Data Engineer (Azure Synapse Analytics, PySpark)

About Sembcorp

Sembcorp is a leading energy and urban solutions provider headquartered in Singapore. Led by its purpose to drive energy transition, Sembcorp delivers sustainable energy solutions and urban developments by leveraging its sector expertise and global track record.

Purpose and Scope

We are seeking a highly skilled and self-driven Azure Data Engineer with expertise in PySpark, Python, and modern Azure data services including Synapse Analytics and Azure Data Explorer. The ideal candidate will design, develop, and maintain scalable data pipelines and architectures, enabling effective data management, analytics, and governance.

Key Roles and Responsibilities

• Design, develop, and maintain scalable and efficient data pipelines (both batch and real-time streaming) using modern data engineering tools.
• Build and manage data lakes, data warehouses, and data marts using Azure Data Services.
• Integrate data from various sources including APIs, structured/unstructured files, IoT devices, and real-time streams.
• Develop and optimize ETL/ELT workflows using tools such as Azure Data Factory, Databricks, and Apache Spark.
• Implement real-time data ingestion and processing using Azure Stream Analytics, Event Hubs, or Kafka.
• Ensure data quality, availability, and security across the entire data lifecycle.
• Collaborate with analysts, data scientists, and engineering teams to deliver business-aligned data solutions.
• Contribute to data governance efforts and ensure compliance with data privacy standards.
• Establish and manage source system connectivity (on-prem, APIs, sensors, etc.).
• Handle deployment and migration of data pipeline artifacts between environments using Azure DevOps.
• Design, develop, and troubleshoot PySpark scripts and orchestration pipelines.
• Perform data integration using database joins and other transformations aligned with project requirements.
• Any assigned ad-hoc duties.

Requirements:

• Bachelor’s Degree in Computer Science, Engineering, or related field.
• 3–5 years of experience in Azure-based data engineering, PySpark, and Big Data technologies.
• Strong hands-on experience with Azure Synapse Analytics for pipeline orchestration and data handling.
• Expertise in SQL, data warehousing, data marts, and ingestion using PySpark and Python.
• Solid experience building and maintaining cloud-based ETL/ELT pipelines, especially with Azure Data Factory or Synapse.
• Familiarity with cloud data environments such as Azure and optionally AWS.
• Experience with Azure DevOps for CI/CD and artifact deployment.
• Excellent communication, problem-solving, and interpersonal skills.

Good to Have:

• 1–2 years of experience working with Azure Data Explorer (including row-level security and access controls).
• Experience with Azure Purview for metadata management, data lineage, governance, and discovery.
• Ability to work independently and take full ownership of assignments.
• Proactive in identifying and resolving blockers and escalating when needed.
• Exposure to real-time processing with tools like Azure Stream Analytics or Kafka.

Our Culture at Sembcorp

At Sembcorp, our culture is shaped by a strong set of shared behaviours that guide the way we work and uphold our commitment to driving the energy transition. We foster an institution-first mindset, where the success of Sembcorp takes precedence over individual interests. Collaboration is at the heart of what we do, as we work seamlessly across markets, businesses, and functions to achieve our goals together. Accountability is a core principle, ensuring that we take ownership of our commitments and deliver on them with integrity and excellence. These values define who we are and create a workplace where our people can thrive while making a meaningful impact on driving energy transition. Join us in making a real impact!