Inviting applications for the role of Lead Data Engineer
In this role, you will design and optimize scalable data pipelines on Azure Databricks using PySpark and Spark SQL, implementing distributed transformations and architecting Delta Lake tables aligned to Medallion (Bronze/Silver/Gold) principles. Develop modular frameworks and orchestrated workflows with incremental processing patterns while ensuring performance optimization, data quality governance, operational logging, and successful migration of legacy workloads into Spark-native solutions. Onsite requirement for 3 to 4 days in London office of client.
Responsibilities
- Advanced proficiency in PySpark and Spark SQL
- Complex DataFrame transformations
- Window functions and advanced aggregations
- Distributed joins and performance tuning
- Error handling and logging patterns
- Strong expertise in Delta Lake
- ACID transactions
- Schema enforcement and evolution
- Time travel and versioning
- MERGE and incremental data processing strategies
- Hands‑on experience with Azure Databricks
- Notebook structuring and modular code design
- Databricks Workflows (job orchestration and task chaining)
- Cluster configuration and optimization
- Integration with Azure storage (ADLS Gen2)
- Solid understanding of data modeling and Lakehouse architecture
- Medallion architecture implementation
- Partitioning and optimization strategies
- Familiarity with CI/CD practices in Databricks environments
- Demonstrate thought leadership and engage with business users such as product owners in determining a solution
- Critical thinking and solid communication skills
Minimum Qualifications
- Bachelor's Degree required. Preferably in Computer Science, Information Systems, or related field.
Preferred Qualifications
- Experience supporting legacy workload modernization (e.g., ETL migration from tools such as Alteryx) is beneficial but not primary)
Genpact is an Equal Opportunity Employer and considers applicants for all positions without regard to race, color, religion or belief, sex, age, national origin, citizenship status, marital status, military/veteran status, genetic information, sexual orientation, gender identity, physical or mental disability or any other characteristic protected by applicable laws. Genpact is committed to creating a dynamic work environment that values respect and integrity, customer focus, and innovation.
Data Engineer employer: Genpact
Genpact is an exceptional employer that fosters a dynamic work culture in Greater London, where innovation and collaboration thrive. Employees benefit from comprehensive growth opportunities, including professional development programmes and mentorship, ensuring they can advance their careers while contributing to impactful projects. With a focus on AI-driven solutions, Genpact offers a unique environment for those looking to make a meaningful difference in the business landscape.
