At a Glance
- Tasks: Design and maintain scalable data pipelines using Databricks and PySpark.
- Company: Join a leading tech firm focused on innovative data solutions.
- Benefits: Attractive salary, flexible working options, and continuous learning opportunities.
- Other info: Collaborative environment with a focus on career advancement.
- Why this job: Make an impact by modernising data pipelines and optimising performance.
- Qualifications: 8+ years in data engineering with strong PySpark and Databricks skills.
The predicted salary is between 70000 - 90000 £ per year.
We are looking for an experienced Senior Data Engineer – Databricks to design, develop, and maintain scalable data pipelines on the Databricks platform. The role requires strong expertise in PySpark, Databricks, and modern data engineering practices, along with experience modernizing legacy data pipelines and building high-performance data solutions in cloud environments.
Key Responsibilities
- Design, build, and maintain scalable data pipelines using Databricks and PySpark.
- Develop end-to-end data workflows including data ingestion, transformation, and consumption.
- Optimize Spark jobs, cluster configurations, and pipeline performance.
- Manage and orchestrate workflows using Databricks Jobs and notebooks.
- Refactor legacy ETL pipelines into modern PySpark-based ELT frameworks.
- Implement data quality checks, monitoring, and error handling mechanisms.
- Design and maintain Delta Lake tables with proper optimization strategies.
- Collaborate with data architects, analysts, and infrastructure teams to deliver reliable data solutions.
- Troubleshoot and resolve production data pipeline issues.
Required Skills & Qualification
- Strong Data Engineering fundamentals including ETL/ELT pipeline design.
- Hands-on experience with PySpark (DataFrames API, Spark SQL, performance tuning).
- Experience with Databricks platform including workspace, clusters, notebooks, and job orchestration.
- Knowledge of Delta Lake features such as ACID transactions and schema evolution.
- Strong Python programming for data processing and automation.
- Experience with data modelling (Dimensional modeling, Data Vault, or Lakehouse architecture).
- Strong SQL skills for data transformation and analysis.
- Experience with cloud platforms (Azure, AWS, or GCP).
- Experience with Git version control and CI/CD practices.
- Minimum 8 years of experience in data engineering or related roles.
- 2–3 years of hands-on experience with Databricks.
- Experience building and maintaining production-grade data pipelines at scale.
- Proven experience modernizing legacy data pipelines.
- Experience working in Agile development environments.
- Certifications (Mandatory) Databricks Certified Data Engineer Associate OR Databricks Certified Data Engineer Professional.
Senior Data Engineer – Databricks employer: ELLIOTT MOSS CONSULTING PTE. LTD.
Join a forward-thinking company that values innovation and collaboration, where as a Senior Data Engineer – Databricks, you will have the opportunity to work with cutting-edge technologies in a dynamic environment. Our commitment to employee growth is reflected in our continuous learning culture, offering ample opportunities for professional development and career advancement. Located in a vibrant area, we provide a supportive work culture that prioritises work-life balance and fosters creativity, making us an excellent employer for those seeking meaningful and rewarding employment.
Contact Details:
ELLIOTT MOSS CONSULTING PTE. LTD. Recruitment Team
