2 Databricks data Engineers
We are seeking 2 Senior Databricks Data Engineers to design, build, and scale a modern Lakehouse platform using Spec-Driven Development (SDD), Apache Airflow, and dbt. In this role, you will translate architectural requirements into reliable production solutions, ensuring data contracts, transformations, and orchestration processes are clearly defined, thoroughly tested, and fully automated.
Job Responsibilities
- Design and develop scalable Databricks data pipelines based on the Medallion Architecture (Bronze, Silver, and Gold layers) using Delta Lake.
- Lead the adoption of Spec-Driven Development (SDD) by creating machine-readable specifications and data contracts before implementation.
- Build, monitor, and optimize workflow orchestration pipelines using Apache Airflow.
- Develop, test, and maintain modular data transformation models with dbt, integrated with the Databricks SQL and Spark ecosystem.
- Optimize PySpark workloads, manage cluster configurations, and control compute and storage costs.
Qualifications
- 5+ years of hands-on data engineering experience, including extensive production experience with Databricks.
- Strong expertise in Apache Airflow for workflow scheduling, dependency management, and error handling.
- Advanced knowledge of dbt modelling, testing practices, and macro development.
- Practical experience with Spec-Driven Development (SDD) and metadata-driven data platform design.
- Expert-level proficiency in Python (PySpark) and SQL.
- Familiarity with Git, CI/CD processes, and infrastructure-as-code practices.
- Multi-Cloud Experience (AWS, Azure, GCP)
- AWS: Configure and manage integrations with Amazon S3, IAM roles, and VPCs to support secure and scalable data platforms.
- Azure: Implement secure data storage using Azure Data Lake Storage Gen2 (ADLS Gen2), integrate with Microsoft Entra ID, and deliver data to analytics platforms such as Power BI.
- GCP: Build and operate data workflows using Google Cloud Storage (GCS) while enabling integration with BigQuery and Vertex AI for downstream analytics and AI use cases.
Company
emagine is a high-end professional services consultancy and solutions firm specialising in providing business and technology services to the financial services sector, we power progress, solve challenges and deliver real results through tailored high-end consulting services and solutions.
We have created a culture of openness and integrity by building genuine and strong relationships and partnerships, enabling us to be uncompromising in our dedication in delivering the optimal service for our clients. Our commitment is not just towards our clients but we aim to foster a positive and equitable working environment with our consultants and colleagues which stems from our core values: Confident, Dedicated, Responsible, Genuine.
#J-18808-Ljbffr
