A hands-on Databricks Lead Engineer is required to lead the delivery of a modern enterprise data platform for a leading financial services organisation.
The role will take technical ownership of Databricks lakehouse solutions, covering ingestion through to curated serving layers. Responsibilities will include implementing Medallion Architecture, Unity Catalog, Declarative Pipelines and scalable Databricks-native ingestion patterns while mentoring engineers and driving production-quality delivery.
What's on offer Up to £575 per day
Inside IR35
Initial 6-month contract with potential extension
Hybrid working with three days per week on-site in London
ASAP start
Technical leadership of a strategic Databricks implementation
Opportunity to shape engineering standards, reusable patterns and delivery practices
What you need At least two years of recent, hands-on Databricks experience
Evidence of delivering Databricks solutions into production environments
An active Databricks Data Engineering, Machine Learning or Generative AI certification
Strong experience implementing Medallion Architecture across Bronze, Silver and Gold layers
Hands-on Unity Catalog experience covering catalog design, access controls, lineage and secure data sharing
Experience developing Declarative Pipelines with Expectations, including failure handling and observability
Knowledge of Auto Loader, Lakeflow Connect and batch, streaming or incremental ingestion patterns
Production-level PySpark and SQL development skills
Strong knowledge of Delta Lake, including MERGE, schema evolution, OPTIMIZE, ZORDER and VACUUM
Experience with Databricks Workflows, orchestration and CI/CD
Understanding of monitoring, alerting, replay, backfill and operational support strategies
Ability to mentor engineers and act as the technical escalation point
Experience with RBAC, data masking, Photon, cluster optimisation or cost management would be beneficial
Exposure to Mosaic AI, Vector Search, model serving, RAG or Lakebase would be advantageous
AMRT1_UKTJ
...
Databricks Lead Engineer - Hybrid - Contract in East Kilbride employer: ANSON MCCADE
Anson McCade is an exceptional employer that prioritises employee well-being and professional growth, offering a flexible remote working environment in the UK. With a strong focus on security and digital transformation, employees benefit from comprehensive training programmes and clear pathways for career advancement, all while contributing to impactful projects alongside talented teams and senior stakeholders.
