Databricks Lead Engineer
££575 GBP
Hybrid WORKING
Location: Central London, Greater London - United Kingdom Type: Contract
NEW CONTRACT ROLE - DATABRICKS LEAD ENGINEER
A new opportunity has just landed for an experienced, hands-on Databricks Lead Engineer to join a major data transformation programme.
We're looking for someone with strong recent experience delivering modern, production-grade data platforms on Databricks. This is a hands-on technical leadership role, so we're looking for someone who can lead engineering delivery and design solutions, rather than simply work from an existing blueprint.
Location: London - 3 days per week on-site
Rate: Up to £575 per day
Contract: Inside IR35
Duration: 6 months initial contract with potential extension
Start: ASAP
Key requirements:
- 2+ years of recent, hands-on Databricks experience delivering production solutions
- Active/current Databricks certification - Data Engineer, Machine Learning or Generative AI
- Strong experience designing and implementing Databricks Lakehouse architectures
- Hands-on implementation of Medallion Architecture - Bronze, Silver and Gold layers
- Experience with Unity Catalog, including governance, access control, lineage and secure data sharing
- Strong experience building Declarative Pipelines / DLT, including Expectations, data quality, failure handling and observability
- Experience with Databricks-native ingestion approaches such as AutoLoader and/or Lakeflow Connect
- Strong PySpark and SQL skills
- Delta Lake experience, including MERGE, schema evolution, optimisation and performance tuning
- Experience with Databricks Workflows and production pipeline orchestration
- Strong understanding of CI/CD, testing, monitoring, alerting, incident triage and operational best practice
- Ability to mentor engineers and act as the technical escalation point within the delivery team
Nice-to-have experience includes:
- Mosaic AI, Vector Search, model serving or RAG pipelines
- Lakeflow ingestion connectors
- Row-level security, column masking and sensitive data handling through Unity Catalog
- Databricks performance optimisation, including Photon, cluster sizing, partitioning and shuffle/skew mitigation
- Cost optimisation and understanding of Databricks compute and DBU drivers
This is a great opportunity for a technically strong Databricks specialist who wants to take ownership of engineering delivery and help shape a modern enterprise data platform.
If you're interested, or know someone who could be a good fit, please get in touch.
#Databricks #DataEngineering #DataEngineer #Lakehouse #PySpark #DeltaLake #UnityCatalog #DataJobs #ContractJobs #LondonTech #InsideIR35
Reference: AMC-AQU-DLEN
Postcode: EC2V 6AA
#adqu
Databricks Lead Engineer in London employer: All The Top Bananas
Join our Specialist Crime Division as a Cyber Operations Development Specialist, where you will not only contribute to public safety but also thrive in a supportive and dynamic work environment. With competitive salaries, flexible working patterns, and extensive opportunities for professional growth, we prioritise your wellbeing and career progression. Experience the unique advantage of being part of a team that is at the forefront of innovative policing technology, making a real difference in communities across the UK.
