At a Glance
- Tasks: Lead the engineering of Databricks lakehouse solutions and mentor fellow engineers.
- Company: Join a forward-thinking tech company focused on modern data platforms.
- Benefits: Competitive salary, flexible working options, and opportunities for professional growth.
- Other info: Dynamic team environment with a focus on innovation and collaboration.
- Why this job: Be at the forefront of data engineering and make a significant impact in the tech world.
- Qualifications: Must have recent Databricks experience and relevant certifications.
The predicted salary is between 70000 - 90000 £ per year.
Data Engineering Consultant (Snr/Lead) We are hiring a hands‑on Databricks Engineer that has experience delivering modern data platforms on Databricks.
This role requires a minimum of 2 years of Databricks experience gained recently (i. e., current/recent projects using modern platform capabilities).
Key Responsibilities Engineering delivery of Databricks lakehouse solutions from ingestion to curated serving layers.
Define and implement Medallion Architecture (Bronze/Silver/Gold) and reusable engineering patterns.
Build scalable ingestion pipelines using Auto Loader, Lakeflow Connect, batch/streaming, and incremental patterns.
Develop Declarative Pipelines with Expectations (DLT) to enforce and monitor data quality, and implement and operate Unity Catalog for governance, access control, lineage, and secure data sharing patterns.
Drive code quality and operational excellence (CI/CD approach, testing strategy, monitoring, incident triage).
Partner with architects, platform teams, and stakeholders to align delivery with enterprise standards.
Mentor engineers and act as the technical escalation point during delivery.
Minimum Experience (Filter Criteria) Hands‑on Databricks experience in recent years (e. g., within the last 2–3 years), demonstrating usage of modern Databricks capabilities and patterns.
Evidence of production delivery (not training‑only or lab‑only exposure).
Must Have (Non‑Negotiable) Databricks certification, at least one of: Databricks Certified Data Engineer Associate/Professional OR Databricks Certified Machine Learning Associate/Professional OR Databricks Certified Generative AI Engineer (Associate).
Unity Catalog hands‑on experience: metastore/catalog design, grants, lineage, and secure access patterns.
Declarative Pipelines with Expectations (DLT): building pipelines, defining expectations, handling failures/quarantines, observability.
Ingestion engineering using Databricks native approaches: Auto Loader and/or Lakeflow Connect, streaming and incremental ingestion patterns.
Medallion Architecture implementation and best practices: designing and implementing Bronze/Silver/Gold with practical decisions (schema evolution, CDC/upserts, SCD patterns, performance strategy).
Should Have Demonstrable use of latest Databricks capabilities (candidate can explain what they used recently and why).
Strong Databricks engineering fundamentals.
Delta Lake (MERGE, schema enforcement/evolution, OPTIMIZE/ZORDER, VACUUM).
Databricks Workflows / job orchestration.
Production‑grade Py Spark/SQL.
Clear understanding of pipeline reliability: observability, alerting, replay/backfill strategies, and operational runbooks.
Nice to Have Lakeflow ingestion connectors (specific connector experience is a plus).
RBAC / masking implementations (row‑level security, column masking, sensitive data handling) using Unity Catalog.
Gen AI on
Databricks: Mosaic AI, Vector Search, model serving, RAG pipelines, AI Functions.
Lake Base awareness or hands‑on experience.
Workload/query optimization: Photon usage, cluster sizing, shuffle/skew mitigation, caching strategy, partitioning, file sizing.
Cost awareness and controls: understanding DBU drivers, job vs. all‑purpose compute, cluster policies, monitoring and chargeback/showback patterns. #J-18808-Ljbffr
