Data lake Engineer in City of Westminster

We are excited to offer a fantastic permanent Data Lake Engineer opportunity offering Hybrid working with travel up to once a week across one of Amey’s offices in Birmingham, London and Liverpool. HOURS OF WORK – Monday – Friday 37.5 Hours Join our vibrant, inclusive community in Group IT, lead and scale our Azure-based data platform. You will own the end-to-end design and optimisation of high-performance, cloud-native data pipelines – ensuring secure, cost-efficient delivery of high-quality datasets across our medallion architecture.

Responsibilities

• Lead the design and implementation of distributed data processing pipelines (batch and streaming) using PySpark/Spark and Python on Azure Databricks.
• Architect and evolve our medallion data platform, ensuring data quality, observability and reliability across all layers (Bronze, Silver, Gold).
• Define and enforce strong data quality controls, validation rules and monitoring strategies to maintain trusted, production‑grade datasets.
• Make technical decisions on cluster strategy and cost optimisation (e.g., interactive vs. job clusters, autoscaling, serverless, pools, spot instances).
• Build and operationalise data workflows using Databricks Workflows, Azure Data Factory, or Fabric, ensuring efficient orchestration, scheduling and alerting.
• Drive API‑based data integrations, ensuring secure, scalable and low‑latency ingestion of data from third‑party and internal systems.
• Champion DevOps standards across the team (CI/CD, TDD, Infrastructure‑as‑Code, Git‑flow, Azure DevOps/GitHub).
• Embed and enforce security and governance best‑practices, including data masking, encryption and RBAC throughout the ETL lifecycle.
• Mentor junior engineers and contribute to technical standards and architecture roadmaps.
• Engage with data scientists, analysts and business stakeholders to deliver high‑value, production‑ready data products.

Qualifications

• Ability to drive technical direction and influence architectural decisions.
• Strong stakeholder management and communication skills.
• Mentoring mindset, collaborative and proactive.
• Operates with a product mindset and ownership mentality.
• Thrives in a fast‑moving environment with ambiguity and growth.
• Extensive experience working with large‑scale data engineering solutions in public cloud (Azure preferred).
• Deep hands‑on experience with Databricks, ADLS, Delta Lake and distributed compute paradigms.
• Prior ownership of mission‑critical production pipelines, including data quality monitoring, automated alerting and remediation.
• Strong track record of cost and performance optimisation at scale, including cluster tuning and workload refactoring.
• Experience with leading orchestration tools (e.g. Databricks Workflows, ADF, Airflow).
• Proven experience integrating APIs and external data services into data platforms.
• Proficiency in SQL, Python, PySpark and software engineering best practices.
• Experience in DevOps, CI/CD, infrastructure automation and data security.
• Exposure to MLOps, ML model deployment or LLM‑based pipelines is a plus.

Benefits

• Remuneration – Enjoy a competitive annual salary with the potential for yearly reviews to ensure you’re rewarded for your contributions.
• Career Growth – Shine in your career with advancement opportunities.
• Training Opportunities – Unlock your potential with comprehensive training, including fully funded leadership programs tailored to your personal growth.
• Holidays – Enjoy at least 24 days of holiday plus bank holidays, and the opportunity to buy further days.
• Pension – Generous pension scheme, with extra contributions from Amey.
• Flexible benefits – Customise your benefits with options such as insurance benefits, Cycle2Work scheme and access to discounted gym membership.
• Exclusive Discounts – Access our online portal filled with discounts from leading retailers, healthcare services, and more, helping you save on the things that matter.
• Give Back to community – Two Social Impact Days each year, for volunteering and fundraising opportunities.
• Family friendly policies for new parents or if you provide care for a dependant.
• Membership of our Affinity Networks who connect, support and inspire diverse communities within Amey.