Lead Data Engineer in London

Must be DV Clearable - 5 days a week on-site.

Hands‐on technical leadership building data systems that actually work. We're looking for a Lead Data Engineer to join an enterprise data engineering capability and play a critical role in designing, building, and improving real‐world data systems. This is not a line‐management role. Instead, it's a senior, hands‐on technical leadership position for someone with deep data engineering experience who leads through delivery, judgement, and technical credibility rather than hierarchy.

You'll work in a complex environment spanning legacy platforms, modern cloud services, and evolving data needs. The challenge isn't greenfield design - it's improving what exists, fixing what's broken, and helping teams use data more effectively and responsibly. This role suits someone who enjoys rolling up their sleeves, tackling difficult problems, and seeing things through end‐to‐end.

What you'll be doing:

• Act as a senior technical anchor for data engineering work, contributing directly while guiding others through example.
• Design, build, and enhance enterprise‐scale data pipelines, platforms, and data services.
• Lead complex data engineering work end‐to‐end - from problem definition and exploration through build, testing, and operational use.
• Work directly with technical and non‐technical stakeholders to translate real operational problems into effective data solutions.
• Analyse and synthesise complex datasets from multiple sources, shaping how data is structured, validated, and consumed.
• Provide technical leadership and direction to data engineers and adjacent roles through design decisions, code, and delivery support.
• Improve data sharing, onboarding of new data sources, and interoperability across teams and systems.
• Introduce and evaluate new tools, patterns, and technologies where they genuinely add value - balancing innovation with pragmatism.
• Build robust, production‐grade solutions, not proofs of concept.
• Treat metadata, lineage, and data quality as first‐class engineering problems.
• Diagnose and resolve complex data and platform issues, often in constrained or imperfect environments.
• Contribute to an inclusive, supportive engineering culture with high technical standards.

What we're looking for:

This role is for a genuine practitioner who enjoys doing the work and raising standards around them.

Essential experience:

• Significant, hands‐on experience as a Senior or Lead Data Engineer working on complex, real‐world systems.
• Strong Python and Spark skills, with evidence of building and maintaining production data pipelines.
• Deep understanding of the full data engineering lifecycle: ingestion, transformation, storage, serving, and reuse.
• Strong experience designing data integrations, including working with diverse data sources and legacy systems.
• Proven ability to design and apply data models that support analysis, operational use, and long‐term maintainability.
• Solid understanding of data security, compliance, and governance, built into systems by design.
• Experience working within established data development practices (version control, testing, deployment, change management).
• Ability to communicate clearly between highly technical teams and non‐technical stakeholders.
• Persistence and resilience when dealing with messy data, legacy constraints, and organisational complexity.

Desirable experience:

• Experience with cloud‐based data ecosystems (including AWS services) alongside existing legacy platforms.
• Exposure to large‐scale enterprise data or analytics platforms.
• Rotating across teams or problem domains, adapting to different delivery contexts.
• Familiarity with emerging or advanced data engineering approaches (including agentic or autonomous patterns), though this is not essential.

If you enjoy solving hard data problems, working closely with people, and delivering practical solutions in imperfect environments, this role offers both challenge and impact. If interested, apply now!