Staff Data Engineer in London

Requirements

• Proven experience operating at staff level (ownership of systems, not just pipelines)
• Experience building and scaling modern data platforms
• A track record of operating at staff or principal level: you've owned systems, shaped technical direction across teams, and influenced how engineering gets done — not just delivered pipelines
• Deep experience building and scaling production data platforms, including high-ingestion time‑series workloads, and strong hands‑on ability in Python and modern data stack components (orchestration, warehousing, observability)
• The ability to design for reliability and scale — you understand the trade‑offs in data system design and have made consequential architecture decisions you can speak to clearly
• A product mindset: you care about whether the data is actually useful and used, not just whether the pipeline ran green
• Experience with cloud data infrastructure (AWS or GCP) and a point of view on what good looks like
• The communication skills to lead without authority — influencing technical direction across teams and making the case for the right thing even when it's harder
• Strong programming skills in Python, with experience building production‑grade data systems
• Experience with modern data stack components (e.g.):

• Orchestration: Airflow / Dagster
• Warehousing: Snowflake / BigQuery / Redshift / ClickHouse
• Streaming (nice to have): Kafka / Flink

What the job involves

We’re looking for a Staff Data Engineer to lead the design and evolution of our data platform. This is a high‑impact, hands‑on role combining technical leadership, system architecture, and product thinking. You’ll work closely with engineering, data science, and energy domain experts to ensure that data is reliable, scalable, and directly drives business value. You’ll work across the data management service team alongside data and analytics engineers, and in close partnership with energy domain experts, data scientists, and the broader engineering organisation. This is a hands‑on senior technical leadership role — you’ll be reviewing pull requests and setting architectural direction in the same week.

What makes this genuinely different: you're not inheriting someone else's vision of what a data platform should be. The cultural norms, the standards, the practices — these are yours to define. If you've wanted to build the right thing from the ground up, this is that opportunity.

Technical Leadership

• Shape the technical direction across batch and streaming pipelines, setting the architecture others build to
• Set standards for pipeline design and data quality
• Lead design reviews and mentor other data engineers
• Evaluate and introduce tooling where it raises the bar — and make the case for when it doesn’t

Data Platform & Pipelines

• Build and maintain robust ETL/ELT pipelines
• Build systems optimised for high‑ingestion, low‑latency querying of time‑series data (TSDS)
• Optimise pipelines for performance, cost, and reliability
• Enable self‑serve analytics and decision‑making across the company

Reliability and observability

• Implement data quality frameworks with real teeth: SLAs, automated testing, lineage, and monitoring
• Establish practices specific to energy data: late arrivals, reprocessing, backfills, and the failure modes that matter in this domain
• Build the observability layer that makes the platform trustworthy without constant human oversight

Scale and performance

• Identify and fix the bottlenecks that constrain us today
• Optimise pipelines for performance, cost, and reliability as data volumes grow
• Architect for the next order of magnitude, not just the next quarter

Technical leadership and culture

• Set engineering standards for pipeline design, data quality, and system observability
• Lead design reviews and mentor data engineers, raising the bar for how the team works
• Act as a multiplier: the people around you should get better because of how you approach problems

What Success Looks Like:

• The data platform handles the team's current scale without firefighting, and is architected for the next phase of growth
• Other teams can access, trust, and use data without routing requests through the data engineering team
• There is a tight, reliable feedback loop between data ingestion and consumption: trading, forecasting, and analytics teams make faster decisions because the data is there when they need it
• The data engineering team has clearer standards, better practices, and higher output than when you arrived