Senior Data Engineer

This is a broad, high-impact engineering role at Defaqto. You will be central to the structural transformation of how we manage and use market data - leading the technical build of a new core data model - while also owning the engineering that powers our research team's day-to-day operations and the wider data strategy.

We are at a pivotal moment in Defaqto's data journey. Our market data estate spans thousands of financial products, maintained by a team of 30+ researchers, and is being redesigned from the ground up into a unified market data model, replacing a flat, siloed structure with a scalable, enrichment-ready architecture. This role sits within the Data Team but works across the organisation, partnering and collaborating closely with Research & Insights, Product, and IT. The successful candidate will help shape and build our next-generation data platform, contributing to architectural and tooling decisions alongside the CTO, Head of Data, and external data architect. Google BigQuery is the current lakehouse platform, but key tooling choices are still being defined. We are looking for an experienced engineer with strong technical judgement who can turn strategy into scalable, production-ready solutions. Alongside platform delivery, this person will lead the development and evolution of our research automation capability and play a key role in Defaqto’s wider data and automation strategy. The initial focus will be implementing the agreed data model and technology approach while progressing automation initiatives in parallel.

Data Model - Technical Build

• Implement the physical data model based on the agreed logical design and technology recommendations agreed during the architecture phase.
• Build and maintain the transformation layer in the agreed tooling: staging models, mart models, tests, and documentation.
• Design and implement compatibility views that allow existing research tooling to continue operating during transition.
• Own the data quality framework - tests, monitoring, and alerting across the data model and downstream consumers.
• Support the phased migration of the research data entry platform to write natively to the new schema.
• Contribute engineering perspective to ongoing architecture decisions as the data model evolves beyond the initial scope.

Research Automation & Engineering

• Assess automation opportunities across research and data workflows and recommend the appropriate technical approach - whether rules-based, deterministic, or AI-assisted - based on the specific problem, data characteristics, and accuracy requirements.
• Build and maintain automation pipelines that reduce manual research effort, selecting from the appropriate toolset including structured transformation, rules engines, and machine learning or LLM-based approaches where warranted.
• Apply LLM and AI tooling to problems where unstructured data, variable formats, or language ambiguity make deterministic approaches insufficient.
• Ensure automation outputs - regardless of the method used - flow into the data model with appropriate quality controls and human-review checkpoints where confidence thresholds require it.
• Be the primary technical partner for the Research & Insights team's data engineering needs.
• Stay current with developments across the automation and AI engineering space and apply them with judgement, not by default.
• Work with Research & Insights and Product stakeholders to identify high-value automation opportunities and maintain a prioritised delivery roadmap.

Data Strategy Implementation

• Contribute to and implement Defaqto's broader data strategy across the data estate.
• Support the integration of acquired datasets (e.g. pricing and claims data) into the data model framework.
• Build and maintain the data catalogue and lineage documentation.
• Work toward an enrichment-ready data architecture that combines product attributes, performance, behaviour, and experience data.
• Participate in evaluating and adopting new data tooling as the stack evolves.

The opportunity

• A genuine opportunity to shape how a data-rich business transforms its core data infrastructure.
• Broad scope - you will work across engineering, strategy, and automation rather than a narrow specialist track.
• Collaborative team environment with a clear data strategy and leadership backing.
• Hybrid working with flexibility.

What you'll need to succeed

Essential requirements

• Strong hands-on experience with SQL and data modelling for analytical workloads (OLAP).
• Proficiency with dbt and familiarity with the broader SQL transformation layer ecosystem.
• Experience with Google BigQuery or an equivalent cloud data warehouse.
• Proven ability to design and implement data pipelines and ETL/ELT processes.
• Experience working with version control (Git) and treating data infrastructure as code.
• Proficiency with Python for data engineering and automation work.
• Experience assessing and implementing automation solutions.
• Practical experience working with LLMs or AI APIs in an engineering context.
• Ability to translate business rules and logic into reliable, testable data models.
• Strong communication skills - able to work with non-technical stakeholders including researchers and product teams.

Desirable requirements

• Experience in financial services, insurance, or fintech data environments.
• Familiarity with data quality frameworks such as Great Expectations or Soda.
• Experience with Google Cloud Platform services beyond BigQuery.
• Exposure to .NET or relational SQL Server environments.
• Understanding of data cataloguing, lineage, and metadata management.
• Experience applying ML or LLM-based approaches to document extraction, classification, or data enrichment problems.
• Familiarity with LLM orchestration frameworks.
• Familiarity with semantic or metrics layers.

Your approach to work: Engineering Standards & Ways of Working

• Champion engineering best practices across the data team: version control, code review, testing, documentation.
• Ensure all data models and pipelines are maintainable, observable, and well-documented.
• Contribute to defining and upholding data engineering standards across the organisation.

Important to know

Location

This is a hybrid role where you'll work from the London office 3 times each week.

Right to Work

Applicants must already hold a legal right to work in the UK without time restrictions and without the need for future sponsorship. We are unable to provide Skilled Worker visa sponsorship.