Senior Data Engineer in Biggleswade

You operate at the frontier of modern data engineering. You understand that AI is not a future consideration - it is a present‑day design constraint. You build data infrastructure that is AI‑ready by default: pipelines that serve feature stores, architectures that can support RAG and LLM applications, and platforms capable of integrating AI‑assisted tooling at every stage of the engineering lifecycle. In a global team spanning Europe, the US, and India, you are a connector – bridging technical depth with business context, and aligning local delivery with global standards.

Technical Architecture & Delivery

• Lead the end‑to‑end design and delivery of complex data engineering solutions on GCP – from architecture through production deployment.
• Architect scalable, cost‑effective data platforms using BigQuery, Dataflow, Cloud Composer, Pub/Sub, Dataplex, and Cloud Storage.
• Design robust data models using Dimensional (Kimball), 3NF, and Data Vault methodologies – selecting the right approach for each use case.
• Implement SCD strategies and historical data management patterns for long‑lived datasets.
• Lead the migration of legacy data structures to GCP, defining parallel testing and data parity validation strategies.
• Provision and govern cloud infrastructure using Terraform; champion IaC as a non‑negotiable standard.
• Design and implement CI/CD pipelines for all data solutions – with automated testing, linting, and deployment gates.

AI‑Era Responsibilities

• AI‑ready architecture: Design every data platform component to be downstream‑AI‑compatible – appropriate partitioning, feature store integration, and schema design for ML consumption.
• GenAI data infrastructure: Architect data pipelines for LLM‑based applications, including embedding generation pipelines, vector store population, and RAG data retrieval layers.
• Feature store engineering: Build and maintain centralised feature stores on Vertex AI, ensuring reproducibility and low‑latency serving for ML models.
• AI‑assisted development leadership: Champion GitHub Copilot, Gemini Code Assist, and Cursor as engineering productivity tools – set standards for how the team uses them responsibly.
• AI‑powered data quality: Design ML‑based anomaly detection into pipeline monitoring – moving beyond threshold alerts to intelligent pattern recognition.
• LLMOps data layer: Build the data infrastructure that underpins model evaluation, fine‑tuning dataset curation, and prompt tracking pipelines.

Leadership & Collaboration

• Lead code reviews; hold the bar for quality, testability, and maintainability.
• Define and document reusable engineering patterns – pipeline templates, transformation standards, naming conventions.
• Actively mentor junior engineers through pairing, structured feedback, and technical design sessions.
• Work closely with global Data Engineering counterparts to align on platform standards.
• Engage directly with senior business stakeholders to translate complex requirements into technical solutions.
• Contribute to hiring: review take‑home tasks, conduct technical interviews, calibrate assessments.
• Define and execute testing strategies for regulated workloads, including parallel‑run validation against legacy systems.

Operational Excellence

• Own pipeline reliability: define SLAs, implement alerting, lead incident resolution.
• Drive DataOps practices: automated testing, data contracts, observability‑first design.
• Monitor and optimise GCP costs; propose and implement efficiency improvements.
• Ensure compliance with data security, encryption, and governance standards in all solutions built.
• Collaboration with global engineering teams across three continents.
• Support and funding for GCP Professional Data Engineer certification and advanced training.
• A clear pathway to Lead Engineer for the right candidate.
• Hybrid working from a modern campus environment.

Qualifications & Experience

• 5+ years of data engineering experience in production, cloud‑native environments.
• 5+ years of advanced SQL – BigQuery specifics, query profiling, partitioning/clustering optimisation, complex analytical queries.
• 3+ years of GCP production experience: architecture design and delivery at scale.
• Deep, hands‑on expertise across: BigQuery, Dataflow, Cloud Composer (Airflow), Pub/Sub, Dataplex, Cloud Storage, Terraform, Cloud Build.
• Mastery of data modelling methodologies: Dimensional/Kimball, 3NF, Data Vault – with real‑world application of each.
• Production‑level Python: OOP design patterns, async processing, unit/integration testing, GCP SDK usage.
• Demonstrated experience designing CI/CD pipelines for data products.
• Track record of leading legacy‑to‑cloud migrations.
• Demonstrated technical leadership: you have designed solutions, led reviews, and raised the quality bar of a team.
• Proven ability to work in high‑ambiguity environments and drive clarity through technical design.
• Strong communication: able to write architecture decision records, run design reviews, and present to non‑technical stakeholders.
• Evidence of mentoring junior engineers and improving team capability.
• Minimum 2:2 degree (or international equivalent) in Computer Science or related technical field; demonstrated professional experience considered in lieu for internal applicants.

Desired

• GCP Professional Data Engineer certification.
• Experience designing AI/ML data pipelines – feature stores, training data pipelines, Vertex AI integration.
• Hands‑on experience with vector databases or embedding pipeline design.
• Active use of AI‑assisted development tools (Copilot, Gemini, Cursor) in production delivery.
• Experience with dbt Core / Dataform in a production, team setting.
• Data engineering experience in a regulated financial environment (banking, insurance, credit).
• Experience designing event‑driven architectures with Pub/Sub and Dataflow.

Location & Working Arrangement

This position is based in Dunton and is expected to attend the Dunton Campus for typically 4 days a week, with flexibility on days required to be in office based on business requirements.

EEO Statement

The Company is committed to diversity and equality of opportunity for all and is opposed to any form of less favourable treatment or harassment on the grounds of race, religion or belief, sex, marriage and civil partnership, pregnancy and maternity, age, sexual orientation, gender reassignment or disability.