Data Engineer in Cardiff

Duration of booking: Expected to last till March 2027 with possible extension

Proposed start date: ASAP

Pay Rates: Up to£28 per hour Umbrella or £26 per hour PAYE inclusive of holidaypay

Hours / Working Days: 37.5 hours per week / Monday to Friday, 9am – 5pm

Based: Office / Hospital

About the Role

We are seeking an Advanced Data Engineer to play a pivotal role in supporting the NHS transition to cloud computing, specifically driving data platform modernization for the Communicable Disease Surveillance Centre (CDSC) within Public Health Wales.

In this role, you will design, build, and optimize scalable data pipelines that support national disease surveillance and public health intelligence. You will lead the migration of mission-critical surveillance data platforms from on-premises infrastructure to Google Cloud Platform (GCP), ensuring seamless data orchestration, rigorous DataOps practices, and adherence to NHS data security and governance standards.

Key Responsibilities

• Data Engineering & Cloud Architecture: Design, build, test, and maintain scalable ETL/ELT pipelines and production-ready data products. Lead the migration of legacy data platforms, pipelines, and services from on-premises infrastructure to GCP.
• Pipeline Orchestration & Optimization: Monitor, troubleshoot, and optimize pipeline performance for large-scale datasets using distributed processing tools (Spark, BigQuery) and workflow orchestrators (Airflow, Cloud Composer, Dagster).
• Software & DataOps Best Practices: Apply CI/CD, containerization (Docker), Git-based version control, and automated testing to build production-quality code. Conduct peer code reviews, refactor existing code, and maintain robust technical documentation.
• Data Modeling & Governance: Develop conceptual, logical, and physical data models. Implement metadata management, data lineage, data profiling, and validation frameworks to maintain high data quality. Ensure full compliance with UK GDPR, the Data Protection Act 2018, and ethical data guidelines.
• Collaboration & Delivery: Work in an Agile environment to translate non-technical business requirements into robust technical solutions. Communicate architectural and data concepts clearly to both technical peers and key business stakeholders.

Requirements & Qualifications

• Technical Core: Strong proficiency in SQL and Python; working knowledge or experience with R is highly desirable.
• Cloud & Infrastructure: Hands-on experience designing cloud-native solutions (preferably GCP), migrating on-premise systems to the cloud, and managing relational databases/data warehouses.
• Data Engineering Stack: Demonstrated experience with big data processing (Spark, Dataproc, BigQuery), orchestration (Airflow/Cloud Composer/Prefect), and API-driven data integration.
• DevOps & Quality: Proven background with Git (GitHub/GitLab/Azure DevOps), Docker, CI/CD automation, and modern DataOps principles.
• Education & Certifications:
• Undergraduate degree in Computer Science, a related technical field, or equivalent practical work experience.
• GCP Data Engineer or GCP Cloud Architect certification is highly desirable.