Lead Data Migration Engineer

We are seeking an accomplished and detail-oriented Lead Data Migration Engineer to join our Data & AI practice. The successful candidate will bring deep expertise in data migration, ETL/ELT pipeline engineering, and cloud-native data platforms, with a strong focus on AWS-based Data Lakehouse architectures. This role is critical in designing, building, and validating end-to-end data migration pipelines, enabling the seamless transition of data from legacy data warehouses to modern cloud platforms. You will play a key role in ensuring data quality, integrity, and performance, particularly through robust testing and validation of ETL pipelines. As a senior practitioner, you will collaborate with architects, engineers, and analysts to deliver secure, scalable, and high-performing migration solutions, leveraging technologies such as AWS Glue, Apache Iceberg, Python/PySpark, SQL, and YAML-based configurations. You will thrive in a collaborative, client‑facing environment, with a passion for solving complex data challenges, ensuring delivery excellence, and driving modernisation through data migration initiatives.

What you'll be doing:

• Client Engagement & Delivery
  • Work as a senior engineer within data migration programmes, supporting delivery across complex transformation initiatives.
  • Collaborate with architects and stakeholders to implement migration strategies and technical solutions.
  • Contribute to project planning, estimation, and execution of migration deliverables.
• Data Migration Engineering
  • Design, build, and optimise data migration pipelines from legacy data warehouses to AWS-based Data Lakehouse platforms.
  • Develop and maintain ETL/ELT pipelines using: AWS Glue, Python / PySpark, SQL, YAML-driven configurations.
  • Implement bulk data migration processes followed by incremental/delta loads.
  • Support pipeline repointing and replatforming to target cloud architectures.
• Data Pipeline Testing & Validation (Core Focus)
  • Test end-to-end ETL data solutions running on AWS services, including AWS Glue and Apache Iceberg.
  • Validate data feed pipeline migrations transitioning to new AWS Data Lakehouse architectures.
  • Test pipelines underpinned by: AI ETL accelerators, Python/PySpark transformations, SQL-based logic, YAML configuration-driven orchestration.
  • Execute and validate: Initial bulk data loads.
  • Develop and run SQL-based validation and reconciliation queries to ensure: Transformation correctness.
  • Support creation of automated testing and validation frameworks.
• AWS Data Platforms & Lakehouse
  • Work hands‑on with AWS cloud services, including: AWS Glue, S3-based data lakes.
  • Support implementation of Data Lakehouse architectures, including Apache Iceberg.
  • Optimise data pipelines for performance, scalability, and reliability in cloud environments.
  • Apply data transformation logic aligned to migration mapping rules.
  • Support data modelling activities required for target-state schemas.
  • Ensure consistency between source and target systems during migration.
  • Collaborate with: Solution Architects, Data Migration Architects, Analysts and QA teams.
  • Promote engineering best practices, reusable components, and automation.
  • Contribute to migration accelerators and reusable frameworks.
• Quality, Governance & Security
  • Ensure data integrity, quality, and traceability throughout migration processes.
  • Follow best practices for secure data handling in regulated environments.
  • Support compliance with: GDPR, UK public sector data requirements (where applicable).
  • Contribute to audit and validation processes.

What experience you'll bring:

• Proven experience in data engineering and data migration delivery, particularly within cloud environments.
• Strong focus on data pipeline testing, validation, and quality assurance.
• Ability to work across the full data lifecycle, with emphasis on migration and transformation.
• Strong analytical, problem-solving, and communication skills.
• Experience working in client‑facing and delivery-focused environments.
• Ability to mentor junior engineers and contribute to team delivery.

Technical Expertise

• Strong hands‑on experience with: AWS cloud services, especially AWS Glue.
• SQL for querying, validation, and reconciliation.
• YAML configuration for pipeline orchestration.
• Experience testing and validating: End-to-end ETL/ELT pipelines.
• Familiarity with: Apache Iceberg and Lakehouse architectures, Distributed processing frameworks (e.g., Apache Spark), Data lake environments.
• Understanding of: Migration performance tuning.
• Experience with version control and CI/CD tools desirable.

What we'll offer you:

We offer a range of tailored benefits that support your physical, emotional, and financial wellbeing. Our Learning and Development team ensures that there are continuous growth and development opportunities for our people. We also offer the opportunity to have flexible work options. We are an equal opportunities employer. We believe in the fair treatment of all our employees and commit to promoting equity and diversity in our employment practices. We are also a proud Disability Confident Committed Employer — we are committed to creating a diverse and inclusive workforce. We actively collaborate with individuals who have disabilities and long‑term health conditions which have an effect on their ability to do normal daily activities, ensuring that barriers are eliminated when it comes to employment opportunities. In line with our commitment, we guarantee an interview to applicants who declare to us, during the application process, that they have a disability and meet the minimum requirements for the role. If you require any reasonable adjustments during the recruitment process, please let us know.