We are seeking an accomplished and detail-oriented Lead Data Engineer – AWS to join our Data & AI practice. The successful candidate will bring deep expertise in data engineering, distributed data processing, and cloud-native platforms, with a strong focus on AWS-based data ecosystems.
This role is critical in designing, building, and optimising end-to-end data pipelines and platforms, enabling scalable data processing, advanced analytics, and AI-driven solutions. You will play a key role in ensuring data quality, integrity, performance, and reliability, supported by strong engineering and testing practices.
As a senior practitioner, you will collaborate with architects, engineers, and analysts to deliver secure, scalable, and high-performing data solutions, leveraging technologies such as AWS Glue, Python/PySpark, SQL, and configuration-driven frameworks (e.g., YAML). You will thrive in a collaborative, client-facing environment, with a passion for solving complex technical challenges, ensuring delivery excellence, and driving modernisation through cloud-native engineering practices.
What you'll be doing:
Client Engagement & Delivery
- Act as a senior engineer within data engineering and cloud platform initiatives, supporting delivery across complex transformation programmes
- Collaborate with architects and stakeholders to define and implement scalable AWS-based data solutions
- Contribute to solution design, estimation, and delivery planning
- Lead engineering workstreams and ensure high-quality technical delivery
- Design, build, and optimise scalable data pipelines and data processing frameworks on AWS
- Develop and maintain ETL/ELT pipelines using:
- AWS Glue
- Python / PySpark
- SQL
- Configuration-driven frameworks (e.g., YAML)
- Implement robust data ingestion, transformation, and processing patterns
- Build reusable data services, components, and frameworks
- Define and implement testing strategies for data pipelines, ensuring reliability and accuracy
- Python / PySpark transformations
- SQL-based validation logic
- Configuration-driven orchestration
- Develop automated testing, monitoring, and alerting solutions
- Ensure:
- Consistent transformation behaviour
- Drive improvements in observability and pipeline resilience
AWS Data Platforms
- Lead development on AWS services including:
- AWS Glue
- S3-based data lakes
- Supporting services within the AWS data ecosystem
- Support implementation of modern data architectures, including data lakes and Lakehouse-style platforms
- Optimise pipelines and jobs for performance, scalability, and cost efficiency
- Define and implement data transformation logic aligned to business requirements
- Support data modelling approaches for analytics and platform use cases
- Ensure consistency, usability, and quality across data assets and pipelines
Collaboration & Technical Leadership
- Collaborate with:
- Solution Architects
- Analysts and ML engineers
- Provide technical leadership and mentoring to engineers within the team
- Promote engineering best practices, automation, and reusable solutions
- Contribute to engineering standards, documentation, and knowledge sharing
Quality, Governance & Security
- Ensure data quality, integrity, and reliability across data platforms
- Implement and enforce secure coding and data handling practices
- Support compliance with:
- GDPR
- Regulated environment standards (where applicable)
- Contribute to monitoring, auditing, and operational processes
What experience you'll bring
- Proven experience in data engineering and cloud-based platform delivery
- Strong understanding of distributed data processing and scalable system design
- Ability to lead delivery while remaining hands-on technically
- Strong analytical, problem-solving, and communication skills
- Experience working in client-facing and delivery-focused environments
- Ability to mentor and develop engineering teams
Technical Expertise
- Strong hands-on experience with:
- AWS cloud services, especially AWS Glue
- SQL for querying, transformation, and validation
- Experience building and operating:
- Familiarity with:
- Data lakes and Lakehouse concepts
- Distributed processing frameworks (e.g., Apache Spark)
- Strong understanding of:
- ETL vs ELT patterns
- Performance tuning and optimisation
- Experience with:
- Version control (Git)
- CI/CD and DevOps practices
What we can offer you
We offer a range of tailored benefits that support your physical, emotional, and financial wellbeing. Our Learning and Development team ensures there are continuous growth and development opportunities for our people. We also offer the opportunity to have flexible work options.
We are an equal opportunities employer. We believe in the fair treatment of all our employees and commit to promoting equity and diversity in our employment practices.
We are a proud Disability Confident Committed Employer – we are committed to creating a diverse and inclusive workforce. We actively collaborate with individuals who have disabilities and long-term health conditions that affect their ability to perform normal daily activities, ensuring that barriers are eliminated when it comes to employment opportunities. In line with our commitment, we guarantee an interview to applicants who declare a disability and meet the minimum requirements for the role.
If you require any reasonable adjustments during the recruitment process, please let us know. Join us in building a truly diverse and empowered team.
