Lead AI Data Engineer in Croydon

Location: Croydon | Manchester | Sheffield | Glasgow

Salary: £62,109 plus skills allowance of up to £18,921

Advert Close: Wednesday 22nd April 2026

Lead Data Engineers deliver the data engineering vision for the organisation as set by the Principal Data Engineer. As a Lead Data Engineer, you will focus on designing and implementing numerous complex data flows to connect operational systems, data for analytics and our emerging AI-enabled services and systems, as part of the AI as a Service platform team. You will identify and deliver new data sources and develop ETL processes designed to enable both human and AI/machine consumption, collaborate with the Data Acquisition, Analytics and Infrastructure teams to deliver the data strategy through the design and implementation of technology systems. You will lead and assure on AI-focused data concerns including (meta) data processing within AI-agent flows, tracking, auditing and assuring AI-related data quality, and recording data provenance across lifecycles that include multiple interactions that bring legacy data sets and AI-based components together. You will lead the data management and data engineering services and analytical projects, working collaboratively with other teams to acquire and ingest new data sources. You will also work with the Principal Data Engineer, Data Architects, Database Administrators, Technical Architects, Solution Architects and Data Analytics team to set the enterprise data strategy and standards and agree implementation and governance.

Main Responsibilities

• Managing data sources in a way to improve data quality and conform them for analysis and processing by various AI and Machine Learning tools and systems.
• Developing new data models and ETL processes, working collaboratively within the AI as a Service team to map data to an appropriate matching data model.
• Exercising key stakeholder management skills to manage expectations, listen and interpret business needs, communicate progress and facilitate difficult conversations, while building a 'service culture' around an organisational shift towards AI-ready data.
• Managing the performance of suppliers and the data team to improve capability, as well as creating the roadmap which balances immediate user needs with long-term investment.
• Working with AI and Machine Learning specialists on your team to develop processes that balance data availability, capacity and data archiving, while continually improving processes and efficiency in the acquisition and ingestion of data.
• Line managing staff, including coaching and professional development.
• Using agreed standards and tools to design, code, test, correct and document moderate to complex programs and scripts from agreed specifications, and deploying tested code to infrastructure hosted over multiple cloud providers including AWS and Azure.
• Facilitating the evaluation and selection of software products, data models, and design patterns by keeping abreast of best practice in industry and across government, to enable and assure enterprise-scale AI-data integrations.

You’ll have a demonstrable passion for Data, with the following skills or some experience in:

• Effectively leading and managing teams, helping them deliver technical products and projects involving complex data processing lifecycles, solution design and end user reporting.
• Advising on best practice IT programming and automation techniques and tools including migrations of relational and NoSQL databases, and possibly time series and vector databases, along with testing and deploying data pipelines via Continuous Integration techniques, e.g., using Github actions / Jenkins / etc.
• Leading the use of Cloud Data technologies, solutions and future Cloud Data Strategies, on at least one major Cloud provider’s infrastructure services, such as Azure or AWS.
• Working with stakeholders to gather requirements and deliver across the product/service development lifecycle including findings, designs, proof of concept Products, up to production ready data pipelines.
• Presenting complex ideas and recommendations that inform decisions in a clear way to technical and non-technical audiences and senior customers.
• Using tools such as industry standard ETL tools, network databases and scheduling and orchestration tools.
• Data engineering in the context of AI, Machine Learning, or automation-enabled products or services.
• Familiarity with MLOps principles underpinning pipelines used for continuously training / validating / deploying / monitoring Machine Learning models and LLMs.
• Data analysis techniques and tools, industry recognised data patterns and standards and their applications.
• Working with Big Data tools and data stores.

Why work for us...

Find out more information at: Benefits - Home Office Careers, but some of the primary ones are:

• A Civil Service Pension with employer contribution rates of at least 28.97%.
• In-year reward scheme for one-off or sustained exceptional personal or team achievements.
• 25 days annual leave on appointment, rising with service.
• 8 days of public holidays, plus 1 additional privilege day.
• Where business needs allow, some roles may be suitable for a combination of office and home-based working. This is a non-contractual arrangement where all employees will be expected to spend a minimum of 60% of their working time in an office.