Lead/Principle Data Engineer in Gloucester

Data Engineer Clearance Required:
Active SC Clearance (Mandatory) The Opportunity We're looking for a talented
Data Engineer
to join a growing team delivering secure, large-scale data solutions for government and public sector customers. This is an exciting opportunity to work on complex data platforms that support critical decision-making and operational outcomes. You'll play a key role in building and enhancing secure batch and real-time data processing capabilities, working with modern cloud technologies and industry-leading open-source tools. As part of a collaborative Agile team, you'll help design, develop, and optimise data pipelines, ensuring data is accurately processed, transformed, and delivered to downstream consumers. This role offers exposure to cutting-edge cloud and data technologies alongside the opportunity to work on meaningful projects within highly secure environments. Please note: Active SC Clearance is an essential requirement for this position. Due to the nature of the work, candidates must already hold current and active SC Clearance. Unfortunately, sponsorship or clearance applications cannot be supported for this role. Key Responsibilities Develop, test, and maintain robust data pipelines that meet both functional and non-functional business requirements. Design, build, and optimise data processing solutions to support batch and real-time analytics capabilities. Contribute to logical and physical database design activities across a range of data platforms. Expand and enhance platform capabilities, enabling the business to solve new and evolving data challenges. Analyse complex datasets and identify opportunities to improve data processing, quality, and performance. Ensure data models align with established data architecture standards, maintaining consistency across entities, relationships, and definitions. Perform root cause analysis on data-related issues, identifying and implementing effective solutions. Collaborate with engineers, architects, and stakeholders to deliver scalable and reliable data solutions. Support continuous improvement initiatives, helping to drive best practices across data engineering and platform development. Produce clear technical documentation covering data pipelines, processes, and system enhancements. Skills & Experience Essential Experience developing and maintaining
data pipelines
using modern data processing technologies and frameworks. Knowledge of
Agile
delivery methodologies and experience working within fast-paced development environments. Strong understanding of
data modelling , data structures, and their advantages and limitations across different use cases. Experience working with
public cloud platforms , particularly
AWS , including services such as
EC2, RDS, Lambda, Athena, and Glue . Experience configuring and optimising
Relational
and
NoSQL
databases, including query tuning and performance optimisation. Proficiency in programming and scripting languages such as
Python . Experience applying
Test-Driven Development (TDD)
methodologies and associated testing frameworks. Strong analytical and problem-solving skills with the ability to work through complex data challenges. Excellent communication skills and the ability to work effectively within multidisciplinary teams. Bachelor's degree or equivalent commercial experience in a relevant discipline. Desirable Experience with CI/CD tooling such as
Jenkins
or
Drone . Knowledge of configuration management tools including
Ansible
and
Packer . Experience using
Terraform
for Infrastructure-as-Code deployments. Understanding of both
Linux
and
Windows
environments. Exposure to modern cloud-native architectures and DevOps practices. Experience working within government, defence, or other highly regulated environments. What You'll Bring A passion for data engineering and solving complex technical problems. A strong attention to detail and commitment to delivering high-quality solutions. A collaborative mindset and the ability to work effectively within Agile teams. A proactive approach to learning new technologies and expanding your technical skillset. The ability to translate business requirements into scalable and efficient data solutions. A desire to contribute to projects that deliver meaningful impact within secure and mission-critical environments. Security Clearance -
Mandatory Requirement Candidates must hold current and active SC (Security Check) Clearance at the time of application. Applications from candidates who are eligible for clearance, willing to undergo clearance, or whose clearance has expired cannot be considered for this position. This is an excellent opportunity to join a high-performing engineering team, working on innovative cloud and data platforms that support critical government programmes while developing your expertise with modern technologies and large-scale data solutions. Reasonable Adjustments: Respect and equality are core values to us. We are proud of the diverse and inclusive community we have built, and we welcome applications from people of all backgrounds and perspectives. Our success is driven by our people, united by the spirit of partnership to deliver the best resourcing solutions for our clients. If you need any help or adjustments during the recruitment process for any reason
,
please let us know when you apply or talk to the recruiters directly so we can support you.
TPBN1_UKTJ