Data Engineer / Senior Data Engineer in Warminster

As a Data Engineer, you will be critical to the successful delivery of the programme, collaborating within a matrix organisation, with multi-disciplinary teams within Engineering. We are looking for individuals who want to serve. You’ll have a mission focus, and the enthusiasm and drive to deliver. You must be eligible and willing to obtain SC clearance and will be based at Warminster working in a hybrid style. You’ll work in a matrix organisation and report operationally through OMNIA Training and functionally to the Data Lead. Ultimately, you’ll work for the British Army, championing innovation, and helping shape the future of military collective training.

Key Responsibilities

• Collaborate with engineering, simulation, customer, third party and training teams to ensure seamless data management and integration across military training environments.
• Ensuring the security and compliance of data environments through the implementation of appropriate security controls and governance frameworks.
• Author, review and contribute to technical documentation.
• Coordinate with cross-functional engineering teams—including QA, development, operations, and business SMEs.
• Assist in the design and implementation of scalable, high-performance data platforms and pipelines.
• Define and enforce data engineering standards, best practices, and governance frameworks including data lifecycle design, implementation and management.
• Implement the design and optimisation of data storage, processing, and retrieval strategies.
• Collaborate with cross-functional teams (data scientists, analysts, software engineers) to align data solutions with business needs.
• Drive adoption of modern data technologies (e.g., cloud-native platforms, streaming, orchestration tools).
• Contribute to data quality, security, compliance, and lineage practices across the organisation.
• Provide technical input and guidance to data engineering team members.
• Evaluate, select, and integrate new tools, frameworks, and platforms for the data ecosystem.
• Act as a subject matter expert in data strategy, influencing design decisions.
• Any other duties required to meet the needs of the programme.

Who we are looking for

The OMNIA Data Engineer will provide hands‑on technical oversight and management across the Army Collective Training Service (ACTS) data solutions. These will be prominently MODCloud hosted utilising Cloud Services from AWS, Azure, OCP where appropriate. Reporting to the Data Architect, you will play a key role in ensuring the effective design, assurance, transformation, and delivery of data across ACTS services and capabilities. This includes supporting the integration and operational use of data from legacy systems through API-driven solutions. This role requires a systems‑thinking mindset, strong stakeholder engagement skills, and the ability to work across engineering teams in a complex and evolving environment. Given the programme’s focus on modelling and simulation, familiarity with relevant standards and technologies is highly desirable.

Essential Skills and Experience

• Hands‑on experience with cloud platforms (AWS, Azure, GCP, OCI) and cloud‑native data services.
• Extensive experience designing and building large‑scale data pipelines and platforms.
• Strong expertise in SQL, data modelling, and database optimisation (relational, vector and NoSQL).
• Proficiency in distributed data processing frameworks (e.g., Spark, Flink, Hadoop).
• Deep knowledge of cloud data platforms (AWS, Azure, or GCP) and associated services.
• Strong programming skills in Python, Java, or Scala for data engineering.
• Hands‑on experience with data orchestration and workflow management tools (e.g., Airflow, Dagster, Prefect).
• Proven track record with data governance, quality, lineage, and security practices.
• Experience with real‑time/streaming data technologies, data ingestion/ETL (e.g., Apache Kafka, Apache NiFi, Kinesis, Pub/Sub).
• Strong proficiency in relational and non‑relational databases (e.g., PostgreSQL, MongoDB, Cassandra).
• Deep knowledge of data transformation and ETL pipelines and APIs.
• Security cleared or ability to obtain (SC or above).

Desirable Skills and Experience

• Degree in Data Engineering or equivalent professional accreditation such as CEng.
• Experience with graph databases and advanced query languages (e.g., Neo4j, Gremlin).
• Knowledge of machine learning data pipelines and MLOps practices.
• Familiarity with data virtualisation and data mesh concepts.
• Hands‑on experience with containerisation and orchestration (Docker, Kubernetes, Red Hat OpenShift and Ceph).
• Exposure to infrastructure‑as‑code tools (Terraform, CloudFormation).
• Experience with BI/visualisation tools (e.g., Tableau, Power BI, Looker, Elastic Stack).
• Knowledge of compliance frameworks (GDPR, HIPAA, CCPA) and their impact on data systems.
• Strong background in performance tuning for high‑throughput, low‑latency data systems.
• Microsoft Certified: Azure Data Engineer Associate, AWS Certified Data Engineer, IBM Data Engineering Professional Certificate or similar.