Data Engineering Manager

This role leads a high-performing data engineering team focused on building and operating modern data platforms using cloud-native technologies. You will oversee the design, development and optimisation of large-scale data pipelines, ensuring robust, secure and efficient data processing across distributed systems. The position requires strong hands-on technical expertise combined with leadership skills to guide engineers, shape technical direction and deliver reliable data solutions. You will work in a modern, cloud-first environment built on Google Cloud Platform, with workloads deployed on GKE and leveraging technologies such as Spark, Python, SQL, Trino, Databricks SQL, Apache Arrow, Apache Kafka and Iceberg. The role involves close collaboration with other engineering and data teams in a professional setting that values reliability, performance and scalability. Working patterns and hours are typically structured but may offer flexibility depending on team practices and project needs, with a focus on maintaining a balanced and sustainable workload.

Responsibilities:

• Lead and mentor a team of data engineers, providing technical guidance, coaching and support to help them grow and deliver high-quality solutions.
• Design, build and maintain scalable data pipelines and workflows using Spark, Python and SQL to support analytics, reporting and data products.
• Oversee the implementation and optimisation of data processing solutions on cloud infrastructure, particularly within GCP and GKE environments.
• Drive the adoption and effective use of Databricks SQL, Trino and Apache Arrow to improve performance, reliability and developer productivity.
• Manage streaming and real-time data processing solutions using Apache Kafka, ensuring resilient, low-latency data flows.
• Implement and govern data storage and table formats such as Iceberg (REST Catalog), ensuring data is organised, discoverable and performant.
• Collaborate with stakeholders to understand data requirements and translate them into robust, scalable technical designs.
• Ensure data quality, security and reliability across all pipelines and platforms, including monitoring, alerting and incident response processes.
• Promote engineering best practices such as code review, automated testing, CI/CD and infrastructure-as-code within the team.
• Work closely with infrastructure and platform teams to optimise resource usage, performance and cost across GCP and containerised workloads.
• Evaluate new tools, frameworks and architectures in areas such as distributed computing, data storage and streaming, and guide their adoption where appropriate.
• Prepare and maintain technical documentation and standards for data engineering solutions and platform components.

Essential Skills:

• Strong hands-on experience with SQL for data modelling, querying and performance optimisation.
• Proficiency in Python for data engineering, scripting and building data processing applications.
• Extensive experience with distributed data processing using Spark.
• Practical experience with Trino for federated querying and analytics.
• Experience using Databricks SQL to develop and optimise data workloads.
• Knowledge of Apache Arrow for efficient in-memory data representation and processing.
• Hands-on experience with Apache Kafka for building and managing streaming data pipelines.
• Experience working with modern table formats such as Iceberg (REST Catalog).
• Strong understanding of cloud infrastructure, particularly Google Cloud Platform (GCP).
• Experience deploying and managing workloads on GKE (Google Kubernetes Engine) or similar container orchestration platforms.
• Demonstrated ability to lead or manage engineering teams in a technical environment.
• Solid understanding of data engineering principles, including ETL/ELT, data warehousing and data pipeline orchestration.

Additional Skills & Qualifications:

• Experience working with large-scale, high-volume data platforms.
• Familiarity with infrastructure-as-code and CI/CD practices for data engineering.
• Understanding of data governance, security and compliance practices in cloud environments.
• Experience collaborating with cross-functional teams such as data science, analytics and product.
• Ability to communicate complex technical concepts clearly to both technical and non-technical stakeholders.