Data Engineering Manager in City of Westminster

Responsibilities

• Team Leadership & Management – Lead, mentor, and develop a team of data engineers, fostering a culture of collaboration, innovation, and continuous improvement. Set clear goals and performance expectations for team members, conducting regular performance reviews and providing constructive feedback.
• Data Pipeline Orchestration – Oversee the design, building, and maintenance of end-to-end data pipelines using Airflow (including Amazon MWAA) to orchestrate, schedule, and monitor batch/streaming workflows. Ensure implementation of DAGs with retry logic, error handling, and alerting.
• Data Ingestion & Transformation – Guide the integration of data from various sources using Airbyte for ingestion and dbt for transformations. Collaborate with Data Analysts and Data Scientists to implement transformations and business logic, ensuring data is analytics-ready.
• Data Modeling & Warehousing – Direct design and implementation of efficient data models for structured and semi-structured data in AWS S3 (data lake) and Snowflake. Utilize Databricks for data processing and analytics.
• Data Governance & Security – Lead the utilization of AWS Lake Formation APIs and best practices to maintain data security, access controls, and compliance. Collaborate with IT security on encryption standards, audit trails, and identity/role-based access.
• Performance Optimization – Oversee optimization of AWS Athena queries, data partitioning, Airflow DAG tuning, Snowflake queries, and Databricks jobs. Monitor for performance and cost efficiency.
• Collaboration & Stakeholder Management – Partner with cross-functional teams to ensure seamless integration of data workflows and systems. Communicate technical solutions to non-technical stakeholders.
• Continuous Improvement – Participate in architecture reviews, code reviews, and troubleshooting sessions. Stay current on emerging trends in data engineering, orchestration tools, and cloud services.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
• 5+ years of experience in data engineering or a similar role, with at least 2 years in a leadership or managerial capacity working with cloud-based data platforms.

Technical Skills

• Cloud & Orchestration – Proficient in Airflow (self-managed or Amazon MWAA) with DAG development, scheduling, and best practices for organization and error handling.
• AWS Expertise – Hands-on experience with AWS Lake Formation, S3, Athena, Lambda, Glue, IAM.
• Snowflake – Proficient in setting up data warehouses, configuring security, and optimizing queries.
• Databricks – Experience with Databricks for data processing, analytics, and real-time data insights.
• Data Ingestion & Transformation – Experience with Airbyte or similar tools for ingestion; dbt or SQL-based frameworks for modular data processing.
• Programming – Proficiency in Python and/or Java/Scala for building data pipelines and custom integrations.
• Query Languages – Advanced knowledge of SQL for data manipulation and analysis.

Soft Skills

• Strong problem-solving and analytical abilities.
• Excellent communication and collaboration skills, able to work effectively in cross-functional teams.
• Ability to operate in a fast-paced, agile environment and manage multiple priorities simultaneously.

Preferred Certifications & Experience

• AWS certifications (e.g., AWS Certified Data Analytics – Specialty, AWS Certified Solutions Architect) are a plus.
• Experience with CI/CD pipelines, containerization (Docker, Kubernetes), and infrastructure as code (Terraform, CloudFormation).
• Familiarity with DevOps best practices for managing Airflow environments (version control, automated testing, monitoring).
• Familiarity with financial services data, especially private equity, is not necessary but beneficial.

Benefits

• Support for professional accreditations such as ACCA and study leave.
• Flexible arrangements, generous holidays, and an additional day off for your birthday.
• Continuous mentoring along your career progression.
• Active sports, events, and social committees across our offices.
• 24/7 support from our Employee Assistance Program.
• Opportunity to invest in our growth through the Employee Share Plan.
• Additional local benefits depending on your location.

We are in the business of equity, in every sense of the word. For us, this means taking action to ensure every colleague has equal opportunity, valuing every voice and experience across our organisation, maintaining an inclusive culture where you can bring your whole self to work, and making Alter Domus a workplace where everyone feels they belong. We celebrate our differences, and understand that our success relies on diverse perspectives and experiences, working towards shared goals and a common purpose. We take pride in creating a workplace where all our people are empowered to be truly invested in the alternative and bring their whole selves to work. We are committed to ensuring a welcoming recruiting and onboarding process for everyone. Please contact our hiring team if you require any accommodations to make our recruitment process more accessible for you.