Lead Data Platform Engineer (5873) in London

We are seeking a highly skilled Lead Data Platform Engineer to join our Data Engineering and Machine Learning team. This role is pivotal in designing, architecting, and delivering robust, scalable, and secure data platforms that enable the firm to manage, analyse, and leverage data effectively while meeting regulatory and client confidentiality requirements.

You will combine hands‐on engineering with strong solution architecture skills, ensuring that data platform solutions are fit‐for‐purpose, well‐governed, and aligned to business needs. A key focus will be on Databricks, Azure Data Factory, and the Lakehouse Medallion architecture, with DevOps and automation at the heart of everything you do.

Key Responsibilities

• Architect and design end‐to‐end data platform solutions, ensuring scalability, reliability, and compliance.
• Lead implementation using Databricks, PySpark, Spark SQL, and Azure Data Factory.
• Develop APIs for data integration and automation.
• Write efficient, maintainable code in PySpark, Python, and SQL.
• Implement and manage CI/CD pipelines and automated deployments via Azure DevOps.
• Build infrastructure‐as‐code solutions (Terraform, ARM templates) for cloud resource provisioning.
• Monitor and optimise platform performance and manage cloud costs.
• Ensure data quality, security, governance, and lineage across all components.
• Collaborate with data engineers, architects, and business stakeholders to translate requirements into effective solutions.
• Maintain comprehensive documentation and stay current with emerging technologies.
• Provide coaching and mentoring to engineers, fostering a culture of continuous learning and technical excellence.

About You

• Proven experience designing and engineering data platforms in cloud environments (preferably Azure).
• Strong hands‐on experience with Databricks, PySpark, Spark SQL, and Azure Data Factory.
• Proficiency in Python and RESTful API development.
• Advanced expertise in DevOps practices for CI/CD and automated deployments.
• Experience with infrastructure‐as‐code tools (Terraform, ARM templates).
• Experience with additional Azure services (Fabric, Functions, Logic Apps).
• Data Lakehouse architecture.
• Background in regulated or professional services environments.

Our Benefits – What We Can Offer You

• 25 days holidays as standard plus bank holidays – You can 'buy' up to 35hrs of extra holiday too.
• Generous and flexible pension schemes.
• Volunteering days – Two days of volunteering every year for a cause of your choice (fully paid).
• Westfield Health membership, offering refunds on medical services alongside our Aviva Digital GP services.
• A wide range of well‐being initiatives to encourage positive mental health both in and out of the workplace.
• Flexible by Choice programme which gives our colleagues more choice over a hybrid way of working subject to role, team and client requirements.

We have been ranked in the Best Workplaces for Wellbeing for Large Organisations for 2024! Our responsible business programmes are fundamental to who we are and our purpose. We are committed to being a diverse and inclusive workplace where our colleagues can flourish, and we have established a number of inclusion network groups across our business to support this aim. Our commitment to Social Responsibility, community investment activity and tackling climate change is a fundamental part of who we are.

As part of the Irwin Mitchell Group's on‐boarding process all successful applicants are required to complete the group’s employment screening process. This process helps to ensure that all new employees meet our standards in relation to honesty and integrity therefore protecting the interests of the Group, colleagues, clients, partners and other stakeholders.

Irwin Mitchell LLP is an equal opportunity employer. We’re proud of our values, and we’re looking for people who share them.