At a Glance
- Tasks: Lead a team to design and develop a modern cloud-based data platform.
- Company: Join a leading insurance provider known for tech innovation.
- Benefits: Enjoy competitive salary, bonuses, and a comprehensive benefits package.
- Other info: Hybrid working model from a modern London office.
- Why this job: Shape the future of data engineering with cutting-edge technologies.
- Qualifications: Experience in Python, PySpark, and Azure; leadership skills preferred.
The predicted salary is between 60000 - 80000 £ per year.
Location
- London - Hybrid (3 days in the office)
- About the Client
Our client is a leading specialist insurance provider, recognised for its innovative approach to technology and data.
Operating globally, they are investing heavily in their modern data platform, leveraging cloud technologies to deliver scalable, high-quality data solutions that support underwriting, analytics and business decision-making.
Role Overview
As a Lead Data Engineer, you'll play a key role in designing and evolving a modern cloud-based data platform built on Databricks and Azure.
You'll lead the development of scalable data pipelines using Python and Py Spark, enabling high-quality data to power underwriting, analytics and business-critical decision-making across the organisation.
Working closely with Product Owners, Architects and business stakeholders, you'll provide technical leadership throughout the full data lifecycle.
Alongside remaining hands‑on, you'll mentor a team of Data Engineers, champion engineering best practices and help shape the future of the organisation's cloud data platform.
Key Responsibilities
- Lead and mentor a team of Data Engineers, supporting the successful delivery of data engineering projects from design through to production.
- Design, develop and optimise scalable data pipelines using Python, Py Spark and Databricks.
- Build and maintain cloud‑native data solutions across the Azure ecosystem, including Azure Functions and related services.
- Collaborate with Product Owners, Architects and business stakeholders to define technical solutions and deliver against business priorities.
- Drive engineering best practices through code reviews, mentoring and continuous improvement initiatives.
- Identify opportunities to improve platform performance, data quality, scalability and operational efficiency.
- Champion automation, testing, CI/CD and modern engineering practices across the data engineering function.
- Key Requirements
- Strong commercial experience building data engineering solutions using Python, Py Spark and Databricks.
- Proven experience designing and optimising complex SQL queries and working with large‑scale datasets.
- Strong experience across the Azure technology stack, including Azure Functions and cloud‑native data services.
- Previous experience leading or mentoring Data Engineers within an Agile environment.
- Strong understanding of distributed data processing, modern data engineering practices and CI/CD.
- Experience building scalable, reliable and high‑performing data platforms.
- Experience within Financial Services or Insurance is highly advantageous.
- Why Join
- Opportunity to shape the future of a modern, cloud‑native data platform built on Databricks and Azure.
- Hands‑on leadership role combining technical delivery with engineering strategy.
- Work with modern technologies including Python, Py Spark, Databricks and the wider Azure ecosystem.
- Collaborative engineering culture focused on innovation, quality and continuous improvement.
- Competitive salary, bonus and comprehensive benefits package.
- Hybrid working from a modern London office.
- Data & Application Notice
We process personal data submitted as part of your application for recruitment purposes in line with UK GDPR.
All applications are reviewed by a member of our team, we do not use automated decision‑making in our selection process.
Your application will be assessed against the requirements of this role.
By applying to this advert, you consent to receiving future relevant roles and industry news and insights from us.
To opt out of this, please contact: compliance@pioneer-search. com.
For more information on how we use your data, including how long we retain it & your rights, please refer to our Privacy Policy: www. pioneer-search. com/privacy-policy.
#J-18808-Ljbffr
Lead Data Engineer employer: Pioneer Search
Join a dynamic and growing insurance business as a Senior Internal Controls Analyst, where you will be part of a collaborative work culture that values innovation and continuous improvement. With opportunities for hybrid working in London, you will engage with senior stakeholders and gain invaluable experience in controls testing and remediation within a regulated environment. The company is committed to employee growth, offering a broad role that enhances your skills in internal controls, audit, and risk management.
