Lead Data Engineer

Lead Data Engineer | Location: London (Hybrid – 1-2 day a month in office) | Salary: £90,000-£110,000 The Opportunity Are you passionate about building cutting-edge data platforms that drive business growth? Our client is seeking a skilled and motivated Lead Data Engineer to play a key role in the creation of a brand-new data platform within the Azure ecosystem and Databricks . This is an exciting opportunity to be at the forefront of data innovation, working within a newly formed Data & Analytics team in a long standing London Market Insurer. You’ll help shape the data strategy, improve data quality, and empower the business to make data-driven decisions. As a Lead Data Engineer, you\’ll work closely with both technical and business stakeholders, leveraging your expertise to design, develop, and optimize a high-performance data platform built on Databricks . This platform will be built to scale, incorporating the latest advancements in data intelligence while supporting strategic business objectives. Key Responsibilities Build & Develop – Design and maintain a robust Databricks Data Platform, ensuring performance, scalability, and availability. Data Pipelines – Connect APIs, databases, and data streams to the platform, implementing ETL/ELT processes. Data Integrity – Embed quality measures, monitoring, and alerting mechanisms. CI/CD & Automation – Create deployment pipelines and automate workflows. Collaboration – Work with stakeholders across Global IT, Data, and Broking teams to translate business requirements into technical solutions. Futureproofing – Drive the evolution of the data platform, ensuring adaptability for new data sources, analytical models, and emerging technologies. What You’ll Bring ✅ Extensive hands-on experience with Databricks , and Microsoft Azure data tools (must-have: Azure Data Factory, Azure Synapse, or Azure SQL). ✅ Dimensional modelling expertise for analytics use cases. ✅ Strong ETL/ELT development skills. ✅ Python scripting experience for data automation. ✅ Experience with CI/CD methodologies for data platforms. ✅ Knowledge of MS SQL Server, SSIS, Visual Studio, and SSDT projects . ✅ Previous London Market Insurance experience Why Join? Greenfield Project – Work on an all-new data platform, shaping its architecture from the ground up. Collaborative Culture – Engage with global teams in an agile, innovative environment. Career Growth – Play a pivotal role in driving data excellence within a forward-thinking business. Cutting-Edge Tech – Work with the latest advancements in Azure, Databricks, and Data Engineering . This is a fantastic opportunity for a Lead Data Engineer looking to make a tangible impact. If you’re ready to take on a challenging and rewarding role, apply today!