Find a job 
Featured Companies 
Featured Universities 
For Companies At a Glance
- Tasks: Design and optimise data pipelines using Databricks and Azure for impactful AI solutions.
- Company: Join a global leader in underwriting with a modern, cloud-first approach.
- Benefits: Hybrid work model, competitive salary, and opportunities for professional growth.
- Other info: Dynamic environment with a focus on innovation and career advancement.
- Why this job: Be at the forefront of data engineering and AI, making a real difference in insurance.
- Qualifications: Strong data engineering experience, especially with Databricks and Azure.
The predicted salary is between 60000 - 80000 £ per year.
Location: UK (hybrid)
We’re partnering with a global underwriting leader to hire a Senior Data Engineer to play a key role in scaling their global data platform. This is an opportunity to work at the intersection of data engineering and AI, delivering high-impact solutions across insurance and reinsurance business units. You’ll be joining a modern, cloud-first environment with significant investment in Databricks, Azure and AI-driven data capabilities.
The Role
- Design, build and optimise enterprise-scale ETL and ELT pipelines using Azure Data Factory, Databricks and Python
- Develop and maintain scalable data solutions delivering high-quality, ML-ready datasets
- Work extensively with Databricks SQL and SQL Server to support analytics and reporting use cases
- Contribute to the ongoing modernisation of the data platform, including migration of legacy systems into Azure-based architectures
- Build robust data models leveraging Delta Lake and modern lakehouse principles
- Implement workflow orchestration using tools such as Apache Airflow and Azure Data Factory
AI & Advanced Analytics
- Support and contribute to AI-enabled data solutions, including LLM and Retrieval-Augmented Generation (RAG) use cases
- Work with vector databases and emerging Databricks AI capabilities
- Help drive intelligent automation across areas such as claims and underwriting through NLP and advanced data pipelines
- Design scalable, cloud-native data architectures across Azure, with some exposure to AWS environments
- Optimise data performance, reliability and cost efficiency across the platform
- Establish and follow best practices across CI/CD, testing and deployment using Azure DevOps and Kubernetes
- Improve pipeline reliability and reduce manual intervention through automation and orchestration
Requirements
- Strong experience in data engineering, ideally within complex, enterprise environments
- Proven expertise with Databricks, including Databricks SQL
- Advanced SQL skills, including SQL Server
- Hands-on experience with ETL and ELT frameworks
- Experience with Azure data stack, including Azure Data Factory and Data Lake
- Experience working with AI/ML pipelines, LLMs or RAG frameworks
- Exposure to NLP or intelligent document processing
- Experience within insurance or financial services
- Familiarity with AWS services such as Lambda, S3 or EC2
Senior Data Engineer - Databricks in London employer: Arthur Recruitment
Contact Detail:
Arthur Recruitment Recruiting Team
StudySmarter Expert Advice 🤫
We think this is how you could land Senior Data Engineer - Databricks in London
✨Tip Number 1
Network like a pro! Reach out to folks in the industry, especially those already working with Databricks or in data engineering roles. A friendly chat can open doors and give you insights that job descriptions just can't.
✨Tip Number 2
Show off your skills! Create a portfolio showcasing your projects, especially those involving Azure, Databricks, and AI solutions. This is your chance to demonstrate what you can do beyond the CV.
✨Tip Number 3
Prepare for interviews by brushing up on your technical knowledge and problem-solving skills. Be ready to discuss your experience with ETL/ELT pipelines and how you've tackled challenges in previous roles.
✨Tip Number 4
Don’t forget to apply through our website! It’s the best way to ensure your application gets the attention it deserves. Plus, we love seeing candidates who are proactive about their job search.
We think you need these skills to ace Senior Data Engineer - Databricks in London
Some tips for your application 🫡
Tailor Your CV: Make sure your CV reflects the skills and experiences that match the Senior Data Engineer role. Highlight your expertise with Databricks, Azure, and any relevant AI projects you've worked on. We want to see how you can contribute to our data platform!
Craft a Compelling Cover Letter: Your cover letter is your chance to shine! Use it to tell us why you're passionate about data engineering and how your background aligns with our needs. Don't forget to mention specific projects or achievements that showcase your skills.
Showcase Your Technical Skills: In your application, be sure to highlight your hands-on experience with ETL/ELT frameworks and your advanced SQL skills. We love seeing examples of how you've built scalable data solutions or optimised data performance in previous roles.
Apply Through Our Website: We encourage you to apply directly through our website for the best chance of getting noticed. It’s super easy, and you'll be able to keep track of your application status. Plus, we love seeing candidates who take the initiative!
How to prepare for a job interview at Arthur Recruitment
✨Know Your Tech Stack
Make sure you’re well-versed in the technologies mentioned in the job description, especially Databricks, Azure Data Factory, and Python. Brush up on your SQL skills too, as they’ll likely test your knowledge during the interview.
✨Showcase Your Projects
Prepare to discuss specific projects where you've designed and built ETL/ELT pipelines. Highlight how you’ve optimised data solutions and any experience with AI/ML pipelines. Real-world examples will make your expertise stand out.
✨Understand the Business Context
Since this role is within the insurance and reinsurance sectors, do a bit of research on how data engineering impacts these industries. Being able to connect your technical skills to business outcomes will impress your interviewers.
✨Ask Insightful Questions
Prepare thoughtful questions about the company’s data strategy, their use of AI, and how they envision the future of their data platform. This shows your genuine interest in the role and helps you assess if it’s the right fit for you.