At a Glance
- Tasks: Design and deliver scalable data flows using Databricks and Azure Data Factory.
- Company: Join a forward-thinking company based in Bournemouth with a collaborative spirit.
- Benefits: Fixed-term contract, competitive pay, and opportunities for professional growth.
- Other info: Exciting chance to work in a dynamic environment with potential career advancement.
- Why this job: Make an impact by transforming raw data into valuable insights for businesses.
- Qualifications: Strong PySpark skills and experience with Databricks and Azure Data Factory required.
The predicted salary is between 48000 - 72000 € per year.
We have an exciting opportunity for a skilled Lead Databricks Engineer with deep PySpark expertise and hands‑on experience in building pipelines across Databricks and Azure Data Factory. The role will be based out of our Bournemouth office and is a fixed‑term contract for 12 months. You will design and deliver robust, scalable, and governed data flows that form the backbone of our Lakehouse platform, ensuring high‑quality, trusted data for analytics, reporting, and regulatory requirements.
Key Responsibilities
- Build and maintain Databricks pipelines (batch and incremental) using PySpark and SQL.
- Orchestrate end-to-end data workflows with Azure Data Factory.
- Develop and optimise Delta Lake tables (partitioning, schema evolution, vacuuming).
- Implement the Medallion Architecture (Bronze, Silver, Gold) to transform raw data into business‑ready datasets.
- Apply row‑count checks, logging, and error‑handling frameworks for pipeline monitoring and governance.
- Integrate Databricks pipelines with downstream systems including Power BI and reporting platforms.
- Collaborate with analysts, business teams, and other engineers to deliver consistent, well‑documented datasets.
- Support deployments and automation via Azure DevOps CI/CD pipelines.
- Gather and refine requirements from business stakeholders.
About You
- Strong PySpark development skills for large‑scale data engineering.
- Proven experience with Databricks pipelines and workflow management.
- Expertise in Azure Data Factory for orchestration and scheduling.
- Solid knowledge of Delta Lake and Lakehouse data modelling principles.
- Hands‑on experience with SQL for data transformation and validation.
- Familiarity with Azure services (ADLS/Blob, Key Vault, SQL).
- Knowledge of ETL/ELT frameworks, error logging, and monitoring.
Desirable skills
- Understanding of Microsoft Fabric.
- Experience with DevOps practices (CI/CD, release pipelines).
- Awareness of data governance and lineage tools.
- Experience working in the Finance sector.
Please note that we are unable to offer Skilled Worker Visa Sponsorship for this role. Therefore, you must ensure that you are eligible to work in the UK without our sponsorship for your application to be considered. Any successful internal colleagues will be offered a secondment opportunity.
Lead Databricks Engineer employer: LV=
Join our Bournemouth office as a Lead Databricks Engineer and be part of a dynamic team that values innovation and collaboration. We offer a supportive work culture that prioritises employee growth, with opportunities for professional development and hands-on experience in cutting-edge technologies like Databricks and Azure. Enjoy the benefits of working in a vibrant coastal city, where you can balance a rewarding career with a fulfilling lifestyle.
StudySmarter Expert Advice🤫
We think this is how you could land Lead Databricks Engineer
✨Tip Number 1
Network like a pro! Reach out to your connections in the data engineering field, especially those who work with Databricks or Azure. A friendly chat can lead to insider info about job openings that aren't even advertised yet.
✨Tip Number 2
Show off your skills! Create a portfolio showcasing your best Databricks projects and PySpark pipelines. This gives potential employers a taste of what you can do and sets you apart from the crowd.
✨Tip Number 3
Prepare for interviews by brushing up on common questions related to Databricks and Azure Data Factory. Practise explaining your past projects and how you tackled challenges—this will help you shine during the interview.
✨Tip Number 4
Don't forget to apply through our website! It’s the best way to ensure your application gets noticed. Plus, we love seeing candidates who are proactive about their job search.
We think you need these skills to ace Lead Databricks Engineer
Some tips for your application 🫡
Tailor Your CV:Make sure your CV highlights your PySpark and Databricks experience. We want to see how your skills align with the role, so don’t be shy about showcasing relevant projects or achievements!
Craft a Compelling Cover Letter:Your cover letter is your chance to shine! Use it to explain why you’re excited about this role and how your background makes you the perfect fit. We love seeing genuine enthusiasm for what we do.
Showcase Your Technical Skills:Be specific about your technical expertise in Azure Data Factory and Delta Lake. We’re looking for solid examples of how you’ve built and optimised data pipelines, so don’t hold back on the details!
Apply Through Our Website:We encourage you to apply directly through our website. It’s the best way for us to receive your application and ensures you’re considered for the role. Plus, it’s super easy!
How to prepare for a job interview at LV=
✨Know Your Tech Inside Out
Make sure you brush up on your PySpark and Databricks skills. Be ready to discuss specific projects where you've built pipelines or worked with Azure Data Factory. The more detailed examples you can provide, the better!
✨Understand the Medallion Architecture
Familiarise yourself with the Medallion Architecture (Bronze, Silver, Gold) and be prepared to explain how you've applied it in past projects. This shows that you not only know the theory but have practical experience too.
✨Showcase Your Collaboration Skills
Since this role involves working with analysts and business teams, think of examples where you've successfully collaborated with others. Highlight how you gathered requirements and delivered well-documented datasets.
✨Prepare for Scenario-Based Questions
Expect questions about error handling and monitoring in your pipelines. Prepare to discuss how you've implemented logging and governance frameworks in your previous roles, as this will demonstrate your attention to detail and commitment to quality.
