Find a job 
Featured Companies 
Featured Universities 
For Companies At a Glance
- Tasks: Design and build cutting-edge Databricks Lakehouse solutions to transform data into insights.
- Company: Join The AA, a beloved brand focused on making data our superpower.
- Benefits: Enjoy 25 days annual leave, free breakdown membership, and diverse learning opportunities.
- Why this job: Be part of a team driving meaningful change with innovative data solutions.
- Qualifications: Proven experience with Azure Databricks, Python, and building production-grade data pipelines.
- Other info: Hybrid working model with a supportive and inclusive company culture.
The predicted salary is between 36000 - 60000 £ per year.
Location: London (hybrid working 3 office days per week)
Employment Type: Permanent, full time
Are you passionate about data and eager to make a significant impact? The AA is a well-loved brand with a range of driver services much wider than most people realise. We have an enviable set of data assets from breakdown, service, repair, insurance, telematics, digital interactions, car dealers and driving school!
If that’s not enough, we’re focused on making Data our new Superpower as one of only 4 strategic priorities for the Group. Our growing team is modernising our data infrastructure to a cutting-edge cloud platform and enabling machine learning and GenAI. Join us at an exciting time and be part of a team that is driving meaningful change for our customers, colleagues and shareholders.
This role is for an experienced Databricks Data Engineer. You will be working day-to-day designing, building and operating production-grade Databricks Lakehouse solutions, including structured streaming pipelines, Unity Catalog governance, and Spark-based data engineering at scale.
This is not a role for someone who has "touched" Databricks occasionally or supported it at the edges. We are looking for someone who has built, owned and evolved Databricks solutions, and who can confidently articulate design decisions, trade-offs, and best practice.
What will I be doing?
- Designing, building and operating production-grade Databricks Lakehouse solutions, including structured streaming pipelines using Python and PySpark
- Owning and evolving Unity Catalog–based governance, ensuring secure, discoverable and well-managed data assets
- Developing and maintaining event-driven data pipelines, integrating closely with backend engineering teams
- Implementing and supporting CI/CD pipelines in Azure DevOps to enable reliable, automated Databricks deployments
- Creating high-quality, analytics-ready datasets that deliver actionable business insight at scale
- Proactively improving performance, reliability, automation and observability across the Databricks data platform
What do I need?
We are intentionally setting a high bar for Databricks experience. You should be able to demonstrate deep, hands-on capability, not just theoretical knowledge.
- Significant, hands-on experience with Azure Databricks in a production environment
- Proven experience building Spark / Databricks pipelines using Python and PySpark
- Strong experience with structured streaming and event-driven architectures
- Practical experience implementing and operating Unity Catalog
- Solid understanding of Lakehouse design principles, including dimensional and analytical modelling
- Experience building and maintaining CI/CD pipelines, ideally using Azure DevOps
- Confidence working with large-scale data, performance tuning, and troubleshooting complex pipelines
What we mean by "Databricks experience"
You have:
- Designed and built Databricks pipelines end-to-end
- Made architectural decisions within Databricks environments
- Worked with Spark internals, optimisation techniques and cluster configuration
- Operated Databricks solutions in live, business-critical contexts
If your Databricks exposure has been limited to minor contributions, proof-of-concepts, or occasional usage alongside other tools, this role is unlikely to be the right fit.
Additional information
We’re always looking to recognise and reward our employees for the work they do. As a valued member of The AA team, you’ll have access to a range of benefits including:
- 25 days annual leave plus bank holidays + holiday buying scheme
- Worksave pension scheme with up to 7% employer contribution
- Free AA breakdown membership from Day 1 plus 50% discount for family and friends
- Discounts on AA products including car and home insurance
- Employee discount scheme that gives you access to a car salary sacrifice scheme plus great discounts on healthcare, shopping, holidays and more
- Company funded life assurance
- Diverse learning and development opportunities to support you to progress in your career
- Dedicated Employee Assistance Programme and a 24/7 remote GP service for you and your family
- Plus, so much more!
We’re an equal opportunities employer and welcome applications from everyone. The AA values diversity and the difference this brings to our culture and our customers. We actively seek people from diverse backgrounds to join us and become part of an inclusive company where you can be yourself, be empowered to be your best and feel like you truly belong. We have five communities to bring together people with shared characteristics and backgrounds and drive positive change.
Data Engineer – Databricks Specialist in London employer: Automobile Association
Contact Detail:
Automobile Association Recruiting Team
StudySmarter Expert Advice 🤫
We think this is how you could land Data Engineer – Databricks Specialist in London
✨Tip Number 1
Network like a pro! Reach out to current employees at The AA on LinkedIn or other platforms. Ask them about their experiences and any tips they might have for the interview process. It’s all about making connections!
✨Tip Number 2
Prepare for technical interviews by brushing up on your Databricks skills. Practice coding challenges related to data engineering, especially using Python and PySpark. Show them you’re not just familiar with Databricks, but that you live and breathe it!
✨Tip Number 3
Don’t forget to showcase your past projects! Bring examples of your work with Databricks, especially those that highlight your ability to design and build production-grade solutions. This is your chance to shine and show what you can bring to the team.
✨Tip Number 4
Finally, apply through our website! It’s the best way to ensure your application gets seen by the right people. Plus, it shows you’re genuinely interested in joining The AA family. Let’s get you that job!
We think you need these skills to ace Data Engineer – Databricks Specialist in London
Some tips for your application 🫡
Show Off Your Databricks Skills: Make sure to highlight your hands-on experience with Databricks in your application. We want to see how you've designed and built pipelines, so don’t hold back on the details!
Be Specific About Your Experience: When you describe your past roles, focus on specific projects where you’ve implemented structured streaming or worked with Unity Catalog. We love seeing concrete examples of your work!
Tailor Your Application: Don’t just send a generic CV! Tailor your application to reflect the skills and experiences mentioned in our job description. This shows us you’re genuinely interested in the role.
Apply Through Our Website: We encourage you to apply directly through our website. It’s the best way for us to receive your application and ensures you’re considered for this exciting opportunity!
How to prepare for a job interview at Automobile Association
✨Know Your Databricks Inside Out
Make sure you can confidently discuss your hands-on experience with Databricks. Be prepared to share specific examples of how you've designed and built pipelines, made architectural decisions, and optimised performance in a production environment.
✨Showcase Your Problem-Solving Skills
During the interview, highlight instances where you've tackled complex data challenges. Discuss how you approached troubleshooting and performance tuning in your previous roles, especially with large-scale data and event-driven architectures.
✨Familiarise Yourself with the Company’s Data Vision
Research The AA's focus on modernising their data infrastructure and their strategic priorities. Understanding their goals around making data a 'superpower' will help you align your answers with their vision and demonstrate your enthusiasm for contributing to their mission.
✨Prepare Questions That Matter
Think of insightful questions to ask your interviewers about their Databricks implementation and future projects. This shows your genuine interest in the role and helps you assess if the company is the right fit for you.
