At a Glance
- Tasks: Design and build innovative Databricks solutions while collaborating with backend teams.
- Company: Join the AA, a beloved brand committed to diversity and inclusion.
- Benefits: Enjoy 25 days leave, pension contributions, and discounts on various services.
- Other info: Dynamic environment with diverse learning opportunities and career progression.
- Why this job: Make a real impact in data engineering on a cutting-edge cloud platform.
- Qualifications: Hands-on experience with Azure Databricks and strong Python skills required.
The predicted salary is between 60000 - 75000 £ per year.
We are an equal opportunities employer and welcome applications from everyone. The AA values diversity and actively seeks people from diverse backgrounds to join us and become part of an inclusive company where you can be yourself, be empowered to be your best and feel like you truly belong.
Responsibilities
- Designing, building and operating production-grade Databricks Lakehouse solutions, including structured streaming pipelines using Python and PySpark
- Owning and evolving Unity Catalog‑based governance, ensuring secure, discoverable and well‑managed data assets
- Developing and maintaining event‑driven data pipelines, integrating closely with backend engineering teams
- Implementing and supporting CI/CD pipelines in Azure DevOps to enable reliable, automated Databricks deployments
- Creating high‑quality, analytics‑ready datasets that deliver actionable business insight at scale
- Proactively improving performance, reliability, automation and observability across the Databricks data platform
Qualifications
- Significant, hands‑on experience with Azure Databricks in a production, enterprise‑scale environment
- Proven experience building Spark/Databricks pipelines using Python and PySpark
- Strong experience with structured streaming and event‑driven architectures
- Practical experience implementing and operating Unity Catalog
- Solid understanding of Lakehouse design principles, including dimensional and analytical modelling
- Experience building and maintaining CI/CD pipelines, ideally using Azure DevOps
- Confidence working with large‑scale data, performance tuning, and troubleshooting complex pipelines
- Strong debugging and problem‑solving skills, including the ability to diagnose and resolve issues in Spark/Databricks pipelines; hands‑on experience configuring and managing Databricks environments, including cluster setup, dependency management and package installation
- Designed and built Databricks pipelines end‑to‑end
- Made architectural decisions within Databricks environments
- Worked with Spark internals, optimisation techniques and cluster configuration
- Operated Databricks solutions in live, business‑critical contexts
Benefits
- 25 days annual leave plus bank holidays and holiday buying scheme
- Worksave pension scheme with up to 7% employer contribution
- Free AA breakdown membership from Day 1 plus 50% discount for family and friends
- Discounts on AA products, including car and home insurance
- Employee discount scheme that gives you access to a car salary sacrifice scheme and other discounts on healthcare, shopping, holidays and more
- Company‑funded life assurance
- Diverse learning and development opportunities to support you to progress in your career
- Dedicated Employee Assistance Programme and a 24/7 remote GP service for you and your family
The AA is a well‑loved brand with a wide range of driver services and a growing data infrastructure that is modernising on a cutting‑edge cloud platform. If you’re passionate about data and eager to make a significant impact, we invite you to apply for this experienced Databricks Data Engineer role.
Data Engineer (Databricks) in City of Westminster employer: The Automobile Association
The AA is an exceptional employer that champions diversity and inclusivity, ensuring every employee feels empowered and valued. With a strong focus on professional growth, employees benefit from extensive learning opportunities, a supportive work culture, and a comprehensive benefits package including generous leave, pension contributions, and health discounts. Located in a dynamic environment, the AA offers the chance to work with cutting-edge technology while making a meaningful impact in the data landscape.
Contact Details:
The Automobile Association Recruitment Team
StudySmarter Expert Advice🤫
We think this is how you could land Data Engineer (Databricks) in City of Westminster
✨Tip Number 1
Network like a pro! Reach out to folks in the industry, attend meetups, and connect with people on LinkedIn. You never know who might have the inside scoop on job openings or can refer you directly.
✨Tip Number 2
Show off your skills! Create a portfolio showcasing your Databricks projects, especially those involving Python and PySpark. This will give potential employers a taste of what you can do and set you apart from the crowd.
✨Tip Number 3
Prepare for interviews by brushing up on your technical knowledge and problem-solving skills. Practice common data engineering scenarios and be ready to discuss your experience with CI/CD pipelines and event-driven architectures.
✨Tip Number 4
Don’t forget to apply through our website! It’s the best way to ensure your application gets seen. Plus, we love seeing candidates who are genuinely interested in joining our diverse and inclusive team.
We think you need these skills to ace Data Engineer (Databricks) in City of Westminster
Some tips for your application 🫡
Tailor Your CV:Make sure your CV reflects the skills and experiences mentioned in the job description. Highlight your hands-on experience with Azure Databricks, Python, and PySpark to show us you’re the right fit for the role.
Craft a Compelling Cover Letter:Use your cover letter to tell us why you're passionate about data engineering and how your background aligns with our values of diversity and inclusion. This is your chance to show us your personality!
Showcase Your Projects:If you've worked on any relevant projects, especially those involving Databricks or CI/CD pipelines, make sure to mention them. We love seeing real-world applications of your skills!
Apply Through Our Website:We encourage you to apply directly through our website. It’s the best way for us to receive your application and ensures you don’t miss out on any important updates from us!
How to prepare for a job interview at The Automobile Association
✨Know Your Databricks Inside Out
Make sure you brush up on your Azure Databricks knowledge. Be ready to discuss your hands-on experience with building Spark/Databricks pipelines using Python and PySpark. Prepare examples of how you've designed and operated production-grade solutions, as this will show your practical understanding of the platform.
✨Showcase Your Problem-Solving Skills
Be prepared to tackle some technical questions or scenarios during the interview. Think about complex pipeline issues you've resolved in the past and how you approached debugging and performance tuning. This will demonstrate your strong problem-solving skills and ability to work under pressure.
✨Familiarise Yourself with CI/CD Pipelines
Since the role involves implementing CI/CD pipelines in Azure DevOps, make sure you can discuss your experience with this process. Highlight any specific projects where you’ve successfully automated Databricks deployments, as this will show your capability to enhance reliability and efficiency.
✨Emphasise Your Collaborative Spirit
The job requires close integration with backend engineering teams, so be ready to talk about your teamwork experiences. Share examples of how you've collaborated with others to create high-quality, analytics-ready datasets, and how you’ve contributed to a positive team environment.
