At a Glance
- Tasks: Design and build innovative Databricks Lakehouse solutions using Python and PySpark.
- Company: Join the AA, a beloved brand with a modern data infrastructure.
- Benefits: Enjoy 25 days leave, pension contributions, and discounts on various services.
- Other info: Diverse learning opportunities and a supportive, inclusive workplace await you.
- Why this job: Make a real impact in a cutting-edge cloud environment while growing your career.
- Qualifications: Hands-on experience with Azure Databricks and strong problem-solving skills required.
The predicted salary is between 60000 - 75000 £ per year.
We are an equal opportunities employer and welcome applications from everyone. The AA values diversity and actively seeks people from diverse backgrounds to join us and become part of an inclusive company where you can be yourself, be empowered to be your best and feel like you truly belong.
Responsibilities
- Designing, building and operating production-grade Databricks Lakehouse solutions, including structured streaming pipelines using Python and PySpark.
- Owning and evolving Unity Catalog‑based governance, ensuring secure, discoverable and well‑managed data assets.
- Developing and maintaining event‑driven data pipelines, integrating closely with backend engineering teams.
- Implementing and supporting CI/CD pipelines in Azure DevOps to enable reliable, automated Databricks deployments.
- Creating high‑quality, analytics‑ready datasets that deliver actionable business insight at scale.
- Proactively improving performance, reliability, automation and observability across the Databricks data platform.
Qualifications
- Significant, hands‑on experience with Azure Databricks in a production, enterprise‑scale environment.
- Proven experience building Spark/Databricks pipelines using Python and PySpark.
- Strong experience with structured streaming and event‑driven architectures.
- Practical experience implementing and operating Unity Catalog.
- Solid understanding of Lakehouse design principles, including dimensional and analytical modelling.
- Experience building and maintaining CI/CD pipelines, ideally using Azure DevOps.
- Confidence working with large‑scale data, performance tuning, and troubleshooting complex pipelines.
- Strong debugging and problem‑solving skills, including the ability to diagnose and resolve issues in Spark/Databricks pipelines; hands‑on experience configuring and managing Databricks environments, including cluster setup, dependency management and package installation.
- Designed and built Databricks pipelines end‑to‑end.
- Made architectural decisions within Databricks environments.
- Worked with Spark internals, optimisation techniques and cluster configuration.
- Operated Databricks solutions in live, business‑critical contexts.
Benefits
- 25 days annual leave plus bank holidays and holiday buying scheme.
- Worksave pension scheme with up to 7% employer contribution.
- Free AA breakdown membership from Day 1 plus 50% discount for family and friends.
- Discounts on AA products, including car and home insurance.
- Employee discount scheme that gives you access to a car salary sacrifice scheme and other discounts on healthcare, shopping, holidays and more.
- Company‑funded life assurance.
- Diverse learning and development opportunities to support you to progress in your career.
- Dedicated Employee Assistance Programme and a 24/7 remote GP service for you and your family.
The AA is a well‑loved brand with a wide range of driver services and a growing data infrastructure that is modernising on a cutting‑edge cloud platform. If you’re passionate about data and eager to make a significant impact, we invite you to apply for this experienced Databricks Data Engineer role.
Databricks Lakehouse Data Engineer — Azure, CI/CD, Impact in City of Westminster employer: The Automobile Association
The AA is an exceptional employer that champions diversity and inclusivity, ensuring every employee feels empowered and valued. With a strong focus on professional growth, employees benefit from extensive learning opportunities, a supportive work culture, and a comprehensive benefits package including generous leave, pension contributions, and health discounts. Located in a dynamic environment, the AA offers the chance to work with cutting-edge technology while making a meaningful impact in the data engineering field.
Contact Details:
The Automobile Association Recruitment Team
StudySmarter Expert Advice🤫
We think this is how you could land Databricks Lakehouse Data Engineer — Azure, CI/CD, Impact in City of Westminster
✨Tip Number 1
Network like a pro! Reach out to folks in the industry, attend meetups, and connect with people on LinkedIn. You never know who might have the inside scoop on job openings or can refer you directly.
✨Tip Number 2
Show off your skills! Create a portfolio showcasing your Databricks projects, especially those involving Python and PySpark. This gives potential employers a taste of what you can do and sets you apart from the crowd.
✨Tip Number 3
Prepare for interviews by brushing up on your technical knowledge and problem-solving skills. Practice common data engineering scenarios and be ready to discuss your experience with CI/CD pipelines and event-driven architectures.
✨Tip Number 4
Don’t forget to apply through our website! It’s the best way to ensure your application gets seen. Plus, we love seeing candidates who are genuinely interested in joining our diverse and inclusive team.
We think you need these skills to ace Databricks Lakehouse Data Engineer — Azure, CI/CD, Impact in City of Westminster
Some tips for your application 🫡
Tailor Your CV:Make sure your CV reflects the skills and experiences that match the job description. Highlight your hands-on experience with Azure Databricks, Python, and CI/CD pipelines to show us you’re the perfect fit!
Craft a Compelling Cover Letter:Use your cover letter to tell us why you're passionate about data engineering and how your background aligns with our values of diversity and inclusion. This is your chance to let your personality shine through!
Showcase Your Projects:If you've worked on relevant projects, don’t hesitate to include them! We love seeing real examples of your work with Databricks, Spark, and event-driven architectures. It helps us understand your practical experience better.
Apply Through Our Website:We encourage you to apply directly through our website. It’s the best way for us to receive your application and ensures you don’t miss out on any important updates from our team!
How to prepare for a job interview at The Automobile Association
✨Know Your Databricks Inside Out
Make sure you brush up on your knowledge of Azure Databricks, especially in a production environment. Be ready to discuss your hands-on experience with building Spark/Databricks pipelines using Python and PySpark, as well as any architectural decisions you've made within Databricks.
✨Showcase Your CI/CD Skills
Since the role involves implementing CI/CD pipelines in Azure DevOps, be prepared to talk about your experience with this. Bring examples of how you've enabled reliable, automated Databricks deployments and any challenges you faced while doing so.
✨Demonstrate Problem-Solving Prowess
Highlight your debugging and problem-solving skills during the interview. Share specific instances where you diagnosed and resolved issues in Spark/Databricks pipelines, and discuss your approach to performance tuning and troubleshooting.
✨Emphasise Collaboration with Backend Teams
The job requires close integration with backend engineering teams, so be ready to discuss your experience developing and maintaining event-driven data pipelines. Talk about how you’ve collaborated with other teams to create high-quality, analytics-ready datasets that drive business insights.
