At a Glance
- Tasks: Build and optimise data pipelines using Azure Databricks and support analytics solutions.
- Company: Join a forward-thinking company focused on modern data platforms.
- Benefits: Flexible day rate, hybrid work model, and opportunities for professional growth.
- Other info: Contract role for UK residents only, with potential for exciting projects.
- Why this job: Be at the forefront of data engineering and make a real impact in a dynamic environment.
- Qualifications: Experience with Azure Databricks, PySpark, and data pipeline development required.
Location: London (Hybrid - 2 days per week onsite)
Contract Length: Initial 6 months
Day Rate: Flexible (Inside & Outside IR35 considered - final determination pending)
We're partnering with a client seeking a skilled Databricks Data Engineer to support the build and evolution of a modern data platform. This is a hands‑on contract role, ideally suited to someone who thrives working across both new platform development and ongoing production support within a Databricks‑driven environment.
The Role
This role sits at the heart of a Databricks Lakehouse implementation, where you'll be responsible for building, optimising, and maintaining scalable data pipelines and datasets to support analytics and business intelligence. You’ll work closely with technical and business stakeholders to deliver high‑quality, performant data solutions while also ensuring the stability and reliability of existing workloads.
Key Responsibilities
- Design, build, and optimise data pipelines within Azure Databricks using PySpark and Spark SQL
- Develop and manage Delta Lake‑based data models, supporting a structured Lakehouse / Medallion architecture (Bronze, Silver, Gold layers)
- Support the ingestion and transformation of large‑scale data from multiple sources into Databricks
- Contribute to the modernisation and migration of legacy SQL Server workloads into Databricks
- Monitor, troubleshoot, and improve the performance of existing data pipelines and jobs
- Work closely with stakeholders to ensure data is reliable, well‑structured, and ready for analytics and reporting
- Integrate and orchestrate workflows using tools such as Azure Data Factory or Databricks Workflows
- Collaborate with BI teams to ensure datasets are optimised for Power BI and downstream consumption
Required Skills & Experience
- Strong commercial experience working with Azure Databricks as a core data processing platform
- Deep expertise in:
- PySpark / Apache Spark for distributed data processing
- Delta Lake (ACID transactions, optimisation, data versioning)
- Spark SQL and advanced SQL techniques
- Python for data engineering and pipeline development
- Hands‑on experience designing and implementing Lakehouse architectures
- Experience migrating data platforms from on‑premises systems into cloud‑based Databricks environments
- Solid understanding of data modelling, ETL/ELT design, and performance optimisation
- Familiarity with orchestration tools such as Azure Data Factory, Airflow, or Databricks Jobs
- Experience working in production environments with both project delivery and BAU responsibilities
Desirable Experience
- Exposure to Unity Catalog, data governance, and fine‑grained access control in Databricks
- Experience implementing CI/CD pipelines for Databricks (e.g. via Azure DevOps or Git integration)
- Knowledge of streaming data pipelines (Spark Structured Streaming / Kafka)
- Experience working with cloud‑native data architectures in Azure
- Prior exposure to Power BI or similar BI/reporting tools
Additional Information
This is a contract role for UK residents only. The role does not offer sponsorship. You must have the right to work in the UK with no restrictions. Some roles may be subject to successful background checks including a DBS and Credit Check.
Contract Databricks Data Engineer in London employer: Nigel Frank
Join a forward-thinking company that values innovation and collaboration, offering a dynamic work culture in the heart of London. As a Contract Databricks Data Engineer, you'll have the opportunity to work on cutting-edge data solutions while enjoying flexible working arrangements and competitive rates. With a strong focus on employee growth and development, this role provides a unique chance to enhance your skills in a supportive environment that champions both personal and professional advancement.
StudySmarter Expert Advice🤫
We think this is how you could land Contract Databricks Data Engineer in London
✨Network Like a Pro
Get out there and connect with people in the industry! Attend meetups, webinars, or even just grab a coffee with someone who works in data engineering. You never know when a casual chat could lead to your next opportunity.
✨Show Off Your Skills
Don’t just talk about your experience; demonstrate it! Create a portfolio showcasing your projects, especially those involving Databricks, PySpark, and Delta Lake. This will give potential employers a clear view of what you can bring to the table.
✨Ace the Interview
Prepare for technical interviews by brushing up on your SQL and data pipeline skills. Be ready to discuss your past projects in detail, focusing on challenges you faced and how you overcame them. Confidence is key!
✨Apply Through Us
Make sure to apply through our website! We’re always on the lookout for talented individuals like you, and applying directly can give you an edge. Plus, we love seeing candidates who are proactive about their job search.
We think you need these skills to ace Contract Databricks Data Engineer in London
Some tips for your application 🫡
Tailor Your CV:Make sure your CV is tailored to the Databricks Data Engineer role. Highlight your experience with Azure Databricks, PySpark, and any relevant projects that showcase your skills in building and optimising data pipelines.
Craft a Compelling Cover Letter:Your cover letter is your chance to shine! Use it to explain why you're the perfect fit for this contract role. Mention specific experiences that align with the job description and show your enthusiasm for working with us at StudySmarter.
Showcase Your Technical Skills:Don’t forget to emphasise your technical expertise in your application. Detail your hands-on experience with Delta Lake, Spark SQL, and any orchestration tools you've used. This will help us see how you can contribute to our team right away!
Apply Through Our Website:We encourage you to apply through our website for a smoother application process. It helps us keep track of your application and ensures you don’t miss out on any important updates from us!
How to prepare for a job interview at Nigel Frank
✨Know Your Databricks Inside Out
Make sure you brush up on your Azure Databricks knowledge. Be ready to discuss your experience with PySpark, Spark SQL, and Delta Lake. Prepare examples of how you've built and optimised data pipelines in previous roles.
✨Showcase Your Problem-Solving Skills
Expect questions about troubleshooting and improving existing data pipelines. Think of specific challenges you've faced and how you resolved them. This will demonstrate your hands-on experience and ability to think critically under pressure.
✨Understand the Business Context
Familiarise yourself with the business side of data engineering. Be prepared to discuss how your work impacts analytics and reporting. Showing that you can bridge the gap between technical and business stakeholders will set you apart.
✨Prepare for Technical Questions
You might face some technical assessments or coding challenges during the interview. Practice common data engineering problems, especially those related to ETL/ELT design and performance optimisation. Being confident in your technical skills will help you shine.
