At a Glance
- Tasks: Build scalable data pipelines in Azure Databricks for high-quality datasets.
- Company: Join a diverse and inclusive team at Ryan Specialty.
- Benefits: Competitive pay, flexible work environment, and growth opportunities.
- Other info: Opportunity to work independently and define technical solutions.
- Why this job: Make an impact by transforming raw data into valuable insights.
- Qualifications: 2+ years of Azure Databricks experience and strong data engineering skills.
The predicted salary is between 50000 - 65000 £ per year.
An experienced data engineer with strong Azure Databricks expertise is required for a 6 month contract to develop and deliver a scalable data pipeline that ingests files and transforms them into standardised, high‑quality datasets for reporting. This is a hands‑on delivery role focused on building a production‑grade pipeline, suited to someone comfortable working independently, defining technical solutions, and rapidly delivering high‑quality data engineering components.
Location: London, UK (Fenchurch).
Key Responsibilities
- Implement a scalable ETL pipeline in Azure Databricks, that transforms raw data into standardised schemas to support reporting and analytics, following a medallion architecture in line with internal design standards.
- Leverage an existing email ingestion tool to extract multiple format source files from a shared mailbox.
- Develop notebooks, jobs and pipelines in Databricks for ETL orchestration.
- Design and implement robust validation rules, monitor pipeline runs and troubleshoot failures, with logging and notifications to support robust data processes.
- Integrate with CI/CD flows across environments using Azure DevOps.
- Produce clear technical documentation of the solution to ensure maintainability post‑delivery.
Experience
- 2+ years hands‑on experience with Azure Databricks, including data validation, transformation, and optimisation.
- Expertise in PySpark/Spark SQL.
- Knowledge of Unity Catalog and modern Azure Databricks features, as well as data best practices and concepts.
- Proficiency building end‑to‑end data pipelines with Delta Lake/Lakehouse architecture.
- Experience implementing CI/CD pipelines (Azure DevOps preferred).
- Experience supporting BI reporting layers (PowerBI or similar) is nice to have.
- Insurance experience is nice to have.
Data Engineer employer: Ryan Specialty Corporate Services Limited
Ryan Specialty is an exceptional employer located in the heart of London, offering a dynamic work culture that fosters innovation and collaboration. With a strong commitment to diversity and inclusion, employees are encouraged to grow their skills and advance their careers while working on impactful projects in data engineering. The company provides competitive benefits and a supportive environment, making it an ideal place for professionals seeking meaningful and rewarding employment.
Contact Details:
Ryan Specialty Corporate Services Limited Recruitment Team
StudySmarter Expert Advice🤫
We think this is how you could land Data Engineer
✨Network Like a Pro
Get out there and connect with folks in the industry! Attend meetups, webinars, or even just grab a coffee with someone who’s already in the data engineering game. You never know who might have the inside scoop on job openings or can put in a good word for you.
✨Show Off Your Skills
Don’t just tell them what you can do; show them! Create a portfolio of your projects, especially those involving Azure Databricks and ETL pipelines. Share your GitHub link or any relevant work during interviews to demonstrate your hands-on experience.
✨Ace the Interview
Prepare for technical interviews by brushing up on your PySpark and data pipeline knowledge. Practice common interview questions and scenarios related to data engineering. Remember, it’s not just about answering questions but also about showcasing your problem-solving skills.
✨Apply Through Our Website
We’ve got some fantastic opportunities waiting for you! Make sure to apply through our website for the best chance at landing that dream job. It’s a straightforward process, and we’re here to help you every step of the way!
We think you need these skills to ace Data Engineer
Some tips for your application 🫡
Tailor Your CV:Make sure your CV highlights your experience with Azure Databricks and data pipelines. We want to see how your skills match the job description, so don’t be shy about showcasing your relevant projects!
Craft a Compelling Cover Letter:Your cover letter is your chance to shine! Use it to explain why you’re the perfect fit for this role. We love seeing enthusiasm and a clear understanding of what we do at StudySmarter.
Showcase Your Technical Skills:When detailing your experience, focus on your hands-on work with PySpark, Delta Lake, and CI/CD pipelines. We’re looking for someone who can hit the ground running, so make those skills pop!
Apply Through Our Website:We encourage you to apply directly through our website. It’s the best way for us to receive your application and ensures you’re considered for the role. Plus, it’s super easy!
How to prepare for a job interview at Ryan Specialty Corporate Services Limited
✨Know Your Tech Inside Out
Make sure you brush up on your Azure Databricks skills, especially around ETL processes and PySpark. Be ready to discuss specific projects where you've implemented these technologies, as this will show your hands-on experience.
✨Showcase Your Problem-Solving Skills
Prepare to talk about how you've tackled challenges in previous roles, particularly around data validation and troubleshooting pipeline failures. Use examples that highlight your ability to think critically and act independently.
✨Understand the Medallion Architecture
Familiarise yourself with the medallion architecture and be prepared to explain how you've applied it in your work. This will demonstrate your understanding of data engineering best practices and your ability to deliver high-quality datasets.
✨Communicate Clearly and Confidently
Since you'll need to produce clear technical documentation, practice explaining complex concepts in simple terms. This will not only help during the interview but also show that you can communicate effectively with both technical and non-technical stakeholders.
