At a Glance
- Tasks: Build scalable data pipelines using Azure Databricks for high-quality reporting datasets.
- Company: Join a diverse and inclusive team at Ryan Specialty in London.
- Benefits: Competitive pay, flexible working options, and opportunities for professional growth.
- Other info: Dynamic role with a focus on innovation and collaboration.
- Why this job: Make an impact by transforming raw data into valuable insights.
- Qualifications: 2+ years of Azure Databricks experience and strong PySpark skills required.
The predicted salary is between 50000 - 65000 £ per year.
An experienced data engineer with strong Azure Databricks expertise is required for a 6 month contract to develop and deliver a scalable data pipeline that ingests files and transforms them into standardised, high-quality datasets for reporting. This is a hands-on delivery role focused on building a production-grade pipeline, suited to someone comfortable working independently, defining technical solutions, and rapidly delivering high-quality data engineering components.
Location: London - UK - Fenchurch
Key Responsibilities:
- Implement a scalable ETL pipeline in Azure Databricks, that transforms raw data into standardised schemas to support reporting and analytics, following a medallion architecture in line with internal design standards.
- Leverage an existing email ingestion tool to extract multiple format source files from a shared mailbox.
- Develop notebooks, jobs and pipelines in Databricks for ETL orchestration.
- Design and implement robust validation rules, monitor pipeline runs and troubleshoot failures, with logging and notifications to support robust data processes.
- Integrate with CI/CD flows across environments using Azure DevOps.
- Produce clear technical documentation of the solution to ensure maintainability post-delivery.
Experience:
- 2+ years hands-on experience with Azure Databricks, including data validation, transformation, and optimisation.
- Expertise in PySpark/ Spark SQL.
- Knowledge of Unity Catalog and modern Azure Databricks features, as well as data best practices and concepts.
- Proficiency building end-to-end data pipelines with Delta Lake/Lakehouse architecture.
- Experience implementing CI/CD pipelines (Azure DevOps preferred).
- Experience supporting BI reporting layers (Power BI or similar) is nice to have.
- Insurance experience is nice to have.
Data Engineer employer: Ryan Specialty
At Ryan Specialty, we pride ourselves on being an exceptional employer, offering a dynamic work environment in the heart of London. Our commitment to employee growth is reflected in our inclusive culture and opportunities for professional development, particularly in cutting-edge technologies like Azure Databricks. Join us to be part of a diverse team that values innovation and collaboration, ensuring your contributions are recognised and rewarded.
StudySmarter Expert Advice🤫
We think this is how you could land Data Engineer
✨Tip Number 1
Network like a pro! Reach out to your connections in the data engineering field, especially those who work with Azure Databricks. A friendly chat can lead to insider info about job openings or even referrals.
✨Tip Number 2
Show off your skills! Create a portfolio showcasing your projects, especially those involving ETL pipelines and Azure Databricks. This gives potential employers a taste of what you can do and sets you apart from the crowd.
✨Tip Number 3
Prepare for interviews by brushing up on common data engineering scenarios. Be ready to discuss how you've tackled challenges in building scalable pipelines and optimising data processes. Confidence is key!
✨Tip Number 4
Don't forget to apply through our website! We love seeing candidates who are genuinely interested in joining us. Plus, it makes tracking your application a breeze!
We think you need these skills to ace Data Engineer
Some tips for your application 🫡
Tailor Your CV:Make sure your CV highlights your experience with Azure Databricks and data engineering. We want to see how your skills match the job description, so don’t be shy about showcasing your relevant projects!
Craft a Compelling Cover Letter:Your cover letter is your chance to shine! Use it to explain why you’re the perfect fit for this role. We love seeing enthusiasm and a clear understanding of what we do at StudySmarter.
Showcase Your Technical Skills:When detailing your experience, focus on your hands-on work with ETL pipelines, PySpark, and CI/CD processes. We’re looking for specifics that demonstrate your ability to deliver high-quality data solutions.
Apply Through Our Website:We encourage you to apply directly through our website. It’s the best way for us to receive your application and ensures you’re considered for the role. Plus, it’s super easy!
How to prepare for a job interview at Ryan Specialty
✨Know Your Tech Inside Out
Make sure you brush up on your Azure Databricks skills, especially around ETL pipelines and PySpark. Be ready to discuss specific projects where you've implemented these technologies, as this will show your hands-on experience.
✨Showcase Your Problem-Solving Skills
Prepare to talk about challenges you've faced in previous roles, particularly around data validation and troubleshooting. Highlight how you approached these issues and the solutions you implemented, as this demonstrates your ability to think critically.
✨Understand the Medallion Architecture
Familiarise yourself with the medallion architecture and be prepared to explain how you've applied it in your work. This shows that you not only understand the theory but can also apply it practically in building scalable data pipelines.
✨Communicate Clearly and Document Well
Since producing clear technical documentation is part of the role, practice explaining complex concepts in simple terms. Bring examples of documentation you've created in the past to illustrate your ability to maintain clarity and ensure future maintainability.
