At a Glance
- Tasks: Design and optimise data pipelines using Databricks and cloud technologies.
- Company: Join a forward-thinking analytics platform with a remote working culture.
- Benefits: Competitive daily rate, flexible work environment, and opportunities for professional growth.
- Other info: Collaborative environment with a focus on cutting-edge technology and career advancement.
- Why this job: Make an impact by building large-scale data solutions in an innovative team.
- Qualifications: Active SC clearance and expertise in Databricks, Python, and AWS required.
We're looking for a Senior Data Engineer – Databricks & Big Data Pipelines to join our team, working in our UK remote working division. This position requires active UK Security Check (SC) clearance and offers the opportunity to build advanced, large-scale data engineering solutions for an innovative analytics platform. You will design, develop and optimize distributed data pipelines on Databricks and cloud ecosystems while ensuring performance, scalability, and compliance. The role combines technical depth with client-facing delivery as part of an agile and collaborative environment.
Responsibilities
- Design, develop and optimize end-to-end data pipelines using Databricks, Spark/PySpark and Python
- Lead the full pipeline lifecycle: analysis, architecture, development, testing, deployment and monitoring
- Implement ingestion, normalization and standardization workflows for high-volume and diverse data sources
- Ensure data platform architecture supports performance, security, and compliance requirements
- Collaborate cross-functionally with engineering, product teams and client stakeholders during project delivery
- Support ML readiness for ingestion and transformation pipelines where required
- Promote data platform best practices across security, governance and cloud adoption strategies
Mandatory Requirements
- Active SC Clearance
- Expertise in Databricks, including Delta Lake and Unity Catalog
- Strong background in Python, PySpark and advanced SQL
- Proven experience in AWS cloud platforms for data engineering
- Track record of building high-performance, large-scale data solutions in production environments
- Ability to lead pipeline lifecycle (design, deploy, monitor) autonomously
- Strong communication skills for stakeholder collaboration and client presentations
SkillSet Overview
- Must Have: Databricks (Delta Lake, Unity Catalog, workflows), Python, Data Engineering Technologies (including PySpark), SQL, AWS
- Nice to Have: AI and ML Ops, PostgreSQL, OpenSearch, Microsoft Azure, Claude Code (AI productivity tooling), Docker / Kubernetes, REST APIs, Cloud-native ML pipeline deployment, Familiarity with managing data platform releases, monitoring and cost governance, Domain understanding in supply chain analytics, compliance or public sector data, Experience integrating AI tools into engineering workflows
Data Engineer (Databricks) - SC Clearance Required in London employer: Korn Ferry
Join our innovative team as a Senior Data Engineer, where you'll have the opportunity to work remotely within the UK while contributing to cutting-edge data engineering solutions. We pride ourselves on fostering a collaborative and agile work culture that encourages professional growth and development, offering competitive rates and the chance to engage with advanced technologies in a supportive environment. With a focus on performance, security, and compliance, we empower our employees to excel in their roles and make a meaningful impact in the analytics space.
StudySmarter Expert Advice🤫
We think this is how you could land Data Engineer (Databricks) - SC Clearance Required in London
✨Tip Number 1
Network like a pro! Reach out to your connections in the data engineering field, especially those who work with Databricks or have SC clearance. A friendly chat can lead to insider info about job openings that aren't even advertised yet.
✨Tip Number 2
Show off your skills! Create a portfolio showcasing your projects, especially those involving Databricks, Python, and AWS. This will give potential employers a taste of what you can do and set you apart from the crowd.
✨Tip Number 3
Prepare for interviews by brushing up on your technical knowledge and soft skills. Be ready to discuss your experience with data pipelines and how you've tackled challenges in previous roles. Confidence is key!
✨Tip Number 4
Don't forget to apply through our website! We love seeing applications directly from candidates who are passionate about data engineering. Plus, it gives us a chance to get to know you better right from the start.
We think you need these skills to ace Data Engineer (Databricks) - SC Clearance Required in London
Some tips for your application 🫡
Tailor Your CV:Make sure your CV highlights your experience with Databricks, Python, and AWS. We want to see how your skills match the job description, so don’t be shy about showcasing your relevant projects!
Craft a Compelling Cover Letter:Your cover letter is your chance to shine! Use it to explain why you’re the perfect fit for our team and how your background aligns with the role. Keep it concise but impactful – we love a good story!
Showcase Your Projects:If you've worked on any cool data engineering projects, make sure to mention them! We’re interested in seeing how you’ve tackled challenges and what technologies you used. Real-world examples can really set you apart.
Apply Through Our Website:We encourage you to apply directly through our website. It’s the best way for us to receive your application and ensures you don’t miss out on any important updates from our team!
How to prepare for a job interview at Korn Ferry
✨Know Your Databricks Inside Out
Make sure you brush up on your Databricks knowledge, especially Delta Lake and Unity Catalog. Be ready to discuss how you've used these tools in past projects, as well as any challenges you faced and how you overcame them.
✨Showcase Your Pipeline Mastery
Prepare to talk about the full lifecycle of data pipelines you've designed and developed. Highlight specific examples where you led the analysis, architecture, development, testing, deployment, and monitoring phases, demonstrating your ability to manage complex projects.
✨Communicate Like a Pro
Since this role involves client-facing delivery, practice articulating your technical knowledge in a way that's easy for non-technical stakeholders to understand. Think about how you can explain complex concepts simply and clearly.
✨Be Ready for Scenario Questions
Expect questions that assess your problem-solving skills in real-world scenarios. Prepare examples that showcase your experience with high-volume data sources, compliance requirements, and how you ensure performance and security in your data solutions.
