At a Glance
- Tasks: Design and optimise data pipelines using Databricks and cloud technologies.
- Company: Join a forward-thinking analytics platform with a remote-first culture.
- Benefits: Competitive daily rate, flexible remote work, and opportunities for professional growth.
- Other info: Collaborative team atmosphere with a focus on cutting-edge technology.
- Why this job: Make an impact by building large-scale data solutions in an innovative environment.
- Qualifications: SC clearance, expertise in Databricks, Python, and AWS required.
We're looking for a Senior Data Engineer - Databricks & Big Data Pipelines to join our team, working in our UK remote working division. This position requires active UK Security Check (SC) clearance and offers the opportunity to build advanced, large-scale data engineering solutions for an innovative analytics platform. You will design, develop and optimize distributed data pipelines on Databricks and cloud ecosystems while ensuring performance, scalability, and compliance. The role combines technical depth with client-facing delivery as part of an agile and collaborative environment.
Responsibilities
- Design, develop and optimize end-to-end data pipelines using Databricks, Spark/PySpark and Python
- Lead the full pipeline lifecycle: analysis, architecture, development, testing, deployment and monitoring
- Implement ingestion, normalization and standardization workflows for high-volume and diverse data sources
- Ensure data platform architecture supports performance, security, and compliance requirements
- Collaborate cross-functionally with engineering, product teams and client stakeholders during project delivery
- Support ML readiness for ingestion and transformation pipelines where required
- Promote data platform best practices across security, governance and cloud adoption strategies
Mandatory Requirements
- Active SC Clearance
- Expertise in Databricks, including Delta Lake and Unity Catalog
- Strong background in Python, PySpark and advanced SQL
- Proven experience in AWS cloud platforms for data engineering
- Track record of building high-performance, large-scale data solutions in production environments
- Ability to lead pipeline lifecycle (design, deploy, monitor) autonomously
- Strong communication skills for stakeholder collaboration and client presentations
SkillSet Overview
Must Have:
- Databricks (Delta Lake, Unity Catalog, workflows)
- Python
- Data Engineering Technologies (including PySpark)
- SQL
- AWS
Nice to Have:
- AI and ML Ops
- PostgreSQL
- OpenSearch
- Microsoft Azure
- Claude Code (AI productivity tooling)
- Docker / Kubernetes
- REST APIs
- Cloud-native ML pipeline deployment
- Familiarity with managing data platform releases, monitoring and cost governance
- Domain understanding in supply chain analytics, compliance or public sector data
- Experience integrating AI tools into engineering workflows
Data Engineer (Databricks) - SC Clearance Required employer: Ports North
Join our innovative team as a Data Engineer, where you'll have the opportunity to work remotely from anywhere in the UK while contributing to cutting-edge data engineering solutions. We pride ourselves on fostering a collaborative and agile work culture that encourages professional growth and development, offering competitive rates and the chance to work with advanced technologies like Databricks and AWS. With a strong focus on security and compliance, we provide a supportive environment for you to thrive and make a meaningful impact in the analytics space.
StudySmarter Expert Advice🤫
We think this is how you could land Data Engineer (Databricks) - SC Clearance Required
✨Tip Number 1
Network like a pro! Reach out to your connections in the data engineering field, especially those who work with Databricks or have SC clearance. A friendly chat can lead to insider info about job openings that aren't even advertised yet.
✨Tip Number 2
Show off your skills! Create a portfolio showcasing your best projects, especially those involving Databricks and big data pipelines. This will give potential employers a taste of what you can do and set you apart from the crowd.
✨Tip Number 3
Prepare for interviews by brushing up on your technical knowledge and soft skills. Be ready to discuss your experience with Python, PySpark, and AWS, and don’t forget to highlight your ability to collaborate with teams and clients.
✨Tip Number 4
Apply through our website! We’ve got loads of opportunities waiting for talented Data Engineers like you. Plus, it’s a great way to ensure your application gets seen by the right people.
We think you need these skills to ace Data Engineer (Databricks) - SC Clearance Required
Some tips for your application 🫡
Tailor Your CV:Make sure your CV highlights your experience with Databricks, Python, and AWS. We want to see how your skills match the job description, so don’t be shy about showcasing your relevant projects!
Craft a Compelling Cover Letter:Your cover letter is your chance to shine! Use it to explain why you’re the perfect fit for our Data Engineer role. Share specific examples of your work with data pipelines and how you've tackled challenges in the past.
Show Off Your Communication Skills:Since this role involves collaboration with various teams, make sure to highlight your communication skills in your application. We love candidates who can articulate their ideas clearly and work well with others!
Apply Through Our Website:We encourage you to apply directly through our website. It’s the best way for us to receive your application and ensures you don’t miss out on any important updates from our team!
How to prepare for a job interview at Ports North
✨Know Your Tech Inside Out
Make sure you brush up on your Databricks, Python, and PySpark skills. Be ready to discuss specific projects where you've designed and optimised data pipelines. The more you can demonstrate your technical expertise, the better!
✨Showcase Your Problem-Solving Skills
Prepare to talk about challenges you've faced in previous roles, especially around data ingestion and normalisation. Use the STAR method (Situation, Task, Action, Result) to structure your answers and highlight how you tackled those issues.
✨Understand the Business Context
Familiarise yourself with the company’s goals and how your role as a Data Engineer fits into their vision. Being able to connect your technical skills to business outcomes will impress interviewers and show that you're not just a techie but also a strategic thinker.
✨Communicate Clearly and Confidently
Since this role involves client-facing delivery, practice explaining complex technical concepts in simple terms. Good communication is key, so be prepared to engage with stakeholders and demonstrate your ability to collaborate effectively.
