Find a job 
Featured Companies 
Featured Universities 
For Companies At a Glance
- Tasks: Build and maintain data pipelines using Apache Spark and AWS services.
- Company: Join Scrumconnect, a leading UK tech consultancy driving digital transformation.
- Benefits: Enjoy a hybrid work model, competitive salary, and opportunities for professional growth.
- Other info: Active SC clearance is mandatory; excellent career growth opportunities await you.
- Why this job: Make a real impact in a cloud-native environment while working with cutting-edge technologies.
- Qualifications: Experience in Python, SQL, and data engineering principles required.
The predicted salary is between 50000 - 65000 £ per year.
A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong analytical and engineering skills to deliver trusted, well-governed data assets in a modern, cloud-native environment.
Active SC clearance is a mandatory, non-negotiable requirement. Candidates must hold current, in-date Security Check (SC) clearance at the time of application. Sponsorship is not available. Applications without active SC clearance will not be considered.
Working arrangement: This role is hybrid. Candidates must be willing and able to travel to the Newcastle office three days per week. Remaining days may be worked remotely from anywhere in the UK.
You will work as a Data Engineer on a complex, cloud-based data programme — designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics, you will help deliver reliable, well-governed data assets to downstream users.
You will apply strong data analysis skills to identify root causes of data issues, work with dimensional data models and slowly changing dimensions, and implement infrastructure as code using Terraform. Familiarity with DWP engineering best practices and the ability to translate customer expectations into applied technical functionality are key to success in this role.
Key responsibilities
- Build and maintain scalable data pipelines using Apache Spark and PySpark, processing and transforming large datasets across distributed cloud infrastructure.
- Configure and manage Apache Airflow DAGs for task orchestration, ensuring reliable scheduling, monitoring, and execution of data processing workflows.
- Perform data analysis to identify and resolve root causes of pipeline failures and data quality issues — including reviewing EMR output logs and CloudWatch metrics.
- Apply understanding of dimensional data models and slowly changing dimensions (SCD) to design and maintain well-structured, analytically trusted data assets.
- Provision and manage cloud infrastructure using Terraform. Containerise solutions using Docker and manage deployments through GitLab CI/CD pipelines and release tagging.
- Apply understanding of both server-side and client-side encryption patterns within AWS. Work within IAM policies and data governance standards appropriate to a regulated government environment.
Technical skills required
- Python — primary language for pipeline development and data processing
- SQL — used for querying, transformation, and validation across data stores
- PySpark — for distributed data processing using Apache Spark on AWS EMR
- Familiarity with basic data structures for constructing robust, scalable solutions
AWS services
- Amazon EMR — running Spark workloads and reviewing output logs
- Amazon Athena — ad hoc querying of data in S3
- Amazon Textract and Comprehend — familiarity with AI/ML document extraction and NLP services
- AWS S3, IAM, CloudWatch, EC2, ECR — core platform services used day-to-day
- AWS console proficiency — navigating, configuring, and monitoring services
- Understanding of server-side and client-side encryption within AWS
Infrastructure, DevOps & delivery
- Terraform — Infrastructure as Code for provisioning and managing AWS environments
- Docker — containerisation of data engineering solutions
- GitLab — source code management, CI/CD pipeline configuration, release tagging, and component versioning
- Familiarity with DWP engineering best practices
- Ability to translate customer expectations into applied, functional technical solutions
Technology stack at a glance
- Python
- PySpark
- SQL
- Apache Spark
- Apache Airflow
- Jupyter Notebooks
- Dimensional modelling / SCD
- AWS EMR
- Amazon Athena
- AWS S3
- AWS IAM
- AWS CloudWatch
- AWS EC2 / ECR
- Amazon Textract
- Amazon Comprehend
- Terraform
- Docker
- GitLab CI/CD
- GitLab Tags
Data Engineer (SC Cleared) in Newcastle upon Tyne employer: Scrumconnect Limited
Contact Detail:
Scrumconnect Limited Recruiting Team
StudySmarter Expert Advice 🤫
We think this is how you could land Data Engineer (SC Cleared) in Newcastle upon Tyne
✨Tip Number 1
Network like a pro! Reach out to folks in the industry, attend meetups, and connect with people on LinkedIn. You never know who might have the inside scoop on job openings or can refer you directly.
✨Tip Number 2
Show off your skills! Create a portfolio showcasing your data pipelines, projects, or any relevant work you've done. This gives potential employers a taste of what you can bring to the table.
✨Tip Number 3
Prepare for interviews by brushing up on your technical skills. Be ready to discuss Apache Spark, AWS services, and your experience with data modelling. Practice common interview questions to boost your confidence.
✨Tip Number 4
Apply through our website! It’s the best way to ensure your application gets seen. Plus, we love seeing candidates who are proactive about their job search.
We think you need these skills to ace Data Engineer (SC Cleared) in Newcastle upon Tyne
Some tips for your application 🫡
Show Off Your Skills: Make sure to highlight your experience with Apache Spark, Python, and AWS in your application. We want to see how you've used these tools in real-world scenarios, so don’t hold back!
Tailor Your Application: Customise your CV and cover letter to match the job description. Use keywords from the listing, like 'data pipelines' and 'cloud-native environment', to show us you’re a perfect fit for the role.
Be Clear and Concise: Keep your application straightforward and to the point. We appreciate clarity, so avoid jargon unless it’s relevant to the role. Make it easy for us to see why you’re the right candidate!
Apply Through Our Website: Don’t forget to submit your application through our website! It’s the best way for us to receive your details and ensures you’re considered for the role. We can’t wait to hear from you!
How to prepare for a job interview at Scrumconnect Limited
✨Know Your Tech Stack
Make sure you’re well-versed in the technologies mentioned in the job description, especially Apache Spark, PySpark, and AWS services. Brush up on your Python skills and be ready to discuss how you've used these tools in past projects.
✨Demonstrate Problem-Solving Skills
Prepare to showcase your analytical abilities by discussing specific instances where you identified and resolved data issues. Be ready to explain your thought process during root cause analysis and how you approached troubleshooting data pipelines.
✨Familiarise with Infrastructure as Code
Since Terraform is a key part of the role, ensure you understand how to provision and manage cloud infrastructure using it. You might be asked to explain how you’ve implemented Infrastructure as Code in previous roles, so have some examples ready.
✨Understand Data Governance
Given the importance of security and data governance in this role, brush up on IAM policies and encryption patterns within AWS. Be prepared to discuss how you would ensure compliance and data integrity in a regulated environment.