AWS Data Engineer | Python, PySpark, Airflow, AWS EMR, Terraform | Award-Winning Public Services Consultancy | ****Active SC Clearence required****
- Salary: up to £80,000
- Location: Newcastle, on site 3 days a week
- ****Active SC required****
Our client is a multi-award-winning digital consultancy that builds and runs services for UK government departments, the sort of platforms tens of millions of people end up using without ever thinking about it. This role sits hands‑on in their data engineering team, building the AWS pipelines that carry that work.
You'll write Python and PySpark for Spark workloads on Amazon EMR, orchestrate them through Apache Airflow DAGs, and provision the whole environment as Infrastructure as code (IaC) with Terraform, containerised in Docker and shipped through GitLab CI/CD.
Non‑Negotiables
- Python: primary language for pipeline development and processing
- PySpark and Apache Spark: distributed processing on AWS EMR
- SQL: querying, transformation and validation across data stores
- Apache Airflow: writing DAGs and orchestrating tasks at scale
- AWS EMR: running Spark workloads and reading output logs
- Terraform and Infrastructure as Code (IaC) on AWS
- GitLab CI/CD: pipeline config, release tagging, component versioning
- Dimensional modelling and slowly changing dimensions (SCD Types 1, 2, 3)
- ****Active SC****
What You'll Work With
- Amazon Athena: ad hoc querying over S3
- AWS S3, IAM, CloudWatch, EC2 and ECR day to day
- Jupyter Notebooks: exploratory analysis and prototyping
- Server‑side and client‑side encryption patterns in AWS
- Root cause analysis across pipeline failures and data quality issues
- Data governance standards in a regulated government setting
- You'll turn customer expectations into working technical solutions
Nice to Haves
- AI and ML services applied to document-heavy datasets
- Public sector or other regulated delivery experience
Why Join / Projects
The data is public service data, so it matters and it gets used. You'll see your pipelines carrying real caseloads rather than sitting in a proof of concept, and you'll move across departments and problems instead of maintaining one warehouse for years.
You own your pipelines end to end. When something breaks you're in the EMR logs and CloudWatch metrics working out why, not raising a ticket for someone else. Infrastructure goes in as Terraform , deployments go out through GitLab , and the engineering standards are yours to Push on where they need tightening.
Employee Benefits
- Salary: up to £80,000
- Private health cover
- Rewards Gateway discounts and perks platform
- 25 days holiday
- Sponsorship through Security Check (SC) clearance
AWS Data Engineer | Python, PySpark, Airflow, AWS EMR, Terraform | Award-Winning Public Services Consultancy | ****Active SC Clearence required****
#J-18808-Ljbffr
AWS Data Engineer in Newcastle upon Tyne employer: Prism Digital
Join a dynamic and innovative AI consultancy in London, where you'll have the opportunity to lead hands-on data projects while working closely with a supportive team. With a flat structure and direct access to leadership, you can expect a collaborative work culture that fosters personal growth and offers a clear path to advancing your career as the company scales. Enjoy a flexible hybrid working model, competitive salary, and performance bonuses that reward your contributions to client success.
