Find a job 
Featured Companies 
Featured Universities 
For Companies At a Glance
- Tasks: Lead the migration of legacy SAS analytics to a modern PySpark ecosystem on AWS.
- Company: Join Randstad Technologies, a leader in tech recruitment and innovation.
- Benefits: Competitive salary, flexible working options, and opportunities for professional growth.
- Why this job: Make a significant impact by transforming financial data processing with cutting-edge technology.
- Qualifications: Expertise in PySpark, Python, and experience with AWS services required.
- Other info: Dynamic role with a focus on engineering leadership and innovative solutions.
The predicted salary is between 43200 - 72000 £ per year.
As the Technical Lead, you will drive the high-stakes migration of legacy SAS analytics to a modern, cloud-native PySpark ecosystem on AWS. This isn't just a lift and shift; you will refactor complex procedural logic into scalable, production-ready distributed pipelines for a Tier-1 financial services environment.
Core Responsibilities
- Engineering Leadership: Design and develop complex ETL/ELT pipelines and Data Marts using PySpark, EMR, and Glue.
- Legacy Modernisation: Architect the conversion of SAS Base/Macros into modular, testable Python code using SAS2PY and manual refactoring.
- Performance Tuning: Optimise Spark execution (partitioning, shuffling, caching) to ensure cost-efficient processing of massive financial datasets.
- Quality & Governance: Implement rigorous CI/CD, unit testing, and data reconciliation frameworks to ensure "penny-perfect" accuracy.
Technical Stack
- Engine: PySpark (Expert), Python (Clean Code/SOLID principles).
- AWS: EMR, Glue, S3, Athena, IAM, Lambda.
- Data Modeling: SCD Type 2, Fact/Dimension tables, Data Vault/Star Schema.
- Legacy: Proficiency in reading/debugging SAS (Base, Macros, DI Studio).
- DevOps: Git-based workflows, Jenkins/GitLab CI, Terraform.
Lead PySpark Engineer in City of London employer: Randstad Technologies Recruitment
Contact Detail:
Randstad Technologies Recruitment Recruiting Team
StudySmarter Expert Advice 🤫
We think this is how you could land Lead PySpark Engineer in City of London
✨Tip Number 1
Network like a pro! Reach out to your connections in the industry, especially those who work with PySpark or in financial services. A friendly chat can lead to insider info about job openings that aren't even advertised yet.
✨Tip Number 2
Show off your skills! Create a portfolio showcasing your PySpark projects, especially any legacy migrations or ETL pipelines you've worked on. This will give potential employers a taste of what you can do and set you apart from the crowd.
✨Tip Number 3
Prepare for technical interviews by brushing up on your knowledge of AWS services like EMR and Glue. Be ready to discuss how you've optimised Spark execution in past projects. We want to see your expertise shine!
✨Tip Number 4
Don't forget to apply through our website! It’s the best way to ensure your application gets noticed. Plus, we love seeing candidates who are proactive about their job search.
We think you need these skills to ace Lead PySpark Engineer in City of London
Some tips for your application 🫡
Tailor Your CV: Make sure your CV highlights your experience with PySpark and AWS. We want to see how you've tackled similar challenges in the past, especially around ETL/ELT pipelines and legacy modernisation.
Showcase Your Projects: Include specific projects where you've refactored complex logic or optimised Spark execution. We love seeing real-world examples of your work that demonstrate your engineering leadership skills.
Be Clear and Concise: When writing your application, keep it straightforward. We appreciate clarity, so avoid jargon unless it's relevant to the role. Make it easy for us to see why you're a great fit!
Apply Through Our Website: Don't forget to submit your application through our website! It helps us keep everything organised and ensures your application gets the attention it deserves.
How to prepare for a job interview at Randstad Technologies Recruitment
✨Know Your Tech Stack Inside Out
Make sure you’re well-versed in PySpark, AWS services like EMR and Glue, and the principles of clean code. Brush up on your knowledge of data modelling techniques and be ready to discuss how you’ve applied these in past projects.
✨Showcase Your Leadership Skills
As a Technical Lead, you’ll need to demonstrate your ability to guide a team. Prepare examples of how you’ve led engineering projects, tackled challenges, and fostered collaboration among team members. Highlight your experience in implementing CI/CD and testing frameworks.
✨Prepare for Problem-Solving Questions
Expect to face technical scenarios where you’ll need to refactor SAS code or optimise Spark execution. Practice explaining your thought process clearly and concisely, as this will showcase your analytical skills and ability to communicate complex ideas effectively.
✨Understand the Business Context
Since this role is within a Tier-1 financial services environment, it’s crucial to grasp the implications of your work on business outcomes. Familiarise yourself with the financial sector's data governance and compliance standards, and be ready to discuss how your technical decisions can drive value.