Find a job 
Featured Companies 
Featured Universities 
For Companies At a Glance
- Tasks: Lead the migration of legacy data workflows to high-performance AWS cloud using PySpark.
- Company: Join a dynamic team in the financial services sector, working remotely.
- Benefits: Enjoy 33 days holiday, competitive pay, and flexible remote work.
- Why this job: Make a significant impact on data modernisation in a fast-paced environment.
- Qualifications: 5+ years of PySpark experience and strong AWS skills required.
- Other info: Collaborate with a talented team and enhance your technical expertise.
The predicted salary is between 48000 - 72000 £ per year.
We are seeking a Lead PySpark Engineer to drive a large-scale data modernisation project, transitioning legacy data workflows into a high-performance AWS cloud environment. This is a hands-on technical role focused on converting legacy SAS code into production-ready PySpark pipelines within a complex financial services landscape.
Key Responsibilities
- Code Conversion: Lead the end-to-end migration of SAS code (Base SAS, Macros, DI Studio) to PySpark using automated tools (SAS2PY) and manual refactoring.
- Pipeline Engineering: Design, build, and troubleshoot complex ETL/ELT workflows and data marts on AWS.
- Performance Tuning: Optimise Spark workloads for execution efficiency, partitioning, and cost-effectiveness.
- Quality Assurance: Implement clean coding principles, modular design, and robust unit/comparative testing to ensure data accuracy throughout the migration.
- Engineering Excellence: Maintain Git-based workflows, CI/CD integration, and comprehensive technical documentation.
Technical Requirements
- PySpark (P3): 5+ years of hands-on experience writing scalable, production-grade PySpark/Spark SQL.
- AWS Data Stack (P3): Strong proficiency in EMR, Glue, S3, Athena, and Glue Workflows.
- SAS Knowledge (P1): Solid foundation in SAS to enable the understanding and debugging of legacy logic for conversion.
- Data Modeling: Expertise in ETL/ELT, dimensions, facts, SCDs, and data mart architecture.
- Engineering Quality: Experience with parameterisation, exception handling, and modular Python design.
Additional Details
- Industry: Financial Services experience is highly desirable.
- Working Pattern: Fully remote with internal team collaboration days.
- Benefits: 33 days holiday entitlement (pro-rata).
PySpark Developer in London employer: Randstad Technologies Recruitment
Contact Detail:
Randstad Technologies Recruitment Recruiting Team
StudySmarter Expert Advice 🤫
We think this is how you could land PySpark Developer in London
✨Tip Number 1
Network like a pro! Reach out to your connections in the industry, especially those who work with PySpark or in financial services. A friendly chat can lead to insider info about job openings that aren't even advertised yet.
✨Tip Number 2
Show off your skills! Create a portfolio showcasing your PySpark projects, especially any cloud migration work. This gives potential employers a taste of what you can do and sets you apart from the crowd.
✨Tip Number 3
Prepare for interviews by brushing up on common technical questions related to PySpark and AWS. We recommend doing mock interviews with friends or using online platforms to get comfortable with the format.
✨Tip Number 4
Don't forget to apply through our website! It’s the best way to ensure your application gets seen. Plus, we love seeing candidates who are proactive and eager to join our team.
We think you need these skills to ace PySpark Developer in London
Some tips for your application 🫡
Read the Job Description Thoroughly: Before you start your application, take a good look at the job description. It’s packed with info about what we’re looking for in a Lead PySpark Engineer, so make sure you understand the key responsibilities and technical requirements.
Tailor Your CV and Cover Letter: Don’t just send out a generic CV! Highlight your experience with PySpark, AWS, and any relevant projects you've worked on. We want to see how your skills match up with what we need, so make it personal!
Showcase Your Technical Skills: Since this role is all about hands-on technical work, be sure to include specific examples of your coding experience, especially with PySpark and AWS. Mention any successful projects where you’ve optimised performance or led migrations.
Apply Through Our Website: We encourage you to apply directly through our website. It’s the best way to ensure your application gets into the right hands quickly. Plus, it shows us you’re keen to join the StudySmarter team!
How to prepare for a job interview at Randstad Technologies Recruitment
✨Know Your PySpark Inside Out
Make sure you brush up on your PySpark skills before the interview. Be ready to discuss your past projects, especially those involving code conversion and pipeline engineering. Highlight specific challenges you faced and how you optimised Spark workloads.
✨Familiarise Yourself with AWS Tools
Since this role involves working with AWS, get comfortable with tools like EMR, Glue, and S3. You might be asked about how you've used these in previous roles, so have some examples ready that showcase your experience and problem-solving skills.
✨Understand the Financial Services Landscape
If you have experience in financial services, make sure to mention it! If not, do a bit of research on common data challenges in this sector. Showing that you understand the industry can set you apart from other candidates.
✨Prepare for Technical Questions
Expect technical questions that test your knowledge of ETL/ELT processes and data modelling. Brush up on clean coding principles and be prepared to discuss how you ensure data accuracy and quality assurance in your work.
