Find a job 
Featured Companies 
Featured Universities 
For Companies At a Glance
- Tasks: Lead the migration of legacy data workflows to high-performance AWS cloud using PySpark.
- Company: Join a dynamic team in the financial services sector focused on data modernisation.
- Benefits: Enjoy 33 days holiday, remote work, and a collaborative team environment.
- Why this job: Make a real impact by transforming data processes with cutting-edge technology.
- Qualifications: 5+ years of PySpark experience and strong AWS data stack knowledge required.
- Other info: Fully remote role with opportunities for professional growth and collaboration.
The predicted salary is between 48000 - 72000 £ per year.
We are seeking a Lead PySpark Engineer to drive a large-scale data modernisation project, transitioning legacy data workflows into a high-performance AWS cloud environment. This is a hands-on technical role focused on converting legacy SAS code into production-ready PySpark pipelines within a complex financial services landscape.
Key Responsibilities
- Code Conversion: Lead the end-to-end migration of SAS code (Base SAS, Macros, DI Studio) to PySpark using automated tools (SAS2PY) and manual refactoring.
- Pipeline Engineering: Design, build, and troubleshoot complex ETL/ELT workflows and data marts on AWS.
- Performance Tuning: Optimise Spark workloads for execution efficiency, partitioning, and cost-effectiveness.
- Quality Assurance: Implement clean coding principles, modular design, and robust unit/comparative testing to ensure data accuracy throughout the migration.
- Engineering Excellence: Maintain Git-based workflows, CI/CD integration, and comprehensive technical documentation.
Technical Requirements
- PySpark (P3): 5+ years of hands-on experience writing scalable, production-grade PySpark/Spark SQL.
- AWS Data Stack (P3): Strong proficiency in EMR, Glue, S3, Athena, and Glue Workflows.
- SAS Knowledge (P1): Solid foundation in SAS to enable the understanding and debugging of legacy logic for conversion.
- Data Modeling: Expertise in ETL/ELT, dimensions, facts, SCDs, and data mart architecture.
- Engineering Quality: Experience with parameterisation, exception handling, and modular Python design.
Additional Details
- Industry: Financial Services experience is highly desirable.
- Working Pattern: Fully remote with internal team collaboration days.
- Benefits: 33 days holiday entitlement (pro-rata).
Pyspark Developer employer: Randstad Technologies Recruitment
Contact Detail:
Randstad Technologies Recruitment Recruiting Team
StudySmarter Expert Advice 🤫
We think this is how you could land Pyspark Developer
✨Tip Number 1
Network like a pro! Reach out to your connections in the industry, especially those who work with PySpark or in financial services. A friendly chat can lead to insider info about job openings that aren't even advertised yet.
✨Tip Number 2
Show off your skills! Create a GitHub repository showcasing your PySpark projects or any relevant code you've worked on. This gives potential employers a sneak peek into your coding style and problem-solving abilities.
✨Tip Number 3
Prepare for technical interviews by brushing up on your PySpark knowledge and AWS tools. Practice common coding challenges and be ready to discuss your past projects in detail. We want you to shine!
✨Tip Number 4
Don't forget to apply through our website! It’s the best way to ensure your application gets seen by the right people. Plus, we love seeing candidates who take the initiative to connect directly with us.
We think you need these skills to ace Pyspark Developer
Some tips for your application 🫡
Tailor Your CV: Make sure your CV is tailored to the Lead PySpark Engineer role. Highlight your experience with PySpark, AWS, and any relevant projects you've worked on. We want to see how your skills match what we're looking for!
Craft a Compelling Cover Letter: Your cover letter is your chance to shine! Use it to explain why you're the perfect fit for this role. Share specific examples of your past work with data migration and pipeline engineering that align with our needs.
Showcase Your Technical Skills: Don’t forget to highlight your technical expertise in PySpark and AWS. Mention any tools or methodologies you’ve used, like SAS2PY for code conversion. We love seeing candidates who can demonstrate their hands-on experience!
Apply Through Our Website: We encourage you to apply through our website for a smoother application process. It helps us keep track of your application and ensures you don’t miss out on any important updates from us!
How to prepare for a job interview at Randstad Technologies Recruitment
✨Know Your PySpark Inside Out
Make sure you brush up on your PySpark skills before the interview. Be ready to discuss your experience with writing scalable, production-grade PySpark code and be prepared to explain how you've optimised Spark workloads in the past.
✨Understand the AWS Data Stack
Familiarise yourself with the AWS services mentioned in the job description, like EMR, Glue, and S3. Having specific examples of how you've used these tools in previous projects will show that you're not just a theoretical expert but someone who can apply their knowledge practically.
✨Showcase Your Code Conversion Experience
Since the role involves converting SAS code to PySpark, be ready to discuss any relevant experiences you have. If you've worked with automated tools like SAS2PY or done manual refactoring, share those stories to demonstrate your hands-on expertise.
✨Emphasise Quality Assurance Practices
Talk about your approach to clean coding principles and testing. Highlight any experience you have with unit testing and ensuring data accuracy during migrations, as this is crucial for the role. Showing that you prioritise engineering excellence will set you apart.
