At a Glance
- Tasks: Lead the migration of legacy data workflows to AWS using PySpark.
- Company: Join a dynamic team in the financial services sector.
- Benefits: Enjoy 33 days holiday and fully remote work flexibility.
- Other info: Collaborate with a talented team while working remotely.
- Why this job: Make a significant impact on data modernisation in a high-performance environment.
- Qualifications: 5+ years of PySpark experience and strong AWS skills required.
The predicted salary is between 48000 - 72000 £ per year.
We are seeking a Lead PySpark Engineer to drive a large-scale data modernisation project, transitioning legacy data workflows into a high-performance AWS cloud environment. This is a hands-on technical role focused on converting legacy SAS code into production-ready PySpark pipelines within a complex financial services landscape.
Key Responsibilities
- Code Conversion: Lead the end-to-end migration of SAS code (Base SAS, Macros, DI Studio) to PySpark using automated tools (SAS2PY) and manual refactoring.
- Pipeline Engineering: Design, build, and troubleshoot complex ETL/ELT workflows and data marts on AWS.
- Performance Tuning: Optimise Spark workloads for execution efficiency, partitioning, and cost-effectiveness.
- Quality Assurance: Implement clean coding principles, modular design, and robust unit/comparative testing to ensure data accuracy throughout the migration.
- Engineering Excellence: Maintain Git-based workflows, CI/CD integration, and comprehensive technical documentation.
Technical Requirements
- PySpark (P3): 5+ years of hands-on experience writing scalable, production-grade PySpark/Spark SQL.
- AWS Data Stack (P3): Strong proficiency in EMR, Glue, S3, Athena, and Glue Workflows.
- SAS Knowledge (P1): Solid foundation in SAS to enable the understanding and debugging of legacy logic for conversion.
- Data Modeling: Expertise in ETL/ELT, dimensions, facts, SCDs, and data mart architecture.
- Engineering Quality: Experience with parameterisation, exception handling, and modular Python design.
Additional Details
- Industry: Financial Services experience is highly desirable.
- Working Pattern: Fully remote with internal team collaboration days.
- Benefits: 33 days holiday entitlement (pro-rata).
Locations
Data & AI - LLM Model Developer(PySpark Engineer) in Hampshire, Portsmouth employer: Randstad Digital
As a leading employer in the financial services sector, we offer a dynamic and supportive work environment that prioritises employee growth and innovation. Our remote working model allows for flexibility while fostering collaboration through regular team engagement, and our commitment to professional development ensures that you will continually enhance your skills in cutting-edge technologies like PySpark and AWS. With generous benefits including 33 days of holiday entitlement, we strive to create a rewarding experience for all our employees.
StudySmarter Expert Advice🤫
We think this is how you could land Data & AI - LLM Model Developer(PySpark Engineer) in Hampshire, Portsmouth
✨Tip Number 1
Network like a pro! Reach out to your connections in the data and AI space, especially those who work with PySpark. A friendly chat can lead to insider info about job openings that aren't even advertised yet.
✨Tip Number 2
Show off your skills! Create a portfolio showcasing your PySpark projects or any relevant work you've done. This gives potential employers a taste of what you can bring to the table, especially for that hands-on role.
✨Tip Number 3
Prepare for interviews by brushing up on common PySpark scenarios and AWS tools. Practise explaining your thought process when tackling complex ETL workflows, as this will demonstrate your expertise and problem-solving skills.
✨Tip Number 4
Don't forget to apply through our website! We love seeing candidates who are genuinely interested in joining us. Plus, it makes it easier for us to keep track of your application and get back to you quickly.
We think you need these skills to ace Data & AI - LLM Model Developer(PySpark Engineer) in Hampshire, Portsmouth
Some tips for your application 🫡
Tailor Your CV:Make sure your CV highlights your experience with PySpark and AWS. We want to see how your skills match the job description, so don’t be shy about showcasing your relevant projects!
Craft a Compelling Cover Letter:Your cover letter is your chance to shine! Use it to explain why you’re the perfect fit for this role. We love seeing passion and enthusiasm, so let us know what excites you about working with us at StudySmarter.
Showcase Your Technical Skills:Since this role is all about technical expertise, make sure to include specific examples of your work with PySpark, AWS, and data migration. We want to see your problem-solving skills in action!
Apply Through Our Website:We encourage you to apply directly through our website. It’s the best way to ensure your application gets into the right hands. Plus, it shows us you’re keen on joining the StudySmarter team!
How to prepare for a job interview at Randstad Digital
✨Know Your PySpark Inside Out
Make sure you brush up on your PySpark skills before the interview. Be ready to discuss your experience with writing scalable, production-grade PySpark code and how you've tackled performance tuning in past projects. They’ll want to see that you can not only code but also optimise workflows effectively.
✨Understand the AWS Data Stack
Familiarise yourself with the AWS services mentioned in the job description, like EMR, Glue, and S3. Prepare to explain how you've used these tools in previous roles, especially in relation to data migration and ETL/ELT processes. Showing that you can navigate the AWS environment will give you a leg up.
✨Showcase Your SAS Knowledge
Since this role involves converting legacy SAS code, it’s crucial to demonstrate your understanding of SAS. Be prepared to discuss any past experiences where you’ve worked with SAS, and how you approached code conversion challenges. This will show that you can bridge the gap between old and new technologies.
✨Emphasise Engineering Excellence
Talk about your commitment to clean coding principles and robust testing practices. Share examples of how you've implemented CI/CD workflows and maintained technical documentation in your previous roles. This will highlight your dedication to engineering quality, which is key for this position.
