Staff Data Engineer | Contract | Up to £1,100/day | Inside IR35 in London

Staff Data Engineer | Contract | Up to £1,100/day | Inside IR35

Location: Hybrid – London (3 days per week on-site)
Rate: Up to £1,100/day
Contract: Initial 6 months
IR35: Inside IR35

We're supporting a major UK financial services organisation through a significant programme of data engineering modernisation and transformation.

We're looking for a genuinely Staff-level Data Engineer who combines exceptional hands-on Python/PySpark engineering with strong architectural thinking and practical experience of AI-assisted software development.

This is not a hands-off architecture role. You need to be equally comfortable setting technical direction, challenging engineering approaches and independently writing, debugging and optimising production-quality code.

The sweet spot is someone who can operate at a strategic level while remaining an outstanding hands-on engineer.

The roleYou'll play a key role in modernising a large-scale enterprise data estate, including the migration and transformation of thousands of data pipelines, development of new data products and adoption of modern, AI-assisted engineering practices.

Responsibilities will include:

• Providing Staff-level technical leadership across complex data engineering and modernisation initiatives.
• Remaining deeply hands-on with Python, PySpark and Apache Spark, independently building and proving engineering solutions.
• Designing and optimising scalable data pipelines and processing frameworks across AWS.
• Tackling complex Spark performance challenges, including shuffles, joins, data skew, partitioning and memory optimisation.
• Designing and challenging engineering approaches across AWS Glue, Apache Airflow and associated technologies.
• Reviewing existing code, identifying performance bottlenecks and driving improvements in engineering quality.
• Establishing reusable engineering patterns, frameworks and approaches to accelerate large-scale data migrations.
• Applying AI-assisted development tools to improve engineering productivity, code quality and delivery speed.
• Challenging technical assumptions, influencing architectural decisions and raising engineering standards across multiple teams.

What we're looking forCore engineering capability

• Exceptional hands-on Python/PySpark and Apache Spark expertise, with the ability to independently write, debug and optimise complex production code.
• Deep understanding of Spark internals and performance optimisation, including execution plans, joins, shuffles, partitioning, memory management and data skew.
• Strong AWS data engineering, particularly AWS Glue.
• Strong Apache Airflow experience.
• Excellent software engineering fundamentals, including code quality, testing, performance and maintainability.
• Experience designing and delivering large-scale data pipeline migrations or modernisation programmes.

Technical leadership and AI-enabled engineering

• Experience operating at Staff, Principal or Senior Lead Engineer level, or demonstrably performing at that standard.
• Ability to independently assess complex engineering challenges, challenge existing approaches and shape scalable solutions.
• Strong architectural and conceptual understanding, underpinned by genuine hands-on delivery experience.
• Demonstrable experience using AI-assisted development tools (such as Claude, GitHub Copilot or similar) to accelerate real-world software engineering delivery.
• Ability to critically review, validate and improve AI-generated code rather than relying on it without understanding.
• Experience influencing engineering practices and technical direction across multiple teams.

Beneficial experience

• Snowflake.
• Legacy ETL modernisation and migration.
• Building reusable migration frameworks or automation to accelerate large-scale transformation programmes.
• Financial services or other complex, highly regulated environments.

What makes someone successful in this role?We're looking for a relatively rare combination of capabilities:
A genuinely excellent hands-on Spark engineer who can think architecturally, solve complex problems independently and use AI to accelerate engineering delivery.

You'll need to be comfortable discussing high-level technical strategy one minute and getting directly into complex PySpark code the next.

This is an opportunity to have a meaningful impact on a major enterprise transformation programme while remaining at the forefront of hands-on engineering.