Senior Data Engineer | Python | Spark | Data Matching | SC Preferred
£500-550 | Inside IR35 | London | Hybrid, 2 days onsite per week
We are looking for a Senior Data Engineer specialising in Entity Resolution and Data Matching to support a major UK Government programme.
This is a senior, hands-on role focused on designing and delivering complex data-matching solutions across large and sensitive datasets. You will work with modern data engineering technologies including Python, PySpark, Apache Spark, AWS and Databricks, while applying advanced matching techniques to improve the accuracy, quality and usability of data.
You'll be working on complex problems involving record linkage, identity matching, deduplication and probabilistic matching, so we're looking for someone who combines strong Data Engineering capability with genuine Entity Resolution expertise.
Key experience
- Strong commercial experience in Entity Resolution, Data Matching or Record Linkage
- Experience designing deterministic, probabilistic and fuzzy matching solutions
- Strong understanding of matching rules, blocking strategies, thresholds and confidence scoring
- Experience analysing and reducing false positives and false negatives
- Advanced Python, PySpark and SQL
- Strong hands-on experience with Apache Spark
- Experience working with large-scale data platforms using AWS EMR and/or Databricks
- AWS experience across technologies such as S3, Glue and Athena
- Strong Data Quality experience including profiling, cleansing, standardisation and deduplication
- Experience working with complex, incomplete or inconsistent datasets
- Good understanding of Data Governance, lineage, metadata, auditability and privacy
- Experience providing technical leadership, assurance and peer review
Highly desirable
- Quantexa or another Entity Resolution platform
- UK Government or Central Government experience
- Previous Home Office, HMRC, DWP, ONS, NHS or similar public-sector experience
- Experience working with population or administrative datasets
- Experience matching personal identifiers such as National Insurance Numbers
- Strong knowledge of Data Quality frameworks and governance controls
- Experience working with sensitive personal data in highly regulated environments
The role
You will help define and implement robust matching methodologies, develop scalable data-processing solutions and provide technical assurance around how records and entities are linked.
You should be comfortable explaining why records have been matched, how match confidence has been calculated and how the solution has been validated. This is not simply an ETL/Data Engineering role. Strong Entity Resolution or Data Matching experience is essential.
Location: London
Working pattern: Hybrid, approximately 2 days onsite per week
IR35: Inside IR35
#J-18808-Ljbffr
Senior Data Engineer | Python | Spark | Data Matching | SC Preferred employer: Venturi
Venturi is an exceptional employer that fosters a collaborative and innovative work culture, allowing you to thrive in a predominantly remote setting while occasionally connecting with colleagues in South London. With a strong emphasis on employee growth, we provide ample opportunities for professional development in cutting-edge AI technologies, ensuring that your contributions are both meaningful and impactful within the tech landscape.
