About the job

Senior Data Engineer – Entity Resolution

£500-550 | Inside IR35 | London | Hybrid, 2 days onsite per week

We are looking for a Senior Data Engineer specialising in Entity Resolution and Data Matching to support a major UK Government programme.

This is a senior, hands-on role focused on designing and delivering complex data-matching solutions across large and sensitive datasets. You will work with modern data engineering technologies including Python, PySpark, Apache Spark, AWS and Databricks, while applying advanced matching techniques to improve the accuracy, quality and usability of data.

You’ll be working on complex problems involving record linkage, identity matching, deduplication and probabilistic matching, so we’re looking for someone who combines strong Data Engineering capability with genuine Entity Resolution expertise.

Key experience

• Strong commercial experience in Entity Resolution, Data Matching or Record Linkage
• Experience designing deterministic, probabilistic and fuzzy matching solutions
• Strong understanding of matching rules, blocking strategies, thresholds and confidence scoring
• Experience analysing and reducing false positives and false negatives
• Advanced Python, PySpark and SQL
• Strong hands-on experience with Apache Spark
• Experience working with large-scale data platforms using AWS EMR and/or Databricks
• AWS experience across technologies such as S3, Glue and Athena
• Strong Data Quality experience including profiling, cleansing, standardisation and deduplication
• Experience working with complex, incomplete or inconsistent datasets
• Good understanding of Data Governance, lineage, metadata, auditability and privacy
• Experience providing technical leadership, assurance and peer review

Highly desirable

• Splink experience
• Quantexa or another Entity Resolution platform
• Experience developing reusable probabilistic matching frameworks
• UK Government or Central Government experience
• Previous Home Office, HMRC, DWP, ONS, NHS or similar public-sector experience
• Experience working with population or administrative datasets
• Experience matching personal identifiers such as National Insurance Numbers
• Strong knowledge of Data Quality frameworks and governance controls
• Experience working with sensitive personal data in highly regulated environments

The role

You will help define and implement robust matching methodologies, develop scalable data-processing solutions and provide technical assurance around how records and entities are linked.

You should be comfortable explaining why records have been matched, how match confidence has been calculated and how the solution has been validated. This is not simply an ETL/Data Engineering role. Strong Entity Resolution or Data Matching experience is essential.

Location: London

Working pattern: Hybrid, approximately 2 days onsite per week

Rate: £550 per day

IR35: Inside IR35

If you have strong Data Engineering + Entity Resolution / Record Linkage experience and are interested in a major UK Government programme, please apply with your latest CV.