EmergingDataset curation and stewardship

Dataset Contamination Analyst

Also called Benchmark Leakage Analyst

A Dataset Contamination Analyst finds where training data overlaps with evaluation sets and recommends fixes. It matters because inflated scores mislead product, research and customer decisions, and the company needs results it can trust.

What they do

The analyst takes a training corpus and a set of evaluations and looks for overlap. That means exact matches, near duplicates, paraphrases and shared sources. They run similarity search at scale, sample and review the hits by hand, and trace each one back to where the data came from. They check dataset catalogues and consent registers to see what is allowed. They write up what they find, rate how serious it is, and agree fixes with the teams who own the data and the evaluations.

What a strong hire delivers

Within the first months, a strong hire gives the company a clear picture of where its main evaluations are exposed. They deliver a contamination report with ranked findings and practical remediation recommendations. They set up repeatable checks that run before new data or new benchmarks are adopted. They leave behind clean documentation, so scores can be defended to customers, partners and the team.

How you can tell they are good

Ask them to walk through a past contamination investigation. A strong candidate explains how they chose a detection method, how they judged false positives, and how they measured coverage. They separate harmless overlap from overlap that changes a result. They can say what they could not detect. Be wary of anyone who treats exact-match deduplication as the whole answer.

Why it is hard to fill

Hardly anyone has this title. The work often sits inside data science, evaluation or data governance jobs, so CVs rarely say it. The skills are split: similarity search and experimental design on one side, provenance, licensing and consent on the other. Candidates usually have one half and need to show the other. Hiring managers often search for the wrong title and miss them.

Where to find them

Look at data scientists who have worked on evaluation or deduplication, and at research engineers who have audited benchmarks. Information scientists, archivists and data governance specialists often bring strong provenance habits. Some sit in data quality teams at model builders, search companies, and organisations that publish or curate large datasets. Many are doing this work under another job title.

Indicative pay

Not enough published evidence yet to give a fair range for this role, which is common for work this new. We benchmark it for you as part of a search.

Interview questions for a Dataset Contamination Analyst

  1. Tell us about a time you found overlap between training data and an evaluation. How did you find it and what changed?

    Look for a specific case, the method used, how hits were verified and what action followed. A strong answer includes the effect on results. Worry if the story is vague, or if nothing was actually remediated.

  2. How would you detect paraphrased or translated copies of benchmark items in a large corpus?

    Good answers cover embeddings, approximate nearest neighbour search, thresholds, and manual review of samples. They admit limits. Worry if they only mention exact string matching or claim full detection is possible.

4 more questions, with what a strong answer covers, are in the Role Kit.

Job description you can post

We are looking for a Dataset Contamination Analyst to make sure our evaluation results can be trusted. You will investigate overlap between our training data and our evaluations, explain what you find, and work with research and data teams to fix it.

What you will do

  • Search training corpora for overlap with evaluation sets, including near duplicates and paraphrases
  • Review and classify findings, and trace them to their data sources
  • Write contamination reports with ranked findings and remediation recommendations
  • Agree fixes with data and evaluation owners and check they worked
  • Build repeatable checks for new datasets and new benchmarks
  • Keep dataset catalogue and consent records accurate and up to date

What you will bring

  • Demonstrated work on deduplication, similarity search or data quality at scale
  • Experience designing evaluations or analysing how data affects results
  • Clear understanding of data provenance and how to document it
  • Working knowledge of licensing and consent constraints on data
  • Ability to write findings that non-specialists can act on
  • Judgement about when overlap matters and when it does not

Nice to have

  • Experience with dataset catalogues or consent registers
  • Background in archives, information science or data governance
  • Experience auditing public benchmarks

Free to use and adapt. If it saves you time, a link back to this page is appreciated.

Skills that matter

  • Data quality and provenanceAdvancedTrack dataset origin, validity and transformations
  • Data stewardship and consentAdvancedMaintain permissions, retention and responsible data access
  • Experimental designWorkingChoose comparisons that identify credible effects
  • Intellectual property and licensingWorkingAssess rights in training inputs and generated outputs

Other roles in dataset curation and stewardship

Hiring nowTraining Dataset CuratorA Training Dataset Curator chooses, documents and maintains the data that models learn from. They make sure each dataset fits the model's goals…Hiring nowData Provenance SpecialistA Data Provenance Specialist traces where every dataset record came from and what was done to it, so the company can show it has the right to use…Hiring nowAI Data StewardAn AI Data Steward controls who can use each dataset, for what, and for how long. That keeps training and evaluation data within its rights, and…Hiring nowTraining Data Strategy LeadThe Training Data Strategy Lead decides what data the company buys, licenses, generates or collects, and manages the vendors and rights behind it…

Related roles in other families

Hiring nowPretraining Research EngineerFoundation model pretrainingA pretraining research engineer runs controlled experiments on how large models learn from broad data. The work decides which architectures, data…Hiring nowModel Architecture ResearcherFoundation model pretrainingA Model Architecture Researcher tests changes to how a model is built, to improve its behaviour or efficiency. The work tells a company which…EmergingScaling Science ResearcherFoundation model pretrainingA Scaling Science Researcher works out how model performance changes with compute, data and model size. Their estimates tell a company how to spend…EmergingTraining Dynamics ResearcherFoundation model pretrainingA Training Dynamics Researcher works out why large training runs wobble, stall or diverge, and fixes the cause. Fewer failed runs means less wasted…

How we wrote this

Role definitions are our own editorial work, checked against occupational classifications, standards, research and employer pages. Titles vary by company, and this is a description of the work, not a job advert.

← Dataset curation and stewardshipAll new job families

Start a search

Hiring a Dataset Contamination Analyst?

Tell us the work. We find the people already doing it, whatever their job title says, and you meet them already interviewed for your role.