EmergingScientific machine learning and research computing

AI Reproducibility Specialist

Also called AI Research Reproducibility Analyst

An AI Reproducibility Specialist independently reruns AI-supported scientific claims and reports what holds and what does not. They protect the company from building products, papers or decisions on results that cannot be repeated.

What they do

They take a claimed result, such as a model prediction, a surrogate simulation or a discovery pipeline, and try to reproduce it from the ground up. They read the paper and the code, rebuild the environment, trace data and parameters back to source, and rerun on notebooks or HPC. They design the replication itself: what to hold fixed, what to vary, and what counts as agreement. When results differ, they work out why. The causes might be seeds, library versions, data leakage, numerical drift or a genuine flaw. They write it up clearly for scientists and engineers alike.

What a strong hire delivers

Within the first months, a strong hire produces a replication package that another team can run without asking questions. They deliver a discrepancy report that separates real deviations from noise and names causes. They agree a standard for what counts as reproduced, with tolerances and uncertainty stated. They also leave behind provenance habits that make later claims easier to check, and they give leadership a clear view of which results can be trusted.

How you can tell they are good

Ask them to walk through a result they tried to reproduce, including one that failed. Good candidates explain how they set tolerances, how they separated randomness from error, and how they recorded provenance. They name what they could not verify. Be cautious of anyone who treats reproduction as simply rerunning code, or who cannot say what would have changed their mind about a claim.

Why it is hard to fill

Hardly anyone holds this title. The work usually sits inside research engineering, scientific computing or a domain lab, so searching by title finds almost nobody. The mix is awkward: someone needs deep learning fluency, real domain knowledge and the discipline of an auditor. Many strong people prefer building new models to checking old ones, and independence makes the role harder to place within a team.

Where to find them

Look at research software engineers, computational scientists and applied mathematicians who have maintained shared codebases or reviewed others' work. Good candidates also come from physics, chemistry, biology or earth science groups that use learning methods. Some have worked on reproducibility initiatives, artefact evaluation or internal model validation. Labs, scientific software teams, university research computing groups and AI for science start-ups are all worth exploring.

Indicative pay

Not enough published evidence yet to give a fair range for this role, which is common for work this new. We benchmark it for you as part of a search.

Interview questions for an AI Reproducibility Specialist

  1. Describe a scientific result you tried to reproduce. What did you do first, and what was the outcome?

    Strong answers start with reading the claim closely, then rebuilding the environment and data lineage before running anything. They give a clear outcome, including partial success. Worry if the story is only about rerunning supplied code.

  2. A rerun gives results close to the claim but not identical. How do you decide whether that counts as reproduced?

    Look for tolerances set in advance, reasoning about seeds, hardware and numerical precision, and use of uncertainty estimates. Worry about vague judgement calls or a demand for exact matches in stochastic settings.

4 more questions, with what a strong answer covers, are in the Role Kit.

Job description you can post

We build AI systems that support scientific modelling and discovery. As claims from our models and papers carry real weight, we need someone independent to reproduce them. You will rerun results from scratch, explain any differences and help us know what we can trust.

What you will do

  • Reproduce AI-supported scientific results independently from code, data and papers
  • Design replication plans with clear tolerances and success criteria
  • Build replication packages others can run end to end
  • Trace data, parameters and environments to establish computational provenance
  • Investigate discrepancies and identify their causes
  • Write discrepancy reports for scientific and engineering audiences
  • Advise teams on making future results easier to reproduce

What you will bring

  • Demonstrated reproduction or validation of complex computational results
  • Strong scientific Python and comfort with notebooks and HPC environments
  • Working knowledge of deep learning and how it can fail
  • Grounding in a scientific discipline or applied mathematics
  • Experience designing experiments and quantifying uncertainty
  • Clear writing that explains technical findings to mixed audiences

Nice to have

  • Experience with differentiable simulators
  • Contributions to research software or reproducibility initiatives
  • Experience reviewing or auditing others' scientific code

Free to use and adapt. If it saves you time, a link back to this page is appreciated.

Skills that matter

  • Scientific domain expertiseAdvancedApply disciplinary theory and experimental standards
  • Deep learningAdvancedBuild and diagnose trainable neural models
  • Experimental designWorkingChoose comparisons that identify credible effects
  • Uncertainty quantificationWorkingEstimate confidence, calibration and model uncertainty

Other roles in scientific machine learning and research computing

Hiring nowScientific Machine Learning ScientistA Scientific Machine Learning Scientist builds learned models that obey the physics, chemistry or biology of a problem. They matter because a model…Hiring nowAI Research Software EngineerAn AI Research Software Engineer turns scientific machine learning ideas into software that other people can run, trust and extend. They give a…EmergingAI Scientific Workflow EngineerAn AI Scientific Workflow Engineer connects data, models and instruments into reproducible science pipelines. They make sure results can be traced…

Related roles in other families

Hiring nowComputer Vision EngineerComputer vision and spatial perceptionA Computer Vision Engineer builds detection and recognition systems that work in a defined setting, and proves how well they work. Companies rely on…Hiring now3D Vision ResearcherComputer vision and spatial perceptionA 3D Vision Researcher recovers spatial structure from images and depth data, then proves it holds up. Companies building robotics, mapping, AR or…Hiring nowVideo Understanding ScientistComputer vision and spatial perceptionA Video Understanding Scientist builds models that recognise actions, events and the dependencies between them in video. Companies hire for this…EmergingOpen-World Perception ResearcherComputer vision and spatial perceptionAn Open-World Perception Researcher builds vision models that know when they are looking at something unfamiliar. That matters because a system that…

How we wrote this

Role definitions are our own editorial work, checked against occupational classifications, standards, research and employer pages. Titles vary by company, and this is a description of the work, not a job advert.

← Scientific machine learning and research computingAll new job families

Start a search

Hiring an AI Reproducibility Specialist?

Tell us the work. We find the people already doing it, whatever their job title says, and you meet them already interviewed for your role.