Hiring nowCapability evaluation and benchmarking

Benchmark Research Scientist

Also called AI Benchmark Scientist · Benchmark Designer · Capability Evaluation Researcher

A Benchmark Research Scientist designs valid tests for a specific capability and shows the test measures what it claims to. Their work tells a company what its systems can really do, and where they fail.

What they do

The work starts with a construct. Someone says they want to measure a capability, and the scientist pins down what that means in practice. They design tasks, write scoring rules and build or source datasets, keeping those datasets secured so they do not leak into training. They run pilots on several systems and look at the spread of results. They check whether scores are stable, whether easy shortcuts exist and whether the test tracks the real skill. They write up what the benchmark can and cannot show. Then they revise it, again and again.

What a strong hire delivers

Within the first months, a strong hire produces a benchmark for one clearly defined capability, with a validation report and a plain account of its limits. Scoring is reproducible by someone else. The team gets a shared definition of the construct, a view on how much to trust the results, and a method it can reuse for the next benchmark.

How you can tell they are good

Ask them to take a vague capability and turn it into a test plan. Good candidates define the construct first, name the ways the test could mislead, and describe how they would check validity and reliability on independent systems. Ask what went wrong in a past benchmark. Worry if they talk only about dataset size or leaderboard rank.

Why it is hard to fill

Almost nobody carries this title, so searching on it finds few people. The skill sits across experimental science, statistics, psychometrics and software testing, and most candidates are strong in one or two. Those who have validated a benchmark that others used are rare, and labs and evaluation organisations tend to keep them. Expect a search to be about spotting transferable measurement judgement.

Where to find them

Look in labs, AI safety and security institutes and independent evaluation organisations first. Beyond those, strong candidates come from experimental science, applied statistics, psychometrics and test development in education or health. Some are software testers who moved into model evaluation. Others are domain experts, such as clinicians or lawyers, who built assessments in their own field and learned the technical side.

Indicative pay

Not enough published evidence yet to give a fair range for this role, which is common for work this new. We benchmark it for you as part of a search.

Interview questions for a Benchmark Research Scientist

  1. Tell me about a benchmark or test you designed. What construct was it meant to measure, and how did you know it did?

    Strong answers define the construct clearly and describe concrete validity checks, such as comparison with other measures or expert review. They admit weaknesses. Worry if the answer centres on dataset size, or if validity was simply assumed.

  2. A benchmark's scores are saturating across several systems. What do you do?

    Good answers ask whether the test is too easy, contaminated or measuring something narrow. They consider harder items, item analysis and retiring the benchmark. Worry about reflexively adding more questions without diagnosing the cause.

4 more questions, with what a strong answer covers, are in the Role Kit.

Job description you can post

We build AI systems and need to know what they can actually do. We are hiring a Benchmark Research Scientist to design valid, reliable tests for specified capabilities and to report honestly on their limits. You will work closely with researchers and engineers who rely on your results.

What you will do

  • Turn a target capability into a clearly defined construct and test plan
  • Design tasks, scoring rules and datasets for new benchmarks
  • Run pilots and analyse results across several systems
  • Validate benchmarks for construct validity and reliability
  • Keep test data secured and scoring reproducible
  • Write validation and limitations reports for technical and non-technical readers

What you will bring

  • Shown work designing or validating tests, benchmarks or experiments
  • Strong grounding in probability, statistics and experimental design
  • Experience with evaluation harnesses and reproducible scoring
  • Care about data quality and provenance
  • Clear written communication about uncertainty and limits
  • Judgement to challenge weak results, including your own

Nice to have

  • Background in psychometrics or assessment design
  • Deep expertise in a specific domain, such as medicine or law
  • Experience evaluating systems from several model providers

Free to use and adapt. If it saves you time, a link back to this page is appreciated.

Skills that matter

  • Model evaluationAdvancedMeasure capability and failure modes credibly
  • Experimental designAdvancedChoose comparisons that identify credible effects
  • Probability and statisticsWorkingQuantify variability and statistical evidence
  • Data quality and provenanceWorkingTrack dataset origin, validity and transformations

Other roles in capability evaluation and benchmarking

Hiring nowModel Evaluation EngineerA Model Evaluation Engineer builds the tests that show what an AI system can do and where it breaks. Without them, a team cannot tell whether a…Hiring nowDangerous Capability EvaluatorA dangerous capability evaluator tests whether frontier AI systems can do things that could cause serious harm, under secured and authorised…Hiring nowDomain AI EvaluatorA Domain AI Evaluator judges model outputs against the standards of a specific profession. They give a company scores it can trust, and they catch…EmergingAgent Performance AnalystAn Agent Performance Analyst monitors and scores the output quality of AI agents running in production, and ties it to business KPIs. They tell the…

Related roles in other families

Hiring nowMachine Learning Research ScientistLearning theory and statistical machine learningA Machine Learning Research Scientist develops and tests learning methods against explicit hypotheses. They give the company evidence about what…Hiring nowStatistical Learning ScientistLearning theory and statistical machine learningA Statistical Learning Scientist builds prediction methods that show their assumptions and their uncertainty. For a company shipping models into…Hiring nowLearning Theory ResearcherLearning theory and statistical machine learningA Learning Theory Researcher works out what learning algorithms can and cannot do, and proves it. For an AI company, that means firmer ground for…Hiring nowAutomated Machine Learning ResearcherLearning theory and statistical machine learningAn Automated Machine Learning Researcher builds search procedures that choose and tune models within a fixed compute budget. It matters because…

How we wrote this

Role definitions are our own editorial work, checked against occupational classifications, standards, research and employer pages. Titles vary by company, and this is a description of the work, not a job advert.

← Capability evaluation and benchmarkingAll new job families

Start a search

Hiring a Benchmark Research Scientist?

Tell us the work. We find the people already doing it, whatever their job title says, and you meet them already interviewed for your role.