Hiring nowCapability evaluation and benchmarking

Model Evaluation Engineer

Also called Model Evals Engineer · Evals Engineer · Agent QA Engineer

A Model Evaluation Engineer builds the tests that show what an AI system can do and where it breaks. Without them, a team cannot tell whether a change helped, hurt or did nothing.

What they do

They design task sets, write the harness that runs a model against them, and score the results in a way that gives the same answer twice. They check that test items have not leaked into training data. They work out how many samples are needed before a difference means anything. When a new model, prompt or agent change lands, they run regression evaluations and report what moved. They read failures by hand, group them, and turn the patterns into new tasks. They also work with product and research colleagues to decide what is worth measuring.

What a strong hire delivers

Within the first months, a strong hire gives the team a working harness that anyone can run and that produces stable scores. They deliver a first task set tied to real product or capability questions, with contamination controls in place. They set up regression checks that run before releases, and they write up results so that decisions about shipping, rolling back or retraining rest on evidence rather than opinion.

How you can tell they are good

Ask them to walk through an evaluation they built and what went wrong with it. A strong candidate talks about noise, variance and leakage without prompting. They can explain how they knew a score change was real. Ask how they would test an agent that behaves differently on each run. Be wary of anyone who only cites leaderboard numbers or cannot describe how their scores were validated.

Why it is hard to fill

Every AI team now needs this person, and few people have done it at depth. Titles vary: evals engineer, agent QA engineer, research engineer. Good candidates often come from experimental science, statistics or software testing, so CVs rarely look alike. The skill is a blend of coding, statistical judgement and test design, and most people have two of the three. Labs and product companies compete for the same small group.

Where to find them

Look at QA and test engineers who have moved into machine learning work, and at research engineers who spent time on benchmarks. Experimental scientists and applied statisticians with strong Python often make the leap well. Domain specialists in fields such as law, medicine or security can also be strong when paired with solid engineering. They sit at frontier labs, evaluation-focused teams and AI product companies.

Indicative pay

Not enough published evidence yet to give a fair range for this role, which is common for work this new. We benchmark it for you as part of a search.

Interview questions for a Model Evaluation Engineer

  1. Describe an evaluation you built from scratch. What question was it meant to answer, and how did you know it answered it?

    Strong answers state the decision the evaluation supported, how tasks were chosen and how scoring was checked against human judgement. Worry if they describe only tooling, or if they cannot say what the evaluation failed to capture.

  2. A score drops two points after a model update. How do you decide whether it is a real regression?

    They should discuss sample size, run-to-run variance, confidence intervals, paired comparisons and reading individual failures. Worry if they treat any drop as real, or dismiss it as noise without checking.

4 more questions, with what a strong answer covers, are in the Role Kit.

Job description you can post

We are building AI systems and need to know, with evidence, what they can do and where they fail. We are looking for a Model Evaluation Engineer to build evaluations the whole team trusts. You will own the harness, the task sets and the scored results that guide our release decisions.

What you will do

  • Build and maintain a reproducible evaluation harness
  • Design task sets that reflect real capability and product questions
  • Run regression evaluations on every meaningful model or system change
  • Put contamination controls and secured datasets in place
  • Analyse failures by hand and turn patterns into new tasks
  • Report results with clear uncertainty so decisions are well founded

What you will bring

  • Strong Python and habits from software testing or experimental work
  • Working command of statistics, including variance and significance
  • Evaluations you have built, with an account of how you validated them
  • Clear understanding of data quality and provenance
  • Ability to explain results plainly to engineers and non-specialists

Nice to have

  • Experience evaluating agents or multi-step systems
  • Domain expertise in an area we build for
  • Experience with automated scoring and checking it against human judgement

Free to use and adapt. If it saves you time, a link back to this page is appreciated.

Skills that matter

  • Model evaluationAdvancedMeasure capability and failure modes credibly
  • Experimental designAdvancedChoose comparisons that identify credible effects
  • Probability and statisticsWorkingQuantify variability and statistical evidence
  • Data quality and provenanceWorkingTrack dataset origin, validity and transformations

Other roles in capability evaluation and benchmarking

Hiring nowBenchmark Research ScientistA Benchmark Research Scientist designs valid tests for a specific capability and shows the test measures what it claims to. Their work tells a…Hiring nowDangerous Capability EvaluatorA dangerous capability evaluator tests whether frontier AI systems can do things that could cause serious harm, under secured and authorised…Hiring nowDomain AI EvaluatorA Domain AI Evaluator judges model outputs against the standards of a specific profession. They give a company scores it can trust, and they catch…EmergingAgent Performance AnalystAn Agent Performance Analyst monitors and scores the output quality of AI agents running in production, and ties it to business KPIs. They tell the…

Related roles in other families

Hiring nowMachine Learning Research ScientistLearning theory and statistical machine learningA Machine Learning Research Scientist develops and tests learning methods against explicit hypotheses. They give the company evidence about what…Hiring nowStatistical Learning ScientistLearning theory and statistical machine learningA Statistical Learning Scientist builds prediction methods that show their assumptions and their uncertainty. For a company shipping models into…Hiring nowLearning Theory ResearcherLearning theory and statistical machine learningA Learning Theory Researcher works out what learning algorithms can and cannot do, and proves it. For an AI company, that means firmer ground for…Hiring nowAutomated Machine Learning ResearcherLearning theory and statistical machine learningAn Automated Machine Learning Researcher builds search procedures that choose and tune models within a fixed compute budget. It matters because…

How we wrote this

Role definitions are our own editorial work, checked against occupational classifications, standards, research and employer pages. Titles vary by company, and this is a description of the work, not a job advert.

← Capability evaluation and benchmarkingAll new job families

Start a search

Hiring a Model Evaluation Engineer?

Tell us the work. We find the people already doing it, whatever their job title says, and you meet them already interviewed for your role.