Hiring nowCapability evaluation and benchmarking

Domain AI Evaluator

Also called Expert AI Evaluator

A Domain AI Evaluator judges model outputs against the standards of a specific profession. They give a company scores it can trust, and they catch the serious errors that generic benchmarks miss.

What they do

The evaluator reads model outputs in their own field and scores them against professional standards. They apply a clear rubric, record why each score was given, and settle disagreements between scorers with written rationale. They work inside evaluation harnesses, often on secured datasets, and keep scoring reproducible so results can be rerun and compared. They flag high-severity errors, such as advice that would cause real harm if followed. They also point out weak test items, unclear criteria and gaps in data provenance, and feed that back to the people who build the evaluations.

What a strong hire delivers

In the first months, a strong hire gives the team scored outputs it can defend. Scoring rubrics become clearer and agreement between scorers improves. High-severity failures are found early and logged with evidence. Weak test items are fixed or removed. Engineers and researchers start to rely on the scores when deciding whether a model is ready to ship.

How you can tell they are good

Ask them to take a model output from their field and score it aloud. A good evaluator separates a minor slip from a serious error and explains why. They can describe how they would check their own consistency, and what they do when two experts disagree. Be wary of anyone who relies on gut feel or cannot write down their reasoning.

Why it is hard to fill

The title is new and rarely used, so searches by job title miss most of the good people. The hire needs two things that seldom sit together: real depth in a professional field and a scientist's discipline about measurement. Domain experts often lack statistical habits. Evaluation specialists often lack domain depth. Candidates with both are rare and usually settled elsewhere.

Where to find them

Look in experimental science, statistics, software testing and applied research. Many come from clinical, legal, financial or engineering practice and later moved into evaluation or quality work. Typical titles include research scientist, quality analyst, test engineer, data scientist and subject matter specialist. They sit in model providers, applied AI companies, research groups and regulated industries.

Indicative pay

Not enough published evidence yet to give a fair range for this role, which is common for work this new. We benchmark it for you as part of a search.

Interview questions for a Domain AI Evaluator

  1. Tell us about a time you scored work against a professional standard. How did you make your judgement consistent?

    Strong answers describe a written rubric, examples that anchor each score, and checks against other scorers. They show how they caught their own drift. Worry if consistency rests on experience or instinct alone.

  2. How do you decide whether an error in a model output is high severity?

    Look for reasoning based on consequence in real professional use, not on how wrong the output looks. Good answers give examples and a clear scale. Worry about treating all errors as equal.

4 more questions, with what a strong answer covers, are in the Role Kit.

Job description you can post

We build AI systems and need someone with deep professional knowledge to judge how well they perform in a specific domain. You will score model outputs against real professional standards, explain your reasoning, and help us trust our evaluation results before we make decisions about releases.

What you will do

  • Score model outputs against professional standards using clear rubrics
  • Write adjudication rationale for each judgement and for disputed cases
  • Identify and log high-severity errors with supporting evidence
  • Run scoring in evaluation harnesses so results are reproducible
  • Work with secured datasets and follow data handling rules
  • Flag weak test items, unclear criteria and provenance gaps
  • Share findings with researchers and engineers

What you will bring

  • Demonstrated professional or research work in a specific domain
  • Examples of consistent, evidence-based scoring or review
  • Working knowledge of probability and statistics
  • Experience with experimental design or software testing
  • Clear written explanations of difficult judgements
  • Care with data quality and provenance

Nice to have

  • Experience with model evaluation or benchmark design
  • Familiarity with evaluation harnesses
  • Experience working with sensitive or secured data

Free to use and adapt. If it saves you time, a link back to this page is appreciated.

Skills that matter

  • Model evaluationAdvancedMeasure capability and failure modes credibly
  • Experimental designAdvancedChoose comparisons that identify credible effects
  • Probability and statisticsWorkingQuantify variability and statistical evidence
  • Data quality and provenanceWorkingTrack dataset origin, validity and transformations

Other roles in capability evaluation and benchmarking

Hiring nowModel Evaluation EngineerA Model Evaluation Engineer builds the tests that show what an AI system can do and where it breaks. Without them, a team cannot tell whether a…Hiring nowBenchmark Research ScientistA Benchmark Research Scientist designs valid tests for a specific capability and shows the test measures what it claims to. Their work tells a…Hiring nowDangerous Capability EvaluatorA dangerous capability evaluator tests whether frontier AI systems can do things that could cause serious harm, under secured and authorised…EmergingAgent Performance AnalystAn Agent Performance Analyst monitors and scores the output quality of AI agents running in production, and ties it to business KPIs. They tell the…

Related roles in other families

Hiring nowMachine Learning Research ScientistLearning theory and statistical machine learningA Machine Learning Research Scientist develops and tests learning methods against explicit hypotheses. They give the company evidence about what…Hiring nowStatistical Learning ScientistLearning theory and statistical machine learningA Statistical Learning Scientist builds prediction methods that show their assumptions and their uncertainty. For a company shipping models into…Hiring nowLearning Theory ResearcherLearning theory and statistical machine learningA Learning Theory Researcher works out what learning algorithms can and cannot do, and proves it. For an AI company, that means firmer ground for…Hiring nowAutomated Machine Learning ResearcherLearning theory and statistical machine learningAn Automated Machine Learning Researcher builds search procedures that choose and tune models within a fixed compute budget. It matters because…

How we wrote this

Role definitions are our own editorial work, checked against occupational classifications, standards, research and employer pages. Titles vary by company, and this is a description of the work, not a job advert.

← Capability evaluation and benchmarkingAll new job families

Start a search

Hiring a Domain AI Evaluator?

Tell us the work. We find the people already doing it, whatever their job title says, and you meet them already interviewed for your role.