Hiring nowCapability evaluation and benchmarking

Dangerous Capability Evaluator

Also called Frontier Capabilities Evaluator · Frontier Risk Evaluator (CBRN/Cyber/Autonomy)

A dangerous capability evaluator tests whether frontier AI systems can do things that could cause serious harm, under secured and authorised protocols. Their evidence helps a company decide what it can safely build, release and defend.

What they do

The work is running controlled tests on models in sensitive areas such as biological and chemical risk, cyber operations and autonomous behaviour. The evaluator designs tasks, builds or adapts evaluation harnesses, and works with domain experts to judge what counts as meaningful uplift. They handle restricted datasets under strict access rules. They run repeated trials, score results in a reproducible way, and check whether a model is truly failing or just being poorly prompted. Findings go into written assessments that safety and leadership teams use to make decisions.

What a strong hire delivers

In the first months, a strong hire delivers a working, secured evaluation process that others can rerun and trust. They produce clear assessments against agreed thresholds, with honest statements of uncertainty. They tighten how test data is handled and recorded. They also leave the team with sharper judgement about which results should trigger escalation and which are noise.

How you can tell they are good

Ask them to walk through an evaluation they designed, and what would have made them distrust the result. Strong people talk about elicitation effort, sample size, contamination and false reassurance without prompting. They explain limits plainly, handle sensitive material with discipline, and can say what a score does not show. Be wary of confident claims with no stated uncertainty.

Why it is hard to fill

The title varies by employer, so searching by it misses most of the pool. The work needs domain depth, evaluation craft and trusted handling of sensitive material, and few people have all three. Many candidates need vetting or clearance, which slows everything. Most good people are already inside labs or government institutes, and they are careful about moving. Expect a small, closed market.

Where to find them

Look at people from model evaluation, red teaming and safety research teams at frontier labs and AI security institutes. Others come from experimental science, applied statistics, offensive security, biosecurity and software testing. Government and defence-adjacent research groups are also worth a look. Domain experts who have since built evaluation skills often do well, as do evaluators who have learned a risk domain in depth.

Indicative pay

Not enough published evidence yet to give a fair range for this role, which is common for work this new. We benchmark it for you as part of a search.

Interview questions for a Dangerous Capability Evaluator

  1. Describe an evaluation you built to detect a risky behaviour. How did you decide it was measuring what you intended?

    A strong answer covers task design, validation against known cases, and checks for leakage or shortcuts. They name what they got wrong. Worry if they treat a single score as proof or cannot explain how validity was tested.

  2. How do you tell whether a model cannot do a task or simply was not elicited well?

    Look for prompting variation, scaffolding, repeated runs, fine-tuning or tool access, and a stated effort budget. They should be cautious about negative results. Worry if a failed attempt is read as evidence of safety.

4 more questions, with what a strong answer covers, are in the Role Kit.

Job description you can post

We are hiring a dangerous capability evaluator to measure whether our frontier models can do things that could cause serious harm. You will work under secured, authorised protocols and produce evidence that informs our decisions on training, release and safeguards.

What you will do

  • Design and run controlled evaluations of high-risk model capabilities
  • Build and maintain evaluation harnesses with reproducible scoring
  • Handle restricted datasets under strict access and recording rules
  • Work with domain experts to define thresholds and judge outputs
  • Test whether results reflect real limits or weak elicitation
  • Write clear risk assessments with stated uncertainty
  • Escalate threshold-relevant findings through agreed processes

What you will bring

  • Demonstrated experience designing and validating model or experimental evaluations
  • Strong grounding in statistics and experimental design
  • Track record of careful work with sensitive or restricted material
  • Ability to write evaluation code that others can rerun
  • Clear written reporting of findings and limits
  • Willingness to undergo vetting or clearance where required

Nice to have

  • Depth in a risk domain such as cyber, biosecurity or autonomy
  • Experience with red teaming or safety evaluation
  • Experience working with government or security institutes

Free to use and adapt. If it saves you time, a link back to this page is appreciated.

Skills that matter

  • Model evaluationAdvancedMeasure capability and failure modes credibly
  • Experimental designAdvancedChoose comparisons that identify credible effects
  • Probability and statisticsWorkingQuantify variability and statistical evidence
  • Data quality and provenanceWorkingTrack dataset origin, validity and transformations

Other roles in capability evaluation and benchmarking

Hiring nowModel Evaluation EngineerA Model Evaluation Engineer builds the tests that show what an AI system can do and where it breaks. Without them, a team cannot tell whether a…Hiring nowBenchmark Research ScientistA Benchmark Research Scientist designs valid tests for a specific capability and shows the test measures what it claims to. Their work tells a…Hiring nowDomain AI EvaluatorA Domain AI Evaluator judges model outputs against the standards of a specific profession. They give a company scores it can trust, and they catch…EmergingAgent Performance AnalystAn Agent Performance Analyst monitors and scores the output quality of AI agents running in production, and ties it to business KPIs. They tell the…

Related roles in other families

Hiring nowMachine Learning Research ScientistLearning theory and statistical machine learningA Machine Learning Research Scientist develops and tests learning methods against explicit hypotheses. They give the company evidence about what…Hiring nowStatistical Learning ScientistLearning theory and statistical machine learningA Statistical Learning Scientist builds prediction methods that show their assumptions and their uncertainty. For a company shipping models into…Hiring nowLearning Theory ResearcherLearning theory and statistical machine learningA Learning Theory Researcher works out what learning algorithms can and cannot do, and proves it. For an AI company, that means firmer ground for…Hiring nowAutomated Machine Learning ResearcherLearning theory and statistical machine learningAn Automated Machine Learning Researcher builds search procedures that choose and tune models within a fixed compute budget. It matters because…

How we wrote this

Role definitions are our own editorial work, checked against occupational classifications, standards, research and employer pages. Titles vary by company, and this is a description of the work, not a job advert.

← Capability evaluation and benchmarkingAll new job families

Start a search

Hiring a Dangerous Capability Evaluator?

Tell us the work. We find the people already doing it, whatever their job title says, and you meet them already interviewed for your role.