Safety and security

Capability evaluation and benchmarking

Capability evaluation and benchmarking is the work of measuring what AI systems can actually do and where they fail. AI companies need it to decide what to ship, what to fix and what to hold back.

Models now ship faster than the tests that judge them. Founders face questions about quality, risk and fitness for a customer's work, and a leaderboard score rarely answers them. The hires are tricky because the skill sits between disciplines. A strong candidate has the rigour of an experimental scientist, the habits of a software tester and enough domain sense to know when a test measures the wrong thing. Many people claim it. Few can show a valid test.

Hiring now 4Emerging 1

Roles in this family

5 roles

Hiring nowModel Evaluation EngineerA Model Evaluation Engineer builds the tests that show what an AI system can do and where it breaks. Without them, a team cannot tell whether a…Hiring nowBenchmark Research ScientistA Benchmark Research Scientist designs valid tests for a specific capability and shows the test measures what it claims to. Their work tells a…Hiring nowDangerous Capability EvaluatorA dangerous capability evaluator tests whether frontier AI systems can do things that could cause serious harm, under secured and authorised…Hiring nowDomain AI EvaluatorA Domain AI Evaluator judges model outputs against the standards of a specific profession. They give a company scores it can trust, and they catch…EmergingAgent Performance AnalystAn Agent Performance Analyst monitors and scores the output quality of AI agents running in production, and ties it to business KPIs. They tell the…

Where the people come from

Experimental science, statistics, software testing or domain evaluation

Not to be confused with. Benchmark performance is not a universal measure of intelligence or safety.

Core skills

  • Probability and statistics
  • Experimental design
  • Data quality and provenance
  • Model evaluation

Where it is used

  • Cross-sector AI platforms
  • Frontier AI governance

More in Safety and security

Alignment and scalable oversight0 rolesInterpretability and model internals0 rolesAI control, monitoring and containment0 rolesAI application security and misuse defence0 rolesAI robustness, safety engineering and incidents0 roles

← All new job families

Start a search

Hiring in capability evaluation and benchmarking?

Tell us the work you need done. We find the people already doing it, whatever their job title says.