EmergingCapability evaluation and benchmarking Part of The agent workforce

Agent Performance Analyst

Also called Agent QA Lead

An Agent Performance Analyst monitors and scores the output quality of AI agents running in production, and ties it to business KPIs. They tell the company whether its agents are actually working, and where they are not.

What they do

They sample live agent output and score it against agreed quality criteria. They build and maintain the scoring methods, from simple checks to structured review, and test whether those methods can be trusted. They track results against business KPIs such as resolution, accuracy or escalation. They look into drops in quality, trace them to a cause, and say whether it sits in the model, the prompt, the data or the process. They report to product, operations and engineering, and flag risks early.

What a strong hire delivers

In the first months, a strong hire gives the company a working picture of agent quality in production. That means defined scoring criteria, a sampling approach that holds up, and a regular report that leaders read and act on. They link quality measures to business KPIs, find the most costly failure patterns, and get fixes into the engineering queue with clear evidence.

How you can tell they are good

Ask them to walk through a scoring method they built and how they checked it was reliable. Strong people talk about sample size, bias, disagreement between reviewers and data provenance without prompting. They separate a real quality shift from noise. They can explain a KPI link and where it breaks. Be wary of anyone who only describes dashboards.

Why it is hard to fill

Almost nobody holds this exact title. The work often sits inside analytics, QA or operations roles, so candidates describe it in very different words. The skills are mixed: statistics, experimental design and a feel for how agents fail. Demand is moderate, but good people are rarely on the market and CVs rarely show the real scope of what they have done.

Where to find them

Look in analytics teams, software testing and QA, and operations groups that have started running agents in live workflows. Good candidates also come from experimental science, applied statistics and domain evaluation, such as claims, support quality or compliance review. Enterprises are the main employers. Many of these people currently carry titles like data analyst, QA lead or quality manager.

Indicative pay

Not enough published evidence yet to give a fair range for this role, which is common for work this new. We benchmark it for you as part of a search.

Interview questions for an Agent Performance Analyst

Tell me about a time you set up quality scoring for a system whose output was hard to judge. How did you decide what good looked like?
A strong answer defines criteria with stakeholders, uses examples, and tests agreement between reviewers. It covers revisions after early results. Worry if they relied on one person's opinion or never checked the scoring itself.
How would you choose a sampling approach for scoring live agent conversations when volume is high and failures are rare?
Look for stratified or targeted sampling, awareness of rare-event problems, and a plan for confidence in the results. They should balance cost and coverage. Worry about purely random sampling with no reasoning about what it misses.

4 more questions, with what a strong answer covers, are in the Role Kit.

Job description you can post

We run AI agents in live business workflows and need someone to tell us how well they perform. You will measure output quality, link it to the outcomes we care about, and help the teams building and running the agents decide what to fix first.

What you will do

Sample and score production agent output against agreed quality criteria
Design and maintain scoring methods, and test their reliability
Track quality against business KPIs and report clearly to leaders
Investigate drops in quality and trace them to a cause
Work with engineering and operations to turn findings into fixes
Check the quality and provenance of the data behind our measures
Flag emerging risks and failure patterns early

What you will bring

Demonstrated work scoring or evaluating complex system output
Solid grounding in statistics and experimental design
Strong SQL and comfort working with messy production data
Experience explaining uncertainty to non-technical colleagues
Evidence of linking quality measures to business outcomes
Clear, concise written reporting
Sound judgement about what a measure can and cannot tell us

Nice to have

Background in software testing or QA
Experience in a domain such as support, claims or compliance review
Familiarity with evaluating language model or agent behaviour

Free to use and adapt. If it saves you time, a link back to this page is appreciated.

Skills that matter

Model evaluationWorkingMeasure capability and failure modes credibly
Experimental designWorkingChoose comparisons that identify credible effects
Probability and statisticsWorkingQuantify variability and statistical evidence
Data quality and provenanceWorkingTrack dataset origin, validity and transformations

Other roles in capability evaluation and benchmarking

Hiring nowModel Evaluation EngineerA Model Evaluation Engineer builds the tests that show what an AI system can do and where it breaks. Without them, a team cannot tell whether a…Hiring nowBenchmark Research ScientistA Benchmark Research Scientist designs valid tests for a specific capability and shows the test measures what it claims to. Their work tells a…Hiring nowDangerous Capability EvaluatorA dangerous capability evaluator tests whether frontier AI systems can do things that could cause serious harm, under secured and authorised…Hiring nowDomain AI EvaluatorA Domain AI Evaluator judges model outputs against the standards of a specific profession. They give a company scores it can trust, and they catch…

Related roles in other families

Hiring nowMachine Learning Research ScientistLearning theory and statistical machine learningA Machine Learning Research Scientist develops and tests learning methods against explicit hypotheses. They give the company evidence about what…Hiring nowStatistical Learning ScientistLearning theory and statistical machine learningA Statistical Learning Scientist builds prediction methods that show their assumptions and their uncertainty. For a company shipping models into…Hiring nowLearning Theory ResearcherLearning theory and statistical machine learningA Learning Theory Researcher works out what learning algorithms can and cannot do, and proves it. For an AI company, that means firmer ground for…Hiring nowAutomated Machine Learning ResearcherLearning theory and statistical machine learningAn Automated Machine Learning Researcher builds search procedures that choose and tune models within a fixed compute budget. It matters because…

How we wrote this

Role definitions are our own editorial work, checked against occupational classifications, standards, research and employer pages. Titles vary by company, and this is a description of the work, not a job advert.

← Capability evaluation and benchmarking All new job families

Start a search

Hiring an Agent Performance Analyst?

Tell us the work. We find the people already doing it, whatever their job title says, and you meet them already interviewed for your role.