EmergingMulti-agent coordination and economicsPart of The agent workforce

Multi-Agent Evaluation Scientist

Also called Agent Interaction Evaluator

A Multi-Agent Evaluation Scientist measures how groups of AI agents behave together and designs tests that expose collusion, cascades and coordination failures before customers or production systems meet them.

What they do

They build evaluation suites for systems where several agents act, negotiate or hand work to each other. They write adversarial interaction scenarios, run them in multi-agent simulators and read the traces to see where behaviour drifts. They separate failures caused by one model from failures that only appear when agents interact. They define what counts as collusion, cascade or deadlock in measurable terms. They work with engineers who build orchestration and with researchers who train the agents, then turn findings into fixes, tests that run on every release and clear reports for people who decide what ships.

What a strong hire delivers

In the first months, a strong hire gives the team a working evaluation suite with a small set of adversarial scenarios that people trust. They agree definitions for the failure types that matter most. They produce a first map of where interaction risk sits in the current system. They set up trace analysis so failures can be explained, not just counted. Release decisions start to use their results.

How you can tell they are good

Ask them to walk through an evaluation they designed for interacting agents. Listen for how they chose scenarios, how they handled randomness, and how they told an emergent failure from a single-agent bug. Ask what they would measure to detect quiet collusion. Strong people show real traces, admit what their tests missed and explain trade-offs plainly.

Why it is hard to fill

The title barely exists yet. The work is usually hidden inside roles such as evaluation engineer, research scientist or distributed systems engineer. Few people have both evaluation craft and a feel for game theory and system behaviour. Candidates often have one half and need to be assessed for how quickly they can learn the other. Job titles tell you very little, so we read the work history instead.

Where to find them

Look at people in agent evaluation teams, reinforcement learning researchers who have worked on multi-agent settings, and distributed systems engineers who moved into AI. Game theorists and mechanism design researchers are another source. They tend to sit in frontier labs, safety and evaluation organisations, agent platform start-ups and research groups, often under a different title.

Indicative pay

Not enough published evidence yet to give a fair range for this role, which is common for work this new. We benchmark it for you as part of a search.

Interview questions for a Multi-Agent Evaluation Scientist

  1. Describe an evaluation you built where several agents interacted. What did it catch that single-agent tests missed?

    A strong answer gives a concrete system, the scenarios used and a specific interaction failure found. It explains why single-agent tests could not see it. Worry if answers stay generic, cite only benchmark scores or describe single-model testing.

  2. How would you detect collusion between agents that never communicate directly?

    Look for ideas such as outcome correlation, comparison against independent baselines, counterfactual runs and trace analysis. A good candidate discusses false positives and what evidence would be convincing. Worry if they rely only on reading messages.

4 more questions, with what a strong answer covers, are in the Role Kit.

Job description you can post

We build systems in which several AI agents work together, and we need to know how they behave as a group before our customers do. We are looking for an evaluation scientist to design and run tests for interaction-level risk, and to help the team fix what those tests uncover.

What you will do

  • Design evaluation suites for systems of interacting agents
  • Write adversarial interaction scenarios that probe collusion, cascades and coordination failures
  • Run experiments in multi-agent simulators and analyse the traces
  • Define measurable criteria for emergent failure modes
  • Work with engineers and researchers to reproduce and fix issues
  • Build tests that run on every release
  • Report findings clearly to technical and non-technical decision makers

What you will bring

  • Evidence of designing evaluations for multi-agent or distributed systems
  • Working knowledge of game theory or decision theory applied in practice
  • Experience analysing agent traces to explain failures
  • Solid engineering skills, including distributed systems fundamentals
  • Experience with agent orchestration or coordination protocols
  • Clear written communication of technical findings

Nice to have

  • Background in reinforcement learning or mechanism design
  • Experience with safety or red-team style evaluation
  • Published or shared work on agent interaction

Free to use and adapt. If it saves you time, a link back to this page is appreciated.

Skills that matter

  • Agent orchestrationAdvancedCoordinate tool-using agent execution
  • Decision theory and game theoryAdvancedAnalyse choices, incentives and strategic interactions
  • Distributed systemsWorkingDesign reliable work across networked machines
  • Model evaluationWorkingMeasure capability and failure modes credibly

Other roles in multi-agent coordination and economics

EmergingMulti-Agent Systems ResearcherA Multi-Agent Systems Researcher studies how several AI agents behave when they work together, and builds methods and benchmarks to make that…EmergingAgent Coordination EngineerAn Agent Coordination Engineer builds the layer that lets several AI agents hand work to each other, share state and settle conflicts. It matters…EmergingAgent Market Mechanism ResearcherThis person designs and tests the rules that govern how AI agents trade, bid and share resources in bounded simulated markets. It matters because…EmergingMulti-Agent Systems ArchitectA Multi-Agent Systems Architect designs how many AI agents work together: who does what, how work passes between them, and how failures stay…

Related roles in other families

EmergingAI Formal Verification EngineerFormal methods and reliable reasoningAn AI Formal Verification Engineer writes precise specifications for AI systems and their outputs, then proves they hold or finds counterexamples…Hiring nowAutomated Theorem Proving ResearcherFormal methods and reliable reasoningAn Automated Theorem Proving Researcher builds and improves methods that find or check mathematical and logical proofs by machine. The company gets…Hiring nowProgram Synthesis ResearcherFormal methods and reliable reasoningA Program Synthesis Researcher builds systems that generate programmes which meet a testable or formal specification. They give the company code…EmergingAI-Assisted Verification SpecialistFormal methods and reliable reasoningAn AI-Assisted Verification Specialist uses AI tools to speed up proof and specification work, then checks every result independently. They help a…

How we wrote this

Role definitions are our own editorial work, checked against occupational classifications, standards, research and employer pages. Titles vary by company, and this is a description of the work, not a job advert.

← Multi-agent coordination and economicsAll new job families

Start a search

Hiring a Multi-Agent Evaluation Scientist?

Tell us the work. We find the people already doing it, whatever their job title says, and you meet them already interviewed for your role.