EmergingAI control, monitoring and containment

AI Monitor Evaluation Engineer

Also called AI Monitoring Evals Engineer

An AI Monitor Evaluation Engineer tests whether the systems that watch model behaviour actually catch dangerous or unauthorised actions. It matters because a monitor you have not attacked is a monitor you cannot trust.

What they do

The work is trying to break monitors before a real failure does. The engineer builds adversarial evaluation harnesses and runs them inside sandboxes. They design attacks that an untrusted model might plausibly use to slip past a monitor, then score what gets caught and what does not. They examine each miss and work out why it happened. They adapt the attacks as monitors change, and they tell the monitor builders what to fix. This work is emerging and often sits inside security, safety or evaluation teams under other titles.

What a strong hire delivers

In the first months, a strong hire gives the company a clear picture of how well its monitors really work. They set up a repeatable evaluation harness and produce a first monitor evaluation and failure-analysis report. That report shows detection and false-negative rates under adaptive attacks, names the weakest points, and recommends fixes. The monitor team then has a ranked list of problems and a way to check whether fixes held.

How you can tell they are good

Look for someone who treats the monitor as an opponent's target, not a product to demo. Ask them to walk through a past evaluation: how they built the attacks, how they adapted them, and what they did with misses. Good candidates explain their scoring choices and limits. Be wary of anyone who reports a single detection number with no mention of attack strength.

Why it is hard to fill

Few people have done this exact job, because it is new and usually hidden inside other titles. The skills come from different worlds: security engineering, formal methods and AI safety evaluation. Most candidates are strong in one and thin in the others. Titles rarely match the work, so searching by job title misses people. We look at what someone has actually attacked and measured.

Where to find them

Look in security engineering teams, especially those doing red teaming or detection work. Formal methods and verification groups are another source. So are AI safety evaluation teams at frontier labs, safety-focused start-ups and independent research organisations. Some come from AI control research. Titles vary widely, including security engineer, evaluation engineer, research engineer and red team engineer.

Indicative pay

Not enough published evidence yet to give a fair range for this role, which is common for work this new. We benchmark it for you as part of a search.

Interview questions for an AI Monitor Evaluation Engineer

  1. Tell us about a monitor or detection system you tried to break. How did you design the attacks?

    A strong answer names a real system, the threat model, and how attacks were chosen and varied. It covers what was missed and why. Worry if they only describe running a standard test set with no adaptation.

  2. How do you decide whether a detection rate is good enough to trust?

    Good answers tie the number to attack strength, cost of misses and the threat model. They discuss confidence, sample size and false positives. Worry about a single headline figure or a fixed threshold claimed as universally right.

4 more questions, with what a strong answer covers, are in the Role Kit.

Job description you can post

We are hiring an AI Monitor Evaluation Engineer to test whether our monitors catch dangerous or unauthorised model behaviour. You will attack our monitoring systems, measure how they fail, and help the team fix them. The work is new, hands-on and matters to how safely we deploy models.

What you will do

  • Build and maintain adversarial evaluation harnesses for monitors
  • Design attacks an untrusted model might use to evade detection
  • Run evaluations in sandboxed environments
  • Measure detection and false-negative rates under adaptive attacks
  • Analyse failures and write clear failure-analysis reports
  • Work with monitor builders to verify fixes
  • Keep attack sets current as monitors and models change

What you will bring

  • Demonstrated work attacking or evaluating detection systems
  • Strong security engineering skills
  • Experience building evaluation or test harnesses in code
  • Clear thinking about scoring, uncertainty and threat models
  • Ability to write reports that engineers and leaders can act on
  • Familiarity with sandboxing and containment practices

Nice to have

  • Experience with formal verification or formal methods
  • Background in AI control or AI safety evaluation
  • Published or shared work on red teaming

Free to use and adapt. If it saves you time, a link back to this page is appreciated.

Skills that matter

  • AI controlAdvancedLimit harm from potentially untrusted AI behaviour
  • Security engineeringAdvancedProtect systems, identities and sensitive assets
  • Model evaluationWorkingMeasure capability and failure modes credibly
  • Formal verificationWorkingProve stated properties under explicit assumptions

Other roles in aI control, monitoring and containment

EmergingAI Control ResearcherAn AI Control Researcher designs the safeguards that limit what a model can do if it turns out to be unreliable or hostile. They keep risk bounded…EmergingAgent Containment EngineerAn Agent Containment Engineer builds the boundaries that keep AI agents from doing harm, even if the model misbehaves or is attacked. They matter…EmergingAI Shutdown and Recovery EngineerAn AI Shutdown and Recovery Engineer builds and tests the ways to stop an AI system mid-action and bring it back to a known good state. It matters…EmergingAI Control EngineerAn AI Control Engineer builds the monitoring, sandboxing and containment that let a company use powerful models safely, even if a model is…

Related roles in other families

EmergingAI Formal Verification EngineerFormal methods and reliable reasoningAn AI Formal Verification Engineer writes precise specifications for AI systems and their outputs, then proves they hold or finds counterexamples…Hiring nowAutomated Theorem Proving ResearcherFormal methods and reliable reasoningAn Automated Theorem Proving Researcher builds and improves methods that find or check mathematical and logical proofs by machine. The company gets…Hiring nowProgram Synthesis ResearcherFormal methods and reliable reasoningA Program Synthesis Researcher builds systems that generate programmes which meet a testable or formal specification. They give the company code…EmergingAI-Assisted Verification SpecialistFormal methods and reliable reasoningAn AI-Assisted Verification Specialist uses AI tools to speed up proof and specification work, then checks every result independently. They help a…

How we wrote this

Role definitions are our own editorial work, checked against occupational classifications, standards, research and employer pages. Titles vary by company, and this is a description of the work, not a job advert.

← AI control, monitoring and containmentAll new job families

Start a search

Hiring an AI Monitor Evaluation Engineer?

Tell us the work. We find the people already doing it, whatever their job title says, and you meet them already interviewed for your role.