EmergingCyber defence and security operations AIPart of The agent workforce

Defensive AI Agent Evaluator

Also called Cyber Defence Agent Evaluator

A Defensive AI Agent Evaluator tests how AI agents behave inside security detection and response work, and shows where they fail. It matters because an agent that oversteps or misses a threat can harm the assets it was meant to protect.

What they do

The work is testing agents that sit inside authorised defensive workflows. The evaluator builds secured scenarios from security telemetry, SIEM alerts and case records, then pushes the agent to see how it behaves. They try misleading inputs, poisoned data and instructions that tempt it past its permissions. They watch which controlled tools it calls and why. They score how useful it is on real triage and response tasks. Each failure is written up with evidence, a way to reproduce it and a suggested control. Findings go to engineers and security leads.

What a strong hire delivers

In the first months, a strong hire gives the team a repeatable way to judge a security agent before and after changes. Expect a set of authorised scenarios that reflect real defensive work, a first boundary-failure report with clear severity, and fixes agreed with engineering. The team should also gain shared language for what the agent may do alone and what needs a human.

How you can tell they are good

Ask them to walk through an evaluation they ran or designed. A strong candidate explains the scenario, the assets at stake, how they kept testing authorised and contained, and how they scored both usefulness and safety. They separate a real boundary failure from noise. They can show a report that an engineer could act on. Be wary of generic jailbreak talk with no defensive context.

Why it is hard to fill

Almost nobody holds this title yet. The work usually sits inside security operations, detection engineering or AI red teaming roles, so CVs rarely say it. Candidates tend to be strong on one side: security people who have not tested agents, or AI testers who have never worked a live incident. The overlap is small, and good people are rarely looking.

Where to find them

Start with security operations analysts and leads, detection engineers and incident responders who have begun working with agents or automation. Also look at AI red teamers and reliability engineers who have worked on security products. They sit in security vendors, managed detection teams, larger in-house security functions and AI companies building tools for defenders.

Indicative pay

Not enough published evidence yet to give a fair range for this role, which is common for work this new. We benchmark it for you as part of a search.

Interview questions for a Defensive AI Agent Evaluator

  1. Describe an agent or automation you tested in a defensive workflow. What did you try to break, and what did you find?

    Look for a specific system, a clear test plan and a real finding with evidence. Strong answers cover impact on assets and what changed afterwards. Worry about vague claims, or testing that never touched realistic security data.

  2. How do you keep an adversarial evaluation of a security agent authorised and contained?

    Strong answers cover written scope, approval, isolated environments, controlled tools, and limits on data and actions. They mention stakeholders and stop conditions. Worry about anyone casual about permissions or testing near production.

4 more questions, with what a strong answer covers, are in the Role Kit.

Job description you can post

We are building AI agents that support security detection and response, and we need someone to test them hard before our customers rely on them. You will challenge agent behaviour in secured, authorised scenarios and tell us clearly where it fails and what to do about it.

What you will do

  • Design authorised defensive scenarios using realistic security telemetry, SIEM data and case records
  • Test agents with adversarial inputs, misleading content and permission-pushing instructions
  • Review how agents use controlled tools and where they cross set boundaries
  • Measure task utility alongside protection of assets under adversarial conditions
  • Write boundary-failure reports with evidence, severity and recommended controls
  • Work with engineers and security leads to fix issues and retest
  • Keep scenario sets current as agents and threats change

What you will bring

  • Demonstrated work in security operations, detection engineering or incident response
  • Experience testing or red teaming AI systems, agents or automation
  • Working knowledge of SIEM, security telemetry and case management systems
  • Clear examples of keeping security testing authorised and contained
  • Ability to write reports that engineers can act on
  • Sound judgement on trade-offs between speed, usefulness and risk

Nice to have

  • Experience with agent orchestration or tool-calling systems
  • Background in reliability engineering or incident review
  • Experience building evaluation harnesses or scoring methods

Free to use and adapt. If it saves you time, a link back to this page is appreciated.

Skills that matter

  • Security engineeringAdvancedProtect systems, identities and sensitive assets
  • AI red teamingAdvancedTest systems adversarially under authorisation
  • Reliability and incident responseWorkingDetect failures, restore service and prevent recurrence
  • Agent orchestrationWorkingCoordinate tool-using agent execution

Other roles in cyber defence and security operations AI

Hiring nowCybersecurity ML ScientistA Cybersecurity ML Scientist builds and tests models that spot malicious activity in authorised security data. They keep detection useful as…Hiring nowAI Detection EngineerAn AI Detection Engineer builds the detections that turn AI-derived signals into alerts a security team can trust. They cut noise, catch real…EmergingAI Security Operations EngineerAn AI Security Operations Engineer builds constrained AI support for security triage and response. It lets a security team move faster on alerts and…

Related roles in other families

Hiring nowAI Security EngineerAI application security and misuse defenceAn AI Security Engineer hardens the applications, interfaces and sensitive assets built around models. They find how attackers could misuse the…Hiring nowAI Red Team EngineerAI application security and misuse defenceAn AI Red Team Engineer runs authorised attacks against model-connected systems to find weaknesses before real adversaries do. They give the company…Hiring nowAI Abuse Detection AnalystAI application security and misuse defenceAn AI Abuse Detection Analyst spots unauthorised and harmful use of a company's models and products, then turns what they find into rules and cases…EmergingModel Supply Chain Security EngineerAI application security and misuse defenceA Model Supply Chain Security Engineer protects model files, datasets and dependencies from tampering. They give the company confidence that what it…

How we wrote this

Role definitions are our own editorial work, checked against occupational classifications, standards, research and employer pages. Titles vary by company, and this is a description of the work, not a job advert.

← Cyber defence and security operations AIAll new job families

Start a search

Hiring a Defensive AI Agent Evaluator?

Tell us the work. We find the people already doing it, whatever their job title says, and you meet them already interviewed for your role.