Hiring nowAI application security and misuse defence

AI Abuse Detection Analyst

Also called AI Trust and Safety Analyst

An AI Abuse Detection Analyst spots unauthorised and harmful use of a company's models and products, then turns what they find into rules and cases. It protects users, limits cost and keeps the product trusted.

What they do

The analyst reads logs, usage patterns and account signals to find people misusing a model-connected product. That might be automated scraping, jailbreak sharing, fraud, or attempts to push an agent into actions it should not take. They investigate suspicious accounts, trace links between them, and write detection rules. They test those rules against real traffic and tune them to cut false alarms. They open cases, record the evidence and escalate serious ones to security, legal or policy colleagues. They also run sandboxed tests to reproduce abuse and check that a fix works.

What a strong hire delivers

In the first months, a strong hire gives the company a clear view of how its product is being misused. They deliver a working set of detection rules with known precision, a case process people actually follow, and escalation records that show what happened and why. Response to new abuse gets faster. Gaps in logging and identity data are named and fixed, and coverage improves in the areas that matter most.

How you can tell they are good

Ask for a real investigation, told from first signal to outcome. A strong candidate explains what they saw, how they separated abuse from odd but legitimate use, and what rule they wrote. They can say how they measured false positives and what they missed. Ask how they would handle a new abuse type with no history. Be wary of answers that are all tools and no reasoning.

Why it is hard to fill

The title is new and rarely means the same thing twice. Most good candidates sit under security, fraud or trust and safety titles, and few have worked on model-specific abuse. The work needs investigative instinct, data skill and enough security grounding to understand agents and tool access. Candidates strong in one of those are common. Those strong in all three are scarce, and employers often wrongly screen on the title alone.

Where to find them

Look in application security, threat detection, fraud and abuse teams, and security research. Trust and safety analysts from large consumer platforms bring useful casework habits. Detection engineers from security operations often adapt well. Many come from companies that run public APIs, marketplaces or developer platforms, where misuse of access is already a daily problem. Red teamers with an interest in defence are also worth a conversation.

Indicative pay

Not enough published evidence yet to give a fair range for this role, which is common for work this new. We benchmark it for you as part of a search.

Interview questions for an AI Abuse Detection Analyst

  1. Walk me through an abuse or fraud investigation you led. What was the first signal and how did it end?

    A strong answer follows a clear chain from signal to evidence to action, and names what was uncertain. It covers how they confirmed intent and what changed afterwards. Worry about vague stories, or credit taken for team work.

  2. How do you decide whether a detection rule is good enough to ship?

    Look for talk of precision, coverage and testing against past traffic. Good candidates mention review of false positives, staged rollout and a plan for tuning. Worry if they only count catches and ignore the cost of wrongly blocking users.

4 more questions, with what a strong answer covers, are in the Role Kit.

Job description you can post

We build products that connect models to users, data and tools. As we grow, so does the number of people who try to misuse them. We are hiring an AI Abuse Detection Analyst to find that misuse early, investigate it carefully and help us respond quickly and fairly.

What you will do

  • Investigate suspicious accounts and usage patterns across our model-connected products
  • Write, test and tune detection rules for unauthorised or harmful use
  • Open and manage cases with clear evidence and decisions
  • Escalate serious incidents to security, legal and policy colleagues
  • Reproduce abuse techniques in sandboxes and check that fixes hold
  • Report on detection precision, coverage and response time
  • Identify gaps in logging and identity data and work with engineers to close them

What you will bring

  • Demonstrated investigations in abuse, fraud, threat detection or security research
  • Strong signal analysis skills using logs and query tools
  • Experience writing detection logic and measuring its false positives
  • Clear written case records and escalation notes
  • Working knowledge of application security and access control
  • Sound judgement when evidence is incomplete

Nice to have

  • Experience with AI red teaming or adversarial testing of models
  • Familiarity with agent tool and protocol integrations
  • Incident response experience

Free to use and adapt. If it saves you time, a link back to this page is appreciated.

Skills that matter

  • Security engineeringAdvancedProtect systems, identities and sensitive assets
  • AI red teamingAdvancedTest systems adversarially under authorisation
  • Tool and protocol integrationWorkingConnect models to external data and capabilities
  • Reliability and incident responseWorkingDetect failures, restore service and prevent recurrence

Other roles in aI application security and misuse defence

Hiring nowAI Security EngineerAn AI Security Engineer hardens the applications, interfaces and sensitive assets built around models. They find how attackers could misuse the…Hiring nowAI Red Team EngineerAn AI Red Team Engineer runs authorised attacks against model-connected systems to find weaknesses before real adversaries do. They give the company…EmergingModel Supply Chain Security EngineerA Model Supply Chain Security Engineer protects model files, datasets and dependencies from tampering. They give the company confidence that what it…Hiring nowAgent Security EngineerAn Agent Security Engineer protects AI agents from prompt injection, data leaks and tool misuse. They let a company ship agents that act on real…Hiring nowModel Weights Security EngineerA Model Weights Security Engineer protects trained model weights and the training infrastructure around them from theft, including by well-resourced…Hiring nowAI Threat Intelligence AnalystAn AI Threat Intelligence Analyst tracks how threat actors use AI to attack, deceive and abuse systems, then turns what they find into detections…EmergingAgent Identity and Permissions ArchitectThis person designs how AI agents prove who they are and what they may do. They keep access narrow as agent numbers grow, so one compromised agent…Hiring nowAI Trust and Safety SpecialistAn AI Trust and Safety Specialist writes and enforces the rules for how a model-connected product can be used, and investigates misuse. They keep…

Related roles in other families

Hiring nowCybersecurity ML ScientistCyber defence and security operations AIA Cybersecurity ML Scientist builds and tests models that spot malicious activity in authorised security data. They keep detection useful as…Hiring nowAI Detection EngineerCyber defence and security operations AIAn AI Detection Engineer builds the detections that turn AI-derived signals into alerts a security team can trust. They cut noise, catch real…EmergingAI Security Operations EngineerCyber defence and security operations AIAn AI Security Operations Engineer builds constrained AI support for security triage and response. It lets a security team move faster on alerts and…EmergingDefensive AI Agent EvaluatorCyber defence and security operations AIA Defensive AI Agent Evaluator tests how AI agents behave inside security detection and response work, and shows where they fail. It matters because…

How we wrote this

Role definitions are our own editorial work, checked against occupational classifications, standards, research and employer pages. Titles vary by company, and this is a description of the work, not a job advert.

← AI application security and misuse defenceAll new job families

Start a search

Hiring an AI Abuse Detection Analyst?

Tell us the work. We find the people already doing it, whatever their job title says, and you meet them already interviewed for your role.