ScenarioGeneral-purpose autonomy assurance

Cross-Domain Capability Certification Designer

Also called AGI Capability Certification Designer

A Cross-Domain Capability Certification Designer would define what a general-purpose AI system can credibly be said to do across consequential domains. The role depends on how AI develops, and no company is hiring for it now.

What they do

The designer would decide what capability claims are worth making and what evidence would support them. They would map the domains a system might be trusted in, then draft a certification scope for each. They would design validation protocols that test whether results in one domain say anything about another. They would work with evaluators, domain experts, risk teams and legal counsel to find where a claim could be stretched too far. They would write down limits plainly and set conditions for when a certificate must be reviewed or withdrawn. They would also adapt the protocol when the system or its use changes.

What a strong hire delivers

In the first months, a strong hire would produce a clear proposed certification scope and a validation protocol that others can run and challenge. They would agree with leadership which claims the company will and will not make. They would surface weak points in existing evaluations and set out how to fix them. They would leave behind a review process that holds up when someone senior wants a stronger claim.

How you can tell they are good

Ask them to take one capability claim and show how it could be wrong. Strong people talk about construct validity, what a test actually measures, and how results fail to carry across domains. They name the evidence they would refuse to accept. Be wary of anyone who relies on a single benchmark score, treats certification as paperwork, or cannot say what would make them withdraw a claim.

Why it is hard to fill

If this role appears, the people would be scarce because it joins measurement science, safety assurance and institutional design, and few have worked across all three. Titles would vary widely and none would signal the skill. Most candidates would be strong in one area and thin in the others. Many would also be tempted to overclaim, so judgement matters as much as method.

Where to find them

Today they sit in AI control, model evaluation, assurance and systems safety teams. Some come from formal methods, safety cases in regulated engineering, or standards and audit work. Others come from AI governance groups, frontier lab evaluation teams, independent evaluators and research institutes. Psychometrics and measurement backgrounds would also be worth a look, especially for construct validity.

Interview questions for a Cross-Domain Capability Certification Designer

  1. Describe a capability claim you helped test. What evidence did it rest on, and where was it weakest?

    A strong answer names a real claim, the evidence used and a candid weak point. They should describe what they changed afterwards. Worry if they cannot name any weakness or only quote headline scores.

  2. How would you decide whether a result in one domain supports a claim in a different domain?

    Look for reasoning about shared task structure, distribution shift and construct validity, plus a plan to test transfer directly. Worry about assuming general ability from narrow success or treating domains as interchangeable.

4 more questions, with what a strong answer covers, are in the Role Kit.

Job description you can post

We are looking for someone to define what our general-purpose AI systems can credibly be said to do across consequential domains. You would design the certification scope and validation protocols behind those claims, and help us avoid claiming more than the evidence supports.

What you will do

  • Draft certification scopes for capability claims across several domains
  • Design validation protocols that test whether results transfer between domains
  • Review existing evaluations for construct validity and gaps
  • Work with evaluators, domain experts and risk teams to challenge proposed claims
  • Set conditions for review, suspension and withdrawal of certifications
  • Document limits and assumptions in plain language for leadership and external readers

What you will bring

  • Demonstrated work designing or critiquing evaluations, assurance cases or safety arguments
  • Clear understanding of construct validity and measurement limits
  • Experience testing AI or complex engineered systems where failure has consequences
  • Ability to explain uncertainty to non-specialists without softening it
  • Record of holding a line on claims under pressure
  • Comfort working across unfamiliar domains with expert input

Nice to have

  • Experience in AI control or formal methods
  • Background in standards, audit or regulated safety cases
  • Familiarity with AI governance and institutional design

Free to use and adapt. If it saves you time, a link back to this page is appreciated.

Depends on a scenario

Broad reliable autonomous delegation

What AI would need to do
Systems complete diverse consequential tasks with sustained reliability beyond existing narrow deployments.
What organisations would need to decide
Organisations explicitly delegate bounded authority and establish enforceable accountability.
What would rule it out
Persistent reliability limits or lack of useful cost advantage prevents broad delegation.

We give no date. The role becomes real if both conditions hold.

Skills that matter

  • AI controlAdvancedLimit harm from potentially untrusted AI behaviour
  • AI governance and assuranceAdvancedAssign accountability and verify lifecycle controls
  • Model evaluationWorkingMeasure capability and failure modes credibly
  • Frontier scenario analysisWorkingExplore conditional outcomes without claiming predictions

Other roles in general-purpose autonomy assurance

ScenarioGeneral-Purpose Autonomy Assurance ArchitectA General-Purpose Autonomy Assurance Architect would design how an organisation gains justified confidence in systems given broad, consequential…ScenarioAutonomous Delegation Accountability OfficerAn Autonomous Delegation Accountability Officer would keep track of who authorised what when AI systems carry out broad work. The role depends on…ScenarioGeneral-Purpose System Decommissioning PlannerA General-Purpose System Decommissioning Planner would prepare the controlled withdrawal of autonomous systems that run deeply inside an…ScenarioHead of Autonomous OperationsA Head of Autonomous Operations would run a business unit staffed mostly by agents with a small human team. It depends on how AI develops: it only…

Related roles in other families

ScenarioRecursive Improvement Change-Control ArchitectRecursive AI development change controlA Recursive Improvement Change-Control Architect would define enforceable limits on fast, automated changes to AI systems. The role depends on how…ScenarioAutomated AI Research Assurance LeadRecursive AI development change controlAn Automated AI Research Assurance Lead would challenge the evidence coming out of heavily automated AI development. The role depends on how AI…ScenarioCapability Acceleration Monitoring SpecialistRecursive AI development change controlA Capability Acceleration Monitoring Specialist would watch for evidence that AI systems are improving unusually fast through automated research. It…ScenarioSelf-Modifying System Verification EngineerRecursive AI development change controlA Self-Modifying System Verification Engineer would check that stated safety properties hold as AI systems change their own software. The role…

How we wrote this

Role definitions are our own editorial work, checked against occupational classifications, standards, research and employer pages. Titles vary by company, and this is a description of the work, not a job advert. Scenario roles describe responsibilities that depend on stated conditions; they are not forecasts.

← General-purpose autonomy assuranceAll new job families

Start a search

Hiring a Cross-Domain Capability Certification Designer?

Tell us the work. We find the people already doing it, whatever their job title says, and you meet them already interviewed for your role.