Hiring nowAI data centre networks and facilities

AI Compute Scheduler Engineer

Also called GPU Cluster Scheduler Engineer

An AI Compute Scheduler Engineer decides which model workloads get which cluster resources, and when. Good scheduling keeps expensive GPUs busy, treats teams fairly and gets training and inference jobs finished sooner.

What they do

The work is deciding how a shared GPU cluster is divided between competing jobs. That means writing and tuning scheduler policy: queues, priorities, quotas, preemption and admission control. They study how real workloads behave, from long training runs to bursty inference, and build evaluations that replay that demand against a proposed policy. They read network telemetry and power and cooling limits to see what the site can actually support. They talk to research and product teams about what they need, explain trade-offs, and handle the awkward cases when two important jobs want the same hardware.

What a strong hire delivers

Within the first months, a strong hire gives the company a written, tested scheduling policy that teams understand and accept. They put in place a workload evaluation method so changes can be judged before they reach the cluster. They find where capacity is being wasted, through fragmentation, idle reservations or poor placement, and cut it. They also set up clear reporting on utilisation, fairness and job completion time.

How you can tell they are good

Ask them to walk through a scheduling decision they made and the evidence behind it. Strong people talk about workload mix, failure cases and who lost out. Ask how they would test a new policy without risking live jobs. Be wary of anyone who only quotes utilisation and ignores fairness, queue wait or the effect on researchers.

Why it is hard to fill

Hardly anyone has this exact title, so job titles mislead. The skill sits between distributed systems, HPC and data centre engineering, and few people have worked on all three sides. Many good schedulers have never touched power or cooling, and many facilities engineers have never written policy. Clusters at real scale are also rare, so hands-on experience is limited to a small pool of people.

Where to find them

Look at HPC engineers who have run batch schedulers for research or industrial compute. Platform and infrastructure engineers from large cloud providers are another source, especially those who worked on orchestration and capacity. Network, electrical, mechanical and facilities engineers who moved towards workload questions can also fit. Frontier labs, model providers, GPU cloud operators and large research institutions are the likeliest places to find them.

Indicative pay

Not enough published evidence yet to give a fair range for this role, which is common for work this new. We benchmark it for you as part of a search.

Interview questions for an AI Compute Scheduler Engineer

  1. Tell me about a scheduling policy you designed or changed on a shared cluster. What problem were you solving?

    A strong answer names the workloads, the competing needs and the measured result. They explain trade-offs and who was affected. Worry if they describe only tools or configuration, with no reasoning about why the policy changed.

  2. How do you balance utilisation against fairness between teams?

    Good answers cover quotas, priorities, preemption and queue wait, and admit the two goals conflict. They give a concrete example. Worry if they treat high utilisation as the only goal or dismiss fairness as politics.

4 more questions, with what a strong answer covers, are in the Role Kit.

Job description you can post

We run a large GPU cluster shared by training and inference workloads, and we need an engineer to decide how that capacity is shared. You will design scheduling policy, test it against real demand and make sure teams get fair, predictable access while the hardware stays well used.

What you will do

  • Design and maintain scheduler policy, including queues, priorities, quotas and preemption
  • Build admission control that protects critical workloads
  • Create workload evaluations that replay real demand against proposed policy changes
  • Track utilisation, fairness and job completion time, and report on them clearly
  • Use network telemetry and power and cooling data to inform placement and limits
  • Work with research and product teams to understand their needs and explain trade-offs
  • Investigate scheduling-related incidents and put lasting fixes in place

What you will bring

  • Demonstrated work running or designing a scheduler for shared compute
  • Solid grounding in distributed systems and cluster orchestration
  • Understanding of GPU hardware and computer architecture
  • Experience analysing workload data to support policy decisions
  • Track record handling incidents and improving reliability
  • Clear communication with technical and non-technical teams

Nice to have

  • Background in HPC, network, electrical, mechanical or facilities engineering
  • Experience with power or cooling models at site level
  • Experience building simulation or replay tools for capacity planning

Free to use and adapt. If it saves you time, a link back to this page is appreciated.

Skills that matter

  • Hardware and computer architectureAdvancedDesign computing structures and hardware interfaces
  • Cloud and orchestrationAdvancedProvision and operate scalable compute services
  • Distributed systemsWorkingDesign reliable work across networked machines
  • Reliability and incident responseWorkingDetect failures, restore service and prevent recurrence

Other roles in aI data centre networks and facilities

Hiring nowAI Cluster Network EngineerAn AI Cluster Network Engineer designs and runs the network that links accelerators during distributed training. A well-built fabric keeps expensive…Hiring nowAI Data Centre Power EngineerAn AI Data Centre Power Engineer plans the electrical capacity and resilience of AI compute sites. They make sure GPU clusters get safe, reliable…Hiring nowAI Thermal Systems EngineerAn AI Thermal Systems Engineer designs and validates the cooling for dense accelerator racks. Good cooling keeps expensive compute inside…Hiring nowAI Data Centre Programme ManagerAn AI Data Centre Programme Manager delivers gigawatt-scale compute sites, covering land, power, cooling and schedule. They turn a company's compute…Hiring nowAI Power and Energy StrategistThe AI Power and Energy Strategist secures the electricity that AI compute sites run on. Without firm power on a workable schedule, GPUs sit idle…

Related roles in other families

Hiring nowMLOps EngineerMLOps and AI platform engineeringAn MLOps Engineer builds the pipeline that packages, tests and releases models in a repeatable way. It matters because releases become predictable…Hiring nowAI Platform EngineerMLOps and AI platform engineeringAn AI Platform Engineer builds the shared services that let teams use models and build AI applications without each reinventing the plumbing. It…EmergingLLMOps EngineerMLOps and AI platform engineeringAn LLMOps Engineer runs prompt, model and retrieval changes as controlled releases. It matters because language model features break quietly, and…Hiring nowModel Registry and Release EngineerMLOps and AI platform engineeringA Model Registry and Release Engineer keeps every model artefact traceable and every deployment approved. It matters because a company that cannot…

How we wrote this

Role definitions are our own editorial work, checked against occupational classifications, standards, research and employer pages. Titles vary by company, and this is a description of the work, not a job advert.

← AI data centre networks and facilitiesAll new job families

Start a search

Hiring an AI Compute Scheduler Engineer?

Tell us the work. We find the people already doing it, whatever their job title says, and you meet them already interviewed for your role.