Hiring nowAI compilers, kernels and runtime systems

ML Runtime Engineer

Also called AI Runtime Engineer

An ML Runtime Engineer builds the scheduling, memory and execution layer that runs models on real hardware. Their work decides whether a model serves reliably, fits in memory and reaches the throughput the business needs.

What they do

They write and maintain the parts of a model runtime that decide what runs, where and when. That means scheduling work across accelerators, managing memory allocation and reuse, and handling workloads that change shape from one request to the next. They profile runs, find stalls and fragmentation, and fix them in code. They work close to compiler output and GPU kernels, so they read intermediate representations and kernel behaviour as well as systems code. They write stress tests that push the runtime past normal load, and correctness tests that check results stay right after every change. They also debug hard failures such as hangs, leaks and rare race conditions.

What a strong hire delivers

In the first months, a strong hire ships a runtime component that is in real use, with stress and correctness tests that others trust. They find the biggest memory and throughput losses in the current execution path and remove some of them. They make failures easier to diagnose through better profiling and logging. They leave the team clearer about where runtime limits sit and which hardware trade-offs matter most.

How you can tell they are good

Ask them to walk through a runtime problem they solved, from first symptom to fix. Strong people show profiler evidence, name the real bottleneck and explain what they rejected. Ask how they proved correctness after the change. Listen for specifics on memory lifetimes, scheduling trade-offs and concurrency bugs. Be wary of answers that only quote benchmark gains without explaining the cause.

Why it is hard to fill

Almost nobody has this title, so searching by title misses most of the pool. The work sits between compilers, operating systems style systems programming and accelerator performance, and few people are strong in all three. Many candidates know kernels well but have never owned scheduling or memory management. Others know systems but have not worked on accelerators. Good ones are usually settled and in demand, so the pitch has to be about the problem.

Where to find them

Look first at compiler engineers, systems programmers and GPU performance engineers. Titles often read as inference engineer, runtime developer, performance engineer or kernel engineer. They come from frontier labs, model providers, accelerator and chip companies, and teams building agent frameworks or inference platforms. Some come from database engines, game engines and high performance computing, where memory and scheduling discipline is already routine.

Indicative pay

Not enough published evidence yet to give a fair range for this role, which is common for work this new. We benchmark it for you as part of a search.

Interview questions for an ML Runtime Engineer

  1. Tell me about a runtime or systems bottleneck you found and fixed. How did you locate it?

    A strong answer names the symptom, the profiler evidence and the actual cause, then the fix and the measured result. It should show method rather than guesswork. Worry if they credit luck or cannot explain why the fix worked.

  2. How would you reduce memory use in a runtime serving requests of very different lengths?

    Good answers cover allocation strategy, fragmentation, reuse, batching and the cost of each option on latency. They should ask about workload and hardware. Worry if they offer one trick as a universal answer.

4 more questions, with what a strong answer covers, are in the Role Kit.

Job description you can post

We are hiring an ML Runtime Engineer to build the scheduling, memory and execution layer that runs our models on accelerators. You will work close to compilers and kernels, and your work will decide how reliably and efficiently our models run in production.

What you will do

  • Implement scheduling, memory management and execution in our model runtime
  • Handle dynamic workloads with changing shapes and request patterns
  • Profile runs and remove bottlenecks in memory use and throughput
  • Write stress and correctness tests for every runtime component you ship
  • Debug hangs, leaks and concurrency faults in production and test
  • Work with kernel and compiler engineers on how generated code is executed

What you will bring

  • Shipped systems code that manages memory, scheduling or concurrency under load
  • Hands-on GPU programming and use of performance profilers
  • Clear understanding of hardware and computer architecture
  • Experience with compilers, intermediate representations or accelerator performance
  • Evidence of testing for correctness as well as speed
  • Comfort with distributed systems concepts

Nice to have

  • Experience with Triton or similar kernel languages
  • Work on inference serving for large models
  • Contributions to open source runtimes or compilers

Free to use and adapt. If it saves you time, a link back to this page is appreciated.

Skills that matter

  • GPU programmingAdvancedOptimise parallel accelerator kernels
  • Hardware and computer architectureAdvancedDesign computing structures and hardware interfaces
  • Linear algebra and optimisationWorkingReason about matrix operations and objective functions
  • Distributed systemsWorkingDesign reliable work across networked machines

Other roles in aI compilers, kernels and runtime systems

Hiring nowAI Inference Compiler EngineerAn AI Inference Compiler Engineer makes trained models run faster and cheaper on real hardware. They change how model computation is compiled and…Hiring nowGPU Kernel EngineerA GPU Kernel Engineer writes and tunes the low-level code that runs model operations on accelerators. Faster, correct kernels cut compute cost and…EmergingAI Performance Portability EngineerAn AI Performance Portability Engineer keeps models fast and accurate across different hardware. They cut dependence on one chip or vendor, which…

Related roles in other families

Hiring nowAI Accelerator ArchitectAI accelerators and hardware co-designAn AI Accelerator Architect decides how a chip or compute system should be built for specific training or inference workloads. Their choices set the…Hiring nowAI Hardware Verification EngineerAI accelerators and hardware co-designAn AI Hardware Verification Engineer proves that accelerator logic and its interfaces behave correctly before silicon is committed. Escaped defects…Hiring nowHardware-Software Co-Design EngineerAI accelerators and hardware co-designA Hardware-Software Co-Design Engineer shapes model software and chip interfaces together, so neither is tuned in isolation. For an AI company, that…EmergingNeuromorphic Computing ResearcherAI accelerators and hardware co-designA neuromorphic computing researcher tests whether event-driven, brain-inspired computation can beat conventional hardware on a real task. The answer…

How we wrote this

Role definitions are our own editorial work, checked against occupational classifications, standards, research and employer pages. Titles vary by company, and this is a description of the work, not a job advert.

← AI compilers, kernels and runtime systemsAll new job families

Start a search

Hiring an ML Runtime Engineer?

Tell us the work. We find the people already doing it, whatever their job title says, and you meet them already interviewed for your role.