Hiring nowAI compilers, kernels and runtime systems

AI Inference Compiler Engineer

Also called Compiler Engineer, AI Inference

An AI Inference Compiler Engineer makes trained models run faster and cheaper on real hardware. They change how model computation is compiled and executed, cutting latency and serving cost without breaking the numbers.

What they do

The work starts with a model graph and ends with code that runs well on an accelerator. Day to day, the engineer reads profiler traces, finds where time and memory go, and decides whether the fix belongs in a graph pass, a fused kernel or the code generator. They write compiler passes for lowering, fusion, layout choice and scheduling. They write or tune GPU kernels where the compiler falls short. They build benchmarks that check speed and numerical agreement against a reference. They also work with model and serving teams to learn which workloads matter most.

What a strong hire delivers

In the first months, a strong hire gives the team a clear picture of where inference time goes and a ranked list of fixes. They ship the first compiler passes or kernel changes that measurably cut latency or raise throughput on the main workloads. They leave behind a benchmark suite that catches regressions in both speed and accuracy, so later work is safer and easier to judge.

How you can tell they are good

Ask them to walk through one optimisation from profile to result. A good candidate names the bottleneck, explains why it existed, shows the measured gain and says how they proved the outputs still matched. Ask what they tried that failed. Strong people are specific about memory movement, precision and hardware limits. Be wary of vague claims about speed-ups with no method.

Why it is hard to fill

Few people sit across compilers, GPU performance and machine learning at once. Most come from one side and are learning the others. Titles are inconsistent: the same work appears as compiler engineer, kernel engineer, performance engineer or runtime engineer. Strong candidates are often tied up in frontier labs and chip companies, and CVs rarely show the depth. Judging them needs someone who can test it.

Where to find them

Look in compiler teams at chip makers and accelerator start-ups, and in machine learning framework and runtime groups. Also look at GPU performance and kernel engineers, high performance computing specialists, and systems engineers at model providers or inference platform companies. Some come from language compiler work such as LLVM-style toolchains and have moved towards machine learning workloads. Open source compiler contributors are worth checking.

Indicative pay

Not enough published evidence yet to give a fair range for this role, which is common for work this new. We benchmark it for you as part of a search.

Interview questions for an AI Inference Compiler Engineer

  1. Walk me through an optimisation you made to a compiled neural workload, from first profile to final result.

    A strong answer names the bottleneck, the evidence, the change and the measured effect. It covers how correctness was checked. Worry if the story is only about the outcome, or if they cannot explain why it worked.

  2. When would you fix a slow operator in a graph pass, and when in a hand-written kernel?

    Good answers weigh generality against control, maintenance cost and how often the pattern appears. They mention fusion, layout and hardware limits. Worry about a fixed preference for one approach regardless of context.

4 more questions, with what a strong answer covers, are in the Role Kit.

Job description you can post

We are hiring an AI Inference Compiler Engineer to make our models run faster and more efficiently on the hardware we use. You will work on how model computation is lowered, optimised and executed, alongside model and serving engineers. Success means lower latency and cost with outputs we can trust.

What you will do

  • Write and improve compiler passes for graph lowering, fusion, layout and scheduling
  • Profile inference workloads and find the main sources of latency and memory cost
  • Write or tune GPU kernels where compiler output is not enough
  • Build benchmarks that track speed and numerical correctness
  • Work with model and serving teams to prioritise the workloads that matter
  • Document optimisations so others can maintain and extend them

What you will bring

  • Demonstrated work on compilers, code generation or accelerator performance
  • Strong systems programming skills in a language such as C++
  • Hands-on GPU programming experience
  • Clear understanding of computer architecture and memory hierarchies
  • Comfort with linear algebra and numerical precision trade-offs
  • Evidence of measuring optimisations rigorously and checking correctness

Nice to have

  • Experience with compiler IRs and writing custom passes
  • Experience with kernel languages such as Triton
  • Exposure to distributed inference
  • Open source contributions to compiler or runtime projects

Free to use and adapt. If it saves you time, a link back to this page is appreciated.

Skills that matter

  • GPU programmingAdvancedOptimise parallel accelerator kernels
  • Hardware and computer architectureAdvancedDesign computing structures and hardware interfaces
  • Linear algebra and optimisationWorkingReason about matrix operations and objective functions
  • Distributed systemsWorkingDesign reliable work across networked machines

Other roles in aI compilers, kernels and runtime systems

Hiring nowGPU Kernel EngineerA GPU Kernel Engineer writes and tunes the low-level code that runs model operations on accelerators. Faster, correct kernels cut compute cost and…Hiring nowML Runtime EngineerAn ML Runtime Engineer builds the scheduling, memory and execution layer that runs models on real hardware. Their work decides whether a model…EmergingAI Performance Portability EngineerAn AI Performance Portability Engineer keeps models fast and accurate across different hardware. They cut dependence on one chip or vendor, which…

Related roles in other families

Hiring nowAI Accelerator ArchitectAI accelerators and hardware co-designAn AI Accelerator Architect decides how a chip or compute system should be built for specific training or inference workloads. Their choices set the…Hiring nowAI Hardware Verification EngineerAI accelerators and hardware co-designAn AI Hardware Verification Engineer proves that accelerator logic and its interfaces behave correctly before silicon is committed. Escaped defects…Hiring nowHardware-Software Co-Design EngineerAI accelerators and hardware co-designA Hardware-Software Co-Design Engineer shapes model software and chip interfaces together, so neither is tuned in isolation. For an AI company, that…EmergingNeuromorphic Computing ResearcherAI accelerators and hardware co-designA neuromorphic computing researcher tests whether event-driven, brain-inspired computation can beat conventional hardware on a real task. The answer…

How we wrote this

Role definitions are our own editorial work, checked against occupational classifications, standards, research and employer pages. Titles vary by company, and this is a description of the work, not a job advert.

← AI compilers, kernels and runtime systemsAll new job families

Start a search

Hiring an AI Inference Compiler Engineer?

Tell us the work. We find the people already doing it, whatever their job title says, and you meet them already interviewed for your role.