Hiring nowInference and model optimisation

Inference Systems Engineer

Also called Model Serving Engineer · Inference Engineer · Serving Engineer · LLM Systems Engineer

An Inference Systems Engineer builds the serving engines and runtime services that run models in production. They keep latency low and cost under control without letting output quality slip, which decides whether a product is affordable to run.

What they do

They design and run the layer between a trained model and the user. That means scheduling requests, batching them sensibly, managing the key-value cache and deciding how work is split across GPUs. They profile a live system, find where time and memory go, and change the code, kernels or architecture to fix it. They try quantisation and other compression, then check the quality cost against a fixed evaluation set. They build workload benchmarks that look like real traffic, not tidy test loads. They also work with product and infrastructure teams on capacity, rollout and failure behaviour.

What a strong hire delivers

In the first months, a strong hire gives the team an honest picture of how the current stack performs, built on benchmarks that reflect real traffic. They then ship measurable gains in tail latency, throughput and cost per request, with quality held at an agreed level. They leave behind a serving stack that others can profile, test and change without relying on one person's memory.

How you can tell they are good

Ask them to walk through a serving system they improved. A good candidate names the workload, the baseline, what the profiler showed and what they changed. They talk about tail latency, not just averages. They can explain the trade-offs in batching and caching and how they proved quality had not dropped. Be wary of vague claims about speed-ups with no measurement behind them.

Why it is hard to fill

Very few people have run large model serving in production, and most of them sit inside labs and AI platforms. The job title varies: serving engineer, inference engineer, LLM systems engineer. The skills are a mix of kernel-level GPU work, distributed systems and ML knowledge, and few people are strong in all three. Many candidates have one side only, so you need to judge which gap the team can cover.

Where to find them

Look in performance engineering, ML systems and distributed services teams. Strong candidates often come from labs, AI platform companies and infrastructure groups that run large GPU fleets. Some move over from search, databases, game engines or high-frequency systems where latency was the whole job. Open-source contributors to serving frameworks are worth a close look, because their work is public and you can read it.

Indicative pay

Not enough published evidence yet to give a fair range for this role, which is common for work this new. We benchmark it for you as part of a search.

Interview questions for an Inference Systems Engineer

  1. Tell me about a serving system you made faster. What was the workload, and how did you measure the result?

    Strong answers give a clear baseline, a realistic workload, the profiling evidence and the specific change. They report tail latency and cost, not only averages. Worry if numbers are vague, or if the benchmark was synthetic and never matched real traffic.

  2. How do you decide batch size and scheduling policy when latency targets and throughput goals pull against each other?

    Look for an understanding of continuous batching, queueing effects and the difference between time to first token and per-token latency. A good answer ties choices to the product's needs. Worry about one fixed answer that ignores the workload.

4 more questions, with what a strong answer covers, are in the Role Kit.

Job description you can post

We are building the serving layer that runs our models for real users. We need an Inference Systems Engineer to make it fast, reliable and affordable without lowering output quality. You will own the serving stack from scheduling through to GPU execution, and work closely with research and product teams.

What you will do

  • Design and run model serving engines and runtime services at scale
  • Improve tail latency, throughput and cost per request at fixed quality
  • Build workload benchmarks that reflect real production traffic
  • Profile GPU and system performance and fix the bottlenecks found
  • Apply and evaluate quantisation and other compression methods
  • Improve scheduling, batching and caching strategies
  • Work with infrastructure and product teams on capacity and rollout

What you will bring

  • Shipped or improved a production model serving system, with measured results
  • Strong grounding in distributed systems and performance engineering
  • Hands-on GPU programming or kernel-level optimisation experience
  • Experience using profilers to find and explain bottlenecks
  • Working knowledge of inference frameworks, batching and cache management
  • Clear judgement about quality trade-offs, backed by evaluation

Nice to have

  • Contributions to open-source serving or compiler projects
  • Experience with model compression and quantisation in production
  • Experience serving across multiple GPUs or nodes

Free to use and adapt. If it saves you time, a link back to this page is appreciated.

Skills that matter

  • Inference servingAdvancedServe models reliably under workload constraints
  • Model compressionAdvancedReduce model compute and storage requirements
  • GPU programmingWorkingOptimise parallel accelerator kernels
  • Distributed systemsWorkingDesign reliable work across networked machines

Other roles in inference and model optimisation

Hiring nowModel Compression EngineerA Model Compression Engineer makes trained models smaller and faster to run without letting quality slip past agreed limits. For a company, that…Hiring nowInference Performance EngineerAn Inference Performance Engineer profiles how a model is served, finds where time and accelerator capacity are lost, and fixes it without hurting…Hiring nowEdge AI EngineerAn Edge AI Engineer gets learned models running on constrained devices and offline environments, within clear limits on power, memory and latency…

Related roles in other families

Hiring nowDistributed Training EngineerDistributed training and research systemsA Distributed Training Engineer scales model training across many accelerators while keeping results correct. They keep expensive training runs…Hiring nowResearch Infrastructure EngineerDistributed training and research systemsA research infrastructure engineer gives researchers dependable compute and software environments, so experiments run, repeat and finish without…EmergingTraining Reliability EngineerDistributed training and research systemsA Training Reliability Engineer keeps long training runs alive and trustworthy. They detect failures, recover quickly and prove the recovered run is…Hiring nowResearch Productivity EngineerDistributed training and research systemsA Research Productivity Engineer builds the tools and automation that let researchers run experiments, analyse results and reproduce them faster. It…

How we wrote this

Role definitions are our own editorial work, checked against occupational classifications, standards, research and employer pages. Titles vary by company, and this is a description of the work, not a job advert.

← Inference and model optimisationAll new job families

Start a search

Hiring an Inference Systems Engineer?

Tell us the work. We find the people already doing it, whatever their job title says, and you meet them already interviewed for your role.