Hiring nowDistributed training and research systems

Distributed Training Engineer

Also called ML Distributed Systems Engineer

A Distributed Training Engineer scales model training across many accelerators while keeping results correct. They keep expensive training runs fast, stable and repeatable, so research ideas reach a trained model sooner.

What they do

They take a training job that works on one machine and make it run across many. That means choosing how to split the model and the data, and setting up communication between accelerators so every step stays in sync. They profile GPU use, find where time is lost, and fix stalls in data loading, networking or memory. When a run hangs or the loss drifts, they trace the cause across nodes. They work with job schedulers and cloud infrastructure, and they write down what they learn so researchers can launch large jobs without them.

What a strong hire delivers

In the first months, a strong hire gets a training stack that researchers can launch across multiple nodes with little hand-holding. They produce a multi-node validation report showing throughput, scaling efficiency and numerical consistency against a smaller baseline. They remove the worst bottlenecks, cut failed or wasted runs, and leave clear runbooks for restarting and debugging jobs.

How you can tell they are good

Ask them to walk through a real scaling problem they solved. Strong people explain why they chose a parallelism strategy, what they measured, and what broke. They can read a profiler trace and say where the time went. Ask how they checked that a distributed run matched a single-node one. Vague talk about speed-ups with no numerical checks is a warning sign.

Why it is hard to fill

Few people have run genuinely large jobs, because few teams have had the hardware. Titles vary widely, and many candidates are strong in one half only: either systems and networking, or machine learning. The useful hire needs both, plus the patience to debug failures that appear once in a long run. Those people are usually already busy and well placed.

Where to find them

Look at high-performance computing engineers, distributed systems engineers moving towards machine learning, and ML infrastructure or ML systems engineers. They come from frontier labs, model providers, large cloud and hardware teams, and research-heavy start-ups. Some come from national labs or scientific computing groups, where large cluster jobs and schedulers are everyday work.

Indicative pay

Not enough published evidence yet to give a fair range for this role, which is common for work this new. We benchmark it for you as part of a search.

Interview questions for a Distributed Training Engineer

  1. Describe a training job you scaled across multiple nodes. What limited it first, and what did you change?

    A strong answer names the actual bottleneck, such as communication, data loading or memory, and the measurement that showed it. They describe the fix and the result. Worry if the story is only about adding more GPUs.

  2. How do you choose between data, tensor and pipeline parallelism for a new model?

    Look for reasoning about model size, memory, interconnect and batch size, with trade-offs stated plainly. Good candidates mention combining strategies and testing. Worry about one favourite approach applied to every case.

4 more questions, with what a strong answer covers, are in the Role Kit.

Job description you can post

We train large models and need an engineer to make that training fast, stable and repeatable across many accelerators. You will work closely with researchers, own our distributed training stack, and help us turn compute into results with fewer wasted runs.

What you will do

  • Scale training jobs across multiple nodes and accelerators
  • Choose and implement parallelism strategies for new models
  • Profile GPU and cluster performance and remove bottlenecks
  • Debug hangs, crashes and numerical differences in distributed runs
  • Maintain job scheduling, checkpointing and resume workflows
  • Produce multi-node validation reports covering throughput, scaling efficiency and consistency
  • Document tooling so researchers can launch jobs independently

What you will bring

  • Demonstrated work scaling training across multiple nodes
  • Hands-on use of distributed training frameworks
  • Experience reading GPU profiler output and acting on it
  • Solid grounding in distributed systems, including communication and failure handling
  • Experience with job schedulers and cloud orchestration
  • Clear written communication of technical findings

Nice to have

  • Custom GPU kernel or communication work
  • Experience running jobs on large shared clusters
  • Background in high-performance computing

Free to use and adapt. If it saves you time, a link back to this page is appreciated.

Skills that matter

  • Distributed systemsAdvancedDesign reliable work across networked machines
  • GPU programmingAdvancedOptimise parallel accelerator kernels
  • Cloud and orchestrationWorkingProvision and operate scalable compute services
  • Research operationsWorkingCoordinate research resources and reproducible execution

Other roles in distributed training and research systems

Hiring nowResearch Infrastructure EngineerA research infrastructure engineer gives researchers dependable compute and software environments, so experiments run, repeat and finish without…EmergingTraining Reliability EngineerA Training Reliability Engineer keeps long training runs alive and trustworthy. They detect failures, recover quickly and prove the recovered run is…Hiring nowResearch Productivity EngineerA Research Productivity Engineer builds the tools and automation that let researchers run experiments, analyse results and reproduce them faster. It…

Related roles in other families

Hiring nowInference Systems EngineerInference and model optimisationAn Inference Systems Engineer builds the serving engines and runtime services that run models in production. They keep latency low and cost under…Hiring nowModel Compression EngineerInference and model optimisationA Model Compression Engineer makes trained models smaller and faster to run without letting quality slip past agreed limits. For a company, that…Hiring nowInference Performance EngineerInference and model optimisationAn Inference Performance Engineer profiles how a model is served, finds where time and accelerator capacity are lost, and fixes it without hurting…Hiring nowEdge AI EngineerInference and model optimisationAn Edge AI Engineer gets learned models running on constrained devices and offline environments, within clear limits on power, memory and latency…

How we wrote this

Role definitions are our own editorial work, checked against occupational classifications, standards, research and employer pages. Titles vary by company, and this is a description of the work, not a job advert.

← Distributed training and research systemsAll new job families

Start a search

Hiring a Distributed Training Engineer?

Tell us the work. We find the people already doing it, whatever their job title says, and you meet them already interviewed for your role.