Hiring nowDistributed training and research systems

Research Infrastructure Engineer

Also called Research Systems Engineer · ML Infrastructure Engineer · Training Infra Engineer · GPU Cluster Engineer

A research infrastructure engineer gives researchers dependable compute and software environments, so experiments run, repeat and finish without infrastructure getting in the way. It protects expensive GPU time and the credibility of results.

What they do

Builds and runs the platform researchers use to launch training jobs and run experiments. Day to day that means scheduling work across GPU clusters, fixing failed or stalled jobs, and tracing slowdowns to networking, storage or drivers. They package environments so runs can be repeated, set quotas and resource controls, and profile jobs that waste hardware. They also talk to researchers constantly, turn repeated complaints into tooling, and decide what to standardise and what to leave flexible.

What a strong hire delivers

In the first months, a strong hire makes failures visible and then rarer. Researchers get a clear way to launch, track and rerun experiments. Resource controls stop one job starving the rest. Common failures get automated recovery or clear alerts. The team can see where GPU time goes and how much is lost to infrastructure, and has a plan to reduce it.

How you can tell they are good

Ask them to walk through a real training failure they diagnosed, from first symptom to root cause. Good candidates name the layer at fault and how they proved it. Ask how they made an experiment reproducible and what they refused to standardise. Be wary of tool lists with no account of trade-offs, or of people who have only used clusters and never run one.

Why it is hard to fill

Very few people have run large GPU clusters for researchers, and most of them sit inside labs, neoclouds and hyperscalers that work hard to keep them. Titles vary wildly, so searching by title misses people. The skills straddle systems, HPC and ML, and candidates often lean one way. Researchers who like infrastructure are rare, and so are infrastructure engineers who respect research habits.

Where to find them

Look at engineers from HPC centres, distributed systems teams and cloud platform groups who have moved toward ML workloads. Others come from training or platform teams at frontier labs, neoclouds and hyperscalers, often titled systems, infra or cluster engineer. Some are ML engineers who got pulled into keeping the cluster alive and found they preferred it. Scheduler and networking specialists are worth a conversation.

Indicative pay

Not enough published evidence yet to give a fair range for this role, which is common for work this new. We benchmark it for you as part of a search.

Interview questions for a Research Infrastructure Engineer

  1. Tell me about a large training job that kept failing. How did you find the cause?

    A strong answer follows a clear method: logs, metrics, isolating hardware, network or software layers, then proof of the fix. Worry if they guess, blame researchers, or cannot say what they personally did.

  2. How would you make a researcher's experiment reproducible across clusters and over time?

    Look for pinned environments, versioned code, data and configuration, recorded seeds and hardware details, and honest limits such as nondeterministic kernels. Worry if the answer is only containers or only documentation.

4 more questions, with what a strong answer covers, are in the Role Kit.

Job description you can post

We train models at a scale where infrastructure decides how fast our researchers learn. We are looking for a research infrastructure engineer to run our compute and build the platform researchers use every day. You will work closely with researchers and own reliability end to end.

What you will do

  • Operate and improve GPU clusters and job scheduling for training and experiments
  • Build a reusable platform for launching, tracking and reproducing experiments
  • Design resource controls, quotas and priorities that researchers see as fair
  • Diagnose training failures and slowdowns across software, hardware and network
  • Automate recovery and alerting for common failure modes
  • Measure time lost to infrastructure and reduce it
  • Work with researchers to turn repeated pain points into tools

What you will bring

  • Demonstrated work running distributed training or large compute systems
  • Hands-on experience with Kubernetes, Slurm or similar schedulers
  • Solid understanding of networking and storage behaviour under heavy load
  • Experience profiling GPU workloads and working with CUDA-level issues
  • Clear examples of making research or compute workflows reproducible
  • Ability to explain technical trade-offs to researchers plainly

Nice to have

  • Experience supporting researchers directly in a lab setting
  • Work on multi-node training performance
  • Contributions to open source infrastructure or scheduling tools

Free to use and adapt. If it saves you time, a link back to this page is appreciated.

Skills that matter

  • Distributed systemsAdvancedDesign reliable work across networked machines
  • GPU programmingAdvancedOptimise parallel accelerator kernels
  • Cloud and orchestrationWorkingProvision and operate scalable compute services
  • Research operationsWorkingCoordinate research resources and reproducible execution

Other roles in distributed training and research systems

Hiring nowDistributed Training EngineerA Distributed Training Engineer scales model training across many accelerators while keeping results correct. They keep expensive training runs…EmergingTraining Reliability EngineerA Training Reliability Engineer keeps long training runs alive and trustworthy. They detect failures, recover quickly and prove the recovered run is…Hiring nowResearch Productivity EngineerA Research Productivity Engineer builds the tools and automation that let researchers run experiments, analyse results and reproduce them faster. It…

Related roles in other families

Hiring nowInference Systems EngineerInference and model optimisationAn Inference Systems Engineer builds the serving engines and runtime services that run models in production. They keep latency low and cost under…Hiring nowModel Compression EngineerInference and model optimisationA Model Compression Engineer makes trained models smaller and faster to run without letting quality slip past agreed limits. For a company, that…Hiring nowInference Performance EngineerInference and model optimisationAn Inference Performance Engineer profiles how a model is served, finds where time and accelerator capacity are lost, and fixes it without hurting…Hiring nowEdge AI EngineerInference and model optimisationAn Edge AI Engineer gets learned models running on constrained devices and offline environments, within clear limits on power, memory and latency…

How we wrote this

Role definitions are our own editorial work, checked against occupational classifications, standards, research and employer pages. Titles vary by company, and this is a description of the work, not a job advert.

← Distributed training and research systemsAll new job families

Start a search

Hiring a Research Infrastructure Engineer?

Tell us the work. We find the people already doing it, whatever their job title says, and you meet them already interviewed for your role.