Hiring nowFoundation model pretraining

Pretraining Research Engineer

Also called Research Engineer, Pretraining · Pre-training Engineer · Research Scientist, Pretraining · Foundation Model Researcher

A pretraining research engineer runs controlled experiments on how large models learn from broad data. The work decides which architectures, data mixes and training recipes earn a full-scale run, and so where the compute budget goes.

What they do

They design small and mid-scale experiments, then read the results carefully. They change one thing at a time: an architecture tweak, a data mix, an optimiser setting, a learning-rate schedule. They watch for loss spikes, divergence and numerical problems, and work out the cause. They fit scaling curves to predict whether a change will hold at larger size. They work with data teams on quality, filtering and provenance. They write up each run so someone else can reproduce it, and they keep checkpoints versioned and documented.

What a strong hire delivers

In the first months, a strong hire leaves behind a trusted experimental baseline that others can reproduce. They deliver at least one clear finding that changes a training decision, backed by scaling evidence. They reduce wasted runs by catching instability early. They hand over versioned checkpoints with training and data documentation that another researcher can pick up and use without asking questions.

How you can tell they are good

Ask them to walk through a training run that went wrong and how they found the cause. Good candidates separate evidence from hunch, and can explain what they would not conclude from a result. Ask how they chose what to test and what they dropped. Be wary of someone who only quotes final scores and cannot describe the controls.

Why it is hard to fill

The global pool is tiny and almost all of it sits inside a handful of frontier labs. Titles vary widely: research engineer, research scientist, foundation model researcher. They often mean the same work. Few people have run large-scale training end to end, so many candidates have only seen part of it. Expect long, competitive conversations and be ready to sell the problem, not just the job.

Where to find them

Most come from frontier labs and from the few well-funded teams that train their own foundation models. Others come from deep-learning research groups in academia, often with publications on optimisation, scaling or architecture. Some come from large-scale training and infrastructure teams who have moved closer to the research. Look at authorship on training reports and open reproducible codebases, not job titles alone.

Indicative pay

Not enough published evidence yet to give a fair range for this role, which is common for work this new. We benchmark it for you as part of a search.

Interview questions for a Pretraining Research Engineer

  1. Describe a training run that diverged or behaved oddly. How did you find the cause?

    A strong answer names the symptoms, the hypotheses ruled out in order, and the evidence that settled it. Look for care with numerics, data and optimiser settings. Worry if the fix was luck or a restart with no explanation.

  2. How do you decide whether a result at small scale will hold at larger scale?

    Good answers cover scaling curves, fitting across several sizes, and the limits of extrapolation. They name effects that change with scale. Worry about blind faith in small-run wins or no mention of uncertainty.

4 more questions, with what a strong answer covers, are in the Role Kit.

Job description you can post

We train foundation models from the ground up. We are looking for a pretraining research engineer to run controlled experiments on how our models learn, and to turn the results into decisions about architecture, data and training recipes. You will work closely with research, data and infrastructure colleagues.

What you will do

  • Design and run controlled pretraining experiments at small and mid scale
  • Investigate training instability and find root causes
  • Run scaling experiments and use them to guide larger runs
  • Work with data colleagues on quality, filtering and provenance
  • Produce versioned checkpoints with training and data documentation
  • Write up results so others can reproduce and challenge them

What you will bring

  • Demonstrated work training deep-learning models at meaningful scale
  • Strong experimental design and clear reasoning from evidence
  • Hands-on depth in transformer modelling
  • Fluent in PyTorch or JAX
  • Track record of reproducible, well-documented research or engineering
  • Experience diagnosing numerical or optimisation problems in training

Nice to have

  • Published or open work on scaling, optimisation or architecture
  • Experience with large distributed training systems
  • Experience with data curation for pretraining

Free to use and adapt. If it saves you time, a link back to this page is appreciated.

Skills that matter

  • Deep learningAdvancedBuild and diagnose trainable neural models
  • Transformer modellingAdvancedImplement attention-based sequence or multimodal models
  • Experimental designWorkingChoose comparisons that identify credible effects
  • Data quality and provenanceWorkingTrack dataset origin, validity and transformations

Other roles in foundation model pretraining

Hiring nowModel Architecture ResearcherA Model Architecture Researcher tests changes to how a model is built, to improve its behaviour or efficiency. The work tells a company which…EmergingScaling Science ResearcherA Scaling Science Researcher works out how model performance changes with compute, data and model size. Their estimates tell a company how to spend…EmergingTraining Dynamics ResearcherA Training Dynamics Researcher works out why large training runs wobble, stall or diverge, and fixes the cause. Fewer failed runs means less wasted…

Related roles in other families

Hiring nowMachine Learning Research ScientistLearning theory and statistical machine learningA Machine Learning Research Scientist develops and tests learning methods against explicit hypotheses. They give the company evidence about what…Hiring nowStatistical Learning ScientistLearning theory and statistical machine learningA Statistical Learning Scientist builds prediction methods that show their assumptions and their uncertainty. For a company shipping models into…Hiring nowLearning Theory ResearcherLearning theory and statistical machine learningA Learning Theory Researcher works out what learning algorithms can and cannot do, and proves it. For an AI company, that means firmer ground for…Hiring nowAutomated Machine Learning ResearcherLearning theory and statistical machine learningAn Automated Machine Learning Researcher builds search procedures that choose and tune models within a fixed compute budget. It matters because…

How we wrote this

Role definitions are our own editorial work, checked against occupational classifications, standards, research and employer pages. Titles vary by company, and this is a description of the work, not a job advert.

← Foundation model pretrainingAll new job families

Start a search

Hiring a Pretraining Research Engineer?

Tell us the work. We find the people already doing it, whatever their job title says, and you meet them already interviewed for your role.