Training Dynamics Researcher

Also called Optimisation Dynamics Scientist

A Training Dynamics Researcher works out why large training runs wobble, stall or diverge, and fixes the cause. Fewer failed runs means less wasted compute and faster, more predictable progress on the model.

What they do

Watches runs as they train and reads the signals that come before trouble: gradient norms, update sizes, loss spikes, activation statistics and sudden shifts in behaviour. Builds diagnostics that make those signals easy to see. Forms a hypothesis about the cause, then designs small, controlled experiments to test it. Tries corrections such as changes to learning rate schedules, initialisation, normalisation or data ordering. Checks that a fix does not quietly harm quality elsewhere. Writes up findings so the wider pretraining team can reuse them. Much of this work sits today inside research engineer or pretraining scientist titles.

What a strong hire delivers

In the first months, a strong hire gives the team a working diagnostic suite that flags instability early and is trusted by the people running training. They explain at least one recurring failure in clear terms and back it with experiment results. They land corrective changes that reduce unstable runs, and they show evidence that evaluation quality held steady. The team also gets clearer habits for logging and comparing runs.

How you can tell they are good

Ask them to walk through a real instability they diagnosed. Listen for the evidence trail: what they measured, what they ruled out, and how they confirmed the cause rather than the symptom. Ask how they checked a fix for hidden regressions. Strong people are careful about seeds, controls and noise. Be wary of anyone who credits a fix to a lucky hyperparameter change without being able to explain why it worked.

Why it is hard to fill

Hardly anyone holds this title, so searching by title finds very little. The skill is usually picked up inside pretraining teams by people called research scientist, research engineer or ML engineer. Candidates need real research judgement and hands-on experience with training runs large enough to misbehave. That experience is concentrated in a small number of teams, and those people are rarely on the market. Many good candidates would not describe themselves this way.

Where to find them

Look at pretraining and scaling teams at frontier labs and well-funded model builders. Also look at academic groups working on optimisation theory, deep learning science or scaling behaviour, where students often run careful diagnostic studies. Research engineers who maintain training infrastructure often develop the instincts too. Titles to search include research scientist, research engineer, optimisation researcher and ML scientist, then screen for evidence of diagnosing runs.

Indicative pay

Not enough published evidence yet to give a fair range for this role, which is common for work this new. We benchmark it for you as part of a search.

Interview questions for a Training Dynamics Researcher

Tell me about a training run that went unstable. How did you find the cause?
A strong answer names specific signals, the hypotheses tested and discarded, and how the cause was confirmed. Worry if the story is only about the fix, or if the cause was guessed and never verified.
Which gradient statistics do you log routinely, and why those?
Good answers cover gradient norms, per layer behaviour, update to weight ratios and noise measures, tied to what each reveals. Worry about a long list with no reasoning, or no awareness of logging cost.

4 more questions, with what a strong answer covers, are in the Role Kit.

Job description you can post

We are building large foundation models and need someone to understand why training runs go wrong. You will study instabilities and transitions during learning, build the diagnostics that expose them, and run the experiments that fix them. The aim is steadier runs without losing model quality.

What you will do

Build and maintain diagnostics for gradient statistics, optimiser behaviour and activation health
Investigate loss spikes, divergence and stalls in pretraining runs
Design controlled experiments to test causes and corrections
Test fixes against broad evaluations to catch hidden quality regressions
Trace instabilities back to data quality and provenance where relevant
Document findings and share practical guidance with the pretraining team

What you will bring

Demonstrated diagnosis of instabilities in real model training
Strong grounding in deep learning and transformer modelling
Rigorous experimental design, including seeds, controls and baselines
Hands-on experience with PyTorch or JAX
Reproducible training practice that others can rerun
Clear written explanations of technical findings

Nice to have

Experience with large-scale distributed training
Published or shared work on optimisation or training behaviour
Familiarity with data curation and provenance tracking

Free to use and adapt. If it saves you time, a link back to this page is appreciated.

Skills that matter

Deep learningAdvancedBuild and diagnose trainable neural models
Transformer modellingAdvancedImplement attention-based sequence or multimodal models
Experimental designWorkingChoose comparisons that identify credible effects
Data quality and provenanceWorkingTrack dataset origin, validity and transformations

Other roles in foundation model pretraining

Hiring nowPretraining Research EngineerA pretraining research engineer runs controlled experiments on how large models learn from broad data. The work decides which architectures, data…Hiring nowModel Architecture ResearcherA Model Architecture Researcher tests changes to how a model is built, to improve its behaviour or efficiency. The work tells a company which…EmergingScaling Science ResearcherA Scaling Science Researcher works out how model performance changes with compute, data and model size. Their estimates tell a company how to spend…

Related roles in other families

Hiring nowMachine Learning Research ScientistLearning theory and statistical machine learningA Machine Learning Research Scientist develops and tests learning methods against explicit hypotheses. They give the company evidence about what…Hiring nowStatistical Learning ScientistLearning theory and statistical machine learningA Statistical Learning Scientist builds prediction methods that show their assumptions and their uncertainty. For a company shipping models into…Hiring nowLearning Theory ResearcherLearning theory and statistical machine learningA Learning Theory Researcher works out what learning algorithms can and cannot do, and proves it. For an AI company, that means firmer ground for…Hiring nowAutomated Machine Learning ResearcherLearning theory and statistical machine learningAn Automated Machine Learning Researcher builds search procedures that choose and tune models within a fixed compute budget. It matters because…

How we wrote this

Role definitions are our own editorial work, checked against occupational classifications, standards, research and employer pages. Titles vary by company, and this is a description of the work, not a job advert.

Careers: jobs, Anthropic
Transformers documentation, Hugging Face

← Foundation model pretraining All new job families

Start a search

Hiring a Training Dynamics Researcher?

Tell us the work. We find the people already doing it, whatever their job title says, and you meet them already interviewed for your role.