Hiring nowCausal inference, decisions and experimentation

Experimentation Data Scientist

Also called Experimentation Scientist

An Experimentation Data Scientist designs and analyses product and operational experiments so the company can tell what actually works. They turn uncertain results into clear decisions, and stop teams shipping changes on noise.

What they do

They take a question such as whether a change helps users or the business and turn it into a test that can answer it. That means choosing the unit of randomisation, setting the metric and guardrails, and working out the sample needed for adequate power. They write the protocol before launch. They watch running tests for sample ratio problems and interference between users. They decide how and when a test can stop, using sequential methods where teams want early looks. After the test they analyse results, explain uncertainty and write a decision report that product and operations teams can act on.

What a strong hire delivers

In the first months, a strong hire gives the company experiments it can trust. They put a clear protocol template in place and agree guardrail metrics with product leads. They fix or flag weak tests already running. They deliver decision reports that state the call, the evidence and the risk. Teams start asking for design input before launch, not analysis after it. Fewer inconclusive tests reach the end of their run.

How you can tell they are good

Ask them to walk through a real experiment from design to decision. Listen for how they chose the randomisation unit, how they sized the test and what they did when results were awkward. A strong candidate explains peeking, multiple comparisons and interference in plain words. They can say when not to run a test. Be wary of anyone who talks only about p-values or tools.

Why it is hard to fill

Titles vary widely. The same work appears as experimentation scientist, decision scientist, product analyst or econometrician. Many candidates can run a standard A/B test, but fewer can handle sequential testing, interference or low-traffic settings. Others have strong theory and little product exposure. The useful hire sits between the two, and a CV rarely shows which side a person is on. We screen on the work they have actually designed.

Where to find them

Look first at data scientists and statisticians in companies that run many tests, such as consumer products, marketplaces and platforms. Economists and econometricians with applied policy or industry experience move across well. Decision scientists and operations researchers are another source. Some come from experiment platform teams, where they have built the methods as well as used them. Academic researchers in applied statistics can fit if they have shipped decisions.

Indicative pay

Not enough published evidence yet to give a fair range for this role, which is common for work this new. We benchmark it for you as part of a search.

Interview questions for an Experimentation Data Scientist

  1. Describe an experiment you designed end to end. What did you decide before launch, and what changed your plan?

    Strong answers cover the hypothesis, unit of randomisation, metrics, guardrails, power and the protocol. They show real adjustments made during the run and a clear final decision. Worry if the story starts at analysis or hides what went wrong.

  2. How do you choose the unit of randomisation, and when does that choice cause trouble?

    Look for clear reasoning about users, sessions, accounts or clusters, and awareness of interference, network effects and repeated exposure. A strong candidate links the choice to the decision being made. Worry about a default answer of randomising by user.

4 more questions, with what a strong answer covers, are in the Role Kit.

Job description you can post

We are hiring an Experimentation Data Scientist to help us decide what to build and ship using sound evidence. You will design and analyse experiments across our product and operations, and turn the results into clear recommendations. We value careful thinking, plain communication and work that holds up.

What you will do

  • Design product and operational experiments, including randomisation, metrics and power
  • Write experiment protocols before launch and agree guardrails with stakeholders
  • Monitor running tests for data quality and validity issues
  • Apply sequential testing methods where early decisions are needed
  • Analyse results and write clear decision reports
  • Advise teams when an experiment is not the right tool
  • Improve our experimentation standards and tooling

What you will bring

  • Experiments you have designed and analysed, with decisions that followed
  • Strong grounding in probability, statistics and experimental design
  • Working knowledge of causal inference methods beyond simple A/B tests
  • Experience with sequential testing or similar stopping rules
  • Fluent in Python or R for analysis
  • Clear written reports for non-specialist readers
  • Good judgement on trade-offs and guardrails

Nice to have

  • Experience with experiment platforms
  • Background in econometrics or decision science
  • Experience testing AI-driven product features

Free to use and adapt. If it saves you time, a link back to this page is appreciated.

Skills that matter

  • Causal inferenceAdvancedSeparate causal claims from predictive associations
  • Experimental designAdvancedChoose comparisons that identify credible effects
  • Probability and statisticsWorkingQuantify variability and statistical evidence
  • Decision theory and game theoryWorkingAnalyse choices, incentives and strategic interactions

Other roles in causal inference, decisions and experimentation

Hiring nowCausal Inference ScientistA Causal Inference Scientist estimates what actually changes when the company acts, under assumptions that are stated and tested. That keeps…Hiring nowDecision ScientistA Decision Scientist turns uncertain evidence into clear, constrained choices. They help an AI company decide what to ship, test, fund or stop, and…Hiring nowAlgorithmic Game Theory ResearcherAn Algorithmic Game Theory Researcher studies how agents respond to incentives inside computational systems. They design mechanisms and strategic…

Related roles in other families

Hiring nowMachine Learning Research ScientistLearning theory and statistical machine learningA Machine Learning Research Scientist develops and tests learning methods against explicit hypotheses. They give the company evidence about what…Hiring nowStatistical Learning ScientistLearning theory and statistical machine learningA Statistical Learning Scientist builds prediction methods that show their assumptions and their uncertainty. For a company shipping models into…Hiring nowLearning Theory ResearcherLearning theory and statistical machine learningA Learning Theory Researcher works out what learning algorithms can and cannot do, and proves it. For an AI company, that means firmer ground for…Hiring nowAutomated Machine Learning ResearcherLearning theory and statistical machine learningAn Automated Machine Learning Researcher builds search procedures that choose and tune models within a fixed compute budget. It matters because…

How we wrote this

Role definitions are our own editorial work, checked against occupational classifications, standards, research and employer pages. Titles vary by company, and this is a description of the work, not a job advert.

← Causal inference, decisions and experimentationAll new job families

Start a search

Hiring an Experimentation Data Scientist?

Tell us the work. We find the people already doing it, whatever their job title says, and you meet them already interviewed for your role.