The dashboard says the variant is up eleven per cent, the confidence bar has turned green, and somebody has already screenshotted it for Thursday's steering call. Nobody has asked how many accounts are in each arm. Product experimentation is the practice of changing one part of your product, holding everything else constant, and using a pre-named metric to decide whether the change stays. The definition is the easy part.
The arithmetic is what catches teams out, and it almost never gets run before the test goes live. Most mid-size product organisations cannot detect the effects they care about at the volumes they actually have. They run the test anyway, read the output as though it carried information, and ship on it.
How do you run product experimentation without enough traffic?
Run product experimentation on a smaller number of larger questions. Pick one metric, pre-commit to a decision rule in writing, accept a lower confidence threshold on purpose, and test changes big enough to move that metric by a wide margin. A test that cannot detect the effect you care about is not evidence.
The standard power calculation makes the constraint concrete. Detecting a five per cent relative improvement on a ten per cent baseline conversion rate, at eighty per cent power and ninety-five per cent two-sided confidence, needs roughly 58,000 users per arm. If your product has four hundred active users in total, that test will never finish.
Run it backwards and the picture becomes more useful. With two hundred users in each arm and the same baseline, the smallest effect you can reliably detect is about eight and a half percentage points. Your feature would need to take conversion from ten per cent to eighteen and a half before the test could see it at all.
For a binary metric, sample size per arm is approximately 16 × p × (1 − p) ÷ d², where p is your baseline rate and d is the absolute difference you want to detect. Rearranged, the minimum detectable effect is the square root of 16 × p × (1 − p) ÷ n. Both assume eighty per cent power and ninety-five per cent two-sided confidence.
Underpowered tests do not fail quietly, they fail in one direction
An underpowered experiment does not politely return "no result". When a small sample does cross the significance threshold, it crosses because the observed effect happened to be large — so the effects you detect are systematically overstated. You ship, the lift does not reproduce, and nobody connects the two events three months later.
Andrew Gelman and John Carlin's 2014 paper on Type S and Type M errors sets the mechanism out formally: at low power, statistically significant estimates are inflated, sometimes several times over. A significant result from a small sample is not a weak version of a strong result, it is a biased one.
The second failure is procedural rather than statistical. Ron Kohavi, Diane Tang and Ya Xu's Trustworthy Online Controlled Experiments (Cambridge University Press, 2020) draws on organisations running more than 20,000 experiments a year, and much of it concerns trust rather than mathematics — whether the number in front of you means what the dashboard claims it means.
Evan Miller showed in 2010 that checking an experiment repeatedly and stopping when it looks significant inflates the false positive rate. Peek ten times and a result you are reading as one per cent significance is closer to five. Every Monday stand-up where somebody opens the experiment dashboard is a peek.
Four hundred users is not four hundred data points
In business-to-business and institutional products, users are not independent of one another. Ten traders at the same firm share a workflow, a training session and an internal champion, so when one changes behaviour the others tend to follow. The statistical unit is the account, not the seat.
The correction has a name and a formula. The design effect is 1 + (m − 1) × ICC, where m is the average number of users per account and the intracluster correlation (ICC) measures how much behaviour clusters inside accounts. At ten users per account and an ICC of 0.2 — illustrative, and you should measure your own — the design effect is 2.8 and four hundred users behave like about 143.
Randomise by seat and you will measure the same firm ten times, then call it a sample.
Randomise by account, count by account, and your sample is the number of logos rather than the number of logins. That figure is usually small enough to end the A/B testing conversation before it starts. It is also the number your sales team already quotes in every board pack, which makes the argument unusually easy to have.
Alternatives to A/B testing, and how each one misleads you
Every alternative to a fixed-horizon A/B test buys you something and charges you something. Sequential tests buy repeated looks and charge wider intervals. Holdouts buy a clean comparison and charge you a group of customers who never receive the feature. None of them removes the need to decide in advance what would change your mind.
| Method | What it actually tells you | How it fails |
|---|---|---|
| Fixed-horizon A/B test | Whether the difference exceeds chance, at a sample size named before launch | The required run time often outlasts the decision. Teams stop early, and the stated confidence becomes decorative. |
| Sequential test | The same, but you may look repeatedly without inflating false positives | Pays for the looking with wider early intervals, so modest effects clear the boundary later than a naive dashboard implies. |
| Bayesian posterior | A probability that the variant is better, which stakeholders read more easily | The prior does real work and is usually whatever the tool shipped with. Stopping on a threshold is not free here either. |
| Before-and-after comparison | That the metric moved in the window following release | Confounded by everything else in that window: a pricing change, quarter end, one large account onboarding, a market event. |
| Holdout group | A running estimate of cumulative effect against users kept on the old behaviour | Sales exempts the accounts that complain. That deletion is not random, and it removes exactly the accounts whose behaviour mattered. |
| Task observation with five users | Whether the feature is usable and understood | Says nothing about revenue or retention, then gets presented in steering decks as though it did. |
The last row is where most teams on low-volume products should be spending their effort, and it pairs naturally with a continuous discovery habit that keeps interviews running between releases. Observation answers "does this work" quickly and cheaply. It does not answer "did this pay", and blurring the two is how a usability finding ends up carrying a business case.
Write the decision rule before anyone sees the feature
A decision rule is a sentence written before launch that names the metric, the threshold, the stop date and what happens under each outcome. It converts an argument about interpretation into an argument about design, which is far cheaper to have. Put it in the ticket, not in a document nobody opens.
- Name one metric and the unit of randomisation — user, account or portfolio. Choosing the metric is the hard part, and the reasoning in the guide to numbers that quietly mislead applies directly.
- Count independent units rather than users, and apply the design effect where users cluster inside accounts.
- Run the power calculation before the ticket is written. If the minimum detectable effect is larger than the change could plausibly deliver, you are not running an experiment and should say so out loud.
- Choose the evidence standard deliberately. Eighty per cent confidence is defensible for a cheap, reversible change; it is not defensible undeclared.
- Write the rule as an if-then, including the number that triggers a rollback and not only the number that counts as a win.
- Circulate it to the people who will argue about the result — sales, second line, the executive who asked for the feature — while changing it is still free.
- Read the result once, on the stop date, and do what the rule says.
Step six is the one teams skip, and it is the only step that changes the outcome of the meeting. A threshold agreed in advance is a constraint. The same threshold produced afterwards is an opinion, and an opinion loses to whoever is most senior in the room.
Regulated products change the sequence, not the method
The statistics of product experimentation are identical in a trading platform and a photo-sharing app. What differs is when you are permitted to start. Anything that varies pricing, credit access, order handling or material a customer could construe as advice needs review before it runs, and that review is not a formality you can schedule around.
Plan for the sequence rather than fighting it. Take the experiment design to second line at the same time as the feature design, because their questions — how are users allocated, what is the detriment to the control group, how will you evidence the decision afterwards — reshape the design in ways that are cheap now and expensive in week three. The same logic runs through working with a compliance function that can veto a release.
The operational failure is duller than the regulatory one. The design goes across on the Tuesday of the last sprint week, comes back with two questions on the Friday, and answering them needs a meeting nobody can book until the following Wednesday. The flag sits at zero per cent for a fortnight while the board pack reports the feature as delivered.
The vocabulary that keeps these arguments honest
Most disagreements about experiment results are really disagreements about terms. Four hundred users against four hundred accounts. Confidence against probability. Significance against size. Agreeing the words in advance removes a surprising share of the argument, and all six of these belong in the ticket rather than in somebody's head.
- Minimum detectable effect (MDE) — the smallest change your sample size can distinguish from noise at your chosen power and confidence.
- Statistical power — the probability your test finds a real effect of a given size; eighty per cent is the common default, which means one real effect in five is missed.
- Unit of randomisation — the thing you assign to treatment or control, which must match the unit whose behaviour you are claiming changed.
- Intracluster correlation (ICC) — how much behaviour within a group, such as an account, resembles itself rather than the wider population.
- Overall Evaluation Criterion (OEC) — Kohavi, Tang and Xu's term for the single combined metric an experiment is judged against, including the guardrails it must not damage.
- Holdout group — users deliberately kept on the previous behaviour after rollout, so cumulative effect can be estimated over time.
Frequently asked questions
How many users do I need to run an A/B test?
It depends on your baseline rate and the size of effect you want to catch, not on a universal minimum. Detecting a five per cent relative lift on a ten per cent baseline, at eighty per cent power and ninety-five per cent confidence, needs roughly 58,000 users per arm. Detecting a doubling needs a few hundred. Calculate before you design.
Can I lower the confidence level to ninety per cent to get a result faster?
Yes, provided you write the choice down before the test starts and say why. Ninety per cent means one false positive in ten rather than one in twenty, which is a reasonable trade for a cheap, reversible change. It is not reasonable for a pricing change, and it is never reasonable to pick the threshold after seeing the data.
What should I do if I cannot reach statistical significance at all?
Stop calling it an experiment and change the evidence standard openly. Use a staged rollout with a named rollback trigger, pair it with task observation and support-ticket monitoring, and record the outcome as a judgement supported by weak evidence. That is an honest position to hold in a steering meeting. An underpowered test reported as a win is not.
Is a before-and-after comparison ever acceptable?
For large, fast effects inside a stable window, sometimes. The risk is that everything else moves too — a release, a pricing update, quarter end, one large customer onboarding. If you use one, name the confounders you know about in writing beforehand, then check whether any of them occurred before you interpret the change.
How long should an experiment run?
At least one full business cycle, so weekly and monthly patterns average out, and never less than the run length implied by your sample size calculation. In institutional products that cycle can be a quarter, because behaviour clusters around reporting dates and month end. Set the stop date before launch and do not move it afterwards.
Do I need an experimentation platform to do this properly?
No. A feature flag, deterministic assignment based on account identifier, and a query you can rerun will cover most low-volume cases. Platforms earn their cost when the marginal effort of running an experiment is what limits how many you run. At a handful of tests a quarter, discipline matters more to product experimentation than tooling does.
Once the decision rule is written, the next argument arrives immediately: which changes deserve one at all. Not every release needs an experiment, and running three underpowered tests a quarter is worse than running one honest staged rollout. That is a prioritisation question rather than a statistical one, and the frameworks people reach for each mislead in their own way.