RecSys Part I · Statement and measurement · chapter 4 of 19
Validation and experiment
A metric is easy to compute and hard to trust. A random split inflates quality through features you never suspected; offline and online diverge systematically; an A/B needs more traffic than it seems, and the habit of glancing at the results turns five percent of false positives into twenty-one. All of this can be computed in advance — and here it is computed.
- Split by time only. With a random split the target leaks through any feature that proxies popularity: the counter already contains future clicks.
- The popularity baseline is mandatory. Not as a rival but as a measuring instrument: it shows how much of the metric is available with no personalisation at all.
- Half the effect — four times the data. Verified numerically: the ratio is 3.98. Which is why «let us measure +0.1%» usually means «we do not have the traffic».
- Peeking breaks the significance level. One check at the end gives an honest 5.0%; a daily check over two weeks gives 21.5% false positives on an A/A test.
1. The split
Why the only honest split is by time
The argument is simple and always applies. Any real dataset has features that proxy popularity one way or another: an impression counter, the number of purchases, an average rating, «how many times it was favourited».
With a random split, part of the interactions of the same item lands in train and part in test. But the feature «number of purchases» is computed over all the data, including the test. So it already contains information about what will happen in the test.
The model uses that — entirely legitimately from the point of view of optimisation: the feature is informative. It is just that in production no such feature exists, because the future has not happened yet.
The leak is all the more insidious because it does not look like one: you did not mix the target into the features by hand, you merely computed an aggregate over the whole dataset. And the metrics will not reveal it — they look excellent precisely because of it.
The right way: pick a global cutoff in time \(T\), train on everything before it, evaluate on everything after. All aggregates and counters are also computed only from data before \(T\) — a discipline of its own that is easy to forget.
The scheme «take each user's last interaction into the test» is popular in academic papers and looks like a split by time. But every user has their own cutoff, which means one user's test moment lies in another user's past — and the aggregates leak again.
This is one of the known methodological problems of comparing recommender models in the literature: part of the published improvements does not reproduce, and the protocol rather than the models is the reason.
The decision about cold entities has to be made explicitly
After a split by time the test will inevitably contain users and items that were not in training. What to do with them is a decision, not an implementation detail, and the number depends on it heavily.
| Option | What it measures | What is wrong with it |
|---|---|---|
| drop cold entities from the test | quality on the familiar audience | inflates: in production cold entities are always there, and their share grows |
| keep them and count them as misses | honest end-to-end quality | mixes two different problems — model quality and cold-start coverage |
| compute two metrics separately | both quantities on their own | more expensive, but the only option you can make decisions from |
The third option is the recommended one: one metric on warm entities, another on cold, and the share of cold as a separate number. Otherwise an improvement in the model and a deterioration in coverage merge into one movement.
2. The baseline you cannot do without
Before comparing models with each other you have to compare them with the absence of a model. The baseline is the popular top over a window.
The popularity baseline answers the question of how much of the metric is collected with no personalisation at all. The answer is usually unpleasantly high — which is exactly why it has to be known.
- If your model beats popularity by a couple of percent, the question «is it worth its complexity» becomes a substantive one.
- If it does not beat it at all — the protocol is almost certainly wrong, not the model.
- If it beats it suspiciously well — look for a leak. Too good a result on the first run is a symptom, not a success.
And the same technique applies to any new stage: before adding a source of candidates, look at what «take more from the existing one» gives you.
3. Why offline and online diverge
The gap between an offline metric and the result of an A/B is not an anomaly but the norm, and its causes are systematic.
| Cause | Mechanism |
|---|---|
| Offline is measured on someone else's policy | the held-out set was collected by the previous system. The new model proposes something different, and how people will react to that is not written in the logs |
| The metric is a proxy | NDCG grows, retention does not move: they are related but not the same thing |
| Positional bias in the labels | offline a model is praised for reproducing the previous order, biases included |
| Product effects outside the model | response speed, layout, the share of ads — offline they do not exist at all |
| Novelty and habituation | a spike in the first days on novelty and a rollback afterwards; offline does not see time |
The practical conclusion is not «offline is useless» but «offline is a filter». It cheaply screens out what is obviously bad; the decision is made by an experiment. A simple habit keeps the link between them: write down the offline prediction before the launch and check it afterwards. After a dozen experiments you will know the conversion factor of your own system, and that is worth more than any single metric.
4. How much traffic is needed
The central piece of arithmetic before an experiment: is there enough data to distinguish the effect you hope for at all.
For a proportion (conversion, CTR) at a significance level of 5% and power of 80%:
$$ n \;=\; \frac{\Bigl(z_{\alpha/2}\sqrt{2\bar p(1-\bar p)} \;+\; z_{\beta}\sqrt{p_0(1-p_0) + p_1(1-p_1)}\Bigr)^2}{(p_1 - p_0)^2} $$where \(p_0\) is the base conversion, \(p_1\) the expected one and \(\bar p\) their average. What matters here is not memorising the formula but seeing the denominator: \(n\) is inversely proportional to the square of the absolute difference.
| Base CTR | a 1% lift | a 2% lift | a 5% lift |
|---|---|---|---|
| 1% | 15.6 M | 3.9 M | 637 K |
| 5% | 3.0 M | 753 K | 122 K |
| 20% | 630 K | 158 K | 26 K |
Per arm; \(\alpha = 5\%\), power 80%. The numbers are reproduced by the script _tools/abtest_demo.py.
Two observations worth taking away:
- The fourfold law. At a base of 5% a 2% lift needs 753 thousand and a 1% lift needs 3.0 M. The ratio is 3.98: half the effect costs four times the data. That is the answer to the proposal to «measure plus one tenth of a percent».
- The rarer the event, the more expensive it is. At the same relative lift, dropping the base conversion from 20% to 1% multiplies the required sample by 25, because the absolute difference \(p_0 \cdot \text{lift}\) became smaller.
Usually the traffic is given, and it is more useful to ask what the minimum effect you can distinguish at all is. At a base CTR of 5%:
| Observations per arm | Detectable lift from |
|---|---|
| 100 K | 5.53% |
| 1 M | 1.73% |
| 10 M | 0.55% |
The MDE has to be computed before the launch. An experiment whose MDE is above the expected effect will not «show the absence of an effect» — it will show nothing at all, and that will cost two weeks.
Peeking
The temptation is understandable: the test is running, the dashboard refreshes, and you want to see whether it has gone green yet. The problem is that a significance level of 5% applies to one check.
A simulation in which both arms are built identically, so the true effect is exactly zero. We count the share of experiments in which the test statistic crossed the threshold at least once:
| How we look | False positives |
|---|---|
| once at the end | 5.0% |
| daily, one week | 16.5% |
| daily, two weeks | 21.5% |
20 000 repetitions per configuration; _tools/abtest_demo.py.
Every new glance is another chance to cross the threshold by accident. A nominal 5% turns into 21%, that is, every fifth «successful» experiment is in fact empty.
- Fix the horizon in advance and look once at the end. Boring and it works.
- Sequential tests (SPRT, always-valid inference) — if you really do need to look as it runs. They are built so that peeking is permitted by construction, at the price of a slightly larger sample.
- Alpha spending — a significance budget laid out in advance across pre-appointed checkpoints.
Separately: stopping «as soon as it goes green» is peeking in its most harmful form, because it systematically inflates the estimate of the effect. You stop at a random maximum.
- The sample-size tab: the curve falls as \(1/\sqrt{n}\). To distinguish half the effect you need four times the data.
- Reduce the base conversion at the same relative lift — the required sample grows, because the absolute difference became smaller.
- The peeking tab is an A/A test. Increase the number of checks and watch the share of false positives move from 5% to over 20%.
What to say in an interview: «We fix the horizon in advance and compute the MDE before the start. If we must look as it runs — a sequential test or alpha spending, otherwise a nominal 5% turns into over 20%».
5. When randomising by user breaks
A classic A/B assumes isolation: what happens in the treatment does not affect the control. In recommender systems that assumption is violated regularly.
| Where it breaks | Mechanism | What fixes it |
|---|---|---|
| Two-sided markets ride-hailing, marketplace, delivery |
the treatment arm buys up a limited resource — couriers, stock in the warehouse — and the control gets less of it | randomisation by city, dark store or time slot; switchback tests |
| Social graphs | a user in the treatment posts and likes more actively, and their friends in the control see it | cluster randomisation over graph components |
| Shared models | the system retrains on the logs of both arms, and the treatment policy mixes its data into training — the control stops being a control | separate training on control data, or a holdout excluded from training |
The third case is particularly nasty because it is invisible: the A/B infrastructure works normally, the assignment is correct, and the leak goes through retraining. It is checked by the question «whose logs is the model that the control sees trained on».
6. The discipline of an experiment
If the list of metrics is wide, you will always find something that went green. With twenty metrics at a level of 5%, one false positive is expected purely by construction.
So before the start these are fixed: the specific hypothesis being tested, one primary decision metric, the horizon and the stopping rule. Everything else is a supporting slice for understanding, not grounds for a rollout.
For every growth metric it is useful to keep a metric that will get worse if the growth was achieved the wrong way.
- CTR grows — look at scroll depth and the share of quick returns;
- time in the app grows — look at retention a week later;
- GMV grows — look at returns and at the share of impressions going to the top 1% of the catalogue;
- any metric grows — look at coverage: was the growth bought by collapsing into the popular.
This is the same technique as in the chapter on the loop: fast metrics move things, slow and antagonistic ones guard them.
Interview questions
Why can't you use a random split?
Because the data almost certainly contains features that proxy popularity: counters of impressions and purchases, an average rating. With a random split they are computed over the whole dataset, test included, and already contain future clicks. The model uses that, the metrics look excellent, and in production no such feature exists.
The right way is a global cutoff in time, and all aggregates and counters are computed only from data before it, otherwise a split by time stops being one.
Leave-one-out looks like a split by time, but every user has their own cutoff, so the aggregates leak again. This is a known methodological problem of academic comparisons.
What do you do with cold users and items in the test set?
Make the decision explicitly, because the number depends on it heavily. Dropping cold entities inflates the quality: in production they are always there. Keeping them and counting them as misses is honest, but it mixes model quality with cold-start coverage.
The working option is to compute two metrics separately, on warm and on cold, plus the share of cold as a number in its own right. Otherwise an improvement in the model and a deterioration in coverage merge into a single movement of the metric.
Why do you need a popularity baseline if it is obviously worse?
It is not a rival but a measuring instrument: it shows how much of the metric is collected with no personalisation at all. The answer is usually unpleasantly high.
After that it reads as a diagnosis. If the model beats popularity by a couple of percent, the question of its complexity becomes substantive. If it does not beat it at all, the protocol is almost certainly wrong. If it beats it suspiciously well, look for a leak: too good a result on the first run is a symptom, not a success.
How much traffic is needed to catch a 1% lift?
It depends on the base conversion, and the dependence is quadratic in the absolute difference. At a base CTR of 5% with the standard 5% significance and 80% power — about 3 M observations per arm. At a CTR of 1% — 15.6 M. At a CTR of 20% — 630 thousand.
The key relation: half the effect requires four times the data (verified numerically: 753 K against 3.0 M, a ratio of 3.98). Which is why the proposal to «measure +0.1%» usually means «we do not have the traffic», and that has to be computed before the launch, not after.
Why can't you peek at A/B results as they come in?
A significance level of 5% applies to one check. Every next one is another chance to cross the threshold by accident.
On an A/A test, where the true effect is exactly zero: one check at the end gives an honest 5.0% of false positives, a daily check over a week gives 16.5%, over two weeks 21.5%. That is, every fifth «successful» experiment is empty.
What to do: fix the horizon in advance; or take a sequential test or alpha spending, where peeking is permitted by construction. And separately — stopping «as soon as it goes green» systematically inflates the estimate of the effect, because you stop at a random maximum.
When does randomising by user give a biased estimate?
When the isolation of the arms is violated. Three typical cases.
Two-sided markets: the treatment arm buys up a limited resource — couriers, stock — and the control gets less. Treated by randomising over cities, dark stores or time slots, and by switchback tests.
Social graphs: the activity of the treatment is seen by friends in the control. Treated by cluster randomisation over graph components.
Shared models: the system retrains on the logs of both arms, and the treatment policy mixes its data into training — the control stops being a control. The least visible case, because the infrastructure works normally throughout.
One-screen cheat sheet
The split
By time only, one global cutoff. Aggregates too are to the left of it. Leave-one-out leaks.
Cold entities in the test
Two metrics separately plus the share of cold. Otherwise quality and coverage merge.
The baseline
Popularity is an instrument, not a rival. Too good a result is a symptom of a leak.
Sample size
At a CTR of 5%: +5% → 122 K, +2% → 753 K, +1% → 3.0 M. Half the effect — four times the data.
Peeking
1 check — 5.0%, daily over two weeks — 21.5%. Fix the horizon in advance.
Network effects
A resource, a graph, a shared model. Treatment: cluster randomisation, switchback, a holdout outside training.
Primary sources
- R. Kohavi, D. Tang, Y. Xu. Trustworthy Online Controlled Experiments, Cambridge University Press 2020 — the practical foundation for A/B: power, peeking, the typical mistakes.
- R. Johari, P. Koomen et al. Always Valid Inference: Continuous Monitoring of A/B Tests — the tests under which looking as it runs is allowed.
- M. Saveski, J. Pouget-Abadie et al. Detecting Network Effects: Randomizing Over Randomized Experiments, KDD 2017 — how a violation of isolation is caught.
- M. F. Dacrema, P. Cremonesi, D. Jannach. Are We Really Making Much Progress?, RecSys 2019 — on reproducibility and the role of the comparison protocol.
- The numbers in this chapter:
_tools/abtest_demo.pyin this repository.