Part I · Statement and measurement · chapter 2 of 19

Data and biases

A recommender system learns not from preferences but from logs — that is, from traces of behaviour filtered by a previous version of itself. Four problems follow from this, each of which opens its own line in the textbook, along with a whole catalogue of biases you will be asked about at any interview. The place to start is one formula.

What to take away

We are not predicting what we think we are. Not $P(\text{click})$ but $P(\text{click} \mid \text{show}, \pi_{\text{log}})$ — everything is conditioned on what the previous version of the system showed.
A missing entry is not a zero, it is an unknown. The absence of a click on an item that was never shown carries no information whatsoever.
The ranker's pool is built from slates. Negatives are what was shown alongside and not chosen, not random items from the catalogue.
The loop closes fast. In a simulation without exploration, catalogue coverage stops at 0.6% after the very first round and never moves again.

1. What it is that we observe

Explicit feedback and why almost nobody trains on it

Explicit feedback is when a person states an opinion outright: stars, a like, a thumbs down. It looks like the ideal signal, and in practice it is rarely used — the reasons are worth knowing.

Problem with explicit feedback	What it consists of
There is almost none of it	a few percent of users leave a rating; on that volume nothing can be trained except the head of the catalogue
It is biased in composition	people rate what provoked an emotion: delight or irritation. The middle, which is the bulk of consumption, is not rated at all
It is biased in people	ratings come from a particular subsample — the active and the motivated. Their tastes do not represent the audience
It disagrees with behaviour	the classic «gave five stars to a documentary and watches comedies». People rate the person they want to be, and behave otherwise
The scale drifts	for one person a four is praise, for another it is disappointment. This is repaired by subtracting a baseline, but the residual noise is large

So in production people train on implicit feedback: a purchase, an add-to-cart, a click, a function of watch time. There is a lot of it, it costs the user no effort, and it is closer to what a person actually does. The price is four problems, and the rest of this chapter is made of them.

Four problems of implicit data

Problem 1. A proxy can be wrong twice over

We proxy relevance with an observable event, and there are two independent levels at which we can miss.

The business goal was proxied badly. We took the click instead of satisfaction — we got clickbait. This is repairable by engineering: change the target, add heads, reweight.
The business goal itself is a poor one. We maximise watch time where value is what is needed. This is not repairable by engineering at all — only at the level of a product decision.

Separating the two levels is useful in an interview: the first question is «is that the right target», the second is «is that the right goal».

Problem 2. The logging policy — the central formula of the chapter

The logs contain data only about the user's interaction with items from the system itself, and only with those that were shown. So we are not training on what we thought we were:

$$ \text{we want } P(\text{click}) \qquad\text{but we predict}\qquad P(\text{click} \mid \text{show},\, \pi_{\text{log}}) $$

where $\pi_{\text{log}}$ is the logging policy: the previous version of the model plus every business constraint, filter and rule that stood in production while the data was collected.

Everything you train is conditioned on what the previous system showed. This is not an interference you can reduce by being careful — it is a structural property of the data, and all further work with biases comes down to either accounting for that condition or changing $\pi_{\text{log}}$.

Problem 3. The loop closes

Being conditioned on the current policy is called exposure bias. If nothing is done, a cycle appears:

the model affects impressions → impressions affect the data → the data affects the next model

Every turn narrows what the system is capable of seeing at all. Below we compute how fast.

Problem 4. Data arrives in slates

A user sees not one item but a whole output — a slate — and sometimes only part of it. Strictly speaking we predict $P(\text{click} \mid \text{show}, \text{slate})$: the probability of a click depends on what stood next to it.

The consequence people skip most often. If the user never saw an item, that item cannot be a good negative for ranking. The absence of a click on something never shown carries no information about preferences — it is an unknown, not a refusal.

Hence the rule for building the training pool

Everything on the left is data. Everything on the right is the unknown, not zeros.

The practical translation

The pool for training the ranker is built from slates. Positives are the things there was a positive interaction with. Negatives are the things shown in the same slate and not chosen.

Random items from the catalogue are legitimate negatives for candidate generation and two-tower models, because those really are applied to the whole catalogue. For the final ranker they are not: it always works on the hundred candidates that reached it, and training it to tell them from random noise means spending capacity on a task that does not exist.

The same argument from the other side is worked through in the chapter on sampling negatives: there you can see what happens when negatives are taken from the wrong place.

2. The catalogue of biases

There are many biases, and at an interview people usually want to hear that you distinguish them by mechanism rather than just listing them. The four main ones:

Bias	Mechanism	What is done about it
Selection bias	the user decides what to rate; the extremes get rated	move away from explicit feedback to implicit — there the choice is made by the system, not the user
Positional bias	a click is likelier on the top positions regardless of content	partly softened by top-heavy metrics; treated with position as a feature, a separate bias tower, or IPS weighting
Exposure bias	the logs contain only what was shown; what was not shown is absent as a class	change the logging policy: exploration, a random quota, randomising adjacent positions
Popularity bias	popular items appear in the data orders of magnitude more often; there is nothing to learn the tail from	loss corrections, embedding normalisation, diversity in re-ranking, exploration

How to tell them apart in one phrase

Selection — the user chose. Exposure — the system chose. Positional — the position spoiled it. Popularity — the volume of data spoiled it.

And the harm they do is of different natures: the first three spoil estimation (we measure what exists incorrectly), while popularity spoils learning (there is nothing for the model to learn the tail from).

Positional bias: a problem of the data, not of the model

People click more often on what stands higher — regardless of content. The key observation: the target itself is spoiled. No model trained on such a target will rid itself of the bias, because the bias is already inside the labels.

Hence three levels of working with it, in order of increasing honesty and cost:

Soften it by choice of metric. Top-heavy metrics — NDCG, MRR — value the top of the output more, and that partly compensates for the top of the output being more influential in the data too. Cheap, but it does not remove the bias.
Model the position explicitly. Position is fed to the model as a feature, and at inference a constant one is substituted. Simple and it works, but the model can «hide» part of relevance inside the position feature.
Estimate and separate. A separate bias tower trained on position and context, or IPS weighting with an estimate of the probability of examination. The most honest and the most expensive: it needs random traffic to estimate the propensity at all.

The mechanics of the third level are worked through where IPS is introduced — the inverse propensity widget on the trainer page shows what the estimate turns into when the probabilities of being shown are small.

Popularity built into the geometry

A separate case that interviewers like, because it is not about the data but about the problem statement.

MIPS is not nearest-neighbour search

Once the embeddings are trained, a recommendation is a search for the maximum inner product:

$$ \operatorname{top-}k(u) \;=\; \operatorname*{arg\,max}_{i \in I}{}^{(k)}\; p_u^{\top} q_i $$

With cosine similarity the length $\lVert q_i\rVert$ cancels out; with an inner product it does not. So MIPS systematically prefers items with a large norm, and the norm grows with popularity during training: a popular item takes part in more gradient steps.

The result is a popularity bias built neither into the data nor into the loss, but arising from the choice of similarity measure. The remedies: normalise the embeddings and switch to cosine; reduce MIPS to nearest-neighbour search by adding a coordinate; or correct for popularity in the loss itself.

3. Closing the loop: how fast

«The model affects the data, the data affects the model» sounds like a distant threat. Let us count.

Simulation: a catalogue of 2000, ten slots, twelve rounds

The model scores items by a smoothed CTR over the collected log, shows the top 10, collects clicks and retrains. Examination of a position decays as $0.85^{\,p}$. No exploration at all.

Round	without exploration		with a 5% random quota
	coverage	share of the ideal	coverage	share of the ideal
1	0.50%	0.37	2.85%	0.43
3	0.60%	0.37	7.20%	0.48
6	0.60%	0.37	13.65%	0.53
12	0.60%	0.37	24.75%	0.70

The numbers are reproduced by the script _tools/feedback.py in this repository.

What matters here

The loop closes on the first round. Coverage of 0.6% is twelve items out of two thousand. Twelve further rounds change nothing: the system shows what it has data about, and gets data only about what it shows.
Quality freezes at 0.37 of the ideal and does not grow. The model has not «under-trained» — it trained perfectly on what it saw. The problem is not the model.
Five percent of random quota is enough to break the circle. Coverage grows to 24.8%, quality to 0.70. Not for free: those 5% of impressions are knowingly worse, and that is exactly how exploration is paid for.

Hence a practical conclusion worth saying out loud: without exploration the offline metrics will grow while the system degrades, and the metrics themselves will not show it — they are computed on data generated by that same policy.

How this is caught in production

Coverage and Gini over impressions across time. If coverage creeps down from release to release, the loop is narrowing. The metric is cheap and almost nobody looks at it.
The share of impressions going to the top 1% of the catalogue. If it grows, the system is collapsing into the popular.
The age of the content shown. If the median age grows, new material has stopped getting through.
Random traffic as a measuring instrument. Even a fraction of a percent of fully random impressions gives an unbiased sample on which you can honestly assess how bad things are.

Interview questions

Why does production train on implicit feedback rather than ratings?

There is little explicit feedback (a few percent of users), it is biased in composition (people rate the extremes, not the middle that makes up the bulk of consumption), biased in people (ratings come from a particular active subsample), and it disagrees with behaviour — a person rates the self they would like to be.

Implicit feedback — purchases, clicks, watch time — is abundant, free for the user and closer to real behaviour. The price: four problems, starting with the fact that all of it is conditioned on the previous impression policy.

What does a model trained on logs actually predict?

Not $P(\text{click})$ but $P(\text{click} \mid \text{show}, \pi_{\text{log}})$, where $\pi_{\text{log}}$ is the logging policy: the previous model plus every filter and business rule that stood in production.

This is a structural property, not an error term: the logs physically contain no data about what the system did not show. All further work with biases comes down to two options — either account for that condition in the estimate (IPS and its relatives) or change $\pi_{\text{log}}$ itself through exploration.

Why can't random catalogue items be used as negatives for the ranker?

Because the ranker is never applied to random items. It works on the hundred candidates the first stage selected, and telling them apart from random noise is a task that does not exist in production.

The rule: the ranker's pool is built from slates. Positives are what was interacted with; negatives are what was shown alongside and not chosen. Random catalogue items are legitimate negatives for candidate generation and two-tower models, because those are precisely what is applied to the whole catalogue.

And separately: the absence of a click on an item that was never shown is an unknown, not a refusal. Putting a zero there means inventing data.

List the main biases and how they differ by mechanism.

Selection — the user chose: they decide what to rate, and they rate the extremes. Exposure — the system chose: the logs hold only what was shown. Positional — the position spoiled it: a click is likelier at the top regardless of content. Popularity — the volume spoiled it: the popular has orders of magnitude more data.

A useful distinction: the first three spoil estimation (we measure what exists incorrectly), popularity spoils learning (there is nothing to learn the tail from).

A separate case is popularity built into the geometry: with an inner product the norm does not cancel, and in training it grows with popularity, so MIPS systematically prefers the popular. Treated by normalisation or by a correction in the loss.

Why can't positional bias be cured by the model?

Because it is already in the target. The click was collected under the influence of position, and any model trained on those labels will reproduce the bias — for the model it is part of the signal.

Three levels of working with it, in order of cost: soften it with top-heavy metrics (cheap, but it does not remove it); feed position as a feature and substitute a constant one at inference (simple, but the model can hide relevance inside the position feature); estimate the probability of examination and separate the effects — a bias tower or IPS weighting, which is the most honest but needs random traffic to estimate the propensity.

How fast does the feedback loop close, and how do you notice it?

Faster than people expect. In a simulation with a catalogue of 2000 items and ten slots, coverage stops at 0.6% — twelve items — after the very first round, and twelve subsequent rounds do not move it. Quality freezes at 0.37 of what is achievable.

And the model is not broken while this happens: it is excellently trained on what it saw. A random quota of 5% lifts coverage to 24.8% and quality to 0.70 — at the price of those 5% of impressions being knowingly worse.

It is caught with cheap metrics over time: coverage and Gini over impressions, the share of impressions going to the top 1% of the catalogue, the median age of what is shown. Plus a fraction of a percent of fully random traffic — an unbiased sample on which you can honestly measure how bad it is.

One-screen cheat sheet

The central formula

We train not $P(\text{click})$ but $P(\text{click} \mid \text{show}, \pi_{\text{log}})$. Everything is conditioned on the previous policy.

Missing ≠ zero

What was not shown is unknown. A zero must not go there.

The ranker's pool

From slates: a negative is what was shown alongside and not chosen. Random catalogue items are negatives for candidate generation, not for the ranker.

Four biases

Selection — the user chose. Exposure — the system chose. Positional — the position spoiled it. Popularity — the volume spoiled it.

Position

The target is spoiled, not the model. Treatment: metrics → position as a feature → a bias tower or IPS.

The loop

Without exploration, coverage of 0.6% from the first round and quality of 0.37 forever. A 5% quota → 24.8% and 0.70.

Primary sources

Y. Hu, Y. Koren, C. Volinsky. Collaborative Filtering for Implicit Feedback Datasets, ICDM 2008 — the «confidence instead of a rating» formulation that all work with implicit feedback grew out of.
T. Joachims, L. Granka et al. Accurately Interpreting Clickthrough Data as Implicit Feedback, SIGIR 2005 — the swapped-results experiments that measured positional bias for the first time.
T. Schnabel, A. Swaminathan et al. Recommendations as Treatments: Debiasing Learning and Evaluation, ICML 2016 — IPS for recommendations.
H. Steck. Training and Testing of Recommender Systems on Data Missing Not at Random, KDD 2010 — why missing entries cannot be treated as missing at random.
The numbers in this chapter: _tools/feedback.py in this repository.