Interview questions

Demand follows a power law: the frequency of the \(k\)-th most popular object is proportional to \(k^{-\alpha}\). On logarithmic axes that is a straight line with slope \(-\alpha\), and that is how it is estimated.

One picture is not enough, you have to name the quantity. About the data — \(\alpha\), or Gini over frequencies. About the output — coverage@k (the share of the catalogue shown at least once), Gini over impressions and the entropy of impressions. The difference matters: a system can deepen the skew or smooth it.

Orders of magnitude: at \(\alpha = 1.0\) the top percent of the catalogue collects about 62% of interactions, at \(\alpha = 1.2\) — already 85%.

It is a direct consequence of the arithmetic. At \(\alpha = 1.2\), a catalogue of 100 thousand and ten million interactions, 74% of items are expected to have fewer than ten interactions. At \(\alpha = 1.0\) — 17%.

That is, most of the catalogue is always in a state where there is practically no collaborative signal, and that is not the problem of a young service but a property of the demand distribution. Hence the need for content features: they let you say something about an item before it has accumulated a history.

Subscription pays for retention: what matters is the probability of renewal, not views. The metric is slow, the feedback cycle is a month.

Advertising pays for attention: impressions, time, returns. A conflict is built in here — what holds attention is not obliged to be useful.

A marketplace pays a commission on turnover, but with two corrections: returns and the health of supply. A seller needs demand, or they leave, and the platform is left without an assortment.

The practical conclusion: borrowing someone else's metric is the most expensive mistake in stating the problem. And in a two-sided market this is a constrained problem, which in practice is reduced to a weighted sum with weights from experiments.

A proxy and the real goal have a shared part and a divergence. Under light pressure they run side by side; when you start pressing, the model finds exactly the divergence — that is the cheapest place to pick the metric up.

Hence clickbait when optimising clicks, the infinite scroll when optimising time, returns when optimising immediate GMV.

The cure is not to give up proxies but to pair them: fast proxies move the system, slow health metrics (cohort retention, the share active on a horizon of months, complaints, returns) guard it. The retention curve is read by its shape: a healthy product flattens out at a non-zero level.

The budget does not add up. With a catalogue of a million and a score costing 100 µs, scoring everything is 100 seconds against a budget of 100 ms — a thousand times over. A gap like that is not closed by constants.

So: narrow cheaply (candidate generation, hundreds out of millions, usually a dot product in an index), then order expensively (ranking on hundreds). Sometimes pre-ranking is added in between and re-ranking on top of the head.

Latency here is not an abstraction: slowing down costs measurable money — the well-known observation about 100 ms and a percent of sales goes back to internal experiments at Amazon that Greg Linden described in 2006.

Recall. The job of the first stage is not to guess the answer but not to lose it: what it did not select, ranking will never show.

It follows that end-to-end quality is bounded above by the recall of the first stage. Even perfect ranking will not lift the system above that ceiling, so the stage to improve is the weakest one, not the most visible one.

Precision at this stage is not merely useless but harmful as a target: it pushes you to select fewer and safer items, that is, straight into the popular ones, collapsing coverage.

There is little explicit feedback (a few percent of users), it is biased in composition (people rate the extremes, not the middle that makes up the bulk of consumption), biased in people (ratings come from a particular active subsample), and it disagrees with behaviour — a person rates the self they would like to be.

Implicit feedback — purchases, clicks, watch time — is abundant, free for the user and closer to real behaviour. The price: four problems, starting with the fact that all of it is conditioned on the previous impression policy.

Not \(P(\text{click})\) but \(P(\text{click} \mid \text{show}, \pi_{\text{log}})\), where \(\pi_{\text{log}}\) is the logging policy: the previous model plus every filter and business rule that stood in production.

This is a structural property, not an error term: the logs physically contain no data about what the system did not show. All further work with biases comes down to two options — either account for that condition in the estimate (IPS and its relatives) or change \(\pi_{\text{log}}\) itself through exploration.

Because the ranker is never applied to random items. It works on the hundred candidates the first stage selected, and telling them apart from random noise is a task that does not exist in production.

The rule: the ranker's pool is built from slates. Positives are what was interacted with; negatives are what was shown alongside and not chosen. Random catalogue items are legitimate negatives for candidate generation and two-tower models, because those are precisely what is applied to the whole catalogue.

And separately: the absence of a click on an item that was never shown is an unknown, not a refusal. Putting a zero there means inventing data.

Selection — the user chose: they decide what to rate, and they rate the extremes. Exposure — the system chose: the logs hold only what was shown. Positional — the position spoiled it: a click is likelier at the top regardless of content. Popularity — the volume spoiled it: the popular has orders of magnitude more data.

A useful distinction: the first three spoil estimation (we measure what exists incorrectly), popularity spoils learning (there is nothing to learn the tail from).

A separate case is popularity built into the geometry: with an inner product the norm does not cancel, and in training it grows with popularity, so MIPS systematically prefers the popular. Treated by normalisation or by a correction in the loss.

Because it is already in the target. The click was collected under the influence of position, and any model trained on those labels will reproduce the bias — for the model it is part of the signal.

Three levels of working with it, in order of cost: soften it with top-heavy metrics (cheap, but it does not remove it); feed position as a feature and substitute a constant one at inference (simple, but the model can hide relevance inside the position feature); estimate the probability of examination and separate the effects — a bias tower or IPS weighting, which is the most honest but needs random traffic to estimate the propensity.

Faster than people expect. In a simulation with a catalogue of 2000 items and ten slots, coverage stops at 0.6% — twelve items — after the very first round, and twelve subsequent rounds do not move it. Quality freezes at 0.37 of what is achievable.

And the model is not broken while this happens: it is excellently trained on what it saw. A random quota of 5% lifts coverage to 24.8% and quality to 0.70 — at the price of those 5% of impressions being knowingly worse.

It is caught with cheap metrics over time: coverage and Gini over impressions, the share of impressions going to the top 1% of the catalogue, the median age of what is shown. Plus a fraction of a percent of fully random traffic — an unbiased sample on which you can honestly measure how bad it is.

Because the jobs differ. The first stage must not lose: what it did not select, the second will never show, and the mistake is unrecoverable. The second must arrange: its mistake is recoverable by re-ranking.

Measuring a candidate generator by precision is harmful: precision pushes you to select fewer and safer items, that is, into the popular, collapsing coverage. And end-to-end quality is bounded above by the recall of the first stage anyway.

Because K is already an engineering decision about output size, and sources cost different amounts: a static top from memory is nearly free, an ANN index costs milliseconds, a neural source noticeably more.

What has to be compared is recall relative to the latency budget: build a Pareto front of «recall against time» and take the point that fits your budget. A source with lower recall but three times cheaper often wins — the time saved adds another source.

Separately, the denominator is worth clarifying: do we count relative to the positives of the dataset (cheap, but inflated — those positives were collected by the previous policy) or jointly with ranking (more honest, but unstable over time).

Precision@k does not see order at all: three arrangements of the same slate with hits at the top, in the middle and at the bottom give an identical 0.300, whereas NDCG gives 1.0000, 0.5508 and 0.4250.

MAP does take order into account, but it is binary by nature: it cannot express graded relevance. NDCG handles a graded gain (including the exponential \(2^{rel}-1\), or the price of a positive directly), an explicit discount by position, and normalisation by the ideal order, which makes queries with different numbers of positives comparable.

A caveat: the logarithmic discount is a modelling assumption about attention, not a measured quantity.

Because of the discount. Swapping adjacent positions 1 and 2 changes NDCG by 0.3691; the same swap at positions 9 and 10 changes it by 0.0120. A difference of 31 times.

It follows that a pairwise loss treating all inversions alike (RankNet) optimises something other than what we measure. The correction is to multiply the gradient of a pair by how much swapping it would move the metric; that is how LambdaRank works.

First, AUC counts all pairs equally: a swap at the first position costs as much as one at the two-hundredth. For an output where the first ten are visible, that is precisely the blindness NDCG is needed for.

Second, AUC is invariant to any monotone transformation of the score and therefore says nothing about calibration. A model can systematically double its probabilities while having an excellent AUC.

It is critical where the score enters arithmetic: an ad auction with bid × CTR, budget allocation, threshold rules. There you look at LogLoss, Brier, ECE and the calibration curve. Calibration is repaired after training and does not spoil AUC.

Over queries (micro) — this measures the average query, and the number is almost entirely determined by active users, who have tens of times more queries. Over users (macro) — this measures the average user, all voices equal.

The practical consequence: a release that kills onboarding looks neutral in micro — newcomers may be half the audience by headcount and a few percent by queries. Macro will sag noticeably.

The answer «I look at both and cut by cohort separately» is the right one: the divergence between micro and macro is itself a diagnosis.

Because the data almost certainly contains features that proxy popularity: counters of impressions and purchases, an average rating. With a random split they are computed over the whole dataset, test included, and already contain future clicks. The model uses that, the metrics look excellent, and in production no such feature exists.

The right way is a global cutoff in time, and all aggregates and counters are computed only from data before it, otherwise a split by time stops being one.

Leave-one-out looks like a split by time, but every user has their own cutoff, so the aggregates leak again. This is a known methodological problem of academic comparisons.

Make the decision explicitly, because the number depends on it heavily. Dropping cold entities inflates the quality: in production they are always there. Keeping them and counting them as misses is honest, but it mixes model quality with cold-start coverage.

The working option is to compute two metrics separately, on warm and on cold, plus the share of cold as a number in its own right. Otherwise an improvement in the model and a deterioration in coverage merge into a single movement of the metric.

It is not a rival but a measuring instrument: it shows how much of the metric is collected with no personalisation at all. The answer is usually unpleasantly high.

After that it reads as a diagnosis. If the model beats popularity by a couple of percent, the question of its complexity becomes substantive. If it does not beat it at all, the protocol is almost certainly wrong. If it beats it suspiciously well, look for a leak: too good a result on the first run is a symptom, not a success.

It depends on the base conversion, and the dependence is quadratic in the absolute difference. At a base CTR of 5% with the standard 5% significance and 80% power — about 3 M observations per arm. At a CTR of 1% — 15.6 M. At a CTR of 20% — 630 thousand.

The key relation: half the effect requires four times the data (verified numerically: 753 K against 3.0 M, a ratio of 3.98). Which is why the proposal to «measure +0.1%» usually means «we do not have the traffic», and that has to be computed before the launch, not after.

A significance level of 5% applies to one check. Every next one is another chance to cross the threshold by accident.

On an A/A test, where the true effect is exactly zero: one check at the end gives an honest 5.0% of false positives, a daily check over a week gives 16.5%, over two weeks 21.5%. That is, every fifth «successful» experiment is empty.

What to do: fix the horizon in advance; or take a sequential test or alpha spending, where peeking is permitted by construction. And separately — stopping «as soon as it goes green» systematically inflates the estimate of the effect, because you stop at a random maximum.

When the isolation of the arms is violated. Three typical cases.

Two-sided markets: the treatment arm buys up a limited resource — couriers, stock — and the control gets less. Treated by randomising over cities, dark stores or time slots, and by switchback tests.

Social graphs: the activity of the treatment is seen by friends in the control. Treated by cluster randomisation over graph components.

Shared models: the system retrains on the logs of both arms, and the treatment policy mixes its data into training — the control stops being a control. The least visible case, because the infrastructure works normally throughout.

By the strength of the normalisation. All three have the form \(|A\cap B| / (|A||B|)^{\alpha}\): cosine \(\alpha = 0.5\) (a square root), Jaccard roughly linear, PMI \(\alpha = 1\) (full normalisation).

In practice: cosine is gentler towards the popular than Jaccard — in the example with a blockbuster of 10⁶ and a niche item of 10³ with an intersection of 900, cosine gives 0.028 and Jaccard 0.0009, a difference of thirty times.

PMI answers a different question: not «how large is the intersection» but «how many times larger is it than expected under independence». So two blockbusters with an intersection of 210 thousand against 200 thousand expected get almost zero from PMI, while two niche items with an intersection of 600 against six expected get 4.61.

It explodes on rare pairs. A pair built on fifteen observations gets a PMI of 5.93 — more than an honest pair on six hundred observations with its 4.61. In the limit: two items watched by exactly the same single person give \(\ln 10^6 = 13.8\), that is, pure noise takes first place.

The reason is that PMI measures a ratio to the expectation, and when the expectation is close to zero any single co-occurrence gives an enormous ratio.

The cure is NPMI (normalising by \(-\log P(A,B)\), which penalises rarity) plus, mandatorily, a threshold on the number of co-occurrences and shrinkage for small counts. NPMI alone is not enough: a pair of fifteen observations still gets 0.534.

No, and this is a common inaccuracy. It is checked directly: take three pairs with the same excess over expectation — a popular one, a middling one and a niche one. NPMI will give them different values, and the popular one will get more, because its denominator \(-\log P(A,B)\) is smaller.

So NPMI introduces a preference of its own, the reverse of what PMI does. In practice that is rather useful — noise is suppressed — but the phrase «cleaned of popularity» is wrong.

The scoring formula is the same for both: walk through the history and add up the weights. The difference is where the weights come from. A computed measure looks at a pair of items in isolation; a regression sees all the predictors together.

That repairs three things. Popularity stops pretending to be a link: an item that lies in every basket gets a near-zero weight, because it adds nothing beyond the rest. Duplicates stop being counted twice: the regression splits the weight between collinear predictors. And negative weights appear — substitutes, which a similarity formula cannot express in principle.

Formally \(B_{ij} = -P_{ij}/P_{jj}\) is the coefficient of a partial regression, that is, the move from marginal to conditional dependence.

EASE is SLIM with the \(L_1\) and the constraint \(B \ge 0\) removed. In exchange there is a closed-form solution: one matrix inversion instead of coordinate descent column by column.

In quality the gap is entirely explained by the sign constraint: EASE with its negative weights zeroed out falls exactly to the level of SLIM (0.402 against 0.401 on ML-20M). About 60% of the weights in EASE are negative — the model spends most of its capacity on «what you do not want».

The choice is by resources: EASE costs \(O(|I|^3)\) in time and \(O(|I|^2)\) in memory, but does not depend on the number of users. For tens of thousands of items, take EASE. If the catalogue is such that a dense matrix does not fit, take SLIM with its sparsity.

They are not alternatives: factorisation is a method inside collaborative filtering. CF splits into neighbour-based approaches (kNN, item-item — similarities computed straight from the matrix) and learned ones, which include factorisation as well as EASE and SLIM. And «learned» and «factorised» are different properties: EASE is trained but does not decompose its matrix.

The substantive difference is in the mechanism of generalisation. Neighbours generalise through direct co-occurrences, factorisation through compression into \(d\) coordinates. So it finds links where there are hardly any co-occurrences: in the example, items that co-occurred 12 times out of 20 000 get a cosine of 0.04 from neighbours and 0.98 from a rank-2 factorisation.

And the converse: for mutually exclusive items neighbours give 0 («no data»), factorisation −1 («opposites»). The second statement is unavailable to neighbours.

A real singular value decomposition requires a complete matrix. Ours is sparse, and filling the gaps with zeros is not allowed: a gap means «it was not shown», not «it was not liked».

What came to be called SVD after the Netflix Prize is Simon Funk's gradient descent over the observed cells, that is, minimising the error only on known pairs with regularisation. The name stuck; the method is a different one.

The problem is not convex in \((P,Q)\) jointly but is convex in each matrix separately. Fixing \(Q\), for each user we get an ordinary ridge regression with the solution \(p_u = (Q_u^\top Q_u + \lambda I)^{-1} Q_u^\top r_u\).

Hence three advantages: every \(p_u\) is computed independently and the task parallelises perfectly; there is no learning rate to tune; convergence takes a dozen iterations. And the system is of size \(d \times d\) — the size of the rank, not of the catalogue.

SGD wins when the matrix is very sparse and online updating is needed, and also when the loss is not quadratic — for BPR or a logistic loss you cannot derive ALS.

Rank is the number of coordinates in each vector, and it is a substantive hypothesis: «any taste is a mixture of \(d\) archetypes». A product of matrices of width \(d\) has rank at most \(d\), so the choice of \(d\) limits what the model is able to express at all.

The price is measurable: the number of parameters is \((|U| + |I|)\cdot d\). On a 30×40 matrix with 378 training observations, rank 6 gives 420 parameters — more than there is data, and the model will fit the training set while learning nothing.

It is chosen on a held-out set. The training error falls monotonically with rank and suggests nothing; only the held-out error has a minimum.

The first, the usual one: shrinkage towards zero. For a user with a single rating it pays to produce an enormous vector that hits it perfectly; the penalty on length makes that unprofitable, and the model says «I do not know» instead of inventing confidently.

The second, the one people forget: conditioning. The matrix \(Q_u^\top Q_u\) for a user with two or three interactions is singular — its rank is less than \(d\) and no inverse exists. Adding \(\lambda I\) makes it invertible always, so at \(\lambda = 0\) the solution for a rare user simply will not compute.

It separates the target from the confidence. The target is binary — «there is a preference» — while the weight \(c_{ui} = 1 + \alpha r_{ui}\) says how confident we are of that: listened a hundred times or once. We learn from all pairs rather than only the observed ones, and the unobserved enter with a weight of 1 — soft negatives that a single observation overturns.

The sum over all pairs does not explode because of the decomposition \(Q^\top C_u Q = Q^\top Q + Q^\top(C_u - I)Q\): the first term does not depend on the user and is computed once per iteration, the second is a sum only over the observed items, of which there are few.

And \(\alpha\) is a genuine hyperparameter, not one: sensible values are of the order of tens.

Fold-in. The item vectors are already trained and change slowly, so it is enough to solve exactly the ALS step responsible for users: \(p_{\text{new}} = (Q_s^\top Q_s + \lambda I)^{-1} Q_s^\top r_s\), where \(Q_s\) are the vectors of what the person interacted with.

This is a \(d \times d\) system — at rank 8, microseconds, and no retraining. On synthetic data with a catalogue of 300 and 25 interactions the top 25 matches the real interests in 17 cases out of 25.

What matters is that the converse does not work: cold start for items is not treated this way. A new item has no vector and there is nowhere to take one from — that needs content features, that is, a different architecture.

The overlap of disjoint groups of users and items in embedding space. The cause is formal: if unshown pairs do not enter the loss, there is not a single constraint between the users of one group and the items of another, the loss falls apart into independent subproblems, and the relative placement of the groups is determined by nothing. One of the equally valid optima is the one where the groups lie on top of each other.

This is degeneracy, not overfitting. It does not bother a ranker: it is applied to candidates from the same system that produced the logs. A candidate generator goes into an ANN over the whole catalogue — and gets items of another group with a high score.

The diagnosis: the difference between HitRate on the original pool and over the whole catalogue. In the simulation, 0.392 against 0.196 with 50.8% of foreign items in the top 5; for the model with catalogue negatives, 0.362 against 0.362 and 0.0%.

Because it is not broken. It spent all its capacity on exactly the task the loss set — telling items apart inside the shown pool — and it does that well: 0.392 against 0.362 for the correctly trained one.

The problem is that outside the training support it is undefined. Widening the candidate set to the catalogue costs it half its quality (0.392 → 0.196), while the correct model loses nothing.

This also explains why folding is invisible offline: the metrics are computed on a held-out set, and that set was collected by the same policy and lies inside the blocks too.

Yes, and that is the main reason retrieval is trained with a batch softmax. A batch is drawn from the global stream, so it contains representatives of every group and the loss creates cross-group constraints.

But what saves you is not «in-batch» — it is global shuffling. If the data is sharded by country or segment, if a separate model is trained per region, or if examples are grouped by session, the batch neighbours are «one's own» again and folding returns: 43.8% of foreign items in the top 5 in the simulation.

And at the level of individual items in-batch fixes nothing: an item with no feedback will never become either a positive or a negative, because only positives get into a batch. Hence Mixed Negative Sampling with a uniform addition from the corpus.

The gradient of a full softmax contains a sum over the catalogue with the factor \(P(d \mid u)\) — the probability the model itself produces. An item with a low score enters with almost zero weight: the model has already understood everything about it.

Numerically, with a positive scoring 3.0: a random negative scoring −2 contributes ×1, a popular one scoring +0.5 contributes ×11, a hard negative scoring +2.8 contributes ×67.

Hence the phrase «a full softmax mines hard negatives by itself»: nobody looks for them, the weights are assigned automatically. That property is what is lost when the catalogue is replaced by a sample, and the choice of negative source is an attempt to get it back.

The gradient of a full softmax needs the expectation of \(\nabla s\) under the target distribution \(P(\cdot \mid u)\). We sample from a proposal \(Q\) and apply importance sampling: the weight becomes \(\omega_d = e^{s}/Q(d) = e^{\,s - \log Q(d)}\).

That is, dividing by \(Q\) inside the exponent is subtracting \(\log Q\) from the logit: \(s^c = s - \log Q(d)\).

The meaning: the more often an item lands in the sample, the more we subtract and the smaller its weight in the denominator — it is punished more weakly by exactly the factor by which it is over-represented. Without the correction the model converges to \(\log p - \log Q\) and systematically underestimates the popular — the bias works against hits, not in their favour.

Estimate not the frequency but the average number of steps between an item's appearances in a batch: \(p = 1/\delta\). Two hash arrays are kept — the step of the last appearance and a running average of the interval, updated as \(B \leftarrow (1-\alpha)B + \alpha(t - A)\).

The scheme works without a fixed vocabulary, adapts to drift, and lives on parameter servers under distributed training.

Collisions understate the interval and therefore overstate the frequency, so several independent pairs of arrays are used and the maximum over them is taken — every individual estimate is understated, and the maximum is closer to the truth.

Normalisation removes the influence of the norm, which grows with popularity during training: without it the inner product systematically prefers the popular regardless of relevance.

But after normalisation the logits are squeezed into \([-1,1]\) and a softmax over them is nearly uniform. On logits \((1,-1,-1,-1,-1)\) the right answer gets only 64.9% of the mass — the gradient is smeared between the right one and the clearly wrong ones.

The temperature \(s/\tau\) brings the sharpness back: at \(\tau = 0.5\) the weight of the right answer is 93.2%. It is fixed, scheduled, or made trainable.

Every object has \(k\) parameters of its own, tied to nothing else, so any arrangement of vectors is reachable; at \(k = |\mathcal{I}|\) any similarity matrix is reproduced.

Hence a consequence: whatever a content encoder produces, the table can produce the same — the encoder's output is one particular assignment, and the table accepts any. The converse is false: the encoder has to give identical vectors to objects with identical features.

That is, their inductive bias is zero. This is the upper bound of expressiveness and at the same time the cause of every problem: there is nothing to hold the model back from memorising noise.

Because a row of the table is trained only on the examples containing that item, and there is nothing to link it to the rows of similar ones. The arithmetic: with a catalogue of 10 M, a billion interactions and dimension 256, 97.7% of items have more free parameters than observations about them.

For such a row the problem is underdetermined. All the model can do is memorise individual observations, and it is in no position to tell a rare but genuine signal from noise — from the outside they look identical.

Quality grows through the first epoch and falls sharply at the start of the second — a break rather than a gradual degradation, unlike computer vision. The practical consequence: the best result often comes from training for exactly one epoch, and industrial pipelines are built that way.

The hypothesised cause: the distribution \((\mathrm{EMB}(x), y)\) differs between seen and new examples, because for the seen ones the embedding has adapted to the label. On the second epoch the MLP adapts to precisely the distribution of the seen ones and loses its applicability to new ones. That is, the overfitting is to an artefact of the model's own table.

The sparsity can be reduced — filter out rare values or hash them into fewer buckets — and the effect really does weaken. But that often worsens the final quality: rare features are at once the source of the overfitting and the source of the signal they were introduced for.

Because the tail of features is shorter and thicker than the tail of items, and that is checkable. On a catalogue of 100 thousand with six features from a vocabulary of 5 thousand: fewer than ten observations for 74.2% of items and for 0.0% of features. The medians are 4.5 for an item against 1275 for a feature, a difference of 283 times.

The mechanism is reuse: a rare item is assembled from frequent features. The rarest item of the catalogue is expected to have two observations, while its features have been seen between 479 and 2.5 million times. Nothing is known about the item itself, everything is known about what it is made of.

Plus the saving: 5 thousand rows instead of 100 thousand, and 3.7% of them have fewer observations than the dimension instead of 98.3%.

A ceiling on the head of the catalogue: two products with identical descriptions get identical vectors, even when there is enough data to know that they convert differently. ID embeddings pick that up, an encoder over features does not.

Plus the quality of the features becomes critical — poor tokenisation limits you more than any architecture — and the vector has to be computed rather than fetched, which is more expensive at inference.

So in production it is a hybrid: content gives generalisation and cold start, ID picks up what is not in the features. The question is not «what instead of what» but in what proportion.

The main thing is that a new object is representable immediately. There is no waiting for the id to enter a vocabulary and for the vocabulary to be rebuilt: the hash is computed at once. For a fast-changing catalogue that is not an optimisation but a condition of the thing working.

The price is collisions: several ids share a row and become indistinguishable. Several independent hash functions make complete indistinguishability a product of probabilities while memory grows linearly — a favourable trade.

And an important detail: collisions hit the tail. A popular item has enough data to pull a shared row towards itself; a rare one does not.

The arithmetic does not add up. With a catalogue of 10 M and dimension 256 that is 5.1 billion operations per request — of the order of 100 ms at 50 GFLOPS, while the retrieval budget is usually 10 ms. A gap of an order of magnitude is not closed by constants.

Plus memory: 10.2 GB in float32 for the vectors alone.

A caveat: if the catalogue fits in GPU memory, exhaustive search can turn out faster than approximate search on a CPU, and then it is preferable — no loss of recall, no index rebuild, and the score need not be an inner product.

With cosine the norm of the vector cancels; with an inner product it does not. So MIPS systematically prefers items with a large norm, and the norm grows with popularity during training. The result is a popularity bias built into the geometry rather than into the data.

The practical consequence: the index has to be built for the measure the model optimises. Either normalise the embeddings and work with cosine, or take an index with an inner-product metric, or reduce MIPS to ordinary neighbour search by adding an extra coordinate.

A multi-layer graph — essentially a skip list generalised to a metric space. A point's level is sampled from a geometric distribution, so each next layer holds about M times fewer points. The search starts at the top, where there are few points, greedily descends along edges and drops a layer. The expected complexity is \(O(\log N)\).

efSearch is the size of the queue on the bottom layer, that is, the «recall against latency» knob on an already-built index. It changes at runtime with no rebuild, which is also why it serves as a mechanism of controlled degradation under load.

The weak spots: memory for the edges (up to M per node per layer) and degradation under deletions, which are done by marking and require periodic rebuilding.

A vector is cut into m subvectors, each encoded by a centroid number from its own codebook. At 256 dimensions with 32 subvectors of 8 bits a vector takes 32 bytes instead of 1024 — 32-fold compression, and a catalogue of 10 M shrinks from 10.2 GB to 0.32 GB.

Plus distances are computed faster: for a query the distances to all centroids are computed once, after which the distance to any vector is m table additions.

The payment is accuracy: the average error per coordinate is about 0.43 with a spread of 1.0. So a two-phase search is used — select hundreds of candidates on the compressed index, then recompute exact distances for those alone.

A hierarchical code produced by residual quantisation: the first codebook approximates the vector, the second the residual, the third the residual of the residual. The item becomes a tuple of several tokens.

The arithmetic: 6 levels of 256 give \(2.8 \cdot 10^{14}\) combinations at a code length of 6 bytes against 1024 for a float32 vector — 171-fold compression. With a catalogue of 10 M, three levels already suffice to address an item; the rest refine.

But the point is not the compression, it is the hierarchy: items close in meaning get a shared prefix. A new item inherits the prefix of its region, and a model that has never seen it already knows the main thing about it. This is the same mechanism as content features — a rare object assembled from frequent pieces.

If an item is a sequence of tokens, recommendation becomes generation: the model predicts the code token by token, the way a language model predicts words. The index disappears entirely: no ANN, no rebuilding, no trade of recall against latency.

The two-tower constraint is lifted at the same time — the score stops being an inner product, because autoregression sees the tokens already generated.

The risk is specific: the model can generate a code that corresponds to no existing item. Cured by constrained decoding over a prefix tree of valid codes. And the approach is worth judging with restraint — few have taken it to the scale of tens of millions of items.

By the set over which a single term of the loss is computed, and everything else follows from that.

Pointwise models relevance itself (BCE on 0/1) — it gives calibrated probabilities but knows nothing about order or the structure of the slate: a mistake at position 1 and at position 50 are punished identically.

Pairwise models the observed order — it is aligned with AUC, because AUC is the share of correctly ordered pairs, but it loses calibration by construction.

Listwise models the structure of the slate — it is closest to NDCG, more expensive to train and even worse calibrated.

We postulate \(P(i^+ \succ i^- \mid u) = \sigma(f(u,i^+) - f(u,i^-))\). The likelihood of all observed preferences is a product of such probabilities; take the logarithm and flip the sign:

\(\mathcal{L}_{\mathrm{BPR}} = -\sum_{i^+ \in P}\sum_{i^- \in N} \log \sigma\bigl(f(u,i^+)-f(u,i^-)\bigr)\)

Two properties have to be named at once. The loss depends only on the difference of scores — so it is invariant to a shift and gives no calibration. And the gradient with respect to the positive's score equals \(-\sigma(-\Delta)\): a pair inverted by 3 weighs 0.95, one already separated by 3 weighs 0.047, that is, 20 times less. Mining of hard pairs is built into the shape of the loss.

Both metrics depend only on the order, and any monotone transformation of the score preserves the order. So a calibrated model and its own score passed through a monotone function are indistinguishable by AUC and NDCG.

It starts to hurt exactly when the score takes part in arithmetic. An example: five products, the ranking by score is the same for both models, but the ranking by expected revenue \(p \cdot \text{price}\) is different; the product that rises to the top has an expected revenue of 30 instead of 90, a loss of 67% at the top position. Log loss meanwhile worsens by 27% — it does see the difference.

Hence the practice: \(pCTR \cdot bid\) in an auction, \(P(\text{buy})\cdot\text{price}\), thresholds for business rules — everywhere a pointwise output is needed, even if the final order is produced by a different loss.

It is a pairwise loss in which every pair is weighted by the change of the metric from swapping it: \(\mathcal{L} = \sum_{(i,j)} \Delta\mathrm{NDCG}_{ij}\log(1+\exp(-s_{ij}(f_i-f_j)))\). The second factor is the same BPR; all the peculiarity is in the first.

Why: NDCG is piecewise constant and not differentiable, so it cannot be optimised directly, but the gradients can be weighted by a pair's contribution to the metric. The spread is enormous — a swap at positions (1,2) gives a ΔNDCG of 0.3691, at (9,10) 0.0120, that is, 31 times less.

And it is listwise because ΔNDCG is computed over the whole slate: you need the positions of all the other documents and the IDCG of the query. A formally pairwise loss carries information about the structure of the entire list.

The loss itself is pairwise, the same BPR. Everything is in the weights \(w_{ij} = N_{ij}\cdot c(l_i,l_j)\).

\(N_{ij}\): the scores are perturbed many times with logistic noise and re-sorted; only the pairs that turn out adjacent get weight, with an addition of \(1/R\) by the position of the adjacency. Hence the position discount is built in, the weight matrix is sparse (no more than 990 pairs out of 4950 at 100 documents and 10 perturbations, that is, linear in n rather than quadratic), and the weights are recomputed at every boosting iteration.

\(c(l_i,l_j)\): the probability that \(i\) really is better than \(j\), given a matrix of labelling errors. The pair «3 against 2» gets a cut weight, the pair «4 against 0» keeps nearly one.

The difference from LambdaRank: the latter weighs a pair by how much the metric changes when it is swapped, proceeding from the assumption that the current order is right. YetiRank weighs it by how likely the pair is to end up adjacent and high at all — that is, it models its own uncertainty explicitly.

Three cases. First, binary labels: the factor \(c(l_i,l_j)\) degenerates and a weighted BPR is all that is left. Second, labels from logs rather than assessors: a click is not «confused», it is biased by position and by impression, and a matrix of labelling errors does not describe that noise; bias corrections are what is needed. Third, when calibration is needed: like any pairwise loss it depends only on differences of scores.

The second case is the most common mistake in applying it: the method is taken into recommendations by analogy with search, where the labels really are from assessors.

The formula is one and the same. The difference is where the set in the denominator came from, and it is fundamental.

In candidate generation the denominator is sampled negatives: we set the distribution ourselves, so we know it and can correct the bias with LogQ. In ranking the denominator is the shown slate, that is, the result of the logging policy; its distribution is not under our control.

Plus a listwise output is particularly badly calibrated: add one strong item to a slate of four and the «probability» of the first falls from 0.4551 to 0.2034 — by 55%, though the item itself did not change.

All of them optimise a proxy under a fixed impression policy. The items the policy never showed are in neither the pointwise sum, nor the pairs, nor the denominator of the softmax — no loss will tell the model about them.

That is the feedback loop: the model learns from what the previous model showed. The circle cannot be broken by the choice of loss — you have to affect the impression policy itself, that is, do exploration.

Three concrete reasons. High cardinality: a one-hot over a million values cannot be built, and target encoding loses the semantics — two items with the same average CTR and different audiences are merged into one number, and that costs 50% of the clicks in a simple example with two segments.

Unstructured data: texts, pictures, history — features can be generated from them, but it works worse.

And the main one: embeddings at the input of a tree work badly in principle. The coordinates mean nothing individually, and a split looks at one coordinate; to distinguish 4 levels along each of 16 dimensions you need 4.29 billion leaves, while a linear layer gets by with 16 multiplications.

Plus two practical arguments: many signals at once, and drift — boosting has to be rebuilt entirely, a network can be fine-tuned.

When not a single point of the checklist holds: plenty of data, important high-cardinality features, unstructured data, many signals, a need for fast fine-tuning. If none of that is there, CatBoost on aggregates will be both cheaper and better.

The main drawback of networks is the barrier to entry: GPUs, distributed training, inference in a runtime. That is a real cost, and pretending it does not exist is not a good look in an interview.

Represent the category by a one-hot vector \(e_i \in \{0,1\}^n\) and multiply by a matrix \(W \in \mathbb{R}^{n \times d}\) with no bias. The result \(e_i W\) is the \(i\)-th row of \(W\). So \(W\) is the embedding table, and a lookup is simply an optimisation of a multiplication by a sparse vector.

A useful consequence: at \(d = 1\) the model learns the average CTR of a value, that is, target encoding is the special case of an embedding of dimension one. Everything an embedding gives beyond that is the dimensions in which «who exactly likes it» fits.

It should depend on the cardinality and on the informativeness of the feature; the same size for everything is suboptimal.

The informational estimate: \(d\cdot s = \log_2 n\) bits. Google's heuristic: \(d = 6\sqrt[4]{n}\). The gap between them is enormous — at \(n=10^7\) that is 23 bits against d = 337 — and it is meaningful: almost all the capacity goes into semantics rather than identification.

The heuristic is not taken literally: at \(n=10^9\) it gives 1067 and 4.3 terabytes for one feature. In production the typical dimensions are 32 to 128, and the honest route is to choose the dimensions during training.

Batched lookup: combine the tables of all features into one matrix and do a single lookup with an offset — otherwise you get many small operations on the GPU with the overhead of a kernel launch.

Sparse gradients: a batch of 4096 samples touches no more than 4096 rows out of 10 M, that is, 0.041% of the table; the rest of the gradients are zeros and need neither updating nor sending between cards. An important subtlety: for such parameters the learning rate is worth increasing, because they are updated far less often than dense weights.

Row-wise optimiser: keep Adam's statistics per row rather than per parameter. A table of 10 M × 256 takes 10.2 GB, with Adam's moments 30.7 GB, and with row-wise statistics 10.32 GB, that is, 0.8% more than the table.

Plus sharding, offloading to the CPU and the hashing trick when the tables do not fit on a GPU.

Because networks are sensitive to scale (large values dominate the gradient), break on outliers and cannot handle missing values. Trees have none of these problems — this is the price of moving to networks.

Five transformations: a logarithm against skew; a sigmoid that squeezes outliers (with normalisation before it, otherwise the gradients vanish); the cumulative distribution function, which brings the feature to a uniform distribution; periodic functions for time; quantisation by quantiles.

Quantisation has a fundamental drawback — the order relation is lost both between bins and inside a bin. Piecewise linear encoding repairs that.

Because in raw form 23 and 0 are the most distant pair possible, though they are adjacent hours. The model is forced to spend capacity on learning the join at the boundary of the day.

A sine-cosine pair of one frequency places time on a circle: the distance between 23 and 0 becomes 0.2611 — exactly the same as between 0 and 1, and the smallest possible. The most distant pair turns out to be 23 and 12, the opposite points of the day, as it should be.

Several frequencies give several circles of different scale — the hour, the day of the week and the season at once. The mechanism is the same as for positional embeddings in a transformer.

Piecewise linear encoding: a vector of length T where to the left of the current bin there are ones, to the right zeros, and exactly one component is fractional — the share of the bin travelled. A linear layer on top of such a vector gives a continuous piecewise linear function.

The difference from a one-hot by bins: inside a bin the output is not a constant but moves linearly. On approximating \(x^2\) with 16 bins the MSE differs by a factor of 857. And more important than the gap is how it grows: for piecewise constant encoding the error falls as \(T^{-2}\), for piecewise linear as \(T^{-4}\).

Because it separates two regimes that are mixed together in a dense vector. A dense content vector is a «soft» feature: the model generalises from it but cannot memorise the specifics of a particular object. A discrete hierarchical code gives both: the shared prefix is responsible for generalising to similar objects, the full code for memorising the particular one.

This is exactly the case where discretisation does not lose information but structures it. The flip side of the same argument is the hashing trick: its identifiers carry no semantics, so for a new object you get a combination the model has never seen — full reliance on memorisation and zero generalisation.

Six steps, each repairing the breakage of the previous. A linear model — no interactions at all. Second-degree cross features — there are interactions, but \(O(n^2)\) parameters and no generalisation to unseen combinations. FM — factorise the weight matrix, memory \(O(nd)\) and generalisation by transitivity, but only the second degree. Early networks (Wide & Deep, DeepFM, DLRM) — put an MLP on top, and it turns out it does not find the pairs by itself. DCN-v2 — write the multiplication straight into the layer. Transformers over features — try to choose the important pairs dynamically.

The practical upshot: by default you take DCN-v2. Everything to the left is history, everything to the right is the research frontier.

Two things, and the second is more serious. Size: \(O(n^2)\) parameters in the total cardinality; at n = 3 M that is 4.5 trillion weights, 18 TB in float32 — and that is for interactions of only the second degree.

Generalisation: the weight of an unseen combination is not trained at all, and the unseen are the overwhelming majority. With 30 fields of 100 thousand values and a billion samples, even in the ideal case no more than 10% of the combinations occur, and given a power-law distribution, orders of magnitude fewer. This is a problem of representation, not of scale: hardware will not solve it.

By factorising the weight matrix: \(w_{ij} \approx \langle v_i, v_j \rangle\). Every feature gets a vector, and the weight of a pair is an inner product.

Memory falls from \(O(n^2)\) to \(O(nd)\): at n = 3 M and d = 32 that is 384 MB instead of 18 TB, 47 thousand times less.

Generalisation arises by transitivity: to estimate the weight of the pair (i, j) it is enough that \(v_i\) was trained on other pairs with i and \(v_j\) on pairs with j. Countably it looks like this: with 1000 features there are 499 500 pairs in all, while FM at d = 16 has just 16 000 parameters, that is, 3.2% — the information from that share of pairs is enough to recover the rest.

Because universality of approximation is a statement about the existence of weights, not about gradient descent finding them on finite data.

A product is awkward for an MLP: it builds it from piecewise linear splines, which takes many neurons and much data. Rendle et al. showed directly that a learned MLP similarity loses to an ordinary inner product.

And in recommendations it is worse still: the features are sparse, a particular combination occurs rarely, and there is simply nothing to learn the product from. Hence the conclusion — write the multiplication into the architecture rather than hoping for it.

\(x_{l+1} = x_0 \odot (W_l x_l + b_l) + x_l\). Three parts. \(W_l x_l\) is a linear combination of the whole concatenation, so inside it one can obtain the inner product of any feature vectors regardless of their dimensions: the DeepFM and DLRM requirement of a single dimension is removed. The elementwise multiplication by \(x_0\) is the product written into the architecture, with each layer raising the degree by one. Plus a residual connection.

L layers give combinations of degree up to L+1, but more than 2 or 3 layers give no gain: the number of monomials grows faster than the data covers them — at the fourth degree with a hundred features it is already 3.9 M combinations.

The layer is heavy: \(W_l\) is d × d over the whole concatenation. Hence the low-rank variant \(W_l \approx U_l V_l^\top\) — at d = 1024 and r = 64 it is 8 times cheaper — and beyond that a mixture of low-rank decompositions.

Stacked usually works better than parallel: the cross network first, then the MLP over its output, with the wide vector narrowed before the MLP.

An important and unobvious detail: a cross network gives no gain on top of abstract vector representations — it has to be applied precisely over a concatenation of feature vectors. If you feed in an already mixed representation, the output of a transformer say, there is nothing left to multiply: the coordinates no longer correspond to features.

And the division of labour: cross models the low orders explicitly, the MLP the high ones implicitly. They complement each other rather than compete.

Because it treats all tokens identically. For text that is right — tokens are homogeneous. In recommendations the features are heterogeneous by nature: age, an artist ID, a price, an embedding from another model. A single transformation imposes a homogeneity that is not there.

Hence Hiformer with separate Q, K, V and its own FFN per feature — the problem is real, but the layers need a great deal of memory. AutoInt, which applies ordinary attention, works worse than DCN-v2.

The gradients vanish: the factors of the Jacobians multiply. If every layer multiplies the gradient by 0.8, then at 32 layers 7.9e-04 remains — a fall by a factor of 1262, and the lower layers practically do not train.

ResNet: \(y = F(x) + x\). The Jacobian of a block is \(I + F'\), and the product contains a term made of ones alone — the path entirely along skip connections, so from below the gradient is bounded by one. A superfluous layer can also be zeroed out: at \(F \equiv 0\) it is the identity. From above the product grows, which is why a skip always comes with normalisation. The downside is that a sum requires matching dimensions, so the layers are heavy.

DenseNet: a concatenation instead of a sum, the dimensions need not match, and F can be a cheap narrowing layer. With an input of 512 and four blocks of growth 64 that is 1.56e+05 parameters against 1.05e+06 for ResNet — 6.7 times fewer. The price is growing width.

Because there are always several signals: engagement (clicks, time), satisfaction (likes, ratings), long-term metrics. A separate model per signal costs several times the resources and is harder to maintain.

Shared bottom gives regularisation, transfer of knowledge and a saving. The trouble arises on conflicting tasks: the gradients of the heads pull the shared representation in different directions. In a simple geometric model where the directions of the tasks differ by an angle α, each gets cos(α/2): at 90° that is 0.71, a loss of 29%, and at 180° zero.

The blunt cure is to widen the trunk so that there is room for both directions. The more careful one is MMoE.

MoE: \(y = \sum_i g_i(x) f_i(x)\), where \(f_i\) are experts and \(g\) is a gate; the key idea is conditional computation, with part of the experts working on an object.

MMoE is multi-gate: every task has its own gate, that is, its own expert weights. If the tasks are unrelated, each learns its own subset of experts and no conflict arises.

An important note for an honest comparison: the gates are of the order of one and a half percent of the parameters, everything else is simply the capacity of the experts. So MMoE has to be compared not with a narrow shared bottom but with one widened to the same capacity — otherwise it is unclear what produced the gain.

The problem is a mismatch between the slice of training and the slice of application. Training P(purchase | impression) directly is blocked by sparsity: at a CTR of 5% and a CVR of 3% the imbalance is 1 to 666, and the signal is too complex besides. Training P(purchase | click) is easier, but such a head is trained on 5% of the data (clicks only) and applied to every impression — 20 times more objects.

Why that breaks ranking: pCVR is a probability conditional on a click. A niche product with a pCTR of 0.01 and a pCVR of 0.40 is the champion by the conditional probability, while by expected purchase it is 0.004 against 0.0072 for a popular one — a loss of 44% at the top position.

ESMM: model the chain impression → click → purchase, P(purchase|impression) = P(click|impression)·P(purchase|click), two heads. And the key technique — the second head is trained on a loss over P(purchase|impression), that is, over the entire space of impressions; the gradient reaches it through the product.

For two different reasons, and they have to be told apart: users click the top more often because they see it sooner and trust it more (that is a bias), and the top really does hold more relevant items (that is the model working). Mixing them means subtracting real relevance along with the bias.

Concretely, the bias reverses the order: an item with a relevance of 0.12 at position 5 gives an observed CTR of 0.039, while an item with a relevance of 0.10 at position 1 gives 0.100. A naive model confidently puts the worse one higher.

Four cures: train on the first position only (with a slate of 20 that leaves 5% of the data), position as a feature with pos=1 substituted at inference (the model can hide relevance in the position feature), IPW (needs random traffic, and a spread of weights of 8 times inflates the variance) and a bias tower — the production standard.

A separate small tower fed with all the context useful for modelling the bias — not only the position but also, for instance, the device: fewer positions are visible on a phone than on a desktop.

During training the outputs are added in the logit: the main model plus the tower. A mandatory element is dropout on the tower's output, otherwise it explains too much by position, pulls the relevance signal onto itself, and the main model is left undertrained. At inference the tower is thrown away and a clean score remains.

The elegance of the idea is that the bias is modelled explicitly and separably: we do not subtract it after the fact, we give it a separate channel to be written off to, and then cut that channel away.

Two reasons. First: a label is binary, a prediction is not. «There was no click» looks identical for an object with a probability of 0.40 and one with 0.02, while the teacher distinguishes those cases. This is dark knowledge — what is passed on is not the answer but the shape of the distribution.

Second: the teacher is a smoothed version of the data. At a true click probability of 0.05 and ten impressions, the standard error of the estimate is 0.069, that is, 138% of the quantity itself: the label says almost nothing about the particular object. The teacher averaged that noise over the whole set, and the student gets a less noisy target — and noise is the main source of overfitting on the tail.

The motivation for the whole technique is trading quality for latency: the teacher is computed offline, the student fits the ranker's budget.

Because the teacher has biases of its own: it was trained on the logs of the same system and inherited its positional bias, its skew towards the popular and its feedback loop. Distilling it directly, we distil that too.

The solution: one head learns from the target labels, the other from the teacher's predictions, and the first is the one used for ranking. The distillation head works as an auxiliary task: it forces the shared representation to absorb the teacher's knowledge without dictating the order. The knowledge passes through the shared trunk, the biases stay in the other head.

The second motive is calibration: the teacher's predictions break it, a head on real labels preserves it. For advertising, where the score is multiplied by a bid, that settles it.

Four places. The teacher drifts when retrained, and the student's target changes abruptly — and monitoring will not show it, since each model on its own looks fine. The teacher has to be computed over the whole training stream — distillation moves the cost from inference into training rather than removing it. The student inherits the teacher's systematic errors and will never surpass it on that subset, though on the real labels it could have. And a closed circle of versions: the teacher is trained on the logs of the previous student, and after a few iterations the system distils itself.

In that the weights depend on the candidate: \(v_U(A) = \sum_j w_j e_j\), where \(w_j = g(e_j, v_A)\). The user no longer has one vector — they have a vector per candidate.

What for: averaging dissolves a rare but decisive event. In a history of 12 events the single music track gets a weight of 0.083 under averaging, while attention gives it 0.835 — ten times more. The score changes sign from −0.56 to +0.83.

When the history is homogeneous the difference is minimal — which is exactly why the problem went unnoticed for so long.

Because the sum of the weights is interpreted as the intensity of the user's interest in the candidate. Under softmax normalisation it is always one, and that information is lost: a user with ten relevant events in their history and a user with one would give a convolution of the same «mass».

That is, normalisation flattens exactly what one would like to distinguish.

It sets how concentrated the attention is. The weights are a softmax of inner products divided by τ, and at both ends of the scale the construction degenerates.

At τ = 2 the entropy of the weights is 2.43 against a maximum of ln(12) = 2.48 — attention turns into ordinary averaging, the very thing the exercise was meant to escape. At τ = 0.05 the entropy is zero and the weight of one event is one — a hard selection of a single element, with the rest of the history thrown away.

The reason is the same as in a two-tower model: the inner products of normalised vectors are squeezed into [−1; 1], and without the division a softmax over them is nearly uniform.

SASRec is a transfer from NLP: items as words, users as sentences, the task next-item prediction, a transformer decoder with a causal mask. BERT4Rec uses bidirectional attention and a cloze task instead of causal prediction; it also trained on a full softmax. BERT4Rec showed better results.

The resolution: SASRec with the right loss beats BERT4Rec. The original advantage was explained not by bidirectional attention but by the loss function — in vanilla SASRec it was binary cross-entropy with one uniform negative.

How bad that is: with a catalogue of a million, the probability that a random negative is one of the hundred nearest to the positive is 0.0001. To see a hard negative even half the time you need almost seven thousand samples. The model almost never sees what it is supposed to learn to distinguish, and the architecture has nothing to do with it.

The lesson: when comparing two papers, check the loss and the negative-sampling scheme first.

Because of load. Under early fusion the user vector depends on the candidate, so the transformer over the history has to be run separately for every candidate.

The bill: a history of 100 events, d = 256, two layers, 500 candidates. One run is 6.3e+07 operations, that is, 1.25 ms at 50 GFLOPS. Five hundred runs are 627 ms, which at a ranking budget of 20 ms is a miss by a factor of 31.

The solution is late fusion: a two-tower architecture where the transformer over history and context is computed once, the item is encoded by its own tower and the score is obtained as an inner product. The same «expressiveness against cost» trade-off as in candidate generation, only now at the ranking stage.

Two arguments. Informational: an impression was chosen by the previous model while a click was chosen by the person; impressions speak about the system, not about the user. Engineering: at a CTR of 5% there are twenty times more impressions, and attention is quadratic in length, so the cost grows 400-fold.

So impressions are both more expensive and less informative — the decision is an easy one.

It cannot be put directly into a model that runs per request: attention is quadratic, and a history of 8000 events is 1113 times more expensive than one of a hundred, with the quadratic term already 94% of the cost there.

Two approaches. Event picking — choose from the deep history by a meaningful principle rather than taking the last n. And an offline loop — compute the vector of the deep history in advance, once a day.

The offline loop has upsides: it is cheap, simpler to deploy, a heavier model fits for the same resources, and the history goes up to two thousand events. The downsides are fundamental: no fresh events and no context. So it does not replace the real-time loop but complements it — the vector of the deep history is fed into the fast transformer.

Nobody reports it directly. It is measured by the return to the results: if the user's next event happened t seconds later, that is roughly how long they spent on the document; the absence of a return is treated separately. The «deep click» target is built on that — a stay longer than N seconds.

This is a good example of a general property of recommendation data: the quantity of interest is not observed, and a proxy with biases of its own is constructed in its place.

Because the ranker scores items one at a time while the user sees the list as a whole, with the items interacting. Two items with a click probability of 0.20 give 0.36 if they are independent and only 0.20 if they are duplicates. And the original plus a less relevant item from another topic (0.12) gives 0.296 — a slate 48% better.

So «choose the best k items» and «assemble the best list of k items» are different problems, and the second does not reduce to the first by sorting.

By the goal, not the mechanism. In exploration we deviate from the greedy top in order to obtain data about what we have not been showing — we pay with present quality for the future quality of the model. In working with diversity we deviate for product reasons: a list of ten nearly identical products is bad here and now, regardless of what data we will collect.

The mechanism is similar, but the budgets and the metrics of success are different.

At three levels. Inside the output — intra-list diversity (the average dissimilarity of pairs) and the entropy of categories. Per user over a period — the share of categories new to them, serendipity. Across the system — coverage and Gini over impressions.

Entropy is more convenient than a simple count of categories because it also penalises a skew: the slates «8 + 1 + 1» and «4 + 3 + 3» both contain three categories, but the entropies are 0.639 against 1.089 — nearly double. And «8 + 1 + 1» is an output from one topic with two random specks.

Greedy selection with a penalty for similarity to what is already chosen: at every step we take \(\arg\max [\lambda\,\mathrm{rel}(u,i) - (1-\lambda)\max_{j\in S}\mathrm{sim}(i,j)]\). The penalty runs over the nearest item already chosen rather than the average: one duplicate in the slate poisons a candidate entirely.

The shape of the trade-off curve is typical: the first few percent of diversity are almost free (all three clusters are covered for 7.6% of relevance, and diversity grows from 0.32 to 0.82), after which the price rises sharply — the remaining 0.07 of diversity costs another 13 percentage points. Work at the knee.

Only by A/B. Offline metrics always vote for λ = 1: at that value the sum of relevance is maximal by definition, and so are NDCG and Recall@k. They evaluate an output as a set of independent items — the very assumption because of which the re-ranking layer was needed at all.

What has to be measured is product metrics: return rate, session length, the share of users interacting with more than one topic.

We define a distribution over subsets with \(P(S) \propto \det(L_S)\), where \(L = \mathrm{diag}(q)\,S\,\mathrm{diag}(q)\), and sample from it. The determinant of a Gram matrix is the squared volume of the parallelepiped on those vectors.

Collinear vectors (nearly identical items) give a volume of zero, and such a subset is never sampled at all. Orthogonal ones give the maximum. For two unit vectors det = sin²θ: 0 at 0°, 0.75 at 60°, 1 at 90°. And the lengths are responsible for relevance — the determinant grows as its square.

The advantage over MMR: one construction accounts for both relevance and diversity, with no manual λ balancing two terms of different natures. But it is rarely used in production — even approximate algorithms require the similarity matrix to be computed explicitly and decomposed, which does not fit the budget.

Because the usefulness of a slate is usually monotone and submodular: every next item adds no more than it would have added earlier — a formalisation of saturation. For such functions the greedy algorithm gives no worse than 1 − 1/e ≈ 0.632 of the optimum.

And that is not a consolation prize: enumerating a slate of 10 from 500 candidates is 2.5·10²⁰ options. At the real sizes of the problem, greediness is the only option that has any lower bound at all.

Yes, if the slate is built autoregressively: solving the problem as predicting the next item given those already chosen. Then the model learns for itself that after a pair of trainers it should not show four more — diversity comes out as a side effect of the formulation, with no manual λ.

The price: k sequential passes instead of one batch, because position k cannot be computed without choosing the previous ones — and that is exactly what the ranking budget cannot bear. Plus the model inherits its diversity from logs collected by a previous policy.

The model is trained on logs collected by a previous version of the same model. Everything it did not show is absent from the data, and a good item with no impressions will stay unknown: to learn a CTR you have to show the item; to show it you have to know its CTR.

The way out is to change the logging policy. Two cheap ways: a random quota (ε-greedy) and softmax sampling with a temperature. They usually take the first 95% of the benefit. Beyond that comes a bandit layer with an estimate of uncertainty.

An important caveat about softmax sampling: it re-ranks only what the funnel has already selected, and the funnel selected greedily. Real exploration requires a mechanism at retrieval too.

From Hoeffding's inequality: for a sample on [0,1] with true mean μ and sample mean \(\hat\mu\), \(\mathbb{P}(\mu \ge \hat\mu + u) \le e^{-2nu^2}\). Set the right-hand side equal to δ and solve for u: \(U = \sqrt{-\ln\delta / 2n}\).

Then choose δ = 1/k^c — demanding an ever more reliable bound as time goes on — and get \(a_k = \arg\max [Q_k(a) + c\sqrt{\log k / n_k(a)}]\).

It reads as «the estimate of the mean reward plus the width of the confidence interval». The bonus decays as \(1/\sqrt{n}\): at 1 impression it is 3.03, at 1000 it is already 0.096. So no separate exploration knob is needed — the algorithm shrinks it as data accumulates.

UCB is deterministic and optimistic: it takes the upper bound of the interval. Thompson is Bayesian: it samples parameters from the posterior and acts greedily with respect to the sample. For a binary reward that is Beta-Bernoulli — a Beta prior is conjugate to the Bernoulli, the update is trivial (success → α+1, failure → β+1), three lines of code and no hyperparameters beyond the prior.

By regret both are sublinear, unlike ε-greedy: for the latter, doubling the interval doubles the increment (73.9 → 142.5), because a fixed share of random traffic is spent forever.

The main practical difference is not in the reward but in measurability, see the next question.

Because of propensities. UCB is deterministic: in a given state the order is the same, so π(i|u) equals zero or one, and IPS, SNIPS and doubly robust turn into a division by zero. A new model cannot be evaluated offline on such logs.

Thompson defines a distribution over permutations: the propensities are positive and can be estimated by repeated sampling at the same μ and σ. One layer closes both problems — it explores and it makes the data suitable for evaluation.

A caveat: positive propensities alone are not enough, their spread matters too. If 1% of the observations have a weight of 100, the effective sample size falls from 1000 to 39 — the estimate is formally unbiased but useless. Hence weight clipping and the requirement that the policy not stray far from the logging one.

A direct transfer does not work: in the classical setting there are a handful of arms pulled thousands of times each, and here there are millions of items and zero impressions in the tail. The options are an item, a source of candidates, the whole policy, a group of items.

Almost always an item is chosen, and that is possible because an impression counter is not the only source of uncertainty. A trained model can answer «I do not know» about an item with unfamiliar features and an empty history. The uncertainty comes from the model, not from a counter.

Hence the right formulation: a bandit is an add-on over the ranker, not a replacement. The ranker is responsible for the mean, the bandit for what to do with the spread around it.

Four engineering reasons. The cost of a mistake is bounded: the layer rearranges a hundred already-filtered candidates, and the worst case is showing the ninth instead of the second; at retrieval the cost is bounded by nothing. All the features are here, and σ is needed about a (user, item) pair in context. There are few candidates: an ensemble of five models over a hundred is affordable, over ten thousand it is not. And this is the point where a decision is being made anyway.

The limitation: the layer explores only what the funnel selected. If retrieval is greedy, the bandit explores inside an already biased pool.

Three sources: an ensemble of models (the sample variance of the predictions), MC-dropout (dropout at inference, several passes of the head) and neural linear (Bayesian linear regression over the last layer, giving an honest \(\sqrt{x^\top A^{-1}x}\)). The first two cost almost nothing in development.

The main thing is not to confuse epistemic uncertainty («the model does not know», decreases with data) with aleatoric («the event is random», never decreases). The temptation to take \(\sigma = \sqrt{p(1-p)}\) is strong — the formula is at hand. But that is pure aleatoric noise: at p = 0.3 it equals 0.4583 both at one impression and at ten thousand. Such a bonus is maximal at p = 0.5 and simply drags mediocrities upwards.

The check: plot σ as a function of the number of impressions. No dependence means you are measuring the wrong thing. And do not build an ensemble from models with a shared embedding table: the predictions will coincide and σ will collapse.

Remember that the dependence is a hump, not a staircase. In the simulation α = 3 gives 0.909 of the ideal output against 0.834 for greedy (plus 9% by rearranging the same pool), while α = 5 already gives 0.891: we start lifting items about which everything is already clear.

And separately about small α. At α = 0.5 the gem looks into the top in 7 catalogues out of 8 but settles at top-1 in only 2: it gets a few impressions, catches an unlucky streak, its estimate sags, and the bonus \(\propto 1/\sqrt{n}\) has already dried up. A small bonus is worse than none — it spends impressions without finishing the job.

When a random quota and softmax sampling are enough — and they are enough to take the first 95% of the benefit. The layer pays for itself under three conditions at once: the catalogue updates fast, there is an honest estimate of uncertainty, and propensities are needed for offline evaluation.

There is also a deterministic alternative: hard-lift exactly one post by a little-known author to a fixed position. That is how it is done in the open-sourced code of the X feed. Predictable, explainable, ten lines — and if the only task is to keep a new author from dying in obscurity, that is enough.

And a methodological trap: the gain from exploration shows up in the quality of future models, while it is measured by a week-long A/B on today's metrics. On that horizon it almost always looks like a loss.

Because different data has different freshness and a different cost of computation. «The user likes dairy» is an aggregate over 90 days and cannot be computed in the 50 ms of a request. «The milk is already in the basket» is visible only in real time. Computing everything in the runtime is expensive; computing everything in advance is stale or impossible.

Hence lambda: batch recomputes the aggregates from scratch once a day, speed updates them incrementally per event, serving keeps two mirrored halves, and a separate profile service merges them, allowing for the fact that batch may already have taken in part of the events from the deltas.

The main pain is that one aggregation logic lives in two places: fix the bug twice, deploy twice, often different languages and different teams.

A discrepancy in the way a feature is computed between training and inference. The canonical example: offline computes the customer's age from the first order, online from the registration. The model trains on one thing and infers on another.

The scale: with a gap of 14 days the share of customers above the threshold shifts by 4.6 pp in the middle of the distribution, where the bulk of them sits. And that is one feature out of hundreds.

Feature skew is the second most frequent cause of a gap between offline and online results, after a wrongly chosen metric. It is cured by a shared feature-computation library that both offline and online call.

Logging gives exactly what the model saw, guarantees the same distribution and lets the prediction be reproduced. The price is storage and delay: at 10⁸ requests, 100 candidates and 500 features that is 20 TB a day and 7.3 PB a year, and a new feature becomes available for training only one training window after you start writing it.

Reconstruction removes both, but restoring the state exactly is nearly impossible, not every feature is restorable (which constrains the design), and a leak from the future is easy to get.

The intermediate solution is to log only what was shown: a tenfold saving, but the information about the candidates that were not shown is lost — and that is what the offline evaluation of policies needs.

Backend: the identifiers of the request, the user and the item, the server-side time, the position, the model's score. Frontend: mandatorily the identifier of the request the item came from, plus the identifiers, plus the time — server-side or by arrival in the storage, since the client's clock cannot be trusted.

They are merged by the request identifier. All else being equal, backend logs are preferred: client-side ones get lost, and a lost click does not become a missing value — it becomes a zero, that is, a false negative. At 10% losses the observed CTR shifts down by exactly 10%, and nothing is left in the data to show it.

The gluing together of one person's identifiers: user_id when logged in, the device_id of various devices, cookie_id. Without it the history is smeared.

Concretely: a user has 60 events and the threshold for a substantial history is 20. Across five identifiers that gives 12 each, and not one reaches the threshold — the person looks like a newcomer on every device. That is the fake cold start: the history exists but is fragmented.

It is especially critical where users deliberately hide their behaviour — private mode, no login. Identifiers have to be handled at the level of the shared feature layer, otherwise the gluing will be implemented differently in three places, and we are back to skew.

The share of requests for which the feature is available. It is watched for two reasons.

The first is the arithmetic of accumulation: with 50 features at a coverage of 0.95 each, the full vector exists for only 7.7% of requests. Handling missing values is the main mode of operation, not an edge case.

The second is a gap between training and production: trained at a coverage of 0.87, in production 0.60, and the share of requests in an unseen regime grew from 13% to 40%. The model's behaviour there is unpredictable.

And a sharp drop in coverage almost always means a breakage upstream — the cheapest alarm signal in existence.

Covariate — the distribution of X changed (a new cohort, a campaign, a season); cured by retraining and fixing the source. Label — the distribution of Y changed (the click rate grew because a loyal audience arrived); the cause has to be worked out. Concept — the relation X → Y itself changed; the hardest, retraining helps only if the content keeps up.

Telling them apart matters because label drift is easy to mistake for a success of the model.

Detection: for features, PSI over the deciles of the training distribution, with thresholds of 0.10 and 0.25. The orders of magnitude are worth remembering: a shift of half a sigma gives a PSI of 0.24, a quarter of a sigma only 0.06. For predictions — the distribution of the scores at inference. For targets — daily CTR by segment.

Three variants of the pipeline: by hand (for a start), on a schedule (enough for most tasks), by trigger — a metric dropped, drift was detected, N examples accumulated (for news and the like).

Neural networks have it easier than boosting here: boosting is trained from scratch every time, while a network can be fine-tuned on top of the old weights with the lower layers frozen.

The risks of online fine-tuning: forgetting patterns that matter for rare segments; strengthening the feedback loop, because the model closes in on its own recommendations faster; and quickly absorbing anomalies such as a bot attack. Usually they are combined: regular full retraining as an anchor plus fast fine-tuning on top.

The main reason is a different load pattern. Features and models are bound by CPU and GPU, candidate generation by memory. In one process the service needs both at once, and it scales horizontally badly: adding replicas for the sake of CPU duplicates the whole index.

The arithmetic: an index of 100 GB, a compute layer of 4 GB. At eight replicas the monolith holds 832 GB and the split system 232 GB, 3.6 times less; the difference is six extra copies of the index.

The other reasons: separation of responsibility, removing a single point of failure, flexibility of experiments, a boundary between teams. The price is possible duplication of part of the data in memory — and at two replicas there is no gain at all.

Immediately after candidate generation. Two reasons.

The first: otherwise we spend expensive inference time on items we know in advance we will not show. With a budget of 500 candidates and 30% rejection a late filter leaves 350 useful ones instead of 500 — that is 43% fewer for the same money.

The second: the output may come out short, which drags the business metrics down. To guarantee 20 positions at 30% rejection you have to score 29, otherwise there is a follow-up request and the response time grows.

The ideal option is to arrange candidate generation so that candidates are born already filtered: no wasted work at all. More expensive to implement.

That all the traffic goes down the stack at once, and the stack is not ready for it. And the dependence is counterintuitive: the better the cache was working, the harder the blow. At a hit rate of 0.95 the layer below is sized for 5% of the traffic and receives 100% — a 20-fold spike; at 0.99 it is 100-fold.

That is, a system with an excellent hit rate looks healthier while in fact being more vulnerable. Hence two rules: choose the cache keys carefully and write separate tests for this. Invalidating the entire cache is not «we will answer more slowly», it is an outage.

k hash functions and a bit array of length m. On insertion we set ones by the hashes, on a check we look at the same positions. The probability of a false positive is \((1 - e^{-kn/m})^k\), the optimum is \(k^* = (m/n)\ln 2\), and at it exactly half the bits are occupied.

The curve over k is U-shaped: at m = 8000 and n = 1000 one hash gives 11.75%, six give 2.16%, twelve give 4.83% again. Too few hashes and there is little discriminating power; too many and the filter fills up.

In recommendations it is for pagination. The system is unstable between requests, so naive pagination produces duplicates; instead a serialised filter with what has already been shown is passed between the frontend and the backend. Bloom in particular fits because there are no false negatives: an error is possible only towards «we will hide something extra», and a duplicate is never shown.

To hold the share of a category in the output when demand changes. A hard quota inserts items at fixed positions and destroys the sense of the output; a controller instead tweaks a bonus to the score, and the output stays sorted by meaning.

The key point is why the integral part is needed. A proportional controller produces an action proportional to the error, so at zero error there is no boost either: the system lives with a permanent shortfall. In the simulation, at I = 0 the share stabilises at 12% instead of 30% — a miss of 17.7 pp, and raising P to 2 only reduces it to 12.7 while adding oscillations. Steady-state error is removed exclusively by I.

The flip side is that too large a P makes the loop swing: at P = 3 the run-up reaches 58% against a target of 30%.

Degrade step by step rather than return a server error. The blender is in pain — switch off the late re-ranking stages and take fewer candidates. Candidate generation is in pain — an emergency ANN or popularity right at the blender. The feature service is in pain — answer with the ANN scores. Everything is in pain — the pumpkin.

Why this is not a rare case: a chain of five services at an availability of 0.999 each gives 0.995, that is, 43.7 hours of downtime a year. And splitting into services worsens this arithmetic — flexibility of scaling is paid for with the number of places that can break.

The transitions between steps have to be thought through in advance, and switched on automatically — with circuit breakers and a tie to utilisation.

A deliberately unintelligent but definitely working mode: popular goods computed once, popular by segment, editorial lists, simple business logic with no personalisation.

The rules: the dumber the pumpkin, the better; for a hard incident it is better to fix a huge list of items in the config so that they are certain to fly through every filter; it is useful to have several pumpkins in sequence; and build them outside the main service — if it is down, it will not serve even a pumpkin.

The main thing: if you have not tested your pumpkin in a long time, you most likely no longer have one.

Besides the ordinary things (latency by percentile, errors, timeouts, cache hits, queue lags) — the specific ones: the shares of the candidate sources, the share of fresh items, the share of fallback answers for each fallback, the coverage by categories, the age of the model, the index and the features, feature coverage and drift.

And the main abstraction is to monitor the whole funnel: how many candidates were pulled up, how many survived the filtering, how many were scored, how many made it into the answer, how long each step took. Such monitoring localises the problem at once, whereas a metric of the final output only says «it got worse».

With five steps in this order: goal setting (business metric → offline metric → the model's target), constraints (catalogue, users, SLA, filtering, intent, biases), the technical architecture, the ML architecture, the experiment setup. Then criticism of what came out and a second pass.

The justification is Gall's law: a complex system that works has always evolved from a simple one that worked, while one designed from scratch does not work and cannot be patched. And the simple solution is needed not as a fallback but as an instrument of understanding: we will criticise on the basis of what we came to understand while building it.

The first things I will ask before any architecture: which business metric, how rare the target event is, the size of the catalogue and the response budget.

Because a crude mistake at this stage will not be rescued by any model below: if the wrong thing is being optimised, the quality of the implementation does not matter.

And it is exactly the stage one wants to skip: many people are impatient to start doing things, and usually that is a useful trait — which is why it is dangerous precisely here. A partial compensation is to build the simple solution first, to reach something working quickly and come back to the goal already understanding the problem.

Compute it. If the ranking model spends around 4 µs per candidate and the budget is 300 ms, a full scan fits up to roughly 75 thousand candidates. A catalogue of 3000 items is scanned in 12 ms — no funnel is needed at all. A catalogue of 500 thousand requires a selection by 7 times, hundreds of millions by thousands of times.

And it matters that refusing the funnel on a small catalogue is not a simplification but the right decision: any funnel loses candidates, and the ceiling of quality is set by its first stage.

Because the event is too rare. The number of observations for a fixed relative effect is proportional to (1−p)/p, so catching the same gain at a conversion of 0.5% instead of 40% costs 133 times more traffic: 1.25 million per group against 9.4 thousand.

A proxy metric is needed — sensitive in a short A/B and correlated with the long-term goal. It is built like this: choose base metrics with a plausible long-term effect (watching series, discovery, frequency of use), assemble candidate combinations out of them, select the best retrospectively and always validate on a golden set of past experiments with a known outcome. The last step is the real validation.

With a 3 + 3 grid. Write out the actors — usually the user, the item and the context — and invent features for each one separately and for each pair.

For example: the user — purchase frequency and average basket; the item — sales and CTR over X days; the context — day of the week and the embedding of the surface; user × item — the dot product of the towers and how many times they took this category; user × context — how much they usually buy at this hour; item × context — the good's CTR at this time.

The grid almost always covers what is essential, and it is convenient to speak through out loud as the structure of the answer.

The situation where two surfaces show one user an overlapping assortment and their gains do not add up. If each in isolation gives +10, then at half overlap together they give +15 rather than +20; at full overlap +10, that is, the second adds nothing.

A classic reporting trap: the teams honestly show the gains of their surfaces while the total turnover does not grow. It is cured by metrics at the level of the user rather than the surface, and by an uplift formulation — a purchase on this surface relative to a purchase at all. And uplifts require a full shared feature layer.

Show the top of the popular, let them choose — and then the key trick: remove from the sample all the users who also liked what was chosen, and rebuild the top.

Why: without the removal the top barely changes and the next question brings no new information. In a simulation on five taste groups the naive scheme asks five times in a row about one and the same group — 1 of 5 covered; with removal all 5 are covered.

The general formulation: we choose the next question so that it splits the remaining uncertainty as much as possible, not so that it is the most popular.

The system stops being two-sided and becomes three-sided. Authors have their own expectations: they count on impressions, and their absence produces complaints. The instruments are item exploration inside the model, PID controllers to stabilise the volume of impressions, explicit quotas.

And the consequence people forget: authors are paid for impressions, so fraud appears — bots and inflated CTR. The target and the objective metric have to be antifraud-aware, otherwise the system will conscientiously optimise the inflation.

The orders of magnitude at a budget of 300 ms: network and parsing 20, the profile 30, candidate generation 40, filtering 15, features 60, ranking 80, re-ranking 25, the answer 20 — 290 in total, a margin of 10 ms.

Two conclusions. First: the margin will go on the tail of the distribution, so the budget is held at p99 rather than at the mean. Second: features and ranking eat 47% of the budget, and speed-ups should be looked for there first, not in candidate generation, where people look instinctively.

Three reasons. A different cost of error: ranking errs by a few percent of engagement, visibility errs on reputation and legal requirements; combining them into one score means mixing incomparable quantities. A different speed of change: visibility rules change within hours, the model is retrained on a schedule. Auditability: a rule can be read and disputed, a score lowered by a model cannot.

Additionally: lowering a score does not guarantee that the post will not be shown. If there are few competitors, a post with a lowered score will end up in the output anyway. A hard requirement of «do not show» is implemented only by a hard filter.

Because it is expensive and is called per «post and viewer» pair. Before ranking there are thousands of candidates, after the selection a hundred. Asking about a hundred is ten times cheaper.

The general rule is the same as in the chapter «The order of filtering»: cheap filters working on data that is already in memory (the age of a post, blocks, what has already been shown) go before ranking; expensive ones, requiring a trip to another service, go after. And one has to remember that filtering after the selection shrinks the output, so a margin is allowed for: the selection is made with a surplus.

It decouples the model from the product policy. The action weights are changed by config and rolled out by experiment, with no retraining. Plus rare negative signals such as a report get a head of their own and do not dissolve in a common loss.

The price: the heads compete for a shared body; rare tasks are prone to overfitting; the weights cannot be derived theoretically and are picked by A/B tests, which is slow. In more detail — the chapter «Multi-task learning» and page x05.

So that the score of a post does not depend on the composition of the batch. Otherwise the same thing with the same viewer gives different numbers, the score cannot be cached, the result is not reproducible, and debugging becomes impossible.

The flip side: the model does not see the whole slate and cannot account for the mutual influence of posts — for instance, that three posts in a row are about the same thing. That task is solved separately, by re-ranking after the scoring. That is, listwise effects are not lost but moved into a separate step where they are easier to control.

Two arguments. Memory: a vocabulary over all the posts and all the authors does not fit, while a hash table has a fixed size. Freshness: a new post is representable immediately after publication, because its hash is computed on the spot, whereas a vocabulary would have to be rebuilt.

The price is collisions. Not just any are dangerous, but collisions of two frequent values: their embeddings get glued together. It is cured by several hash functions — then indistinguishability requires a coincidence in all the tables at once. In detail and with an interactive — the chapter «Categorical features and memory».

A trick question, and the right answer is «neither on its own». The pipeline code shows which decisions were made and where the levers are. The model weights show what the system learned from the data. Only the first is open, and that is a fundamental limit on transparency: one can check that a rule exists and how it is written, but not what the model learned from the logs.

That is exactly why the code release comes with a tool that shows a user the labels on their account: the code plus observable outputs give more than the code alone.

Query hydration → candidate sources → candidate hydration → cheap filters → scoring → selection of the top-K → expensive hydration → expensive filters → the answer → side effects in the background.

The key justifications for the order: sources do not depend on each other and go in parallel; filters do depend and go in turn, from cheap to expensive; everything that requires a trip to another service per «object and user» pair is placed after the selection, because there are an order of magnitude fewer objects there; everything the current request does not need, but the next one does, goes into the background after the answer has been sent.

The request should complete on the remaining sources. A recommendation is not a transaction: an incomplete feed is better than an error. In the code we looked at this is literally one line: the results of the sources pass through flatten, which discards the errors.

But a mandatory addition: the degradation has to be visible. Metrics per source are needed — how many times it was called, how many times an error came back, how many candidates were returned. Without that, a source failing on a few percent of the traffic will never be discovered, while some of the users will systematically get a worse feed.

For experiments. The method receives the request and decides by it — inside, it reads configuration parameters, and those are rolled out to a percentage of the traffic. In this way any stage can be switched on or off for part of the users without branching the code and without shipping a new version.

The second use is conditional logic: a source of topics only works for topical requests, a video filter only when the client asked to exclude video. Instead of branches inside the stage, the condition is lifted to the level of the framework, where it also lands in the metrics automatically.

Because some hydrators depend on the results of others. Mutual follows cannot be computed without having the list of follows; the sequence of actions for the model is built taking the already shown posts into account.

Inside a round the hydrators are parallel, between rounds there is a barrier. This is a simplified substitute for a dependency graph: a real graph would require the links to be described explicitly, while two levels cover almost every case and require nothing beyond putting a hydrator into the right list.

The measurements have to be built into the framework rather than placed by hand across the stages. In the code we looked at, every type of stage has a run wrapper that calls the substantive method and along the way records the time, the number of input and output candidates, and the name of the stage.

From that, the answers to the two main operational questions come for free: where the bottleneck in time is and which filter drops how much. And under parallel execution it is the worst stage that has to be optimised, not the average one — the total time equals the time of the slowest.

Each has its own blind spot. The retrieval model looks for things similar to the user's history, but its index is refreshed only when a checkpoint is saved — the very freshest posts are not there. The source of posts from follows sees only the follows. The cluster source leans on the structure of the graph and knows nothing about the new, because the clusters are recomputed once a week.

Remove any one and you get a class of posts the system stops finding in principle. This is exactly the argument about multi-source retrieval from the chapter «Kinds of candidate generators»: different generators are optimised for different types of connection, and their union covers more than the best of them.

Do not search, but keep them in memory laid out by author. In the system we looked at this is a separate service: a map «post identifier → compact record» plus indexes «author → the queue of their recent posts». A request comes down to walking the list of follows and reading ready-made queues.

Three decisions make that possible. No text at all — only identifiers, times and flags, otherwise the memory grows by an order of magnitude. A retention window — the old is cleaned out by a background task. Separate indexes for original posts, replies with reposts, and video — so that filtering is a choice of structure rather than a pass with discarding.

It is filled by a stream of events, not by a periodic dump: the delay from publication to appearance in memory is seconds.

It is a clustering of the social graph: 20 million notable accounts are laid out across 145 thousand communities by sparse binary factorisation of the similarity graph. An author gets a set of communities they are known for; a user gets a set of communities they are interested in, as a sum over those they follow; a post gets a vector over communities.

The differences from ordinary factorisation: the values are binary rather than real; there are few non-zero coordinates; and, most importantly, a coordinate is interpretable — it is a particular community with members that can be looked at with your own eyes. Besides, what is factorised is not the interaction matrix but the similarity graph of accounts.

For stability and explainability. A slowly changing source is insurance for the case when the model starts behaving unexpectedly after retraining: part of the feed stays predictable.

Explainability matters no less: «this post is popular in a community you belong to» can be shown to a user and presented to a regulator, while «the dot product of the embeddings is 0.83» cannot.

The price is a week's delay: a new account will not get into the clusters at once. That is why the source is not the only one.

Only if the score depends on the «user and post» pair and on nothing else. In the system we looked at that is guaranteed by the attention mask: candidates do not see each other, so the score does not depend on the composition of the batch. Thanks to that, the pipeline has a separate source returning already ranked posts from a cache.

The second condition is that the features must not change faster than the cache lives. That is why the age of a post is fed to the model with a granularity of an hour: a continuously growing value would devalue the cache every minute.

The practical meaning: a person scrolls the feed, every screen is a new request, and recomputing the model on the same candidates makes no sense.

Three reasons. Cold start: a new user would get a random vector, whereas a history of a single action already gives a meaningful representation. Freshness: a learnable vector reflects the user as of the retraining, the history reflects them right now. Memory: a table of hundreds of millions of rows that has to be synchronised between replicas.

In essence the model is moved from transductive to inductive: it can compute a vector for a person it did not see during training. The price is that stable long-term interests not manifested in recent actions are lost. For a feed where content lives for hours the trade-off is justified.

It is a code from residual quantisation: a multimodal embedding of the post is encoded by six levels of 256 codes. At each level the nearest codebook vector is taken, subtracted, and the residual is encoded by the next level.

The difference from a hash is fundamental. A hash is arbitrary — two posts about the same thing land in unrelated rows, and the model has to learn each separately. A semantic ID is derived from the content, so similar posts share a prefix, and the knowledge accumulated on some posts transfers to new ones with the same prefix. Besides, a new post is given a code immediately after publication, with no retraining.

They have different distributions. In-batch ones are other people's positives, that is, a distribution proportional to popularity: the negatives come out hard but systematically biased towards the head. Global ones are sampled from the whole corpus and cover the tail, which practically never lands in a batch.

Using only in-batch means teaching the model to tell popular things apart from one another and to know nothing about the tail. Only global means getting negatives that are too easy and a weak gradient. In this config in-batch plus 64 global per example are used, and each kind has its own LogQ correction.

In-batch negatives come from the popularity distribution \(Q\). Without a correction the sampled softmax converges to \(\log p - \log Q\), that is, the model systematically under-rates the popular. The correction subtracts \(\log Q\) from the logit and gives back \(\log p\).

A scale of 2.0 instead of the theoretical 1.0 is an over-correction. The motive: unbiasedness is achieved with respect to the distribution that produced the log, and the log was produced by a previous version of the system, which itself liked the popular. The over-correction crudely presses that inherited skew down and pushes retrieval into the tail; the risk is small, because ranking works afterwards.

Because retrieval and ranking have different tasks. Retrieval has to sketch out the area of interests roughly — what matters here is recall, not subtlety. Ranking will then work out what inside that area is better.

Using every signal at once at the retrieval stage would mean mixing things that mean different things (a click and a completed view say different things) and complicating the task where the complexity does not pay off. Besides, a like is the most frequent explicit signal, that is, it gives the most training examples.

Note the filter separately: a positive counts only if there was no negative action on the same post, including «did not dwell». That cuts off clickbait — a post that was liked and immediately scrolled past.

For consistency. The weights of the tower and the index are always from one moment of training — a situation where the user vector is computed by a new tower while the index was built by an old one is impossible. A dot product between inconsistent representations is meaningless, and that is a classic source of silent quality degradation.

The side benefits: one artefact instead of two, rolling the model back rolls the index back, and the start-up of the serving does not require hours of re-indexing. The price is that the index is refreshed only on a save, so fresh posts come from other sources.

The dot product becomes a cosine. The first consequence is that popularity bias goes away: in the unnormalised case the length of a vector learns popularity, and the dot product systematically prefers the popular regardless of direction.

The second consequence is engineering: on the unit sphere \(\|a-b\|^2 = 2 - 2\langle a,b\rangle\), that is, maximising the dot product reduces to finding the Euclidean nearest neighbour. Any approximate-search library can be used without requiring MIPS support from it.

The price is that the signal the norm carried is lost. After normalisation there is nothing left to tell a reliable, many times confirmed vector from the noisy vector of a cold item.

By one condition in the attention kernel: a query sees a key only if the key belongs to the history or the key is the query itself. Candidates physically cannot look at each other; the «candidate × another candidate» blocks are not even loaded into the computation.

It gives four things. Cacheability: the score depends only on the (user, post) pair. Reproducibility: the order does not depend on the split into batches. Explainability: the reason for a position does not contain the words «another post happened to be next to it». Speed: the quadratic term in the candidates disappears.

The price is that the model does not see the slate and cannot account for the mutual influence of posts. That task is moved into a separate re-ranking step after the scoring.

Because «there was no conversion» on a post without a click does not mean a refusal — the user had no opportunity to perform the action. Training on such examples teaches the model that a lack of opportunity equals a negative answer, and produces systematic under-estimation.

This is the classic sample selection bias, the same one ESMM was invented for. Here the path of masking the loss is chosen: the head gets a gradient only on the subsample with a click, while the shared body of the model sees all the examples, so the representations are learned on the full data.

The frequency with which they receive a gradient differs by orders of magnitude. A dense matrix is updated on every example of the batch; a particular row of the embedding table only when its item lands in the batch.

So the embeddings need a large step and adaptation per row — row-wise Adagrad at a rate of 0.28, while the dense part is trained with Muon at 7.1e-4, almost four hundred times smaller. A single optimiser would either strangle the embeddings with a small step or wreck the dense part with a large one.

A separate subtlety is the half-life of Adagrad's statistics of 2500 steps: without forgetting, the denominator grows monotonically, the effective step goes to zero and the model stops adapting to changed behaviour.

Dwell time, time after a click, active seconds — eight continuous quantities. They cannot be forced into a binary formulation without losing the main thing: the difference between «watched for three seconds» and «watched for thirty». Any threshold erases that difference.

The price is a heavy-tailed distribution and the need to reconcile the scale with the probabilities in the common sum. The first is cured by metrics robust to outliers, the second by a small weight on the continuous term in the score formula.

Because an hour is an artificial discretisation of continuous time. The difference between 13:59 and 14:01 is nil for behaviour, but for the feature it is a transition between values, and the model learns sharp boundaries at round hours — an artefact of the encoding, not a property of the users.

A ten-percent jitter blurs the boundaries and forces a smooth dependence to be learned. The same motive as in piecewise-linear encoding of continuous features, only the trick is cruder and more universal: it does not require changing the architecture.

Because of the number of objects each has to process. The retriever produces embeddings for 28 million posts; making its tower more expensive is multiplied by that corpus. The ranker works with hundreds of candidates per request, and the cost of extra width is incomparably smaller.

This is the general principle of multi-stage systems we discussed in the chapter «The multi-stage funnel»: the further down the funnel, the fewer objects and the more expensive the model that is affordable. The selection stage is obliged to be simple, the ranking stage can afford complexity.

No. A leak would be if the model used information from the future relative to the moment of the prediction. Here the whole history is the past with respect to the candidates being scored; inside it, an event from the middle looking at a later one is entirely legitimate.

A causal mask is needed by autoregressive models predicting the next element of a sequence. Here the task is different: encode the history as a whole, and the prediction is taken from the candidate positions, which stand after all the history.

Because the weight is multiplied by the predicted probability of the action, not by the number of actions that happened. On an ordinary post the probability of a report is two or three orders of magnitude smaller than the probability of a like, so the actual contributions are comparable: roughly \(-0.012\) against \(+0.010\).

The ratio of 468 is reached exactly when the model considers a report as likely as a like. But such a post should go down — in that case the mechanism works as intended, not as an injustice.

Because further on it is multiplied by corrections smaller than one: the decay on the author, the discount for being outside the follows. Multiplying a negative number by 0.25 increases it — a penalty would turn into a reward, and a repeated post by a bad author would rise.

The mapping is arranged so that the order is preserved completely: negative scores are squeezed into \([0;\,0.000894]\), positive ones start at 0.001. A good post cannot end up below a bad one.

By a multiplier \(m(k) = (1-\text{floor})\cdot\text{decay}^{k} + \text{floor}\) with parameters 0.5 and 0.25, where \(k\) is the number of posts by the same author higher up by score. That gives 1.0, 0.625, 0.438, 0.344 and so on, with a floor of 0.25.

The advantage over a quota: it is a price, not a ban. A good enough fifth post by an author can still get through if it overtakes other people's even with a coefficient of 0.25. A hard quota would cut it off regardless of quality. The floor is needed so that an author does not disappear entirely — without it the multiplier would go to zero.

Because a follow is an agreement to read a person's posts, not everything they comment on and forward. Without such a discount the follow feed would fill up with other people's conversations that got there through a single acquaintance.

Formally it is the same multiplier of 0.75 as for posts outside the follows; in the code the condition is combined. It is controlled by a separate flag, which is on by default.

One post that passes the thresholds (few followers for the author, few impressions for the post, fresh) is lifted to the score of position 15–16. This is explicit exploration: we spend a position to get a signal about a post there is no data on.

Alongside in the code lies an alternative — choosing that post by Thompson sampling with a prior of \(\text{Beta}(0.75,\ 49.25)\) instead of «take the best by score». By default it is off, but the formulation itself is acknowledged as a bandit one outright. The prior corresponds to an expected response of about 1.5% with the strength of belief of 50 observations.

These are different kinds of uniformity. The decay keeps one person from taking over the feed. The DPP over embeddings keeps one topic from taking it over — even if the posts are written by different authors.

Besides, the DPP solves a task inaccessible to the model itself: the candidates in the transformer do not see each other, so the model fundamentally does not know that three posts are about the same thing. Listwise effects are moved into a separate step, where they can be governed by two parameters without retraining.

You will not know analytically — the weights are picked by experiment. That is the main price of a multi-task scheme: every change of the weights requires an A/B test, and there are not enough tests for the whole grid of values.

In practice people look not at engagement in the moment but at long-term indicators — return rate, time to the next session, the share of negative reactions. This is exactly the conversation about proxy metrics and their divergence from the real goal that was in the chapter «Proxy metrics and long-term goals» and the chapter «True relevance and its proxies». The very fact that the weight of a report is so large, and the sum of the negative weights exceeds the sum of the positive ones ninefold, is precisely an attempt to defend against optimising momentary engagement.

Along two axes at once. By cost: first those that need no external data, then those requiring a trip to another service. By rejection per unit of cost: if a filter is cheap and throws out a fifth, it should be at the front, to reduce the work for everything that follows.

Deduplication usually goes first of all: the sources work in parallel and return overlapping sets. Filters that fire only for part of the requests are reasonably placed closer to the end — most of the time they do nothing.

A separate boundary is the selection of the top-K. Everything that requires a request per «object and user» pair is placed after it: the difference between thousands of candidates and dozens of selected ones gives a saving of tens of times.

Because more filters work after the selection, and how much they will throw out is not known in advance. In the system we looked at, 50 are selected and 35 go to the screen — roughly a twofold margin against the rejection of the last stage.

If exactly 35 were selected, any firing of the visibility filter would leave the user with an incomplete screen. A practical rule: every new rule applied after ranking is a tax on the size of the selection, and it has to be paid for either by increasing K or by knowingly accepting the risk.

You cannot guarantee it with one mechanism — the recording of impressions happens after the answer has been sent, in the background, and may not happen at all. In the system we looked at, four independent mechanisms work on this task: a compact Bloom filter arriving with the request, two filters from different journals of impressions, and a separate filter over what has already been served in the current scrolling session. Plus the list of what has been shown is passed into the source of posts from follows, and it does not return them at all.

The logic is that the data paths differ: for a duplicate to break through, all of them have to fail at once. The price is small, while a repeat impression is noticed by the user instantly.

By observation — you cannot. The share of engagement collected by, say, replies over-states their contribution: remove them and you free up positions that other posts will take, and part of the engagement will flow there.

An experiment is needed: remove that type for part of the traffic and compare. In the code we looked at there is a separate filter for this, deterministically hiding a set percentage of posts. Two important details of the implementation: the decision depends on a hash of the «post and viewer» pair, so it is stable — one and the same post does not flicker between requests; and it is personal — the post is hidden for a particular person rather than globally, so the post's overall statistics are not distorted.

What is unusual here is that the unit of randomisation is a «user and object» pair rather than a user. That makes it possible to measure the value of a type of inventory rather than of a piece of functionality.

We are, and knowingly. It is a product decision about what a news feed is: content older than two days has no place in it, however good it may be.

Engineering-wise the decision has pleasant consequences. It bounds the volume the service of fresh posts has to keep in RAM. It gives an upper bound on the size of the journals of impressions. And it is almost free: the creation time is sewn into the post's identifier, no data has to be requested — which is why the filter stands third, right after the deduplication.

The flip side is that the system is structurally incapable of showing you a good post from a week ago. For a feed that is right, for a video service, say, it would be a catastrophe. A threshold of this kind always follows from the product rather than from the technology.

The intermediate option — show behind an interstitial — hands the decision to the user and sharply lowers the cost of a classifier's error. In a two-answer system a false positive means either a hidden normal post or a shown unacceptable one; with an interstitial it costs one extra tap.

The practical consequence: the detector's threshold can be set more aggressively and catch more without paying for it in user irritation. That is, the third answer is not about the interface but about the operating point on the precision-recall curve.

Apply them in different situations, going by the asymmetry in the cost of an error. In the system we looked at, only the precise detector is applied to posts from accounts the viewer follows: a false positive is expensive here, the person chose to read this author themselves. To recommendations from unfamiliar accounts the high-recall detector is added as well: a false positive is almost free, because the user was not expecting this post, while letting through spam that the system itself pushed at them is noticeable.

Instead of choosing one point on the curve, two are taken and applied where each is appropriate. An important detail: the additional set of rules can only forbid — strictness grows as you move away from the circle of follows, but never falls.

In the system we looked at there are three, and they lean on fundamentally different data. By other people's reaction: the ratio of reports to likes over 30 days — a ratio precisely, otherwise large accounts would be punished for their size. By the structure of the graph: PageRank over follows and interactions, then the logarithm of the mass. By the account's own behaviour: a transformer over the sequence of its actions.

Several are needed because each has its own vulnerability. Other people's reaction is faked by coordinated reports. The graph is faked by inflating followers, though badly — PageRank takes the weight of the source into account, and the mass of bots is close to zero. Behaviour is faked by imitating a human rhythm, which is expensive at scale. Deceiving all three at once is considerably harder than any one of them.

Architecturally they are one and the same: a transformer over a user's sequence of actions. The difference is only in the target variable: the recommender model predicts what the person will do next, the detector whether this is a person at all.

The common intuition is one too: a sequence contains information that aggregates destroy. For recommendations that is the order and the tempo of interests, for detection it is the rhythm: even intervals, no pauses for sleep, repeating chains of actions. Every individual action of a bot looks normal; what gives it away is precisely the sequence.

Four reasons, and all are visible in the code. A different cost of error — lowering a score and forbidding an impression are incomparable in their consequences. A different speed of change — the rules change under legal requirements within hours, the model is retrained on a schedule. Auditability — a rule can be read and disputed, a lowered score cannot. A guarantee — lowering a score does not guarantee that the post will not be shown: if there are few competitors, it will end up in the output anyway.

The last point is often missed, and it is the decisive one. A hard requirement of «do not show» is implemented only by a hard filter; a soft demotion gives no such promise.

As a separate stage after ranking, by rules rather than by a common score. The reasons: the score of a post and an advertiser's bid have no common unit of measurement, and the conversion coefficient would have to be assigned arbitrarily; ads carry contractual obligations that cannot be outbid by a score; the share of ads is a strategic decision that cannot be handed to a model.

In the system we looked at, the number of ads is the minimum of three constraints: how many there are, what the spacing step allows, and how many brand-safe posts there are in the feed, the last of which gives a ceiling of one half. Plus a threshold: if there are fewer than five posts, there are no ads at all.

To honour the obligations to advertisers about adjacency and at the same time to tie monetisation to the quality of the output. The mechanism is hard: bad content reduces not the placement but the total number of ads placed.

Economically that means a feed with problematic content automatically loses the ability to earn. Such a feedback loop works more reliably than any declaration, because it is built into the code rather than into a policy.

Not by score but at a fixed position with a fatigue mechanism. In the system we looked at the position is the sixth — within the first screen but not in first place — and the interval before showing it again is 30 hours.

Its not being a multiple of a day is no accident here: an interval of 24 hours would tie the impression to one and the same time of day, while 30 hours give a drift, and the block lands in different contexts. The sense of the restriction is that the module's value lies not in the frequency of impressions but in it working at least once; a daily impression would breed blindness.

Everything the current request does not need but the next ones do: recording the impressions, updating the caches, publishing the training logs, metrics, events for billing. In the system we looked at there are thirteen such tasks, and they start in the background without waiting for a result.

A mandatory consequence to keep in mind: since the result is not checked, all of it is unreliable. That is why several independent mechanisms are kept as a guard against a repeat impression. And for the training logs there is no reliable solution at all — losing them biases the sample for future models and is invisible in any online metric.

Everything that may become the subject of an experiment is moved into parameters read from an external system per request rather than globally. Then two simultaneous requests may have different values of one parameter, and that gives the split into groups with no branching in the code.

Second: every stage of the pipeline gets a switch that reads a parameter. That makes it possible to experiment not only with values but with the presence of a stage — a new source or filter is switched on for a percentage of the traffic.

Third, and often forgotten: the measurements have to be taken at the level of the framework, otherwise along with every experiment you will have to add the measurement of its effect by hand.

The price of the approach is that the discipline shifts from the code into the process of working with the configuration: a change to the product's behaviour no longer passes code review.

On its own — little: you would see the structure and not the numbers. That is why the repository we looked at states separately that the defaults are brought to the main production values by a script on a schedule.

It is important to understand correctly what that gives. The formulation «this is how it works for most users as of the last synchronisation» is true. The formulation «this is how it works» is not: part of the traffic is in experiments, and between a change in production and its appearance in the code there is a lag.

A good example is right in this repository. In the summer of 2026 a boost was introduced to the weight of the probability of a reply for posts from people with a mutual follow: first a value of 20, then lowered to 15.

The reason for the reduction was not the metrics. Judging by the description they were good: people were talking with people they know more actively. The problem surfaced from feedback: during the World Cup users started seeing too little discussion of the event, because many relevant posts were written by accounts they do not follow, and strengthening mutual follows automatically weakened everything else.

The moral is exactly the one from the conversation about proxy metrics: the optimised quantity was growing while the product's function — to show what is happening in the world — was degrading, and no engagement metric showed that.

A zero usually means one of three things, and telling them apart matters. The mechanism was tested and did not roll out — that is the case in the repository we looked at with the boost to the weight of dwell time: it was tested in the same experiment as the boost to replies but was not rolled out widely. The head is trained but does not enter the score — as with a click on a profile: the prediction is computed, the weight is zero. The mechanism is waiting to be switched on — as with the inventory holdout with its percentages at zero.

What the three cases have in common is that a zero is the position of a knob, not the absence of a knob. The code of the mechanism is written, tested and ready; tomorrow the value may become non-zero without a single line of change.

With two separations that determine everything else.

First: ranking apart from visibility. The order is decided by the model, the right to be shown by a separate service with rules. A different cost of error, a different speed of change, a different auditability. And most importantly: lowering a score does not guarantee that a post will not be shown.

Second: posts apart from non-posts. Ads, blocks of account recommendations and promos do not take part in the common ranking — they have no common unit of measurement with the score of a post, and their share is a business decision.

After that the standard funnel: several sources with different blind spots → hydration → cheap filters → the model → a selection with a margin → expensive filters → blending. And side effects after the answer.

Not by learnable vectors. The user is the output of a transformer over the sequence of their actions: that works from the very first action and updates in real time. The item is a code derived from the content, for example residual quantisation of a multimodal embedding: a new post gets a representation right after publication, and posts close in meaning share a prefix of the code, so knowledge transfers.

The general rule: a learnable vector is justified when the object lives longer than the retraining period. If it lives for less, it does not manage to be learned and is useless.

By a weighted sum with the weights in the configuration rather than in the loss. That decouples the model from the product policy: changing the priorities becomes changing a number and rolling out an experiment rather than retraining.

Three things worth saying separately. The weights are multiplied by probabilities, not by counters, so no conclusion about influence can be drawn from the ratio of the weights. The result is worth bringing into the non-negative range if it is afterwards multiplied by corrections smaller than one. And the weights cannot be derived theoretically — only by experiment, which is the main price of the scheme.

By a soft multiplier with a floor rather than by a quota: \(m(k) = (1-\text{floor})\cdot\text{decay}^k + \text{floor}\), where \(k\) is how many posts by the same author already stand higher. At the values 0.5 and 0.25 that gives 1.0 → 0.625 → 0.438 → 0.344 with a floor of 0.25.

The advantage over a quota: it is a price, not a ban. A good enough fifth post by an author will get through if it overtakes other people's even with a coefficient of 0.25. The floor is needed so that the author does not disappear entirely.

And it is worth saying separately that this is only one axis of diversity. Uniformity by topic is solved by a different mechanism — selection through a determinantal process after the scoring.

Give exactly one of their posts a guaranteed position in the middle of the output rather than a multiplier on all of them. The implementation: out of the suitable posts (few impressions, few followers for the author, fresh) the best by score is taken, and its score is set equal to the score of position 15–16.

Two properties make that manageable. A guarantee of a position rather than of an increment: the result is predictable and does not depend on who else is in the output. Exactly one post: the price of the mechanism is bounded above by one position regardless of the number of newcomers.

In substance this is exploration: we spend a position to get a signal about a post there is no data on. Alongside in the code lies a Bayesian variant of the choice through Thompson sampling with a prior of Beta(0.75, 49.25), switched off by default.

You have to start from the fact that the recording of impressions almost certainly happens after the answer has been sent, in the background, and its result is not checked. So the impression may not have been recorded: a restart of the service, an overflowing queue, a failure of the store.

Hence the practice: several independent mechanisms with different data paths. In the system we looked at there are four — a compact filter arriving with the request, two journals of impressions from different stores and a session list. Plus the list of what has been shown is passed into the source of posts from follows.

The diagnosis accordingly: compare what was recorded in each of the journals and look for a discrepancy rather than looking for a bug in the filter.

Only by experiment, not by observation: by removing a type of content you free up positions that other posts will take, and part of the engagement will flow there. The observed share over-states the contribution.

The mechanism: deterministically hide a set percentage of such posts and compare. Two important details — the decision has to depend on a hash of the «object and user» pair in order to be stable between requests, and it has to be personal so as not to distort the object's overall statistics.

The unit of randomisation here is a «user and object» pair rather than a user. That makes it possible to measure the value of inventory rather than of functionality.

Admit that the metric measures something other than what the product exists for, and look for which function of the product degraded.

A concrete case from this repository: strengthening the weight of a reply for mutual follows gave good engagement metrics — people were talking with people they know more actively. But the complaints were about something else: during a major sporting event the feed showed little discussion, because the relevant posts were written by accounts they do not follow, and strengthening one group automatically weakened the others. The value was lowered from 20 to 15.

The general conclusion: strengthening any group is a weakening of all the others, and it becomes noticeable where there are no metrics. It is useful to keep metrics of coverage and diversity alongside metrics of engagement — those are exactly what catch such shifts.

A question about maturity of judgement, and answering «everything is right» is bad. Three substantive directions.

Silent degradation. Source errors are swallowed silently. That is the right decision in substance, but it requires metrics and alerts on every source — otherwise a failure on a couple of percent of the traffic will not be discovered. The open code shows that measurements exist; whether such failures are visible through them cannot be judged from the repository.

The reliability of the training logs. The events for future models are written by the same unreliable background means as everything else. Losing part of the logs biases the sample and is not diagnosed by online metrics. Here I would separate logging for the product from logging for training, giving the second delivery guarantees.

The absence of explicit position debiasing. The model is trained on the logs of its own output, where the upper positions get more attention simply because of position. No explicit correction is visible in the open code — perhaps it is in the unpublished part, but if not, that is a noticeable source of bias.