RecSys · textbook
Trainer
Revision
About
All chapters
The catalogue
26 interactive widgets. Each answers a concrete interview question — not «show me something pretty» but «turn the knob and see why it is so». Every number in the captions comes from running the widgets themselves, and the formulas are checked against an independent implementation.
The things you are asked to compute and explain in almost every interview.
Sixteen types of task, a check of the answer with an explanation, statistics by type that finish off your weak spots, and a session mode of 10 tasks.
Rearrange the positives: precision and recall will not budge, while RR, AP and NDCG grow. That is the whole difference between metrics of coverage and metrics of order.
The threshold does not affect the AUC at all, imbalance barely affects it — while AP collapses from 0.89 to 0.34 because of it.
Any monotone transformation of the score leaves the AUC exactly the same while breaking the calibration. Where a prediction is multiplied by money, an AUC is not enough.
Newcomers are half the headcount and three percent of the queries. A degradation on them is visible only when averaging over users.
IPS is unbiased, but under weak overlap of the policies the ESS falls from 400 to 25 and the estimate becomes useless. Clipping has an optimum by RMSE.
The MDE falls as the square root of n: an effect twice as small costs four times as much data. Daily peeking turns 5% of false positives into 22%.
Candidate generators are compared not at an equal K but at an equal latency budget — otherwise different engineering solutions are being compared.
With a cosine the logits are squeezed into [-1;1], and without a temperature the softmax is almost uniform while the gradients are weak.
Real training of two models on one stream of batches: without a correction the score converges to log p − log Q and sinks the popular, with it to log p.
For ε-greedy the regret is linear because of the fixed share of random traffic, for UCB and Thompson it is sublinear.
How the systems themselves are built: from the long tail to semantic IDs.
At α = 1.5 the top 4% of the catalogue collects 93% of the views. Under a weak skew recommendations are not needed, under a strong one they degenerate into popularity.
End-to-end recall is the product of the stages’ recalls. What is lost at candidate generation, no ranker will bring back.
The RMSE on the held-out set has its minimum exactly at the true rank. A cold user gets zero factors and a prediction equal to the average rating.
The «dot or cosine» argument is an argument about whether the norm of the vector is needed. On normalised vectors all three measures give one order.
Two disjoint groups and two-dimensional embeddings. With negatives from impressions the groups overlap and half of the catalogue-wide top-5 belongs to the other group; with negatives from the catalogue it is zero. Inside its own group the «impression» model is even better, while over the catalogue it loses exactly half of the HitRate.
Recall grows from 0.10 to 1.00 as efSearch goes from 1 to 20, while the number of visited nodes goes from 25 to 68. Approximateness is a tunable parameter, not a failure.
The first percent of diversity costs 7% of relevance, the last 21%. The DPP tab shows the same through the determinant of the kernel: a set volume of 1.99e-2 against 4.15e-4 for the greedy top.
A catalogue of 40 items with a new item of the best CTR hidden in it. Greedy sorting never finds it; sorting by μ + α·σ finds it and gives +9% to the output, while too large an α spoils everything again. The premature abandonment of an item by a deterministic UCB is visible too.
The source of the negatives sets the model’s implicit prior: in-batch gives a 1.75× skew towards the popular, pure hard ones drag in false negatives.
It is not any collision that is dangerous but a collision of two frequent values. Several hash functions make the probability of full indistinguishability a product.
A raw feature gives only a straight line, one-hot gives steps, PLE gives a continuous piecewise-linear function with no loss of resolution inside a bin.
The errors are one-sided: if it said «not seen», it was definitely not seen. The curve over k is U-shaped, with the optimum at exactly half the bits filled.
Without the integral part the controller misses the target by 17.7 pp — at zero error it produces no action and cannot live without a miss.
The weights depend on the candidate, so the single relevant event of the history gets a weight of 0.835 instead of a uniform 0.083.
Every level encodes the residual of the previous one, close items share a prefix, and a new item gets an ID straight away from its content vector.