Metrics: compute them by hand and turn the knobs

1. Take the preset «positives at the bottom» and lift both positives to the top with the arrows. Precision@K and Recall@K will not change at all — they do not see the order. RR, AP and NDCG will grow.
2. Reduce \(K\): recall falls, precision may grow — the classic trade-off.
3. The preset «one positive»: RR becomes exactly \(1/\text{position}\), and NDCG the same number, only with a logarithmic discount instead of a linear one.
4. Switch on graded relevance — now a click on a row cycles 0 → 1 → 2. Precision and recall simply binarise the grades, while NDCG really does tell «2» from «1».
5. Then switch to gain = 2^rel − 1. The numerator in DCG changes, and NDCG with it — under one and the same ranking. There is no single «correct» gain, it is a convention, and in an interview it is spelled out.
6. Note the names: there is one query here, so it is honest to write RR and AP. The letter M in MRR and MAP is mean over queries, and what you average over changes the result: a separate widget below.

1. Move the threshold: precision and recall go in opposite directions, while the ROC-AUC does not change at all — it does not depend on the threshold, it is a property of the ranking rather than of the decision rule.
2. Move the share of positives at a fixed model quality: from 5% to 60% the ROC-AUC travels 0.868 → 0.852 (sixteen thousandths), while AP collapses from 0.890 to 0.341. Hence the rule: on strongly imbalanced tasks you look at PR, not at ROC.
3. The tab «AUC as a share of pairs»: count the green cells and see that the share equals the ROC-AUC on the same points exactly.
4. The data here is deterministic: the slider picks points out of one and the same fixed pool rather than resampling. Otherwise the noise of the estimate (at 5% positives the standard deviation of the ROC-AUC itself reaches 0.145) would completely swallow the effect you are meant to see.

1. By headcount half the users are newcomers, while the queries from them are a few percent. So micro is almost entirely determined by the active ones.
2. Drop the metric on the newcomers to zero. Micro sags by hundredths, macro by tenths. A release that kills onboarding looks neutral by micro.
3. The converse is also true: macro over-rates the contribution of those who made one query and left. So both are looked at — and slices by cohort are cut separately.

What to say in an interview: «Averaging over queries measures the average query, averaging over users the average user. With a heavy tail of activity these are different numbers, and the degradation of newcomers is visible only in the second».

1. Both sliders are monotone transformations of the score. The order of the objects does not change at all, so the ROC-AUC stands dead still at one and the same number.
2. Meanwhile ECE, LogLoss and Brier wander by multiples. Compare the «average prediction» and the «observed frequency»: the discrepancy is the systematic error of calibration.
3. Press Platt scaling. ECE falls from about 0.16 to 0.02, LogLoss from 0.65 to 0.48 — while the AUC does not change by a single unit in the fourth digit. Calibration is fixed after training and does not interfere with the ranking.

What to say in an interview: «AUC is invariant to any monotone transformation of the score, so it says nothing about calibration. Where a prediction is multiplied by money — an auction, a budget, expected revenue — LogLoss, ECE and a calibration curve are needed».

1. The cloud of points is 200 independent runs. The mean of IPS lies exactly on the true value (it is unbiased), but the cloud is wide: on a single log you can be off by a factor of several.
2. Pull the divergence of the policies: the ESS falls from 400 to 25 out of 500 records, and the spread of IPS grows fourfold. That is what «the estimate falls apart under weak overlap» means.
3. Switch clipping on. At \(w \le 15\) the bias is −0.053, but the RMSE improves; at \(w \le 5\) the bias is −0.164 and the RMSE is already worse than with no clipping. The threshold has an optimum.
4. DM is biased but stable. DR usually wins on RMSE — it combines a reward model with a correction through the weights.

What to say in an interview: «IPS is unbiased provided the policies overlap and the propensities are known. Its trouble is variance: the weight \(1/p\) explodes on rare actions. It is cured by self-normalisation, clipping or DR, and every way pays with bias».

1. The MDE curve falls as \(1/\sqrt{n}\). To tell apart an effect twice as small you need four times as much data — that is the answer to «let us go and measure +0.1%».
2. Reduce the base conversion at the same relative lift: the required sample grows, because the absolute difference \(p_0 \cdot \text{lift}\) became smaller.
3. The tab «peeking» is an A/A test, there is no real effect at all. Checking once at the end gives an honest ≈5%, while checking daily over two weeks gives about 22%. Every new look is one more chance of crossing the threshold by accident.

What to say in an interview: «We fix the horizon in advance and compute the MDE before the start. If we have to look along the way, we take a sequential test or alpha spending, otherwise the nominal 5% turns into 20-plus%».

1. The teal points (with the correction) lie on the diagonal «learned = truth». The pink ones do not, and the bigger the point — that is, the more popular the item — the further down it has travelled.
2. Check the numbers: without the correction the correlation of the learned score with \(\log p - \log Q\) is ≈ 0.99. The model learned exactly what theory predicts for a sampled softmax without a correction — it is simply not what we need.
3. The rank correlation with the truth: ≈0.57 without the correction against ≈0.96 with it. In the top-5 of the model without the correction, rare items climb in.
4. Set \(\alpha = 0\) — popularity becomes uniform, \(Q\) a constant, and the correction stops changing anything. That is a check of understanding: the correction cures precisely the skew of \(Q\), not «noise».