RecSys Supplement · The X algorithm · page 11 of 11
What this code teaches about the theory
We have gone through the system along the path of a request. Now let us put it back together — by topic. For each one: what the theory says about it, how it is done in production and, most importantly, where practice diverges from theory and why. The divergences are more interesting than the matches.
In brief
- Almost all of the theory turned up in live code — two-tower retrieval, semantic IDs, hash embeddings, multi-task learning, DPP, a Bloom filter, PageRank, smoothed ratios.
- The main divergence is about embeddings. The course teaches representing a user by a learnable vector; here there is none at all.
- The second is about listwise ranking. The course says seeing the whole slate is useful; here that is deliberately forbidden for the sake of cacheability.
- The third is about LogQ. Theory gives a correction scale of 1, production has 2.
- What is not in textbooks at all: the safety loop, the report to the user, the synchronisation of the configuration with the open code.
1. Topic by topic
| Topic | How it is in X | Where we discuss it |
|---|---|---|
| The multi-stage funnel | Sources → hydration → filters → scoring → a selection of 50 → filters → 35 to the screen | x01, x06 |
| MIPS and similarity measures | The dot product of normalised vectors, that is, the cosine | x03 |
| Matrix factorisation | Not used for the feed; its role was taken by a two-tower transformer. A related idea lives in SimClusters — as sparse binary factorisation of the graph | x02 |
| Kinds of candidate generators | Five sources with different blind spots: the memory of follows, retrieval, clusters, topics, the cache | x02 |
| Recall@K of candidate generation | Not explicitly measured in the open code; instead, source limits as a product decision | x02 |
| Ranking metrics | The training code has NDCG, AUC, calibration, relative cross-entropy | x04 |
| The problems of implicit data | A positive counts only in the absence of a negative. Clickbait is cut off by the «did not dwell» label | x03, x04 |
| MMR and DPP | DPP over embeddings in a separate service, θ = 0.65, the first 150 positions are rearranged | x05 |
| Biases and the feedback loop | An over-corrected LogQ, the out-of-network discount, the boost for a little-known author | x03, x05 |
| Two-tower models | Exactly so, including the normalisation and the cosine. A corpus of 28.7 mln, the index inside the checkpoint | x03 |
| Sampling negatives | In-batch plus 64 global per example, each kind with its own correction | x03 |
| The LogQ correction | Present, with a scale of 2.0 instead of the theoretical one | x03 |
| Learnable embeddings | Absent for the user and for the post alike. Only the history and the semantic ID | x03 |
| Inductive bias, cold start | The model is inductive by construction; plus in 10% of the examples the history is zeroed out deliberately | x03 |
| Hashing categories | Two hash functions per entity, a shared table with zone offsets | x03 |
| Unified Embedding | That is exactly what it is: indistinguishability requires a collision in both tables | x03 |
| Multi-task learning | A head per action, combined by weights outside the model | x05 |
| ESMM and sample selection bias | A loss mask: the conversion heads learn only on clicked posts | x04 |
| Position debiasing | No explicit correction is visible in the open code; there are, however, the out-of-network discount and the decay on the author | x05 |
| Two loops, the lambda architecture | Online: the memory of follows from Kafka. Offline: the clusters and the reputation once a week | x02, x07 |
| Logging and its gaps | Side effects in the background with no check of the result — hence the fourfold guard against repeats | x01, x08 |
| The architecture of the runtime | Rust on the request track, Python for the model, Scala for the batches | x00 |
| The order of filtering | 18 cheap ones before scoring, 3 expensive ones after the selection | x06 |
| The Bloom filter | Arrives with the request as a hydrator, alongside three more mechanisms | x01, x06 |
| Resilience | Source errors are swallowed, the feed is assembled from the survivors | x01 |
| Transformers over history | A sequence of profile + 1022 events + 64 candidates, 8 layers, a width of 2560 | x04 |
| Bandits and Thompson sampling | Present in the newcomer boost with a prior of Beta(0.75, 49.25), but off by default | x05 |
| Experiments | 191 parameters, a switch on every stage, the history of a change day by day | x09 |
| Layers with different goals | Ads and modules are a separate pipe, rules instead of a common score | x08 |
| Semantic IDs | Residual quantisation of 6 levels by 256 codes on top of a multimodal embedding | x03 |
2. Five places where practice diverges from theory
Matches are useful but teach nothing. Let us take the divergences — in them you can see how practice differs from an exposition.
Divergence 1. The user has no embedding
The course says: represent the user by a learnable vector, the item by a learnable vector, and the score is their product.
In production: neither of the two. The user is the output of a transformer over their history, the post is a set of codes from residual quantisation.
The course starts from matrix factorisation, where learnable vectors are the very essence of the method. Production starts from the properties of the domain: content lives for hours, new users arrive every day, the catalogue does not fit into a table.
Both are right within their frames. A learnable embedding is better when the objects are long-lived and there are not many of them — a catalogue of films, categories of goods. It loses when the objects are ephemeral: by the time the table is updated the post is no longer relevant.
What to take away: the question «a learnable vector or a computed representation» is decided not by theory but by the lifetime of the object relative to the retraining period. If the object lives for less, a learnable vector is useless.
Divergence 2. Listwise ranking is forbidden deliberately
The course says: listwise approaches see the whole slate and are therefore stronger than pointwise ones.
In production: the candidates physically do not see each other — the attention mask forbids it.
The course compares by quality all else being equal. Production takes into account what is not in that comparison: cacheability, reproducibility and explainability.
Note that the listwise effects are not lost — they are moved into a separate re-ranking step. That is, what was chosen is not «pointwise instead of listwise» but a division: the model answers for the «user and post» pair, a separate service for how the posts look together.
What to take away: an architectural decision has dimensions besides quality. A score that depends on the composition of the batch cannot be cached — and the cache here became a full-fledged source of candidates.
Divergence 3. LogQ with a scale of 2
The course says: subtract \(\log Q\) and get an unbiased estimate.
In production: \(2\log Q\) is subtracted.
Unbiasedness is achieved with respect to the distribution that produced the log. And the log was produced by a previous version of the system, which itself preferred the popular. A model unbiased with respect to the log conscientiously reproduces the inherited skew.
What to take away: an «unbiased estimate» is always unbiased with respect to something. In recommendations that is «with respect to what we showed yesterday», and if yesterday's policy was biased, formal correctness does not save you.
Divergence 4. Diversity is brought in twice
The course says: for diversity, take MMR or DPP.
In production: both, but along different axes — the decay on a repeated author separately, the DPP over embeddings separately.
The course discusses diversity as one notion. In the feed of a social network there are two: not getting stuck on one person and not getting stuck on one topic. These are different problems with different mechanisms: the first is solved by a cheap multiplier right in the scoring, the second by an expensive determinant in a separate service.
What to take away: before choosing a method of diversity, the axis has to be named. «A diverse output» is not a statement of the task.
Divergence 5. The newcomer boost is not a multiplier
The course says: cold start is cured by priority — blending in, a boost, exploration.
In production: exactly one suitable post is taken and its score is set equal to the score of position 15–16.
A multiplier gives an unpredictable result: where the post ends up depends on who else is in the output. Setting it equal to the score of a target position gives a guarantee of a place rather than a guarantee of an increment. The product promise is formulated as «one position in the middle of the feed», and the code expresses exactly that.
Plus «exactly one» is an upper bound on the price of the mechanism: however many newcomers there are, the feed loses one position, not ten.
What to take away: when a requirement is formulated in positions, implement it in positions, not in multipliers on the score.
3. What is not in textbooks at all
| What | Why it matters |
|---|---|
| The safety loop — classifiers, account reputation, the rules engine, three kinds of answer | By volume of code it is comparable with the recommender core. In any public system it is half the work, and interviewers ask about it |
| The report to the user about labels | Transparency as part of the product rather than as documentation. Technically it rests on preserving the reason for the verdict |
| Synchronising the configuration with the open code | Opening the code and not opening the values is almost meaningless. A separate mechanism solves that |
| Measuring the value of inventory by a holdout on a «user and object» pair | Causal measurement with a unit of randomisation other than the user |
| Limiting ads by the quality of the output | An economic loop built into the code: bad content reduces the advertising inventory |
Interview questions about the system as a whole
Questions to which this system gives a ready and concrete answer. The value is that instead of «they probably do something like this» one can say «in the open code of X it is done like this, and here is why».
Design the feed of a social network. Where do you start?
With two separations that determine everything else.
First: ranking apart from visibility. The order is decided by the model, the right to be shown by a separate service with rules. A different cost of error, a different speed of change, a different auditability. And most importantly: lowering a score does not guarantee that a post will not be shown.
Second: posts apart from non-posts. Ads, blocks of account recommendations and promos do not take part in the common ranking — they have no common unit of measurement with the score of a post, and their share is a business decision.
After that the standard funnel: several sources with different blind spots → hydration → cheap filters → the model → a selection with a margin → expensive filters → blending. And side effects after the answer.
How do you represent a user and an item if content lives for hours?
Not by learnable vectors. The user is the output of a transformer over the sequence of their actions: that works from the very first action and updates in real time. The item is a code derived from the content, for example residual quantisation of a multimodal embedding: a new post gets a representation right after publication, and posts close in meaning share a prefix of the code, so knowledge transfers.
The general rule: a learnable vector is justified when the object lives longer than the retraining period. If it lives for less, it does not manage to be learned and is useless.
The model predicts a dozen different actions. How do you combine them into one score?
By a weighted sum with the weights in the configuration rather than in the loss. That decouples the model from the product policy: changing the priorities becomes changing a number and rolling out an experiment rather than retraining.
Three things worth saying separately. The weights are multiplied by probabilities, not by counters, so no conclusion about influence can be drawn from the ratio of the weights. The result is worth bringing into the non-negative range if it is afterwards multiplied by corrections smaller than one. And the weights cannot be derived theoretically — only by experiment, which is the main price of the scheme.
How do you keep one author from taking over the whole feed?
By a soft multiplier with a floor rather than by a quota: \(m(k) = (1-\text{floor})\cdot\text{decay}^k + \text{floor}\), where \(k\) is how many posts by the same author already stand higher. At the values 0.5 and 0.25 that gives 1.0 → 0.625 → 0.438 → 0.344 with a floor of 0.25.
The advantage over a quota: it is a price, not a ban. A good enough fifth post by an author will get through if it overtakes other people's even with a coefficient of 0.25. The floor is needed so that the author does not disappear entirely.
And it is worth saying separately that this is only one axis of diversity. Uniformity by topic is solved by a different mechanism — selection through a determinantal process after the scoring.
How do you support new authors without breaking the feed?
Give exactly one of their posts a guaranteed position in the middle of the output rather than a multiplier on all of them. The implementation: out of the suitable posts (few impressions, few followers for the author, fresh) the best by score is taken, and its score is set equal to the score of position 15–16.
Two properties make that manageable. A guarantee of a position rather than of an increment: the result is predictable and does not depend on who else is in the output. Exactly one post: the price of the mechanism is bounded above by one position regardless of the number of newcomers.
In substance this is exploration: we spend a position to get a signal about a post there is no data on. Alongside in the code lies a Bayesian variant of the choice through Thompson sampling with a prior of Beta(0.75, 49.25), switched off by default.
A user complains that they see the same post twice. Where do you look?
You have to start from the fact that the recording of impressions almost certainly happens after the answer has been sent, in the background, and its result is not checked. So the impression may not have been recorded: a restart of the service, an overflowing queue, a failure of the store.
Hence the practice: several independent mechanisms with different data paths. In the system we looked at there are four — a compact filter arriving with the request, two journals of impressions from different stores and a session list. Plus the list of what has been shown is passed into the source of posts from follows.
The diagnosis accordingly: compare what was recorded in each of the journals and look for a discrepancy rather than looking for a bug in the filter.
How do you measure how much value a particular type of content brings?
Only by experiment, not by observation: by removing a type of content you free up positions that other posts will take, and part of the engagement will flow there. The observed share over-states the contribution.
The mechanism: deterministically hide a set percentage of such posts and compare. Two important details — the decision has to depend on a hash of the «object and user» pair in order to be stable between requests, and it has to be personal so as not to distort the object's overall statistics.
The unit of randomisation here is a «user and object» pair rather than a user. That makes it possible to measure the value of inventory rather than of functionality.
The metrics went up but users are complaining. What do you do?
Admit that the metric measures something other than what the product exists for, and look for which function of the product degraded.
A concrete case from this repository: strengthening the weight of a reply for mutual follows gave good engagement metrics — people were talking with people they know more actively. But the complaints were about something else: during a major sporting event the feed showed little discussion, because the relevant posts were written by accounts they do not follow, and strengthening one group automatically weakened the others. The value was lowered from 20 to 15.
The general conclusion: strengthening any group is a weakening of all the others, and it becomes noticeable where there are no metrics. It is useful to keep metrics of coverage and diversity alongside metrics of engagement — those are exactly what catch such shifts.
What would you do differently from the way it is done in X?
A question about maturity of judgement, and answering «everything is right» is bad. Three substantive directions.
Silent degradation. Source errors are swallowed silently. That is the right decision in substance, but it requires metrics and alerts on every source — otherwise a failure on a couple of percent of the traffic will not be discovered. The open code shows that measurements exist; whether such failures are visible through them cannot be judged from the repository.
The reliability of the training logs. The events for future models are written by the same unreliable background means as everything else. Losing part of the logs biases the sample and is not diagnosed by online metrics. Here I would separate logging for the product from logging for training, giving the second delivery guarantees.
The absence of explicit position debiasing. The model is trained on the logs of its own output, where the upper positions get more attention simply because of position. No explicit correction is visible in the open code — perhaps it is in the unpublished part, but if not, that is a noticeable source of bias.
One-screen cheat sheet
Matched the theory
Two towers, semantic IDs, hash embeddings, multi-task learning, DPP, Bloom, PageRank.
Diverged: embeddings
There are no learnable vectors for the user or the post. The reason is the lifetime of the object.
Diverged: listwise
Forbidden deliberately for cacheability; moved into a separate re-ranking step.
Diverged: LogQ
A scale of 2 instead of 1 — against the bias inherited from the logging policy.
Diverged: diversity
Two axes, two mechanisms: by author with a multiplier, by topic with a determinant.
Diverged: cold start
A guaranteed position for one post instead of a multiplier for all.
Not in the theory
The safety loop, transparency as a product, the synchronisation of the configuration.
The main lesson
Architectural decisions are chosen not by quality alone: cacheability and explainability weigh no less.
The second lesson
Strengthening any group is a weakening of the others, and it shows up where there are no metrics.
Primary sources
- xai-org/x-algorithm — the whole repository, the walkthrough follows the commit
28e414f, Apache 2.0. - The pages of the supplement: the overview, the pipeline, the sources, retrieval, ranking, scoring, the filters, visibility, blending, the configuration.
- The course: all thirteen weeks, the metrics trainer, the widget catalogue.