RecSys Supplement · The X algorithm · page 1 of 11
The open-sourced «For You» feed: what it is and how to read it
Theory explains why recommender systems are built the way they are. Here is a rare chance to look at how that looks when it is written in earnest and runs for hundreds of millions of users. X opened the code of its main feed: 2028 files, about 360 thousand lines of Rust, Python, Scala and Java.
In brief
- This is not a toy example. The repository holds production code: the feed pipeline, the ranking model together with its training, the serving, the filters, the visibility system. Not everything, but the core is real.
- The system is assembled on every request out of two things: posts from the accounts you follow, and posts the model found among those you do not follow. Both are ranked by one and the same model.
- Ranking and visibility are kept apart. Ranking decides the order. Whether a post may be shown at all is decided by a separate service under separate rules. These are two different services with different inputs.
- The model predicts not «relevance» but the probabilities of specific actions — a like, a reply, a repost, a report, a block — and they are combined into a single score by explicit weights that sit right in the code.
- Almost everything here is familiar theory. Two-tower retrieval, semantic IDs, hash embeddings, multi-task heads, DPP, a Bloom filter, PageRank. The difference is that here you can see what it all costs.
A map of the supplement
1. What exactly was opened
The phrase «X opened its algorithm» sounds vague, so let us fix the boundaries at once. The repository holds the code that determines which posts you see in the «For you» feed: how the candidates are collected, how they are scored, how they are sorted, what is thrown out of them and why. This is not a description and not a specification — these are the sources of the services, including the training of the ranking model and its serving.
What is there
- The whole feed pipeline — from the hydration of the request to the side effects after the answer. That is
home-mixer/and thecandidate-pipeline/framework it is built on. - The Phoenix model — both retrieval and ranking, with training code in JAX, with a generator of synthetic data and with a serving engine in Rust. It is stated to be the production implementation rather than a demonstration one: the only disclosed replacement is the optimiser for configs on a deprecated slot.
- The weights with which the predictions are combined into one score — as ordinary numbers in the source.
- The visibility system — the rules by which a post is hidden or put behind an interstitial, and the systems that produce the labels for those rules.
- SimClusters — the good old source of candidates based on clustering accounts by who interacts with whom.
What is not there — and it matters to understand this
- The prompts of the LLM classifiers (
grox/) are not published. That is, you can see that a classifier is invoked and where its verdict goes, but not what exactly the model is asked. - Part of the labelling rules (
botmaker-rules/) is absent — it is stated directly that this is to reduce the risk of circumvention. - The model weights are not published. The training code is there, the checkpoints are not; to run anything you are invited to generate a synthetic world and train a small model yourself.
- The infrastructure scaffolding — deployment, orchestration, internal clients — is replaced by stubs or missing. You cannot assemble this and bring it up at home in one piece.
- The data, naturally, is not there either. And without interaction logs a recommender system is an empty skeleton: all the value is in what the model has learned, not in how the pipeline is written.
The last point is worth spelling out separately, because there is more confusion around it than anywhere else. Open-sourced code of a recommender system does not allow it to be reproduced. It allows one to understand which decisions were made and at what cost. That is exactly what we need: the principles are known, and here we look at their implementation.
Every piece of code on the pages of the supplement comes with its path in the repository and a link to the permanent address of the commit 28e414f. The code belongs to X and is distributed under the Apache License 2.0; the fragments are quoted for analysis. The numeric constants in the text are not retyped by hand — they are pulled out of the sources by a script and checked automatically, so that they do not drift apart from the code.
2. Two tracks: the request and the labelling
The first thing to get straight is that the system has two independent tracks, and they cross at exactly one point.
The request track works while you wait for the feed. It has a hard time budget: collect the candidates, pull features onto them, filter, run the model, sort, mix in the ads, return. All in tens of milliseconds.
The labelling track works continuously and has nothing to do with your request. Classifiers go through published posts, batch jobs compute account reputation, a rules engine attaches labels. The result is put into a store.
They cross at the moment when the posts have already been ranked: the visibility service reads those labels and answers for every post — show it, put it behind an interstitial, or drop it. This is exactly the multi-stage design familiar from the theory of the funnel, only here the stage «may this be shown at all» has honestly been moved out into a separate service with its own rules.
The temptation to do the opposite is strong: teach the ranking model simply to score bad things lower, and no separate service is needed. It is not done, for three reasons.
- Different requirements on errors. Ranking errs softly: a slightly less interesting post was shown, a little engagement was lost. Visibility errs harshly: something that must not be shown was shown, and that is a different class of problem. Mixing quantities with such different costs of error into one score is a bad idea.
- Different speeds of change. Visibility rules change under legal and product requirements, sometimes within hours. The ranking model is retrained on its own schedule. Tying them together means that any change of the rules requires retraining.
- Auditability. The rule «if an account has such a label and the viewer does not follow it, do not show it» can be read, explained and disputed. «The model lowered the score» cannot be explained. For a system that faces claims from regulators and users, that is the decisive argument.
We made the same argument in the chapter «A summary of biases», when discussing why business rules are not sewn into the loss but moved out into re-ranking.
3. A map of the repository
2028 files are easy to be scared by, but the structure is simple: one directory is one service or one library. Here is what lies where, by language and by volume.
| Language | Files | Lines | What is written in it |
|---|---|---|---|
| Rust | 451 | 140 106 | Everything on the request track: the feed pipeline, the store of fresh posts, the re-ranker, the visibility service, the model serving |
| Python | 410 | 103 315 | The Phoenix model: definitions, training in JAX, generation of synthetic data, attention kernels |
| Scala | 556 | 80 987 | Batch jobs: SimClusters, account reputation, aggregation of labels |
| Java | 313 | 29 421 | The labelling rules engine and its scaffolding |
| CUDA and C++ | 24 | 4 004 | Attention kernels for a particular generation of GPUs |
The choice of languages is instructive in itself and fits what was said in the chapter «The architecture of the runtime». Rust is where every millisecond counts and where a service holds millions of objects in memory. Python is where the speed of a researcher's iterations matters rather than the speed of execution. Scala is where data is processed offline in large batches. This is that same gap between the online and offline loops out of which the lambda architecture grows.
The components by role
| Directory | Role | Where we discuss it |
|---|---|---|
candidate-pipeline/ | The framework of stages: source, hydrator, filter, scorer, selector, side effect | x01 |
home-mixer/ | The feed itself: which stages, in what order, with which parameters | x01, x05, x06 |
thunder/ | Fresh posts from follows, laid out in memory | x02 |
simclusters/ | Clustering of accounts and the search for candidates by cluster | x02 |
phoenix/ | The model: two-tower retrieval and a transformer ranker, training and serving | x03, x04 |
phoenix-rankall/ | The index of posts that retrieval queries | x03 |
vm-ranker/ | The re-ranker: selection through a determinantal process over embeddings | x05 |
visibility-filtering/ | Show, hide or drop — by rules and labels | x07 |
grox/, clip/, media-model-proxy/ | Understanding the content: classifiers of text, images and video | x07 |
agatha/, bdsm/, user-cred-v2/ | Account reputation: by the reaction of others, by behaviour, by the graph | x07 |
botmaker/, scarecrow/ | The language of the labelling rules, its compiler and its executor | x07 |
under-the-hood/ | The report to a user about the labels on their account | x07 |
4. Five decisions that define the whole system
In the README they are given as a list. Let us take each one: what it means, what it is paid for with and what the theory says about it.
Decision 1. We predict actions, not relevance
The model does not produce one number saying «how good the post is». It produces the probability of every action separately: will like, will reply, will repost, will quote, will share, will click, will watch the video through, will follow the author, will report, will block, will hide. Combining these probabilities into one score is a separate explicit step with weights that live in the config.
This is exactly the multi-task formulation, taken to its logical conclusion. The benefit is twofold:
- The product policy changes and the model is not touched. Decided that replies are worth more than likes — change a number in the config and roll it out through an experiment. If the model were trained on a single score, every such decision would mean retraining.
- Rare signals do not drown. A report happens several orders of magnitude less often than a like. In a single loss it simply would not affect the gradient. A separate head learns to predict it as a task of its own, and the weight then sets how strongly it influences the output.
The price is the one every multi-task setup pays: the heads compete for a shared body, rare tasks overfit, and the weights have to be picked by experiment, because they cannot be derived from first principles. In detail and with real numbers — on page x05.
Decision 2. The candidates do not see each other
The ranking model is a transformer, and all the candidates are fed to it at once together with the viewer's history. But the attention mask is arranged so that a candidate may look at the viewer's context but not at the other candidates.
Why such a restriction? Without it the score of a post would depend on who else ended up in the same batch. The consequences:
- The score cannot be cached — on the next request the set of neighbours is different and the number changes.
- The result stops being reproducible: two identical requests give a different order because of the randomness of batching.
- Debugging turns into hell. «Why did this post go down?» — «Because another post happened to be next to it».
Here a classic trade-off is visible. Listwise models, which we mentioned in the chapter «Listwise losses», gain precisely from seeing the whole slate: they can account for mutual influence and for diversity. X gives that gain up for the sake of consistency and cacheability — and picks diversity back up after ranking, in a separate re-ranking step. The mechanics of the mask are discussed on x04, and there is a widget there where it can be switched off to see what breaks.
Decision 3. Hash embeddings instead of a vocabulary
Both retrieval and ranking look embeddings up through several hash functions rather than through a table «identifier → row». No vocabulary exists at all.
We covered this in the chapter «Categorical features and memory»: the hashing trick trades memory for collisions, and several hash functions make the probability of full indistinguishability a product of probabilities. But in the feed of a social network this trick has a second, more important meaning: a new post becomes representable immediately. There is no need to wait for its identifier to get into a vocabulary, for the vocabulary to be rebuilt, for the model to be retrained. The post is published — its hash is computed — the embedding is available.
For a system where half the content lives for hours, that is not an optimisation but a necessary condition. Compare it with the classical matrix factorisation from the chapter «Matrix factorisation», where a new item gets no vector at all without retraining.
Decision 4. Ranking and visibility are different systems
Discussed above in the section on the two tracks. Let us add only the consequence that is easy to miss: visibility filtering happens after the sorting, not before it. First the posts are ranked, then the top is selected, and only then permission is asked about each of them.
The order is not obvious — it would seem cheaper to throw out the superfluous first and then rank less. But a call to the visibility service is expensive and is made per «post and viewer» pair, so it pays better to ask about a hundred selected posts than about a thousand candidates. This is the same argument as in the chapter «The order of filtering»: cheap filters go before ranking, expensive ones after.
Decision 5. The pipeline is assembled from standard stages
The whole feed is described in terms of six types of stage, and the framework runs them itself, parallelising the independent ones and surviving the failure of individual ones. To add a source of candidates or a filter means writing one structure and adding it to a list.
It sounds like ordinary engineering, but it is exactly what makes everything else possible: thirty filters, eighteen sources, experiments over each stage separately. Discussed on x01.
5. How to read this
- Start with
home-mixer/params/param.rs. It is the most informative file: 191 parameters, and their names alone show what is tunable in the system at all. - Then
home-mixer/scorers/ranking_scorer.rs— the arithmetic of the final score in full, including every boost. - Then
home-mixer/candidate_pipeline/phoenix_candidate_pipeline.rs— the list of stages in order. It is the table of contents of the whole feed. - Then
visibility-filtering/rules/registry.rs— the visibility rules in order of application. - And only after that
phoenix/, if the model is of interest. It is the largest and the most specific.
The mod.rs files in Rust are simply enumerations of a directory's modules, there is nothing substantial in them. The *.thrift and *.proto files are data schemas: useful for understanding which fields a post and a request have at all.
6. A glossary of the repository
The terms that occur in the code constantly and without which it is hard to read on.
| Term | What it means |
|---|---|
| Home Mixer | The service that assembles the «For you» feed. The name is historical: it «mixes» posts with the other elements of the feed |
| Candidate | A post that has come under consideration. It stays a candidate right to the end — until it is chosen or dropped |
| Hydration | Pulling data onto an object: onto the request — the lists of follows and blocks, onto a candidate — the text, the author, the counters. A term of the same family as feature enrichment |
| In-network / Out-of-network | Posts from those the viewer follows and from those they do not. Abbreviated OON. The key division in the whole system |
| Slate | The set of posts shown at one time. The same notion as in ranking metrics |
| Impression | The fact that a post was shown to a viewer. Stored so as not to show it twice |
| Label | A mark on a post or an account, put there by the labelling system. Not to be confused with a class label in training |
| VF | Visibility Filtering, the visibility service. In the code it occurs as a prefix: VFFilter, VFCandidateHydrator |
| SID | Semantic ID — the code of a post from residual quantisation. Discussed on x03, the theory is in the chapter «Generative retrieval and semantic IDs» |
| Param | A tunable value read from the configuration system. In the code it is declared by the param! macro, and the default is synchronised with the production one |
| Side effect | An action after the answer has been sent: record the impressions, update the cache, emit events. It does not influence the output |
Common misconceptions
- «The action weights show how many times worse a report is than a like». No. The weight is multiplied by the predicted probability of the action, not by the number of actions. The analysis with numbers is on x05, and there is a widget there where it is immediately visible.
- «Open code makes it possible to game the feed». Ranking is personalised: the reaction of particular accounts mostly influences what is shown to those who resemble them. Besides, a significant part of the labelling systems is deliberately not published.
- «This is the whole X algorithm». This is the «For you» feed. Search, notifications, trends, «Following» are other systems and are not here.
- «The code in the repository is what runs in production right now». The parameter defaults are synchronised with the production scripts, but part of the traffic is always in experiments. About that — x09.
- «Since the code is open, the system can be reproduced». Without interaction logs and trained weights, no. The value of the repository is that the decisions are visible, not that it can be run.
Interview questions
Why separate ranking and visibility filtering? Why not just lower the score?
Three reasons. A different cost of error: ranking errs by a few percent of engagement, visibility errs on reputation and legal requirements; combining them into one score means mixing incomparable quantities. A different speed of change: visibility rules change within hours, the model is retrained on a schedule. Auditability: a rule can be read and disputed, a score lowered by a model cannot.
Additionally: lowering a score does not guarantee that the post will not be shown. If there are few competitors, a post with a lowered score will end up in the output anyway. A hard requirement of «do not show» is implemented only by a hard filter.
Why does visibility filtering come after ranking rather than before?
Because it is expensive and is called per «post and viewer» pair. Before ranking there are thousands of candidates, after the selection a hundred. Asking about a hundred is ten times cheaper.
The general rule is the same as in the chapter «The order of filtering»: cheap filters working on data that is already in memory (the age of a post, blocks, what has already been shown) go before ranking; expensive ones, requiring a trip to another service, go after. And one has to remember that filtering after the selection shrinks the output, so a margin is allowed for: the selection is made with a surplus.
What does predicting separate actions give instead of one relevance score?
It decouples the model from the product policy. The action weights are changed by config and rolled out by experiment, with no retraining. Plus rare negative signals such as a report get a head of their own and do not dissolve in a common loss.
The price: the heads compete for a shared body; rare tasks are prone to overfitting; the weights cannot be derived theoretically and are picked by A/B tests, which is slow. In more detail — the chapter «Multi-task learning» and page x05.
Why are candidates forbidden to look at each other in the transformer?
So that the score of a post does not depend on the composition of the batch. Otherwise the same thing with the same viewer gives different numbers, the score cannot be cached, the result is not reproducible, and debugging becomes impossible.
The flip side: the model does not see the whole slate and cannot account for the mutual influence of posts — for instance, that three posts in a row are about the same thing. That task is solved separately, by re-ranking after the scoring. That is, listwise effects are not lost but moved into a separate step where they are easier to control.
Why hash embeddings, if a vocabulary of identifiers can be kept?
Two arguments. Memory: a vocabulary over all the posts and all the authors does not fit, while a hash table has a fixed size. Freshness: a new post is representable immediately after publication, because its hash is computed on the spot, whereas a vocabulary would have to be rebuilt.
The price is collisions. Not just any are dangerous, but collisions of two frequent values: their embeddings get glued together. It is cured by several hash functions — then indistinguishability requires a coincidence in all the tables at once. In detail and with an interactive — the chapter «Categorical features and memory».
What matters more for understanding the system: the pipeline code or the model weights?
A trick question, and the right answer is «neither on its own». The pipeline code shows which decisions were made and where the levers are. The model weights show what the system learned from the data. Only the first is open, and that is a fundamental limit on transparency: one can check that a rule exists and how it is written, but not what the model learned from the logs.
That is exactly why the code release comes with a tool that shows a user the labels on their account: the code plus observable outputs give more than the code alone.
One-screen cheat sheet
What is open
The feed pipeline, the model with training and serving, the weights combining the actions, the visibility rules, the labelling systems.
What is closed
Classifier prompts, part of the labelling rules, model weights, the data, deployment.
Two tracks
The request one assembles the feed in tens of milliseconds. The labelling one runs continuously and puts labels into a store.
The junction
After the sorting: the visibility service answers for every post — show, hide behind an interstitial, or drop.
Multi-task
The model predicts the probabilities of actions; combining them into a score is a separate step with weights from the config.
Isolated candidates
In the attention mask candidates do not see each other: the score is consistent and cacheable.
Hash embeddings
No vocabulary, several hash functions. A new post is representable at once.
Languages
Rust — online, Python — the model, Scala — offline jobs, Java — the rules engine.
Where to start reading
param.rs → ranking_scorer.rs → the list of pipeline stages → the registry of visibility rules.
Primary sources
- xai-org/x-algorithm — the whole repository, Apache 2.0. The analysis follows the commit
28e414f. - README — the authors' description of the architecture, including diagrams of both tracks and lists of the components.
- phoenix/README.md — the description of the model: two-tower retrieval, the ranking transformer, the configs.
- docs/BIDIRECTIONAL_BOOST_CHANGE.md — the history of one parameter, discussed on x09.