RecSys Supplement · The X algorithm · page 4 of 11
Retrieval: two towers, hashes and semantic IDs
How to pull a few hundred candidates out of 28 million posts in a matter of milliseconds. Let us go through the two-tower Phoenix model: why the user has no embedding of their own, how a post is represented without a vocabulary of identifiers, what a semantic ID is and how the training here differs from the way two-tower models are usually taught.
In brief
- A user is their history. There is no learnable user embedding in the model at all (
use_user_embedding = False). There is a transformer over the sequence of interactions plus one token with coarse profile features. - A post is its semantic ID and the hashes of its author. There is no learnable post embedding either (
use_post_embedding = False). A post is encoded by six levels of residual quantisation of 256 codes each, derived from a multimodal embedding. - Hashes, not a vocabulary. Every identifier is turned into two rows of the table by two different hash functions. No vocabulary exists, and a new item is representable at once.
- The candidate index lives inside the checkpoint. On every save the candidate tower is run over the corpus, and the resulting 28.7 mln vectors are written into the same file. The serving indexes nothing at start-up.
- Training is a sampled softmax with two kinds of negatives and a LogQ correction, exactly as in the chapter «The LogQ correction», only with a correction scale of 2.0 and separate learning rates for the embeddings and for the rest of the network.
A map of the page
1. Why two towers at all
Let us repeat the logic of the chapter «Why two-tower models are needed» in the scenery of this system, because here it is especially visible.
The corpus of candidates is 28 672 000 posts (the constant max_posts in the config). A few hundred have to be returned per request. If the model computed a joint function \(f(\text{user}, \text{post})\), it would have to be called 28 million times — impossible in any reasonable time.
Two towers are a restriction imposed deliberately: the score is obliged to decompose into the product of two independently computed vectors,
$$ s(u, i) \;=\; \langle \mathbf{q}_u,\ \mathbf{p}_i \rangle. $$The price is known: the towers do not see each other, so no «cross» features — a match between the language of the user and of the post, the time since following the author — can be expressed by the model. The gain is known too: \(\mathbf{p}_i\) for all the posts are computed in advance, and on a request all that remains is to compute \(\mathbf{q}_u\) once and find the nearest neighbours.
There is a function in the code with a telling name:
def _l2_normalize_candidates(embeddings: jax.Array) -> jax.Array:A fragment of recsys_two_tower_model.py · code by X, Apache 2.0, commit 28e414f
That is, the vectors are brought to unit length, and the dot product turns into a cosine. We discussed this in the chapter «The cosine and the temperature» and felt it by hand in the widget of similarity measures: the norm of a vector in models like this learns popularity, and the cosine throws it away.
Here the decision has a second, purely engineering meaning as well. A nearest-neighbour search by dot product over normalised vectors is equivalent to a search by Euclidean distance, because \(\|a-b\|^2 = 2 - 2\langle a,b\rangle\). Which means any approximate-search library can be used without worrying about whether it supports maximum inner product search — the task reduces to an ordinary nearest-neighbour one.
2. The user has no embedding
The most unexpected decision in this model, and in the config it is written down outright:
"use_user_embedding": False,
"use_post_embedding": False,A fragment of xrecsys_two_tower.py · code by X, Apache 2.0, commit 28e414f
Neither the user nor the post has a learnable row of their own in a table. The user is represented exclusively by what they interacted with: a sequence of up to 1023 history events plus one token with coarse profile features — country, language, location, gender, age, installed apps.
Recall the chapter «Learnable embeddings», where we contrasted learnable embeddings with content-based representations. A learnable user embedding is memorisation: the model remembers a particular person. That has three troubles, and all three are critical for the feed of a social network.
- A new user gets a random vector. Until they accumulate a history, their embedding is noise, and the recommendations match. Here, by contrast, a new user is represented by their very first action: a history of one event already gives a meaningful vector.
- The vector goes stale. A learnable embedding reflects the user as of the last training run. If you took up a new topic yesterday, the table will learn about it only after retraining. The history, on the other hand, is updated in real time: an action was taken — the next request already computes the vector with it.
- A table of hundreds of millions of rows. It has to be stored, updated and synchronised between serving replicas.
Giving up the user embedding moves the model from transductive to inductive: it can compute a vector for a person it has never seen, from their actions alone. That is exactly what we called an inductive bias — the model is obliged to generalise rather than to memorise.
There is a price too, and it is worth stating honestly. Stable long-term preferences that are not expressed in recent actions cannot be expressed by the model. A person who has loved astronomy for years but spent the last week reading about football is, to this model, a football fan. A learnable embedding would have remembered that. The trade-off is chosen in favour of freshness, and for a feed where content lives for hours it is justified.
"empty_history_user_dropout_rate": 0.1,A fragment of xrecsys_two_tower.py · code by X, Apache 2.0, commit 28e414f
In ten percent of the training examples the history is zeroed out deliberately. The model is forced to learn to work with a user about whom nothing is known — otherwise it would rely entirely on the history and fall apart on exactly those who need recommendations most: the newcomers.
This is the same trick as feature dropout, only applied to a whole modality. And it is a direct answer to the question «how does a two-tower model with no user embedding work with a cold viewer» — it is specially taught to.
3. Hash embeddings: how to do without a vocabulary
The author of a post, the user, the IP — all of these are categorical features with an enormous number of values. The classical approach — a vocabulary «value → row index» — does not work here: a vocabulary has to be built, stored, rebuilt, while new values appear every second.
Instead of a vocabulary, universal hashing is used. The function in the code looks like this:
raw = (ids[i] * scales[j] + biases[j]) % modulusA fragment of recsys_embedding.py · code by X, Apache 2.0, commit 28e414f
that is, \(h_j(\text{id}) = \bigl((\text{id} \cdot a_j + b_j) \bmod M\bigr) \bmod m\), where \(a_j, b_j\) are fixed constants, \(M\) a large prime, \(m\) the number of rows in the table. And the key point: there are two such functions.
| Entity | Hash functions | Multipliers \(a_j\) | Modulus \(M\) |
|---|---|---|---|
| user | 2 | 196 742 702 and 1 852 108 266 | 2 859 568 897 |
| post | 2 | 2 161 410 491 and 1 754 358 832 | 2 361 375 383 |
| author | 2 | 371 965 780 and 328 930 218 | 631 860 353 |
This is exactly the scheme we discussed in the chapter «Unified Embedding» under the name Unified Embedding, and it is also what stands behind the hashing widget: the probability that two values turn out indistinguishable is not \(p\) but \(p^2\). At a table load of, say, 5% that gives a fall from 5% to 0.25%.
A collision of two frequent authors is still possible and still harmful: their embeddings add up into a shared signal, and the model stops telling them apart. Two hash functions lower the probability but do not eliminate it.
That is why real systems usually keep a small explicit vocabulary on top of the hash for the most frequent values — the ones where the cost of an error is high. No such mechanism is visible in the published config, and this is one of the places where the limit of our knowledge should be admitted: perhaps it exists in the internal configuration, perhaps collisions of frequent authors were judged acceptable.
4. The semantic ID: a post with no identifier
The most interesting part. A post used to be represented by the hashes of its identifier. Now it is a semantic ID, and in the config that is written like this:
"use_post_embedding": False,
"use_post_sid": True,
"sid_num_levels": 6,
"sid_codebook_size": 256,
"sid_embed_dim": 1024,A fragment of xrecsys_two_tower.py · code by X, Apache 2.0, commit 28e414f
How such a code is produced
The mechanism is residual quantisation, exactly what we discussed in the chapter «Generative retrieval and semantic IDs». A multimodal embedding of the post (text plus image) of dimension 1024 is taken and encoded by six levels:
- The first codebook of 256 vectors. We find the nearest one — that is the first token \(c_1\).
- We subtract it from the original vector. What remains is the residual — what the first level did not explain.
- The second codebook of 256 vectors, but now for residuals. The nearest is the token \(c_2\).
- And so on six times.
A post turns into a tuple of six numbers, each from 0 to 255: for example, \((37,\ 210,\ 4,\ 118,\ 250,\ 91)\). The total number of distinguishable codes is \(256^6 \approx 2.8 \cdot 10^{14}\), with an enormous margin for any catalogue.
Let us compare three ways of representing a post.
- A learnable embedding by ID. A table for the whole catalogue, and a new post is a random vector until retraining. For a feed where a post lives for hours, useless.
- Hashes of the identifier. A table of fixed size, a new post representable at once — but the representation is arbitrary. Two posts about space get unrelated rows of the table, and everything the model knows about them it has to learn separately for each.
- A semantic ID. The code is derived from the content. A new post gets its code immediately after publication, by running it through the multimodal encoder and the codebooks. And — most importantly — that code is meaningfully related to the codes of similar posts.
The third point is worth spelling out. If the model has learned on thousands of posts with the prefix (37, 210), then a new post with the same prefix inherits that knowledge for free. This is what the README calls compositional generalization: a post the model has never seen is representable not at random but as a combination of parts it already understands.
In practice this solves the item cold start — a task matrix factorisation cannot handle in principle. The mechanics of residual quantisation can be turned by hand in the «Quantisation» widget in the catalogue.
Two more lines of the config deserve an explanation:
sid_hash_level: True— the codes of different levels do not share one space but are separated by hashing: code 37 of the first level and code 37 of the second are different vectors. Logical, since they encode different things: the first a coarse topic, the second a refinement of the residual.sid_cross_attn: True— the six tokens of the code are not simply summed but processed by attention. The levels can look at each other, and the model itself decides which level matters more for a particular post.
5. Training: a sampled softmax with two kinds of negatives
Here we meet in live code almost everything we derived in the chapter «Two-tower models». Let us go through it piece by piece.
What counts as a positive
The naive answer is «an interaction». The real one turns out to be stricter:
valid_positive_mask = (
has_positive_actions
& ~has_hard_negative_actions
& ~has_soft_negative_actions
& ad_mask_candidates
& candidate_padding_mask[:, :C]
)A fragment of recsys_two_tower_model.py · code by X, Apache 2.0, commit 28e414f
A positive is a post on which a positive action happened and not a single negative one did. And the negative ones are split into two sorts:
| Sort | Actions |
|---|---|
| Positive | a like — and nothing else |
| Hard negative | a report, «not interested», «show less», unfollowing the author, a block, hiding, «not relevant» |
| Soft negative | did not dwell on the post |
Note the first row: in the base configuration retrieval is trained on likes and on nothing else. Not on reposts, not on clicks, not on watch time. This is a deliberate narrowing: retrieval has to sketch out the area of interests roughly, while the fine distinctions are the work of ranking.
A post a person liked and then reported — what is that? Formally a positive. To the model it is garbage: it will get the signal «such things should be retrieved», while in fact such things should not be retrieved.
We discussed this in the chapter «The problems of implicit data» in the section on the problems of implicit data: a click does not equal satisfaction. Here the same thought has been taken to a rule: an event counts as a positive only if nothing went wrong.
And it also quietly solves the problem of clickbait. A post with an enticing headline will collect a like and immediately a «did not dwell». A soft negative strikes it out of the positives, and retrieval does not learn to fetch such things.
Two kinds of negatives
The negatives are taken from two sources at once, and that differs from what is usually taught:
| Source | How many | Where it comes from |
|---|---|---|
| In-batch | the whole batch | The positives of the other examples in the same batch. Free: their embeddings have already been computed |
| Global | 64 per example | Separately sampled posts from the corpus, computed specially |
Why both, if in-batch ones are free? Because they have different distributions, and we discussed that in detail in the chapter «Sampling negatives».
- In-batch negatives come from the popularity distribution: the more often a post is liked, the more often it is someone's positive and therefore someone else's negative. They are automatically «hard» — the popular really is usually not bad — but systematically biased.
- Global negatives are sampled separately and cover the whole corpus, including the tail, which practically never lands in a batch. They are easy, but they are exactly what keeps the model from collapsing onto the head of the distribution.
Incidentally, an interesting division of roles is visible in the config: num_negatives_per_example: 0 alongside num_global_negatives_per_example: 64. That is, there are no additional «local» negatives at all — in-batch plus 64 global ones do the work.
The LogQ correction — in live code
And here is what it was worth getting here for. In-batch negatives are biased towards the popular, and without a correction the model would learn «popular means bad». The correction is in the code:
def _apply_logq_correction(
local_batch, local_neg, local_batch_correction, global_batch_correction
):
if use_in_batch_negatives:
local_batch_correction = local_batch_correction.reshape((1, -1))
local_batch += local_batch_correction
if N > 0:
local_neg += global_batch_correction.reshape((1, -1))
return local_batch, local_negA fragment of recsys_two_tower_model.py · code by X, Apache 2.0, commit 28e414f
Note two things. The first: the correction is added to the logits — that is exactly the \(s - \log Q\) of the derivation of the LogQ correction, up to the sign sewn into how the correction itself is computed. The second: there are two different corrections — one for in-batch negatives and one for global ones, because their distributions differ and one correction cannot straighten both.
The scale of the correction is a parameter of its own:
logq_correction_scale: float = 2.0A fragment of recsys_two_tower_model.py · code by X, Apache 2.0, commit 28e414f
That is, the correction is applied not as \(s - \log Q\) but as \(s - \lambda \log Q\) with \(\lambda = 2\). Theory says an unbiased estimate is obtained at \(\lambda = 1\); a value of 2 means a deliberate over-correction — the popular is pushed down harder than the mathematics requires.
The theoretical \(\lambda = 1\) makes retrieval unbiased with respect to the distribution that produced the log. But the log was produced by a previous version of the system, which itself preferred the popular. That is, even a model unbiased with respect to the log reproduces the popularity bias inherited from the logging policy.
This is that same feedback loop. An over-correction of \(\lambda = 2\) is a crude but practical way of pressing it down: we pay with the accuracy of the estimate for retrieval reaching into the tail more actively. And since ranking works after retrieval and will lift the popular anyway, the risk is small.
What the correction does and what happens without it can be seen in the LogQ widget: there two models learn on one stream of batches, and without the correction the score converges to \(\log p - \log Q\).
Two learning rates
"learning_rate": 2e-3,
"emb_learning_rate": 0.1,A fragment of xrecsys_two_tower.py · code by X, Apache 2.0, commit 28e414f
The learning rate of the embeddings is 50 times higher than that of the rest of the network. This is standard practice for recommender models, and the reason is the sparsity of the gradient: dense layers receive a gradient on every example of the batch, while a particular row of the embedding table receives one only when its item lands in the batch, that is, thousands of times less often. For it to manage to learn, the step has to be larger.
6. The index: the serving computes nothing
How do these embeddings get into the search? The answer in the README is unexpected and elegant: the index lies inside the checkpoint.
On every save of the model the trainer runs the candidate tower over the whole corpus and writes the resulting 28.7 million vectors into the same file as the weights. The serving loads the checkpoint — and the index is already there.
- Consistency. The weights of the tower and the index are always from one moment of training. The classic trouble with separate storage is an index computed by an old version of the tower and a user vector by a new one; the dot product between them is meaningless. Here that is impossible by construction.
- Simplicity of release. One artefact instead of two with their synchronisation. Rolling the model back automatically rolls the index back.
- Speed of start-up. Nothing is indexed at load time — 28.7 million runs of the tower is hours.
The price is the size of the checkpoint and the fact that the index is refreshed only when the model is saved. A post published after the last checkpoint is absent from this index. That is exactly why a separate system maintaining an index of fresh posts in real time exists alongside — it is discussed on the page about the sources.
7. The configuration of the production model on one screen
| Parameter | Value | What it means |
|---|---|---|
history_seq_len | 1023 | How many of the user's latest events the tower sees |
candidate_seq_len | 64 | How many candidates are in a training example |
num_global_negatives_per_example | 64 | Global negatives per example |
num_layers | 8 | Transformer layers in the user tower |
emb_size, emb_table_width | 1024 | The dimension of the representations |
query_heads / kv_heads | 16 / 4 | Grouped-query attention: one key head per four query heads |
max_posts | 28 672 000 | The size of the indexed corpus |
attn_logit_cap | 80.0 | A cap on the attention logits — protection against divergence |
effective_sequence_len | 513 | The average length after packing the sequences |
num_candidate_heads | 2 | Two candidate heads: home and immersive |
The last row is curious. The candidate tower has two heads with different definitions of a positive: for the feed a positive is a like, for the full-screen video mode it is a like, a reply, a quote, a repost, a quality view, a follow, a bookmark or a share. One model, two different notions of what «good» means, depending on the surface. This is the same argument about different surfaces and different objective functions.
16 query heads to 4 key and value heads means that every group of four query heads shares the same keys. What for: at inference the cache of keys and values is the main consumer of memory and bandwidth, and cutting it fourfold turns directly into throughput. Quality sags little in the process.
This is a purely engineering optimisation from the world of large language models, which moved into recommendations together with transformers — a good illustration of what was said in the chapter «Transformers over history»: recommender architectures today borrow almost everything from language ones.
Common mistakes and hidden rocks
- «A two-tower model is about embeddings of users and items». Here there is neither one nor the other as learnable tables. The user is a transformer over history, the post is its semantic ID. Two towers are about the separateness of the computation, not about representations being obliged to be learnable rows.
- Confusing a semantic ID with a hash. A hash is arbitrary: similar posts get unrelated rows. A semantic ID is derived from the content: similar posts share a prefix. That is the difference between «a unique name» and «an address in a space of meanings».
- Thinking that a positive is any interaction. A positive here is a like in the absence of any negative action, including «did not dwell».
- Considering the LogQ correction exact. A scale of 2.0 instead of the theoretical 1.0 is a deliberate over-correction against inherited popularity bias, not a mistake.
- Forgetting that the index is refreshed only when the model is saved. Fresh posts are absent from it, and other sources are responsible for them.
- Thinking that «no user embedding» = «no personalisation». The personalisation is complete, it is just built on actions rather than on a memorised identifier. And it works instantly for a newcomer — unlike a table.
Interview questions
Why was the learnable user embedding given up in the two-tower model?
Three reasons. Cold start: a new user would get a random vector, whereas a history of a single action already gives a meaningful representation. Freshness: a learnable vector reflects the user as of the retraining, the history reflects them right now. Memory: a table of hundreds of millions of rows that has to be synchronised between replicas.
In essence the model is moved from transductive to inductive: it can compute a vector for a person it did not see during training. The price is that stable long-term interests not manifested in recent actions are lost. For a feed where content lives for hours the trade-off is justified.
What is a semantic ID and why is it better than a hash of the identifier?
It is a code from residual quantisation: a multimodal embedding of the post is encoded by six levels of 256 codes. At each level the nearest codebook vector is taken, subtracted, and the residual is encoded by the next level.
The difference from a hash is fundamental. A hash is arbitrary — two posts about the same thing land in unrelated rows, and the model has to learn each separately. A semantic ID is derived from the content, so similar posts share a prefix, and the knowledge accumulated on some posts transfers to new ones with the same prefix. Besides, a new post is given a code immediately after publication, with no retraining.
Why are both in-batch and global negatives needed, if in-batch ones are free?
They have different distributions. In-batch ones are other people's positives, that is, a distribution proportional to popularity: the negatives come out hard but systematically biased towards the head. Global ones are sampled from the whole corpus and cover the tail, which practically never lands in a batch.
Using only in-batch means teaching the model to tell popular things apart from one another and to know nothing about the tail. Only global means getting negatives that are too easy and a weak gradient. In this config in-batch plus 64 global per example are used, and each kind has its own LogQ correction.
Why the LogQ correction, and why is its scale equal to two?
In-batch negatives come from the popularity distribution \(Q\). Without a correction the sampled softmax converges to \(\log p - \log Q\), that is, the model systematically under-rates the popular. The correction subtracts \(\log Q\) from the logit and gives back \(\log p\).
A scale of 2.0 instead of the theoretical 1.0 is an over-correction. The motive: unbiasedness is achieved with respect to the distribution that produced the log, and the log was produced by a previous version of the system, which itself liked the popular. The over-correction crudely presses that inherited skew down and pushes retrieval into the tail; the risk is small, because ranking works afterwards.
Why is a positive for retrieval only a like rather than any interaction?
Because retrieval and ranking have different tasks. Retrieval has to sketch out the area of interests roughly — what matters here is recall, not subtlety. Ranking will then work out what inside that area is better.
Using every signal at once at the retrieval stage would mean mixing things that mean different things (a click and a completed view say different things) and complicating the task where the complexity does not pay off. Besides, a like is the most frequent explicit signal, that is, it gives the most training examples.
Note the filter separately: a positive counts only if there was no negative action on the same post, including «did not dwell». That cuts off clickbait — a post that was liked and immediately scrolled past.
Why store the candidate index inside the model checkpoint?
For consistency. The weights of the tower and the index are always from one moment of training — a situation where the user vector is computed by a new tower while the index was built by an old one is impossible. A dot product between inconsistent representations is meaningless, and that is a classic source of silent quality degradation.
The side benefits: one artefact instead of two, rolling the model back rolls the index back, and the start-up of the serving does not require hours of re-indexing. The price is that the index is refreshed only on a save, so fresh posts come from other sources.
The vectors of both towers are normalised. What does that change?
The dot product becomes a cosine. The first consequence is that popularity bias goes away: in the unnormalised case the length of a vector learns popularity, and the dot product systematically prefers the popular regardless of direction.
The second consequence is engineering: on the unit sphere \(\|a-b\|^2 = 2 - 2\langle a,b\rangle\), that is, maximising the dot product reduces to finding the Euclidean nearest neighbour. Any approximate-search library can be used without requiring MIPS support from it.
The price is that the signal the norm carried is lost. After normalisation there is nothing left to tell a reliable, many times confirmed vector from the noisy vector of a cold item.
One-screen cheat sheet
The task
28 672 000 posts → hundreds of candidates. The score is obliged to decompose into \(\langle q_u, p_i\rangle\).
The user
No learnable embedding. A transformer over a history of up to 1023 events + a profile token.
The post
No learnable embedding. A semantic ID 6 × 256 + the author's hashes.
Hashes
Two functions of the form \((a\cdot id + b) \bmod M\). Indistinguishability needs a collision in both: \(p^2\).
Semantic ID
Residual quantisation. Similar posts share a prefix — hence the transfer of knowledge to new ones.
A positive
A like and no negative actions, including «did not dwell».
Negatives
In-batch (∝ popularity) plus 64 global. Each kind has its own LogQ correction.
LogQ
A scale of 2.0 — an over-correction against inherited bias towards the popular.
The index
Computed on a save and lying inside the checkpoint. The serving indexes nothing.
Primary sources
- recsys_two_tower_model.py — the towers, the loss, the LogQ correction, the definition of a positive.
- xrecsys_two_tower.py — the production configs: lengths, dimensions, the number of negatives, the semantic ID parameters.
- recsys_embedding.py — the hash functions, the sizes of the tables, the offsets of the zones.
- phoenix/README.md — the authors' description of the architecture.
- The course: the chapter «Two-tower models» in full, the chapter «Inductive bias and content» on inductive bias, the chapter «Unified Embedding» on Unified Embedding, the chapter «Generative retrieval and semantic IDs» on semantic IDs.