Part IV · Sequences and the slate · chapter 14 of 19

Transformers over history

A user's history is the most informative feature we have and the most awkward: it is a sequence of variable length, and a model needs a vector. This whole chapter is about how that sequence is folded up, what is lost by simple averaging, and why production ends up with not one model but two loops recomputed at different rates.

What to take away

Averaging dissolves a rare decisive event. The single music track in a history of twelve events gets a weight of 0.835 instead of 0.083, and the score changes from −0.56 to +0.83.
Early fusion costs exactly as much as there are candidates. With 500 candidates the encoder is run 500 times: 627 ms instead of 1.25, that is, 31 times past the budget.
Attention is quadratic in length. A history of 8000 events is 1113 times more expensive than one of a hundred — hence the split into an offline loop and a real-time one.
The central methodological lesson: the advantage of BERT4Rec over SASRec was explained not by bidirectional attention but by the loss function. When comparing papers, check the loss and the negative-sampling scheme first.

1. The starting point: averaging

The classic construction: the user vector is the average of the embeddings of the last 50 views plus the average of the embeddings of the last search queries. The key property the whole chapter grows out of: the vector is formed independently of the current candidate.

What gets lost in the process

Take a history of 12 events: 8 about sport, 2 about cooking, 1 about technology and exactly one music track. And four candidates, one per topic.

Candidate	Max. weight in the history	Uniform weight	Score with attention	Score with the average
sport	0.120	0.083	+0.97	+0.60
cooking	0.293	0.083	+0.73	+0.35
tech	0.685	0.083	+0.73	−0.34
music	0.835	0.083	+0.83	−0.56

The numbers are reproduced by the script _tools/history_demo.py in this repository.

Look at the last row. Attention finds the single music event and gives it a weight 10 times higher than the uniform one. The score changes sign: +0.83 against −0.56.

Averaging did not «underrate» that signal — it destroyed it. One event out of twelve is indistinguishable from noise in an average, and all the information that the person listens to music at all was lost at the pooling step.

Note the «sport» row too: there the difference between attention and the average is minimal. When the history is homogeneous, averaging works fine — and that is exactly why the problem went unnoticed for so long.

What to look for here

Switch the candidate — the weights over the history are recomputed, because attention is computed relative to it. The grey dashed line (the plain average) does not move at all: it is the same for every candidate.
Choose the «music» candidate — the very case from the table above.
The τ slider shows the second half of the story: attention degenerates at both ends of the scale.

What to say in an interview: «Target attention differs from pooling in that the weights depend on the candidate. So a rare but decisive event of the history does not dissolve in the average — DIN, BST and TransAct are built on that».

2. DIN: attention that depends on the candidate

Target-aware attention

Let $e_1, \dots, e_H$ be the embeddings of the historical events and $v_A$ the embedding of the candidate. Then the user vector is:

$$ v_U(A) = \sum_{j=1}^{H} w_j\, e_j, \qquad w_j = g(e_j, v_A) $$

Formally this is the same pooling of events — but the weights stop being uniform and depend on who is being scored. Hence the notation $v_U(A)$: the user no longer has one vector, they have a vector per candidate.

A detail people like to ask about

The authors of DIN do not use softmax normalisation of the attention scores. That looks like carelessness and is in fact a substantive decision.

The sum $\sum_j w_j$ is interpreted as the intensity of the user's interest in the candidate. Under softmax normalisation the sum is always one — and that information is lost: a user with ten relevant events in their history and a user with one would get a convolution of the same «mass».

Normalisation is not free here: it flattens precisely what we would like to distinguish.

The setup from the paper as a specimen: predict the probability of a click on a product advertisement; the history is the user's clicks, the vector of an event is a concatenation of trainable embeddings of the product, the shop and the category; 16 feature groups with an embedding of dimension 12; two weeks for training, one day for the test, 2 billion examples. DIN was followed by a whole family — Deep Interest Evolution, Adaptive Interest, Multi-Interest Network and so on.

3. BST and TransAct: order and real time

The next breakage is obvious: pooling does not take the order of events into account. «Bought a phone, then a case» and «bought a case, then a phone» are the same thing to it.

Behavior Sequence Transformer: add positional embeddings to the event vectors and put a transformer in as the encoder. The hidden representation of the target item is used as the user vector.

TransAct is the submodule of Pinterest's ranking model responsible for real time. Its design is worth remembering as a specimen of an engineering solution:

the user's last 100 actions;
a trainable embedding of the type of action, concatenated to the item vector — preserving what it was, not only what it was with;
the item embedding comes from a separate graph model;
a two-layer transformer;
the candidate vector is concatenated to the event vectors — early fusion;
the user vector is a concatenation of the last $K$ vectors and max pooling.

Note the second-to-last point: early fusion buys expressiveness, and it has to be paid for. The bill arrives in section 6.

4. The temperature of attention

Attention degenerates at both ends of the scale

The weights are a softmax of inner products divided by a temperature. Let us look at the entropy of the weights — it measures how «smeared» the attention is. The maximum with 12 events is $\ln 12 = 2.48$.

$\tau$	Entropy of the weights	Maximum weight
2.00	2.43	0.173
1.00	2.20	0.317
0.35	0.66	0.835
0.10	0.01	0.999
0.05	0.00	1.000

The numbers are reproduced by the script _tools/history_demo.py.

At $\tau = 2$ an entropy of 2.43 is nearly the maximum of 2.48 — attention has turned into ordinary averaging, the very thing the whole exercise was meant to escape. At $\tau = 0.05$ the entropy is zero and the weight of one event equals one — attention has turned into a hard selection of a single element of the history, with all the rest thrown away.

This is the same temperature for the same reason as in a two-tower model: the inner products of normalised vectors are squeezed into $[-1; 1]$, and without the division a softmax over them is nearly uniform.

5. SASRec, BERT4Rec and the central lesson

A separate line is transformers not for ranking but for candidate generation. The transfer from language processing is direct: words = items, sentences = users, the task is next-item prediction, the architecture a transformer decoder with a causal mask. That is SASRec.

A useful lens for holding the construction in your head: a language model is a two-tower model with trainable word embeddings, where the left tower processes the context and the right one reduces to an embedding table.

What in the setup of SASRec is out of date

Formally the task is the same as for two-tower candidate generation models: pairs (user, positive), where the user is a sequence of past interactions. But the practices used are older than the ones we worked through:

binary cross-entropy instead of a softmax;
a single negative is sampled, and uniformly at that.

That became the source of years of methodological confusion in the literature.

BERT4Rec applies bidirectional attention instead of causal. To train on the same task it would have had to give up teacher forcing, which is very slow — so it uses a cloze task: mask random tokens and predict them. It also trained on a full softmax with no negative sampling. Its results came out better than SASRec's.

The resolution that makes the story worth remembering

It turned out that SASRec with the right loss is much better than the vanilla one — and beats BERT4Rec. That is, BERT4Rec's original advantage was explained not by bidirectional attention but by its training on a full softmax, while SASRec trained on binary cross-entropy with one uniform negative.

Years of comparing architectures were comparing loss functions.

How useless one uniform negative is

A catalogue of 10⁶. Call «hard» the 100 items nearest to the positive — the ones the model is supposed to learn to tell apart. The probability that a uniformly random negative turns out to be hard: 0.0001.

Negatives per positive	Probability of seeing at least one hard one
1	0.0001
100	0.0100
1000	0.0952
10000	0.6321
100000	1.0000

The numbers are reproduced by the script _tools/history_demo.py.

To see a hard negative even half the time you need 6931 samples instead of one. A full softmax includes all 10⁶.

That is why changing the loss gave more than changing the architecture: with one uniform negative the model almost never sees what it is supposed to learn to distinguish. No architecture will repair that.

A lesson worth taking far beyond this chapter: in recommendation models the training task and the loss function often decide more than the architecture. When comparing two papers, look first at whether their loss and negative-sampling scheme are the same.

6. Early fusion against late fusion

Now about how all this survives under load. The production line of development shows well that the architecture is dictated not by quality but by the budget.

The bill for early fusion

A history of 100 events, dimension 256, two layers, 500 candidates per request.

Fusion	Encoder runs	Operations	At 50 GFLOPS
Late	1	6.27e+07	1.25 ms
Early	500	3.13e+10	627 ms

The numbers are reproduced by the script _tools/history_demo.py.

With a ranking budget of 20 ms, late fusion fits with room to spare and early fusion misses by a factor of 31. The difference is exactly the number of candidates — because under early fusion the user vector depends on the candidate and cannot be reused.

Hence the construction: a two-tower architecture where the transformer over history and context is computed once, the item is encoded by a separate tower, and the score is an inner product. The same «expressiveness against cost» trade-off as in candidate generation, only now arriving at the ranking stage.

Why the history holds clicks rather than impressions

Two arguments, and the second is purely arithmetical.

Informational: an impression was chosen by the previous model, while a click was chosen by the person. Impressions speak about the system, clicks about the user.

Engineering: at a CTR of 5% there are 20 times more impressions, and attention is quadratic in length:

Clicks over a period	Impressions over the same period	Attention costlier by
100	2 000	400 times
500	10 000	400 times
2000	40 000	400 times

The numbers are reproduced by the script _tools/history_demo.py.

Moving from clicks to impressions costs a 400-fold growth in the cost of attention — while carrying less information. The decision is an easy one.

Two practical techniques from the same line

How to measure time on a document. Nobody reports it directly. The standard technique is to measure by the return to the results: if the user's next event happened $t$ seconds later, that is roughly how long they spent on the document. The absence of a return is treated separately. That is how the «deep click» target is built — a stay longer than $N$ seconds.

A neural network as a feature for boosting. An important deployment pattern: the transformer's prediction is fed as a feature into the ranking CatBoost. The network does not replace boosting, it becomes a strong feature for it — which is cheaper and safer than changing the whole stack at once.

7. The length of the history: two loops

We would like to analyse deep history, but it cannot be put into a model that runs per request.

The quadratic term takes its due

Length of history	Operations	Costlier than at 100 events
100	6.27e+07	1.0
500	5.18e+08	8.3
2000	5.14e+09	82.1
8000	6.97e+10	1112.7

The numbers are reproduced by the script _tools/history_demo.py.

At 8000 events the quadratic term is already 94% of the cost. Note the character of the growth: up to a few hundred events the projections dominate and the cost is nearly linear, and beyond that attention eats everything.

There are two ways out, and in practice they are combined.

Event picking — take not the last $n$ events but choose from the deep history by a cleverer principle.
An offline model — compute the vector over the deep history in advance.

The loops do not compete: the vector of the deep history is fed into the fast transformer, which adds the fresh events and the context.

What the offline loop costs and what it gives

The upsides: user embeddings are recomputed once a day; it is noticeably simpler to develop and deploy; for the same resources a heavier model can be used. That is exactly how the history is taken up to two thousand events and the model made several times larger.

The downsides are exactly two and both are fundamental: fresh history is not taken into account and context is not taken into account. A user who started looking for something new ten minutes ago has not changed as far as the offline vector is concerned.

So an offline model does not replace the real-time loop, it complements it.

8. Where this is going

The general direction of development reads in one line: a sum of embeddings → target-aware attention → a full transformer → a single model over the features of the user, the context, the item and the history at once.

Direction	What is done	The limitation
A single transformer	history and features in one model, separate parameters for feature tokens and shared ones for the sequence, a causal mask, SEP between sequences, pyramidal layers	very expensive computationally
Mixing instead of attention	history as keys and values in cross-attention, with the features as queries; cheap token mixing instead of attention	scales better, the line is young
Cross-domain models	history from one service is reused in another	needs a shared identifier infrastructure
A single model for search and recommendations	one catalogue, a merged history, only the context differs	—

The last row deserves separate attention: it literally realises the claim that search and recommendations are one problem with a different query. If the query is simply one more input, the two systems merge naturally into one.

Everything listed is done in the ordinary supervised paradigm. The next step now being tried is to reformulate ranking as a generative problem: autoregressive training, a history length of up to eight thousand events, an order of magnitude more parameters.

Interview questions

How does target attention differ from averaging the history?

In that the weights depend on the candidate: $v_U(A) = \sum_j w_j e_j$, where $w_j = g(e_j, v_A)$. The user no longer has one vector — they have a vector per candidate.

What for: averaging dissolves a rare but decisive event. In a history of 12 events the single music track gets a weight of 0.083 under averaging, while attention gives it 0.835 — ten times more. The score changes sign from −0.56 to +0.83.

When the history is homogeneous the difference is minimal — which is exactly why the problem went unnoticed for so long.

Why does DIN not normalise the attention weights with a softmax?

Because the sum of the weights is interpreted as the intensity of the user's interest in the candidate. Under softmax normalisation it is always one, and that information is lost: a user with ten relevant events in their history and a user with one would give a convolution of the same «mass».

That is, normalisation flattens exactly what one would like to distinguish.

What does the temperature of attention do?

It sets how concentrated the attention is. The weights are a softmax of inner products divided by τ, and at both ends of the scale the construction degenerates.

At τ = 2 the entropy of the weights is 2.43 against a maximum of ln(12) = 2.48 — attention turns into ordinary averaging, the very thing the exercise was meant to escape. At τ = 0.05 the entropy is zero and the weight of one event is one — a hard selection of a single element, with the rest of the history thrown away.

The reason is the same as in a two-tower model: the inner products of normalised vectors are squeezed into [−1; 1], and without the division a softmax over them is nearly uniform.

Tell us about SASRec, BERT4Rec and how their story ended.

SASRec is a transfer from NLP: items as words, users as sentences, the task next-item prediction, a transformer decoder with a causal mask. BERT4Rec uses bidirectional attention and a cloze task instead of causal prediction; it also trained on a full softmax. BERT4Rec showed better results.

The resolution: SASRec with the right loss beats BERT4Rec. The original advantage was explained not by bidirectional attention but by the loss function — in vanilla SASRec it was binary cross-entropy with one uniform negative.

How bad that is: with a catalogue of a million, the probability that a random negative is one of the hundred nearest to the positive is 0.0001. To see a hard negative even half the time you need almost seven thousand samples. The model almost never sees what it is supposed to learn to distinguish, and the architecture has nothing to do with it.

The lesson: when comparing two papers, check the loss and the negative-sampling scheme first.

Why did advertising move away from early fusion?

Because of load. Under early fusion the user vector depends on the candidate, so the transformer over the history has to be run separately for every candidate.

The bill: a history of 100 events, d = 256, two layers, 500 candidates. One run is 6.3e+07 operations, that is, 1.25 ms at 50 GFLOPS. Five hundred runs are 627 ms, which at a ranking budget of 20 ms is a miss by a factor of 31.

The solution is late fusion: a two-tower architecture where the transformer over history and context is computed once, the item is encoded by its own tower and the score is obtained as an inner product. The same «expressiveness against cost» trade-off as in candidate generation, only now at the ranking stage.

Why does the history take clicks rather than impressions?

Two arguments. Informational: an impression was chosen by the previous model while a click was chosen by the person; impressions speak about the system, not about the user. Engineering: at a CTR of 5% there are twenty times more impressions, and attention is quadratic in length, so the cost grows 400-fold.

So impressions are both more expensive and less informative — the decision is an easy one.

How do you work with a very long history?

It cannot be put directly into a model that runs per request: attention is quadratic, and a history of 8000 events is 1113 times more expensive than one of a hundred, with the quadratic term already 94% of the cost there.

Two approaches. Event picking — choose from the deep history by a meaningful principle rather than taking the last n. And an offline loop — compute the vector of the deep history in advance, once a day.

The offline loop has upsides: it is cheap, simpler to deploy, a heavier model fits for the same resources, and the history goes up to two thousand events. The downsides are fundamental: no fresh events and no context. So it does not replace the real-time loop but complements it — the vector of the deep history is fed into the fast transformer.

How is time spent on a document measured?

Nobody reports it directly. It is measured by the return to the results: if the user's next event happened t seconds later, that is roughly how long they spent on the document; the absence of a return is treated separately. The «deep click» target is built on that — a stay longer than N seconds.

This is a good example of a general property of recommendation data: the quantity of interest is not observed, and a proxy with biases of its own is constructed in its place.

One-screen cheat sheet

Averaging

The vector does not depend on the candidate. The single track out of 12 events: weight 0.083 against 0.835.

DIN

$v_U(A)=\sum_j w_j e_j$, $w_j=g(e_j,v_A)$. No softmax — the sum of weights is the intensity of interest.

BST, TransAct

Positional embeddings and a transformer; the type of action as a separate embedding; early fusion.

Temperature

τ=2 → entropy 2.43 against a maximum of 2.48, averaging again. τ=0.05 → a hard selection.

The central lesson

BERT4Rec won by the loss, not by attention. One uniform negative is hard with probability 0.0001.

Fusion

Early: 500 runs, 627 ms. Late: one run, 1.25 ms. The budget is 20 ms.

Clicks, not impressions

There are 20 times more impressions, attention is 400 times costlier, and there is less information.

Two loops

Offline — depth without freshness or context; per request — freshness without depth. Add them together.

Primary sources

G. Zhou et al. Deep Interest Network for Click-Through Rate Prediction, KDD 2018.
Q. Chen et al. Behavior Sequence Transformer for E-commerce Recommendation in Alibaba, DLP-KDD 2019.
W.-C. Kang, J. McAuley. Self-Attentive Sequential Recommendation, ICDM 2018 — SASRec.
F. Sun et al. BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer, CIKM 2019.
A. Petrov, C. Macdonald. Turning Dross Into Gold Loss: is BERT4Rec really better than SASRec?, RecSys 2023 — the resolution from section 5.
X. Xia et al. TransAct: Transformer-based Realtime User Action Model for Recommendation at Pinterest, KDD 2023.
P. Covington, J. Adams, E. Sargin. Deep Neural Networks for YouTube Recommendations, RecSys 2016 — the starting point with averaging.
The numbers in this chapter: _tools/history_demo.py in this repository.