RecSys Part V · Engineering · chapter 17 of 19
Data and logging
The models are over — what they stand on begins. This chapter is about the layer that papers and talks usually leave out and on which more deployments break than on the choice of architecture: where features come from at the moment of a request, what exactly to write to the logs, and why an offline metric can be excellent while the online one is not.
- Data differs in freshness and in the cost of computing it — that, not a love of architecture, is where the two loops come from. Everything in the runtime is expensive; everything precomputed is stale.
- Feature skew is the second most frequent cause of an offline-online gap after a wrongly chosen metric. A discrepancy of one formula moves the share above a threshold by 4.6 pp, and offline metrics will not show it at all.
- Coverage is the metric people forget. With 50 features at a coverage of 0.95, the full vector exists for only 7.7% of requests.
- A fragmented history produces a fake cold start. A person with 60 events spread over five identifiers looks like a newcomer under each of them.
1. Two timelines
Let us start with a concrete example. What the model needs at the moment of a request — and what that means technically.
| The product requirement | The technical reality |
|---|---|
| The user likes dairy — show it higher | «likes dairy» is an aggregate over 90 days; it cannot be computed in the 50 ms of a request |
| The milk is already in the basket — stop showing it | visible only in real time |
| It is Friday evening — more beer and snacks | the context of the request, known instantly |
- Different data has different freshness and a different cost of computation.
- You cannot compute everything in the runtime — too expensive.
- You cannot precompute everything — it goes stale, or it is simply impossible.
The basic split: offline — item embeddings and user statistics, refreshed once a day or on triggers; online — the last clicks in the session, geolocation, device, time of day.
An important caveat: the boundary is more often an engineering one than a conceptual one. The same feature can be implemented on either side — it will just cost different money and give different freshness. And if the offline batches are recomputed often, the boundary blurs entirely.
The lambda architecture and its main pain
- Batch, once a day: read all the orders over 30 days, count them, write to the KV store.
- Speed, on every event: a new order lands in the queue, the stream updates the counter incrementally and writes a delta.
- UPS: takes the profile from the batch half, looks at when it was updated, and picks up the required deltas from the speed half.
A subtlety that is easy to miss: an increment for a sliding window is not trivial. You must not only add what is new but subtract what has dropped out — which means storing the events themselves.
108 users, 3 events a day, a window of 30 days:
| an honest window: a ring buffer of events | 9.0e+09 events × 16 bytes = 144.0 GB |
| exponential decay: one number per user | 108 × 8 bytes = 800.0 MB |
| the difference | 180× |
If a stateless update is wanted, the counter is replaced by exponential decay over time. The constant is picked through a half-life: 30 days give \(\lambda = 0.02310\).
| Days ago | 0 | 7 | 30 | 60 | 90 |
|---|---|---|---|---|---|
| weight of the event | 1.0000 | 0.8507 | 0.5000 | 0.2500 | 0.1250 |
The numbers are reproduced by the script _tools/data_demo.py in this repository.
The price: decay has no sharp window boundary, and «over 30 days» turns into «over roughly a month». For a product rule («refund if the purchase was within a month») that is unacceptable; for a model feature it is almost always fine.
The temptation is clear: drop batch altogether, compute everything in the stream, and for a historical recomputation run the same code over a replay of the old data. One codebase, the pain is gone.
Why it is rarely done:
- you still cannot process as much data as a batch would, and by hardware it costs more;
- adding a new feature becomes an engineering adventure always, not only when the feature is a fresh one.
In practice most runtimes live with some kind of lambda — possibly less academic than the diagrams.
The User Profile Service
Data is usually not fetched from the KV stores directly: the merge logic is moved out into a separate service. It receives an identifier, reads everything known about the user from the mirrors, performs a careful merge so as not to double-count — batch may already have taken in part of the events that are also present in the deltas — keeps fallbacks for unavailability, and returns a serialised profile.
The point of extracting a service is exactly one: the merge logic is complicated and must live in a single place. Spread across clients, it will drift apart — for the same reason batch and speed drift apart.
2. Feature skew
The offline job computes the age of a customer as (now − first_order_date).days. The online code computes the same thing as (now − registration_date).days. The numbers came out different — not out of malice: the offline code was written by one team, the online code by another, they agreed to do it the same way, and over time they drifted.
The model trains on one thing and infers on another.
Suppose there are 14 days on average between registration and the first order, and the customer's age is distributed with a mean of 180 and a spread of 120:
| Threshold | Share above it in training | In production | Difference |
|---|---|---|---|
| 30 | 0.894 | 0.914 | +2.0 pp |
| 90 | 0.773 | 0.807 | +3.4 pp |
| 180 | 0.500 | 0.546 | +4.6 pp |
| 365 | 0.062 | 0.077 | +1.6 pp |
The numbers are reproduced by the script _tools/data_demo.py.
A shift of fourteen days moves the share above a threshold by whole percentage points — and that is on one feature out of hundreds. Note where the effect is largest: in the middle of the distribution, where the bulk of the users sits.
- The experiment is rolled out: the offline metrics are good, the model on validation is excellent.
- In the A/B the online metrics are worse — and it is not clear why.
- Random things get debugged for a long time.
- Feature values offline are painstakingly compared against online, and the problem is found.
Feature skew is the second cause of a gap between offline and online results, after a wrongly chosen metric. That is a sentence worth being able to say word for word in an interview.
A Feature Store is the general name for the abstractions that guarantee one and the same way of computing a feature across all clients. Usually it is lambda-architecture processing with a shared feature-computation library: one implementation of the logic, called both offline and online.
3. What to write to the logs
To train, you have to reconstruct the state of the system at the moment of the impression: which items were shown, which features the model saw when it built that ranking, and what the user did. The trouble is that the user's features change all the time. There are two approaches.
| Log the features | Reconstruct them | |
|---|---|---|
| How | at inference time write down the values that were used for the prediction | log only the client-side part and restore the values offline |
| For | exactly what the model saw; no risk of a different distribution; the prediction is reproducible | new features are added instantly; no spend on storage |
| Against | expensive in storage; a new feature has to wait until it has been logged | restoring it exactly is nearly impossible; not every feature is restorable, and that constrains the design; a leak is easy to get |
108 requests a day, 100 candidates per request, 500 features of 4 bytes each:
| What is logged | Per day | Per year |
|---|---|---|
| the features of all the candidates | 20.0 TB | 7.3 PB |
| only the top-10 that were shown | 2.0 TB | 730.0 TB |
The numbers are reproduced by the script _tools/data_demo.py.
From which it is immediately clear that the first decision is not «to log or not» but what exactly to log: narrowing down to what was shown gives a tenfold saving, but at the same time throws away the information about the candidates that were not shown — the information needed for the offline evaluation of policies.
And the second, less obvious price is delay. With a training window of 30 days, a new feature becomes available for training 30 days after you start writing it. That is not about money but about the speed of experiments.
Backend logs: the identifiers of the request, the user and the item; the server-side time; the position; the model's score.
Frontend logs: most importantly the identifier of the request the item came from; the identifiers of the item and the user; the time — server-side or by the moment the log reached the storage (the client's clock cannot be trusted).
The two streams are merged by the request identifier — that is the only honest way to get a true picture of what happened.
Client-side events get lost: the network, a closed tab, blockers. At a true CTR of 0.050:
| Loss of frontend logs | 0% | 2% | 5% | 10% | 20% |
|---|---|---|---|---|---|
| observed CTR | 0.0500 | 0.0490 | 0.0475 | 0.0450 | 0.0400 |
| bias | — | −2.0% | −5.0% | −10.0% | −20.0% |
The numbers are reproduced by the script _tools/data_demo.py.
The key thing here is the mechanism, not the magnitude. A lost click does not become a missing value — it becomes a zero. That is, it turns into a false negative and shifts the target down by exactly the share of the losses, and does so silently: nothing is left in the data to say the event ever happened.
One frontend event is then used in four places: building the dataset for the model, computing features, the A/B for the experiment's metrics, and monitoring and dashboards.
On top of that, frontend logs live long and pass through generations of development. If the schema is not fixed, a year from now it will turn out that a «click» in the dashboard and a «click» in the dataset are different events.
The ID Graph
While assembling the pool it turns out that one real person has several identifiers: a logged-in user_id, the device_id of a phone, the device_id of a laptop, a cookie_id. The history is smeared across them.
A user has 60 events over the period. The threshold beyond which the model considers a history substantial is 20 events.
| Identifiers | Events per identifier | Threshold reached | History visible |
|---|---|---|---|
| 1 | 60.0 | yes | 100% |
| 2 | 30.0 | yes | 50% |
| 3 | 20.0 | yes | 33% |
| 5 | 12.0 | NO | 20% |
The numbers are reproduced by the script _tools/data_demo.py.
With five identifiers not one of them reaches the threshold: a person with sixty events looks like a newcomer on each of their devices. That is the fake cold start — the history exists, but it is fragmented.
The case everyone understands: a week of browsing without logging in, a history accumulated, then registration — now there is a user_id and the history has to be glued to it.
And the hardest case is platforms where users deliberately hide their behaviour: they browse in private mode (a new identifier every time) and do not log in. Without identifier matching everyone will have a permanently cold profile, and the quality of the recommendations will hit a ceiling that has nothing to do with the model.
The conclusion worth stating: working with the right identifiers has to happen at the level of the shared feature layer, not separately in every pipeline. Otherwise the gluing will be implemented differently in three places — and we are back to the story about skew.
4. Data quality and drift
The problems already met: skew, the loss of client-side events, leaks when reconstructing features, the loss of history because of multiple identifiers. Now — how to keep an eye on all that.
Coverage is the share of requests for which a feature is available. And the first thing worth computing is how it behaves as features accumulate:
| Features | All present at a coverage of 0.99 | At 0.95 | At 0.90 |
|---|---|---|---|
| 1 | 0.9900 | 0.9500 | 0.9000 |
| 5 | 0.9510 | 0.7738 | 0.5905 |
| 10 | 0.9044 | 0.5987 | 0.3487 |
| 20 | 0.8179 | 0.3585 | 0.1216 |
| 50 | 0.6050 | 0.0769 | 0.0052 |
The numbers are reproduced by the script _tools/data_demo.py.
With 50 features at a coverage of 0.95 the full vector exists for only 7.7% of requests. Which means handling missing values is not an edge case but the main mode of operation, and it has to be designed accordingly.
A separate danger is a gap between training and production. The model was trained at a coverage of 0.87, while in production the feature is available for 0.60 of requests: the share of requests in an unseen regime grew from 13% to 40%, 3.1 times. The behaviour on such examples is unpredictable.
A sharp drop in coverage almost always means a breakage upstream — a broken pipeline, a changed format, a source that fell off. It is the cheapest alarm signal in existence.
Three kinds of drift
| Kind | What changed | Example | The cure |
|---|---|---|---|
| Covariate | the distribution of \(X\) | a new cohort of users, an ad campaign, a season | retraining on fresh data plus fixing the source |
| Label | the distribution of \(Y\) | the click rate grew because a loyal audience arrived, not because the model got better | work out the cause: seasonality, the product, or a bug |
| Concept | the relation \(X \to Y\) itself | what people used to like they no longer do; a trend has gone stale | the hardest: retraining helps only if the content keeps up |
Telling them apart matters because the second kind is easy to mistake for success. The metric went up, so the model must be good? Not necessarily: the distribution of the target changed and the model had nothing to do with it.
It is computed over bins, usually the deciles of the training distribution. The thresholds accepted in the industry: up to 0.10 — stable, 0.10–0.25 — be on guard, above 0.25 — the distribution has changed.
| Shift of the distribution | PSI over 10 bins | Verdict |
|---|---|---|
| 0.00 σ | 0.0000 | stable |
| 0.10 σ | 0.0096 | stable |
| 0.25 σ | 0.0598 | stable |
| 0.50 σ | 0.2377 | be on guard |
| 1.00 σ | 0.9261 | changed |
The numbers are reproduced by the script _tools/data_demo.py.
The orders of magnitude are worth knowing: a shift of half a sigma trips it confidently, a quarter of a sigma does not yet. The metric is sensitive, and that is a merit: a shift of a quarter of a standard deviation is almost invisible to the eye on a histogram.
For the model's predictions the distribution of the scores at inference is monitored — the mean, the spread, the percentiles. For the targets — daily CTR and conversion by segment.
Before the holidays the patterns change radically and predictably. Which means one can prepare: train a holiday model, or raise the weight of the corresponding period of the previous year.
It is the rare case where the drift is known in advance — it would be a sin not to use that.
Retraining pipelines
| Variant | When it fits |
|---|---|
| By hand — train when you feel like it, deploy manually | good for a start, bad in a complex multi-part production |
| Once a period — a scheduled pipeline | enough for most tasks |
| By trigger — a metric dropped, drift was detected, \(N\) examples accumulated | where the data changes very fast: news, support tickets |
Boosting is trained from scratch every time — it cannot be fine-tuned on small batches. A neural network can be fine-tuned on top of the old weights, with the lower layers frozen and only the head touched, and the embeddings of new items updated incrementally.
This is exactly the checklist item from the chapter on features: «has to fine-tune quickly under drift».
- Forgetting. The network loses patterns that mattered a month ago — critical for rare segments that are barely represented in a fresh window.
- The feedback loop gets stronger. The model recommends → users watch what was recommended → the data is biased → the model amplifies the bias. Online fine-tuning speeds that circle up.
- Anomalies. A bot attack or an outage, and the model absorbs the anomaly quickly — the fresher the data, the quicker.
Usually a combination is used: regular full retraining plus fast fine-tuning on top. The full pass works as an anchor, the fine-tuning as freshness.
Interview questions
Why are two loops of feature computation needed?
Because different data has different freshness and a different cost of computation. «The user likes dairy» is an aggregate over 90 days and cannot be computed in the 50 ms of a request. «The milk is already in the basket» is visible only in real time. Computing everything in the runtime is expensive; computing everything in advance is stale or impossible.
Hence lambda: batch recomputes the aggregates from scratch once a day, speed updates them incrementally per event, serving keeps two mirrored halves, and a separate profile service merges them, allowing for the fact that batch may already have taken in part of the events from the deltas.
The main pain is that one aggregation logic lives in two places: fix the bug twice, deploy twice, often different languages and different teams.
What is feature skew and why does it matter?
A discrepancy in the way a feature is computed between training and inference. The canonical example: offline computes the customer's age from the first order, online from the registration. The model trains on one thing and infers on another.
The scale: with a gap of 14 days the share of customers above the threshold shifts by 4.6 pp in the middle of the distribution, where the bulk of them sits. And that is one feature out of hundreds.
Feature skew is the second most frequent cause of a gap between offline and online results, after a wrongly chosen metric. It is cured by a shared feature-computation library that both offline and online call.
Log the features or restore them later?
Logging gives exactly what the model saw, guarantees the same distribution and lets the prediction be reproduced. The price is storage and delay: at 10⁸ requests, 100 candidates and 500 features that is 20 TB a day and 7.3 PB a year, and a new feature becomes available for training only one training window after you start writing it.
Reconstruction removes both, but restoring the state exactly is nearly impossible, not every feature is restorable (which constrains the design), and a leak from the future is easy to get.
The intermediate solution is to log only what was shown: a tenfold saving, but the information about the candidates that were not shown is lost — and that is what the offline evaluation of policies needs.
What is the minimal set of logs needed?
Backend: the identifiers of the request, the user and the item, the server-side time, the position, the model's score. Frontend: mandatorily the identifier of the request the item came from, plus the identifiers, plus the time — server-side or by arrival in the storage, since the client's clock cannot be trusted.
They are merged by the request identifier. All else being equal, backend logs are preferred: client-side ones get lost, and a lost click does not become a missing value — it becomes a zero, that is, a false negative. At 10% losses the observed CTR shifts down by exactly 10%, and nothing is left in the data to show it.
What is an ID Graph and what is it for?
The gluing together of one person's identifiers: user_id when logged in, the device_id of various devices, cookie_id. Without it the history is smeared.
Concretely: a user has 60 events and the threshold for a substantial history is 20. Across five identifiers that gives 12 each, and not one reaches the threshold — the person looks like a newcomer on every device. That is the fake cold start: the history exists but is fragmented.
It is especially critical where users deliberately hide their behaviour — private mode, no login. Identifiers have to be handled at the level of the shared feature layer, otherwise the gluing will be implemented differently in three places, and we are back to skew.
What is feature coverage and why is it watched?
The share of requests for which the feature is available. It is watched for two reasons.
The first is the arithmetic of accumulation: with 50 features at a coverage of 0.95 each, the full vector exists for only 7.7% of requests. Handling missing values is the main mode of operation, not an edge case.
The second is a gap between training and production: trained at a coverage of 0.87, in production 0.60, and the share of requests in an unseen regime grew from 13% to 40%. The model's behaviour there is unpredictable.
And a sharp drop in coverage almost always means a breakage upstream — the cheapest alarm signal in existence.
What kinds of drift are there and how are they detected?
Covariate — the distribution of X changed (a new cohort, a campaign, a season); cured by retraining and fixing the source. Label — the distribution of Y changed (the click rate grew because a loyal audience arrived); the cause has to be worked out. Concept — the relation X → Y itself changed; the hardest, retraining helps only if the content keeps up.
Telling them apart matters because label drift is easy to mistake for a success of the model.
Detection: for features, PSI over the deciles of the training distribution, with thresholds of 0.10 and 0.25. The orders of magnitude are worth remembering: a shift of half a sigma gives a PSI of 0.24, a quarter of a sigma only 0.06. For predictions — the distribution of the scores at inference. For targets — daily CTR by segment.
How is retraining arranged and what does online fine-tuning risk?
Three variants of the pipeline: by hand (for a start), on a schedule (enough for most tasks), by trigger — a metric dropped, drift was detected, N examples accumulated (for news and the like).
Neural networks have it easier than boosting here: boosting is trained from scratch every time, while a network can be fine-tuned on top of the old weights with the lower layers frozen.
The risks of online fine-tuning: forgetting patterns that matter for rare segments; strengthening the feedback loop, because the model closes in on its own recommendations faster; and quickly absorbing anomalies such as a bot attack. Usually they are combined: regular full retraining as an anchor plus fast fine-tuning on top.
One-screen cheat sheet
Two loops
Different freshness and different cost. All in the runtime is expensive, all precomputed is stale.
Lambda's pain
One logic in two places. Kappa removes it at the price of every new feature being hard.
The window
An honest window is 144 GB of buffer; decay is 800 MB. 180×, but the boundary blurs.
Skew
The second cause of an offline-online gap. 14 days of difference → 4.6 pp at the threshold.
Logs
20 TB a day for all the features. Merge by request_id. Never trust the client's clock.
Frontend losses
A lost click becomes a zero, not a missing value: the bias equals the share of the losses.
Coverage
50 features at 0.95 → a full vector for 7.7%. A drop in coverage = a breakage upstream.
Drift
Covariate / label / concept. PSI: 0.24 at half a sigma. Label drift is easy to read as success.
Primary sources
- D. Sculley et al. Hidden Technical Debt in Machine Learning Systems, NeurIPS 2015 — the canonical work on the model being a small part of the system.
- N. Marz, J. Warren. Big Data: Principles and Best Practices of Scalable Realtime Data Systems, Manning 2015 — the primary source of the lambda architecture.
- J. Kreps. Questioning the Lambda Architecture, 2014 — where the kappa architecture is posed.
- M. Haldar et al. Applying Deep Learning to Airbnb Search, KDD 2019 — the section on the offline-online gap in practice.
- Z. Liu et al. Monolith: Real Time Recommendation System With Collisionless Embedding Table, 2022 — online fine-tuning in production.
- The numbers in this chapter:
_tools/data_demo.pyin this repository.