Runtime

The main reason is a different load pattern. Features and models are bound by CPU and GPU, candidate generation by memory. In one process the service needs both at once, and it scales horizontally badly: adding replicas for the sake of CPU duplicates the whole index.

The arithmetic: an index of 100 GB, a compute layer of 4 GB. At eight replicas the monolith holds 832 GB and the split system 232 GB, 3.6 times less; the difference is six extra copies of the index.

The other reasons: separation of responsibility, removing a single point of failure, flexibility of experiments, a boundary between teams. The price is possible duplication of part of the data in memory — and at two replicas there is no gain at all.

Immediately after candidate generation. Two reasons.

The first: otherwise we spend expensive inference time on items we know in advance we will not show. With a budget of 500 candidates and 30% rejection a late filter leaves 350 useful ones instead of 500 — that is 43% fewer for the same money.

The second: the output may come out short, which drags the business metrics down. To guarantee 20 positions at 30% rejection you have to score 29, otherwise there is a follow-up request and the response time grows.

The ideal option is to arrange candidate generation so that candidates are born already filtered: no wasted work at all. More expensive to implement.

That all the traffic goes down the stack at once, and the stack is not ready for it. And the dependence is counterintuitive: the better the cache was working, the harder the blow. At a hit rate of 0.95 the layer below is sized for 5% of the traffic and receives 100% — a 20-fold spike; at 0.99 it is 100-fold.

That is, a system with an excellent hit rate looks healthier while in fact being more vulnerable. Hence two rules: choose the cache keys carefully and write separate tests for this. Invalidating the entire cache is not «we will answer more slowly», it is an outage.

k hash functions and a bit array of length m. On insertion we set ones by the hashes, on a check we look at the same positions. The probability of a false positive is \((1 - e^{-kn/m})^k\), the optimum is \(k^* = (m/n)\ln 2\), and at it exactly half the bits are occupied.

The curve over k is U-shaped: at m = 8000 and n = 1000 one hash gives 11.75%, six give 2.16%, twelve give 4.83% again. Too few hashes and there is little discriminating power; too many and the filter fills up.

In recommendations it is for pagination. The system is unstable between requests, so naive pagination produces duplicates; instead a serialised filter with what has already been shown is passed between the frontend and the backend. Bloom in particular fits because there are no false negatives: an error is possible only towards «we will hide something extra», and a duplicate is never shown.

To hold the share of a category in the output when demand changes. A hard quota inserts items at fixed positions and destroys the sense of the output; a controller instead tweaks a bonus to the score, and the output stays sorted by meaning.

The key point is why the integral part is needed. A proportional controller produces an action proportional to the error, so at zero error there is no boost either: the system lives with a permanent shortfall. In the simulation, at I = 0 the share stabilises at 12% instead of 30% — a miss of 17.7 pp, and raising P to 2 only reduces it to 12.7 while adding oscillations. Steady-state error is removed exclusively by I.

The flip side is that too large a P makes the loop swing: at P = 3 the run-up reaches 58% against a target of 30%.

Degrade step by step rather than return a server error. The blender is in pain — switch off the late re-ranking stages and take fewer candidates. Candidate generation is in pain — an emergency ANN or popularity right at the blender. The feature service is in pain — answer with the ANN scores. Everything is in pain — the pumpkin.

Why this is not a rare case: a chain of five services at an availability of 0.999 each gives 0.995, that is, 43.7 hours of downtime a year. And splitting into services worsens this arithmetic — flexibility of scaling is paid for with the number of places that can break.

The transitions between steps have to be thought through in advance, and switched on automatically — with circuit breakers and a tie to utilisation.

A deliberately unintelligent but definitely working mode: popular goods computed once, popular by segment, editorial lists, simple business logic with no personalisation.

The rules: the dumber the pumpkin, the better; for a hard incident it is better to fix a huge list of items in the config so that they are certain to fly through every filter; it is useful to have several pumpkins in sequence; and build them outside the main service — if it is down, it will not serve even a pumpkin.

The main thing: if you have not tested your pumpkin in a long time, you most likely no longer have one.

Besides the ordinary things (latency by percentile, errors, timeouts, cache hits, queue lags) — the specific ones: the shares of the candidate sources, the share of fresh items, the share of fallback answers for each fallback, the coverage by categories, the age of the model, the index and the features, feature coverage and drift.

And the main abstraction is to monitor the whole funnel: how many candidates were pulled up, how many survived the filtering, how many were scored, how many made it into the answer, how long each step took. Such monitoring localises the problem at once, whereas a metric of the final output only says «it got worse».