RecSys Supplement · The X algorithm · page 7 of 11
Filtering: twenty-nine reasons not to show a post
Ranking decides in what order. The filters decide whether a post gets into the feed at all. There are twenty-nine of them, they go strictly in turn, and the order is set by cost rather than by taste. Let us see why some stand before scoring and others after, why one task has three independent mechanisms, and how the filters can leave the feed short.
In brief
- The filters go strictly in turn, and the order is meaningful: first the cheap and mass ones, then the expensive and pointwise ones.
- The boundary runs at the selection of the top-50. Everything that requires a trip to another service per «post and viewer» pair stands after it — there are an order of magnitude fewer objects there.
- Three independent mechanisms against a repeat impression: a Bloom filter in the hydrators and two filters from different journals. That is a direct admission that the recording of impressions is unreliable.
- 50 posts are selected and 35 are shown. The margin exists because three more filters work after the selection and it is not known in advance how much they will throw out.
- There is a filter that throws posts away deliberately and for no reason — the deterministic inventory holdout. It is an instrument of measurement, not of filtering.
1. The order and its logic
Eighteen filters work before scoring, three after the selection, the rest are built into other stages. Here is the full list before scoring, in the order in which they stand in the code.
| # | Filter | What it throws out | Cost |
|---|---|---|---|
| 1 | DropDuplicatesFilter | One and the same post returned by several sources | a set in memory |
| 2 | CoreDataHydrationFilter | Posts whose text or author did not load | a field check |
| 3 | AgeFilter | Older than 48 hours | arithmetic on the identifier |
| 4 | SelfTweetFilter | The viewer's own posts | a comparison of numbers |
| 5 | OONRetweetReplyFilter | Other people's replies and reposts, and replies with a lost parent | a field check |
| 6 | OONNsfwSimclustersFilter | Adult content from the cluster source for non-followers | a flag check |
| 7 | RetweetDeduplicationFilter | Repeated reposts of one and the same post | a set in memory |
| 8 | IneligibleSubscriptionFilter | Paid content without a subscription | a flag check |
| 9–11 | PreviouslySeen…, PreviouslyServed… | What has already been shown — from three different journals | sets obtained during hydration |
| 12 | ViewerMutedKeywordFilter | A match with muted words | a string search |
| 13 | AuthorSocialgraphFilter | Blocks and muted accounts | sets from hydration |
| 14 | Brazil2026ElectionFilter | Accounts under a requirement of the Brazilian electoral court | a check against a list |
| 15–16 | VideoFilter, TopicIdsFilter | The wrong type of content for this request | a flag check |
| 17 | NewUserMinEngagementFilter | For entirely new accounts — weakly engaging posts from outside the follows | a comparison of numbers |
| 18 | InventoryHoldoutFilter | A set percentage of posts, deterministically | a hash |
Look at the last column: not a single filter before scoring goes to another service. Everything they need either already lies in the candidate after hydration or is computed by arithmetic. That is exactly why eighteen of them can be afforded on thousands of candidates.
Three considerations, and all three are visible in the list.
First: deduplication goes first. The sources work in parallel and do not know about each other, so one and the same post easily arrives three times. By discarding the duplicates at once we reduce the work for all seventeen following filters. Deduplication is the most profitable position in the queue.
Second: age stands third. It throws out the largest share — everything older than 48 hours — and does so practically for free, because the creation time is sewn into the post's identifier and requires no data. The rule is simple: the maximum rejection per unit of cost goes first.
Third: what depends on the request comes closer to the end. The video and topic filters fire only for part of the requests, and the holdout is off by default entirely. There is no point in putting them first: most of the time they do nothing.
This is exactly the principle of ordering filters that we discussed in the chapter «The order of filtering». Here it is visible line by line.
2. The whole funnel
Let us gather the numbers from the configuration: the sources return up to 1200, 1000, 800 and 200 posts, the scoring threshold is 2800, the top-50 is selected, and 35 go to the screen.
- Switch three of the four sources off. The result barely changes: there are still more than fifty candidates left, and the extra ones simply do not reach the selection. The number of sources influences quality, not fill rate.
- Now pull the slider for the harshness of the filters after the selection. At a multiplier of about 3 fewer than 35 posts remain, and the feed does not fill up. That is what has a direct effect.
- Click on the row «older than 48 hours» and see how much wider the funnel becomes. One filter throws away a fifth of everything.
- Note the three rows about «already shown» in a row — the next section is about them.
What to say in an interview: «Filtering after ranking has to be built into the size of the selection. If you select exactly as many as you show, any spike of rejection at the last stage leaves the user with an incomplete screen».
3. Three mechanisms against one repeat
The most noticeable duplication in the system. Against a repeat impression there work at the same time:
- A Bloom filter arriving with the request at the hydration stage;
PreviouslySeenPostsFilter— the main journal of impressions;PreviouslySeenPostsBackupFilter— the backup journal;- plus
PreviouslyServedPostsFilter— what has already been served in this scrolling session.
And separately: the list of what has already been shown is passed into the source of posts from follows, and it does not return them at all.
Recall how impressions are recorded. On the page about the pipeline we saw that it is a side effect: it starts after the answer has been sent, in the background, and its result is not checked — literally let _ = join_all(futures).await in the code.
So the recording of an impression may not happen. The service restarted, the queue overflowed, the request to the store fell over — the user saw the post, and the system did not learn about it. And on the next request it will honestly show it again.
Hence the strategy: several independent mechanisms, each with its own data path. The Bloom filter arrives with the request, the journals are read from different stores, the session list lives in the request itself. For a duplicate to break through, all of them have to fail at once.
The price is extra work and extra kilobytes in every request. But a repeat impression is noticed by the user instantly and irritates strongly, while an extra filter costs almost nothing. The trade-off is obvious.
4. The filter that throws posts away for no reason
The most unusual of the twenty-nine. InventoryHoldoutFilter removes a set percentage of posts — not bad ones, not old ones, not forbidden ones, just a percentage.
fn holdout_bucket(post_id: u64, viewer_id: u64) -> u64 {
let mut z = post_id
.wrapping_mul(0x9E37_79B9_7F4A_7C15)
.wrapping_add(viewer_id.rotate_left(32).wrapping_mul(0xD1B5_4A32_D192_ED03))
.wrapping_add(0x9E37_79B9_7F4A_7C15);
z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9);
z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB);
z ^= z >> 31;
z % 100
}
fn is_held_out(post_id: u64, viewer_id: u64, percent: u32) -> bool {
Self::holdout_bucket(post_id, viewer_id) < percent as u64
}A fragment of inventory_holdout_filter.rs · code by X, Apache 2.0, commit 28e414f
Let us take it apart. Out of the pair «post identifier and viewer identifier» a number from 0 to 99 is produced by hashing. If it is smaller than the given percentage, the post is thrown out. The percentages are set separately for original posts, replies and reposts; by default they are all zeros, that is, the mechanism is off.
A question that cannot be answered by observation: how much are replies in the feed actually worth? One can look at how much they are clicked — but that is not an answer. If replies are removed, other posts will take their place and part of the engagement will flow there. The observed share of engagement of replies over-states their contribution.
The only way to learn the real contribution is to remove them for part of the traffic and compare. That is exactly what the holdout does: a randomly chosen share of posts is deprived of the chance to be shown, and the difference in behaviour gives a causal estimate.
Note two properties of the implementation. Determinism: the decision depends only on the pair of identifiers, so one and the same post is stably hidden from one and the same viewer over any number of requests — otherwise the user would see flickering. Dependence on both: a post is hidden not globally but for a particular person, so the experiment does not distort the post's overall statistics.
This is the same trick we discussed in the chapter «Exploration in practice»: to measure the value of a mechanism, it is switched off for part of the traffic. What is unusual here is that the unit of randomisation is not a user but a «user and object» pair. That makes it possible to measure the value of a type of inventory rather than of a feature.
5. The filters after the selection
After the sorting and the taking of the top-50, three more filters work, and they are fundamentally more expensive.
| Filter | What it does | Why after the selection |
|---|---|---|
VFFilter | Removes posts for which the visibility service answered «do not show» | A call to a separate service for every «post and viewer» pair |
AncillaryVFFilter | Removes posts whose parent, quoted or reposted post is hidden | Requires the results of the previous filter |
DedupConversationFilter | Collapses several branches of one conversation into one | Makes sense only on the final order |
The first row is the main one. Asking the visibility service about 2800 candidates and about 50 selected ones is a fiftyfold difference in the number of requests. Hence the decision to place it after the sorting, discussed on the overview page.
The third row is subtler. Deduplicating a conversation makes sense only when the order has already been determined: the best branch has to be kept, and which one is best is known only after the scoring.
50 are selected, 35 are shown. The difference is not rounding but a computed margin: three filters after the selection throw out an amount that is not known in advance, and if the margin is not enough, the user gets an incomplete screen.
The widget above shows where the boundary runs: at ordinary harshness about 44 posts remain, at triple harshness already fewer than 35. That is, the system is designed for roughly a twofold margin of safety on the rejection at the last stage.
A generalisation worth carrying away: every filtering after ranking is a tax on the size of the selection. By adding a new visibility rule you are obliged either to increase the selection or to accept the risk of an incomplete output.
Common mistakes and hidden rocks
- Putting expensive filters before ranking. A call to another service per «object and user» pair has to go after the selection: there are an order of magnitude fewer objects there.
- Selecting exactly as many as you show. Any rejection after the selection will leave the screen incomplete. A margin is mandatory and has to be computed, not guessed.
- Considering the triple guard against repeats paranoia. The recording of impressions is a side effect whose result is not checked. One mechanism on such a task is unreliable by construction.
- Putting rarely firing filters at the head of the queue. At the front should be those with the maximum rejection per unit of cost.
- Confusing the inventory holdout with a filter. It protects the user from nothing — it is an instrument of causal measurement that happens to live in the list of filters.
- Forgetting about deduplication between sources. The sources work in parallel and do not know about each other; without the first filter one post travels on in three copies.
Interview questions
In what order do you put filters in a recommender pipeline?
Along two axes at once. By cost: first those that need no external data, then those requiring a trip to another service. By rejection per unit of cost: if a filter is cheap and throws out a fifth, it should be at the front, to reduce the work for everything that follows.
Deduplication usually goes first of all: the sources work in parallel and return overlapping sets. Filters that fire only for part of the requests are reasonably placed closer to the end — most of the time they do nothing.
A separate boundary is the selection of the top-K. Everything that requires a request per «object and user» pair is placed after it: the difference between thousands of candidates and dozens of selected ones gives a saving of tens of times.
Why are more posts selected than are shown?
Because more filters work after the selection, and how much they will throw out is not known in advance. In the system we looked at, 50 are selected and 35 go to the screen — roughly a twofold margin against the rejection of the last stage.
If exactly 35 were selected, any firing of the visibility filter would leave the user with an incomplete screen. A practical rule: every new rule applied after ranking is a tax on the size of the selection, and it has to be paid for either by increasing K or by knowingly accepting the risk.
How do you guarantee that a user will not see the same post twice?
You cannot guarantee it with one mechanism — the recording of impressions happens after the answer has been sent, in the background, and may not happen at all. In the system we looked at, four independent mechanisms work on this task: a compact Bloom filter arriving with the request, two filters from different journals of impressions, and a separate filter over what has already been served in the current scrolling session. Plus the list of what has been shown is passed into the source of posts from follows, and it does not return them at all.
The logic is that the data paths differ: for a duplicate to break through, all of them have to fail at once. The price is small, while a repeat impression is noticed by the user instantly.
How do you measure how much value a particular type of content brings to a feed?
By observation — you cannot. The share of engagement collected by, say, replies over-states their contribution: remove them and you free up positions that other posts will take, and part of the engagement will flow there.
An experiment is needed: remove that type for part of the traffic and compare. In the code we looked at there is a separate filter for this, deterministically hiding a set percentage of posts. Two important details of the implementation: the decision depends on a hash of the «post and viewer» pair, so it is stable — one and the same post does not flicker between requests; and it is personal — the post is hidden for a particular person rather than globally, so the post's overall statistics are not distorted.
What is unusual here is that the unit of randomisation is a «user and object» pair rather than a user. That makes it possible to measure the value of a type of inventory rather than of a piece of functionality.
The age filter throws out everything older than 48 hours. Are we not losing good things?
We are, and knowingly. It is a product decision about what a news feed is: content older than two days has no place in it, however good it may be.
Engineering-wise the decision has pleasant consequences. It bounds the volume the service of fresh posts has to keep in RAM. It gives an upper bound on the size of the journals of impressions. And it is almost free: the creation time is sewn into the post's identifier, no data has to be requested — which is why the filter stands third, right after the deduplication.
The flip side is that the system is structurally incapable of showing you a good post from a week ago. For a feed that is right, for a video service, say, it would be a catastrophe. A threshold of this kind always follows from the product rather than from the technology.
One-screen cheat sheet
How many
29 filters: 18 before scoring, 3 after the selection, the rest built into other stages.
The order
Deduplication → the cheap with a large rejection → the request-dependent → the holdout.
The boundary
The selection of the top-50. After it — only what goes to other services.
Age
48 hours. Almost free: the time is sewn into the post's identifier.
Against repeats
A Bloom filter + two journals + the session list. The recording of impressions is unreliable.
The margin
50 are selected, 35 are shown. The difference is the price of the filters after the selection.
The holdout
A hash of the «post and viewer» pair → 0..99. Deterministic, personal, 0% by default.
The rule
Every rule after ranking is a tax on the size of the selection.
What the filters do not do
Not one before scoring goes to another service. Otherwise there would not be eighteen of them.
Primary sources
- home-mixer/filters/ — all twenty-nine filters.
- inventory_holdout_filter.rs — the deterministic holdout and its hash.
- params/config.rs — the constants: the age of a post, the size of the selection, the size of the answer.
- phoenix_candidate_pipeline.rs — the order of the filters in the pipeline.
- The course: the chapter «The order of filtering» on filtering and its place in the funnel, the same chapter on the Bloom filter, the chapter «Exploration in practice» on experiments.