RecSys Supplement · The X algorithm · page 8 of 11
Labelling and visibility: may this post be shown
Ranking has decided in what order. What remains is to decide whether it may be shown at all. That is the job of a separate service with separate rules and a separate input — the labels that content classifiers and account reputation models attach continuously. Let us take both halves: how the decision is made and where the data for it comes from.
In brief
- Three possible answers: show, show behind an interstitial, do not show. Not two — the intermediate option exists and matters.
- The first ban breaks off the check. The remaining rules are no longer evaluated, and the name of the one that decided is written into the verdict.
- For recommendations an additional set of rules applies, and it can only forbid. One and the same post is shown to a follower and not to a stranger.
- The reputation of an account is computed in three different ways: by other people's reaction to its posts, by the structure of the follow graph and by the sequence of its own actions.
- The bot detector is the same transformer over a sequence of actions as the recommender model. One architecture, opposite tasks.
1. Three answers instead of two
pub enum VfAction {
Allow,
Drop(FilteredReason),
Interstitial(FilteredReason),
}A fragment of mod.rs · code by X, Apache 2.0, commit 28e414f
The intermediate option — an interstitial the user can tap to see the content — is not an interface detail but an important design decision.
Imagine a system with only two answers. Then for every borderline case — an explicit image, a scene of violence, a shocking photo — one has to choose between «show it to everyone» and «show it to no one». Both options are bad: the first hits those who do not want such things, the second those who deliberately subscribed to them.
The interstitial breaks that dilemma by handing the decision to the user. The system says «there is something here that may be unpleasant for you», and the person decides for themselves.
The engineering consequence matters more than the ethical one: with three answers the cost of a classifier's error falls sharply. A model that identifies explicit content with an accuracy of 90% would, in a two-answer system, on every tenth error either show something it should not or hide something normal. With an interstitial an error of the first kind costs one extra tap. That allows the threshold to be set more aggressively and to catch more.
2. How the verdict is computed
fn evaluate_rules(rules: &[Box<dyn Rule>], context: &RuleContext<'_>) -> Verdict {
let mut worst = VfAction::Allow;
let mut decided_by = None;
for rule in rules {
match rule.evaluate(context) {
VfAction::Drop(reason) => {
return Verdict { action: VfAction::Drop(reason), decided_by: Some(rule.name()) };
}
VfAction::Interstitial(reason) => {
if matches!(worst, VfAction::Allow) {
worst = VfAction::Interstitial(reason);
decided_by = Some(rule.name());
}
}
VfAction::Allow => {}
}
}
...
}A fragment of mod.rs · code by X, Apache 2.0, commit 28e414f
Three properties, each with a meaning.
- A ban breaks off the check immediately. As soon as a rule has said «do not show», the function returns — the remaining twenty-eight rules are not evaluated. That is a saving: the rules go to fetch labels, and superfluous calls cost money.
- An interstitial does not break it off. It is remembered as the current worst verdict, but the loop continues — because a ban may come later, and a ban is stricter.
- The first interstitial wins, not the last. The condition
if matches!(worst, VfAction::Allow)means that a second rule with an interstitial will change nothing.
The verdict records decided_by — the name of the rule. It would seem pointless: the result is the same either way.
The point is that without this field the system becomes undebuggable. The question «why is my post not being shown» has, without it, the answer «one of twenty-nine rules fired». With it — «this particular one fired».
And it is exactly this field that makes possible the report to the user described below. Transparency here is not bolted on the side but follows from the verdict preserving the reason from the very start.
3. Two sets of rules: why a follower is shown more
The rules are split into policies by level of strictness:
pub struct Policies {
filter_all: Vec<Box<dyn Rule>>,
timeline_home: Vec<Box<dyn Rule>>,
timeline_home_recommendations: Vec<Box<dyn Rule>>,
}A fragment of registry.rs · code by X, Apache 2.0, commit 28e414f
The base set is 29 rules applied to everything in the feed. The second set is an additional one, only for recommendations, that is, posts from people the viewer does not follow.
Look at the contents of the second set — the words «high recall» appear in it systematically:
| Rule | What it means |
|---|---|
SPAM_HIGH_RECALL_DROP | Spam caught by a detector with high recall |
NSFW_HIGH_RECALL_DROP | Adult content caught with high recall |
NSFW_HIGH_PRECISION_DROP | The same, but with high precision |
DO_NOT_AMPLIFY_DROP | An explicit «do not amplify» mark |
MALICIOUS_URL_DROP | A malicious link |
COMPROMISED_USER_DROP | An account judged to be compromised |
We went through this trade-off in the chapter «AUC and calibration» and felt it in the AUC widget: moving the threshold, one cannot increase precision and recall at the same time. A high-recall detector catches almost all the spam, but a lot of ordinary posts along with it. A high-precision detector errs rarely, but also misses a lot.
The classic dilemma: which threshold to choose? Here an unexpected answer is given — both, but for different situations.
- A post from an account you follow: only the precise detector is applied. A false positive here is expensive — you yourself chose to read this person, and hiding their post on a suspicion would be an intrusion into your choice.
- A post from an unfamiliar account: the high-recall detector is applied too. A false positive is almost free — you were not expecting this post and will not notice its absence. Whereas letting through spam that the system itself pushed at you is noticeable and unpleasant.
The asymmetry in the cost of an error is turned directly into an asymmetry of threshold. An elegant trick: instead of choosing one point on the curve, two are taken and applied where each is appropriate.
Note the second consequence too: the additional set can only forbid. It cannot allow what the base set forbade. Strictness only grows as you move away from the circle of follows.
4. Where the labels come from
The rules read labels. The labels are produced by the second track of the system — the one that runs continuously and has nothing to do with your request.
Reputation by other people's reaction
The first way of judging an account is to look at how people react to its posts. The names of the metrics are visible in the code: ReportsPerFav, SpamReportsPerFav, with an observation window of 30 days.
The key thing here is a ratio, not an absolute number. A large account gets more reports simply because it is seen more. Counting reports in units would mean punishing size. The ratio of reports to likes does not depend on size.
This is exactly the thought we went through in the smoothed CTR: a raw ratio is noisy on small numbers — an account with two likes and one report has a ratio of 0.5 — and it is smoothed with a prior. The same trick, the same motive.
Reputation by the graph
The second way does not look at the content at all. It runs PageRank over the graph of follows and interactions, obtaining a «mass» for every account, and out of it a score:
private val ScoreSlope = 7.07
private val ScoreIntercept = 165.2
val score = if (mass <= 0) 0.0 else ScoreIntercept + ScoreSlope * scala.math.log(mass)A fragment of UserCredV2.scala · code by X, Apache 2.0, commit 28e414f
The PageRank mass is distributed by a power law — the very one taken apart in the chapter «The long tail and the power law» and which can be turned by hand in the long-tail widget. Between the largest account and the average one the difference is many orders of magnitude.
Such a quantity cannot be used as a feature directly: any linear model will see only the top, and the differences inside the tail will merge into zero for it. A logarithm turns a multiplicative scale into an additive one — a difference of «ten times» becomes a constant shift, no matter where on the scale we are.
The coefficients 7.07 and 165.2 are chosen so that the result lands in a convenient range. Note that there is no substantive meaning in the particular numbers — this is a calibration of the scale, not a discovery about the nature of the graph.
The value of this signal is that it is expensive to fake. Writing spammy text costs nothing, while getting into the centre of the follow graph is impossible without real live people actually following you. Inflating followers with bots hardly helps here: PageRank takes the weight of the source into account, and the mass of the bots themselves is close to zero.
Reputation by an account's own behaviour
The third way is the most interesting for us, because architecturally it repeats the recommender model. From the description in the repository:
A transformer over a sequence of actions that identifies inauthentic accounts — bots, spam, coordinated behaviour — from the stream of their behavioural events.
Let us stop here, because the parallel is striking.
The recommender model takes a user's sequence of actions and predicts what they will do next. The bot detector takes the very same sequence of actions and predicts whether this is a person at all.
One and the same input, one and the same architecture, and even one and the same intuition: a sequence of actions contains more information than any aggregates over it. We justified that in the chapter «Transformers over history» for recommendations — «liked 50 posts about space in an hour» and «liked one post a week for a year» give identical counters but entirely different sequences.
For bot detection it works even more strongly. A bot gives itself away not by what it does — every individual action looks normal — but by rhythm and order: even intervals, no pauses for sleep, identical chains. Exactly the information aggregates destroy and a transformer over a sequence sees.
A practical conclusion for an interview: if you can build a recommender transformer over history, you can build a detector of anomalous behaviour. It is one and the same task with a different target variable.
The rules engine
The labels are attached not by the models directly but by a rules engine: botmaker — a language, a compiler and an executor, scarecrow — the service that applies the rules to events as they arrive. A rule reads as «on such an event, if such conditions hold, attach such a label».
Why a separate language, if an ordinary one would do? For the same reason the visibility rules were moved out of the model: so that a change can be rolled out without shipping code, so that it is readable not only by programmers, and so that it can be checked and disputed.
Part of the rules is absent from the repository — it is stated outright that this is to reduce the risk of circumvention. That is an honest boundary of the openness, and it is worth keeping in mind: the code of the visibility rules is open in full, while part of the logic producing the labels is not.
5. The report to the user as part of the system
Closing the construction is a tool that shows a person which labels are on their account and posts. Daily jobs collect the labels that have been applied, and the server side aggregates them over a period.
Let us return to the limitation we noted on the overview page: open code shows which decisions were made, but not what the model learned from the data. A rule can be read. The weights of a spam classifier cannot, and they are not in the repository.
The report closes exactly that hole, but from the other side. You do not see how the classifier works — but you do see its output as applied to you. And then, from the code of the rules, you can trace what exactly that label does to the visibility of your posts.
The result is a pairing: the code explains the mechanics, the report gives the observable result. Separately each half is almost useless, together they are verifiable. And note that technically all of this rests on one field, decided_by, in the verdict: without preserving the reason, no report could be built.
Common mistakes and hidden rocks
- Thinking there are two decisions. The third option — the interstitial — is fundamental: it allows the classifier's threshold to be set more aggressively, because the cost of a false positive falls to one tap.
- Thinking the rules are the same for all posts. For recommendations an additional set applies, including the high-recall detectors. One and the same post will be shown to a follower and not to a stranger.
- Counting reports in units. A large account gets more reports simply because it is seen more. What is counted is the ratio to likes, and it is smoothed on small numbers.
- Feeding the PageRank mass into a model as it is. It is distributed by a power law; without a logarithm the feature distinguishes only the top.
- Not preserving the reason for the verdict. Without the name of the rule that fired, the system is undebuggable and a report to the user is impossible.
- Confusing the openness of the rules with the openness of the models. The visibility rules are open in full; part of the logic producing the labels is deliberately unpublished.
Interview questions
Why does a moderation system have three possible answers rather than two?
The intermediate option — show behind an interstitial — hands the decision to the user and sharply lowers the cost of a classifier's error. In a two-answer system a false positive means either a hidden normal post or a shown unacceptable one; with an interstitial it costs one extra tap.
The practical consequence: the detector's threshold can be set more aggressively and catch more without paying for it in user irritation. That is, the third answer is not about the interface but about the operating point on the precision-recall curve.
How do you use two detectors of the same thing at once — a high-precision one and a high-recall one?
Apply them in different situations, going by the asymmetry in the cost of an error. In the system we looked at, only the precise detector is applied to posts from accounts the viewer follows: a false positive is expensive here, the person chose to read this author themselves. To recommendations from unfamiliar accounts the high-recall detector is added as well: a false positive is almost free, because the user was not expecting this post, while letting through spam that the system itself pushed at them is noticeable.
Instead of choosing one point on the curve, two are taken and applied where each is appropriate. An important detail: the additional set of rules can only forbid — strictness grows as you move away from the circle of follows, but never falls.
How do you judge an account's reputation, and why are several ways needed?
In the system we looked at there are three, and they lean on fundamentally different data. By other people's reaction: the ratio of reports to likes over 30 days — a ratio precisely, otherwise large accounts would be punished for their size. By the structure of the graph: PageRank over follows and interactions, then the logarithm of the mass. By the account's own behaviour: a transformer over the sequence of its actions.
Several are needed because each has its own vulnerability. Other people's reaction is faked by coordinated reports. The graph is faked by inflating followers, though badly — PageRank takes the weight of the source into account, and the mass of bots is close to zero. Behaviour is faked by imitating a human rhythm, which is expensive at scale. Deceiving all three at once is considerably harder than any one of them.
How is a bot detector similar to a recommender model?
Architecturally they are one and the same: a transformer over a user's sequence of actions. The difference is only in the target variable: the recommender model predicts what the person will do next, the detector whether this is a person at all.
The common intuition is one too: a sequence contains information that aggregates destroy. For recommendations that is the order and the tempo of interests, for detection it is the rhythm: even intervals, no pauses for sleep, repeating chains of actions. Every individual action of a bot looks normal; what gives it away is precisely the sequence.
Why is the visibility decision not built into the ranking model?
Four reasons, and all are visible in the code. A different cost of error — lowering a score and forbidding an impression are incomparable in their consequences. A different speed of change — the rules change under legal requirements within hours, the model is retrained on a schedule. Auditability — a rule can be read and disputed, a lowered score cannot. A guarantee — lowering a score does not guarantee that the post will not be shown: if there are few competitors, it will end up in the output anyway.
The last point is often missed, and it is the decisive one. A hard requirement of «do not show» is implemented only by a hard filter; a soft demotion gives no such promise.
One-screen cheat sheet
Three answers
Show · behind an interstitial · do not show. The third lowers the cost of a classifier's error.
The order of evaluation
The first ban breaks off the check. An interstitial is remembered, but the loop goes on.
Two sets
29 base rules plus additional ones for recommendations. The second can only forbid.
High recall against high precision
The recall detector only for unfamiliar accounts, the precise one for everybody.
Reputation: reaction
The ratio of reports to likes over 30 days, smoothed. Not absolute counters.
Reputation: the graph
PageRank → mass → \(165.2 + 7.07\ln(\text{mass})\). The logarithm because of the power law.
Reputation: behaviour
A transformer over a sequence of actions. The same architecture as in recommendations.
The rules engine
A separate language: rollout without shipping code, readability, auditability.
Transparency
The code explains the mechanics, the report gives the observable result. It rests on the «who decided» field.
Primary sources
- rules/registry.rs — all the rules and the split into policies.
- UserCredV2.scala — turning the PageRank mass into a score.
- bdsm/README.md — the transformer over an account's sequence of actions.
- agatha/scalding/RateBasedBuildJob.scala — metrics of the «reports per like» kind over 30 days.
- The course: the chapter «AUC and calibration» on the trade-off between precision and recall, the chapter «Transformers over history» on transformers over sequences, the chapter «The long tail and the power law» on power laws.