How We Built a Recommendation System (And What Actually Moved the Needle)

Every video platform eventually asks the same question: what should the next video be?

We started with a simple feed built on one fixed rule: overlap with your most recently watched tags, plus a tag affinity weighted by recency. It worked well enough to ship, but it was a fixed rule. It could not learn, and it ignored two of the most useful signals a feed has, namely what is popular right now, and how long people actually watch once they click.

So we built a learned ranker to replace it. This post is the honest version of that project, including the parts where our own numbers lied to us, and the fix that mattered more than the model.

The model

We tried a neural setup with two separate towers first and abandoned it, since there was not enough data density yet to justify the extra complexity over a tree model. We settled on LightGBM with a lambdarank objective: for each user, take one video they actually watched (the positive) and a handful they did not (negatives), and train the model to rank the positive above the negatives.

The negatives matter more than they sound like they should. Early on we sampled them uniformly at random from the catalog. A random video is almost always a video with few views, so "rank the video with more views higher" separated positive from negative almost every time, and the model took that shortcut and became, in effect, a popularity predictor wearing a machine learning costume. Mixing in negatives sampled proportional to view count forced the model to actually discriminate between popular videos, and personalization's share of the model's decisions roughly tripled as a result.

We caught our own metrics lying to us

Here is the part we almost did not write about, because it is not flattering. It is also the most useful thing we learned.

Three separate times during tuning, we got a great validation score, got excited, and then found out the number was fake.

Bug one: early stopping was wired to LightGBM's default ranking metric, not our real evaluation metric (deduplicated NDCG per user on a split ordered by time). The default metric peaked in about three rounds of training, so every "tuned" model we produced had only three trees, and our tuner kept cranking the learning rate to its ceiling to chase it. It looked like progress. It was the tuner overfitting to the wrong yardstick.

Bug two, the sneaky one: our evaluator ranked with a strict "greater than." If a model gave a whole block of videos the exact same score, every relevant video in that block got ranked as if it were first, because nothing scored strictly higher than it. A model that learned to lump fifty videos into one identical score looked like it had solved recommendations. Once we fixed ties to split rank credit fairly, that "great" model's score went from a headline 0.354 down to a flat 0.000. It had not learned to rank anything. It had learned to stop trying.

Bug three: an earlier "good" number came from mixing a fresh data export with an unrelated feature fix landing at the same time, so there were two changes behind one number, with no way to tell which one caused it.

After fixing all three, the honest validation score for our actual shipped model dropped from headline numbers in the 0.15 to 0.35 range down to about 0.115. That is a much less exciting number to put in a deck. It is also the true one, and it is the one we shipped on.

The lesson here generalizes past machine learning: if a metric moved by more than noise and you cannot explain the mechanism, distrust the metric before you trust the win. We now treat any change smaller than about 0.005 on this metric as noise, full stop, and we look for the plumbing bug before we look for a story.

Did it actually beat the old feed?

Offline, yes, clearly. Against the same evaluation, on the same users:

Roughly 4x better at surfacing what a user goes on to actually watch. Reasons why: about half our users have too little history for a fixed rule to say anything useful about them, so the old feed returned nothing meaningful for them, while the ranker still had popularity and general behavior signals to fall back on. The old feed also had exactly one rule; the ranker learns the weighting between a dozen signals instead of hardcoding it.

We shipped it as a 50/50 experiment against the incumbent feed.

It came back flat. No real difference in watch time after a full week.

The bug that mattered more than the model

This is the part of the project we are proudest of, and it has nothing to do with machine learning.

We went looking for why a model that clearly won offline was a tie in production, and found the actual cause a few layers up the stack, before the ranker ever touches a video: the candidate pool it was ranking was a uniformly random 200 videos out of a catalog of 6,600.

A ranker can only rank what it is handed. We measured it directly: that random pool of 200 covered only about 8% of what users actually went on to watch. A pool built from popularity plus recent uploads covered about 67%, a roughly 9x improvement in the ceiling on what the ranker could possibly find, before it makes a single ranking decision.

That is a bigger lever than any amount of feature engineering or hyperparameter tuning we could have done on the ranker itself. We had built a genuinely better judge and then only shown it a handful of random candidates to judge between. The fix was one query change, swapping the random sample for a pool built from popularity and recency, and it shipped in a single commit. Every offline number in this post that mattered was measured after that fix; the flat A/B result before it is void, because it was never really testing the model.

We also tried adding matrix factorization personalization to that candidate pool, expecting it to help further. It did not move coverage at all, since popular videos are just where watches of 30 seconds or more concentrate, and personalized picks mostly displaced popular ones with a lower hit rate instead of adding new ones. We kept it simple: popularity plus recency, no extra infrastructure. The unglamorous fix beat the interesting one.

The feature that looked great and then looked worse

We spent real time on matrix factorization "taste" features for the ranker itself, hoping to replace some of the model's reliance on memorizing specific video IDs (which cannot generalize to new content). First attempt: it looked incredible, 70% of the model's decisions and a huge validation jump. Also fake, since the factorization had been fit on the same data it was being scored against, so it was partly reading its own answer key.

We fixed that leak by fitting five different versions of the factorization, each blind to the exact user and video pair it is about to score. The inflated win disappeared, and once it disappeared honestly, the feature was neutral to negative at every size we tried. We reverted it from the ranker entirely. It survives only as an input to candidate selection, where it earned its keep on real data the ranker never saw during training, not the ranker, where it did not.

Two different features, same shape of mistake: a number that looks too good, sourced from data the model was never supposed to see. We now treat "surprisingly great result" as a signal to go looking for a leak before we go looking for a celebration.

Where it stands

• Offline: the learned ranker beats the old feed by roughly 4x on the same evaluation, held out from training.
• The single biggest lever we found was not the model. It was fixing what videos the model was even allowed to consider.
• We caught three metric bugs and one data leak before they shipped as "wins," each by refusing to trust a number we could not explain the mechanism behind.
• The live experiment restarted from the ship date of the candidate pool fix; that is the read that actually counts, and it is still running.

TL;DR

We replaced a "For You" feed built on one fixed rule with a LightGBM ranker that beat it 4x offline, but three separate metric bugs (wrong early stopping signal, unfair tie handling, a mixed up data export) had produced fake wins along the way, and we caught all three before shipping. The bigger lesson: our first live A/B came back flat, and the cause was not the ranker at all. It was that production was only showing the model a random 200 videos out of 6,600 to choose from. Fixing that one candidate selection bug mattered more than every change to the model combined.

Building something similar

If you are working on a feed, a search ranking, or any product that needs to learn from user behavior instead of running on fixed rules, we have already paid for the mistakes described in this post so you do not have to. That includes catching a fake metric before it ships, and knowing to check the candidate pool before touching the model.

If you want help building this, reviewing an existing recommender, or just want to compare notes on evaluation methodology, email us at [email protected].