A recommender whose most important finding is that recency-weighted popularity wins
A production-style two-stage recommender over 1.3M products, evaluated on a strict global timeline
Summary
- Problem
- Recommenders are easy to over-credit when evaluation leaks future data into training. The goal is a ranked, explained list per user that holds up under time-respecting evaluation, for an Amazon-scale catalogue.
- My role
- Sole developer
- Team
- Personal project
- Status
- In progress
- Timeline
- October 2026
- Stack
- Python, PyTorch, LightGBM (LambdaMART), FastAPI, Parquet, Redis (adapter), Docker
- Next
- Fix the re-ranker's training objective, which likely doesn't match how it is used at serving time, and re-measure.
What shipped
- Showed that recency-weighted popularity (HR@20 0.024 on 20,000 warm users) beat every learned model, including two-tower at 0.00025, by evaluating on a strict global timeline.
- Raised personalisation sixfold, from 3 to 18 distinct top-1 items over 2,000 users, at a 25% HR@20 cost, by fusing popularity with item co-occurrence.
- Served 1.33 million items at 33.0 ms p95 for warm users and 310 requests/s in one process by embedding the feature store in memory.
- Warm-user HR@20, best source
- 0.024
- Baseline: Two-tower 0.00025, ALS 0.0053
- Recency-weighted popularity on 20,000 warm users, exact full-catalogue ranking, one held-out target each (results/evaluation_electronics_0core.json); every learned model's paired-bootstrap difference against it excludes zero
- Personalisation from fusion
- 6× more distinct top-1 items
- Baseline: 25% lower HR@20
- Shipped fusion of recency popularity with co-occurrence gave 18 distinct top-1 items over 2,000 warm users against 3, at HR@20 0.015 against 0.020 (results/fusion_electronics_0core.json); taken deliberately
- Warm-user latency p95
- 33.0 ms
- Baseline: 150 ms target
- 400 requests at concurrency 8, k=20, one process, 1,334,914 items served (results/load_test_electronics_0core.json); all five latency targets met, 310 requests a second
On this page
Source: results/evaluation_electronics_0core.json in the RankStack repository
Context and constraints
Given a user, the service returns the products they are likely to want next, each with a short explanation and the model version that produced it. The data is Amazon Reviews 2023, Electronics, with 43.4M interactions after validation, and raw files are never committed. Every split, feature and metric is defined against a global timeline, every dataset build writes a manifest of hashes, counts and versions, and CI fails if a documented number drifts from its result file.
Architecture
Retrieval narrows the catalogue to a few hundred candidates. Stage 2 scores them with LambdaMART over 18 point-in-time features, then applies a seen filter, a freshness bonus, a category cap and maximal marginal relevance. Features are served from an embedded in-memory store by default, with a Redis adapter behind the same interface for the distributed case. Every failure mode falls down a degradation ladder that returns a working list rather than an error. Explanations come from templates, with no language model in the request path.
Request flows
- Serve a ranked list: Client → Recommendation API → Retrieval (recency popularity + co-occurrence, fused) → LambdaMART re-ranker and business rules → Template explanations
- Load features: Parquet data and manifests → Embedded online feature store
Engineering decisions
How should train and test be split?
- Options
- Leave-one-out per user
- Random split
- A global timeline on absolute timestamps
- Chose
- One global timeline, with the cut points recorded in every result file and leakage tested automatically
- Trade-off
- Numbers are lower and harder to compare with papers that use leave-one-out.
- Evidence
- 57% of test users have no training history and 41% target an item unseen at training time, which a per-user split would hide.
What happens when the learned models lose?
- Options
- Drop them quietly
- Tune until they win
- Report the loss and ship what won
- Chose
- Report the loss and ship what won
- Trade-off
- The headline result is a negative one.
- Evidence
- Two-tower retrieval scored HR@20 0.00025 and the LambdaMART re-ranker scored −0.0173 against its own stage 1; the hand-written rule layer changed no hit rate at all.
Accuracy or personalisation?
- Options
- Serve recency-weighted popularity alone
- Fuse it with item-to-item co-occurrence
- Chose
- Reciprocal-rank fusion of the two
- Trade-off
- HR@20 falls by 25% (paired difference −0.005, 95% CI −0.0095 to −0.0010).
- Evidence
- Fusion gives 18 distinct top-1 items across 2,000 warm users instead of 3, so users stop seeing the same few products.
Should a language model write the explanations?
- Options
- Generate explanations with a language model
- Deterministic templates from the evidence the model used
- Chose
- Deterministic templates
- Trade-off
- Explanations are less varied.
- Evidence
- Every explanation is filled from evidence the ranking actually used, so it can't describe a reason that wasn't there.
Evaluation
Every source is scored by the same harness on the same frozen cohorts, with exact full-catalogue ranking and no sampled negatives.
| Source | HR@20, warm users | HR@20, all users |
|---|---|---|
| Recency-weighted popularity | 0.02395 | 0.016055 |
| Popularity | 0.0069 | 0.006733 |
| ALS | 0.00525 | 0.00587 |
| Co-occurrence | 0.00295 | 0.001985 |
| Two-tower | 0.00025 | 0.000604 |
Against recency-weighted popularity, paired bootstrap over 1,000 resamples gives ALS −0.0187 and two-tower −0.0237, both excluding zero. Nothing learned here beats recency-weighted popularity on this corpus, and that is the result worth stating plainly. A bug that attached every per-item feature to the wrong item was found and fixed; the re-ranking figures were re-measured afterwards, and re-ranking is still worse than stage 1. The results are in results/ in the RankStack repository.
Deployment and operations
400 requests at concurrency 8 met all five latency targets in one process: warm users at 25.1 ms p50 and 33.0 ms p95, cold start at 29.7 ms p95, and the degraded path at 13.9 ms p95, at 310 requests a second over 1,334,914 items. A serving runbook covers every degradation and its alert. CI requires 85% test coverage across unit, property, integration and end-to-end tests.
Results
The service answers within its targets and explains every answer, but two of its three learned components lost their measurements: two-tower retrieval and the LambdaMART re-ranker. The re-ranker’s likely problem is a mismatch between what it was trained on and how it is used. Rather than drop those components quietly, the repository documents them, and the request path serves the fusion that measured best for the trade it makes.
What I'd do next
Retrain the re-ranker on the candidate lists it actually reorders at serving time, then test whether a sequence encoder or an item text tower closes the gap to recency-weighted popularity.
Talk about this project
I'm happy to walk through any decision here in more depth.