Skip to content
Novandri.

Building a Recommendation Engine with Python: End-to-End Pipeline

A recommender is a pipeline, not a model. Read the stages from event logging to serving and where each one quietly fails.

Rachmadwipa Novandri7 min read

Tutorials on building a recommender usually start with a matrix and end with a ranked list, which is a fair description of the model and a poor description of the system. A production recommender is an event pipeline, a training job, a serving path, and a measurement loop — and most of the failures happen at the boundaries between those pieces.

Direct answer: The pipeline runs from event capture through dataset construction, model training, offline evaluation, serving, and online measurement. The critical design decisions are made at the seams: what counts as an interaction, how the training snapshot is built, whether the serving code computes features identically to training, and whether the evaluation split matches the time at which predictions happen. Python is the glue; the correctness lives in those definitions.

A four-step flow: define the interaction, build the training snapshot, train and rank, then serve and measure.
Conceptual: correctness lives at the seams between stages. Everything downstream depends on how an interaction is defined.

Stage one: what is an interaction

Everything downstream depends on how an interaction is defined, and the definition is a business decision disguised as a data field.

A click is not a purchase; a purchase is not a retained customer. A model trained on clicks learns what attracts attention, and a model trained on purchases learns what creates value. The choice determines the whole system, and mixing them without weighting produces a ranker that optimises the easy signal.

The underlying unit is an event. scikit-learn's pipelines assume structured rows; event systems produce a stream that has to be reduced to training examples. The logging has to capture more than the positive event — the candidate set, the position, the time, and the context — because without them you cannot correct for position bias or reconstruct the decision later. This is the same principle as event instrumentation in analytics: what you did not record cannot be analysed.

Stage two: building the training snapshot

The off-by-one errors that quietly destroy recommenders happen here.

For each training example, every feature must have been knowable at the prediction moment. A user's "average purchase value" computed over all their history including the target purchase is a leak. A rolling statistic whose window ends after the prediction timestamp is a leak. A target-encoded item category computed on the full dataset is a leak.

scikit-learn's guidance states the rule generally — "data leakage occurs when information that would not be available at prediction time is used when building the model" — and prescribes splitting before preprocessing. For a recommender this has to be implemented as a time-aware snapshot: take the state of the world as it was at time $t$, compute features from events strictly before $t$, and predict the outcome after $t$.

The split also needs to match deployment. scikit-learn's TimeSeriesSplit builds expanding folds in which training precedes testing — an early fold trains on indices 0–5 and tests on 6–7, a later fold on 0–7 testing on 8–9 — and supports a gap for late-arriving labels.

Stage three: model and ranking

The model stage is the one tutorials emphasise, and in a well-built system it is often the smallest part.

A reasonable progression is: a popularity baseline, a content-based model using item attributes, a collaborative model using interactions, and only then a hybrid or neural model such as the approaches represented by NGCF and DeepFM. Each step has to beat the previous one on the same time-aware evaluation, or it is added complexity without evidence.

DeepFM is built to learn "both low- and high-order feature interactions" end to end, and NGCF encodes the interaction graph in the embedding process. Both can improve ranking, and both require enough data to learn their additional structure — which is why the simpler baselines come first.

Stage four: serving without drift

A model trained correctly can still behave wrongly in production if the features are computed differently at serving time. This is the most common operational failure in recommender systems and the hardest to see, because both paths produce plausible numbers.

The fix is structural: share the feature computation code between training and serving, or generate the serving features from the same pipeline rather than reimplementing them in another language. A feature store is one way of enforcing this; disciplined code sharing is another. What does not work is hoping the two implementations agree.

Serving also needs a fallback. When the model is unavailable, the system must return something sensible — popularity, editorial selection, or the previous ranking — and the fallback should be logged so the measurement does not silently mix two systems.

Stage five: measurement closes the loop

Offline metrics are a filter, not a decision. A recommender that wins offline can lose online because of position bias, interference between users, or a feedback loop that amplifies early clicks.

The online test has to log the candidate set and the position, randomise at a unit that matches the intervention, and run long enough to see retained behaviour. CUPED is relevant to the statistical side: it "utilizes data from the pre-experiment period to reduce metric variability" and reports about a 50% variance reduction on Bing, which is often the difference between detecting a modest ranking improvement and not.

A build order that avoids rework

  1. Define the interaction and write it down. Everything else depends on it.
  2. Instrument the events — candidate set, position, context, time, outcome.
  3. Build a time-aware snapshot builder and test it on known leakage cases.
  4. Implement the baselines first and record their scores.
  5. Add complexity only when the evaluation improves, with the same split and metric.
  6. Share the feature code with serving and verify with a parity test.
  7. Run an online experiment with position logging before claiming the model works.

For the experiment design, see A/B Testing Recommendation Systems; for the model families, see Deep Learning for Recommendations.

A worked example

The following is hypothetical and illustrative. A team is adding a neural ranker to a catalogue recommender and expects an offline win to carry online. Three checks come from the pipeline seams. First, fix the interaction: the event log must record the candidate set, position, time, and context, since a model trained on clicks learns what attracts attention rather than what creates value. Second, rebuild the training snapshot as a time-aware fold, following scikit-learn's rule that preprocessing comes after the split; every feature must be knowable at prediction time, with a gap for late labels as TimeSeriesSplit allows. Third, run a serving parity test on shared feature code, because a model can be trained correctly yet served with silently different features. If any check fails, the neural model is not promoted; the last passing baseline stays live. The bounded outcome is a staged rollout with position logging and an online test, where CUPED-style variance reduction might make a modest lift detectable, or the team keeps the baseline.

A counterargument

A reasonable critic could push back that the article's "correctness lives in the definitions" framing treats the pipeline seams as the whole problem while the cited deep-learning sources point the other way.

DeepFM and NGCF are included as a "reasonable progression," yet they receive little attention beyond a note that they need enough data - the one stage tutorials emphasise is compressed into two sentences, and the claim that the model stage is "often the smallest part" is asserted rather than demonstrated. scikit-learn's cross-validation and common-pitfalls docs ground the leakage and time-split warnings, but they are generic ML guidance, not evidence specific to recommenders; position bias and feedback loops, raised in the measurement stage, are not addressed by those sources at all.

References & Further Reading