Skip to content
Novandri.

Feature Engineering: Turning Raw Data into Predictive Signals

Features are where domain knowledge enters a model. Read how to build them without leaking the future or inflating your validation.

Rachmadwipa Novandri7 min read

Feature engineering is usually described as a creative step — the place where a practitioner's intuition turns raw columns into useful signals. That is true, and it is also the source of most silent failures, because a feature is a claim about the world that has to be true at prediction time, not just in the training notebook.

Direct answer: Feature engineering is "adjusting and reworking the predictors to enable models to better uncover predictor-response relationships" — the phrase comes from Kuhn and Johnson's open reference on the subject. It works when the transformation encodes real domain structure and when the entire transformation is fitted inside the training fold. It fails when a feature uses information that will not be available at prediction time, or when preprocessing was applied before the split, which makes the validation score look excellent and the deployed model look random.

A four-step flow: name the signal, fit inside the fold, check time order, then stop when done.
Conceptual: the transformation belongs inside the training fold. Could I have computed this exactly as I did in training?

What a feature actually is

A raw column is a measurement. A feature is a representation built from measurements to make a relationship easier for a model to learn.

A timestamp is a measurement; the hour of day, the day of week, and the distance from a promotion date are features. A price is a measurement; the ratio to a rolling median and the change over seven days are features. Each transformation encodes an assumption about what drives the outcome — that time of day matters, that relative price matters more than absolute — and that assumption is the contribution.

Kuhn and Johnson frame the goal as providing tools for re-representing predictors while placing them "in the context of a good predictive modeling framework" — and that second clause is where most tutorials stop too early. A clever feature inside a broken evaluation is worse than a simple feature inside a sound one, because the clever feature makes the broken evaluation harder to notice.

The five families

Most useful transformations fall into a handful of groups:

Numeric transforms handle scale and shape. Standardisation, log transforms for skewed positive quantities, and binning for nonlinear relationships all change how much a model can learn from a variable. Tree-based models are largely indifferent to monotonic scaling; linear and distance-based models are not.

Categorical encoding turns labels into numbers. One-hot encoding works for low cardinality; target or mean encoding can handle high cardinality but introduces a serious leak risk if computed on the full training set. Hashing trades interpretability for compactness.

Time features encode calendar structure and recency — hour, weekday, time since last event, rolling statistics over defined windows. Rolling features are particularly easy to get wrong: the window must end at or before the prediction moment, and the model must see the same windows in production.

Text and image features come from pretrained models or classical representations such as TF-IDF. They are powerful and they inherit the biases and domain mismatch of the model that produced them.

Interactions combine variables — a ratio, a product, a difference. These are the features where domain knowledge pays most, because a model would need far more data to discover the interaction on its own.

The leak is the main danger

scikit-learn states the rule without ambiguity: "This situation is called overfitting. To avoid it, it is common practice when performing a (supervised) machine learning experiment to hold out part of the available data as a test set". The same documentation warns that "data leakage occurs when information that would not be available at prediction time is used when building the model", and prescribes splitting before any preprocessing.

Feature engineering creates three specific leak risks:

Computing a statistic on the full dataset. A target-encoded category built on all rows includes the validation labels. The fix is to compute it inside the training fold, using out-of-fold encoding for the training rows themselves.

Using future values in a rolling window. A 30-day average ending after the prediction date is a lookahead, even if it is only by one day. scikit-learn's TimeSeriesSplit supports a gap parameter precisely for the related case where recent data is not yet usable.

Creating a feature from an outcome that arrives late. A "customer lifetime value to date" feature that includes events after the prediction moment is unavailable in production by definition. If an event cannot be observed at the moment of prediction, it cannot be a feature.

The discipline is simple: every feature needs a stated answer to the question "at prediction time, could I have computed this exactly as I did in training?"

Time makes the problem harder

A rolling statistic computed across a random split can see values from the future fold.

Expanding-window validation — training on earlier periods and testing on later ones — matches the deployment. scikit-learn's TimeSeriesSplit does this with documented folds in which the training set grows: an early fold trains on indices 0–5 and tests on 6–7, a later fold trains on 0–7 and tests on 8–9. Adding a gap inserts a period between training and test, which matters when labels take time to arrive.

The practical consequence for feature work is that the feature computation has to be refitted inside each fold, not computed once for the whole dataset. A pipeline object is the mechanism that makes this automatic rather than a matter of discipline.

When to stop

Feature engineering has diminishing and then negative returns. More features increase the chance of finding a spurious relationship, complicate the pipeline, and create more places for a leak to hide.

A reasonable stopping rule is to add a feature only when you can state the mechanism in one sentence and test its contribution in a clean validation. If you cannot say why it should matter, it is a candidate for noise, and if you cannot measure its contribution, you do not know whether it helped.

For the evaluation framework, see MLOps; for the broader modelling context, see Data Analytics vs Data Science.

A worked example

The following is hypothetical and illustrative. A team predicts weekly demand for a product and proposes a 30-day rolling average of recent orders as a feature. Check one, availability at prediction time: the team writes the feature's one-sentence mechanism and confirms the window ends at or before the prediction moment, so no future order leaks in - the rule scikit-learn states when it warns that data leakage uses information unavailable at prediction time. Check two, leakage inside the split: target encoding for the product category is computed only within the training fold, using out-of-fold values for training rows, because computing it on all rows folds validation labels into the feature. Check three, time-ordered validation: the pipeline refits inside each expanding fold via TimeSeriesSplit with a gap for late-arriving labels, rather than computing the rolling window once over the full dataset. The bounded outcome is a validation score the team can defend as honest and a deployed feature computed the same way; a higher score from a random split or a pre-split scaler would be treated as a leak, not a gain.

For an application that ablates feature blocks under a leakage-controlled walk-forward protocol, see the BTC Monthly ML case study: removing the month-to-date block cost 5.3 points, while removing event features improved accuracy.

A counterargument

A reasonable critic can object that the article's stopping rule - add a feature only when its mechanism fits one sentence - is stricter than its sources support. Kuhn and Johnson present tools for re-representing predictors and for selection, and scikit-learn documents leakage and validation mechanics, but neither establishes that a useful feature must have a one-sentence human explanation. The article's own list includes text and image features from pretrained models, whose value often lies in representations no one can narrate simply.

The objection is not that unexplained features are safer; it is that the rule converts a heuristic for avoiding noise into a test of legitimacy the references do not provide.

Under conditions where a feature improves out-of-fold performance but resists a crisp story, the framework would discard a real signal in the name of interpretability.

References & Further Reading