Skip to content
Novandri.

Why Data Quality Matters More Than Model Complexity

A better model cannot repair a broken measurement. Read where data quality failures enter and how to find them before training.

Rachmadwipa Novandri7 min read

Teams respond to disappointing model performance by changing the model. Sometimes that is right, and more often the ceiling was set upstream: the target was defined inconsistently, a field means something different in two systems, or the label arrives after the prediction moment. No architecture recovers information that was never correctly recorded.

Direct answer:

Model quality is bounded by the quality of the target and features, and the most damaging data problems are semantic rather than technical — a label that does not mean what the team thinks, an identifier that joins the wrong records, a field whose meaning changed without a version bump.

The practical response is to audit the target and the joins before tuning anything, to treat preprocessing as part of the evaluated pipeline, and to monitor the data after deployment as carefully as the model.

A four-step flow: audit the target, check identifiers, check timing, then monitor the data.
Conceptual: the ceiling is set upstream. The most damaging data problems are semantic rather than technical.

The failure modes are mostly invisible

A missing value is a visible problem, and most pipelines handle it. The dangerous failures do not announce themselves:

A target with a delayed definition. A "churned customer" label computed from today's data may include customers who have not yet had time to churn. Training on it teaches the model to predict the observation window rather than the behaviour. This is the same issue that target-trial design surfaces in causal work — the time zero has to be defined before the outcome can be measured.

A join that duplicates or drops records. Joining an events table to a many-row dimension table silently multiplies the training set, and the model sees repeated examples that distort its weighting. A left join that fails quietly removes a segment.

A field whose meaning changed. A product category is recoded, a status field gains a value, a sensor is replaced. The pipeline still runs, the schema still validates, and the model's input distribution has shifted. Sculley et al. list "changes in the external world" among the sources of hidden technical debt in ML systems — and a change in your own upstream system is the same class of problem.

An important behaviour that was never logged. scikit-learn defines an event as a measurable "interaction or occurrence" — a page load, a link click, a purchase, an app crash. If the behaviour that would predict churn was never recorded as an event, no feature can encode it, and the model will do the best it can with proxies.

The leak is a data quality problem

scikit-learn's guidance states that "data leakage occurs when information that would not be available at prediction time is used when building the model", and prescribes splitting before any preprocessing. That rule is about data, not modelling: a scaler or imputer fitted on the full dataset has used validation information, and the resulting score is false.

Target encoding carries the same risk in a more seductive form. A category's mean target value computed over all rows is a powerful feature and a leak, because each row contributed to its own encoding. The fix is out-of-fold computation inside the training split — an operation on data, not a change to the algorithm.

This is why a pipeline object matters: it keeps the transformation attached to the fold, so the discipline does not depend on anyone remembering.

How to audit before you model

The checks that pay off are mundane, and they should happen before the first model run:

  1. Write the target definition in one sentence, including the time at which it becomes knowable. Then check the data against it.
  2. Inspect the distributions of every feature by segment and by time. A field whose distribution shifts sharply at a date is a candidate for a definition change.
  3. Check row counts through each join. If a join changes the count, understand exactly why before proceeding.
  4. Look for impossible values and impossible combinations — a future timestamp, a negative duration, a status that contradicts another field.
  5. Check label availability. For each training row, ask whether every feature was known at the moment the label was defined.
  6. Measure a trivial baseline. A model that cannot beat a simple rule is often being asked to learn from data that does not contain the signal.
  7. Keep a data manifest — the version of every source, the query, and the time of extraction. Without it, a result cannot be reproduced. Datasheets for datasets make the same case for documentation at collection time.

After deployment, the data is still the problem

A model's inputs can change distribution long before its accuracy visibly falls, especially when labels arrive late.

Monitoring should therefore include the input features, their ranges and missingness, and the share of rows falling into segments that were rare in training. When a drift signal fires, the response is to inspect the data — a new source version, a changed category, a broken join — before assuming the model needs retraining.

The SRE practice of alerting on burn rate is a useful analogue. Burn rate is defined as "how fast, relative to the SLO, the service consumes the error budget" — and it exists because a slow drift should be quiet while a fast one should be loud. Data-quality alerts deserve the same design: a small shift is a log, a large one is a page.

A useful reframe

The comparison between model complexity and data quality is not really a contest. A more complex model is a way of extracting more from the data you have; a data-quality fix changes what is available to every model, including the simple one. If you improve the data, every subsequent experiment inherits the gain, which is why the work compounds.

For the pipeline mechanics this belongs to, see MLOps; for what to monitor after launch, see Model Monitoring.

A worked example

The following is hypothetical and illustrative. Suppose a team's churn model is disappointing and the proposed fix is a more complex algorithm. Before retuning, three data checks run. The first is the target definition written in one sentence with the time it becomes knowable: the current "churned customer" label, computed from today's snapshot, includes customers who have not yet had time to churn, so the model is learning the observation window. The second is row counts through each join: an events table joined to a many-row dimension table multiplies training rows, duplicating examples and distorting weights, which the audit treats as a stop-and-explain signal. The third is label availability: for each row, check whether every feature was known at the moment the label was defined, and split before any preprocessing so a scaler fitted on all rows does not leak validation information, as scikit-learn's guidance requires. The outcome is bounded: redefining the label with a proper time zero and fixing the join improves the simple model, so no architecture change is made - the gain comes from data the team already had.

For a case where two independent price feeds disagreed on the label sign and three months were dropped rather than silently accepted, see the BTC Monthly ML case study.

A counterargument

A fair objection is that data quality versus model complexity is a false contest, and the article's own sources weaken the binary. Kuhn and Johnson treat feature construction, resampling, and selection as a discipline inseparable from model choice, which makes the boundary between data work and modelling porous rather than hierarchical. Sculley et al. locate failure in the surrounding system - serving, monitoring, configuration - none of which is purely a data-quality problem, so naming data the dominant variable leaves out the operational layer. A critic would add that the audit checklist carries opportunity costs the piece never weighs: once distributions are inspected and joins counted, further cleanup can flatten in return while a deliberate architecture change unlocks a step gain.

The article rarely shows the model as the binding constraint, yet in some projects that is exactly where a skeptic would look first.

References & Further Reading