Skip to content
Novandri.

What Is Model Collapse and Why It Matters

Training on generated data can narrow a model's view of the world. Read the mechanism, the paper's evidence, and what it does to a content pipeline.

Rachmadwipa Novandri5 min read

The internet is filling with text written by models. If future models are trained on that text, a natural question arises: does the process degrade? The model-collapse literature says it can, and the mechanism is more specific than "AI training on AI".

Direct answer: Model collapse is the degradation that occurs when a model is trained recursively on data generated by previous models. The paper The Curse of Recursion reports that this process causes models to progressively lose information, particularly in the tails of the distribution — the rare cases that make a dataset diverse. The boundary: the result is a controlled experimental finding about recursive training, not a claim that all current models are collapsing, and it does not make synthetic data useless.

The mechanism

Training a model on a finite sample approximates the underlying distribution. When the model generates new data, its output reflects that approximation — with the rare events underrepresented, because rare events are poorly estimated from finite samples. Training the next model on this generated data compounds the loss.

A loop: train on a finite sample, generate replacement data, train the next generation, and repeat.
Conceptual: recursive replacement erodes rare tails across generations.

Three consequences follow:

Tails disappear first. Common patterns survive because they appear often in both real and generated data. Rare patterns — unusual phrasings, minority viewpoints, edge cases — are precisely what a finite sample misses, so they vanish over generations.

Variance shrinks. Generated data tends toward the model's average. A dataset built from it becomes more uniform, and a model trained on uniform data loses sensitivity to variation.

Errors compound. A small bias in one generation becomes a larger bias in the next. This is the same compounding principle as error propagation in agent workflows.

The paper's result is a warning about a feedback loop, not a claim about a single training run.

A model trained mostly on human-authored data behaves differently from one trained on successive generations of synthetic output.

Why it matters for a content pipeline

Consider an organisation that uses a model to generate product descriptions, then feeds those descriptions into a future training set. Three risks emerge:

  1. Loss of specificity. Distinctive product attributes are often rare in the text. If they disappear from generated descriptions, a future model cannot learn them.
  2. Homogenisation. Descriptions converge on a house style. That may be desirable for branding and undesirable for recall of unusual features.
  3. Undetected drift. Because the generated text looks fluent, the degradation is not visible in a spot check. Only a comparison against the original distribution reveals it.

The data-curation article covers deduplication and quality filtering as countermeasures. The key practice is provenance: know which parts of a training set are human-authored, which are generated, and in what generation.

A worked example of tail loss

The following is hypothetical and illustrative.

A dataset of 100,000 customer reviews contains 2% — 2,000 reviews — mentioning a rare compatibility issue. A model trained on it generates a replacement corpus of equal size.

If the model underrepresents that 2% and produces it at 0.5%, the replacement corpus has 500 mentions. A second generation trained on the replacement may produce 0.1%, or 100 mentions. After three generations, the pattern is effectively gone.

Three controls change the outcome:

  • Preserve an anchor of original data. Keep a fixed fraction of human-authored examples in every generation. The collapse result is strongest when generated data fully replaces real data.
  • Track the tail explicitly. Measure the frequency of the rare pattern at each generation. An aggregate quality metric will not show its disappearance.
  • Use generation for specific purposes. Synthetic data is valuable for balancing a class or creating edge cases deliberately, as the synthetic-data article discusses. The danger is uncontrolled replacement, not all synthetic data.

The lesson is that collapse is a distributional problem.

What the result does not say

It does not mean all synthetic data is harmful. Deliberate generation with verified labels can improve a dataset. The paper studies recursive replacement under controlled conditions.

It does not prove current models are collapsing. The experiment isolates a mechanism; whether it dominates in a particular training corpus is an empirical question with access to data most researchers do not have.

It does not make human data automatically correct. Human-authored data contains errors and biases. The point is about diversity and provenance, not purity.

The counterargument — that new data will always be generated from human sources — is partly true and complacent.

As generated content grows, the proportion of human-anchored text in some domains falls, which is exactly when provenance and tail measurement matter.

A decision test

  1. Label the provenance of every training source — human, generated, and which generation.
  2. Keep a fixed anchor of original data across retraining cycles.
  3. Measure tail slices explicitly, not only aggregate quality.
  4. Check for homogenisation by comparing diversity across generations.
  5. Use synthetic data deliberately with verified labels and a defined purpose.
  6. Monitor drift after each retraining, with a rollback path.
  7. Treat "it reads fine" as no evidence about distributional health.

For the data practices, see Data Curation for LLM Pretraining; for the related risk, see Data Quality.

References & Further Reading