Skip to content
Novandri.

How to Evaluate Landing Page Testing

A landing page test measures a variant, not a strategy. Read how to evaluate one, a worked example, and why a win can be a measurement artefact.

Rachmadwipa Novandri5 min read

Landing page tests are often judged by the reported lift, which ignores whether the test was valid. The A/B-testing entry describes controlled comparison, as the Google Ads experiment guide offers a design for effects, as the attribution article warns about credit rules, as the evaluation-rubric article requires concrete criteria, as the publication standard forbids guaranteed outcomes.

Direct answer:

Evaluate a landing page test by checking the assignment, the sample, the metric, and the comparison — then deciding whether the result changes a decision.

Wikipedia's A/B-testing entry provides the method, and the experiment guidance offers a design. The boundary: a lift in a short window does not guarantee a durable effect, which the time-series article warns about non-stationarity, as the risk article requires residual risk.

A four-step flow: check assignment, sample, metric, then comparison.
Conceptual: a win can be a measurement artefact. A three-day lift is a starting question, which the checks reveal.

The checks

Assignment and sample. Were visitors randomly assigned, and was the sample large enough, which the A/B-testing entry describes, as the sample-size literature advises, as the data-governance article requires dated figures, as the rubric article requires concrete criteria, as the risk article requires residual risk, as the decision-memo article requires for a record.

Metric definition. What counts as a conversion, which the funnel article treats as a stage measure, as the evaluation-rubric article requires for concrete criteria, as the email-metrics article treats intermediate signals, as the dashboard article argues for metrics tied to decisions, as the attribution article warns about credit rules, as the decision-memo article requires a question.

Comparison period. Was the test run long enough to cover normal variation, which the time-series article warns about non-stationarity, as the experiment guidance advises, as the monitoring article advises for review, as the data-governance article requires dated figures, as the risk article requires residual risk, as the rubric article requires criteria.

Downstream effect. Did the change help beyond the immediate metric, which the retention and unit-economics articles require, as the cohort method advises, as the analytics article recommends for observed behaviour, as the attribution article warns about credit rules, as the decision-memo article requires for a record, as the risk article requires residual risk.

Three consequences follow. The metric comes before the test, as the rubric and dashboard articles advise, as the funnel article treats stages, as the email-metrics article treats signals, as the attribution article warns, as the decision-memo article requires a question, as the data-governance article requires dated figures, as the time-series article warns, as the experiment guidance advises, as the monitoring article advises, as the risk article requires residual risk, as the retention and unit-economics articles require, as the cohort and analytics articles advise, as the decision-memo article requires for a record.

A worked example

The following is hypothetical and illustrative. A test reports a 12% lift in signups from a new headline after three days.

Three checks decide. First, was assignment random and the sample sufficient, as the A/B-testing entry advises, with dated figures, as data governance requires? Second, what is the conversion metric, as the rubric and funnel articles require, with a decision behind it, as the dashboard article argues? Third, did the lift hold across a full cycle, as the time-series and experiment guidance advise, with downstream value, as the unit-economics article requires, as the risk article requires residual risk, as the decision-memo article requires for a record?

Metric definition matters, as the rubric and funnel articles advise. And the decision should be recorded, as the decision-memo article requires.

What landing page testing cannot do

It cannot prove a durable effect from a short window, which the time-series article warns about non-stationarity, as the experiment guidance advises, as the publication standard forbids guaranteed outcomes, as the risk article requires residual risk. It does not replace a clear metric, which the rubric and dashboard articles require, as the funnel article treats stages, as the attribution article warns, as the decision-memo article requires a question. And it cannot fix a weak value proposition, which the value-proposition and positioning articles treat as prior, as the rubric article requires criteria, as the decision-memo article requires for a record, as the risk article requires residual risk.

The counterargument that any lift is worth shipping is fair for reversible changes and incomplete when the metric is a proxy, which the worked example shows, as the rubric and unit-economics articles advise, as the time-series article warns, as the risk article requires residual risk.

A decision test

  1. Define the conversion metric, as the rubric article requires.
  2. Check assignment and sample, as the A/B-testing entry advises.
  3. Run across a full cycle, as the time-series article advises.
  4. Check downstream value, as the unit-economics article requires.
  5. Use dated figures, as data governance requires.
  6. Treat tracked credit as a claim, as the attribution article warns.
  7. Record the decision and residual risk, as the decision-memo and risk articles require.

For the method, see A/B Testing; for the page, see Landing Page Best Practices.

References & Further Reading