Skip to content
Novandri.

How to Evaluate AI Search

AI answer engines are hard to measure. Read what can actually be observed, a worked evaluation, and why a tool claiming precision deserves doubt.

Rachmadwipa Novandri4 min read

Evaluating AI search is mostly a problem of honesty. The systems personalise answers, change without notice, and do not expose a public log of why a source was chosen. The result is that many tools sell precision where only directional evidence is available. A useful evaluation starts by separating what can be observed from what must be inferred.

Direct answer:

Evaluate AI search with a defined question set, manual observation across engines and runs, and behavioural signals such as branded search and direct traffic with a comparison.

The measuring-GEO article covers what can and cannot be counted, and AAPOR's standards apply to any survey used. The boundary: no tactic guarantees inclusion or citation, which the publication standard states explicitly for GEO, and attribution is weak, as that article warns for answer-engine traffic.

A four-step flow: define a question set, observe across runs, check behavioural signals, then doubt precision claims.
Conceptual: a citation is an observation, not a guarantee. A citation should be treated as an observation, not a guarantee.

What can be observed

Whether an answer appears. Ask a defined set of questions in the engines you care about, on a schedule, and record the sources cited, as the measuring-GEO article recommends for observable signals. A single check is anecdote; a dated series is evidence.

What the engine cites. The domains cited for your questions tell you whether your material enters the answer set, which the AI-search keyword article treats as a result of clear, extractable content.

Behavioural signals. Branded search, direct traffic, and later conversions, compared with a baseline or holdout, as the brand-lift article explains for separating effect from background. These are indirect and should be labelled as such, as the attribution article warns about credit rules.

Three consequences follow. The question set should be fixed in advance, as AAPOR's standards require for wording and population in a survey. The engines should be named, because results differ. And the horizon should be long enough for preference to accumulate, which the brand-awareness article treats as a quarter-level signal.

A worked evaluation

The following is hypothetical and illustrative. A company defines fifty questions its buyers ask, grouped by decision stage, and checks them monthly across three engines.

After a quarter, its domain is cited in 30% of checks, up from 12%, which the dated observation supports. Branded search rises 18% against a category trend of 5%, which the geo-experiment guidance helps interpret. Three checks decide what can be claimed. First, was the question set fixed before the work, as AAPOR's standards require for comparable wording? Second, is the citation change consistent across engines, or driven by one, as the measuring-GEO article advises for multiple sources? Third, is the branded-search change plausibly linked, or could it be a product launch, as the attribution article warns about alternative causes?

Three conclusions follow. The evidence is directional, which the measuring-GEO article treats as the honest standard. The citation gain is real for the tested set and should not be generalised to all questions, as Keyword Research for AI Search Answers advises for observable signals. And the decision to invest further should be recorded with the limits, as the decision-memo article requires.

What AI-search evaluation cannot do

It cannot tell you why an engine chose a source, which the measuring-GEO article treats as closed. It cannot guarantee a future citation, which the publication standard forbids claiming. And it cannot replace good content, which the content-marketing article argues is the reason to be cited at all.

The counterargument that no precise measurement means no measurement is wrong.

Directional evidence with stated limits is more useful than a fabricated number, which the publication standard requires for all claims.

A decision test

  1. Fix a question set from real buyer language, as AAPOR's wording standards advise.
  2. Name the engines and check on a schedule, as the measuring-GEO article recommends.
  3. Record citations with dates, as the data-governance article requires for relied-upon records.
  4. Pair with behavioural signals and a comparison, as the brand-lift article advises.
  5. Label attribution as weak, as the attribution article warns.
  6. Report direction, not precision, as the measuring-GEO article requires.
  7. Record the decision with its limits, as the decision-memo article advises.

For the measurement limits, see Measuring GEO; for the content method, see Keyword Research for AI Search Answers.

References & Further Reading

  • American Association for Public Opinion Research, 2026. AAPOR — The professional association for public opinion and survey research; its standards and resources underpin the survey-design discussion.
  • Brian Vaver and Koehler, 2011. Measuring Ad Effectiveness Using Geo Experiments — Geo experiments for incrementality, the cleanest way to attribute marketing spend.