Skip to content
Novandri.

Structured Outputs Costs Explained

Structured outputs trade reliability for token and schema overhead. Learn the source mechanisms, a worked cost example, and where the approach breaks.

Rachmadwipa Novandri7 min read

Structured outputs are often adopted as a free reliability upgrade, but every constrained response carries a cost. The OpenAI guide documents structured model outputs, and the Anthropic documentation describes tool use with a defined input schema. The real problem is that teams see the cleaner data and miss the token overhead, latency, and schema upkeep behind it.

Direct answer: Structured outputs force a model to return data that matches a supplied schema, which raises reliability for downstream code. The sources document the two mechanisms. The bounded answer is that this reliability is paid for with schema tokens, added latency, and maintenance, so it should be applied selectively.

Delta: This article adds a claim-status table and a cost decision test beyond the sources: it maps each schema choice to a reliability versus cost decision, with claim status explicit.

The mechanism

The OpenAI guide describes structured model outputs as responses constrained to a supplied schema, often a JSON Schema. Constrained decoding makes the model produce text that conforms to the definition, so a parser can rely on the shape of the result. The practical consequence is that downstream code no longer needs to repair malformed output. The cost is that the schema is part of the request, consuming tokens, and that decoding to a strict format can add processing time.

A flow from a supplied schema through constrained decoding to parseable output, with a retry path on mismatch.
Conceptual: a reliable shape costs tokens, latency, and upkeep.

The Anthropic documentation describes tool use, where a model is given tools with defined input schemas and returns arguments that fit those definitions. The mechanism is similar: the tool definition tells the model what valid input looks like, and the response is meant to match it. The practical consequence is that an agent can call software with structured arguments rather than free text. Again, the definitions consume context, and a mismatch can trigger a retry.

A team that adopts structure everywhere pays for it on every call. A simple classification that could return one word may carry a large schema, a longer prompt, and a validation step. The mechanism that makes the trade-off manageable is to match the strictness to the consumer: strict schemas where code must parse the result, loose formats where a human reads it. Each layer of constraint has a price in tokens and latency, and each layer buys a specific kind of reliability.

The output is a decision about where structure is worth its cost. The sources document the mechanisms; they do not set a budget or a threshold. Those are local choices. The discipline is to record why a schema exists, what it costs, and when it should be revisited, so a reviewer can tell whether the constraint is earning its keep.

A worked example

The following is hypothetical and illustrative. A hypothetical pipeline extracts invoice fields from documents and feeds a ledger. Three checks frame the review. First, does the schema include only fields the ledger needs, rather than every field on the page? Second, how many input tokens does the schema add to each call, and how does that scale across volume? Third, what retry rate follows from validation failures when the document is incomplete? The review finds a large schema driving extra tokens and frequent retries on blank fields. The team trims optional fields and retries only on parse failure. The outcome is a cost decision, not a model change. The numbers are hypothetical; the verification method is the contribution.

Evidence status

ClaimStatusSourceVerifiedRecheck when
Structured outputs constrain responses to a supplied schemasource-reportedOpenAI, Structured outputs2026-11-18API behavior changes
Tool use sends structured arguments defined by a tool schemasource-reportedAnthropic, Tool use2026-11-18Tool API changes
Stricter schemas raise reliability at a token and latency costinferredDerived from the sources2026-11-18Pricing or decoding changes

The first two rows report the mechanisms as documented by each provider. The third is an inference about the cost side, because neither guide states a budget or measures latency for every schema. Recheck the first two claims when the provider documentation changes and the inferred claim when pricing or decoding behavior changes.

What it cannot do

Structured outputs do not make a model correct. They constrain the shape of a response, not its truth. A field can be perfectly formed and still contain a wrong value, because the schema says nothing about the facts. The sources document format constraints, not accuracy guarantees. Treating a valid JSON object as a verified answer is a category error. The reliability gained is syntactic: downstream code stops breaking on malformed text, but the content still needs independent checking.

The hidden cost is schema maintenance. A schema that mirrors a changing business object must be updated whenever that object changes, and every change can ripple through prompts, validators, and tests. There is also a retry cost: when a model fails to satisfy a strict schema, the call is repeated, and repeated calls multiply token spend.

The expense is paid in engineering time and in quiet token waste.

A common failure mode is over-constraint. A schema that is too strict for the task can cause the model to refuse, truncate, or invent values to satisfy the format. The result is clean-looking data that is worse than a loose answer. A related failure is applying structure to a task where a human reads the output, so the constraint buys nothing that matters. Both failures come from treating the schema as free rather than as a design choice with a cost.

There is a boundary around the approach. It covers the format of a response, not the quality of the underlying reasoning, the correctness of a tool implementation, or the safety of the action a tool performs. It does not replace input validation, access control, or a review step before an output is used. A structured argument that fits the schema can still instruct a tool to do the wrong thing, so the boundary must be enforced outside the schema.

Because models and pricing change, the cost decision needs revisiting. A schema that was expensive under one model may be cheap under another, and a format that once needed retries may become stable. Each schema should carry a recheck condition tied to a model or pricing change, and the retry rate should be watched as a live signal. Without that review, a team keeps paying for constraints that no longer buy reliability.

The counterargument that strict schemas are fair for machine-parsed pipelines is incomplete when the added tokens and retries dominate a low-value call.

A decision test

  1. Identify what consumes the model output and whether it needs a strict schema.
  2. List only the fields the consumer actually uses.
  3. Measure the token cost of the schema per call.
  4. Measure validation failures and the resulting retry rate.
  5. Compare strict and loose versions on the same task.
  6. Set a recheck condition tied to a model or pricing change.
  7. Record why each constraint exists and who owns it.

Structure is worth paying for only when the consumer needs it.

The test turns a default choice into a reviewable trade-off between reliability and cost, so a reviewer can see which constraint earns its tokens. For further reading, see Data Quality in Practice.

References & Further Reading