Skip to content
Novandri.

LLM Inference Cost Pitfalls

Inference cost is set by how tokens are served, not just by the price per token. Read the pitfalls, a worked example, and why cache misses are expensive.

Rachmadwipa Novandri4 min read

Teams budget inference by multiplying tokens by a published price, which misses the operational factors that decide the bill: batching, memory, repeated prefixes, and idle capacity. The PagedAttention paper explains how memory management changes serving efficiency, as the AWS cost framework treats waste as a design problem, as LLM Cost Estimator: How to Predict Your Monthly Bill requires an honest total, as the serving-frameworks article treats batching as a lever.

Direct answer: Control inference cost by measuring tokens per request, reusing cached prefixes, batching where possible, and counting idle capacity — not by looking at a single token price. PagedAttention is the primary source on serving memory, and the AWS cost framework covers waste. The boundary: this is educational orientation, not a guarantee — prices and workloads change, as the benchmarks article warns about broad results, as the publication standard forbids guaranteed savings.

A four-step flow: measure per request, reuse cached prefixes, batch where possible, then count idle capacity.
Conceptual: a cheaper answer that fails costs more. The same system prompt sent on every request is paid for every time.

The pitfalls

Ignoring repeated prefixes.

Assuming perfect batching. Serving frameworks batch requests to raise throughput, which the serving-frameworks article describes, as the PagedAttention paper explains memory fragmentation, as the capacity-planning article advises, as the AWS cost framework treats waste.

Paying for idle capacity. Reserved GPUs cost money when traffic is low, which the cloud-cost article advises, as the AWS cost framework treats utilisation, as the unit-economics article requires for contribution, as the cash-flow article requires for cash spent.

Counting tokens, not outcomes.

A cheaper answer that fails the task costs more, which the cost estimator treats as cost per correct answer, as the evaluation-rubric article requires for concrete criteria, as the routing article advises, as the decision-memo article requires for a recorded judgement.

Three consequences follow. The measurement should be cost per successful task, as the cost estimator requires, as the routing article advises, as the rubric article requires. Capacity should be reviewed against traffic, as the AWS cost framework advises, as the monitoring article treats as a trigger. And the result should be recorded, as the decision-memo article requires.

A worked example

The following is hypothetical and illustrative. A service sends a 2,000-token system prompt with every request and runs an always-on GPU.

Three checks decide. First, can the repeated prefix be cached, as Prompt Caching: Reducing Cost on Repeated Context advises, with the saving measured, as the cost estimator requires? Second, is the GPU utilised enough to justify the reservation, as the AWS cost framework and cloud-cost article ask, with capacity reviewed, as the capacity-planning article advises? Third, what is the cost per successful answer, as the cost estimator and routing article require, with a concrete criterion, as the rubric article requires?

Three conclusions follow. The repeated prefix is a direct saving, which the cache check reveals. Idle capacity is a real cost, as the AWS framework advises. And the decision should be recorded with the metric, as the decision-memo article requires, as the monitoring article advises for review.

What inference-cost analysis cannot do

It cannot guarantee a saving, which the publication standard forbids claiming, as the benchmarks article warns about broad results, as the risk article requires residual risk. It does not replace quality measurement, which the evaluation-rubric article requires, as the routing article treats cost per correct answer, as the hallucination article warns about wrong answers. And it cannot fix a workload that needs the model, which the value-proposition article treats as prior, as the build-versus-buy article advises.

The counterargument that token price is the whole cost is fair as a starting number and incomplete once caching, batching, and idle capacity are counted, which the worked example shows, as the cost estimator advises.

A decision test

  1. Measure tokens per request, as the cost estimator requires.
  2. Reuse cached prefixes, as Prompt Caching: Reducing Cost on Repeated Context advises.
  3. Check batching behaviour, as the serving-frameworks article describes.
  4. Review capacity against traffic, as the AWS cost framework advises.
  5. Count idle cost, as the cloud-cost article advises.
  6. Compute cost per successful answer, as the routing and rubric articles require.
  7. Record the decision and review, as the decision-memo article requires.

For the mechanism, see Prompt Caching; for the total, see LLM Cost Estimator.

References & Further Reading