LLM Cost Engineering & Token Economics

Your ShopFlow team just promoted a semantic answer cache for its large language model (LLM) support assistant. Safe public return-policy hits can now return an evaluated answer without generating new text. That doesn't make the bill disappear. Live order checks, exception cases, and cache misses still invoke a model, and nobody can defend the next release with "it should be cheaper."

This lesson builds the missing artifact: a cost ledger for one evaluated support release. It starts from provider-reported token usage, accounts for two very different forms of caching, tests output savings against the answer contract, defers only eligible offline work, and produces a promotion decision. The following chapter will turn that budget contract into gateway routing and fallback policy. We won't build a router here.

The accounting boundary

Cost engineering begins after a request decision is known:

This boundary matters. A semantic answer cache can skip generation. A provider prompt cache reuses matching prefix work but still generates a new output. Treating both as "cache hits" hides both correctness risk and spend.

Use a dated rate card, not remembered prices

Provider pricing is configuration data. For this lab, the teaching fixture snapshots OpenAI's GPT-5.4 short-context rates from May 31, 2026: standard input costs $2.50 per million tokens, standard cached input costs $0.25 per million tokens, and standard output costs $15.00 per million tokens. OpenAI's pricing page publishes Batch rates as a separate row, so the ledger stores those values explicitly too. This dated fixture teaches the ledger shape; it isn't a permanent price promise. Refresh every production rate card from current provider pricing documentation before forecasting a release.^[1]

The usage trace, not a character-count estimate, is the source of truth after a model call. A provider can expose total input tokens, the cached subset, and output tokens. The fresh-input charge is the total input minus the cached subset.

Price one generated answer

ShopFlow's public-policy-answer path uses a long evaluated instruction prefix and cites policy evidence in every generated response. A cold request reports 1,800 input tokens and 180 output tokens. Split each billable category rather than multiplying one blended token count by one price.

$$\begin{aligned} \text{cost} = &\frac{\text{fresh input tokens} \times \text{fresh input price}}{1{,}000{,}000} \\ &+ \frac{\text{cached input tokens} \times \text{cached input price}}{1{,}000{,}000} \\ &+ \frac{\text{output tokens} \times \text{output price}}{1{,}000{,}000} \end{aligned}$$

The output token rate is higher in this rate card, although this long-input request still spends more dollars on input overall. That distinction is why both token volume and per-category rate belong in the ledger. Decode is autoregressive and serving systems manage growing KV state during generation; those mechanisms help explain why output can be priced differently, while the current rate card remains the billing authority.^[2][1]

Prefix reuse is still a generated answer

OpenAI's documented prompt caching applies automatically for prompts of at least 1,024 tokens, depends on exact matching at the beginning of the prompt, and reports cached prefix tokens in usage.prompt_tokens_details.cached_tokens.^[3] That gives the application one actionable design rule: put stable instructions and shared evidence first, then dynamic customer data.

A prefix hit isn't a stored response. The support model still answers the current question and still incurs output spend.

Carry forward the semantic-cache decision

The preceding lesson promoted semantic response reuse only for stable public-policy answers under one evaluated release scope. This lesson consumes that decision; it doesn't rebuild its vector index or threshold. A stored-answer hit records zero generated tokens, while a prompt-prefix hit records a newly generated answer at a reduced input cost.

Replay a day of traffic

A per-request example is useful, but a release decision needs traffic shape. For a ShopFlow baseline replay day, before testing a shorter exception answer:

• 3,200 stable policy questions safely hit the semantic answer cache.
• 1,800 policy questions still generate, but reuse the evaluated prompt prefix.
• 3,000 live-order questions must generate from current tool evidence.
• 500 return exceptions generate a longer cited answer with the stable prefix.

The semantic-hit counterfactual answers "what cost did safe reuse avoid?" Actual spend answers "what appears on this release ledger?"

Avoided generation is useful evidence, but it isn't a refund and shouldn't be mixed into actual spend. If an answer hit later fails review, the evaluation consequence matters before its apparent savings.

Reduce output only under an answer contract

Output reductions can be valuable in a rate card where output is expensive. Blind truncation isn't a cost strategy. ShopFlow's cited support answer must state eligibility, cite the policy evidence, and name the customer's next step. Compare three candidate formats generated on the same input:

The cheapest candidate is rejected. Cost reductions count only after the output still meets the same correctness and evidence contract.

Batch only work that can wait

OpenAI's Batch documentation describes a separate asynchronous workload path with a 50% discount compared with synchronous APIs and a 24-hour completion window.^[4] The pricing page publishes Batch token rates explicitly, so the ledger reads that row instead of deriving every category from one multiplier.^[1] That path is useful for nightly regression evaluation of the support release. It isn't appropriate for a customer waiting in a live conversation.

Attribute cost to decisions, not just models

An invoice tells you how much you spent. A release trace tells you why. Each recorded row should name the evaluated release, feature, decision, pricing mode, token usage, and output-contract evidence. Recording only a model name can't distinguish an unsafe answer-cache promotion from a harmless increase in live-order traffic. Make contract evidence mandatory on every trace instead of defaulting it to a pass.

Forecast and gate the release

A cost target without quality is an incentive to produce short, wrong answers. A quality target without cost leaves a production team unable to plan capacity or margins. Require both.

The canary report below evaluates the concise cited-answer policy for return exceptions. The replay's request mix is projected across 30 days, including its nightly offline evaluation run.

Produce the next policy input

The next lesson will implement model gateway routing and fallbacks. Give it a budget contract, not a half-built router embedded in cost analysis. The contract records what a future route must preserve: citation schema, maximum generated-answer spend, and the release evidence behind the limit.

What the ledger proves

This release did not become cheaper because of one vague "cache" switch:

Don't infer a pricing benefit from a prompt rewrite until usage traces show cached tokens. Don't count semantic-cache savings unless accepted-hit quality remains valid. Don't approve a shorter output merely because it costs less.

Mastery check

Key concepts

• Version rate cards because provider prices and discount rules change.
• Price provider-reported fresh input, cached input, and output separately.
• A semantic response hit can skip generation; a prompt-prefix hit can't.
• Output reduction is valid only after the answer schema and evidence requirements still pass.
• Batch is a workload-class decision: asynchronous evaluation can wait; live support can't.
• Store explicit rates for each provider pricing mode instead of assuming every discount composes arithmetically.
• A release promotion needs both a quality gate and a budget gate.

Practice tasks

1. Add a new warranty-claim-answer traffic slice with longer cited output. Decide whether it breaks the monthly budget.
2. Change the rate card to a newly verified provider/model rate card and rerun the ledger. Explain which request class moves the forecast most.
3. Add one unsafe semantic answer hit to the quality report. Verify that apparent savings don't allow promotion.
4. Replace the fixed replay with exported usage traces from an application you control, keeping rate-card identity and release identity explicit.

Evaluation rubric

• Foundational: Computes a generated request charge from fresh input, cached input, and output categories.
• Foundational: Explains why answer reuse and prefix reuse can't share one cache_hit metric.
• Intermediate: Attributes spend by feature and decision, including avoided generation without treating it as actual spend.
• Intermediate: Tests an output optimization against a required answer contract.
• Advanced: Promotes only when a dated rate card, measured traffic replay, quality report, and budget policy all pass.
• Advanced: Hands a budget contract to the routing layer without prematurely implementing routing in this lesson.

Self-check questions

Common pitfalls

Estimating the bill from characters

Symptom: Forecast misses invoice even when request count is correct. Cause: Character length or visible response length was treated as provider usage. Fix: Store billable usage categories returned by the provider and price them with a versioned rate card.

Calling all reuse a cache hit

Symptom: Team believes generations were skipped, but output spend remains high. Cause: Prefix-cache reuse was confused with stored-answer reuse. Fix: Record separate decisions and cost semantics for each layer.

Cutting output below correctness

Symptom: Spend falls while support answers omit evidence or next actions. Cause: Output cap was promoted without evaluating the required answer schema. Fix: Make the quality contract a hard promotion gate.

Batching interactive requests

Symptom: Forecast looks excellent while customer response latency becomes unacceptable. Cause: A discount was modeled without preserving completion-time requirements. Fix: Batch only explicitly asynchronous work such as nightly evaluation.

Deriving every pricing mode from one multiplier

Symptom: Forecast drifts from provider billing even though token counts are correct. Cause: Standard, Batch, or other processing-mode prices were inferred from one discount instead of loaded from the provider's dated table. Fix: Version explicit rates for every pricing mode used by the release and refresh them from current provider documentation.