LLM Cost Engineering & Token Economics

Your PolicyOps team just promoted a semantic answer cache for its large language model (LLM) support assistant. Safe public access-policy hits can now return an evaluated answer without generating new text. That doesn't make the bill disappear. Live account checks, exception cases, and cache misses still invoke a model, and nobody can defend the next release with "it should be cheaper."

The missing artifact is a cost ledger for one evaluated support release. It starts from provider-reported token usage, accounts for two very different forms of caching, tests output savings against the answer contract, defers only eligible offline work, and produces a promotion decision. The next chapter will turn that budget contract into gateway routing and fallback policy. We won't build a router here.

The accounting boundary

Cost engineering begins after a request decision is known:

This boundary matters. A semantic answer cache can skip generation. A provider prompt cache reuses matching prefix work but still generates a new output. Treating both as "cache hits" hides both correctness risk and spend.

Use a dated rate card, not remembered prices

Provider pricing is configuration data. For this lab, the teaching fixture snapshots OpenAI's GPT-5.4 short-context rates from May 31, 2026: standard input costs $2.50 per million tokens, standard cached input costs $0.25 per million tokens, and standard output costs $15.00 per million tokens. OpenAI's pricing page publishes Batch rates as a separate row, so the ledger stores those values explicitly too. Its displayed Batch cached-input rate is $0.13 per million tokens; don't replace the published row with a local "divide every rate by two" rule. This dated fixture teaches the ledger shape; it isn't a permanent price promise. Refresh every production rate card from current provider pricing documentation before forecasting a release.^{[1]Reference 1OpenAI API Pricinghttps://developers.openai.com/api/docs/pricing}

The usage trace, not a character-count estimate, is the source of truth after a model call. A provider can expose total input tokens, the cached subset, and output tokens. The fresh-input charge is the total input minus the cached subset.

For reasoning models such as GPT-5.4, provider-reported output_tokens already includes hidden reasoning tokens billed at the output rate. output_tokens_details.reasoning_tokens is useful diagnostic detail, but adding it to output_tokens again would double-charge the ledger.^{[2]Reference 2Reasoning modelshttps://developers.openai.com/api/docs/guides/reasoning}

Price one generated answer

PolicyOps's public-policy-answer path uses a long evaluated instruction prefix and cites policy evidence in every generated response. A cold request reports 1,800 input tokens and 180 output tokens. Split each billable category rather than multiplying one blended token count by one price.

$$\begin{aligned} \text{cost} = &\frac{\text{fresh input tokens} \times \text{fresh input price}}{1{,}000{,}000} \\ &+ \frac{\text{cached input tokens} \times \text{cached input price}}{1{,}000{,}000} \\ &+ \frac{\text{output tokens} \times \text{output price}}{1{,}000{,}000} \end{aligned}$$

The output token rate is higher in this rate card, although this long-input request still spends more dollars on input overall. That distinction is why both token volume and per-category rate belong in the ledger. Decode is autoregressive and serving systems manage growing KV-cache state during generation; those mechanisms help explain why output can be priced differently, while the current rate card remains the billing authority.^{[3]Reference 3Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180[1]Reference 1OpenAI API Pricinghttps://developers.openai.com/api/docs/pricing}

Prefix reuse is still a generated answer

OpenAI's documented prompt caching applies automatically for prompts of at least 1,024 tokens, depends on exact matching at the beginning of the prompt, and reports the cached prefix count under cached_tokens in the response usage details.^{[4]Reference 4Prompt cachinghttps://developers.openai.com/api/docs/guides/prompt-caching} The application design rule is direct: put stable instructions and shared evidence first, then dynamic customer data.

A prefix hit isn't a stored response. The support model still answers the current question and still incurs output spend.

Carry forward the semantic-cache decision

The preceding lesson promoted semantic response reuse only for stable public-policy answers under one evaluated release scope. Cost accounting consumes that decision; it doesn't rebuild its vector index or threshold. A stored-answer hit records zero generated tokens, while a prompt-prefix hit records a newly generated answer at a reduced input cost.

Replay a day of traffic

A per-request example is useful, but a release decision needs traffic shape. For a PolicyOps baseline replay day, before testing a shorter exception answer:

• 3,200 stable policy questions safely hit the semantic answer cache.
• 1,800 policy questions still generate, but reuse the evaluated prompt prefix.
• 3,000 live-order questions must generate from current tool evidence.
• 500 return exceptions generate a longer cited answer with the stable prefix.

The semantic-hit counterfactual answers "what cost did safe reuse avoid?" Actual spend answers "what appears on this release ledger?"

Avoided generation is useful evidence, but it isn't a credit and shouldn't be mixed into actual spend. If an answer hit later fails review, the evaluation consequence matters before its apparent savings.

Reduce output only under an answer contract

Output reductions can matter in a rate card where output is expensive. Blind truncation isn't a cost strategy. PolicyOps's cited support answer must state eligibility, cite the policy evidence, and name the requester's next step. Compare three candidate formats generated on the same input:

The cheapest candidate is rejected. Cost reductions count only after the output still meets the same correctness and evidence contract.

Batch only work that can wait

OpenAI's Batch documentation describes a separate asynchronous workload path with a 50% discount compared with synchronous APIs and a 24-hour completion window.^{[5]Reference 5OpenAI Batch API Guidehttps://developers.openai.com/api/docs/guides/batch} The pricing page publishes Batch token rates explicitly, so the ledger reads that row instead of deriving every category from one multiplier.^{[1]Reference 1OpenAI API Pricinghttps://developers.openai.com/api/docs/pricing} In this fixture, the displayed $0.13 Batch cached-input row means the replay total isn't the exact result of halving every Standard-row decimal locally. That path is useful for nightly regression evaluation of the support release. It isn't appropriate for a customer waiting in a live conversation.

Attribute cost to decisions, not models alone

An invoice tells you how much you spent. A release trace tells you why. Each recorded row should name the evaluated release, feature, decision, pricing mode, rate-card identity, token usage, and output-contract evidence. Recording only a model name can't distinguish an unsafe answer-cache promotion from a harmless increase in live-order traffic. Make contract evidence and rate-card identity mandatory on every trace instead of silently repricing old usage with whichever table is loaded now.

Forecast and gate the release

A cost target without quality is an incentive to produce short, wrong answers. A quality target without cost leaves a production team unable to plan capacity or margins. Require both.

The canary report below evaluates the concise cited-answer policy for return exceptions. The replay's request mix is projected across 30 days, including its nightly offline evaluation run.

This forecast repeats the measured replay day 30 times. Only 4.5% uniform traffic growth would consume the remaining budget, so production planning should also replay low, expected, and high request-volume scenarios. A point forecast that barely passes isn't a comfortable operating margin.

Produce the next policy input

The next lesson will implement model gateway routing and fallbacks. Give it a budget contract, not a half-built router embedded in cost analysis. The contract records what a future route must preserve: citation schema, an approved generated-answer ceiling, and the release evidence behind the limit. The ceiling must be a policy value that the observed maximum fits under, not the observed maximum relabeled as a budget.

What the ledger proves

This release did not become cheaper because of one vague "cache" switch:

Wait for usage traces before claiming a prompt rewrite improved pricing through cached tokens. Count semantic-cache savings only when accepted-hit quality remains valid. Never approve a shorter output merely because it costs less.

Mastery check

Key concepts

• Version rate cards because provider prices and discount rules change.
• Price provider-reported fresh input, cached input, and output separately.
• Reasoning tokens are already included in billed output tokens; don't add them twice.
• A semantic response hit can skip generation; a prompt-prefix hit can't.
• Output reduction is valid only after the answer schema and evidence requirements still pass.
• Batch is a workload-class decision: asynchronous evaluation can wait; live support can't.
• Store explicit rates for each provider pricing mode instead of assuming every discount composes arithmetically.
• Pin rate_card_id on every usage trace so historical spend can't be repriced silently.
• A release promotion needs both a quality check and a budget check.

Practice tasks

1. Add a new warranty-claim-answer traffic slice with longer cited output. Decide whether it breaks the monthly budget.
2. Change the rate card to a newly verified provider/model rate card and rerun the ledger. Explain which request class moves the forecast most.
3. Change one stored trace to a stale rate_card_id. Verify that the ledger rejects repricing it against the loaded table.
4. Add one unsafe semantic answer hit to the quality report. Verify that apparent savings don't allow promotion.
5. Increase every traffic slice by 5%. Verify that the monthly forecast now crosses the $750 budget.
6. Replace the fixed replay with exported usage traces from an application you control, keeping rate-card identity and release identity explicit.

Evaluation rubric

• Foundational: Computes a generated request charge from fresh input, cached input, and output categories.
• Foundational: Uses total output tokens for billing while keeping reasoning-token detail diagnostic.
• Foundational: Explains why answer reuse and prefix reuse can't share one cache_hit metric.
• Intermediate: Attributes spend by feature and decision, including avoided generation without treating it as actual spend.
• Intermediate: Tests an output optimization against a required answer contract.
• Advanced: Rejects historical usage when its pinned rate-card identity doesn't match the loaded pricing table.
• Advanced: Promotes only when a dated rate card, measured traffic replay, quality report, and budget policy all pass.
• Advanced: Hands a budget contract to the routing layer without prematurely implementing routing here.

Self-check questions

Common pitfalls

Estimating the bill from characters

• Symptom: Forecast misses invoice even when request count is correct.
• Cause: Character length or visible response length was treated as provider usage.
• Fix: Store billable usage categories returned by the provider and price them with a versioned rate card.

Calling all reuse a cache hit

• Symptom: Team believes generations were skipped, but output spend remains high.
• Cause: Prefix-cache reuse was confused with stored-answer reuse.
• Fix: Record separate decisions and cost semantics for each layer.

Cutting output below correctness

• Symptom: Spend falls while support answers omit evidence or next actions.
• Cause: Output cap was promoted without evaluating the required answer schema.
• Fix: Make the quality contract a hard promotion gate.

Batching interactive requests

• Symptom: Forecast looks excellent while customer response latency becomes unacceptable.
• Cause: A discount was modeled without preserving completion-time requirements.
• Fix: Batch only explicitly asynchronous work such as nightly evaluation.

Deriving every pricing mode from one multiplier

• Symptom: Forecast drifts from provider billing even though token counts are correct.
• Cause: Standard, Batch, or other processing-mode prices were inferred from one discount instead of loaded from the provider's dated table.
• Fix: Version explicit rates for every pricing mode used by the release and refresh them from current provider documentation.

Repricing historical traces with the current table

• Symptom: The same saved trace produces a different historical spend number after a rate-card refresh.
• Cause: Usage rows stored token categories but omitted the rate-card identity used for the original ledger decision.
• Fix: Pin rate_card_id on each trace and reject mismatched tables instead of silently repricing.

Double-counting reasoning tokens

• Symptom: Ledger output spend is higher than provider billing for a reasoning model.
• Cause: output_tokens_details.reasoning_tokens was added on top of output_tokens, even though total output already includes it.
• Fix: Price output_tokens once. Store reasoning-token detail only for diagnosis and policy analysis.