LLM Cost Engineering & Token Economics

Your PolicyOps team just promoted a semantic answer cache for its large language model (LLM) support assistant. Safe public access-policy hits can now return an evaluated answer without generating new text. That doesn't make the bill disappear. Live account checks, exception cases, and cache misses still invoke a model, and nobody can defend the next release with "it should be cheaper."

The missing artifact is a cost ledger for one evaluated support release. It starts from provider-reported token usage, accounts for two very different forms of caching, tests output savings against the answer contract, defers only eligible offline work, and produces a promotion decision. The next chapter will turn that budget contract into gateway routing and fallback policy. We won't build a router here.

The accounting boundary

Cost engineering begins after a request decision is known:

This boundary matters. A semantic answer cache can skip generation. A provider prompt cache reuses matching prefix work but still generates a new output. Treating both as "cache hits" hides both correctness risk and spend.

Use a dated rate card, not remembered prices

Provider pricing is configuration data. For this lab, the teaching fixture snapshots OpenAI's GPT-5.4 short-context rates from May 31, 2026: standard input costs $2.50 per million tokens, standard cached input costs $0.25 per million tokens, and standard output costs $15.00 per million tokens. OpenAI's pricing page publishes Batch rates as a separate row, so the ledger stores those values explicitly too. Its displayed Batch cached-input rate is $0.13 per million tokens; don't replace the published row with a local "divide every rate by two" rule. This dated fixture teaches the ledger shape; it isn't a permanent price promise. Refresh every production rate card from current provider pricing documentation before forecasting a release.^{[1]Reference 1OpenAI API Pricinghttps://developers.openai.com/api/docs/pricing}

The usage trace, not a character-count estimate, is the source of truth after a model call. A provider can expose total input tokens, the cached subset, and output tokens. The fresh-input charge is the total input minus the cached subset.

For reasoning models such as GPT-5.4, provider-reported output_tokens already includes hidden reasoning tokens billed at the output rate. output_tokens_details.reasoning_tokens is useful diagnostic detail, but adding it to output_tokens again would double-charge the ledger.^{[2]Reference 2Reasoning modelshttps://developers.openai.com/api/docs/guides/reasoning}

Price one generated answer

PolicyOps's public-policy-answer path uses a long evaluated instruction prefix and cites policy evidence in every generated response. A cold request reports 1,800 input tokens and 180 output tokens. Split each billable category rather than multiplying one blended token count by one price.

$$\begin{aligned} \text{cost} = &\frac{\text{fresh input tokens} \times \text{fresh input price}}{1{,}000{,}000} \\ &+ \frac{\text{cached input tokens} \times \text{cached input price}}{1{,}000{,}000} \\ &+ \frac{\text{output tokens} \times \text{output price}}{1{,}000{,}000} \end{aligned}$$

The output token rate is higher in this rate card, although this long-input request still spends more dollars on input overall. That distinction is why both token volume and per-category rate belong in the ledger. Decode is autoregressive and serving systems manage growing KV-cache state during generation; those mechanisms help explain why output can be priced differently, while the current rate card remains the billing authority.^{[3]Reference 3Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180[1]Reference 1OpenAI API Pricinghttps://developers.openai.com/api/docs/pricing}

Prefix reuse is still a generated answer

For GPT-5.4, treat prompt-cache eligibility as observed behavior, not a 1,024-token promise. OpenAI's current guidance, checked on July 31, 2026, places GPT-5.4 with GPT-5.5-and-earlier models: the minimum cacheable prefix varies by model from 1,024 to 2,048 tokens, and a prompt just above 1,024 tokens may not cache consistently. Prompt caching still depends on an exact matching prefix at the beginning of the prompt. Count a cache read only when returned usage reports cached_tokens > 0.^{[4]Reference 4GPT-5.4 Modelhttps://developers.openai.com/api/docs/models/gpt-5.4[5]Reference 5Prompt cachinghttps://developers.openai.com/api/docs/guides/prompt-caching} The application design rule is direct: put stable instructions and shared evidence first, then dynamic customer data.

A prefix hit isn't a stored response. The support model still answers the current question and still incurs output spend. The fixture below represents a returned cache read of 1,280 tokens; it doesn't assert that a prompt at any specific threshold will hit.

Why does a provider prompt-cache hit still have an output charge?

Answer

It reuses processing for an exactly matching input prefix. The model still generates the new response, so output tokens remain billable.

Carry forward the semantic-cache decision

The preceding lesson promoted semantic response reuse only for stable public-policy answers under one evaluated release scope. Cost accounting consumes that decision; it doesn't rebuild its vector index or threshold. A stored-answer hit records zero generated tokens, while a prompt-prefix hit records a newly generated answer at a reduced input cost.

Replay a day of traffic

A per-request example is useful, but a release decision needs traffic shape. For a PolicyOps baseline replay day, before testing a shorter exception answer:

• 3,200 stable policy questions safely hit the semantic answer cache.
• 1,800 policy questions still generate, but reuse the evaluated prompt prefix.
• 3,000 live-order questions must generate from current tool evidence.
• 500 return exceptions generate a longer cited answer with the stable prefix.

The semantic-hit counterfactual answers "what cost did safe reuse avoid?" Actual spend answers "what appears on this release ledger?"

Avoided generation is useful evidence, but it isn't a credit and shouldn't be mixed into actual spend. If an answer hit later fails review, the evaluation consequence matters before its apparent savings.

Reduce output only under an answer contract

Output reductions can matter in a rate card where output is expensive. Blind truncation isn't a cost strategy. PolicyOps's cited support answer must state eligibility, cite the policy evidence, and name the requester's next step. Compare three candidate formats generated on the same input:

The cheapest candidate is rejected. Cost reductions count only after the output still meets the same correctness and evidence contract.

Batch only work that can wait

OpenAI's Batch documentation describes a separate asynchronous workload path with a 50% discount compared with synchronous APIs and a 24-hour completion window.^{[6]Reference 6OpenAI Batch API Guidehttps://developers.openai.com/api/docs/guides/batch} The pricing page publishes Batch token rates explicitly, so the ledger reads that row instead of deriving every category from one multiplier.^{[1]Reference 1OpenAI API Pricinghttps://developers.openai.com/api/docs/pricing} In this fixture, the displayed $0.13 Batch cached-input row means the replay total isn't the exact result of halving every Standard-row decimal locally. That path is useful for nightly regression evaluation of the support release. It isn't appropriate for a customer waiting in a live conversation.

Your nightly evaluation workload is cheaper through Batch. Should the interactive support endpoint use the same path?

Answer

No. The Batch completion window changes latency semantics. Only work that can wait belongs on that path.

Attribute cost to decisions, not models alone

An invoice tells you how much you spent. A release trace tells you why. Each recorded row should name the evaluated release, feature, decision, pricing mode, rate-card identity, token usage, and output-contract evidence. Recording only a model name can't distinguish an unsafe answer-cache promotion from a harmless increase in live-order traffic. Make contract evidence and rate-card identity mandatory on every trace instead of silently repricing old usage with whichever table is loaded now.

Forecast and gate the release

A cost target without quality is an incentive to produce short, wrong answers. A quality target without cost leaves a production team unable to plan capacity or margins. Require both.

The canary report below evaluates the concise cited-answer policy for return exceptions. The replay's request mix is projected across 30 days, including its nightly offline evaluation run.

This forecast repeats the measured replay day 30 times. Only 4.5% uniform traffic growth would consume the remaining budget, so production planning should also replay low, expected, and high request-volume scenarios. A point forecast that barely passes isn't a comfortable operating margin.

Two routes have the same cost per attempt, but one often needs a retry before producing a valid answer. Which cost should the release decision compare?

Answer

Compare cost per successful answer or task, including retries and failed attempts. Cost per request would hide the extra spend caused by lower reliability.

Produce the next policy input

The next lesson will implement model gateway routing and fallbacks. Give it a budget contract, not a half-built router embedded in cost analysis. The contract records what a future route must preserve: citation schema, an approved generated-answer ceiling, and the release evidence behind the limit. The ceiling must be a policy value that the observed maximum fits under, not the observed maximum relabeled as a budget.

Treat that ceiling as a lane-admission artifact for one successful generation, not as a hard bound on total request spend. A failed primary can still bill before timeout, and a contract-preserving fallback bills again. Incident TCO can exceed maximum_generated_answer_usd when attempt costs stack; the gateway lesson tracks request-level attempt spend separately from the per-answer admission ceiling.

What the ledger proves

This release did not become cheaper because of one vague "cache" switch:

Wait for usage traces before claiming a prompt rewrite improved pricing through cached tokens. Count semantic-cache savings only when accepted-hit quality remains valid. Never approve a shorter output merely because it costs less.