LearnApplied LLM EngineeringLLM Cost Engineering & Token Economics

⚙️MediumMLOps & Deployment

LLM Cost Engineering & Token Economics

Build an auditable LLM cost ledger from usage traces, cache decisions, output contracts, offline batch work, and release budget gates.

20 min read

Learning path

Step 83 of 196 in the full curriculum

Semantic Caching & Cost Optimization Model Gateways, Routing, and Fallbacks

Personalize this lesson

Adapt explanations and teaching visuals to your background and preferred voice.

In our worked PolicyOps release receipt, an on-call alert fires at 09:12: one traffic day plus its nightly evaluation costs $23.92. A 30-day projection still fits under the $750 budget, but leaves only $32.44. The release can pass its hard gate and still deserve a closer look.

The dashboard labels 3,200 requests as cache hits. Some returned an already evaluated answer and made no model call. Others reused a provider prompt prefix and still generated new text. Collapse those events into one label and you can't tell avoided spend from discounted spend, or know which correctness check failed.

PolicyOps's support assistant is built around a large language model (LLM). It also handles live-order lookups and cited exception answers. We need a cost ledger for one evaluated release: provider-reported token usage, both cache kinds, output savings that preserve the answer contract, work that can wait, and a promotion decision. The next lesson will consume that budget contract when it routes and falls back. First classify each call.

The accounting boundary

Before pricing, classify what happened. "Cache hit" can mean no model call or a generated answer with cheaper input. Use one row for each outcome:

Decision	What executes?	What belongs in this ledger?
Semantic answer hit from the previous lesson	Stored evaluated answer returned	No generation charge; record avoided generation separately
Generated answer with provider prompt-cache hit	Model still produces a new answer	Fresh input, cached input, and output token charges
Generated answer without prompt-cache hit	Model reads full input and produces a new answer	Fresh input and output token charges
Offline evaluation batch	Model runs later, outside interactive path	Batch-eligible token spend and completion contract

Only the semantic answer hit has no generated usage. A provider prompt cache reuses matching prefix work, then generates a fresh answer. That distinction determines both the invoice row and the correctness test you need.

Ask two questions in order: did we return an evaluated answer, and if not, did the provider reuse an input prefix? The flow below turns those answers into billable categories.

Diagram showing Support request, Safe stored answer?, No generation charge, and Generate new answer. — Support request, Safe stored answer?, No generation charge, and Generate new answer.

Use a dated rate card, not remembered prices

Treat provider pricing as versioned configuration. This lab's teaching fixture freezes OpenAI's GPT-5.4 short-context row. Checked against the official pricing and model pages on August 21, 2026, it matches the May 31, 2026 snapshot used in the code: standard input $2.50 per million tokens, cached input $0.25 per million, and output $15.00 per million.^{[1]Reference 1OpenAI API Pricinghttps://developers.openai.com/api/docs/pricing}^{[2]Reference 2GPT-5.4 Modelhttps://developers.openai.com/api/docs/models/gpt-5.4}

OpenAI also publishes a long-context row. Above 272K input tokens, the full session uses $5.00 / $0.50 / $22.50 for input, cached input, and output. Every trace here stays below that cliff, so the ledger selects the short-context row before doing any token arithmetic.^{[1]Reference 1OpenAI API Pricinghttps://developers.openai.com/api/docs/pricing}^{[2]Reference 2GPT-5.4 Modelhttps://developers.openai.com/api/docs/models/gpt-5.4}

Batch rates are another published row, not a local "divide every Standard rate by two" rule. The displayed Batch cached-input rate is $0.13 per million tokens; halving $0.25 gives $0.125, so local arithmetic would lose the displayed rate. This dated fixture teaches ledger shape, not a permanent price promise. Refresh production cards from current provider documentation before forecasting a release.^{[3]Reference 3OpenAI Batch API Guidehttps://developers.openai.com/api/docs/guides/batch}^{[1]Reference 1OpenAI API Pricinghttps://developers.openai.com/api/docs/pricing}

The usage trace, not a character estimate, settles the bill after a model call. Read total input tokens, the cached subset, and output tokens from that trace. Fresh input is total input minus cached input.

Reasoning models add a tempting accounting trap. For GPT-5.4, provider-reported output_tokens already includes hidden reasoning tokens billed at the output rate. output_tokens_details.reasoning_tokens explains part of that total; adding it again would double-charge. A call can also hit max_output_tokens while thinking, leaving no visible answer while its output tokens still bill.^{[4]Reference 4Reasoning modelshttps://developers.openai.com/api/docs/guides/reasoning}

Before running the first cell, predict the split for a 1,800-token trace with 400 cached tokens: 1,400 fresh tokens and 400 cached tokens. The rate card and usage objects make that subtraction explicit.

01-rate-card-and-usage.py

from collections import defaultdict
from dataclasses import dataclass, replace
from decimal import Decimal, ROUND_HALF_UP

MILLION = Decimal("1000000")
CENT = Decimal("0.01")

def dollars(amount: Decimal) -> Decimal:
    return amount.quantize(CENT, rounding=ROUND_HALF_UP)

@dataclass(frozen=True)
class TokenRates:
    input_per_1m: Decimal
    cached_input_per_1m: Decimal
    output_per_1m: Decimal

@dataclass(frozen=True)
class RateCard:
    rate_card_id: str
    standard: TokenRates
    batch: TokenRates

@dataclass(frozen=True)
class Usage:
    input_tokens: int
    cached_input_tokens: int
    output_tokens: int
    reasoning_tokens: int = 0

    def __post_init__(self) -> None:
        if not 0 <= self.cached_input_tokens <= self.input_tokens:
            raise ValueError("cached input must be a subset of input tokens")
        if min(self.input_tokens, self.output_tokens) < 0:
            raise ValueError("token counts cannot be negative")
        if not 0 <= self.reasoning_tokens <= self.output_tokens:
            raise ValueError("reasoning tokens must be a subset of output tokens")

rate_card = RateCard(
    rate_card_id="openai-gpt-5.4-short-context-2026-05-31",
    standard=TokenRates(
        input_per_1m=Decimal("2.50"),
        cached_input_per_1m=Decimal("0.25"),
        output_per_1m=Decimal("15.00"),
    ),
    batch=TokenRates(
        input_per_1m=Decimal("1.25"),
        cached_input_per_1m=Decimal("0.13"),
        output_per_1m=Decimal("7.50"),
    ),
)

print(rate_card.rate_card_id)
print(f"standard_input_per_1m=${rate_card.standard.input_per_1m}")
print(f"standard_cached_input_per_1m=${rate_card.standard.cached_input_per_1m}")
print(f"standard_output_per_1m=${rate_card.standard.output_per_1m}")
print(f"batch_input_per_1m=${rate_card.batch.input_per_1m}")
print(f"batch_cached_input_per_1m=${rate_card.batch.cached_input_per_1m}")
print(f"batch_output_per_1m=${rate_card.batch.output_per_1m}")

Output

openai-gpt-5.4-short-context-2026-05-31
standard_input_per_1m=$2.50
standard_cached_input_per_1m=$0.25
standard_output_per_1m=$15.00
batch_input_per_1m=$1.25
batch_cached_input_per_1m=$0.13
batch_output_per_1m=$7.50

Price one generated answer

PolicyOps's public-policy-answer path uses a long evaluated instruction prefix and cites policy evidence in every generated response. Take the stable public question "How long are revoked access tokens retained?" A cold request reports 1,800 input tokens and 180 output tokens. Before looking at the formula, predict which side costs more: output has the higher rate, but input has ten times as many tokens.

Split each billable category instead of multiplying one blended token count by one price.

\begin{aligned} \text{cost} = &\frac{\text{fresh input tokens} \times \text{fresh input price}}{1{,}000{,}000} \\ &+ \frac{\text{cached input tokens} \times \text{cached input price}}{1{,}000{,}000} \\ &+ \frac{\text{output tokens} \times \text{output price}}{1{,}000{,}000} \end{aligned}

02-price-one-generated-answer.py

@dataclass(frozen=True)
class PricedUsage:
    fresh_input_usd: Decimal
    cached_input_usd: Decimal
    output_usd: Decimal

    @property
    def total_usd(self) -> Decimal:
        return self.fresh_input_usd + self.cached_input_usd + self.output_usd

def price_usage(usage: Usage, card: RateCard = rate_card, *, batch: bool = False) -> PricedUsage:
    rates = card.batch if batch else card.standard
    fresh_input_tokens = usage.input_tokens - usage.cached_input_tokens
    return PricedUsage(
        fresh_input_usd=Decimal(fresh_input_tokens) * rates.input_per_1m / MILLION,
        cached_input_usd=Decimal(usage.cached_input_tokens) * rates.cached_input_per_1m / MILLION,
        output_usd=Decimal(usage.output_tokens) * rates.output_per_1m / MILLION,
    )

cold_policy_answer = Usage(input_tokens=1_800, cached_input_tokens=0, output_tokens=180)
cold_cost = price_usage(cold_policy_answer)

assert cold_cost.total_usd == Decimal("0.0072")
assert rate_card.standard.output_per_1m > rate_card.standard.input_per_1m

print(f"fresh_input=${cold_cost.fresh_input_usd:.6f}")
print(f"output=${cold_cost.output_usd:.6f}")
print(f"total=${cold_cost.total_usd:.6f}")
print(f"output_share={(cold_cost.output_usd / cold_cost.total_usd):.1%}")

Output

fresh_input=$0.004500
output=$0.002700
total=$0.007200
output_share=37.5%

The output token rate is higher, but this 1,800-input-token request still spends more dollars on input because it has ten times as many input tokens as output tokens. Both volume and per-category rate belong in the ledger.

That split also gives you two different latency levers. Prefix reuse can reduce repeated input processing and first-token delay, but it doesn't remove decode. A shorter compliant answer can reduce output spend and often decode time. Measure time to first token, output tokens, and end-to-end latency separately instead of treating one cost change as a latency result.^{[5]Reference 5Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180}^{[1]Reference 1OpenAI API Pricinghttps://developers.openai.com/api/docs/pricing}^{[6]Reference 6Prompt cachinghttps://developers.openai.com/api/docs/guides/prompt-caching}

Reasoning tokens ride inside that output charge. In OpenAI's published usage shape, output_tokens=1186 includes reasoning_tokens=1024, leaving 162 output tokens outside the reasoning detail in this fixture. Predict the price before running the cell: charge 1,186 output tokens once, not 1,186 plus 1,024. An incomplete call can spend all 1,024 output tokens on hidden reasoning and still bill decode without returning a visible answer.

02b-price-reasoning-without-double-counting.py

reasoning_answer = Usage(
    input_tokens=1_800,
    cached_input_tokens=0,
    output_tokens=1_186,
    reasoning_tokens=1_024,
)
incomplete_reasoning = Usage(
    input_tokens=1_800,
    cached_input_tokens=0,
    output_tokens=1_024,
    reasoning_tokens=1_024,
)
reasoning_cost = price_usage(reasoning_answer)
incomplete_cost = price_usage(incomplete_reasoning)
double_counted_output = Decimal("2210") * rate_card.standard.output_per_1m / MILLION

assert reasoning_cost.output_usd == Decimal("1186") * rate_card.standard.output_per_1m / MILLION
assert reasoning_cost.output_usd != double_counted_output
assert incomplete_cost.output_usd == Decimal("1024") * rate_card.standard.output_per_1m / MILLION
assert incomplete_cost.total_usd > Decimal("0")

print(f"visible_plus_reasoning_output_tokens={reasoning_answer.output_tokens}")
print(f"reasoning_detail={reasoning_answer.reasoning_tokens}")
print(f"priced_output=${reasoning_cost.output_usd:.6f}")
print(f"double_counted_output=${double_counted_output:.6f}")
print(f"incomplete_hidden_output_tokens={incomplete_reasoning.output_tokens}")
print(f"incomplete_total=${incomplete_cost.total_usd:.6f}")

Output

visible_plus_reasoning_output_tokens=1186
reasoning_detail=1024
priced_output=$0.017790
double_counted_output=$0.033150
incomplete_hidden_output_tokens=1024
incomplete_total=$0.019860

The incomplete row still includes the $0.004500 fresh-input charge. Hidden thinking isn't free, and reasoning_tokens isn't a second billable bucket. Keep the detail field for diagnosis: it explains why a response was expensive or stopped before visible output.

Prefix reuse is still a generated answer

Prompt-cache eligibility is observed behavior, not a budget promise. OpenAI's guidance, checked on August 21, 2026, gives GPT-5.6 and later a 1,024-visible-token minimum and charges cache writes at 1.25× the uncached input rate. GPT-5.4 follows earlier-model behavior: 2,048 visible tokens is the documented minimum, some earlier models may occasionally cache shorter prefixes, and there is no additional cache-write fee. Copying the GPT-5.4 ledger onto GPT-5.6 without a write column would under-count.^{[2]Reference 2GPT-5.4 Modelhttps://developers.openai.com/api/docs/models/gpt-5.4}^{[6]Reference 6Prompt cachinghttps://developers.openai.com/api/docs/guides/prompt-caching}

Caching also depends on an exact matching rendered prefix. Count a read only when returned usage reports cached_tokens > 0; put stable instructions and shared evidence first, then dynamic customer data. A prefix hit isn't a stored response, so the support model still answers the current question and still incurs output spend.^{[6]Reference 6Prompt cachinghttps://developers.openai.com/api/docs/guides/prompt-caching}

The fixture below records an observed 1,280-token cache read. Treat that number as trace data, not as a guarantee that a prompt near any threshold will hit. Predict the arithmetic first: 1,800 total input tokens means 520 fresh tokens, while output remains 180 tokens. Any latency improvement here comes from input reuse, not from skipping decode.

03-account-for-prefix-reuse.py

prefix_cached_policy_answer = Usage(
    input_tokens=1_800,
    cached_input_tokens=1_280,
    output_tokens=180,
)
prefix_cost = price_usage(prefix_cached_policy_answer)
savings = cold_cost.total_usd - prefix_cost.total_usd

assert prefix_cost.total_usd == Decimal("0.004320")
assert prefix_cost.output_usd == cold_cost.output_usd
assert savings == Decimal("0.002880")

print(f"cold_generation=${cold_cost.total_usd:.6f}")
print(f"prefix_cached_generation=${prefix_cost.total_usd:.6f}")
print(f"saved_per_generated_answer=${savings:.6f}")
print(f"still_generated={prefix_cost.output_usd > 0}")

Output

cold_generation=$0.007200
prefix_cached_generation=$0.004320
saved_per_generated_answer=$0.002880
still_generated=True

Carry forward the semantic-cache decision

The preceding lesson approved semantic response reuse only for stable public-policy answers within one evaluated release scope. This ledger records that decision; it doesn't rebuild the vector index or threshold.

For "How long are revoked access tokens retained?", a stored-answer hit records zero generated tokens. A prompt-prefix hit records a new answer with cheaper input. Predict those rows before running the next cell: the first has $0 actual generation and a non-zero counterfactual, while the second has $0.004320 actual generation.

04-separate-answer-hits-from-prefix-hits.py

@dataclass(frozen=True)
class RequestOutcome:
    feature: str
    decision: str
    usage: Usage | None
    counterfactual_usage: Usage | None = None

def generated_cost(outcome: RequestOutcome) -> Decimal:
    if outcome.usage is None:
        return Decimal("0")
    return price_usage(outcome.usage).total_usd

semantic_hit = RequestOutcome(
    feature="public-policy-answer",
    decision="SEMANTIC_ANSWER_HIT",
    usage=None,
    counterfactual_usage=prefix_cached_policy_answer,
)
prefix_hit = RequestOutcome(
    feature="public-policy-answer",
    decision="GENERATE_PREFIX_HIT",
    usage=prefix_cached_policy_answer,
)

avoided_generation = price_usage(semantic_hit.counterfactual_usage).total_usd
assert generated_cost(semantic_hit) == Decimal("0")
assert generated_cost(prefix_hit) == Decimal("0.004320")

print(f"semantic_hit_generation=${generated_cost(semantic_hit):.6f}")
print(f"semantic_hit_avoided=${avoided_generation:.6f}")
print(f"prefix_hit_generation=${generated_cost(prefix_hit):.6f}")

Output

semantic_hit_generation=$0.000000
semantic_hit_avoided=$0.004320
prefix_hit_generation=$0.004320

Three dollar stacks for the public-policy question How long are revoked access tokens retained? Cold generation bills $0.007200 as $0.004500 fresh input plus $0.002700 output. Prefix reuse bills $0.004320 as $0.001300 fresh, $0.000320 cached, and the same $0.002700 output. A stored-answer hit realizes $0.000000 and keeps the $0.004320 prefix bill as avoided, not charged. — Same public-policy question, three ledgers. Prefix reuse shrinks input and still bills the $0.002700 decode. A stored answer realizes $0.000000; the $0.004320 bar is avoided spend, not a credit.

Replay a day of traffic

A per-request price hides traffic shape. Replay one PolicyOps day before testing a shorter exception answer:

3,200 stable policy questions safely hit the semantic answer cache.
1,800 policy questions still generate, but reuse the evaluated prompt prefix.
3,000 live-order questions must generate from current tool evidence.
500 access exceptions generate a longer cited answer with the stable prefix.

Before running the replay, predict the ranking. Semantic hits are numerous but have no generated spend. Live-order calls always generate from current evidence, so their output and volume may dominate the billed total. Keep two answers separate: the counterfactual says what safe reuse avoided; actual spend says what belongs on this release ledger.

05-price-a-traffic-replay.py

@dataclass(frozen=True)
class TrafficSlice:
    outcome: RequestOutcome
    requests: int

daily_replay = [
    TrafficSlice(semantic_hit, requests=3_200),
    TrafficSlice(prefix_hit, requests=1_800),
    TrafficSlice(
        RequestOutcome(
            feature="live-order-answer",
            decision="GENERATE_LIVE_DATA",
            usage=Usage(input_tokens=900, cached_input_tokens=0, output_tokens=95),
        ),
        requests=3_000,
    ),
    TrafficSlice(
        RequestOutcome(
            feature="access-exception-answer",
            decision="GENERATE_PREFIX_HIT",
            usage=Usage(input_tokens=2_200, cached_input_tokens=1_280, output_tokens=220),
        ),
        requests=500,
    ),
]

actual_daily_spend = sum(
    generated_cost(slice_.outcome) * slice_.requests
    for slice_ in daily_replay
)
avoided_daily_generation = sum(
    price_usage(slice_.outcome.counterfactual_usage).total_usd * slice_.requests
    for slice_ in daily_replay
    if slice_.outcome.counterfactual_usage is not None
)

assert actual_daily_spend == Decimal("21.761000")
assert avoided_daily_generation == Decimal("13.824000")

print(f"generated_daily_spend=${dollars(actual_daily_spend)}")
print(f"semantic_hit_avoided_daily=${dollars(avoided_daily_generation)}")
print(f"request_count={sum(slice_.requests for slice_ in daily_replay)}")

Output

generated_daily_spend=$21.76
semantic_hit_avoided_daily=$13.82
request_count=8500

The replay reports $21.76 of generated spend and $13.82 of avoided generation. Avoided generation is evidence, not an invoice credit, so keep it out of actual spend. If an answer hit later fails review, its quality failure matters before its apparent savings.

Reduce output only under an answer contract

Output reductions can matter when output tokens carry the highest rate. Blind truncation isn't a cost strategy. PolicyOps's cited support answer must state eligibility, cite policy evidence, and name the requester's next step. Compare three formats generated from the same input:

Candidate	Intended change	Safe for automatic support?
Verbose	Full explanation with repetition	Correct but unnecessarily long
Concise cited	One decision, one citation, one next step	Candidate for release
Truncated	Hard stop before citation and action	Reject

The contract is three booleans, not a token target. Predict the selection before running the cell: the shortest candidate that keeps all three fields should win, while a 55-token answer that drops evidence must be rejected.

06-optimize-output-under-contract.py

@dataclass(frozen=True)
class AnswerCandidate:
    label: str
    output_tokens: int
    states_decision: bool
    cites_policy: bool
    gives_next_step: bool

def satisfies_answer_contract(candidate: AnswerCandidate) -> bool:
    return (
        candidate.states_decision
        and candidate.cites_policy
        and candidate.gives_next_step
    )

candidates = [
    AnswerCandidate("verbose", 220, True, True, True),
    AnswerCandidate("concise_cited", 130, True, True, True),
    AnswerCandidate("truncated", 55, True, False, False),
]
safe_candidates = [candidate for candidate in candidates if satisfies_answer_contract(candidate)]
selected_answer = min(safe_candidates, key=lambda candidate: candidate.output_tokens)

verbose_usage = Usage(input_tokens=2_200, cached_input_tokens=1_280, output_tokens=220)
concise_usage = Usage(input_tokens=2_200, cached_input_tokens=1_280, output_tokens=selected_answer.output_tokens)
saved_per_exception = price_usage(verbose_usage).total_usd - price_usage(concise_usage).total_usd

assert selected_answer.label == "concise_cited"
assert not satisfies_answer_contract(candidates[-1])
assert saved_per_exception == Decimal("0.001350")

print(f"selected={selected_answer.label}")
print(f"rejected={candidates[-1].label}")
print(f"saved_per_exception=${saved_per_exception:.6f}")
print(f"saved_per_replay_day=${dollars(saved_per_exception * 500)}")

Output

selected=concise_cited
rejected=truncated
saved_per_exception=$0.001350
saved_per_replay_day=$0.68

The cheapest candidate is rejected. The concise form saves $0.001350 per exception, or $0.68 across this 500-request slice, while preserving the contract. A compliant shorter answer may also reduce decode time, but verify p95 latency and retry rate instead of inferring either from token count alone.

Batch only work that can wait

Batch is a scheduling choice before it's a pricing choice. OpenAI documents an asynchronous path with 50% lower costs than synchronous APIs and completion within 24 hours.^{[3]Reference 3OpenAI Batch API Guidehttps://developers.openai.com/api/docs/guides/batch} A live customer can't accept that latency contract, but a nightly regression run can.

The pricing page publishes Batch token rates explicitly, so the ledger reads that row instead of deriving every category from one multiplier.^{[1]Reference 1OpenAI API Pricinghttps://developers.openai.com/api/docs/pricing} In this fixture, the displayed $0.13 cached-input rate means the replay total isn't the exact result of halving every Standard-row decimal locally. Predict the eligibility flags before running the cell: live support stays Standard; nightly evaluation can use Batch.

07-separate-interactive-and-batch-work.py

@dataclass(frozen=True)
class Workload:
    name: str
    interactive: bool
    deadline_hours: int
    endpoint_supported: bool
    daily_requests: int
    usage: Usage

def batch_eligible(workload: Workload) -> bool:
    return (
        not workload.interactive
        and workload.deadline_hours >= 24
        and workload.endpoint_supported
    )

live_support = Workload(
    name="live-support-answer",
    interactive=True,
    deadline_hours=0,
    endpoint_supported=True,
    daily_requests=3_000,
    usage=Usage(input_tokens=900, cached_input_tokens=0, output_tokens=95),
)
nightly_eval = Workload(
    name="nightly-release-eval",
    interactive=False,
    deadline_hours=24,
    endpoint_supported=True,
    daily_requests=2_000,
    usage=Usage(input_tokens=1_800, cached_input_tokens=1_280, output_tokens=80),
)

nightly_sync_cost = price_usage(nightly_eval.usage).total_usd * nightly_eval.daily_requests
nightly_batch_cost = price_usage(nightly_eval.usage, batch=True).total_usd * nightly_eval.daily_requests

assert not batch_eligible(live_support)
assert batch_eligible(nightly_eval)
assert nightly_batch_cost == Decimal("2.832800")
assert nightly_batch_cost < nightly_sync_cost

print(f"live_support_batchable={batch_eligible(live_support)}")
print(f"nightly_eval_batchable={batch_eligible(nightly_eval)}")
print(f"nightly_eval_sync=${dollars(nightly_sync_cost)}")
print(f"nightly_eval_batch=${dollars(nightly_batch_cost)}")

Output

live_support_batchable=False
nightly_eval_batchable=True
nightly_eval_sync=$5.64
nightly_eval_batch=$2.83

Attribute cost to decisions, not models alone

An invoice tells you how much you spent. A release trace tells you why. Each row should name the evaluated release, feature, decision, pricing mode, rate-card identity, token usage, and output-contract evidence.

Recording only a model name can't distinguish an unsafe answer-cache promotion from a harmless increase in live-order traffic. Make contract evidence and rate-card identity mandatory instead of silently repricing old usage with whichever table happens to be loaded.

Sort billed dollars by feature before changing prompts. In this fixture, live-order answers own $11.03 of $23.92, or 46.1%. If a feature crosses a team-defined triage threshold, such as 40% for this release, inspect its volume, output tokens, retries, and tool path first. That threshold prioritizes investigation; it doesn't prove the feature is inefficient.

08-build-feature-cost-telemetry.py

@dataclass(frozen=True)
class LedgerTrace:
    release_id: str
    feature: str
    decision: str
    requests: int
    usage: Usage | None
    rate_card_id: str
    contract_passed: bool
    contract_evidence_id: str
    batch: bool = False

def trace_unit_spend(trace: LedgerTrace, card: RateCard = rate_card) -> Decimal:
    if trace.rate_card_id != card.rate_card_id:
        raise ValueError("trace rate card does not match pricing table")
    if trace.usage is None:
        return Decimal("0")
    return price_usage(trace.usage, card, batch=trace.batch).total_usd

def trace_spend(trace: LedgerTrace, card: RateCard = rate_card) -> Decimal:
    return trace_unit_spend(trace, card) * trace.requests

release_id = "support-release-2026-05-cost-v1"
optimized_replay = [
    TrafficSlice(
        RequestOutcome(
            feature=item.outcome.feature,
            decision=item.outcome.decision,
            usage=concise_usage if item.outcome.feature == "access-exception-answer" else item.outcome.usage,
            counterfactual_usage=item.outcome.counterfactual_usage,
        ),
        requests=item.requests,
    )
    for item in daily_replay
]
traces = [
    LedgerTrace(
        release_id=release_id,
        feature=item.outcome.feature,
        decision=item.outcome.decision,
        requests=item.requests,
        usage=item.outcome.usage,
        rate_card_id=rate_card.rate_card_id,
        contract_passed=True,
        contract_evidence_id=(
            "approved-public-policy-cache@v1"
            if item.outcome.decision == "SEMANTIC_ANSWER_HIT"
            else "cited-support-answer-v3@canary"
        ),
    )
    for item in optimized_replay
]
traces.append(
    LedgerTrace(
        release_id=release_id,
        feature=nightly_eval.name,
        decision="BATCH_OFFLINE_EVAL",
        requests=nightly_eval.daily_requests,
        usage=nightly_eval.usage,
        rate_card_id=rate_card.rate_card_id,
        contract_passed=True,
        contract_evidence_id="nightly-release-eval@batch-eligible",
        batch=True,
    )
)

spend_by_feature: defaultdict[str, Decimal] = defaultdict(lambda: Decimal("0"))
for trace in traces:
    spend_by_feature[trace.feature] += trace_spend(trace)

total_with_eval = sum(spend_by_feature.values())
trace_contracts_complete = all(
    trace.contract_passed
    and trace.contract_evidence_id
    and trace.rate_card_id == rate_card.rate_card_id
    for trace in traces
)
stale_trace = replace(traces[0], rate_card_id="openai-gpt-5.4-short-context-2026-04-30")
assert spend_by_feature["public-policy-answer"] == Decimal("7.776000")
assert actual_daily_spend - sum(trace_spend(trace) for trace in traces[:-1]) == Decimal("0.675000")
assert trace_contracts_complete

for feature in sorted(spend_by_feature):
    print(f"{feature}=${dollars(spend_by_feature[feature])}")
print(f"daily_total_with_eval=${dollars(total_with_eval)}")
print(f"trace_contracts_complete={trace_contracts_complete}")
try:
    trace_spend(stale_trace)
except ValueError as error:
    print(f"stale_rate_card_rejected={error}")

Output

access-exception-answer=$2.29
live-order-answer=$11.03
nightly-release-eval=$2.83
public-policy-answer=$7.78
daily_total_with_eval=$23.92
trace_contracts_complete=True
stale_rate_card_rejected=trace rate card does not match pricing table

Ranked daily spend for support-release-2026-05-cost-v1. Live-order answers are $11.03 (46.1 percent), generated public-policy answers $7.78 (32.5 percent), nightly eval $2.83 (11.8 percent), and access-exception answers $2.29 (9.6 percent). The $23.92 total is actual billed spend, including batch eval, not avoided semantic-cache savings. — Rank billed dollars, not avoided cache savings. Live-order generation is 46% of the $23.92 day, so that's the first lane to inspect when the forecast gets tight.

Forecast and gate the release

A release gate answers two separate questions: can we afford this traffic, and can we trust the answers? A cost target alone rewards short, wrong responses. A quality target alone leaves no capacity or margin plan. Require both.

The canary report evaluates the concise cited-answer policy for access exceptions. It projects the replay mix across 30 days and includes the nightly offline evaluation run. Before running it, predict the result: $23.9188 per day becomes a $717.56 monthly forecast, and promotion still requires the quality checks to pass.

09-evaluate-a-release-budget.py

@dataclass(frozen=True)
class QualityReport:
    cited_answer_pass_rate: Decimal
    unsafe_cache_hits: int
    evaluated_answers: int

@dataclass(frozen=True)
class PromotionPolicy:
    monthly_budget_usd: Decimal
    minimum_cited_answer_pass_rate: Decimal
    maximum_unsafe_cache_hits: int
    minimum_evaluated_answers: int

quality = QualityReport(
    cited_answer_pass_rate=Decimal("0.997"),
    unsafe_cache_hits=0,
    evaluated_answers=2_000,
)
promotion_policy = PromotionPolicy(
    monthly_budget_usd=Decimal("750.00"),
    minimum_cited_answer_pass_rate=Decimal("0.995"),
    maximum_unsafe_cache_hits=0,
    minimum_evaluated_answers=2_000,
)
monthly_forecast = total_with_eval * Decimal("30")
budget_headroom = promotion_policy.monthly_budget_usd - monthly_forecast
uniform_volume_growth_headroom = budget_headroom / monthly_forecast

sample_size_passed = (
    quality.evaluated_answers >= promotion_policy.minimum_evaluated_answers
)
quality_passed = (
    trace_contracts_complete
    and sample_size_passed
    and quality.cited_answer_pass_rate >= promotion_policy.minimum_cited_answer_pass_rate
    and quality.unsafe_cache_hits <= promotion_policy.maximum_unsafe_cache_hits
)
budget_passed = monthly_forecast <= promotion_policy.monthly_budget_usd

assert dollars(monthly_forecast) == Decimal("717.56")
assert dollars(budget_headroom) == Decimal("32.44")
assert quality_passed and budget_passed

print(f"monthly_forecast=${dollars(monthly_forecast)}")
print(f"budget_headroom=${dollars(budget_headroom)}")
print(f"uniform_volume_growth_headroom={uniform_volume_growth_headroom:.1%}")
print(f"sample_size_passed={sample_size_passed}")
print(f"quality_passed={quality_passed}")
print(f"budget_passed={budget_passed}")

Output

monthly_forecast=$717.56
budget_headroom=$32.44
uniform_volume_growth_headroom=4.5%
sample_size_passed=True
quality_passed=True
budget_passed=True

This forecast repeats one measured replay day 30 times. The $32.44 remainder represents only 4.5% uniform growth, so replay low, expected, and high volume scenarios before ramping traffic. The hard budget gate passes, but a team could add a soft alert at 90% budget consumption; this fixture is already at 95.7%, so it deserves an owner and a volume plan even with budget_passed=True.

Release gate for support-release-2026-05-cost-v1. The $750 monthly budget is $717.56 forecast spend plus $32.44 headroom, so 4.5 percent uniform growth fills the remainder. Quality still passes: cited-answer rate 99.7 percent over a 99.5 percent floor, zero unsafe cache hits, and 2,000 evaluated answers. Promotion requires both receipts. — The stacked $750 bar makes the remainder visible: $32.44 is only 4.5% uniform growth. Quality also clears, so the cost policy can promote, but volume scenarios still need replay.

Produce the next policy input

The next lesson implements model gateway routing and fallbacks. Hand it a budget contract, not a half-built router inside cost analysis. The contract says what every future route must preserve: citation schema, an approved generated-answer ceiling, and the release evidence behind that limit.

Set the ceiling as a policy value that the observed maximum fits under. Treat it as a lane-admission limit for one successful generation, not a bound on total request spend. A failed primary can bill before timeout, and a contract-preserving fallback can bill again. Incident TCO can exceed maximum_generated_answer_usd when attempts stack, so the gateway lesson must track request-level spend separately.

Before running the final cell, predict the handoff: the observed maximum is $0.004570, the policy ceiling is $0.005000, and the status can promote only because quality and budget gates already passed.

10-emit-the-budget-contract.py

@dataclass(frozen=True)
class GatewayBudgetContract:
    release_id: str
    required_answer_schema: str
    maximum_generated_answer_usd: Decimal
    monthly_forecast_usd: Decimal
    rate_card_id: str
    status: str

observed_max_generated_cost = max(
    trace_unit_spend(trace)
    for trace in traces
    if trace.usage is not None and not trace.batch
)
per_answer_budget_usd = Decimal("0.005000")
assert observed_max_generated_cost <= per_answer_budget_usd

status = "PROMOTE_COST_POLICY" if quality_passed and budget_passed else "HOLD_RELEASE"
gateway_contract = GatewayBudgetContract(
    release_id=release_id,
    required_answer_schema="cited-support-answer-v3",
    maximum_generated_answer_usd=per_answer_budget_usd,
    monthly_forecast_usd=dollars(monthly_forecast),
    rate_card_id=rate_card.rate_card_id,
    status=status,
)

assert gateway_contract.status == "PROMOTE_COST_POLICY"
assert observed_max_generated_cost == Decimal("0.004570")
assert gateway_contract.maximum_generated_answer_usd == Decimal("0.005000")

print(f"status={gateway_contract.status}")
print(f"required_schema={gateway_contract.required_answer_schema}")
print(f"observed_max_generated_answer=${observed_max_generated_cost:.6f}")
print(f"max_generated_answer=${gateway_contract.maximum_generated_answer_usd:.6f}")
print("next_step=apply_contract_in_gateway_routing")

Output

status=PROMOTE_COST_POLICY
required_schema=cited-support-answer-v3
observed_max_generated_answer=$0.004570
max_generated_answer=$0.005000
next_step=apply_contract_in_gateway_routing

What the ledger proves

By now, "cache hit" isn't an explanation by itself. Each claim about this release needs the receipt that supports it:

Evidence	Correct conclusion
Semantic answer hit with valid release scope	New generation was avoided for an already approved answer class
Provider `cached_tokens` on a generated answer	Repeated prefix processing was charged at the cached-input rate
Concise cited-answer candidate passes evaluation	Output reduction is eligible for release
Nightly evaluation classified as noninteractive	Batch discount can be modeled without changing customer latency
Every usage trace pins the loaded rate-card identity	Historical spend remains reproducible instead of being silently repriced
Monthly forecast and quality check pass	Cost policy can be promoted and handed to gateway design

Wait for usage traces before claiming a prompt rewrite improved pricing through cached tokens. Count semantic-cache savings only while accepted-hit quality remains valid. Never approve a shorter output merely because it costs less.

Next Step

Continue to Model Gateways, Routing, and Fallbacks

You now have a pinned budget contract: schema `cited-support-answer-v3`, a `$0.005000` per-answer admission ceiling, and `PROMOTE_COST_POLICY`. Next you'll use that contract to filter lanes and fallbacks without dropping citations, privacy, or review when a cheaper path appears.

PreviousSemantic Caching & Cost Optimization

Share this article