Calling LLM APIs in Production

A large language model (LLM) answered correctly in your local notebook. In the deployed application, the exact same request stalls, times out, or throws an HTTP 429. Did the remote server finish? Did it charge your credit balance? Should your client retry right away, or will repeating the call knock down an already struggling provider? What should the user see while you don't know?

In Prompt Engineering Fundamentals, citing a policy line didn't make an unsupported 90-minute deadline true. We'll carry that discipline into an operational security task: deciding whether a stale service-account key on account acct_10234 meets rotation policy P-7. The trusted key age in the database is 45 days; the policy threshold is 30 days. The model can propose an eligibility candidate, but the server checks it against trusted facts and never creates a rotation job as an untracked side effect.

We'll build that boundary layer by layer, then exercise an actual provider SDK through a mocked HTTP transport. All runnable examples execute locally without live credentials or billable model calls. For a simple threshold rule, ordinary backend code could check the integers directly; the model candidate gives us a compact, checkable environment to master the defensive contracts needed whenever a larger workflow summarizes, extracts, or reasons over unstructured text.

Keep credentials on the server

A hosted model call needs an API key or an IAM role token. That key authorizes billable requests against your organization's quota. Client-side browser bundles, mobile apps, git repositories, and build artifacts are dangerous places for it. Keep provider keys strictly on the server.

Local code can read a credential from an environment variable. A deployed service should inject it from the platform's secret manager (such as AWS Secrets Manager, Google Secret Manager, or HashiCorp Vault) directly into the server process. If you prefix a variable with frontend exposure tags like NEXT_PUBLIC_ or VITE_, build tools bake the secret into the static JavaScript shipped to every browser tab. Anyone opening browser DevTools can steal that key within seconds.

The lab injects a placeholder into a local test dictionary rather than mutating the process environment. Production code passes os.environ, so a missing deployment secret fails fast on startup instead of silently falling back to a dummy value.

The following check rejects an absent or blank configuration value. It can't prove a key has the right model permissions; only the remote provider can verify authorization. Notice that neither output line prints the key itself.

Both verification branches stay visible, while the secret value never appears in stdout or log sinks. A production service calls require_server_key(os.environ) during startup, keeps separate credentials for staging and production, and rotates compromised credentials immediately upon any suspected leak.

A task contract for the model

A server-side secret protects the credential from theft, but it doesn't protect the application from erratic responses. Your internal application routes need a typed task contract, not a disorganized collection of vendor SDK parameters.

When a browser client asks whether account acct_10234 needs key rotation, the frontend identifies the account and transmits the user's intent. After authenticating the caller and checking tenant access, the backend loads the trusted key age and rotation policy from its database. Client-supplied status flags or timestamps must never become authoritative merely because they arrived in a JSON payload.

The boundary separates several core responsibilities:

The model handles raw text and token probabilities; your application enforces contracts and invariants. That separation is the foundation of dependable AI engineering.

Name the application's request

A vendor request format changes often: Anthropic uses messages with specific system parameters, OpenAI uses messages or responses formats, and local inference engines use OpenAI-compatible HTTP schemas. If your route handlers call vendor SDKs directly, vendor-specific parameters bleed into your business logic.

This internal contract encapsulates intent, evidence IDs, deadline tiers, a trace ID, and a prompt version. It doesn't mention any specific vendor SDK, and it doesn't grant permission to execute a rotation job. Python's dataclass annotations document the internal contract clearly.

LLMTask describes application intent rather than a specific vendor's wire schema. It works equally well with OpenAI, Anthropic, a local vLLM instance, or an API gateway.

Notice that LLMTask separates the read-only eligibility inquiry from the write action. Deciding eligibility is a safe read; creating a rotation job is an authorized mutation.

overall_deadline_seconds specifies the total latency budget for the end-to-end task. If the wrapper retries after a transient network drop, each subsequent attempt receives only the time that remains. Three 20-second attempts must not silently consume a full minute of user waiting time. A retry count without a shared deadline budget isn't a resilient architecture.

What belongs in the wrapper

The wrapper encapsulates operational policies that must remain consistent across all application routes:

If individual route handlers call vendor SDKs directly, these policies drift. One handler retries five times on HTTP 500 while another fails immediately; one route logs raw customer emails while another logs nothing. A centralized wrapper makes operational behavior testable and uniform.

A provider adapter translates that domain contract into provider wire format. Here, required_response_fields makes our validation requirements explicit before mapping them to a concrete SDK:

The adapter injects actual policy text and trusted facts, not bare IDs that the model can't resolve. Here, rotation_window_days denotes the minimum key age required for rotation under policy P-7 (30 days), matching the contract we'll verify.

What the wrapper does

Follow one request through the wrapper. In modern provider APIs, tool calling requests that an application run an external tool, while Structured Outputs constrains generation grammar via token-level masking.^{[1]Reference 1Function callinghttps://developers.openai.com/api/docs/guides/function-calling[2]Reference 2Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs} Both mechanisms still produce an untrusted candidate; validating that candidate against source facts remains the server's responsibility.

Every model invocation serves two distinct audiences:

1. The product caller requires a validated typed object or a structured failure it can present to the user.
2. The operations team requires a telemetry record capturing latency, token usage, and status, even when the model fails or returns garbage.

Validate a candidate before returning it

Start with a fixed candidate so validation mechanics stay transparent. We verify the root dictionary, required keys, strict types, cited source IDs, and whether the proposed decision matches trusted database facts. Notice that Python's bool is a subclass of int (isinstance(True, int) evaluates to True), so we verify type(data["rotation_window_days"]) is int to reject booleans pretending to be integers.

The caller receives a validated RotationDecision rather than raw model text. If an unknown account ID is provided, the database lookup fails before dispatching a billable request to the provider.

Timeouts, rate limits, and resilience

Hosted LLMs take seconds to generate completions. If a provider stalls under traffic, one lingering call can monopolize a web worker and freeze the UI. Defending against that requires tiered timeouts and intelligent rate-limit classification.

Tiered timeouts: connect, TTFT, read, and total budget

Production networking requires setting multiple timeout tiers rather than a single coarse timer:

1. Connect timeout (1s to 3s): Time spent resolving DNS and completing the TCP/TLS handshake. If this fails, the provider gateway is unreachable or your network routing has dropped. Fail fast here.
2. Time-to-First-Token (TTFT) timeout (5s to 10s): Time spent waiting for the provider's scheduler to allocate GPU capacity and run the prompt prefill phase. If TTFT exceeds 10 seconds, the provider's queue is backlogged.
3. Read / socket timeout (2s to 4s): The maximum allowable pause between consecutive streamed tokens or response bytes. This catches silent socket drops where the connection hangs without sending an explicit TCP RST.
4. Overall task deadline (e.g. 20s): The total end-to-end latency budget across all attempts. If your overall deadline is 20 seconds and attempt 1 consumed 14 seconds before failing, attempt 2 gets at most 6 seconds.

Enforce deadlines around an asynchronous provider call using asyncio.wait_for:

asyncio.wait_for requests task cancellation and awaits coroutine termination. It can't interrupt blocking synchronous code or confirm that the remote provider ceased token generation. Always configure underlying transport timeouts in your HTTP client alongside high-level asyncio deadlines.^{[3]Reference 3Coroutines and Taskshttps://docs.python.org/3/library/asyncio-task.html}

Rate limit dynamics: RPM, TPM, and 429 response handling

API providers enforce rate limits along two distinct dimensions using token bucket or sliding window limiters:

• Requests Per Minute (RPM): Limits the number of discrete HTTP requests initiated in a 60-second window.
• Tokens Per Minute (TPM): Limits the cumulative count of input and output tokens processed in that same window.

In production, TPM limits are hit much faster than RPM limits. A prompt containing 16,000 tokens of retrieved documents drains 16,000 tokens from your TPM bucket on a single request. If your tier has an 80,000 TPM limit, just five concurrent requests exhaust your quota for the entire minute, even though your RPM utilization is only 5 out of 500 allowed requests.

When a provider rejects a call, inspect the HTTP status and body headers before deciding whether to retry:

1. Temporary rate limits (HTTP 429): The token bucket is momentarily dry. Headers indicate when capacity replenishes. This is safe to retry with backoff.
2. Permanent billing and quota exhaustion (HTTP 429): The provider returns errors such as credit_balance_exhausted, insufficient_quota, organization_spend_limit_exceeded, or monthly_spend_limit_exceeded.^{[4]Reference 4API Error Codeshttps://developers.openai.com/api/docs/guides/error-codes} These indicate an unpaid invoice, empty credit balance, or hard spend cap. Retrying these calls burns server CPU, inflates latency, and guarantees another 429. Fail immediately and alert operators.
3. Response headers to inspect:
   • Retry-After: The minimum duration (in seconds or an HTTP-date timestamp) the client must sleep before re-attempting.
   • x-ratelimit-remaining-requests and x-ratelimit-reset-requests: Remaining request allowance and replenishment window.
   • x-ratelimit-remaining-tokens and x-ratelimit-reset-tokens: Remaining token balance and reset timing.^{[5]Reference 5Rate Limitshttps://developers.openai.com/api/docs/guides/rate-limits}

A provider returns HTTP 429 with error code credit_balance_exhausted. Should the wrapper retry with backoff?

Answer

No. Credit exhaustion is a permanent account failure, not a transient rate limit. Retrying will burn latency and fail every time until an operator adds funds.

The classifier below categorizes HTTP responses into distinct operational actions:

Exponential backoff with full and decorrelated jitter

When an API blips, immediate retries transform a momentary hiccup into a thundering herd that knocks the provider down completely. If 500 client workers encounter an error at the exact same moment and use standard exponential backoff (e.g. sleep 1s, then 2s, then 4s), all 500 workers wake up simultaneously at $t = 1.0\text{s}$ and slam the provider again in a synchronized shockwave.

Jitter math: full jitter vs decorrelated jitter

AWS distributed systems research identified two primary randomization strategies to break synchronized retry waves:^{[6]Reference 6Exponential Backoff And Jitterhttps://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/[7]Reference 7Timeouts, retries, and backoff with jitterhttps://builder.aws.com/content/3EumjoZascWd1oZiEgL8ORlv3qE/timeouts-retries-and-backoff-with-jitter}

1. Full Jitter: Sample uniformly between zero and the current exponential delay ceiling: $\text{delay} = \text{random}(0, \min(\text{cap}, \text{base} \times 2^{\text{attempt}-1}))$ Full jitter guarantees that no two workers wake at the exact same moment, spreading load across the entire interval.
2. Decorrelated Jitter: Instead of calculating powers of two from the attempt index, sample uniformly between the base delay and three times the prior sleep interval: $\text{sleep}_{i+1} = \min(\text{cap}, \text{random}(\text{base}, \text{sleep}_i \times 3))$ Decorrelated jitter produces high entropy between successive retries, preventing clusters from reforming even over extended retry sequences.

Compare both algorithms side by side:

Respecting Retry-After and deadline caps

If the provider includes a Retry-After header, that value represents an absolute floor. You must never shorten the requested sleep duration to squeeze in an extra attempt. Instead, calculate your jitter on top of that floor, and verify that the combined sleep fits within your remaining task deadline budget:

When only 1.2 seconds remain in the task budget, sleeping 2.0 seconds is impossible. The wrapper halts immediately and emits a deadline timeout rather than embarking on an attempt that can't return in time.

Circuit breakers, fallback providers, and multi-region routing

When a provider region enters hard failure (such as an extended 5xx outage or sustained 100% rate-limiting), retrying every incoming request worsens downstream latency and exhausts your server's thread pool. A circuit breaker prevents an unhealthy upstream dependency from destabilizing your entire architecture.

The three-state circuit breaker

A circuit breaker transitions across three well-defined states:

1. Closed (Healthy): Normal operations. All requests route to the primary provider. Consecutive successes reset the failure counter to zero.
2. Open (Tripped): When the failure count or error rate breaches a configured threshold (such as 5 consecutive 5xx errors), the breaker trips to Open. Incoming calls don't touch the primary provider; they fail fast or route immediately to a secondary fallback provider. This protects the primary provider from load so it can recover.
3. Half-Open (Canary Probing): After a cooldown window (e.g. 30 seconds), the circuit permits a small canary fraction (such as 10% of traffic) to test the primary provider. If the canary requests succeed, the breaker resets to Closed. If any canary call fails, the breaker snaps back to Open for another cooldown period.

Fallback providers and multi-region routing

When the breaker opens, a model router can divert requests along two dimensions:

• Multi-region routing: Divert from openai/us-east to openai/us-west or azure-openai/eastus2. Regional outages frequently spare secondary availability zones.
• Cross-provider fallback: Divert from an unavailable primary model (such as Claude 3.5 Sonnet) to a secondary model (such as GPT-4o).

When configuring cross-provider fallbacks, enforce strict contract normalization: both providers must produce outputs that parse cleanly into the same RotationDecision schema. Never return raw fallback text that bypasses semantic verification.

Implement a circuit breaker with automatic routing to a fallback provider:

Idempotency and side-effect isolation

A timeout describes local uncertainty: your client gave up waiting for a response, but the remote provider or downstream worker might still have processed the request.

Checking rotation eligibility is a read operation: retrying it carries no risk of corrupting database records. In contrast, creating a rotation job or triggering an AWS IAM key deactivation is an action with real side effects. Retrying an unconfirmed action without an idempotency key can spawn duplicate rotation tickets, confuse operations teams, or trigger multiple credential revokes in production.

Always enforce durable idempotency keys on mutating operations. The following in-memory demonstration records arguments alongside each idempotency key:

This dictionary demonstrates the sequential concept. In production, use durable database storage with an atomic unique constraint on (tenant_id, idempotency_key) to prevent concurrent workers from both executing the mutation. Scope the key to the specific user intent; using an account ID alone would block all future legitimate key rotations for that account.

From text to typed objects

Free-form natural language is fine for human reading, but programmatic software requires typed structures.

For key rotation, downstream systems need exact fields: decision, rotation_window_days, and source_line_ids. In OpenAI's API, Structured Outputs with a strict JSON schema uses constrained decoding: during generation, an underlying finite state machine (FSM) masks tokens that would violate the schema grammar. JSON mode (json_object) guarantees only syntactically valid JSON, allowing missing keys, type mismatches, and hallucinated enums.^{[2]Reference 2Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs}

Neither mode proves that the output is semantically truthful or compliant with policy. Check schema adherence first, then verify business invariants before creating a RotationDecision:

Schema checks vs semantic checks

Run the parser from minimal-wrapper.py against five distinct candidates in the same session. All five contain valid JSON. Two have the wrong schema shape; two have plausible fields but contradict trusted database facts or policy definitions. Only one passes:

The second candidate has the right shape and cites P-7, but claims not_eligible when trusted records show a 45-day-old stale key (which meets the 30-day threshold). The third candidate claims a 90-day threshold that P-7 never specified. Syntax validation checks shape; semantic validation checks truth.

Exercise a real SDK without spending tokens

OpenAI's Responses API accepts a strict schema in text.format; its SDK exposes generated text through response.output_text.^{[8]Reference 8OpenAI Python API libraryhttps://github.com/openai/openai-python} The adapter below maps our contract to those fields, verifying completion status and checking for refusals before calling parse_decision.

We use httpx.MockTransport to intercept every outgoing HTTP request locally. The API key and model ID are placeholders. Four test fixtures exercise success, an unsupported policy threshold, a model refusal, and an incomplete response truncated by token limits:^{[9]Reference 9Transportshttps://www.python-httpx.org/advanced/transports/}

Only the first candidate reaches application code as a validated RotationDecision. The other three fail cleanly rather than returning fabricated defaults.

In production, remove the mock transport, load credentials from secret storage, and configure long-lived HTTP client connection pools. Notice store=False: this disables provider retention for prompt evaluation, helping satisfy data privacy requirements.^{[10]Reference 10Data controls in the OpenAI platformhttps://developers.openai.com/api/docs/guides/your-data}

Server-Sent Events streaming vs buffered responses

When invoking an LLM, you can buffer the entire response on the server or stream tokens incrementally over HTTP using Server-Sent Events (SSE).^{[11]Reference 11Using server-sent eventshttps://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events[12]Reference 12HTML Standard: Server-sent eventshttps://html.spec.whatwg.org/multipage/server-sent-events.html}

In an SSE connection, the client issues an HTTP POST request with Accept: text/event-stream. The provider keeps the TCP connection open and emits event chunks formatted with data: prefixes, ending with a sentinel data: [DONE]\n\n:

TTFT vs inter-token latency

Streaming splits response time into two distinct metrics:

1. Time-to-First-Token (TTFT): The duration from request dispatch to the arrival of the very first token chunk. TTFT reflects network transit, provider scheduling queues, and prompt prefill time.
2. Inter-Token Latency (ITL): The interval between consecutive tokens during the autoregressive decode phase, inversely related to tokens per second (TPS).

In human-facing chat interfaces, streaming creates the illusion of speed: a user starts reading within 300 ms (TTFT), even if the full answer takes 6 seconds to generate. For automated programmatic pipelines, streaming adds complexity without value: you can't parse incomplete JSON without speculative partial parsers, and partial tokens must never cross an action boundary.

The streaming client state machine

A robust streaming client transitions through distinct states:

1. Connect: Initiate the HTTP connection with Accept: text/event-stream.
2. First token milestone: Measure TTFT. If no byte arrives before the TTFT budget, abort the socket.
3. Buffer deltas: Append incremental text deltas to an in-memory buffer while monitoring read timeouts between chunks.
4. Handle socket drop: If the TCP connection terminates prematurely, mark the buffer unverified and fail closed.
5. Terminal validation: When data: [DONE] arrives, run schema and semantic checks on the complete text before committing the state.

Test delta buffering and failure handling during a mid-stream connection drop:

When the connection drops mid-stream, the client flags the accumulated buffer as failed and display-only. Partial sentences must never authorize a backend action.

Production telemetry and cost attribution

After every request, operators need to know what model version executed, how many tokens were billed, how long generation took, and why the call stopped.

Under the OpenTelemetry GenAI Semantic Conventions, model calls are instrumented using standardized attribute namespaces:^{[13]Reference 13OpenTelemetry GenAI Semantic Conventionshttps://github.com/open-telemetry/semantic-conventions-genai}

• gen_ai.system: Provider identifier (e.g. openai, anthropic).
• gen_ai.request.model: The specific model ID requested (e.g. gpt-4o, claude-3-5-sonnet-20241022).
• gen_ai.usage.input_tokens: Count of prompt tokens billed.
• gen_ai.usage.output_tokens: Count of completion tokens billed.
• gen_ai.response.finish_reasons: Terminal reason (e.g. stop, length, content_filter).

Tracing spans should follow a strict hierarchy:

Token accounting and cost attribution

Hosted model providers bill separately for prompt tokens and output tokens. If the model supports prompt caching, cached prompt tokens receive substantial discounts (typically 50% to 75% cheaper than fresh input tokens).

At illustrative baseline rates of $2.00 per million input tokens and $6.00 per million output tokens:

While less than a tenth of a cent per call sounds negligible, a cluster processing 200,000 automated policy evaluations daily spends nearly $6,000 monthly. Telemetry must track token consumption by tenant and task to support cost attribution.

Redacting customer text

Prompts can contain customer names, proprietary code, or personal data. OWASP's Top 10 for LLM Applications highlights sensitive information disclosure and prompt injection as top vulnerabilities.^{[14]Reference 14OWASP Top 10 for Large Language Model Applicationshttps://genai.owasp.org/llm-top-10/} Copying raw prompts into log collectors turns your monitoring platform into a high-risk data store.^{[15]Reference 15Handling sensitive datahttps://opentelemetry.io/docs/security/handling-sensitive-data/}

Use keyed HMAC fingerprints for diagnostic correlation. The fingerprint lets you identify whether identical prompts caused identical failures without storing customer text:

Choosing the execution shape

The server boundary supports different execution patterns depending on user expectations and latency constraints:

Interactive draft generation benefits from streaming responsiveness, eligibility verification requires atomic schema validation, and nightly compliance scans belong in an offline batch queue. Choose your execution shape to match the consumer's constraints.

Carry the boundary into an app

Production systems require an authenticated backend environment. Before exposing /rotation-decision to the web, authenticate the user session, verify tenant permissions, and apply rate limiting at your API gateway. Hiding a provider key on the server doesn't protect you if an unprotected public endpoint lets anyone burn your budget through your proxy.

We've validated the core boundary components: secret isolation, typed contracts, tiered deadlines, rate-limit classification, full and decorrelated jitter backoff, circuit breakers, and redacted telemetry. The next lesson integrates these components into an end-to-end fullstack application.