Calling LLM APIs in Production

You can test the stale-key prompt from the previous lesson and get the right answer once. That doesn't make it a production feature.

A production large language model (LLM) call has to survive slow networks, invalid outputs, user cancellation, private data, and retries. Suppose Luna's support screen asks whether stale service-account key acct_10234 is eligible under policy line P-7. The answer must arrive through a boundary that protects credentials, validates evidence, records failure, and keeps rotation-job creation separate. Build that boundary.

Application boundary: Put the model call behind one small application boundary. The rest of the app calls your boundary, not a provider SDK (Software Development Kit) directly.

Keep credentials on the server

A hosted model request usually uses an API key or another server-side credential. If you paste it into source code or send it to a browser, anyone who obtains it may be able to make requests against your account. Keep provider keys out of browser code, mobile clients, repositories, and logs.

Locally, read a credential from an environment variable. In a deployed service, use the platform's secret manager and inject the value only into the server process. Don't put a provider key in a public frontend variable.

The lab uses a placeholder rather than a real key and proves only that server configuration is present.

This pattern avoids printing even part of the secret while still proving that server configuration is present. A real app should also separate development and production credentials and rotate a key after any suspected leak.

A task contract for the model

Treat an LLM API call like a typed task crossing a service boundary.

You don't paste a private support thread into an arbitrary chat box and hope the answer is safe. You send a typed task contract with an account ID, requested decision, policy constraints, deadline, and maybe a "run once" marker. When the result comes back, code checks that it matches the request before committing it.

An LLM request needs the same idea:

The model sees text and tools. Your application sees contracts. That separation is the first professional habit.

Building a request contract

Start by naming what your app owns.

A provider request may use messages, tools, response formats, model IDs, and sampling settings. Those fields differ across providers and change over time. Keep those details out of route handlers.

Instead, define a small internal request.

This object doesn't know whether you use OpenAI, Anthropic, a local model, or a gateway.

It describes application intent. Notice what it doesn't contain: permission to create a rotation job or grant approval. Deciding eligibility and executing an action are separate boundaries.

What belongs in the wrapper

The wrapper owns the parts that must be consistent across the product:

If route handlers call the provider directly, these rules drift: one route retries five times while another never retries; one logs raw prompts while another logs nothing; one validates JSON while another trusts a string. That inconsistency becomes technical debt fast.

The provider adapter can turn the application contract into its own request payload. The teaching payload below is deliberately provider-shaped rather than SDK-specific: a real adapter maps its messages and required fields into the schema format its provider supports. Keep that translation testable too:

What the wrapper does

The flow below is provider-agnostic. In OpenAI's API, function calling and Structured Outputs are provider features that can express tools or output schemas; application validation and action authorization still belong outside the model response.^[1][2]

Notice the two outputs:

1. The user-facing result.
2. The operational record.

Production work needs both.

Minimal wrapper

The next snippet uses a fake provider function so you can run the shape without credentials. It carries the exact policy decision from the previous lesson through parse, business validation, logging, and return.

This example is small, but it already has three production habits:

• The parser owns the response contract.
• The wrapper logs a trace ID and status.
• The caller receives a typed eligibility decision, not raw model text.

Timeouts, retries, and backoff

Each network call needs a timeout.

Without a timeout, one slow provider request can hold a web worker, keep a browser spinner open, and make the app look broken. A beginner often sets an overlong timeout to avoid errors. That hides the problem instead of solving it.

Use a timeout that matches the product moment:

This runnable timeout keeps the user-facing deadline outside the fake provider. A slow response becomes a named failure state rather than an endless spinner:

Retries need more care.

Retrying a temporary network error can be appropriate for a read-only decision call. Retrying an action after a timeout is different: the server may have created a rotation job even though the client never received the response. If the workflow can call a tool or update a database, enforce idempotency at that action boundary.

Not every error should be retried. A status code alone may not classify the failure: a provider can use the same code for a temporary rate limit and exhausted quota. Inspect provider error details too. Authentication errors, quota exhaustion, and malformed requests need a fix, not more traffic:

Exponential backoff with jitter

A good retry policy adds delay between attempts. If retries fire immediately, a brief provider outage becomes a thundering herd from your own servers. Exponential backoff waits a little longer each time: a base near 1 second, then 2, then 4. Backoff alone isn't enough. If a thousand workers all back off by exactly 1, 2, and 4 seconds, they still retry in synchronized waves. Adding random jitter spreads those retries across the window so the herd disperses. The common form is full jitter: pick a random delay between zero and the current exponential ceiling.^[3]

Before you run the next snippet, predict the ceiling for attempt 1 and attempt 2.

This fake provider fails on the first two calls, then succeeds. The wrapper computes a longer ceiling each time it retries, then samples a jittered delay below it. In this testable version we inject a no-op sleep function and a fixed random source so the output stays deterministic while you run it.

Also log the final status. A retry that succeeds is still operationally important. A provider SDK may implement retry behavior, but your application still owns timeout, retry cap, and idempotency decision.

A timeout doesn't prove nothing happened

The eligibility decision is read-only. Creating a rotation job isn't. The action below stores an idempotency key before returning a job ID. If the network drops after the first call, retrying the same key returns the existing rotation job rather than creating another one.

The dictionary makes sequential retries visible, but production code needs a durable, atomic claim. Store the idempotency key in a database row protected by a unique constraint, or use a provider-supported idempotency mechanism. Otherwise concurrent retries can both pass the first check and create duplicates.

From text to typed objects

Free-form text is useful for humans. Applications often need objects.

If the app asks for a rotation decision, downstream code wants fields such as decision, rotation_window_days, and source_line_ids. In OpenAI's API, Structured Outputs with a supported strict JSON schema enforces supported schema adherence, while JSON mode targets valid JSON and still has documented edge cases that callers must handle.^[2] Neither mode proves an answer follows workspace policy. Your application still handles refusals or incomplete output and checks facts and permissions.

A common shortcut that breaks

The tempting shortcut is stopping once json.loads succeeds. Run both candidates below: both are valid JSON, but one contradicts policy P-7.

The second object parses correctly and still fails. Shape validation and business validation solve different problems.

When to stream

Streaming sends partial output as it's generated.

Use it when the user is waiting and partial text is useful: drafting a customer-facing explanation, a long answer, code generation, or voice. Don't make streaming your default execution mode. Eligibility extraction often works better as a normal request because the app needs one final validated object.

Streaming adds state:

• partial content buffer
• cancellation path
• final validation step
• error display after partial output
• logs that mark started, streaming, completed, or failed

If streaming output later fails validation, the UI needs a plan. For a draft reply, you may show partial text and mark it unapproved. For a rotation-job action, hold the action until final validation passes.

A tiny streaming handler can expose a draft while keeping action authorization outside the text stream:

Provider streaming endpoints commonly send incremental events over a persistent HTTP response. Your job is to buffer them, handle cancellation, and validate the final assembled result before you act on it.

What to log and what to hide

You can't manage what you don't measure.

Each call records enough information to debug, compare, and budget:

Estimate usage without logging customer text

Hosted APIs may charge for input and output tokens. Rates depend on provider and model, so keep the calculation separate from current pricing. With illustrative rates of $2 per million input tokens and $6 per million output tokens, one rotation-decision request costs:

That looks tiny, but 10,000 decisions per day becomes about $9.80 per day, or $3,577 per year at those illustrative rates. Log usage fields per task and join them to the rate card selected by your application.

The log record should help operators without copying customer text into telemetry. If you need to correlate repeated inputs, use a keyed fingerprint rather than a plain hash. The lab key below is another placeholder; production code injects it from secret storage.

Don't log raw prompts by default. A keyed fingerprint isn't anonymization either. Keep it only when correlation value justifies retention, protect its key, and apply access and retention controls.

Prompts can contain names, emails, API keys, private documents, or customer messages. OWASP's 2025 Top 10 for LLM Applications lists prompt injection and sensitive information disclosure among the central application risks; logs that retain prompt data extend the places where that data must be protected.^[4] NIST's AI Risk Management Framework organizes risk work around governance, mapping, measuring, and managing risk, which supports traceable controls without defaulting to raw-input retention.^[5]

---

Log IDs, usage, failure classes, and carefully redacted excerpts only when you have a clear retention policy. Add a keyed fingerprint only when you need repeat-input correlation.

Choosing the execution shape

Once the wrapper works, choose the execution shape that matches the job.

The beginner mistake is trying to use one shape for all work.

A support-agent draft, nightly audit-log enrichment job, and bulk rotation-policy evaluation don't need the same execution pattern.

Spot the problems

Read this bad design and write down at least three problems before you look at the answer:

Take a minute to think. What's risky? What's missing?

Strong answers usually include these issues:

If you named three of those, you've located real production failure paths.

Carry the boundary into an app

A model call behaves like a production dependency only when the wrapper owns the operational contract: keep credentials on the server, bound waiting time, retry transient failures with jitter, make actions idempotent, validate policy-grounded outputs, and record useful telemetry without storing raw customer text.

The next lesson will put this wrapper behind a small web workflow. Its form and database record are trustworthy only if this boundary already returns explicit success and failure states.

Mastery check

Key concepts

• Secure API key handling and environment variables
• Provider-agnostic LLM request contract
• Timeouts, retry classification, jitter, and idempotent actions
• Structured outputs and schema validation
• Streaming response handling
• Cost, latency, and error logging

Evaluation rubric

• Foundational: Stores API keys in environment variables and keeps them out of source code
• Foundational: Separates application intent from provider-specific API fields
• Foundational: Turns a slow provider response into a named timeout state
• Foundational: Retries only transient failures with bounded jittered backoff
• Intermediate: Keeps side-effecting actions idempotent when a response may have been lost
• Intermediate: Validates structured responses before trusting model output
• Intermediate: Logs latency, token usage, model version, and failure reasons for each call
• Intermediate: Explains when to use streaming, batch work, prompt caching, or background jobs

Follow-up questions

Common pitfalls