Multi-Tenant LLM Platform

A shared large language model (LLM) platform serves one hundred workspaces. Tenant A submits a long design review; Tenant B needs a short incident summary. Both wait, although the accelerator dashboard looks only moderately busy. Then B receives a passage from A's private postmortem.

These symptoms require different responses. Queueing might come from compute, memory, admission policy, or scheduling. Foreign evidence is an authorization failure. Adding capacity might address the first problem; it won't repair the second.

The code-completion system gave one product a serving path. This capstone turns that path into shared infrastructure. We'll keep two requests in view: A's long review and B's short summary. They may share frozen weights and execution steps, but each needs its own authorized evidence, request state, resource allowance, and usage record.

Define what sharing may change

A synthetic dense 72-billion-parameter model at two bytes per weight needs about 144 GB of weight storage. One hundred independent copies would need 14.4 TB before runtime state. Sharing a copy across compatible requests can reduce duplication, although a fleet still needs replicas for capacity and availability. A model replica can itself span several graphics processing units (GPUs).

That saving isn't a tenant-isolation mechanism. Write down the invariants before picking a serving engine:

The control plane owns policy, deployment, artifact authorization, and worker membership. Data-plane services carry requests and run inference. Both the gateway and worker enforce relevant policy; a protected gateway is insufficient if clients can call a worker's administrative endpoints directly.

Define service-level objectives (SLOs) by tier and workload. In this platform, time to first token (TTFT) starts at external request arrival and ends at the first usable output, so it includes queueing, prompt processing, and adapter loading. An engine metric may start later; name the timing boundary before comparing it with the gateway's result. Inter-token latency (ITL) also needs a sampling unit: current vLLM records gaps between streamed output events, which can bundle tokens. Its request-level time per output token (TPOT) instead uses (completion latency − TTFT) / (output tokens − 1) for more than one output token.^{[1]Reference 1Metricshttps://docs.vllm.ai/en/stable/design/metrics/} Also report completion latency, error rate, and throughput under the same offered load. A request with a great first-token time and a failed stream isn't a successful completion.

For example, a tier might target p95 TTFT below 800 ms and p95 streamed-output gaps below 120 ms. These are invented design inputs, not performance results. A percentile across all output gaps can overweight requests with more events. For a goodput metric, separately specify a per-request success/deadline rule and count completions satisfying it. A fleet percentile isn't a per-request gate.

Make the distinction concrete. One synthetic request emits a first token at 100 ms, bundles four more tokens into an event at 220 ms, and completes at 240 ms. Its observed output-event gap is 120 ms, while request TPOT is (240 − 100) / (5 − 1) = 35 ms. Neither number reconstructs the individual decoding gaps behind that bundle. Specify whether a tenant's streaming objective constrains observed output gaps, a request mean, or another defined rule before declaring success.

Establish identity before touching tenant state

An incoming Tenant-ID: A header is a request to select a workspace, not proof of membership. Authenticate the principal, check its current workspace grants, and construct the internal context from that decision. Validate authorization on every protected operation, with denial as the default.^{[2]Reference 2Authorization Cheat Sheethttps://cheatsheetseries.owasp.org/cheatsheets/Authorization_Cheat_Sheet.html}

The internal envelope should carry the resolved tenant and principal, request and attempt IDs, policy revision, authorized model/adapter revisions, deadline, and output cap. Protect it with authenticated service communication and narrowly scoped credentials. A signature protects integrity, but the receiver must still validate issuer, audience, expiry, and permission for the operation. Don't let a downstream service overwrite tenant identity from an untrusted body field.

This fixture starts after credential verification. Its principal names and grants are hand-authored, not a real identity provider. A caller can select an authorized workspace, but claiming a different one must fail before any lookup.

from dataclasses import dataclass

@dataclass(frozen=True)
class Context:
    principal: str
    tenant: str

GRANTS = {"alice": {"A"}, "bob": {"B"}}

def resolve_context(authenticated_principal, requested_tenant):
    if requested_tenant not in GRANTS.get(authenticated_principal, set()):
        raise PermissionError("workspace access denied")
    return Context(authenticated_principal, requested_tenant)

ctx = resolve_context("alice", "A")
assert ctx.tenant == "A"
for principal, tenant in (("alice", "B"), ("unknown", "A")):
    try:
        resolve_context(principal, tenant)
    except PermissionError:
        print(f"denied: {principal} acting for {tenant}")
    else:
        raise AssertionError("unauthorized context created")

denied: alice acting for B
denied: unknown acting for A

Tenant identity is only the first permission dimension. A can contain documents visible to Alice but not another A employee. Retrieval, tool calls, cached answers, and exported logs must honor those finer grants too. Never ask the language model to decide whether a document belongs to the caller.

Reserve work before admitting it

Different limits answer different questions:

Tokenize the fully assembled request with the deployed tokenizer, including templates and authorized retrieved content. Enforce byte and retrieval-size limits before that work too. Reserve input tokens plus the maximum output allowance under the chosen quota policy. If tool iterations or server-added context can expand the request, reserve each extra step or stop before exceeding the remaining allowance.

A period quota has a useful conservation rule. Start with 1,000 units; reserve 300 input plus 400 maximum output. Available allowance becomes 300. If trusted completion reports 120 output tokens, actual usage is 420, so release the unused 280 and leave 580 available. A duplicate completion must leave 580 unchanged.

$\text{limit}=\text{available}+\text{held reservations}+\text{settled usage}.$

This example charges all input tokens, including cached input, against its synthetic quota. A different product policy can choose different units, but it must declare them. This isn't yet a TPM implementation: a rolling-window limiter also needs authoritative time, window attribution, and a defined rule for long-running requests and late reconciliation. Refunding an old reservation into an unrelated new window can manufacture allowance.

Make reservation and settlement idempotent

An idempotency key must be scoped to the tenant and bound to a fingerprint of the resolved execution request. Repeating the same key and request returns the same reservation. Reusing it for a different prompt, model revision, or output cap is a conflict, not free work. This prevents duplicate admission, but a durable dispatcher must also prevent two workers from executing the same reservation.

The next cell implements a single-process, fixed-period ledger with integer token and page units. Its lock makes local check-and-update atomic. It doesn't provide durable storage, distributed locking, authentication, execution ownership, or a rolling time window. Only trusted gateway/worker code may call it.

from dataclasses import dataclass
from threading import Lock

@dataclass(frozen=True)
class Spec:
    fingerprint: str  # trusted canonical request fingerprint, not a client assertion
    input_tokens: int
    max_output: int
    kv_pages: int

@dataclass
class Entry:
    spec: Spec
    output: int | None = None  # None means held, including uncertain execution

class Ledger:
    def __init__(self, tenants, token_limit=1000, concurrency=2, page_limit=20):
        for value in (token_limit, concurrency, page_limit):
            if type(value) is not int or value <= 0:
                raise ValueError("limits must be positive integers")
        self.limits = (token_limit, concurrency, page_limit)
        self.used = {tenant: 0 for tenant in tenants}
        self.entries = {}
        self.lock = Lock()

    def _held(self, tenant):
        return [entry for (owner, _), entry in self.entries.items()
                if owner == tenant and entry.output is None]

    def available(self, tenant):
        with self.lock:
            return self.limits[0] - self.used[tenant] - sum(
                e.spec.input_tokens + e.spec.max_output for e in self._held(tenant))

    def reserve(self, tenant, request_id, spec):
        if tenant not in self.used:
            raise PermissionError("unknown tenant")
        if not request_id or not spec.fingerprint:
            raise ValueError("identity and fingerprint are required")
        if any(type(x) is not int or x < 0
               for x in (spec.input_tokens, spec.max_output, spec.kv_pages)):
            raise ValueError("reservation units must be nonnegative integers")
        if spec.kv_pages == 0:
            raise ValueError("this fixture requires a positive page reservation")
        key = (tenant, request_id)
        with self.lock:
            if key in self.entries:
                if self.entries[key].spec != spec:
                    raise ValueError("idempotency conflict")
                return key
            held = self._held(tenant)
            tokens = sum(e.spec.input_tokens + e.spec.max_output for e in held)
            pages = sum(e.spec.kv_pages for e in held)
            limit, concurrency, page_limit = self.limits
            if (self.used[tenant] + tokens + spec.input_tokens + spec.max_output > limit
                    or len(held) >= concurrency
                    or pages + spec.kv_pages > page_limit):
                raise ValueError("admission limit exceeded")
            self.entries[key] = Entry(spec)
            return key

    def settle(self, tenant, request_id, output_tokens):
        # Called only after authoritative usage AND execution retirement are known.
        with self.lock:
            entry = self.entries[(tenant, request_id)]
            if type(output_tokens) is not int or not 0 <= output_tokens <= entry.spec.max_output:
                raise ValueError("invalid usage")
            if entry.output is not None:
                if entry.output != output_tokens:
                    raise ValueError("conflicting settlement")
                return
            self.used[tenant] += entry.spec.input_tokens + output_tokens
            entry.output = output_tokens

ledger = Ledger(["A", "B"])
spec = Spec("resolved-request-v1", 300, 400, 5)
assert ledger.reserve("A", "r1", spec) == ("A", "r1")
assert ledger.reserve("A", "r1", spec) == ("A", "r1")
print("after reservation:", ledger.available("A"))
ledger.settle("A", "r1", 120)
ledger.settle("A", "r1", 120)  # duplicate delivery must be harmless
assert ledger.available("A") == 580
assert ledger.available("B") == 1000
print("after repeated settlement:", ledger.available("A"))

after reservation: 300
after repeated settlement: 580

Test the failure paths, not just the arithmetic. This continuation races two reservations that would jointly exceed the allowance. Exactly one can succeed. The test exercises a local lock, not cross-host consensus.

from concurrent.futures import ThreadPoolExecutor
from threading import Barrier

def must_reject(call):
    try:
        call()
    except (ValueError, PermissionError):
        return
    raise AssertionError("invalid operation succeeded")

before = ledger.available("A")
must_reject(lambda: ledger.reserve("A", "r1", Spec("different-request", 300, 400, 5)))
must_reject(lambda: ledger.settle("A", "r1", 121))
must_reject(lambda: ledger.reserve("A", "bad", Spec("bad", -1, 10, 1)))
must_reject(lambda: ledger.reserve("unknown", "r1", spec))
assert ledger.available("A") == before

race = Ledger(["A"])
barrier = Barrier(2)

def contender(request_id):
    barrier.wait()
    try:
        race.reserve("A", request_id, spec)
        return True
    except ValueError:
        return False

with ThreadPoolExecutor(max_workers=2) as pool:
    accepted = list(pool.map(contender, ["one", "two"]))
assert sum(accepted) == 1
assert race.available("A") == 300
uncertain = Ledger(["A"])
uncertain.reserve("A", "lost-response", spec)
# A missing client receipt supplies no authoritative retirement or usage event.
must_reject(lambda: uncertain.reserve("A", "new-attempt", spec))
assert uncertain.reserve("A", "lost-response", spec) == ("A", "lost-response")
assert uncertain.available("A") == 300
uncertain.settle("A", "lost-response", 120)  # trusted retirement finally arrives
assert uncertain.available("A") == 580
print("concurrent reservations accepted:", sum(accepted))
print("conflicts and invalid input leave the ledger unchanged")
print("missing receipt: held 700; authoritative retirement: available", uncertain.available("A"))

concurrent reservations accepted: 1
conflicts and invalid input leave the ledger unchanged
missing receipt: held 700; authoritative retirement: available 580

The missing-receipt trace deliberately keeps 700 units held. A new identity cannot spend that allowance again, while replaying the original reservation changes nothing. The final settle call represents trusted evidence of both usage and execution retirement; calling it with zero merely because the client disconnected would invent evidence the ledger cannot check itself. This fixture still has no durable worker-ownership protocol.

Across gateways, use a durable transaction, conditional update, or equivalent atomic service for the allowance and idempotency record. “Read balance, then write balance” on separate connections isn't sufficient. Serializable database transactions can reject conflicting work and require whole-transaction retries; design retries to preserve the same idempotency identity.^{[3]Reference 3Transaction Isolationhttps://www.postgresql.org/docs/current/transaction-iso.html} Publish dispatch only from committed reservation state, through a recoverable delivery protocol. Aborting a database transaction doesn't undo GPU work already launched as a side effect.

Tenant allowance and physical worker capacity are separate reservations. Either coordinate them transactionally or use a recoverable staged protocol: reserve tenant allowance, obtain worker capacity, dispatch once, and release the first reservation if placement definitely failed. Don't release it merely because an acknowledgment timed out. Execution may already be running.

Budget memory before promising concurrency

The KV cache stores attention keys and values for active context. For a synthetic full-attention model with 80 layers, 8 KV heads, head dimension 128, and two-byte values, each cached token uses:

$2\times80\times8\times128\times2=327{,}680\text{ bytes}=0.3125\text{ MiB}.$

At 4,000 tokens, that's 1.2207 GiB per sequence before metadata and other memory. The leading factor counts keys and values. Use KV-head count, not query-head count, for grouped-query attention. Other architectures and cache layouts need their own formula.

PagedAttention maps logical token blocks to non-contiguous physical blocks.^{[4]Reference 4Efficient Memory Management for Large Language Model Serving with PagedAttentionhttps://arxiv.org/abs/2309.06180} With 16-token blocks, our shape uses 5 MiB per block. A's 47-token sequence requires three blocks and B's 31-token sequence requires two. They use 78 of 80 reserved token slots, leaving two slots unused. Paging limits this tail waste; it doesn't authorize block-table access.

BYTES_PER_TOKEN = 2 * 80 * 8 * 128 * 2
BLOCK_TOKENS = 16

def reserve_pages(input_tokens, max_output):
    if any(type(n) is not int or n < 0 for n in (input_tokens, max_output)):
        raise ValueError("token counts must be nonnegative integers")
    return (input_tokens + max_output + BLOCK_TOKENS - 1) // BLOCK_TOKENS

for tenant, tokens in (("A", 47), ("B", 31)):
    pages = reserve_pages(tokens, 0)
    print(f"{tenant}: {pages} pages, {pages * BLOCK_TOKENS - tokens} unused token slots")
print("MiB per page:", BYTES_PER_TOKEN * BLOCK_TOKENS / 2**20)
print("GiB for 4000 cached tokens:", round(BYTES_PER_TOKEN * 4000 / 2**30, 4))
assert reserve_pages(16, 1) == 2  # output growth crosses a page boundary
assert reserve_pages(0, 0) == 0
assert reserve_pages(2**54 + 1, 0) == 2**50 + 1  # exact integer arithmetic

A: 3 pages, 1 unused token slots
B: 2 pages, 1 unused token slots
MiB per page: 5.0
GiB for 4000 cached tokens: 1.2207

Keep page arithmetic in integers. Converting 2**54 + 1 tokens to a float before rounding would lose one token and underreserve a page. That regression input is far beyond a real model's context; deployment must enforce the actual context/output caps before allocation.

The formula above is aggregate logical KV storage. Per-device fit depends on tensor sharding or replicated KV heads, layer placement, precision, and engine metadata. Reserve weights, adapters, activations, graph buffers, and headroom too. A fleet-wide free-memory total doesn't prove that the selected worker has the required shape available.

Reserving the full input-plus-output cap is conservative. Incremental page reservation can admit more work, but then the platform must define what happens when growth fails: queue, preempt, swap, recompute, or terminate under an explicit policy. Don't promise unlimited streaming after reserving only the prompt.

For shared prefix blocks, distinguish physical occupancy from tenant accounting. One physical block may have several references. A policy might charge each tenant logical usage or apportion shared occupancy; either can be coherent if declared. Summing per-request logical bytes overstates physical occupancy when blocks are shared, while forgetting shared references can release live state too early.

Remote KV adds another placement option, not free memory. Authorize the trust scope and verify model, adapter, token sequence, position, and cache-layout compatibility before transfer. Compare lookup and transfer time with recomputation under the deadline. Retire ownership only after readers finish; network timeouts don't establish that a remote reader has stopped.

Give admitted tenants a bounded turn

Continuous batching removes finished sequences and admits new work at iteration boundaries.^{[5]Reference 5Orca: A Distributed Serving System for Transformer-Based Generative Models.https://www.usenix.org/conference/osdi22/presentation/yu} B shouldn't wait for A's entire generation merely because they started together. But continuous batching alone doesn't prevent A's long prompt from monopolizing a step.

Chunked prefill splits prompt work so it can interleave with decode. In current vLLM documentation, max_num_batched_tokens bounds tokens processed per iteration; changing it trades prefill throughput and first-token delay against streaming latency.^{[6]Reference 6Optimization and Tuning.https://docs.vllm.ai/en/latest/configuration/optimization.html} It isn't a tenant quota or a latency guarantee.

Use per-tenant queues and explicit service shares. A priority class can have a larger share without giving it unbounded priority forever. Otherwise sustained premium traffic can starve B's lower tier. Add maximum waiting time, aging, or reserved service for that tier, then declare what happens when offered load exceeds capacity.

The miniature deficit scheduler below charges synthetic work units, not requests or measured GPU time. A receives four units per round and B two. A's chunks cost three units each; B's cost one. Unused credit carries forward while the queue remains backlogged.

from collections import deque

queues = {"A": deque([3] * 10), "B": deque([1] * 20)}
quantum = {"A": 4, "B": 2}
credit = {tenant: 0 for tenant in queues}
served = {tenant: 0 for tenant in queues}
jobs = {tenant: 0 for tenant in queues}
for _ in range(3):
    for tenant, queue in queues.items():
        if not queue:
            credit[tenant] = 0
            continue
        credit[tenant] += quantum[tenant]
        while queue and queue[0] <= credit[tenant]:
            cost = queue.popleft()
            credit[tenant] -= cost
            served[tenant] += cost
            jobs[tenant] += 1
        if not queue:
            credit[tenant] = 0
assert served == {"A": 12, "B": 6}
assert jobs == {"A": 4, "B": 6}
print("served work units:", served)
print("completed chunks:", jobs)

served work units: {'A': 12, 'B': 6}
completed chunks: {'A': 4, 'B': 6}

The intended 2:1 work share is not a 2:1 request-count share. Real prefill and decode tokens have different, context-dependent costs, and co-batching changes those costs. Calibrate a scheduling cost model and reconcile estimates with measurements. Bound chunk size and batch-building time; a fair average over three rounds doesn't bound tail latency.

Speculative decoding adds proposal and verification work to that budget. Reserve the positions and temporary state actually processed, not just the tokens eventually accepted. Keep the proposal path inside the tenant's data permissions and verify compatibility with its target and adapter. A single-stream improvement doesn't establish better latency when other tenants compete for those same positions.

Preemption also has a cost. Preserve the generated prefix and execution state needed to resume, account for repeated prefill, and avoid evicting the same victim indefinitely. A victim-selection function doesn't free memory by itself. Retire or swap its actual blocks, update allocator state, and reserve the newcomer's blocks atomically with respect to competing scheduling decisions. An asynchronous device operation may need to complete before reuse is safe.

Keep adapter residency separate from permission

Low-Rank Adaptation (LoRA) adds small learned matrices to selected frozen base layers.^{[7]Reference 7LoRA: Low-Rank Adaptation of Large Language Models.https://arxiv.org/abs/2106.09685} Compatible requests can share base execution while using different adapters. The adapter is a behavior change, not an authorization boundary or a substitute for document access controls.

Resolve a tenant-facing alias such as docs-current into an approved immutable artifact revision and compatible base-model revision. Check permission before consulting a hot cache. Bind the resolved revision to the request for its whole lifetime, including retries and cached KV. A mutable alias mustn't silently change an active request halfway through generation.

As checked on October 10, 2026, vLLM supports per-request LoRA selection and warns that runtime loading should be used only in an isolated, fully trusted environment.^{[8]Reference 8LoRA Adaptershttps://docs.vllm.ai/en/stable/features/lora/} Don't expose raw load/unload endpoints, filesystem paths, or arbitrary artifact URLs to tenants. An adapter service should validate ownership, format, size, rank, base compatibility, and artifact integrity before a worker loads it.

Cache eviction needs reference tracking. If A's adapter is least recently used but still referenced by an in-flight batch, eviction can't invalidate that batch's handle. Evict an unreferenced artifact, wait, or use another worker. A least-recently-used list alone doesn't implement safe concurrent hot-swapping.

Scope retrieval and caching to the actual reader

A relevant document isn't necessarily an authorized document. Apply tenant and principal-level predicates inside the trusted retrieval operation, before evidence leaves that boundary. Revalidate sensitive permissions before materializing content if grants can change during the request. A broad top-K followed by filtering can lose all authorized candidates and may expose foreign IDs or scores to a component that wasn't allowed to see them.

The boundary matters more than a particular database feature. A trusted retrieval service can perform the check internally; a tenant-facing application mustn't receive forbidden evidence and promise to discard it later. Defense-in-depth checks at content fetch remain useful. In PostgreSQL, enabled row-level security without a policy denies ordinary row access, but superusers and BYPASSRLS roles bypass it. Table owners normally bypass it too unless subject to FORCE ROW LEVEL SECURITY. Test the actual role used by the retrieval service; enabling a policy doesn't establish that this role obeys it.^{[9]Reference 9PostgreSQL Row Security Policieshttps://www.postgresql.org/docs/current/ddl-rowsecurity.html}

This continuation of the identity fixture models retrieval selection and a cache namespace. It operates on in-memory rows with invented relevance scores, not a vector database. Alice's own tenant contains a second document she isn't allowed to read.

documents = [
    {"id": "A-public", "tenant": "A", "readers": {"alice"}, "score": 0.80},
    {"id": "A-private", "tenant": "A", "readers": {"carol"}, "score": 0.95},
    {"id": "B-secret", "tenant": "B", "readers": {"bob"}, "score": 0.99},
]

def search(context, top_k):
    if type(top_k) is not int or top_k <= 0:
        raise ValueError("top_k must be a positive integer")
    allowed = [d for d in documents if d["tenant"] == context.tenant
               and context.principal in d["readers"]]
    return [d["id"] for d in sorted(allowed, key=lambda d: d["score"], reverse=True)[:top_k]]

def namespace(context, acl_revision, adapter_revision, token_ids):
    # A conservative identity tuple, not vLLM's wire-level block-hash implementation.
    return (context.tenant, context.principal, acl_revision, "base-v3",
            adapter_revision, "tokenizer-v3", "kv-layout-v1", tuple(token_ids))

assert search(ctx, 1) == ["A-public"]
old = namespace(ctx, "acl-v1", "docs-v2", [1, 5, 9])
cache = {old: "private-cache-handle"}
assert namespace(ctx, "acl-v2", "docs-v2", [1, 5, 9]) not in cache
assert namespace(ctx, "acl-v1", "docs-v3", [1, 5, 9]) not in cache
bob = resolve_context("bob", "B")
assert namespace(bob, "acl-v1", "docs-v2", [1, 5, 9]) not in cache
print("Alice's result:", search(ctx, 1))
print("ACL revision, adapter revision, and cross-tenant misses verified")

Alice's result: ['A-public']
ACL revision, adapter revision, and cross-tenant misses verified

A key is a compatibility partition, not permission to fetch its value. The cache service must authenticate and authorize its caller. Don't accept a caller-selected trust-group ID or cache salt without resolving membership. Cached answers, semantic caches, and KV prefixes have different compatibility needs; an answer cache may also depend on retrieval corpus revision, tool state, and generation settings.

vLLM's prefix caching hashes token blocks with extra state such as LoRA identity and supports a first-block cache_salt to separate reuse groups.^{[10]Reference 10Automatic Prefix Cachinghttps://docs.vllm.ai/en/stable/design/prefix_caching/} This helps control prefix reuse and associated timing exposure; it doesn't replace API authorization, secure transport, or process isolation. Identical prompt text alone doesn't establish identical token IDs, positions, model behavior, or permission to share.

For private retrieval, default to a conservative principal or permission-set scope. Broader tenant-wide reuse needs a demonstrated shared-visibility contract. After revocation, stop new reads under stale grants and invalidate or make stale cache entries unreachable according to the retention policy. Existing streams and historical logs need their own revocation/retention decisions.

Choose the runtime boundary from the threat model

Namespaces, scheduler quotas, and cache salts protect different paths. They don't create separate operating-system or GPU security domains. Avoid a single “isolation level” that hides which resources remain shared.

NVIDIA distinguishes GPU instances from compute instances inside them. Compute instances within one GPU instance share its memory and some engines; they aren't equivalent to separate GPU-instance memory boundaries.^{[11]Reference 11Conceptshttps://docs.nvidia.com/datacenter/tesla/mig-user-guide/latest/concepts.html} A MIG partition isn't automatically a dedicated node or a virtual machine, and a large tensor-parallel replica may not fit the chosen partitioning scheme.

Encryption and personally identifiable information (PII) masking are additional controls, not replacements for these boundaries. A masked prompt can still retrieve another tenant's postmortem. A redaction service can also miss sensitive data; protect its placeholder map, logs, and error paths. Don't infer a specific regulatory requirement from the word “regulated”; document the actual data and adversary assumptions.

Separate usage, allocated cost, and the customer bill

Record immutable usage events with tenant, request, attempt, reservation, model/adapter revision, input/output counts, cached-input counts, terminal status, and metering-policy revision. A retry is a new execution attempt under the same logical request. Generated output, delivered output, and billable output can differ after disconnection; state which the contract charges.

A receipt saying “this request took 200 ms” doesn't mean it consumed 200 exclusive GPU-ms. Several requests may share a step, and a replica may span multiple GPUs. Shared execution cost is measured at the worker/step level; assigning it to requests is an allocation policy, not a uniquely observable physical truth.

Suppose four GPUs run a shared step for 0.2 seconds. That is 0.8 GPU-seconds. At an invented rate of $2 per GPU-hour, the step costs about $0.000444. The equivalent four-GPU node rate is $8 per node-hour: multiply that by 0.2 node-seconds, not 0.8 GPU-seconds. Mixing those units quadruples the charge.

This calculation assigns 25% of shared cost to A and 75% to B. The shares are policy inputs, not measured request-level utilization. Conservation should hold before rounding individual invoices.

from decimal import Decimal as D

seconds, gpus = D("0.2"), D(4)
per_gpu_hour = D(2)  # invented price, not a provider quote
node_hour = per_gpu_hour * gpus
step_cost = seconds * gpus * per_gpu_hour / D(3600)
assert step_cost == seconds * node_hour / D(3600)
shares = {"A": D("0.25"), "B": D("0.75")}
assert sum(shares.values()) == 1
allocated = {tenant: step_cost * share for tenant, share in shares.items()}
assert sum(allocated.values()) == step_cost
wrong = seconds * gpus * node_hour / D(3600)
assert wrong == 4 * step_cost
print("step cost USD:", f"{step_cost:.9f}")
print("allocated USD:", {t: f"{cost:.9f}" for t, cost in allocated.items()})
print("mixing GPU-seconds with node-hour price overcharges by:", f"{wrong / step_cost:.0f}")

step cost USD: 0.000444444
allocated USD: {'A': '0.000111111', 'B': '0.000333333'}
mixing GPU-seconds with node-hour price overcharges by: 4

Queue wait consumes deadline budget but isn't busy accelerator time. Idle replicas, CPU work, storage, transfer, and failed/recomputed attempts still cost money; track them separately and declare their allocation rule. A customer token price can differ from internal allocated cost. Compare the two to detect an unsustainable workload mix, without pretending the tariff is a hardware measurement.

Recover without losing ownership

A durable request state machine should distinguish reserved, queued, running, cancel-requested, usage-uncertain, and terminal states. Dispatch and terminal events need replay-safe processing. Cancellation isn't complete until the worker has stopped generation and retired device/state references, or recovery has established an equivalent safe boundary.

Before streaming begins, a bounded retry may be transparent if request ownership prevents duplicate execution. After output has been delivered, restarting can duplicate or contradict text. Use an explicit resumable-stream protocol with sequence numbers and compatible state, or terminate the stream with a clear partial-result error. Don't silently append a fresh answer to an old prefix.

Quarantine poison requests after bounded retries, but avoid copying raw prompts into broadly accessible dead-letter queues. Store a scoped identifier and protected diagnostic metadata. Worker readiness establishes that a worker can accept work; it doesn't prove model quality or tenant isolation.

For rollout, resolve immutable model and adapter revisions, isolate incompatible cache namespaces, and send only authorized shadow traffic to approved destinations. Shadow inference still consumes data, quota policy decisions, and compute. Change routing for new requests, drain old references, and retain a rollback path. A base-model revision needs compatible loaded weights and state, often in a separate pool; the essential requirement is compatibility, not a universal deployment topology.

Diagnose the opening incident with evidence

First stop the cross-tenant path, contain access, preserve scoped audit evidence, and inspect identity propagation, retrieval predicates, and cache ownership. Don't treat it as a latency tuning exercise.

Then explain why A and B wait. Compare queue age, admitted work, active sequences, KV occupancy, preemptions, adapter misses, and per-stage latency by tenant and route. A generic utilization percentage can hide a memory or policy bottleneck. Scale only when additional ready capacity addresses the measured constraint; otherwise repair admission, batch construction, or state handling.

A representative load test should include short summaries, long prompts, adapter churn, cache misses, disconnects, worker loss, quota-service outages, and retries. Name hardware/count/topology, engine version, model and precision, prompt/output distributions, warmup, concurrency, arrival process, and baseline. Measure overload behavior and per-tenant tails, not just aggregate throughput. The local examples above verify arithmetic and control logic only; they don't establish serving speed, isolation against a hostile runtime, or distributed correctness.

Follow-up questions

How would you recover a reservation whose worker disappeared after producing some tokens?

Keep its usage uncertain rather than refunding immediately. Reconcile durable dispatch/attempt records with worker events, establish that the old execution can't continue, and apply the documented partial-usage policy. A replacement attempt needs ownership control and a defined streaming behavior.

What would justify sharing cached prefixes across two principals in one tenant?

Both must be authorized for the prefix's full data and execution context, with compatible model/adapter/token state. Resolve a deliberate shared trust group, authorize cache reads, and specify how revocation invalidates reuse. Tenant membership alone doesn't prove shared document visibility.

What strong answers show

• Trusted identity is resolved before retrieval, adapter lookup, admission, cache access, and billing; worker access can't bypass the gateway.
• Quota and capacity reservations are atomic or recoverably staged, idempotent, and safe under uncertain execution.
• Memory calculations use the actual cache layout and placement, while scheduling charges bounded work rather than assuming requests cost the same.
• Logical cache/tenant controls are distinguished from process, GPU-instance, VM, and host boundaries.
• Usage events, internal cost allocation, and customer pricing have explicit units and duplicate-event semantics.
• Rollout, cancellation, and retry protocols preserve execution ownership and state compatibility, with production claims limited to measured evidence.

Check your understanding