Agent Failure & Recovery

A deploy-status agent asks the controller to queue a rollback for RUN-842. The controller saves job job-1. Before its reply reaches the agent, the connection drops.

The agent sees a timeout. The controller sees a queued job. If the agent treats those observations as equivalent to “the rollback failed,” it may submit another job. If it treats them as success, it may tell the user that a rollout has finished when it hasn't even started.

Both mistakes come from filling a gap in evidence with a convenient story. Recovery means finding out which state you're in, then choosing an action that's valid in that state.

Agent memory preserves information across turns. It doesn't make that information true or make an external operation safe to repeat. The agent runtime, the code around model calls and tools, needs to preserve those distinctions. The examples below use synthetic CI (continuous integration) records and a local SQLite controller fixture. They make no model calls and perform no deployments.

A status question doesn't authorize a rollback

Start one step before the timeout. The user asks, “What happened to run RUN-842?” That authorizes a status lookup, not a production mutation. A generated plan saying “roll back the failed run” doesn't expand that scope.

A tool call is a proposal. Before dispatch, the runtime checks the registered tool, argument shape, resource identifier, and caller permissions. A valid JSON object proves none of the other three. A rollback would require a separately authorized action with its exact target and current preconditions checked at execution.

For the read-only question, a small admission function can make the allowed action explicit. The authorized run IDs below stand in for a trusted permission lookup, not a field supplied by the model.

import re

class RejectedProposal(Exception):
    pass

def admit_lookup(tool, args, requested_run, authorized_runs):
    if tool != "get_run":
        raise RejectedProposal("only get_run is allowed for this task")
    if not isinstance(args, dict) or set(args) != {"id"}:
        raise RejectedProposal("get_run requires exactly one field: id")
    run_id = args["id"]
    if not isinstance(run_id, str) or not re.fullmatch(r"RUN-[0-9]{3,}", run_id):
        raise RejectedProposal("id must be a run identifier")
    if run_id != requested_run or run_id not in authorized_runs:
        raise RejectedProposal("run is outside the authorized task")
    return {"id": run_id}  # Copy only admitted fields.

for tool, args in [
    ("get_run", {"run_id": "RUN-842"}),
    ("create_release", {}),
    ("get_run", {"id": "RUN-843"}),
    ("get_run", {"id": "RUN-842"}),
]:
    try:
        print("admitted:", admit_lookup(tool, args, "RUN-842", {"RUN-842"}))
    except RejectedProposal as exc:
        print("rejected:", exc)

rejected: get_run requires exactly one field: id
rejected: only get_run is allowed for this task
rejected: run is outside the authorized task
admitted: {'id': 'RUN-842'}

The first rejection supplies a useful repair: change run_id to id. The second and third are scope violations, not reasons to expand the tool permissions. Keep correction feedback small and structured; don't echo secrets or an entire internal traceback into the next prompt.

A repaired proposal must pass admission again. This loop differs from retrying an already admitted network request: the model may change the payload, so it creates a new proposal for review.

Choose recovery from the failed boundary

Several failures can look like “the agent couldn't finish.” Locate the failed boundary before choosing a remedy:

1. Proposal validation: Missing arguments, an unknown tool, or a schema mismatch can be rejected before dispatch. If the action remains in scope, supply a bounded repair and validate the new proposal. Temperature zero doesn't guarantee identical output, and a repeated or changed response still needs admission.^{[1]Reference 1Reproducible outputs with the seed parameterhttps://developers.openai.com/cookbook/examples/reproducible_outputs_with_the_seed_parameter} A runtime schema change may instead require repairing the adapter or tool definition; another model call can't fix an incompatible deployment.
2. Dependency access: Connections can drop; services can report overload or rate limits. Decide whether the error is transient and whether repeating this operation is safe. A read may allow a bounded retry with jitter. A dispatched write may already have committed, even if its response is a 503. Backoff controls load; it doesn't settle the write's outcome.
3. Result interpretation: A tool can return HTTP 200 with an application-level failure, or confirm a queued job without confirming execution. Parse its documented response contract and verify the claim. An uncertain operation needs reconciliation or a valid repeat-safety contract. Compensation is appropriate only if confirmed partial effects call for a separately permitted recovery action.

A status code alone doesn't settle the side-effect question. A server error can arrive after a write commits, and a timeout can occur while the remote operation continues in the background. Likewise, some rate-limit responses describe an exhausted monthly quota rather than a brief burst overload. Classify typed errors and documented provider codes; don't search arbitrary exception messages for the word “timeout.”

The uncertain write has an identity

Return to the opening incident, now assuming an authorized operator approved queuing a rollback to revision rev-17. The runtime creates an operation key once, persists it with the approved payload, and sends both to the controller.

An idempotent operation has the same intended effect when repeated. An idempotency-key API gives the server a way to recognize repeated attempts at one logical operation. The server must implement that contract; attaching a random string to an ordinary request doesn't create it. AWS describes caller-provided request identifiers, parameter matching, and service-side atomicity as parts of safe retry design.^{[2]Reference 2Making retries safe with idempotent APIshttps://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/}

Three different identifiers serve different jobs:

The operation key is neither approval nor evidence of completion. Preserve it across process restarts. If the target revision changes, that's a new proposal, not another attempt with the old key.

Lose the reply after a real local commit

The fixture below uses two tables: jobs holds queued work; operations binds a key and payload to a job. Both inserts commit in one SQLite transaction. A fault inside that transaction rolls both back. A fault after it commits loses the reply but leaves both records present.

Predict the result before running it: after a lost reply, will submitting the same key and payload create job-2?

import json
import sqlite3
from contextlib import closing
from pathlib import Path
from tempfile import TemporaryDirectory

class KeyConflict(Exception):
    pass

class Controller:
    """Local database fixture. Enqueues jobs; does not execute a rollback."""
    def __init__(self, path):
        self.path = path
        with closing(sqlite3.connect(path)) as db, db:
            db.executescript("""
                CREATE TABLE IF NOT EXISTS jobs (
                    id INTEGER PRIMARY KEY, payload TEXT NOT NULL,
                    state TEXT NOT NULL
                );
                CREATE TABLE IF NOT EXISTS operations (
                    key TEXT PRIMARY KEY, payload TEXT NOT NULL,
                    job_id INTEGER NOT NULL REFERENCES jobs(id)
                );
            """)

    def enqueue(self, key, run_id, revision, fault=None):
        if not all(isinstance(v, str) and v.strip() for v in (key, run_id, revision)):
            raise ValueError("key, run_id, and revision must be nonempty strings")
        if fault not in {None, "before_commit", "after_commit"}:
            raise ValueError("unknown fault location")
        payload = json.dumps({"run_id": run_id, "revision": revision}, sort_keys=True)
        with closing(sqlite3.connect(self.path)) as db, db:
            db.execute("BEGIN IMMEDIATE")  # Serialize this fixture's writers.
            existing = db.execute(
                "SELECT payload, job_id FROM operations WHERE key = ?", (key,)
            ).fetchone()
            if existing:
                if existing[0] != payload:
                    raise KeyConflict("same key, different payload")
                return f"job-{existing[1]}"
            job_id = db.execute(
                "INSERT INTO jobs(payload, state) VALUES (?, 'queued')", (payload,)
            ).lastrowid
            db.execute("INSERT INTO operations VALUES (?, ?, ?)", (key, payload, job_id))
            if fault == "before_commit":
                raise TimeoutError("fault before transaction commit")
        if fault == "after_commit":
            raise TimeoutError("reply lost after transaction commit")
        return f"job-{job_id}"

    def lookup(self, key):
        with closing(sqlite3.connect(self.path)) as db:
            row = db.execute("""
                SELECT jobs.id, jobs.state FROM operations
                JOIN jobs ON jobs.id = operations.job_id WHERE operations.key = ?
            """, (key,)).fetchone()
            return None if row is None else (f"job-{row[0]}", row[1])

    def job_count(self):
        with closing(sqlite3.connect(self.path)) as db:
            return db.execute("SELECT COUNT(*) FROM jobs").fetchone()[0]

with TemporaryDirectory() as directory:
    path = Path(directory) / "controller.sqlite"
    controller = Controller(path)
    try:
        controller.enqueue("rb-842-1", "RUN-842", "rev-17", fault="after_commit")
    except TimeoutError:
        print("client: outcome unknown")
    # Reopen the file to model a new client/controller instance.
    controller = Controller(path)
    print("reconciled:", controller.lookup("rb-842-1"))
    print("same-key retry:", controller.enqueue("rb-842-1", "RUN-842", "rev-17"))
    assert controller.job_count() == 1
    try:
        controller.enqueue("rb-842-1", "RUN-842", "rev-16")
    except KeyConflict as exc:
        print("rejected:", exc)
    try:
        controller.enqueue("rb-843-1", "RUN-843", "rev-17", fault="before_commit")
    except TimeoutError:
        pass
    assert controller.lookup("rb-843-1") is None
    assert controller.job_count() == 1
    print("queued jobs:", controller.job_count())

    # Two workers submit a new logical operation concurrently.
    from concurrent.futures import ThreadPoolExecutor
    from threading import Barrier
    start = Barrier(2)
    def competing_send(_):
        start.wait(timeout=5)
        return controller.enqueue("rb-844-1", "RUN-844", "rev-17")
    with ThreadPoolExecutor(max_workers=2) as pool:
        results = list(pool.map(competing_send, range(2)))
    print("concurrent sends:", results)
    print("total queued jobs:", controller.job_count())
    assert results == ["job-2", "job-2"] and controller.job_count() == 2

client: outcome unknown
reconciled: ('job-1', 'queued')
same-key retry: job-1
rejected: same key, different payload
queued jobs: 1
concurrent sends: ['job-2', 'job-2']
total queued jobs: 2

The same-key retry returns job-1. The correct user-facing claim is “rollback job queued,” not “rollback completed.” Completion requires another observation from the component that executes the job.

The two competing workers each open their own database connection for the fresh rb-844-1 operation. BEGIN IMMEDIATE admits one writer at a time, so the second checks the ledger after the first has committed and receives the same job-2.^{[3]Reference 3Transactionhttps://www.sqlite.org/lang_transaction.html} There are two total jobs for two logical operations. A check performed outside the write transaction could let both workers observe an absent key; catching a duplicate-key exception later isn't enough if an external effect already occurred.

This short fixture uses the connection's default lock-wait timeout. Under sustained contention, BEGIN IMMEDIATE can fail with SQLITE_BUSY instead of waiting indefinitely.^{[3]Reference 3Transactionhttps://www.sqlite.org/lang_transaction.html} Any lock retry must fit the operation's deadline and preserve its key; serialization doesn't remove overload.

This is real transaction and file-reopen behavior, but the failures are deliberately injected Python exceptions. It doesn't test power loss, a remote controller, replication, or deployment execution. The transaction protects only records in this database. If a worker later calls an external deploy API, that call needs its own repeat-safety and reconciliation design.

The fixture uses one operation-key namespace with no expiration. It omits authentication and authorization, which must precede admission in a real service. A multi-tenant API needs a documented key scope and must check caller access before returning a previously stored result.

When replay is safe, and when it isn't

An uncertain write doesn't always forbid retry. Under a valid server-side idempotency contract, repeating the same key and same payload can be the prescribed recovery mechanism, including when the first attempt committed. Querying an operation-status endpoint may instead avoid another send or reveal that work is still running.

Read the exact API contract. Stripe, for example, stores the first executed request's status and body, including 500 responses; it rejects changed parameters for the same key. Its documentation permits pruning keys after at least 24 hours, after which a reused key can represent a new request. Validation failures and concurrent-request conflicts don't save an idempotent result because endpoint execution hasn't begun.^{[4]Reference 4Idempotent requestshttps://docs.stripe.com/api/idempotent_requests} A saved 500 can therefore repeat on replay; it isn't a reason to mint a new key. Those rules aren't universal defaults for other services.

Without such a contract, a status lookup returning “not found” may still be inconclusive: a replica can lag or the original request can be in flight. Don't turn absence from an eventually consistent lookup into proof that a second non-idempotent write is safe. Preserve the unknown state and reconcile or escalate.

Suppose an approved maintenance workflow cordons a Kubernetes node and evicts pods, then discovers that its intended replacement image is unavailable. The node may still be unschedulable. Controllers may already have created replacement pods elsewhere. Uncordoning permits new scheduling onto the node; it doesn't restore the evicted pods to their former placement.^{[5]Reference 5Safely Drain a Nodehttps://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/} First inspect what actually happened and whether the node is safe to accept work.

The Saga pattern can coordinate confirmed steps and their compensations, but compensation isn't mandatory after every failure and needn't run in exact reverse order. Domain rules may favor forward repair, parallel independent compensations, or human review.^{[6]Reference 6Compensating Transaction patternhttps://learn.microsoft.com/en-us/azure/architecture/patterns/compensating-transaction} Uncordoning might be an authorized recovery step; it might also reopen an unhealthy node. Persist the chosen compensation and its progress, recheck its current permissions and preconditions, and design repeat safety at the destination. An idempotency-key API is one mechanism, not a requirement every compensating tool supports.

A compensation is another action, not an undo button. Canceling a queued job might be possible; undoing a completed rollout might require a new reviewed deployment; retracting a notification doesn't make recipients forget it. A timeout during compensation leaves another uncertain effect to reconcile.

A checkpoint isn't a transaction with the outside world

A checkpoint stores workflow state so a worker can resume. Suppose the last checkpoint says “about to submit rb-842-1,” but the controller has already queued its job. Resuming that checkpoint restores the old local position; it doesn't remove the remote job.

Persist operation identity before dispatch, then record the confirmed result when it arrives. The gap between the external commit and the local result record still exists. Idempotency or reconciliation handles that gap.

LangGraph checkpointers preserve thread-scoped graph state at super-step boundaries, with pending writes supporting recovery of successful nodes within a failed super-step. A persistent backend is needed to retain it across process restarts.^{[7]Reference 7LangGraph Checkpointershttps://docs.langchain.com/oss/python/langgraph/checkpointers} Temporal reconstructs workflow state from event history and requires deterministic workflow behavior.^{[8]Reference 8Temporal Workflow Execution Overviewhttps://docs.temporal.io/workflow-execution} Recorded Activity completions aren't re-executed during replay, but an Activity whose effect occurred before its completion was reported can be retried. Side-effecting Activities therefore need repeat safety; workflow replay isn't an exactly-once external-effect guarantee.^{[9]Reference 9Activity Definitionhttps://docs.temporal.io/activity-definition}

Neither framework decides whether an old approval remains valid after a restart. Recheck authorization and action preconditions, preserve unresolved operations, and distinguish replaying recorded results from re-executing an unrecorded side effect. A generated summary isn't a substitute for this durable state.

Back off without multiplying the damage

For a read-only status request, repeating the operation is usually much simpler. Even so, retry needs a stopping rule. Three attempts means one initial call plus at most two retries, not three retries after the first call.

If 500 clients receive overload responses together and choose the same deterministic delay, their retries can arrive in another burst. Network and scheduling variance don't reliably disperse it. Backoff reduces retry frequency; jitter reduces alignment between clients.

Full jitter samples a delay across the entire exponential window.^{[10]Reference 10Exponential Backoff And Jitterhttps://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/} For zero-indexed failure count $k \ge 0$:

Here $T_0$ is the base retry delay and $T_{\max}$ is the ceiling. Independent random choices reduce coordinated bursts; they don't guarantee smooth arrivals or spare server capacity. A shared random seed can defeat the purpose.

Retry-After can be delay seconds or an HTTP date. The HTTP standard describes an indicated wait, not permission to repeat any operation.^{[11]Reference 11HTTP Semantics, Section 10.2.3: Retry-Afterhttps://www.rfc-editor.org/rfc/rfc9110.html#section-10.2.3} Our policy treats a valid provider hint as a minimum delay, even if it exceeds the local backoff cap. If waiting doesn't fit the caller's deadline, stop.

There's a catch: max(retry_after, jitter) gives every client the same delay when the hint exceeds every sampled jitter value. Predict what happens to these eight synthetic clients:

import random

retry_after, ceiling = 5.0, 0.2
samples = [random.Random(client).uniform(0.0, ceiling) for client in range(8)]
clamped = [max(retry_after, sample) for sample in samples]
spread = [retry_after + sample for sample in samples]
assert len(set(clamped)) == 1
assert len(set(spread)) == 8
assert all(5.0 <= delay <= 5.2 for delay in spread)
print("clamped: unique waits =", len(set(clamped)))
print("floor plus jitter: unique waits =", len(set(spread)))

clamped: unique waits = 1
floor plus jitter: unique waits = 8

Adding jitter after the minimum avoids that collapse in this fixture. It adds latency and can exceed the caller's deadline, so the helper below checks the combined wait. The 0.2-second window is illustrative; suitable dispersion depends on traffic, capacity, and the provider contract. These distinct synthetic floats aren't evidence that a live fleet avoids collisions.

The helper below is deliberately read-only. A provider adapter must classify an error as transient before wrapping it in TransientReadError; unknown exceptions propagate. The adapter also converts any supported retry hint to a nonnegative delay. The callable receives its remaining timeout budget and must pass that limit to its actual I/O operation.

import math
import random
import time

class TransientReadError(Exception):
    def __init__(self, message, retry_after=0.0):
        super().__init__(message)
        if not math.isfinite(retry_after) or retry_after < 0:
            raise ValueError("retry_after must be finite and nonnegative")
        self.retry_after = retry_after

class DeadlineExceeded(Exception):
    pass

def retry_read(call, *, attempts=3, timeout=10.0, base=0.2, cap=2.0,
               clock=time.monotonic, sleep=time.sleep, rng=None):
    if type(attempts) is not int or not 1 <= attempts <= 100:
        raise ValueError("attempts must be an integer in [1, 100]")
    if any(not math.isfinite(v) or v <= 0 for v in (timeout, base, cap)):
        raise ValueError("timeout, base, and cap must be finite and positive")
    rng = random.Random() if rng is None else rng
    deadline = clock() + timeout
    ceiling = min(base, cap)
    for attempt in range(attempts):
        remaining = deadline - clock()
        if remaining <= 0:
            raise DeadlineExceeded("no time for another call")
        try:
            result = call(remaining)
        except TransientReadError as exc:
            if attempt + 1 == attempts:
                raise
            delay = exc.retry_after + rng.uniform(0.0, ceiling)
            if delay >= deadline - clock():
                raise DeadlineExceeded("retry wait exceeds remaining time") from exc
            sleep(delay)
            ceiling = min(cap, ceiling * 2)
        else:
            if clock() >= deadline:
                raise DeadlineExceeded("read completed after its deadline")
            return result

class FakeClock:
    def __init__(self):
        self.now = 0.0
        self.waits = []
    def __call__(self):
        return self.now
    def sleep(self, seconds):
        self.waits.append(seconds)
        self.now += seconds

clock = FakeClock()
remaining_budgets = []
def status_read(remaining):
    remaining_budgets.append(remaining)
    if len(remaining_budgets) < 3:
        raise TransientReadError("temporary overload", retry_after=0.25)
    return "RUN-842 failed in unit-tests"

print(retry_read(status_read, clock=clock, sleep=clock.sleep, rng=random.Random(7)))
print("attempts:", len(remaining_budgets), "waits:", [round(v, 3) for v in clock.waits])
assert remaining_budgets == sorted(remaining_budgets, reverse=True)

# A delay requirement that doesn't fit must not produce an early send.
clock = FakeClock()
calls = []
def long_hint(remaining):
    calls.append(remaining)
    raise TransientReadError("wait required", retry_after=20.0)
try:
    retry_read(long_hint, timeout=3.0, clock=clock, sleep=clock.sleep)
except DeadlineExceeded:
    pass
else:
    raise AssertionError("deadline should reject the retry")
assert len(calls) == 1 and clock.waits == []

RUN-842 failed in unit-tests
attempts: 3 waits: [0.315, 0.31]

The fake clock makes waiting testable; the default implementation really sleeps and doesn't give every client the same random seed. This wrapper can't interrupt a callable that ignores its timeout argument. Production I/O needs connect/read/total timeouts or cancellation support, and cancellation doesn't prove that remote work stopped.

Choose who owns each retry and share the overall deadline and budget. Two nested three-attempt policies can permit up to nine sends when both retry the relevant failures; nine isn't the observed count unless those sends actually occurred. A validation library may re-ask after a parsed but invalid response while leaving transport failures to the SDK. Current Instructor documentation distinguishes these layers and notes that its elapsed-time and token retry limits don't interrupt a call or enforce a hard cumulative spend cap.^{[12]Reference 12Retry Logic with Tenacityhttps://python.useinstructor.com/concepts/retrying/} Inspect installed versions and instrument transport sends; don't infer them from a wrapper's attempt counter.

Repeated calls aren't always a loop

An agent reads RUN-842 twice and gets the same failure status. That may be wasted work. A paginated log reader makes ten calls to the same tool and obtains ten new pages. That may be exactly the right plan.

Tool name, argument hash, or embedding similarity alone can't distinguish them. Track progress against the task: new relevant log pages, a newly established failure cause, or a confirmed operation transition. Don't count a new timestamp or a rewritten summary as progress merely because its text changed.

The fixture below accepts a trusted, task-specific set of evidence IDs. Repeated IDs consume a no-progress allowance; new IDs reset it. A separate call cap still bounds a process that keeps finding new material forever.

class StopRun(Exception):
    pass

class ProgressBudget:
    def __init__(self, max_calls=8, max_stalls=2):
        if any(type(v) is not int or v <= 0 for v in (max_calls, max_stalls)):
            raise ValueError("limits must be positive integers")
        self.max_calls, self.max_stalls = max_calls, max_stalls
        self.calls = self.stalls = 0
        self.seen = set()

    def admit(self):
        if self.calls >= self.max_calls or self.stalls >= self.max_stalls:
            raise StopRun("call or no-progress budget exhausted")
        self.calls += 1  # Count the admitted call even if its result later fails.

    def observe(self, evidence_ids):
        if not isinstance(evidence_ids, set) or any(
            not isinstance(v, str) or not v for v in evidence_ids
        ):
            raise ValueError("expected a set of nonempty evidence IDs")
        new = evidence_ids - self.seen
        self.stalls = 0 if new else self.stalls + 1
        self.seen.update(evidence_ids)

budget = ProgressBudget()
for evidence in [{"status-v1"}, {"status-v1"}, {"log-page-1"},
                 {"log-page-1"}, {"log-page-1"}]:
    budget.admit()
    budget.observe(evidence)
try:
    budget.admit()
except StopRun as exc:
    print("stopped after", budget.calls, "calls:", exc)
assert budget.calls == 5 and budget.stalls == 2

stopped after 5 calls: call or no-progress budget exhausted

An invented evidence ID would fool this counter. In a real runtime, a trusted tool adapter and task-specific verifier decide which observations count. Hashes and semantic similarity can flag suspicious repetition for that verifier; they aren't proof that two requests have the same purpose. Record blocked proposals separately from dispatched calls so the audit trail doesn't disappear when a guard fires.

If the status endpoint lacks detail, change strategy to the failed job's logs rather than fetching the status again. Reflexion formalizes one broader pattern: use task feedback to generate a verbal lesson, store it in episodic memory, and apply it to a later trial.^{[13]Reference 13Reflexion: Language Agents with Verbal Reinforcement Learning.https://arxiv.org/abs/2303.11366} A deterministic “missing field id” error usually doesn't need that extra machinery. Neither a reflection nor a pinned reminder can authorize a write or establish that it committed.

Context capacity and cumulative spend are different limits

A model's context window limits one request. A run's token or money budget limits accumulated work across requests. Compaction can reduce the next prompt; it can't refund tokens already spent.

Suppose a fixed prefix has h tokens and each completed turn adds s tokens that later requests resend. Over n requests, input tokens total:

With h = 100, s = 50, and n = 4, the prompts contain 100, 150, 200, and 250 tokens: 700 total. This simplified calculation excludes output tokens and provider-specific cache pricing. It explains why a run with a modest final context can still have substantial cumulative usage.

Check a conservative input estimate and reserve a bounded output allowance before dispatch. Account for actual usage afterward. If the next request needs at most 120 input tokens plus 80 output tokens, a run with only 150 tokens left can't admit it under that reservation rule. Raising an exception after spending 200 tokens is accounting, not prevention.

Parallel calls need atomic reservations from the same budget. Unknown pricing shouldn't silently become zero cost. An ambiguous provider timeout may also leave usage unknown, so retain a conservative reservation until reconciliation. A local reservation is only as strict as its tokenizer estimate, provider accounting, and enforced output limit.

Stop rediscovering an outage

Retries give a dependency a few chances to recover. A circuit breaker prevents every subsequent request from spending that same allowance on a dependency that's persistently failing.

In the breaker below, closed admits calls; open rejects calls during a cooldown; half-open admits one probe. A successful probe closes it. A counted dependency failure or cancelled probe reopens it. Other errors propagate and release the probe slot while leaving it half-open; they supply no health verdict. Other policies allow several probes or use rolling failure rates.^{[14]Reference 14Circuit Breaker patternhttps://learn.microsoft.com/en-us/azure/architecture/patterns/circuit-breaker} One probe and a counter reset by successful calls are deliberate choices here.

The subtle part is concurrent completion. Call A starts while the breaker is closed. Call B then fails and opens it. A succeeds late. That old success may be a valid result for A, but it shouldn't erase the newer open state. Each admission therefore carries a generation number; opening the circuit invalidates older health updates.

Cancellation creates another trap. If the half-open probe is cancelled, leaving its “in flight” flag set would block every future probe. This implementation conservatively reopens for another cooldown and propagates cancellation. It doesn't claim cancellation means a remote side effect failed.

import asyncio
import math
import time

class CircuitOpen(Exception):
    pass

class CircuitBreaker:
    def __init__(self, threshold=2, cooldown=5.0, clock=time.monotonic):
        if type(threshold) is not int or threshold <= 0:
            raise ValueError("threshold must be a positive integer")
        if not math.isfinite(cooldown) or cooldown <= 0:
            raise ValueError("cooldown must be finite and positive")
        self.threshold, self.cooldown, self.clock = threshold, cooldown, clock
        self.state, self.failures, self.generation = "closed", 0, 0
        self.opened_at = 0.0
        self.probe = False
        self.lock = asyncio.Lock()

    def _open(self):
        self.state = "open"
        self.opened_at = self.clock()
        self.probe = False
        self.generation += 1

    async def call(self, fn):
        async with self.lock:
            if self.state == "open":
                if self.clock() - self.opened_at < self.cooldown:
                    raise CircuitOpen("cooldown active")
                self.state = "half_open"
            if self.state == "half_open" and self.probe:
                raise CircuitOpen("probe already in flight")
            is_probe = self.state == "half_open"
            if is_probe:
                self.probe = True
            generation = self.generation
        try:
            result = await fn()
        except BaseException as exc:
            async with self.lock:
                if generation == self.generation:
                    if is_probe:
                        self.probe = False
                    if isinstance(exc, (TimeoutError, ConnectionError)):
                        self.failures += 1
                        if is_probe or self.failures >= self.threshold:
                            self._open()
                    elif is_probe and isinstance(exc, asyncio.CancelledError):
                        self._open()
            raise  # Includes cancellation; never turn it into a successful fallback.
        async with self.lock:
            if generation == self.generation:
                self.state, self.failures, self.probe = "closed", 0, False
        return result

async def exercise_breaker():
    now = [0.0]
    breaker = CircuitBreaker(threshold=1, clock=lambda: now[0])
    started, release = asyncio.Event(), asyncio.Event()
    async def slow_success():
        started.set()
        await release.wait()
        return "old result"
    async def unavailable():
        raise TimeoutError("controller unavailable")
    old = asyncio.create_task(breaker.call(slow_success))
    await started.wait()
    try:
        await breaker.call(unavailable)
    except TimeoutError:
        pass
    release.set()
    assert await old == "old result"
    assert breaker.state == "open"
    print("late success:", breaker.state)

    now[0] = 5.0
    started.clear()
    release.clear()
    probe = asyncio.create_task(breaker.call(slow_success))
    await started.wait()
    try:
        await breaker.call(unavailable)
    except CircuitOpen:
        print("second probe: rejected")
    probe.cancel()
    try:
        await probe
    except asyncio.CancelledError:
        pass
    assert breaker.state == "open" and not breaker.probe
    now[0] = 10.0
    release.set()
    assert await breaker.call(slow_success) == "old result"
    assert breaker.state == "closed"
    print("after cancellation and a new probe:", breaker.state)

    try:
        await breaker.call(unavailable)
    except TimeoutError:
        pass
    now[0] = 15.0
    async def invalid_result():
        raise ValueError("adapter rejected the result")
    try:
        await breaker.call(invalid_result)
    except ValueError:
        pass
    assert breaker.state == "half_open" and not breaker.probe
    assert await breaker.call(slow_success) == "old result"
    assert breaker.state == "closed"
    print("non-health error: propagated; next probe admitted")

asyncio.run(exercise_breaker())

late success: open
second probe: rejected
after cancellation and a new probe: closed
non-health error: propagated; next probe admitted

The broad exception clause exists to clean up and re-raise, not to swallow programming errors. Only the two selected dependency exceptions count toward the threshold. Schema and permission failures aren't evidence that a dependency is unhealthy. A provider adapter can add documented overload errors, scoped carefully so one tenant's quota doesn't open a circuit for everyone.

This is an in-process, single-event-loop breaker. It doesn't coordinate a fleet, persist across restarts, or prevent an already-running request from completing. Its cleanup covers ordinary task cancellation, not process termination or repeated cancellation during cleanup. Every admitted I/O call still needs its own timeout. Test any shared-store or service-mesh implementation under its actual concurrency and failure model.

Tiered model fallbacks and graceful capability degradation

When a provider is unavailable, continuing with a backup is one option. Waiting, returning an unavailable result, reducing scope, or handing off may fit the task better. A configured cascade might include:

• A second provider: Normalize message roles, tool formats, schema support, context and output limits, and refusal or incomplete-response handling. Translation alone doesn't establish equivalent behavior. Evaluate this route on the actual task before relying on it.
• A narrower model task: For example, classify a supplied log excerpt while disabling mutation tools. Small models aren't universally distilled, cheaper, or incapable of multi-step planning; frontier size doesn't certify safe autonomy either. Capability and authority need task-specific evaluation and host controls.
• A cache or human handoff: Report a scoped, dated observation or package unresolved work for review. A cache can preserve facts observed earlier without supplying the current evidence the task needs.

Routes share the remaining budget and cancellation state. Switching models doesn't enlarge action authority or reset an unresolved write. Two model providers may also share an infrastructure or tool dependency, so “cross-provider” doesn't guarantee independent failure modes.

Suppose the live controller is unavailable. A cache says job-1 was queued forty minutes ago. That may answer “what was last observed?” It doesn't answer “has the rollback finished now?”

Don't accept has_live_evidence=True from generated text. Evidence must arrive through a trusted adapter, identify the correct run and operation, and meet the claim's freshness requirements. A backup model can't manufacture missing controller observations.

The example uses an injected trusted observation and produces the final sentence deterministically. Its age threshold is a task policy, not a universal definition of “live.” Even a fresh read describes an observed state at a time, not an eternal guarantee.

from dataclasses import dataclass

@dataclass(frozen=True)
class Observation:
    run_id: str
    job_id: str
    state: str
    observed_at: int
    source: str

def answer_status(observation, *, run_id, job_id, now, max_age=30):
    if type(now) is not int or type(max_age) is not int or max_age < 0:
        raise ValueError("now and max_age must be integer seconds; max_age >= 0")
    if observation is None:
        return "Current rollback status is unavailable."
    if not isinstance(observation, Observation):
        raise ValueError("expected a trusted adapter observation")
    if (observation.run_id, observation.job_id) != (run_id, job_id):
        return "Current rollback status is unavailable."
    if observation.source != "controller" or observation.state not in {"queued", "running", "completed", "failed"}:
        return "Current rollback status is unavailable."
    if type(observation.observed_at) is not int:
        raise ValueError("observed_at must be integer seconds")
    age = now - observation.observed_at
    if age < 0 or age > max_age:
        return "Current rollback status is unavailable."
    return f"{job_id} was {observation.state} when checked {age}s ago."

fresh = Observation("RUN-842", "job-1", "queued", 990, "controller")
stale = Observation("RUN-842", "job-1", "completed", 100, "controller")
for observation in (fresh, stale, None):
    print(answer_status(observation, run_id="RUN-842", job_id="job-1", now=1000))

job-1 was queued when checked 10s ago.
Current rollback status is unavailable.
Current rollback status is unavailable.

A Python class isn't an authenticity boundary: malicious code can construct one. The fixture tests the selection and wording rules, not whether a network response is genuine. Real evidence must come from an authenticated, authorized controller client with a defined timestamp and consistency model. A stored observation can also become stale between validation and delivery.

Fallback routes must preserve permissions and schema rules. A cached response must be scoped to the caller; a second provider must be allowed to receive the data. If no route can meet the claim, return a clear unavailable result or hand off to a reviewer. Don't turn a cancelled request into an unsolicited fallback attempt.

Measure recovery without hiding unfinished work

In a synthetic report of 100 runs requiring intervention, sixty completed the original task, twelve returned a useful but explicitly degraded answer, eighteen were handed off safely, and ten remained unresolved. Those outcomes tell different stories:

You could define a broad safe-disposition rate as 90/100, but calling it a 90% task-success rate would hide the lost capability and review load. Count unique runs in the denominator, not retry attempts, and define how pending runs and timeouts are reported. With no intervened runs, the rate is undefined, not zero or one.

Alongside outcome counts, record attempts per logical operation, deduplicated requests, reconciliation duration, unknown-write outcomes, rejected proposals, no-progress stops, and circuit-open rejections. Separate user cancellations from dependency failures. Keep operation keys out of high-cardinality metric labels; use them in appropriately protected traces.

An escalation packet should contain the authorized task, operation key and exact proposed payload, last confirmed observation, unresolved state, attempts already made, and why automation stopped. Copy allowlisted, redacted fields instead of attaching a raw transcript. Human approval still needs to bind to the action that will execute, and stale preconditions must be rechecked.

Fault injection: change the failure point

The examples establish specific runtime properties. They don't establish that a language model picks the right strategy, that a judge recognizes every false claim, or that a production controller survives a partition. Extend the checks at the boundary you intend to rely on:

For a service integration, also test competing workers, key expiration, crash/restart boundaries, replica lag, authorization changes, and the worker that executes queued jobs. Keep expected outcomes in the harness, not in the agent's prompt or memory during measurement. A harness that supplies the missing production evidence is testing a different system.

Mastery check

Start from the lost reply in the opening. Explain why retrying with a new key and declaring success are both wrong. Then identify what each component can establish: the proposal validator, the controller's operation record, the workflow checkpoint, and the final status observation.

A good answer distinguishes admission, queue acceptance, workflow progress, and deployment completion. They aren't interchangeable. Recovery preserves operation identity, obtains suitable evidence, and stops when a safe next action can't be established.