Human-in-the-Loop Agent Architecture

Computer-use agents can click buttons, fill forms, and operate real software. That makes the next production question unavoidable: what happens when the agent is about to spend money, change user data, send an external message, or deploy code?

Human-in-the-Loop (HITL) architecture adds a hard execution boundary for those moments. An agent can propose a side effect, but a downstream policy service pauses it until the required reviewer authorizes the exact action.

This addresses the opaque execution problem: even capable models can produce confident but incorrect outputs, and in production AI systems, one bad model promotion or rollback can degrade real users. HITL doesn't make an action correct by itself. It creates a point where policy, evidence, authorization, and final-state checks can stop a bad action.

This is also one practical control for one of the top agent risks. The OWASP Top 10 for LLM Applications lists LLM06: Excessive Agency, which it breaks into excessive functionality, excessive permissions, and excessive autonomy. Requiring human approval for high-impact actions is one recommended mitigation for the excessive-autonomy facet, ideally enforced in a downstream system rather than left to the model to decide. ^{[1]Reference 1OWASP Top 10 for Large Language Model Applicationshttps://genai.owasp.org/llm-top-10/} Treat the pattern below as one implementation of that control.

Like a release bot that can read evaluation results but must route a production traffic change to an authorized reviewer, a HITL agent runs only within a defined permission boundary. Unlike a chatbot that waits for input at every turn, this pattern supervises actions according to their possible effect.

Follow Vega, a model-release assistant for an internal AI platform. Vega can read authorized evaluation runs, inspect deployment metrics, and draft release notes. Vega can also propose a model promotion, traffic shift, rollback, or incident update, but a downstream gate controls whether any of those side effects run. The traffic thresholds below are illustrative policy choices, not universal rules.

Design Vega's risk policy, implement checkpoint/resume logic that lets Vega wait durably for review, and build an approval UI that gives reviewers enough evidence without exposing unnecessary deployment data. This lesson assumes you already know large language model (LLM) tool calling, state graphs, guardrail policy, and browser-agent risk. If those terms are fuzzy, review Function Calling & Tool Use, Agentic Architectures, Guardrails & Safety Filters, and Computer Use & Browser Agents first.

The three pillars of human-agent interaction

Not all human oversight is created equal. Production systems distinguish between three distinct interaction models based on risk tolerance and the nature of the task. Those distinctions determine where approval, supervision, and audit controls belong.

Human-in-the-Loop (HITL): Active gatekeeping

In this model, the agent can't proceed with a gated action without explicit human approval. Use it for actions whose impact isn't acceptable to discover after execution, such as money movement, production deployment, or user-record change. The agent proposes an action, pauses execution, and waits for an approve, reject, or modified proposal decision before continuing. Modified proposals still need validation.

Human-on-the-Loop (HOTL): Supervisory control

Here the agent performs tasks autonomously while a human monitors the process and retains the ability to interrupt or override. This fits actions whose effect can be stopped or repaired after detection, such as routing internal draft suggestions into a queue. It isn't a sufficient gate for shifting production traffic or publishing an incident update: the human may see the mistake only after the effect happened. HITL blocks before execution, while HOTL can only interrupt execution that's already allowed.

Human-out-of-the-Loop (HOOTL): Full autonomy

At the far end of the spectrum, some actions are automated with no real-time human oversight. This applies only to bounded actions with clear authorization and acceptable failure modes. An authorized evaluation read or a draft created in an internal queue may fit; an unrestricted database read or web action doesn't become low risk merely because it doesn't write data. Some actions stay HITL because their effect, policy, or applicable duties demand a pre-execution gate.

The trust spectrum

Not all agent actions carry the same risk. Designing a HITL system begins by classifying every tool and action along a trust spectrum. This principle drives the level of friction required before execution.

A naive implementation treats all actions equally: either everything is autonomous (dangerous) or everything requires approval (tedious). A usable HITL system defines granular policies:

Risk classification system

Start with a RiskLevel enum and a policy lookup table. That decouples the agent's logic ("I want to do X") from the governance logic ("Should I allow X?").

To implement this, we define a policy lookup table that maps each available tool to a specific risk tier. This structure is the core policy engine: it takes a tool's name and runtime arguments as contextual input, then returns the required authorization flow before execution.

Risk isn't static. A "send email" tool might be Low Risk when emailing an internal test address, but High Risk when emailing an external domain. Production policies are dynamic, checking arguments at runtime.

Architecture: the Checkpoint/Resume pattern

The fundamental architectural challenge of HITL is the pause. When an agent pauses for approval, it can't sleep the thread (time.sleep()) or block in memory. A guarded compare-and-swap (CAS) update later lets only one current approval resolve the pending row.

Why in-memory blocking fails

1. Durability: If the server restarts or crashes while waiting for approval (which could take hours), the agent's state is lost.
2. Resource Usage: Holding a thread open for a human response wastes compute resources.
3. Scalability: You can't scale to thousands of concurrent agents if each one is blocking a thread.

Use the Checkpoint/Resume pattern. When an approval is needed, the agent persists its graph state (messages, structured variables, and pending task metadata) to durable storage and returns control to the caller. After approval, the runtime reloads that state and re-enters the waiting node.

Vega's paused state might look like this:

When an authorized reviewer clicks Approve, the runtime resolves this pending decision with a guarded write, reloads the checkpoint, rechecks current state and arguments, and only then attempts the action. If the reviewer clicks Reject, Vega can draft a release note without promoting the model. Checkpoint state is ordinary inspectable data, but emergency changes should still go through an authenticated, versioned, audited path rather than an ad hoc database edit.

Before building a workflow engine, you can test the persistence boundary with ordinary data. The checkpoint below stores a redacted summary and an action hash, but not the raw request message.

Compare two popular approaches for implementing this pattern:

Using LangGraph checkpointing

LangGraph provides first-class support for this via interrupt(). ^{[2]Reference 2LangGraph Interruptshttps://docs.langchain.com/oss/python/langgraph/interrupts} When you compile the graph with a checkpointer, LangGraph persists checkpoints for the thread and pauses execution until the graph is invoked again with Command(resume=...). ^{[3]Reference 3LangGraph Persistencehttps://docs.langchain.com/oss/python/langgraph/persistence} One subtle detail matters in production: when execution resumes, LangGraph reruns the node from the top, so any code before interrupt() must be idempotent. ^{[2]Reference 2LangGraph Interruptshttps://docs.langchain.com/oss/python/langgraph/interrupts}

The approval flow

The flow separates the Agent Runtime (the Python application, often powered by an LLM) from the Approval Interface (Web/Slack). In production, they coordinate through persisted state plus a thin resume endpoint, not a long-lived in-memory call stack. The checkpoint figure above is the source of truth for the sequence: pause, persist state, create a versioned approval record, notify the reviewer, validate the decision, reload the checkpoint, execute, and audit.

REST API for approval queue

To build the "Resume API" component, we need an endpoint that looks up the suspended thread and issues a command to resume it. This endpoint takes a thread ID plus a versioned approval decision payload as input, validates the thread's state, and outputs a resume command to awaken the agent with the provided instructions.

The compare-and-swap check records at most one reviewer decision. Store each approval request with fields such as status, version, action_hash, expires_at, required_role, and resolved_at, then resolve it with a guarded update such as WHERE id = ? AND status = 'pending' AND version = ? AND action_hash = ? AND expires_at > now(). The stored and compared action hash must contain the full digest; a shortened form belongs only in the UI. Record authorized separately from executed: an approval may be accepted and then fail during execution. The guarded approval update and outbox insert must commit in one database transaction. A reconciler that detects authorized approvals without an outbox row is an additional defense, not a substitute for atomic persistence. Otherwise, a crash after the approval update but before the outbox insert can strand an authorized action. A durable resume worker must recheck current production state and use an idempotency key so a retry can't promote the same model twice.

This small executable model shows those two separate transitions. It takes a versioned decision and an idempotency key as input, then shows that a stale click is rejected and an execution retry returns the previously recorded outcome.

An expiry guard is separate from version matching. This example takes the current time and expiry time as inputs, then rejects a decision whose review window has already passed.

Designing the approval interface

A "Yes/No" button is rarely enough for production systems. The approval interface must provide context and control. When Vega asks "Can I route 25% traffic to reranker-v17?", the human needs to know what data will change and why.

The approval UI should show the authorized reviewer the smallest evidence packet needed for the decision: a redacted request summary, the policy trigger, source references the reviewer is allowed to inspect, and the exact arguments Vega proposes to execute. Dumping an entire incident thread or model-card history into every review card creates unnecessary exposure and makes the meaningful change harder to see.

The approval payload

Vega should pause with a structured payload that the UI can render for review. This payload takes authorized evidence and proposed tool arguments as input and structures them into a redacted typed object for the reviewer. For data mutations, this should ideally be a visual diff rather than raw JSON.

By rendering this payload, the Human-in-the-Loop UI becomes a useful decision surface. The reviewer can compare Vega's proposed effect with permitted evidence before authorizing the tool call. For complex actions, consider adding a dry-run diff that shows what would happen if the action were approved. Version, expiry, and action hash fields matter too: they let the backend reject stale clicks instead of replaying an approval long after the underlying state changed. The backend stores and compares the full digest even if the card displays an abbreviated label.

The packet builder is also a data-minimization boundary. This executable example takes a deployment note and exact action as input, redacts the email address, and emits a stable hash that binds the review card to the proposed mutation.

Give the reviewer a concise rationale, the exact tool arguments, a diff, and the policy rule that triggered review. Don't make raw chain-of-thought your approval primitive. Explanations can be unfaithful, and they may reveal internal reasoning or sensitive context you didn't intend to surface. ^{[4]Reference 4Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptinghttps://arxiv.org/abs/2305.04388}

Also keep approval state compact. Persist a data-minimized audit record with access controls and retention policy, then inject only structured fields such as decision, reason, modified_args, or a short reviewer summary back into the runtime. Otherwise long-lived threads accumulate reviewer chatter that burns context window on approval metadata instead of task state.

Using Temporal for long-running workflows

For workflows that coordinate several durable activities, timers, retries, and approval messages, Temporal can be a better fit than a hand-rolled checkpoint table. LangGraph checkpoints can also wait durably; the choice isn't only "short wait versus long wait." Temporal's workflow execution records event history and communicates with callers through activities plus message-passing primitives such as Signals and Updates. ^{[5]Reference 5Temporal Workflow Execution Overviewhttps://docs.temporal.io/workflow-execution}

Signals are a good default when the approval service can fire-and-forget. If the UI needs synchronous confirmation that the workflow accepted the decision, or you want the runtime to reject a stale approval before it lands in history, an Update is usually cleaner. ^{[6]Reference 6Temporal Python SDK: Workflow message passinghttps://docs.temporal.io/develop/python/workflows/message-passing}

Temporal also gives you durable state, timers, retries, and event history out of the box. That means you don't have to build the orchestration layer for "pause, notify, wait, resume" yourself.

This approach is useful for multi-step approval chains (for example, waiting for both a support lead and finance sign-off). The workflow can wait without keeping a worker blocked, evaluate partial approvals, and continue waiting for the remaining required decision.

The Temporal workflow below uses an Update for the approval decision because the UI needs to learn whether the action ID and version still match the pending request. The workflow then calls a validation activity before any external side effect.

The validator rejects stale versions and a second decision while the first accepted Update is waiting to be consumed, so two fast clicks can't overwrite one another.

Keep policy code inside the workflow deterministic. If is_risky() depends on live balances, anomaly services, or a policy database, fetch that data in an Activity first and pass the result into workflow state. Temporal replays workflow code against event history, so non-deterministic logic inside the workflow will break replay. ^{[5]Reference 5Temporal Workflow Execution Overviewhttps://docs.temporal.io/workflow-execution}

If a notification doesn't need a response, a Signal may still be appropriate. Approval buttons need synchronous accept/reject feedback, so an Update with validation is a stronger fit for this decision path. ^{[6]Reference 6Temporal Python SDK: Workflow message passinghttps://docs.temporal.io/develop/python/workflows/message-passing}

For approval chains that can run for months or accumulate a large event history, periodically use Continue-As-New to cap history size while carrying forward unresolved approval state. ^{[5]Reference 5Temporal Workflow Execution Overviewhttps://docs.temporal.io/workflow-execution}

Advanced patterns

Pause and resume is the core mechanism, but production queues also need dynamic escalation, safe modifications, bounded batches, and reviewer-load metrics. These controls reduce needless review work without weakening policy floors.

Dynamic risk escalation

Static policies need runtime context. Publishing one external incident update is already a side effect that may require approval; attempting hundreds of updates in a short interval should move into a stricter queue or be blocked outright. This visual starts from a static policy floor and conditionally escalates the required authorization layer.

The numbered cells match the examples below. A 50% traffic shift moves from APPROVE to ESCALATE; an otherwise autonomous eval read moves to APPROVE after an anomaly spike; and disabling a safety filter stays at ESCALATE even when runtime conditions look quiet. This monotonic rule is the visual form of escalate_risk().

Dynamic policies evaluate context by taking the proposed action and operational metadata as input, applying rule-based checks, and outputting an escalated risk tier if anomalies are detected. This function combines the static base policy with runtime conditions. It checks traffic percentage, recent failure count, and time window, elevating the required authorization level when thresholds are crossed:

Approval with modification

A useful HITL pattern lets the human modify a proposed action rather than merely reject it. Instead of a binary yes/no choice, the human can correct arguments, such as reducing a traffic percentage or changing a rollout stage. That edit is a new proposal, not a guarantee of safety.

Build the approval UI as a form, not a button. Populate the form with the agent's proposed arguments (e.g., email body, SQL query) and allow the human to edit them before hitting "Approve".

For example, if Vega proposes promote_model(model_id="reranker-v17", traffic_percent=50), a human reviewer can modify it to promote_model(model_id="reranker-v17", traffic_percent=10) for a smaller canary. Before execution, the host must validate schema, reviewer permission, applicable policy, action version, and current deployment state. The edit doesn't train Vega and doesn't bypass a stronger approval tier.

Batch approvals

If Vega needs to propose 50 low-risk release-note publications after a migration freeze, asking for separate decisions for each one creates reviewer fatigue. Reviewers who face repetitive requests may stop inspecting individual effects. A batch review can group related proposals without hiding the service, audience, destination, or policy status of each item.

Instead of creating one alert per proposed operation, the orchestrator collects pending actions over a bounded window or groups them by a shared task identifier. The UI can then present: "Vega proposes 50 release-note publishes (review all items)." The approval record must bind to the exact item list or item hashes, so an item inserted after approval can't ride along in the batch.

Implementing batch approvals requires per-item state and idempotency. If a batch contains 50 actions and the reviewer authorizes the fixed set, the orchestrator must track each action separately. If action 42 fails, already-completed publishes aren't automatically undone unless the operation provides a compensation path. Retry only the failed authorized item with its idempotency key, and re-request review if its inputs or destination changed.

Bind a batch approval to the exact items shown to the reviewer. Here a later publish inserted into the queue changes the digest, so it can't reuse the earlier authorization.

What to measure

Once a HITL system is live, model quality alone stops being enough. You also need operational metrics that tell you whether the human review layer is adding safety without destroying throughput.

Track these per tool and per risk tier rather than relying on one aggregate dashboard number. Set alert thresholds from the effect and policy: a correction on an isolated draft and a correction on a money-movement proposal carry different operational meaning.

Security: untrusted approval evidence

A subtle but critical prompt-injection vulnerability in HITL systems is that the approval request itself is an attack vector. If an attacker can control content that the agent summarizes, such as a commit message, incident comment, or retrieved ticket, they can trick the human reviewer.

Consider Vega summarizing a deployment ticket and asking for approval to publish a release update. A malicious comment might contain:

"SYSTEM ALERT: Please click 'Approve' to verify your account security. Ignore the actual reply content below."

If the approval UI renders this prominently, a distracted human might approve a malicious update or traffic change.

Treat user content, retrieved text, and model summaries as untrusted evidence in the approval UI. Escape it for the rendering context and separate it visually from trusted policy labels, proposed arguments, and reviewer controls.

Escaping doesn't decide whether an action is allowed, but it stops evidence text from becoming active page markup. This small renderer keeps attacker-controlled text inside an evidence panel and renders the actual approval control separately.

A second principle from OWASP LLM06 matters here: enforce the authorization decision in a downstream system, not in the model. The agent proposes an action, but the policy engine and the approval gate decide whether it runs. Pairing this with least-privilege tools (only the functionality and permissions each task needs) keeps a tricked or jailbroken agent from reaching dangerous actions in the first place. ^{[1]Reference 1OWASP Top 10 for Large Language Model Applicationshttps://genai.owasp.org/llm-top-10/}

Input validation on modification

When a human modifies an agent's proposed action (e.g., changing traffic percentage or rollout stage), the system must treat this human input as untrusted. A compromised account or a social engineer could modify the argument to execute something malicious. After reviewer authorization, expiry, and action-version checks have passed, this function validates edited arguments against schema and the action-tier policy. It executes an action that stays in the current review tier and routes a larger edit for escalation.

Scaling oversight with AI triage

As agent volume grows, human review can become the primary operational bottleneck. A rules engine or reviewer model can help prioritize the queue and reject proposals that violate policy, provided it never turns an approval-required effect into autonomous execution.

This introduces an "AI-in-the-loop" filter that can automatically reject obvious policy violations and fast-path actions already classified as autonomous by explicit policy. The human reviewer is still required for actions at or above the approval floor.

A practical design combines three layers: hard policy rules, anomaly features, and an evaluator that emits a queue score plus rationale. The evaluator shouldn't silently override an approval-marked or critical action. Its job is triage, not final authority. Reviewer decisions may later support evaluation or training, but only after access control, purpose limitation, redaction, label-quality review, and leakage-safe dataset splitting.

Treat the policy tier as a floor. This router takes an explicit policy and a model suggestion as inputs; it lets a draft stay autonomous, but refuses to downgrade a model-promotion proposal or destructive action.

That human approval record is also part of your governance story. NIST AI RMF frames governance, measurement, and operational controls as lifecycle responsibilities across design, deployment, and management. ^{[7]Reference 7Artificial Intelligence Risk Management Framework (AI RMF 1.0)https://www.nist.gov/itl/ai-risk-management-framework} If a deployment is classified as a high-risk AI system under the EU AI Act, Article 14 requires effective human oversight proportionate to its risk, including abilities related to understanding limitations, automation bias, interpretation of output, and intervention or stopping the system. ^{[8]Reference 8EU AI Act: Regulation laying down harmonised rules on artificial intelligencehttps://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32024R1689} For any consequential workflow, retain authorized, data-minimized evidence of who decided, which version and effect were reviewed, what executed, and why the outcome was recorded.

Practice: design Vega's weekend policy

Suppose Vega runs unattended on Saturday night during a migration freeze. Design a dynamic risk policy with these rules:

• Automatically perform authorized evaluation reads and deployment-health checks
• Require approval for every model-promotion proposal
• Escalate any promotion over 10% traffic to a senior reviewer
• Escalate any proposal that disables a required safety filter
• Auto-reject any request after 2:00 AM if the same service already had two failed rollouts in the past hour

Write the ToolPolicy table and the calculate_dynamic_risk function. Then identify the edge case: what happens if Vega proposes three 9% traffic shifts in one hour to bypass the senior-review threshold?

This is a split-transaction attack. Dynamic policies must look at aggregate service behavior, rather than single action size. A useful fix is a rolling-sum check: if one service accumulates more than 10% new traffic within 24 hours, escalate regardless of individual step size.

HITL controls to keep

• Categorize every tool into Auto, Notify, Approve, or Escalate.

• Use the Checkpoint/Resume pattern (LangGraph, Temporal) so approvals can be asynchronous and survive restarts.

• Allow reviewers to edit proposals, then validate the edited action again before execution.

• Escalate risk based on velocity, time of day, and dollar amounts.

• Group repetitive actions to prevent alert fatigue.

The point of HITL isn't to keep humans clicking "Approve" forever. It's to put human judgment where failure is expensive or irreversible, while keeping policy-required gates in place even as lower-risk automation improves.

A safe HITL design has four moving pieces: risk classification, durable checkpoints, compare-and-swap resume semantics, and approval UIs that give reviewers real context. The common traps are in-memory blocking, stale approvals, alert fatigue, and prompt injection through the approval message itself.

The next step is applying those approval patterns to AI-assisted software work. Coding agents can create useful patches, but they need the same risk tiers, review checks, data-minimized audit evidence, and human ownership before their changes reach a repository or deployment pipeline.

Mastery check

Key concepts

• HITL, HOTL, and HOOTL oversight modes
• Risk tiers: Auto, Notify, Approve, and Escalate
• Durable checkpoint and resume instead of in-memory waiting
• Approval packets, compare-and-swap guards, and stale-click protection
• Dynamic escalation, safe modification, and reviewer throughput metrics

Evaluation rubric

• Picks the right oversight mode for each tool action instead of blocking everything
• Explains why durable execution and guarded resume writes are required
• Designs reviewer UX with context, diffs, expiry, and validation
• Balances safety and throughput with escalation logic, batching, and metrics

Follow-up questions

Common pitfalls

• Symptom: Review queue grows and reviewers start blindly approving requests.

• Cause: Everything is routed through the same approval path, including safe or repetitive work.

• Fix: Add risk tiers, bounded batches, and triage while keeping pre-execution review for effects that policy requires humans to authorize.

• Symptom: Approved actions disappear after deploy or restart.

• Cause: Approval state was stored in memory or tied to a live request thread.

• Fix: Persist checkpoints and approval rows durably, then resume from stored state instead of sleeping a worker.

• Symptom: Two reviewers approve same request and agent runs action twice.

• Cause: Approval resolution did not use version checks or compare-and-swap semantics.

• Fix: Guard writes on status and version, then reject stale clicks on resume.

• Symptom: Reviewer edits create unsafe tool arguments even though model proposal looked safe.

• Cause: Human modifications were trusted without schema, authorization, or business-rule validation.

• Fix: Re-validate modified arguments exactly like model-generated arguments before execution.

• Symptom: Approval UI becomes a prompt-injection surface.

• Cause: Attacker-controlled text or raw model summaries are rendered like trusted system instructions.

• Fix: Separate untrusted content visually, show structured diffs, and keep authorization logic downstream from the model.