LearnAdvanced Agents & RetrievalComputer-Use / GUI / Browser Agents

🤖HardLLM Agents & Tool Use

Computer-Use / GUI / Browser Agents

Build browser and desktop agents whose proposed clicks and keystrokes remain behind host policy, approval, verification, and sandbox controls.

28 min read

Learning path

Step 121 of 177 in the full curriculum

Code Generation & Sandboxing Human-in-the-Loop Agent Architecture

Personalize this lesson

Adapt explanations and teaching visuals to your background and preferred voice.

Code-generation agents need sandboxes because generated commands can affect files, networks, and secrets. Computer-use agents add a different risk surface: they can propose interactions with a visible browser or desktop. Extend sandboxing discipline to CI dashboards, internal applications, and workflows where a model sees pixels and proposes mouse or keyboard actions.

You don't need to study image encoders in detail before starting. The practical requirement is simpler: understand that the agent grounds actions from screenshots, text, and accessibility metadata, then let the host wrap that perception loop in policy and sandbox controls.

An infrastructure team receives an incident: "Why did deploy run RUN-78432 fail after the test stage?" Assume the legacy CI dashboard exposes the failing step, screenshot, and redacted log excerpt in a browser portal but doesn't provide a supported API for that evidence. A read-only agent pilot could open the permitted portal session, locate the failing job and traceback, and prepare evidence for a reviewer.

If stable API or selector-based automation works, use it first. Computer use becomes relevant when an approved workflow depends on visual evidence or a legacy interface that deterministic automation can't reliably address. Even then, the model should propose steps while a host process controls credentials, action policy, and any record mutation.

Computer-use tools let models receive screenshots, and sometimes supporting browser state or accessibility metadata, then propose input events such as moving a virtual mouse, clicking, typing a run ID, scrolling, or requesting another screenshot.^{[1]Reference 1Introducing computer use, a new Claude 3.5 Sonnet capabilityhttps://www.anthropic.com/news/3-5-models-and-computer-use} The host decides whether and where a proposed event executes.

This interface can cover permitted legacy tools or visual portal steps with no suitable API. It also creates risk: a mis-grounded click, an overbroad session, or a leaked credential can change external state. Treat computer use as delegated access, not as a convenience wrapper around screenshots.

From curated tools to full computer control

Ordinary function calling gives a model a curated set of typed operations that the developer can separately authorize. Typed doesn't automatically mean safe, but it creates a narrow action surface. Computer use exposes a broader action surface: from a screenshot, sometimes augmented with an accessibility tree or optical character recognition (OCR) output, the model proposes primitive input events.

The shift changes the threat model. A function-calling agent can only call the tools you exposed. A computer-use agent can, in principle, open any application visible on the virtual desktop, click any pixel, and type into any field. That reach makes the pattern useful for the long tail of workflows that have no API, and explains why every production deployment must enforce a permission boundary with multiple independent layers of control.

The observe-act loop

Computer-use agents follow a sense-plan-act cycle. The host captures the screen, the model proposes the next low-level action, the host validates and possibly executes that action, and the next observation returns to the model.

The loop repeats until the host verifies completion, a limit expires, or a safety boundary intervenes. Each additional observation and model decision consumes latency and model usage, which is why bounded trajectories matter.

The main illustration below shows the architecture for the running CI incident example. An incident enters at the top, then the agent captures an observation, the model proposes an action, and the host validates and executes only permitted steps inside a restricted session. Policy, approval, audit evidence, and the runtime boundary surround the loop.

Host-controlled computer-use loop where observation leads to one proposed GUI step, policy validation gates sandbox execution, verification either returns to observation or completes through teardown. — Computer use is a host-controlled loop. The model proposes one GUI step, but policy, sandbox execution, human review, audit logging, and final verification stay outside the model.

The CI dashboard runs inside a restricted browser session because the task depends on visual evidence rather than an available API. A policy engine sits between the model's proposal and execution. Any proposed incident update or deploy action is routed to a reviewer with the relevant observation and intended change. Audit records should capture policy evidence without copying secrets or more log data than retention policy permits.

A host can make that split concrete before integrating a model. In this miniature policy, reading a permitted CI page can execute, while mutating incident state pauses for approval.

route-gui-action-proposals.py

from dataclasses import dataclass

@dataclass(frozen=True)
class Action:
    domain: str
    kind: str
    target: str

ALLOWED_DOMAINS = {"ci.example", "incidents.example"}
READ_ACTIONS = {"view_log", "open_run", "scroll", "capture_reference"}
WRITE_ACTIONS = {"update_incident", "rerun_deploy", "submit_form"}

def route(action: Action) -> str:
    if action.domain not in ALLOWED_DOMAINS:
        return "blocked: domain outside task scope"
    if action.kind in READ_ACTIONS:
        return "execute read-only step"
    if action.kind in WRITE_ACTIONS:
        return "pause: reviewer approval required"
    return "blocked: unknown action kind"

for proposal in [
    Action("ci.example", "view_log", "RUN-78432"),
    Action("incidents.example", "update_incident", "INC-2048"),
    Action("ci.example", "erase_run", "RUN-78432"),
    Action("mail.example", "open", "inbox"),
]:
    print(route(proposal))

Output

execute read-only step
pause: reviewer approval required
blocked: unknown action kind
blocked: domain outside task scope

The same pattern becomes easier to reason about when you strip away the story details and look at the reusable production architecture.

Computer-use architecture: grounded observations feed model reasoning, the model proposes one GUI primitive, and host controls validate, sandbox-execute, audit, and apply stop rules outside the model. — Treat the model as a decision engine, not the executor. Stable grounding signals go in, primitive actions come out, and the host keeps policy, sandbox, audit, and stop rules.

A concrete five-step trace for RUN-78432

To make the loop tangible, walk through the CI failure inspection task step by step using proposed GUI actions.

Step 1. The host opens a fresh restricted browser profile and establishes an authorized CI session through a credential broker. The model doesn't receive the password or propose typing it. Once authenticated, the host captures the permitted dashboard view and relevant grounding metadata.

Step 2. After the login succeeds, the host captures the next screenshot showing the run search form. The model outputs type("RUN-78432") into the run field and a left_click on the search button.

Step 3. The results page loads with failing test output visible. After seeing the log region, the model outputs a mouse_move over the failed step, a left_click to expand it, and a scroll action to bring the traceback into view. The host captures the new screenshot and returns it.

Step 4. The model reads visible evidence and proposes preparing a draft incident note with the failing command, traceback line, and artifact link. The host policy permits extracting that evidence but doesn't allow the model to rerun deploys or edit incident severity from the page alone.

Step 5. The host presents the gathered evidence to a reviewer. If the reviewer approves a specified incident update, a separately authorized write records the note and status. The host verifies the intended incident changed and that unrelated incidents did not change before tearing down the session.

At each step the policy engine inspected the proposal. Navigation and evidence collection stayed read-only; the incident write required approval. The audit record stores action proposals, policy outcomes, relevant observation identifiers, approval, and final verification, with secrets and unnecessary raw log content excluded.

Don't represent authentication as text the model should type. Represent it as a host-owned setup step that returns only a session outcome.

keep-credentials-host-side.py

def establish_session(task: str, credential_handle: str) -> dict[str, str]:
    assert credential_handle.startswith("broker://")
    return {"task": task, "session": "authenticated", "visible_secret": "none"}

model_context = {"task": "inspect CI failure evidence", "run_id": "RUN-78432"}
session = establish_session(model_context["task"], "broker://ci-readonly")

print("model fields:", sorted(model_context))
print("session:", session["session"])
print("password exposed:", session["visible_secret"] != "none")

Output

model fields: ['run_id', 'task']
session: authenticated
password exposed: False

Grounding pixels to intent

The hardest part of the observe step is turning raw pixels into reliable intent. A screenshot is only a grid of colored pixels. The model must map regions of that grid to UI elements, understand which element corresponds to "the failed test details for this run," and decide the next physical action that will advance the goal.

Pure coordinate-based clicking is extremely brittle. If the CI dashboard runs an A/B test that moves the "Failed tests" tab twenty pixels down, or if a different viewport collapses the sidebar, or if a responsive layout kicks in on a slightly narrower window, the coordinates the model learned on the previous run become invalid. The model then clicks empty space and the loop stalls.

That's why a browser-agent host can augment screenshots with an accessibility tree, OCR, or element metadata. The model can then refer to role, name, or visible text instead of only fragile (x, y) coordinates. These signals improve grounding, but page content is still untrusted and can't authorize a side effect.

Grounding metadata is useful only when it selects one target. If two buttons share the same label, the host should request a new observation or human help instead of clicking a guess.

resolve-accessible-target.py

from dataclasses import dataclass

@dataclass(frozen=True)
class Element:
    role: str
    name: str
    element_id: str

def resolve(elements: list[Element], role: str, name: str) -> str:
    matches = [element.element_id for element in elements if element.role == role and element.name == name]
    if len(matches) != 1:
        return f"blocked: expected one target, found {len(matches)}"
    return f"click element {matches[0]}"

page = [
    Element("button", "Failed tests", "failures-1"),
    Element("button", "Failed tests", "failures-2"),
]
print(resolve(page[:1], "button", "Failed tests"))
print(resolve(page, "button", "Failed tests"))

Output

click element failures-1
blocked: expected one target, found 2

How providers expose computer use

Anthropic documents computer use as a versioned tool in the Messages API. Its documentation defines tool versions, supported model pairings, an agent loop, and actions such as mouse movement, clicking, typing, key presses, scrolling, waiting, and requesting screenshots.^{[2]Reference 2Computer Use Toolhttps://docs.anthropic.com/en/docs/agents-and-tools/tool-use/computer-use-tool} Check its current compatibility table when implementing rather than hard-coding a tool identifier into architectural guidance.

Your client code is responsible for executing an allowed primitive inside the controlled environment, capturing the next observation, and returning the result in the next tool_result message.

OpenAI documents computer use through the Responses API. Its guide shows computer_call items that contain proposed actions, and the host answers with computer_call_output plus the next screenshot. The host still owns browser isolation, step execution, and confirmation before risky actions.^{[3]Reference 3Computer Usehttps://developers.openai.com/api/docs/guides/tools-computer-use}

Google documents computer use through the Gemini API. The model analyzes the user request plus screenshots, returns UI-action function calls, and can attach a safety_decision; your client still decides whether to execute that step, captures the result, and sends the next observation back.^{[4]Reference 4Computer Usehttps://ai.google.dev/gemini-api/docs/computer-use}

Across all three provider patterns, the developer owns the execution environment. The model proposes actions; application code enforces boundaries and decides when to stop or escalate to a human.

Browser agents versus full desktop agents

Many real production workloads stay inside the browser. When that's true, you have lighter options.

Use Playwright or Selenium directly when an approved target exposes stable selectors and deterministic automation satisfies the workflow. Browser-agent harnesses and evaluation environments can add observation/action adapters for model-driven experiments. Prefer the narrowest control surface that meets the task rather than moving to pixel-level control by default.

Consider full computer-use primitives only when authorized tasks require visual interpretation, browser UI interaction, or a native desktop application and no narrower supported integration is suitable. Don't use computer use to evade a site's access controls or anti-bot policy.

Full desktop computer use, in the style of OSWorld, becomes necessary when the job spans a browser window plus a thick client that has no web equivalent. The sandbox requirements grow because you must now present a believable desktop to both the browser and the native application while still keeping every input observable and revocable by the host.

Automation escalation ladder from direct APIs to Playwright to browser agents to desktop agents, with illustrative risk and action-surface bars rising as control broadens. — Move right only when narrower surfaces can't satisfy the authorized task. Each step adds a broader proposed-action surface and more observation handling.

Sandboxing and safety boundaries

The key engineering decision isn't only the prompt you give the model. It's the environment in which approved actions run.

For a privileged engineering workflow, begin each task in a fresh restricted browser profile or disposable runtime appropriate to the threat model. Expose only the applications and domains needed for the task. Don't expose personal email, password managers, general-purpose terminals, or package installation to a dashboard-inspection session.

Route network traffic through policy that permits only task-required domains. Establish credentials host-side rather than placing passwords in model-visible text. An action validator should inspect each proposed target, action class, active domain, and requested side effect before execution.

High-risk actions, including deploy reruns, production mutations, or incident-record writes, require explicit human approval before execution. Show the reviewer the relevant redacted observation and intended change. For audits and debugging, retain action proposals, policy outcomes, approval decisions, and verifier evidence according to a retention policy; don't assume storing every raw screenshot or keystroke is appropriate.

The session manifest should be inspectable before the browser starts. This example admits a read-only CI lookup and rejects a session that requests an unrelated domain or write capability.

admit-gui-session.py

from dataclasses import dataclass

@dataclass(frozen=True)
class SessionManifest:
    domains: tuple[str, ...]
    allowed_effects: tuple[str, ...]
    browser_profile: str

TASK_DOMAINS = {"ci.example", "incidents.example"}
READ_ONLY_EFFECTS = {"browse_url", "read_visible_text", "capture_reference"}

def admit(manifest: SessionManifest) -> str:
    if manifest.browser_profile != "fresh":
        return "blocked: reused browser profile"
    if not set(manifest.domains) <= TASK_DOMAINS:
        return "blocked: unrelated domain"
    if not set(manifest.allowed_effects) <= READ_ONLY_EFFECTS:
        return "blocked: write capability requested"
    return "admitted: read-only session"

safe = SessionManifest(("ci.example",), ("browse_url", "capture_reference"), "fresh")
reused_profile = SessionManifest(("ci.example",), ("browse_url",), "reused")
unrelated_domain = SessionManifest(("ci.example", "mail.example"), ("browse_url",), "fresh")
write_capability = SessionManifest(("ci.example",), ("rerun_deploy",), "fresh")
print(admit(safe))
print(admit(reused_profile))
print(admit(unrelated_domain))
print(admit(write_capability))

Output

admitted: read-only session
blocked: reused browser profile
blocked: unrelated domain
blocked: write capability requested

Audit data must also be minimized. The log needs evidence of the policy decision, not copied private email addresses or secret fields.

redact-gui-audit-event.py

import re

def audit_event(note: str, action: str, decision: str) -> dict[str, str]:
    redacted = re.sub(r"[\w.+-]+@[\w.-]+", "[EMAIL]", note)
    redacted = re.sub(r"password=[^\s]+", "password=[REDACTED]", redacted)
    return {"action": action, "decision": decision, "note": redacted}

event = audit_event(
    "notify [email protected] password=portal-secret",
    action="capture_reference",
    decision="allowed_read_only",
)
print(event["note"])
print("secret logged:", "portal-secret" in str(event))

Output

notify [EMAIL] password=[REDACTED]
secret logged: False

Common pitfalls include reusing a browser profile across unrelated tasks, letting the model alone decide when the task is finished, and running write-enabled sessions without rate limits or a kill switch.

GUI-agent isolation adds browser-profile state, visible credential surfaces, timing-dependent interfaces, and clicks that can mutate external application state. Those risks require controls beyond filesystem and process limits: domain policy, credential brokering, action gating, final-state verification, and cleanup of session state.

Defense-in-depth sandbox for computer-use agents: fresh VM, restricted apps, egress allow-list, action validator, human review, and audit replay each catch different failures. — Sandbox boundaries must be independent. If prompt rules fail, the action validator can still block a step, and if the validator misses something, the VM, network policy, and human review still limit blast radius.

Evaluation of computer-use agents

Final-task accuracy alone isn't enough. Evaluate both functional success and operational safety.

Task success rate asks whether the final observable state of the system matches the goal. WebArena and OSWorld supply programmatic validators for this. Step efficiency measures how many actions and model calls were required; long trajectories burn tokens and raise the probability of eventual failure. Safety violation rate counts how often the agent attempted a blocked action or required human intervention. Grounding accuracy compares proposed click coordinates against ground-truth element locations. For open-ended tasks, blind human raters compare two agent trajectories on the same goal and express preference.

Useful evaluation suites include:

WebArena for realistic browser environments with self-hostable e-commerce, GitLab, and Reddit clones.^{[5]Reference 5WebArena: A Realistic Web Environment for Building Autonomous Agentshttps://arxiv.org/abs/2307.13854}
OSWorld for open-ended desktop tasks across real Ubuntu, Windows, and macOS applications using actual mouse, keyboard, and screenshot interactions.^{[6]Reference 6OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentshttps://arxiv.org/abs/2404.07972}
Mind2Web for large-scale element grounding and multi-step navigation across 137 live websites.^{[7]Reference 7Mind2Web: Towards a Generalist Agent for the Webhttps://arxiv.org/abs/2306.06070}
GAIA for general assistant tasks that combine web browsing, code execution, and file operations.^{[8]Reference 8GAIA: A Benchmark for General AI Assistantshttps://arxiv.org/abs/2311.12983}
AgentDojo as a complementary tool-use security environment for prompt-injection and harmful-action testing.^{[9]Reference 9AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentshttps://arxiv.org/abs/2406.13352}

Public benchmark numbers change as systems and harnesses change. Treat WebArena, OSWorld, and related suites as regression harnesses and failure-mode maps, not as a full production readiness certificate. A score under a documented harness doesn't determine whether your CI dashboard, incident workflow, and reviewer queue satisfy policy.

Failure modes specific to GUI agents

Computer-use agents fail in characteristic ways that pure API agents rarely encounter.

UI drift occurs when an A/B test, font change, or responsive layout moves an element by forty pixels; the model clicks empty space. Visual ambiguity appears when two buttons look identical at the screenshot resolution the model received; it chooses the wrong one because lighting or compression hid the label. Timing and animation problems cause clicks before a menu finishes expanding or before a loading spinner disappears. Captcha and anti-bot systems are deliberately designed to defeat this class of agent and should route to human handling, not automated bypass. Credential leakage happens when the model is asked to log in and types a real password into a visible field that ends up in the audit log. Scope creep occurs when the model decides "while I am here I should also clean up the user's profile" and edits fields that were never part of the original task.

Indirect prompt injection is a central web-agent failure. A computer-use agent reads screen content as input, and a page can contain attacker-written text or images that instruct it to take an unrelated action. AgentDojo studies prompt-injection attacks against tool-using agents.^{[9]Reference 9AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentshttps://arxiv.org/abs/2406.13352} Provider guidance likewise places responsibility on the host: Anthropic recommends isolated environments and confirmation for consequential actions, OpenAI documents confirmation and isolation practices, and Gemini can return a per-step safety_decision requiring confirmation.^{[2]Reference 2Computer Use Toolhttps://docs.anthropic.com/en/docs/agents-and-tools/tool-use/computer-use-tool}^{[3]Reference 3Computer Usehttps://developers.openai.com/api/docs/guides/tools-computer-use}^{[4]Reference 4Computer Usehttps://ai.google.dev/gemini-api/docs/computer-use} Treat page content as untrusted evidence, not authority for writes or production mutations.

Text banners are only one visual channel. Also assume:

Spoofed system chrome: a page or screenshot paints a fake OS or browser "Allow" dialog; host policy must ignore on-screen approve affordances and only honor your out-of-band approval UI.
OCR / accessibility-tree poison: a visible label looks benign while the a11y name or OCR string carries instructions.
Pixel-level lies: compression, stego-like cues, or adversarial patterns at screenshot resolution; pixels can mislead the model even when DOM text looks clean.

Host policy owns approval. Never treat a painted button, dialog, or OCR string as an authorization event.

A policy can carry the observation taint of untrusted screen content into the next proposed action (the same first-class property named for non-GUI tool results in ReAct architectures). Here, seeing a suspicious dashboard banner doesn't prevent continued reading, but it blocks a deploy-rerun proposal until a reviewer approves it.

gate-actions-after-page-injection.py

from dataclasses import dataclass

@dataclass(frozen=True)
class Observation:
    contains_untrusted_instruction: bool

@dataclass(frozen=True)
class Proposal:
    action: str
    mutates_production: bool = False

def authorize(observation: Observation, proposal: Proposal) -> str:
    if observation.contains_untrusted_instruction and proposal.mutates_production:
        return "blocked: untrusted page cannot authorize deploy action"
    return "allowed: read-only continuation"

banner = Observation(contains_untrusted_instruction=True)
print(authorize(banner, Proposal("read_failed_step")))
print(authorize(banner, Proposal("rerun_deploy", mutates_production=True)))

Output

allowed: read-only continuation
blocked: untrusted page cannot authorize deploy action

Treat each computer-use run as a security-sensitive operation. Retain enough redacted evidence to explain policy outcomes and support incident response, and maintain an operator stop control for active sessions.

Production patterns that work

A useful production pattern is hybrid, not pure computer use.

Use ordinary tool calling, Model Context Protocol (MCP) servers, direct APIs, or deterministic browser automation whenever they satisfy the authorized workflow. Invoke computer use only for the narrow portion that requires visual reasoning or legacy GUI access. Bound sessions by step count and wall-clock time. Require approval for actions whose blast radius includes secrets, incident state, or production infrastructure.

Use the same engineering discipline as for AI coding agents. Speed and generality matter only when they remain inside explicit boundaries and human accountability.

Make that escalation rule explicit in routing rather than leaving it to a model's preference.

choose-narrowest-control-surface.py

from dataclasses import dataclass

@dataclass(frozen=True)
class Workflow:
    name: str
    api_available: bool
    stable_selector_flow: bool
    visual_evidence_required: bool

def control_surface(workflow: Workflow) -> str:
    if workflow.api_available:
        return "direct API"
    if workflow.stable_selector_flow and not workflow.visual_evidence_required:
        return "deterministic browser automation"
    return "computer use with host policy"

workflows = [
    Workflow("fetch run status", True, False, False),
    Workflow("download known artifact", False, True, False),
    Workflow("inspect failed test screenshot", False, False, True),
]
for workflow in workflows:
    print(workflow.name, "->", control_surface(workflow))

Output

fetch run status -> direct API
download known artifact -> deterministic browser automation
inspect failed test screenshot -> computer use with host policy

A minimal host executor sketch

Start with the control logic before connecting a real browser driver. The model's proposals are fixture data; the host policy refuses an unapproved write and accepts completion only after observable state matches the read-only goal.

host-owned-gui-loop.py

from dataclasses import dataclass

@dataclass(frozen=True)
class Proposal:
    action: str
    writes_record: bool = False

proposals = [
    Proposal("open_run_page"),
    Proposal("capture_traceback_reference"),
    Proposal("update_incident_status", writes_record=True),
]
evidence_collected = False

for step, proposal in enumerate(proposals, start=1):
    if proposal.writes_record:
        print(f"step {step}: paused for approval before {proposal.action}")
        break
    print(f"step {step}: executed {proposal.action}")
    evidence_collected = evidence_collected or proposal.action == "capture_traceback_reference"

print("read-only goal verified:", evidence_collected)
print("session teardown: requested")

Output

step 1: executed open_run_page
step 2: executed capture_traceback_reference
step 3: paused for approval before update_incident_status
read-only goal verified: True
session teardown: requested

The host, not the model, owns the stop condition, the policy checks, the human gate, the audit log, and the final verification.

Risk classification table for CI dashboard actions

In the CI dashboard and incident-management example, not every action carries the same blast radius. A production policy engine classifies proposed actions into risk tiers before any execution happens.

Risk Tier	Example Actions in CI Dashboard	Example Actions in Incident UI	Human Gate?	Typical Policy Response
Low	View run page, scroll, read failing-step metadata	View incident record, read prior notes	No	Execute immediately, log only
Medium	Expand log panel, copy artifact reference	Stage a draft note without submitting it	Policy-dependent	Permit only if retention and data policy allow it
High	Download full logs, open deploy controls, rerun production job	Submit note, update severity, page an owner, close incident	Yes	Pause, present redacted observation + intended change, wait for approval

The table makes the defense-in-depth concrete. Low-risk navigation and reading stay fast. High-risk production or incident mutation steps pause for a human who can see the model's observation and the proposed state change.

This tiny policy router makes that escalation rule concrete:

risk-router.py

from dataclasses import dataclass

@dataclass(frozen=True)
class ActionProposal:
    target: str
    kind: str

LOW_RISK_ACTIONS = {"scroll", "view_run", "read_incident"}
MEDIUM_RISK_ACTIONS = {"expand_log", "copy_reference", "prepare_note"}
HIGH_RISK_ACTIONS = {"download_logs", "rerun_deploy", "submit_incident", "update_severity"}

def classify(proposal: ActionProposal) -> tuple[str, bool, str]:
    if proposal.kind in HIGH_RISK_ACTIONS:
        return "high", True, "pause"
    if proposal.kind in MEDIUM_RISK_ACTIONS:
        return "medium", False, "policy_check"
    if proposal.kind in LOW_RISK_ACTIONS:
        return "low", False, "execute"
    return "blocked", False, "unknown_action"

samples = [
    ActionProposal(target="run_page", kind="scroll"),
    ActionProposal(target="log_panel", kind="copy_reference"),
    ActionProposal(target="deploy_controls", kind="rerun_deploy"),
    ActionProposal(target="run_page", kind="erase_run"),
]

for sample in samples:
    tier, needs_human, decision = classify(sample)
    print(f"{sample.target}: tier={tier}, human_gate={needs_human}, decision={decision}")

Output

run_page: tier=low, human_gate=False, decision=execute
log_panel: tier=medium, human_gate=False, decision=policy_check
deploy_controls: tier=high, human_gate=True, decision=pause
run_page: tier=blocked, human_gate=False, decision=unknown_action

Approval is necessary but not sufficient. After an approved write, the host must check both the intended record and the absence of unrelated changes.

verify-scoped-record-update.py

before = {
    "INC-2048": "traceback_collected",
    "INC-9911": "open",
}
expected_after = {
    "INC-2048": "review_approved",
    "INC-9911": "open",
}

def verify(after: dict[str, str]) -> str:
    return "verified" if after == expected_after else "blocked: unexpected final state"

approved_write = {"INC-2048": "review_approved", "INC-9911": "open"}
wrong_scope = {"INC-2048": "review_approved", "INC-9911": "closed"}

print("approved write:", verify(approved_write))
print("wrong scope:", verify(wrong_scope))

Output

approved write: verified
wrong scope: blocked: unexpected final state

Cost and latency realities

A direct API request can often answer a structured lookup in one operation. A computer-use trajectory may require multiple observations, model calls, action executions, retries, and possibly human review. Measure these costs on your own workload and selected model rather than carrying over generic per-task estimates.

Reserve computer use for cases that require it, such as permitted visual evidence inspection or a legacy CI dashboard without a supported integration. Route structured lookups and authorized writes through narrower typed surfaces when available.

How to start a minimal pilot with one CI dashboard

Begin with a controlled portal workflow and limit initial scope to read-only evidence lookup. Expand to an approved write only after the team defines acceptance criteria, measures failures, red-teams page injection, and confirms reviewer and rollback procedures.

Run each session in a fresh restricted environment appropriate to its risk. Retain redacted action evidence, policy outcomes, and final host-verification results according to privacy and incident-response requirements. Review failures with the owning engineering team before expanding the action space or adding another workflow.

Common implementation gotchas with Playwright inside VMs

When a browser driver supplies observations inside a restricted runtime, keep viewport size, device scale, fonts, and browser extensions deterministic for the evaluation or workflow configuration. Don't rely on a generic networkidle event as proof that a dynamic application is ready; wait for task-specific visible state and re-observe after each action. Layout drift can otherwise move a target between proposal and execution.

Trajectories as future training data

A verified computer-use session can be useful evaluation or improvement data when retention, privacy, and authorization permit it. Store only the data required for the stated purpose, with approval decisions and final-verifier results attached; a completed session isn't automatically a representative training demonstration.

Practice: design a safe CI failure inspector

Your platform team needs an agent that can open the legacy CI dashboard for a given run ID, locate the failed test step, capture the relevant traceback and artifact reference, update the incident note, and route the investigation to the owning team.

The CI dashboard has no API and lacks reliable selector-based automation.

Complete these tasks:

Write the exact policy rules the action validator must enforce before any mouse movement is allowed. Cover navigation limits, credential handling, and which UI elements are off-limits.
Decide which actions require human approval and what evidence (screenshot, proposed diff, run ID, incident ID) the reviewer must see before approving.
Specify the acceptance criteria the host must check after the agent claims "done". The criteria should include incident note changed correctly, note contains the failing-step reference, and no unrelated incidents or deploys were touched.
List three failure modes you realistically expect in the first week of production and the concrete guardrails that would catch each one.
Produce a one-page reviewer checklist that a human can use consistently before approving a write.

Write the policy and checklist as documents another engineer could hand to an incident reviewer on day one.

Computer-use agents propose actions on visible interfaces that can expose privileged state. Treat that ability like delegated access: least privilege, redacted evidence, restricted sessions, host verification, and approval for irreversible effects.

Next Step

Continue to Human-in-the-Loop Agent Architecture

Computer-use tasks that reach a policy boundary need explicit approval checkpoints and durable evidence across steps. Human-in-the-loop architecture makes those supervision and approval patterns durable across a longer workflow.

PreviousCode Generation & Sandboxing

Share this article