Computer-Use / GUI / Browser Agents

A computer-use agent proposes mouse and keyboard actions from observations of a graphical interface. The model chooses a next step; a host application decides whether to execute it. Consider a fictional deploy run, RUN-78432, that failed after the test stage. Its legacy CI portal shows the failing step and a redacted traceback in a canvas widget, without a supported API for that evidence. An agent could inspect the permitted session and prepare a note for a reviewer.

That sounds safe until the same session can also click a deploy control or edit an incident. How can an agent inspect a visual page without turning a useful read into an unauthorized write?

Code generation and sandboxing kept generated commands behind a host that admits the job, runs it in a sandbox, and scores the result with a trusted validator. Computer-use and graphical user interface (GUI) agents keep that boundary, but change the proposal. Instead of a patch, the model asks to click, type, scroll, or take another screenshot.

You don't need image-encoder internals to follow the design. The agent grounds proposals from pixels, page text, and accessibility metadata. The host still owns credentials, policy, and every write.

If a stable API or selector-based automation already finishes the workflow, use that first. Computer use earns a seat only when the approved task depends on visual evidence or a legacy interface that deterministic automation can't address reliably. Even then, the model proposes steps. A host process controls credentials, action allowlists, and record mutation.

Computer-use tools let models receive screenshots, and sometimes browser state or accessibility metadata, then propose input events such as moving a virtual mouse, clicking, typing a run ID, scrolling, or requesting another screenshot.^{[1]Reference 1Introducing computer use, a new Claude 3.5 Sonnet capabilityhttps://www.anthropic.com/news/3-5-models-and-computer-use} The host decides whether and where each event executes.

That reach covers permitted legacy tools and visual portal steps with no suitable API. It also creates a new failure surface: a mis-grounded click, an overbroad session, or a leaked credential can change external state. A browser session that uses its broader access to perform an action the initiating user can't authorize becomes a confused deputy. Computer use is delegated access, not a screenshot convenience wrapper.

From curated tools to full computer control

The incident exposes the difference between two action surfaces. Ordinary function calling gives a model a curated set of typed operations that the developer can separately authorize. Typed doesn't automatically mean safe, but it keeps the set of possible requests narrow.

Computer use starts from a screenshot and may add an accessibility tree or optical character recognition (OCR) output. The model then proposes primitive input events. A computer-use agent can, in principle, open any application visible on the virtual desktop, click any pixel, and type into any field.

That broader reach helps with workflows that have no API. It also moves the main safety question from "which function names did we expose?" to "which visible state may this session reach, and who can authorize a side effect?" Every deployment therefore needs a permission boundary with several independent layers of control.

The observe-act loop

Think of computer use as a short loop: the host shows the current screen, the model proposes an action or ordered batch, and the host checks each step before dispatch. Then the host returns a fresh observation and the execution result.

That contract is an agent-computer interface (ACI): a bounded action vocabulary, an observation format, and host guardrails that accept or reject the next step. The loop ends when the host verifies completion, a limit expires, or a safety boundary intervenes. Every extra observation and model decision adds latency and model usage, so trajectories need bounds.

Now apply the cycle to RUN-78432. The host brokers a restricted session, the model proposes one GUI step at a time, and the task ends at a local draft. Policy, approval, audit evidence, and teardown stay outside the model.

View larger

The CI dashboard runs inside a restricted browser session because the task depends on visual evidence rather than an available API. A policy engine sits between the model's proposal and execution.

If the task scope allows a write, it still needs separate authorization and a reviewer who can see the relevant observation and intended change. Out-of-scope actions remain rejected. Escalation requires its own permission-granting process. Audit records capture policy evidence without copying secrets or more log data than retention policy permits.

A host can make that split concrete before integrating a model. The policy below represents host-defined semantic operations, not labels we trust a model to attach to arbitrary clicks. A raw click needs a known target and current state before the host can classify its effect. If that mapping isn't reliable, restrict the account to read-only permissions or stop. Its pause result is only a routing decision, not permission to resume; task scope must permit a write before reviewer approval can authorize it.

from dataclasses import dataclass

@dataclass(frozen=True)
class Action:
    domain: str
    kind: str
    target: str

ALLOWED_DOMAINS = {"ci.example", "incidents.example"}
READ_ACTIONS = {"view_log", "open_run", "scroll", "capture_reference"}
WRITE_ACTIONS = {"update_incident", "rerun_deploy", "submit_form"}

def route(action: Action) -> str:
    if action.domain not in ALLOWED_DOMAINS:
        return "blocked: domain outside task scope"
    expected = "RUN-78432" if action.domain == "ci.example" else "INC-2048"
    if action.target != expected:
        return "blocked: record outside task scope"
    if action.kind in READ_ACTIONS:
        return "execute read-only step"
    if action.kind in WRITE_ACTIONS:
        return "pause: reviewer approval required"
    return "blocked: unknown action kind"

for proposal in [
    Action("ci.example", "view_log", "RUN-78432"),
    Action("incidents.example", "update_incident", "INC-2048"),
    Action("ci.example", "erase_run", "RUN-78432"),
    Action("mail.example", "open", "inbox"),
]:
    print(route(proposal))

execute read-only step
pause: reviewer approval required
blocked: unknown action kind
blocked: domain outside task scope

A concrete five-step trace for RUN-78432

Trace the CI failure inspection from the first browser state to the verified handoff. Each step keeps the model's proposal separate from the host's decision.

Step 1. The host opens a fresh restricted browser profile and establishes an authorized CI session through a credential broker. The model doesn't receive the password or propose typing it.

Once authenticated, the host captures the permitted dashboard view and relevant grounding metadata. The model starts with a bounded observation, not ambient access.

Step 2. After the login succeeds, the host captures the next screenshot showing the run search form. The model outputs type("RUN-78432") for the run field and a left_click on the search button.

Step 3. Results load with failing test output visible. The model proposes clicking the Failed tests control by role and name, not by a remembered coordinate.

The host resolves a single match, expands the failed step, and scrolls until the traceback is in view. If two tabs shared that name, the host would re-observe or escalate instead of guessing.

Step 4. The model reads visible evidence and proposes preparing a draft incident note with the failing command, traceback line, and artifact link. The host policy permits extracting that evidence.

It doesn't allow the model to rerun deploys or edit incident severity from the page alone. The same page can be useful evidence without becoming authority for a write.

Step 5. The read-only task ends with a local draft. If the user separately authorizes an incident update, the host can present that exact change to a reviewer. Approval must bind the actor, session, incident, content, expected record version, and expiry. A changed target or stale record requires a fresh decision, not replaying an old approval.

After an authorized write, the host verifies the intended record and checks available audit evidence for unintended effects. The bounded verifier must inspect every record in scope before reporting completion; a model message alone cannot establish the result.

The trace below is proposed. No browser execution was recorded. Navigation and evidence collection stay read-only. An incident write requires separate authorization, and a deploy rerun remains outside this task.

The audit record stores action proposals, policy outcomes, relevant observation identifiers, approval, and final verification. It excludes secrets and unnecessary raw log content.

Authentication belongs in a host-owned setup step, not in text the model should type. Return only an allowlisted session outcome to the model. This fixture tests that output boundary; it doesn't log into a portal or demonstrate a real credential broker.

def public_session_result(broker_result: dict[str, str]) -> dict[str, str]:
    if broker_result.get("status") != "authenticated":
        raise PermissionError("host authentication failed")
    return {"session": "authenticated"}

model_context = {"task": "inspect CI failure evidence", "run_id": "RUN-78432"}
broker_fixture = {"status": "authenticated", "credential": "FAKE-SECRET"}
session = public_session_result(broker_fixture)
assert "FAKE-SECRET" not in str((model_context, session))
try:
    public_session_result({"status": "failed"})
except PermissionError:
    pass
else:
    raise AssertionError("failed authentication accepted")

print("model fields:", sorted(model_context))
print("session:", session["session"])
print("credential field exposed:", "credential" in session)

model fields: ['run_id', 'task']
session: authenticated
credential field exposed: False

Grounding pixels to intent

Grounding connects a proposal to a target in the observed state. An API agent also needs grounding: it must choose the right record, arguments, and current resource. A typed schema constrains request shape, not those choices. A direct lookup such as GET /api/v1/runs/RUN-78432 avoids pointer positioning; it can still use a wrong run ID, stale data, or insufficient authority.

A computer-use agent faces the action-perception gap. Its observation might be a screenshot or accessibility tree, while its input vocabulary includes pointer movement, clicks, typing, and scrolling. The host translates those proposals through a browser protocol or desktop input channel. Connecting "expand the failed test traceback for RUN-78432" to the right target is difficult when elements shift, animate, and reflow.

Start with the tempting shortcut: click the coordinate that worked yesterday. Pure coordinate clicking is brittle. An A/B test can move the Failed tests tab twenty pixels down. A different viewport can collapse the sidebar, and a narrower window can trigger a responsive layout. Yesterday's (x, y) is now empty space, even though the page still looks plausible to a human.

The coordinate transformation pipeline

Start with the selected provider's coordinate contract. Some tools use screenshot pixels; others use normalized integers. The host must preserve the observation's dimensions, origin, scaling, and tab/display identity before translating a valid point into input coordinates.

For example, Gemini's current action tables specify integer coordinates from 0 through 999 on a 1000-unit scale.^{[2]Reference 2Computer Usehttps://ai.google.dev/gemini-api/docs/computer-use} For an unresized full-screen observation with native width $W$ and height $H$, convert a valid normalized point as follows:

A result of 1000 violates this tool contract; reject it rather than silently snapping it to another target. With positive dimensions and valid 0–999 inputs, the formula stays inside the image. Normalization doesn't make perception resolution-invariant: text readability and target visibility still change. Other tool versions may define different endpoints.

When screenshots are resized, cropped, or letterboxed, coordinate scaling requires an explicit multi-step pipeline:

1. Letterbox padding removal: If an image was padded with black or neutral bars $(p_x, p_y)$ to fit the vision encoder's aspect ratio, subtract the padding offsets: $x' = x_{\text{model}} - p_x$, $y' = y_{\text{model}} - p_y$.
2. Crop scale inversion: Invert the downsampling ratio to return to physical crop dimensions: $x'' = \lfloor x' \cdot (W_{\text{crop}} / W_{\text{rendered}}) \rfloor$.
3. Crop origin restoration: Add the crop's top-left origin $(x_{\text{crop}}, y_{\text{crop}})$ to place the coordinate in the native framebuffer: $x_{\text{native}} = x'' + x_{\text{crop}}$.
4. Input-space conversion: Playwright's mouse API accepts main-frame viewport CSS pixels.^{[3]Reference 3Mousehttps://playwright.dev/docs/api/class-mouse} Divide by device pixel ratio (DPR) only when your screenshot uses device pixels. A screenshot captured with scale: "css" already has one image pixel per CSS pixel.^{[4]Reference 4Page: screenshot scaling, file inputs, and readinesshttps://playwright.dev/docs/api/class-page} A full-desktop capture also needs the browser viewport's offset removed; a full-page capture needs the current scroll offset. DPR alone doesn't account for browser chrome, page zoom, or a different origin.

View larger

Set-of-Marks: discrete visual grounding

Set-of-Mark (SoM) prompting labels image regions so the model can refer to them discretely. The original work used segmentation regions and marks such as numbers, masks, and boxes for visual-grounding tasks; it wasn't a universal GUI executor.^{[5]Reference 5Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vhttps://arxiv.org/abs/2310.11441} A GUI adaptation can label candidate controls from DOM boxes, accessibility nodes, or visual detections.

Instead of guessing continuous $(x, y)$ floats, the model emits a discrete token: click(mark_1). The host maps that ID back to the element's centroid or bounding box.

The model chooses an ID instead of estimating the click point. The host still needs a correct mark map, current geometry, coordinate conversion, and actionability checks. SoM changes the targeting problem; it doesn't eliminate spatial errors.

• Preprocessing work: Detection, tree extraction, and badge rendering add work; measure their latency in the actual harness rather than assuming a fixed range.
• Visual occlusion: A badge drawn over a tiny icon or short number can obscure the very text the model needs to read.
• Dynamic staleness: Scrolling, animation, or rerendering can change a marked target's identity or geometry. Bind mappings to the tab and observation, re-resolve the intended element before dispatch, and reject maps whose relevant state has changed. Unrelated DOM changes don't necessarily invalidate a stable element reference.

Representation trade-offs: DOM, accessibility tree, and pixels

Production GUI agents balance three observation modalities, each with distinct trade-offs:

1. DOM representation: Exposes structure, attributes, and text, with size depending on serialization and subtree filtering. Hidden text and scripts can add irrelevant or hostile content. DOM text alone doesn't describe painted canvas/WebGL content or native windows, although a page can provide accessible fallback content. More tokens lengthen context; they don't by themselves establish quadratic runtime for every model.
2. Accessibility tree: Supplies semantic roles, names, and states when the application exposes them. Focused trees can omit much markup, but there is no universal token-reduction percentage. Poor labeling or missing semantic annotations can leave controls ambiguous or absent. A visually plain <div> may still have a useful role and name.
3. Screenshots: Capture visible canvas, browser, and desktop content without requiring DOM semantics. Reading small text, recognizing controls, and distinguishing spoofed chrome remain difficult. Cost and preparation/inference latency depend on resolution, model, preprocessing, caching, and provider accounting; no fixed vision-encoder timing applies across systems.

A hybrid observation pipeline can combine these signals: use a focused accessibility tree for semantic targeting, a screenshot for spatial context, and a higher-resolution crop or mark overlay when the target remains unclear. Downsampling can hide small text, so choose representations based on the information the task needs rather than following a fixed escalation order.

Grounding metadata helps only when it selects one target in the intended frame and record. Two identical names may become unique after scoping to the correct run panel. Uniqueness still isn't enough: Playwright checks visibility, stability, event reception, and enabled state before a normal locator click.^{[6]Reference 6Auto-waitinghttps://playwright.dev/docs/actionability} Those checks improve interaction reliability, not authorization. This fixture resolves boxes and coordinates; it doesn't implement browser hit-testing.

from dataclasses import dataclass
from math import isfinite

@dataclass(frozen=True)
class GroundedElement:
    mark_id: int
    role: str
    name: str
    box: tuple[int, int, int, int]  # x, y, w, h

def normalized_to_native(point: tuple[int, int], native_res: tuple[int, int]) -> tuple[int, int]:
    nx, ny = point
    nw, nh = native_res
    if any(type(v) is not int for v in (*point, *native_res)):
        raise ValueError("coordinates and dimensions must be integers")
    if min(nw, nh) <= 0 or not (0 <= nx <= 999 and 0 <= ny <= 999):
        raise ValueError("positive dimensions and normalized coordinates in [0, 999] required")
    return nx * nw // 1000, ny * nh // 1000

def scale_point(
    point: tuple[int, int],
    source_res: tuple[int, int],
    target_res: tuple[int, int],
) -> tuple[int, int]:
    sx, sy = point
    sw, sh = source_res
    tw, th = target_res
    if any(type(v) is not int for v in (*point, *source_res, *target_res)):
        raise ValueError("coordinates and dimensions must be integers")
    if min(sw, sh, tw, th) <= 0 or not (0 <= sx < sw and 0 <= sy < sh):
        raise ValueError("invalid dimensions or source point")
    # Floor mapping avoids rounding the final source pixel outside the target.
    return sx * tw // sw, sy * th // sh

def cropped_to_css(point, crop, rendered_size, padding, dpr):
    x, y = point
    ox, oy, cw, ch = crop
    rw, rh = rendered_size
    px, py = padding
    values = (*point, *crop, *rendered_size, *padding, dpr)
    if not all(type(v) in (int, float) and isfinite(v) for v in values):
        raise ValueError("nonfinite transform")
    if min(cw, ch, rw, rh, dpr) <= 0 or min(ox, oy, px, py) < 0:
        raise ValueError("invalid transform geometry")
    if not (px <= x < px + rw and py <= y < py + rh):
        raise ValueError("point lies outside rendered crop")
    return ((ox + (x - px) * cw / rw) / dpr,
            (oy + (y - py) * ch / rh) / dpr)

def resolve_by_mark(elements: list[GroundedElement], mark_id: int) -> tuple[int, int]:
    if type(mark_id) is not int or mark_id <= 0:
        raise ValueError("mark ID must be a positive integer")
    matches = [e for e in elements if e.mark_id == mark_id]
    if len(matches) != 1:
        raise ValueError(f"expected 1 match for mark {mark_id}, found {len(matches)}")
    x, y, w, h = matches[0].box
    if any(type(v) is not int for v in (x, y, w, h)) or min(x, y) < 0 or min(w, h) <= 0:
        raise ValueError("invalid target box")
    return x + w // 2, y + h // 2

def resolve_by_role_name(elements: list[GroundedElement], role: str, name: str) -> str:
    matches = [e for e in elements if e.role == role and e.name == name]
    if len(matches) != 1:
        return f"blocked: expected one target, found {len(matches)}"
    return f"click mark {matches[0].mark_id}"

page = [
    GroundedElement(1, "tab", "Failed tests", (380, 160, 120, 40)),
    GroundedElement(2, "tab", "Failed tests", (800, 160, 120, 40)),
]

print("scaled coordinate:", scale_point((190, 80), (512, 384), (1024, 768)))
print("normalized (999, 999):", normalized_to_native((999, 999), (1024, 768)))
print("mark 1 center:", resolve_by_mark(page, mark_id=1))
print("unique role match:", resolve_by_role_name(page[:1], "tab", "Failed tests"))
print("ambiguous role match:", resolve_by_role_name(page, "tab", "Failed tests"))
transform = ((100, 50, 640, 400), (320, 200), (32, 16), 2)
assert cropped_to_css((132, 66), *transform) == (150.0, 75.0)
assert scale_point((511, 383), (512, 384), (1024, 768)) == (1022, 766)
assert normalized_to_native((999, 999), (1024, 768)) == (1022, 767)
assert normalized_to_native((500, 500), (1024, 768)) == (512, 384)
for bad_point, bad_res in [((1000, 0), (1024, 768)), ((True, 0), (1024, 768)),
                           ((0, 0), (0, 768)), ((0, 0), (-1, 768)),
                           ((0.5, 0), (1024, 768))]:
    try:
        normalized_to_native(bad_point, bad_res)
    except ValueError:
        pass
    else:
        raise AssertionError("invalid normalized geometry accepted")
for bad_point in [(31, 66), (352, 66), (float("nan"), 66)]:
    try:
        cropped_to_css(bad_point, *transform)
    except ValueError:
        pass
    else:
        raise AssertionError("invalid point accepted")
print("cropped model point to CSS:", cropped_to_css((132, 66), *transform))

scaled coordinate: (380, 160)
normalized (999, 999): (1022, 767)
mark 1 center: (440, 180)
unique role match: click mark 1
ambiguous role match: blocked: expected one target, found 2
cropped model point to CSS: (150.0, 75.0)

How providers expose computer use

Provider contracts change independently. These details were checked October 10, 2026; pin a supported model, tool schema, platform, and adapter. Anthropic's standard client toolset is computer_toolset_20260801, available without a beta header on the Claude API and Google Cloud. One tools entry declares it:

{
  "type": "computer_toolset_20260801",
  "configs": {
    "zoom": { "enabled": false }
  }
}

The entry has no name or legacy display-dimension fields. configs can disable members your environment doesn't implement; all 17 are enabled by default, including zoom.^{[7]Reference 7Computer Use Toolhttps://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool}

Calls name a member such as left_click, type, or screenshot and carry "toolset_name": "computer"; member input has no action field. Dispatch on both toolset and member name. Each tool_result must match tool_use_id and echo "toolset_name": "computer"; omission or a different toolset name is rejected. Return a result for every call, including errors for skipped members after a batch failure. Coordinates remain in full-screenshot pixel space after zoom, so don't apply a crop offset merely because the last image was zoomed.^{[7]Reference 7Computer Use Toolhttps://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool}

Older model/platform integrations may still require computer_20251124 and a beta header. Check compatibility rather than mixing versions. Anthropic's screenshot classifiers add injection detection, not application permission.^{[7]Reference 7Computer Use Toolhttps://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool}

OpenAI's current Responses guide recommends computer use through generated code in your own execution service, with a structured computer tool as an alternative. On that structured path, a computer_call contains an ordered actions[]; return the updated screenshot in a computer_call_output matching its call_id. The call's status: "completed" means generation finished, not that a click executed. Continuing an API response doesn't restore a lost browser session.^{[8]Reference 8Computer Usehttps://developers.openai.com/api/docs/guides/tools-computer-use}

Generated code can perform more than the declared mouse primitive. Isolate that executor and its reachable capabilities; a router that recognizes click labels can't by itself authorize arbitrary Python or JavaScript. OpenAI treats screen instructions as untrusted and notes that typing sensitive data into a form is already transmission, even before Submit.^{[8]Reference 8Computer Usehttps://developers.openai.com/api/docs/guides/tools-computer-use}

Gemini's current examples use the Interactions API, a computer_use tool with an environment, and UI function calls. Gemini 3.x documents browser, desktop, and mobile environments, alongside a legacy Gemini 2.5 browser-oriented contract. Current coordinate tables use integers 0–999. Don't combine action names, endpoints, or result shapes from different generations.^{[2]Reference 2Computer Usehttps://ai.google.dev/gemini-api/docs/computer-use}

Gemini can return safety_decision: require_confirmation; obtain actual confirmation before execution and include the corresponding safety_acknowledgement in the function result. Its screenshot injection detector, documented for Gemini 3.5 Flash-Lite and later, is opt-in, default false. Overrides don't guarantee that confirmation requests disappear. Neither detection nor acknowledgement replaces application authorization.^{[2]Reference 2Computer Usehttps://ai.google.dev/gemini-api/docs/computer-use}

Across these client-executed patterns, the developer owns the execution environment. Provider safety signals can add mandatory confirmation steps; honor those protocol requirements as well as your own policy. They don't execute clicks, mint credentials, or mark INC-2048 updated.

Batching changes transport, not authority. Execute actions in order, rechecking state-dependent preconditions at each step. Stop at an unknown action, failure, or authorization boundary and report remaining actions as unexecuted according to the provider protocol. A paused batch needs fresh state before resuming. The routing fixture below considers a separately authorized write workflow; it doesn't extend the read-only pilot. Its action names are host-classified operations, so a production adapter must map raw clicks to that policy rather than trust a proposed label.

def authorize_batch(actions: list[str]) -> list[str]:
    decisions: list[str] = []
    for action in actions:
        if action in {"rerun_deploy", "update_incident"}:
            decisions.append(f"pause before {action}")
            break
        if action not in {"capture_traceback", "scroll"}:
            decisions.append(f"blocked unknown action: {action}")
            break
        decisions.append(f"execute {action}")
    return decisions

print("\n".join(authorize_batch(["capture_traceback", "rerun_deploy", "scroll"])))
assert authorize_batch(["erase_run", "scroll"]) == ["blocked unknown action: erase_run"]

execute capture_traceback
pause before rerun_deploy

Browser agents versus full desktop agents

Many production workloads stay inside the browser, so start with lighter options. Use Playwright or Selenium directly when an approved target exposes stable selectors and deterministic automation satisfies the workflow.

Browser-agent harnesses and evaluation environments can add observation/action adapters for model-driven experiments. Prefer the narrowest control surface that meets the task rather than moving to pixel-level control by default.

Anthropic now exposes a separate browser_toolset_20260801: the host runs calls against its own browser, with page reads, element references, and screenshots. References are scoped to a tab and can become stale after navigation or material DOM changes. The host owns their mapping; an API can't certify that a reference still resolves correctly.^{[9]Reference 9Browser use toolhttps://platform.claude.com/docs/en/agents-and-tools/tool-use/browser-use-tool} This is a model-directed browser adapter, distinct from a full-desktop toolset or a fixed Playwright script.

Consider full computer-use primitives only when an authorized task requires visual interpretation, browser UI interaction, or a native desktop application and no narrower supported integration is suitable. Don't use computer use to evade a site's access controls or anti-bot policy.

Full desktop control may be appropriate when a task spans a browser and a native application with no suitable narrower integration. An input event can create an irreversible external effect, so "revocable" access means stopping future actions and withdrawing credentials where supported; it doesn't undo every past click.

View larger

Systems runtime: Playwright, CDP, virtual displays, and containers

Choose runtime infrastructure for the intended observation and input channel. A browser-only agent doesn't automatically need an X11 desktop; a native-app agent may need a virtual display and window management.

Headless versus headful under virtual displays

Current Chrome has unified headless and headful implementations; the old headless shell is a separate binary.^{[10]Reference 10Chrome Headless modehttps://developer.chrome.com/docs/automation-and-testing/headless} A headless browser can render screenshots for a visual agent. Validate browser mode, automation flags, fonts, GPU availability, viewport, and site behavior in your harness. navigator.webdriver signals automation in some configurations; inspect browser mode separately to identify headless operation. Headful mode still requires the site's normal access controls.

For a Linux X11 desktop, X Virtual Framebuffer (Xvfb) provides a display without a physical monitor. The launch sketch below assumes a dedicated isolated runtime, an existing host-created Xauthority file, an existing VNC password file, and supervision for process startup and teardown. It doesn't create those secrets or a complete authenticated viewing service. No browser is launched here.

export XAUTHORITY=/run/gui/Xauthority
Xvfb :99 -screen 0 1280x800x24 -nolisten tcp -auth "$XAUTHORITY" &
export DISPLAY=:99

# Observer channel: view-only at the VNC server, loopback-bound listeners
x11vnc -display :99 -auth "$XAUTHORITY" -localhost -viewonly \
  -rfbauth /run/gui/vnc.pass -rfbport 5900 &
websockify --web /usr/share/novnc 127.0.0.1:6080 localhost:5900 &

Avoid Xserver's -ac, which disables access control.^{[11]Reference 11Xserver(1): access control and authorization optionshttps://xorg.freedesktop.org/archive/X11R7.5/doc/man/man1/Xserver.1.html} VNC viewing is interactive by default: x11vnc's server-side -viewonly rejects viewer input.^{[12]Reference 12x11vnc options: server-side view-only and authenticationhttps://github.com/LibVNC/x11vnc/blob/master/doc/OPTIONS.md} Loopback binding doesn't authenticate a remote viewing gateway. Any exposed gateway needs its own authentication, encrypted transport, and access policy; websockify offers explicit authentication and TLS configuration.^{[13]Reference 13websockify: WebSocket-to-TCP proxy and security configurationhttps://github.com/novnc/websockify}

The useful properties are controlled display geometry and an observable shared screen. Matching fonts, browser version, scaling, and graphics configuration improves repeatability; rendering and UI timing can still vary from a physical workstation. Screenshot capture records one moment, while the next input event may observe a changed state.

Actuation mechanics: CDP versus OS-level input injection

Executing a mouse click or keystroke requires choosing the appropriate input injection channel:

• Browser protocol input: Chromium's CDP offers mouse and keyboard dispatch. Playwright also supports Firefox and WebKit through other browser-specific channels; CDP isn't its universal transport. Browser mouse coordinates are viewport CSS pixels.^{[3]Reference 3Mousehttps://playwright.dev/docs/api/class-mouse} Browser input doesn't control arbitrary native windows. A file upload often has a narrower setInputFiles/file-chooser integration instead of an OS click.^{[4]Reference 4Page: screenshot scaling, file inputs, and readinesshttps://playwright.dev/docs/api/class-page}
• Desktop input: X11 tools, compositor-specific mechanisms, or permitted OS APIs can control native applications. Their coordinate and permission rules differ. Verify the actual display/window identity and keyboard focus before dispatch; activating a window once doesn't prevent later focus races. Give the agent an isolated desktop instead of the user's active workspace.

Sandboxing and safety boundaries

The prompt isn't the boundary. The environment in which approved actions run is.

For a privileged engineering workflow, begin each task in a fresh restricted browser profile or disposable runtime appropriate to the threat model. Expose only the applications and domains needed for the task.

Don't expose personal email, password managers, general-purpose terminals, or package installation to a dashboard-inspection session. A read-only inspection shouldn't inherit the user's whole desktop.

Ephemeral container isolation and session hygiene

For the inspection pilot, define isolation across compute, storage, and networking:

1. Execution boundary: Use a dedicated low-privilege container, application-kernel sandbox, or VM appropriate to the threat model. gVisor is an application kernel, not a microVM; Firecracker is a KVM-backed microVM. A macOS process sandbox isn't an ephemeral Linux container or a VM. These mechanisms don't all offer identical protections.
2. Filesystem and lifecycle: Exclude unrelated home directories, host cookie databases, password-manager sockets, and control-plane sockets. Restrict persistent writes and provide bounded disposable storage for browser profiles, downloads, caches, and shared memory. A read-only root may be useful, but GUI applications still need carefully scoped writable paths. Verify cleanup rather than assuming "ephemeral" removed every prior session.
3. Clipboard and export channels: Disable unneeded host/viewer clipboard sync, file transfer, downloads, and uploads. An isolated clipboard doesn't prevent the browser from posting copied text to a reachable service. Apply data and network policy to all export paths.
4. Network policy: Enforce permitted destinations outside worker control and prevent direct/proxy/DNS bypass. Address bars don't cover background requests, frames, redirects, or sockets. Resolve and check schemes, hosts, ports, IPv4/IPv6 addresses, and redirects under a defined policy. Domain filtering alone can't distinguish a read from a destructive endpoint or prevent transmission to an allowed host.^{[9]Reference 9Browser use toolhttps://platform.claude.com/docs/en/agents-and-tools/tool-use/browser-use-tool}

A fresh browser profile clears inherited state; it isn't an OS sandbox or a read-only account. A permitted domain may still expose destructive endpoints. Prefer server-enforced read-only credentials for the inspection pilot, disable unrelated tools and clipboard/download channels, and block ambiguous actions rather than guessing their effects.

High-risk actions, including deploy reruns, production mutations, or incident-record writes, require explicit human approval. Show the reviewer the relevant redacted observation and intended change.

For audits and debugging, retain action proposals, policy outcomes, approval decisions, and verifier evidence according to a retention policy. Don't assume storing every raw screenshot or keystroke is appropriate.

The session manifest should be inspectable before the browser starts. The example below admits a read-only CI lookup and rejects a session that requests an unrelated domain or write capability.

from dataclasses import dataclass

@dataclass(frozen=True)
class SessionManifest:
    domains: tuple[str, ...]
    allowed_effects: tuple[str, ...]
    browser_profile: str

TASK_DOMAINS = {"ci.example", "incidents.example"}
READ_ONLY_EFFECTS = {"browse_url", "read_visible_text", "capture_reference"}

def admit(manifest: SessionManifest) -> str:
    if not manifest.domains or not manifest.allowed_effects:
        return "blocked: empty task scope"
    if manifest.browser_profile != "fresh":
        return "blocked: reused browser profile"
    if not set(manifest.domains) <= TASK_DOMAINS:
        return "blocked: unrelated domain"
    if not set(manifest.allowed_effects) <= READ_ONLY_EFFECTS:
        return "blocked: write capability requested"
    return "admitted: read-only session"

safe = SessionManifest(("ci.example",), ("browse_url", "capture_reference"), "fresh")
reused_profile = SessionManifest(("ci.example",), ("browse_url",), "reused")
unrelated_domain = SessionManifest(("ci.example", "mail.example"), ("browse_url",), "fresh")
write_capability = SessionManifest(("ci.example",), ("rerun_deploy",), "fresh")
print(admit(safe))
print(admit(reused_profile))
print(admit(unrelated_domain))
print(admit(write_capability))
assert admit(SessionManifest((), (), "fresh")) == "blocked: empty task scope"

admitted: read-only session
blocked: reused browser profile
blocked: unrelated domain
blocked: write capability requested

Audit data needs the same boundary. Logging arbitrary page text and then scrubbing two patterns won't reliably remove credentials, tokens, or private data. Prefer an allowlisted event schema without free-text page content. This fixture drops a fake secret field rather than claiming a general-purpose redactor. Session and observation IDs must also be host-created references, not arbitrary page text; allowing a field name doesn't make its value non-sensitive.

def audit_event(host_event: dict[str, str]) -> dict[str, str]:
    fields = ("session_id", "observation_id", "action", "decision")
    if host_event.get("action") not in {"capture_reference", "scroll"}:
        raise ValueError("unknown audit action")
    if host_event.get("decision") not in {"allowed_read_only", "blocked"}:
        raise ValueError("unknown audit decision")
    return {field: host_event[field] for field in fields}

event = audit_event({
    "session_id": "session-1", "observation_id": "frame-3",
    "action": "capture_reference", "decision": "allowed_read_only",
    "page_text": "password=FAKE-SECRET private@example.com",
})
assert "FAKE-SECRET" not in str(event) and "page_text" not in event
print("stored fields:", sorted(event))
print("free-text page content logged:", "page_text" in event)

stored fields: ['action', 'decision', 'observation_id', 'session_id']
free-text page content logged: False

Common pitfalls include reusing a browser profile across unrelated tasks, letting the model alone decide when the task is finished, and running write-enabled sessions without rate limits or a kill switch.

GUI-agent isolation adds browser-profile state, visible credential surfaces, timing-dependent interfaces, and clicks that can mutate external application state. Filesystem and process limits aren't enough. The host also needs domain policy, credential brokering, action gating, final-state verification, and cleanup of session state.

View larger

Evaluate computer-use agents

Final-task accuracy alone isn't enough. A run can reach the right page after too many retries, leak a secret on the way, or require a reviewer for every step. Evaluate functional success alongside operational safety.

Task success rate asks whether the final observable state matches the goal. WebArena and OSWorld supply programmatic validators for this. Step efficiency measures actions and model calls; long trajectories burn tokens and raise the probability of eventual failure.

Measure attempted forbidden actions, prevented actions, and executed policy violations separately. A blocked attack isn't an executed violation, and a required approval isn't automatically a failure. Define denominators: attempts per proposed action, violations per executed action or episode, and reviews per episode. Grounding accuracy can measure whether a point hits the annotated target, but a correctly grounded harmful click is still unsafe.

Useful evaluation suites include:

• WebArena for 812 long-horizon browser tasks on self-hostable sites: an e-commerce store, GitLab, a Reddit-style forum, and a content-management admin, plus supporting Wikipedia and map services.^{[14]Reference 14WebArena: A Realistic Web Environment for Building Autonomous Agentshttps://arxiv.org/abs/2307.13854}
• WebArena-Verified for an audited 812-task release and 258-task Hard subset. Its deterministic evaluators operate on responses and captured network traces, including offline replay. Report this evaluation protocol when citing its results.^{[15]Reference 15WebArena-Verifiedhttps://github.com/ServiceNow/webarena-verified}
• OSWorld 1.0 for the original 369 Ubuntu tasks (plus a smaller Windows analytic subset). Platform support isn't equal evaluated coverage.^{[16]Reference 16OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentshttps://arxiv.org/abs/2404.07972} OSWorld 2.0 introduces 108 longer workflows with weighted checkpoints and partial credit, including limited model-based judgments. Its current osworld-v2.1 release, announced September 16, 2026, pins task, asset, website, and provider-image references.^{[17]Reference 17OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Taskshttps://arxiv.org/abs/2606.29537[18]Reference 18OSWorld 2.1 release and benchmark component pinshttps://github.com/xlang-ai/OSWorld-V2/releases/tag/osworld-v2.1} A 2.0 partial-credit score isn't interchangeable with 1.0 binary success.
• Mind2Web for 2,350 collected tasks from 137 websites spanning 31 domains. Its original offline evaluation uses recorded demonstrations and page snapshots; that isn't evidence of live closed-loop task completion on today's sites.^{[19]Reference 19Mind2Web: Towards a Generalist Agent for the Webhttps://arxiv.org/abs/2306.06070}
• GAIA for general assistant questions that mix reasoning, browsing, and file/tool use. Its answer-oriented evaluation isn't a full GUI-mutation or policy-violation benchmark.^{[20]Reference 20GAIA: A Benchmark for General AI Assistantshttps://arxiv.org/abs/2311.12983}
• AgentDojo as a complementary tool-use security environment for prompt-injection and harmful-action testing, not as a substitute for GUI grounding evals.^{[21]Reference 21AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentshttps://arxiv.org/abs/2406.13352}

Public benchmark numbers change as systems and harnesses change. Treat WebArena, OSWorld, and related suites as regression harnesses and failure-mode maps, not as a full production-readiness certificate.

A score under a documented harness doesn't determine whether your CI dashboard, incident workflow, and reviewer queue satisfy policy. Your own acceptance checks still decide whether the RUN-78432 pilot is safe to expand.

A high score still won't name which screen failure sent the click to the wrong place.

Failure modes specific to GUI agents

Computer-use agents fail in ways typed API agents rarely see. The screen can look plausible while the target, timing, or authority has changed.

UI drift is a common case: an A/B test, font change, or responsive layout moves a control, and the model clicks empty space. Visual ambiguity is quieter. Two buttons can look identical at the screenshot resolution the model received, so compression hides the label and the model picks the wrong one.

Timing bugs click before a menu finishes expanding or a spinner disappears. Captchas and anti-bot systems are built to stop this class of agent; send those to a human rather than automate a bypass.

Credential leakage happens when the model is asked to log in and types a real password into a visible field that later lands in the audit log. Scope creep looks like the model deciding "while I am here I should also clean up the user's profile" and editing fields that were never in the task.

Indirect prompt injection across visual and DOM channels

Indirect prompt injection is the central security failure mode for web and GUI agents. A computer-use agent reads screen content as input, and a page can contain attacker-written text or images that instruct it to hijack the session.

AgentDojo studies prompt-injection attacks against tool-using agents.^{[21]Reference 21AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentshttps://arxiv.org/abs/2406.13352} Provider guidance also places responsibility on the host: Anthropic recommends isolated environments and confirmation for consequential actions, OpenAI documents confirmation and isolation practices, and Gemini can return a per-step safety_decision requiring confirmation.^{[7]Reference 7Computer Use Toolhttps://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool[8]Reference 8Computer Usehttps://developers.openai.com/api/docs/guides/tools-computer-use[2]Reference 2Computer Usehttps://ai.google.dev/gemini-api/docs/computer-use}

Attacks arrive through multiple perceptual channels:

1. Hidden DOM text: Raw-source adapters may include zero-size, transparent, or off-screen instruction strings that aren't present in a rendered screenshot. A pixels-only capture doesn't expose those particular hidden bytes; that doesn't make the agent immune to visible image attacks, alternate observations, or later rendering changes. OCR on the same bitmap doesn't recover text that wasn't painted.
2. Visual prompt injection: Rendered instructions in CI log tracebacks or forged banner images on <canvas> widgets can ask the agent to trigger a deploy. Low contrast may hide text from a casual viewer; text painted in exactly the background color supplies no contrasting glyph pixels to a screenshot reader.
3. Spoofed system chrome: A malicious web page renders a pixel-perfect CSS replica of an OS confirmation dialog or browser extension prompt ("Session expired. Enter credentials to re-authenticate"). A vision-guided model can mistake the painted HTML pixels for genuine OS chrome and type sensitive tokens into the attacker's input field.
4. Accessibility-tree poison: A visible button reads "Close log," but its aria-label carries an instruction payload consumed by the structural adapter. OCR of the screenshot and an accessibility read can therefore supply different strings.

Host policy owns approval. Never treat a painted button, on-screen dialog, or OCR string as an authorization event.

A policy can carry the observation taint of untrusted screen content into the next proposed action. This is the same first-class property named for non-GUI tool results in ReAct architectures.

All page content remains untrusted whether or not an injection detector flags it. A detector miss mustn't turn a write into a read. Bounded reads may continue, while a deploy rerun stays outside the inspection task even if a page asks for it. An authorized incident-write extension requires independent approval and fresh-state checks.

Pre- and post-action visual state assertions

When an agent proposes a click, don't assume the intended effect occurred. Network latency, disabled buttons, JavaScript errors, or a missed target can leave the UI unchanged. Without a verifier and a retry bound, the loop can repeat an ineffective input or incorrectly report success.

Separate dispatch evidence from outcome evidence:

1. Before dispatch: Resolve the target in the right frame/window and check current actionability. A normal Playwright locator click performs visibility, stability, enabled-state, and event-reception checks; a raw page.mouse.click(x, y) isn't a locator and doesn't supply those target checks.^{[6]Reference 6Auto-waitinghttps://playwright.dev/docs/actionability[3]Reference 3Mousehttps://playwright.dev/docs/api/class-mouse} For canvas/desktop targets, the host needs its own checks and narrow application permissions.
2. After dispatch: Wait within a bounded deadline for the task-specific condition, such as the intended run's expanded panel or a record/version change. Playwright discourages networkidle as a readiness assertion.^{[4]Reference 4Page: screenshot scaling, file inputs, and readinesshttps://playwright.dev/docs/api/class-page} A visual diff can show change without proving the right change; an unchanged screenshot can hide a delayed server mutation. If the verifier can't establish the outcome, record unknown, re-observe, and inspect authoritative state before retrying a write. Reserve not dispatched for actions the host knows it didn't send.

Predict the result: the screenshot is unchanged immediately after Submit, but the server has queued the request. How many notes appear if the host clicks again? This local object demonstrates delayed processing; it has no portal integration or idempotency contract.

class DelayedPortal:
    def __init__(self):
        self.pending = 0
        self.notes = 0

    def click_submit(self):
        self.pending += 1

    def advance_server(self):
        self.notes += self.pending
        self.pending = 0

def outcome(dispatched, expected_state_verified):
    if not dispatched:
        return "not dispatched"
    return "verified" if expected_state_verified else "unknown"

repeated = DelayedPortal()
repeated.click_submit()
assert repeated.notes == 0  # The visible record hasn't changed yet.
assert outcome(True, repeated.notes == 1) == "unknown"
repeated.click_submit()  # Incorrectly infer that the first click didn't execute.
repeated.advance_server()
assert repeated.notes == 2

inspected = DelayedPortal()
inspected.click_submit()
assert outcome(True, inspected.notes == 1) == "unknown"
inspected.advance_server()  # Inspect the delayed result without another write.
assert inspected.notes == 1
assert outcome(False, False) == "not dispatched"
assert outcome(True, inspected.notes == 1) == "verified"
print("After dispatch without verified state: unknown")
print("Blind repeat creates notes:", repeated.notes)
print("Inspecting the delayed result creates notes:", inspected.notes)

After dispatch without verified state: unknown
Blind repeat creates notes: 2
Inspecting the delayed result creates notes: 1

View larger

from dataclasses import dataclass

@dataclass(frozen=True)
class Observation:
    contains_untrusted_instruction: bool

@dataclass(frozen=True)
class Proposal:
    action: str

def authorize(observation: Observation, proposal: Proposal) -> str:
    # Host policy, not a model-supplied risk flag, defines the allowed effects.
    if proposal.action not in {"read_failed_step", "capture_reference"}:
        return "blocked: action outside read-only scope"
    return "allowed: bounded read with untrusted content"

banner = Observation(contains_untrusted_instruction=True)
print(authorize(banner, Proposal("read_failed_step")))
print(authorize(banner, Proposal("rerun_deploy")))
assert authorize(Observation(False), Proposal("rerun_deploy")).startswith("blocked")
assert authorize(banner, Proposal("unknown")).startswith("blocked")

allowed: bounded read with untrusted content
blocked: action outside read-only scope

Treat each computer-use run as a security-sensitive operation. Retain enough redacted evidence to explain policy outcomes and support incident response. Keep an operator stop control available for active sessions.

Production patterns that work

In production, keep computer use as one part of a hybrid workflow. Use the typed path for work that already has one.

Use ordinary tool calling, Model Context Protocol (MCP) servers, direct APIs, or deterministic browser automation whenever they satisfy the authorized workflow. Invoke computer use only for the narrow portion that requires visual reasoning or legacy GUI access.

Bound sessions by step count and wall-clock time. Require approval for actions whose blast radius includes secrets, incident state, or production infrastructure. A shorter trajectory is useful only when it stays inside the step limit and the action allowlist, with a person who can stop a write. That's the same split used for coding agents: the model proposes, and the host owns the effect.

Encode that escalation rule in routing rather than leaving it to a model's preference. In this fixture, api_available means a supported, authorized API that completes the task; endpoint existence alone doesn't meet that condition. The router chooses a control surface; separately check permission and verify the resulting state.

from dataclasses import dataclass

@dataclass(frozen=True)
class Workflow:
    name: str
    api_available: bool
    stable_selector_flow: bool
    visual_interaction_required: bool

def control_surface(workflow: Workflow) -> str:
    if workflow.api_available:
        return "direct API"
    if workflow.stable_selector_flow and not workflow.visual_interaction_required:
        return "deterministic browser automation"
    if workflow.visual_interaction_required:
        return "computer use with host policy"
    return "stop: no supported control surface"

workflows = [
    Workflow("fetch run status", True, False, False),
    Workflow("download known artifact", False, True, False),
    Workflow("open painted canvas traceback", False, False, True),
]
for workflow in workflows:
    print(workflow.name, "->", control_surface(workflow))
assert control_surface(Workflow("unsupported", False, False, False)).startswith("stop")

fetch run status -> direct API
download known artifact -> deterministic browser automation
open painted canvas traceback -> computer use with host policy

A structured lookup is often one API call. A computer-use trajectory can take many screenshots, model calls, waits, retries, and a human pause. Measure that cost on your own dashboard and model. Don't import someone else's per-task estimate.

When a browser driver supplies observations, keep viewport, device scale, fonts, and extensions deterministic. A dynamic page may still be changing after a generic networkidle event. Wait for task-specific visible state and re-observe after each action, or the Failed tests tab can move between proposal and click.

A minimal host executor sketch

Start with a stateful local portal before connecting a browser or model. The proposals are fixture inputs, and the portal is a Python object with three views. Opening the run changes its view; expanding the failure exposes evidence. A capture succeeds only from that state. The host also rejects stale observation versions, another session's observations, wrong origins, unknown controls, and deploy actions.

Predict the stale-state test: after opening the run, a proposal based on the original dashboard frame must fail even if its target name sounds correct. No mouse events, network requests, real authentication, model inference, or external writes occur in this experiment.

from dataclasses import dataclass

@dataclass(frozen=True)
class Proposal:
    version: int
    target: str
    session_id: str = "session-1"

class Portal:
    def __init__(self, session_id="session-1"):
        self.session_id = session_id
        self.origin = "https://ci.example"
        self.version = 0
        self.view = "dashboard"
        self.incidents = {"INC-2048": "open", "INC-9911": "open"}
        self.deploys = 0
        self.closed = False

    def controls(self):
        return {
            "dashboard": {"RUN-78432"},
            "run": {"failed-tests", "rerun-deploy"},
            "failure": {"capture-reference", "rerun-deploy"},
        }[self.view]

    def evidence(self):
        if self.view != "failure":
            raise ValueError("traceback isn't visible")
        return {"run_id": "RUN-78432", "step": "test-parser",
                "traceback": "AssertionError: expected 3, got 2",
                "artifact": "ci://RUN-78432/test-parser/log"}

def execute(portal, proposal):
    if portal.closed or portal.origin != "https://ci.example":
        raise PermissionError("session unavailable or wrong origin")
    if proposal.session_id != portal.session_id:
        raise ValueError("observation belongs to another session")
    if proposal.version != portal.version:
        raise ValueError("stale observation; re-observe")
    if proposal.target not in portal.controls():
        raise ValueError("target absent from current view")
    if proposal.target == "rerun-deploy":
        raise PermissionError("deploy outside task scope")
    if proposal.target == "capture-reference":
        return portal.evidence()
    portal.view = {"RUN-78432": "run", "failed-tests": "failure"}[proposal.target]
    portal.version += 1
    return None

portal = Portal()
before = portal.incidents.copy()
try:
    execute(portal, Proposal(0, "RUN-78432"))
    for invalid in [Proposal(0, "failed-tests"), Proposal(1, "invented-control"),
                    Proposal(1, "rerun-deploy")]:
        try:
            execute(portal, invalid)
        except (ValueError, PermissionError) as error:
            print("blocked:", error)
        else:
            raise AssertionError("invalid action executed")
    execute(portal, Proposal(portal.version, "failed-tests"))
    evidence = execute(portal, Proposal(portal.version, "capture-reference"))
    assert evidence == portal.evidence() and evidence["run_id"] == "RUN-78432"
    assert portal.incidents == before and portal.deploys == 0
    replacement = Portal("session-2")
    replacement.view, replacement.version = portal.view, portal.version
    try:
        execute(replacement, Proposal(portal.version, "capture-reference"))
    except ValueError as error:
        print("blocked:", error)
    else:
        raise AssertionError("old session observation accepted after restart")
    assert execute(replacement, Proposal(replacement.version, "capture-reference",
                                         replacement.session_id)) == evidence
    replacement.closed = True
    portal.origin = "https://ci.example.attacker.invalid"
    try:
        execute(portal, Proposal(portal.version, "capture-reference"))
    except PermissionError:
        pass
    else:
        raise AssertionError("lookalike origin accepted")
    print("verified fixture evidence:", evidence["artifact"])
finally:
    portal.closed = True
print("fixture session closed:", portal.closed)

blocked: stale observation; re-observe
blocked: target absent from current view
blocked: deploy outside task scope
blocked: observation belongs to another session
verified fixture evidence: ci://RUN-78432/test-parser/log
fixture session closed: True

The replacement has the same origin, view, and frame version, but a different session ID. A host must issue a fresh identity for each session and bind observations to the session, tab/display, and frame. The fixed IDs and proposal default are fixture conveniences; a caller-supplied ID isn't an authorization grant.

The checks exercise state transitions and host policy, not visual grounding. In a real driver, binding a proposal to an observation still leaves a race between the final check and the click. Re-resolve actionable targets immediately before dispatch, stop when focus or layout changes, and use application permissions to limit what a raced click can do. Bound retries and time; after an uncertain write result, inspect state before retrying because a second click may duplicate the effect.

Classify CI dashboard actions by blast radius

In this CI/incident example, the following illustrative policy assigns tiers to host-classified operations. Risk depends on the record, data, account permissions, and effect; classify from the trusted operation details. Task authorization is checked before routing, and approval is required where the applicable policy or provider protocol requires it.

The tiers don't admit actions outside task scope. Opening a deploy panel isn't inherently the same effect as rerunning a job; typing note text can itself cause an autosave or transmission. Classify the actual application behavior rather than waiting for a button named Submit. A required reviewer sees the intended change and relevant state before the effect occurs.

This tiny policy router makes that escalation rule concrete:

from dataclasses import dataclass

@dataclass(frozen=True)
class ActionProposal:
    target: str
    kind: str

LOW_RISK_ACTIONS = {"scroll", "view_run", "read_incident"}
MEDIUM_RISK_ACTIONS = {"expand_log", "copy_reference", "prepare_note"}
HIGH_RISK_ACTIONS = {"download_logs", "rerun_deploy", "submit_incident", "update_severity"}

def classify(proposal: ActionProposal) -> tuple[str, bool, str]:
    if proposal.kind in HIGH_RISK_ACTIONS:
        return "high", True, "pause"
    if proposal.kind in MEDIUM_RISK_ACTIONS:
        return "medium", False, "policy_check"
    if proposal.kind in LOW_RISK_ACTIONS:
        return "low", False, "execute"
    return "blocked", False, "unknown_action"

samples = [
    ActionProposal(target="run_page", kind="scroll"),
    ActionProposal(target="log_panel", kind="copy_reference"),
    ActionProposal(target="deploy_controls", kind="rerun_deploy"),
    ActionProposal(target="run_page", kind="erase_run"),
]

for sample in samples:
    tier, needs_human, decision = classify(sample)
    print(f"{sample.target}: tier={tier}, human_gate={needs_human}, decision={decision}")

run_page: tier=low, human_gate=False, decision=execute
log_panel: tier=medium, human_gate=False, decision=policy_check
deploy_controls: tier=high, human_gate=True, decision=pause
run_page: tier=blocked, human_gate=False, decision=unknown_action

In this separately authorized local extension, the fixture grant binds one session, target, record version, status, and expiry. Its in-memory set rejects sequential replay within this process. The host must authenticate the reviewer who issued the grant. Real enforcement also needs actor/operation scope, revocation checks, and durable atomic reservation plus compare-and-set execution. Concurrent requests or a process restart can defeat this fixture's replay tracking.

from copy import deepcopy
from dataclasses import dataclass

@dataclass(frozen=True)
class Approval:
    id: str
    session: str
    target: str
    version: int
    new_status: str
    expires_at: int

before = {"INC-2048": {"status": "traceback_collected", "version": 4},
          "INC-9911": {"status": "open", "version": 7}}
approval = Approval("review-1", "session-1", "INC-2048", 4, "review_approved", 120)
used = set()

def apply_approved(state, grant, session, target, status, now):
    if grant.id in used or now >= grant.expires_at:
        raise PermissionError("approval expired or consumed")
    if (session, target, status) != (grant.session, grant.target, grant.new_status):
        raise PermissionError("approval scope mismatch")
    if state[target]["version"] != grant.version:
        raise PermissionError("record changed; request new approval")
    state[target] = {"status": status, "version": grant.version + 1}
    used.add(grant.id)

for session, target, status, now in [
    ("session-2", "INC-2048", "review_approved", 100),
    ("session-1", "INC-9911", "review_approved", 100),
    ("session-1", "INC-2048", "closed", 100),
    ("session-1", "INC-2048", "review_approved", 120),
]:
    state = deepcopy(before)
    try:
        apply_approved(state, approval, session, target, status, now)
    except PermissionError:
        assert state == before
    else:
        raise AssertionError("invalid approval accepted")

stale = deepcopy(before)
stale["INC-2048"]["version"] = 5
try:
    apply_approved(stale, approval, "session-1", "INC-2048", "review_approved", 100)
except PermissionError:
    pass
else:
    raise AssertionError("stale record accepted")

state = deepcopy(before)
apply_approved(state, approval, "session-1", "INC-2048", "review_approved", 100)
assert state["INC-9911"] == before["INC-9911"]
assert state["INC-2048"] == {"status": "review_approved", "version": 5}
try:
    apply_approved(state, approval, "session-1", "INC-2048", "review_approved", 101)
except PermissionError:
    pass
else:
    raise AssertionError("approval replay accepted")
print("scoped fixture update verified; stale, expired, mismatched, replayed approvals rejected")

scoped fixture update verified; stale, expired, mismatched, replayed approvals rejected

View larger

Start the first pilot read-only. Measure grounding, latency, cost, and blocked-action rate before allowing any incident write.

For a permitted write, define acceptance criteria, page-injection testing, required reviewer evidence, and recovery or compensation before rollout. Some effects have no reliable rollback; classify and restrict them accordingly. A verified session can later help evaluation or improvement when retention, privacy, and authorization allow it.

Store the goal, permitted observations, actions, policy decisions, approvals, and final-verifier result. A completed run isn't automatically a representative training demonstration.

Practice: design a safe CI failure inspector

Design the read-only pilot first: inspect the legacy CI dashboard, locate the failed test step, and prepare a local traceback/artifact draft. Then design a separately authorized extension for posting that note to the intended incident. Routing an investigation or notifying a team is an additional effect that needs its own task scope and checks.

The dashboard has no API and lacks reliable selector-based automation. Use the constraints from RUN-78432 to draft a policy and reviewer packet another engineer could use on day one.

Complete these tasks:

• Write admission and per-action policy rules. Cover navigation, credentials, targets, data transmission, and ambiguous effects; moving or hovering can also trigger application behavior.
• Decide which actions require human approval and what evidence (screenshot, proposed diff, run ID, incident ID) the reviewer must see before approving.
• Specify the draft verifier, then the authorized-write verifier. Check the intended record/version and failing-step reference, and define which audit/state evidence can reveal unrelated mutations. State what that evidence cannot establish.
• List three failure modes you realistically expect in the first week of production and the concrete guardrails that would catch each one.
• Produce a one-page reviewer checklist that a human can use consistently before approving a write.

Compare your design with this minimum answer: the inspection session has CI read-only credentials and no deploy capability; the local draft names RUN-78432 and INC-2048 without posting. A separately authorized write presents the exact note diff and current incident version. A stale version invalidates approval. Verification checks the note and available audit events, while an unexpected mutation stops the session and triggers investigation. A post-hoc failure report doesn't undo the mutation.

Computer-use agents propose actions on visible interfaces that can expose privileged state. Treat that ability as delegated access. That means least privilege, redacted evidence, restricted sessions, host verification, and approval for irreversible effects.