Computer-Use Agent Systems
Computer-use agent systems: screenshot and DOM perception, action loops, sandboxed browsers, confirmation gates, recovery, evals, and when APIs beat UI agents.
- Read time
- 17 min
- Level
- Advanced
- Updated
- 2026-08-25
- Components
- Perception (DOM/screenshot) · Action planner · Sandboxed executor
A computer-use agent system is a closed loop that perceives a graphical interface (DOM, accessibility tree, screenshot, or all three), plans a UI action, executes it in a sandbox, and re-perceives until the task succeeds, fails a budget, or hits a confirmation gate. It is how you automate software that has no usable API — and it is the wrong default whenever an API exists.
Key takeaways
- 01
Prefer an API or documented protocol over computer use; UI agents exist for the leftover surfaces.
- 02
Perception should fuse structured trees (DOM, AX) with pixels; screenshots-only agents break on icon-only controls and fail evals you cannot replay.
- 03
The executor runs in a disposable sandbox with no production credentials and no unconstrained network — the agent will click whatever it hallucinates.
- 04
Irreversible actions (submit, pay, send, delete) pause for a human or a policy check; reversible navigation does not.
- 05
Evaluate task success, unsafe-action rate, and recovery from injected UI changes — not whether the trajectory “looks reasonable.”
Anatomy: perception, plan, act, check
Each step is an observation–action pair. Perception builds a state: a reduced DOM or accessibility tree, a screenshot, the current URL, and a list of candidate targets with stable selectors where they exist. The planner — an LLM, often multimodal — chooses one action from a small schema: click, type, scroll, select, wait, keychord, done, fail, or escalate. The executor performs that action against a real browser or OS session. A checker compares the new observation to the goal and to invariants (still on the expected origin, no unexpected permission prompt).
Keep the action schema small and typed. Free-form “do whatever” tools turn the model into an un-auditable remote desktop. Record every pair: observation hash, action, result, timestamp. That trace is how you debug, how you train, and how you prove to security what the agent touched.
Data flow through a step
Goal and policy sit outside the model: a task object with success criteria, a domain allow-list, a step budget, and a list of actions that require confirmation. On each tick the system captures observation, redacts secrets on screen (known password fields, SSO cookies), and sends a compact view to the planner — not the raw 4K screenshot plus the entire DOM. The planner returns a structured action. Policy checks it (origin, action type, rate). If it is reversible, the executor runs it; if it is irreversible, the confirmation gate blocks until a human or a second policy says yes. After execution, recovery logic decides whether to continue, retry, re-plan, or stop.
Targeting order matters. Prefer accessibility names and test ids over (x, y) coordinates; coordinates break on resolution, zoom, and animation. Fall back to vision when the tree is empty or lies (canvas, remote desktop, custom widgets). Log which targeting mode fired — a sudden spike in coordinate clicks is an early warning that the UI tree disappeared.
- Task object: goal, success check, domain allow-list, step budget
- Observe → redact → compact → plan → policy → act → check
- Selectors first, coordinates last
- Confirm on send/pay/delete/submit; auto-continue on navigation
- Trace every observation–action pair
Sandbox, credentials, and blast radius
Computer-use agents operate a real browser or OS. That is a remote-control channel, not a chatbot. Run sessions in disposable VMs or containers: fresh profile, no extensions, no access to the operator’s cookies, network egress allow-listed to the target app plus required IdP. Production data stores are not mounted. If the task needs a login, inject a short-lived, scoped credential from a vault the model never sees.
Assume prompt injection from the page. Visible text, alt text, and PDFs inside the UI are untrusted instructions. The policy layer — not the planner prompt — is what prevents “download this binary” and “email the customer list.” Disable file downloads, OS-level shell, and extra tabs unless the task type explicitly allows them. Snapshot the VM; on budget exceeded or invariant break, kill the session rather than letting it improvise.
Recovery and confirmation
UI automation fails in ordinary ways: spinner never ends, selector gone after a deploy, modal covering the target, session expired, captcha, A/B variant. A recovery controller owns these, not the planner. Timeouts get a wait-and-reobserve. Missing selectors trigger a re-perception with a screenshot fallback, then at most N re-plans. Auth walls escalate; they are not something the agent should “click through” with stored passwords. Captchas and security challenges fail closed to a human.
Confirmation gates sit on irreversible effects. The gate shows the human the goal, the proposed action, and the screenshot crop — not a wall of DOM. Batch low-risk confirmations in shadow mode during rollout; do not remove the gate on payments, messages, or deletes because the demo was smooth. Loop detection (same selector, same action, K times) is a hard stop.
Failure modes
Wrong-target clicks are the headline: the model describes the right button and hits the adjacent one. Unbounded loops burn the budget while looking busy. Agents that accept on-page instructions will exfiltrate data or navigate off-origin. Session reuse across tasks leaks the previous user’s state. Coordinate-only control looks fine in a recording and fails in production at a different DPI. Success checks that only look for a banner string are gamed by the agent opening a cached confirmation page.
Verify outcomes outside the UI when you can: the record exists in the API, the email hit the outbox, the ticket left the queue. If the only verifier is the screenshot, you are trusting the same channel the agent can manipulate.
- Off-by-one clicks and hallucinated controls
- Prompt injection from page content
- Session and credential reuse across tasks
- Looping on spinners or failed selectors
- Success checks that trust the UI the agent controls
Evals, and when not to use computer use
Golden tasks are recorded expert trajectories plus an outcome verifier. Report task success, steps-to-success, confirmation rate, unsafe-action attempts (blocked off-origin, blocked download, blocked send), and recovery success after you mutate the UI (renamed button, extra modal). Run the suite in the same sandbox class as production. Video is for humans; the gate is the verifier.
Do not use computer use when a stable API, MCP tool, or RPA connector already performs the workflow — it will be slower, flakier, and harder to permission. Do not point an agent at an unconstrained desktop “to see what it can do.” Use it for vendor portals, legacy thick clients, and internal apps that will not get APIs this year, with a sandbox, a domain allow-list, and a confirmation gate on anything that leaves the machine.