How to Build a Computer-Use Agent
How to build a computer-use agent: screenshot and DOM loop, browser sandbox, write confirmations, and evals on task success — not a scripted RPA recording.
To build a computer-use agent, run a model in a sandbox that observes the screen (screenshot, DOM, or accessibility tree), emits one action at a time, confirms writes, and scores itself on task success — not on whether a scripted UI selector still matches.
What you’ll build
- A perception-action loop over screenshot, DOM, or accessibility tree
- An allowlisted action space with timeouts, max steps, and domain locks
- A sandbox that dies at the end of the task and cannot pivot to your intranet
- Confirmations on writes, payments, and credential fields
- An eval set scored on task success, not on whether the trajectory 'looked reasonable'
Before you start
- 01A list of tasks the agent must complete, with start URL and a machine-checkable success condition
- 02A disposable browser or VM image that cannot reach production data by default
- 03An LLM or VLM endpoint that can emit structured actions (click, type, scroll, wait)
- 04A human confirmation channel for any action that submits a form or spends money
- 05Logging that stores screenshots and DOM snapshots without shipping raw PII to a third-party trainer
Key takeaways
- 01
A computer-use agent is a perception-action loop, not a replay of recorded clicks.
- 02
Prefer a disposable browser or VM with an egress allowlist over the user's real session.
- 03
Cap steps, lock domains, and require confirmation before submit, pay, or delete.
- 04
DOM plus screenshot beats screenshot-only on dense enterprise UIs; screenshot beats DOM on canvas and remote desktops.
- 05
Ship only when a frozen task set reports success rate and unsafe-action rate you can defend.
How to build a computer-use agent (and when not to)
A computer-use agent looks at a graphical interface and acts: click, type, scroll, wait. It exists for systems with no API — a vendor portal, a legacy thick client, a browser flow that changes weekly. If an API exists, call the API. UI-level agents are slower, more expensive, and easier to jailbreak into a click on the wrong button.
Do not confuse this with classic RPA. RPA replays selectors. A browser agent / UI agent driven by an LLM re-plans when the button moves. That is the value, and also the risk: the same flexibility can click Pay Now. The rest of this guide is the loop, the sandbox, and the evals that keep that risk inside a box.
Screenshot, DOM, and the observation loop
Each turn, the agent needs a compact observation. Full-resolution screenshots burn tokens and hide small text. Crop to the viewport, annotate numbered candidate elements, and send the image plus a text list of roles and names from the accessibility tree. On internal admin tools with real ARIA labels, DOM-only is often enough and cheaper. On canvas-heavy SaaS and Citrix-style desktops, you need pixels.
Keep history short. Send the last few actions and a digest, not fifty screenshots. If the page did not change after a click, tell the model it missed rather than letting it click the same pixel again. Stamp every observation with URL, tab title, and whether a modal is open so the policy model can refuse off-domain navigation.
- Observe: screenshot crop + accessibility snapshot + URL
- Plan: one structured action, not a paragraph of intent
- Act: executor clicks or types inside the sandbox
- Check: success predicate or 'no change' signal before the next turn
Sandboxes, domain locks, and confirmations
The executor must not be the employee's laptop. Use a headless browser with a throwaway profile, or a VM with a snapshot restore. Block file system, webcam, and local network. Allowlist DNS to the target vendor and your identity provider if login is required. If the task needs a login, inject a short-lived task account, not the user's cookies.
Hard limits: max steps, max wall-clock, max tabs, and a domain lock. Leaving the allowlisted host ends the task. Password managers in the sandbox should be empty. For any action that submits a form, triggers a payment, or types into a field with type=password, pause and show a human the screenshot plus the proposed action. No confirmation, no click.
Stuck states, captchas, and recovery
Agents get stuck on cookie banners, session timeouts, and 'did you mean' dialogs. Encode recovery as tools: dismiss known banners by role name, refresh once on a 5xx, and stop on captcha or 2FA that the task account cannot pass. Do not let the model invent a phone number to satisfy an SMS prompt.
When the DOM is a virtualized grid, clicks miss. Fall back to keyboard navigation or to an API if you discover one mid-project. Log the trajectory either way. The cheapest fix for a chronically flaky UI is often a vendor API ticket, not a larger vision model.
Evals on task success, not cinematic demos
Build a set of frozen tasks: start URL, credentials for a lab account, and a checker. Score binary success, steps-to-success, dollars of model spend, and unsafe-action count (submit without confirm, domain escape, credential field). Re-run on every prompt and model pin change. Public benchmarks are useful for research; your vendor's portal is the only score that matters.
Keep traces: screenshot per step, action JSON, and the checker result. Humans review failures twice a week and add new tasks when the vendor ships a UI change. If you cannot name the success rate this week, you do not have a computer-use agent in production — you have a demo.
Implementation notes
ReinforcedX builds computer-use agents in the client's cloud: sandbox image, action executor, confirmation UI, and the task-success suite. We are model-agnostic — Anthropic computer-use, OpenAI operators, Gemini, or an open-source VLM — and we do not mark up tokens. You own the trajectories, evals, and runbooks.
Four weeks is the standard path: task inventory and threat model, sandbox and loop in staging, shadow-mode on lab accounts, handover. Thirty days on-call covers the first vendor UI change. If the workflow has a stable API, we will tell you to use the API instead of a UI agent.
Step-by-step build
- 1
Define tasks and success checks
Write each workflow as start state, allowed domains, and a checker (URL, DOM assertion, or API side-effect) so you are not grading vibes.
- 2
Choose perception: screenshot, DOM, or both
Use DOM/accessibility roles where the app is well-tagged; add screenshots for icons, canvases, and remote desktops; fuse them when enterprise pages mix both.
- 3
Lock the action space
Expose click, type, scroll, wait, tab, and back as structured tools with coordinates or element refs; forbid shell, file, and arbitrary network tools in v1.
- 4
Put the loop in a sandbox
Launch a fresh browser profile or VM per task, allowlist egress, strip SSO cookies, set a wall-clock and step budget, and destroy the environment on exit.
- 5
Gate writes and credentials
Detect password fields, payment widgets, and submit buttons; pause for a human confirmation with a screenshot of the pending action.
- 6
Evaluate on task success
Run the frozen task set on every model or prompt change, record trajectories, and fail the release if success drops or an unsafe action fires without confirmation.
Common pitfalls
The mistakes that show up in real deployments — each one costs a week if you learn it the hard way.
Treating a UI agent as recorded RPA
A selector that worked on Tuesday breaks on Wednesday's CSS. Computer-use agents recover from layout change by seeing the screen; if you hard-code XPaths you have rebuilt brittle RPA with extra latency.
Giving the agent the operator's laptop
A model with the user's cookies can drain a bank or mail a customer list. Run inside a locked browser profile or VM with no SSO cookies, no local files, and an egress allowlist.
No stop condition
Without a max-step cap and a success checker, the agent clicks in circles and runs up the vision bill. Kill the session at N steps, on navigation off the allowlisted domains, or on a captcha.
Auto-submitting forms
Typing into a search box can be autonomous. Clicking Pay or Send needs a human. Put a confirmation gate on submit, purchase, delete, and any field that looks like a password.
Evaluating demos instead of tasks
A slick video of booking a flight is not a metric. Score binary task success on a frozen set of sites and workflows, plus unsafe-action rate.