Computer Use: When the Agent Drives a Whole Desktop

Giving an agent a screen, a mouse, and a keyboard is the most general capability available — anything a person can do on a computer becomes reachable, including in software with no API and no automation surface. It's also the capability with the widest gap between the demo and a system you'd run unattended.

The failures aren't the ones people anticipate.

Perception is the bottleneck, not action

Clicking and typing are easy. Knowing what to click is the hard part, and it's a perception problem the agent solves from a screenshot.

Where that breaks:

• Small or low-contrast text may not be legible at the resolution the model receives.
• Visually identical elements — three "Delete" buttons in a list — need positional reasoning that's fragile.
• State that isn't visible. Whether a field is focused, whether a dropdown is open, whether a background operation is running.
• Coordinate precision. Being a few pixels off is a different button, and the agent may not realize it clicked the wrong one.

→ Anything with an accessibility tree should use it rather than pixels. It's structured, it carries roles and labels, and it eliminates most of the above. Screenshots are for confirming state, not for locating targets.

The verify-after-act discipline

The single biggest reliability difference: never assume an action worked.

click(save_button)
screenshot()                       # what happened?
assert_visible("Saved")            # or handle the alternative

Without verification, an agent that clicks a button which didn't register proceeds as though it did, and every subsequent step operates on a false model of the screen. Errors compound silently and the final state can be badly wrong with no error anywhere.

⚠️ Verification also has to handle timing. UIs animate, load, and update asynchronously. A screenshot taken immediately after a click frequently shows the previous state — so "wait for an expected element" beats "wait a fixed time," which beats "don't wait" by a large margin.

Timing and modality are the recurring failures

Beyond the general slowness, three specific problems recur:

Unexpected modals. A cookie banner, a session-expiry prompt, an update notification, an autosave dialog. Each blocks the intended flow, and each looks to the agent like an unfamiliar screen. Handle them as a class: before every action, check for known blocking overlays and dismiss them.

Focus loss. Something steals focus and keystrokes go elsewhere. Anything typed into the wrong field can be anything from harmless to catastrophic.

Scroll state. The target exists but is off-screen. An agent that concludes "not present" when it means "not visible" takes a wrong branch, and this is a common source of confident errors.

Isolation is not optional

A computer-use agent operates a real machine, so containment is a design requirement rather than a hardening step:

• A dedicated VM or container, never a person's actual desktop.
• Only the accounts the task needs, in a fresh profile. A logged-in browser session is authority over everything it's signed into.
• Network restricted to what the task requires.
• Filesystem scoped, with nothing sensitive mounted.
• Ephemeral — destroyed after the run, so nothing carries over between tasks.

And the same untrusted-content problem as any browsing agent, amplified: every pixel of text on screen is potential instruction, from any application, including ones the task didn't intend to open.

✅ Where it genuinely fits

• Legacy software with no API and no prospect of one. The canonical case, and often the only option.
• One-off or low-volume tasks where building an integration costs more than the work.
• Exploration — using the UI to discover how something works, then encoding the stable part as a real integration.
• Cross-application flows where the value is in bridging tools that don't talk to each other.

❌ Where it doesn't

• Anything with an API. Slower, more expensive, and far less reliable for no benefit.
• High-volume repetitive work, where per-step cost multiplies.
• Anything requiring precision, where a mis-click is expensive.
• Unattended operation on consequential actions, without staging and review.

The takeaway

Computer use trades reliability for reach. Perception is the constraint, so prefer accessibility trees over pixels; verify after every action and wait on expected elements rather than on time; handle modals, focus, and scroll as recurring classes rather than as surprises. Isolate it in an ephemeral VM with minimal credentials, and treat everything on screen as untrusted. Then use it to reach what nothing else can — and promote anything stable to a real integration as soon as you understand it.