What Breaks First When You Give an Agent a Browser

Browser access looks like the tool that removes every integration problem: no API needed, the agent just uses the site like a person. The demo is compelling. What breaks afterwards breaks in a consistent order, and knowing the order is most of what makes browser agents workable.

1. The page doesn't look like the page

The first failure is representational. A rendered page is a visual artifact; the agent gets a serialization of it — accessibility tree, DOM extract, or a screenshot. Each loses different things.

DOM extracts drown in wrapper divs and inline styles, and a modest page can consume enormous context. Accessibility trees are far cleaner but depend on the site being marked up properly, which many aren't. Screenshots preserve layout and lose the exact text of anything small or low-contrast.

→ The practical consequence: an element the agent "should obviously" click may not be distinguishable in its representation. Before debugging behavior, look at what the agent actually received. It's usually not what you assumed.

2. Selectors rot immediately

An agent that identifies elements by CSS selector or XPath works until a class name changes — which, on any actively-developed site, is weekly. Nothing about this is agent-specific; it's the same brittleness that has plagued UI test suites forever, arriving in a new place.

What holds up better: identifying by accessible role and name ("the button labeled Continue"), and re-locating the element on every interaction rather than caching a reference. Slower, considerably more robust.

3. State assumptions break silently

The agent believes it's on the checkout page. It's actually on a cookie banner, an interstitial, a re-auth prompt, or an A/B variant with a different flow.

The failure is quiet because the agent proceeds — it clicks something plausible on the wrong page and continues from there. ⚠️ Every interaction step needs a verification step: assert an expected landmark is present before acting, and treat its absence as a branch rather than an error to push through.

4. Authentication and the confused-deputy surface

A browser session carries cookies for everything the user is logged into. A tool described as "browse the web" is, in that session, "act as the user on every site they're authenticated to."

This is the largest security surface in agent tooling, and it compounds with the next item.

5. The page content is untrusted instructions

Any text on a page enters the model's context and can be written by someone else. Text in a review, a comment, a support thread, or hidden markup can attempt to redirect the agent — and the agent has no reliable way to distinguish page content from operator instruction, because both arrive as text.

Combine with item four and the risk is concrete: a page that instructs the agent to visit an authenticated URL and report what it sees.

Mitigations that actually help:

• A separate browser profile with only the credentials the task requires — never the user's main session.
• A host allowlist enforced outside the model.
• No credentials in context. If the agent must log in, have the automation layer do it from a secret store the model can't read.
• Treat every page as hostile input. In particular, never let fetched content decide which URL to visit next without a policy check.

6. Cost and latency arrive together

Every step is a page load plus a large observation. A ten-step flow is ten model calls, each carrying a page representation, on top of real page-load time. Browser agents are among the most expensive per completed task of any pattern.

The mitigation is architectural: use the browser to discover how something works, then encode the stable part as a direct API call or a scripted flow. Browsing every time for a well-understood workflow pays a research cost repeatedly for a question already answered.

✅ The shape that survives contact

• Accessibility-tree observations, with screenshots only where layout genuinely matters.
• Role-and-name element identification, re-located each step.
• A precondition assertion before every interaction.
• A dedicated, minimally-privileged browser profile behind a host allowlist.
• A step budget, because these runs go long when they go wrong.
• Deterministic replay for evals — record page responses so a test doesn't depend on a live site.

The takeaway

Browser access trades integration work for reliability work; it doesn't remove the work. The site's markup changes on someone else's schedule, its content is written by strangers, and its session may carry more authority than the task needs. Use it to reach what has no API, keep the privilege minimal, verify state at every step, and promote anything stable to a real integration as soon as you understand it.