A Browser Agent With 99% Per-Step Accuracy — Why Does It Only Succeed 36% of the Time After 100 Steps?

Running a browser agent demo is easy — a single website, a single run, everything going smoothly while you stare at the screen and pray nothing breaks. Production is a completely different story: thousands of unattended runs, websites that can change at any moment, and models that occasionally have bad days. Derek Megan, a software engineer at BrowserBase, addressed this head-on at the AI Engineer Conference, zeroing in on a core tension that many people overlook — costs accumulate continuously, while value is only realized at the finish line.

How Browser Agents Actually Interact With Web Pages

It helps to think of the model's output as a probability distribution: input tokens represent the page state, the overall goal, and the steps already taken; the model samples the most likely action from that distribution; that action is then translated into a structured tool call and deterministically executed in the browser.

But the browser interface itself is deeply complex, built from multiple layers: network requests flowing in and out of the browser, the DOM (the HTML representation of the page), page screenshots, the accessibility tree (a semantic text representation of the page), and browser state including storage, cookies, the console, URLs, and tabs. Underneath all of this sits a code execution runtime that can interact with the browser via arbitrary JavaScript or CDP.

The industry has converged on three main strategies for wiring these layers together. The first is building a text representation of the browser, typically mixing HTML state and the accessibility tree into a single representation. The second is computer use, representing the browser through a series of page screenshots. The third — growing in popularity — is called de-harnessing: removing custom tooling and instead letting the agent write arbitrary code against the browser in a dynamic execution environment. In practice, most production systems rely on the first two approaches or some combination of them.

Derek also categorizes browser trajectories by their degree of "agentiness" into three types: at one end, purpose-built browser trajectories designed for a single task; at the other end, the most agentic form — ad-hoc browser automation, where the task is unknown in advance and provided by the user on the fly; and in the middle, using the browser as an implementation detail in service of a larger goal, like deep research or competitive analysis. For at-scale deployments, he focuses specifically on transactional workflows — not because they're simple, but because they represent the most common form of large-scale real-world deployment.

Accessibility Tree is the semantic structure of a page that browsers expose for assistive technologies like screen readers. It transforms visual DOM elements into tree nodes with attributes like role, name, and state. Compared to raw HTML, the accessibility tree filters out most purely stylistic information while preserving a semantic description of what the page is doing — making it a better model input with fewer tokens and a higher signal-to-noise ratio. CDP (Chrome DevTools Protocol) is the low-level control protocol provided by Chrome/Chromium; automation libraries like Puppeteer and Playwright are built on top of it, allowing programs to directly control tabs, intercept network requests, and inject JavaScript. It is the "primitive" layer through which browser agents communicate with the browser engine. Understanding both layers helps explain why browser interfaces are far more complex than ordinary web APIs: agents must simultaneously process visual, semantic, network, and storage signals — all heterogeneous in nature.

Why Production Is So Hard

The difficulty comes down to one critical dynamic: costs accumulate continuously, while value is realized only at the end. Every step in a trajectory incurs an incremental cost and adds an incremental failure risk — but you only extract value from a trajectory once every single step has been completed successfully. In other words, browser agents offer no partial credit — and that's true of the vast majority of web automation tasks.

Here's a sobering thought experiment: suppose your browser agent has an independent per-step success rate of 99% — that sounds highly reliable. But if a trajectory spans 100 steps, the overall success rate drops to roughly 36%. It works only about one in three times. That's not good enough to deliver reliable results for customers in production.

Follow this logic to its conclusion — costs keep accumulating, risk keeps building, and the probability of value realization is vanishingly low — and you might conclude you should never use browser agents at all. But Derek emphasizes that while this dynamic is real, the math is not the whole story. It's time to step back and rethink how agents actually create value.

The mathematical core here is exponential decay through compounding probabilities. If each step has an independent success rate of p, the overall success rate of an n-step trajectory is p^n. With p=0.99 and n=100, that's 0.99^100 ≈ 0.366. Drop p to 0.95 and the same 100-step trajectory falls to roughly 0.006 — practically unusable. This property shows up in any system that requires all steps to succeed, and has long been discussed in contexts like database transactions and distributed Saga patterns. What makes browser agents distinctive is that the number of steps is usually dictated by the task itself and can't be arbitrarily reduced without changing the task's semantics — and the failure modes at each step are highly heterogeneous (network timeouts, element location failures, model hallucinations, anti-bot blocking…), making it impossible to cover everything with a single error-tolerance mechanism. Improving per-step success rate and reducing total step count are therefore two equally important but mechanistically distinct optimization paths.

Redefining Value: Profit = Revenue − Cost

Derek puts forward a deceptively simple formula that many people undervalue: Profit = Revenue − Cost. The implication is that your agent is just another line item on a financial statement — it should give you more than it takes from you.

So what does a browser agent actually give you? From the perspective of BrowserBase customers, they're typically not automating the web for themselves — they're performing actions on the web on behalf of their customers. What a browser agent truly delivers is "completing some task for your customers, repeatedly" — and doing so reliably every time, as long as the environment doesn't change.

Agent quality can be evaluated along three dimensions, ranked by importance:

• Performance: The primary success metric. Once you've confirmed the agent can complete the task, cost and maintainability become optimization problems — and optimization problems can be solved with engineering.
• Cost: This breaks down into per-run cost (currently dominated by model cost), infrastructure and compute, and additional integrations and tooling. As open-source models drive prices down, model cost will represent a shrinking share of the total.
• Maintainability: This includes observability investment, developer time, and re-evaluation as the problem evolves.

On measuring success, Derek offers two key principles. First, success should correspond to a concrete artifact produced by the run: paying a bill means a confirmation email; placing an order means an order ID or receipt; submitting a form means a new record in the system that can be deterministically queried. Second, don't measure success rate per run — instead, allow retries and measure success per transaction. If an agent has a 50% single-run success rate but a transaction allows up to 4 attempts, the per-transaction success rate quickly climbs to roughly 94%. Customers don't care how many retries it took — only whether the workflow was reliably completed.

Let the Model Do Only What It Should

Once performance is acceptable, everything else is an optimization problem. Derek walks through a real example to show how to drive success rates up — automating a health insurance portal to log in and download an Explanation of Benefits (EOB) document. Every architectural change should move at least one of the three levers: cost, performance, or maintainability.

Start with the simplest possible agent: a request comes in, the agent takes actions in the browser. This is the minimal system that can "do something."

Wrap complex operations into a single tool: downloading an EOB involves programmatically manipulating the page and retrieving the downloaded file from storage back into the agent runtime. Rather than having the model make decisions on the fly, encapsulate this entire complex operation into a single tool call.

Give the agent real-time validation: add an OCR tool that deterministically extracts entities from the document and compares them against the system of record. Now the agent has a deterministic mechanism to actually know whether it did its job correctly.

Strip stable logic out of the model: the portal's business logic may be ambiguous, but the authentication flow almost never changes. Pull the authentication logic out of the model's responsibility and turn it into a standalone function. This reduces the number of steps the agent must take and can be reused across multiple business workflows on the same portal — lower unit cost, better performance, easier to maintain.

Use a "skill" to constrain the critical path: finally, narrow the agent's responsibility to the areas that are genuinely ambiguous, and draft a "skill" for it. Think of it like a Standard Operating Procedure (SOP) written for the agent rather than for a human. Going back to the probability distribution framing — when the agent knows it's on step 2 and the skill explicitly states that step 3 comes next, the probability that it takes step 3 is much higher. Ambiguity is removed from the decision process.

The result is a fairly complex but highly efficient system: a request comes in, a serverless authentication function handles browser authentication first, then hands the browser off to the agent runtime; the agent navigates the portal according to its skill, calls the deterministic download function, hands the file to the validation tool, and can now authoritatively determine whether the trajectory succeeded or failed. The system moves steps that should never have been the model's responsibility out of the model's scope, reduces the total step count, and still achieves the goal — more maintainable, higher performance, and a way out of the browser agent "36% trap."

The Skill concept described here is essentially a structured context injection mechanism. Traditional RPA relies on recorded deterministic scripts; LLM-based agents rely on the model sampling actions from a probability distribution at each step. "Skills" sit between the two: not step-by-step hard-coded scripts, but operating procedures written in natural language or pseudocode that explicitly tell the agent "in this context, take this path" — thereby compressing the variance in the model's action distribution at critical decision points. This resembles Chain-of-Thought prompting in some ways, but emphasizes workflow constraints over reasoning chains. The effect is to make steps that have a correct answer — but which the model might not reliably infer on its own — behave in a near-deterministic fashion, while preserving the model's flexibility to handle genuinely ambiguous situations. This is the key engineering technique for improving success rates without reverting to fully hard-coded scripts.

Don't Forget the Risk Factors That Push Back

Even after a solid system is built, the durability of these metrics over time faces several threats: the environment fights back (anti-bot mechanisms, web pages that aren't designed to be friendly to agents); tasks change (the underlying work shifts beneath the agent's feet); better approaches emerge (a website you're automating today might ship an API tomorrow — though for most use cases, APIs aren't arriving anytime soon); and models drift (their inherent uncertainty means they won't always take the critical path). Fortunately, several of these factors will ease as model capabilities improve — stronger models are better at understanding user intent and self-correcting when things go wrong.

Derek closes with a half-joking observation: welcome to production — congratulations, you're on-call now. That line captures the truth of operationalizing browser agents: the gap between demo and production is bridged by systematic engineering discipline over probability, cost, and risk.