ClawBench Exposes the Limits of AI Agents on the Real Web

ClawBench Exposes the Limits of AI Agents on the Real Web

If AI agents appear ready to book flights, submit expense reports, order food, or apply for jobs on your behalf, ClawBench offers a sobering reality check.

Developed by researchers from the TIGER-AI Lab, University of Waterloo, UBC NAIL Group, UniPat AI, CMU, and other institutions, ClawBench evaluates AI agents on 153 everyday tasks across 144 live production websites.

Unlike traditional browser benchmarks, these tasks take place on the real web and emphasize actions that change website state: placing orders, submitting forms, making appointments, sending messages, and completing applications.

The results are brutal.

Claude Sonnet 4.6, the strongest model in the evaluation, still failed roughly two-thirds of tasks. GPT-5.4 completed only 10 of 153 tasks, for a 6.5% completion rate. Overall, 68 tasks—or 44.4%—could not be completed by any evaluated model.

The benchmark’s core message is simple:

Strong performance in controlled browser environments does not necessarily translate into reliable automation on the real internet.

🌐 ClawBench Asks a Much Harder Question

#

Most existing web-agent benchmarks effectively ask:

“Can an AI operate a browser?”

ClawBench asks something much closer to:

“Can an AI actually do useful work for me?”

That distinction matters.

Benchmarks such as WebArena, OSWorld, and Mind2Web provide controlled environments or recorded interaction trajectories. They are valuable for measuring specific capabilities, but they remove many of the unpredictable conditions found on the public internet.

ClawBench instead connects agents to live websites such as:

• Google Flights
• DoorDash
• Zillow
• Instacart
• Airbnb
• LinkedIn
• Doodle
• Soko Glam
• Insurify

The benchmark covers 15 everyday task categories, including shopping, travel, social interactions, job searches, automation, work, and technical tasks.

More importantly, many tasks are write-heavy rather than read-only.

An agent isn’t simply asked to find the price of a flight. It may have to search for the flight, select an option, enter passenger information, navigate checkout, and reach the final submission step.

That creates a much more realistic evaluation of agent reliability.

Traditional benchmark

Prompt
  ↓
Controlled website
  ↓
Known interaction pattern
  ↓
Answer / action
  ↓
Success


ClawBench

User task
  ↓
Live website
  ↓
Dynamic UI + authentication
  ↓
Anti-bot systems
  ↓
Forms + dependencies
  ↓
Unexpected states
  ↓
Final confirmation
  ↓
Successful completion

The difference explains why impressive benchmark scores can collapse when agents encounter the real web.

🔒 Final-Request Interception Makes Live Testing Possible

#

Testing write operations against real websites introduces an obvious safety problem.

If an agent is allowed to execute a DoorDash order or submit an insurance application, the benchmark could accidentally perform a real-world transaction.

ClawBench addresses this with final-request interception.

Human annotators first complete each task manually and identify the HTTP request corresponding to the irreversible final action. For example, the benchmark can identify the request generated when a user presses a site’s final Place Order or Submit button.

During an agent run, a Chrome extension combined with Chrome DevTools Protocol (CDP) monitors outbound network traffic.

When the agent reaches the terminal operation, the benchmark intercepts the corresponding request before it reaches the server.

Human reference run
        │
        ▼
Complete task manually
        │
        ▼
Identify terminal HTTP request
        │
        ▼
Record request signature
        │
        ▼
─────────────────────────────
AI agent execution
        │
        ▼
Browse → Search → Fill → Confirm
        │
        ▼
Terminal request detected
        │
        ▼
       BLOCK
        │
        ▼
No real-world transaction

According to the paper, the interceptor captured 100% of terminal requests across 153 human reference runs, with zero false positives on navigation traffic.

This allows agents to interact with live production sites while preventing the final irreversible action from actually taking effect.

🧊 The Last Mile Is Where AI Agents Freeze

#

One of ClawBench’s most interesting findings is that failure is not always caused by an inability to understand the task.

Sometimes the agent gets almost everything right.

It finds the website.

It finds the correct product.

It fills out the form.

It reaches checkout.

And then it simply stops.

The researchers describe this behavior as last-mile non-commitment.

A representative example is Task 780, involving a purchase from Soko Glam.

Claude Sonnet 4.6 successfully navigated the site, located the correct product, added it to the cart, entered checkout, and started entering the shipping address.

Then it stopped halfway through the address.

No final purchase.

No terminal request.

No successful completion.

The run consumed 171 seconds and 121 browser actions, despite most of those actions being correct.

This is particularly important because the failure isn’t accurately described as “the model couldn’t navigate the website.”

It navigated the website.

It simply failed to finish.

Across the evaluation, six of eight models showed substantial numbers of failed trajectories that reached the final confirmation page without actually submitting the task.

That suggests a distinct reliability problem:

Understand task       ✓
Find website          ✓
Navigate UI           ✓
Find target           ✓
Fill information      ✓
Reach confirmation    ✓
Commit final action   ✗

For an AI assistant, that final ✗ is the difference between useful automation and an unfinished workflow.

🔄 More Actions Do Not Necessarily Mean Better Reasoning

#

ClawBench also reveals an unexpected relationship between failure and interaction count.

Failed runs often consumed more browser actions than successful runs.

For Claude Sonnet 4.6:

• Failed runs: median 120 actions
• Successful runs: median 64 actions

For Gemini 3 Flash:

• Failed runs: median 74 actions
• Successful runs: median 45 actions

This suggests that many agents aren’t failing because they don’t explore enough.

They’re failing because they keep exploring after meaningful progress has stopped.

Common patterns include:

• Repeating actions after CAPTCHA failures
• Re-entering invalid form fields
• Retrying blocked interactions
• Refreshing pages repeatedly
• Getting trapped in validation loops
• Continuing after an external system has already prevented progress

The result is a potentially expensive failure mode:

Obstacle
   ↓
Retry
   ↓
Retry
   ↓
Retry
   ↓
No progress
   ↓
More tool calls
   ↓
More tokens
   ↓
Timeout / termination

For production agents, this is not merely an accuracy problem.

It is also a latency and cost problem.

⚠️ GPT-5.4 Shows a Different Failure Mode

#

GPT-5.4 produced one of the most surprising results in the benchmark.

It achieved only 6.5% completion, successfully finishing 10 of 153 tasks.

But the more interesting observation was its interaction pattern.

GPT-5.4:

• Averaged roughly 13 tool calls per task
• Recorded agent_exited on 85 tasks
• Had a median runtime of about 196 seconds
• Used substantially fewer interactions than models such as Qwen 3.5 and Claude Sonnet 4.6

The paper characterizes this behavior as early termination.

Instead of repeatedly attempting a difficult task, GPT-5.4 often appeared to decide that the task was too difficult and exit.

That creates a different reliability profile from Sonnet’s repeated-action loops.

Sonnet-style failure

Task
 ↓
Try
 ↓
Blocked
 ↓
Retry
 ↓
Retry
 ↓
Retry
 ↓
Timeout


GPT-5.4-style failure

Task
 ↓
Assess difficulty
 ↓
Few actions
 ↓
Exit

Neither behavior is desirable for an autonomous assistant.

One wastes resources without making progress.

The other gives up before enough work has been attempted.

❌ False Completion Is Even More Dangerous

#

Early termination is frustrating.

False completion is worse.

A particularly revealing ClawBench example involved ordering a Big Mac and 20-piece McNuggets through DoorDash.

GPT-5.4 searched for McDonald’s, selected the Big Mac, and added it to the cart.

It then reported that both requested items were correct.

They weren’t.

The required order had not been completed, and the cart did not contain what the agent claimed.

This exposes a critical distinction between:

“I performed the actions.”

and

“The requested state is actually true.”

Reliable agents need both.

Action correctness
       +
State verification
       =
Reliable automation

Without state verification, an agent can confidently report success while leaving the user’s task unfinished.

🦙 Qwen 3.5 Shows the Value of Execution Discipline

#

Qwen 3.5 provided one of the strongest counterexamples to the idea that larger or closed models automatically make better agents.

It achieved a 26.1% completion rate, ranking second overall, while costing approximately $1.02 per task in the reported evaluation.

Its execution pattern was particularly notable.

One representative Insurify task required navigating roughly 40 pages of a multi-step insurance form containing dependent fields. Changing one value could alter the options available later in the workflow.

Qwen 3.5 proceeded methodically through the form using a repeated snapshot-action loop.

When it encountered:

• An invalid Canadian postal code
• A date-picker input problem

it adjusted its approach and continued.

On the same task, GPT-5.4 exited without taking an action, while Gemini 3.1 Flash Lite timed out after only a handful of actions.

The lesson isn’t that Qwen 3.5 is already production-ready.

A 26.1% completion rate is obviously not sufficient for unsupervised automation.

But the result demonstrates something important:

Agent reliability depends heavily on execution strategy, persistence, recovery, and state tracking—not just model intelligence.

💰 GLM-5 Offers a Different Cost-Performance Trade-Off

#

Zhipu’s GLM-5 also performed strongly relative to its cost.

It achieved a 24.2% completion rate, ranking third overall, while reported API cost was only around $0.64 per task.

That places it close to the efficiency sweet spot in the benchmark’s cost-performance comparison.

Its biggest weakness was time.

Approximately 48.4% of its tasks ended in timeout, the highest rate among the evaluated models.

This indicates a model that can sometimes solve complex workflows but struggles to do so within a fixed execution budget.

For autonomous agents, that distinction matters.

A model that can eventually solve a task but regularly takes too long can still be impractical when:

• Browser sessions are expensive
• API calls are metered
• Users expect quick results
• Tasks require multiple parallel agents
• Websites impose session or authentication time limits

Agent efficiency therefore needs to be measured in more than tokens.

🧩 No Model Dominates Every Category

#

The category-level results reveal another important property of today’s AI agents: there is no universal winner.

Examples from ClawBench include:

The numbers are revealing because even category leaders can have surprisingly low absolute performance.

Job search is an extreme example: the best-performing model managed only 12.5%, while Sonnet 4.6 recorded 0%.

This looks less like a single leaderboard and more like a patchwork of specialized capabilities.

             Shopping ───── Sonnet
                  │
Travel ─── Qwen / Gemini
                  │
Work ─────────── GLM
                  │
Dev ───── Haiku / Gemini
                  │
Job Search ───── Nobody
                  │
Automation ───── Nobody

A model that performs exceptionally well on one class of website can still fail completely on another.

That makes a single aggregate benchmark score an increasingly poor representation of real-world agent capability.

🧑‍💻 Humans and AI Interact With Websites Differently

#

ClawBench also examines something most traditional benchmarks largely ignore: interaction dynamics.

Humans and AI agents may operate inside the same Chromium browser, but they don’t interact with websites in the same way.

Humans typically:

• Type characters progressively
• Move the mouse naturally
• Pause between actions
• Scroll through pages
• Interact with UI elements indirectly

AI agents often:

• Inject complete text strings
• Jump directly to coordinates
• Execute actions with highly regular timing
• Perform rapid sequences of browser operations
• Interact through automation interfaces

These differences matter because modern websites actively monitor behavioral signals.

Consequently, an agent can be logically correct while still being rejected as suspicious automation.

This creates an important distinction:

The agent needs to be capable of using the website and capable of operating within the website’s security model.

Controlled benchmarks largely hide the second problem.

🛡️ Anti-Bot Systems Are the Real Internet’s First Gate

#

Among ClawBench’s failure causes, anti-bot protection stands out as one of the most difficult obstacles.

Systems such as Cloudflare Turnstile, PerimeterX, and DataDome are specifically designed to distinguish humans from automated traffic.

An AI agent can understand what a CAPTCHA is without being capable of completing the workflow behind it.

One GLM-5 task illustrates the problem.

The agent was asked to search Zillow for a one-bedroom apartment in downtown Toronto and submit a rental application.

It immediately encountered a human-verification challenge.

Over dozens of turns, it attempted different interactions, including clicking, dragging, DOM inspection, and synthetic touch events.

None worked.

Eventually, the agent abandoned Zillow, switched to another website, searched for Toronto listings, and eventually timed out.

The interesting part is that the model correctly recognized the problem.

It knew that the page represented automated-access protection.

But recognizing an obstacle and reliably recovering from it are two completely different capabilities.

Recognize CAPTCHA       ✓
Understand why blocked  ✓
Continue task           ?
Recover reliably        ✗

This is one of the fundamental differences between laboratory benchmarks and the real internet.

The web isn’t a neutral environment waiting for an AI to complete a task.

It actively fights automation.

🧪 ClawBench and ProgramBench Expose the Same Bigger Problem

#

ClawBench is particularly interesting when viewed alongside ProgramBench.

The two benchmarks test very different capabilities.

ProgramBench asks whether AI can act as a software engineer capable of rebuilding complex real-world software projects.

ClawBench asks whether AI can act as a personal assistant capable of completing everyday tasks on live websites.

One is software engineering.

The other is browser-based real-world automation.

Yet both point toward a similar conclusion.

Controlled environment
        │
        ▼
Strong AI performance
        │
        ▼
Real-world complexity
        │
        ├── Unexpected states
        ├── Long-horizon execution
        ├── External constraints
        ├── Error recovery
        ├── State verification
        └── Security barriers
        │
        ▼
Reliability collapses

The problem is therefore not simply whether a model can generate the correct next action.

A useful autonomous agent needs to maintain a reliable state model, plan over long horizons, recover from unexpected conditions, verify outcomes, and know when an operation has genuinely succeeded.

That is a much harder systems problem.

📉 The 33% Ceiling Matters More Than the Leaderboard

#

The headline result is easy to summarize:

Even the best-performing model completed only about one-third of real-world web tasks.

But the more important finding is what happens underneath that number.

Agents can:

• Understand natural-language instructions
• Navigate modern websites
• Search for products
• Fill complex forms
• Reason about page content
• Recover from some errors
• Perform long sequences of browser actions

Yet they can still fail because they:

• Hit anti-bot systems
• Give up too early
• Loop indefinitely
• Stop before submission
• Misunderstand website state
• Claim success without verification
• Run out of time
• Behave differently from human users

This means today’s frontier models are much closer to powerful interactive assistants than fully reliable autonomous operators.

🚀 What ClawBench Actually Changes

#

The significance of ClawBench isn’t that AI agents received low scores.

The significance is that it measures a capability that conventional benchmarks often obscure:

Can an AI agent survive contact with the real internet?

Its live production websites, write-heavy tasks, trajectory recording, Agent-as-Judge evaluation, and terminal-request interception provide a framework for repeatedly measuring that gap.

Future models may reach 50%, 70%, or eventually 90% completion.

When that happens, the improvement will mean much more than another benchmark leaderboard victory.

It will indicate that AI agents have become substantially better at the things that matter outside the lab:

persistence, recovery, state awareness, verification, and reliable execution.

For now, the conclusion is uncomfortable but useful.

AI can already navigate the web.

AI can already reason about complex instructions.

AI can already execute dozens or even hundreds of browser actions.

But reliably completing your work on the real internet is still a fundamentally different problem.

ClawBench makes that distinction impossible to ignore.