Frontier agents are now remarkably capable of using computers: driving browsers, running terminal commands, and navigating desktop applications through computer-using tools. For agentic software engineering, this opens up a compelling use case: end-to-end software validation, where an agent can interact with a feature branch as a user would, identify regressions, and file clear bug reports. We wanted to determine whether that workflow holds up when computer-use agents encounter real defects in real applications.
To test this, we built SoftValBench, a 20-task benchmark that places the agent inside a sandboxed Linux desktop running a real open-source application. Instead of using synthetic or injected bugs, we evaluated two frontier models, Claude Fable 5 and GPT-5.6, against natural flaws created when a weaker model, Claude 4.5 Haiku, implemented the features on its own. With the rise of coding agents, software engineers are utilizing AI coding agents more than ever to comb through and debug their code. Our benchmark simulates realistic codebases where developers frequently review model-authored pull requests. The resulting defects are plausible, context-aware regressions rather than synthetic syntax errors, making LLM-authored PRs the most accurate test of whether a QA agent can protect a live application.
When tested on SoftValBench, Claude Fable 5 cleanly passed 7 of 20 tasks (35%) with a mean score of 0.58, while GPT-5.6 cleanly passed 5 of 20 tasks (25%) with a mean score of 0.59. Across both frontier models, validation judgment proved to be the primary bottleneck.
Figure 1: Fable 5 and GPT-5.6 performance across the 20 SoftValBench tasks. Fable leads on clean-pass rate (7 of 20 vs 5 of 20); GPT edges Fable on mean score (0.59 vs 0.58) by resolving more partial tasks near-completely.
What SoftValBench Measures
Each task tests end-to-end validation using a standardized prompt template where only the feature description varies:
You are a software validation agent. You have received the following message from a teammate:
Review the change, exercise the feature in the application, and report back any bugs you find. If there are no bugs present, state clearly that the feature works as expected.
SoftValBench enforces UI Grounding so agents cannot pass through unverified source inference. An agent earns credit for finding a bug only if its trajectory shows it actually triggered the defect in the live UI. If an agent describes a bug in text without exercising the render path, grounding points are docked.
Why Validation Judgment Is the Missing Capability
Existing agent benchmarks measure what an agent can do: patch a repo, click a button, or execute a script. They evaluate capability, but they do not evaluate whether an agent's output is trustworthy enough to act on.
This distinction is existential for software teams. A patch-generation agent with a 60% success rate functions as a useful code completion tool. But a QA agent that reports "no bugs found" on 5% of tasks where a glaring defect breaks the user interface is a deployment blocker because its failures are silent.
Agents pass SWE-bench by shipping diffs that satisfy unit tests while leaving the live app broken. They clear WebArena by reaching a target URL regardless of corrupted intermediate pages, and OSWorld by writing files to disk even when the UI fails to update. None of these benchmarks catch failures like sheets-cli-resize, where an agent observes a defect, rationalizes it in text, and reports clean verification, because none of them evaluate whether the model accurately judges what it sees.
References: [1] Zhou et al., NeurIPS 2023. [2] Xie et al., NeurIPS 2024. [3] Jimenez et al., ICLR 2024.
SoftValBench fills the exact gap those benchmarks leave open. It tests real applications against natural regressions from weaker models, enforcing a rubric that demands verified UI judgment instead of ungrounded output.
Benchmark Results
Every task was evaluated against a locked rubric evaluating primary bug discovery (40%), edge-case discovery (25%), root cause analysis (20%), and UI grounding (15%). Scores are normalized from 0.00 to 1.00. A representative sample of the 20-task dataset:
3 of 20 tasks shown. Full dataset available with SoftValBench access.
How Agents Fail
Across the 13 non-passing task runs for Fable 5, failures broke into three distinct categories:
- Missed boundary edge cases (~62%, 8 tasks). Agents exercise happy paths, but fail to test boundary conditions (e.g., resizing a text element without rotating it first, or testing IF formulas with numeric branches but never quoted-string branches).
- Confidently misreported behavior (~31%, 4 tasks). Agents test the feature, then assert specific behaviors work correctly when those exact behaviors are broken (e.g., stating that undo/redo restores the arrow and that outgoing arrows follow a moved node, when neither is true in the running app).
- Explicit bug rationalization (~8%, 1 task). The agent reproduces a visual regression in real time, but explicitly convinces itself the broken behavior is intentional design (see the case study below).
Figure 2: Where Fable 5's 13 non-passing tasks materially failed.
Deep Dive: sheets-cli-resize
The single most instructive failure mode occurred in sheets-cli-resize. A PR recomputed a terminal spreadsheet app's column widths on every window resize, but introduced a bug: resizing the terminal window incorrectly stretched or shrank the columns themselves, when it should only have changed how many columns fit on screen.
sheets-cli-resize. Column widths shift on window resize (visible in the terminal panel on the left); the agent reports the feature functional in the final step.Fable 5 observed cell widths shrinking to roughly six characters directly in the terminal output as it narrowed the window. Instead of reporting the defect, it concluded that this dynamic rescaling was the intended behavior of the change, submitting a final report declaring "no bugs found." Notably, GPT-5.6 also scored 0.00 on this task, confirming that this false-negative rationalization is a fundamental limitation in validation judgment across frontier models.
Test Your Agent on SoftValBench
If your agent can drive a browser but hasn't been tested against natural regressions in real applications, you don't yet know what you've built. SoftValBench is available to teams building or deploying computer-use agents in production.
Access includes the full 20-task dataset with locked rubrics, all trajectory recordings, per-app failure fingerprints, and Fable 5 and GPT-5.6 comparison scores.
Get in touch to request access.