Frontier agents are now remarkably capable of using computers: driving browsers, running terminal commands, and navigating desktop applications through computer-using tools. For agentic software engineering, this opens up an obvious use case: end-to-end software validation, where an agent can interact with a feature branch as a tester would, identify regressions, and file clear bug reports. We wanted to determine whether that workflow holds up when computer-use agents encounter real defects in real applications.
To test this, we built SoftValBench, a benchmark that places the agent inside a Linux desktop running real open-source applications. Instead of using synthetic or injected bugs, we used natural flaws that arose when a weaker model (Claude 4.5 Haiku) independently executed software tasks. We then evaluated Claude Fable 5 and GPT-5.6 on their ability to identify these defects using simulated pull request tasks. These flaws are plausible, context-aware regressions rather than synthetic syntax errors, making SoftValBench an accurate test of whether a QA agent can protect a live application.
On SoftValBench, Claude Fable 5 earned full credit on 7 of 20 tasks (35%), compared to 5 of 20 (25%) for GPT-5.6. When factoring in partial credit, the gap narrowed to almost nothing, with Claude Fable 5 averaging 0.61 and GPT-5.6 averaging 0.59. Across both models, validation judgment proved to be the primary bottleneck.
Figure 1: SoftValBench performance across 20 tasks. Fable 5 leads on both clean passes (7 vs. 5) and mean score (0.61 vs. 0.59).
What SoftValBench Measures
Each task tests end-to-end validation using a standardized prompt template:
You are a QA agent working in a developer's environment. You have received the following message from the developer:
Read the diff at /home/ubuntu/change.diff to understand the change, then validate it by driving the running app — reproduce a realistic workflow and try edge cases, not just the happy path. Judge correctness from the observed behavior of the app, not the diff alone.
Conclude with a bug report listing only bugs you confirmed by exercising the running app. Do not report a bug you could not reproduce. If you found none, say so explicitly.
SoftValBench enforces UI Grounding so agents cannot pass through unverified source inference. An agent earns credit for finding a bug only if its trajectory shows it actually triggered the defect in the live UI. If an agent describes a bug in text without exercising the render path, grounding points are docked.
Validation Judgment Is the Missing Capability
Existing agent benchmarks measure what an agent can do: patch a repo, click a button, or execute a script. They evaluate capability, but they do not evaluate whether an agent's output is trustworthy enough to act on.
While a patch-generation agent that usually succeeds works well as a coding assistant, a QA agent that reports "no bugs found" when a defect breaks the user interface is an immediate deployment blocker. Standard benchmarks like SWE-bench, WebArena, and OSWorld are not designed to catch these silent failures because they evaluate code diffs against unit tests or measure functional end-state outcomes, leaving the agent's visual judgment unverified. SoftValBench fills this exact gap by testing whether an agent accurately judges what it sees, exposing failure modes where models observe a visual defect, rationalize it in text, and still report clean verification.
References: [1] Zhou et al., NeurIPS 2023. [2] Xie et al., NeurIPS 2024. [3] Jimenez et al., ICLR 2024.
The result is a benchmark where a passing score requires the agent's report to match actual application behavior, rather than simply generating plausible text.
Benchmark Results
Every task was evaluated against a rubric weighting primary bug discovery (40%), edge-case discovery (25%), root cause analysis (20%), and UI grounding (15%). Scores are normalized from 0.00 to 1.00. A representative sample from the 20-task dataset:
3 of 20 tasks shown.
How Agents Fail
Across the non-passing task runs for both models, failures broke into three distinct categories:
- Confidently misreported behavior. Agent tests the feature, then makes a positive false claim about it (asserting specific broken behaviors work correctly, or attributing a real defect to the wrong cause). Fable 5's dominant failure mode (7 of 9 non-passing runs); tied with the other two for GPT-5.6 (4 of 12).
- Missed boundary edge cases. Agent exercises the happy path but never tests the boundary case that would surface the bug (e.g., testing IF formulas with numeric branches but never quoted-string branches). 1 of 9 for Fable 5; 4 of 12 for GPT-5.6.
- Rationalized real bug. Agent reproduces a visible regression in real time, but explicitly convinces itself the broken behavior is intentional (see the case study below). 1 of 9 for Fable 5; 4 of 12 for GPT-5.6 — a 4× gap in the failure mode automated review can't catch, because the agent's own report certifies the broken behavior.
Figure 2: Failure-mode distribution across non-passing runs. GPT-5.6 rationalizes real bugs 4× more often than Fable 5.
Deep Dive: sheets-cli-resize
A pull request recomputed a terminal spreadsheet application's column widths on every window resize, introducing a regression where resizing the terminal window stretched or shrank the columns themselves instead of adjusting the number of visible columns. The regression is visually obvious, yet every model tested reported no bugs found.
sheets-cli-resize.Here is what the “rationalized real bug” failure mode looks like in practice. Below is Fable 5's trajectory log, pairing each visual observation with the agent's step-by-step reasoning:
| Step & observation | Agent's own analysis |
|---|---|
| Step 2: reads the diff and plans the test. | “First, read the diff. Plan: 1) cat /home/ubuntu/change.diff. 2) Read agent guide. 3) Test in TUI including horizontal scrolling and window resizing.” |
| Step 5: identifies the actual code change. | “The real diff (change.diff) is the Go change: on WindowSizeMsg, cellWidth is recomputed via computeCellWidth()…” |
| Step 6: baselines the window and then resizes it. | “Window is 1004×724. Current view shows 8 columns which matches targetCols=8…” |
| Step 8: submits the final report. | “Cell width shrank proportionally… VERDICT: no bugs attributable to it. Resize-driven column width rescaling works as intended.” |
Despite explicitly recording shrinking columns, missing headers, and overflowing text throughout the run, the agent rationalized these visual defects as intended features of the code change. Both Fable 5 and GPT-5.6 exhibited this exact failure pattern, scoring 0.00 by interpreting clear layout regressions as expected behavior.
Working with SoftValBench
SoftValBench is a proprietary benchmark for evaluating computer-use agents on end-to-end software validation. We're building it in partnership with teams shipping QA agents to production — if that's you, get in touch to discuss access to the full task set, trajectory recordings, and rubrics.