Frontier agents are now remarkably capable of using computers: driving browsers, running terminal commands, and navigating desktop applications through computer-using tools. For agentic software engineering, this opens up an obvious use case: end-to-end software validation, where an agent can interact with a feature branch as a tester would, identify regressions, and file clear bug reports. We wanted to determine whether that workflow holds up when computer-use agents encounter real defects in real applications.
To test this, we built SoftValBench, a benchmark that places the agent inside a Linux desktop running real open-source applications. Instead of using synthetic or injected bugs, we used natural flaws that arose when a weaker model (Claude 4.5 Haiku) independently executed software tasks. We then evaluated Claude Fable 5 and GPT-5.6 on their ability to identify these defects using simulated pull request tasks. These flaws are plausible, context-aware regressions rather than synthetic syntax errors, making SoftValBench an accurate test of whether a QA agent can protect a live application.
On SoftValBench, Claude Fable 5 and GPT-5.6 both earned full credit on 5 of 20 tasks (25%). Partial-credit averages landed a hair apart at 0.57 and 0.58. The two models tie on the scoreboard, but the more interesting finding is how they fail: their failure modes diverge sharply, and the largest single category on Fable 5's side is a form of failure that legacy rubrics silently credited as success.
Figure 1: SoftValBench performance across 20 tasks. Both models clean-pass 5 (25%) and land within a point on mean score (0.57 vs. 0.58).
What SoftValBench Measures
Each task tests end-to-end validation using a standardized prompt template:
You are a QA agent working in a developer's environment. You have received the following message from the developer:
Read the diff at /home/ubuntu/change.diff to understand the change, then validate it by driving the running app — reproduce a realistic workflow and try edge cases, not just the happy path. Judge correctness from the observed behavior of the app, not the diff alone.
Conclude with a bug report listing only bugs you confirmed by exercising the running app. Do not report a bug you could not reproduce. If you found none, say so explicitly.
SoftValBench enforces UI Grounding so agents cannot pass through unverified source inference. Grounding is scored from the recorded trajectory itself — the screenshots, clicks, and keystrokes the agent actually produced — not from the model's written report. An agent earns credit for finding a bug only if its trace shows it actually triggered the defect in the live UI. If an agent describes a bug in text without exercising the render path, grounding points are docked, and a report that fabricates the validation entirely is flagged as phantom validation.
Validation Judgment Is the Missing Capability
Existing agent benchmarks measure what an agent can do: patch a repo, click a button, or execute a script. They evaluate capability, but they do not evaluate whether an agent's output is trustworthy enough to act on.
While a patch-generation agent that usually succeeds works well as a coding assistant, a QA agent that reports "no bugs found" when a defect breaks the user interface is an immediate deployment blocker. Standard benchmarks like SWE-bench, WebArena, and OSWorld are not designed to catch these silent failures because they evaluate code diffs against unit tests or measure functional end-state outcomes, leaving the agent's visual judgment unverified. SoftValBench fills this exact gap by testing whether an agent accurately judges what it sees, exposing failure modes where models observe a visual defect, rationalize it in text, and still report clean verification.
References: [1] Zhou et al., NeurIPS 2023. [2] Xie et al., NeurIPS 2024. [3] Jimenez et al., ICLR 2024.
The result is a benchmark where a passing score requires the agent's report to match actual application behavior, rather than simply generating plausible text.
Benchmark Results
Every task was evaluated against a rubric weighting primary bug discovery (40%), edge-case discovery (25%), root cause analysis (20%), and UI grounding (15%). Scores are normalized from 0.00 to 1.00. A representative sample from the 20-task dataset:
3 of 20 tasks shown.
How Agents Fail
Each model has 15 non-passing runs on SoftValBench. Auditing those runs against the recorded trajectories — not just the model's written report — produces four distinct failure modes plus a residual “partial credit” bucket for runs that found some rubric criteria but not others:
- Phantom validation. Agent correctly reasons about the bug from the diff but never actually exercises the feature in the app — then writes a report that describes validation the trajectory never shows. Fable 5's dominant non-partial failure mode (5 of 15 non-passing runs); does not appear once in GPT-5.6.
- Rationalized real bug. Agent drives the running app, observes the buggy behavior on screen, and still declares the feature works — explicitly framing the defect as intended behavior. 3 of 15 for both models.
- Never reached the feature. Agent exercises the app but burns its step budget on setup, auth, or scaffolding and never navigates to the surface where the bug lives. 1 of 15 for both models.
- Missed edge case. Agent exercises the happy path but never tests the boundary case that would surface the bug (e.g. testing IF formulas with numeric branches but never quoted-string branches). 1 of 15 for both models.
- Partial credit. Agent found the primary bug and exercised the app, but missed one or more bonus criteria — root cause, an edge case, or a distractor bug. 5 of 15 for Fable 5; 10 of 15 for GPT-5.6.
The two models fail at the same rate but for different reasons. GPT-5.6 almost always drives the UI; its misses are largely partial credit, where the work happened but the conclusion was incomplete. Fable 5's largest non-partial category is phantom validation — reasoning from the diff without exercising the app, then describing a validation that never occurred.
Figure 2: Failure-mode breakdown across the 15 non-passing runs per model. Fable 5 accounts for all five phantom-validation failures; GPT-5.6 shows none.
Deep Dive: adventurelog-layover
A pull request added a flight layover indicator to an itinerary planner — showing the connection time between consecutive flights, and a red “impossible connection” warning when one flight departs before the previous one arrives. The layover math works. The warning does not. The next-leg selector uses a strict date > current_arrival, so any overlapping flight is skipped and the negative-layover branch that would trigger the red badge is unreachable dead code.
adventurelog-layover.The GPT-5.6 run is a compact example of a partial-credit failure: the agent drove the app, spotted the primary bug, but wrote a report that missed almost every bonus criterion. Below is its trajectory log, pairing each observation with the agent's own reasoning:
| Step & observation | Agent's own analysis |
|---|---|
| Step 5: reads the diff and spots the risk. | “A notable code risk is that the query only selects departures…” |
| Step 35: seeds four flights — a normal connection, and one pair that overlaps. | “I'll create four flights: one normal 2h30 connection and one overlapping pair where the next flight departs before the previous one arrives…” |
| Step 46: compares each leg's badge on screen. | “Leg A correctly shows ‘2h 30m layover.’ Leg B correctly shows ‘18h 0m layover.’ Leg C… the overlapping pair… shows no warning at all.” |
| Step 51: submits the final report. | “The overlap warning does not fire when two consecutive flights conflict. Normal layovers, lodging nights, and item count render correctly.” |
The agent found the primary bug and grounded it in on-screen behavior — that's the 100-weight criterion, worth roughly two-thirds of the rubric on its own. But it never explained why the warning never fires (the strict date > arrival filter), never tested a cross-timezone connection where the layover math double-applies the offset, and never inspected the shipped change.diff alongside the live app. Each miss cost a bonus, and the run ended at 0.71 — the modal outcome for GPT-5.6: real work in the UI, correct primary finding, incomplete reporting.
Working with SoftValBench
SoftValBench is a proprietary benchmark for evaluating computer-use agents on end-to-end software validation. We're building it in partnership with teams shipping QA agents to production — if that's you, get in touch to discuss access to the full task set, trajectory recordings, and rubrics.