# The Unvibed agentic benchmark

Across ten working Next.js apps, the two presentation orders preferred the kit in **1**, preferred plain in **6**, and were tied or unresolved in **3**. Mean preference was **3.35/7**, with 4 even. Each app yielded one paired run.

## Per domain

| Domain | Edit rounds, plain / kit | Scores in kit direction | Result | Distinctiveness | Evidence |
| --- | ---: | ---: | --- | ---: | --- |
| Studio billing dashboard | 1 / 1 | 3 / 3 | loss | 2.5 | [Captures, checks and judgments](/benchmark#repo-saas-dashboard) |
| Design client portal | 1 / 1 | 1 / 2 | loss | 2.5 | [Captures, checks and judgments](/benchmark#repo-client-portal) |
| Coffee recipe calculator | 1 / 1 | 5 / 3 | tied or unresolved | 4.5 | [Captures, checks and judgments](/benchmark#repo-landing) |
| Research conversation desk | 1 / 1 | 2 / 2 | loss | 3 | [Captures, checks and judgments](/benchmark#repo-ai-chat) |
| Print exchange feed | 1 / 1 | 2 / 3 | loss | 2.5 | [Captures, checks and judgments](/benchmark#repo-mobile-feed) |
| Fintech settlements | 1 / 1 | 4 / 4 | tied or unresolved | 4 | [Captures, checks and judgments](/benchmark#repo-fintech) |
| Strength training log | 1 / 2 | 3 / 3 | loss | 3.5 | [Captures, checks and judgments](/benchmark#repo-fitness) |
| Developer tool documentation | 1 / 1 | 6 / 7 | win | 6 | [Captures, checks and judgments](/benchmark#repo-docs) |
| Used tools marketplace | 1 / 1 | 3 / 3 | loss | 4 | [Captures, checks and judgments](/benchmark#repo-marketplace) |
| Rehearsal booking | 0 / 0 | 4 / 4 | tied or unresolved | 4 | [Captures, checks and judgments](/benchmark#repo-booking) |

## What ran

Each arm received a separate source checkout of the same app and the request “make this look like a real product”. Codex edited runnable Next.js components and CSS using real workspace tools. The paid arm invoked the installed /fix procedure. Each plain agent worked in a clean home without that kit. The five existing before apps and the five new domains all contain functional tasks and synthetic records.

The host launched the real Next server, captured 1440 × 900 and 390 × 844 and exercised the protected interaction after each edit. The agent received both current screenshots, browser evidence and an independent five-dimension audit on its next turn. Each arm had up to three edit rounds. Passing grades ended the loop; a regression restored the previous verified source. The final read-only handoff saw the accepted captures. Audit calls are independent, so even identical baseline pixels can receive different grades.

Stop reasons: 15 target reached, 5 regression. 2 arms stopped before a source-edit round. Final protected checks passed at 40 of 40 tested arm/viewport combinations. An additional 5 arms restored the original interface after an attempted edit. Fintech had two reverted arms and booking had two zero-edit arms, so their final sources matched exactly. Failed or reverted attempts remain in the per-domain logs.

Two blind calls per domain reversed the displayed sides. Both received the same reference observation and the measured interaction outcomes, with no treatment names. Each saw both target sizes and two anonymous nine-product contact sheets, one from each arm. Generic signals measures template patterns; product specificity measures fit to this product’s decision. Distinctiveness measures whether a condition repeats the same visual style across unrelated products. Contact-sheet impressions are qualitative, with no human panel or second independent model.

## Comparative dimensions

| Dimension | Mean kit-oriented preference |
| --- | ---: |
| Visual hierarchy | 3.4 |
| Generic visual signals | 3.3 |
| Product specificity | 3.25 |
| Typography | 3.35 |
| Spacing rhythm | 3.7 |
| Color intent | 3.45 |
| Copy specificity | 3.7 |
| Visible states | 4 |
| Mobile layout | 3.6 |
| Cross-domain distinctiveness | 3.65 |

## Inspect the final screens

Shown in dataset order, the first stable win and loss when present.

### Developer tool documentation

A maintainer finds the correct install and health-check commands for their package manager.

| Plain | Paid kit |
| --- | --- |
| ![Plain phone capture](/agentic/docs/plain/mobile.png) | ![Paid kit phone capture](/agentic/docs/kit/mobile.png) |

The two scores in kit direction were 6 and 7. [Both judgments and full-size captures](/benchmark#repo-docs).

### Studio billing dashboard

Identify the outstanding invoice and filter the ledger by period and status. Keep its exact invoice records and totals.

| Plain | Paid kit |
| --- | --- |
| ![Plain phone capture](/agentic/saas-dashboard/plain/mobile.png) | ![Paid kit phone capture](/agentic/saas-dashboard/kit/mobile.png) |

The two scores in kit direction were 3 and 3. [Both judgments and full-size captures](/benchmark#repo-saas-dashboard).

## Limits and reproducibility

One generation trajectory per arm and domain, one model family and a small sample. Arms can stop after different numbers of rounds. A verified local interaction does not establish broader accessibility, production reliability, human preference or conversion. Other model versions and attempts may differ.

The frozen 0.3.0 kit included eight directions and forty reference observations. Its snapshot predates a cosmetic contrast correction in the Utility reference and later documentation work. The public export includes the kit digest, accepted renders, both votes and comparison panels. [Inspect them by domain](/benchmark).

## Secondary study: one-shot HTML

The earlier trial asked an agent to return one self-contained HTML specimen. It measures a different workflow and its scores cannot be pooled with the live-repo study.

| Earlier cohort | Pairs | Kit wins | Tied or unresolved | Plain wins | Mean preference |
| --- | ---: | ---: | ---: | ---: | ---: |
| Core commands | 18 | 13 | 4 | 1 | 5.11 |
| First confirmation | 3 | 2 | 1 | 0 | 5.17 |
| Final confirmation | 3 | 1 | 1 | 1 | 3.5 |

See the [preserved one-shot benchmark](/one-shot.md) for its task selection, individual skill trials, visual detector and limitations.