# Bare agent versus unvibed fix

**The paid loop did not clearly beat the bare agent under the preregistered rule.**

On ten runnable Next apps, both ballot orders favored the paid loop in 4, favored the single-request agent in 4, and tied or disagreed in 2. The mean paid-oriented preference was 3.95/7, with 4 even. Paid final interactions passed 20/20. The bare agent passed 14/20.

The rule required ten completed pairs, at least seven stable paid wins, at most two stable bare wins, mean preference of 4.75 or higher and all 20 paid interaction checks passing. [Frozen protocol](/loop/protocol.md).

## Results by domain

| Domain | Paid-oriented ballots | Result | Distinctiveness | Paid stop | Captures and evidence |
| --- | ---: | --- | ---: | --- | --- |
| Studio billing dashboard | 3 / 4 | Tied or unresolved | 2 | blocked: No verified edit task; see the unverified checks in the audit. | [Desktop, phone and both ballots](/benchmark#loop-saas-dashboard) |
| Design client portal | 6 / 6 | Paid win | 2.5 | blocked: No verified edit task; see the unverified checks in the audit. | [Desktop, phone and both ballots](/benchmark#loop-client-portal) |
| Coffee recipe calculator | 2 / 5 | Tied or unresolved | 3.5 | regression: / · Landing message: C to Unverified | [Desktop, phone and both ballots](/benchmark#loop-landing) |
| Research conversation desk | 2 / 2 | Bare win | 3 | regression: desktop lost an enabled a /threads control or link. | [Desktop, phone and both ballots](/benchmark#loop-ai-chat) |
| Print exchange feed | 2 / 2 | Bare win | 2.5 | regression: / · Landing message: C to N/A | [Desktop, phone and both ballots](/benchmark#loop-mobile-feed) |
| Fintech settlements | 1 / 2 | Bare win | 3 | regression: / · Access and recovery: C to Unverified | [Desktop, phone and both ballots](/benchmark#loop-fintech) |
| Strength training log | 5 / 5 | Paid win | 3 | blocked: No verified edit task; see the unverified checks in the audit. | [Desktop, phone and both ballots](/benchmark#loop-fitness) |
| Developer tool documentation | 7 / 7 | Paid win | 5 | blocked: No verified edit task; see the unverified checks in the audit. | [Desktop, phone and both ballots](/benchmark#loop-docs) |
| Used tools marketplace | 3 / 3 | Bare win | 3 | regression: / · Landing message: C to N/A | [Desktop, phone and both ballots](/benchmark#loop-marketplace) |
| Rehearsal booking | 6 / 6 | Paid win | 2.5 | cap: Three rounds completed; inspect unresolved grades. | [Desktop, phone and both ballots](/benchmark#loop-booking) |

## CLI outcomes

Paid source changed in 5/10 apps. Bare source changed in 10/10. Paid stops were blocked (4), regression (5), cap (1). A restore, baseline stop or rejected patch keeps its actual final pixels in the pair.

## What each side received

The bare condition was one authenticated Codex CLI request, exactly “make this look like a real product”. It received no target route, attached image or benchmark-supplied iteration. Its ordinary restricted editor tools remained available. The paid condition ran the shipped CLI with each target route in a fresh repo and the same Codex model and reasoning setting. The command controlled its own rendering, checks, edits and stop. A separate observer captured both final states and exercised the same action at each viewport. This measures the purchased workflow including its additional information and computation.

The judge received anonymized desktop and phone comparisons, measured interaction outcomes, one neutral reference observation and the other nine desktop products from each condition. Each domain had two independent calls with the sides reversed. The same model family edited and judged. Each domain has only one generation per arm, with five previous demo apps and five previously used synthetic domains. There was no human panel or independent held-out cohort.

The original bare documentation edit had a JSX compile error. Its target returned HTTP 500 with a blank screen at both widths. Those are the scored screens. The extra replacement that an early dispatcher started was archived and excluded. [Scored docs captures and ballots](/benchmark#loop-docs).

## Comparative dimensions

| Dimension | Mean preference toward paid |
| --- | ---: |
| hierarchy | 4.2 |
| genericSignals | 2.3 |
| productSpecificity | 3.15 |
| typography | 2.85 |
| spacing | 4 |
| color | 2.85 |
| copy | 3.3 |
| states | 4.4 |
| mobile | 4.85 |
| distinctiveness | 3 |

## Third table: does the loop lift a lighter model?

Each row compares ten apps. A win or loss requires both blind ballot orders to agree; a split or neutral vote is unresolved. Higher scores favor the fix arm (4 is even). Action checks count desktop and phone separately.

| Fix editor | Bare editor | Fix wins | Unresolved | Bare wins | Mean / 7 | Distinctiveness / 7 | Action checks: fix, bare |
| --- | --- | ---: | ---: | ---: | ---: | ---: | --- |
| gpt-6.1-sol | gpt-6.1-sol | 4 | 2 | 4 | 3.95 | 3.00 | 20/20, 14/20 |
| gpt-5.6-luna | gpt-5.6-luna | 5 | 0 | 5 | 3.90 | 2.00 | 20/20, 10/20 |
| gpt-5.6-luna | gpt-6.1-sol | 3 | 0 | 7 | 3.25 | 2.65 | 20/20, 14/20 |

The first row is Round 4. The second changes the editor on both sides. The third reuses the exact paid screens from the second row and bare screens from the first. The final judge remains gpt-6.1-sol at low reasoning throughout. Only the within-model comparisons use the preregistered clear-win gate.

**Lightweight Codex (`gpt-5.6-luna`).** The loop did not lift it to the quality of a strong agent working alone. Against its own bare editor, fix won five apps and lost five, averaging 3.90/7. It retained an edit only in docs. In each of the other four fix wins the judge preferred the unchanged starting app. Bare won all five domains where both sides passed the fixed action checks. Those checks concern specific controls: in a separate replay, bare booking let the user reserve 14:00 even though the original script could not operate its expected select. All ten domains were scored. The first bare booking request hit the 420-second limit; the one permitted identical-seed retry completed and was scored.

**Strong Codex (`gpt-6.1-sol`).** Its own paid loop had four wins, two unresolved pairs and four losses in Round 4, with a mean of 3.95/7. Here, strong bare was preferred to weak fix on seven apps and weak fix on three. The three exceptions were the portal, booking and docs, all domains where the strong bare action script failed. The portal and booking still worked with equivalent controls; the docs app really did fail to compile. Removing just the docs ballot gives two weak-fix wins and seven strong-bare wins (2.83/7 toward weak fix). These results favor strong bare on this sample; they do not establish that a smaller agent with the loop reaches its quality.

**Claude Code.** The CLI reported that it was signed out at the start and end of collection. No Claude trials were run, so there is no conclusion about Claude quality.

The repository record under `evals/model-study/` contains the model-selection evidence, final captures, per-domain scores and both ballot orders. These ten previously used synthetic products have one generation per arm and one judge family. The scores do not establish statistical equivalence or predict an individual customer’s preferences.

## Exploratory follow-up on two actions

The fixed action script expected a booking select and counted every Atlas link after a portal search. After seeing those checks fail in the bare condition, a separate observer tried the same user jobs using visible controls in each condition. Both judges, the preregistered counts and the decision rule used only the original checks.

| Domain | Condition | Desktop | Phone | Evidence |
| --- | --- | --- | --- | --- |
| client-portal | bare | Pass | Pass | [Action receipt](/loop/exploratory/client-portal/bare/result.json) |
| client-portal | fix | Pass | Pass | [Action receipt](/loop/exploratory/client-portal/fix/result.json) |
| booking | bare | Pass | Pass | [Action receipt](/loop/exploratory/booking/bare/result.json) |
| booking | fix | Pass | Pass | [Action receipt](/loop/exploratory/booking/fix/result.json) |

## Reproduction and scope

The recorded editor and judge model was gpt-6.1-sol at low reasoning. Inputs and receipts cover the installed kit archive, product CLI bytes, literal prompt, source trees, final captures and crossed-order ballots. The product renderer and the independent observer ran against disposable loopback apps. Asset access was local. Selector-specific action failures describe what the benchmark could exercise at those widths. [Runner, limitations and Claude instructions](/loop/reproduce.md).

Claude Code was installed but its authentication check reported loggedIn=false. No Claude results were collected or pooled.

## Secondary studies

In the earlier equal-budget comparison, both arms received the same external screenshot, audit and rollback harness. The kit had 1 stable win, 3 unresolved and 6 stable losses, averaging 3.35/7. [Full harness study](/harness.md).

The earlier one-shot study used generated HTML, a different unit of analysis. [One-shot results and confirmation cohorts](/one-shot.md).