# Omni UI Harness v0 Pilot Results

Updated: 2026-06-28

## Current Result

Latest iteration: `ui-reporting-repairmode-k1-20260628T012257Z`

Task: `reporting-dashboard`

Mode: build-first single builder + same-builder gate repair loop (`fullMode=repair`)

Cell: `claude:sonnet/medium`

Verdict after corrected geometry scoring: `revise_full_harness`

The repair-mode direction is promising but not shippable. Full/sculptor cleared
raw's source/style failures and showed much better process discipline, but the
mobile screenshot still has visible KPI/table overlaps. The scorer now treats
rendered overlap, viewport overflow, and clipping as severe geometry defects
instead of soft flags.

Corrected score read:

| Metric | raw_ui_agent | full_ui_harness | Full - raw |
| --- | ---: | ---: | ---: |
| Corrected effective | 50 | 60 | +10 |
| Recorded effective before severity fix | 50 | 76 | +26 |
| Judge blend | 74 | 76 | +2 |
| Code judge | 79 | 79 | 0 |
| Visual judge | 66 | 69 | +3 |
| Severe/static blocking | 20 | 5 | -15 |
| Flags | 25 | 23 | -2 |
| Self-gate runs | 0 | 5 | +5 |
| Turns | 8 | 19 | +11 |
| Cost | $0.3604 | $0.5780 | +$0.2176 |

Conclusion: build-first + self-gating is a better direction than read-first
contract mapping, but the full harness still needs a real mobile-overlap repair
pass before it can be considered a UI-quality lift.

Artifacts:

- Analysis: `runs/arena/ui-harness/ui-reporting-repairmode-k1-20260628T012257Z/analysis.json`
- Active scores: `runs/arena/omni-ui-harness-v0-reporting-dashboard.scores.json`
- Screenshots: `runs/arena/ui-harness/ui-reporting-repairmode-k1-20260628T012257Z/{raw_ui_agent-s1,full_ui_harness-s1}/{desktop,mobile}.png`

## Previous Proof-Gate Result

Iteration: `ui-patient-intake-proofgate-20260627T_go`

Latest rescore: `rescore-20260627T194100Z-proofgate-fixed`

Verdict: `contract_failed_after_rescore`

The active result is now the proof-gate iteration. The full arm passed the
executable duplicate-before-create proof, but the result is gate-enforced
compliance, not a promotion. The ungated payer identity contract failed, and
the broader mobile replay proof still needs hardening.

Current write-up:
`docs/v2/OMNI-UI-HARNESS-V0-PROOFGATE-ITERATION.md`.

## Previous Mutation-Boundary Iteration

Run: `ui-patient-intake-mutation-boundary-20260627T_go`

Rescore: `rescore-20260627T173700Z-mutation-boundary-fixed`

Verdict: `contract_failed_after_rescore`

The mutation-boundary prompt rail did not work: full/sculptor still missed
proactive duplicate lookup and tied the frozen raw baseline at `50`. It did,
however, leave us with a corrected critical-contract scorer.

## Previous Single-Builder Rerun

Run: `ui-patient-intake-singlebuilder-contract-20260627T150851Z`

Rescore: `rescore-20260627T161100Z-golden-driverfix3`

Verdict: `tie_no_lift_after_rescore`

The single-builder contract-discovery rerun showed that full/sculptor improved
UI/process behavior in one seed, but both candidate arms still failed the
critical pre-create duplicate contract. The golden reference proved that
contract was reachable and exposed a scorer rank inversion that the later
mutation-boundary rescore fixed.

## Previous Comparable Run

Run: `ui-patient-intake-20260627T035800Z`

Task: `patient-intake`

Arms: `raw_ui_agent` vs `full_ui_harness`

Cell: `claude:sonnet/medium`, one seed per arm, symmetric post-run lab gates, code judge, visual judge, and rendered screenshots.

Verdict: `revise_harness_and_task_floor`

This is a clear comparable `n=1` run, not a promotion result. Quality is not yet measurable because the visual proof is blind to the headline duplicate-candidate workflow, and the code judge preferred raw on this seed. The full harness's defensible win is process and cost only: fewer UI lint flags, more self-gates, rendered self-gates, fewer turns, and lower cost.

Updated rescore: `docs/v2/OMNI-UI-HARNESS-V0-RESCORE-RESULTS.md`.

The improved testing harness replayed the saved candidates through duplicate, invalid-format, long-payer, and create-error browser states. That rescore broke the old tie: raw won the product-contract result, while full won the UI-adherence lane.

## Score Read

| Metric | raw_ui_agent | full_ui_harness | Full - raw |
| --- | ---: | ---: | ---: |
| Calibrated effective | 71 | 72 | +1 |
| Original strict effective | 50 | 50 | 0 |
| Judge blend | 71 | 72 | +1 |
| Code judge | 71 | 66 | -5 |
| Visual judge | 71 | 82 | +11 |
| Blocking issues | 0 | 0 | 0 |
| Non-blocking flags | 16 | 4 | -12 |
| Self-gate runs | 3 | 7 | +4 |
| Rendered self-gate runs | 0 | 3 | +3 |
| Turns | 52 | 38 | -14 |
| Cost | $2.2076 | $1.8629 | -$0.3447 |

The original strict effective score floored both arms to 50 because non-blocking UI flags made the strict lab gate fail. Opus called that a floor artifact, so the analysis now reports calibrated effective separately: floor only on blocking static/rendered issues, eslint failure, mount failure, leak, or infra error. Both scoring methods agree on the important point: strict scoring says tie at 50, calibrated scoring says +1 for full, so neither supports a quality-lift claim.

## What The Run Actually Shows

- Full harness process discipline improved: fewer flags, more explicit gates, more rendered self-checking, lower cost, fewer turns.
- Full harness quality did not clearly improve: the judge blend delta is only +1 and the code judge preferred raw by 5 points.
- The visual judge preferred the full harness by 11 points, but that is confounded by missing duplicate-candidate screenshots. The visual judge saw calmer layout and fewer visible defects, not the core duplicate-detection workflow.
- Raw appears to preserve more real task wiring in code on this seed, including duplicate pre-check style behavior and DOB/age handling. Full is cleaner visually but riskier functionally.

## Missing Proof

- `n=1` per arm, so this remains a methods shakedown.
- No screenshot state renders a duplicate candidate, even though that is the headline requirement for the task.
- The invalid-submit screenshots prove required-field handling, but not populated invalid DOB/email/zip/phone validation.
- Non-blocking flags are useful diagnostics, but should not hard-cliff the score.

## Artifacts

- Final HTML report: `docs/ui-harness-ui-patient-intake-20260627T035800Z.html`
- Final analysis JSON: `runs/arena/ui-harness/ui-patient-intake-20260627T035800Z/analysis.json`
- Active dashboard scores: `runs/arena/omni-ui-harness-v0-patient-intake.scores.json`
- Discarded process-timeout iteration report: `docs/ui-harness-ui-patient-intake-20260627T031211Z.html` (kept as evidence, but not used as the current result because the full arm timed out after gates and lost cost/turn capture)

## Next Harness Change

Before spending on more seeds or more tasks:

1. Add a duplicate-candidate proof state for `patient-intake`.
2. Add a populated invalid-format state for DOB, email, zip, and phone.
3. Keep non-blocking UI lint flags as diagnostics or score shading, not as the hard effective-score floor.
4. Then rerun `patient-intake` raw vs full. If the proof states are solid, expand to the next UI task slice.