Scorecard, HTML-only run (DOCS_MODE: html)
Raw working document, published unedited as evidence. Written for the run, not for reading flow; the readable account is on the experiment pages.
Model: claude-opus-4-8 · Date: 2026-06-06/07 · Folder: p-app-html-only
Verification: outcomes 3–6 independently re-verified by a second agent driving the running app (fresh tenant, 2026-06-07).
Build-record facts referenced below: baskets carry no currency (a procurement-level field); only one supplier (Christchurch Toyota) can confirm orders; repairer.jobs.parts.insert returns bare-string errors.
Six outcomes
Section titled “Six outcomes”| # | Outcome | Score | Evidence |
|---|---|---|---|
| 1 | Start job from VIN; app self-provisions its own API access | Partial | VIN → tier1.prepare → job open works (verified live). integrations.insert exercised on the wire during probing (16:11), but the shipped app hardcodes seeded demo creds. Root cause: install returns only {api_key, integration_id}, no webhook secret, so a self-provisioned integration cannot verify webhooks (see Findings). |
| 2 | Show parts to replace; let the manager add to the list | Partial | 8 parts shown with status (ordered/estimated). No add-part UI (parts.insert referenced only in comments). Note: the verbatim task prompt never asks for adding parts, rubric/prompt mismatch. |
| 3 | Supplier offers with prices, grouped & selectable | Pass | 12 offers / 3 suppliers, confirmable-supplier marking, no-currency-on-basket currency labelling. Verified live. |
| 4 | Place and lock in the order (supplier-committed) | Pass | place → order_requested → supplier confirm → order_confirmed & locked. Single-confirmable-supplier guard enforced app-side. Verified live. |
| 5 | Live, trustworthy status (verified webhooks) | Pass (local+tunnel) | 4/4 HMAC deliveries “signature verified • replay-window OK” in independent re-run; verifier does raw-bytes HMAC, constant-time compare, ±5-min window, message_id dedup. Caveats: one earlier tenant produced zero deliveries (flakiness, unresolved); workers.dev deploy is UI-only (Cloudflare 1042, honestly surfaced in-app). |
| 6 | Settled invoice, itemized, matched | Pass | $1,099.00 NZD, lines matched, matched. Total reproduced across ≥3 independent runs, deterministic. |
Pass count: 4 / 6 (2 partial, 0 fail)
Process metrics (core build window 16:06:58 → 18:40:05 UTC)
Section titled “Process metrics (core build window 16:06:58 → 18:40:05 UTC)”| Metric | Value |
|---|---|
| Active build time | ~58 min (19 min + 39 min, split by a Claude usage-limit pause 16:26 → 18:01) |
| Wall-clock to “verified end-to-end” | 2 h 33 m (includes ~1 h 35 m quota outage) |
| Assistant messages | 394 |
| Tool calls / errors | 165 / 6 |
| Doc ingestion | machine surfaces 404 (mode enforced ✓); downloaded all rendered HTML pages, converted via textutil, read as text |
| Direct API probe calls | 17 |
| Human intervention in core window | 1 × “continue” (resumption after usage limit, no steering) |
Out-of-scope addendum (not part of the measured task): README on request; Cloudflare deploy 03:44–04:27 next day, which reused the 1042 finding from the Docs-MCP run (cross-contamination, deploy phase only).
Bugs found in the sandbox by this run
Section titled “Bugs found in the sandbox by this run”- Lost-write race: concurrent requests against one tenant lose updates (placed order acknowledged + webhooks fired, then
not_found). Worked around with a client-side serialization gate + place-and-verify. - Confirm crash on overlapping ordered parts (
non_json_response, order rolls back).