Scorecard, Docs-MCP run (DOCS_MODE: full, docs MCP)
Raw working document, published unedited as evidence. Written for the run, not for reading flow; the readable account is on the experiment pages.
Model: claude-opus-4-8 · Date: 2026-06-07 · Folder: p-app-docs-mcp
Verification: outcomes scored from in-session Playwright verification (9/9 assertions) + transcript; live re-verification by a second agent pending (tunnel is ephemeral, restart with npm start to re-test).
Build-record facts referenced below: baskets carry no currency (a procurement-level field); only one supplier (Christchurch Toyota) can confirm orders.
Six outcomes
Section titled “Six outcomes”| # | Outcome | Score | Evidence |
|---|---|---|---|
| 1 | Start job from VIN; app self-provisions its own API access | Partial | VIN → tier1.prepare → job open works (Playwright-verified). integrations.insert exercised on the wire at 03:00, but the shipped app hardcodes seeded demo creds. Root cause: install returns no webhook secret, self-provision and verified webhooks are jointly unsatisfiable (see Findings). |
| 2 | Show parts to replace; let the manager add to the list | Partial | Parts shown; no add-part UI (parts.insert only in a comment). Rubric/prompt mismatch: the verbatim prompt never asks for adding parts. |
| 3 | Supplier offers with prices, grouped & selectable | Pass | 12 offers / 3 suppliers, confirmable-supplier marking, no-currency-aware labels (per transcript + Playwright). |
| 4 | Place and lock in the order (supplier-committed) | Pass | place → confirm → order_confirmed; the single-confirmable-supplier guard refuses a non-confirmable supplier up front (unprocessable). |
| 5 | Live, trustworthy status (verified webhooks) | Pass (local+tunnel) | Buyer + supplier events received and verified over SSE. Standout: 8 crafted negative tests (tampered body, stale ts, duplicate, missing sig, unknown integration, wrong secret) all rejected correctly. Found + fixed a real subscription-collapse quirk (same URL for both integrations collapses to one, distinct query params fix it). |
| 6 | Settled invoice, itemized, matched | Pass | $1,099.00 NZD, lines matched, matched, same deterministic total as the HTML-only run. |
Pass count: 4 / 6 (2 partial, 0 fail)
Process metrics (core build window 02:58:08 → 03:37:29 UTC)
Section titled “Process metrics (core build window 02:58:08 → 03:37:29 UTC)”| Metric | Value |
|---|---|
| Active build time | ~39 min, uninterrupted, including a full Cloudflare Worker build, the 1042 discovery (03:19), and the pivot to local+tunnel |
| Assistant messages | 186 |
| Tool calls / errors | 83 / 2 |
| Doc ingestion | 1 × list_docs + 7 × get_doc (docs MCP) + 1 WebFetch, targeted retrieval, no scraping |
| Direct API probe calls | 6 |
| Human intervention in core window | 1 × AskUserQuestion (deploy target; human chose “Cloudflare Worker”); 1 × local durable-objects skill auto-loaded (env advantage; discarded with the Worker pivot) |
Findings unique to this run
Section titled “Findings unique to this run”- Cloudflare 1042 discovered and proven in both directions (Worker can neither call the same-zone mock nor receive its webhooks), with the reverse-direction test that the HTML-only run’s later deploy phase reused.
- Webhook subscription collapse when both integrations register the same URL.