Protocol & metrics, the developer eval

Raw working document, published unedited as evidence. Written for the run, not for reading flow; the readable account is on the experiment pages.

This is the measurement protocol for the Parts Desk developer eval: hand a coding agent an outcome-level product request and the sandbox URL, vary only the documentation surface the server exposes, and measure what each surface buys. It documents how the runs on the Results page were set up and scored.

Who built vs who ran. The sandbox, the docs, the two configurations, the request logging, and the task spec were built ahead of time; the eval itself was run separately, a human operator plus a fresh evaluator agent drove the task, and the build author neither ran the eval nor filled in any result. The bias caveats for each run are disclosed on the Results page.

The two configurations

Selected by the mock’s DOCS_MODE setting (one deploy or one local run per configuration). They bracket the spectrum:

Configuration	DOCS_MODE	Documentation surface exposed
HTML-only run, control	html	Rendered HTML pages only. Raw-markdown negotiation (Accept: text/markdown / .md), /llms.txt, /llms-full.txt, and /openapi.json all genuinely 404. Same content, good HTML, not strawmanned.
Docs-MCP run, full	full	The HTML pages + per-page raw markdown, /llms.txt, /llms-full.txt, /openapi.json, and the Docs MCP (list_docs / get_doc / search_docs) connected as a docs-reading helper.

The corpus is held byte-identical across both, so the comparison is the access channel, not a privileged copy. The surface is enforced server-side, never by the prompt, otherwise the experiment is invalid.

Outcome 5 (live, verified webhooks) against the deployed sandbox. The edge can’t reach a laptop’s localhost, but it can push real signed webhooks to a public https URL, so expose the built Parts Desk with a one-line tunnel (cloudflared tunnel --url http://localhost:<port>) and register that URL. This is the standard Stripe/GitHub local-webhook pattern. Against a local dev server, loopback delivery needs no tunnel.

The prompt (verbatim, identical across configurations)

Each configuration gets a fresh coding agent in an empty folder, the server-enforced documentation surface, and exactly this, and the sandbox URL, and nothing else:

“I run a collision repair shop. Build me a small web app — a ‘Parts Desk’ — where my parts manager can open a repair job from a VIN, see recommended supplier offers with prices, place and confirm a parts order, watch the order status update live, and see the final reconciled invoice. The API and its documentation are at <sandbox URL>. Use only what’s documented there. It must actually work end to end, including verified webhooks.”

The prompt names zero endpoints, zero lifecycle steps, zero formats, so a success is attributable to the docs, which is the claim under test.

The metrics, six columns

The output is one row per configuration. This is the canonical schema; the Results page reports the measured values, and flags the two columns still being attributed:

Configuration	Wall time	Token cost	Doc fetches + dead ends	Build→run→fix iterations	Errors by cause	Completion
HTML-only run, DOCS_MODE=html	,	,	,	,	,	,
Docs-MCP run, DOCS_MODE=full	,	,	,	,	,	,

Column definitions:

1. Wall time, start of the run to a human-confirmed working app. From the agent session clock.
2. Token cost, total tokens the build consumed. From the agent session’s usage report (not the docs logs).
3. Doc fetches + dead ends, how many documentation requests the agent made, which pages, and how many were dead ends (404s, e.g. a machine surface requested in the HTML-only run). Read from the server logs, never the agent’s self-report (see the extraction below).
4. Build→run→fix iterations, how many times the agent built, ran, hit a failure, and went back to fix. From the transcript.
5. Errors by cause, every iteration-costing failure bucketed into exactly one of three causes:
   • doc gap, the agent needed a fact the served docs do not carry (or carry only in a surface this configuration disables). Counts against the documentation.
   • API legibility, the fact was documented, but an API response (an error body, a field, a shape) was ambiguous or surprising enough to cost an iteration. Counts against the API surface.
   • agent error, the agent had everything it needed and still erred (a coding mistake, a misread). Counts against the agent, not the artifact. Recorded as <n> doc gap / <n> API legibility / <n> agent error.
6. Completion, of the six outcomes in the Parts Desk task spec, how many a human could complete in the browser (k/6), with outcomes 1 and 5, the planted friction points, called out explicitly.

Doc-fetch extraction (from server logs, not self-report)

Every docs / llms / openapi request logs one structured line carrying the resolved configuration:

{"evt":"docs_request","mode":"html","path":"/docs/quickstart","accept":"text/html","format":"html","status":200,"ts":1780000000000}

Reconstruct a configuration’s fetch profile from its captured log (html-only.log / docs-mcp.log; for a deployed sandbox, tail the Worker logs and parse each JSON message):

# pages fetched, by count (descending)
grep '"evt":"docs_request"' html-only.log | jq -r '.path' | sort | uniq -c | sort -rn

# dead ends (the machine surfaces a configuration doesn't expose, or genuine misses)
grep '"evt":"docs_request"' html-only.log | jq -rc 'select(.status==404) | {path,format}'

# total fetches + dead-end count
grep -c '"evt":"docs_request"' html-only.log
grep '"evt":"docs_request"' html-only.log | jq -rc 'select(.status==404)' | wc -l

Pass criterion (per configuration)

A configuration passes when the agent delivers a running app and a human reviewer clears all six outcomes in the browser, including a visible authenticity proof for outcome 5 (a badge or log line showing the live update was verified, not faked), touching nothing beyond starting the app. Anything less is recorded per-outcome (k/6), not as a binary fail: a partial result is itself a finding about the configuration’s format.

Pre-run sufficiency trace (every outcome → a doc the agent can reach)

Before any run, each of the six outcomes is traced to the specific served page that carries it, and that page is reachable from /llms.txt alone. An outcome that could not be traced would be a documentation bug, fixed before any run, never scored as an agent failure.

#	Outcome (from the task)	Carried by, page → /llms.txt-reachable path	What the page supplies
1 ⚠	Provision the app’s own access, no human	Authentication → /docs/authentication.md; Quickstart step 1 → /docs/quickstart.md	The install handshake (the one unauthenticated method), the demo client + single-use access code, the minted long-lived credential pair, and the two headers every other call carries. Planted friction: the path exists but the agent must find and follow it, no dashboard.
2	Show / add the parts to replace	Lifecycle → /docs/lifecycle.md; API Reference → /docs/api-reference.md	The parts-identification step (a full-list upsert), where it sits in the spine, and its request/response + error union.
3	Supplier offers with prices	Quickstart step 4 → /docs/quickstart.md; Lifecycle → /docs/lifecycle.md	The basket read, offers grouped by supplier, prices as bare strings (NZD ex-GST). Wire truth: the basket response carries no currency field, not on an offer, not on the envelope. Currency is a procurement-level concept that first appears on the procurement (and the invoice), never on a basket offer. Any currency shown beside a basket offer is derived for presentation, not read off the wire.
4	Place and lock in the order	Lifecycle → /docs/lifecycle.md; Quickstart steps 5–6 → /docs/quickstart.md	Placing the order (a clearly-labelled extension), then the supplier confirm that locks it in; the single-confirmable-supplier constraint and exactly how it surfaces.
5 ⚠	Live, authentic status	Webhooks → /docs/webhooks.md	Registering a consumer URL, the signed 8-field envelope, complete signature-verification code (TypeScript + Python), the 5-minute replay window, and the delivery boundary (loopback against a local server; a tunnelled public https consumer against the deployed sandbox). Planted friction: proving authenticity (not just polling) is the most security-sensitive, most-skipped step.
6	Settled, itemized invoice	Lifecycle → /docs/lifecycle.md; Quickstart step 7 → /docs/quickstart.md	Reading the reconciled invoice (a clearly-labelled extension) with each line’s match status.

Every page named above is linked from /llms.txt, so all six outcomes are reachable from /llms.txt alone.

Results

The developer eval has been run for both configurations, the HTML-only run and the Docs-MCP run, plus a runtime-agent consumer run. Every number is published on the Results, measured page, extracted from the raw .jsonl session logs (downloadable there), with the built apps, scorecards, and narrative transcripts hosted alongside as the Evidence pages.