The Parts Desk task spec

Raw working document, published unedited as evidence. Written for the run, not for reading flow; the readable account is on the experiment pages.

This spec is deliberately excluded from the sandbox docs corpus the agent reads, so the agent under test can never read the answer key. The agent receives only the short outcome prompt in §2 plus the sandbox URL, never this spec. The measured runs and results are on The numbers; the protocol, metrics, and pass criteria are in Protocol and metrics.

1. What this measures

Earlier runs showed that an AI agent can operate the parts platform at runtime (see the operator run transcript). This task measures the other consumer class: can a coding agent build a working integration from the documentation alone?

The agent is handed a product to build, described purely as product outcomes, no endpoints, no method names, no headers, no data formats. If it ships a working app, the documentation carried it there. That isolation is the entire experiment, so the task is phrased the way a shop owner would phrase it, and nothing more. The same task runs against the same sandbox under two documentation configurations, with only the documentation surface varied by server config (§6). The comparison is the evidence.

2. The task (verbatim prompt to the agent)

Each run gives a fresh coding agent, in an empty folder, exactly this prompt and the sandbox URL, and nothing else:

“I run a collision repair shop. Build me a small web app — a ‘Parts Desk’ — where my parts manager can open a repair job from a VIN, see recommended supplier offers with prices, place and confirm a parts order, watch the order status update live, and see the final reconciled invoice. The API and its documentation are at <sandbox URL>. Use only what’s documented there. It must actually work end to end, including verified webhooks.”

The prompt names zero endpoints, zero lifecycle steps, zero data formats. Everything needed to turn it into a working integration must come from the served docs.

3. The six outcomes (how a human judges “done”)

“Done” is judged by a human clicking through the running app in a browser, not by reading its code. A finished Parts Desk delivers all six outcomes below. Each is stated as a user-visible result, never as an API step.

1. Start a job from a vehicle. ⚠ Planted friction, see §5. A parts manager types in a vehicle identifier (and the shop’s own claim / reference numbers) and the app opens a repair job for that vehicle. The app must obtain its own access to the parts platform on first run, with nothing hand-copied, no keys pasted in by hand, no manual dashboard click, no human walking through a portal to provision it.
2. Show the parts to replace. The job’s list of damaged parts is displayed, and the manager can add to it.
3. Show supplier offers with prices. The app fetches and shows what suppliers offer for those parts, grouped by supplier, each offer with a visible price, and lets the manager pick which offers to buy.
4. Place and lock in the order. From the picked offers, the app places the parts order and gets it locked in by the supplying side, so it is a real, committed order and not just a draft request.
5. Live, trustworthy status. ⚠ Planted friction, see §5. When the order’s state changes, the Parts Desk updates to the new state on its own, without the manager reloading the page, and the app can show that each such update genuinely came from the parts platform and was not forged, surfaced as a visible trust signal (a badge, an indicator, or a log line the human can point at).
6. Show the settled invoice. The app displays the final invoice for the order, itemized, with each line showing whether it matched what was ordered.

4. Rules of the run

• The agent reads only the served documentation. For the configuration under test, the only permitted source of contract knowledge is the documentation the sandbox serves at its docs URL (and the machine-readable surfaces it links). No other web sources, no transcribing from memory, and no reading this sandbox’s source repository.
• The app is the integration, it must call the platform itself. The Parts Desk has to talk to the parts platform’s API directly (write the client). It must not offload the procurement workflow to the platform’s runtime agent-facing connector; that is a different consumer surface, measured separately in the operator test. This task measures building a client from the docs. (Using a docs-reading helper to navigate the documentation is fine and is exactly what the agent-native configuration provides.)
• It must actually run. Not a mockup, not stubbed responses, a real app a human can drive end to end against the live sandbox. A human evaluator clicks through all six outcomes in a browser; nothing beyond starting the app may be done at a terminal.
• Measurement comes from logs, not self-report. Wall time, doc pages fetched, dead ends, build-run-fail iterations, and errors-by-cause are read from the sandbox’s server logs and the session transcript, not from the agent’s own summary. (The sandbox logs every documentation request with the configuration that served it; see Protocol & metrics.)

5. The two planted friction points (do not smooth them away)

Outcomes 1 and 5 are the two places where human-first documentation typically loses an integrator. They are kept as honest friction so the experiment can measure whether each configuration’s format gets the agent across them. They are not tricks, the docs are sufficient to clear both (§7), they are simply not pre-chewed.

• Outcome 1, provisioning without a human. The agent must get the app its own credentials programmatically, on first run. There is no dashboard and no hand-pasted key; the path exists in the docs but the agent has to find and follow it. Human-first docs usually assume a person clicks a console here.
• Outcome 5, proving the live update is authentic. “Status updates live” is easy to fake by polling and pretending. The bar is higher: the app must show that an update truly originated from the platform and was not fabricated by the app, and prove it, visibly. The verification recipe is fully documented, but only if the agent reaches that page; this is the single most security-sensitive thing an integrator must get right, and the most commonly skipped.

6. The two configurations (how the documentation surface is varied)

The two configurations differ only in what documentation surface the server exposes, server-enforced, never prompt-enforced (otherwise the experiment is invalid). They are selected by the sandbox’s DOCS_MODE config, and they bracket the spectrum:

• HTML-only run, DOCS_MODE=html (control): human-rendered HTML pages only. No raw-markdown negotiation, no /llms.txt, no machine-readable spec, the machine surfaces genuinely return 404. Same content, good HTML; it is not strawmanned.
• Docs-MCP run, DOCS_MODE=full (agent-native): the same HTML pages plus the full agent-native surface, raw markdown per page, /llms.txt + the full-corpus digest, the machine-readable /openapi.json, and a small docs-reading helper (the Docs MCP) the agent can connect to navigate and search the documentation.

The corpus is held byte-identical across both, so the comparison is the access channel, not a privileged copy. The exact deploy/run instructions per configuration are in Protocol & metrics and the README.

7. Honesty: the docs are sufficient, not smoothed

Every one of the six outcomes is reachable from the served documentation, each outcome was traced to the specific page (and machine-readable surface) that carries it before any run; that sufficiency trace is recorded in Protocol & metrics. The docs are deliberately not smoothed past the two friction points in §5: if a configuration’s agent cannot deliver an outcome, that is a finding about the documentation format, not a gap engineered into the content. An outcome that turned out to be untraceable to a doc would be a documentation bug, fixed before any run, not scored as an agent failure.

8. How it’s judged

Per configuration: did the agent deliver a running app that clears all six outcomes under a human’s hands, and at what cost (the metrics in §4)? The output is the comparison table (published on the Results, measured page), plus per-configuration notes and written findings. The central question the table answers: which documentation investments actually move a coding agent from prose to a working, secure integration, and by how much?

The build produces everything this task needs to be runnable (the sandbox, the docs, the two configurations, the request logging, this spec, and the empty results scaffold). It does not run the eval or fill in any result, a human evaluator and a fresh evaluator agent run it afterward, and the build author neither runs the eval nor fills in any number.