The builder test: Parts Desk

The Overview gave the result in four rows. This page is the test itself: what the agent was asked, what counted as done, and what happened in each run.

The task

A coding agent starts in an empty folder with this prompt and a documentation URL. Nothing else: no endpoint names, no field formats, no hints.

“I run a collision repair shop. Build me a small web app — a ‘Parts Desk’ — where my parts manager can open a repair job from a VIN, see recommended supplier offers with prices, place and confirm a parts order, watch the order status update live, and see the final reconciled invoice. The API and its documentation are at <sandbox URL>. Use only what’s documented there. It must actually work end to end, including verified webhooks.”

The prompt names zero endpoints and zero formats on purpose. Everything the agent needs has to come from the served documentation, so whatever it ships traces back to the docs. That isolation is the experiment.

What counts as done

A human clicks through the running app in a browser. Six outcomes, each stated as what the parts manager sees, never as an API step:

Open a repair job from a VIN, with the app provisioning its own API access on first run.
See the job’s damaged-parts list, and add to it.
See supplier offers with prices, grouped by supplier, selectable.
Place the order and have the supplying side lock it in.
Watch order status update live, with visible proof that each update came from the platform and was not forged.
See the final invoice, itemized, each line matched against what was ordered.

Two outcomes were deliberately left unsmoothed. Outcome 1 requires the agent to find and run the install handshake on its own; there is no dashboard and no hand-pasted key. Outcome 5 requires verifying webhook signatures, the most security-sensitive step in the integration. The documentation covers both. It just gives them no special signposting. Before any run, each outcome was traced to the documentation page that carries it, so a miss would be a documentation bug rather than an agent failure: Task spec and Protocol and metrics.

The two configurations

The same task ran twice: same sandbox, same model, same verbatim prompt. The server controlled which documentation surface existed, so the agent could not cheat its assignment:

Human-style docs. Rendered HTML pages only. The machine surfaces return 404.
Agent-native docs. The same content, plus raw markdown per page, /llms.txt, a typed /openapi.json, and a docs MCP with list, get, and search tools.

A middle configuration, markdown surfaces without the MCP, is part of the design but has not been run.

What happened

Human-style run: ~58 minutes active, 394 messages

Read the docs the only way it could: downloaded every HTML page, converted each to plain text, and reconstructed the contract before writing code.
Shipped a verified app clearing the same outcomes as the other run.
Along the way, isolated a real concurrency bug in the sandbox, proved it methodically, and built a client-side workaround.

Agent-native run: ~39 minutes uninterrupted, 186 messages

Asked the documentation questions instead: one listing call, then seven targeted page reads.
First successful API call about two minutes in.
Changed architecture mid-build when its first deployment target turned out unable to receive webhooks, and still finished without a pause.

Both apps ended at the same invoice, to the cent. Outcome-by-outcome scores, the bugs found, every caveat, and the raw session logs: The numbers.

Run it yourself

The test is repeatable in about five minutes.

Optional, but it recreates the agent-native configuration: give the agent the docs as tools.

claude mcp add --transport http partifact-docs \
  "https://partifact-docs-mcp.thanhvuttv.workers.dev/mcp"

In an empty folder, hand Claude Code the prompt above with the sandbox URL: https://partifact-mock.thanhvuttv.workers.dev.
Watch the opening move. With nothing hand-pasted, the agent should find the install handshake in the docs and provision its own credentials before building anything.

Connection options for claude.ai and Claude Desktop, and every test credential: Try it.

The operator test, or skip ahead to The numbers.