The numbers
The builder test and operator test pages tell what was run. This page reports what the logs say: builder runs on 7 June 2026, operator run on 5 June 2026.
Builder test
Section titled “Builder test”| Metric | Human-style docs | Agent-native docs |
|---|---|---|
| Zero to first successful API call | not measured | ~2 min |
| Zero to verified end-to-end app | ~58 min active ¹ | ~39 min uninterrupted ² |
| Assistant messages | 394 | 186 |
| Tool calls / tool errors | 165 / 6 | 83 / 2 |
| How it read the docs | downloaded every HTML page, converted to text, read it all | 1 listing call + 7 targeted page reads |
| Direct API probe calls | 17 | 6 |
| Outcomes | 4 pass, 2 partial | 4 pass, 2 partial |
| Final invoice | $1,099.00 NZD | $1,099.00 NZD |
¹ Split 19 + 39 minutes by an unrelated usage-limit pause. The only human message in the window was “continue”.
² Includes discovering that the first deployment target could not receive webhooks, and the pivot to a local server with a tunnel.
The six outcomes
Section titled “The six outcomes”Both runs scored identically: outcomes 3 to 6 pass (offers with prices, supplier-locked order, live verified status, reconciled invoice), outcomes 1 and 2 partial.
- Outcome 1, self-provisioned credentials: partial. Both agents performed the install handshake on the wire during probing, then shipped the seeded demo credentials in the app. The root cause sits in the contract, not the agents: the install call returns no webhook secret, so a self-provisioned integration cannot verify webhooks. Outcomes 1 and 5 are jointly unsatisfiable as specified. Both agents found the same workaround independently.
- Outcome 2, the manager can add parts: partial. Neither app built it. The verbatim prompt never asks for it; the scoring rubric does. A rubric and prompt mismatch on our side, recorded as such.
Per-run scorecards: human-style · agent-native.
Defects the test surfaced in the sandbox
Section titled “Defects the test surfaced in the sandbox”Three real bugs, each found by an agent in normal use, none previously known:
- The webhook-secret gap described above. Fix is on the contract backlog: the install call should return, or allow retrieving, a webhook secret.
- A lost-write race. Concurrent requests against one sandbox could lose updates; a placed order was acknowledged, fired its webhooks, then read back as not found. The human-style run isolated it methodically and built a client-side workaround.
- A crash on confirm when an order overlaps parts that were already ordered.
Operator test
Section titled “Operator test”- Full lifecycle completed unaided, from tool descriptions alone, ending at a reconciled invoice with every line matched. (This run also confirmed the pre-seeded procurement, so its transcript shows two invoices totalling $3,387.00 NZD; later runs confirm only their own $1,099.00 order.)
- Recovered from the planted mixed-supplier 422 using only the error message.
- Flagged the two beyond-contract steps each time it used them.
- Fed three fixes back into the sandbox: supplier seed coverage made visible in the tools; 422 messages now lead with the specific violation; currency removed from the basket, since it belongs to the procurement (Real vs. simulated).
The run’s caveat and its mitigation are on the operator test page. Full play-by-play: run transcript.
Read before quoting
Section titled “Read before quoting”All of these are true at once; none of them erase the result.
- One run per configuration, same human operator, same model, about 11 hours apart. Directional, not statistical.
- The human-style run’s clock includes the usage-limit pause and time spent debugging the two genuine sandbox bugs its approach happened to trigger.
- The agent-native run had two small environmental advantages: one human answer to its own clarifying question about deploy target, and a locally installed deployment skill in context. Both became irrelevant when that deploy path was abandoned.
- Deployment was a post-task addendum in both runs. The human-style run’s deploy phase reused a finding from the other session; the contamination is confined to that addendum.
- Verification is asymmetric. The human-style run’s outcomes were re-verified by an independent agent driving the live app; the agent-native run’s rest on its in-session checks (9 of 9 assertions), pending the same treatment.
- One sandbox produced zero webhook deliveries before a fresh one delivered 4 of 4 verified. Unexplained, under investigation.
- The narrative transcripts are the agents’ own writing. Every number on this page comes from the raw JSONL logs instead.
A middle documentation configuration (markdown surfaces without the docs MCP) is designed but not run. The table stays incomplete rather than extrapolated.
Provenance
Section titled “Provenance”Each run is published here in full: the filled scorecard, the agent’s narrative transcript, and the raw JSONL session log that is the source of every number above. The built app source lives in a private repository, available on request.
- Human-style run: scorecard · transcript · raw session log
- Agent-native run: scorecard · transcript · raw session log
- Operator run: transcript
The seeded invoice total, $1,099.00 NZD, is deterministic. Run either test yourself and a different total is a bug report we want.