Nio

Nio v0.1.236: half the cost per finished task — October 15, 2026

On the same two tasks, Nio v0.1.236 finishes a task for $0.23 on average, down from $0.47, and every run still passes all of its checks. Each run ends only after an independent review and a passing build (“Nio CI: passed”). Nothing stops early to look cheap.

• Before: a v0.1.235 build with the approval fixes from this release, so tasks finish instead of pausing for approval (run 2026-10-15T21-51-53)
• After: Nio v0.1.236 (run 2026-10-15T23-40-32)
• Harness: our nio-vs-raw benchmark (fix a broken Next.js site; build an app from a spec), 3 runs of each task, Nio arm, automatic routing
• Models used, as reported by the gateway: Claude Sonnet 5.5 (builder), GPT-5.1 (reviewer); no Claude Opus call in the v0.1.236 runs
• Total API spend for this study (three full benchmark runs): $5.84

Results

Costs are list prices, including prompt-cache reads and writes.

Task	Checks passed	Cost per run	Model requests	Time
Fix a broken website (fix-site)	8/8 in 3 of 3 → 8/8 in 3 of 3	$0.76 → $0.36	25 → 12	288 s → 227 s
Build an app from a spec (build-app)	11/11 in 3 of 3 → 11/11 in 3 of 3	$0.18 → $0.11	13 → 7	124 s → 94 s
Per finished task	6/6 → 6/6	$0.47 → $0.23 (−51%)

Every run (before and after) passed on its first turn. Every v0.1.236 run ended with “Nio CI: passed”; one earlier run had ended before its CI gate.

Single runs, v0.1.236:

Task	Run 1	Run 2	Run 3
fix-site	$0.370 · 12 requests	$0.360 · 13 requests	$0.354 · 12 requests
build-app	$0.132 · 7 requests	$0.093 · 7 requests	$0.094 · 6 requests

Against a raw model

No raw-model arm ran in this study. In the October 12 run of the same harness (v0.1.226), a direct Claude Sonnet 4.6 call did not fix the website in three turns ($0.66; checks 5/8, 0/8, 7/8), while Nio fixed it on the first turn for $0.64. Nio now does that task for $0.36.

For a small app built from scratch, a direct call is still cheaper: about $0.07 against Nio’s $0.11. Nio’s extra cost there pays for running the build and an independent review.

Where the requests went, and what changed

We read all 115 model replies from the “before” run. Nearly half of the fix-site requests did no useful work:

• Tool calls written in another format were dropped. 13 of 115 replies started with a call in the model’s native XML form; Nio skipped it and the model had to send it again. Nio now runs both forms.
• Command output was hidden. Page checks such as status=200 nav=1 were redacted as if they were secrets, so the model re-ran them to read the numbers. Plain counts and true/false flags now stay readable; settings values are still redacted.
• The reviewer raised false alarms. It reads code only, and on every fix-site run it reported a duplicated header that the page checks had already shown appears once. The builder then spent 6 to 10 requests re-checking. The reviewer now sees what the running app returned, and a rejected point is answered in one sentence.
• The most expensive model was called when nothing was wrong. Re-checking work that already passes no longer brings in the planner (Opus); it cost about $0.15 per fix-site run.
• A stale error blocked the finish. A page check that failed while the server was restarting stayed “open” after later checks passed. All checks of the running app now count as one.
• Smaller repeats removed: items the reviewer already checked are not asked about again; a web-tool fetch of localhost runs from the shell instead of failing; Nio restarts a Next.js server after a build itself; and a file whose markup sits only in a string (an RSS helper) is no longer treated as a page that needs restyling.

Approvals are unchanged: commands that delete files or stop programs by name still ask first. A set of 177 shell commands written by models in these benchmarks is checked offline on every build.

Limits

• Two tasks, three runs each; build-app costs vary by about ±25% on identical code.
• On every fix-site run the reviewer still repeats one false alarm. It now costs one request instead of up to ten.
• The raw-model comparison comes from an earlier version (v0.1.226).

This site is open source. Improve this page.