Nio vs direct provider calls — October 19, 2026
We repeated the October 5 benchmark on Nio
v0.1.249: the same coding task, the same eight models, two ways. Once through
Nio, and once as a direct call to the provider’s API. Nio produced a complete
app in 8 of 12 attempts; direct calls produced one in 1 of 11. Nio cost about
3.4 times as much in total, and less than half as much per complete app.
- Nio version: v0.1.249
- 23 arms across 8 models; Sonnet 5.5 and GPT-5 mini ran 3 times each
- Total API spend for the whole study: $15.87
Summary
| |
Through Nio |
Direct call |
| Complete apps (score 100) |
8 of 12 |
1 of 11 |
| Average score (out of 100) |
88 |
67 |
| Total cost |
$12.24 |
$3.63 |
| Cost per complete app |
$1.53 |
$3.63 |
| Total time, all arms |
125 min |
65 min |
Results by model
Scores are out of 100; 100 means every requirement was met. “Tests” is the
app’s own npm test, re-run independently after the benchmark. Costs use list
prices and include prompt-cache reads and writes.
| Model |
Mode |
Runs |
Score |
Tests |
Cost |
Time |
| Claude Opus 5.5 |
Nio |
1 |
100 |
56/56 pass |
$4.70 |
22.4 min |
| Claude Opus 5.5 |
Direct |
1 |
0 |
no app: response cut off at 32K tokens |
$0.66 |
4.5 min |
| Claude Sonnet 5.5 |
Nio |
3 |
100, 100, 100 |
all pass |
$1.06 avg |
10.3 min avg |
| Claude Sonnet 5.5 |
Direct |
3 |
76, 76, 0 |
1 run returned a cut-off response, no app |
$0.54 avg |
5.6 min avg |
| Claude Sonnet 4.6 |
Nio |
1 |
100 |
83/83 pass |
$1.76 |
19.0 min |
| Claude Sonnet 4.6 |
Direct |
1 |
72 |
61/61 pass |
$0.31 |
10.1 min (timed out) |
| GPT-5 mini |
Nio |
3 |
0, 100, 100 |
1 run stalled with no app |
$0.10 avg |
7.0 min avg |
| GPT-5 mini |
Direct |
3 |
93, 100, 88 |
1 run fails 1 test |
$0.08 avg |
6.4 min avg |
| GPT-4.1 mini |
Nio |
1 |
94 |
1/1 pass |
$0.09 |
2.3 min |
| GPT-4.1 mini |
Direct |
1 |
88 |
test suite crashes |
$0.06 |
2.4 min |
| GPT-5.1 |
Nio |
1 |
76 |
9/9 pass |
$0.36 |
4.4 min |
| GPT-5.1 |
Direct |
1 |
93 |
18/19 pass |
$0.49 |
5.4 min |
| Claude Haiku 4.5 |
Nio |
1 |
89 |
48/50 pass |
$1.10 |
20.0 min |
| Claude Haiku 4.5 |
Direct |
1 |
48 |
13/23 pass |
$0.26 |
6.7 min |
| Nio auto (mixed models) |
Nio |
1 |
100 |
26/26 pass |
$0.74 |
5.4 min |
Compared with earlier versions
Same task and harness each time. Average scores; “Nio cost” is the average per
run.
| Model |
Nio score |
Direct score |
Nio cost |
| Claude Opus 5.5 |
100 → 100 → 100 |
0 → 0 → 0 |
$4.15 → $4.97 → $4.70 |
| Claude Sonnet 5.5 |
100 → 100 → 100 |
71 → 51 → 51 |
$1.00 → $1.05 → $1.06 |
| Claude Sonnet 4.6 |
100 → 100 → 100 |
72 → 78 → 72 |
$1.55 → $1.88 → $1.76 |
| GPT-5 mini |
94 → 65 → 67 |
96 → 96 → 94 |
$0.23 → $0.10 → $0.10 |
| GPT-4.1 mini |
88 → 94 → 94 |
93 → 79 → 88 |
$0.12 → $0.19 → $0.09 |
| GPT-5.1 |
88 → 0 → 76 |
93 → 100 → 93 |
$0.65 → $0.26 → $0.36 |
| Claude Haiku 4.5 |
78 → 100 → 89 |
91 → 89 → 48 |
$2.21 → $1.57 → $1.10 |
| Nio auto |
93 → 100 → 100 |
— |
$1.46 → $0.86 → $0.74 |
Each cell reads v0.1.208 (October 5) → v0.1.244 (October 18–19) → v0.1.249
(October 19).
| |
v0.1.208 |
v0.1.244 |
v0.1.249 |
| Complete apps, Nio vs direct |
7/12 vs 1/12 |
8/12 vs 3/11 |
8/12 vs 1/11 |
| Average score, Nio vs direct |
94 vs 78 |
82 vs 72 |
88 vs 67 |
| Cost per complete app, Nio vs direct |
$1.97 vs $4.94 |
$1.67 vs $1.33 |
$1.53 vs $3.63 |
Where Nio helps
- Strong models finish the job. Sonnet 5.5 scored 100 in all nine Nio runs
across the three versions; the direct call never reached 100. Opus 5.5 and
Sonnet 4.6 scored 100 through Nio every time.
- Large tasks. A direct Opus 5.5 call ran out of room in one 32K-token
response and returned no app, as in every earlier run. This time direct
Sonnet 5.5 and Haiku 4.5 also returned cut-off responses. Nio builds the app
in steps.
- Lower cost per finished app. $1.53 through Nio against $3.63 direct.
- Fewer requests. Nio auto and Sonnet 4.6 reached 100 with 15 and 25
requests, down from 23 and 45 on v0.1.244.
- Repeated requests. When the identical feature request was sent again, Nio
reused its verified result with no model call in about 1 second in 4 of the 11
arms that reached the repeat. Direct calls paid $0.01–0.12 and 18–110 seconds
for each repeat.
- Prompt caching. 63–82% of Nio’s input was served from the provider’s
prompt cache, billed at a tenth of the input price.
Where it doesn’t, yet
- GPT-5 mini. One of three Nio runs stalled before writing the app and
scored 0; the other two scored 100. Direct scored 93, 100 and 88 for a bit
less money.
- GPT-5.1 scored 76 through Nio, leaving out several requested features,
against 93 direct.
- Total cost and speed. Nio spent 3.4 times as much as direct across all
arms and took about twice as long, up to 22 minutes against 4.5 for Opus 5.5.
We are looking into the GPT-5 mini and GPT-5.1 results next.
How it was run
- Task: build a dependency-free Node.js calculator web app, then add operator
precedence and parentheses, scientific functions, memory keys, a persisted
history, a theme toggle, accessibility and unit tests.
- Every arm received the build, the enhancement, two repair rounds, and one
identical repeat of the enhancement.
- Nio arms ran through a local Nio gateway with disposable Pro accounts.
Direct arms sent the same prompts to the provider API and wrote the files
from the response.
- The score comes from the benchmark rubric: required files, features found in
the source, and whether the generated tests pass.
Limits of this data
- Most models ran once per version. Only Sonnet 5.5 and GPT-5 mini have three
runs each, so a single good or bad run moves a model’s result a lot.
- Claude Haiku 4.5 ran directly once here, against twice on October 5, so this
study has 11 direct arms.
- The rubric checks source patterns and can miss or misjudge features.
- The direct baseline is the model answering in one response with no tool
loop, capped at 32K output tokens. It measures the raw model, not another
agent.
- Nio’s automatic mode mixes several models, so it has no direct counterpart.