Nio v0.1.210 vs v0.1.209 — October 6, 2026
v0.1.210 raises the average score on our benchmark tasks from 70 to 81
and lowers Nio’s API cost by 15%, with no task getting worse on
Claude Sonnet 5.5. Most of the gain comes from bugs the benchmark itself
exposed; they are listed below.
- Before: Nio v0.1.209 (calculator: v0.1.208 3-run averages, same pinned models)
- After: Nio v0.1.210
- Models: Claude Sonnet 5.5 and GPT-5 mini, pinned
- Total API spend for this study: $7.86
Results
Scores are out of 100 (100 = every requirement met). Costs are list prices
including prompt-cache reads and writes.
| Task |
Sonnet 5.5 before → after |
GPT-5 mini before → after |
| Calculator: build, then extend |
100 → 100 · $1.00 → $0.95 · 7.4 → 6.1 min |
94 → 100 · $0.23 → $0.18 · 14.6 → 10 min |
| Fix 3 bugs in an existing project |
100 → 100 · 1.7 → 0.7 min |
25 → 35 (other runs: 100, 25) |
| Consolidate a function copied in 4 files |
100 → 100 · $0.23 → $0.11 · 1.4 → 0.6 min |
100 → 100, 100 |
| Python REST API with tests |
34 → 76 (88 in another run) |
10 → 40 |
Average over the 8 arms: 70 → 81. Nio’s total cost for the 8 arms:
$2.22 → $1.88.
What changed
- No “done” while tests fail. Nio tracks the checks it runs. If the model
finishes while one is red, Nio sends it back (on the stronger model from
the second time) or ends with “Status: NOT DONE” naming the failing check.
- Requirement checklist. Items listed in the request are checked against
the code before finishing; missing ones are worked on or reported as
“Possibly incomplete”.
- Earlier escalation to the stronger role model when a check stops
improving. BYOK stays on the user’s own keys.
- Better OpenAI prompt caching: stable instructions first and a
prompt_cache_key for api.openai.com.
- Plan tokens weighted by model price, so cheap models use less of an
allowance and expensive ones more. Plan prices and sizes are unchanged.
Takes effect when the hosted gateway runs v0.1.210.
- Release gate: Nio’s offline CLI test suites run on every release and
block publishing if one fails.
Limits
- Most arms ran once; GPT-5 mini varies run to run.
- GPT-5 mini through Nio is still weak on the Python task (40) and uneven on
the bug-fix task. On these small tasks, a direct GPT-5 mini call scored
100 for less money.
- Nio still leaves servers it started during a task running afterwards.