Through v0.1.251, GPT-5.1 did worse through Nio than called directly on our calculator benchmark (70–78 against 93–100). It answered with plans instead of making the changes, and reran failing tests without editing them.
The cause was a setting, not the model. GPT-5.1 is the first OpenAI model that
defaults to no reasoning in the API, and Nio didn’t set it, so every step of
a Nio task ran without reasoning. In v0.1.252, Nio’s agent turns ask GPT-5.1
(and later GPT-5.x point releases) for medium reasoning, as they already do
for Claude, and give each of its replies up to 6 minutes instead of 2.
Calculator task, three runs each. The direct calls also used medium
reasoning and the same 6-minute limit per request, so both sides think the
same amount. Scores are out of 100; 100 means every requirement was met.
| Run 1 | Run 2 | Run 3 | Average | Complete | Cost per run | |
|---|---|---|---|---|---|---|
| Nio v0.1.252 | 100 | 100 | 94 | 98 | 2 of 3 | $1.79 |
| Direct call | 100 | 100 | 72 | 91 | 2 of 3 | $0.64 |
| Nio v0.1.251 (no reasoning) | 65 | 76 | 69 | 70 | 0 of 3 | $0.87 |
Same calculator task and harness as the October 19 benchmark: the build, the enhancement, two repair rounds and one repeat of the enhancement. Three runs per side, run side by side. Three runs are still few: one run can move the average by ten points or more.