I measured whether a spec file makes the model more accurate. It doesn't.

In 18 out of 30 cells, the arm with spec structured in markdown and the arm with exactly the same information in continuous paragraph produced the same result. In the slugify task with devstral:24b at temperature 0, both returned byte-by-byte identical code.

In July I wrote that replacing the prose prompt with a spec file — contract in table, edge cases, out of scope, acceptance criteria — made the agent get it right more often. I closed that post admitting that the gain was my hypothesis, with the table cells marked to fill later.

Now they are filled. And they refute the hypothesis.

What changes the result is the information being written down. Between a one-sentence request and the same task specified, edge cases jump from 46% to 90%. Between the structured spec and the same information in continuous text, the difference disappears within the noise.

The middle arm is the experiment

Comparing “one sentence” with a “sixty-line spec” doesn’t answer anything. More information helps — that’s obvious, not measurement. That’s why there are three arms, and the middle one is what gives meaning to the rest:

• prose — one-sentence request, only happy path, no edge cases mentioned.
• long prose — exactly the same information as the spec, in continuous text, without structure.
• spec — the same information in structured markdown: contract in table, edge cases, out of scope, acceptance criteria.

Thus prose against long prose measures information, and long prose against spec measures form — which is the claim I needed to sustain and didn’t sustain.

The rest of the design exists to rule out easy outs. The tests for each task are separated into happy path and edge cases, and each edge case corresponds 1:1 to an item that the spec declares and the short prose omits. The output format instruction is identical in all three arms — if the requested format varied along with the content, I would be measuring format. Each task brings a reference implementation that needs to pass 100% of its own tests before the run begins; if the reference fails, the test is wrong. The generated code runs in an isolated process with a 10s timeout because model code isn’t trustworthy: it enters loops, calls process.exit, blows up memory.

The entire harness is in the repository: scripts/bancada-spec.mjs runs the race, scripts/bancada-spec/tarefas.mjs stores the ten tasks with the three prompts and tests for each one, scripts/bancada-spec/executor.mjs runs a candidate in its own process.

$ pnpm bancada --modelos qwen3-coder:30b,devstral:24b,qwen2.5-coder:32b --amostras 3 --temperatura 0.7

references validated: 10 tasks, all at 100% bench: 3 models × 10 tasks × 3 arms, temperature 0.7

--retomar uses what’s already run because 270 generations don’t fit in a session without something dropping in the middle.

Race 1: temperature 0, 90 generations

One request per cell, no randomness. This is the race that answers “with the same prompt, what comes out”.

Arm	Happy Path	Edge Cases	% edge	Tasks 100%
prose	78/116	93/200	47%	0/30
long prose	116/120	185/207	89%	18/30
spec	110/120	183/207	88%	14/30

The first column is what I expected and isn’t news: the happy path the model gets almost always, in any arm. Everyone understands “format a number as Brazilian Real”.

The edge case column is where information shows up — 47% against 89%. It’s the same phenomenon from the July post, now with numbers: the model doesn’t err due to stupidity; it errs because it closed the gap that the request left.

And then the bottom line. The structured spec was one point below the same information in paragraph form. It’s not a victory for anyone.

Race 2: 270 generations at temperature 0.7

A deterministic race doesn’t distinguish real difference from luck. I repeated three times at 0.7, which is where these tools usually run in practice.

Arm	Edge Cases	% edge	Per repetition
prose	277/607	46%	46%, 48%, 43%
long prose	554/614	90%	88.0%, 91.3%, 91.3%
spec	538/621	87%	87.4%, 86.5%, 86.0%

Dispersion matters more than the average. Short prose oscillates 5 points between repetitions; spec, just over 1. All three spec repetitions were below the worst long prose repetition, which means that the 3.6 point difference isn’t an artifact of a bad roll.

Per model, with the same race:

Model	prose	long prose	spec
qwen3-coder:30b	48%	95%	95%
devstral:24b	39%	92%	85%
qwen2.5-coder:32b	49%	84%	81%

The strongest model ties both arms at 95%. In the other two, long prose leads by 7 and 3 points. No model put spec ahead.

The data that decides interpretation

The average hides what happened. In the deterministic race, cell by cell — 3 models × 10 tasks —, spec and long prose gave identical result in 18 of the 30 cells. In the remaining 12, long prose won 8 and spec won 4.

I went to look at one of the identical ones by hand. Task slugify, devstral:24b, temperature 0: both arms produced code byte-by-byte identical. Two prompts of completely different size and layout, with the same content, collapsing into the same output.

formatarBRL: zero out of 11 tests, in all three models, due to one character

The most interesting finding from the race is qualitative and doesn’t appear in any average.

In the task formatarBRL, the prose arm scored 0 out of 11 in all three models. Absolute zero, even on the happy path. Single cause, identical in all three:

Intl.NumberFormat('pt-BR', { style: 'currency', currency: 'BRL' }).format(1234.56);
// '$ 1,234.56' — visually correct, and fails any string comparison

// the character between "R$" and "1" is not a space:
// code 160 (U+00A0, non-breaking space), not 32

The model wrote the idiomatic solution, the one any human reviewer would approve in a glance. And it fails against assert.equal with 'R$ 1.234,56' with common space because Intl returns non-breaking space.

With the information written down — “prefix R$ followed by a single common space, don’t use non-separable space” —, two of the three models start handling this: they use Intl and replace the character, or format it manually. The qwen2.5-coder:32b continues calling Intl without treatment and fails in all three arms, including with spec in front of it.

This is exactly the plausible error from the July post, at the character level. It’s not absurd code that I would catch in ten seconds. It’s the line everyone would write, silently deciding something the request didn’t decide — and that only shows up when the value goes to a CSV or to a string comparison in another system.

What spec costs

In the task truncar, the short prose prompt used 94 input tokens. The same task in spec used 556 — about 6× — for a difference in accuracy that, against long prose, didn’t show up.

With local model, those extra 462 tokens are prefill time: 4ms more per generation on this board (median of 141ms vs. 145ms, five measurements each). With API model, they’re a bill at the end of the month, multiplied by every call where spec enters context.

The cost is the same in both informed arms because the information is the same. What measurement says is that it buys the jump from 46% to 90% — and doesn’t buy anything beyond that just for being in a table.

Aider, Ollama and the trap that invalidates measurement

In practice I don’t talk to the model via fetch. I drive the local model through Aider:

# directly in Ollama
aider --model ollama_chat/qwen2.5-coder:32b --read docs/specs/truncar.md src/texto.js

# or through an OpenAI-compatible gateway
export OPENAI_API_BASE=http://localhost:4000/v1
aider --model openai/qwen2.5-coder:32b --read docs/specs/truncar.md src/texto.js

The --read keeps being the detail that matters most: loads spec into context without making it editable. An agent that can edit spec “resolves” divergence by rewriting reference, which is the worst possible outcome.

And that’s why this bench didn’t measure through Aider. Aider adds repo map, edit format and retry. Measuring via it, the number would be from Aider, not the model. The bench talks to the model via raw API, one turn, no history — it answers “did the model understand the requirement?”, not “did the tool finish the task?”. Those are different questions and the second isolates nothing.

Where small models truly break in 2025, in daily use, isn’t writing the function: it’s applying the patch in search-and-replace format that Aider expects. The block returns with indentation changed or a context snippet that doesn’t match byte-by-byte with the file, and editing fails. The code was right. This doesn’t show up in any function’s accuracy rate — and is most of real friction.

Experiment limits

• Ten small pure functions aren’t real repo tasks. There are no dependencies between files, project conventions, legacy code to respect. The “out of scope” section of spec, which I called the one that pays most in July, has almost nothing to do here — there’s no neighbor for the model to refactor on the fly. It’s plausible that structure pays precisely where this experiment doesn’t look.
• One turn only, no agent loop. No test running, no error coming back, no correction. Much of the value of an executable acceptance criterion is exactly in the loop this bench doesn’t have.
• Local models from 24 to 32B. Nothing says the result transports to large API model or smaller one.
• String comparison is fragile, and formatarBRL proves it. An invisible character zeroed out an entire task. Where my test is too rigid, I fail code that would serve; where it’s loose, I pass code that doesn’t serve. The 3.6 points between spec and long prose are small enough to fit in this fragility.
• Three repetitions give idea of dispersion, not confidence interval. I didn’t do statistical testing.

What I changed my mind about

I wrote in July that spec made the agent get it right more often. It’s wrong, as I wrote it. What makes the model get it right more often is me having decided and written down what happens when value is negative, when input is null, when page goes past end. Spec is where I usually do that — it’s not the mechanism.

What stands from the previous post: the edge cases section is what carries the result. That’s where the 44 points appeared. What falls: the idea that table, headers and markdown make a difference to the model. To the model, as far as I measured, they don’t.

So what’s spec for if it’s not accuracy per generation? This here is judgment, not measurement — I didn’t test any of what comes below:

• Versioning. Spec in prompt dies with session. In repository, it enters diff and someone reviews decision before it becomes behavior.
• Review against reference. The question at review stops being “does this look right?” and becomes “does this match the file next to it?”. The second has an answer.
• Reuse between tools. Same file serves Aider, Claude Code, Cursor and me reading. Prose pasted in chat doesn’t even serve a second time.
• Longevity. Model changes every few months. File that says what system does keeps being valid after change; prompt tuned for specific model, not.

Structure is for me, and the next person who opens file. That seems enough to keep writing spec. It’s just not what I said it was.

If you’re repeating this: pnpm bancada, with --modelos, --amostras, --temperatura e --retomar. Tasks are in scripts/bancada-spec/tarefas.mjs, with the three prompts side by side — the part most worth reviewing before believing any number from this post.