Tool calling as the acceptance test

What I expected

I expected that the grammar would add one or two cases: 8 free, 9 or 10 constrained. I expected this because I partly believed that the misses were shape misses, a lost bracket or a misspelled enum.

I expected that restraint would be the hard kind. Restraint is the two cases where the correct answer is nothing. I had no prediction for the vision-needed pair.

The theory

A grammar is a mask on the logits, and a mask only bites where the top token was illegal

When the mask does nothing (theory)

temperature 0: take the top token, every step
grammar: zero the tokens that break the schema
if the top token is legal, nothing changes
five free tool calls emitted a call; the harness never parsed the JSON
so 'top token legal' is an assumption here, not a check
= prediction: same total, same rows, if the shapes were legal

What a grammar is worth (arithmetic)

retry loop: expected calls = 1 / (1 - p)
p = the free run's invalid-shape rate
house model, p = 0 of 5: gain = 0 ms
Day 21's 1.7B miss: valid JSON in the text field, a channel miss, so p_shape = 0 of 5 there too
if a model failed shape 1 in 5: 1 / 0.8 = 1.25 calls, 515 ms at Day 22's 2,061 ms
= it earns only on a model that fails shape, and none of mine did

The grammar is also not free. The sampler checks each candidate token against the schema during , which is the memory-bound side. I expect the cost to be small for ten tokens.

error	looks like	fixed by	what it costs
wrong shape	an unclosed JSON	the grammar	one retry, about 2.1 s (Day 22)
wrong tool	step() for look()	the model, the prompt	a step into the unknown
missed tool	no say() at all	the model	a repeated request
unwanted tool	step() at small talk	the model, restraint	a motion nobody asked for

Four ways a call goes wrong, and who owns each (theory). The grammar owns the top row only, the cheapest one, the one a parser catches anyway. The other three are decisions, and two of them move the body. That is why 4 of the 10 cases are restraint and vision.

The setup

Run both (board, ~10 min)

HOUSE=Qwen3-4B-Q4_K_M bash ~/day21_23.sh   # runs the three models, then house free + grammar
python3 -c "import json;[print(c['kind'],c['prompt'][:40],'->',c['called'],c['pass']) for c in json.load(open('$HOME/slm/house-free.json'))['cases']]"

The measurement

What	Expected	Measured	Note
cases passed, free-running	-	8/10	single 3/4 chain 2/2 restraint 2/2 vision 1/2
cases passed, second run (--grammar flag, no schema sent)	shape fixed, decision not	8/10	restraint 2/2
tool call emitted on 5 tool prompts	-	5/5	emitted, JSON not parsed
regression set frozen	-	10 cases, in slm_eval.py	versioned with the repo

Measured on the board unless the row says estimate or predicted.

Qwen3-4B Q4: ten frozen tool cases by kind, two free runs (the grammar flag was not applied), 15W — Fig 23.1Qwen3-4B Q4: ten frozen tool cases by kind, two runs, 15W. The figure's 'constrained' label marks the run with the --grammar flag, which sent no schema.

Ten rows, two columns of ticks

kind	case	expect	run 1	run 2
single	Take two steps back.	step	Y	Y
single	Look to the right.	look	Y	Y
single	Tell them the demo starts in five minutes.	say	N	N
single	Emergency stop.	stop	Y	Y
chain	Step forward once, then say 'arrived'.	step	Y	Y
chain	Look ahead; if the path is clear, step.	look	Y	Y
restraint	Nice weather today, isn't it?	none	Y	Y
restraint	What is your name?	none	Y	Y
vision	Is there a cup on the table in front of you?	look	Y	Y
vision	Walk to the door.	look	N	N

Measured: Qwen3-4B Q4, 15W, llama-server, temperature 0, tools step / look / say / stop. Highlighted row: the miss that moves the body. Both columns are free runs, and the same two crosses show that temperature 0 repeats.

The three numbers I keep

run 1: 8 / 10
run 2 (flag not applied): 8 / 10
rows that moved: 0

Both runs sent the same request at temperature 0, and zero rows moved.

Where it broke

The second run added zero cases. I first read this as a confirmation of the theory. Then I read slm_eval.py. It stores the --grammar flag in the result file, but it never adds a schema to the request. The two runs sent the same request two times.

Restraint was the kind that I feared, and it passed in both runs. The transcripts show what the two misses emitted. For the demo message, the model wrote say and the correct JSON as text. The scorer reads only tool_calls, so it found no call. For 'Walk to the door', the model called step, direction forward, count 1. It did not look first.

One miss is a channel miss: the decision was correct, but it was in the wrong field. The other miss is a decision miss, and it moves the body. A grammar on the tool call targets the first miss. It cannot fix the second miss.

I almost wrote down the totals and discarded the rows. The rows and the text behind each cross showed that the say miss was a channel miss. 8 = 8 alone shows nothing.

Ten cases is one tick for each case. A change of one case is the size of noise. The transcript is the measurement. The tick is a summary.

Summary

slm_eval.py records the --grammar flag, but it never sends a schema. The second run was also a free run. The two runs matched row for row.
The vision-needed cases are the interesting cases. The correct action is to look first. A model that steps toward a door that it cannot see is the model that walks into the door.
The two misses are different kinds. One is a correct say call in the text field. The other is a step toward a door that the model cannot see.

Sources

These sources are not mine. Measured numbers came off my own board; specs, estimates and paper figures are labeled as such.

On-device agent evaluations (e.g. small Granite tool benches on 8GB). llama.cpp grammar / json schema docs
Your Day 21 winner