Prefill, decode, and the wrong headline

What I expected

I wrote down 20 to 40 tokens for a tool call. I expected that the decode term would be most of the wait. 30 tokens at 10.7 tok/s is 2.8 s, and I imagined a prefill of a few hundred ms. I expected that decode would be the term to attack.

I expected that the TTFT of the tool prompt would be higher than the TTFT of chat. I did not write a number for the difference.

The theory

Two terms, and they live on opposite sides of the ridge

4 B params x 2 flop = 8 GFLOP per token
fp32 peak 1.25 TFLOP/s: 6.4 ms per token
820 - 162 = 658 ms extra for the schema
658 / 6.4 = about 103 schema tokens
AI = 100 tok x 2 flop / 0.6 B per weight = 333 flop per byte
= 333 >> ridge 18: compute side

1 / 10.7 tok/s = 93.5 ms per token (chat)
weights 4 B x 4.9 bit / 8 = 2.45 GB (est.)
2.45 GB / 93.5 ms = 26 GB/s moved
Day 2's SLM pull: 28 GB/s. same place
AI = 2 flop / 0.6 B = 3.3 flop per byte
= 3.3 << ridge 18: memory side

One tool call reads the full model again ten times: 10 x 2.45 GB = 24.5 GB of DRAM traffic for approximately forty characters of JSON. The roof at 68 GB/s would give 36 ms per token, which is 28 tok/s. The board runs at 2.6x that floor.

The setup

python3 -c "import json;s=json.load(open('$HOME/slm/qwen3-4b.json'));print([round(c['ttft_ms']) for c in s['chat']],[round(t['ttft_ms']) for t in s['tool']],[t['tokens'] for t in s['tool']])"

~/llama.cpp/build/bin/llama-server -m ~/models/X.gguf -ngl 99 -c 4096 --port 8080 --cache-prompt   # then send the same system prompt twice and compare TTFT

The measurement

The clock: a shaded prefill, then ten ticks

Where it broke

The JSON is 10 tokens, not 30. I imagined pretty-printed JSON with spaces and newlines. The tokenizer reads a one-line object as ten pieces. So decode is about 1.3 s, not 2.8 s. On the median call the split was 796 ms of prefill and 1,265 ms of decode, 39 and 61 percent.

I first printed t_tool as 820 + 10 x 93.5 = 1,755 ms, with the chat decode rate, and tagged it measured. The log has the five totals: 3,172 / 1,813 / 2,061 / 1,432 / 4,071 ms, median 2,061. Decode during a tool call ran at 6.0 tok/s, median, not the chat's 10.7. Most likely the server's tool-call parsing per token; not tested. On 2026-10-04 I ran the same harness three more times: the run medians were 2,070, 2,061 and 2,066 ms, each prompt within 10 ms of its first time, 15 of 15 calls a valid tool_call, TTFT median 820 ms and decode 6.0 tok/s during the calls. The spread on this page is between prompts, not between runs.

The schema costs 4x the chat prefill: 658 / 162 = 4.1x, and the whole tool prefill is 5.1x. That is 658 ms for each call, for bytes that never change. That term has a free fix, so prefill is the first target.

My prior for the JSON was 3x too long, because I never tokenized the target.

Summary

• tok/s is not what the robot waits for. The robot waits for TTFT plus the tokens until the tool call is complete. A model can stream prose at 30 tok/s and take 900 ms to start. For the robot, it is slower than a model at 15 tok/s that starts in 200.
• Vision tokens are prefill tokens. An attached image has the same cost as a long pasted prompt. Chapter 19 measures exactly how long.
• The JSON was 10 tokens, not the 20 to 40 that I guessed. It still cost 1,265 ms on the median call, 61 percent of it. The schema's 658 ms of prefill repeats on each call, and that repetition is why I attack it first.

Sources

These sources are not mine. Measured numbers came off my own board; specs, estimates and paper figures are labeled as such.

• Standard LLM serving literature (prefill vs decode)
• Edge-LLM features: speculative / EAGLE3 rows in the Nano table
• Your Day 2 roofline
• day22/rerun-2026-10-04/summary.json: slm_eval.py run three more times on 2026-10-04 (rerun1004-day22-run1..3.json, tegrastats beside each)