VLM latency is not VLA latency

What I expected

Same weights, same pass: I expected that an action token would cost exactly the same as a caption token. Each token is one pass over the weights, and the pass is -bound (Chapter 2). I expected only one difference. It was in , where the action prompt is longer.

I also had a number from the papers. On Orin-class boards, the papers put most of a step, approximately 75%, into the production of the action. I expected that my proxy would give a value near that number.

The theory

Decode per token = weight bytes / bandwidth, then check it twice

Qwen2.5-vl-3b, the floor (estimate)

LM weights: 3 B x 4.5 bit / 8 = 1.7 GB
one decode pulls 28 GB/s (Day 2)
floor = 1.7 GB / 28 GB/s = 61 ms per token
at the 68 GB/s roof it would be 25 ms
= caption: 1,054 / 16 = 66 ms per token

The action, counted honestly

budget 16 tokens, stopped at the brace
the JSON is 10 tokens (eval_tokens in the log)
638 ms / 10 = 64 ms per token
caption 66, action 64
= same per token, as predicted

The 500M model is 3 ms per token above its own floor. The measured value is 211 / 16 = 13 ms. The floor is 0.5 B x 4.5 bit / 8 = 0.28 GB / 28 GB/s = 10 ms. My guess is launch overhead, the cost that Chapter 7 removed from the preprocess by .

The step, split (measured, 3B, the action run)

prefill: 1,979 ms
decode: 638 ms
decode share: 24%

Arithmetic: 638 / (1,979 + 638) = 24%. The papers said about 75% in the action phase; my proxy says the opposite. The reason is length: my action is 10 tokens and a VLA chunk is many more. The image is Chapter 19's 640x360 frame, 367 prompt tokens on the action run, not the 320x180 I set as the robot default; at 320x180 Chapter 19's TTFT was 656 ms against 1,935, so this prefill share is the high-resolution figure.

The setup

Run (board, ~10 min)

python3 ~/day26_vla_proxy.py
cat ~/day26/results.json | head -40

From results.json, the file the run wrote

"server": "~/llama.cpp/build/bin/llama-server"
"image": "~/day19/img_hi.jpg"
"note": "16 new tokens each. Same image. Caption vs forced JSON action. The delta in prefill+...

The measurement

What	Expected	Measured	Note
SmolVLM-500M: caption / action decode ms (16 tok)	same per token	211 / 213	prefill 402 / 409 ms; JSON valid 0/5
Qwen2.5-VL-3B: caption / action decode ms (16 tok)	same per token	1054 / 638	prefill 1918 / 1979 ms; JSON valid 5/5

Measured on the board unless the row says estimate or predicted.

Fig 26.1Same image, same VLM, 16 new tokens: caption vs forced JSON action, prefill and decode (ms).

One step, image to action (3B, measured)

Imagevision tokens
Prefill1,979 ms
Decode638 ms
JSON action10 tokens

2,617 ms end to end at 640x360, 76% of it before the first action token exists. The image is paid for on every step: Chapter 19's context tax, landing on the action path.

Prefill and decode, caption against action (3B)

cap prefill

1,918 ms

cap decode

1,054 ms

act prefill

1,979 ms

act decode

638 ms

Prefill grew 61 ms for the longer prompt; decode shrank because the action ended early. For the 500M the same four are 402 / 211 / 409 / 213: flat, because both of its outputs ran to the 16-token ceiling.

model	cap pre	cap dec	act pre	act dec	JSON /5
SmolVLM-500M	402	211	409	213	0
Qwen2.5-VL-3B	1,918	1,054	1,979	638	5

Measured: median ms at 15W, 16-token budget, temperature 0, 3 caption runs and 5 action runs, image 640x360 (Chapter 19's img_hi.jpg). The 75% action-phase prior is a paper figure. Timing it here needs a real VLA, and Chapter 27 picks it.

Where it broke

The 638 misled me for a short time. The action was forty percent cheaper than the caption, and this reads as if acting is cheaper than talking. It is not. The action ended early at the closing brace. A budget is a limit, not a count, and I compared two limits. The numbers for each token are the numbers that agree.

The second result is that the 500M gave 0 valid JSON in 5 tries. It gave a prose sentence instead, at the same cost for each token. The model could not keep the shape. Talking and acting differ in exactness, not in milliseconds. Exactness comes from the model, not from the stop string.

Summary

Per token, a caption and an action cost the same on the 3B: 66 ms and 64 ms. My action was 10 tokens, the log's eval_tokens, and stopped at the closing brace. Decode was 24% of the step, not the papers' 75%.
Prefill is the other half. The action prompt is longer, and vision tokens (Chapter 19) are in front of it on each step.
The headline Hz of a VLA counts chunks, one chunk for each forward. Chunks per second x actions per chunk gives action throughput. It does not give latency. The first action of a chunk waits for the full forward. Compare that forward time with the two clocks on Chapter 29. Do not compare the share of each action.

Sources

These sources are not mine. Measured numbers came off my own board; specs, estimates and paper figures are labeled as such.

arXiv 2603.02271 VLA characterization on Orin/Thor (action-phase bottleneck ~75%)
Your Day 22 prefill/decode split applied to actions instead of words