The fit equation: weights + KV + vision

What I expected

I computed these values from the model configs before the run.

• Qwen3-1.7B: KV per token (f16): 2x28x8x128x2 B
• Qwen3-4B: KV per token (f16): 2x36x8x128x2 B
• Qwen2.5-7B: KV per token (f16): 2x28x4x128x2 B
• Llama-3.2-3B: KV per token (f16): 2x28x8x128x2 B
• gemma-3n-E2B: KV per token (f16): 2x30x2x256x2 B
• Qwen3-1.7B pile at ctx 512: 2656 MB (w+1.5 GB+KV)
• Qwen3-1.7B pile at ctx 16384: 4392 MB (w+1.5 GB+KV)
• Qwen3-4B pile at ctx 512: 4062 MB (w+1.5 GB+KV)
• Qwen3-4B pile at ctx 16384: 6294 MB (w+1.5 GB+KV)
• Qwen2.5-7B pile at ctx 512: 6208 MB (w+1.5 GB+KV)
• Qwen2.5-7B pile at ctx 16384: 7076 MB (w+1.5 GB+KV)
• Qwen3-4B at ctx 32768 (the wall): 2.49 GB + 4.6 GB KV + scratch

The theory

Why the cache exists

For each new word, attention examines each earlier token. Without a cache, the model would compute all of them again at each step.

The model keeps the key and the value of each token, for each layer and for each KV head. This is the KV cache. The cache uses memory to get speed. It grows by one entry for each token.

Where the three numbers live (config.json of any model)

Num_hidden_layers = 36, num_key_value_heads = 8 and head_dim = 128. These are the three numbers.

The number of bytes per token is layers x kv_heads x head_dim x 2 x 2.

THE CHECK, from llama.cpp's own log

The log gives Llama_kv_cache: size = 576 MiB at n_ctx 4096.

144 KB x 4096 = 589,824 KB = 576 MiB. The match is exact to the byte.

THE ARITHMETIC, worked for the 4B

7,620 is usable.

Subtract 1,500 for the idle floor. After a day of loading, the driver keeps pages.

Subtract 2,064 for the weights as loaded. This is the measured delta at 512 tokens, minus its very small KV.

Subtract the 1,024 that llama.cpp keeps free before it refuses.

The result is 3,032 MB of headroom for conversation. 3,032 MB / 144 KB = ~21,500 tokens. This is the floor formula, not a run.

I asked for 32,768. The live run climbed to 7,350 MB, and my guard at 7,200 MB killed the server. A second attempt (wall.txt) exited on its own at load, peak 6477 MB. llama.cpp said it must reduce device memory by 2288 MiB to keep 1024 MiB free. KV at 32k is 4608 MiB, so about 16.5k tokens would fit (arithmetic). No context between 16k and 32k ran. Rerun 2026-10-04: contexts 20,480 and 24,576 loaded and answered, peak 6,493 and 7,064 MB; 26,624 and 28,672 hit the 7,200 MB guard at 7,291 and 7,422 MB. The 'reduce device memory' line printed at every one of those contexts and the load went on: it is a warning about a 1,024 MiB free target, not a refusal, so the 16.5k arithmetic was wrong. The wall is between 24,576 and 26,624 tokens.

The setup

~/llama.cpp/build/bin/llama-server -v -m ~/models/Qwen3-4B-Q4_K_M.gguf -c 4096 2>&1 | grep -E 'n_layer|n_head_kv|n_embd_head_k|KV buffer'
# KB/token = 2 x n_layer x n_head_kv x n_embd_head_k x 2 bytes / 1024   (f16 K and V)

bash ~/day03_sweep.sh
cat ~/day03/sweep.tsv
# pile = tegrastats RAM used with the server up at that -c

cat ~/day03/wall.txt
# a loop kills the server if RAM used passes 7200 MB. Day 1 taught why.

"model": "Qwen3-4B Q4_K_M"
"size_gb": 2.497
"kb_per_token": 144
"events": [ ... ]
"samples": [ ... ]

The measurement

Two receipts side by side

layers          x 36
KV heads        x 8
head size       x 128
key AND value   x 2
f16 bytes       x 2
= = 147,456 B = 144 KB

layers          x 28
KV heads        x 4
head size       x 128
key AND value   x 2
f16 bytes       x 2
= = 57,344 B = 56 KB

The larger model has the smaller entry. It has 4 KV heads, not 8.

FOUR BLOCKS, one per load (rectangles, taller as the number grows)

Three bars: tokens of conversation before the wall

Where it broke

The KV arithmetic held: 144 KB x 4096 = 576 MiB, and llama.cpp allocated 576 MiB. All six pile predictions were too high, by 74 to 788 MB, because my 1.5 GB floor was larger than the idle the sweep saw. The 4B at 32k did not fit. My guard killed it at 7,350 MB, and llama.cpp's own check said 2288 MiB too much. On Chapter 3 I never ran a context between 16k and 32k, so the practical maximum was arithmetic. On 2026-10-04 I did: 24,576 loaded and answered at a 7,064 MB peak and 26,624 hit the guard at 7,291. The shortfall line was a warning, not a refusal, and the wall is near 24.5k, not 16.5k.

My notes from Day 3

Written by hand on 2026-09-02, before this chapter was typed. These are my original notes from that day. Where the chapter corrects them, the chapter is current.

Summary

• Set the context length deliberately. It is the second largest memory setting, after the model itself. It is the only setting that grows while the robot talks.
• Use KB per token to judge the memory of a model. Do not use the parameter count. The 7B used less memory per token than the 4B.
• A default is (model, quantization, context, camera on). Without all of these items, it is not a default. My default for now is: Qwen3-4B, Q4_K_M, 4k, camera on.

Sources

These sources are not mine. Measured numbers came off my own board; specs, estimates and paper figures are labeled as such.

• Llama / Qwen model cards (GQA)
• Jetson AI Lab SLM guide: Nano 8GB comfort zone 1–4B Q4
• 7B Q4 is a science project with a camera
• NVIDIA posts listing Gemma 4 E2B as the Nano Gemma
• day03/rerun-2026-10-04/summary.json: Qwen3-4B Q4_K_M at 20480, 24576, 26624 and 28672 tokens on 2026-10-04, one try each, guard at 7,200 MB (wall.tsv, server logs, tegrastats)