The 8 GB Lab Book · Chapter 14
Quantization Orin can execute
- Board
- Jetson Orin Nano Super
- Mode
- 15W
- Summary
- Q4_K_M decoded 1.18x faster than Q8_0, not 2x, and scored the same 8/10 on the tool cases.
What I expected
I wrote '~2x tok/sTokens per second: how fast a language model reads a prompt (prefill) or writes output (decode); the two rates are different numbers. of Q8' in the predicted column before the run. That was Prior 1: half the bytes give twice the tokens. Chapter 13 showed three runtimes at one effective memory bandwidthHow many bytes per second the memory system can deliver; on a shared-memory board the CPU, GPU and video engines all draw from the same supply.. I expected tok/s to be bandwidth divided by bytes and nothing else.
Prior 2 was that the two files give the same effective GB/s, because the bus and the engine are the same.
Prior 3 was that Q4 is slightly worse on the 10 regression cases. I expected it to become worse first on the 2 restraint cases.
The theory
Bits per weight from the file, then a bus floor per token
Bits per weight (arithmetic)
4.02 B params in the checkpoint Q8_0: 4.27 GB x 8 / 4.02 B = 8.50 bits Q4_K_M: 2.49 GB x 8 / 4.02 B = 4.96 bits a plain 4.5-bit file would be 2.26 GB = the M keeps some tensors at 6 bits: +0.23 GB
BUS FLOOR PER TOKEN, 68 GB/s ROOF
Q8: 4.27 GB / 68 GB/s = 63 ms Q4: 2.49 / 68 = 37 ms measured: 1/9.6 = 104 ms, 1/11.3 = 89 ms the rest: Q8 41 ms, Q4 52 ms = the not-bus part GREW by 11 ms
'the rest' is all that is not weight bytes. It is attention, norms and launches. For Q4, it is also the unpacking of 4.96-bit blocks into numbers that the cores can multiply. That unpacking is compute. At 612 MHz, compute has a cost.
Prefill: which silicon is running it (arithmetic from measured tok/s)
- Q8 prefill
- 3.75 TF/s
- Q4 prefill
- 3.34 TF/s
- fp32 core peak
- 1.25 TF/s
The setup
bash tools/day14_quant.sh
cat ~/day14/ram.txt; ls ~/slm/"tag": "q4"
"model": "~/models/Qwen3-4B-Q4_K_M.gguf"
"ctx": 4096
"start_ms": 31228.43
"grammar": false
"chat": [ ... ]
"tool": [ ... ]
"cases": [ ... ]
"summary": { "ttft_ms_chat_median": 159.921, "decode_tps_chat_median": 10.697, "tool_valid": ...- Script: tools/day14_quant.sh, which calls tools/slm_eval.py
- Wrote: day14/bench_Q8_0.json, bench_Q4_K_M.json, ram.txt
- And: slm/q8.json, slm/q4.json (quality)
- llama-bench gives prefill and decode tok/s, tegrastats gives the RAM pile with each server up, and slm_eval.py runs the same 10 prompts and 10 tool cases on both files.
# from day14/bench_Q4_K_M.json and bench_Q8_0.json (this run)
"build_commit": "3466812", "backends": "CUDA", "gpu_info": "Orin"
"n_gpu_layers": 99, "n_threads": 6, "type_k": "f16", "type_v": "f16"
llama-bench -ngl 99 -p 512 -n 128 -r 5 (tools/day14_quant.sh)
# board state captured 2026-09-29 (days/environment/board-2026-09-29.txt), not the state of this day's run
L4T R36.4.2, kernel 5.15.148-tegra
NV Power Mode: 15W
CUDA 12.6 (V12.6.68), TensorRT 10.3.0.30, cuDNN 9.3.0.75
The measurement
| What | Expected | Measured | Note |
|---|---|---|---|
| Q8_0 size / decode / prefill | 2x bytes of Q4 | 4.27 GB / 9.6 / 467 tok/s | pile 5259 MiB at 4k, idle 1519 |
| Q4_K_M size / decode / prefill | ~2x tok/s of Q8 | 2.49 GB / 11.3 / 416 tok/s | pile 4171 MiB at 4k, idle 1519 |
| B_eff Q8 / Q4 | same | 41.1 / 28.2 GB/s | same bus |
| tool call emitted 5 / regression 10 / restraint 2, Q8 | - | 5 / 8 / 2 | slm_eval.py |
| tool call emitted 5 / regression 10 / restraint 2, Q4 | slightly worse | 5 / 8 / 2 | slm_eval.py |
| Native precisions on this board | - | FP16, INT8, INT4 | sm_87; no NVFP4/FP8/MXFP4 |
What sm_87 executes; cross out the rest
| quant | tools /5 | cases /10 | restraint /2 | chat TTFT ms |
|---|---|---|---|---|
| Q8_0 | 5 | 8 | 2 | 162 |
| Q4_K_M | 5 | 8 | 2 | 160 |
What 10 cases can see (arithmetic)
say Q4 truly fails 1 in 20 more (p = 0.05) 10 cases show nothing: 0.95^10 = 0.60 2 restraint cases show nothing: 0.95^2 = 0.90 to catch p at 95%: n = ln 0.05 / ln 0.95 = 59 cases with n = 10, the smallest p I catch at 95%: 26% = this set is blind below ~26% per case
On this set, the two files scored 8/10 and failed the same two cases. The set detects a loss for each case only at approximately 26% or more.
Where it broke
For Prior 1, I expected 2x and measured 1.18x. My model had no constant term. It is also not one shared constant. The line through the two measured points needs 1.78 GB / 15.5 ms = 115 GB/s of weight streaming. That value is above the roof. The Q4 kernel has its own extra work.
For Prior 2, the result was 41.1 against 28.2 GB/s effective. The bus is the same, but the kernel that uses it is different. For Prior 3, the scores are identical, 5/5, 8/10, 2/2, and the same two cases failed.
I predicted 2x from a model with no constant term. I did this one day after Chapter 13 gave me a 25 GB/s limit. That limit IS a constant term in a different form. I trusted this measure yesterday, and I forgot it today.
The pile is the whole board, so I subtract the idle figure from the same run (ram.txt: idle 1519). Q8: 5259 - 1519 = 3740 MiB vs MBA mebibyte (MiB) is 2^20 bytes and a megabyte (MB) is 10^6 bytes; some tools label MiB as "MB", so check which one a tool actually counts.. Q4: 4171 - 1519 = 2652 MiB. tegrastatsNVIDIA's Jetson monitoring tool, which prints RAM, swap, CPU and GPU load, clock speeds, temperatures and power readings at a fixed interval. counts MiB, so the files are 4.27 GB = 4076 MiB and 2.49 GB = 2376 MiB. KV at -c 4096 is 4096 x 144 KiB = 576 MiB (604 MB), the Chapter 3 figure, allocated at start in both. So the Q8 pile sits 336 MiB under its own file, and the Q4 pile 276 MiB over it, before KV and context. The used-RAM counter does not show all the mmapMapping a file into a process's address space so its pages load on demand, which also makes those pages count toward the process's memory once touched.'d weight pages. Most likely they count as file cache, which tegrastats excludes. Not tested.
Summary
- Q4_K_MA llama.cpp quantization type that stores most weights as 4-bit values, but per-block scales and a few tensors kept at higher precision push the average above four bits per weight. is not 4 bits. 2.49 GB for 4.02B weights is approximately 5 bits for each weight. K-quants keep wider blocks where the model is sensitive. The file size is the only reliable source.
- Q4 and Q8 had the same scores, 5/5, 8/10 and 2/2, and failed the same two cases. Ten cases detect a loss for each case only at approximately 26% or more. The restraint cases did not become worse first.
- sm_87 has tensor-core paths for FP1616-bit floating point (half precision): half the bytes of FP32, and tensor cores run it at much higher throughput than FP32., INT8An 8-bit integer number format; it shrinks weights and activations relative to floating point but needs calibration or scaling to keep accuracy. and INT4 (spec). It has no path for NVFP4, FP8 or MXFP4. Those are Hopper and Blackwell formats.
Sources
These sources are not mine. Measured numbers came off my own board; specs, estimates and paper figures are labeled as such.
- TensorRT Edge-LLM supported-models / precision notes (Orin: FP16, INT8, INT4 only)
- GGUF quant table (Q4_K_M as the usual default)
- AWQ paper for the idea of protecting salient weights