Power modes, Super, and why TOPS lie

What I expected

I took these predictions from the poster numbers, before the run.

• GPU clock cap, 15W mode: 1020 MHz (Super)
• Memory bus cap, 15W mode: 3200 MHz EMC clock (Super)
• Bandwidth roof at 15W: 102 GB/s poster
• INT8 peak at 15W: 67 TOPS poster
• Ridge point I* (ops/BW): 657 ops/B MAXN (my prior; sparse INT8 TOPS, not FLOP)
• Decode matvec intensity, Q4_K_M: 2/(4.5/8) = 3.6 FLOP/B
• Qwen3-1.7B decode tok/s: 37.1 / 64.9 (0.4x / 0.7x poster)
• Qwen3-4B decode tok/s: 16.4 / 28.7
• B_eff across 1B..7B: 0.4-0.7 x 102
• Soak, Qwen3-4B 30x128 tok, 15W: droop if throttled

The theory

What TOPS measures, what decode needs

A chip has two limits. The first limit is how fast it can compute (TOPS). The second limit is how fast it can move data (GB/s).

To generate one word, the chip reads each weight of the model one time. It does two very small multiplies for each weight that it reads.

That is a memory job, not a math job. The work reaches the GB/s limit first, and it never uses the TOPS.

THE ARITHMETIC: the ridge point

The ridge point is peak FLOP/s / peak bandwidth. Use the number format that the work uses.

The poster gives 67 TOPS / 102 GB/s = 657. But those TOPS are sparse INT8. So 657 is INT8 ops per byte. It is not a FLOP ridge.

At 15W, fp32 on the CUDA cores gives 1024 cores x 2 FLOP x 612 MHz = 1.25 TFLOP/s. This is spec arithmetic. The ridge is 1.25 TFLOP/s / 68 GB/s = 18 FLOP per byte.

Decode needs 3.6 FLOP per byte. This is 5x below the 15W fp32 ridge.

The 3200 and 2133 are EMC clock frequencies in MHz, from nvpmodel.conf. LPDDR5 moves two transfers for each clock. So 3200 MHz is 6400 MT/s, and 128 bits x 6400 MT/s = 102 GB/s. The 15W roof is 102 x 2133 / 3200 = 68 GB/s.

The setup

nvpmodel -q                                   # -> 15W (mode 0)
cat /sys/class/devfreq/17000000.gpu/max_freq   # -> 612000000  (available up to 1020000000)
grep -A30 'POWER_MODEL ID=0' /etc/nvpmodel.conf | grep -E 'GPU MAX|EMC MAX'   # 612 MHz, 2133 MHz
# roof at 15W = 102 GB/s x 2133/3200 = 68 GB/s.   INT8 = 67 x 612/1020 = 40 TOPS.

B=~/llama.cpp/build/bin/llama-bench; MD=~/models
for f in Llama-3.2-1B-Instruct Qwen3-1.7B Llama-3.2-3B-Instruct Qwen3-4B Qwen2.5-7B-Instruct; do
  $B -m $MD/$f-Q4_K_M.gguf -ngl 99 -p 512 -n 128 -r 3 -o json > ~/day02/bench_$f.json; done
# tg128 avg_ts = decode tok/s.  B_eff = avg_ts x model_size bytes.   pp512 = prefill.

tegrastats --interval 2000 > ~/day02/tegra_soak.log &
$B -m $MD/Qwen3-4B-Q4_K_M.gguf -ngl 99 -p 0 -n 128 -r 30 -o json > ~/day02/soak_4b.json
kill %1;  python3 -c "import json;print(json.load(open('$HOME/day02/soak_4b.json'))[0]['samples_ts'])"
# droop = first five vs last five.  gpu@..C and VDD_IN in the tegra log.

Environment and scripts

• Board: Jetson Orin Nano Super, L4T R36.4.2 (from measured.json).
• Power mode: 15W (nvpmodel 0), found, not pinned. Caps: GPU 612 MHz, memory 2133 MHz.
• llama.cpp: build 3466812 (build_commit in each bench JSON).
• Decode and prefill: llama-bench, the loop above, one bench_<model>.json per model.
• Soak: llama-bench plus tegrastats, soak_4b.json and tegra_soak.log.
• Live token trace: tools/day02_live.py, live.json.

"prompt": "Explain in three sentences why an 8 GB Jetson shares one memory between the CPU an...
"runs": [ ... ]
"t0": 65039.38
"samples": [ ... ]

The measurement

TWO RECEIPTS: POSTER CLOCK vs THE MODE I RUN

GPU clock    1020 MHz
memory bus   3200 MHz
= = 67 TOPS, 102 GB/s

GPU clock     612 MHz
memory bus   2133 MHz
= = ~40 TOPS, 68 GB/s

Super is a clock setting on the same chip. It is not a different chip.

ONE DECODE STEP, weight by weight

EFFECTIVE BANDWIDTH vs THE 68 GB/s ROOF

Where it broke

The poster said 102 GB/s and 67 TOPS. The board at 15W caps the GPU at 612 MHz and the memory clock at 2133 MHz, so the roof is 68 GB/s. My tok/s predictions came from the poster and were too high: 16.4 against 11.3 for the 4B. Decode used 27 to 37 GB/s of the 68. The soak did not droop: the last five runs were 0.1% faster than the first five.

My notes from Day 2

Written by hand on 2026-09-02, before this chapter was typed. These are my original notes from that day. Where the chapter corrects them, the chapter is current.

Summary

• Tok/s x model bytes gives the bandwidth that decode used. Here it was 27 to 37 GB/s. That is 39% to 54% of the 68 GB/s roof at 15W.
• Larger models use the bus better. Small models have a fixed cost for each token. This cost has no relation to bandwidth.
• Prefill is the read of the prompt. Compute is important in prefill. The 1B read 1450 tok/s of prompt and decoded 34.3 tok/s.

Sources

These sources are not mine. Measured numbers came off my own board; specs, estimates and paper figures are labeled as such.

• JetPack 6.2 Super-mode blog (Orin Nano 8GB 15/25W/MAXN SUPER, 67 TOPS sparse INT8, ~102 GB/s)
• Roofline: Williams, Waterman, Patterson
• Orin Nano datasheet clocks (GPU 625 MHz base / 1020 MHz Super class)
• Orin Nano Super spec: 1024 CUDA cores (fp32 peak = cores x 2 FLOP x clock)