Instrumentation is the only legal language

What I expected

Each prediction here comes from the short runs of the power-mode chapter.

• 1B decode, 30 samples: median: 34 (Chapter 2, n=3)
• Ruler noise (p90-p10)/median: < 5%
• 1B prefill pp512: 1450 (Chapter 2)
• Effective bandwidth: 27.5 (Chapter 2, tok/s x model bytes)

The theory

What a harness is

A harness is a script. It always records the same five things before it measures anything. These are the power mode, the GPU clock cap, the OS release, the code commit and the idle memory.

Then it runs the benchmark 30 times, not one time. It writes the median, the 10th percentile and the 90th percentile. This is the spread, not only one number that was good by chance.

WHY p10/p90, NOT JUST ONE NUMBER

The mean and a single run hide the noise. Then you will incorrectly think that this noise is a real increase in speed.

The correct error bars are p10 and p90. 80% of the runs are between them.

THE ARITHMETIC: noise as a percent

The spread is p90 - p10 = 34.49 - 34.27 = 0.22 tok/s.

The noise is 0.22 / 34.36 (median) = 0.6%.

A later claim that is within 0.6% is not a result. The 30 samples climbed from 34.01 to 34.49 in run order. So this spread is mostly warm-up drift, not random noise. The harness does not yet discard warm-up runs.

The setup

#!/usr/bin/env bash
# Day 4 harness: the ONLY legal metrics language for this series.
# Paste this block every day. Every posted number comes from here at a pinned power mode.
#
#   bash bench.sh <model.gguf> [tag]            # decode+prefill, 30 samples, tegrastats beside it
#   bash bench.sh --idle                        # the idle line only
#
# Output: ~/bench/<date>_<tag>/  {mode.txt, idle.txt, bench.json, tegra.log, summary.txt}
set -u
B=${LLAMA_BENCH:-$HOME/llama.cpp/build/bin/llama-bench}
OUT=$HOME/bench/$(date +%Y%m%d_%H%M%S)_${2:-run}
mkdir -p "$OUT"

# 1. pin + record the mode (pin with nvpmodel -m 0 and jetson_clocks; the harness records whatever it finds)
{ nvpmodel -q 2>/dev/null | head -1
  echo "gpu_max_hz $(cat /sys/class/devfreq/17000000.gpu/max_freq)"
  echo "gpu_cur_hz $(cat /sys/class/devfreq/17000000.gpu/cur_freq)"
  cat /etc/nv_tegra_release | head -1
  echo "llama.cpp $(cd "$HOME/llama.cpp" && git rev-parse --short HEAD)"
  echo "date $(date -Is)"; } > "$OUT/mode.txt"
cat "$OUT/mode.txt"

# 2. idle line (5 samples, take the median)
tegrastats --interval 1000 2>/dev/null | head -5 | awk '{print $3,$4,$7,$8}' > "$OUT/idle.txt"
echo "idle: $(sort "$OUT/idle.txt" | sed -n 3p)"
[ "${1:-}" = "--idle" ] && exit 0

# 3. bench with tegrastats beside it
tegrastats --interval 2000 > "$OUT/tegra.log" 2>/dev/null & TG=$!
timeout 900 "$B" -m "$1" -ngl 99 -p 512 -n 128 -r 30 -o json > "$OUT/bench.json" 2> "$OUT/bench.err"
kill $TG 2>/dev/null

# 4. summary: median, p10, p90 of the 30 decode samples; prefill; effective bandwidth; temp/power
python3 - "$OUT" <<'PY'
import json, re, sys, statistics as st
o=sys.argv[1]; d=json.load(open(f"{o}/bench.json"))
pp=[t for t in d if t["n_prompt"]>0][0]; tg=[t for t in d if t["n_gen"]>0][0]
s=sorted(tg["samples_ts"]); n=len(s); gb=tg["model_size"]/1e9
T=[];P=[]
for ln in open(f"{o}/tegra.log"):
    m=re.search(r"gpu@([\d.]+)C",ln); p=re.search(r"VDD_IN (\d+)mW",ln)
    if m: T.append(float(m.group(1)))
    if p: P.append(int(p.group(1)))
lines=[f"model {tg['model_filename'] if 'model_filename' in tg else ''} {tg['model_n_params']/1e9:.2f}B {gb:.2f} GB",
       f"decode tok/s  median {st.median(s):.2f}  p10 {s[n//10]:.2f}  p90 {s[9*n//10]:.2f}  n={n}",
       f"prefill tok/s {pp['avg_ts']:.1f}",
       f"B_eff {st.median(s)*gb:.1f} GB/s",
       f"gpu C {T[0] if T else '?'} -> {max(T) if T else '?'}   VDD_IN W median {st.median(P)/1000 if P else '?':.2f}"]
open(f"{o}/summary.txt","w").write("\n".join(lines)+"\n"); print("\n".join(lines))
PY
echo "-> $OUT"

bash ~/bench.sh ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf day04
cat ~/bench/*_day04/summary.txt

Environment, from the run's own mode.txt

NV Power Mode: 15W
gpu_max_hz 612000000
gpu_cur_hz 306000000
# R36 (release), REVISION: 4.2, GCID: 38685322, BOARD: generic, EABI: aarch64, DATE: Fri Dec 13 00:16:27 UTC 2024
llama.cpp 3466812
date 2026-09-02T00:59:56+00:00

• Script: tools/bench.sh. Outputs: mode.txt, idle.txt, bench.json, tegra.log, summary.txt.
• Model: Llama-3.2-1B-Instruct Q4_K_M, llama-bench -p 512 -n 128 -r 30.
• Power mode: 15W, found, not pinned. jetson_clocks was not run.

The measurement

The harness, four steps

PREDICTED vs MEASURED, TODAY'S CALIBRATION

Where it broke

The ruler was tighter than my 5% guess: 0.6% between p10 and p90. But the 30 samples were not random around the median. They spread from 34.01 to 34.49 and rose with run order. The spread is warm-up, and a harness that discards the first runs would show less. The prefill came in at 1480 tok/s against 1450 from Chapter 2. The 27.5 GB/s is the median times the model bytes, so it is the same measurement written twice.

My notes from Day 4

Written by hand on 2026-09-02, before this chapter was typed. These are my original notes from that day. Where the chapter corrects them, the chapter is current.

Summary

• A benchmark number needs its mode line. At 15W, with a GPU cap of 612 MHz, the 1B decoded at 34.36 tok/s median.
• Report p10 and p90 with the median. Over 30 runs, the 1B spread was 0.22 tok/s. That is 0.6% of the median. The samples climbed from 34.01 to 34.49 in run order, so this spread is mostly warm-up drift. A smaller change is not a result.
• The idle line must be in each measurement. Idle on this board rises after CUDA processes run. Today it was 1346 MB with the desktop on, higher than in Chapter 1. The harness now records it.

Sources

These sources are not mine. Measured numbers came off my own board; specs, estimates and paper figures are labeled as such.

• NVIDIA tegrastats documentation
• Nsight Systems / Compute user guides
• Jetson AI Lab benchmark scripts (they pin MLC/INT4 and publish Super vs non-Super)
• Standard LLM serving split: prefill vs decode