The 8 GB Lab Book · Chapter 13

Name the runtime correctly

Board
Jetson Orin Nano Super
Mode
15W
Summary
llama.cpp and ollama decoded Qwen3-1.7B at 23.4 and 22.7 tok/s. Eager fp16 PyTorch gave 7.4, mostly from 3.1x the bytes.

What I expected

I started Prior 1 from Chapter 4. Llama-3.2-1B Q4 gave 34.36 . It has 1.24 B params (its real count), and Qwen3-1.7B has 1.7 B. 34.36 x 1.24 / 1.7 = 25 tok/s for . is the same engine, so the prior for ollama is the same 25.

Prior 2 was that eager would stay at its own ceiling, approximately 20 tok/s (below). I expected that eager would be 20 to 25 percent slower than the Q4 engine. I did not expect a factor of three.

The theory

Decode is a weight stream; the runtime only decides how many bytes

Bytes per token (arithmetic)

fp16: 1.7 B params x 2 B = 3.4 GB
(matches the 3.4 GB download)
Q4: ~1.1 GB, est. (the size ollama pulled)
roof 68 GB/s (Day 2) / bytes:
= ceiling fp16 20 tok/s, Q4 62 tok/s

What each one actually pulled

llama.cpp 23.4 tok/s x 1.1 GB = 26 GB/s
ollama 22.7 x 1.1 = 25 GB/s
PyTorch 7.4 x 3.4 GB = 25 GB/s
each about 37% of the 68 roof
= one wall, 25 GB/s, three runtimes

Each value is the measured tok/s multiplied by the estimated bytes. On Chapter 2, an already used approximately 28 GB/s. Each runtime here gets that number. The fp16 runtime only moves three times the bytes at that bandwidth.

EFFECTIVE GB/s PER RUNTIME (arithmetic from measured tok/s)

llama.cpp
26
ollama
25
PyTorch fp16
25
roof, Day 2
68
The number I was ready to call 'eager is slow' is the same bandwidth the engines get. Per token: 3.4 GB / 68 GB/s = 50 ms if the bus were the only cost; the board took 135 ms (1 / 7.4). The Q4 engine: 1.1 / 68 = 16 ms floor, 43 ms taken. Same ratio, 2.7x, both times.

The setup

Run the three (board, ~15 min incl. pull)
python3 tools/day13_runtimes.py
cat ~/day13/results.json | head -40
ollama without root
cd ~/ollama && OLLAMA_MODELS=~/ollama/models LD_LIBRARY_PATH=~/ollama/lib/ollama:~/ollama/lib/ollama/cuda_jetpack6 bin/ollama serve &
OLLAMA_HOST=localhost:11434 bin/ollama run qwen3:1.7b
From results.json, the file the run wrote
"prompt": "Explain in three sentences why an 8 GB Jetson shares memory between the CPU and th...
"max_tokens": 96
"not_run": { "TensorRT-LLM": "datacenter product; not supported on Jetson. Named here so nobo...
  • Script: tools/day13_runtimes.py
  • Wrote: day13/results.json, plus the llama.cpp and ollama server logs
  • Ollama: started by the script itself; the block above is the same start by hand
  • One prompt, 96 new tokens, temperature 0, three runs per runtime, each server warmed with one short request first. tegrastats runs beside it for the RAM deltas.
Environment
# from tools/day13_runtimes.py and day13/results.json (this run)
llama-server -m Qwen3-1.7B-Q4_K_M.gguf -ngl 99 -c 4096
ollama: qwen3:1.7b, cuda_jetpack6 libraries
PyTorch: Qwen3-1.7B, torch_dtype=float16, device_map=cuda
# this run logged no version strings
# board state captured 2026-09-29 (days/environment/board-2026-09-29.txt), not the state of this day's run
L4T R36.4.2, kernel 5.15.148-tegra
NV Power Mode: 15W
CUDA 12.6 (V12.6.68), TensorRT 10.3.0.30, cuDNN 9.3.0.75
llama.cpp 3466812 (2026-09-01), torch 2.8.0

The measurement

WhatExpectedMeasuredNote
llama.cpp: TTFT / decode-88 ms / 23.4 tok/sRAM +2108 MB, setup ~0.7 (clone + CUDA build 25 min + GGUF download)
ollama: TTFT / decode-63 ms / 22.7 tok/sRAM +1314 MB, setup ~0.3 (tarball, no root; pull 1.1 GB)
PyTorch + transformers (fp16, no quant): TTFT / decode-235 ms / 7.4 tok/sRAM +5310 MB, setup ~0.2 (pip + 3.4 GB download)
TensorRT-LLM-not on Jetsondatacenter product
Measured on the board unless the row says estimate or predicted.
Qwen3-1.7B in three runtimes on the same board: decode tok/s and time to first token, 15W
Fig 13.1Qwen3-1.7B in three runtimes on the same board: decode tok/s and time to first token, 15W.

Six names, one column that says 'engine?'

runtimeon Orinengine?tok/sTTFT msRAM +MB
TensorRT-LLMno----
TRT Edge-LLMyesenginenot set up--
llama.cppyesggml+CUDA23.4882108
ollamayessame one22.7631314
PyTorch fp16yeseager7.42355310
MLC-LLMmaybeenginenot set up--
Numbers in rows 3 to 5 are from the board, 15W, 96 new tokens, median of 3. Edge-LLM and MLC were not set up.
  1. Request
  2. Go server(ollama)
  3. Ggml engine(llama.cpp)
  4. GPUweight stream
The coat: ollama adds the first box only. 23.4 to 22.7 tok/s is the price of the coat, 3 percent. Anything bigger than that between the two is quant or server settings, not the engine; both ran the same context size.

RAM: where the bytes went

PyTorch: +5310 measured, 3400 weights
  leftover 1.9 GB: CUDA context, workspace, KV
llama.cpp: +2108 measured, ~1100 weights (est.)
  leftover ~1.0 GB
ollama: +1314 measured, ~1100 weights
  leftover ~0.2 GB
= same engine, 800 MB apart

The 800 MB is not the ollama wrapper. It is also not the context size, because the two servers ran the same context. My suspect is the process that the system charged for the CUDA context.

Where it broke

Prior 1 was correct. I predicted 25 and measured 23.4, a difference of 7 percent. Parameter scaling from Chapter 4 is a usable measure for Q4 decode.

Prior 2 was incorrect, but not in the way that I expected. Eager decreased to 7.4. fp16 has 3.1x the bytes, and the effective bandwidth did not change, so the bytes explain most of the decrease.

There was a small surprise. The engine is the same, but the time to first token was 88 ms against 63 ms. With the same warm-up request and the same context size on the two servers, ollama gave its first token 25 ms sooner.

I almost wrote 'eager is 3x slower'. The arithmetic shows 3x the bytes at the same 25 GB/s. A fair runtime test keeps the precision constant: fp16 in llama.cpp against fp16 in PyTorch.

My notes from Day 13

Written by hand on 2026-09-14, before this chapter was typed. These are my original notes from that day. Where the chapter corrects them, the chapter is current.

Handwritten notes, Day 13, page 1
Day 13, page 12026-09-14
Handwritten notes, Day 13, page 2
Day 13, page 22026-09-14

Summary

  • Setup time is also a column. llama.cpp took approximately 0.7 h, with a 25 min CUDA build. ollama took approximately 0.3 h from a tarball without root. Eager PyTorch took approximately 0.2 h, with a 3.4 GB download.
  • ollama wraps the same llama.cpp engine: 22.7 against 23.4 tok/s, 3 percent apart. With the same warm-up and the same context size, its first token came 25 ms sooner.
  • The 3.16x gap between eager fp16 and the Q4 engine changed the runtime and the precision at the same time. Most of it is the bytes of fp16, not Python.

Sources

These sources are not mine. Measured numbers came off my own board; specs, estimates and paper figures are labeled as such.

  • NVIDIA TensorRT-LLM GitHub discussion: Jetson not supported on mainline
  • TensorRT Edge-LLM README: C++ inference for Jetson/DRIVE/Physical AI
  • Jetson AI Lab SLM + Edge-LLM tutorials. llama.cpp Jetson containers