The 8 GB Lab Book

Thirty days on one 8 GB NVIDIA Jetson, studied and written out by hand each day: unified memory, CUDA kernels, TensorRT engines, the camera path, small language models, and vision-language-action policies for a robot. What the spec sheet says, what the board does, and the rule I took from each run.

Chapters
30
Measurements
182
Board
Jetson Orin Nano Super, 15W
Author
Aryan Dhawan

Each chapter has the same parts: what I expected, the theory, the setup, the measurement, where it broke, what is not measured, and one rule. Measured numbers are from runs on my own board; specs, estimates and paper figures are labeled as such. I studied each day's topic by hand first, on scrap paper, and copied some days onto my tablet. Those tablet pages are shown at the end of their chapters.

Part IConstraint

  1. There is no VRAM on this boardThe board shows 7,620 MB of one shared pool, not 8,000. The 7B plus the 3B reached 7,441 MB, and swap took 659 MB more. The model load deltas subtract the 643 MB idle of that run.
  2. Power modes, Super, and why TOPS lieAt 15W the caps are 612 MHz GPU and 2133 MHz memory clock, not the poster clocks. Decode used 27 to 37 GB/s of a 68 GB/s roof, and the 4B ran at 11.3 tok/s, not the predicted 16.4.
  3. The fit equation: weights + KV + visionKV cost per token comes from three numbers in the model file: 144 KB for Qwen3-4B, 56 KB for Qwen2.5-7B. llama.cpp's log matched to the byte. The 4B at 32,768 tokens passed my 7,200 MB guard at 7,350 MB and was killed; llama.cpp's fit check said 2288 MiB too much, and I read that as a wall near 16.5k tokens. On 2026-10-04 the same model loaded and answered at 24,576 tokens; my guard killed it at 26,624.
  4. Instrumentation is the only legal languageA harness records the mode, clock cap, release, commit and idle, then takes 30 samples. The 1B decoded at 34.36 tok/s median, p10 34.27, p90 34.49: a 0.6% spread, most of it warm-up drift in run order.
  5. System occupancy by subtraction43 services run at idle with the desktop up. The 'N' processes in the top 15 by RSS add up to 71 MB. I expected 0.5 to 1 GB.

Part IIKernels

  1. Arithmetic intensity of the camera pathThe same 720p frame became a GPU tensor in 11.35 ms with CPU work and a copy. It took 2.27 ms with one uint8 copy and 1.38 ms with no copy. One copy is about 0.1% of the bus. Bandwidth was not the limit.
  2. A fused Ampere preprocess kernelFusing three kernels into one cut the preprocess from 479.9 us to 205.5 us per frame. That is 2.34x, close to the 2.43x of the byte ledger, not 3x. Both versions move bytes at 31 to 33 GB/s, under half the 68 GB/s roof. Three reruns on 2026-10-04 gave 2.33x to 2.34x with the same 32.6 to 32.7 against 31.4 GB/s.
  3. TensorRT plugins when the graph has a holeFolding the preprocess into the resnet18 engine cut end-to-end time from 6.27 ms to 5.31 ms per frame. The torch chain moves 50.7 MB per frame: 0.75 ms at the 68 GB/s roof, 1.55 to 1.6 ms at the 31 to 33 GB/s Chapter 7 measured. The 0.96 ms hole is a lower bound on its cost.
  4. Rust as host, CUDA as deviceSame fused kernel, two hosts, a pageable copy inside the timed loop. Rust's tight-loop p50 was 967 us against 1777 us for Python. In the paced 30 Hz loop the same calls took 4284 us in Rust against 2327 us in Python. Rust's worst tick was 1168 us late against 40 us. The Rust host reached its first kernel in 94 ms against 294 ms. Three reruns on 2026-10-04: tight p50 962 to 995 us Rust against 1,593 to 1,660 Python; paced work 4,102 to 4,295 against 2,394 to 2,418 us, 1.7x to 1.8x; Rust's worst tick 0.5, 0.5 and 31 us, so the 1168 did not repeat.
  5. A reproducible kernel artifactDays 7 to 9 frozen at commit a23e893 in jetson-8gb-kernels. The README table has three rows, one per day. The two parity checks (0.000000) and the occupancy readout are exact. The timings are within noise. The Rust worst tick is not promised. All come from one board.

Part IIIEngines

  1. ONNX contracts and CUDA GraphsBoth engines captured a CUDA graph. The graph saved 0.25 ms on the static engine and 0.27 on the dynamic one. The dynamic engine still lost 0.38 ms, and the graph was not the cause.
  2. FP16 baseline, INT8 on trialINT8 ran 1.38x faster than FP16 on resnet18 and agreed with FP32 on 82.4% of images.
  3. Name the runtime correctlyllama.cpp and ollama decoded Qwen3-1.7B at 23.4 and 22.7 tok/s. Eager fp16 PyTorch gave 7.4, mostly from 3.1x the bytes.
  4. Quantization Orin can executeQ4_K_M decoded 1.18x faster than Q8_0, not 2x, and scored the same 8/10 on the tool cases.
  5. Builder memory vs. runtime memoryThe build peaked at 3167 MB of process RSS, under the 4096 MB workspace flag. The runtimes held 787 and 1172 MB.

Part IVEyes

  1. Keep the frame on the GPU sideThe CPU JPEG path, jpegenc then jpegdec, took 16.95 ms per 720p frame: 51% of the frame budget.
  2. Embeddings as the cheap semantic layerViT-B/32 embeddings ran at 68 fps in 500 MB. At the best threshold, one accepted answer in three was wrong.
  3. Detectors under a RAM capyolo11n ran 19.5 ms slower than yolov8n with fewer flops. The first detector cost 732 MB of RAM, the second 12.
  4. Vision tokens are context tokensFor Qwen2.5-VL-3B, 4.4x the vision tokens gave 2.9x the wait. Peak RAM did not rise with the tokens: the KV was allocated at start, at -c 4096.
  5. The live pipeline examp99 was 18.8 ms against a 33.3 ms period. Only the first two frames were late.

Part VLanguage

  1. Choose the house SLMQwen3-4B Q4 is the house model: tools 5 of 5, restraint 2 of 2. My score picked the 1.7B, and a gate I wrote after the runs overruled it.
  2. Prefill, decode, and the wrong headlineOne Qwen3-4B tool call took 2,061 ms on the board, the median of five, and three reruns on 2026-10-04 gave 2,070 / 2,061 / 2,066 ms (15 calls, p10 1,431, p90 4,076): 796 ms to the first token, then 1,265 ms for 10 JSON tokens at 7.1 tok/s on that call. The 1,755 ms I first printed was arithmetic at the chat decode rate. By subtraction, about 658 ms of the 820 ms median TTFT is the fixed schema.
  3. Tool calling as the acceptance testQwen3-4B passed 8 of 10 frozen tool cases, restraint 2 of 2, in two free runs that matched row for row. The second run carried a --grammar flag that sent no schema.
  4. Serving shape: process, not notebookOn a 10-minute soak, llama-server and ollama started in 40,020 and 41,998 ms and decoded at 10.8 and 11.2 tok/s. TTFT was 126 ms on the cold first request and then a flat band, median 184 ms over 65 requests: the per-request series, read on 2026-10-04, shows no drift. RSS never showed the weights.
  5. Packing STT + SLM + TTS + eyesThe floor and three residents, desktop up, took 5,178 of 7,620 MiB and left 2,442. RSS saw 21% of the SLM's cost and 90% of the eyes'. A tool call with the brain resident took 1,532 ms in the resident run; the first run read 4,689 with thinking left on. Two stacked reruns on 2026-10-04 read the pile at 5,174 and 5,017 MiB and the tool call at 1,530 and 1,534 ms.

Part VIAction

  1. VLM latency is not VLA latencyOn Qwen2.5-VL-3B one image-to-action step took 2,617 ms, 76% of it prefill. Per token, the JSON action cost what a caption cost.
  2. The Nano-legal VLA menuTwo tests, a named board and a GB that fits, left two names on the menu, NanoVLA and a planner. SmolVLA stays on it because I ran it, not because anyone named this board: 1,602 to 1,744 MiB as a process and 1,265 ms per 50-action chunk at 15W, 7.6x the SwiftVLA authors' Orin time. Beside the full brain it did not fit: the stacked run stopped at 326 MiB available, and two reruns on 2026-10-04 stopped at 422 and 363. SmolVLA plus the house SLM is the pair that fits, 6,853 of 7,620.
  3. Architecture that buys HertzIn the sim, torque held 30 ms stale made the biped fall at 0.83 s, before its first planned step. Staleness cost phase, not distance.
  4. Policy Hz vs. control HzIn the sim the same controller walked 1.19 m at 1000 Hz and fell at every rate I tried from 200 Hz down. 200 Hz lasted longest, 2.46 s.
  5. The budget I would defend in a roomDay 25's residents took 5,178 of 7,620 usable MiB and left 2,442. Day 27's stacked run showed that wedge is not room for a VLA: SmolVLA beside the brain stopped at 326 MiB available, and at 422 and 363 in two reruns on 2026-10-04. Three holes stay unmeasured: the real inner loop, a real camera, and the power mode.