FP16 baseline, INT8 on trial

What I expected

The poster gives 2x: tensor cores run at twice the rate. INT8 should halve the FP16 time. For accuracy, I expected only a rounding effect, with 1 or 2% of answers lost.

I also expected that the would run on the CUDA cores at 1.25 TFLOP/s. That gives approximately 2.9 ms for each image at 224x224. I wrote that value down as a floor.

The theory

Three floors, then the bytes, then what a cosine is in degrees

FLOORS AT 224x224 (arithmetic)

3.6 GFLOP per image (1.81 GMAC, published)
CUDA cores fp32: 3.6 G / 1.25 T = 2.9 ms
tensor fp16: 3.6 G / 10 T (est.) = 0.36 ms
tensor int8: 3.6 G / 20 T (est.) = 0.18 ms
= board: 2.67 / 1.34 / 0.97 ms

Bytes per image, batch 1 (arithmetic)

weights 11.7 M x 4 / 2 / 1 B = 46.8 / 23.4 / 11.7 MB
engine files 47.1 / 23.7 / 12.3 MB: the file is the weights
weight floor at 68 GB/s: 0.69 / 0.34 / 0.17 ms
activations not counted: est. 1 to 2x the weights
= byte floors sit on the compute floors (0.36 / 0.18 est.): neither is shown to bind

If the advantage of INT8 is bytes, not , the limit is 2x from half the weights and half the activations. But the byte floors (0.34 and 0.17 ms) are at the level of the compute floors (0.36 and 0.18 ms, est.). The board runs about 4x to 6x above both, so this run does not show which floor binds. 1.38x means that something did not halve. Launches are the same in each engine, 28 x 9 us = 0.25 ms (Chapter 11), and some layers may stay in FP16 (estimate). Without the launches, 1.38x becomes (1.34 - 0.25) / (0.97 - 0.25) = 1.5x.

Why cosine 0.9884 and 82.4% agreement are the same fact

Cos = 0.9884 is an angle of 8.7 degrees between the INT8 and FP32 logit vectors (arithmetic). The 1.0000 of FP16 is less than half a degree. A tilt of 8.7 degrees flips the top-1 when the top two logits are nearer than the distance that the tilt moves them. On out-of-domain images (CIFAR upscaled to 224), the top two are frequently near, so 17.6% flip. The same tilt on confident in-domain frames would flip fewer.

Agreement is a property of the tilt and the data. Cosine is a property of the tilt alone. For this reason, 0.9884 looked like a pass, but it was not a pass.

The trial (measured, 300 runs, 500 eval images, 15W)

INT8 over FP16: 1.38x
FP16 agreement: 99.6%
INT8 agreement: 82.4%
INT8 cosine: 0.9884

The speedup and the flips are two prices of one trade, so they go on the same line.

The setup

Run (board, ~10 min)

python3 tools/day12_int8.py
cat ~/day12/results.json | head -40

From results.json, the file the run wrote

"trt_version": "10.3.0"
"notes": "eval images are CIFAR10 upscaled to 224x224, not real ImageNet or campus frames -- ...
"onnx_ok": true

Script: tools/day12_int8.py
Wrote: day12/results.json
Engines: resnet18_fp32, _fp16, _int8 .engine, plus calib.cache
The script exports resnet18 (ImageNet weights) to static ONNX, builds three engines with the TensorRT Python API, calibrates INT8 on 256 CIFAR10 images at 224, and scores agreement with FP32 on 500 more.

Environment

# from day12/results.json (this run)
"trt_version": "10.3.0"
# board state captured 2026-09-29 (days/environment/board-2026-09-29.txt), not the state of this day's run
L4T R36.4.2, kernel 5.15.148-tegra
NV Power Mode: 15W
CUDA 12.6 (V12.6.68), TensorRT 10.3.0.30, cuDNN 9.3.0.75

The measurement

What	Expected	Measured	Note
FP32: median / p99 ms	-	2.67 / 2.73	engine 47.1 MB, build 24 s
FP16: median / p99 ms	-	1.34 / 1.39	engine 23.7 MB, build 35 s
INT8: median / p99 ms	-	0.97 / 1.04	engine 12.3 MB, build 77 s
FP16 top-1 agreement / cosine vs FP32	-	99.6 % / 1.0000	500 CIFAR test images upscaled
INT8 top-1 agreement / cosine vs FP32	-	82.4 % / 0.9884	500 CIFAR test images upscaled
INT8 speedup over FP16	2x poster	1.38x	resnet18

Measured on the board unless the row says estimate or predicted.

Fig 12.1resnet18 on TensorRT 10.3: latency and agreement with FP32 for FP16 and INT8, 15W.

Three engines on one ruler, and what each one costs in answers

FP32 (TF32?)

2.67 ms

FP16

1.34 ms, 99.6% agree

INT8

0.97 ms, 82.4% agree

Median ms per image, 300 runs. FP32 to FP16 halved the bar, 2.67 to 1.34, as the weights halved. FP16 to INT8 halved the weights again but cut the bar only 1.38x, to 0.97. The agreement only falls at the last step.

256 calibrationimages (CIFAR)
Entropy scalesper tensor
INT8 engine12.3 MB
500 eval images:82.4% agree

The first box decides the third: 256 calibration images set every INT8 scale.

what	my prior	the board
FP32 median / p99	2.9 ms floor	2.67 / 2.73 ms, 47.1 MB, 24 s
FP16 median / p99	-	1.34 / 1.39 ms, 23.7 MB, 35 s
INT8 median / p99	-	0.97 / 1.04 ms, 12.3 MB, 77 s
FP16 agreement / cosine vs FP32	rounding	99.6% / 1.0000
INT8 agreement / cosine vs FP32	1 to 2% lost	82.4% / 0.9884
INT8 speedup over FP16	2x poster	1.38x

Measured: 15W, resnet18 ImageNet weights, TensorRT Python API. The build column is a unit check: 77 - 35 = 42 s is calibration plus int8 tactic search.

Where it broke

FP32 was below its CUDA-core floor, with 2.67 ms against 2.9. At the 612 MHz of Chapter 4, that cannot occur on the CUDA cores. The FP32 build passed no precision flag, and turns on by default for FP32 networks (spec). This puts the convolutions on the tensor cores with a 10-bit mantissa. My 'FP32 baseline' is a TF32 baseline.

Also, INT8 lost 17.6% of the answers, not 2%. The calibrator saw 256 CIFAR images. 256 images from the incorrect domain give 8.7 degrees of tilt on the logits.

I used a poster number as a prior for a batch-1 network. Its byte floor and its compute floor are at the same level. Bytes alone limit the gain to 2x, and launches used a third of that. A TOPS number needs a batch size with it.

I called the FP32 engine a baseline, but I did not compare it with its own floor. It was faster than the floor. It is not the thing that I called it. Label the arithmetic that ran, not the flag that I passed.

0.9884 looked like a pass. The 8.7 degrees and the 17.6% flips are two different measures. Cosine gives the angle between the logit vectors. Agreement counts the answers that changed. Only agreement showed the loss.

My notes from Day 12

Written by hand on 2026-09-13, before this chapter was typed. These are my original notes from that day. Where the chapter corrects them, the chapter is current.

Handwritten notes, Day 12, page 1 — Day 12, page 12026-09-13

Handwritten notes, Day 12, page 2 — Day 12, page 22026-09-13

Summary

The calibrator saw 256 CIFAR images. INT8 then agreed with FP32 on 82.4% of 500 images.
Cosine 0.9884 looked like a pass, but 17.6% of the top-1 answers flipped. When there are no labels, I report agreement with FP32 first and cosine second.
The FP32 engine ran in 2.67 ms, below its 2.9 ms CUDA-core floor. TensorRT kept TF32 on by default, so this baseline is TF32.

Sources

These sources are not mine. Measured numbers came off my own board; specs, estimates and paper figures are labeled as such.

TensorRT quantization / calibrator docs
TensorRT Model Optimizer notes
NVIDIA Developer Forum thread on INT8 regression for ViT-S+DPT on Orin Nano
Ampere tensor-core precision list