TensorRT plugins when the graph has a hole

What I expected

I expected that dispatch caused the hole. There are a small number of launches and one sync before the engine can start. I expected a gap of a few tenths of a millisecond. That is approximately the cost of the unfused preprocess on Chapter 7 (0.48 ms).

I also expected that a C++ plugin was necessary. It was not necessary. Each op in the preprocess is already an ONNX op. The graph could include the preprocess, and I wrote nothing at the engine level.

The theory

The hole as a byte ledger: op order is a byte multiplier

cast u8 to f32, 720p: 2.76 + 11.06 = 13.8 MB
permute: a view, 0 MB
resize in f32: 11.06 + 3.69 = 14.7 MB
/255, -mean, /std: 3 x 7.4 = 22.1 MB
= = 50.7 MB and 5 launches per frame

read u8 once: 2.76 MB
write f32 once: 3.69 MB
resize before the cast, in registers

= = 6.45 MB and 1 launch

This is 8x the bytes for the same tensor, because .float() runs before the resize. It makes an 11 MB copy of a 2.76 MB frame. The floor at the 68 GB/s roof is 50.7 / 68 = 0.75 ms. At the 31 to 33 GB/s that Chapter 7 measured, the same bytes take 1.55 to 1.6 ms. The hole measured 0.96 ms, so the hole is less than the chain at measured bus speed.

Two unit checks: the engine file, and the logit noise

resnet18 has 11.7 M weights (torchvision). In fp16 that is 23.4 MB, and the engine file is 23.7 MB. The engine is the weights plus 0.3 MB of plan. The fold of the preprocess added 0.12 MB of plan and 30 s of build. Five cheap ops took 43% more build time: 99 s against 69 s. My guess is the tactic search, not the flops. I did not measure it.

Fp16 keeps about 3 significant digits (10 mantissa bits, spec). A logit of size 1 to 10 carries rounding of 0.0005 to 0.004, half an ulp. 0.0083 is about two ulp at the top of that range, which fits fp16 noise plus two resize implementations disagreeing at the last bit; I did not separate the two. A half-pixel convention error would be ten times that (Chapter 7).

The setup

cd jetson-8gb-kernels/day08 && export PATH=/usr/local/cuda/bin:$PATH && bash run.sh
# export_onnx.py -> build.sh (trtexec) -> e2e.py; writes results.json
python3 -m json.tool results.json | head -60

cat jetson-8gb-kernels/day08/contract.md

Environment and scripts

• Script: jetson-8gb-kernels/day08/run.sh (export_onnx.py, build.sh, e2e.py). Output: results.json.
• Board: Orin Nano Super, 15W (measured.json). Clocks not logged.
• TensorRT 10.3, FP16 engines, resnet18 with random weights.
• L4T and CUDA: not in the Chapter 8 logs. The sources give the board state captured 2026-09-29.

The measurement

The hole is everything between the frame and the engine

Where it broke

The hole is twice the unfused preprocess of Chapter 7. The ledger gives the most likely cause: bytes, not launches. 50.7 MB is 0.75 ms at the roof. At the 31 to 33 GB/s the board gave on Chapter 7, it is 1.55 to 1.6 ms, more than the whole 0.96 ms hole. So the hole cannot be the full cost of the chain. My order was the opposite of the correct order. Bytes are first and dispatch is second.

Also, 0.96 is a lower bound. A minus B does not show the cost that the engine pays for its own inside preprocess. The torch chain costs a minimum of 0.96 ms. If it ran at the Chapter 7 bus speed, it cost about 1.6 ms, and the engine hides about 0.6 ms of inside preprocess. Not measured. The correction is not a faster kernel. The correction is to resize before the cast.

The parser wrote null for the trtexec latency medians, and my notes showed 0.00 ms. The throughput fields did parse: 220.6 qps with the model alone and 196.7 qps with the preprocess inside, both with a CUDA graph. A measurement of exactly zero comes from a parser, not from a board. A null field stays a null field.

The title of Chapter 8 says plugins, but I wrote no plugin. The work is ONNX-op folding. The plugin would be the Chapter 7 kernel in an IPluginV3 wrapper.

My notes from Day 8

Written by hand on 2026-09-08, before this chapter was typed. These are my original notes from that day. Where the chapter corrects them, the chapter is current.

Summary

• The gap between A and B is most likely bytes, not dispatch. The torch chain moves 50.7 MB for each frame. That is 0.75 ms at the roof and 1.55 to 1.6 ms at the bus speed Chapter 7 measured, more than the 0.96 ms gap.
• The engine that owns the camera tensor has a uint8 input. A camera driver can satisfy that contract directly. Only a Python script can satisfy the contract of a float NCHW tensor.
• The rule for a before/after comparison is the same input hash. The two paths ran the same frame ring. The logit difference is 0.0083, within fp16 rounding plus two resize implementations.

Sources

These sources are not mine. Measured numbers came off my own board; specs, estimates and paper figures are labeled as such.

• TensorRT Developer Guide: plugins, precision constraints, tactic selection
• TensorRT 10.3 release notes (the version on this board). Plugin ABIs change between versions.
• Board state captured 2026-09-29, not the state of the Day 8 run: L4T R36.4.2, CUDA 12.6, TensorRT 10.3.0.30, torch 2.8.0