Arithmetic intensity of the camera path

What I expected

I assumed that the copy was the cost. Three copies of a frame at 30 fps is a large quantity of data. I expected that bandwidth would limit the slow path.

I also assumed that the forward pass would take much more time than the work on the frame before it. The two assumptions were incorrect, for different reasons.

The theory

One identity gives the time of the full path. You can measure each term. The rule is simple. If the preparation and the copy of the frame take more time than the model, the model is not the problem.

t_frame = t_grab + t_prep + t_h2d + t_infer + t_post

Arithmetic intensity shows which limit an operation reaches first. It is the work for each byte that moves. For the preprocess of one frame:

outputs   3 x 480 x 640 = 0.92 M values
work      about 10 flop each (4-tap resize + normalize) = 9.2 Mflop
bytes     2.76 MB in + 3.69 MB out = 6.45 MB
intensity 9.2 M / 6.45 M = 1.4 flop per byte

The board has a ridge. At this intensity, compute and memory reach their limits at the same time:

compute   1024 CUDA cores x 2 x 0.612 GHz = 1.25 TFLOP/s
memory    68 GB/s at 15W (Chapter 2)
ridge     1.25 T / 68 G = 18 flop per byte

1.4 is much less than 18. The preprocess is memory-bound. Its floor is the time to move its bytes one time: 6.45 MB / 68 GB/s = 0.09 ms. This is the theoretical time for one frame.

The board shares one pool of DRAM, but the CPU and the GPU keep their own caches and their own view of a buffer. A copy here is a cache flush, a DMA, and a sync that makes one engine wait for the other. My hypothesis is that the sync is the expensive part, because it serialises two things that could overlap. This run did not split the sync from the copy.

The setup

The input is the same 1280 x 720 frame. The output is the same float32 3 x 480 x 640 tensor. There are three paths between them. The output spec is the same on the three paths. A comparison is correct only in this condition.

• Path A: the CPU resizes, normalizes and lays out the frame, then the float tensor is copied to the GPU.
• Path B: the raw uint8 frame is copied once from pinned memory, and the GPU does the rest.
• Path C: the frame is already on the GPU side. Nothing crosses.

def path_A(f):
    r = cv2.resize(f, (DST_W, DST_H), interpolation=cv2.INTER_LINEAR)
    x = (r.astype(np.float32) / 255.0 - MEAN) / STD
    x = np.ascontiguousarray(x.transpose(2, 0, 1))[None]
    t = torch.from_numpy(x).to(dev, non_blocking=False)  # f32 tensor: 3*480*640*4 B
    torch.cuda.synchronize()
    return t

pinned = torch.empty((SRC_H, SRC_W, 3), dtype=torch.uint8).pin_memory()

def path_B(f):
    pinned.copy_(torch.from_numpy(f))
    g = pinned.to(dev, non_blocking=True)  # u8 frame: 1280*720*3 B
    t = gpu_pre(g)
    torch.cuda.synchronize()
    return t

def path_C(i):
    t = gpu_pre(gpu_frames[i % 8])  # no bus copy
    torch.cuda.synchronize()
    return t

"A_cpu_then_h2d_f32": { "p50": 11.347, "p99": 12.017, "mean": 11.394 }
"B_h2d_u8_then_gpu":  { "p50":  2.266, "p99":  3.669, "mean":  2.301 }
"C_gpu_resident":     { "p50":  1.376, "p99":  1.578, "mean":  1.384 }
"note": "synthetic 1280x720 frames, 8 distinct rolling patterns, cv2=4.8.0"

I measured 1000 frames for each path, after 50 warm-up frames that I discarded. The frames use 8 different patterns in rotation.

Environment and script

• Script: tools/day06_camera_path.py. Output: day06_results.json.
• Board: Orin Nano Super, L4T R36.4.2, 15W (measured.json). Clocks not logged.
• Libraries: torch 2.8, cv2 4.8.0 (cv2 version from the results note).
• Frames: synthetic 1280x720, 8 distinct rolling patterns, no camera attached.

The measurement

Where it broke

Bandwidth was not the limit. One copy of a frame uses approximately 0.1% of the bus. The cost is that each path is a sequence of short waits. A CPU loop does the pixels one row at a time. There is a sync before the copy and a launch after it. The correct measure is latency for each frame, not bytes for each second.

The theory gave a tenth of a millisecond. The best result on the board was 15 times that value. The worst result was 120 times that value. This difference is the finding.

The second result is about the tail. Path A does its work on the CPU, and its p99 is 12.02 ms. On path C the frame stays on the GPU side, and its p99 is 1.58 ms.

My notes from Day 6

Written by hand on 2026-09-06, before this chapter was typed. These are my original notes from that day. Where the chapter corrects them, the chapter is current.

Summary

• On a board with shared memory, one copy uses a small part of the bus. But path B copies one time and takes 2.27 ms. Path C does not copy and takes 1.38 ms.
• The same frame takes 11.35 ms, 2.27 ms or 1.38 ms. The time changes with the location of the work and the number of copies.
• The preprocess has an intensity of 1.4 flop per byte, and the ridge is 18. It is memory-bound on paper. Its floor is 0.09 ms, and no path came near this value.
• A loop at 30 fps must plan for the p99. Path A does its work on the CPU, and its p99 is 12.02 ms.
• Path A takes 9.97 ms more for each frame than path C, for the same output tensor. Chapter 7 examines the 1.38 ms of path C.

Sources

These sources are not mine. Measured numbers came off my own board; specs, estimates and paper figures are labeled as such.

• NVIDIA Jetson camera and NvBufSurface documentation
• NVIDIA DeepStream, zero-copy paths
• CUDA memory management notes for Tegra
• NVIDIA Developer Forum: pinned memory is not always the lowest latency on Orin