Keep the frame on the GPU side

What I expected

Chapter 6 charged 2.27 ms for one u8 crossing plus GPU work. For Prior 1, I budgeted approximately a millisecond for each hop. That is near 3 ms for P2 (convert, cross) and near 2 for P3. For P4 I guessed 10 ms. A software JPEG decoder at approximately 100 Mpx/s (est.) needs 9 ms for 0.92 Mpx.

Prior 2 was that G_cam, which Chapter 1 left as a letter, would be in the tens of MB. That is a small number of RGBA frames in transit.

Prior 3 was that P3 is faster than P2 by one full crossing, two to three ms.

The theory

Bytes per frame, then a two-point line for each path

720p RGBA: 1280 x 720 x 4 B = 3.69 MB
720p I420 source: x 1.5 B = 1.38 MB
480p RGBA: 640 x 480 x 4 = 1.23 MB
one 720p crossing at 68 GB/s: 0.054 ms
= bus time is nothing; Day 6 again

pixels: 0.922 M vs 0.307 M, a 3.0x step
P2: (5.73 - 1.70) / 0.614 Mpx = 6.6 ns/px
  fixed: 1.70 - 6.6 x 0.307 = -0.3 ms, so none
P4: (16.95 - 8.84) / 0.614 = 13.2 ns/px
  fixed: 8.84 - 13.2 x 0.307 = 4.8 ms
= P2 is pure per-pixel; JPEG has a 4.8 ms handshake

6.6 ns per 4-byte pixel is 0.6 GB/s. This rate suggests a per-pixel pass, not a DMA copy of a buffer. The 4.8 ms fixed part of JPEG is for each frame, and the size does not change it.

The setup

python3 ~/day16_nvmm.py
python3 -c "import json;[print(p['name'],p['resolution'],round(p['latency_p50_ms'],2),round(p['latency_p99_ms'],2),round(p['g_cam_mb'],1)) for p in json.load(open('day16/results.json'))['pipelines']]"

python3 ~/day16_nvmm.py --device /dev/video0
# MJPEG mode, the line from the script's notes:
# v4l2src device=/dev/video0 io-mode=2 ! image/jpeg,width=1280,height=720,framerate=30/1 ! jpegdec ! video/x-raw,format=I420 ! nvvidconv ! ...
# a UVC camera lands frames in system memory; it cannot hand nvvidconv an NVMM buffer, so P2 is its best case

"meta": { "script": "day16_nvmm.py", "res_flag": "both", "warmup": 60, "frames": 600 }
"pipelines": [ ... ]
"notes": [ ... ]

The measurement

Memory type on every pad; circle the one crossing

Where it broke

For Prior 1, P2 at 5.73 is double my 3 ms. The error is for each pixel, not for each hop. A convert-and-cross costs 6.6 ns for each pixel at all sizes. P4 at 16.95 was near my 10 only by chance. 4.8 ms of it, 28 percent, is a fixed cost, and I had no model for that part. At 480p that fixed part is 54 percent of 8.84.

For Prior 3, the result was 1.04 ms, not 2 to 3. The resize also has a cost, so the saving is a crossing minus a resize. There is a check that this model is correct. At 480p there is nothing to resize, so P3 should be equal to P2. The result is 1.74 against 1.70. The two values agree.

Prior 2 was correct, but only by a small margin. 27 MB / 3.69 MB = 7.3, so the pool of P2 holds seven RGBA frames. I read the 8 MB of P3 as one 720p frame plus three small frames.

I modelled hops again, after Chapter 6 told me to model waits. The two-resolution line is the tool that I should have used first. One subtraction separates the per-pixel cost from the per-frame cost. The price is one extra run.

The source was a test pattern, not a sensor, so 4.69 ms is the cost of the pipeline alone. A C920 adds a USB transfer and a YUY2 unpack or a CPU jpegdec, and none of these four paths contain them.

results.json records the pipeline string of P4: jpegenc, then jpegdec. The two are software elements on the CPU. The 4.8 ms fixed cost is therefore CPU software, not a hardware block. P4 also paid for an encode, not only a decode.

Summary

• NVMM is not faster memory. It is the same LPDDR5. With NVMM, there is no CPU color pass and no copy on each frame. P3 is faster than P2 because the resize occurs before the buffer crosses. The buffer that crosses is small.
• The JPEG path is the cost of a "read frames from files" demo at 30 fps. It uses 51 percent of the frame budget for a CPU encode and decode. 4.8 ms of it (28 percent) is a fixed cost for each frame. At 480p the fixed part is 54 percent of 8.84 ms.
• The pool numbers are the G_cam term from Chapter 1, which now has a measured value. They are small: single-digit to low-double-digit MB. The cost of the camera is latency, not RAM.
• A USB UVC camera like the C920 delivers YUY2 or MJPEG to system memory. P2 is its predicted best case. P4 is an upper limit for its MJPEG case, because that case needs only the decode. CSI through Argus can use P3.

Sources

These sources are not mine. Measured numbers came off my own board; specs, estimates and paper figures are labeled as such.

• NVIDIA NvBufSurface API
• Jetson CSI camera guide
• DeepStream buffer management
• GStreamer nvvidconv docs
• USB vs CSI distinction in Jetson camera architecture