Embeddings as the cheap semantic layer

What I expected

ViT-B/32 sees 50 tokens where B/16 saw 197, so it has approximately a quarter of the flops. That was Prior 1. The B/16 of Chapter 15 ran at 3.5 TFLOP/s. At that rate this model takes 2.5 ms, which is 400 fps. I expected a torch number near 5 ms.

Prior 2 was that the memory is near the weights plus a context. The weights are 88 M params x 2 B = 176 MB (nameplate). I used 400 MB as the estimate.

Prior 3 was that the 50 out-of-set frames would be much lower in cosine than the in-set frames. Some tau would separate them clearly, with precision near 0.9 while recall stays near 0.9.

The theory

Flops of B/32, derived; then P and R as counts

MACs, ViT-B/32 AT 224 (derived)

224 / 32 = 7: 49 patches + cls = 50 tokens
linears: 12 x 50 x 768^2 x 12 layers = 4.25 G
attention: 2 x 50^2 x 768 x 12 = 0.05 G
4.3 GMAC = 8.6 GFLOP per frame
= 8.6 GFLOP / 14.7 ms = 0.59 TFLOP/s

P and r are counts (arithmetic)

at tau 0.500: P 0.500, R 1.000
everything accepted, half correct:
  so 50 in-set queries, 50 out-of-set
TP = 50 x R; accepted = TP / P
false accepts = accepted - TP
= recall moves in steps of 1/50 = 0.02

The result is 0.59 TFLOP/s against the 3.5 of Chapter 15 on the same tensor cores, six times less. The model has a quarter of the flops and takes 1.5x the time of the B/16 . This looks like the eager overhead of Chapter 13, now on a vision model. 12 layers x approximately 15 kernels = 180 launches (est.), and 14.7 ms / 180 = 80 us each.

The 68 FPS, placed (measured, then arithmetic)

per frame: 14.7 ms
at 3.5 TF/s: 2.5 ms
of a 30 fps period: 44%
with Day 16's P3: 58%

2.5 ms: what a TensorRT engine of this model should give at Chapter 15's rate (arithmetic). 68 fps is enough for a walker, so I am not chasing it today; the 6x sits there for the day the budget gets tight.

The setup

Run (board, ~5 min)

python3 ~/day17_embed.py
cat ~/day17/results.json | head -40

From results.json, the file the run wrote

"script": "day17_embed"
"note": "CIFAR10 classes are stand-ins for campus classes; no campus dataset available."
"model": { "arch": "ViT-B-32", "pretrained": "laion2b_s34b_b79k", "device": "cuda", "dtype": ...
"gallery": { "gallery_per_class": 20, "centroid_from_first_n_gallery": 10 }
"tau_sweep": [ ... ]
"best_tau": { "tau": 0.68, "precision": 0.648, "recall": 0.92, "f1": 0.76 }
"fps_bench": { "input_res": 224, "batch": 1, "frames": 200, "median_latency_ms": 14.723 }
"memory": { "tegrastats_ram_before_mb": 1637, "tegrastats_ram_after_mb": 2137, "tegrastats_ra...

The measurement

What	Expected	Measured	Note
best tau: P / R / F1	-	0.680: 0.65 / 0.92 / 0.76	5 CIFAR classes as stand-ins
embedding fps / ms	-	68 / 14.7	ViT-B/32 fp16, batch 1
MB (tegrastats delta / torch peak)	-	500 / 309

Measured on the board unless the row says estimate or predicted.

Fig 17.1Precision and recall vs accept threshold tau, ViT-B/32 embeddings, 5 classes + none-class.

The tau sweep, in counts

tau	P	R	TP	accepted	wrong accepts
0.500	0.500	1.000	50	100	50
0.600	0.515	1.000	50	97	47
0.660	0.620	0.980	49	79	30
0.680	0.648	0.920	46	71	25
0.700	0.646	0.840	42	65	23
0.720	0.673	0.740	37	55	18

P and R measured; the counts are them multiplied out. From 0.680 to 0.720 the threshold removes 7 wrong answers and 9 right ones. Above the F1 peak, tau throws away more truth than error.

right, accepted

46

wrong, accepted

25

refused

29

tau 0.680, per 100 glances. The third bar is 'I do not know'. The confusion matrix in results.json puts all 25 wrong accepts in the none-of-these row: 12 called bird, 11 cat, 1 airplane, 1 automobile. No in-set frame was accepted as a wrong class; the 4 lost in-set frames were refused. 25 of the 50 strangers, 50%, walk through.

Where it broke

For Prior 1, the result was 14.7 ms, three times my 5. This is the same error as in the Chapter 13 PyTorch column. I predicted with the rate of an engine, and I ran without an engine.

For Prior 2, gave 500 MB and torch gave 309. The 176 MB of weights is a third of what the board lost. The torch counter does not include the 191 MB that the driver and the context use. The pie gets the 500.

Prior 3 had the largest error. At the F1 peak, one accepted answer in three is wrong (25 of 71, above). The out-of-set frames are not far from the centroids. They are other CIFAR animals and vehicles at 32 px, enlarged to 224. The is also not sure.

I called 'precision 0.65' a result. In counts, it is 25 confident wrong answers for each 100 glances. All of them are out-of-set frames that the threshold accepted. A walker that acts on those answers is worse than a walker that says nothing. The product is not tau. The product is the refusal rate that I can accept.

5 classes x 10 query frames is 50 positives, so recall moves in steps of 0.02. The F1 'peak' at 0.680 is a tie with 0.660 at 0.760. The sample size causes this plateau, not the model.

I ran eager torch and compared it mentally with a TensorRT engine. The 6x comes from that mix, not from the model. Chapter 19 must compare engine with engine, or eager with eager.

Summary

The threshold sets the refusal rate. At tau 0.680, 25 of 100 glances were confident wrong answers. All 25 were out-of-set frames that the threshold accepted, half the strangers.
68 fps for 500 MB is the reference. If an item on Chapter 19 costs more, it must justify the cost with an answer that the embedding cannot give.
The test frames were CIFAR10 stand-ins at 32 px, enlarged to 224. Out-of-set frames sat close to the class centroids, so cosine alone did not separate them.

Sources

These sources are not mine. Measured numbers came off my own board; specs, estimates and paper figures are labeled as such.

CLIP / DINOv2 papers (high level)
Jetson AI Lab vision benchmark table (ViT, DINOv2, CLIP fps on Nano Super vs original)
Hierarchical VLA / C-VLA idea of a cheap gate