← Actions or scenes?

Method and results

Three lines: four video classifiers fine-tuned on HMDB51 and evaluated on ordered and frame-permuted clips over three splits, behind a gate that must reject a shuffled-label control. The shuffle gap separates the 3D convnets (10.8 to 16.7 points) from the transformer (2.2 to 5.0) on every split. The export path found one model that quantises losslessly and one that collapses, and the collapse is explained by the activation statistics of a single tensor.

Protocol

• Data. Official HMDB51 archive and the official three splits. Decoded once to 16 uniformly sampled frames at short side 256 into a memory-mapped array with a manifest; each model crops to its native input at train time. Contamination check: identical decoded frame 0 and frame 8 across train and test found two duplicate clips (one each in splits 1 and 2); test copies dropped.¹
• Models. ImageNet ResNet-18 with per-frame logits averaged (11.2 M parameters, the control: frame-order invariant by construction); S3D (8.0 M); R(2+1)D-18 (31.3 M, 112 px); Swin3D-T (27.9 M); the 3D models from Kinetics-400 weights. Full fine-tune, new 51-way head, AdamW, cosine schedule, bf16, 30 epochs, batch 32. The test split doubles as validation; no third split exists.
• Instrument. Shuffle gap = top-1 on ordered frames − top-1 on the same clips with a fixed, seeded permutation of their 16 frames. The seed-pair noise floor (absolute top-1 difference between seeds) is measured before any gap is read; a gap under twice the floor is reported as “within noise”.
• Gate. Before any result is read: a control trained on labels permuted across clips must score at chance (≤ 3 × 1/51) and the control itself must reach ≥ 5 × chance. Labels are permuted across clips, not renamed per class: a consistent renaming is learnable and the control could not fail.
• Provenance. Each phase is a systemd-sandboxed unit launched by a per-host runner under a Temporal workflow that holds one GPU lease across the GPU phases. A unit writes metrics before its done marker; a rejected gate exits with metrics and no marker; a stamp is single-use.

Results: appearance versus motion

Split 1, 30 epochs, two seeds. Gate 0 at 10 epochs: permuted-label controls 1.9 / 2.2 % (chance 1.96 %); single middle frame 44 %; temporal noise floor 2.3 points.

Across splits (S3D and Swin3D-T, two seeds each):

Reading. The 2D ceiling is 48 to 50 %. Every 3D model with permuted frames falls to 51 to 54 %, just above the control, which bounds what the backdrop is worth. Over twelve cells (three splits, two seeds, two models) the S3D gap lies in 10.8 to 13.2 and the Swin3D-T gap in 2.2 to 5.0; the transformer’s advantage on ordered clips is therefore mostly appearance modelling. It is also the best calibrated on every split (ECE 0.04 to 0.07 against 0.14 to 0.19 for S3D).

Results: frame count

R(2+1)D-18 (S3D’s pretrained weights need at least 14 frames), split 1, two seeds, the 16 decoded frames uniformly subsampled for training and every evaluation view:

At four frames the model converges on the 2D control’s 48 to 50 % and its gap falls inside the seed-pair noise floor; the gate’s own verdict flips to “appearance-solvable” for that configuration. The permuted-frame accuracy is flat across the three settings (49 to 57 %), which is the backdrop term; the ordered accuracy is what the frames buy.

Results: edge cost

Batch-one p50 latency on the training card (a datacenter proxy shared with two other workloads; ONNX Runtime and TensorRT rows include host copies inside the timed call; “fp32” may execute in TF32; power is whole-card):

R(2+1)D-18 INT8 (entropy calibration, 192 clips) is 3.4 × fp32 at batch one and 4.5 × in batch-32 throughput with no loss. The 2D model is fastest in plain torch fp16; engine overhead dominates at that size.

The S3D INT8 collapse: mechanism

Per-layer ablation (512-clip subset, fp32 reference 67.4 %): global changes fail (MinMax calibration 2.2 %, 640 calibration clips 1.8 %, FP16 fallback 1.2 %, pooling / concatenation / last-8-layers in fp32 0.8 to 2.9 %); keeping any one of the first three layers in fp32 restores about 68 %; no later single layer helps. The tensor between those layers is the stem’s first activation (64 channels × 16 × 112²).⁴

Its statistics over 192 calibration clips (2.5 × 10⁹ values, reservoir percentiles):

Half of the tensor’s values lie within about one quantisation step of zero at any per-tensor scale that retains the tails, and the per-channel ranges span 60 ×. TensorRT quantises activations per tensor, so no calibrator can serve both the bulk and the tails, and the error compounds through the 174 downstream layers; those layers do not show the spread and quantise cleanly. Fix: the first convolution in fp32 through precision constraints (about 2 % of FLOPs). Citable run:

Both calibrators recover once the layer is pinned, which is consistent with the mechanism being distributional rather than a threshold choice. The repaired INT8 engine is slower than FP16 for this model, so the deployment recommendation for S3D is FP16.

Results: test-time robustness

Checkpoints retrained on split 1 (one seed), then evaluated on the full test split under degradations applied at evaluation time only: darkening (x → x^γ), spatial Gaussian blur, H.264 re-encode at a target bitrate (achieved 572 and 185 kbps at 340 × 256), and frame hold (every k-th frame repeated k times, the 16-frame length kept). Each level is scored on ordered and permuted frames so a gap is reported per level. Top-1 / permuted / gap:

Reading. Darkening, blur and compression cost the per-frame control up to 29 points, while the convnets’ gaps stay within 13 to 18 points through every level except 15-pixel blur, where R(2+1)D-18’s gap halves: the motion signal survives appearance degradation that the backdrop does not. Frame hold inverts this: the control loses under 2 points at a quarter of the frame rate, the convnets lose 19 to 25 and fall to their permuted scores, and Swin3D-T’s gap turns negative at half rate (ordered-but-held frames score below permuted ones). Models trained at the full rate treat held frames as out-of-distribution motion. In absolute terms Swin3D-T is the most robust to every appearance degradation, consistent with reading larger-scale appearance.

Results: induced long tail

Training split resampled to Zipf(α = 1), 413 of 3,570 clips; S3D; two seeds. Head = the ten largest classes; tail = the 41 classes with ≤ 10 clips.

The first run of this table used the inverted sign (13.6 %, tail 0.078); it is kept as a negative result on the losses page.

Limitations

Two to three seeds (one for the robustness checkpoints); one dataset; clip-level evaluation only (16, 8 and 4 frames); degradations synthetic, not captured; a shared datacenter card as the edge proxy; no per-channel activation quantisation or QAT attempted for S3D; no robustness or anonymisation runs. The scene-bias framing follows the literature on the reading page.

Cost

About seventeen GPU-hours in total across all runs, two of them the one-off decode. The largest single unit was one Swin3D-T seed at 30 epochs, twice the wall time and about four and a half times the memory of the convnets.³