Actions or scenes?

status

closed · mixed

when

2026-09 → 2026-09

came from

Bench lane, Marilyn

A small-model action-recognition study: does a video model read motion or backdrop, and what does motion cost at the edge?

Why it started

Scene bias in action datasets is documented (reading). I wanted to measure it on a fixed budget with controls that can fail, a noise floor stated before any comparison, and every number produced by a workflow with a run id, so a reviewer can re-derive any cell. The first vision project here.

What it is

Kinetics-pretrained video classifiers (S3D, R(2+1)D-18, Swin3D-T) and an ImageNet ResNet-18 per-frame control, fine-tuned on HMDB51 (6,766 clips, 51 classes). The instrument is the shuffle gap: top-1 on ordered frames minus top-1 on the same clips with frames permuted, read against the seed-pair noise floor. A gate must reject a shuffled-label control and confirm the control learns before anything downstream is read. Around it: three splits, two to three seeds, a frame-count axis, an induced Zipf long tail with two remedies, and an export path (ONNX Runtime, TensorRT) with a post-mortem of one quantisation failure. Every phase is a sandboxed unit under a Temporal workflow holding a GPU lease on monte, coordinated from carlo.

Where it stands

Closed. Across three splits and two seeds the shuffle gap is 10.8 to 13.2 points for S3D and 2.2 to 5.0 for Swin3D-T, with the transformer the most accurate (70 to 74 %) and best calibrated (ECE 0.04 to 0.07). The two ranges do not overlap. Backdrop alone is worth about 48 % to a per-frame model. Under darkening, blur and compression the motion signal survives while the backdrop degrades; under frame hold the reverse.¹

At the edge, R(2+1)D-18 in INT8 is lossless at 0.77 ms per clip, 3.4 times fp32. S3D collapsed to chance under per-tensor INT8; the cause is the dynamic-range structure of one stem activation tensor (median 0.46 against an absmax of 45.9, 60× per-channel spread), the fix is one layer in fp32, and the repaired engine is still slower than FP16.¹

Outcome is mixed: two of my controls could not fail and a spec carried a sign error; see losses.

Lineage

• Came from: Bench lane (quantisation deltas with the noise floor stated) and Marilyn (gate on invariants; Temporal only).
• Led to: nothing yet.

Sub-pages

• Method and results · Wins · Losses · Rabbit holes · Reading