← Actions or scenes?

Reading that shaped it

Ideas here were taken from published work. Each entry says what was taken and what I found when I tried it, so the people who published the idea get the credit and the reader gets the result.

Where this sits in the scene-bias literature

The question is not new; the point of this study was to ask it with controls that can fail.

• Why Can’t I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognition, Jinwoo Choi and colleagues, arXiv:1912.05534, 2019-12-11. Taken: the framing that action datasets are solvable from the backdrop and that a model’s reliance on it is a measurable quantity. Found: on HMDB51 the backdrop is worth about 48 % top-1 to a per-frame model and 53 to 54 % to a 3D model with frames shuffled; the authors’ debiasing losses were not tried here.¹
• Only Time Can Tell: Discovering Temporal Data for Temporal Modeling, Laura Sevilla-Lara and colleagues, arXiv:1907.08340, 2019-07-19. Taken: the method of classifying clips by whether temporal order matters, using shuffled frames as the probe; my shuffle gap is their per-clip test aggregated per model. Found: HMDB51 is only partly temporal by this test, which agrees with their placement of it; the interesting residual is that the transformer’s gap is a quarter of the convnets’.¹
• What Makes a Video a Video: Analyzing Temporal Information in Video Understanding Models and Datasets, De-An Huang and colleagues, CVPR 2018 (paper). Taken: measuring the value of motion at the clip level (16 frames) rather than after video-level pooling, which is the granularity used here. Found: their conclusion that much of the accuracy is available from few frames holds; a single middle frame reached 44 % against 48 % for 16.²
• RESOUND: Towards Action Recognition without Representation Bias, Yingwei Li and colleagues, ECCV 2018 (paper). Taken: the notion that a dataset has a representation bias when a representation other than the intended one scores well on it. Found: by that definition HMDB51 carries a static bias of roughly half the achievable accuracy for these models.

Reading that shaped the models and the long tail

• Long-tail learning via logit adjustment, Aditya Krishna Menon and colleagues, Google, arXiv:2007.07314, 2020-07-14. Taken: the loss-time correction, add τ·log prior to the logits during training and predict with raw logits. Found: it gave the best tail recall of the remedies tried (0.334 against 0.244 with none) at a head cost, once I had fixed the sign; my spec said “subtract”, the implementation obeyed, and the first run collapsed the tail to 0.078. Both rows are kept.²
• Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification, Saining Xie and colleagues, Google, arXiv:1712.04851, 2017-12-13. Taken: S3D, the separable 3D convnet, as the small temporal model and the gate’s temporal role. Found: 66 % top-1 with a 13-point shuffle gap, the worst calibration of the four, and the one model whose INT8 engine collapsed; its separable stem is where the collapse lives.
• A Closer Look at Spatiotemporal Convolutions for Action Recognition, Du Tran and colleagues, Facebook AI Research, arXiv:1711.11248, 2017-11-30. Taken: R(2+1)D-18 as the factorised convnet. Found: the largest shuffle gap (16.5 points) and a lossless INT8 engine 3.4 times faster than fp32; the edge winner.
• Video Swin Transformer, Ze Liu and colleagues, Microsoft Research Asia, arXiv:2106.13230, 2021-06-24. Taken: Swin3D-T as the transformer. Found: the highest accuracy (73 % mean) and the smallest shuffle gap (3.9 points); on this dataset its advantage is appearance, not motion.¹