Losses
- My first shuffled-label control could not fail. It renamed classes consistently, which a model learns as easily as the real labels, so its score depended on how many classes kept their own name rather than on whether the pipeline worked. Caught in adversarial review before any GPU time; replaced by shuffling labels across clips. Cost: nothing but pride.1
- My original “appearance-solvable” test was true by construction. The 2D control averages per-frame logits, so its shuffle gap is zero on every dataset, and the design would have called every dataset appearance-solvable. Caught in the same review; the finding now comes from a 3D model measured against its own noise floor. The live run confirmed the invariance exactly.1
- I wrote the logit-adjustment sign backwards in the spec. The implementation followed it faithfully and produced 13.6 % top-1 with a collapsed tail; the published method adds τ·log prior at train time rather than subtracting it. Cost: one extra workflow run of about an hour. The wrong-sign row stays in that run’s report as the negative result it is.2
- A weight-loading bug the tests could not see. The rule that tests never download weights meant every training test monkeypatched the loader, and the one registry test saved the wrapper’s own state and reloaded it. The first run on the card failed at the first checkpoint load. Lesson: when a rule forbids the real side effect, test the real shape of the artefact it would have produced. Cost: one smoke run.2
- “Always Temporal” cost a full rerun per fix. The coordinator cannot rerun one phase, so every post-fix number needed a fresh run through gate 0 and a training cell first; the INT8 fix spent about 45 GPU-minutes to obtain a three-minute export. The rule was still right: every number here has a run id, a config hash and a clean commit. The missing phase-resume mode is the design gap it exposed.3
- A measurement that killed its own unit. My first activation probe concatenated 2.5 × 10⁹ values in host memory and the export unit was OOM-killed with an empty log. Cost: one run and the rerun of a full workflow. Became a streaming estimator and a recorded memory peak.4
- Two process slips of my own. Syncing code to the training host with a directory excluded that held tracked files dirtied one run’s provenance and forced a cancel; and one diagnostic ran on the card without a lease, labelled as exploration but a bend of the fleet rule. Both are in the run log; the second is why the citable INT8 number comes from a later workflow run.4
Sources
Numbers on this page are quoted from the lab's own records. The records are private; each note gives the record's date.