Losses

The incident

A local process renamed a parent of the home directory on 2026-07-18. Every terminal session and every Python job died; the operating notes for the AI sessions themselves were lost and had to be reconstructed. The root cause was never established; there was no audit daemon then. Cost: days of work and a week of rebuilding. Fixed by everything on the operating stack page: sandboxed units, an audit watch, backups proven by restore. ¹
An out-of-memory kill took down a machine-learning session on 2026-07-25. Fixed: RAM-heavy jobs run under a memory cap or on the big box, never uncapped on the workstation. ²

Two corpora were backed up nowhere while the nightly reported success. No script sourced them, no snapshot in any of three restic repositories, no S3 prefix. Fixed the same day and restore-verified. See a green exit is not done.³
Two backup mechanisms were built in the same hour by two different sessions, one as restic tags and one as a Temporal child, overlapping on the same data. Cost: the derived arXiv prose stored twice offsite until reconciled. Taught: one lane per concern, and check what the other session is doing. ⁴
The nightly hung for 19 hours on an unregistered child workflow. Fixed, and the symptom signature (tiny history, huge state transitions) is now in the runbook.⁵
Ubuntu’s node exporter excludes /mnt by default, so the corpus SSD at 86% and every dataset mount were never in Prometheus. Fixed by overriding the exclusion.⁶
A root filesystem at 89% silently killed CI for every amd64 repo: the kubelet set a disk-pressure taint and the runner could never reschedule. 17 GB of package cache was the culprit. Fixed; the post-reboot checklist now checks taints.⁷

A bare Temporal command targets production. The CLI’s stored defaults point at the live cluster and namespace; every command now passes address and namespace explicitly.⁸
Marking both network interfaces optional silently moved all NFS traffic to the wrong card. The wait-online unit was skipped, mounts raced the 10 GbE link, and NFS trunking pinned every mount to the slow card; only the client address revealed it, and a remount does not undo it. Fixed by dropping the optional flag and bounding the wait.⁹
A gitignored workspace file made lint lie in the working tree; only a clean-clone check is authoritative. ¹⁰
A YAML scalar with a colon-space in it makes the git server silently create no workflow run at all. ¹¹

Numbers on this page are quoted from the lab's own records. The records are private; each note gives the record's date.