Wins

Backups you can prove

Backed up means restored. A gap audit on 2026-07-25 found two corpora (19 GB and 89 GB) backed up nowhere, proven four ways. Both were covered the same day, and the irreplaceable part, about 5 GB of derived arXiv prose, was restore-verified by diff and checksum on three files including a 60 MB shard. ¹
One nightly workflow for everything. Datasets, restic to three targets, the git server, a disaster-recovery kit and recovery secrets, then the offsite mirror, as one Temporal parent with typed children. A 19-hour hang in August was traced to an unregistered child workflow, recognisable afterwards by a tiny history and a huge state-transition count. ²
The Python nightly was retired on 2026-08-29 after a Go worker ran the same job in parallel for enough green nights to earn the cutover. ³

A weekly fire drill rebuilds the git server from the offsite copy on a disposable cloud box, validates a clone on the box, tears down, and pages on failure or on silence. About a cent a run. ⁴
Break-glass in two regions, the second on another continent because the first shares an earthquake zone with the house. Each region can restore from its own offsite copies, no secret ever on the idle volume, about six dollars a month. The second-region restore has been rehearsed. ⁵
Two drills passed on the first weekend they existed: corpus restore from the Mac, and a full git-server rebuild on a disposable EC2 box. ⁶

A shared Go Temporal library plus tiny single-purpose workers (backup, lease manager, unit runner, deployer), semver-tagged so each worker pins what it depends on. Quality bar: per-file test coverage at 98% or better, measured on a clean clone because the working tree lies. ⁷
A review found seven high-severity issues and all seven were remediated, including a GPU false-kill, before the workers touched production. ⁸
GPU leases with heartbeat and release on every card, proven on carlo in both dev and live clusters, with a read-only skill to check every card in one pass.⁹

Most of carlo’s disks had no exported temperature and one was in no monitoring list at all. A fleet-wide disk collector fixed it; every disk readable, none missing.¹⁰
The paging path was silently dropping alerts. Any Grafana alert with more than two or three instances exceeded the 4 KB message limit and was rejected as an attachment. Fixed fleet-wide the same day. ¹¹
A storage benchmark that measured the ceiling instead of guessing it: 977 MB/s shared across four USB drives, and the hub, not the port, is the limit.¹²
The fleet monitor rebuilt and pushed after the wipe destroyed a tree that had never been in git. ¹³

Every workflow unit runs with the home directory read-only, and an audit watch on /home is live. ¹⁴
Keys rotated the same day, the fleet swept for credentials in URLs, and a fail-closed secret guard on the nightly state publish that aborts if anything key-shaped is staged.¹⁵
A worker restart is safe for live training: the unit launcher reattaches by name, so the orchestrator can be redeployed under a running job. ¹⁶
Network paths that survive a reboot. After a power outage left NFS down for 18 hours, monte’s interfaces were pinned by MAC address and the boot waits for them; verified by rebooting. ¹⁷
Cloud GPU on demand, rent, run, always return, proven live in a day. See the operating stack. ¹⁸

Numbers on this page are quoted from the lab's own records. The records are private; each note gives the record's date.