Losing $HOME

On 2026-07-18, one afternoon, a local process running as me renamed a parent directory of my home. Every tmux session died. Every running Python job died. Whatever had not been pushed to git or captured by a backup was gone, including the operating notes I had been keeping for the AI sessions themselves.

The root cause was never established; there was no audit daemon watching at the time.

What came out of it, in order:

Keys rotated the same day.
An incident write-up and recovery runbook.
Nightly backups as a Temporal workflow with an offsite copy and a restore drill that runs weekly on a disposable cloud box, so “backed up” means “restored this week”.
Sandboxing for every workflow unit so a job cannot rename or delete the home directory, and an audit watch on /home.
A rule I now apply everywhere: long jobs run as workflows that survive a session, not as nohup in a terminal.

See Fleet & operations and the lesson a green exit is not done.

What changed: the lab stopped being a laptop with GPUs and became something that could be rebuilt from zero.