Rabbit holes
Dead ends
- A chat server and a message bus as the fleet’s nervous system. Mattermost for the agents’ commons, an ActiveMQ Artemis design to make wake-ups structural. Dead end: the org chart was the workflow engine and could not say what “done” was. Became Temporal. Full story on Rabbit Holing. 1
- The control plane on a Mac in a container runtime. Ran for sixteen days. Dead end: deleted at the platform cutover; carlo has been the control plane since.2
- Trusting a composite temperature sensor on USB-bridged drives. The enclosure’s bridge reports a composite that does not track the per-sensor values. Dead end for that alert; the rule is never alert on composite for those drives.3
- A fast drive behind USB for small I/O. 77 IOPS at queue depth one, behind the spinning disk’s 112. Dead end: over USB the cost is round-trip latency, not the medium.4
- Running a frontier open model’s weights locally at a quarter of a token per second. Dead end as anything but an insurance path. 5
Detours that became the standard
- Python orchestration ported to Go, twice. First for the shell-model trainer, then as the shared worker library the whole fleet pins. Became the worker restructure.6
- A one-off restic tag for two unprotected corpora, made by hand on the evening the gap was found. Became the pattern the nightly adopted, after a review of the overlap with the other session’s mechanism. 7
- “Can the drive enclosure be walked to another machine?” A portability drill was written for the ZFS pools: export on carlo, import on monte with nothing from carlo, walk it back. Scheduled after v5 lands. 8
- Renting a single cheap GPU to build the rental pattern. An A10 at $1.29 an hour became a workflow that scales to any rig by changing one field, with a reaper so nothing keeps billing. See the operating stack. 9
- A second AWS region because the first is in the earthquake zone. Started as a what-if during the drill work; became the two-region break-glass at six dollars a month.10
Still open
- Node workers on monte and markov, and moving the trainer onto the lease protocol.
Sources
Numbers on this page are quoted from the lab's own records. The records are private; each note gives the record's date.