thebeast runbook: VM drill passed, data copies done 2026-10-14

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6HNx6cyr4J4CiMGCy1wkF

migration/THEBEAST-K8S-WORKER.md

@@ -10,6 +10,11 @@ years, no backup wanted) · cluster = prod k8s0 · OS = Fedora `k8s-worker`
 (thebeast is mainly an AMD-GPU machine: 2× AMD + 1× RTX 3080 Ti, no unified
 memory) · the node will carry a dedicated taint and also serve CI/CD.
 
+**VM-drilled 2026-10-14** (on thebeast itself, evidence:
+`drill-evidence/data-disks-2026-10-14.txt`): fresh install lands the exact
+layout below, and a flagless reprovision rediscovers both data disks by label
+and remounts them without reformatting (UUIDs unchanged, marker files intact).
+
 ## Disk map — ALWAYS by-id, never nvme0n1
 
 Kernel NVMe names are **not stable across boots on this machine** (observed).
@@ -35,13 +40,14 @@ importable alone. Finish the copies first, then never boot the desktop again.
 Already safe (verified 2026-10-14): `~/developer` (all 113 repos) + the
 credential/dotfile set → bazzite (Oct 12 rsync, 1 file of drift).
 
-Still to copy before shutdown (≈27 GB + small dirs):
-- `~/keyssecrets` (57K) and `~/dicesecrets` (5.8M) — **explicitly flagged
-  "not copied" in the Oct 12 audit**
-- `~/Videos/canaries` (24G drone footage), `~/Pictures` (479M),
-  `~/Videos/{Webcam,subscribescreen,Screencasts,music}` (~370M),
-  `~/Documents` (6.8M), `~/opt/vaultbackup` (3.1M, OpenBao backups),
-  `~/migration-backup` (1.1G, archival), `~/backups/sogo-final-2026-09-13`
+Copied to bazzite and rsync-verified 2026-10-14: `~/keyssecrets`,
+`~/dicesecrets`, `~/Videos/canaries` (24G), `~/Pictures`, `~/Documents`.
+`~/opt/vaultbackup` and `~/migration-backup` declared done/not-needed (user,
+2026-10-14).
+
+Still open before shutdown:
+- `~/Videos/{Webcam,subscribescreen,Screencasts,music}` (~370M) and
+  `~/backups/sogo-final-2026-09-13` — small, not yet copied
 - Verify the Syncthing replicas on synology0-nas for `~/Logseq`,
   `~/DICESharedBoostNote`, `~/Boostnote`, `~/MangaReader_backup` — thebeast's
   syncthing shows 0 connected peers, so the NAS copy is **unverified**

migration/drill-evidence/data-disks-2026-10-14.txt

@@ -0,0 +1,40 @@
+Dedicated data disks drill — fresh install + flagless reprovision, real VM
+run finished: 2026-10-14T23:30+01:00   host: thebeast
+test: bastion/tests/integration/data-disks.test.ts (1 passed, 1164s)
+topology: 1 VM, role k8s-worker schema v2, 1000G OS disk + 2x 20G data disks
+addressed as /dev/disk/by-id/virtio-LHDRILL7F600711 (longhorn) and
+/dev/disk/by-id/virtio-RNDRILL7FD4070C (rancher) — the same by-id contract
+thebeast's NVMe serials use.
+
+Proven:
+- fresh install: labvg = home/root/srv/swap/var/varlog ONLY (no longhorn,
+  no rancher LV); /var/lib/longhorn and /var/lib/rancher are whole-disk xfs
+  partitions labelled lab-longhorn / lab-rancher, each verified to sit on
+  the disk with the matching serial; lab-schema-v2 tag present; fstab by UUID.
+- reprovision with NO dataDisks flags: both disks rediscovered by label,
+  remounted --noformat (filesystem UUIDs identical before/after), marker
+  files intact in /home /srv /var/lib/longhorn /var/lib/rancher, LV set and
+  schema tag unchanged.
+
+=== checkpoints ===
+[22:10:40] [dd-worker] creating VM (1000GB disk + 2 data disk(s), role=k8s-worker, schema=v2)
+[22:13:02] [dd-worker] queueing install...
+[22:15:25] [dd-worker] installing (up to 60 min)...
+[22:21:24] [dd-worker] up at 192.168.251.194
+[22:21:24] [dd-worker] on-disk LVs: home=8G root=32G srv=16G swap=16G var=48G varlog=16G
+[22:21:27] [dd-worker] wrote marker preserve-1792016486955-iu1rih into /home, /srv, /var/lib/longhorn, /var/lib/rancher
+[22:21:27] [dd-worker] reprovisioning (role=k8s-worker, schema: whatever is on disk)...
+[22:29:58] [dd-worker] back up at 192.168.251.194 after reinstall
+[22:29:59] [dd-worker] markers after reinstall: all four mounts carry preserve-1792016486955-iu1rih
+[22:29:59] [dd-worker] on-disk LVs: home=8G root=32G srv=16G swap=16G var=48G varlog=16G
+✓ k8s-worker with longhorn+rancher disks: install, then flagless reprovision rediscovers both (1161s)
+
+Run notes (cost of learning, so nobody pays it twice):
+- the drill machine's bastion imports resolve @lab/shared to dist/ -- run
+  `pnpm build` after switching branches or the API answers with the OLD role
+  registry ("invalid role: 'k8s-worker'").
+- a passphrase-protected default ~/.ssh/id_rsa fails as "SSH not available"
+  with a perfectly healthy machine (BatchMode never offers it without an
+  agent). Use LAB_TEST_SSH_KEY with a dedicated passphrase-less key.
+- run: cd bastion && LAB_TEST_SSH_KEY=~/.ssh/lab-integration_ed25519 \
+    sudo -E npx vitest run -c tests/integration/vitest.config.ts tests/integration/data-disks.test.ts