66182c8cb8
A machine with spare NVMe bays (thebeast: 1TB OS + 2TB + 2TB) wants the
opposite trade from the fixed 250G/300G LVs that exist to FIT a 1TB disk:
give /var/lib/longhorn and /var/lib/rancher a whole disk each and keep the
OS VG small. Until now every generator was single-disk
(ignoredisk --only-use=$DISK, one VG) and had no way to say that.
New in the shared layout (v2 only):
- DataDisks { longhorn?, rancher? }: /dev/disk/by-id paths ONLY, because
nvme0n1/nvme1n1 enumeration is not stable across boots on multi-NVMe
hosts and a swapped kernel name formats the wrong disk.
- v2Layout() drops the LV whose mount a dedicated disk supplies.
- validateDataDisks() shared by HTTP API, websocket handler and CLI, so
all three reject with the same words: by-id enforced, capability-gated
(no longhorn disk on a control-plane-only node, no rancher disk without
k3s), no double-booking one disk.
Fedora kickstart (the only generator extended; dataDisks on ubuntu/vyos
are rejected, not dropped):
- %pre resolves each by-id path to tonight's kernel name; a missing disk
fails the install loudly instead of continuing single-disk.
- Fresh disk: ONE xfs partition spanning it, labelled lab-longhorn /
lab-rancher. --grow is correct here -- consuming the disk is the point,
unlike LVs competing inside a shared VG.
- The label is on-disk truth, exactly like the lab-schema-v2 VG tag:
a reprovision rediscovers labelled disks via blkid with NO flags and
mounts them --noformat; a fresh install refuses to wipe or shadow a
labelled disk it was not told about; a label on the wrong disk, or a
data LV coexisting with a dedicated disk, aborts rather than guesses.
- --disk now also accepts a by-id path (resolved in %pre), and auto-detect
can never pick a data disk as the OS disk.
Plumbing: InstallConfig.dataDisks, POST /api/install validation,
labctl provision install --longhorn-disk/--rancher-disk, labd forwarding.
The labd command-install path also silently DROPPED schemaVersion (an
explicit --schema-version 2 on a legacy role became v1 on the far side);
it now travels with dataDisks in the protocol message.
migration/THEBEAST-K8S-WORKER.md is the concrete runbook: disk map by
serial, data still to copy off the degraded (zero-redundancy) rpool,
VLAN move, the exact labctl command, and the k8s0 join with CI/CD taint.
569 tests pass (34 new), tsc --build clean, no new lint errors.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6HNx6cyr4J4CiMGCy1wkF
4.7 KiB
thebeast → prod k8s worker — conversion runbook
Converting the old desktop thebeast (currently Fedora/ZFS desktop at
10.0.1.22) into a k8s-worker node of the prod cluster (k8s0) via the
PXE bastion, using the dedicated-data-disk layout (--longhorn-disk /
--rancher-disk, introduced with this runbook).
Decisions already taken (2026-10-14): Windows disk is sacrificed (unused for
years, no backup wanted) · cluster = prod k8s0 · OS = Fedora k8s-worker
(thebeast is mainly an AMD-GPU machine: 2× AMD + 1× RTX 3080 Ti, no unified
memory) · the node will carry a dedicated taint and also serve CI/CD.
Disk map — ALWAYS by-id, never nvme0n1
Kernel NVMe names are not stable across boots on this machine (observed).
Every command below uses /dev/disk/by-id.
| target | disk | by-id |
|---|---|---|
| OS (wipes Windows) | Samsung 970 EVO Plus 1TB | nvme-Samsung_SSD_970_EVO_Plus_1TB_S4EWNJ0N138234M |
| /var/lib/longhorn | Sabrent Rocket 4.0 2TB | nvme-Sabrent_Rocket_4.0_2TB_7F600711179D89261778 |
| /var/lib/rancher | Sabrent Rocket 4.0 2TB | nvme-Sabrent_Rocket_4.0_2TB_7FD4070C1EDA92197301 |
| DISCONNECT after final shutdown | Samsung 970 EVO Plus 2TB S4J4NF0NC00809J |
sits on its own motherboard add-on card, PCIe x4, model discontinued |
NICs: enp69s0 34:97:f6:32:76:e9 (current uplink — this MAC is what the
bastion will see), enp70s0 34:97:f6:31:b3:99 (down).
0. Before the final shutdown — data
rpool is a 4-wide raidz1 already DEGRADED (member
S4J4NF0R102776W removed since 2026-09-14). There is zero redundancy
left: one more failure loses the pool, and a disk pulled out of it is NOT
importable alone. Finish the copies first, then never boot the desktop again.
Already safe (verified 2026-10-14): ~/developer (all 113 repos) + the
credential/dotfile set → bazzite (Oct 12 rsync, 1 file of drift).
Still to copy before shutdown (≈27 GB + small dirs):
~/keyssecrets(57K) and~/dicesecrets(5.8M) — explicitly flagged "not copied" in the Oct 12 audit~/Videos/canaries(24G drone footage),~/Pictures(479M),~/Videos/{Webcam,subscribescreen,Screencasts,music}(~370M),~/Documents(6.8M),~/opt/vaultbackup(3.1M, OpenBao backups),~/migration-backup(1.1G, archival),~/backups/sogo-final-2026-09-13- Verify the Syncthing replicas on synology0-nas for
~/Logseq,~/DICESharedBoostNote,~/Boostnote,~/MangaReader_backup— thebeast's syncthing shows 0 connected peers, so the NAS copy is unverified - Final re-sync of
~/.claude+ atuin (activity postdates the Oct 12 copy)
1. Physical + network
- Final desktop shutdown.
- Disconnect the Samsung 2TB
S4J4NF0NC00809J(the add-on card). - Move thebeast's switch port to the
192.168.8.0/24VLAN (same as k8s0 and labmaster). The bastion's proxy-DHCP (dhcp-range=192.168.8.0,proxy) coexists with the real DHCP there — no DHCP changes needed. - Make sure network boot is enabled/first in firmware (UEFI PXE, IPv4).
2. Discovery
PXE boot. Unknown MAC → discovery image → the machine self-reports:
labctl provision list # thebeast appears under discovered (MAC 34:97:f6:32:76:e9)
3. Install
labctl provision install 34:97:f6:32:76:e9 thebeast \
--role k8s-worker \
--disk /dev/disk/by-id/nvme-Samsung_SSD_970_EVO_Plus_1TB_S4EWNJ0N138234M \
--longhorn-disk /dev/disk/by-id/nvme-Sabrent_Rocket_4.0_2TB_7F600711179D89261778 \
--rancher-disk /dev/disk/by-id/nvme-Sabrent_Rocket_4.0_2TB_7FD4070C1EDA92197301
(k8s-worker is v2-only, so no --schema-version needed.)
What the layout does: OS disk gets EFI + /boot + labvg with the v2 base LVs
only — no longhorn/rancher LVs. Each 2 TB disk gets ONE xfs partition
spanning it, labelled lab-longhorn / lab-rancher, mounted at
/var/lib/longhorn / /var/lib/rancher. The ZFS signatures on those disks
are wiped by the install — which is the point of section 0.
Reinstall semantics from then on: the labels are on-disk truth. A reprovision
rediscovers both data disks by label (no flags needed) and mounts them
--noformat; a fresh install refuses to wipe or shadow a labelled disk it was
not told about.
4. Join k8s0
As on every k8s-worker, k3s is installed with INSTALL_K3S_SKIP_START; the
join is deliberate (see /root/README on the node):
curl -sfL https://get.k3s.io | K3S_URL=https://<k8s0-server>:6443 K3S_TOKEN=<token> sh -
Add the agreed CI/CD taint/labels at join time (k3s agent flags), e.g.:
--node-taint dedicated=cicd:NoSchedule --node-label itaz.eu/cicd=true
(exact taint TBD with the Pulumi model; GPU bits — ROCm for the 2 AMD cards, nvidia for the 3080 Ti — are cluster-side follow-ups in thelab-kubernetes-pulumi, not part of the install.)