dedicated data disks: whole-disk longhorn/rancher volumes, by stable by-id path
Some checks failed
CI/CD / typecheck (pull_request) Failing after 13s
CI/CD / test (pull_request) Failing after 10s
CI/CD / lint (pull_request) Failing after 24s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped

A machine with spare NVMe bays (thebeast: 1TB OS + 2TB + 2TB) wants the
opposite trade from the fixed 250G/300G LVs that exist to FIT a 1TB disk:
give /var/lib/longhorn and /var/lib/rancher a whole disk each and keep the
OS VG small. Until now every generator was single-disk
(ignoredisk --only-use=$DISK, one VG) and had no way to say that.

New in the shared layout (v2 only):
- DataDisks { longhorn?, rancher? }: /dev/disk/by-id paths ONLY, because
  nvme0n1/nvme1n1 enumeration is not stable across boots on multi-NVMe
  hosts and a swapped kernel name formats the wrong disk.
- v2Layout() drops the LV whose mount a dedicated disk supplies.
- validateDataDisks() shared by HTTP API, websocket handler and CLI, so
  all three reject with the same words: by-id enforced, capability-gated
  (no longhorn disk on a control-plane-only node, no rancher disk without
  k3s), no double-booking one disk.

Fedora kickstart (the only generator extended; dataDisks on ubuntu/vyos
are rejected, not dropped):
- %pre resolves each by-id path to tonight's kernel name; a missing disk
  fails the install loudly instead of continuing single-disk.
- Fresh disk: ONE xfs partition spanning it, labelled lab-longhorn /
  lab-rancher. --grow is correct here -- consuming the disk is the point,
  unlike LVs competing inside a shared VG.
- The label is on-disk truth, exactly like the lab-schema-v2 VG tag:
  a reprovision rediscovers labelled disks via blkid with NO flags and
  mounts them --noformat; a fresh install refuses to wipe or shadow a
  labelled disk it was not told about; a label on the wrong disk, or a
  data LV coexisting with a dedicated disk, aborts rather than guesses.
- --disk now also accepts a by-id path (resolved in %pre), and auto-detect
  can never pick a data disk as the OS disk.

Plumbing: InstallConfig.dataDisks, POST /api/install validation,
labctl provision install --longhorn-disk/--rancher-disk, labd forwarding.
The labd command-install path also silently DROPPED schemaVersion (an
explicit --schema-version 2 on a legacy role became v1 on the far side);
it now travels with dataDisks in the protocol message.

migration/THEBEAST-K8S-WORKER.md is the concrete runbook: disk map by
serial, data still to copy off the degraded (zero-redundancy) rpool,
VLAN move, the exact labctl command, and the k8s0 join with CI/CD taint.

569 tests pass (34 new), tsc --build clean, no new lint errors.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6HNx6cyr4J4CiMGCy1wkF
This commit is contained in:
Michal
2026-10-14 18:58:37 +01:00
parent 84836cb8f2
commit 66182c8cb8
17 changed files with 741 additions and 33 deletions

View File

@@ -0,0 +1,107 @@
# thebeast → prod k8s worker — conversion runbook
Converting the old desktop **thebeast** (currently Fedora/ZFS desktop at
`10.0.1.22`) into a `k8s-worker` node of the **prod cluster (k8s0)** via the
PXE bastion, using the dedicated-data-disk layout (`--longhorn-disk` /
`--rancher-disk`, introduced with this runbook).
Decisions already taken (2026-10-14): Windows disk is sacrificed (unused for
years, no backup wanted) · cluster = prod k8s0 · OS = Fedora `k8s-worker`
(thebeast is mainly an AMD-GPU machine: 2× AMD + 1× RTX 3080 Ti, no unified
memory) · the node will carry a dedicated taint and also serve CI/CD.
## Disk map — ALWAYS by-id, never nvme0n1
Kernel NVMe names are **not stable across boots on this machine** (observed).
Every command below uses `/dev/disk/by-id`.
| target | disk | by-id |
|---|---|---|
| OS (wipes Windows) | Samsung 970 EVO Plus 1TB | `nvme-Samsung_SSD_970_EVO_Plus_1TB_S4EWNJ0N138234M` |
| /var/lib/longhorn | Sabrent Rocket 4.0 2TB | `nvme-Sabrent_Rocket_4.0_2TB_7F600711179D89261778` |
| /var/lib/rancher | Sabrent Rocket 4.0 2TB | `nvme-Sabrent_Rocket_4.0_2TB_7FD4070C1EDA92197301` |
| DISCONNECT after final shutdown | Samsung 970 EVO Plus 2TB `S4J4NF0NC00809J` | sits on its own motherboard add-on card, PCIe x4, model discontinued |
NICs: `enp69s0` `34:97:f6:32:76:e9` (current uplink — this MAC is what the
bastion will see), `enp70s0` `34:97:f6:31:b3:99` (down).
## 0. Before the final shutdown — data
**rpool is a 4-wide raidz1 already DEGRADED** (member
`S4J4NF0R102776W` removed since 2026-09-14). There is **zero redundancy
left**: one more failure loses the pool, and a disk pulled out of it is NOT
importable alone. Finish the copies first, then never boot the desktop again.
Already safe (verified 2026-10-14): `~/developer` (all 113 repos) + the
credential/dotfile set → bazzite (Oct 12 rsync, 1 file of drift).
Still to copy before shutdown (≈27 GB + small dirs):
- `~/keyssecrets` (57K) and `~/dicesecrets` (5.8M) — **explicitly flagged
"not copied" in the Oct 12 audit**
- `~/Videos/canaries` (24G drone footage), `~/Pictures` (479M),
`~/Videos/{Webcam,subscribescreen,Screencasts,music}` (~370M),
`~/Documents` (6.8M), `~/opt/vaultbackup` (3.1M, OpenBao backups),
`~/migration-backup` (1.1G, archival), `~/backups/sogo-final-2026-09-13`
- Verify the Syncthing replicas on synology0-nas for `~/Logseq`,
`~/DICESharedBoostNote`, `~/Boostnote`, `~/MangaReader_backup` — thebeast's
syncthing shows 0 connected peers, so the NAS copy is **unverified**
- Final re-sync of `~/.claude` + atuin (activity postdates the Oct 12 copy)
## 1. Physical + network
1. Final desktop shutdown.
2. Disconnect the Samsung 2TB `S4J4NF0NC00809J` (the add-on card).
3. Move thebeast's switch port to the `192.168.8.0/24` VLAN (same as k8s0 and
labmaster). The bastion's proxy-DHCP (`dhcp-range=192.168.8.0,proxy`)
coexists with the real DHCP there — no DHCP changes needed.
4. Make sure network boot is enabled/first in firmware (UEFI PXE, IPv4).
## 2. Discovery
PXE boot. Unknown MAC → discovery image → the machine self-reports:
```
labctl provision list # thebeast appears under discovered (MAC 34:97:f6:32:76:e9)
```
## 3. Install
```
labctl provision install 34:97:f6:32:76:e9 thebeast \
--role k8s-worker \
--disk /dev/disk/by-id/nvme-Samsung_SSD_970_EVO_Plus_1TB_S4EWNJ0N138234M \
--longhorn-disk /dev/disk/by-id/nvme-Sabrent_Rocket_4.0_2TB_7F600711179D89261778 \
--rancher-disk /dev/disk/by-id/nvme-Sabrent_Rocket_4.0_2TB_7FD4070C1EDA92197301
```
(`k8s-worker` is v2-only, so no `--schema-version` needed.)
What the layout does: OS disk gets EFI + /boot + `labvg` with the v2 base LVs
only — **no** longhorn/rancher LVs. Each 2 TB disk gets ONE xfs partition
spanning it, labelled `lab-longhorn` / `lab-rancher`, mounted at
`/var/lib/longhorn` / `/var/lib/rancher`. The ZFS signatures on those disks
are wiped by the install — which is the point of section 0.
Reinstall semantics from then on: the labels are on-disk truth. A reprovision
rediscovers both data disks by label (no flags needed) and mounts them
`--noformat`; a fresh install refuses to wipe or shadow a labelled disk it was
not told about.
## 4. Join k8s0
As on every k8s-worker, k3s is installed with `INSTALL_K3S_SKIP_START`; the
join is deliberate (see `/root/README` on the node):
```
curl -sfL https://get.k3s.io | K3S_URL=https://<k8s0-server>:6443 K3S_TOKEN=<token> sh -
```
Add the agreed CI/CD taint/labels at join time (k3s agent flags), e.g.:
```
--node-taint dedicated=cicd:NoSchedule --node-label itaz.eu/cicd=true
```
(exact taint TBD with the Pulumi model; GPU bits — ROCm for the 2 AMD cards,
nvidia for the 3080 Ti — are cluster-side follow-ups in
thelab-kubernetes-pulumi, not part of the install.)