dedicated data disks: whole-disk longhorn/rancher volumes, by stable by-id path
A machine with spare NVMe bays (thebeast: 1TB OS + 2TB + 2TB) wants the
opposite trade from the fixed 250G/300G LVs that exist to FIT a 1TB disk:
give /var/lib/longhorn and /var/lib/rancher a whole disk each and keep the
OS VG small. Until now every generator was single-disk
(ignoredisk --only-use=$DISK, one VG) and had no way to say that.
New in the shared layout (v2 only):
- DataDisks { longhorn?, rancher? }: /dev/disk/by-id paths ONLY, because
nvme0n1/nvme1n1 enumeration is not stable across boots on multi-NVMe
hosts and a swapped kernel name formats the wrong disk.
- v2Layout() drops the LV whose mount a dedicated disk supplies.
- validateDataDisks() shared by HTTP API, websocket handler and CLI, so
all three reject with the same words: by-id enforced, capability-gated
(no longhorn disk on a control-plane-only node, no rancher disk without
k3s), no double-booking one disk.
Fedora kickstart (the only generator extended; dataDisks on ubuntu/vyos
are rejected, not dropped):
- %pre resolves each by-id path to tonight's kernel name; a missing disk
fails the install loudly instead of continuing single-disk.
- Fresh disk: ONE xfs partition spanning it, labelled lab-longhorn /
lab-rancher. --grow is correct here -- consuming the disk is the point,
unlike LVs competing inside a shared VG.
- The label is on-disk truth, exactly like the lab-schema-v2 VG tag:
a reprovision rediscovers labelled disks via blkid with NO flags and
mounts them --noformat; a fresh install refuses to wipe or shadow a
labelled disk it was not told about; a label on the wrong disk, or a
data LV coexisting with a dedicated disk, aborts rather than guesses.
- --disk now also accepts a by-id path (resolved in %pre), and auto-detect
can never pick a data disk as the OS disk.
Plumbing: InstallConfig.dataDisks, POST /api/install validation,
labctl provision install --longhorn-disk/--rancher-disk, labd forwarding.
The labd command-install path also silently DROPPED schemaVersion (an
explicit --schema-version 2 on a legacy role became v1 on the far side);
it now travels with dataDisks in the protocol message.
migration/THEBEAST-K8S-WORKER.md is the concrete runbook: disk map by
serial, data still to copy off the degraded (zero-redundancy) rpool,
VLAN move, the exact labctl command, and the k8s0 join with CI/CD taint.
569 tests pass (34 new), tsc --build clean, no new lint errors.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6HNx6cyr4J4CiMGCy1wkF
2026-10-14 18:58:37 +01:00
# thebeast → prod k8s worker — conversion runbook
Converting the old desktop **thebeast ** (currently Fedora/ZFS desktop at
`10.0.1.22` ) into a `k8s-worker` node of the **prod cluster (k8s0) ** via the
PXE bastion, using the dedicated-data-disk layout (`--longhorn-disk` /
`--rancher-disk` , introduced with this runbook).
Decisions already taken (2026-10-14): Windows disk is sacrificed (unused for
years, no backup wanted) · cluster = prod k8s0 · OS = Fedora `k8s-worker`
(thebeast is mainly an AMD-GPU machine: 2× ×
memory) · the node will carry a dedicated taint and also serve CI/CD.
2026-10-14 23:31:15 +01:00
**VM-drilled 2026-10-14** (on thebeast itself, evidence:
`drill-evidence/data-disks-2026-10-14.txt` ): fresh install lands the exact
layout below, and a flagless reprovision rediscovers both data disks by label
and remounts them without reformatting (UUIDs unchanged, marker files intact).
dedicated data disks: whole-disk longhorn/rancher volumes, by stable by-id path
A machine with spare NVMe bays (thebeast: 1TB OS + 2TB + 2TB) wants the
opposite trade from the fixed 250G/300G LVs that exist to FIT a 1TB disk:
give /var/lib/longhorn and /var/lib/rancher a whole disk each and keep the
OS VG small. Until now every generator was single-disk
(ignoredisk --only-use=$DISK, one VG) and had no way to say that.
New in the shared layout (v2 only):
- DataDisks { longhorn?, rancher? }: /dev/disk/by-id paths ONLY, because
nvme0n1/nvme1n1 enumeration is not stable across boots on multi-NVMe
hosts and a swapped kernel name formats the wrong disk.
- v2Layout() drops the LV whose mount a dedicated disk supplies.
- validateDataDisks() shared by HTTP API, websocket handler and CLI, so
all three reject with the same words: by-id enforced, capability-gated
(no longhorn disk on a control-plane-only node, no rancher disk without
k3s), no double-booking one disk.
Fedora kickstart (the only generator extended; dataDisks on ubuntu/vyos
are rejected, not dropped):
- %pre resolves each by-id path to tonight's kernel name; a missing disk
fails the install loudly instead of continuing single-disk.
- Fresh disk: ONE xfs partition spanning it, labelled lab-longhorn /
lab-rancher. --grow is correct here -- consuming the disk is the point,
unlike LVs competing inside a shared VG.
- The label is on-disk truth, exactly like the lab-schema-v2 VG tag:
a reprovision rediscovers labelled disks via blkid with NO flags and
mounts them --noformat; a fresh install refuses to wipe or shadow a
labelled disk it was not told about; a label on the wrong disk, or a
data LV coexisting with a dedicated disk, aborts rather than guesses.
- --disk now also accepts a by-id path (resolved in %pre), and auto-detect
can never pick a data disk as the OS disk.
Plumbing: InstallConfig.dataDisks, POST /api/install validation,
labctl provision install --longhorn-disk/--rancher-disk, labd forwarding.
The labd command-install path also silently DROPPED schemaVersion (an
explicit --schema-version 2 on a legacy role became v1 on the far side);
it now travels with dataDisks in the protocol message.
migration/THEBEAST-K8S-WORKER.md is the concrete runbook: disk map by
serial, data still to copy off the degraded (zero-redundancy) rpool,
VLAN move, the exact labctl command, and the k8s0 join with CI/CD taint.
569 tests pass (34 new), tsc --build clean, no new lint errors.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6HNx6cyr4J4CiMGCy1wkF
2026-10-14 18:58:37 +01:00
## Disk map — ALWAYS by-id, never nvme0n1
Kernel NVMe names are **not stable across boots on this machine ** (observed).
Every command below uses `/dev/disk/by-id` .
| target | disk | by-id |
|---|---|---|
| OS (wipes Windows) | Samsung 970 EVO Plus 1TB | `nvme-Samsung_SSD_970_EVO_Plus_1TB_S4EWNJ0N138234M` |
| /var/lib/longhorn | Sabrent Rocket 4.0 2TB | `nvme-Sabrent_Rocket_4.0_2TB_7F600711179D89261778` |
| /var/lib/rancher | Sabrent Rocket 4.0 2TB | `nvme-Sabrent_Rocket_4.0_2TB_7FD4070C1EDA92197301` |
| DISCONNECT after final shutdown | Samsung 970 EVO Plus 2TB `S4J4NF0NC00809J` | sits on its own motherboard add-on card, PCIe x4, model discontinued |
NICs: `enp69s0` `34:97:f6:32:76:e9` (current uplink — this MAC is what the
bastion will see), `enp70s0` `34:97:f6:31:b3:99` (down).
## 0. Before the final shutdown — data
**rpool is a 4-wide raidz1 already DEGRADED** (member
`S4J4NF0R102776W` removed since 2026-09-14). There is **zero redundancy
left**: one more failure loses the pool, and a disk pulled out of it is NOT
importable alone. Finish the copies first, then never boot the desktop again.
Already safe (verified 2026-10-14): `~/developer` (all 113 repos) + the
credential/dotfile set → bazzite (Oct 12 rsync, 1 file of drift).
2026-10-14 23:31:15 +01:00
Copied to bazzite and rsync-verified 2026-10-14: `~/keyssecrets` ,
`~/dicesecrets` , `~/Videos/canaries` (24G), `~/Pictures` , `~/Documents` .
`~/opt/vaultbackup` and `~/migration-backup` declared done/not-needed (user,
2026-10-14).
Still open before shutdown:
- `~/Videos/{Webcam,subscribescreen,Screencasts,music}` (~370M) and
`~/backups/sogo-final-2026-09-13` — small, not yet copied
dedicated data disks: whole-disk longhorn/rancher volumes, by stable by-id path
A machine with spare NVMe bays (thebeast: 1TB OS + 2TB + 2TB) wants the
opposite trade from the fixed 250G/300G LVs that exist to FIT a 1TB disk:
give /var/lib/longhorn and /var/lib/rancher a whole disk each and keep the
OS VG small. Until now every generator was single-disk
(ignoredisk --only-use=$DISK, one VG) and had no way to say that.
New in the shared layout (v2 only):
- DataDisks { longhorn?, rancher? }: /dev/disk/by-id paths ONLY, because
nvme0n1/nvme1n1 enumeration is not stable across boots on multi-NVMe
hosts and a swapped kernel name formats the wrong disk.
- v2Layout() drops the LV whose mount a dedicated disk supplies.
- validateDataDisks() shared by HTTP API, websocket handler and CLI, so
all three reject with the same words: by-id enforced, capability-gated
(no longhorn disk on a control-plane-only node, no rancher disk without
k3s), no double-booking one disk.
Fedora kickstart (the only generator extended; dataDisks on ubuntu/vyos
are rejected, not dropped):
- %pre resolves each by-id path to tonight's kernel name; a missing disk
fails the install loudly instead of continuing single-disk.
- Fresh disk: ONE xfs partition spanning it, labelled lab-longhorn /
lab-rancher. --grow is correct here -- consuming the disk is the point,
unlike LVs competing inside a shared VG.
- The label is on-disk truth, exactly like the lab-schema-v2 VG tag:
a reprovision rediscovers labelled disks via blkid with NO flags and
mounts them --noformat; a fresh install refuses to wipe or shadow a
labelled disk it was not told about; a label on the wrong disk, or a
data LV coexisting with a dedicated disk, aborts rather than guesses.
- --disk now also accepts a by-id path (resolved in %pre), and auto-detect
can never pick a data disk as the OS disk.
Plumbing: InstallConfig.dataDisks, POST /api/install validation,
labctl provision install --longhorn-disk/--rancher-disk, labd forwarding.
The labd command-install path also silently DROPPED schemaVersion (an
explicit --schema-version 2 on a legacy role became v1 on the far side);
it now travels with dataDisks in the protocol message.
migration/THEBEAST-K8S-WORKER.md is the concrete runbook: disk map by
serial, data still to copy off the degraded (zero-redundancy) rpool,
VLAN move, the exact labctl command, and the k8s0 join with CI/CD taint.
569 tests pass (34 new), tsc --build clean, no new lint errors.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6HNx6cyr4J4CiMGCy1wkF
2026-10-14 18:58:37 +01:00
- Verify the Syncthing replicas on synology0-nas for `~/Logseq` ,
`~/DICESharedBoostNote` , `~/Boostnote` , `~/MangaReader_backup` — thebeast's
syncthing shows 0 connected peers, so the NAS copy is **unverified **
- Final re-sync of `~/.claude` + atuin (activity postdates the Oct 12 copy)
## 1. Physical + network
1. Final desktop shutdown.
2. Disconnect the Samsung 2TB `S4J4NF0NC00809J` (the add-on card).
3. Move thebeast's switch port to the `192.168.8.0/24` VLAN (same as k8s0 and
labmaster). The bastion's proxy-DHCP (`dhcp-range=192.168.8.0,proxy` )
coexists with the real DHCP there — no DHCP changes needed.
4. Make sure network boot is enabled/first in firmware (UEFI PXE, IPv4).
## 2. Discovery
PXE boot. Unknown MAC → discovery image → the machine self-reports:
```
labctl provision list # thebeast appears under discovered (MAC 34:97:f6:32:76:e9)
```
## 3. Install
```
labctl provision install 34:97:f6:32:76:e9 thebeast \
--role k8s-worker \
--disk /dev/disk/by-id/nvme-Samsung_SSD_970_EVO_Plus_1TB_S4EWNJ0N138234M \
--longhorn-disk /dev/disk/by-id/nvme-Sabrent_Rocket_4.0_2TB_7F600711179D89261778 \
--rancher-disk /dev/disk/by-id/nvme-Sabrent_Rocket_4.0_2TB_7FD4070C1EDA92197301
```
(`k8s-worker` is v2-only, so no `--schema-version` needed.)
What the layout does: OS disk gets EFI + /boot + `labvg` with the v2 base LVs
only — **no ** longhorn/rancher LVs. Each 2 TB disk gets ONE xfs partition
spanning it, labelled `lab-longhorn` / `lab-rancher` , mounted at
`/var/lib/longhorn` / `/var/lib/rancher` . The ZFS signatures on those disks
are wiped by the install — which is the point of section 0.
Reinstall semantics from then on: the labels are on-disk truth. A reprovision
rediscovers both data disks by label (no flags needed) and mounts them
`--noformat` ; a fresh install refuses to wipe or shadow a labelled disk it was
not told about.
## 4. Join k8s0
As on every k8s-worker, k3s is installed with `INSTALL_K3S_SKIP_START` ; the
join is deliberate (see `/root/README` on the node):
```
curl -sfL https://get.k3s.io | K3S_URL=https://<k8s0-server>:6443 K3S_TOKEN=<token> sh -
```
Add the agreed CI/CD taint/labels at join time (k3s agent flags), e.g.:
```
--node-taint dedicated=cicd:NoSchedule --node-label itaz.eu/cicd=true
```
(exact taint TBD with the Pulumi model; GPU bits — ROCm for the 2 AMD cards,
nvidia for the 3080 Ti — are cluster-side follow-ups in
thelab-kubernetes-pulumi, not part of the install.)
