66182c8cb8
Some checks failed
CI/CD / typecheck (pull_request) Failing after 13s
CI/CD / test (pull_request) Failing after 10s
CI/CD / lint (pull_request) Failing after 24s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
A machine with spare NVMe bays (thebeast: 1TB OS + 2TB + 2TB) wants the
opposite trade from the fixed 250G/300G LVs that exist to FIT a 1TB disk:
give /var/lib/longhorn and /var/lib/rancher a whole disk each and keep the
OS VG small. Until now every generator was single-disk
(ignoredisk --only-use=$DISK, one VG) and had no way to say that.
New in the shared layout (v2 only):
- DataDisks { longhorn?, rancher? }: /dev/disk/by-id paths ONLY, because
nvme0n1/nvme1n1 enumeration is not stable across boots on multi-NVMe
hosts and a swapped kernel name formats the wrong disk.
- v2Layout() drops the LV whose mount a dedicated disk supplies.
- validateDataDisks() shared by HTTP API, websocket handler and CLI, so
all three reject with the same words: by-id enforced, capability-gated
(no longhorn disk on a control-plane-only node, no rancher disk without
k3s), no double-booking one disk.
Fedora kickstart (the only generator extended; dataDisks on ubuntu/vyos
are rejected, not dropped):
- %pre resolves each by-id path to tonight's kernel name; a missing disk
fails the install loudly instead of continuing single-disk.
- Fresh disk: ONE xfs partition spanning it, labelled lab-longhorn /
lab-rancher. --grow is correct here -- consuming the disk is the point,
unlike LVs competing inside a shared VG.
- The label is on-disk truth, exactly like the lab-schema-v2 VG tag:
a reprovision rediscovers labelled disks via blkid with NO flags and
mounts them --noformat; a fresh install refuses to wipe or shadow a
labelled disk it was not told about; a label on the wrong disk, or a
data LV coexisting with a dedicated disk, aborts rather than guesses.
- --disk now also accepts a by-id path (resolved in %pre), and auto-detect
can never pick a data disk as the OS disk.
Plumbing: InstallConfig.dataDisks, POST /api/install validation,
labctl provision install --longhorn-disk/--rancher-disk, labd forwarding.
The labd command-install path also silently DROPPED schemaVersion (an
explicit --schema-version 2 on a legacy role became v1 on the far side);
it now travels with dataDisks in the protocol message.
migration/THEBEAST-K8S-WORKER.md is the concrete runbook: disk map by
serial, data still to copy off the degraded (zero-redundancy) rpool,
VLAN move, the exact labctl command, and the k8s0 join with CI/CD taint.
569 tests pass (34 new), tsc --build clean, no new lint errors.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6HNx6cyr4J4CiMGCy1wkF
108 lines
4.7 KiB
Markdown
108 lines
4.7 KiB
Markdown
# thebeast → prod k8s worker — conversion runbook
|
||
|
||
Converting the old desktop **thebeast** (currently Fedora/ZFS desktop at
|
||
`10.0.1.22`) into a `k8s-worker` node of the **prod cluster (k8s0)** via the
|
||
PXE bastion, using the dedicated-data-disk layout (`--longhorn-disk` /
|
||
`--rancher-disk`, introduced with this runbook).
|
||
|
||
Decisions already taken (2026-10-14): Windows disk is sacrificed (unused for
|
||
years, no backup wanted) · cluster = prod k8s0 · OS = Fedora `k8s-worker`
|
||
(thebeast is mainly an AMD-GPU machine: 2× AMD + 1× RTX 3080 Ti, no unified
|
||
memory) · the node will carry a dedicated taint and also serve CI/CD.
|
||
|
||
## Disk map — ALWAYS by-id, never nvme0n1
|
||
|
||
Kernel NVMe names are **not stable across boots on this machine** (observed).
|
||
Every command below uses `/dev/disk/by-id`.
|
||
|
||
| target | disk | by-id |
|
||
|---|---|---|
|
||
| OS (wipes Windows) | Samsung 970 EVO Plus 1TB | `nvme-Samsung_SSD_970_EVO_Plus_1TB_S4EWNJ0N138234M` |
|
||
| /var/lib/longhorn | Sabrent Rocket 4.0 2TB | `nvme-Sabrent_Rocket_4.0_2TB_7F600711179D89261778` |
|
||
| /var/lib/rancher | Sabrent Rocket 4.0 2TB | `nvme-Sabrent_Rocket_4.0_2TB_7FD4070C1EDA92197301` |
|
||
| DISCONNECT after final shutdown | Samsung 970 EVO Plus 2TB `S4J4NF0NC00809J` | sits on its own motherboard add-on card, PCIe x4, model discontinued |
|
||
|
||
NICs: `enp69s0` `34:97:f6:32:76:e9` (current uplink — this MAC is what the
|
||
bastion will see), `enp70s0` `34:97:f6:31:b3:99` (down).
|
||
|
||
## 0. Before the final shutdown — data
|
||
|
||
**rpool is a 4-wide raidz1 already DEGRADED** (member
|
||
`S4J4NF0R102776W` removed since 2026-09-14). There is **zero redundancy
|
||
left**: one more failure loses the pool, and a disk pulled out of it is NOT
|
||
importable alone. Finish the copies first, then never boot the desktop again.
|
||
|
||
Already safe (verified 2026-10-14): `~/developer` (all 113 repos) + the
|
||
credential/dotfile set → bazzite (Oct 12 rsync, 1 file of drift).
|
||
|
||
Still to copy before shutdown (≈27 GB + small dirs):
|
||
- `~/keyssecrets` (57K) and `~/dicesecrets` (5.8M) — **explicitly flagged
|
||
"not copied" in the Oct 12 audit**
|
||
- `~/Videos/canaries` (24G drone footage), `~/Pictures` (479M),
|
||
`~/Videos/{Webcam,subscribescreen,Screencasts,music}` (~370M),
|
||
`~/Documents` (6.8M), `~/opt/vaultbackup` (3.1M, OpenBao backups),
|
||
`~/migration-backup` (1.1G, archival), `~/backups/sogo-final-2026-09-13`
|
||
- Verify the Syncthing replicas on synology0-nas for `~/Logseq`,
|
||
`~/DICESharedBoostNote`, `~/Boostnote`, `~/MangaReader_backup` — thebeast's
|
||
syncthing shows 0 connected peers, so the NAS copy is **unverified**
|
||
- Final re-sync of `~/.claude` + atuin (activity postdates the Oct 12 copy)
|
||
|
||
## 1. Physical + network
|
||
|
||
1. Final desktop shutdown.
|
||
2. Disconnect the Samsung 2TB `S4J4NF0NC00809J` (the add-on card).
|
||
3. Move thebeast's switch port to the `192.168.8.0/24` VLAN (same as k8s0 and
|
||
labmaster). The bastion's proxy-DHCP (`dhcp-range=192.168.8.0,proxy`)
|
||
coexists with the real DHCP there — no DHCP changes needed.
|
||
4. Make sure network boot is enabled/first in firmware (UEFI PXE, IPv4).
|
||
|
||
## 2. Discovery
|
||
|
||
PXE boot. Unknown MAC → discovery image → the machine self-reports:
|
||
|
||
```
|
||
labctl provision list # thebeast appears under discovered (MAC 34:97:f6:32:76:e9)
|
||
```
|
||
|
||
## 3. Install
|
||
|
||
```
|
||
labctl provision install 34:97:f6:32:76:e9 thebeast \
|
||
--role k8s-worker \
|
||
--disk /dev/disk/by-id/nvme-Samsung_SSD_970_EVO_Plus_1TB_S4EWNJ0N138234M \
|
||
--longhorn-disk /dev/disk/by-id/nvme-Sabrent_Rocket_4.0_2TB_7F600711179D89261778 \
|
||
--rancher-disk /dev/disk/by-id/nvme-Sabrent_Rocket_4.0_2TB_7FD4070C1EDA92197301
|
||
```
|
||
|
||
(`k8s-worker` is v2-only, so no `--schema-version` needed.)
|
||
|
||
What the layout does: OS disk gets EFI + /boot + `labvg` with the v2 base LVs
|
||
only — **no** longhorn/rancher LVs. Each 2 TB disk gets ONE xfs partition
|
||
spanning it, labelled `lab-longhorn` / `lab-rancher`, mounted at
|
||
`/var/lib/longhorn` / `/var/lib/rancher`. The ZFS signatures on those disks
|
||
are wiped by the install — which is the point of section 0.
|
||
|
||
Reinstall semantics from then on: the labels are on-disk truth. A reprovision
|
||
rediscovers both data disks by label (no flags needed) and mounts them
|
||
`--noformat`; a fresh install refuses to wipe or shadow a labelled disk it was
|
||
not told about.
|
||
|
||
## 4. Join k8s0
|
||
|
||
As on every k8s-worker, k3s is installed with `INSTALL_K3S_SKIP_START`; the
|
||
join is deliberate (see `/root/README` on the node):
|
||
|
||
```
|
||
curl -sfL https://get.k3s.io | K3S_URL=https://<k8s0-server>:6443 K3S_TOKEN=<token> sh -
|
||
```
|
||
|
||
Add the agreed CI/CD taint/labels at join time (k3s agent flags), e.g.:
|
||
|
||
```
|
||
--node-taint dedicated=cicd:NoSchedule --node-label itaz.eu/cicd=true
|
||
```
|
||
|
||
(exact taint TBD with the Pulumi model; GPU bits — ROCm for the 2 AMD cards,
|
||
nvidia for the 3080 Ti — are cluster-side follow-ups in
|
||
thelab-kubernetes-pulumi, not part of the install.)
|