6fe71eba63
All checks were successful
CI/CD / lint (pull_request) Successful in 46s
CI/CD / typecheck (pull_request) Successful in 1m17s
CI/CD / test (pull_request) Successful in 1m12s
CI/CD / build (pull_request) Successful in 1m23s
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V6HNx6cyr4J4CiMGCy1wkF
114 lines
4.9 KiB
Markdown
114 lines
4.9 KiB
Markdown
# thebeast → prod k8s worker — conversion runbook
|
||
|
||
Converting the old desktop **thebeast** (currently Fedora/ZFS desktop at
|
||
`10.0.1.22`) into a `k8s-worker` node of the **prod cluster (k8s0)** via the
|
||
PXE bastion, using the dedicated-data-disk layout (`--longhorn-disk` /
|
||
`--rancher-disk`, introduced with this runbook).
|
||
|
||
Decisions already taken (2026-10-14): Windows disk is sacrificed (unused for
|
||
years, no backup wanted) · cluster = prod k8s0 · OS = Fedora `k8s-worker`
|
||
(thebeast is mainly an AMD-GPU machine: 2× AMD + 1× RTX 3080 Ti, no unified
|
||
memory) · the node will carry a dedicated taint and also serve CI/CD.
|
||
|
||
**VM-drilled 2026-10-14** (on thebeast itself, evidence:
|
||
`drill-evidence/data-disks-2026-10-14.txt`): fresh install lands the exact
|
||
layout below, and a flagless reprovision rediscovers both data disks by label
|
||
and remounts them without reformatting (UUIDs unchanged, marker files intact).
|
||
|
||
## Disk map — ALWAYS by-id, never nvme0n1
|
||
|
||
Kernel NVMe names are **not stable across boots on this machine** (observed).
|
||
Every command below uses `/dev/disk/by-id`.
|
||
|
||
| target | disk | by-id |
|
||
|---|---|---|
|
||
| OS (wipes Windows) | Samsung 970 EVO Plus 1TB | `nvme-Samsung_SSD_970_EVO_Plus_1TB_S4EWNJ0N138234M` |
|
||
| /var/lib/longhorn | Sabrent Rocket 4.0 2TB | `nvme-Sabrent_Rocket_4.0_2TB_7F600711179D89261778` |
|
||
| /var/lib/rancher | Sabrent Rocket 4.0 2TB | `nvme-Sabrent_Rocket_4.0_2TB_7FD4070C1EDA92197301` |
|
||
| DISCONNECT after final shutdown | Samsung 970 EVO Plus 2TB `S4J4NF0NC00809J` | sits on its own motherboard add-on card, PCIe x4, model discontinued |
|
||
|
||
NICs: `enp69s0` `34:97:f6:32:76:e9` (current uplink — this MAC is what the
|
||
bastion will see), `enp70s0` `34:97:f6:31:b3:99` (down).
|
||
|
||
## 0. Before the final shutdown — data
|
||
|
||
**rpool is a 4-wide raidz1 already DEGRADED** (member
|
||
`S4J4NF0R102776W` removed since 2026-09-14). There is **zero redundancy
|
||
left**: one more failure loses the pool, and a disk pulled out of it is NOT
|
||
importable alone. Finish the copies first, then never boot the desktop again.
|
||
|
||
Already safe (verified 2026-10-14): `~/developer` (all 113 repos) + the
|
||
credential/dotfile set → bazzite (Oct 12 rsync, 1 file of drift).
|
||
|
||
Copied to bazzite and rsync-verified 2026-10-14: `~/keyssecrets`,
|
||
`~/dicesecrets`, `~/Videos/canaries` (24G), `~/Pictures`, `~/Documents`.
|
||
`~/opt/vaultbackup` and `~/migration-backup` declared done/not-needed (user,
|
||
2026-10-14).
|
||
|
||
Still open before shutdown:
|
||
- `~/Videos/{Webcam,subscribescreen,Screencasts,music}` (~370M) and
|
||
`~/backups/sogo-final-2026-09-13` — small, not yet copied
|
||
- Verify the Syncthing replicas on synology0-nas for `~/Logseq`,
|
||
`~/DICESharedBoostNote`, `~/Boostnote`, `~/MangaReader_backup` — thebeast's
|
||
syncthing shows 0 connected peers, so the NAS copy is **unverified**
|
||
- Final re-sync of `~/.claude` + atuin (activity postdates the Oct 12 copy)
|
||
|
||
## 1. Physical + network
|
||
|
||
1. Final desktop shutdown.
|
||
2. Disconnect the Samsung 2TB `S4J4NF0NC00809J` (the add-on card).
|
||
3. Move thebeast's switch port to the `192.168.8.0/24` VLAN (same as k8s0 and
|
||
labmaster). The bastion's proxy-DHCP (`dhcp-range=192.168.8.0,proxy`)
|
||
coexists with the real DHCP there — no DHCP changes needed.
|
||
4. Make sure network boot is enabled/first in firmware (UEFI PXE, IPv4).
|
||
|
||
## 2. Discovery
|
||
|
||
PXE boot. Unknown MAC → discovery image → the machine self-reports:
|
||
|
||
```
|
||
labctl provision list # thebeast appears under discovered (MAC 34:97:f6:32:76:e9)
|
||
```
|
||
|
||
## 3. Install
|
||
|
||
```
|
||
labctl provision install 34:97:f6:32:76:e9 thebeast \
|
||
--role k8s-worker \
|
||
--disk /dev/disk/by-id/nvme-Samsung_SSD_970_EVO_Plus_1TB_S4EWNJ0N138234M \
|
||
--longhorn-disk /dev/disk/by-id/nvme-Sabrent_Rocket_4.0_2TB_7F600711179D89261778 \
|
||
--rancher-disk /dev/disk/by-id/nvme-Sabrent_Rocket_4.0_2TB_7FD4070C1EDA92197301
|
||
```
|
||
|
||
(`k8s-worker` is v2-only, so no `--schema-version` needed.)
|
||
|
||
What the layout does: OS disk gets EFI + /boot + `labvg` with the v2 base LVs
|
||
only — **no** longhorn/rancher LVs. Each 2 TB disk gets ONE xfs partition
|
||
spanning it, labelled `lab-longhorn` / `lab-rancher`, mounted at
|
||
`/var/lib/longhorn` / `/var/lib/rancher`. The ZFS signatures on those disks
|
||
are wiped by the install — which is the point of section 0.
|
||
|
||
Reinstall semantics from then on: the labels are on-disk truth. A reprovision
|
||
rediscovers both data disks by label (no flags needed) and mounts them
|
||
`--noformat`; a fresh install refuses to wipe or shadow a labelled disk it was
|
||
not told about.
|
||
|
||
## 4. Join k8s0
|
||
|
||
As on every k8s-worker, k3s is installed with `INSTALL_K3S_SKIP_START`; the
|
||
join is deliberate (see `/root/README` on the node):
|
||
|
||
```
|
||
curl -sfL https://get.k3s.io | K3S_URL=https://<k8s0-server>:6443 K3S_TOKEN=<token> sh -
|
||
```
|
||
|
||
Add the agreed CI/CD taint/labels at join time (k3s agent flags), e.g.:
|
||
|
||
```
|
||
--node-taint dedicated=cicd:NoSchedule --node-label itaz.eu/cicd=true
|
||
```
|
||
|
||
(exact taint TBD with the Pulumi model; GPU bits — ROCm for the 2 AMD cards,
|
||
nvidia for the 3080 Ti — are cluster-side follow-ups in
|
||
thelab-kubernetes-pulumi, not part of the install.)
|