lab/migration/THEBEAST-K8S-WORKER.md

# thebeast → prod k8s worker — conversion runbook

Converting the old desktop **thebeast** (currently Fedora/ZFS desktop at
`10.0.1.22`) into a `k8s-worker` node of the **prod cluster (k8s0)** via the
PXE bastion, using the dedicated-data-disk layout (`--longhorn-disk` /
`--rancher-disk`, introduced with this runbook).

Decisions already taken (2026-10-14): Windows disk is sacrificed (unused for
years, no backup wanted) · cluster = prod k8s0 · OS = Fedora `k8s-worker`
(thebeast is mainly an AMD-GPU machine: 2× AMD + 1× RTX 3080 Ti, no unified
memory) · the node will carry a dedicated taint and also serve CI/CD.

## Disk map — ALWAYS by-id, never nvme0n1

Kernel NVMe names are **not stable across boots on this machine** (observed).
Every command below uses `/dev/disk/by-id`.

| target | disk | by-id |
|---|---|---|
| OS (wipes Windows) | Samsung 970 EVO Plus 1TB | `nvme-Samsung_SSD_970_EVO_Plus_1TB_S4EWNJ0N138234M` |
| /var/lib/longhorn | Sabrent Rocket 4.0 2TB | `nvme-Sabrent_Rocket_4.0_2TB_7F600711179D89261778` |
| /var/lib/rancher | Sabrent Rocket 4.0 2TB | `nvme-Sabrent_Rocket_4.0_2TB_7FD4070C1EDA92197301` |
| DISCONNECT after final shutdown | Samsung 970 EVO Plus 2TB `S4J4NF0NC00809J` | sits on its own motherboard add-on card, PCIe x4, model discontinued |

NICs: `enp69s0` `34:97:f6:32:76:e9` (current uplink — this MAC is what the
bastion will see), `enp70s0` `34:97:f6:31:b3:99` (down).

## 0. Before the final shutdown — data

**rpool is a 4-wide raidz1 already DEGRADED** (member
`S4J4NF0R102776W` removed since 2026-09-14). There is **zero redundancy
left**: one more failure loses the pool, and a disk pulled out of it is NOT
importable alone. Finish the copies first, then never boot the desktop again.

Already safe (verified 2026-10-14): `~/developer` (all 113 repos) + the
credential/dotfile set → bazzite (Oct 12 rsync, 1 file of drift).

Still to copy before shutdown (≈27 GB + small dirs):
- `~/keyssecrets` (57K) and `~/dicesecrets` (5.8M) — **explicitly flagged
  "not copied" in the Oct 12 audit**
- `~/Videos/canaries` (24G drone footage), `~/Pictures` (479M),
  `~/Videos/{Webcam,subscribescreen,Screencasts,music}` (~370M),
  `~/Documents` (6.8M), `~/opt/vaultbackup` (3.1M, OpenBao backups),
  `~/migration-backup` (1.1G, archival), `~/backups/sogo-final-2026-09-13`
- Verify the Syncthing replicas on synology0-nas for `~/Logseq`,
  `~/DICESharedBoostNote`, `~/Boostnote`, `~/MangaReader_backup` — thebeast's
  syncthing shows 0 connected peers, so the NAS copy is **unverified**
- Final re-sync of `~/.claude` + atuin (activity postdates the Oct 12 copy)

## 1. Physical + network

1. Final desktop shutdown.
2. Disconnect the Samsung 2TB `S4J4NF0NC00809J` (the add-on card).
3. Move thebeast's switch port to the `192.168.8.0/24` VLAN (same as k8s0 and
   labmaster). The bastion's proxy-DHCP (`dhcp-range=192.168.8.0,proxy`)
   coexists with the real DHCP there — no DHCP changes needed.
4. Make sure network boot is enabled/first in firmware (UEFI PXE, IPv4).

## 2. Discovery

PXE boot. Unknown MAC → discovery image → the machine self-reports:

```
labctl provision list        # thebeast appears under discovered (MAC 34:97:f6:32:76:e9)
```

## 3. Install

```
labctl provision install 34:97:f6:32:76:e9 thebeast \
  --role k8s-worker \
  --disk /dev/disk/by-id/nvme-Samsung_SSD_970_EVO_Plus_1TB_S4EWNJ0N138234M \
  --longhorn-disk /dev/disk/by-id/nvme-Sabrent_Rocket_4.0_2TB_7F600711179D89261778 \
  --rancher-disk /dev/disk/by-id/nvme-Sabrent_Rocket_4.0_2TB_7FD4070C1EDA92197301
```

(`k8s-worker` is v2-only, so no `--schema-version` needed.)

What the layout does: OS disk gets EFI + /boot + `labvg` with the v2 base LVs
only — **no** longhorn/rancher LVs. Each 2 TB disk gets ONE xfs partition
spanning it, labelled `lab-longhorn` / `lab-rancher`, mounted at
`/var/lib/longhorn` / `/var/lib/rancher`. The ZFS signatures on those disks
are wiped by the install — which is the point of section 0.

Reinstall semantics from then on: the labels are on-disk truth. A reprovision
rediscovers both data disks by label (no flags needed) and mounts them
`--noformat`; a fresh install refuses to wipe or shadow a labelled disk it was
not told about.

## 4. Join k8s0

As on every k8s-worker, k3s is installed with `INSTALL_K3S_SKIP_START`; the
join is deliberate (see `/root/README` on the node):

```
curl -sfL https://get.k3s.io | K3S_URL=https://<k8s0-server>:6443 K3S_TOKEN=<token> sh -
```

Add the agreed CI/CD taint/labels at join time (k3s agent flags), e.g.:

```
--node-taint dedicated=cicd:NoSchedule --node-label itaz.eu/cicd=true
```

(exact taint TBD with the Pulumi model; GPU bits — ROCm for the 2 AMD cards,
nvidia for the 3080 Ti — are cluster-side follow-ups in
thelab-kubernetes-pulumi, not part of the install.)