dedicated data disks: whole-disk longhorn/rancher volumes, by stable by-id path
Some checks failed
CI/CD / typecheck (pull_request) Failing after 13s
CI/CD / test (pull_request) Failing after 10s
CI/CD / lint (pull_request) Failing after 24s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped

A machine with spare NVMe bays (thebeast: 1TB OS + 2TB + 2TB) wants the
opposite trade from the fixed 250G/300G LVs that exist to FIT a 1TB disk:
give /var/lib/longhorn and /var/lib/rancher a whole disk each and keep the
OS VG small. Until now every generator was single-disk
(ignoredisk --only-use=$DISK, one VG) and had no way to say that.

New in the shared layout (v2 only):
- DataDisks { longhorn?, rancher? }: /dev/disk/by-id paths ONLY, because
  nvme0n1/nvme1n1 enumeration is not stable across boots on multi-NVMe
  hosts and a swapped kernel name formats the wrong disk.
- v2Layout() drops the LV whose mount a dedicated disk supplies.
- validateDataDisks() shared by HTTP API, websocket handler and CLI, so
  all three reject with the same words: by-id enforced, capability-gated
  (no longhorn disk on a control-plane-only node, no rancher disk without
  k3s), no double-booking one disk.

Fedora kickstart (the only generator extended; dataDisks on ubuntu/vyos
are rejected, not dropped):
- %pre resolves each by-id path to tonight's kernel name; a missing disk
  fails the install loudly instead of continuing single-disk.
- Fresh disk: ONE xfs partition spanning it, labelled lab-longhorn /
  lab-rancher. --grow is correct here -- consuming the disk is the point,
  unlike LVs competing inside a shared VG.
- The label is on-disk truth, exactly like the lab-schema-v2 VG tag:
  a reprovision rediscovers labelled disks via blkid with NO flags and
  mounts them --noformat; a fresh install refuses to wipe or shadow a
  labelled disk it was not told about; a label on the wrong disk, or a
  data LV coexisting with a dedicated disk, aborts rather than guesses.
- --disk now also accepts a by-id path (resolved in %pre), and auto-detect
  can never pick a data disk as the OS disk.

Plumbing: InstallConfig.dataDisks, POST /api/install validation,
labctl provision install --longhorn-disk/--rancher-disk, labd forwarding.
The labd command-install path also silently DROPPED schemaVersion (an
explicit --schema-version 2 on a legacy role became v1 on the far side);
it now travels with dataDisks in the protocol message.

migration/THEBEAST-K8S-WORKER.md is the concrete runbook: disk map by
serial, data still to copy off the degraded (zero-redundancy) rpool,
VLAN move, the exact labctl command, and the k8s0 join with CI/CD taint.

569 tests pass (34 new), tsc --build clean, no new lint errors.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6HNx6cyr4J4CiMGCy1wkF
This commit is contained in:
Michal
2026-10-14 18:58:37 +01:00
parent 84836cb8f2
commit 66182c8cb8
17 changed files with 741 additions and 33 deletions

View File

@@ -40,8 +40,14 @@ _labctl() {
"app labcontroller status")
COMPREPLY=($(compgen -W "--user -h --help" -- "$cur"))
return ;;
"app k3s evacuate")
COMPREPLY=($(compgen -W "--user --allow-single-replica -h --help" -- "$cur"))
return ;;
"app k3s admit")
COMPREPLY=($(compgen -W "--user -h --help" -- "$cur"))
return ;;
"app k3s install")
COMPREPLY=($(compgen -W "--role --user --k3s-server --k3s-token -h --help" -- "$cur"))
COMPREPLY=($(compgen -W "--role --user --k3s-server --k3s-token --k3s-version -h --help" -- "$cur"))
return ;;
"app k3s health")
COMPREPLY=($(compgen -W "--user -h --help" -- "$cur"))
@@ -59,7 +65,7 @@ _labctl() {
COMPREPLY=($(compgen -W "-h --help" -- "$cur"))
return ;;
"provision install")
COMPREPLY=($(compgen -W "--role --os --disk --vyos-mgmt --vyos-mgmt-address --vyos-bond --vyos-bond-address --vyos-bond-vrrp --vlan-vip --vyos-vrrp-priority --vyos-mgmt-vlan --vlan --vyos-password --vyos-hwid --vyos-fresh-config -h --help" -- "$cur"))
COMPREPLY=($(compgen -W "--role --os --disk --schema-version --longhorn-disk --rancher-disk --vyos-mgmt --vyos-mgmt-address --vyos-bond --vyos-bond-address --vyos-bond-vrrp --vlan-vip --vyos-vrrp-priority --vyos-mgmt-vlan --vlan --vyos-password --vyos-hwid --vyos-fresh-config --vyos-bundle --vyos-api-key --vyos-api-listen -h --help" -- "$cur"))
return ;;
"provision reprovision")
COMPREPLY=($(compgen -W "--role --os --disk --user -h --help" -- "$cur"))
@@ -101,7 +107,7 @@ _labctl() {
COMPREPLY=($(compgen -W "deploy status -h --help" -- "$cur"))
return ;;
"app k3s")
COMPREPLY=($(compgen -W "install health list kubeconfig -h --help" -- "$cur"))
COMPREPLY=($(compgen -W "evacuate admit install health list kubeconfig -h --help" -- "$cur"))
return ;;
"version")
COMPREPLY=($(compgen -W "-h --help" -- "$cur"))

View File

@@ -131,9 +131,12 @@ complete -c labctl -n "__labctl_using_cmd provision" -a makeiso -d 'Generate a U
complete -c labctl -n "__labctl_using_cmd provision" -a recheck -d 'Refresh hardware info for all installed machines via SSH'
# provision install options
complete -c labctl -n "__labctl_in_cmd provision install" -l role -d 'Machine role (see below)' -xa 'vanilla worker infra labcontroller'
complete -c labctl -n "__labctl_in_cmd provision install" -l role -d 'Machine role (see below)' -xa 'vanilla worker infra labcontroller standalone-cluster k8s-controller k8s-worker k8s-combo'
complete -c labctl -n "__labctl_in_cmd provision install" -l os -d 'Operating system' -xa 'fedora-43 ubuntu-26.04 vyos-rolling'
complete -c labctl -n "__labctl_in_cmd provision install" -l disk -d 'Target disk device (auto-detect if omitted)' -x
complete -c labctl -n "__labctl_in_cmd provision install" -l disk -d 'Target OS disk: kernel name or stable /dev/disk/by-id/ path (auto-detect if omitted)' -x
complete -c labctl -n "__labctl_in_cmd provision install" -l schema-version -d 'Disk layout for a FRESH install (default 1). Ignored on reprovision: the layout already on the disk wins.' -xa '1 2'
complete -c labctl -n "__labctl_in_cmd provision install" -l longhorn-disk -d 'Dedicated whole disk for /var/lib/longhorn, as a stable /dev/disk/by-id/ path (fedora, schema v2 only)' -x
complete -c labctl -n "__labctl_in_cmd provision install" -l rancher-disk -d 'Dedicated whole disk for /var/lib/rancher, as a stable /dev/disk/by-id/ path (fedora, schema v2 only)' -x
complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-mgmt -d 'VyOS: untagged interface the machine PXE boots from (default eth0)' -x
complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-mgmt-address -d 'VyOS: CIDR for the management interface, or \'dhcp\' (default dhcp)' -x
complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-bond -d 'VyOS: comma-separated LACP bond members (must exclude the PXE NIC)' -x
@@ -146,9 +149,12 @@ complete -c labctl -n "__labctl_in_cmd provision install" -l vlan -d 'VyOS: tagg
complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-password -d 'VyOS: password for the \'vyos\' user' -x
complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-hwid -d 'VyOS: pin an interface name to a MAC via hw-id (repeatable)' -x
complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-fresh-config -d 'VyOS: on reinstall, overwrite the preserved config with the generated one'
complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-bundle -d 'VyOS: apply a Pulumi-rendered bundle verbatim (kubernetes-deployment/infra/vyos/vyos-bundle.json). Replaces the derived --vyos-bond/--vlan/... config; secret values are left unset for `pulumi up`.' -x
complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-api-key -d 'VyOS: enable the HTTP API with this key so Pulumi can manage the router from first boot' -x
complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-api-listen -d 'VyOS: address the HTTP API binds to (default: the static management address). Required when management is DHCP; the API is never bound to all interfaces.' -x
# provision reprovision options
complete -c labctl -n "__labctl_in_cmd provision reprovision" -l role -d 'Machine role (see below)' -xa 'vanilla worker infra labcontroller'
complete -c labctl -n "__labctl_in_cmd provision reprovision" -l role -d 'Machine role (see below)' -xa 'vanilla worker infra labcontroller standalone-cluster k8s-controller k8s-worker k8s-combo'
complete -c labctl -n "__labctl_in_cmd provision reprovision" -l os -d 'Operating system' -xa 'fedora-43 ubuntu-26.04 vyos-rolling'
complete -c labctl -n "__labctl_in_cmd provision reprovision" -l disk -d 'Target disk device (auto-detect if omitted)' -x
complete -c labctl -n "__labctl_in_cmd provision reprovision" -l user -d 'SSH user for the reboot (default: vyos for VyOS machines, else current user)' -x
@@ -157,7 +163,7 @@ complete -c labctl -n "__labctl_in_cmd provision reprovision" -l user -d 'SSH us
complete -c labctl -n "__labctl_in_cmd provision debug" -l pxe-boot -d 'Boot installed system via PXE (kernel+initrd from network, root from NVMe)'
# provision register options
complete -c labctl -n "__labctl_in_cmd provision register" -l role -d 'Machine role' -xa 'vanilla worker infra labcontroller'
complete -c labctl -n "__labctl_in_cmd provision register" -l role -d 'Machine role' -xa 'vanilla worker infra labcontroller standalone-cluster k8s-controller k8s-worker k8s-combo'
complete -c labctl -n "__labctl_in_cmd provision register" -l ip -d 'Machine IP address' -x
# provision logs options
@@ -200,16 +206,26 @@ complete -c labctl -n "__labctl_in_cmd app labcontroller deploy" -l crdb-replica
complete -c labctl -n "__labctl_in_cmd app labcontroller status" -l user -d 'SSH user' -x
# app k3s subcommands
complete -c labctl -n "__labctl_using_cmd app k3s" -a evacuate -d 'Cordon, evict Longhorn replicas and drain so a node can be safely wiped'
complete -c labctl -n "__labctl_using_cmd app k3s" -a admit -d 'Return an evacuated node to service: uncordon, clear eviction, restore its Longhorn disk'
complete -c labctl -n "__labctl_using_cmd app k3s" -a install -d 'Install k3s on a target machine (hostname, IP, or MAC)'
complete -c labctl -n "__labctl_using_cmd app k3s" -a health -d 'Check k3s health (all hosts if no target given)'
complete -c labctl -n "__labctl_using_cmd app k3s" -a list -d 'List installed machines and their k3s status'
complete -c labctl -n "__labctl_using_cmd app k3s" -a kubeconfig -d 'Fetch kubeconfig from a target and merge into ~/.kube/config'
# app k3s evacuate options
complete -c labctl -n "__labctl_in_cmd app k3s evacuate" -l user -d 'SSH user' -x
complete -c labctl -n "__labctl_in_cmd app k3s evacuate" -l allow-single-replica -d 'permit evacuating volumes that have only one replica (rebuilds from the only copy; there is no backup target)'
# app k3s admit options
complete -c labctl -n "__labctl_in_cmd app k3s admit" -l user -d 'SSH user' -x
# app k3s install options
complete -c labctl -n "__labctl_in_cmd app k3s install" -l role -d 'k3s role: infra (server) or worker (agent)' -x
complete -c labctl -n "__labctl_in_cmd app k3s install" -l role -d 'k3s role: infra, labcontroller, standalone-cluster, k8s-controller, k8s-combo (servers) or worker, k8s-worker (agents)' -x
complete -c labctl -n "__labctl_in_cmd app k3s install" -l user -d 'SSH user' -x
complete -c labctl -n "__labctl_in_cmd app k3s install" -l k3s-server -d 'k3s server URL (required for worker role)' -x
complete -c labctl -n "__labctl_in_cmd app k3s install" -l k3s-token -d 'k3s join token (required for worker role)' -x
complete -c labctl -n "__labctl_in_cmd app k3s install" -l k3s-server -d 'k3s server URL (required for agent roles, and to JOIN as a server)' -x
complete -c labctl -n "__labctl_in_cmd app k3s install" -l k3s-token -d 'k3s join token; on a first server it SEEDS the cluster token' -x
complete -c labctl -n "__labctl_in_cmd app k3s install" -l k3s-version -d 'exact k3s version, e.g. v1.34.6+k3s1. Unset installs LATEST, which on a rejoining node can be several minor versions ahead of the cluster' -x
# app k3s health options
complete -c labctl -n "__labctl_in_cmd app k3s health" -l user -d 'SSH user' -x