The fleet's disk layout was decided by literal role-name comparisons at 17
sites across 8 files. That is why `labcontroller` got no rancher LV (it
matched neither "infra" nor "worker"), why labmaster carried an empty 728 GiB
Longhorn LV, and why a 62 GiB container image tipped that node into
DiskPressure and evicted bastion, labd, cockroach and openbao.
Three generators each hardcoded their own sizes and disagreed: the kickstart
said rancher=120G, ubuntu-autoinstall said 20G, asahi-firstboot said 20G under
a comment asserting the sizes matched install.ks.ts. They did not.
ROLE_REGISTRY already carried `k3s` and `parent` and nothing read them. This
makes capabilities authoritative, moves every size into one shared module, and
adds schema v2 (fixed 250G rancher / 300G longhorn, never --grow). Existing
machines stay v1 forever: schema is detected from the on-disk VG tag, so a v1
box stays v1 even when v2 is requested.
Proven on real VMs, not just unit tests: all four topologies installed and had
their layouts read back off the machines; markers in /home, /srv,
/var/lib/longhorn and /var/lib/rancher survived reinstall; and a 4-node cluster
formed, took real data, and had all three servers rolled one at a time through
evacuate / wipe / reinstall / rejoin with checksums verified before and after
each roll. Drill: 10/10, evidence in migration/drill-evidence/.
Production defects the drill found, none of which a unit test could reach:
- sshOpts() capped every operation at 30s while drainNode asked kubectl for
--timeout=15m, so the transport killed the drain after evicting most pods and
returned a transcript that read like success. `labctl app k3s evacuate` could
never have drained a real node.
- generateAgentConfig put node-role.kubernetes.io/worker in the kubelet's
node-label list, which NodeRestriction rejects, so no agent installed through
K3sModule could start. systemd reported only "did not take the steps required
by its unit configuration".
- toOpContext built K3sConfig by enumerating three keys and silently dropped
the rest, so the k3s version pin did nothing and --allow-single-replica was a
no-op that claimed to override a data-loss guard.
- The Longhorn disk annotation was mangled by shell quoting; with
createDefaultDiskLabeledNodes the node then gets no disk at all, joins, goes
Ready, and silently hosts no replica.
- evacuate drained before evicting, which blocks on Longhorn's PDBs and would
destroy a single-replica volume's only source.
- etcd member ids were parsed from JSON, losing uint64 precision and emitting
decimal where `member remove` needs hex -- at the moment a node is stopped
and about to be wiped.
- get.k3s.io installs latest, two minor versions ahead of the fleet, so a
reinstalled node would rejoin on a newer k3s than its peers and roll an
unplanned two-version control-plane upgrade. K3sConfig.k3sVersion now pins it.
- `labctl app k3s install --role x` mapped anything that was not "worker" to
"infra", so the runbook's own `--role k8s-combo` would have installed a
different role with different disk expectations and no warning.
Also: /var/lib/longhorn/replicas must never be deleted, only its contents, or
every replica placed on the node dies on a non-recursive mkdir -- for ever,
while the node reports Ready, Schedulable and healthy.
etcd is now in the kickstart's package set for k3s server roles, because
`k3s etcdctl` is not a command and a member that needs to recover its own
membership has no internet.
542 unit tests / 39 files.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Exports generateServerConfig/generateAgentConfig (previously module-private) and
adds bin/render-config.ts, which produces /etc/rancher/k3s/config.yaml from env
vars using the PRODUCTION generator, with no SSH/OperationContext.
This is the linchpin of the labsim 3-server-etcd rehearsal: the sim must drive
its nodes through this exact generator, not a parallel set of INSTALL_K3S_EXEC
flags, or it rehearses a mechanism production does not run. Verified rendering
all three shapes -- cluster-init server, joining server (server:+token:), and
dual-stack (dual node-ip + cluster/service CIDRs).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
Most of this estate is SSH-onboard, not PXE-provisioned, and that changes where
the dual-stack guarantee has to come from. os-install-research.md classifies the
two paths: Asahi cannot PXE at all, and the DGX Sparks (spark-2935 and
aitopatom-3a1c, both NVIDIA_DGX_Spark on NVIDIA's own OS) must never be
reinstalled -- see project_dgx_spark_kernel_recovery. Of the five nodes, at most
worker0 and worker2 ever run our install templates.
So for the majority the templates govern nothing, this module is the only
labctl touchpoint, and nothing upstream can promise the vendor OS took its
DHCPv6 lease. Writing node-ip for a missing address does not fail here; it fails
later when k3s will not start, and it reads as a bind error rather than as a
missing address. One ssh round trip turns that into a sentence naming the
address and telling you to check the kea reservation.
This is what makes "out of the box" true for heterogeneous hardware rather than
just for the nodes we image ourselves.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
Phase 2d of cluster dual-stack. This is the file that makes a NEW node correct
by construction: it already owns /etc/rancher/k3s/config.yaml for every node, so
once it emits dual values, a joining node is dual-stack with nothing else to do.
Before this there was nowhere for a node's IPv6 to come from -- K3sConfig had no
v6 field and generateAgentConfig() took no arguments at all.
The property that matters most is the negative one: with no ipv6 and no CIDRs,
addressFamilyLines() returns "" and the output is BYTE-IDENTICAL to what the
five existing nodes already have. Verified by diffing the generator's output for
worker0's parameters against worker0's live config.yaml -- identical. So this
change is inert until dual-stack config is supplied, rather than a flag day that
rewrites five healthy nodes and restarts the cluster to tell it what it knew.
node-ip is written only once there is a second family to name. k3s auto-detects
a sensible IPv4 by itself, and emitting it unconditionally would be that same
pointless rewrite. IPv4 stays first: the supported single-to-dual-stack
conversion preserves the primary family, which is what lets existing Services
keep their ClusterIP.
Agents get their own node-ip and no CIDRs. An agent without one joins a
dual-stack cluster as IPv4-only, gets no IPv6 pod CIDR, and fails later as pods
unreachable over v6 while the node itself reads Ready.
The IPv6 address is also added as a TLS SAN. Without it anything reaching the
apiserver over v6 -- a peer server joining, kubectl against the v6 address --
fails verification with an error naming the certificate rather than the missing
SAN.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
Every cluster this bastion builds was pinned to ipam=kubernetes, which makes it
permanently single-stack: Cilium reads node.spec.podCIDRs, the controller-manager
writes that once at node join and never revises it, so adding IPv6 later fails
with `required IPv6 PodCIDR not available` and needs a rebuild.
cluster-pool puts allocation in the CiliumNode CRD, where the operator can add
an address family to a running node. Proven in labsim (842408c). It is also
Cilium's own default -- ipam=kubernetes was the deviation.
Pool matches the k3s cluster-cidr (10.42.0.0/16, /24 per node), so a node gets
the same CIDR shape it would have had. IPv4-only for now: the IPv6 pool wants an
apiserver carrying an IPv6 service CIDR, which is a separate change.
Mirrors @michal/cilium-values ciliumBaseValues (cilium-values@11ee509), which
these paths track but do not import.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
Replace the CIS-style disableSwap op with enableSwap: activate the
labvg-swap LV with an fstab entry (kubelet runs failSwapOn=false; zram
stays the fast tier, the LV is overflow before OOM kill). Add
growRancherLv: extend labvg/rancher to 120G when the VG has free space,
covering nodes installed before the kickstart sizing change and vanilla
nodes converted to k8s; skips with a clear message when the VG is full.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017f6jyeeDqP4ufyeL3UER9w
The macvlan/VLAN-10 foundation HA depends on (Multus meta-CNI + a lan10
sub-interface + reference CNI plugins) was applied by hand during the HA
migration. Codify both as idempotent lab operations in the networking group,
after installCilium (which already sets cni.exclusive=false + bpf.vlanBypass={10}).
A fresh cluster now reproduces the full macvlan stack.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two changes prompted by today's etcd raft panic on worker1-k8s0
(tocommit out of range, lost-write on follower) and the cascading
disk pressure that surfaced underneath it.
Audit logs to journald
- kube-apiserver now uses audit-log-path=- so audit events flow to
k3s.service stdout and into journald instead of growing files in
/var/log/kubernetes. The previous setup combined apiserver's
internal rotation with a logrotate *.log glob that double-rotated
the rotated files into permanent orphans (observed: 7+ GB).
- New journald-limits operation writes a SystemMaxUse=2G drop-in so
audit volume cannot fill /var/log even under bursty load.
- log-rotation operation repurposed to decommission the obsolete
logrotate rule and reap leftover audit files. Idempotent: no-op
on fresh installs.
Etcd member recovery
- New recoverEtcdMember(broken, peer, hostname) codifies the
documented k3s recovery: stop k3s, etcdctl member remove, wipe
/var/lib/rancher/k3s/server/{db,tls,cred}, restart, poll for
rejoin. Refuses to operate when cluster size < 3 to preserve
quorum.
Tests
- 7 new unit tests covering both decommission paths and the
recovery procedure (54 total, all green).
- install.test.ts asserts the file-based audit args are gone.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add `labctl provision recheck` to refresh hardware info via SSH
- Preserve hardware info in InstalledInfo when install completes
- Fix /ks-auto: run nested %pre scripts from included kickstarts
- Add command-discover WebSocket routing for hw info updates
- Fix k3s join: clean stale TLS/cred when joining existing cluster
- Add --tls-verify=false for internal HTTP registry pushes
- Add fix-ssh-root.sh script for root SSH access on all nodes
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Skip Cilium install for joining servers (already in cluster via daemonset)
- Longhorn annotation for workers: SSH to server node from CLI to apply
kubectl annotation (workers don't have kubectl access)
- Default SSH user for k3s/app commands changed to 'root' (operations
need root privileges, using 'lab' user broke installs)
- k3s server config: cluster-init for initial server, server+token for joins
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Server config now uses cluster-init: true for initial server (enables
embedded etcd). Joining servers get server: + token: in config.
- Cilium install already checks for existing installation, so joining
servers skip it gracefully (the "release name in use" error is non-fatal)
Cluster rebuilt as etcd HA:
worker0-k8s0 control-plane,etcd (initial server, cluster-init)
worker1-k8s0 control-plane,etcd (joined server, Mac Studio aarch64)
spark-2935 worker (DGX Spark, aarch64)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Replace sed with grep -v / awk for fstab manipulation (Asahi Fedora's
sed doesn't support \| delimiter or \? quantifier)
- Use idempotent write_lab_fstab function: removes all old entries first,
comments out conflicting btrfs subvol entries, adds fresh LVM entries
- Fix sed for SSH hardening: use #* instead of \? (POSIX compatible)
- Tested on Mac Studio: no duplicate fstab entries after multiple runs
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Firstboot script defaults admin user to 'lab' instead of bastion's
config.adminUser (which was 'michal' from host system)
- iSCSI OS detection uses case-insensitive match for 'fedora'
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
k3s host prep:
- Add iSCSI initiator install+enable (Fedora: iscsi-initiator-utils,
Ubuntu: open-iscsi) — required by Longhorn
- Add Longhorn disk label to k3s server+agent configs
- Add Longhorn disk annotation operation in post-install hardening
CLI:
- Add `labctl provision asahi` command with interactive install guide
- Change default SSH user from "michal" to "lab" in all commands
- Change admin user in bastion progress callback to "lab"
Asahi provisioning fixes:
- Download installer_data.json locally (installer reads it as file)
- Use REPO_BASE to serve upstream ZIP from bastion (LAN speed)
- Fix ZIP32 vs ZIP64: serve original upstream ZIP unmodified
(our repackaged ZIP used ZIP64 which breaks Asahi urlcache)
- Add /data/asahi-repo fallback path for k3s container PVC mount
- Deploy script syncs asahi-repo to bastion pod after deployment
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove hardcoded devices/directRoutingDevice from Cilium install (let
Cilium auto-detect per node — needed for heterogeneous NICs like eno1 vs enP7s7)
- Set k8sServiceHost=127.0.0.1 k8sServicePort=6444 so Cilium init
containers can reach the API via k3s agent's local LB proxy
- Add node-role.kubernetes.io/worker label to agent config
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Kickstart installs on real hardware failed silently — no error reporting,
only 3 progress callbacks, zero log streaming. This overhaul makes every
install fully observable.
Kickstart improvements:
- Error trapping in %pre and %post (trap ERR sends failure details to bastion)
- 12+ granular progress stages (was 3): SSH, hostname, k3s prep, EFI boot, metadata
- Background log streamer: tails %post output and batch-sends to /api/log
- bastion_log() function for explicit log lines from kickstart scripts
Bastion API:
- POST /api/log — receives raw log lines from kickstart (single or batch)
- InstallLogBuffer — per-MAC ring buffer (2000 lines) + file persistence
- GET /api/logs/:mac — now returns log_lines + log_total alongside stages
- SSE /api/logs/:mac/follow — uses named events (event: stage vs event: log)
- Progress events forwarded to labd via bastion-progress WebSocket message
- Post-provision k3s logs routed through progressBus (was console-only)
dnsmasq fixes found during VM testing:
- HTTP Boot filename: ipxe-real.efi → ipxe.efi (leftover from old 2-stage approach)
- pxe-service directives: only in proxy mode (breaks OVMF PXE in full mode)
- PXEClient vendor class echo for UEFI firmware compatibility
Integration tests:
- PXE boot test: blank UEFI VM → dnsmasq → HTTP Boot → iPXE → bastion → install
- ISO boot test: blank VM boots from bastion-generated ISO → same flow
- Shared helpers: pxe-network (no DHCP, nftables fix), pxe-vm (UEFI + ISO boot)
- test-provision.sh: runs both PXE + ISO tests with prerequisite checks
- 250GB sparse QCOW2 disk (LVM layout needs ~204GB)
201 unit tests passing (11 new).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
