6 Commits

Author	SHA1	Message	Date
Michal	84836cb8f2	v2 partition schema: capability model, fixed sizing, and a proven rolling migration ... The fleet's disk layout was decided by literal role-name comparisons at 17 sites across 8 files. That is why `labcontroller` got no rancher LV (it matched neither "infra" nor "worker"), why labmaster carried an empty 728 GiB Longhorn LV, and why a 62 GiB container image tipped that node into DiskPressure and evicted bastion, labd, cockroach and openbao. Three generators each hardcoded their own sizes and disagreed: the kickstart said rancher=120G, ubuntu-autoinstall said 20G, asahi-firstboot said 20G under a comment asserting the sizes matched install.ks.ts. They did not. ROLE_REGISTRY already carried `k3s` and `parent` and nothing read them. This makes capabilities authoritative, moves every size into one shared module, and adds schema v2 (fixed 250G rancher / 300G longhorn, never --grow). Existing machines stay v1 forever: schema is detected from the on-disk VG tag, so a v1 box stays v1 even when v2 is requested. Proven on real VMs, not just unit tests: all four topologies installed and had their layouts read back off the machines; markers in /home, /srv, /var/lib/longhorn and /var/lib/rancher survived reinstall; and a 4-node cluster formed, took real data, and had all three servers rolled one at a time through evacuate / wipe / reinstall / rejoin with checksums verified before and after each roll. Drill: 10/10, evidence in migration/drill-evidence/. Production defects the drill found, none of which a unit test could reach: - sshOpts() capped every operation at 30s while drainNode asked kubectl for --timeout=15m, so the transport killed the drain after evicting most pods and returned a transcript that read like success. `labctl app k3s evacuate` could never have drained a real node. - generateAgentConfig put node-role.kubernetes.io/worker in the kubelet's node-label list, which NodeRestriction rejects, so no agent installed through K3sModule could start. systemd reported only "did not take the steps required by its unit configuration". - toOpContext built K3sConfig by enumerating three keys and silently dropped the rest, so the k3s version pin did nothing and --allow-single-replica was a no-op that claimed to override a data-loss guard. - The Longhorn disk annotation was mangled by shell quoting; with createDefaultDiskLabeledNodes the node then gets no disk at all, joins, goes Ready, and silently hosts no replica. - evacuate drained before evicting, which blocks on Longhorn's PDBs and would destroy a single-replica volume's only source. - etcd member ids were parsed from JSON, losing uint64 precision and emitting decimal where `member remove` needs hex -- at the moment a node is stopped and about to be wiped. - get.k3s.io installs latest, two minor versions ahead of the fleet, so a reinstalled node would rejoin on a newer k3s than its peers and roll an unplanned two-version control-plane upgrade. K3sConfig.k3sVersion now pins it. - `labctl app k3s install --role x` mapped anything that was not "worker" to "infra", so the runbook's own `--role k8s-combo` would have installed a different role with different disk expectations and no warning. Also: /var/lib/longhorn/replicas must never be deleted, only its contents, or every replica placed on the node dies on a non-recursive mkdir -- for ever, while the node reports Ready, Schedulable and healthy. etcd is now in the kickstart's package set for k3s server roles, because `k3s etcdctl` is not a command and a member that needs to recover its own membership has no internet. 542 unit tests / 39 files. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-10-09 22:33:59 +01:00
Michal	df2dfc5d71	fix(bastion): pin the VyOS boot NIC by MAC, and detect pre-installer stalls ... Both Protectli VP2440s failed to install on real hardware: they fetched kernel+initrd and then went silent. The console showed why — Looking for a connected Ethernet interface ... e2 ? e3 ? e4 ? e5 ? Connected e4 found Connected e5 found [4.595647] igc 0000:02:00.0 e2: NIC Link is Up IP-Config: e4 ... no response after 15 secs - giving up Unable to find a live file system on the network live-boot picks the first *connected* interface. The i40e SFP+ pair links before the igc copper port (up at 4.6s), so it chose the fiber ports, which have no DHCP, and never tried the NIC that actually PXE booted. Fix: pass BOOTIF=01-<mac> on the kernel cmdline. live-boot's Device_from_bootif() (verified present in this image) matches it against /sys/class/net and sets DEVICE directly. The MAC comes from the dispatch key — i.e. exactly the NIC that PXE booted — which is more reliable than iPXE's ${net0} on a box where the booting NIC may not be net0. Why the integration test missed it: the VM had ONE NIC, so "first connected interface" was trivially correct, and virtio links instantly so there was no negotiation race. createPxeVm now takes decoyNics, attaching extra NICs ahead of the PXE NIC on a network with no route to the bastion; the VyOS test uses 2. Without BOOTIF that reproduces the hardware failure. getVmMac is network-aware so it still returns the booting NIC. Also: the bastion had every clue and said nothing — it logged INSTALL STARTED, served kernel+initrd, then nothing for 7 minutes. dispatch now stamps dispatched_at, and /api/logs/:mac returns stalled_for_s / stalled (8 min threshold, sized for the ~600MB squashfs fetch), so a machine wedged before the installer environment comes up is diagnosable without a console. Verified on hardware: both firewalls installed, bond0 802.3ad + VLANs 2/3/9/10/200 + VRRP (priority 200/100, VIP .254 per VLAN) applied, and /config/lab-provisioned written. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-12 12:35:09 +01:00
Michal	a664074fa3	wip: save current ks debugging state before bisect revert ... All accumulated changes to kickstart template, test infrastructure, and dnsmasq config. None of these produce a clean boot yet — saving state before reverting to baseline for bisection. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-03-28 20:24:14 +00:00
Michal	cc289c0f94	feat: serial console on test VMs for debugging without SSH ... - VMs get serial console on TCP (PXE: port 4555, ISO: port 4556) - serialExec() helper: runs commands via telnet when SSH/network is down - PXE test: on SSH failure, dumps hostname, IP, NetworkManager, sshd, failed units, and fstab via serial console before failing - Kickstart enables serial-getty@ttyS0 for auto-login on serial Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-03-27 15:22:43 +00:00
Michal	7446d669c1	feat: ARM ISO boot integration test, OVMF boot fixes ... ARM integration test: - arm-iso-provision.test.ts: aarch64 VM boots from bastion-generated ISO - Uses QEMU aarch64 emulation (slow but validates the R1 scenario) - Generous timeouts for emulated boot (15min discovery, 60min install) - test-provision.sh updated: `sudo ./scripts/test-provision.sh arm` VM boot fixes: - setBootDisk() preserves UEFI loader/nvram when switching to disk boot - /boot/efi mount gets nofail in fstab (prevents emergency mode in VMs) - chronyd enable uses || true (fails in kickstart chroot) - createIsoVm supports arch parameter for ARM VMs Note: SSH-after-reboot in OVMF VMs still fails — OVMF doesn't respect efibootmgr changes and loops PXE/HTTP Boot. Real hardware works fine. The install flow itself (discovery → kickstart → complete) is validated. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-03-27 00:26:12 +00:00
Michal	46b017d77e	feat: install logging, error trapping, PXE/ISO integration tests ... Kickstart installs on real hardware failed silently — no error reporting, only 3 progress callbacks, zero log streaming. This overhaul makes every install fully observable. Kickstart improvements: - Error trapping in %pre and %post (trap ERR sends failure details to bastion) - 12+ granular progress stages (was 3): SSH, hostname, k3s prep, EFI boot, metadata - Background log streamer: tails %post output and batch-sends to /api/log - bastion_log() function for explicit log lines from kickstart scripts Bastion API: - POST /api/log — receives raw log lines from kickstart (single or batch) - InstallLogBuffer — per-MAC ring buffer (2000 lines) + file persistence - GET /api/logs/:mac — now returns log_lines + log_total alongside stages - SSE /api/logs/:mac/follow — uses named events (event: stage vs event: log) - Progress events forwarded to labd via bastion-progress WebSocket message - Post-provision k3s logs routed through progressBus (was console-only) dnsmasq fixes found during VM testing: - HTTP Boot filename: ipxe-real.efi → ipxe.efi (leftover from old 2-stage approach) - pxe-service directives: only in proxy mode (breaks OVMF PXE in full mode) - PXEClient vendor class echo for UEFI firmware compatibility Integration tests: - PXE boot test: blank UEFI VM → dnsmasq → HTTP Boot → iPXE → bastion → install - ISO boot test: blank VM boots from bastion-generated ISO → same flow - Shared helpers: pxe-network (no DHCP, nftables fix), pxe-vm (UEFI + ISO boot) - test-provision.sh: runs both PXE + ISO tests with prerequisite checks - 250GB sparse QCOW2 disk (LVM layout needs ~204GB) 201 unit tests passing (11 new). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-03-26 22:26:33 +00:00