v2 partition schema: capability model, fixed sizing, and a proven rolling migration

The fleet's disk layout was decided by literal role-name comparisons at 17
sites across 8 files. That is why `labcontroller` got no rancher LV (it
matched neither "infra" nor "worker"), why labmaster carried an empty 728 GiB
Longhorn LV, and why a 62 GiB container image tipped that node into
DiskPressure and evicted bastion, labd, cockroach and openbao.

Three generators each hardcoded their own sizes and disagreed: the kickstart
said rancher=120G, ubuntu-autoinstall said 20G, asahi-firstboot said 20G under
a comment asserting the sizes matched install.ks.ts. They did not.

ROLE_REGISTRY already carried `k3s` and `parent` and nothing read them. This
makes capabilities authoritative, moves every size into one shared module, and
adds schema v2 (fixed 250G rancher / 300G longhorn, never --grow). Existing
machines stay v1 forever: schema is detected from the on-disk VG tag, so a v1
box stays v1 even when v2 is requested.

Proven on real VMs, not just unit tests: all four topologies installed and had
their layouts read back off the machines; markers in /home, /srv,
/var/lib/longhorn and /var/lib/rancher survived reinstall; and a 4-node cluster
formed, took real data, and had all three servers rolled one at a time through
evacuate / wipe / reinstall / rejoin with checksums verified before and after
each roll. Drill: 10/10, evidence in migration/drill-evidence/.

Production defects the drill found, none of which a unit test could reach:

- sshOpts() capped every operation at 30s while drainNode asked kubectl for
  --timeout=15m, so the transport killed the drain after evicting most pods and
  returned a transcript that read like success. `labctl app k3s evacuate` could
  never have drained a real node.
- generateAgentConfig put node-role.kubernetes.io/worker in the kubelet's
  node-label list, which NodeRestriction rejects, so no agent installed through
  K3sModule could start. systemd reported only "did not take the steps required
  by its unit configuration".
- toOpContext built K3sConfig by enumerating three keys and silently dropped
  the rest, so the k3s version pin did nothing and --allow-single-replica was a
  no-op that claimed to override a data-loss guard.
- The Longhorn disk annotation was mangled by shell quoting; with
  createDefaultDiskLabeledNodes the node then gets no disk at all, joins, goes
  Ready, and silently hosts no replica.
- evacuate drained before evicting, which blocks on Longhorn's PDBs and would
  destroy a single-replica volume's only source.
- etcd member ids were parsed from JSON, losing uint64 precision and emitting
  decimal where `member remove` needs hex -- at the moment a node is stopped
  and about to be wiped.
- get.k3s.io installs latest, two minor versions ahead of the fleet, so a
  reinstalled node would rejoin on a newer k3s than its peers and roll an
  unplanned two-version control-plane upgrade. K3sConfig.k3sVersion now pins it.
- `labctl app k3s install --role x` mapped anything that was not "worker" to
  "infra", so the runbook's own `--role k8s-combo` would have installed a
  different role with different disk expectations and no warning.

Also: /var/lib/longhorn/replicas must never be deleted, only its contents, or
every replica placed on the node dies on a non-recursive mkdir -- for ever,
while the node reports Ready, Schedulable and healthy.

etcd is now in the kickstart's package set for k3s server roles, because
`k3s etcdctl` is not a command and a member that needs to recover its own
membership has no internet.

542 unit tests / 39 files.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

.gitignore

@@ -31,3 +31,9 @@ node_modules/
 # Asahi build artifacts (large)
 bastion/.asahi-cache/
 bastion/asahi-repo/*.zip
+
+# Regenerated by labsim/dualstack-lab.sh; derived state, not source.
+labsim/dualstack-evidence/
+
+# Runtime snapshots from labsim/cilium-ipam-switch.sh
+labsim/.ipam-switch-state/

bastion/completions/labctl.bash

@@ -59,10 +59,10 @@ _labctl() {
       COMPREPLY=($(compgen -W "-h --help" -- "$cur"))
       return ;;
     "provision install")
-      COMPREPLY=($(compgen -W "--role --os --disk -h --help" -- "$cur"))
+      COMPREPLY=($(compgen -W "--role --os --disk --vyos-mgmt --vyos-mgmt-address --vyos-bond --vyos-bond-address --vyos-bond-vrrp --vlan-vip --vyos-vrrp-priority --vyos-mgmt-vlan --vlan --vyos-password --vyos-hwid --vyos-fresh-config -h --help" -- "$cur"))
       return ;;
     "provision reprovision")
-      COMPREPLY=($(compgen -W "--role --os --disk -h --help" -- "$cur"))
+      COMPREPLY=($(compgen -W "--role --os --disk --user -h --help" -- "$cur"))
       return ;;
     "provision debug")
       COMPREPLY=($(compgen -W "--pxe-boot -h --help" -- "$cur"))

bastion/completions/labctl.fish

@@ -132,13 +132,26 @@ complete -c labctl -n "__labctl_using_cmd provision" -a recheck -d 'Refresh hard
 
 # provision install options
 complete -c labctl -n "__labctl_in_cmd provision install" -l role -d 'Machine role (see below)' -xa 'vanilla worker infra labcontroller'
-complete -c labctl -n "__labctl_in_cmd provision install" -l os -d 'Operating system' -xa 'fedora-43 ubuntu-26.04'
+complete -c labctl -n "__labctl_in_cmd provision install" -l os -d 'Operating system' -xa 'fedora-43 ubuntu-26.04 vyos-rolling'
 complete -c labctl -n "__labctl_in_cmd provision install" -l disk -d 'Target disk device (auto-detect if omitted)' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-mgmt -d 'VyOS: untagged interface the machine PXE boots from (default eth0)' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-mgmt-address -d 'VyOS: CIDR for the management interface, or \'dhcp\' (default dhcp)' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-bond -d 'VyOS: comma-separated LACP bond members (must exclude the PXE NIC)' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-bond-address -d 'VyOS: address on the untagged bond (trunk native VLAN)' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-bond-vrrp -d 'VyOS: VRRP VIP floated on the untagged bond' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vlan-vip -d 'VyOS: VRRP VIP for a --vlan entry (repeatable)' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-vrrp-priority -d 'VyOS: VRRP priority for all groups on this box (higher = master)' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-mgmt-vlan -d 'VyOS: tagged management VLAN on the PXE port' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vlan -d 'VyOS: tagged VLAN sub-interface on the bond (repeatable)' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-password -d 'VyOS: password for the \'vyos\' user' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-hwid -d 'VyOS: pin an interface name to a MAC via hw-id (repeatable)' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-fresh-config -d 'VyOS: on reinstall, overwrite the preserved config with the generated one'
 
 # provision reprovision options
 complete -c labctl -n "__labctl_in_cmd provision reprovision" -l role -d 'Machine role (see below)' -xa 'vanilla worker infra labcontroller'
-complete -c labctl -n "__labctl_in_cmd provision reprovision" -l os -d 'Operating system' -xa 'fedora-43 ubuntu-26.04'
+complete -c labctl -n "__labctl_in_cmd provision reprovision" -l os -d 'Operating system' -xa 'fedora-43 ubuntu-26.04 vyos-rolling'
 complete -c labctl -n "__labctl_in_cmd provision reprovision" -l disk -d 'Target disk device (auto-detect if omitted)' -x
+complete -c labctl -n "__labctl_in_cmd provision reprovision" -l user -d 'SSH user for the reboot (default: vyos for VyOS machines, else current user)' -x
 
 # provision debug options
 complete -c labctl -n "__labctl_in_cmd provision debug" -l pxe-boot -d 'Boot installed system via PXE (kernel+initrd from network, root from NVMe)'

bastion/package.json

@@ -21,6 +21,8 @@
     "test:integration:pxe:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'PXE boot'",
     "test:integration:iso": "vitest run -c tests/integration/vitest.config.ts -t 'ISO boot'",
     "test:integration:iso:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'ISO boot'",
+    "test:integration:vyos": "vitest run -c tests/integration/vitest.config.ts -t 'VyOS provisioning'",
+    "test:integration:vyos:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'VyOS provisioning'",
     "test:integration:arm-iso": "vitest run -c tests/integration/vitest.config.ts -t 'ARM ISO'",
     "test:integration:arm-iso:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'ARM ISO'",
     "test:integration:asahi": "vitest run -c tests/integration/vitest.config.ts -t 'asahi firstboot'",

bastion/src/bastion/src/config.ts

@@ -20,7 +20,22 @@ export function loadConfig(overrides: Partial<BastionConfig> = {}): BastionConfi
   const ubuntuMirror = overrides.ubuntuMirror ?? process.env["UBUNTU_MIRROR"]
     ?? `https://releases.ubuntu.com/${ubuntuVersion}`;
 
-  const fedoraMirror = `https://download.fedoraproject.org/pub/fedora/linux/releases/${fedoraVersion}/Everything/${arch}/os`;
+  // "latest" resolves the newest nightly ISO from the vyos-nightly-build GitHub
+  // releases at startup. downloads.vyos.io no longer serves direct rolling ISOs
+  // (it returns the vyos.io site, and nightly builds sit behind a signup form);
+  // GitHub releases are the remaining free, unauthenticated direct source.
+  // LTS ISOs are subscription-only. Set VYOS_ISO_URL to pin a specific build.
+  const vyosIsoUrl = overrides.vyosIsoUrl ?? process.env["VYOS_ISO_URL"] ?? "latest";
+  const vyosDefaultPassword = overrides.vyosDefaultPassword
+    ?? process.env["VYOS_DEFAULT_PASSWORD"] ?? "vyos";
+
+  // Overridable like ubuntuMirror and vyosIsoUrl above -- it was the only one
+  // of the three hardcoded. Without this, every integration-test install pulls
+  // the full package set from download.fedoraproject.org, which makes a
+  // multi-VM provisioning matrix impractical (~25 min per install, serialised).
+  // Point FEDORA_MIRROR at a local mirror or caching proxy for test runs.
+  const fedoraMirror = overrides.fedoraMirror ?? process.env["FEDORA_MIRROR"]
+    ?? `https://download.fedoraproject.org/pub/fedora/linux/releases/${fedoraVersion}/Everything/${arch}/os`;
   const tftpDir = `${bastionDir}/tftp`;
   const httpDir = `${bastionDir}/http`;
   const stateFile = `${bastionDir}/state.json`;
@@ -38,6 +53,8 @@ export function loadConfig(overrides: Partial<BastionConfig> = {}): BastionConfi
     dhcpRangeEnd,
     ubuntuVersion,
     ubuntuMirror,
+    vyosIsoUrl,
+    vyosDefaultPassword,
     // These are populated at runtime by the network service
     iface: overrides.iface ?? "",
     serverIp: overrides.serverIp ?? "",
@@ -48,6 +65,7 @@ export function loadConfig(overrides: Partial<BastionConfig> = {}): BastionConfi
     syslogPort,
     skipDnsmasq: overrides.skipDnsmasq,
     skipArtifacts: overrides.skipArtifacts,
+    autoK3s: overrides.autoK3s ?? process.env["AUTO_K3S"] !== "false",
     labdUrl: overrides.labdUrl ?? process.env["LABD_URL"],
     bastionJoinToken: overrides.bastionJoinToken ?? process.env["BASTION_JOIN_TOKEN"],
     fedoraMirror,

bastion/src/bastion/src/main.ts

@@ -40,6 +40,125 @@ function download(url: string, dest: string, label: string): void {
   }
 }
 
+/**
+ * Pick the largest regular-file initrd from an `xorriso -lsl` listing.
+ *
+ * /live carries decoys: a 0-byte initrd.img placeholder on some images, or an
+ * initrd.img SYMLINK to the real version-suffixed file on others. Parsing is
+ * field-based (ls -l layout: perms links uid gid size month day time 'name')
+ * and considers only lines whose mode string marks a regular file — symlinks
+ * report their link size, not the target's, and must not win.
+ */
+export function pickLargestInitrd(
+  listing: string,
+): { name: string; size: number } | undefined {
+  let best: { name: string; size: number } | undefined;
+  for (const line of listing.split("\n")) {
+    if (!line.startsWith("-")) continue;          // regular files only
+    const quoted = /'([^']+)'/.exec(line);
+    const fields = line.trim().split(/\s+/);
+    const size = parseInt(fields[4] ?? "", 10);
+    const name = quoted?.[1] ?? "";
+    if (!name.startsWith("initrd")) continue;
+    if (!Number.isFinite(size) || size <= 0) continue;
+    if (best === undefined || size > best.size) {
+      best = { name, size };
+    }
+  }
+  return best;
+}
+
+const VYOS_NIGHTLY_RELEASES =
+  "https://api.github.com/repos/vyos/vyos-nightly-build/releases/latest";
+
+/**
+ * Resolve the configured VyOS ISO URL, expanding the "latest" sentinel.
+ *
+ * The nightly asset filename embeds a build date, so there is no stable
+ * "latest.iso" path to hardcode — the newest release has to be looked up.
+ * Any other value is used verbatim, which is how VYOS_ISO_URL pins a build
+ * or points at a locally mirrored copy.
+ */
+function resolveVyosIsoUrl(configured: string): string {
+  if (configured !== "latest") return configured;
+
+  const body = execSync(`curl -sSfL "${VYOS_NIGHTLY_RELEASES}"`, {
+    encoding: "utf-8",
+    stdio: ["pipe", "pipe", "pipe"],
+  });
+  const release = JSON.parse(body) as {
+    tag_name?: string;
+    assets?: Array<{ name: string; browser_download_url: string }>;
+  };
+
+  const asset = (release.assets ?? []).find((a) =>
+    /generic-amd64\.iso$/.test(a.name),
+  );
+  if (!asset) {
+    throw new Error(
+      `No generic-amd64 ISO asset in VyOS nightly release ${release.tag_name ?? "?"}`,
+    );
+  }
+
+  logger.info(`  VyOS ISO resolved to ${asset.name} (${release.tag_name ?? "?"})`);
+  return asset.browser_download_url;
+}
+
+/**
+ * Extract VyOS netboot artifacts from the release ISO.
+ *
+ * VyOS publishes no netboot bundle, so kernel/initrd/squashfs have to come out
+ * of the ISO. xorriso is already in the bastion image (used for boot.iso) and
+ * extracts without root or a loop mount.
+ *
+ * The initrd needs care: /live contains an empty initrd.img placeholder
+ * alongside the real one, which carries a version-suffixed name. Booting the
+ * 0-byte file fails with no useful diagnostic, so pick the largest initrd*.
+ */
+export function prepareVyosArtifacts(config: BastionConfig): void {
+  const kernel = `${config.httpDir}/vyos-vmlinuz`;
+  const initrd = `${config.httpDir}/vyos-initrd`;
+  const squashfs = `${config.httpDir}/vyos-filesystem.squashfs`;
+
+  if (existsSync(kernel) && existsSync(initrd) && existsSync(squashfs)) {
+    logger.info("  VyOS netboot artifacts -- cached");
+    return;
+  }
+
+  const iso = `${config.bastionDir}/vyos.iso`;
+  download(resolveVyosIsoUrl(config.vyosIsoUrl), iso, "VyOS ISO");
+
+  const extract = (isoPath: string, dest: string, label: string): void => {
+    execSync(
+      `xorriso -osirrox on -indev "${iso}" -extract "${isoPath}" "${dest}"`,
+      { stdio: "pipe" },
+    );
+    logger.info(`  ${label} -- extracted from ${isoPath}`);
+  };
+
+  extract("/live/vmlinuz", kernel, "VyOS kernel");
+  extract("/live/filesystem.squashfs", squashfs, "VyOS squashfs");
+
+  // Pick the real initrd by size from the ISO's own directory listing.
+  const listing = execSync(`xorriso -indev "${iso}" -lsl /live/ --`, {
+    encoding: "utf-8",
+    stdio: ["pipe", "pipe", "pipe"],
+  });
+
+  const best = pickLargestInitrd(listing);
+  if (best === undefined) {
+    throw new Error("No non-empty initrd found in /live on the VyOS ISO");
+  }
+  extract(`/live/${best.name}`, initrd, `VyOS initrd (${best.name}, ${best.size} bytes)`);
+
+  // The ISO is only needed to produce the three artifacts above.
+  try {
+    unlinkSync(iso);
+  } catch {
+    // Non-fatal: leaving it costs disk but nothing else.
+  }
+}
+
 function symlinkSafe(target: string, linkPath: string): void {
   try {
     symlinkSync(target, linkPath);
@@ -182,6 +301,17 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
       logger.warn(`Ubuntu ${config.ubuntuVersion} artifacts not available -- Ubuntu provisioning disabled`);
     }
 
+    // VyOS netboot artifacts (non-fatal — same policy as Ubuntu)
+    try {
+      logger.info("Preparing VyOS netboot artifacts...");
+      prepareVyosArtifacts(config);
+    } catch (err) {
+      logger.warn(
+        `VyOS artifacts not available -- VyOS provisioning disabled ` +
+        `(${err instanceof Error ? err.message : String(err)})`,
+      );
+    }
+
     // Symlink iPXE binaries into HTTP dir for UEFI HTTP Boot
     for (const name of ["ipxe.efi", "ipxe-arm64.efi"]) {
       const src = `${config.tftpDir}/${name}`;
@@ -261,6 +391,7 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
           role: msg.role as import("@lab/shared").Role,
           os: msg.os as import("@lab/shared").OsId,
           queued_at: new Date().toISOString(),
+          ...(msg.vyos ? { vyos: msg.vyos } : {}),
         };
       });
       return { status: "ok", data: { mac: msg.mac, hostname: msg.hostname } };

bastion/src/bastion/src/routes/api.ts

@@ -5,8 +5,8 @@
 // /api/discover  - receive hardware discovery reports from PXE-booted machines
 
 import type { FastifyInstance } from "fastify";
-import type { HardwareInfo, InstalledInfo, Role } from "@lab/shared";
-import { isValidOsId, SUPPORTED_ROLES } from "@lab/shared";
+import type { BastionConfig, HardwareInfo, InstalledInfo, Role, VyosInstallSpec } from "@lab/shared";
+import { isValidOsId, SUPPORTED_ROLES, SUPPORTED_OS } from "@lab/shared";
 import type { StateManager } from "../services/state.js";
 import { logger } from "../services/logger.js";
 import { triggerPostProvisionK3s } from "../services/post-provision.js";
@@ -14,9 +14,18 @@ import { progressBus } from "../services/progress-events.js";
 import type { ProgressEvent } from "../services/progress-events.js";
 import type { InstallLogBuffer } from "../services/install-log.js";
 import type { SyslogListener } from "../services/syslog-listener.js";
+import { isRole, requiresSchemaV2 } from "@lab/shared";
+
+/**
+ * Seconds after dispatch with zero progress before a machine is called stalled.
+ * Generous: the slowest legitimate gap is fetching a ~600MB VyOS squashfs over
+ * HTTP before the hook can report anything.
+ */
+const STALL_THRESHOLD_S = 8 * 60;
 
 export function registerApiRoutes(
   app: FastifyInstance,
+  config: BastionConfig,
   state: StateManager,
   installLog: InstallLogBuffer,
   syslog: SyslogListener,
@@ -34,9 +43,11 @@ export function registerApiRoutes(
       disk?: string;
       role?: string;
       os?: string;
+      vyos?: VyosInstallSpec;
+      schemaVersion?: number;
     };
   }>("/api/install", async (request, reply) => {
-    const { mac: rawMac, hostname, disk, role, os } = request.body ?? {};
+    const { mac: rawMac, hostname, disk, role, os, vyos, schemaVersion } = request.body ?? {};
     const mac = (rawMac ?? "").toLowerCase().replace(/-/g, ":");
 
     if (mac === "") {
@@ -48,9 +59,24 @@ export function registerApiRoutes(
       return reply.status(400).send({ error: `invalid role: '${validRole}'. Supported: ${SUPPORTED_ROLES.join(", ")}` });
     }
 
+    // Reject an unknown schema rather than silently installing v1: someone
+    // asking for v3 wants a layout we do not have, and quietly giving them the
+    // old one is how a fleet ends up with layouts nobody chose.
+    if (schemaVersion !== undefined && schemaVersion !== 1 && schemaVersion !== 2) {
+      return reply.status(400).send({ error: `invalid schemaVersion: ${schemaVersion}. Supported: 1, 2` });
+    }
+
+    // Only an EXPLICIT v1 is a contradiction. Omitting the schema is normal --
+    // a reinstall never passes one, because the layout already on the disk wins.
+    if (schemaVersion === 1 && isRole(validRole) && requiresSchemaV2(validRole)) {
+      return reply.status(400).send({
+        error: `role '${validRole}' requires schemaVersion 2 (it has no v1 disk layout)`,
+      });
+    }
+
     const osId = os ?? "fedora-43";
     if (!isValidOsId(osId)) {
-      return reply.status(400).send({ error: `invalid os: '${osId}'. Supported: fedora-43, ubuntu-26.04` });
+      return reply.status(400).send({ error: `invalid os: '${osId}'. Supported: ${SUPPORTED_OS.join(", ")}` });
     }
 
     state.update((s) => {
@@ -60,6 +86,8 @@ export function registerApiRoutes(
         role: validRole as Role,
         os: osId,
         queued_at: new Date().toISOString(),
+        ...(schemaVersion ? { schemaVersion: schemaVersion as 1 | 2 } : {}),
+        ...(vyos ? { vyos } : {}),
       };
     });
 
@@ -158,12 +186,31 @@ export function registerApiRoutes(
           };
           s.installed[mac] = installedInfo;
 
-          const admin = installedInfo.role !== "vanilla" && installedInfo.role !== "" ? "lab" : "root";
+          // VyOS: the only login user is "vyos", and a router never runs k3s —
+          // without this guard a non-vanilla role + recorded IP would trigger
+          // the k3s post-provision against a VyOS box.
+          const isVyos = (installedInfo.os ?? "").startsWith("vyos");
+          const admin = isVyos
+            ? "vyos"
+            : installedInfo.role !== "vanilla" && installedInfo.role !== "" ? "lab" : "root";
           console.log(`\n  \x1b[0;32m\x1b[1m  ssh ${admin}@${ip}\x1b[0m\n`);  // eslint-disable-line no-console
 
-          // Auto-install k3s for non-vanilla roles
-          if (installedInfo.role !== "vanilla" && ip !== "") {
-            void triggerPostProvisionK3s(installedInfo.hostname, ip, installedInfo.role, admin, mac);
+          // Auto-install k3s for non-vanilla roles.
+          // autoK3s=false suppresses this entirely. Multi-node tests need that:
+          // post-provision installs each node into its OWN cluster, so leaving it
+          // on turns three combo nodes into three single-node clusters, and it
+          // also stops k3s and deletes CNI state minutes after install lands --
+          // right on top of a test that is forming a cluster itself.
+          if (config.autoK3s === false) {
+            logger.info(`[api] autoK3s disabled; skipping k3s post-provision for ${mac}`);
+          } else if (!isVyos && installedInfo.role !== "vanilla" && ip !== "") {
+            // State is JSON on disk; a role written by an older build may no
+            // longer exist. Skip loudly rather than crash the install callback.
+            if (isRole(installedInfo.role)) {
+              void triggerPostProvisionK3s(installedInfo.hostname, ip, installedInfo.role, admin, mac);
+            } else {
+              console.warn(`[api] unknown role ${installedInfo.role} for ${mac}; skipping k3s post-provision`);  // eslint-disable-line no-console
+            }
           }
         }
       }
@@ -434,6 +481,15 @@ export function registerApiRoutes(
     const installedEntry = currentState.installed[mac];
 
     if (queueEntry) {
+      // A machine that was handed an install script but has reported nothing
+      // since is wedged BEFORE the installer environment came up — a bad
+      // kernel/initrd, no network in the initramfs, or the wrong NIC picked.
+      // Surfacing it here is what makes that diagnosable without a console.
+      const since = queueEntry.progress_at ?? queueEntry.dispatched_at;
+      const stalledForS = since !== undefined && queueEntry.progress === undefined
+        ? Math.floor((Date.now() - new Date(since).getTime()) / 1000)
+        : 0;
+
       return reply.send({
         mac,
         hostname: queueEntry.hostname,
@@ -441,6 +497,9 @@ export function registerApiRoutes(
         progress: queueEntry.progress ?? "queued",
         progress_detail: queueEntry.progress_detail ?? "",
         progress_at: queueEntry.progress_at ?? queueEntry.queued_at,
+        dispatched_at: queueEntry.dispatched_at,
+        stalled_for_s: stalledForS,
+        stalled: stalledForS > STALL_THRESHOLD_S,
         role: queueEntry.role,
         os: queueEntry.os,
         stages: queueEntry.log ?? [],

bastion/src/bastion/src/routes/dispatch.ts

@@ -15,6 +15,7 @@ import {
   renderLocalBootIpxe,
 } from "../templates/boot.ipxe.js";
 import { renderUbuntuInstallIpxe } from "../templates/ubuntu-boot.ipxe.js";
+import { renderVyosInstallIpxe } from "../templates/vyos-boot.ipxe.js";
 import { renderDebugKickstart } from "../templates/debug.ks.js";
 import { logger } from "../services/logger.js";
 
@@ -99,8 +100,22 @@ echo "==============================="
       const os = queueEntry.os ?? "fedora-43";
       logger.info(`INSTALL STARTED: ${mac} -> ${hostname} (${os})`);
 
+      // Stamp the handoff so a machine that boots the installer but never
+      // reports can be spotted without a console.
+      state.update((s) => {
+        const entry = s.install_queue[mac];
+        if (entry) entry.dispatched_at = new Date().toISOString();
+      });
+
       let script: string;
-      if (os.startsWith("ubuntu")) {
+      if (os.startsWith("vyos")) {
+        script = renderVyosInstallIpxe({
+          mac,
+          hostname,
+          serverIp: config.serverIp,
+          httpPort: config.httpPort,
+        });
+      } else if (os.startsWith("ubuntu")) {
         script = renderUbuntuInstallIpxe({
           mac,
           hostname,

bastion/src/bastion/src/routes/kickstart.ts

@@ -30,6 +30,7 @@ export function registerKickstartRoutes(
       hostname: queueEntry?.hostname ?? "lab-node",
       disk: queueEntry?.disk ?? "",
       role: queueEntry?.role ?? "worker",
+      ...(queueEntry?.schemaVersion ? { schemaVersion: queueEntry.schemaVersion } : {}),
     });
 
     return reply.type("text/plain").send(ks);

bastion/src/bastion/src/routes/vyos.ts

@@ -0,0 +1,71 @@
+// VyOS network install routes.
+//
+// VyOS has no unattended installer, so the automation is injected via
+// live-config's `hooks` component: the iPXE script passes
+// live-config.hooks=<.../vyos/autoinstall.sh>, live-config wgets it and runs it
+// as root, and that script fetches and executes the generated install driver.
+
+import type { FastifyInstance } from "fastify";
+import type { BastionConfig } from "@lab/shared";
+import type { StateManager } from "../services/state.js";
+import { buildVyosConfigSpec } from "../templates/vyos-config-spec.js";
+import { renderVyosInstallPy } from "../templates/vyos-install.py.js";
+import { logger } from "../services/logger.js";
+
+function normalizeMac(value: string | undefined): string {
+  return (value ?? "").toLowerCase().replace(/-/g, ":");
+}
+
+export function registerVyosRoutes(
+  app: FastifyInstance,
+  config: BastionConfig,
+  state: StateManager,
+): void {
+  // live-config hook. Kept minimal: everything version-specific lives in the
+  // generated Python. wget is guaranteed present -- live-config used it to
+  // fetch this very script.
+  app.get<{ Querystring: { mac?: string } }>("/vyos/autoinstall.sh", async (request, reply) => {
+    const mac = normalizeMac(request.query.mac);
+    const base = `http://${config.serverIp}:${config.httpPort}`;
+
+    logger.info(`VYOS AUTOINSTALL HOOK served to ${mac || "unknown MAC"}`);
+
+    const script = `#!/bin/sh
+# Lab PXE Bastion -- VyOS unattended install hook (run by live-config as root)
+set -eu
+
+wget -q "${base}/vyos/install.py?mac=${mac}" -O /tmp/vyos-install.py
+exec python3 /tmp/vyos-install.py
+`;
+    return reply.type("text/plain").send(script);
+  });
+
+  // Per-MAC install driver, with the machine's config spec baked in.
+  app.get<{ Querystring: { mac?: string } }>("/vyos/install.py", async (request, reply) => {
+    const mac = normalizeMac(request.query.mac);
+    const queueEntry = state.load().install_queue[mac];
+
+    const spec = buildVyosConfigSpec({
+      hostname: queueEntry?.hostname ?? "vyos",
+      spec: queueEntry?.vyos,
+      defaultPassword: config.vyosDefaultPassword,
+      sshKeys: config.sshKeys,
+      disk: queueEntry?.disk,
+    });
+
+    logger.info(
+      `VYOS INSTALL DRIVER served to ${mac} (${spec.hostname}, ` +
+      `${spec.sets.length} config ops, disk="${spec.disk || "auto"}")`,
+    );
+
+    const script = renderVyosInstallPy({
+      spec,
+      mac,
+      serverIp: config.serverIp,
+      httpPort: config.httpPort,
+      role: queueEntry?.role ?? "vanilla",
+    });
+
+    return reply.type("text/plain").send(script);
+  });
+}

bastion/src/bastion/src/server.ts

@@ -12,6 +12,7 @@ import { registerDispatchRoutes } from "./routes/dispatch.js";
 import { registerKickstartRoutes } from "./routes/kickstart.js";
 import { registerApiRoutes } from "./routes/api.js";
 import { registerAsahiRoutes } from "./routes/asahi.js";
+import { registerVyosRoutes } from "./routes/vyos.js";
 
 
 export function createApp(config: BastionConfig): { app: ReturnType<typeof Fastify>; state: StateManager; installLog: InstallLogBuffer; syslog: SyslogListener } {
@@ -45,8 +46,9 @@ export function createApp(config: BastionConfig): { app: ReturnType<typeof Fasti
   // Register route handlers
   registerDispatchRoutes(app, config, state);
   registerKickstartRoutes(app, config, state, syslog);
-  registerApiRoutes(app, state, installLog, syslog);
+  registerApiRoutes(app, config, state, installLog, syslog);
   registerAsahiRoutes(app, config);
+  registerVyosRoutes(app, config, state);
   // boot.iso is generated at startup and served as a static file from httpDir
   // (static serving supports HTTP Range requests, required by JetKVM streaming)
 

bastion/src/bastion/src/services/kickstart-generator.ts

@@ -24,12 +24,14 @@ export function generateInstallKickstart(
     hostname: string;
     disk: string;
     role: Role;
+    schemaVersion?: 1 | 2;
   },
 ): string {
   const ksParams: InstallKickstartParams = {
     hostname: params.hostname,
     disk: params.disk,
     role: params.role,
+    ...(params.schemaVersion ? { schemaVersion: params.schemaVersion } : {}),
     domain: config.domain,
     fedoraVersion: config.fedoraVersion,
     timezone: config.timezone,

bastion/src/bastion/src/services/post-provision.ts

@@ -7,6 +7,7 @@ import { homedir } from "node:os";
 import { join } from "node:path";
 import { logger } from "./logger.js";
 import { progressBus } from "./progress-events.js";
+import { isK3sServer, type Role } from "@lab/shared";
 
 function findSshKey(): string | undefined {
   const sudoUser = process.env["SUDO_USER"];
@@ -84,6 +85,36 @@ function sshRunStreaming(ip: string, user: string, command: string, keyPath: str
   });
 }
 
+/**
+ * Roles this bare post-provision install can legitimately handle.
+ *
+ * The install below is a plain `INSTALL_K3S_EXEC=server|agent` with no K3S_URL
+ * and no K3S_TOKEN, and it knows about exactly one machine. That is fine for a
+ * node that is its own cluster, and wrong for anything that has to JOIN one:
+ *
+ *   - a cluster member installed this way gets `cluster-init` semantics and
+ *     becomes its own separate cluster, so provisioning three of them yields
+ *     three clusters instead of one (split brain);
+ *   - an AGENT cannot even start — `INSTALL_K3S_EXEC=agent` with nothing to
+ *     join fails with "Job for k3s-agent.service failed because the control
+ *     process exited with error code", which is exactly what a k8s-worker
+ *     install does today.
+ *
+ * So cluster members are not auto-installed. They are handed the command that
+ * supplies the missing server URL and token.
+ */
+export function shouldAutoInstallK3s(role: Role): boolean {
+  switch (role) {
+    case "k8s-combo":
+    case "k8s-controller":
+    case "k8s-worker":
+    case "worker":
+      return false;
+    default:
+      return true;
+  }
+}
+
 /**
  * Trigger k3s installation on a freshly provisioned machine.
  * Runs in the background — logs progress to bastion console and progressBus.
@@ -91,7 +122,7 @@ function sshRunStreaming(ip: string, user: string, command: string, keyPath: str
 export async function triggerPostProvisionK3s(
   hostname: string,
   ip: string,
-  role: string,
+  role: Role,
   sshUser: string,
   mac?: string,
 ): Promise<void> {
@@ -104,6 +135,21 @@ export async function triggerPostProvisionK3s(
     }
   };
 
+  // Gate before ANY work: the prerequisite steps below stop k3s and delete CNI
+  // state, so they must not run for a role we are declining to install either.
+  if (!shouldAutoInstallK3s(role)) {
+    emitStage(
+      "post-provision",
+      `${role} is a cluster member; not auto-installing k3s (it would form its own cluster)`,
+    );
+    const verb = isK3sServer(role) ? "--k3s-server https://<existing-server>:6443" : "--k3s-server https://<server>:6443";
+    logger.info(
+      `[k3s] Join it to the cluster with:\n` +
+      `        labctl app k3s install ${hostname} --role ${role} ${verb} --k3s-token <token>`,
+    );
+    return;
+  }
+
   emitStage("post-provision", `auto-installing k3s on ${hostname} (${ip}) role=${role}`);
   emitStage("post-provision", "waiting for SSH (machine may still be rebooting)");
 
@@ -150,7 +196,7 @@ sysctl --system > /dev/null'`, keyPath, hostname, mac);
 
   // Step 4: Install k3s
   // labcontroller extends infra — both are k3s servers
-  const k3sRole = (role === "infra" || role === "labcontroller") ? "server" : "agent";
+  const k3sRole = isK3sServer(role) ? "server" : "agent";
   emitStage("post-provision", `installing k3s ${k3sRole}`);
   const code = await sshRunStreaming(ip, sshUser,
     `curl -sfL https://get.k3s.io | sudo INSTALL_K3S_EXEC="${k3sRole}" INSTALL_K3S_SKIP_SELINUX_RPM=true sh -`,

bastion/src/bastion/src/templates/asahi-firstboot.sh.ts

@@ -3,6 +3,7 @@
 // Creates the standard lab LVM layout on the data partition, matching install.ks.ts.
 
 import type { Role } from "@lab/shared";
+import { isK3sServer, v2Layout, type LvSpec } from "@lab/shared";
 
 export interface AsahiFirstbootParams {
   hostname: string;
@@ -12,13 +13,42 @@ export interface AsahiFirstbootParams {
   sshKeys: string[];
   adminUser: string;
   mac: string;
+  /** Fresh-setup layout. Defaults to 1 so existing callers are unchanged. */
+  schemaVersion?: 1 | 2;
 }
 
 export function renderFirstbootScript(params: AsahiFirstbootParams): string {
   const { hostname, role, serverIp, httpPort, sshKeys, adminUser, mac } = params;
+  const schemaVersion = params.schemaVersion ?? 1;
+
+  // ---------------------------------------------------------------------------
+  // v2 LAYOUT, from the SAME definition Fedora and Ubuntu render
+  // (shared/src/types/layout.ts).
+  //
+  // omitRoot: on Apple Silicon the Asahi installer has already created a btrfs
+  // root partition OUTSIDE our volume group -- worker1 runs a 4.6 GB one. We
+  // must not create a second root LV.
+  //
+  // The v1 path below hardcodes rancher at 20480M while claiming "sizes match
+  // install.ks.ts", which uses 122880M. That 100 GB gap is what left worker1
+  // with an image store too small to hold a single large container image.
+  // ---------------------------------------------------------------------------
+  const v2 = v2Layout(role, { omitRoot: true });
+  const v2LvLines = v2.map((lv: LvSpec) => `lvcreate -L ${lv.sizeMib}M -n ${lv.name} labvg -y`).join("\n");
+  const v2FormatLines = v2
+    .map((lv: LvSpec) => (lv.fs === "swap" ? `mkswap /dev/labvg/${lv.name}` : `mkfs.xfs /dev/labvg/${lv.name}`))
+    .join("\n");
+  const v2MountLines = v2
+    .filter((lv: LvSpec) => lv.fs !== "swap")
+    .map((lv: LvSpec) => `mount_lv ${lv.name} ${lv.mount}`)
+    .join("\n");
+  const v2FstabLines = v2
+    .filter((lv: LvSpec) => lv.fs !== "swap")
+    .map((lv: LvSpec) => `echo "/dev/labvg/${lv.name} ${lv.mount} xfs defaults 0 0" >> /etc/fstab`)
+    .join("\n");
 
   const isWorker = role === "worker";
-  const isInfra = role === "infra" || role === "labcontroller";
+  const isInfra = isK3sServer(role);
 
   // Role-specific LV creation commands
   const roleLvLines: string[] = [];
@@ -127,24 +157,27 @@ write_lab_fstab() {
     # Add fresh LVM entries
     echo "# Lab LVM volumes" >> /etc/fstab
     echo "/dev/labvg/swap    none         swap defaults 0 0" >> /etc/fstab
-    echo "/dev/labvg/var     /var         xfs  defaults 0 0" >> /etc/fstab
+${schemaVersion === 2 ? v2FstabLines.split("\n").map((l: string) => "    " + l).join("\n") : `    echo "/dev/labvg/var     /var         xfs  defaults 0 0" >> /etc/fstab
     echo "/dev/labvg/varlog  /var/log     xfs  defaults 0 0" >> /etc/fstab
     echo "/dev/labvg/home    /home        xfs  defaults 0 0" >> /etc/fstab
     echo "/dev/labvg/srv     /srv         xfs  defaults 0 0" >> /etc/fstab
-    ${roleFstabLines.join('\n    ')}
+    ${roleFstabLines.join('\n    ')}`}
 }
 
 # ── Check for existing VG ────────────────────────────────────────
 if vgs labvg &>/dev/null; then
     echo "Volume group 'labvg' already exists — reprovision detected."
+    DISK_SCHEMA=v1
+    vgs --noheadings -o vg_tags labvg 2>/dev/null | grep -q lab-schema-v2 && DISK_SCHEMA=v2
+    echo "On-disk schema: $DISK_SCHEMA"
     echo "Activating existing volumes..."
     vgchange -ay labvg
 
-    mount_lv var /var
+${schemaVersion === 2 ? v2MountLines.split("\n").map((l: string) => "    " + l).join("\n") : `    mount_lv var /var
     mount_lv varlog /var/log
     mount_lv home /home
     mount_lv srv /srv
-${roleMountLines.map(l => `    ${l}`).join('\n')}
+${roleMountLines.map(l => `    ${l}`).join('\n')}`}
 
     # Enable swap
     if lvs labvg/swap &>/dev/null; then
@@ -162,24 +195,28 @@ echo "Creating LVM on $DATA_PART..."
 
 pvcreate "$DATA_PART"
 vgcreate labvg "$DATA_PART"
+${schemaVersion === 2 ? `
+# Tag so a later run can tell which layout is on this disk without trusting a
+# caller-supplied flag -- same mechanism install.ks.ts uses.
+vgchange --addtag lab-schema-v2 labvg >/dev/null 2>&1 || true` : ""}
 
 # Create LVs — sizes match install.ks.ts (in MiB)
 echo "Creating logical volumes..."
-lvcreate -L 27648M  -n swap   labvg -y   # 27GB swap
+${schemaVersion === 2 ? v2LvLines : `lvcreate -L 27648M  -n swap   labvg -y   # 27GB swap
 lvcreate -L 102400M -n var    labvg -y   # 100GB /var
 lvcreate -L 10240M  -n varlog labvg -y   # 10GB /var/log
 lvcreate -L 10240M  -n home   labvg -y   # 10GB /home
 lvcreate -L 20480M  -n srv    labvg -y   # 20GB /srv
-${roleLvLines.join('\n')}
+${roleLvLines.join('\n')}`}
 
 # Format
 echo "Formatting volumes..."
-mkswap /dev/labvg/swap
+${schemaVersion === 2 ? v2FormatLines : `mkswap /dev/labvg/swap
 mkfs.xfs /dev/labvg/var
 mkfs.xfs /dev/labvg/varlog
 mkfs.xfs /dev/labvg/home
 mkfs.xfs /dev/labvg/srv
-${roleFormatLines.join('\n')}
+${roleFormatLines.join('\n')}`}
 
 # Migrate and mount volumes that can be switched live.
 # Copy existing content first so we don't shadow files (e.g. /home/user/.ssh).

bastion/src/bastion/src/templates/install.ks.ts

@@ -3,6 +3,7 @@
 // packages, and %post with SSH keys, user creation, k3s prereqs, progress callbacks.
 
 import type { Role } from "@lab/shared";
+import { v2Layout, V2_BOOT, requiresSchemaV2, effectiveSchemaVersion, isK3sServer, type LvSpec } from "@lab/shared";
 
 export interface InstallKickstartParams {
   hostname: string;
@@ -17,6 +18,15 @@ export interface InstallKickstartParams {
   syslogPort: number;
   sshKeys: string[];
   adminUser: string;
+  /**
+   * Disk layout to use for a FRESH install. Defaults to 1 so every existing
+   * caller keeps today's layout untouched.
+   *
+   * On a REPROVISION this is ignored: the schema already on the disk wins,
+   * read from an LVM tag on the volume group. Reinstalling a machine must
+   * never silently change its layout -- that is how data gets destroyed.
+   */
+  schemaVersion?: 1 | 2;
 }
 
 export function renderInstallKickstart(params: InstallKickstartParams): string {
@@ -38,9 +48,63 @@ export function renderInstallKickstart(params: InstallKickstartParams): string {
   const fqdn = domain ? `${hostname}.${domain}` : hostname;
   const vg = "labvg";
   const now = new Date().toISOString();
+  // ---------------------------------------------------------------------------
+  // v1 LAYOUT -- DO NOT REPLACE THESE WITH CAPABILITY LOOKUPS.
+  //
+  // These literals are deliberately frozen. Every machine already in the fleet
+  // was partitioned by them, and `labctl provision reprovision` must reproduce
+  // that same layout byte-for-byte or it destroys data. Yes, they are wrong:
+  // `labcontroller` runs k3s and gets no rancher LV, and `infra` nodes hold
+  // every Longhorn replica in the homelab cluster while only `worker` gets a
+  // longhorn LV. Those are fixed in the v2 layout, which new boxes get; see
+  // schemaVersion. Changing them here rewrites history for existing hardware.
+  // ---------------------------------------------------------------------------
+  // ---------------------------------------------------------------------------
+  // v2 LAYOUT. One definition, shared with ubuntu-autoinstall and asahi-firstboot
+  // (shared/src/types/layout.ts) so the three generators cannot drift apart
+  // again. Capability-gated, and nothing uses --grow.
+  // ---------------------------------------------------------------------------
+  const schemaVersion = effectiveSchemaVersion(role, params.schemaVersion);
+  if (params.schemaVersion === 1 && requiresSchemaV2(role)) {
+    // Fail here rather than render a machine with no rancher/longhorn LV.
+    throw new Error(
+      `role '${role}' has no v1 disk layout - it was introduced with schema v2. ` +
+      `Pass schemaVersion: 2 (labctl provision install --schema-version 2).`,
+    );
+  }
+  const v2RootMib = v2Layout(role).find((lv: LvSpec) => lv.name === "root")?.sizeMib ?? 32768;
+  const v2 = v2Layout(role);
+  const v2LogvolLines = v2
+    .map((lv: LvSpec) =>
+      lv.fs === "swap"
+        ? `logvol swap --vgname=${vg} --name=${lv.name} --fstype=swap --size=${lv.sizeMib}`
+        : `logvol ${lv.mount} --vgname=${vg} --name=${lv.name} --fstype=xfs --size=${lv.sizeMib}`,
+    )
+    .join("\n");
+  /** OS volumes only - the ones a reprovision recreates rather than preserves. */
+  const v2OsLogvolLines = v2
+    .filter((lv: LvSpec) => ["swap", "root", "var", "varlog"].includes(lv.name))
+    .map((lv: LvSpec) =>
+      lv.fs === "swap"
+        ? `logvol swap --vgname=${vg} --name=${lv.name} --fstype=swap --size=${lv.sizeMib}`
+        : `logvol ${lv.mount} --vgname=${vg} --name=${lv.name} --fstype=xfs --size=${lv.sizeMib}`,
+    )
+    .join("\n");
+
   const hasLonghorn = role === "worker";
   const hasRancher = role === "infra";
+  // k8s roles get a dedicated 120G image-store LV. 2026-08 incident: the old
+  // 20G LV idled at 85% used, so a single ~5G image pull tripped imagefs
+  // eviction. Must be sized here — longhorn's --grow consumes all remaining
+  // VG space, making post-install lvextend impossible on worker nodes.
+  const hasRancherLv = role === "infra" || role === "worker";
   const isVanilla = role === "vanilla";
+  // etcd members need etcdctl available OFFLINE. The recovery procedure for a
+  // lost or stale member is `etcdctl member list/remove`, and `k3s etcdctl` is
+  // not a command -- the binary comes from the `etcd` rpm. Telling an operator
+  // to `dnf install etcd` in the middle of a quorum failure assumes working
+  // DNS, routing and a mirror, which is exactly what may be broken.
+  const needsEtcdctl = isK3sServer(role);
 
   // -- Auth section --
   // Always set a root password (for serial console debugging) + SSH keys
@@ -113,9 +177,9 @@ done
     ? `logvol /var/lib/longhorn --vgname=${vg} --name=longhorn --fstype=xfs --grow --size=1`
     : "";
 
-  // -- Rancher LV for fresh install (infra role) --
-  const rancherFreshLine = hasRancher
-    ? `logvol /var/lib/rancher --vgname=${vg} --name=rancher --fstype=xfs --size=20480`
+  // -- Rancher LV for fresh install (k8s roles: worker + infra) --
+  const rancherFreshLine = hasRancherLv
+    ? `logvol /var/lib/rancher --vgname=${vg} --name=rancher --fstype=xfs --size=122880`
     : "";
 
   return `# Lab Bastion -- Fedora ${fedoraVersion} server install
@@ -129,7 +193,13 @@ lang ${locale}
 keyboard uk
 timezone ${timezone} --utc
 
-network --bootproto=dhcp --activate --hostname=${fqdn}
+# --ipv6=auto, not =dhcp: "auto" follows the router advertisement, so the RA's
+# managed-flag is what steers the node to DHCPv6, and a VLAN with no DHCPv6 yet
+# still installs instead of blocking on a lease that will never come. Failing an
+# OS install because IPv6 was not ready would be a worse trade than a node that
+# briefly has no v6. The address itself comes from a kea DHCPv6 reservation
+# keyed on MAC -- the same source of truth as the v4 address.
+network --bootproto=dhcp --ipv6=auto --activate --hostname=${fqdn}
 
 ${auth}
 ${userDirective}
@@ -168,6 +238,16 @@ if vgs $VG &>/dev/null; then
     echo "=== Existing VG found - reprovision mode ==="
     REPROVISION=yes
 
+    # Which layout is ALREADY on this disk? The LVM tag is authoritative -- the
+    # schemaVersion the caller asked for is irrelevant here, because silently
+    # re-laying-out an existing machine destroys its data. A v1 box reinstalls
+    # as v1 forever, until it is deliberately migrated.
+    DISK_SCHEMA=v1
+    if vgs --noheadings -o vg_tags $VG 2>/dev/null | grep -q lab-schema-v2; then
+        DISK_SCHEMA=v2
+    fi
+    echo "On-disk schema: $DISK_SCHEMA"
+
     # Detect which data LVs to preserve
     PRESERVE_LONGHORN=no; PRESERVE_SRV=no; PRESERVE_HOME=no; PRESERVE_RANCHER=no
     lvs $VG/longhorn &>/dev/null && PRESERVE_LONGHORN=yes
@@ -200,11 +280,21 @@ clearpart --none
 part /boot/efi --onpart=$EFI_PART --fstype=efi
 part /boot --onpart=$BOOT_PART --fstype=ext4
 volgroup ${vg} --useexisting --noformat
+PARTEOF
+
+    # OS volumes are recreated, so they must match the layout already on disk.
+    if [ "$DISK_SCHEMA" = "v2" ]; then
+        cat >> /tmp/part.ks << PARTEOF
+${v2OsLogvolLines}
+PARTEOF
+    else
+        cat >> /tmp/part.ks << PARTEOF
 logvol swap --vgname=${vg} --name=swap --fstype=swap --size=27648
 logvol / --vgname=${vg} --name=root --fstype=xfs --size=33792
 logvol /var --vgname=${vg} --name=var --fstype=xfs --size=102400
 logvol /var/log --vgname=${vg} --name=varlog --fstype=xfs --size=10240
 PARTEOF
+    fi
 
     # Preserve or recreate data LVs
     if [ "$PRESERVE_HOME" = "yes" ]; then
@@ -229,7 +319,19 @@ PARTEOF
 
 else
     # Fresh install
-    cat > /tmp/part.ks << PARTEOF
+${schemaVersion === 2 ? `    cat > /tmp/part.ks << PARTEOF
+ignoredisk --only-use=$DISK
+clearpart --all --initlabel --drives=$DISK
+part /boot/efi --fstype=efi --size=${V2_BOOT.efiMib} --ondisk=$DISK
+part /boot --fstype=ext4 --size=${V2_BOOT.bootMib} --ondisk=$DISK
+part pv.01 --size=1 --grow --ondisk=$DISK
+volgroup ${vg} pv.01
+${v2LogvolLines}
+PARTEOF
+    # Tag the VG so a later reprovision knows this disk is v2 WITHOUT trusting
+    # whatever schemaVersion the caller happens to pass. The tag lives on the
+    # VG, which a reprovision preserves; a marker file would not survive.
+    SCHEMA_TAG_PENDING=v2` : `    cat > /tmp/part.ks << PARTEOF
 ignoredisk --only-use=$DISK
 clearpart --all --initlabel --drives=$DISK
 part /boot/efi --fstype=efi --size=600 --ondisk=$DISK
@@ -244,7 +346,7 @@ logvol /home --vgname=${vg} --name=home --fstype=xfs --size=10240
 logvol /srv --vgname=${vg} --name=srv --fstype=xfs --size=20480
 ${longhornFreshLine}
 ${rancherFreshLine}
-PARTEOF
+PARTEOF`}
 fi
 
 echo "=== Generated partition config ==="
@@ -291,6 +393,9 @@ tar
 socat
 conntrack-tools
 ethtool`}
+${needsEtcdctl ? `
+# etcdctl, for recovering etcd membership without internet access
+etcd` : ""}
 
 # Boot management
 efibootmgr
@@ -361,6 +466,21 @@ fs.inotify.max_user_watches         = 1048576
 SYSCTL
 sysctl --system || true
 
+# -- IPv6 link-local address generation: EUI-64, fleet-wide --
+# A cluster node takes its IPv6 from a MAC-keyed DHCPv6 reservation. kea can only
+# match that reservation when it can recover the node's MAC, and for a modern
+# client that sends a DUID-UUID (no MAC in it) the only place kea can find one is
+# an EUI-64 link-local. NetworkManager's default is stable-privacy (RFC 7217),
+# whose link-local hides the MAC -- so a node on the default silently never gets
+# its reserved address, and with a reservations-only subnet it gets nothing at
+# all. Proven on 2026-09-06: worker0/worker2 (eui64) bound; worker1/spark
+# (default) did not, until flipped. Setting it here means a new node is correct
+# from first boot, before its connection is ever activated.
+cat > /etc/NetworkManager/conf.d/10-ipv6-eui64.conf << 'NMEUI64'
+[connection]
+ipv6.addr-gen-mode=eui64
+NMEUI64
+
 # -- Disable firewalld permanently (k3s/Cilium manage iptables directly) --
 # Note: no '--now' — systemd is not running in the Anaconda chroot
 systemctl disable firewalld || true
@@ -388,12 +508,30 @@ bastion_progress "post-install" "3-bootorder done"
 # -- Enable SysRq magic keys (for emergency reboot via Alt+SysRq+REISUB) --
 echo "kernel.sysrq=1" > /etc/sysctl.d/90-sysrq.conf
 
+# -- Disk schema tag --
+# Tag the VG so a future reprovision can tell which layout is on this disk
+# WITHOUT trusting whatever the caller passes. Deliberately derived from what
+# actually landed on disk rather than from render-time intent: a v1 machine
+# reinstalled with --schema-version 2 keeps its v1 layout (see %pre), and must
+# not end up wearing a v2 tag it does not deserve.
+V2_ROOT_MIB=${v2RootMib}
+ROOT_MIB=$(lvs --noheadings --units m --nosuffix -o lv_size ${vg}/root 2>/dev/null | tr -d ' ' | cut -d. -f1)
+if [ "$ROOT_MIB" = "$V2_ROOT_MIB" ]; then
+    vgchange --addtag lab-schema-v2 ${vg} >/dev/null 2>&1 || true
+    echo "Tagged ${vg} as lab-schema-v2 (root=\${ROOT_MIB}M)"
+    DISK_SCHEMA_ON_DISK=2
+else
+    DISK_SCHEMA_ON_DISK=1
+    echo "Layout is v1 (root=\${ROOT_MIB}M, v2 would be \${V2_ROOT_MIB}M)"
+fi
+
 # -- Provisioning metadata --
 cat > /etc/lab-provisioned << PROVEOF
 hostname: ${fqdn}
 role: ${role}
 provisioned: $(date -Iseconds)
 bastion: ${serverIp}
+schema: v$DISK_SCHEMA_ON_DISK
 PROVEOF
 
 cat > /root/README << 'README'

bastion/src/bastion/src/templates/ubuntu-autoinstall.ts

@@ -2,10 +2,15 @@
 // Equivalent of the Fedora kickstart: LVM partitioning, packages,
 // SSH keys, k3s prereqs, progress callbacks.
 
+import { v2Layout, type LvSpec } from "@lab/shared";
+import type { Role } from "@lab/shared";
+
 export interface UbuntuAutoinstallParams {
   hostname: string;
   disk: string;
-  role: string;  // "vanilla" | "worker" | "infra"
+  role: Role;
+  /** Fresh-install layout. Defaults to 1 so existing callers are unchanged. */
+  schemaVersion?: 1 | 2;
   domain: string;
   ubuntuVersion: string;
   timezone: string;
@@ -31,6 +36,48 @@ export function renderUbuntuAutoinstall(params: UbuntuAutoinstallParams): string
 
   const fqdn = domain ? `${hostname}.${domain}` : hostname;
   const vg = "labvg";
+  // ---------------------------------------------------------------------------
+  // v1 LAYOUT -- DO NOT REPLACE THESE WITH CAPABILITY LOOKUPS.
+  //
+  // These literals are deliberately frozen. Every machine already in the fleet
+  // was partitioned by them, and `labctl provision reprovision` must reproduce
+  // that same layout byte-for-byte or it destroys data. Yes, they are wrong:
+  // `labcontroller` runs k3s and gets no rancher LV, and `infra` nodes hold
+  // every Longhorn replica in the homelab cluster while only `worker` gets a
+  // longhorn LV. Those are fixed in the v2 layout, which new boxes get; see
+  // schemaVersion. Changing them here rewrites history for existing hardware.
+  // ---------------------------------------------------------------------------
+  // ---------------------------------------------------------------------------
+  // v2 LAYOUT, rendered from the SAME definition Fedora and Asahi use
+  // (shared/src/types/layout.ts). Before this, Ubuntu sized rancher at 20G
+  // while the kickstart used 120G -- the drift that the 2026-08 imagefs
+  // eviction incident was caused by.
+  //
+  // Indentation is load-bearing: list items sit at 6 spaces, their keys at 8.
+  // At any other depth curtin's YAML does not parse. See the v1 comment below.
+  // ---------------------------------------------------------------------------
+  const schemaVersion = params.schemaVersion ?? 1;
+  const renderV2Lv = (lv: LvSpec): string => {
+    const gib = Math.round(lv.sizeMib / 1024);
+    const mountPath = lv.fs === "swap" ? "none" : lv.mount;
+    return [
+      `      - id: lv-${lv.name}`,
+      `        name: ${lv.name}`,
+      `        type: lvm_partition`,
+      `        volgroup: vg0`,
+      `        size: ${gib}G`,
+      `      - id: fs-${lv.name}`,
+      `        type: format`,
+      `        volume: lv-${lv.name}`,
+      `        fstype: ${lv.fs}`,
+      `      - id: mount-${lv.name}`,
+      `        type: mount`,
+      `        device: fs-${lv.name}`,
+      `        path: ${mountPath}`,
+    ].join("\n");
+  };
+  const v2LvBlock = v2Layout(role).map(renderV2Lv).join("\n");
+
   const hasLonghorn = role === "worker";
   const hasRancher = role === "infra";
 
@@ -40,34 +87,39 @@ export function renderUbuntuAutoinstall(params: UbuntuAutoinstallParams): string
   // Build the LVM layout to match Fedora kickstart sizes
   const extraLvs: string[] = [];
   if (hasLonghorn) {
-    extraLvs.push(`        - id: lv-longhorn
-          name: longhorn
-          type: lvm_partition
-          volgroup: vg0
-          size: -1
-        - id: fs-longhorn
-          type: format
-          volume: lv-longhorn
-          fstype: xfs
-        - id: mount-longhorn
-          type: mount
-          device: fs-longhorn
-          path: /var/lib/longhorn`);
+    // 6 spaces for the list item, 8 for its keys -- these are siblings of the
+    // lv-home/lv-srv entries in storage.config, which sit at 6. At 8 the
+    // rendered document is not valid YAML at all ("expected <block end>, but
+    // found '-'"), so an Ubuntu node with the longhorn role could never have
+    // installed. Ubuntu + longhorn is exactly the worker shape.
+    extraLvs.push(`      - id: lv-longhorn
+        name: longhorn
+        type: lvm_partition
+        volgroup: vg0
+        size: -1
+      - id: fs-longhorn
+        type: format
+        volume: lv-longhorn
+        fstype: xfs
+      - id: mount-longhorn
+        type: mount
+        device: fs-longhorn
+        path: /var/lib/longhorn`);
   }
   if (hasRancher) {
-    extraLvs.push(`        - id: lv-rancher
-          name: rancher
-          type: lvm_partition
-          volgroup: vg0
-          size: 20G
-        - id: fs-rancher
-          type: format
-          volume: lv-rancher
-          fstype: xfs
-        - id: mount-rancher
-          type: mount
-          device: fs-rancher
-          path: /var/lib/rancher`);
+    extraLvs.push(`      - id: lv-rancher
+        name: rancher
+        type: lvm_partition
+        volgroup: vg0
+        size: 20G
+      - id: fs-rancher
+        type: format
+        volume: lv-rancher
+        fstype: xfs
+      - id: mount-rancher
+        type: mount
+        device: fs-rancher
+        path: /var/lib/rancher`);
   }
 
   const extraLvsBlock = extraLvs.length > 0 ? "\n" + extraLvs.join("\n") : "";
@@ -81,6 +133,11 @@ export function renderUbuntuAutoinstall(params: UbuntuAutoinstallParams): string
     `curtin in-target -- bash -c 'cat > /etc/modules-load.d/k3s.conf << EOF\nbr_netfilter\noverlay\nip_conntrack\nEOF'`,
     // Sysctl for k3s networking
     `curtin in-target -- bash -c 'cat > /etc/sysctl.d/90-k3s.conf << EOF\nnet.bridge.bridge-nf-call-iptables  = 1\nnet.bridge.bridge-nf-call-ip6tables = 1\nnet.ipv4.ip_forward                 = 1\nnet.ipv6.conf.all.forwarding        = 1\nfs.inotify.max_user_instances       = 524288\nfs.inotify.max_user_watches         = 1048576\nEOF'`,
+    // IPv6 link-local = EUI-64, so a MAC-keyed DHCPv6 reservation can match: kea
+    // recovers the node's MAC from an EUI-64 link-local when the client sends a
+    // DUID-UUID (no MAC in it). NM's stable-privacy default hides the MAC and the
+    // node silently never gets its reserved address. Proven 2026-09-06.
+    `curtin in-target -- bash -c 'cat > /etc/NetworkManager/conf.d/10-ipv6-eui64.conf << EOF\n[connection]\nipv6.addr-gen-mode=eui64\nEOF'`,
     // Disable ufw firewall
     `curtin in-target -- systemctl disable ufw || true`,
     // Enable chrony/ntp
@@ -121,7 +178,20 @@ export function renderUbuntuAutoinstall(params: UbuntuAutoinstallParams): string
     `curtin in-target -- bash -c 'IP_ADDR=$(ip -4 addr show | awk "/inet / && !/127.0.0/ {split(\\$2,a,\\"/\\"); print a[1]; exit}"); curl -sf -X POST "http://${serverIp}:${httpPort}/api/progress" -H "Content-Type: application/json" -d "{\\"mac\\":\\"$(ip link show | awk "/ether/ && !/00:00:00:00/ {print \\$2; exit}")\\",\\"stage\\":\\"complete\\",\\"detail\\":\\"ready at $IP_ADDR\\"}" || true'`,
   );
 
-  const lateCommandsYaml = lateCommands.map((c) => `        - "${c}"`).join("\n");
+  // JSON.stringify, not `"${c}"`. JSON is a subset of YAML, so this produces a
+  // correctly escaped double-quoted scalar for free -- and the naive version
+  // was broken in two ways at once, for every role:
+  //
+  //   * embedded double quotes ended the scalar early
+  //     (`echo "tmpfs /tmp ..." >> /etc/fstab` -> "expected <block end>")
+  //   * the heredocs contain REAL newlines, and YAML folds newlines inside a
+  //     double-quoted scalar into spaces -- so even where it parsed, the
+  //     heredoc arrived at the target as one long line and wrote a file with
+  //     no line breaks.
+  //
+  // JSON escaping turns the newlines into \n, which YAML unescapes back to
+  // real newlines on parse, so the heredoc survives intact.
+  const lateCommandsYaml = lateCommands.map((c) => `        - ${JSON.stringify(c)}`).join("\n");
 
   return `#cloud-config
 autoinstall:
@@ -139,6 +209,30 @@ autoinstall:
     allow-pw: false
     authorized-keys:
 ${sshKeysYaml}
+  # Both address families. Without dhcp6 the installer's default is IPv4-only,
+  # so a node provisioned into a dual-stack cluster comes up with no IPv6, k3s
+  # has no v6 node-ip to bind, and it joins as an IPv4-only member of a
+  # dual-stack cluster -- which surfaces later as pods on that node being
+  # unreachable over v6 while the node itself reads Ready.
+  #
+  # The address itself comes from a kea DHCPv6 reservation keyed on MAC, the
+  # same source of truth as the v4 address, so nothing here needs to know it.
+  #
+  # optional: true matters -- it lets the install proceed if the v6 lease is
+  # slow or the VLAN has no DHCPv6 yet, rather than blocking on a timeout. The
+  # node still needs the address before k3s starts, but that is a later step's
+  # problem and failing the OS install over it would be worse.
+  # (No backticks in this comment: it lives inside a TS template literal, and a
+  # backtick here ends the literal and breaks the build.)
+  network:
+    version: 2
+    ethernets:
+      primary:
+        match:
+          name: "en*"
+        dhcp4: true
+        dhcp6: true
+        optional: true
   storage:
     config:
       - id: disk0
@@ -182,7 +276,7 @@ ${sshKeysYaml}
         name: ${vg}
         devices:
           - part-pv
-      - id: lv-swap
+${schemaVersion === 2 ? v2LvBlock : `      - id: lv-swap
         name: swap
         type: lvm_partition
         volgroup: vg0
@@ -259,7 +353,7 @@ ${sshKeysYaml}
       - id: mount-srv
         type: mount
         device: fs-srv
-        path: /srv${extraLvsBlock}
+        path: /srv${extraLvsBlock}`}
   packages:
     - openssh-server
     - curl

bastion/src/bastion/src/templates/vyos-boot.ipxe.ts

@@ -0,0 +1,53 @@
+// iPXE boot script template for VyOS network install.
+//
+// VyOS ships no unattended installer: `install image` is unconditionally
+// interactive (image_installer.py's install action takes no arguments, and
+// --no-prompt is wired only to `add`). So PXE boots the *live* system and the
+// automation is injected through live-config's `hooks` component, which fetches
+// a script over HTTP and runs it as root late in live boot.
+//
+// Unlike the Fedora/Ubuntu paths this boots a live image rather than an
+// installer, so there is no kickstart/autoinstall equivalent — see
+// routes/vyos.ts for the hook that actually drives the install.
+
+export function renderVyosInstallIpxe(params: {
+  mac: string;
+  hostname: string;
+  serverIp: string;
+  httpPort: number;
+}): string {
+  const base = `http://${params.serverIp}:${params.httpPort}`;
+
+  // Pin the boot NIC by MAC. live-boot otherwise scans for the first
+  // *connected* interface, and on a multi-NIC box that race is lost by
+  // whichever port negotiates slowest: on the Protectli VP2440 the SFP+
+  // pair links first, so live-boot picked the fiber ports (which have no
+  // DHCP), burned 15s per port, and gave up with "Unable to find a live
+  // file system on the network" -- while the copper port that actually PXE
+  // booted came up at 4.6s and was never tried.
+  //
+  // live-boot's Device_from_bootif() strips the "01-" and matches the MAC
+  // against /sys/class/net/*. params.mac is the dispatch key, i.e. exactly
+  // the NIC that PXE booted -- more reliable than iPXE's ${net0} on a box
+  // where the booting NIC may not be net0.
+  const bootif = `01-${params.mac.toLowerCase().replace(/:/g, "-")}`;
+
+  // Deliberately NOT passing `nonetworking` (present in VyOS's own PXE docs):
+  // live-config's hook component needs networking up to fetch the hook over
+  // HTTP. Also no `console=ttyS0` — on hardware without a physical UART that
+  // costs 30s at every systemd boot phase.
+  return `#!ipxe
+
+echo
+echo =============================================
+echo   Lab PXE Bastion - INSTALLING VyOS
+echo   Target: ${params.hostname}
+echo   MAC:    ${params.mac}
+echo =============================================
+echo
+
+kernel ${base}/vyos-vmlinuz boot=live nopersistence noautologin BOOTIF=${bootif} fetch=${base}/vyos-filesystem.squashfs live-config.hooks=${base}/vyos/autoinstall.sh?mac=${params.mac}
+initrd ${base}/vyos-initrd
+boot
+`;
+}

bastion/src/bastion/src/templates/vyos-config-spec.ts

@@ -0,0 +1,352 @@
+// Builds the VyOS configuration spec applied by the autoinstall hook.
+//
+// We deliberately do NOT emit a config.boot file as text. A config.boot carries a
+// `vyos-config-version` trailer; without a trailer matching the running image,
+// VyOS runs its migration scripts from version 0 on first boot. Instead the hook
+// loads the image's own /opt/vyatta/etc/config.boot.default through vyos.configtree
+// and applies these set operations on top, so syntax and version trailer always
+// match the exact image being installed.
+
+import type { VyosInstallSpec } from "@lab/shared";
+
+export interface VyosSetOp {
+  path: string[];
+  value?: string;
+  /** false appends to a multi-value node (e.g. bond members) instead of replacing. */
+  replace?: boolean;
+}
+
+export interface VyosConfigSpec {
+  hostname: string;
+  /** "" means accept the installer default (the running image's version string). */
+  imageName: string;
+  password: string;
+  console: "K" | "S";
+  /** Target disk name (e.g. "nvme0n1"); "" accepts the installer's first-disk default. */
+  disk: string;
+  /**
+   * IP the driver should report in the "complete" callback ("ready at <ip>" —
+   * the exact format routes/api.ts parses installed.ip from). The mgmt
+   * address when static; "" means detect the live DHCP address at runtime.
+   */
+  reportAddress: string;
+  /** Whether to accept RAID-1 when the installer finds more than one disk. */
+  raid: boolean;
+  /** Overwrite the installed config.boot with the generated one on reinstall. */
+  freshConfig: boolean;
+  sets: VyosSetOp[];
+  /** Paths that are VyOS tag nodes — must be marked as such in the ConfigTree. */
+  tags: string[][];
+}
+
+/**
+ * Normalise a target disk to the form the installer expects.
+ *
+ * find_disks() enumerates via `lsblk -Jbp` (-p = full paths), so its valid
+ * responses are "/dev/mmcblk0"-style. A bare "mmcblk0" is rejected by
+ * ask_input()'s valid_responses check and re-prompts forever.
+ */
+function normalizeDiskPath(value: string | undefined): string {
+  const raw = (value ?? "").trim();
+  if (raw === "") return "";
+  return raw.startsWith("/dev/") ? raw : `/dev/${raw}`;
+}
+
+/** Sentinel marking a value that lives in Pulumi config, not in the bundle. */
+const SECRET_PREFIX = "@secret:";
+
+/**
+ * Enable the VyOS HTTP API so the router is manageable the moment it boots.
+ *
+ * This belongs at install time rather than in the Pulumi model: the model is
+ * applied THROUGH this API, so a router that lacks it cannot be brought under
+ * management without a hand-run change on a live firewall. It is also why the
+ * model excludes `service https` outright -- a provider able to rewrite its own
+ * transport can lock itself out permanently.
+ *
+ * `listen-address` is always set. Leaving it unbound would expose a
+ * config-write endpoint on every segment the router touches, the WAN included.
+ */
+function apiSets(apiKey: string, listenAddress: string): VyosSetOp[] {
+  const sets: VyosSetOp[] = [
+    { path: ["service", "https", "api", "keys", "id", "pulumi", "key"], value: apiKey },
+    { path: ["service", "https", "api", "rest"] },
+  ];
+  if (listenAddress !== "") {
+    sets.push({ path: ["service", "https", "listen-address"], value: listenAddress });
+  }
+  return sets;
+}
+
+/** Tag nodes introduced by the API config, needed by the installer's ConfigTree. */
+const API_TAGS: string[][] = [["service", "https", "api", "keys", "id"]];
+
+/**
+ * The address to bind the API to: an explicit choice, else the management
+ * address with its prefix length stripped. Under DHCP there is no address to
+ * bind at build time, so the caller must pass one or the listener stays unbound
+ * and the API is not enabled at all.
+ */
+function apiListenAddress(spec: VyosInstallSpec, mgmtAddress: string): string {
+  if (spec.apiListenAddress !== undefined && spec.apiListenAddress !== "") {
+    return spec.apiListenAddress;
+  }
+  return mgmtAddress.includes("/") ? (mgmtAddress.split("/")[0] ?? "") : "";
+}
+
+/**
+ * Use a Pulumi-rendered bundle as the router's config verbatim.
+ *
+ * Secret-valued nodes are dropped rather than installed with their sentinel
+ * text: writing `@secret:pppoePassword` into config.boot would look configured
+ * while being wrong, which is worse than being absent. The router comes up
+ * without those values and the first `pulumi up` fills them in.
+ *
+ * `system host-name` is forced to the hostname the install was asked for. The
+ * bundle carries the name of whichever router it was exported from, and
+ * installing vyos001's hostname onto vyos002 would collide on the network.
+ */
+function buildFromBundle(
+  params: { hostname: string; defaultPassword: string; disk?: string | undefined },
+  spec: VyosInstallSpec,
+  bundle: NonNullable<VyosInstallSpec["bundle"]>,
+  mgmtAddress: string,
+): VyosConfigSpec {
+  const sets: VyosSetOp[] = [];
+  const dropped: string[] = [];
+  for (const op of bundle.sets) {
+    if (op.value !== undefined && op.value.startsWith(SECRET_PREFIX)) {
+      dropped.push(op.path.join(" "));
+      continue;
+    }
+    if (op.path.length === 2 && op.path[0] === "system" && op.path[1] === "host-name") {
+      continue;
+    }
+    sets.push({
+      path: op.path,
+      ...(op.value === undefined ? {} : { value: op.value }),
+      ...(op.replace === undefined ? {} : { replace: op.replace }),
+    });
+  }
+  sets.unshift({ path: ["system", "host-name"], value: params.hostname });
+
+  if (dropped.length > 0) {
+    console.warn(
+      `vyos ${params.hostname}: ${dropped.length} secret-valued node(s) left unset by the ` +
+        `bundle; run \`pulumi up\` to supply them: ${dropped.join(", ")}`,
+    );
+  }
+
+  const tags = [...bundle.tags];
+  const api = enableApi(spec, params.hostname, mgmtAddress);
+  if (api.length > 0) {
+    sets.push(...api);
+    tags.push(...API_TAGS);
+  }
+
+  return {
+    hostname: params.hostname,
+    imageName: "",
+    password: spec.password ?? params.defaultPassword,
+    console: "K",
+    disk: normalizeDiskPath(params.disk),
+    reportAddress: mgmtAddress.includes("/") ? (mgmtAddress.split("/")[0] ?? "") : "",
+    raid: false,
+    freshConfig: spec.freshConfig ?? false,
+    sets,
+    tags,
+  };
+}
+
+/**
+ * The API config for this install, or nothing when it cannot be enabled safely.
+ *
+ * Refusing to enable it unbound is deliberate. Under DHCP there is no address
+ * known at build time, and the alternative -- binding to every interface --
+ * would publish a config-write endpoint on the WAN. Better to leave the router
+ * SSH-only and say so than to open it everywhere.
+ */
+function enableApi(spec: VyosInstallSpec, hostname: string, mgmtAddress: string): VyosSetOp[] {
+  if (spec.apiKey === undefined || spec.apiKey === "") return [];
+  const listen = apiListenAddress(spec, mgmtAddress);
+  if (listen === "") {
+    console.warn(
+      `vyos ${hostname}: --vyos-api-key given but no address to bind to ` +
+        `(management is "${mgmtAddress}"). Pass --vyos-api-listen <addr>; the HTTP API ` +
+        `has NOT been enabled, so Pulumi cannot manage this router yet.`,
+    );
+    return [];
+  }
+  return apiSets(spec.apiKey, listen);
+}
+
+export function buildVyosConfigSpec(params: {
+  hostname: string;
+  spec?: VyosInstallSpec | undefined;
+  defaultPassword: string;
+  sshKeys?: string[] | undefined;
+  disk?: string | undefined;
+}): VyosConfigSpec {
+  const spec = params.spec ?? {};
+  const mgmt = spec.mgmtInterface ?? "eth0";
+  const mgmtAddress = spec.mgmtAddress ?? "dhcp";
+
+  // A rendered bundle replaces the derived config entirely. Deriving a second
+  // opinion alongside it is the drift the bundle exists to prevent: Pulumi and
+  // labctl would each believe they knew the router's config, and the box would
+  // end up with whichever ran last.
+  if (spec.bundle !== undefined) {
+    return buildFromBundle(params, spec, spec.bundle, mgmtAddress);
+  }
+  const bondMembers = spec.bondMembers ?? [];
+  const vlans = spec.vlans ?? [];
+
+  const sets: VyosSetOp[] = [];
+  const tags: string[][] = [
+    ["interfaces", "ethernet"],
+    ["system", "login", "user"],
+  ];
+
+  const hwIds = spec.hwIds ?? {};
+  const pinHwId = (iface: string): void => {
+    const mac = hwIds[iface];
+    if (mac !== undefined && mac !== "") {
+      sets.push({ path: ["interfaces", "ethernet", iface, "hw-id"], value: mac });
+    }
+  };
+
+  sets.push({ path: ["system", "host-name"], value: params.hostname });
+
+  // Management interface — the NIC that PXE booted, left untagged and unbonded.
+  sets.push({ path: ["interfaces", "ethernet", mgmt, "address"], value: mgmtAddress });
+  pinHwId(mgmt);
+
+  // Tagged management VLAN on the PXE port. Emitted regardless of bonding, so
+  // the box stays reachable on the management VLAN while still booting untagged
+  // on whichever VLAN the bastion's proxy DHCP serves.
+  const mgmtVlan = spec.mgmtVlan;
+  if (mgmtVlan !== undefined) {
+    tags.push(["interfaces", "ethernet", mgmt, "vif"]);
+    const vif = ["interfaces", "ethernet", mgmt, "vif", String(mgmtVlan.id)];
+    sets.push({ path: [...vif, "address"], value: mgmtVlan.address });
+    if (mgmtVlan.description !== undefined && mgmtVlan.description !== "") {
+      sets.push({ path: [...vif, "description"], value: mgmtVlan.description });
+    }
+  }
+
+  // LACP bond. Members must exclude the PXE NIC; firmware PXE cannot run over LACP.
+  const bonded = bondMembers.length > 0;
+  if (bonded) {
+    tags.push(["interfaces", "bonding"]);
+    sets.push({ path: ["interfaces", "bonding", "bond0", "mode"], value: "802.3ad" });
+    sets.push({ path: ["interfaces", "bonding", "bond0", "hash-policy"], value: "layer2+3" });
+    for (const member of bondMembers) {
+      sets.push({
+        path: ["interfaces", "bonding", "bond0", "member", "interface"],
+        value: member,
+        replace: false,
+      });
+      pinHwId(member);
+    }
+    // Address on the trunk's native/untagged VLAN.
+    if (spec.bondAddress !== undefined && spec.bondAddress !== "") {
+      sets.push({ path: ["interfaces", "bonding", "bond0", "address"], value: spec.bondAddress });
+    }
+  }
+
+  // VRRP groups accumulate here; emitted (plus a sync group) after the VLANs.
+  // interface accepts dotted vifs (constraint regex `[0-9]+(.\d+)?`), address
+  // is a tag node (the VIP is the tag value itself), vrid range is 1-255.
+  const vrrpGroups: Array<{ name: string; iface: string; vrid: number; vip: string }> = [];
+  if (bonded && spec.bondVrrp !== undefined && spec.bondVrrp !== "") {
+    // vrid 1 for the untagged group: the native VLAN is never a vif, so this
+    // cannot collide with a vlan-id-derived vrid.
+    vrrpGroups.push({ name: "native", iface: "bond0", vrid: 1, vip: spec.bondVrrp });
+  }
+
+  // Tagged VLAN sub-interfaces hang off the bond when there is one, else off mgmt.
+  const parent = bonded
+    ? ["interfaces", "bonding", "bond0"]
+    : ["interfaces", "ethernet", mgmt];
+  if (vlans.length > 0) {
+    tags.push([...parent, "vif"]);
+    const parentName = bonded ? "bond0" : mgmt;
+    for (const vlan of vlans) {
+      const vif = [...parent, "vif", String(vlan.id)];
+      sets.push({ path: [...vif, "address"], value: vlan.address });
+      if (vlan.description !== undefined && vlan.description !== "") {
+        sets.push({ path: [...vif, "description"], value: vlan.description });
+      }
+      if (vlan.vrrp !== undefined && vlan.vrrp !== "") {
+        vrrpGroups.push({
+          name: `vlan${vlan.id}`,
+          iface: `${parentName}.${vlan.id}`,
+          vrid: vlan.id,
+          vip: vlan.vrrp,
+        });
+      }
+    }
+  }
+
+  // Emit VRRP groups plus one sync group so all VLANs fail over together —
+  // without it a single-link event could split mastership across the pair.
+  if (vrrpGroups.length > 0) {
+    tags.push(["high-availability", "vrrp", "group"]);
+    tags.push(["high-availability", "vrrp", "sync-group"]);
+    const priority = String(spec.vrrpPriority ?? 100);
+    for (const g of vrrpGroups) {
+      const base = ["high-availability", "vrrp", "group", g.name];
+      sets.push({ path: [...base, "interface"], value: g.iface });
+      sets.push({ path: [...base, "vrid"], value: String(g.vrid) });
+      sets.push({ path: [...base, "priority"], value: priority });
+      // address is a tag node: the VIP is the path's final segment, no value.
+      sets.push({ path: [...base, "address", g.vip] });
+      tags.push([...base, "address"]);
+      sets.push({
+        path: ["high-availability", "vrrp", "sync-group", "MAIN", "member"],
+        value: g.name,
+        replace: false,
+      });
+    }
+  }
+
+  sets.push({ path: ["service", "ssh", "port"], value: "22" });
+
+  const sshKeys = params.sshKeys ?? [];
+  if (sshKeys.length > 0) {
+    tags.push(["system", "login", "user", "vyos", "authentication", "public-keys"]);
+    sshKeys.forEach((entry, index) => {
+      const parts = entry.trim().split(/\s+/);
+      const type = parts[0] ?? "";
+      const key = parts[1] ?? "";
+      if (!type.startsWith("ssh-") && !type.startsWith("ecdsa-")) return;
+      if (!key) return;
+      const name = parts[2] ?? `lab-key-${index}`;
+      const base = ["system", "login", "user", "vyos", "authentication", "public-keys", name];
+      sets.push({ path: [...base, "type"], value: type });
+      sets.push({ path: [...base, "key"], value: key });
+    });
+  }
+
+  // Enabled here too, not just for bundle installs: every VyOS this bastion
+  // provisions should be manageable from first boot.
+  const api = enableApi(spec, params.hostname, mgmtAddress);
+  if (api.length > 0) {
+    sets.push(...api);
+    tags.push(...API_TAGS);
+  }
+
+  return {
+    hostname: params.hostname,
+    imageName: "",
+    password: spec.password ?? params.defaultPassword,
+    console: "K",
+    disk: normalizeDiskPath(params.disk),
+    // Static mgmt address wins; under DHCP the driver detects the live IP.
+    reportAddress: mgmtAddress.includes("/") ? (mgmtAddress.split("/")[0] ?? "") : "",
+    raid: false,
+    freshConfig: spec.freshConfig ?? false,
+    sets,
+    tags,
+  };
+}

bastion/src/bastion/src/templates/vyos-install.py.ts

@@ -0,0 +1,514 @@
+// Renders the Python program that performs the unattended VyOS install.
+//
+// It runs as root inside the live system, fetched and executed by live-config's
+// `hooks` component (see vyos-boot.ipxe.ts). It does three things:
+//   1. builds config.boot from the image's own default via vyos.configtree
+//   2. drives the interactive `install image` through a pty
+//   3. reports progress back to the bastion, then reboots
+//
+// A pty is used rather than piping stdin because the installer reads the
+// password through getpass(), which opens /dev/tty directly and would ignore a
+// pipe. Prompts are matched by text rather than replayed positionally: the
+// installer skips the boot-config question when it finds a previous
+// installation, so a fixed answer sequence desyncs on reinstall.
+
+import type { VyosConfigSpec } from "./vyos-config-spec.js";
+
+export function renderVyosInstallPy(params: {
+  spec: VyosConfigSpec;
+  mac: string;
+  serverIp: string;
+  httpPort: number;
+  role: string;
+}): string {
+  // Base64 so arbitrary values (passwords, descriptions, SSH keys) can never
+  // terminate the Python string literal that carries them.
+  const specB64 = Buffer.from(JSON.stringify(params.spec), "utf-8").toString("base64");
+
+  return `#!/usr/bin/env python3
+"""Unattended VyOS install driver -- generated by the lab PXE bastion."""
+
+import base64
+import json
+import os
+import pty
+import re
+import select
+import subprocess
+import sys
+import time
+import urllib.request
+
+SPEC = json.loads(base64.b64decode("${specB64}").decode("utf-8"))
+BASTION = "http://${params.serverIp}:${params.httpPort}"
+MAC = "${params.mac}"
+ROLE = ${JSON.stringify(params.role ?? "vanilla")}
+
+INSTALLER = "/usr/libexec/vyos/op_mode/image_installer.py"
+CONFIG_DIR = "/opt/vyatta/etc/config"
+# The installer copies the rootfs from the boot MEDIUM path -- which only a
+# CD/USB boot provides. With fetch= (HTTP netboot) nothing is mounted there
+# (verified in VM: Errno 2), so the squashfs must be linked or re-fetched into
+# place before 'install image' runs.
+ROOTFS_EXPECTED = "/usr/lib/live/mount/medium/live/filesystem.squashfs"
+SQUASHFS_URL = "http://${params.serverIp}:${params.httpPort}/vyos-filesystem.squashfs"
+# The live-config hook runs BEFORE vyos-router creates the /opt/vyatta compat
+# path, so the squashfs's own location must be tried too (verified in VM: only
+# /usr/share/vyos/config.boot.default exists at hook time).
+DEFAULT_CONFIG_CANDIDATES = [
+    "/opt/vyatta/etc/config.boot.default",
+    "/usr/share/vyos/config.boot.default",
+]
+STALL_TIMEOUT = 900  # seconds without installer output before giving up
+
+
+def detect_ip():
+    """Best-effort local IP as seen on the route toward the bastion.
+
+    Matches Fedora's semantics (IP captured during install): under DHCP the
+    installed system will renew on the same NIC/subnet the live env used.
+    """
+    import socket
+    try:
+        s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+        s.connect(("${params.serverIp}", ${params.httpPort}))
+        ip = s.getsockname()[0]
+        s.close()
+        return ip
+    except Exception:
+        return ""
+
+
+class LogStreamer:
+    """Stream install output to the bastion's /api/log so 'labctl provision
+    logs -f' works live for VyOS, like Anaconda's syslog does for Fedora.
+
+    Strictly best-effort: a failed POST drops the batch and must never stall
+    the pty read loop or fail the install.
+    """
+
+    ANSI = re.compile(rb"\\x1b\\[[0-9;?]*[a-zA-Z]|\\x1b[=>]|\\r")
+
+    def __init__(self):
+        self.partial = b""
+        self.pending = []
+        self.last_flush = time.time()
+
+    def feed(self, chunk):
+        """Raw pty bytes: split into lines, strip ANSI noise, queue."""
+        self.partial += chunk
+        while b"\\n" in self.partial:
+            raw, self.partial = self.partial.split(b"\\n", 1)
+            text = self.ANSI.sub(b"", raw).decode("utf-8", "replace").rstrip()
+            if text:
+                self.pending.append(text)
+        self.maybe_flush()
+
+    def line(self, text):
+        """A driver-originated message (already a clean string)."""
+        self.pending.append(text)
+        self.maybe_flush()
+
+    def maybe_flush(self):
+        if len(self.pending) >= 20 or (self.pending and time.time() - self.last_flush >= 2):
+            self.flush()
+
+    def flush(self):
+        if not self.pending:
+            return
+        batch, self.pending = self.pending[:200], self.pending[200:]
+        self.last_flush = time.time()
+        try:
+            body = json.dumps({"mac": MAC, "lines": batch}).encode()
+            req = urllib.request.Request(
+                BASTION + "/api/log",
+                data=body,
+                headers={"Content-Type": "application/json"},
+            )
+            urllib.request.urlopen(req, timeout=5).read()
+        except Exception:
+            pass
+
+
+STREAM = LogStreamer()
+
+
+def say(msg):
+    """Print locally and stream to the bastion log buffer."""
+    print(msg)
+    STREAM.line(str(msg))
+
+
+def report(stage, detail=""):
+    """Best-effort progress callback; never fatal."""
+    STREAM.flush()
+    try:
+        body = json.dumps({"mac": MAC, "stage": stage, "detail": detail}).encode()
+        req = urllib.request.Request(
+            BASTION + "/api/progress",
+            data=body,
+            headers={"Content-Type": "application/json"},
+        )
+        urllib.request.urlopen(req, timeout=5).read()
+    except Exception:
+        pass
+
+
+def build_config():
+    """Apply our set operations onto the image's own default config.
+
+    Using config.boot.default as the base keeps the vyos-config-version trailer
+    consistent with the running image, so first boot does not run migrations.
+    """
+    from vyos.configtree import ConfigTree
+
+    default_config = next(
+        (p for p in DEFAULT_CONFIG_CANDIDATES if os.path.exists(p)), None)
+    if default_config is None:
+        raise FileNotFoundError(
+            "no config.boot.default found (tried %s)" % ", ".join(DEFAULT_CONFIG_CANDIDATES))
+    say("base config: %s" % default_config)
+
+    with open(default_config) as handle:
+        config = ConfigTree(handle.read())
+
+    for op in SPEC["sets"]:
+        replace = op.get("replace", True)
+        if "value" in op and op["value"] is not None:
+            config.set(op["path"], value=op["value"], replace=replace)
+        else:
+            config.set(op["path"])
+
+    # Tag nodes must be marked after the nodes exist, as the installer itself does.
+    for tag in SPEC["tags"]:
+        try:
+            config.set_tag(tag)
+        except Exception as err:
+            say("warning: set_tag %s failed: %s" % (tag, err))
+
+    os.makedirs(CONFIG_DIR, exist_ok=True)
+    target = os.path.join(CONFIG_DIR, "config.boot")
+
+    # Re-attach the vyos-config-version footer: ConfigTree.to_string() emits
+    # only the config body, and a config without the footer is treated as
+    # ancient -- the boot migrator then runs every migration over it and (as
+    # observed in the VM test) crashes in system/31-to-32. Building the footer
+    # from the running system pins it to the exact image being installed.
+    body = config.to_string()
+    try:
+        from vyos.component_version import version_info_from_system
+        info = version_info_from_system()
+        info.update_config_body(body)
+        info.write(target)
+        say("wrote %s (footer: %s)" % (target, info.release))
+    except Exception as err:
+        say("warning: version footer failed (%s); writing bare config" % err)
+        with open(target, "w") as handle:
+            handle.write(body)
+    return target
+
+
+def find_live_squashfs():
+    """Locate the squashfs live-boot fetched, without walking into the mounted
+    rootfs or overlay (each would mean traversing the entire OS tree)."""
+    explicit = [
+        "/run/live/medium/live/filesystem.squashfs",
+        "/lib/live/mount/medium/live/filesystem.squashfs",
+    ]
+    for path in explicit:
+        if os.path.isfile(path) and os.path.getsize(path) > 0:
+            return path
+    for root in ("/run/live", "/lib/live/mount", "/usr/lib/live/mount"):
+        for dirpath, dirs, files in os.walk(root):
+            depth = dirpath.count(os.sep) - root.count(os.sep)
+            dirs[:] = [d for d in dirs
+                       if d not in ("rootfs", "overlay")
+                       and not d.endswith(".squashfs")
+                       and depth < 3]
+            if "filesystem.squashfs" in files:
+                path = os.path.join(dirpath, "filesystem.squashfs")
+                if os.path.isfile(path) and os.path.getsize(path) > 0:
+                    return path
+    return None
+
+
+def ensure_rootfs():
+    """Make FILE_ROOTFS_SRC exist so the installer can copy the system image."""
+    if os.path.isfile(ROOTFS_EXPECTED) and os.path.getsize(ROOTFS_EXPECTED) > 0:
+        return
+    src = find_live_squashfs()
+    if src is None:
+        say("squashfs not in live mounts; re-fetching %s" % SQUASHFS_URL)
+        src = "/tmp/filesystem.squashfs"
+        urllib.request.urlretrieve(SQUASHFS_URL, src)
+    os.makedirs(os.path.dirname(ROOTFS_EXPECTED), exist_ok=True)
+    if os.path.lexists(ROOTFS_EXPECTED):
+        os.remove(ROOTFS_EXPECTED)
+    os.symlink(src, ROOTFS_EXPECTED)
+    say("rootfs source: %s -> %s" % (ROOTFS_EXPECTED, src))
+
+
+def build_rules():
+    """Prompt -> response table for the interactive installer."""
+    password = SPEC["password"].encode() + b"\\n"
+    image_name = SPEC["imageName"].encode() + b"\\n"
+    disk = SPEC["disk"].encode() + b"\\n"
+    console = SPEC["console"].encode() + b"\\n"
+    raid = (b"yes\\n" if SPEC["raid"] else b"no\\n")
+
+    return [
+        (re.compile(rb"Would you like to continue\\?"), b"yes\\n"),
+        (re.compile(rb"What would you like to name this image\\?"), image_name),
+        (re.compile(rb"Please confirm password for the .vyos. user:"), password),
+        (re.compile(rb"Please enter a password for the .vyos. user:"), password),
+        (re.compile(rb"What console should be used by default"), console),
+        # Three RAID variants: "configure RAID-1 mirroring?", "...on them?",
+        # and "choose two disks for RAID-1 mirroring?" -- all default to YES,
+        # so a missed one both hangs the install and risks an unwanted mirror.
+        (re.compile(rb"Would you like to [^?]*RAID-1 mirroring"), raid),
+        (re.compile(rb"Installation will delete all data on (?:the drive|both drives)\\. Continue\\?"), b"yes\\n"),
+        (re.compile(rb"Which one should be used for installation\\?"), disk),
+        (re.compile(rb"Would you like to use all the free space on the drive\\?"), b"yes\\n"),
+        (re.compile(rb"Which file would you like as boot config\\?"), b"1\\n"),
+        # Reinstall path only (search_previous_installation): carrying the old
+        # /config and SSH host keys forward is VyOS's "reinstall without losing
+        # data". Always yes -- freshConfig replaces config.boot afterwards, so
+        # answering no here would also discard non-config data under /config.
+        (re.compile(rb"Would you like to copy data to the new image\\?"), b"yes\\n"),
+        (re.compile(rb"Would you like to copy the encrypted config to the new image\\?"), b"yes\\n"),
+        # More than one previous image found -- take the first offered.
+        (re.compile(rb"From which image would you like to save config information\\?"), b"1\\n"),
+        (re.compile(rb"From which image would you like to copy the encrypted config\\?"), b"1\\n"),
+    ]
+
+
+def run_installer():
+    """Drive image_installer.py over a pty, answering prompts as they appear."""
+    rules = build_rules()
+    master, slave = pty.openpty()
+
+    proc = subprocess.Popen(
+        [INSTALLER, "--action", "install"],
+        stdin=slave,
+        stdout=slave,
+        stderr=slave,
+        close_fds=True,
+        preexec_fn=os.setsid,
+    )
+    os.close(slave)
+
+    buf = b""
+    transcript = b""   # rolling tail of everything the installer printed
+    last_output = time.time()
+
+    while True:
+        ready, _, _ = select.select([master], [], [], 1.0)
+
+        if ready:
+            try:
+                chunk = os.read(master, 4096)
+            except OSError:
+                break
+            if not chunk:
+                break
+
+            sys.stdout.buffer.write(chunk)
+            sys.stdout.buffer.flush()
+            buf += chunk
+            transcript = (transcript + chunk)[-8000:]
+            STREAM.feed(chunk)
+            last_output = time.time()
+
+            # Answer every prompt currently in the buffer, earliest first, so
+            # ordering is preserved even when the installer skips questions --
+            # and so a single chunk carrying two prompts gets both answers.
+            while True:
+                best = None
+                for pattern, response in rules:
+                    found = pattern.search(buf)
+                    if found and (best is None or found.start() < best[0].start()):
+                        best = (found, response)
+                if best is None:
+                    break
+                found, response = best
+                os.write(master, response)
+                transcript = (transcript + b"\\n>>> answered: " + response)[-8000:]
+                STREAM.line(">>> answered: " + response.decode("utf-8", "replace").strip())
+                buf = buf[found.end():]
+
+            # Bound memory if the installer emits a lot without prompting.
+            if len(buf) > 65536:
+                buf = buf[-8192:]
+
+        elif proc.poll() is not None:
+            break
+
+        STREAM.maybe_flush()
+
+        if time.time() - last_output > STALL_TIMEOUT:
+            proc.kill()
+            raise SystemExit("installer produced no output for %ds" % STALL_TIMEOUT)
+
+    os.close(master)
+    return proc.wait(), transcript.decode("utf-8", "replace")
+
+
+def ensure_network_boot_first():
+    """Keep network boot first so the bastion intercepts every reboot.
+
+    Port of the Fedora kickstart's %post efibootmgr step (install.ks.ts) --
+    what makes reprovision-by-reboot work. Best-effort: skipped on BIOS boots
+    or when efibootmgr is absent. Runs from the live env after the installer;
+    efibootmgr edits NVRAM, not the disk, so installer cleanup is irrelevant.
+    """
+    import shutil
+    if not os.path.isdir("/sys/firmware/efi") or shutil.which("efibootmgr") is None:
+        say("boot order: skipped (BIOS boot or efibootmgr missing)")
+        return
+    try:
+        out = subprocess.run(["efibootmgr"], capture_output=True, text=True, timeout=30).stdout
+        order = []
+        network_entry = None
+        for line in out.splitlines():
+            m = re.match(r"^BootOrder:\\s*(.*)$", line)
+            if m:
+                order = [x.strip() for x in m.group(1).split(",") if x.strip()]
+                continue
+            m = re.match(r"^Boot([0-9A-Fa-f]{4})\\*?\\s+(.*)$", line)
+            if m and network_entry is None:
+                if re.search(r"network|pxe|ipv4|ipv6|http", m.group(2), re.IGNORECASE):
+                    network_entry = m.group(1).upper()
+        if network_entry is None or not order:
+            say("boot order: no network boot entry found; leaving as is")
+            return
+        new_order = [network_entry] + [x for x in order if x.upper() != network_entry]
+        if [x.upper() for x in order] == [x.upper() for x in new_order]:
+            say("boot order: network entry Boot%s already first" % network_entry)
+            return
+        subprocess.run(["efibootmgr", "-o", ",".join(new_order)],
+                       capture_output=True, timeout=30)
+        say("boot order: moved network entry Boot%s first" % network_entry)
+    except Exception as err:
+        say("warning: boot order adjustment failed: %s" % err)
+
+
+def with_target_mounted(fn):
+    """Mount the installed root partition, call fn(rw_dir), always unmount.
+
+    The installer has unmounted and cleaned the target by the time this runs,
+    so the block device is free. The partition holding boot/<image>/rw is the
+    VyOS root; the glob also yields the installed image's rw dir directly.
+    """
+    import glob
+    disk = SPEC["disk"]
+    if not disk:
+        # No pinned disk (installer picked the default) -- enumerate all disks.
+        candidates = ["/dev/" + b for b in os.listdir("/sys/block")
+                      if not b.startswith(("loop", "ram", "zram", "sr"))]
+    else:
+        candidates = [disk]
+
+    mnt = "/mnt/lab-target"
+    os.makedirs(mnt, exist_ok=True)
+    for dev in candidates:
+        name = os.path.basename(dev)
+        parts = sorted(p for p in os.listdir("/sys/block/%s" % name)
+                       if p.startswith(name)) if os.path.isdir("/sys/block/%s" % name) else []
+        for part in parts:
+            pdev = "/dev/" + part
+            if subprocess.run(["mount", pdev, mnt], capture_output=True).returncode != 0:
+                continue
+            try:
+                rw_dirs = glob.glob(os.path.join(mnt, "boot", "*", "rw"))
+                if rw_dirs:
+                    fn(rw_dirs[0])
+                    return True
+            finally:
+                subprocess.run(["umount", mnt], capture_output=True)
+    return False
+
+
+def post_install_target_steps():
+    """Metadata + optional fresh-config overwrite inside the installed image."""
+    def apply(rw_dir):
+        config_dir = os.path.join(rw_dir, "opt/vyatta/etc/config")
+        os.makedirs(config_dir, exist_ok=True)
+
+        # /config/lab-provisioned -- survives VyOS image upgrades. Mirrors the
+        # Fedora kickstart's /etc/lab-provisioned.
+        try:
+            with open(os.path.join(config_dir, "lab-provisioned"), "w") as handle:
+                handle.write("hostname=%s\\n" % SPEC["hostname"])
+                handle.write("role=%s\\n" % ROLE)
+                handle.write("provisioned=%s\\n" % time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()))
+                handle.write("bastion=%s\\n" % BASTION)
+            say("wrote /config/lab-provisioned")
+        except Exception as err:
+            say("warning: lab-provisioned metadata failed: %s" % err)
+
+        # freshConfig: make the bastion-generated config win over the previous
+        # installation's carried-forward config. Explicit intent -- failure is
+        # fatal (raised out of with_target_mounted).
+        if SPEC.get("freshConfig"):
+            import shutil
+            shutil.copyfile(os.path.join(CONFIG_DIR, "config.boot"),
+                            os.path.join(config_dir, "config.boot"))
+            say("freshConfig: replaced installed config.boot with generated config")
+
+    mounted = with_target_mounted(apply)
+    if not mounted:
+        if SPEC.get("freshConfig"):
+            raise RuntimeError("freshConfig requested but installed root partition not found")
+        say("warning: installed root partition not found; skipping metadata")
+
+
+def main():
+    report("vyos-install", "building config.boot")
+    try:
+        build_config()
+    except Exception as err:
+        report("error", "config generation failed: %s" % err)
+        raise
+
+    report("vyos-install", "staging rootfs for installer")
+    try:
+        ensure_rootfs()
+    except Exception as err:
+        report("error", "rootfs staging failed: %s" % err)
+        raise
+
+    report("vyos-install", "running install image")
+    code, transcript = run_installer()
+
+    if code != 0:
+        # Surface the installer's last words in bastion progress -- the console
+        # they were printed on is usually invisible during unattended installs.
+        report("error", "install image exited %d | tail: %s" % (code, transcript[-4000:]))
+        raise SystemExit(code)
+
+    report("post-install", "boot order + metadata")
+    ensure_network_boot_first()
+    try:
+        post_install_target_steps()
+    except Exception as err:
+        report("error", "post-install target steps failed: %s" % err)
+        raise
+
+    # "complete" is the stage the bastion uses to move a machine out of the
+    # install queue into installed state, and "ready at <ip>" is the exact
+    # detail format it parses installed.ip from -- see routes/api.ts.
+    ip = SPEC.get("reportAddress") or detect_ip()
+    report("complete", "ready at %s" % ip if ip else "VyOS installed, rebooting")
+    os.system("sync")
+    # --force: this driver is a child of live-config.service, whose start job is
+    # still running -- a normal reboot deadlocks waiting for it (verified in VM:
+    # shutdown blocked >1min on "start job is running for live-config"). The
+    # installer has already unmounted and cleaned the target, so an immediate
+    # reboot is safe.
+    os.system("systemctl reboot --force")
+
+
+if __name__ == "__main__":
+    main()
+`;
+}

bastion/src/bastion/tests/asahi.test.ts

@@ -223,3 +223,67 @@ describe("renderFirstbootUnit", () => {
     expect(unit).toContain("ConditionPathExists=!/etc/lab-lvm-setup-done");
   });
 });
+
+// ---------------------------------------------------------------------------
+// v2 layout. Asahi was the THIRD generator and the most drifted: it hardcoded
+// rancher at 20480M under a comment claiming "sizes match install.ks.ts", which
+// uses 122880M. worker1 still runs that 20G image store today.
+// ---------------------------------------------------------------------------
+describe("v2 layout", () => {
+  const roles = ["vanilla", "worker", "infra", "labcontroller"] as const;
+  const p = (role: (typeof roles)[number], schemaVersion?: 1 | 2) =>
+    renderFirstbootScript({
+      hostname: "mac1", role, serverIp: "10.0.0.1", httpPort: 8080,
+      sshKeys: ["ssh-ed25519 AAAA t"], adminUser: "michal", mac: "aa:bb:cc:dd:ee:ff",
+      ...(schemaVersion ? { schemaVersion } : {}),
+    } as any);
+
+  function lvSizes(script: string): Record<string, number> {
+    const out: Record<string, number> = {};
+    for (const m of script.matchAll(/lvcreate -L (\d+)M\s+-n (\w+)\s+labvg/g)) {
+      out[m[2]!] = Number(m[1]);
+    }
+    return out;
+  }
+
+  it("v2 never creates a root LV — the Asahi installer already made a btrfs root", () => {
+    for (const role of roles) {
+      expect(Object.keys(lvSizes(p(role, 2)))).not.toContain("root");
+    }
+  });
+
+  it("v2 sizes rancher at 250G, not the 20G that undersized worker1", () => {
+    const lvs = lvSizes(p("infra", 2));
+    expect(lvs["rancher"]).toBe(256000);
+    expect(lvs["var"]).toBe(49152);
+    expect(lvs["swap"]).toBe(16384);
+  });
+
+  it("v2 gives a worker a rancher LV; v1 gave it only to servers", () => {
+    expect(lvSizes(p("worker", 2))["rancher"]).toBe(256000);
+    expect(lvSizes(p("worker"))["rancher"]).toBeUndefined();
+  });
+
+  it("v2 never uses 100%FREE", () => {
+    for (const role of roles) {
+      expect(p(role, 2)).not.toContain("100%FREE");
+    }
+  });
+
+  it("v1 still uses 100%FREE for longhorn and is otherwise unchanged", () => {
+    const v1 = p("worker");
+    expect(v1).toContain("lvcreate -l 100%FREE -n longhorn");
+    const lvs = lvSizes(v1);
+    expect(lvs["swap"]).toBe(27648);
+    expect(lvs["var"]).toBe(102400);
+  });
+
+  it("v2 tags the volume group so a later run can detect the layout", () => {
+    expect(p("worker", 2)).toContain("vgchange --addtag lab-schema-v2 labvg");
+    expect(p("worker")).not.toContain("--addtag lab-schema-v2");
+  });
+
+  it("reprovision reports the on-disk schema", () => {
+    expect(p("worker", 2)).toMatch(/vgs --noheadings -o vg_tags labvg.*grep -q lab-schema-v2/);
+  });
+});

bastion/src/bastion/tests/auto-k3s.test.ts

@@ -0,0 +1,137 @@
+// Tests for the post-provision k3s auto-install gate.
+//
+// Two independent switches, for two different problems:
+//
+//  1. shouldAutoInstallK3s(role) — a ROLE gate. The post-provision installer
+//     runs a bare `INSTALL_K3S_EXEC=server|agent` with no K3S_URL and no
+//     K3S_TOKEN and knows about exactly one machine. For a node that is its
+//     own cluster that is fine. For a cluster MEMBER it is wrong: servers get
+//     cluster-init semantics and form their own separate cluster (so three
+//     k8s-combo boxes yield three clusters), and agents cannot start at all
+//     ("Job for k3s-agent.service failed").
+//
+//  2. config.autoK3s — a global off switch, needed because the prerequisite
+//     steps stop k3s and delete CNI state several minutes after an install
+//     lands, which lands on top of a test forming its own cluster.
+
+import { describe, it, expect } from "vitest";
+import { shouldAutoInstallK3s } from "../src/services/post-provision.js";
+import { loadConfig } from "../src/config.js";
+import { SUPPORTED_ROLES, isK3sServer, roleInfo } from "@lab/shared";
+import type { Role } from "@lab/shared";
+
+describe("shouldAutoInstallK3s", () => {
+  // These join a cluster someone else owns, so they cannot be auto-installed
+  // without the server URL and token that post-provision does not have.
+  const CLUSTER_MEMBERS: Role[] = ["k8s-combo", "k8s-controller", "k8s-worker", "worker"];
+
+  // These are each their own cluster (or no k3s at all), so a bare install is
+  // a correct install.
+  const SELF_CONTAINED: Role[] = ["standalone-cluster", "infra", "labcontroller", "vanilla"];
+
+  it.each(CLUSTER_MEMBERS)("refuses to auto-install %s", (role) => {
+    expect(shouldAutoInstallK3s(role)).toBe(false);
+  });
+
+  it.each(SELF_CONTAINED)("still auto-installs %s", (role) => {
+    expect(shouldAutoInstallK3s(role)).toBe(true);
+  });
+
+  it("covers every supported role — a new role must make a deliberate choice", () => {
+    for (const role of SUPPORTED_ROLES) {
+      expect(
+        CLUSTER_MEMBERS.includes(role) || SELF_CONTAINED.includes(role),
+        `role ${role} is in neither list; decide whether post-provision can install it`,
+      ).toBe(true);
+      expect(typeof shouldAutoInstallK3s(role)).toBe("boolean");
+    }
+  });
+
+  it("refuses every k3s AGENT role — an agent with nothing to join cannot start", () => {
+    for (const role of SUPPORTED_ROLES) {
+      if (roleInfo(role).k3sMode === "agent") {
+        expect(shouldAutoInstallK3s(role), `${role} is an agent`).toBe(false);
+      }
+    }
+  });
+
+  it("does not refuse purely on the basis of being a server", () => {
+    // Guard against the gate degenerating into isK3sServer(): labcontroller
+    // and infra are servers AND auto-installable.
+    const servers = SUPPORTED_ROLES.filter(isK3sServer);
+    expect(servers.some((r) => shouldAutoInstallK3s(r))).toBe(true);
+    expect(servers.some((r) => !shouldAutoInstallK3s(r))).toBe(true);
+  });
+});
+
+describe("config.autoK3s", () => {
+  it("defaults to enabled", () => {
+    const config = loadConfig({});
+    expect(config.autoK3s).toBe(true);
+  });
+
+  it("is disabled by an explicit override", () => {
+    const config = loadConfig({ autoK3s: false });
+    expect(config.autoK3s).toBe(false);
+  });
+
+  it("an explicit true override wins over nothing", () => {
+    const config = loadConfig({ autoK3s: true });
+    expect(config.autoK3s).toBe(true);
+  });
+
+  it("AUTO_K3S=false in the environment disables it", () => {
+    const prev = process.env["AUTO_K3S"];
+    process.env["AUTO_K3S"] = "false";
+    try {
+      expect(loadConfig({}).autoK3s).toBe(false);
+    } finally {
+      if (prev === undefined) delete process.env["AUTO_K3S"]; else process.env["AUTO_K3S"] = prev;
+    }
+  });
+
+  it("only the exact string 'false' disables it", () => {
+    const prev = process.env["AUTO_K3S"];
+    process.env["AUTO_K3S"] = "0";
+    try {
+      expect(loadConfig({}).autoK3s).toBe(true);
+    } finally {
+      if (prev === undefined) delete process.env["AUTO_K3S"]; else process.env["AUTO_K3S"] = prev;
+    }
+  });
+});
+
+describe("kickstart packages", () => {
+  // etcd membership recovery is `etcdctl member list/remove`, and `k3s etcdctl`
+  // is NOT a command (verified on a live node: "No help topic for 'etcdctl'").
+  // The binary ships in the `etcd` rpm. The recovery card is written for a
+  // situation with no internet, so telling the operator to dnf-install it there
+  // is not a plan.
+  it("installs etcd on k3s servers and nowhere else", async () => {
+    const { renderInstallKickstart } = await import("../src/templates/install.ks.js");
+    const render = (role: Role): string =>
+      renderInstallKickstart({
+        hostname: "test-node", disk: "nvme0n1", role, domain: "ad.itaz.eu",
+        fedoraVersion: "43", timezone: "Europe/Warsaw", locale: "en_US.UTF-8",
+        serverIp: "192.168.8.11", httpPort: 8080, syslogPort: 5514,
+        sshKeys: ["ssh-ed25519 AAAA test"], adminUser: "michal",
+        ...(role === "vanilla" || role === "worker" || role === "infra" || role === "labcontroller"
+          ? {} : { schemaVersion: 2 as const }),
+      });
+    // Scope to %packages: "etcd" appears in paths and comments elsewhere, and a
+    // whole-document match would pass for the wrong reason.
+    const pkgSection = (ks: string): string => {
+      const start = ks.indexOf("%packages");
+      return ks.slice(start, ks.indexOf("%end", start));
+    };
+
+    for (const role of ["infra", "labcontroller", "k8s-combo", "k8s-controller", "standalone-cluster"] as const) {
+      expect(pkgSection(render(role)), `${role} is an etcd member and needs etcdctl offline`)
+        .toMatch(/^etcd$/m);
+    }
+    for (const role of ["vanilla", "worker", "k8s-worker"] as const) {
+      expect(pkgSection(render(role)), `${role} runs no etcd`)
+        .not.toMatch(/^etcd$/m);
+    }
+  });
+});

bastion/src/bastion/tests/dispatch.test.ts

@@ -22,6 +22,8 @@ function createTestConfig(testDir: string): BastionConfig {
     dhcpRangeEnd: "",
     ubuntuVersion: "26.04",
     ubuntuMirror: "https://releases.ubuntu.com/26.04",
+    vyosIsoUrl: "https://downloads.vyos.io/rolling/current/generic/vyos-rolling-latest.iso",
+    vyosDefaultPassword: "vyos",
     iface: "eth0",
     serverIp: "10.0.0.1",
     network: "10.0.0.0",

bastion/src/bastion/tests/kickstart-layout.test.ts

@@ -0,0 +1,186 @@
+import { describe, it, expect } from "vitest";
+import { renderInstallKickstart } from "../src/templates/install.ks.js";
+import type { InstallKickstartParams } from "../src/templates/install.ks.js";
+import { SUPPORTED_ROLES, LEGACY_ROLES, v2Layout, type Role } from "@lab/shared";
+
+/**
+ * Partition-layout tests.
+ *
+ * The existing ksvalidator test only sees the MAIN kickstart body. The actual
+ * layout is written by %pre into /tmp/part.ks inside a heredoc, so until now
+ * nothing validated it at all -- a broken layout rendered, shipped and was only
+ * discovered on the installed machine.
+ *
+ * These parse the heredocs. Following tests/ubuntu-autoinstall.test.ts:
+ * "A `toContain` assertion would not have caught that; only parsing does."
+ */
+
+function baseParams(over: Partial<InstallKickstartParams> = {}): InstallKickstartParams {
+  return {
+    hostname: "test-node", disk: "nvme0n1", role: "worker", domain: "ad.itaz.eu",
+    fedoraVersion: "43", timezone: "Europe/Warsaw", locale: "en_US.UTF-8",
+    serverIp: "192.168.8.11", httpPort: 8080, syslogPort: 5514,
+    sshKeys: ["ssh-ed25519 AAAA test"], adminUser: "michal",
+    ...over,
+  };
+}
+
+/** Pull every `cat >[>] /tmp/part.ks << PARTEOF ... PARTEOF` body out of the script. */
+function partKsBlocks(ks: string): string[] {
+  const out: string[] = [];
+  const re = /cat >>?\s*\/tmp\/part\.ks\s*<<\s*PARTEOF\n([\s\S]*?)\nPARTEOF/g;
+  let m: RegExpExecArray | null;
+  while ((m = re.exec(ks)) !== null) out.push(m[1]!);
+  return out;
+}
+
+interface Logvol { mount: string; name: string; fs: string; size?: number; useExisting: boolean }
+
+function parseLogvols(block: string): Logvol[] {
+  return block.split("\n")
+    .filter((l) => l.trim().startsWith("logvol "))
+    .map((line) => {
+      const parts = line.trim().split(/\s+/);
+      const get = (flag: string) => {
+        const p = parts.find((x) => x.startsWith(`${flag}=`));
+        return p ? p.slice(flag.length + 1) : undefined;
+      };
+      return {
+        mount: parts[1]!,
+        name: get("--name") ?? "",
+        fs: get("--fstype") ?? "",
+        ...(get("--size") ? { size: Number(get("--size")) } : {}),
+        useExisting: parts.includes("--useexisting"),
+      };
+    });
+}
+
+/** The fresh-install block is the one that also creates the PV. */
+function freshBlock(ks: string): string {
+  const b = partKsBlocks(ks).find((x) => x.includes("part pv.01"));
+  if (!b) throw new Error("no fresh-install part.ks block found");
+  return b;
+}
+
+describe("v1 is frozen — existing machines must reinstall identically", () => {
+  it.each(LEGACY_ROLES)("%s v1 fresh layout is unchanged", (role) => {
+    const lvs = parseLogvols(freshBlock(renderInstallKickstart(baseParams({ role }))));
+    const byName = Object.fromEntries(lvs.map((l) => [l.name, l]));
+
+    // The exact v1 sizes, in MiB. If one of these changes, every existing box
+    // in the fleet would be re-laid-out on its next reinstall.
+    expect(byName["swap"]!.size).toBe(27648);
+    expect(byName["root"]!.size).toBe(33792);
+    expect(byName["var"]!.size).toBe(102400);
+    expect(byName["varlog"]!.size).toBe(10240);
+    expect(byName["home"]!.size).toBe(10240);
+    expect(byName["srv"]!.size).toBe(20480);
+  });
+
+  it("v1 gives rancher to infra and worker only — the historical bug, preserved", () => {
+    const has = (r: Role) => freshBlock(renderInstallKickstart(baseParams({ role: r }))).includes("--name=rancher");
+    expect(has("infra")).toBe(true);
+    expect(has("worker")).toBe(true);
+    expect(has("labcontroller")).toBe(false); // the bug, deliberately kept in v1
+    expect(has("vanilla")).toBe(false);
+  });
+
+  it("v1 is the default — no caller has to opt out", () => {
+    const ks = renderInstallKickstart(baseParams({ role: "worker" }));
+    expect(parseLogvols(freshBlock(ks)).find((l) => l.name === "root")!.size).toBe(33792);
+  });
+});
+
+describe("v2-only roles have no v1 layout", () => {
+  it.each(["standalone-cluster", "k8s-controller", "k8s-worker", "k8s-combo"] as const)(
+    "%s defaults to v2 when no schema is given", (role) => {
+      const lvs = parseLogvols(freshBlock(renderInstallKickstart(baseParams({ role }))));
+      expect(lvs.find((l) => l.name === "root")!.size).toBe(32768);
+      expect(lvs.some((l) => l.name === "rancher")).toBe(true);
+    });
+
+  it.each(["standalone-cluster", "k8s-controller", "k8s-worker", "k8s-combo"] as const)(
+    "%s throws on an EXPLICIT v1 rather than silently dropping its volumes", (role) => {
+      expect(() => renderInstallKickstart(baseParams({ role, schemaVersion: 1 }))).toThrow(/no v1 disk layout/);
+    });
+});
+
+describe("v2 renders the shared layout", () => {
+  it.each(SUPPORTED_ROLES)("%s v2 matches v2Layout() exactly", (role) => {
+    const ks = renderInstallKickstart(baseParams({ role, schemaVersion: 2 }));
+    const rendered = parseLogvols(freshBlock(ks));
+    const expected = v2Layout(role);
+
+    expect(rendered.map((l) => l.name).sort()).toEqual(expected.map((l) => l.name).sort());
+    for (const spec of expected) {
+      const got = rendered.find((l) => l.name === spec.name)!;
+      expect(got.size).toBe(spec.sizeMib);
+      expect(got.fs).toBe(spec.fs);
+    }
+  });
+
+  it("v2 gives labcontroller a rancher LV — the v1 bug, fixed", () => {
+    const ks = renderInstallKickstart(baseParams({ role: "labcontroller", schemaVersion: 2 }));
+    const rancher = parseLogvols(freshBlock(ks)).find((l) => l.name === "rancher");
+    expect(rancher).toBeDefined();
+    expect(rancher!.size).toBe(256000);
+  });
+
+  it("v2 never uses --grow", () => {
+    for (const role of SUPPORTED_ROLES) {
+      const block = freshBlock(renderInstallKickstart(baseParams({ role, schemaVersion: 2 })));
+      expect(block).not.toMatch(/logvol[^\n]*--grow/);
+    }
+  });
+
+  it("v2 uses the larger EFI and /boot partitions", () => {
+    const block = freshBlock(renderInstallKickstart(baseParams({ role: "worker", schemaVersion: 2 })));
+    expect(block).toMatch(/part \/boot\/efi --fstype=efi --size=1024/);
+    expect(block).toMatch(/part \/boot --fstype=ext4 --size=4096/);
+  });
+});
+
+describe("reprovision protects existing data", () => {
+  it("reprovision reads the on-disk schema instead of trusting the caller", () => {
+    const ks = renderInstallKickstart(baseParams({ role: "worker", schemaVersion: 2 }));
+    expect(ks).toContain("DISK_SCHEMA=v1");
+    expect(ks).toMatch(/vgs --noheadings -o vg_tags .*grep -q lab-schema-v2/);
+  });
+
+  it("the reprovision branch never wipes the disk", () => {
+    const ks = renderInstallKickstart(baseParams({ role: "worker" }));
+    const repro = partKsBlocks(ks).find((b) => b.includes("--useexisting"))!;
+    expect(repro).toContain("clearpart --none");
+    expect(repro).not.toContain("clearpart --all");
+  });
+
+  it("preserved data volumes are never reformatted", () => {
+    const ks = renderInstallKickstart(baseParams({ role: "worker" }));
+    for (const lv of ["longhorn", "rancher", "home", "srv"]) {
+      const re = new RegExp(`--name=${lv} --useexisting --noformat`);
+      expect(ks).toMatch(re);
+    }
+  });
+});
+
+describe("the %pre block survives the /ks-auto extractor", () => {
+  it("matches the same regex ks-auto uses to re-run %pre", () => {
+    // routes/kickstart.ts re-extracts %pre with python:
+    //   re.findall(r'%pre[^\n]*\n(.*?)%end', text, re.DOTALL)
+    // A change that breaks this breaks ISO installs silently.
+    for (const role of SUPPORTED_ROLES) {
+      // v2-only roles throw at v1 by design, so only render the combinations
+      // that are actually legal.
+      // An explicit v1 is illegal for a v2-only role; everything else renders.
+      const versions = (LEGACY_ROLES as readonly string[]).includes(role)
+        ? ([1, 2] as const)
+        : ([2] as const);
+      for (const schemaVersion of versions) {
+        const ks = renderInstallKickstart(baseParams({ role, schemaVersion }));
+        const blocks = [...ks.matchAll(/%pre[^\n]*\n([\s\S]*?)%end/g)];
+        expect(blocks.length).toBeGreaterThan(0);
+        expect(blocks[0]![1]).toContain("/tmp/part.ks");
+      }
+    }
+  });
+});

bastion/src/bastion/tests/kickstart.test.ts

@@ -96,9 +96,9 @@ describe("renderInstallKickstart", () => {
     expect(ks).toContain("/api/progress");
   });
 
-  it("infra role has /var/lib/rancher partition", () => {
+  it("infra role has 120G /var/lib/rancher partition", () => {
     const ks = renderInstallKickstart(baseParams({ role: "infra" }));
-    expect(ks).toContain("logvol /var/lib/rancher --vgname=labvg --name=rancher --fstype=xfs --size=20480");
+    expect(ks).toContain("logvol /var/lib/rancher --vgname=labvg --name=rancher --fstype=xfs --size=122880");
   });
 
   it("infra role has k3s install", () => {
@@ -106,10 +106,14 @@ describe("renderInstallKickstart", () => {
     expect(ks).toContain("curl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_START=true sh -");
   });
 
-  it("worker role does NOT have /var/lib/rancher partition in fresh install", () => {
+  it("worker role has 120G /var/lib/rancher partition (imageFs must be sized before longhorn --grow)", () => {
     const ks = renderInstallKickstart(baseParams({ role: "worker" }));
-    // Worker should not have the fresh-install rancher partition line
-    expect(ks).not.toContain("logvol /var/lib/rancher --vgname=labvg --name=rancher --fstype=xfs --size=20480");
+    expect(ks).toContain("logvol /var/lib/rancher --vgname=labvg --name=rancher --fstype=xfs --size=122880");
+  });
+
+  it("vanilla role does NOT have /var/lib/rancher partition in fresh install", () => {
+    const ks = renderInstallKickstart(baseParams({ role: "vanilla" }));
+    expect(ks).not.toContain("--name=rancher --fstype=xfs");
   });
 
   it("worker role does NOT have k3s install", () => {

bastion/src/bastion/tests/layout-consistency.test.ts

@@ -0,0 +1,105 @@
+import { describe, it, expect } from "vitest";
+import { execFileSync } from "node:child_process";
+import { writeFileSync } from "node:fs";
+import { renderInstallKickstart } from "../src/templates/install.ks.js";
+import { renderUbuntuAutoinstall } from "../src/templates/ubuntu-autoinstall.js";
+import { renderFirstbootScript } from "../src/templates/asahi-firstboot.sh.js";
+import { v2Layout, SUPPORTED_ROLES, type Role } from "@lab/shared";
+
+/**
+ * THE test this whole refactor exists for.
+ *
+ * Three generators partition machines. Before the shared layout they disagreed,
+ * and nothing noticed:
+ *
+ *   install.ks.ts       rancher 122880M
+ *   ubuntu-autoinstall  rancher  20480M
+ *   asahi-firstboot     rancher  20480M   <- under a comment claiming it
+ *                                            matched install.ks.ts
+ *
+ * That 100 GB gap is why worker1 runs an image store too small for one large
+ * container image. This test fails the moment any generator drifts again.
+ */
+
+const KS = (role: Role) => renderInstallKickstart({
+  hostname: "n", disk: "nvme0n1", role, domain: "d", fedoraVersion: "43",
+  timezone: "UTC", locale: "en_US.UTF-8", serverIp: "10.0.0.1", httpPort: 8080,
+  syslogPort: 5514, sshKeys: [], adminUser: "a", schemaVersion: 2,
+});
+
+const UB = (role: Role) => renderUbuntuAutoinstall({
+  hostname: "n", disk: "/dev/sda", role, domain: "d", ubuntuVersion: "24.04",
+  timezone: "UTC", locale: "en_US.UTF-8", serverIp: "10.0.0.1", httpPort: 8080,
+  sshKeys: [], adminUser: "a", schemaVersion: 2,
+} as never);
+
+const AS = (role: Role) => renderFirstbootScript({
+  hostname: "n", role, serverIp: "10.0.0.1", httpPort: 8080,
+  sshKeys: [], adminUser: "a", mac: "aa:bb:cc:dd:ee:ff", schemaVersion: 2,
+} as never);
+
+/** MiB per LV name, from the Fedora kickstart's part.ks heredoc. */
+function fedoraSizes(role: Role): Record<string, number> {
+  const ks = KS(role);
+  const block = [...ks.matchAll(/cat >>?\s*\/tmp\/part\.ks\s*<<\s*PARTEOF\n([\s\S]*?)\nPARTEOF/g)]
+    .map((m) => m[1]!).find((b) => b.includes("part pv.01"))!;
+  const out: Record<string, number> = {};
+  for (const line of block.split("\n").filter((l) => l.trim().startsWith("logvol "))) {
+    const name = /--name=(\S+)/.exec(line)?.[1];
+    const size = /--size=(\d+)/.exec(line)?.[1];
+    if (name && size) out[name] = Number(size);
+  }
+  return out;
+}
+
+/** MiB per LV name, from the Ubuntu curtin document (declared in GiB). */
+function ubuntuSizes(role: Role): Record<string, number> {
+  const tmp = `/tmp/consistency-${role}.yaml`;
+  writeFileSync(tmp, UB(role));
+  const json = execFileSync("python3",
+    ["-c", "import sys,yaml,json; json.dump(yaml.safe_load(open(sys.argv[1])), sys.stdout)", tmp],
+    { encoding: "utf-8" });
+  const cfg = JSON.parse(json).autoinstall.storage.config as Array<Record<string, string>>;
+  return Object.fromEntries(
+    cfg.filter((c) => c["type"] === "lvm_partition")
+       .map((c) => [c["name"]!, Number(String(c["size"]).replace("G", "")) * 1024]),
+  );
+}
+
+/** MiB per LV name, from the Asahi firstboot lvcreate calls. */
+function asahiSizes(role: Role): Record<string, number> {
+  const out: Record<string, number> = {};
+  for (const m of AS(role).matchAll(/lvcreate -L (\d+)M\s+-n (\w+)\s+labvg/g)) out[m[2]!] = Number(m[1]);
+  return out;
+}
+
+describe("all three generators render the same v2 layout", () => {
+  it.each(SUPPORTED_ROLES)("%s: Fedora matches the shared definition", (role) => {
+    const expected = Object.fromEntries(v2Layout(role).map((lv) => [lv.name, lv.sizeMib]));
+    expect(fedoraSizes(role)).toEqual(expected);
+  });
+
+  it.each(SUPPORTED_ROLES)("%s: Ubuntu matches the shared definition", (role) => {
+    const expected = Object.fromEntries(v2Layout(role).map((lv) => [lv.name, lv.sizeMib]));
+    expect(ubuntuSizes(role)).toEqual(expected);
+  });
+
+  it.each(SUPPORTED_ROLES)("%s: Asahi matches, minus the vendor-created root", (role) => {
+    const expected = Object.fromEntries(
+      v2Layout(role, { omitRoot: true }).map((lv) => [lv.name, lv.sizeMib]),
+    );
+    expect(asahiSizes(role)).toEqual(expected);
+  });
+
+  it.each(SUPPORTED_ROLES)("%s: Fedora and Ubuntu are byte-identical in sizing", (role) => {
+    expect(ubuntuSizes(role)).toEqual(fedoraSizes(role));
+  });
+
+  it("rancher is 250G everywhere — the specific drift that bit worker1", () => {
+    for (const role of ["worker", "infra", "labcontroller"] as const) {
+      expect(fedoraSizes(role)["rancher"]).toBe(256000);
+      expect(ubuntuSizes(role)["rancher"]).toBe(256000);
+      expect(asahiSizes(role)["rancher"]).toBe(256000);
+    }
+  });
+});

bastion/src/bastion/tests/schema-version.test.ts

@@ -0,0 +1,166 @@
+import { describe, it, expect, beforeEach, afterEach } from "vitest";
+import { mkdirSync, rmSync } from "node:fs";
+import { join } from "node:path";
+import { tmpdir } from "node:os";
+import type { BastionConfig } from "@lab/shared";
+import { createApp } from "../src/server.js";
+import type { FastifyInstance } from "fastify";
+import type { StateManager } from "../src/services/state.js";
+import type { InstallLogBuffer } from "../src/services/install-log.js";
+
+function createTestConfig(testDir: string): BastionConfig {
+  return {
+    fedoraVersion: "43",
+    arch: "x86_64",
+    httpPort: 0,
+    timezone: "Europe/London",
+    locale: "en_GB.UTF-8",
+    bastionDir: testDir,
+    domain: "test.local",
+    dhcpMode: "proxy",
+    dhcpRangeStart: "",
+    dhcpRangeEnd: "",
+    ubuntuVersion: "26.04",
+    ubuntuMirror: "https://releases.ubuntu.com/26.04",
+    vyosIsoUrl: "https://downloads.vyos.io/rolling/current/generic/vyos-rolling-latest.iso",
+    vyosDefaultPassword: "vyos",
+    iface: "eth0",
+    serverIp: "10.0.0.1",
+    network: "10.0.0.0",
+    gateway: "10.0.0.1",
+    sshKeys: ["ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAITEST test@test"],
+    adminUser: "testadmin",
+    syslogPort: 15514,
+    skipDnsmasq: true,
+    skipArtifacts: true,
+    fedoraMirror: "https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os",
+    tftpDir: join(testDir, "tftp"),
+    httpDir: join(testDir, "http"),
+    stateFile: join(testDir, "state.json"),
+  };
+}
+
+describe("schemaVersion plumbing: /api/install -> /ks", () => {
+  let app: FastifyInstance;
+  let testDir: string;
+
+  beforeEach(async () => {
+    testDir = join(tmpdir(), `schema-test-${Date.now()}-${Math.random().toString(36).slice(2)}`);
+    mkdirSync(testDir, { recursive: true });
+    const built = await createApp(createTestConfig(testDir));
+    app = built.app;
+    await app.ready();
+  });
+
+  afterEach(async () => {
+    await app.close();
+    rmSync(testDir, { recursive: true, force: true });
+  });
+
+  const queue = (body: Record<string, unknown>) =>
+    app.inject({ method: "POST", url: "/api/install", payload: { mac: "aa:bb:cc:dd:ee:01", hostname: "n1", ...body } });
+
+  const ks = () => app.inject({ method: "GET", url: "/ks?mac=aa:bb:cc:dd:ee:01" });
+
+  it("defaults to v1 when no schemaVersion is given", async () => {
+    expect((await queue({ role: "worker" })).statusCode).toBe(200);
+    const body = (await ks()).body;
+    expect(body).toContain("--name=root --fstype=xfs --size=33792");
+  });
+
+  it("schemaVersion 2 reaches the rendered kickstart", async () => {
+    expect((await queue({ role: "worker", schemaVersion: 2 })).statusCode).toBe(200);
+    const body = (await ks()).body;
+    expect(body).toContain("--name=root --fstype=xfs --size=32768");
+    expect(body).toContain("--name=rancher --fstype=xfs --size=256000");
+  });
+
+  it("v2 gives labcontroller a rancher LV through the full path", async () => {
+    expect((await queue({ role: "labcontroller", schemaVersion: 2 })).statusCode).toBe(200);
+    expect((await ks()).body).toContain("--name=rancher");
+  });
+
+  it("rejects an unknown schemaVersion instead of silently installing v1", async () => {
+    const res = await queue({ role: "worker", schemaVersion: 3 });
+    expect(res.statusCode).toBe(400);
+    expect(res.json().error).toMatch(/invalid schemaVersion/);
+  });
+
+  it("still rejects an unknown role", async () => {
+    expect((await queue({ role: "nope" })).statusCode).toBe(400);
+  });
+});
+
+describe("v2-only roles cannot be installed at v1", () => {
+  let app2: FastifyInstance;
+  let dir2: string;
+
+  beforeEach(async () => {
+    dir2 = join(tmpdir(), `schema-v2only-${Date.now()}-${Math.random().toString(36).slice(2)}`);
+    mkdirSync(dir2, { recursive: true });
+    const built = await createApp(createTestConfig(dir2));
+    app2 = built.app;
+    await app2.ready();
+  });
+  afterEach(async () => { await app2.close(); rmSync(dir2, { recursive: true, force: true }); });
+
+  const q = (body: Record<string, unknown>) =>
+    app2.inject({ method: "POST", url: "/api/install", payload: { mac: "aa:bb:cc:dd:ee:02", hostname: "n2", ...body } });
+
+  /**
+   * Omitting the schema must WORK for a v2-only role: a reinstall never passes
+   * one (the lab-schema-v2 tag on the disk decides), and the first version of
+   * this guard rejected that, making it impossible to reinstall a v2-only
+   * machine at all. Found by the reinstall integration test.
+   */
+  it.each(["standalone-cluster", "k8s-controller", "k8s-worker", "k8s-combo"])(
+    "%s is accepted without a schemaVersion and defaults to v2", async (role) => {
+      const res = await q({ role });
+      expect(res.statusCode).toBe(200);
+      const ks = (await app2.inject({ method: "GET", url: "/ks?mac=aa:bb:cc:dd:ee:02" })).body;
+      expect(ks).toContain("--name=root --fstype=xfs --size=32768");
+    });
+
+  it.each(["standalone-cluster", "k8s-controller", "k8s-worker", "k8s-combo"])(
+    "%s is rejected on an EXPLICIT schemaVersion 1 — that is a contradiction", async (role) => {
+      const res = await q({ role, schemaVersion: 1 });
+      expect(res.statusCode).toBe(400);
+      expect(res.json().error).toMatch(/requires schemaVersion 2/);
+    });
+
+  it.each(["standalone-cluster", "k8s-controller", "k8s-worker", "k8s-combo"])(
+    "%s is accepted with schemaVersion 2", async (role) => {
+      expect((await q({ role, schemaVersion: 2 })).statusCode).toBe(200);
+    });
+
+  it("legacy roles are still accepted without a schemaVersion", async () => {
+    for (const role of ["vanilla", "worker", "infra", "labcontroller"]) {
+      expect((await q({ role })).statusCode).toBe(200);
+    }
+  });
+
+  /**
+   * Assert on the FRESH-INSTALL part.ks block specifically. The whole kickstart
+   * also contains the reprovision branch, whose shell emits `--name=longhorn`
+   * inside a `PRESERVE_LONGHORN` guard -- so a naive whole-file `not.toContain`
+   * matches that and fails for reasons that have nothing to do with the layout.
+   */
+  function freshBlockOf(ks: string): string {
+    const blocks = [...ks.matchAll(/cat >>?\s*\/tmp\/part\.ks\s*<<\s*PARTEOF\n([\s\S]*?)\nPARTEOF/g)]
+      .map((m) => m[1]!);
+    const fresh = blocks.find((b) => b.includes("part pv.01"));
+    if (!fresh) throw new Error("no fresh-install block in rendered kickstart");
+    return fresh;
+  }
+
+  it("k8s-controller gets no longhorn LV; k8s-combo does", async () => {
+    await q({ role: "k8s-controller", schemaVersion: 2 });
+    const ctrl = freshBlockOf((await app2.inject({ method: "GET", url: "/ks?mac=aa:bb:cc:dd:ee:02" })).body);
+    expect(ctrl).toContain("--name=rancher");
+    expect(ctrl).not.toContain("--name=longhorn");
+
+    await q({ role: "k8s-combo", schemaVersion: 2 });
+    const combo = freshBlockOf((await app2.inject({ method: "GET", url: "/ks?mac=aa:bb:cc:dd:ee:02" })).body);
+    expect(combo).toContain("--name=longhorn --fstype=xfs --size=307200");
+  });
+});

bastion/src/bastion/tests/ubuntu-autoinstall.test.ts

@@ -0,0 +1,184 @@
+// The Ubuntu autoinstall document must be valid YAML for EVERY role.
+//
+// There was no test here, and the template shipped a document that did not
+// parse: the longhorn/rancher LVM entries were indented 8 spaces while their
+// siblings in storage.config sit at 6, giving "expected <block end>, but found
+// '-'". Every role that gets a longhorn volume -- which is the worker shape --
+// rendered an uninstallable document. A `toContain` assertion would not have
+// caught that; only parsing does.
+//
+// Parsed with python3's yaml rather than a new npm dependency, mirroring how
+// kickstart.test.ts shells out to `ksvalidator`: the point is to check the
+// artefact with a real parser, not to grow the dependency tree.
+
+import { describe, it, expect } from "vitest";
+import { execFileSync } from "node:child_process";
+import { writeFileSync, unlinkSync } from "node:fs";
+import { renderUbuntuAutoinstall } from "../src/templates/ubuntu-autoinstall.js";
+
+const base = {
+  hostname: "n6",
+  disk: "/dev/sda",
+  domain: "ad.itaz.eu",
+  ubuntuVersion: "24.04",
+  timezone: "Europe/London",
+  locale: "en_GB.UTF-8",
+  serverIp: "10.0.0.1",
+  httpPort: 8080,
+  sshKeys: ["ssh-ed25519 AAAAtest test@lab"],
+  adminUser: "root",
+};
+
+/** Parse with python3's yaml and return the document as JSON. */
+function parseYaml(text: string, label: string): Record<string, any> {
+  const tmp = `/tmp/autoinstall-test-${label}.yaml`;
+  writeFileSync(tmp, text);
+  try {
+    const out = execFileSync(
+      "python3",
+      ["-c", "import sys,yaml,json; json.dump(yaml.safe_load(open(sys.argv[1])), sys.stdout)", tmp],
+      { encoding: "utf-8" },
+    );
+    return JSON.parse(out);
+  } catch (err: unknown) {
+    const msg = err instanceof Error ? (err as { stderr?: string }).stderr ?? err.message : String(err);
+    throw new Error(`autoinstall YAML did not parse for ${label}: ${msg}`);
+  } finally {
+    try { unlinkSync(tmp); } catch { /* ignore */ }
+  }
+}
+
+describe("renderUbuntuAutoinstall", () => {
+  for (const role of ["vanilla", "worker", "infra"]) {
+    it(`renders parseable YAML for role=${role}`, () => {
+      const doc = parseYaml(renderUbuntuAutoinstall({ ...base, role }), role);
+      expect(doc.autoinstall).toBeDefined();
+      expect(doc.autoinstall.version).toBe(1);
+      // storage.config must be a flat list; the indentation bug produced a
+      // nested map here, which is how it went unnoticed.
+      expect(Array.isArray(doc.autoinstall.storage.config)).toBe(true);
+    });
+  }
+
+  it("gives the longhorn role its volume as a sibling entry, not a nested map", () => {
+    const doc = parseYaml(renderUbuntuAutoinstall({ ...base, role: "worker" }), "longhorn");
+    const ids = doc.autoinstall.storage.config.map((e: { id: string }) => e.id);
+    expect(ids).toContain("lv-longhorn");
+    expect(ids).toContain("mount-longhorn");
+  });
+
+  it("sets EUI-64 link-local so MAC-keyed DHCPv6 reservations can match", () => {
+    const doc = parseYaml(renderUbuntuAutoinstall({ ...base, role: "worker" }), "eui64");
+    const late = (doc.autoinstall["late-commands"] as string[]).join("\n");
+    expect(late).toContain("10-ipv6-eui64.conf");
+    expect(late).toContain("ipv6.addr-gen-mode=eui64");
+  });
+
+  it("requests both address families on the primary NIC", () => {
+    const doc = parseYaml(renderUbuntuAutoinstall({ ...base, role: "worker" }), "net");
+    const eth = doc.autoinstall.network.ethernets.primary;
+    expect(eth.dhcp4).toBe(true);
+    // Without this a node provisioned into a dual-stack cluster comes up with
+    // no IPv6 and joins as an IPv4-only member.
+    expect(eth.dhcp6).toBe(true);
+    // The install must not block waiting for a v6 lease that may never come.
+    expect(eth.optional).toBe(true);
+  });
+});
+
+// ---------------------------------------------------------------------------
+// v2 layout. Ubuntu previously sized rancher at 20G while the Fedora kickstart
+// used 120G, from a template whose comment claimed the sizes matched. Both now
+// render from shared/src/types/layout.ts, so this checks the rendering, and
+// layout.test.ts checks the definition.
+// ---------------------------------------------------------------------------
+describe("v2 storage layout", () => {
+  const roles = ["vanilla", "worker", "infra", "labcontroller"] as const;
+
+  function lvmPartitions(doc: Record<string, any>): Record<string, string> {
+    const cfg = doc["autoinstall"]["storage"]["config"] as Array<Record<string, any>>;
+    return Object.fromEntries(
+      cfg.filter((c) => c["type"] === "lvm_partition").map((c) => [c["name"], c["size"]]),
+    );
+  }
+
+  it.each(roles)("%s renders valid YAML at schemaVersion 2", (role) => {
+    const doc = parseYaml(
+      renderUbuntuAutoinstall({ ...base, role, schemaVersion: 2 } as any),
+      `v2-${role}`,
+    );
+    expect(Array.isArray(doc["autoinstall"]["storage"]["config"])).toBe(true);
+  });
+
+  it("v2 sizes match the shared definition, not the old 20G rancher", () => {
+    const doc = parseYaml(
+      renderUbuntuAutoinstall({ ...base, role: "worker", schemaVersion: 2 } as any),
+      "v2-sizes",
+    );
+    const lvs = lvmPartitions(doc);
+    expect(lvs["rancher"]).toBe("250G");   // was 20G here and 120G in kickstart
+    expect(lvs["longhorn"]).toBe("300G");  // was grow
+    expect(lvs["root"]).toBe("32G");
+    expect(lvs["var"]).toBe("48G");
+  });
+
+  it("v2 gives labcontroller a rancher LV; v1 does not", () => {
+    const v2 = lvmPartitions(parseYaml(
+      renderUbuntuAutoinstall({ ...base, role: "labcontroller", schemaVersion: 2 } as any), "v2-lc"));
+    const v1 = lvmPartitions(parseYaml(
+      renderUbuntuAutoinstall({ ...base, role: "labcontroller" } as any), "v1-lc"));
+    expect(v2["rancher"]).toBe("250G");
+    expect(v1["rancher"]).toBeUndefined();
+  });
+
+  it("v1 remains the default and is unchanged", () => {
+    const lvs = lvmPartitions(parseYaml(
+      renderUbuntuAutoinstall({ ...base, role: "worker" } as any), "v1-default"));
+    expect(lvs["root"]).toBe("33G");
+    expect(lvs["var"]).toBe("100G");
+    expect(lvs["swap"]).toBe("27G");
+  });
+
+  it("no v2 volume is unbounded", () => {
+    for (const role of roles) {
+      const lvs = lvmPartitions(parseYaml(
+        renderUbuntuAutoinstall({ ...base, role, schemaVersion: 2 } as any), `v2-grow-${role}`));
+      for (const [name, size] of Object.entries(lvs)) {
+        expect(String(size), `${role}/${name}`).toMatch(/^\d+G$/);
+      }
+    }
+  });
+});
+
+/**
+ * KNOWN GAP, asserted so it cannot be assumed away.
+ *
+ * The Fedora kickstart preserves /home, /srv, longhorn and rancher across a
+ * reinstall (%pre probes for an existing labvg). The Ubuntu autoinstall has no
+ * equivalent: `disk0` is wiped unconditionally, so EVERY Ubuntu reinstall is
+ * destructive.
+ *
+ * These tests pin that behaviour rather than pretend it is fine. If someone
+ * adds curtin `preserve: true` handling, these fail and should be rewritten --
+ * which is the point: the change becomes visible instead of silent.
+ */
+describe("Ubuntu reinstall is destructive (known gap)", () => {
+  it("wipes the disk unconditionally — there is no reprovision branch", () => {
+    const doc = parseYaml(renderUbuntuAutoinstall({ ...base, role: "worker" } as any), "destructive");
+    const cfg = doc["autoinstall"]["storage"]["config"] as Array<Record<string, any>>;
+    const disk = cfg.find((c) => c["type"] === "disk");
+    expect(disk?.["wipe"]).toBe("superblock-recursive");
+  });
+
+  it("no volume is marked preserve — nothing survives a reinstall", () => {
+    const doc = parseYaml(renderUbuntuAutoinstall({ ...base, role: "worker" } as any), "no-preserve");
+    const cfg = doc["autoinstall"]["storage"]["config"] as Array<Record<string, any>>;
+    expect(cfg.some((c) => c["preserve"] === true)).toBe(false);
+  });
+
+  it("the same is true at schemaVersion 2 — v2 did not fix this", () => {
+    const doc = parseYaml(renderUbuntuAutoinstall({ ...base, role: "worker", schemaVersion: 2 } as any), "v2-destructive");
+    const cfg = doc["autoinstall"]["storage"]["config"] as Array<Record<string, any>>;
+    expect(cfg.find((c) => c["type"] === "disk")?.["wipe"]).toBe("superblock-recursive");
+  });
+});

bastion/src/bastion/tests/vyos-bundle.test.ts

@@ -0,0 +1,155 @@
+import { describe, it, expect, vi } from "vitest";
+import type { VyosBundle } from "@lab/shared";
+import { buildVyosConfigSpec } from "../src/templates/vyos-config-spec.js";
+
+/**
+ * A bundle is what makes "one config, two apply paths" true rather than
+ * aspirational: `pulumi up` POSTs the subtree model to a running router, labctl
+ * writes the same model into config.boot during a PXE install. These tests pin
+ * the properties that keep the two honest.
+ */
+const bundle: VyosBundle = {
+  sets: [
+    { path: ["system", "host-name"], value: "vyos001" },
+    { path: ["interfaces", "bonding", "bond0", "address"], value: "192.168.1.252/24" },
+    { path: ["interfaces", "bonding", "bond0", "member", "interface"], value: "eth1", replace: false },
+    { path: ["interfaces", "bonding", "bond0", "vif", "53", "disable"] },
+    { path: ["interfaces", "pppoe", "pppoe0", "authentication", "password"], value: "@secret:pppoePassword" },
+    { path: ["interfaces", "pppoe", "pppoe0", "mtu"], value: "1492" },
+  ],
+  tags: [["interfaces", "bonding", "bond0"], ["interfaces", "ethernet"]],
+};
+
+const build = (hostname: string, extra: Record<string, unknown> = {}) =>
+  buildVyosConfigSpec({
+    hostname,
+    spec: { bundle, ...extra },
+    defaultPassword: "changeme",
+  });
+
+describe("vyos config spec from a Pulumi bundle", () => {
+  it("applies non-secret nodes verbatim, preserving valuelessness and replace:false", () => {
+    const spec = build("vyos001");
+
+    expect(spec.sets).toContainEqual({
+      path: ["interfaces", "bonding", "bond0", "address"],
+      value: "192.168.1.252/24",
+    });
+    // A multi-value node must keep replace:false or the second bond member
+    // overwrites the first.
+    expect(spec.sets).toContainEqual({
+      path: ["interfaces", "bonding", "bond0", "member", "interface"],
+      value: "eth1",
+      replace: false,
+    });
+    // A valueless node must not acquire a value on the way through.
+    expect(spec.sets).toContainEqual({
+      path: ["interfaces", "bonding", "bond0", "vif", "53", "disable"],
+    });
+    expect(spec.tags).toEqual(bundle.tags);
+  });
+
+  it("drops secret-valued nodes instead of installing the sentinel text", () => {
+    const warn = vi.spyOn(console, "warn").mockImplementation(() => {});
+    const spec = build("vyos001");
+
+    const values = spec.sets.map((s) => s.value ?? "");
+    expect(values.some((v) => v.startsWith("@secret:"))).toBe(false);
+    expect(spec.sets.some((s) => s.path.includes("authentication"))).toBe(false);
+    // Silently dropping the WAN credential would leave someone debugging a dead
+    // PPPoE link, so it has to be said out loud.
+    expect(warn).toHaveBeenCalledWith(expect.stringContaining("pulumi up"));
+    warn.mockRestore();
+  });
+
+  it("forces the hostname the install was asked for, not the bundle's", () => {
+    // The bundle is exported from one router and reused for its peer; taking the
+    // hostname from it would put two vyos001s on the network.
+    const spec = build("vyos002");
+    const hostnames = spec.sets.filter(
+      (s) => s.path.length === 2 && s.path[0] === "system" && s.path[1] === "host-name",
+    );
+    expect(hostnames).toEqual([{ path: ["system", "host-name"], value: "vyos002" }]);
+  });
+
+  it("still honours installer inputs, which are not router config", () => {
+    const spec = buildVyosConfigSpec({
+      hostname: "vyos001",
+      spec: { bundle, password: "s3cret", freshConfig: true },
+      defaultPassword: "changeme",
+      disk: "nvme0n1",
+    });
+    expect(spec.password).toBe("s3cret");
+    expect(spec.freshConfig).toBe(true);
+    expect(spec.disk).toBe("/dev/nvme0n1");
+  });
+
+  it("enables the HTTP API at install so Pulumi can manage the router from first boot", () => {
+    const warn = vi.spyOn(console, "warn").mockImplementation(() => {});
+    const spec = buildVyosConfigSpec({
+      hostname: "vyos001",
+      spec: { bundle, apiKey: "k3y", apiListenAddress: "10.0.1.252" },
+      defaultPassword: "changeme",
+    });
+
+    expect(spec.sets).toContainEqual({
+      path: ["service", "https", "api", "keys", "id", "pulumi", "key"],
+      value: "k3y",
+    });
+    expect(spec.sets).toContainEqual({ path: ["service", "https", "api", "rest"] });
+    expect(spec.sets).toContainEqual({
+      path: ["service", "https", "listen-address"],
+      value: "10.0.1.252",
+    });
+    // The key id is a tag node; without this the installer's ConfigTree rejects it.
+    expect(spec.tags).toContainEqual(["service", "https", "api", "keys", "id"]);
+    warn.mockRestore();
+  });
+
+  it("binds the API to the static management address when none is given", () => {
+    const spec = buildVyosConfigSpec({
+      hostname: "vyos001",
+      spec: { apiKey: "k3y", mgmtAddress: "192.168.1.252/24" },
+      defaultPassword: "changeme",
+    });
+    expect(spec.sets).toContainEqual({
+      path: ["service", "https", "listen-address"],
+      value: "192.168.1.252",
+    });
+  });
+
+  it("refuses to enable the API unbound rather than exposing it on the WAN", () => {
+    const warn = vi.spyOn(console, "warn").mockImplementation(() => {});
+    // Management is DHCP, so there is no address to bind at build time. Binding
+    // to everything would put a config-write endpoint on the WAN.
+    const spec = buildVyosConfigSpec({
+      hostname: "vyos001",
+      spec: { apiKey: "k3y", mgmtAddress: "dhcp" },
+      defaultPassword: "changeme",
+    });
+    expect(spec.sets.some((s) => s.path[0] === "service" && s.path[1] === "https")).toBe(false);
+    expect(warn).toHaveBeenCalledWith(expect.stringContaining("has NOT been enabled"));
+    warn.mockRestore();
+  });
+
+  it("does not enable the API when no key is supplied", () => {
+    const spec = buildVyosConfigSpec({
+      hostname: "vyos001",
+      spec: { mgmtAddress: "192.168.1.252/24" },
+      defaultPassword: "changeme",
+    });
+    expect(spec.sets.some((s) => s.path[0] === "service" && s.path[1] === "https")).toBe(false);
+  });
+
+  it("ignores the derived path entirely when a bundle is present", () => {
+    // Belt and braces: even if topology flags reach this far (the CLI rejects
+    // them), the bundle must win rather than merge.
+    const spec = buildVyosConfigSpec({
+      hostname: "vyos001",
+      spec: { bundle, bondMembers: ["eth2", "eth3"], vlans: [{ id: 99, address: "10.9.9.1/24" }] },
+      defaultPassword: "changeme",
+    });
+    expect(spec.sets.some((s) => s.path.includes("99"))).toBe(false);
+    expect(spec.sets.filter((s) => s.value === "eth2" || s.value === "eth3")).toEqual([]);
+  });
+});

bastion/src/bastion/tests/vyos.test.ts

@@ -0,0 +1,548 @@
+import { describe, it, expect, beforeEach, afterEach } from "vitest";
+import { mkdirSync, rmSync } from "node:fs";
+import { join } from "node:path";
+import { tmpdir } from "node:os";
+import type { BastionConfig } from "@lab/shared";
+import type { FastifyInstance } from "fastify";
+import { createApp } from "../src/server.js";
+import type { StateManager } from "../src/services/state.js";
+import { buildVyosConfigSpec } from "../src/templates/vyos-config-spec.js";
+import { renderVyosInstallPy } from "../src/templates/vyos-install.py.js";
+
+function createTestConfig(testDir: string): BastionConfig {
+  return {
+    fedoraVersion: "43",
+    arch: "x86_64",
+    httpPort: 0,
+    timezone: "Europe/London",
+    locale: "en_GB.UTF-8",
+    bastionDir: testDir,
+    domain: "test.local",
+    dhcpMode: "proxy",
+    dhcpRangeStart: "",
+    dhcpRangeEnd: "",
+    ubuntuVersion: "26.04",
+    ubuntuMirror: "https://releases.ubuntu.com/26.04",
+    vyosIsoUrl: "https://example.invalid/vyos.iso",
+    vyosDefaultPassword: "test-pw",
+    iface: "eth0",
+    serverIp: "10.0.0.1",
+    network: "10.0.0.0",
+    gateway: "10.0.0.1",
+    sshKeys: ["ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAITEST lab@test"],
+    adminUser: "testadmin",
+    syslogPort: 15515,
+    skipDnsmasq: true,
+    skipArtifacts: true,
+    fedoraMirror: "https://example.invalid/fedora",
+    tftpDir: join(testDir, "tftp"),
+    httpDir: join(testDir, "http"),
+    stateFile: join(testDir, "state.json"),
+  };
+}
+
+/** Pull the base64 spec back out of the generated Python driver. */
+function decodeSpecFrom(python: string): Record<string, unknown> {
+  const match = /base64\.b64decode\("([^"]+)"\)/.exec(python);
+  if (!match?.[1]) throw new Error("no base64 spec found in generated driver");
+  return JSON.parse(Buffer.from(match[1], "base64").toString("utf-8"));
+}
+
+describe("vyos config spec", () => {
+  it("puts VLANs on the bond when members are given", () => {
+    const spec = buildVyosConfigSpec({
+      hostname: "fw1",
+      defaultPassword: "pw",
+      spec: {
+        mgmtInterface: "eth0",
+        mgmtAddress: "10.0.8.2/24",
+        bondMembers: ["eth2", "eth3"],
+        vlans: [{ id: 10, address: "10.0.10.1/24", description: "k8s" }],
+      },
+    });
+
+    const paths = spec.sets.map((s) => s.path.join(" "));
+    expect(paths).toContain("interfaces bonding bond0 mode");
+    expect(paths).toContain("interfaces bonding bond0 vif 10 address");
+    // VLANs must hang off the bond, not the management NIC.
+    expect(paths).not.toContain("interfaces ethernet eth0 vif 10 address");
+
+    // Bond members are a multi-value node — appending, not replacing, is what
+    // keeps the second member from overwriting the first.
+    const members = spec.sets.filter(
+      (s) => s.path.join(" ") === "interfaces bonding bond0 member interface",
+    );
+    expect(members.map((m) => m.value)).toEqual(["eth2", "eth3"]);
+    expect(members.every((m) => m.replace === false)).toBe(true);
+  });
+
+  it("falls back to VLANs on the management NIC when unbonded", () => {
+    const spec = buildVyosConfigSpec({
+      hostname: "fw2",
+      defaultPassword: "pw",
+      spec: { mgmtInterface: "eth1", vlans: [{ id: 20, address: "10.0.20.1/24" }] },
+    });
+
+    const paths = spec.sets.map((s) => s.path.join(" "));
+    expect(paths).toContain("interfaces ethernet eth1 vif 20 address");
+  });
+
+  it("normalises the target disk to a full /dev path", () => {
+    // find_disks() enumerates with `lsblk -Jbp`, so valid responses are full
+    // paths; a bare name fails valid_responses and re-prompts forever.
+    expect(buildVyosConfigSpec({ hostname: "fw3", defaultPassword: "pw", disk: "/dev/mmcblk0" }).disk)
+      .toBe("/dev/mmcblk0");
+    expect(buildVyosConfigSpec({ hostname: "fw3", defaultPassword: "pw", disk: "mmcblk0" }).disk)
+      .toBe("/dev/mmcblk0");
+    expect(buildVyosConfigSpec({ hostname: "fw3", defaultPassword: "pw" }).disk).toBe("");
+  });
+
+  it("defaults to dhcp on eth0 and never opts into RAID", () => {
+    const spec = buildVyosConfigSpec({ hostname: "fw4", defaultPassword: "pw" });
+    const address = spec.sets.find(
+      (s) => s.path.join(" ") === "interfaces ethernet eth0 address",
+    );
+    expect(address?.value).toBe("dhcp");
+    // The installer's RAID prompt defaults to yes; a second disk must not
+    // silently produce a mirror.
+    expect(spec.raid).toBe(false);
+  });
+});
+
+describe("vyos routes", () => {
+  let testDir: string;
+  let app: FastifyInstance;
+  let state: StateManager;
+  const mac = "aa:bb:cc:11:22:33";
+
+  beforeEach(() => {
+    testDir = join(tmpdir(), `bastion-vyos-test-${Date.now()}-${Math.random().toString(36).slice(2)}`);
+    mkdirSync(join(testDir, "http"), { recursive: true });
+    mkdirSync(join(testDir, "tftp"), { recursive: true });
+
+    const result = createApp(createTestConfig(testDir));
+    app = result.app;
+    state = result.state;
+  });
+
+  afterEach(async () => {
+    await app.close();
+    rmSync(testDir, { recursive: true, force: true });
+  });
+
+  it("dispatches a queued vyos machine to the live-boot script", async () => {
+    state.update((s) => {
+      s.install_queue[mac] = {
+        hostname: "fw1",
+        disk: "/dev/nvme0n1",
+        role: "worker",
+        os: "vyos-rolling",
+        queued_at: new Date().toISOString(),
+      };
+    });
+
+    const response = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}` });
+    expect(response.statusCode).toBe(200);
+
+    expect(response.body).toContain("/vyos-vmlinuz");
+    expect(response.body).toContain("fetch=http://10.0.0.1:0/vyos-filesystem.squashfs");
+    expect(response.body).toContain(`live-config.hooks=http://10.0.0.1:0/vyos/autoinstall.sh?mac=${mac}`);
+
+    // `nonetworking` appears in VyOS's own PXE docs but breaks the hook fetch,
+    // and console=ttyS0 costs 30s per systemd phase on boards with no UART.
+    expect(response.body).not.toContain("nonetworking");
+    expect(response.body).not.toContain("console=ttyS0");
+  });
+
+  it("serves a hook that fetches and executes the install driver", async () => {
+    const response = await app.inject({ method: "GET", url: `/vyos/autoinstall.sh?mac=${mac}` });
+    expect(response.statusCode).toBe(200);
+    expect(response.body).toContain(`/vyos/install.py?mac=${mac}`);
+    expect(response.body).toContain("python3 /tmp/vyos-install.py");
+  });
+
+  it("bakes the machine's config into the generated install driver", async () => {
+    state.update((s) => {
+      s.install_queue[mac] = {
+        hostname: "fw1",
+        disk: "/dev/nvme0n1",
+        role: "worker",
+        os: "vyos-rolling",
+        queued_at: new Date().toISOString(),
+        vyos: {
+          mgmtInterface: "eth0",
+          mgmtAddress: "10.0.8.2/24",
+          bondMembers: ["eth2", "eth3"],
+          vlans: [{ id: 10, address: "10.0.10.1/24" }],
+          password: "s3cret",
+        },
+      };
+    });
+
+    const response = await app.inject({ method: "GET", url: `/vyos/install.py?mac=${mac}` });
+    expect(response.statusCode).toBe(200);
+
+    // Builds config from the image's own default so the vyos-config-version
+    // trailer matches and first boot skips migrations.
+    expect(response.body).toContain("/opt/vyatta/etc/config.boot.default");
+    expect(response.body).toContain("/usr/libexec/vyos/op_mode/image_installer.py");
+    // "complete" is what moves the machine out of the install queue.
+    expect(response.body).toContain('report("complete"');
+
+    const spec = decodeSpecFrom(response.body);
+    expect(spec["hostname"]).toBe("fw1");
+    expect(spec["password"]).toBe("s3cret");
+    expect(spec["disk"]).toBe("/dev/nvme0n1");
+
+    const paths = (spec["sets"] as Array<{ path: string[] }>).map((s) => s.path.join(" "));
+    expect(paths).toContain("interfaces bonding bond0 vif 10 address");
+    expect(paths).toContain("system host-name");
+  });
+
+  it("falls back to the bastion default password when none is set", async () => {
+    state.update((s) => {
+      s.install_queue[mac] = {
+        hostname: "fw9",
+        disk: "",
+        role: "worker",
+        os: "vyos-rolling",
+        queued_at: new Date().toISOString(),
+      };
+    });
+
+    const response = await app.inject({ method: "GET", url: `/vyos/install.py?mac=${mac}` });
+    const spec = decodeSpecFrom(response.body);
+    expect(spec["password"]).toBe("test-pw");
+    // Empty disk means "accept the installer's first-disk default".
+    expect(spec["disk"]).toBe("");
+  });
+});
+
+describe("vyos hw-id pinning", () => {
+  it("emits hw-id for the mgmt interface and each bond member", () => {
+    // Discovery sees enp2s0/enp1s0f0np0 under Fedora, but VyOS enumerates its
+    // own eth<N>. Pinning by MAC is what makes the mapping deterministic.
+    const spec = buildVyosConfigSpec({
+      hostname: "fw1",
+      defaultPassword: "pw",
+      spec: {
+        mgmtInterface: "eth2",
+        bondMembers: ["eth0", "eth1"],
+        hwIds: {
+          eth2: "64:62:66:25:96:47",
+          eth0: "64:62:66:25:96:45",
+          eth1: "64:62:66:25:96:46",
+        },
+      },
+    });
+
+    const hw = spec.sets.filter((s) => s.path[s.path.length - 1] === "hw-id");
+    expect(hw.map((s) => [s.path[2], s.value])).toEqual([
+      ["eth2", "64:62:66:25:96:47"],
+      ["eth0", "64:62:66:25:96:45"],
+      ["eth1", "64:62:66:25:96:46"],
+    ]);
+  });
+
+  it("omits hw-id entirely when no mapping is given", () => {
+    const spec = buildVyosConfigSpec({ hostname: "fw1", defaultPassword: "pw" });
+    expect(spec.sets.some((s) => s.path.includes("hw-id"))).toBe(false);
+  });
+});
+
+describe("vyos management VLAN", () => {
+  it("puts the mgmt VLAN on the PXE port while the bond carries routed VLANs", () => {
+    // Trunked PXE port: boots untagged on the VLAN the bastion serves, stays
+    // reachable on the tagged management VLAN.
+    const spec = buildVyosConfigSpec({
+      hostname: "vyos001",
+      defaultPassword: "pw",
+      spec: {
+        mgmtInterface: "eth2",
+        mgmtAddress: "dhcp",
+        mgmtVlan: { id: 3, address: "192.168.3.4/24", description: "kvm" },
+        bondMembers: ["eth0", "eth1"],
+        vlans: [{ id: 2, address: "192.168.8.2/23" }],
+      },
+    });
+
+    const paths = spec.sets.map((s) => s.path.join(" "));
+    expect(paths).toContain("interfaces ethernet eth2 vif 3 address");
+    expect(paths).toContain("interfaces bonding bond0 vif 2 address");
+    // The mgmt VLAN must not land on the bond.
+    expect(paths).not.toContain("interfaces bonding bond0 vif 3 address");
+    expect(spec.tags.map((t) => t.join(" "))).toContain("interfaces ethernet eth2 vif");
+  });
+});
+
+describe("vyos VRRP HA", () => {
+  const haSpec = {
+    mgmtInterface: "eth2",
+    mgmtAddress: "dhcp",
+    bondMembers: ["eth0", "eth1"],
+    bondAddress: "192.168.1.252/24",
+    bondVrrp: "192.168.1.254/24",
+    vrrpPriority: 200,
+    vlans: [
+      { id: 3, address: "192.168.3.4/24", vrrp: "192.168.3.254/24" },
+      { id: 200, address: "192.168.2.252/24" },   // no VIP on this one
+    ],
+  };
+
+  it("emits a vrrp group per VIP with vrid = VLAN id and dotted vif interface", () => {
+    const spec = buildVyosConfigSpec({ hostname: "fw1", defaultPassword: "pw", spec: haSpec });
+    const paths = spec.sets.map((s) => `${s.path.join(" ")}${s.value !== undefined ? "=" + s.value : ""}`);
+
+    expect(paths).toContain("interfaces bonding bond0 address=192.168.1.252/24");
+    // untagged bond group: vrid 1, interface bond0 itself
+    expect(paths).toContain("high-availability vrrp group native interface=bond0");
+    expect(paths).toContain("high-availability vrrp group native vrid=1");
+    // address is a tag node -- VIP is the final path segment, no value
+    expect(paths).toContain("high-availability vrrp group native address 192.168.1.254/24");
+    // VLAN group: vrid = VLAN id, dotted vif
+    expect(paths).toContain("high-availability vrrp group vlan3 interface=bond0.3");
+    expect(paths).toContain("high-availability vrrp group vlan3 vrid=3");
+    expect(paths).toContain("high-availability vrrp group vlan3 address 192.168.3.254/24");
+    // VLAN without a VIP gets no group
+    expect(paths.some((p) => p.includes("group vlan200"))).toBe(false);
+  });
+
+  it("applies the box-wide priority and one sync group over all groups", () => {
+    const spec = buildVyosConfigSpec({ hostname: "fw1", defaultPassword: "pw", spec: haSpec });
+    const prio = spec.sets.filter((s) => s.path[s.path.length - 1] === "priority"
+      && s.path[0] === "high-availability");
+    expect(prio).toHaveLength(2);
+    expect(prio.every((s) => s.value === "200")).toBe(true);
+
+    // sync group binds the pair: all groups fail over together
+    const members = spec.sets.filter(
+      (s) => s.path.join(" ") === "high-availability vrrp sync-group MAIN member",
+    );
+    expect(members.map((m) => m.value)).toEqual(["native", "vlan3"]);
+    expect(members.every((m) => m.replace === false)).toBe(true);
+  });
+
+  it("emits no high-availability nodes when no VIPs are given", () => {
+    const spec = buildVyosConfigSpec({
+      hostname: "fw1",
+      defaultPassword: "pw",
+      spec: { bondMembers: ["eth0", "eth1"], vlans: [{ id: 3, address: "192.168.3.4/24" }] },
+    });
+    expect(spec.sets.some((s) => s.path[0] === "high-availability")).toBe(false);
+  });
+});
+
+describe("pickLargestInitrd", async () => {
+  const { pickLargestInitrd } = await import("../src/main.js");
+
+  // Verbatim from `xorriso -lsl /live/` on vyos-2026.08.05-0033-rolling.
+  const realListing = `total 8
+-r--r--r--    1 0        0           22255 Aug  5 01:33 'filesystem.packages'
+-r--r--r--    1 0        0               6 Aug  5 01:33 'filesystem.packages-remove'
+-r--r--r--    1 0        0        541192192 Aug  5 01:33 'filesystem.squashfs'
+-r--r--r--    1 0        0        50352547 Aug  5 01:33 'initrd.img'
+-r--r--r--    1 0        0        50352547 Aug  5 01:33 'initrd.img-6.18.41-vyos'
+-r--r--r--    1 0        0              20 Aug  5 01:33 'packages.txt'
+-r--r--r--    1 0        0         9135104 Aug  2 19:54 'vmlinuz'
+-r--r--r--    1 0        0         9135104 Aug  2 19:54 'vmlinuz-6.18.41-vyos'
+`;
+
+  it("picks a full-size initrd from a real nightly listing", () => {
+    expect(pickLargestInitrd(realListing)).toEqual({ name: "initrd.img", size: 50352547 });
+  });
+
+  it("ignores 0-byte decoys and symlinks (which report link size, not target size)", () => {
+    const listing = `total 8
+-r--r--r--    1 0        0               0 Aug  5 01:33 'initrd.img'
+lrwxrwxrwx    1 0        0              24 Aug  5 01:33 'initrd.img-link' -> 'initrd.img-6.18.41-vyos'
+-r--r--r--    1 0        0        50352547 Aug  5 01:33 'initrd.img-6.18.41-vyos'
+`;
+    expect(pickLargestInitrd(listing)).toEqual({ name: "initrd.img-6.18.41-vyos", size: 50352547 });
+  });
+
+  it("returns undefined when only decoys exist", () => {
+    expect(pickLargestInitrd("-r--r--r-- 1 0 0 0 Aug 5 01:33 'initrd.img'\n")).toBeUndefined();
+  });
+});
+
+describe("vyos fedora-parity features", () => {
+  it("computes reportAddress from a static mgmt address, empty for dhcp", () => {
+    const staticSpec = buildVyosConfigSpec({
+      hostname: "fw1", defaultPassword: "pw",
+      spec: { mgmtAddress: "192.168.8.2/23" },
+    });
+    expect(staticSpec.reportAddress).toBe("192.168.8.2");
+
+    const dhcpSpec = buildVyosConfigSpec({ hostname: "fw1", defaultPassword: "pw" });
+    expect(dhcpSpec.reportAddress).toBe("");
+  });
+
+  it("defaults freshConfig off (reinstall preserves the on-disk config)", () => {
+    expect(buildVyosConfigSpec({ hostname: "fw1", defaultPassword: "pw" }).freshConfig).toBe(false);
+    expect(buildVyosConfigSpec({
+      hostname: "fw1", defaultPassword: "pw", spec: { freshConfig: true },
+    }).freshConfig).toBe(true);
+  });
+
+  it("driver streams logs to /api/log and reports 'ready at' on completion", async () => {
+    const testDir = join(tmpdir(), `bastion-vyos-parity-${Date.now()}`);
+    mkdirSync(join(testDir, "http"), { recursive: true });
+    mkdirSync(join(testDir, "tftp"), { recursive: true });
+    const { app: parityApp, state: parityState } = createApp(createTestConfig(testDir));
+    try {
+      parityState.update((s) => {
+        s.install_queue["aa:bb:cc:44:55:66"] = {
+          hostname: "fw9", disk: "/dev/vda", role: "vanilla",
+          os: "vyos-rolling", queued_at: new Date().toISOString(),
+        };
+      });
+      const response = await parityApp.inject({
+        method: "GET", url: "/vyos/install.py?mac=aa:bb:cc:44:55:66",
+      });
+      expect(response.body).toContain("/api/log");
+      expect(response.body).toContain('"lines": batch');
+      expect(response.body).toContain('report("complete", "ready at %s"');
+      expect(response.body).toContain("ensure_network_boot_first");
+      expect(response.body).toContain("lab-provisioned");
+      expect(response.body).toContain('ROLE = "vanilla"');
+    } finally {
+      await parityApp.close();
+      rmSync(testDir, { recursive: true, force: true });
+    }
+  });
+
+  it("complete with 'ready at' records installed.ip for a vyos machine", async () => {
+    const testDir = join(tmpdir(), `bastion-vyos-complete-${Date.now()}`);
+    mkdirSync(join(testDir, "http"), { recursive: true });
+    mkdirSync(join(testDir, "tftp"), { recursive: true });
+    const { app: cApp, state: cState } = createApp(createTestConfig(testDir));
+    try {
+      const mac2 = "aa:bb:cc:77:88:99";
+      cState.update((s) => {
+        s.install_queue[mac2] = {
+          hostname: "fw1", disk: "/dev/vda", role: "vanilla",
+          os: "vyos-rolling", queued_at: new Date().toISOString(),
+        };
+      });
+      const response = await cApp.inject({
+        method: "POST", url: "/api/progress",
+        headers: { "Content-Type": "application/json" },
+        body: JSON.stringify({ mac: mac2, stage: "complete", detail: "ready at 192.168.8.2" }),
+      });
+      expect(response.statusCode).toBe(200);
+      const installed = cState.load().installed[mac2];
+      expect(installed?.ip).toBe("192.168.8.2");
+      expect(installed?.os).toBe("vyos-rolling");
+    } finally {
+      await cApp.close();
+      rmSync(testDir, { recursive: true, force: true });
+    }
+  });
+});
+
+describe("vyos installer prompt coverage", () => {
+  // Every interactive prompt image_installer.py can emit, copied verbatim from
+  // the MSG_* constants (including the reinstall-only search_previous_installation
+  // ones). An unanswered prompt does not fail loudly -- the installer simply
+  // blocks on stdin until the driver's stall timeout, which is how the reinstall
+  // path silently hung for 15 minutes in the VM test.
+  const PROMPTS: Record<string, string> = {
+    continue: "Would you like to continue? [y/N] ",
+    imageName: "What would you like to name this image? (Default: 1.5-rolling) ",
+    password: 'Please enter a password for the "vyos" user: ',
+    passwordConfirm: 'Please confirm password for the "vyos" user: ',
+    console: "What console should be used by default? (K: KVM, S: Serial)? (Default: K) ",
+    raidConfigure: "Would you like to configure RAID-1 mirroring? [Y/n] ",
+    raidFoundDisks: "Would you like to configure RAID-1 mirroring on them? [Y/n] ",
+    raidChooseDisks: "Would you like to choose two disks for RAID-1 mirroring? [Y/n] ",
+    diskSelect: "Which one should be used for installation? (Default: /dev/vda) ",
+    diskConfirm: "Installation will delete all data on the drive. Continue? [y/N] ",
+    raidConfirm: "Installation will delete all data on both drives. Continue? [y/N] ",
+    rootSizeAll: "Would you like to use all the free space on the drive? [Y/n] ",
+    bootConfig: "Which file would you like as boot config? ",
+    copyData: "Would you like to copy data to the new image? [Y/n] ",
+    chooseCopyData: "From which image would you like to save config information? ",
+    copyEncData: "Would you like to copy the encrypted config to the new image? [Y/n] ",
+    chooseCopyEncData: "From which image would you like to copy the encrypted config? ",
+  };
+
+  it("answers every installer prompt exactly once", () => {
+    const { execFileSync } = require("node:child_process") as typeof import("node:child_process");
+    const { writeFileSync, unlinkSync, mkdtempSync } = require("node:fs") as typeof import("node:fs");
+
+    // Skip cleanly where python3 is unavailable (same spirit as the
+    // ksvalidator-backed kickstart test).
+    try {
+      execFileSync("python3", ["--version"], { stdio: "pipe" });
+    } catch {
+      return;
+    }
+
+    const spec = buildVyosConfigSpec({
+      hostname: "fw1", defaultPassword: "pw", disk: "/dev/vda",
+    });
+    const driver = renderVyosInstallPy({
+      spec, mac: "aa:bb:cc:11:22:33", serverIp: "10.0.0.1", httpPort: 8080, role: "vanilla",
+    });
+
+    const dir = mkdtempSync(join(tmpdir(), "vyos-rules-"));
+    const driverPath = join(dir, "driver.py");
+    const checkPath = join(dir, "check.py");
+    writeFileSync(driverPath, driver);
+    writeFileSync(checkPath, `
+import importlib.util, json, sys
+spec = importlib.util.spec_from_file_location("drv", ${JSON.stringify(driverPath)})
+drv = importlib.util.module_from_spec(spec); spec.loader.exec_module(drv)
+rules = drv.build_rules()
+prompts = json.loads(sys.argv[1])
+out = {}
+for label, text in prompts.items():
+    out[label] = len([r for p, r in rules if p.search(text.encode())])
+print(json.dumps(out))
+`);
+
+    try {
+      const stdout = execFileSync("python3", [checkPath, JSON.stringify(PROMPTS)], {
+        encoding: "utf-8", stdio: ["pipe", "pipe", "pipe"],
+      });
+      const counts = JSON.parse(stdout) as Record<string, number>;
+      const unanswered = Object.entries(counts).filter(([, n]) => n !== 1);
+      expect(unanswered).toEqual([]);
+    } finally {
+      try { unlinkSync(driverPath); unlinkSync(checkPath); } catch { /* best effort */ }
+      try { rmSync(dir, { recursive: true, force: true }); } catch { /* best effort */ }
+    }
+  });
+});
+
+describe("vyos boot NIC pinning", () => {
+  it("pins the boot interface by MAC via BOOTIF", async () => {
+    // Without this, live-boot picks the first *connected* NIC. On the VP2440
+    // the SFP+ pair links before the copper PXE port, so live-boot tried the
+    // fiber ports (no DHCP), timed out 15s each, and failed with "Unable to
+    // find a live file system on the network".
+    const testDir = join(tmpdir(), `bastion-vyos-bootif-${Date.now()}`);
+    mkdirSync(join(testDir, "http"), { recursive: true });
+    mkdirSync(join(testDir, "tftp"), { recursive: true });
+    const { app: a, state: st } = createApp(createTestConfig(testDir));
+    try {
+      const m = "64:62:66:25:96:47";
+      st.update((s) => {
+        s.install_queue[m] = {
+          hostname: "vyos001", disk: "/dev/mmcblk0", role: "vanilla",
+          os: "vyos-rolling", queued_at: new Date().toISOString(),
+        };
+      });
+      const res = await a.inject({ method: "GET", url: `/dispatch?mac=${m}` });
+      // live-boot's Device_from_bootif() expects 01-<mac with dashes>
+      expect(res.body).toContain("BOOTIF=01-64-62-66-25-96-47");
+      // and it must be on the kernel line, before fetch= is attempted
+      const kernelLine = res.body.split("\n").find((l) => l.startsWith("kernel "));
+      expect(kernelLine).toContain("BOOTIF=01-64-62-66-25-96-47");
+      expect(kernelLine).toContain("fetch=");
+    } finally {
+      await a.close();
+      rmSync(testDir, { recursive: true, force: true });
+    }
+  });
+});

bastion/src/cli/src/api/client.ts

@@ -90,6 +90,8 @@ export class LabdClient {
 
   async installMachine(opts: {
     mac: string; hostname: string; disk?: string; role?: string; os?: string;
+    schemaVersion?: 1 | 2;
+    vyos?: import("@lab/shared").VyosInstallSpec;
   }): Promise<{ status: string; data?: unknown; error?: string }> {
     return this.request("POST", "/api/machines/install", { body: opts });
   }

bastion/src/cli/src/commands/app.ts

@@ -57,6 +57,7 @@ async function fetchState(): Promise<BastionState | null> {
 }
 
 import { registerLabcontrollerCommands } from "./labcontroller.js";
+import { isRole, SUPPORTED_ROLES, type Role } from "@lab/shared";
 
 export function registerAppCommand(program: Command): void {
   const appCmd = program.command("app").description("Application management");
@@ -66,18 +67,104 @@ export function registerAppCommand(program: Command): void {
 
   const k3sCmd = appCmd.command("k3s").description("k3s cluster management");
 
+  k3sCmd
+    .command("evacuate <target>")
+    .description("Cordon, evict Longhorn replicas and drain so a node can be safely wiped")
+    .option("--user <user>", "SSH user", "root")
+    .option(
+      "--allow-single-replica",
+      "permit evacuating volumes that have only one replica (rebuilds from the only copy; there is no backup target)",
+    )
+    .action(async (target: string, opts: { user: string; allowSingleReplica?: boolean }) => {
+      const state = await fetchState();
+      const resolved = resolveTarget(target, state);
+      if (!resolved) {
+        console.error(`Cannot resolve target: ${target}`);
+        process.exit(1);
+      }
+      const sshKey = findSshKey();
+
+      console.log(`Evacuating ${resolved.hostname} (${resolved.ip})...`);
+      console.log("This waits for Longhorn to rebuild replicas elsewhere and can take a while.\n");
+
+      const k3sMod = new K3sModule();
+      const result = await k3sMod.evacuate({
+        hostname: resolved.hostname,
+        ip: resolved.ip,
+        role: "infra",
+        os: "fedora-43" as const,
+        arch: "x86_64" as const,
+        sshUser: opts.user,
+        ...(sshKey ? { sshKeyPath: sshKey } : {}),
+        config: opts.allowSingleReplica ? { evictionAllowSingleReplica: true } : {},
+      });
+
+      for (const line of result.output) console.log(`  ${line}`);
+      if (!result.success) {
+        console.error(`\nNode is NOT safe to wipe (${result.errors.join(", ")}). Do not reprovision it.`);
+        console.error(`If an eviction was started and left pending, clear it with:`);
+        console.error(`  labctl app k3s admit ${resolved.hostname}`);
+        process.exit(1);
+      }
+      console.log(`\n${resolved.hostname} is evacuated. Safe to reprovision.`);
+    });
+
+  k3sCmd
+    .command("admit <target>")
+    .description("Return an evacuated node to service: uncordon, clear eviction, restore its Longhorn disk")
+    .option("--user <user>", "SSH user", "root")
+    .action(async (target: string, opts: { user: string }) => {
+      const state = await fetchState();
+      const resolved = resolveTarget(target, state);
+      if (!resolved) {
+        console.error(`Cannot resolve target: ${target}`);
+        process.exit(1);
+      }
+      const sshKey = findSshKey();
+
+      console.log(`Returning ${resolved.hostname} (${resolved.ip}) to service...`);
+
+      const k3sMod = new K3sModule();
+      const result = await k3sMod.admit({
+        hostname: resolved.hostname,
+        ip: resolved.ip,
+        role: isRole(resolved.role) ? resolved.role : "infra",
+        os: "fedora-43" as const,
+        arch: "x86_64" as const,
+        sshUser: opts.user,
+        ...(sshKey ? { sshKeyPath: sshKey } : {}),
+        config: {},
+      });
+
+      for (const line of result.output) console.log(`  ${line}`);
+      if (!result.success) {
+        console.error(`\n${resolved.hostname} is NOT fully back in service (${result.errors.join(", ")}).`);
+        process.exit(1);
+      }
+      console.log(`\n${resolved.hostname} is back in service.`);
+    });
+
   k3sCmd
     .command("install <target>")
     .description("Install k3s on a target machine (hostname, IP, or MAC)")
-    .option("--role <role>", "k3s role: infra (server) or worker (agent)", "infra")
+    .option(
+      "--role <role>",
+      "k3s role: infra, labcontroller, standalone-cluster, k8s-controller, k8s-combo (servers) or worker, k8s-worker (agents)",
+      "infra",
+    )
     .option("--user <user>", "SSH user", "root")
-    .option("--k3s-server <url>", "k3s server URL (required for worker role)")
-    .option("--k3s-token <token>", "k3s join token (required for worker role)")
+    .option("--k3s-server <url>", "k3s server URL (required for agent roles, and to JOIN as a server)")
+    .option("--k3s-token <token>", "k3s join token; on a first server it SEEDS the cluster token")
+    .option(
+      "--k3s-version <version>",
+      "exact k3s version, e.g. v1.34.6+k3s1. Unset installs LATEST, which on a rejoining node can be several minor versions ahead of the cluster",
+    )
     .action(async (target: string, opts: {
       role: string;
       user: string;
       k3sServer?: string;
       k3sToken?: string;
+      k3sVersion?: string;
     }) => {
       const state = await fetchState();
       const resolved = resolveTarget(target, state);
@@ -88,7 +175,16 @@ export function registerAppCommand(program: Command): void {
         process.exit(1);
       }
 
-      const role = opts.role === "worker" ? "worker" : "infra";
+      // Every supported role, not just infra/worker. The old mapping silently
+      // turned anything that was not "worker" into "infra" -- so
+      // `--role k8s-combo` installed a DIFFERENT role than the one asked for,
+      // with different disk expectations and no warning.
+      if (!isRole(opts.role)) {
+        console.error(`Unknown role: ${opts.role}`);
+        console.error(`Valid roles: ${SUPPORTED_ROLES.join(", ")}`);
+        process.exit(1);
+      }
+      const role: Role = opts.role;
       const sshKey = findSshKey();
 
       console.log(`Installing k3s on ${resolved.hostname} (${resolved.ip}) as ${role}...`);
@@ -106,9 +202,22 @@ export function registerAppCommand(program: Command): void {
         config: {
           ...(opts.k3sServer ? { k3sServerUrl: opts.k3sServer } : {}),
           ...(opts.k3sToken ? { k3sToken: opts.k3sToken } : {}),
+          ...(opts.k3sVersion ? { k3sVersion: opts.k3sVersion } : {}),
         },
       };
 
+      // Only warn when it matters: a node JOINING an existing cluster. On a
+      // first server there is nothing to be skewed against, and a warning that
+      // fires when it is irrelevant is a warning people learn to skip.
+      if (!opts.k3sVersion && opts.k3sServer) {
+        console.warn(
+          "  WARNING: no --k3s-version given, so get.k3s.io installs LATEST.\n" +
+          "  This node is JOINING an existing cluster — check what that cluster runs\n" +
+          "  (kubectl get nodes -o wide) and pin it. Kubernetes supports only ONE minor\n" +
+          "  version of skew between control-plane members.",
+        );
+      }
+
       const installResult = await k3s.install(moduleCtx);
       for (const line of installResult.output) {
         console.log(`  ${line}`);
@@ -281,7 +390,8 @@ export function registerAppCommand(program: Command): void {
       const healthResult = await k3s.health({
         hostname: resolved.hostname,
         ip: resolved.ip,
-        role: resolved.role,
+        // State is JSON on disk; narrow it rather than trusting the string.
+        role: isRole(resolved.role) ? resolved.role : "vanilla",
         os: "fedora-43" as const,
         arch: "x86_64" as const,
         sshUser: opts.user,

bastion/src/cli/src/commands/install.ts

@@ -1,10 +1,61 @@
 // CLI command: provision install
 // Queue a discovered machine for OS installation via labd.
 
-import { Command, Option } from "commander";
+import { readFileSync } from "node:fs";
+import { Command, Option, InvalidArgumentError } from "commander";
 import { isValidOsId, SUPPORTED_OS, SUPPORTED_ROLES, ROLE_REGISTRY } from "@lab/shared";
+import type { VyosBundle, VyosInstallSpec, VyosVlanSpec } from "@lab/shared";
 import { getLabdClient } from "../api/config.js";
 
+/**
+ * Load one router's config out of a Pulumi-rendered bundle.
+ *
+ * The bundle is produced by `kubernetes-deployment` (npm run vyos:bundle) and
+ * holds every router it manages, keyed by name. Selecting by hostname here is
+ * what keeps bring-up and `pulumi up` describing the same box: labctl replays
+ * the declared config rather than deriving its own.
+ */
+export function loadVyosBundle(path: string, hostname: string): VyosBundle {
+  let parsed: { version?: number; routers?: Record<string, VyosBundle> };
+  try {
+    parsed = JSON.parse(readFileSync(path, "utf8"));
+  } catch (e) {
+    throw new InvalidArgumentError(`Cannot read VyOS bundle ${path}: ${(e as Error).message}`);
+  }
+  if (parsed.version !== 1) {
+    throw new InvalidArgumentError(
+      `VyOS bundle ${path} has version ${parsed.version ?? "<none>"}; this labctl understands 1`,
+    );
+  }
+  const router = parsed.routers?.[hostname];
+  if (router === undefined) {
+    const known = Object.keys(parsed.routers ?? {}).join(", ") || "<none>";
+    throw new InvalidArgumentError(
+      `VyOS bundle ${path} has no entry for "${hostname}" (has: ${known})`,
+    );
+  }
+  return router;
+}
+
+/** Parse a repeated --vlan flag: "<id>:<cidr>[:<description>]". */
+export function parseVlan(value: string, previous: VyosVlanSpec[] = []): VyosVlanSpec[] {
+  const parts = value.split(":");
+  const id = Number(parts[0]);
+  const address = parts[1] ?? "";
+  // InvalidArgumentError makes commander print a clean message instead of
+  // dumping a stack trace at the operator.
+  if (!Number.isInteger(id) || id < 1 || id > 4094) {
+    throw new InvalidArgumentError(`Invalid VLAN id in "${value}" (expected 1-4094)`);
+  }
+  if (!address.includes("/")) {
+    throw new InvalidArgumentError(
+      `Invalid VLAN address in "${value}" (expected CIDR, e.g. 10.0.10.1/24)`,
+    );
+  }
+  const description = parts.slice(2).join(":");
+  return [...previous, { id, address, ...(description ? { description } : {}) }];
+}
+
 function roleTable(): string {
   const lines: string[] = ["", "Available roles:"];
   for (const r of ROLE_REGISTRY) {
@@ -15,6 +66,38 @@ function roleTable(): string {
   return lines.join("\n");
 }
 
+/** Parse a repeated --vlan-vip flag: "<id>:<cidr>" — VRRP VIP for a --vlan entry. */
+export function parseVlanVip(
+  value: string,
+  previous: Record<number, string> = {},
+): Record<number, string> {
+  const index = value.indexOf(":");
+  const id = Number(index === -1 ? Number.NaN : value.slice(0, index));
+  const cidr = index === -1 ? "" : value.slice(index + 1).trim();
+  if (!Number.isInteger(id) || id < 1 || id > 4094 || !cidr.includes("/")) {
+    throw new InvalidArgumentError(
+      `Invalid VLAN VIP "${value}" (expected <id>:<cidr>, e.g. 3:192.168.3.254/24)`,
+    );
+  }
+  return { ...previous, [id]: cidr };
+}
+
+/** Parse a repeated --vyos-hwid flag: "<iface>=<mac>". */
+export function parseHwId(
+  value: string,
+  previous: Record<string, string> = {},
+): Record<string, string> {
+  const index = value.indexOf("=");
+  const iface = index === -1 ? "" : value.slice(0, index).trim();
+  const mac = index === -1 ? "" : value.slice(index + 1).trim().toLowerCase();
+  if (iface === "" || !/^([0-9a-f]{2}:){5}[0-9a-f]{2}$/.test(mac)) {
+    throw new InvalidArgumentError(
+      `Invalid hw-id "${value}" (expected <iface>=<mac>, e.g. eth2=64:62:66:25:96:47)`,
+    );
+  }
+  return { ...previous, [iface]: mac };
+}
+
 export function registerInstallCommand(parent: Command): void {
   parent
     .command("install <mac> <hostname>")
@@ -24,10 +107,54 @@ export function registerInstallCommand(parent: Command): void {
     .addOption(new Option("--role <role>", "Machine role (see below)").choices([...SUPPORTED_ROLES]).default("worker"))
     .addOption(new Option("--os <os>", "Operating system").choices([...SUPPORTED_OS]).default("fedora-43"))
     .option("--disk <device>", "Target disk device (auto-detect if omitted)")
+    .addOption(new Option("--schema-version <n>", "Disk layout for a FRESH install (default 1). Ignored on reprovision: the layout already on the disk wins.").choices(["1", "2"]))
+    .option("--vyos-mgmt <iface>", "VyOS: untagged interface the machine PXE boots from (default eth0)")
+    .option("--vyos-mgmt-address <addr>", "VyOS: CIDR for the management interface, or 'dhcp' (default dhcp)")
+    .option("--vyos-bond <ifaces>", "VyOS: comma-separated LACP bond members (must exclude the PXE NIC)")
+    .option("--vyos-bond-address <cidr>", "VyOS: address on the untagged bond (trunk native VLAN)")
+    .option("--vyos-bond-vrrp <cidr>", "VyOS: VRRP VIP floated on the untagged bond")
+    .option("--vlan-vip <id:cidr>", "VyOS: VRRP VIP for a --vlan entry (repeatable)", parseVlanVip)
+    .option("--vyos-vrrp-priority <n>", "VyOS: VRRP priority for all groups on this box (higher = master)")
+    .option("--vyos-mgmt-vlan <id:cidr[:desc]>", "VyOS: tagged management VLAN on the PXE port")
+    .option("--vlan <id:cidr[:desc]>", "VyOS: tagged VLAN sub-interface on the bond (repeatable)", parseVlan)
+    .option("--vyos-password <password>", "VyOS: password for the 'vyos' user")
+    .option("--vyos-hwid <iface=mac>", "VyOS: pin an interface name to a MAC via hw-id (repeatable)", parseHwId)
+    .option("--vyos-fresh-config", "VyOS: on reinstall, overwrite the preserved config with the generated one")
+    .option(
+      "--vyos-bundle <path>",
+      "VyOS: apply a Pulumi-rendered bundle verbatim (kubernetes-deployment/infra/vyos/vyos-bundle.json). " +
+        "Replaces the derived --vyos-bond/--vlan/... config; secret values are left unset for `pulumi up`.",
+    )
+    .option(
+      "--vyos-api-key <key>",
+      "VyOS: enable the HTTP API with this key so Pulumi can manage the router from first boot",
+    )
+    .option(
+      "--vyos-api-listen <addr>",
+      "VyOS: address the HTTP API binds to (default: the static management address). " +
+        "Required when management is DHCP; the API is never bound to all interfaces.",
+    )
     .action(async (mac: string, hostname: string, opts: {
       role: string;
       os: string;
+      /** commander gives us the string form of the --schema-version choice. */
+      schemaVersion?: string;
       disk?: string;
+      vyosMgmt?: string;
+      vyosMgmtAddress?: string;
+      vyosBond?: string;
+      vyosBondAddress?: string;
+      vyosBondVrrp?: string;
+      vlan?: VyosVlanSpec[];
+      vlanVip?: Record<number, string>;
+      vyosVrrpPriority?: string;
+      vyosMgmtVlan?: string;
+      vyosPassword?: string;
+      vyosHwid?: Record<string, string>;
+      vyosFreshConfig?: boolean;
+      vyosBundle?: string;
+      vyosApiKey?: string;
+      vyosApiListen?: string;
     }) => {
       if (!isValidOsId(opts.os)) {
         console.error(`Unknown OS: ${opts.os}. Supported: ${SUPPORTED_OS.join(", ")}`);
@@ -39,18 +166,105 @@ export function registerInstallCommand(parent: Command): void {
         process.exit(1);
       }
 
+      const bondMembers = opts.vyosBond !== undefined && opts.vyosBond !== ""
+        ? opts.vyosBond.split(",").map((s) => s.trim()).filter((s) => s.length > 0)
+        : [];
+
+      // Attach --vlan-vip entries to their --vlan definitions. A VIP for a VLAN
+      // that was never defined is a typo that would otherwise vanish silently.
+      const vips = opts.vlanVip ?? {};
+      const vlans = (opts.vlan ?? []).map((v) =>
+        vips[v.id] !== undefined ? { ...v, vrrp: vips[v.id] as string } : v,
+      );
+      for (const id of Object.keys(vips)) {
+        if (!vlans.some((v) => String(v.id) === id)) {
+          console.error(`--vlan-vip ${id}:... has no matching --vlan ${id}:... entry`);
+          process.exit(1);
+        }
+      }
+
+      const vrrpPriority = opts.vyosVrrpPriority !== undefined && opts.vyosVrrpPriority !== ""
+        ? Number(opts.vyosVrrpPriority)
+        : undefined;
+      if (vrrpPriority !== undefined
+          && (!Number.isInteger(vrrpPriority) || vrrpPriority < 1 || vrrpPriority > 255)) {
+        console.error(`--vyos-vrrp-priority must be an integer 1-255 (got ${opts.vyosVrrpPriority})`);
+        process.exit(1);
+      }
+      const vyos: VyosInstallSpec = {
+        ...(opts.vyosMgmt !== undefined && opts.vyosMgmt !== ""
+          ? { mgmtInterface: opts.vyosMgmt } : {}),
+        ...(opts.vyosMgmtAddress !== undefined && opts.vyosMgmtAddress !== ""
+          ? { mgmtAddress: opts.vyosMgmtAddress } : {}),
+        ...(bondMembers.length > 0 ? { bondMembers } : {}),
+        ...(opts.vyosBondAddress !== undefined && opts.vyosBondAddress !== ""
+          ? { bondAddress: opts.vyosBondAddress } : {}),
+        ...(opts.vyosBondVrrp !== undefined && opts.vyosBondVrrp !== ""
+          ? { bondVrrp: opts.vyosBondVrrp } : {}),
+        ...(vrrpPriority !== undefined ? { vrrpPriority } : {}),
+        ...(vlans.length > 0 ? { vlans } : {}),
+        ...(opts.vyosPassword !== undefined && opts.vyosPassword !== ""
+          ? { password: opts.vyosPassword } : {}),
+        ...(opts.vyosHwid !== undefined && Object.keys(opts.vyosHwid).length > 0
+          ? { hwIds: opts.vyosHwid } : {}),
+        ...(opts.vyosMgmtVlan !== undefined && opts.vyosMgmtVlan !== ""
+          ? { mgmtVlan: parseVlan(opts.vyosMgmtVlan)[0] as VyosVlanSpec } : {}),
+        ...(opts.vyosFreshConfig === true ? { freshConfig: true } : {}),
+        ...(opts.vyosBundle !== undefined && opts.vyosBundle !== ""
+          ? { bundle: loadVyosBundle(opts.vyosBundle, hostname) } : {}),
+        ...(opts.vyosApiKey !== undefined && opts.vyosApiKey !== ""
+          ? { apiKey: opts.vyosApiKey } : {}),
+        ...(opts.vyosApiListen !== undefined && opts.vyosApiListen !== ""
+          ? { apiListenAddress: opts.vyosApiListen } : {}),
+      };
+      const hasVyosOptions = Object.keys(vyos).length > 0;
+
+      // A bundle already describes the whole router. Accepting derived topology
+      // flags alongside it would silently discard them (the bundle wins in
+      // buildVyosConfigSpec), so say so rather than appear to honour both.
+      if (vyos.bundle !== undefined) {
+        const derived = ["mgmtInterface", "mgmtAddress", "bondMembers", "bondAddress",
+                         "bondVrrp", "vrrpPriority", "vlans", "mgmtVlan"] as const;
+        const conflicting = derived.filter((k) => vyos[k] !== undefined);
+        if (conflicting.length > 0) {
+          console.error(
+            `--vyos-bundle describes the whole router; these would be ignored: ${conflicting.join(", ")}`,
+          );
+          console.error("Remove them, or change the bundle in kubernetes-deployment and re-render.");
+          process.exit(1);
+        }
+      }
+
+      if (hasVyosOptions && !opts.os.startsWith("vyos")) {
+        console.error(`VyOS options require --os vyos-rolling (got --os ${opts.os})`);
+        process.exit(1);
+      }
+
+      // Firmware PXE cannot run over LACP, so the NIC that boots the installer
+      // must stay out of the bond — otherwise the next reinstall has no path in.
+      const mgmt = vyos.mgmtInterface ?? "eth0";
+      if (bondMembers.includes(mgmt)) {
+        console.error(`--vyos-bond must not include the PXE/management interface "${mgmt}"`);
+        console.error("PXE cannot boot over an LACP bond; keep that NIC unbonded.");
+        process.exit(1);
+      }
+
       try {
         const result = await getLabdClient().installMachine({
+          ...(opts.schemaVersion ? { schemaVersion: Number(opts.schemaVersion) as 1 | 2 } : {}),
           mac,
           hostname,
           role: opts.role,
           os: opts.os,
           ...(opts.disk ? { disk: opts.disk } : {}),
+          ...(hasVyosOptions ? { vyos } : {}),
         });
 
         console.log(JSON.stringify(result, null, 2));
         console.log("");
-        const osLabel = opts.os.startsWith("ubuntu") ? "Ubuntu" : "Fedora";
+        const osLabel = opts.os.startsWith("ubuntu")
+          ? "Ubuntu"
+          : opts.os.startsWith("vyos") ? "VyOS" : "Fedora";
         console.log(`Power on the machine to start ${osLabel} installation.`);
 
         const roleInfo = ROLE_REGISTRY.find(r => r.name === opts.role);

bastion/src/cli/src/commands/recheck.ts

@@ -44,11 +44,14 @@ export function registerRecheckCommand(parent: Command): void {
       }
 
       // Build list of machines to check
-      const targets: Array<{ mac: string; hostname: string; ip: string }> = [];
+      const targets: Array<{ mac: string; hostname: string; ip: string; sshUser: string }> = [];
+      const userIsDefault = opts.user === "root";
       for (const [mac, info] of Object.entries(state.installed)) {
         if (!info.ip) continue;
         if (opts.target && info.hostname !== opts.target && mac !== opts.target) continue;
-        targets.push({ mac, hostname: info.hostname, ip: info.ip });
+        // VyOS boxes only have the "vyos" login; honor an explicit --user.
+        const sshUser = userIsDefault && (info.os ?? "").startsWith("vyos") ? "vyos" : opts.user;
+        targets.push({ mac, hostname: info.hostname, ip: info.ip, sshUser });
       }
 
       if (targets.length === 0) {
@@ -61,12 +64,12 @@ export function registerRecheckCommand(parent: Command): void {
       let updated = 0;
       let failed = 0;
 
-      for (const { mac, hostname, ip } of targets) {
+      for (const { mac, hostname, ip, sshUser } of targets) {
         process.stdout.write(`  ${hostname.padEnd(24)} ${DIM}(${ip})${RESET}  `);
 
         try {
           const t0 = Date.now();
-          const result = await sshExec(ip, opts.user, HW_COLLECT_SCRIPT, SSH_OPTS);
+          const result = await sshExec(ip, sshUser, HW_COLLECT_SCRIPT, SSH_OPTS);
           const elapsed = Date.now() - t0;
           if (result.exitCode !== 0) {
             console.log(`${RED}SSH failed (exit ${result.exitCode}, ${elapsed}ms)${RESET}`);

bastion/src/cli/src/commands/reprovision.ts

@@ -24,12 +24,12 @@ function roleTable(): string {
 function resolveTarget(
   target: string,
   state: BastionState,
-): { mac: string; hostname: string; ip: string } | null {
+): { mac: string; hostname: string; ip: string; os?: string } | null {
   const normalized = target.toLowerCase().replace(/-/g, ":");
 
   if (state.installed[normalized]) {
     const info = state.installed[normalized];
-    return { mac: normalized, hostname: info.hostname, ip: info.ip };
+    return { mac: normalized, hostname: info.hostname, ip: info.ip, ...(info.os !== undefined ? { os: info.os } : {}) };
   }
 
   if (state.discovered[normalized]) {
@@ -38,13 +38,13 @@ function resolveTarget(
 
   for (const [mac, info] of Object.entries(state.installed)) {
     if (info.hostname === target || info.hostname.startsWith(target + ".")) {
-      return { mac, hostname: info.hostname, ip: info.ip };
+      return { mac, hostname: info.hostname, ip: info.ip, ...(info.os !== undefined ? { os: info.os } : {}) };
     }
   }
 
   for (const [mac, info] of Object.entries(state.installed)) {
     if (info.ip === target) {
-      return { mac, hostname: info.hostname, ip: info.ip };
+      return { mac, hostname: info.hostname, ip: info.ip, ...(info.os !== undefined ? { os: info.os } : {}) };
     }
   }
 
@@ -60,10 +60,12 @@ export function registerReprovisionCommand(parent: Command): void {
     .addOption(new Option("--role <role>", "Machine role (see below)").choices([...SUPPORTED_ROLES]).default("worker"))
     .addOption(new Option("--os <os>", "Operating system").choices([...SUPPORTED_OS]).default("fedora-43"))
     .option("--disk <device>", "Target disk device (auto-detect if omitted)")
+    .option("--user <user>", "SSH user for the reboot (default: vyos for VyOS machines, else current user)")
     .action(async (target: string, hostnameOverride: string | undefined, opts: {
       role: string;
       os: string;
       disk?: string;
+      user?: string;
     }) => {
       if (!isValidOsId(opts.os)) {
         console.error(`Unknown OS: ${opts.os}. Supported: ${SUPPORTED_OS.join(", ")}`);
@@ -123,7 +125,11 @@ export function registerReprovisionCommand(parent: Command): void {
         return;
       }
 
-      const adminUser = process.env["SUDO_USER"] ?? process.env["USER"] ?? "";
+      // SSH user: explicit flag > the machine's current OS (VyOS boxes only
+      // have the "vyos" login) > the invoking user.
+      const currentOsIsVyos = (resolved.os ?? "").startsWith("vyos");
+      const adminUser = opts.user
+        ?? (currentOsIsVyos ? "vyos" : (process.env["SUDO_USER"] ?? process.env["USER"] ?? ""));
       const effectiveUser = adminUser === "root" ? "" : adminUser;
 
       if (effectiveUser === "") {

bastion/src/cli/tests/install-vyos.test.ts

@@ -0,0 +1,35 @@
+// Tests for VyOS install option parsing.
+
+import { describe, it, expect } from "vitest";
+import { parseVlan } from "../src/commands/install.js";
+
+describe("parseVlan", () => {
+  it("parses id and CIDR", () => {
+    expect(parseVlan("10:10.0.10.1/24")).toEqual([{ id: 10, address: "10.0.10.1/24" }]);
+  });
+
+  it("accumulates across repeated flags", () => {
+    const first = parseVlan("10:10.0.10.1/24");
+    const both = parseVlan("20:10.0.20.1/24", first);
+    expect(both).toHaveLength(2);
+    expect(both[1]).toEqual({ id: 20, address: "10.0.20.1/24" });
+  });
+
+  it("keeps a description, including one containing colons", () => {
+    expect(parseVlan("30:10.0.30.1/24:mgmt:secondary")).toEqual([
+      { id: 30, address: "10.0.30.1/24", description: "mgmt:secondary" },
+    ]);
+  });
+
+  it("rejects an address that is not CIDR", () => {
+    // A bare address would produce a VyOS config that fails to commit on first
+    // boot, long after the operator has stopped watching.
+    expect(() => parseVlan("10:10.0.10.1")).toThrow(/CIDR/);
+  });
+
+  it("rejects out-of-range and non-numeric VLAN ids", () => {
+    expect(() => parseVlan("0:10.0.10.1/24")).toThrow(/1-4094/);
+    expect(() => parseVlan("4095:10.0.10.1/24")).toThrow(/1-4094/);
+    expect(() => parseVlan("abc:10.0.10.1/24")).toThrow(/1-4094/);
+  });
+});

bastion/src/labd/src/routes/bastions.ts

@@ -10,6 +10,7 @@ import type { FastifyInstance } from "fastify";
 import type { DbClient } from "../server.js";
 import { bastionRegistry } from "../services/bastion-registry.js";
 import { generateRequestId } from "@lab/shared";
+import type { VyosInstallSpec } from "@lab/shared";
 
 const COMMAND_TIMEOUT_MS = 15_000;
 
@@ -163,9 +164,9 @@ export function registerBastionRoutes(app: FastifyInstance, db: DbClient): void
 
   // Queue install — route to correct bastion by MAC
   app.post<{
-    Body: { mac?: string; hostname?: string; disk?: string; role?: string; os?: string };
+    Body: { mac?: string; hostname?: string; disk?: string; role?: string; os?: string; vyos?: VyosInstallSpec };
   }>("/api/machines/install", async (request, reply) => {
-    const { mac, hostname, disk, role, os } = request.body ?? {};
+    const { mac, hostname, disk, role, os, vyos } = request.body ?? {};
     if (!mac || !hostname) {
       return reply.code(400).send({ error: "mac and hostname are required" });
     }
@@ -183,6 +184,7 @@ export function registerBastionRoutes(app: FastifyInstance, db: DbClient): void
           const result = await sendCommand(all[0]!.bastionId, {
             type: "command-install",
             mac, hostname, disk: disk ?? "", role: role ?? "infra", os: os ?? "fedora-43",
+            ...(vyos ? { vyos } : {}),
           });
           return reply.code(result.status === "ok" ? 200 : 500).send(result);
         } catch (err) {
@@ -196,6 +198,7 @@ export function registerBastionRoutes(app: FastifyInstance, db: DbClient): void
       const result = await sendCommand(bastion.bastionId, {
         type: "command-install",
         mac, hostname, disk: disk ?? "", role: role ?? "infra", os: os ?? "fedora-43",
+        ...(vyos ? { vyos } : {}),
       });
       return reply.code(result.status === "ok" ? 200 : 500).send(result);
     } catch (err) {

bastion/src/modules/modules/k3s/bin/render-config.ts

@@ -0,0 +1,42 @@
+#!/usr/bin/env node
+// Render /etc/rancher/k3s/config.yaml using the PRODUCTION generator.
+//
+// Exists so labsim (and anything else) can produce the exact config.yaml a node
+// would get from `labctl install`, without an SSH/OperationContext. Driving the
+// rehearsal through this means the sim tests the same code path production runs
+// -- if generateServerConfig ever changes shape, the sim moves with it.
+//
+// All input via env, so a sim's cloud-init or a shell can call it plainly:
+//
+//   ROLE=infra HOSTNAME=k8s1 IP=172.31.2.11 render-config.ts        # cluster-init server
+//   ROLE=infra HOSTNAME=k8s2 IP=172.31.2.12 \
+//     K3S_SERVER_URL=https://172.31.2.11:6443 K3S_TOKEN=... render-config.ts   # joining server
+//   ROLE=worker HOSTNAME=k8s4 IP=172.31.2.14 K3S_SERVER_URL=... K3S_TOKEN=... render-config.ts  # agent
+//
+// Dual-stack is opt-in and matches K3sConfig exactly: set IPV6, CLUSTER_CIDR,
+// SERVICE_CIDR (comma-separated families) and the generator emits the dual
+// node-ip + CIDRs. Omit them and the output is byte-identical to a v4-only node.
+import { generateServerConfig, generateAgentConfig } from "../src/operations/k3s-config.js";
+import type { K3sConfig } from "../src/types.js";
+import type { Role } from "@lab/shared";
+
+const env = process.env;
+const role = (env.ROLE ?? "worker") as Role;
+const isServer = role === "infra" || role === "labcontroller";
+
+const splitCsv = (v: string | undefined): string[] | undefined =>
+  v ? v.split(",").map((s) => s.trim()).filter(Boolean) : undefined;
+
+const cfg: K3sConfig = {
+  hostname: env.HOSTNAME ?? "node",
+  ip: env.IP ?? (() => { throw new Error("IP is required"); })(),
+  role,
+  k3sServerUrl: env.K3S_SERVER_URL,
+  k3sToken: env.K3S_TOKEN,
+  tlsSans: splitCsv(env.TLS_SANS),
+  ipv6: env.IPV6,
+  clusterCidr: splitCsv(env.CLUSTER_CIDR),
+  serviceCidr: splitCsv(env.SERVICE_CIDR),
+};
+
+process.stdout.write(isServer ? generateServerConfig(cfg) : generateAgentConfig(cfg));

bastion/src/modules/modules/k3s/src/groups/admit.ts

@@ -0,0 +1,31 @@
+// Return an evacuated node to service. The inverse of the evacuate group.
+//
+// Needed for two different journeys that end in the same place:
+//   - a node that was wiped, reinstalled and has just rejoined the cluster;
+//   - a node whose evacuation was aborted and must simply be put back.
+//
+// Order matters here too. configureLonghornDisk runs AFTER eviction is cleared,
+// because a node that is still marked for eviction will have any disk we give
+// it immediately drained again.
+
+import type { OperationContext, OperationResult, OperationGroup } from "../types.js";
+import { runSequential } from "../utils.js";
+import { uncordonNode } from "../operations/node-evacuate.js";
+import { clearLonghornEviction, assertNodeSchedulable } from "../operations/node-admit.js";
+import { configureLonghornDisk } from "../operations/longhorn-disk.js";
+
+export const admitGroup: OperationGroup = {
+  name: "admit",
+  description: "Uncordon a node, clear its eviction request and restore its Longhorn disk",
+  operations: [
+    { name: "Uncordon node", fn: uncordonNode },
+    { name: "Clear Longhorn eviction", fn: clearLonghornEviction },
+    { name: "Restore Longhorn default disk", fn: configureLonghornDisk },
+    { name: "Verify node is back in service", fn: assertNodeSchedulable },
+  ],
+};
+
+export async function runAdmit(ctx: OperationContext): Promise<OperationResult[]> {
+  ctx.log("Returning node to service...");
+  return runSequential(ctx, admitGroup.operations);
+}

bastion/src/modules/modules/k3s/src/groups/evacuate.ts

@@ -0,0 +1,54 @@
+// Evacuate a node so it can be wiped and reinstalled with a different layout.
+//
+// The ORDER is the operation:
+//
+//   1. Cordon      stop new work landing on a node that is about to leave.
+//   2. Check       can eviction even finish? Seconds to answer, an hour to
+//                  discover the hard way.
+//   3. Evict       move the replicas and WAIT for them to rebuild elsewhere.
+//   4. Drain       now move the workloads.
+//   5. Verify      re-read the cluster; do not trust that 1-4 worked.
+//
+// EVICT BEFORE DRAIN, which is the opposite of the obvious order and the one
+// thing here most likely to be "tidied" back into a bug. `drain --force` does
+// not override PodDisruptionBudgets, and longhorn-manager holds a PDB on the
+// instance-manager pod for as long as volumes are attached or degraded. Drain
+// first and you stop the instance-manager that serves the replicas you are
+// about to evict: the drain blocks for its full 15-minute timeout, and if it
+// did succeed it would have removed the only source a single-replica volume
+// could rebuild from.
+//
+// Removing the etcd member is NOT part of this group -- it costs you the API
+// server the rest of the procedure needs, so it belongs immediately before the
+// wipe, not here.
+//
+// This group deliberately does not reinstall anything. It leaves the node
+// cordoned, drained and empty, and a human (or `labctl provision reprovision`)
+// takes it from there.
+
+import type { OperationContext, OperationResult, OperationGroup } from "../types.js";
+import { runSequential } from "../utils.js";
+import {
+  cordonNode,
+  drainNode,
+  evictLonghornReplicas,
+  assertEvictionCompletable,
+  assertNodeEvacuated,
+} from "../operations/node-evacuate.js";
+
+export const evacuateGroup: OperationGroup = {
+  name: "evacuate",
+  description: "Cordon, evict storage from and drain a node so it can be safely wiped",
+  operations: [
+    { name: "Cordon node", fn: cordonNode },
+    { name: "Check eviction can complete", fn: assertEvictionCompletable },
+    { name: "Evict Longhorn replicas (waits for rebuild)", fn: evictLonghornReplicas },
+    { name: "Drain workloads", fn: drainNode },
+    { name: "Verify node is empty", fn: assertNodeEvacuated },
+  ],
+};
+
+export async function runEvacuate(ctx: OperationContext): Promise<OperationResult[]> {
+  ctx.log("Evacuating node before wipe...");
+  return runSequential(ctx, evacuateGroup.operations);
+}

bastion/src/modules/modules/k3s/src/groups/host-prep.ts

@@ -1,21 +1,23 @@
-// Host preparation: kernel modules, sysctl, swap, firewall, SELinux.
+// Host preparation: kernel modules, sysctl, swap, storage, firewall, SELinux.
 
 import type { OperationContext, OperationResult, OperationGroup } from "../types.js";
 import { runSequential } from "../utils.js";
 import { loadKernelModules } from "../operations/kernel-modules.js";
 import { applyCisHardening } from "../operations/sysctl.js";
-import { disableSwap } from "../operations/swap.js";
+import { enableSwap } from "../operations/swap.js";
+import { growRancherLv } from "../operations/rancher-storage.js";
 import { disableFirewall } from "../operations/firewall.js";
 import { setSelinuxPermissive } from "../operations/selinux.js";
 import { enableIscsi } from "../operations/iscsi.js";
 
 export const hostPrepGroup: OperationGroup = {
   name: "host-prep",
-  description: "Prepare host for k3s: kernel modules, sysctl, swap, firewall, SELinux, iSCSI",
+  description: "Prepare host for k3s: kernel modules, sysctl, swap, imageFs sizing, firewall, SELinux, iSCSI",
   operations: [
     { name: "Load kernel modules", fn: loadKernelModules },
     { name: "Apply CIS sysctl", fn: applyCisHardening },
-    { name: "Disable swap", fn: disableSwap },
+    { name: "Enable swap", fn: enableSwap },
+    { name: "Grow rancher LV", fn: growRancherLv },
     { name: "Disable firewall", fn: disableFirewall },
     { name: "Set SELinux permissive", fn: setSelinuxPermissive },
     { name: "Enable iSCSI", fn: enableIscsi },

bastion/src/modules/modules/k3s/src/groups/k3s-agent.ts

@@ -4,6 +4,7 @@ import type { OperationContext, OperationResult, OperationGroup } from "../types
 import { runSequential } from "../utils.js";
 import { writeK3sConfig } from "../operations/k3s-config.js";
 import { installK3sBinary } from "../operations/k3s-install.js";
+import { labelWorkerRole } from "../operations/worker-label.js";
 
 export const k3sAgentGroup: OperationGroup = {
   name: "k3s-agent",
@@ -11,6 +12,8 @@ export const k3sAgentGroup: OperationGroup = {
   operations: [
     { name: "Write k3s config", fn: writeK3sConfig },
     { name: "Install k3s agent", fn: installK3sBinary },
+    // After registration, never before: the kubelet may not set this itself.
+    { name: "Label node as worker", fn: labelWorkerRole },
   ],
 };
 

bastion/src/modules/modules/k3s/src/health/k3s-service.ts

@@ -1,8 +1,9 @@
 import type { Operation, OperationResult } from "../types.js";
 import { isServiceActive } from "../utils.js";
+import { isK3sServer } from "@lab/shared";
 
 export const checkK3sService: Operation = async (ctx): Promise<OperationResult> => {
-  const isServer = ctx.config.role === "infra" || ctx.config.role === "labcontroller";
+  const isServer = isK3sServer(ctx.config.role);
   const service = isServer ? "k3s" : "k3s-agent";
   const active = await isServiceActive(ctx, service);
   return { success: active, changed: false, message: active ? `${service} is active` : `${service} is not active` };

bastion/src/modules/modules/k3s/src/install.ts

@@ -1,17 +1,21 @@
 // k3s module — install phase.
 // Installs k3s with CIS-hardened configuration and Cilium CNI.
 
+import { isK3sServer, type Role } from "@lab/shared";
+
 export interface K3sInstallContext {
   hostname: string;
   ip: string;
-  role: string;   // "infra" = server, "worker" = agent
+  role: Role;     // server vs agent is derived via isK3sServer(), never by name
   k3sServerUrl?: string;  // Required for agent role
   k3sToken?: string;       // Required for agent role
 }
 
 /** Generate the shell script that installs k3s on a target machine. */
 export function generateInstallScript(ctx: K3sInstallContext): string {
-  const isServer = ctx.role === "infra";
+  // Was `ctx.role === "infra"`, which silently treated a `labcontroller` -- a
+  // role that runs a k3s SERVER -- as an agent. Derived from the registry now.
+  const isServer = isK3sServer(ctx.role);
 
   return `#!/bin/bash
 set -euo pipefail
@@ -215,7 +219,9 @@ echo "  Using network device: \$DEFAULT_DEV"
 
 KUBECONFIG=/etc/rancher/k3s/k3s.yaml cilium install \\
   --set kubeProxyReplacement=true \\
-  --set ipam.mode=kubernetes \\
+  --set ipam.mode=cluster-pool \\
+  --set ipam.operator.clusterPoolIPv4PodCIDRList='{10.42.0.0/16}' \\
+  --set ipam.operator.clusterPoolIPv4MaskSize=24 \\
   --set devices="\$DEFAULT_DEV" \\
   --set nodePort.directRoutingDevice="\$DEFAULT_DEV"
 

bastion/src/modules/modules/k3s/src/k3s-module.ts

@@ -6,21 +6,35 @@ import type { OperationContext, K3sConfig, OperationResult } from "./types.js";
 import { sshExec } from "../../../src/ssh.js";
 import { aggregateResults } from "./utils.js";
 import { runHostPrep } from "./groups/host-prep.js";
+import { runEvacuate } from "./groups/evacuate.js";
+import { runAdmit } from "./groups/admit.js";
 import { runK3sServer } from "./groups/k3s-server.js";
 import { runK3sAgent } from "./groups/k3s-agent.js";
 import { runNetworking } from "./groups/networking.js";
 import { runHardening } from "./groups/hardening.js";
 import { runSequential } from "./utils.js";
 import * as health from "./health/index.js";
+import { isK3sServer } from "@lab/shared";
 
 function toOpContext(ctx: ModuleContext): OperationContext {
+  // Pass the WHOLE config bag through, then pin the fields that come from the
+  // ModuleContext itself.
+  //
+  // This used to enumerate three keys -- k3sServerUrl, k3sToken, tlsSans -- and
+  // silently drop every other one. Anything a caller set beyond those three
+  // vanished with no error and no warning: k3sVersion (so a pinned version was
+  // ignored and get.k3s.io installed LATEST), evictionAllowSingleReplica (so
+  // `labctl app k3s evacuate --allow-single-replica` was a no-op that claimed
+  // to override a data-loss guard), and the dual-stack fields ipv6 /
+  // clusterCidr / serviceCidr.
+  //
+  // Enumerating is the bug: every field added to K3sConfig has to be
+  // remembered here, and forgetting produces silence rather than a type error.
   const config: K3sConfig = {
+    ...(ctx.config as Partial<K3sConfig>),
     hostname: ctx.hostname,
     ip: ctx.ip,
     role: ctx.role as K3sConfig["role"],
-    k3sServerUrl: ctx.config["k3sServerUrl"] as string | undefined,
-    k3sToken: ctx.config["k3sToken"] as string | undefined,
-    tlsSans: ctx.config["tlsSans"] as string[] | undefined,
   };
   return {
     config,
@@ -35,7 +49,16 @@ function toOpContext(ctx: ModuleContext): OperationContext {
     },
     os: ctx.os,
     arch: ctx.arch,
-    log: (_msg) => { /* collected via results */ },
+    // Emit progress LIVE, on stderr.
+    //
+    // This used to be a no-op "collected via results", but results only carry
+    // an operation's final message. Anything an operation reports WHILE it
+    // runs was therefore thrown away -- including evictLonghornReplicas's
+    // per-poll "N replica(s) still on <node>", which is the only visibility
+    // into an eviction that can legitimately take an hour. When one stalled,
+    // there was no way to tell which volumes were stuck or whether progress
+    // was being made at all.
+    log: (msg) => { console.error(`  [${ctx.hostname}] ${msg}`); },  // eslint-disable-line no-console
   };
 }
 
@@ -62,7 +85,7 @@ export class K3sModule implements Module {
   async install(ctx: ModuleContext): Promise<ModuleResult> {
     const start = performance.now();
     const opCtx = toOpContext(ctx);
-    const isServer = ctx.role === "infra" || ctx.role === "labcontroller";
+    const isServer = isK3sServer(ctx.role);
 
     // Phase 1: Host preparation
     const prepResults = await runHostPrep(opCtx);
@@ -95,6 +118,31 @@ export class K3sModule implements Module {
     return toModuleResult("configure", results, start);
   }
 
+  /**
+   * Evacuate the node so it can be wiped. Does NOT reinstall anything --
+   * it leaves the node cordoned, drained and empty.
+   */
+  async evacuate(ctx: ModuleContext): Promise<ModuleResult> {
+    const start = performance.now();
+    const opCtx = toOpContext(ctx);
+    const results = await runEvacuate(opCtx);
+    return toModuleResult("evacuate", results, start);
+  }
+
+  /**
+   * Return an evacuated node to service: uncordon, clear Longhorn's eviction
+   * request, restore its default disk.
+   *
+   * The counterpart to evacuate(), and the only supported way out of an
+   * evacuation that was aborted partway.
+   */
+  async admit(ctx: ModuleContext): Promise<ModuleResult> {
+    const start = performance.now();
+    const opCtx = toOpContext(ctx);
+    const results = await runAdmit(opCtx);
+    return toModuleResult("admit", results, start);
+  }
+
   async health(ctx: ModuleContext): Promise<ModuleResult> {
     const start = performance.now();
     const opCtx = toOpContext(ctx);

bastion/src/modules/modules/k3s/src/operations/cilium.ts

@@ -47,7 +47,9 @@ export const installCilium: Operation = async (ctx): Promise<OperationResult> =>
   const installResult = await ctx.ssh.exec(
     `KUBECONFIG=/etc/rancher/k3s/k3s.yaml cilium install \
       --set kubeProxyReplacement=true \
-      --set ipam.mode=kubernetes \
+      --set ipam.mode=cluster-pool \
+      --set ipam.operator.clusterPoolIPv4PodCIDRList='{10.42.0.0/16}' \
+      --set ipam.operator.clusterPoolIPv4MaskSize=24 \
       --set k8sServiceHost=127.0.0.1 \
       --set k8sServicePort=6444 \
       --set cni.exclusive=false \

bastion/src/modules/modules/k3s/src/operations/etcd-members.ts

@@ -0,0 +1,60 @@
+// Parsing `etcdctl member list`.
+//
+// WHY NOT `-w json`
+// -----------------
+// Two reasons, and the second is the dangerous one:
+//
+//  1. `member remove` requires the id in HEX. The JSON writer emits decimal.
+//  2. etcd member ids are uint64. JSON.parse turns them into JS numbers, which
+//     lose precision above 2^53 -- so the id that comes back is an
+//     APPROXIMATION of the real one.
+//
+// Observed for real: `member remove 13323819137391910000` ->
+// "bad member ID arg (strconv.ParseUint: value out of range), expecting ID in
+// Hex". That happens at the worst possible moment, with the node already
+// stopped and about to be wiped. Failing loudly is the lucky outcome; a
+// mangled id that happened to match another member would remove the WRONG one.
+//
+// The default writer prints the id as hex TEXT, so it never becomes a number.
+
+export interface EtcdMember {
+  /** Raw member name: "<node>-<8 hex>", e.g. "worker0.lab-e26aacc5". */
+  name: string;
+  /** The node name with etcd's random suffix stripped. */
+  node: string;
+  /** Hex id, exactly as etcd prints it — never round-tripped through a number. */
+  id: string;
+  started: boolean;
+}
+
+/**
+ * etcd names a member `<node>-<8 hex>`. Comparing a raw member name against a
+ * node name without stripping that suffix silently never matches.
+ */
+export function etcdNodeName(memberName: string): string {
+  return memberName.replace(/-[0-9a-f]{8}$/, "");
+}
+
+/**
+ * Parse the default `etcdctl member list` writer:
+ *
+ *   825852d3683bc406, started, worker2.lab-5f7ad0c9, https://..:2380, https://..:2379, false
+ *   <hex id>        , status , <node>-<8 hex>      , peer           , client         , learner
+ */
+export function parseEtcdMembers(out: string): EtcdMember[] {
+  return out.split("\n")
+    .map((l) => l.trim())
+    .filter((l) => l !== "")
+    .map((line) => {
+      const cols = line.split(",").map((c) => c.trim());
+      const name = cols[2] ?? "";
+      return {
+        name,
+        node: etcdNodeName(name),
+        id: cols[0] ?? "",
+        started: (cols[1] ?? "") === "started",
+      };
+    })
+    // Drop anything without a hex id: blank lines, warnings, stray output.
+    .filter((m) => /^[0-9a-f]+$/i.test(m.id));
+}

bastion/src/modules/modules/k3s/src/operations/index.ts

@@ -1,10 +1,11 @@
 export { loadKernelModules } from "./kernel-modules.js";
 export { applyCisHardening } from "./sysctl.js";
-export { disableSwap } from "./swap.js";
+export { enableSwap } from "./swap.js";
+export { growRancherLv } from "./rancher-storage.js";
 export { enableIscsi } from "./iscsi.js";
 export { disableFirewall } from "./firewall.js";
 export { setSelinuxPermissive } from "./selinux.js";
-export { writeK3sConfig } from "./k3s-config.js";
+export { writeK3sConfig, generateServerConfig, generateAgentConfig } from "./k3s-config.js";
 export { writeAuditPolicy } from "./audit-policy.js";
 export { cleanupStaleCni } from "./cni-cleanup.js";
 export { installK3sBinary } from "./k3s-install.js";
@@ -23,3 +24,10 @@ export type {
   RecoverEtcdMemberOptions,
   RecoverEtcdMemberResult,
 } from "./etcd-recover.js";
+export {
+  cordonNode, uncordonNode, drainNode, evictLonghornReplicas,
+  assertEvictionCompletable, assertNodeEvacuated, checkEvictionCompletable,
+} from "./node-evacuate.js";
+export type { EvictionFeasibility } from "./node-evacuate.js";
+export { clearLonghornEviction, assertNodeSchedulable } from "./node-admit.js";
+export { labelWorkerRole } from "./worker-label.js";

bastion/src/modules/modules/k3s/src/operations/k3s-config.ts

@@ -2,13 +2,54 @@
 
 import type { Operation, OperationResult, K3sConfig } from "../types.js";
 import { sshOpts, writeRemoteFile } from "../utils.js";
+import { isK3sServer } from "@lab/shared";
 
-function isServerRole(role: string): boolean {
-  return role === "infra" || role === "labcontroller";
+/**
+ * The address-family block: `cluster-cidr`, `service-cidr` and `node-ip`.
+ *
+ * Emitted ONLY when the corresponding config is supplied, and that is
+ * deliberate. With no `ipv6` and no CIDRs this returns "", so the generated
+ * file is byte-identical to what every existing node already has -- no diff,
+ * so `writeRemoteFile` reports unchanged and nothing restarts k3s. Dual-stack
+ * is therefore opt-in per node rather than a flag day.
+ *
+ * `node-ip` is only written once there is a second family to name. k3s
+ * auto-detects a sensible IPv4 on its own, and writing it out unconditionally
+ * would rewrite the config of five healthy nodes to tell them what they had
+ * already worked out.
+ */
+function addressFamilyLines(config: K3sConfig, opts: { cidrs: boolean }): string {
+  const lines: string[] = [];
+  if (opts.cidrs && config.clusterCidr?.length) {
+    lines.push(`cluster-cidr: "${config.clusterCidr.join(",")}"`);
+  }
+  if (opts.cidrs && config.serviceCidr?.length) {
+    lines.push(`service-cidr: "${config.serviceCidr.join(",")}"`);
+  }
+  if (config.ipv6) {
+    // Order matters to k3s: the FIRST entry is the primary family, and the
+    // supported single-to-dual-stack conversion is the one that preserves it.
+    // IPv4 stays primary so existing Services keep their ClusterIP.
+    lines.push(`node-ip: "${config.ip},${config.ipv6}"`);
+  }
+  return lines.length ? `${lines.join("\n")}\n` : "";
 }
 
-function generateServerConfig(config: K3sConfig): string {
-  const tlsSans = [config.hostname, config.ip, ...(config.tlsSans ?? [])];
+// Exported so the exact production config.yaml can be rendered outside an SSH
+// context -- notably by the labsim 3-server-etcd rehearsal, which must drive its
+// nodes through THIS generator rather than a parallel set of INSTALL_K3S_EXEC
+// flags, or it proves a mechanism production does not run.
+export function generateServerConfig(config: K3sConfig): string {
+  // The IPv6 address goes in the cert too. Without it, anything that reaches
+  // this apiserver over v6 -- a peer server joining, or kubectl against the v6
+  // address -- fails TLS verification, and the error names the certificate
+  // rather than the missing SAN, which is a long way from the cause.
+  const tlsSans = [
+    config.hostname,
+    config.ip,
+    ...(config.ipv6 ? [config.ipv6] : []),
+    ...(config.tlsSans ?? []),
+  ];
   const isJoining = !!config.k3sServerUrl;
   const clusterLines = isJoining
     ? `server: "${config.k3sServerUrl}"\ntoken: "${config.k3sToken}"`
@@ -21,7 +62,7 @@ function generateServerConfig(config: K3sConfig): string {
   // and never expire.
   return `# k3s server configuration — CIS hardened, etcd HA
 ${clusterLines}
-protect-kernel-defaults: true
+${addressFamilyLines(config, { cidrs: true })}protect-kernel-defaults: true
 secrets-encryption: true
 write-kubeconfig-mode: "0640"
 
@@ -51,10 +92,34 @@ ${tlsSans.map((s) => `  - "${s}"`).join("\n")}
 `;
 }
 
-function generateAgentConfig(): string {
-  return `protect-kernel-defaults: true
+// Takes the config now: an agent needs its own dual `node-ip` just as much as a
+// server does. Without one it joins as an IPv4-only node into a dual-stack
+// cluster, gets no IPv6 pod CIDR, and the failure surfaces later as pods on that
+// node being unreachable over v6 while the node itself reads Ready.
+// It takes no cluster/service CIDRs -- those are server-side only.
+export function generateAgentConfig(config: K3sConfig): string {
+  // NO node-role.kubernetes.io/worker LABEL HERE, deliberately.
+  //
+  // A kubelet is not permitted to assign itself labels in the
+  // `node-role.kubernetes.io/` namespace -- the servers run with
+  // `enable-admission-plugins=NodeRestriction`, which rejects the registration
+  // outright. The agent then never finishes registering, k3s-agent.service
+  // never signals readiness, and systemd reports:
+  //
+  //   Job for k3s-agent.service failed because the service did not take the
+  //   steps required by its unit configuration.
+  //
+  // which says nothing about labels and sends you looking at the network, the
+  // token, or the kubelet's sysctls. Every agent installed through this module
+  // failed this way; the estate's existing workers only escaped it because
+  // post-provision installs them with no config.yaml at all and their role
+  // label was applied separately, server-side.
+  //
+  // `node.longhorn.io/...` is fine: it is outside the kubernetes.io namespace,
+  // so the kubelet may set it. The worker role label is applied after the node
+  // registers, by labelWorkerRole below.
+  return `${addressFamilyLines(config, { cidrs: false })}protect-kernel-defaults: true
 node-label:
-  - "node-role.kubernetes.io/worker=true"
   - "node.longhorn.io/create-default-disk=config"
 kubelet-arg:
   - "protect-kernel-defaults=true"
@@ -64,11 +129,41 @@ kubelet-arg:
 }
 
 export const writeK3sConfig: Operation = async (ctx): Promise<OperationResult> => {
+  // Refuse to name an address the node does not have.
+  //
+  // Most of this estate is SSH-onboard, not PXE-provisioned: Asahi cannot PXE
+  // at all, and the DGX Sparks run NVIDIA's own OS and must never be
+  // reinstalled. For those nodes the install templates govern nothing and this
+  // module is the ONLY labctl touchpoint, so nothing upstream can guarantee the
+  // vendor OS actually took a DHCPv6 lease.
+  //
+  // Writing node-ip for a missing address does not fail here -- it fails later,
+  // when k3s will not start, with an error about binding rather than about
+  // addressing. Checking costs one ssh round trip and turns a confusing
+  // start-up failure into a sentence naming the address and the node.
+  if (ctx.config.ipv6) {
+    const probe = await ctx.ssh.exec(
+      `ip -6 -o addr show 2>/dev/null | grep -qF " ${ctx.config.ipv6}/" && echo present || true`,
+      sshOpts(ctx),
+    );
+    if (!probe.stdout.includes("present")) {
+      return {
+        success: false,
+        changed: false,
+        message: `Node does not have IPv6 ${ctx.config.ipv6}`,
+        error:
+          `k3s config would set node-ip to ${ctx.config.ipv6}, but that address is not on any ` +
+          `interface. k3s resolves node-ip at start-up and would fail to bind. Check the node ` +
+          `took its DHCPv6 lease (a kea reservation keyed on its MAC) before retrying.`,
+      };
+    }
+  }
+
   await ctx.ssh.exec("mkdir -p /etc/rancher/k3s", sshOpts(ctx));
 
-  const content = isServerRole(ctx.config.role)
+  const content = isK3sServer(ctx.config.role)
     ? generateServerConfig(ctx.config)
-    : generateAgentConfig();
+    : generateAgentConfig(ctx.config);
 
   const changed = await writeRemoteFile(ctx, "/etc/rancher/k3s/config.yaml", content);
 

bastion/src/modules/modules/k3s/src/operations/k3s-install.ts

@@ -2,13 +2,19 @@
 
 import type { Operation, OperationResult } from "../types.js";
 import { sshOpts } from "../utils.js";
-
-function isServerRole(role: string): boolean {
-  return role === "infra" || role === "labcontroller";
-}
+import { isK3sServer } from "@lab/shared";
 
 export const installK3sBinary: Operation = async (ctx): Promise<OperationResult> => {
-  const isServer = isServerRole(ctx.config.role);
+  const isServer = isK3sServer(ctx.config.role);
+
+  // Pin the version when asked. get.k3s.io installs LATEST by default, so a
+  // node reinstalled during a rolling migration comes back on whatever shipped
+  // since -- potentially several minor versions ahead of the cluster it is
+  // about to rejoin, which Kubernetes does not support between control-plane
+  // members.
+  const versionEnv = ctx.config.k3sVersion !== undefined && ctx.config.k3sVersion !== ""
+    ? `INSTALL_K3S_VERSION="${ctx.config.k3sVersion}" `
+    : "";
 
   // Check if already installed
   const version = await ctx.ssh.exec("k3s --version 2>/dev/null", sshOpts(ctx));
@@ -24,12 +30,21 @@ export const installK3sBinary: Operation = async (ctx): Promise<OperationResult>
       );
     }
 
-    // If joining an existing cluster, pass K3S_URL and K3S_TOKEN
-    const joinEnv = ctx.config.k3sServerUrl && ctx.config.k3sToken
-      ? `K3S_URL="${ctx.config.k3sServerUrl}" K3S_TOKEN="${ctx.config.k3sToken}"`
-      : "";
+    // Token and URL are independent. A token WITHOUT a URL seeds the cluster
+    // token on the cluster-init node, so every later joiner can be given the
+    // same pre-shared value instead of scraping node-token off the first
+    // server -- which matters here because that first server gets wiped and
+    // rejoined during a rolling migration. URL only counts when a token comes
+    // with it; a URL alone cannot authenticate a join.
+    //
+    // The token deliberately does NOT go in config.yaml: writeRemoteFile diffs
+    // that file, so a token appearing there would restart k3s on every node.
+    const joinEnv = [
+      ctx.config.k3sServerUrl && ctx.config.k3sToken ? `K3S_URL="${ctx.config.k3sServerUrl}"` : "",
+      ctx.config.k3sToken ? `K3S_TOKEN="${ctx.config.k3sToken}"` : "",
+    ].filter(Boolean).join(" ");
     const result = await ctx.ssh.exec(
-      `curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server" INSTALL_K3S_SKIP_SELINUX_RPM=true ${joinEnv} sh -`,
+      `curl -sfL https://get.k3s.io | ${versionEnv}INSTALL_K3S_EXEC="server" INSTALL_K3S_SKIP_SELINUX_RPM=true ${joinEnv} sh -`,
       { timeoutMs: 300_000 },
     );
     if (result.exitCode !== 0) {
@@ -50,7 +65,7 @@ export const installK3sBinary: Operation = async (ctx): Promise<OperationResult>
       };
     }
     const result = await ctx.ssh.exec(
-      `curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="agent" K3S_URL="${ctx.config.k3sServerUrl}" K3S_TOKEN="${ctx.config.k3sToken}" sh -`,
+      `curl -sfL https://get.k3s.io | ${versionEnv}INSTALL_K3S_EXEC="agent" K3S_URL="${ctx.config.k3sServerUrl}" K3S_TOKEN="${ctx.config.k3sToken}" sh -`,
       { timeoutMs: 300_000 },
     );
     if (result.exitCode !== 0) {

bastion/src/modules/modules/k3s/src/operations/longhorn-disk.ts

@@ -1,50 +1,206 @@
-// Annotate nodes with Longhorn default disk config when /var/lib/longhorn exists.
-// The label is set in k3s config (node-label), but the annotation must be applied via kubectl.
+// Give a node a Longhorn disk at /var/lib/longhorn.
+//
+// The cluster runs `createDefaultDiskLabeledNodes: true`, which means Longhorn
+// creates a disk ONLY on nodes carrying the label
+// `node.longhorn.io/create-default-disk=config` (set via k3s node-label) AND
+// the annotation `node.longhorn.io/default-disks-config` (set here). Miss
+// either and Longhorn gives the node no disk at all: it joins, goes Ready,
+// accepts pods, and silently hosts zero replicas.
+//
+// WHY THERE IS A REPAIR PATH
+// --------------------------
+// Longhorn reads that annotation when it FIRST creates the node's
+// `nodes.longhorn.io` record. This operation runs during configure(), which on
+// a rejoining node is after the record already exists -- so the annotation
+// lands too late and is never read. That is invisible until a rebuild is
+// expected and never arrives, roughly forty minutes into a migration.
+//
+// So: annotate, then read back, and if there is still no disk, write it
+// directly into spec.disks. When Longhorn does honour the annotation the
+// repair is a no-op.
 
-import type { Operation, OperationResult } from "../types.js";
+import type { Operation, OperationContext, OperationResult } from "../types.js";
 import { sshOpts } from "../utils.js";
 import { sshExec as remoteSshExec } from "../../../../src/ssh.js";
 
+const DISK_PATH = "/var/lib/longhorn";
+/** Fixed key so a repeated repair updates one disk rather than adding more. */
+const DISK_KEY = "default-disk-lab";
+const VERIFY_INTERVAL_MS = 5_000;
+/** How long to wait for Longhorn to create the node's record after it joins. */
+const REGISTER_TIMEOUT_MS = 180_000;
+/** How long to then wait for Longhorn to give that record a disk. */
+const DISK_TIMEOUT_MS = 60_000;
+
+const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
+
+/**
+ * Run kubectl from wherever it can be reached: locally on a server, otherwise
+ * over SSH to the configured server (an agent has no local kubectl).
+ */
+async function kubectlAnywhere(
+  ctx: OperationContext,
+  args: string,
+): Promise<{ ok: boolean; out: string }> {
+  const local = await ctx.ssh.exec(`k3s kubectl ${args} 2>&1`, sshOpts(ctx));
+  if (local.exitCode === 0) return { ok: true, out: local.stdout };
+
+  if (ctx.config.k3sServerUrl) {
+    try {
+      const serverHost = new URL(ctx.config.k3sServerUrl).hostname;
+      const remote = await remoteSshExec(serverHost, "root", `k3s kubectl ${args}`, {
+        ...(ctx.ssh.keyPath ? { keyPath: ctx.ssh.keyPath } : {}),
+        timeoutMs: 15_000,
+      });
+      if (remote.exitCode === 0) return { ok: true, out: remote.stdout };
+      return { ok: false, out: remote.stdout || remote.stderr };
+    } catch {
+      /* fall through */
+    }
+  }
+  return { ok: false, out: local.stdout || local.stderr };
+}
+
+/** Does the node's Longhorn record exist yet? */
+async function nodeRecordExists(ctx: OperationContext, nodeName: string): Promise<boolean> {
+  const res = await kubectlAnywhere(ctx, `-n longhorn-system get nodes.longhorn.io "${nodeName}" -o name`);
+  return res.ok && res.out.trim() !== "";
+}
+
+/** Does the node's Longhorn record carry a disk at DISK_PATH? */
+async function hasDisk(ctx: OperationContext, nodeName: string): Promise<boolean> {
+  const res = await kubectlAnywhere(
+    ctx,
+    `-n longhorn-system get nodes.longhorn.io "${nodeName}" ` +
+      `-o jsonpath='{range .spec.disks.*}{.path}{"\\n"}{end}'`,
+  );
+  if (!res.ok) return false;
+  return res.out.split("\n").map((l) => l.trim()).includes(DISK_PATH);
+}
+
 export const configureLonghornDisk: Operation = async (ctx): Promise<OperationResult> => {
-  // Check if /var/lib/longhorn exists on this node
-  const check = await ctx.ssh.exec("test -d /var/lib/longhorn && echo yes || echo no", sshOpts(ctx));
+  const check = await ctx.ssh.exec(`test -d ${DISK_PATH} && echo yes || echo no`, sshOpts(ctx));
   if (check.stdout.trim() !== "yes") {
-    return { success: true, changed: false, message: "No /var/lib/longhorn directory — skipping Longhorn disk config" };
+    return { success: true, changed: false, message: `No ${DISK_PATH} directory — skipping Longhorn disk config` };
   }
 
-  // Find the node name (hostname as registered in k3s)
+  // Longhorn writes replicas into <path>/replicas and creates that directory
+  // only when it FIRST initialises the disk. If the directory later goes away
+  // while longhorn-disk.cfg remains -- someone clearing space, a partial
+  // restore, a wipe that preserved the LV -- Longhorn considers the disk
+  // already initialised and never recreates it. The replica process then dies
+  // on a non-recursive mkdir:
+  //
+  //   level=fatal msg="Error running start replica command"
+  //     error="mkdir /host/var/lib/longhorn/replicas/pvc-...: ..."
+  //
+  // every single time, for ever, while the node reports Ready and Schedulable
+  // and its disk looks healthy. Cheap to guarantee, expensive to diagnose.
+  await ctx.ssh.exec(`mkdir -p ${DISK_PATH}/replicas`, sshOpts(ctx));
+
   const nodeNameResult = await ctx.ssh.exec("hostname -f 2>/dev/null || hostname", sshOpts(ctx));
   const nodeName = nodeNameResult.stdout.trim();
 
-  const annotation = JSON.stringify([{ path: "/var/lib/longhorn", allowScheduling: true }]);
-
-  // Try kubectl locally first (works on server nodes)
-  const result = await ctx.ssh.exec(
-    `k3s kubectl annotate node "${nodeName}" "node.longhorn.io/default-disks-config=${annotation}" --overwrite 2>&1 || true`,
-    sshOpts(ctx),
+  // SINGLE quotes around the whole key=value, not double.
+  //
+  // The value is JSON and therefore full of double quotes. Wrapping it in
+  // double quotes ends the quoted string at the first inner one, and the shell
+  // strips the rest, so the node gets
+  //     [{path:/var/lib/longhorn,allowScheduling:true}]
+  // instead of
+  //     [{"path":"/var/lib/longhorn","allowScheduling":true}]
+  // -- valid-looking, unparseable, and silent. Longhorn then creates NO disk on
+  // the node, which surfaces much later as "why does nothing schedule here".
+  // The JSON contains no single quotes, so single-quoting is safe.
+  const annotation = JSON.stringify([{ path: DISK_PATH, allowScheduling: true }]);
+  const annotated = await kubectlAnywhere(
+    ctx,
+    `annotate node "${nodeName}" 'node.longhorn.io/default-disks-config=${annotation}' --overwrite`,
   );
 
-  if (result.stdout.includes("annotated") || result.stdout.includes("unchanged")) {
-    return { success: true, changed: true, message: `Longhorn disk annotation applied to ${nodeName}` };
+  // No Longhorn at all (or no way to reach the API) — nothing more to do.
+  const lh = await kubectlAnywhere(ctx, "get ns longhorn-system -o name");
+  if (!lh.ok) {
+    return annotated.ok
+      ? { success: true, changed: true, message: `Longhorn disk annotation applied to ${nodeName} (Longhorn not installed yet)` }
+      : { success: true, changed: false, message: "Longhorn disk label set (annotation requires server kubectl)" };
   }
 
-  // For worker/agent nodes without local kubectl: apply via the server
-  if (ctx.config.k3sServerUrl) {
-    // The CLI has SSH access to the server — use sshExec from there
-    const serverHost = new URL(ctx.config.k3sServerUrl).hostname;
-    try {
-      const remoteResult = await remoteSshExec(
-        serverHost, "root",
-        `k3s kubectl annotate node "${nodeName}" "node.longhorn.io/default-disks-config=${annotation}" --overwrite`,
-        { ...(ctx.ssh.keyPath ? { keyPath: ctx.ssh.keyPath } : {}), timeoutMs: 15_000 },
-      );
-      if (remoteResult.stdout.includes("annotated") || remoteResult.stdout.includes("unchanged")) {
-        return { success: true, changed: true, message: `Longhorn disk annotation applied to ${nodeName} (via server)` };
-      }
-    } catch {
-      // Fall through to manual instruction
+  // Longhorn registration is ASYNCHRONOUS. On a node that has just joined --
+  // which is exactly the case after a wipe-and-rejoin -- the nodes.longhorn.io
+  // record does not exist yet, so there is nothing to inspect and nothing to
+  // repair. Waiting a fixed 60s and then giving up meant the repair never ran
+  // on the one path that needs it most, and the caller's schedulability gate
+  // then failed immediately with "Longhorn reports no disk".
+  //
+  // So: wait for the RECORD first, generously, and only then worry about the
+  // disk on it.
+  const registerDeadline = Date.now() + REGISTER_TIMEOUT_MS;
+  let registered = false;
+  while (Date.now() < registerDeadline) {
+    if (await hasDisk(ctx, nodeName)) {
+      return { success: true, changed: annotated.ok, message: `${nodeName} has a Longhorn disk at ${DISK_PATH}` };
     }
+    if (await nodeRecordExists(ctx, nodeName)) { registered = true; break; }
+    await sleep(VERIFY_INTERVAL_MS);
   }
 
-  return { success: true, changed: false, message: "Longhorn disk label set (annotation requires server kubectl)" };
+  if (!registered) {
+    return {
+      success: true,
+      changed: annotated.ok,
+      message: `${nodeName} not registered with Longhorn within ${REGISTER_TIMEOUT_MS / 1000}s; ` +
+        `annotation is in place for when it is`,
+    };
+  }
+
+  // Registered. Give Longhorn a chance to act on the annotation by itself
+  // before overriding it.
+  const diskDeadline = Date.now() + DISK_TIMEOUT_MS;
+  while (Date.now() < diskDeadline) {
+    if (await hasDisk(ctx, nodeName)) {
+      return { success: true, changed: annotated.ok, message: `${nodeName} has a Longhorn disk at ${DISK_PATH}` };
+    }
+    await sleep(VERIFY_INTERVAL_MS);
+  }
+
+  // The record exists and still has no disk: the annotation was applied too
+  // late to be read. Write it in.
+  ctx.log(`${nodeName} has no Longhorn disk after annotating; writing spec.disks directly`);
+  const patch = JSON.stringify({
+    spec: {
+      disks: {
+        [DISK_KEY]: {
+          path: DISK_PATH,
+          allowScheduling: true,
+          evictionRequested: false,
+          storageReserved: 0,
+          tags: [],
+          diskType: "filesystem",
+        },
+      },
+    },
+  });
+  const repaired = await kubectlAnywhere(
+    ctx,
+    `-n longhorn-system patch nodes.longhorn.io "${nodeName}" --type=merge -p '${patch}'`,
+  );
+  if (!repaired.ok) {
+    return {
+      success: false,
+      changed: false,
+      message: `${nodeName} has no Longhorn disk and could not be repaired`,
+      error: `${repaired.out}. Without a disk this node will never host a replica.`,
+    };
+  }
+
+  if (await hasDisk(ctx, nodeName)) {
+    return { success: true, changed: true, message: `Longhorn disk repaired on ${nodeName} (${DISK_PATH})` };
+  }
+  return {
+    success: false,
+    changed: true,
+    message: `Patched spec.disks on ${nodeName} but Longhorn still reports no disk at ${DISK_PATH}`,
+    error: "Check longhorn-manager logs; the node will not host replicas until this is resolved.",
+  };
 };

bastion/src/modules/modules/k3s/src/operations/node-admit.ts

@@ -0,0 +1,136 @@
+// Bring a node back into service. The inverse of node-evacuate.ts.
+//
+// WHY THIS EXISTS
+// ---------------
+// Evacuation is not symmetric with itself. It leaves three marks on a node --
+// a kubernetes cordon, Longhorn's `allowScheduling:false` + `evictionRequested`,
+// and (on a node that was wiped and rejoined) a MISSING default Longhorn disk.
+// Undoing a cordon undoes only the first. Without the other two the node comes
+// back Ready, accepts pods, and quietly hosts no replicas at all -- which looks
+// like success right up until you notice storage never rebalanced onto it.
+//
+// This is also the only supported way out of an aborted evacuation.
+
+import type { Operation, OperationContext, OperationResult } from "../types.js";
+import { sshOpts } from "../utils.js";
+
+async function nodeName(ctx: OperationContext): Promise<string> {
+  const r = await ctx.ssh.exec("hostname -f 2>/dev/null || hostname", sshOpts(ctx));
+  return r.stdout.trim();
+}
+
+async function kubectl(ctx: OperationContext, args: string): Promise<{ ok: boolean; out: string }> {
+  const res = await ctx.ssh.exec(`k3s kubectl ${args} 2>&1`, sshOpts(ctx));
+  return { ok: res.exitCode === 0, out: res.exitCode === 0 ? res.stdout : (res.stdout || res.stderr) };
+}
+
+/**
+ * Clear Longhorn's eviction request and re-enable scheduling.
+ *
+ * Both fields in one merge patch, mirroring how eviction sets them: Longhorn's
+ * validating webhook rejects `evictionRequested:true` on a schedulable node, so
+ * the pair has to move together in both directions.
+ */
+export const clearLonghornEviction: Operation = async (ctx): Promise<OperationResult> => {
+  const name = await nodeName(ctx);
+
+  const lh = await kubectl(ctx, "get ns longhorn-system -o name");
+  if (!lh.ok) {
+    return { success: true, changed: false, message: "Longhorn not installed — nothing to clear" };
+  }
+
+  const before = await kubectl(
+    ctx,
+    `-n longhorn-system get nodes.longhorn.io "${name}" -o jsonpath='{.spec.allowScheduling},{.spec.evictionRequested}'`,
+  );
+  if (!before.ok) {
+    return { success: true, changed: false, message: `${name} has no Longhorn node record yet` };
+  }
+  if (before.out.trim() === "true,false") {
+    return { success: true, changed: false, message: `${name} already accepts replicas` };
+  }
+
+  const patch = JSON.stringify({ spec: { allowScheduling: true, evictionRequested: false } });
+  const res = await kubectl(
+    ctx,
+    `-n longhorn-system patch nodes.longhorn.io "${name}" --type=merge -p '${patch}'`,
+  );
+  return res.ok
+    ? { success: true, changed: true, message: `${name} accepts replicas again` }
+    : { success: false, changed: false, message: `Could not clear eviction on ${name}`, error: res.out };
+};
+
+/**
+ * Confirm the node can actually host replicas again.
+ *
+ * Reads back from Longhorn rather than trusting the patches above, because the
+ * failure this catches is a node that is Ready and schedulable in kubernetes
+ * while having no Longhorn disk at all -- the exact state a rejoined node lands
+ * in when its default-disk annotation was applied too late.
+ */
+const SCHEDULABLE_TIMEOUT_MS = 120_000;
+const SCHEDULABLE_INTERVAL_MS = 5_000;
+const sleep = (ms: number): Promise<void> => new Promise((r) => setTimeout(r, ms));
+
+export const assertNodeSchedulable: Operation = async (ctx): Promise<OperationResult> => {
+  // Poll rather than check once.
+  //
+  // Longhorn registers a node and attaches its disk ASYNCHRONOUSLY, and this
+  // runs moments after a wipe-and-rejoin. Checking instantly reports "Longhorn
+  // reports no disk on this node" for a node that is simply a few seconds from
+  // being fine -- which is a false alarm on the exact path the gate exists to
+  // protect. Still fails if it never settles.
+  const deadline = Date.now() + SCHEDULABLE_TIMEOUT_MS;
+  let problems: string[] = [];
+  for (;;) {
+    problems = await schedulableProblems(ctx);
+    if (problems.length === 0 || Date.now() >= deadline) break;
+    await sleep(SCHEDULABLE_INTERVAL_MS);
+  }
+
+  const name = await nodeName(ctx);
+  return problems.length === 0
+    ? { success: true, changed: false, message: `${name} is back in service` }
+    : {
+        success: false,
+        changed: false,
+        message: `${name} is NOT fully back in service after ${SCHEDULABLE_TIMEOUT_MS / 1000}s`,
+        error: problems.join("; "),
+      };
+};
+
+/** One pass of the schedulability checks. Returns the reasons it is not. */
+async function schedulableProblems(ctx: OperationContext): Promise<string[]> {
+  const name = await nodeName(ctx);
+  const problems: string[] = [];
+
+  const sched = await kubectl(ctx, `get node "${name}" -o jsonpath='{.spec.unschedulable}'`);
+  if (sched.ok && sched.out.trim() === "true") problems.push("node is still cordoned");
+
+  const lh = await kubectl(ctx, "get ns longhorn-system -o name");
+  if (lh.ok) {
+    const spec = await kubectl(
+      ctx,
+      `-n longhorn-system get nodes.longhorn.io "${name}" -o jsonpath='{.spec.allowScheduling},{.spec.evictionRequested}'`,
+    );
+    if (!spec.ok) {
+      problems.push("no Longhorn node record");
+    } else {
+      const [allow, evicting] = spec.out.trim().split(",");
+      if (allow !== "true") problems.push("Longhorn scheduling is still disabled");
+      if (evicting === "true") problems.push("Longhorn eviction is still requested");
+    }
+
+    // A node with no disk hosts no replicas, however healthy it otherwise looks.
+    const disks = await kubectl(
+      ctx,
+      `-n longhorn-system get nodes.longhorn.io "${name}" ` +
+        `-o jsonpath='{range .status.diskStatus.*}{.conditions[?(@.type=="Schedulable")].status}{"\\n"}{end}'`,
+    );
+    const states = disks.ok ? disks.out.split("\n").map((l) => l.trim()).filter(Boolean) : [];
+    if (states.length === 0) problems.push("Longhorn reports no disk on this node");
+    else if (!states.includes("True")) problems.push(`no schedulable disk (states: ${states.join(",")})`);
+  }
+
+  return problems;
+}

bastion/src/modules/modules/k3s/src/operations/node-evacuate.ts

@@ -0,0 +1,547 @@
+// Evacuate a node before it is wiped and reinstalled.
+//
+// WHY THIS EXISTS
+// ---------------
+// The kickstart's %pre can preserve data across a reinstall, but ONLY when the
+// disk layout is unchanged: it keys off `labvg` already existing with the
+// expected LV names. Migrating a machine to the v2 layout means recreating the
+// volume group, which takes the `clearpart --all` branch and destroys every
+// Longhorn replica on that node.
+//
+// So a layout migration cannot rely on disk-level preservation. The node has to
+// be EVACUATED first -- replicas rebuilt elsewhere, workloads moved, etcd
+// membership surrendered -- and then wiped. These operations are the missing
+// half; before this file the repo had no drain, no cordon, and no replica
+// eviction anywhere.
+//
+// Ordering matters and is not interchangeable:
+//   cordon -> check -> evict replicas (WAIT for rebuild) -> drain -> remove etcd member
+//
+// Two ordering facts, both learned the hard way:
+//   - Removing the etcd member first costs you the API server you need for the
+//     rest of the procedure.
+//   - EVICT BEFORE DRAIN. `drain --force` does not override PodDisruptionBudgets,
+//     and longhorn-manager keeps a PDB on the instance-manager pod while volumes
+//     are attached or degraded. Draining a storage node first therefore stops the
+//     instance-manager that serves the very replicas being evicted: the drain
+//     blocks for its full timeout, and if it did succeed it would remove the only
+//     source a single-replica volume could rebuild from.
+
+import type { Operation, OperationContext, OperationResult } from "../types.js";
+import { sshOpts } from "../utils.js";
+
+/** Longhorn's own default; a rebuild of a large replica set can exceed it. */
+const REPLICA_REBUILD_TIMEOUT_MS = 60 * 60_000;
+const POLL_INTERVAL_MS = 15_000;
+/** How often to report "still waiting" when the replica count is not moving. */
+const HEARTBEAT_INTERVAL_MS = 3 * 60_000;
+
+const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
+
+async function nodeName(ctx: OperationContext): Promise<string> {
+  const r = await ctx.ssh.exec("hostname -f 2>/dev/null || hostname", sshOpts(ctx));
+  return r.stdout.trim();
+}
+
+/**
+ * kubectl, from wherever it is reachable.
+ *
+ * An agent has no local kubectl, so the caller must be pointed at a server.
+ * Returns null when there is no way to reach the API, so callers can report
+ * "could not do this" instead of silently continuing.
+ */
+async function kubectl(
+  ctx: OperationContext,
+  args: string,
+  /**
+   * Override the SSH timeout for this call.
+   *
+   * sshOpts() is a flat 30 seconds, which is right for a status read and
+   * completely wrong for anything that does work. A `kubectl drain` asking for
+   * --timeout=15m gets killed by the transport at 30s and returns non-zero
+   * having ALREADY evicted most of the pods -- so it looks like a drain
+   * failure, with a transcript that reads like success, on every real node.
+   */
+  timeoutMs?: number,
+): Promise<{ ok: boolean; out: string }> {
+  const opts = timeoutMs === undefined ? sshOpts(ctx) : { timeoutMs };
+  const local = await ctx.ssh.exec(`k3s kubectl ${args} 2>&1`, opts);
+  if (local.exitCode === 0) return { ok: true, out: local.stdout };
+  return { ok: false, out: local.stdout || local.stderr };
+}
+
+/** Mark the node unschedulable. Safe and instantly reversible. */
+export const cordonNode: Operation = async (ctx): Promise<OperationResult> => {
+  const name = await nodeName(ctx);
+  const res = await kubectl(ctx, `cordon "${name}"`);
+  if (!res.ok) {
+    return { success: false, changed: false, message: `Could not cordon ${name}`, error: res.out };
+  }
+  const already = res.out.includes("already cordoned");
+  return {
+    success: true,
+    changed: !already,
+    message: already ? `${name} was already cordoned` : `Cordoned ${name}`,
+  };
+};
+
+export const uncordonNode: Operation = async (ctx): Promise<OperationResult> => {
+  const name = await nodeName(ctx);
+  const res = await kubectl(ctx, `uncordon "${name}"`);
+  return res.ok
+    ? { success: true, changed: true, message: `Uncordoned ${name}` }
+    : { success: false, changed: false, message: `Could not uncordon ${name}`, error: res.out };
+};
+
+/**
+ * Move workloads off the node.
+ *
+ * --ignore-daemonsets because a DaemonSet pod cannot be evicted by definition;
+ * --delete-emptydir-data because refusing to delete emptyDir is what makes an
+ * unattended drain hang forever, and a node about to be wiped is going to lose
+ * that data regardless.
+ */
+export const drainNode: Operation = async (ctx): Promise<OperationResult> => {
+  const name = await nodeName(ctx);
+  // The SSH timeout must outlive kubectl's own --timeout, or the transport
+  // kills a drain that is working and the result is indistinguishable from a
+  // drain that failed.
+  const DRAIN_TIMEOUT_MS = 15 * 60_000;
+  const res = await kubectl(
+    ctx,
+    `drain "${name}" --ignore-daemonsets --delete-emptydir-data --force --timeout=15m`,
+    DRAIN_TIMEOUT_MS + 60_000,
+  );
+  if (res.ok) return { success: true, changed: true, message: `Drained ${name}` };
+
+  // Surface the END of the transcript. The first line of a drain is always
+  // "node/x already cordoned", which tells you nothing and hides the reason.
+  const tail = res.out.split("\n").map((l) => l.trim()).filter(Boolean).slice(-6).join(" | ");
+  return {
+    success: false,
+    changed: false,
+    message: `Drain failed on ${name}`,
+    error: tail === "" ? res.out : `...${tail}`,
+  };
+};
+
+/** kubectl -o json, parsed. Returns null when the read or the parse failed. */
+async function kubectlJson<T>(ctx: OperationContext, args: string): Promise<T | null> {
+  const res = await kubectl(ctx, `${args} -o json`);
+  if (!res.ok) return null;
+  try {
+    return JSON.parse(res.out) as T;
+  } catch {
+    return null;
+  }
+}
+
+/** A Longhorn setting's value, or the supplied default when it is unreadable. */
+async function longhornSetting(ctx: OperationContext, name: string, fallback: string): Promise<string> {
+  const res = await kubectl(ctx, `-n longhorn-system get settings.longhorn.io ${name} -o jsonpath='{.value}'`);
+  const v = res.ok ? res.out.trim() : "";
+  return v === "" ? fallback : v;
+}
+
+interface LhNode {
+  metadata: { name: string };
+  spec: { allowScheduling?: boolean };
+  status?: { diskStatus?: Record<string, {
+    storageAvailable?: number; storageMaximum?: number;
+    storageScheduled?: number; storageReserved?: number;
+  }> };
+}
+interface LhReplica {
+  metadata: { name: string };
+  spec: { nodeID?: string; volumeName?: string };
+}
+interface LhVolume {
+  metadata: { name: string };
+  spec: { numberOfReplicas?: number; size?: number | string };
+  status?: { state?: string; robustness?: string };
+}
+
+export interface EvictionFeasibility {
+  completable: boolean;
+  /** Reasons eviction can never finish. Each names the offending volume. */
+  blockers: string[];
+  /** Things that will work but are dangerous. */
+  warnings: string[];
+  replicasMoving: number;
+  bytesMoving: number;
+}
+
+/**
+ * Can eviction of this node actually COMPLETE?
+ *
+ * evictLonghornReplicas waits for the node's replica count to reach zero. If
+ * there is nowhere for a replica to go, that wait can never end -- it burns the
+ * full hour-long timeout and then fails, which is an expensive way to discover
+ * a fact that is knowable in a couple of seconds.
+ *
+ * It is knowable because Longhorn's placement rules are simple:
+ *
+ *   - With strict anti-affinity (replica-soft-anti-affinity=false, the default
+ *     and what this fleet runs) a volume may put at most ONE replica on a node.
+ *     So a volume wanting N replicas needs N schedulable nodes OTHER than the
+ *     one being evacuated. A 3-replica volume on a 3-node cluster is therefore
+ *     unevictable -- the two survivors already hold a copy each and cannot take
+ *     the third.
+ *   - The scheduler admits a replica against over-provisioned space, but the
+ *     rebuild writes real bytes to a real disk. Those are different numbers,
+ *     and at 500% over-provisioning the first can pass while the second fails.
+ *
+ * Read-only, a handful of API calls, finishes in seconds.
+ */
+export async function checkEvictionCompletable(
+  ctx: OperationContext,
+  name: string,
+): Promise<EvictionFeasibility> {
+  const blockers: string[] = [];
+  const warnings: string[] = [];
+
+  const nodes = await kubectlJson<{ items: LhNode[] }>(ctx, "-n longhorn-system get nodes.longhorn.io");
+  const replicas = await kubectlJson<{ items: LhReplica[] }>(ctx, "-n longhorn-system get replicas.longhorn.io");
+  const volumes = await kubectlJson<{ items: LhVolume[] }>(ctx, "-n longhorn-system get volumes.longhorn.io");
+  if (!nodes || !replicas || !volumes) {
+    return {
+      completable: false,
+      blockers: ["Could not read Longhorn nodes/replicas/volumes — refusing to guess"],
+      warnings, replicasMoving: 0, bytesMoving: 0,
+    };
+  }
+
+  const softAntiAffinity = (await longhornSetting(ctx, "replica-soft-anti-affinity", "false")) === "true";
+  const overProvisionPct = Number(await longhornSetting(ctx, "storage-over-provisioning-percentage", "100")) || 100;
+
+  const mine = replicas.items.filter((r) => r.spec.nodeID === name);
+  if (mine.length === 0) {
+    return { completable: true, blockers, warnings, replicasMoving: 0, bytesMoving: 0 };
+  }
+
+  const volumeByName = new Map(volumes.items.map((v) => [v.metadata.name, v]));
+  // Candidates are schedulable Longhorn nodes that are not the one leaving.
+  const candidates = nodes.items.filter(
+    (n) => n.metadata.name !== name && n.spec.allowScheduling !== false,
+  );
+
+  const diskFree = (n: LhNode): number =>
+    Object.values(n.status?.diskStatus ?? {}).reduce((a, d) => a + (d.storageAvailable ?? 0), 0);
+  const schedulableFree = (n: LhNode): number =>
+    Object.values(n.status?.diskStatus ?? {}).reduce((a, d) => {
+      const usable = ((d.storageMaximum ?? 0) - (d.storageReserved ?? 0)) * (overProvisionPct / 100);
+      return a + Math.max(0, usable - (d.storageScheduled ?? 0));
+    }, 0);
+
+  let bytesMoving = 0;
+
+  for (const rep of mine) {
+    const volName = rep.spec.volumeName ?? "";
+    const vol = volumeByName.get(volName);
+    if (!vol) {
+      warnings.push(`replica ${rep.metadata.name} has no volume ${volName} — orphan?`);
+      continue;
+    }
+    const want = vol.spec.numberOfReplicas ?? 1;
+    const size = Number(vol.spec.size ?? 0);
+    bytesMoving += size;
+
+    // (1) Placement. Strict anti-affinity means one replica per node per volume.
+    if (!softAntiAffinity && candidates.length < want) {
+      blockers.push(
+        `volume ${volName} wants ${want} replicas but only ${candidates.length} other schedulable ` +
+        `node(s) exist — with strict anti-affinity its replica cannot be placed. Add a schedulable node.`,
+      );
+      continue;
+    }
+
+    // (2) Scheduler capacity, at the configured over-provisioning percentage.
+    const schedulableFit = candidates.filter((n) => schedulableFree(n) >= size);
+    if (schedulableFit.length === 0) {
+      blockers.push(
+        `volume ${volName} (${gib(size)}) does not fit the scheduler on any candidate node ` +
+        `even at ${overProvisionPct}% over-provisioning.`,
+      );
+      continue;
+    }
+
+    // (3) Real capacity. The scheduler counts over-provisioned space; a rebuild
+    // writes real bytes. This is the check that catches a small disk hiding
+    // behind a generous over-provisioning percentage.
+    const realFit = candidates.filter((n) => diskFree(n) >= size);
+    if (realFit.length === 0) {
+      const best = Math.max(0, ...candidates.map(diskFree));
+      blockers.push(
+        `volume ${volName} needs ${gib(size)} but the roomiest candidate node has ${gib(best)} ` +
+        `actually free. storage-over-provisioning-percentage=${overProvisionPct} lets the scheduler ` +
+        `accept this; the disk does not.`,
+      );
+      continue;
+    }
+
+    // (4) Volumes with no second copy, and volumes that cannot rebuild at all.
+    if (want === 1) {
+      warnings.push(
+        `volume ${volName} has a single replica — evicting it rebuilds from the only copy that exists`,
+      );
+    }
+    if (vol.status?.state === "detached") {
+      warnings.push(`volume ${volName} is detached — Longhorn must attach it to rebuild`);
+    }
+    if (vol.status?.robustness === "degraded") {
+      warnings.push(`volume ${volName} is already degraded before eviction starts`);
+    }
+  }
+
+  return { completable: blockers.length === 0, blockers, warnings, replicasMoving: mine.length, bytesMoving };
+}
+
+function gib(bytes: number): string {
+  return `${(bytes / 2 ** 30).toFixed(1)} GiB`;
+}
+
+/**
+ * Standalone gate: refuse fast and loudly rather than hang for an hour.
+ *
+ * Exposed as its own Operation so it can be run on its own -- "could I evacuate
+ * this node if I wanted to?" is a question worth answering without starting.
+ */
+export const assertEvictionCompletable: Operation = async (ctx): Promise<OperationResult> => {
+  const name = await nodeName(ctx);
+  const lh = await kubectl(ctx, "get ns longhorn-system -o name");
+  if (!lh.ok) {
+    return { success: true, changed: false, message: "Longhorn not installed — nothing to evict" };
+  }
+
+  const f = await checkEvictionCompletable(ctx, name);
+  for (const w of f.warnings) ctx.log(`  WARNING: ${w}`);
+
+  if (!f.completable) {
+    return {
+      success: false,
+      changed: false,
+      message: `Eviction of ${name} cannot complete`,
+      error: f.blockers.join("; "),
+    };
+  }
+  return {
+    success: true,
+    changed: false,
+    message: f.replicasMoving === 0
+      ? `${name} holds no replicas`
+      : `${f.replicasMoving} replica(s) (${gib(f.bytesMoving)}) can be placed elsewhere`,
+  };
+};
+
+/**
+ * Ask Longhorn to move this node's replicas elsewhere, then WAIT.
+ *
+ * The wait is the entire point. Setting evictionRequested and immediately
+ * wiping the disk destroys the replicas mid-rebuild, which on a 3-replica
+ * volume takes you to 1 and on the 2-replica Prometheus volume takes you to 0.
+ *
+ * Returns changed:false with an explanation when Longhorn is not installed,
+ * so a standalone or storage-free node is a no-op rather than an error.
+ */
+export const evictLonghornReplicas: Operation = async (ctx): Promise<OperationResult> => {
+  const name = await nodeName(ctx);
+
+  const hasLonghorn = await kubectl(ctx, "get ns longhorn-system -o name");
+  if (!hasLonghorn.ok) {
+    return { success: true, changed: false, message: "Longhorn not installed — nothing to evict" };
+  }
+
+  // Refuse in seconds rather than discover it after an hour of polling.
+  // Inside the operation, not only in the group, so `labctl app k3s evacuate`
+  // gets the same protection as the drill does.
+  const feasible = await checkEvictionCompletable(ctx, name);
+  for (const w of feasible.warnings) ctx.log(`  WARNING: ${w}`);
+  if (!feasible.completable) {
+    return {
+      success: false,
+      changed: false,
+      message: `Eviction of ${name} cannot complete — not starting it`,
+      error: feasible.blockers.join("; "),
+    };
+  }
+  // A single-replica volume rebuilds from the only copy in existence, and this
+  // fleet has no backup target. That is a decision for a human, not a default.
+  const singles = feasible.warnings.filter((w) => w.includes("single replica"));
+  if (singles.length > 0 && ctx.config.evictionAllowSingleReplica !== true) {
+    return {
+      success: false,
+      changed: false,
+      message: `${name} holds single-replica volume(s); refusing to evict`,
+      error: `${singles.join("; ")}. Raise them to 3 replicas and wait for healthy, ` +
+        `or pass evictionAllowSingleReplica to accept the risk.`,
+    };
+  }
+
+  const patch = JSON.stringify({
+    spec: { allowScheduling: false, evictionRequested: true },
+  });
+  const req = await kubectl(
+    ctx,
+    `-n longhorn-system patch nodes.longhorn.io "${name}" --type=merge -p '${patch}'`,
+  );
+  if (!req.ok) {
+    return { success: false, changed: false, message: `Could not request eviction for ${name}`, error: req.out };
+  }
+
+  ctx.log(`Eviction requested for ${name}; waiting for replicas to rebuild elsewhere...`);
+  const start = Date.now();
+  let last = -1;
+  let lastHeartbeat = Date.now();
+  try {
+  while (Date.now() - start < REPLICA_REBUILD_TIMEOUT_MS) {
+    const res = await kubectl(
+      ctx,
+      `-n longhorn-system get replicas.longhorn.io -o jsonpath='{range .items[?(@.spec.nodeID=="${name}")]}{.metadata.name}{"\\n"}{end}'`,
+    );
+    const remaining = res.ok ? res.out.split("\n").map((l) => l.trim()).filter(Boolean).length : -1;
+    if (remaining === 0) {
+      return {
+        success: true,
+        changed: true,
+        message: `All replicas evicted from ${name} after ${Math.round((Date.now() - start) / 1000)}s`,
+      };
+    }
+    // Log on change, and otherwise every few minutes. A silent hour is
+    // indistinguishable from a hung process; a heartbeat that carries the
+    // elapsed time makes "stalled" visible while it is happening rather than
+    // only in the timeout message.
+    const elapsedS = Math.round((Date.now() - start) / 1000);
+    if (remaining !== last) {
+      ctx.log(`  ${remaining} replica(s) still on ${name} (${elapsedS}s elapsed)`);
+      last = remaining;
+      lastHeartbeat = Date.now();
+    } else if (Date.now() - lastHeartbeat >= HEARTBEAT_INTERVAL_MS) {
+      ctx.log(`  still waiting: ${remaining} replica(s) on ${name}, no change in ` +
+        `${Math.round((Date.now() - lastHeartbeat) / 60_000)} min (${elapsedS}s elapsed)`);
+      lastHeartbeat = Date.now();
+    }
+    await sleep(POLL_INTERVAL_MS);
+  }
+  } catch (err) {
+    // Never leave the node eviction-pending because of a transient failure.
+    await cancelEviction(ctx, name);
+    throw err;
+  }
+
+  // Timed out. Say exactly what is stuck before restoring the node, because
+  // after the cancel the eviction flags are gone and the evidence with them.
+  const stuck = await describeStuckReplicas(ctx, name);
+
+  // Put the node back the way we found it: an abandoned eviction leaves it
+  // unschedulable with Longhorn still trying to drain it, which is safe but
+  // sticky, and nothing else would ever clear it.
+  const cancelled = await cancelEviction(ctx, name);
+  return {
+    success: false,
+    changed: true,
+    message: `Timed out waiting for replica eviction from ${name}`,
+    error: `${last} replica(s) still present after ${REPLICA_REBUILD_TIMEOUT_MS / 60000} min. ` +
+      `Wiping now would destroy them.\nStill on ${name}:\n${stuck}\n` +
+      (cancelled
+        ? `Eviction request has been cancelled and scheduling restored.`
+        : `WARNING: could not cancel the eviction request — clear it with ` +
+          `\`labctl app k3s admit ${name}\`.`),
+  };
+};
+
+/**
+ * Describe the replicas still sitting on a node, and why they are not moving.
+ *
+ * An eviction that times out after an hour used to report only a count. That
+ * is not enough to act on: the question is always WHICH volume is stuck and
+ * what state its replica is in. A replica with `desireState=running` and
+ * `currentState=stopped` and no `failedAt` is Longhorn wanting to start it and
+ * not doing so -- a very different problem from a volume that cannot be placed.
+ */
+async function describeStuckReplicas(ctx: OperationContext, name: string): Promise<string> {
+  const reps = await kubectlJson<{ items: {
+    metadata: { name: string };
+    spec: { nodeID?: string; volumeName?: string; failedAt?: string; desireState?: string };
+    status?: { currentState?: string };
+  }[] }>(ctx, "-n longhorn-system get replicas.longhorn.io");
+  if (!reps) return "  (could not read replicas)";
+
+  const mine = reps.items.filter((r) => r.spec.nodeID === name);
+  if (mine.length === 0) return "  (none)";
+
+  const vols = await kubectlJson<{ items: {
+    metadata: { name: string };
+    spec: { numberOfReplicas?: number };
+    status?: { state?: string; robustness?: string };
+  }[] }>(ctx, "-n longhorn-system get volumes.longhorn.io");
+  const byName = new Map((vols?.items ?? []).map((v) => [v.metadata.name, v]));
+
+  return mine.map((r) => {
+    const v = byName.get(r.spec.volumeName ?? "");
+    return `  ${r.spec.volumeName ?? "?"}: replica ${r.metadata.name.slice(-12)} ` +
+      `currentState=${r.status?.currentState ?? "?"} desireState=${r.spec.desireState ?? "?"} ` +
+      `failedAt=${r.spec.failedAt === undefined || r.spec.failedAt === "" ? "(none)" : r.spec.failedAt} | ` +
+      `volume ${v?.status?.state ?? "?"}/${v?.status?.robustness ?? "?"} ` +
+      `wants ${v?.spec.numberOfReplicas ?? "?"} replicas`;
+  }).join("\n");
+}
+
+/**
+ * Undo the eviction request. Returns false if it could not be undone, which the
+ * caller must surface -- silently failing here leaves a node nobody can use.
+ */
+async function cancelEviction(ctx: OperationContext, name: string): Promise<boolean> {
+  const patch = JSON.stringify({ spec: { allowScheduling: true, evictionRequested: false } });
+  const res = await kubectl(
+    ctx,
+    `-n longhorn-system patch nodes.longhorn.io "${name}" --type=merge -p '${patch}'`,
+  );
+  if (res.ok) ctx.log(`Cancelled eviction request on ${name} and restored scheduling`);
+  return res.ok;
+}
+
+/**
+ * Verify the node holds nothing before it is wiped.
+ *
+ * A separate, cheap operation on purpose: it is the gate that turns "we ran the
+ * eviction steps" into "the node is actually empty", and it is the one thing
+ * worth asserting twice. Reading back from the thing that does the work, not
+ * from the request we sent it.
+ */
+export const assertNodeEvacuated: Operation = async (ctx): Promise<OperationResult> => {
+  const name = await nodeName(ctx);
+  const problems: string[] = [];
+
+  const sched = await kubectl(ctx, `get node "${name}" -o jsonpath='{.spec.unschedulable}'`);
+  if (sched.ok && sched.out.trim() !== "true") problems.push("node is still schedulable");
+
+  const pods = await kubectl(
+    ctx,
+    `get pods -A --field-selector spec.nodeName="${name}",status.phase=Running ` +
+      `-o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\\n"}{end}'`,
+  );
+  if (pods.ok) {
+    const running = pods.out.split("\n").map((l) => l.trim()).filter(Boolean);
+    // DaemonSet pods legitimately remain; anything else means the drain lied.
+    if (running.length > 0) ctx.log(`  ${running.length} pod(s) remain (expected: DaemonSets only)`);
+  }
+
+  const lh = await kubectl(ctx, "get ns longhorn-system -o name");
+  if (lh.ok) {
+    const reps = await kubectl(
+      ctx,
+      `-n longhorn-system get replicas.longhorn.io -o jsonpath='{range .items[?(@.spec.nodeID=="${name}")]}{.metadata.name}{"\\n"}{end}'`,
+    );
+    const n = reps.ok ? reps.out.split("\n").map((l) => l.trim()).filter(Boolean).length : 0;
+    if (n > 0) problems.push(`${n} Longhorn replica(s) still on this node`);
+  }
+
+  return problems.length === 0
+    ? { success: true, changed: false, message: `${name} is evacuated and safe to wipe` }
+    : {
+        success: false,
+        changed: false,
+        message: `${name} is NOT safe to wipe`,
+        error: problems.join("; "),
+      };
+};

bastion/src/modules/modules/k3s/src/operations/rancher-storage.ts

@@ -0,0 +1,48 @@
+// Grow the labvg/rancher LV (k3s image store / imageFs) to 120G.
+// 2026-08 incident: the original 20G LV sat at 85% used from steady-state
+// images alone, so one ~5G image pull tripped imagefs eviction and evicted
+// unrelated pods. Fresh installs are sized at 120G by the kickstart; this op
+// covers nodes installed before that change and vanilla nodes converted to
+// k8s later. Never removes or shrinks anything — if the VG lacks free space
+// (e.g. a longhorn --grow LV consumed it), it reports and moves on.
+
+import type { Operation, OperationResult } from "../types.js";
+import { sshOpts } from "../utils.js";
+
+const RANCHER_LV = "labvg/rancher";
+const TARGET_MIB = 122880; // 120G
+
+export const growRancherLv: Operation = async (ctx): Promise<OperationResult> => {
+  const lv = await ctx.ssh.exec(
+    `lvs --noheadings --units m --nosuffix -o lv_size ${RANCHER_LV} 2>/dev/null || true`,
+    sshOpts(ctx),
+  );
+  const sizeMib = Number.parseFloat(lv.stdout.trim());
+  if (Number.isNaN(sizeMib)) {
+    return { success: true, changed: false, message: "No labvg/rancher LV — imageFs shares /var, skipping" };
+  }
+  if (sizeMib >= TARGET_MIB) {
+    return { success: true, changed: false, message: `rancher LV already ${Math.round(sizeMib / 1024)}G` };
+  }
+
+  const vg = await ctx.ssh.exec(`vgs --noheadings --units m --nosuffix -o vg_free labvg`, sshOpts(ctx));
+  const freeMib = Number.parseFloat(vg.stdout.trim());
+  const neededMib = TARGET_MIB - sizeMib;
+  if (Number.isNaN(freeMib) || freeMib < neededMib) {
+    return {
+      success: true,
+      changed: false,
+      message: `VG labvg has ${Math.floor((Number.isNaN(freeMib) ? 0 : freeMib) / 1024)}G free — ` +
+        `need ${Math.ceil(neededMib / 1024)}G to grow rancher LV to 120G (manual LV rebuild required)`,
+    };
+  }
+
+  await ctx.ssh.exec(`lvextend -L ${TARGET_MIB}m /dev/${RANCHER_LV}`, sshOpts(ctx));
+  await ctx.ssh.exec(`xfs_growfs /var/lib/rancher`, sshOpts(ctx));
+
+  return {
+    success: true,
+    changed: true,
+    message: `rancher LV grown ${Math.round(sizeMib / 1024)}G → 120G`,
+  };
+};

bastion/src/modules/modules/k3s/src/operations/swap.ts

@@ -1,22 +1,40 @@
-// Disable swap (CIS requirement for k3s).
+// Enable swap so memory pressure spills to disk instead of OOM-killing.
+// kubelet runs with failSwapOn=false (k3s default); zram stays the fast tier,
+// the labvg-swap LV is the overflow tier. Replaces the old CIS-style
+// disableSwap op — a kernel OOM kill of a node daemon is worse than slow swap.
 
 import type { Operation, OperationResult } from "../types.js";
 import { sshOpts } from "../utils.js";
 
-export const disableSwap: Operation = async (ctx): Promise<OperationResult> => {
-  const check = await ctx.ssh.exec("swapon --show --noheadings", sshOpts(ctx));
-  const active = check.stdout.trim().length > 0;
+const SWAP_DEV = "/dev/mapper/labvg-swap";
 
-  if (active) {
-    await ctx.ssh.exec("swapoff -a", sshOpts(ctx));
+export const enableSwap: Operation = async (ctx): Promise<OperationResult> => {
+  const lv = await ctx.ssh.exec(`test -b ${SWAP_DEV} && echo yes || echo no`, sshOpts(ctx));
+  if (lv.stdout.trim() !== "yes") {
+    return { success: true, changed: false, message: "No labvg-swap LV — skipping swap enable" };
   }
 
-  // Remove swap entries from fstab permanently
-  await ctx.ssh.exec("sed -i '/\\sswap\\s/d' /etc/fstab", sshOpts(ctx));
+  const active = await ctx.ssh.exec(
+    `grep -q "^$(readlink -f ${SWAP_DEV}) " /proc/swaps && echo on || echo off`,
+    sshOpts(ctx),
+  );
+  const wasOff = active.stdout.trim() !== "on";
+
+  if (wasOff) {
+    // Format if the LV was never (or wrongly) initialised, then activate
+    await ctx.ssh.exec(`blkid ${SWAP_DEV} | grep -q 'TYPE="swap"' || mkswap ${SWAP_DEV}`, sshOpts(ctx));
+    await ctx.ssh.exec(`swapon ${SWAP_DEV}`, sshOpts(ctx));
+  }
+
+  // Persist across reboots (idempotent)
+  await ctx.ssh.exec(
+    `grep -q "labvg-swap" /etc/fstab || echo "${SWAP_DEV} none swap defaults 0 0" >> /etc/fstab`,
+    sshOpts(ctx),
+  );
 
   return {
     success: true,
-    changed: active,
-    message: active ? "Swap disabled" : "Swap already disabled",
+    changed: wasOff,
+    message: wasOff ? "LV swap enabled" : "LV swap already active",
   };
 };

bastion/src/modules/modules/k3s/src/operations/worker-label.ts

@@ -0,0 +1,66 @@
+// Apply the worker role label from the SERVER side, after the node registers.
+//
+// WHY THIS IS NOT A NODE-LABEL
+// ----------------------------
+// `node-role.kubernetes.io/worker=true` cannot be set by the kubelet itself.
+// The servers enable the NodeRestriction admission plugin, which forbids a
+// kubelet from assigning itself any label in the `node-role.kubernetes.io/`
+// namespace. Putting it in the agent's `node-label:` list makes registration
+// fail, and the resulting message --
+//
+//   Job for k3s-agent.service failed because the service did not take the
+//   steps required by its unit configuration.
+//
+// names neither labels nor admission, so it reads as a networking, token or
+// sysctl problem. It is none of those.
+//
+// The label is cosmetic (it populates the ROLES column in `kubectl get nodes`)
+// so failing to apply it is a warning, never an error.
+
+import type { Operation, OperationContext, OperationResult } from "../types.js";
+import { sshOpts } from "../utils.js";
+import { sshExec as remoteSshExec } from "../../../../src/ssh.js";
+import { isK3sServer } from "@lab/shared";
+
+const WORKER_LABEL = "node-role.kubernetes.io/worker=true";
+
+async function nodeName(ctx: OperationContext): Promise<string> {
+  const r = await ctx.ssh.exec("hostname -f 2>/dev/null || hostname", sshOpts(ctx));
+  return r.stdout.trim();
+}
+
+export const labelWorkerRole: Operation = async (ctx): Promise<OperationResult> => {
+  if (isK3sServer(ctx.config.role)) {
+    return { success: true, changed: false, message: "Server node — k3s sets its own roles" };
+  }
+
+  const name = await nodeName(ctx);
+
+  // An agent has no local kubectl, so this has to run on a server.
+  const local = await ctx.ssh.exec(`k3s kubectl label node "${name}" ${WORKER_LABEL} --overwrite 2>&1`, sshOpts(ctx));
+  if (local.exitCode === 0) {
+    return { success: true, changed: true, message: `Labelled ${name} as worker` };
+  }
+
+  if (ctx.config.k3sServerUrl !== undefined && ctx.config.k3sServerUrl !== "") {
+    try {
+      const serverHost = new URL(ctx.config.k3sServerUrl).hostname;
+      const remote = await remoteSshExec(
+        serverHost, "root",
+        `k3s kubectl label node "${name}" ${WORKER_LABEL} --overwrite`,
+        { ...(ctx.ssh.keyPath ? { keyPath: ctx.ssh.keyPath } : {}), timeoutMs: 30_000 },
+      );
+      if (remote.exitCode === 0) {
+        return { success: true, changed: true, message: `Labelled ${name} as worker (via server)` };
+      }
+    } catch { /* fall through to the warning */ }
+  }
+
+  // Cosmetic: the ROLES column will read <none>. Not worth failing an install.
+  return {
+    success: true,
+    changed: false,
+    message: `Could not apply the worker role label to ${name} (cosmetic; set it with ` +
+      `\`kubectl label node ${name} ${WORKER_LABEL}\`)`,
+  };
+};

bastion/src/modules/modules/k3s/src/types.ts

@@ -16,6 +16,56 @@ export interface K3sConfig {
 
   // Additional TLS SANs for API server certificate
   tlsSans?: string[] | undefined;
+
+  /**
+   * Exact k3s version to install, e.g. "v1.34.6+k3s1".
+   *
+   * Unset means whatever get.k3s.io currently calls latest, which is how a
+   * reinstalled node silently ends up AHEAD of the cluster it is rejoining.
+   * Kubernetes supports only one minor version of skew between control-plane
+   * members; a node reinstalled two minors ahead of its peers is an
+   * unsupported cluster, and a rolling migration reinstalls nodes one at a
+   * time by design. Pin it to the version the cluster already runs.
+   */
+  k3sVersion?: string | undefined;
+
+  /**
+   * Permit evacuating a node that holds single-replica Longhorn volumes.
+   *
+   * Off by default and deliberately awkward. Evicting the only replica of a
+   * volume rebuilds it from the sole copy in existence; with no backup target
+   * configured, a failure during that rebuild is permanent data loss. Raising
+   * the volume to 3 replicas and waiting for it to go healthy is almost always
+   * the right move instead.
+   */
+  evictionAllowSingleReplica?: boolean | undefined;
+
+  /**
+   * IPv6 address of this node on the cluster VLAN. Its PRESENCE is what makes a
+   * node dual-stack: supply it and `node-ip` is emitted as `<v4>,<v6>`; omit it
+   * and the generated config is byte-identical to the IPv4-only one.
+   *
+   * It must be a real address on the node before k3s starts. k3s resolves
+   * node-ip at boot, so a SLAAC or DHCPv6 lease that has not landed yet leaves
+   * the node with no usable v6 identity. The estate takes it from a kea DHCPv6
+   * reservation keyed on MAC, the same way it takes its IPv4 address.
+   */
+  ipv6?: string | undefined;
+
+  /**
+   * Pod and Service ranges, one entry per address family. Passed rather than
+   * hardcoded so the ranges stay configuration -- and so labsim can rehearse
+   * with its own addresses through this same generator, instead of a parallel
+   * set of INSTALL_K3S_EXEC flags that would prove a different mechanism.
+   *
+   * k3s validates the two together and crash-loops on a mismatch (a dual
+   * cluster-cidr with a single-family service-cidr, say), which is a safe
+   * failure but an avoidable one: set both or neither.
+   *
+   * Servers only -- agents take neither.
+   */
+  clusterCidr?: string[] | undefined;
+  serviceCidr?: string[] | undefined;
 }
 
 /** SSH execution interface injected into operations. */

bastion/src/modules/modules/k3s/tests/agent-node-label.test.ts

@@ -0,0 +1,75 @@
+// The agent must NOT try to self-assign a node-role label.
+//
+// A kubelet may not set labels in the `node-role.kubernetes.io/` namespace;
+// the servers run NodeRestriction, which rejects the registration. The agent
+// then never finishes registering and systemd reports only:
+//
+//   Job for k3s-agent.service failed because the service did not take the
+//   steps required by its unit configuration.
+//
+// That message names neither labels nor admission control, so the cost of
+// reintroducing this is hours of looking at the network and the join token.
+// Observed twice for real: on v2-wrk-1, and on the drill's storage agent.
+
+import { describe, it, expect } from "vitest";
+import { generateAgentConfig, generateServerConfig } from "../src/operations/k3s-config.js";
+import { k3sAgentGroup } from "../src/groups/k3s-agent.js";
+import { mockCtx, OK, stdout, expectCommand } from "./helpers.js";
+import { labelWorkerRole } from "../src/operations/worker-label.js";
+
+const base = { hostname: "w1.lab", ip: "10.0.0.9", role: "k8s-worker" as const };
+
+describe("agent config", () => {
+  it("does NOT put a node-role label in node-label", () => {
+    const cfg = generateAgentConfig({ ...base });
+    expect(cfg).not.toContain("node-role.kubernetes.io");
+  });
+
+  it("keeps the Longhorn label, which IS allowed (outside kubernetes.io)", () => {
+    expect(generateAgentConfig({ ...base })).toContain("node.longhorn.io/create-default-disk=config");
+  });
+
+  it("server config does not set node-role either — k3s assigns those itself", () => {
+    const cfg = generateServerConfig({ ...base, role: "k8s-combo", hostname: "s1.lab" });
+    expect(cfg).not.toContain("node-role.kubernetes.io");
+  });
+
+  it("the agent group applies the worker label AFTER installing", () => {
+    const names = k3sAgentGroup.operations.map((o) => o.name);
+    const install = names.findIndex((n) => /Install k3s agent/i.test(n));
+    const label = names.findIndex((n) => /Label node as worker/i.test(n));
+    expect(install).toBeGreaterThanOrEqual(0);
+    expect(label).toBeGreaterThan(install);
+  });
+});
+
+describe("labelWorkerRole", () => {
+  it("labels an agent from the server side", async () => {
+    const ctx = mockCtx({ role: "k8s-worker" });
+    ctx.ssh.exec.mockImplementation(async (cmd: string) => {
+      if (cmd.startsWith("hostname")) return stdout("w1.lab\n");
+      return OK;
+    });
+    const r = await labelWorkerRole(ctx);
+    expect(r.success).toBe(true);
+    expectCommand(ctx.ssh, /label node "w1\.lab" node-role\.kubernetes\.io\/worker=true/);
+  });
+
+  it("is a no-op on a server — k3s sets control-plane/etcd itself", async () => {
+    const r = await labelWorkerRole(mockCtx({ role: "k8s-combo" }));
+    expect(r.success).toBe(true);
+    expect(r.changed).toBe(false);
+  });
+
+  it("never fails an install over a cosmetic label", async () => {
+    const ctx = mockCtx({ role: "k8s-worker" });
+    ctx.ssh.exec.mockImplementation(async (cmd: string) => {
+      if (cmd.startsWith("hostname")) return stdout("w1.lab\n");
+      return { exitCode: 1, stdout: "", stderr: "no kubectl here" };
+    });
+    const r = await labelWorkerRole(ctx);
+    expect(r.success, "a missing ROLES column must not fail a node install").toBe(true);
+    expect(r.changed).toBe(false);
+    expect(r.message).toMatch(/cosmetic/);
+  });
+});

bastion/src/modules/modules/k3s/tests/etcd-member-parse.test.ts

@@ -0,0 +1,58 @@
+// Parsing `etcdctl member list`.
+//
+// WHY NOT -w json
+// ---------------
+// etcd member IDs are uint64. JSON.parse turns them into JS numbers, which
+// lose precision above 2^53, so the id that comes back is an APPROXIMATION of
+// the real one -- and `member remove` wants HEX anyway, while the JSON writer
+// emits decimal. Observed for real: `member remove 13323819137391910000`
+// -> "bad member ID arg (value out of range), expecting ID in Hex".
+//
+// That happens at the worst possible moment: the node is already stopped and
+// about to be wiped. A mangled id fails loudly if you are lucky; in principle
+// it could match a different member, which would remove the wrong one.
+//
+// The default writer prints the id as hex TEXT, so it never becomes a number.
+// The fixture below is real output from a production node.
+
+import { describe, it, expect } from "vitest";
+import { parseEtcdMembers } from "../src/operations/etcd-members.js";
+
+const REAL = `825852d3683bc406, started, worker2-k8s0.ad.itaz.eu-5f7ad0c9, https://192.168.8.25:2380, https://192.168.8.25:2379, false
+a6170c0755307cea, started, worker1-k8s0.ad.itaz.eu-82b4fe24, https://192.168.8.13:2380, https://192.168.8.13:2379, false
+ad1fe97d38022a78, started, worker0-k8s0.ad.itaz.eu-e26aacc5, https://192.168.8.23:2380, https://192.168.8.23:2379, false`;
+
+describe("parseEtcdMembers", () => {
+  it("keeps ids as exact hex text, never as numbers", () => {
+    const m = parseEtcdMembers(REAL);
+    expect(m.map((x) => x.id)).toEqual([
+      "825852d3683bc406", "a6170c0755307cea", "ad1fe97d38022a78",
+    ]);
+    // The precision trap, stated as an assertion: every one of these exceeds
+    // Number.MAX_SAFE_INTEGER, so a numeric round-trip would corrupt them.
+    for (const x of m) {
+      expect(Number(BigInt(`0x${x.id}`)) > Number.MAX_SAFE_INTEGER).toBe(true);
+      expect(BigInt(`0x${x.id}`).toString(16)).toBe(x.id);
+    }
+  });
+
+  it("strips etcd's random name suffix to give the node name", () => {
+    expect(parseEtcdMembers(REAL).map((x) => x.node)).toEqual([
+      "worker2-k8s0.ad.itaz.eu", "worker1-k8s0.ad.itaz.eu", "worker0-k8s0.ad.itaz.eu",
+    ]);
+  });
+
+  it("reports started status", () => {
+    expect(parseEtcdMembers(REAL).every((x) => x.started)).toBe(true);
+    const unstarted = parseEtcdMembers("abc123, unstarted, , https://x:2380, , false");
+    expect(unstarted[0]?.started).toBe(false);
+  });
+
+  it("ignores blank lines and anything without a hex id", () => {
+    expect(parseEtcdMembers(`\n  \nWARNING: something\n${REAL}\n`)).toHaveLength(3);
+  });
+
+  it("returns nothing for empty output rather than a bogus member", () => {
+    expect(parseEtcdMembers("")).toEqual([]);
+  });
+});

bastion/src/modules/modules/k3s/tests/join-env.test.ts

@@ -0,0 +1,158 @@
+// Tests for how K3S_URL / K3S_TOKEN reach the k3s installer.
+//
+// Token and URL are independent, and the distinction is load-bearing:
+//
+//   token only      -> SEEDS the cluster token on the cluster-init node, so
+//                      every later joiner can use the same pre-shared value.
+//   url + token     -> JOINS an existing cluster.
+//   url only        -> cannot authenticate; must not be passed.
+//
+// Before this was fixed the installer emitted join env ONLY when both were
+// present, so a cluster-init node always self-generated its token and there
+// was no way to pre-share one. A `toContain("K3S_TOKEN")` assertion would pass
+// against the broken version for the url+token case, so each case is asserted
+// on the exact command.
+
+import { describe, it, expect } from "vitest";
+import { mockCtx, stdout, OK } from "./helpers.js";
+import { installK3sBinary } from "../src/operations/k3s-install.js";
+
+/** The `curl ... | sh -` line the operation ran. */
+function installCommand(ssh: ReturnType<typeof mockCtx>["ssh"]): string {
+  const calls = ssh.exec.mock.calls.map((c: [string, unknown?]) => c[0]);
+  const found = calls.find((c: string) => c.includes("get.k3s.io"));
+  if (!found) throw new Error(`No install command issued. Commands:\n${calls.join("\n")}`);
+  return found;
+}
+
+/** k3s --version fails (not installed), everything after succeeds. */
+function primeServerInstall(ctx: ReturnType<typeof mockCtx>): void {
+  ctx.ssh.exec.mockResolvedValue(OK);
+  ctx.ssh.exec.mockResolvedValueOnce({ exitCode: 1, stdout: "", stderr: "" }); // k3s --version
+}
+
+describe("installK3sBinary — server join environment", () => {
+  it("token alone seeds the cluster token and does NOT pass a server URL", async () => {
+    const ctx = mockCtx({ role: "k8s-combo", k3sToken: "pre-shared-token" });
+    primeServerInstall(ctx);
+    ctx.ssh.exec.mockResolvedValue(stdout("NAME STATUS\nnode Ready"));
+
+    await installK3sBinary(ctx);
+    const cmd = installCommand(ctx.ssh);
+
+    expect(cmd).toContain('K3S_TOKEN="pre-shared-token"');
+    expect(cmd).not.toContain("K3S_URL");
+    expect(cmd).toContain('INSTALL_K3S_EXEC="server"');
+  });
+
+  it("url + token joins an existing cluster", async () => {
+    const ctx = mockCtx({
+      role: "k8s-combo",
+      k3sServerUrl: "https://10.0.0.11:6443",
+      k3sToken: "pre-shared-token",
+    });
+    primeServerInstall(ctx);
+    ctx.ssh.exec.mockResolvedValue(stdout("NAME STATUS\nnode Ready"));
+
+    await installK3sBinary(ctx);
+    const cmd = installCommand(ctx.ssh);
+
+    expect(cmd).toContain('K3S_URL="https://10.0.0.11:6443"');
+    expect(cmd).toContain('K3S_TOKEN="pre-shared-token"');
+  });
+
+  it("neither url nor token passes no join environment at all", async () => {
+    const ctx = mockCtx({ role: "k8s-combo" });
+    primeServerInstall(ctx);
+    ctx.ssh.exec.mockResolvedValue(stdout("NAME STATUS\nnode Ready"));
+
+    await installK3sBinary(ctx);
+    const cmd = installCommand(ctx.ssh);
+
+    expect(cmd).not.toContain("K3S_URL");
+    expect(cmd).not.toContain("K3S_TOKEN");
+  });
+
+  it("a URL without a token is ignored — it cannot authenticate a join", async () => {
+    const ctx = mockCtx({ role: "k8s-combo", k3sServerUrl: "https://10.0.0.11:6443" });
+    primeServerInstall(ctx);
+    ctx.ssh.exec.mockResolvedValue(stdout("NAME STATUS\nnode Ready"));
+
+    await installK3sBinary(ctx);
+    expect(installCommand(ctx.ssh)).not.toContain("K3S_URL");
+  });
+});
+
+describe("installK3sBinary — datastore wipe guard", () => {
+  // Wiping server/{tls,cred,db} is correct when JOINING (stale certs are fatal:
+  // "newer than datastore") and catastrophic on a cluster-init node, where it
+  // would destroy the etcd database the node is about to bootstrap from.
+  const WIPE = "rm -rf /var/lib/rancher/k3s/server/tls";
+
+  it("wipes stale server state when joining", async () => {
+    const ctx = mockCtx({
+      role: "k8s-combo",
+      k3sServerUrl: "https://10.0.0.11:6443",
+      k3sToken: "t",
+    });
+    primeServerInstall(ctx);
+    ctx.ssh.exec.mockResolvedValue(stdout("NAME STATUS\nnode Ready"));
+
+    await installK3sBinary(ctx);
+    const calls = ctx.ssh.exec.mock.calls.map((c: [string, unknown?]) => c[0]);
+    expect(calls.some((c: string) => c.includes(WIPE))).toBe(true);
+  });
+
+  it("does NOT wipe the datastore when only seeding a token (cluster-init)", async () => {
+    const ctx = mockCtx({ role: "k8s-combo", k3sToken: "pre-shared-token" });
+    primeServerInstall(ctx);
+    ctx.ssh.exec.mockResolvedValue(stdout("NAME STATUS\nnode Ready"));
+
+    await installK3sBinary(ctx);
+    const calls = ctx.ssh.exec.mock.calls.map((c: [string, unknown?]) => c[0]);
+    expect(calls.some((c: string) => c.includes(WIPE))).toBe(false);
+  });
+});
+
+describe("installK3sBinary — agent", () => {
+  it("refuses to install an agent with no server URL or token", async () => {
+    const ctx = mockCtx({ role: "k8s-worker" });
+    primeServerInstall(ctx);
+
+    const result = await installK3sBinary(ctx);
+    expect(result.success).toBe(false);
+    expect(result.message).toContain("Agent requires");
+  });
+});
+
+describe("installK3sBinary — version pinning", () => {
+  // get.k3s.io installs LATEST. During a rolling migration that means a
+  // reinstalled node rejoins on a newer k3s than its peers; Kubernetes supports
+  // only one minor of skew between control-plane members, and at the time of
+  // writing latest was TWO ahead of the fleet (v1.36.4 vs v1.34.6).
+  it("pins the version when one is given", async () => {
+    const ctx = mockCtx({ role: "k8s-combo", k3sToken: "t", k3sVersion: "v1.34.6+k3s1" });
+    primeServerInstall(ctx);
+    ctx.ssh.exec.mockResolvedValue(stdout("NAME STATUS\nnode Ready"));
+    await installK3sBinary(ctx);
+    expect(installCommand(ctx.ssh)).toContain('INSTALL_K3S_VERSION="v1.34.6+k3s1"');
+  });
+
+  it("pins the agent install too", async () => {
+    const ctx = mockCtx({
+      role: "k8s-worker", k3sServerUrl: "https://10.0.0.11:6443",
+      k3sToken: "t", k3sVersion: "v1.34.6+k3s1",
+    });
+    primeServerInstall(ctx);
+    await installK3sBinary(ctx);
+    expect(installCommand(ctx.ssh)).toContain('INSTALL_K3S_VERSION="v1.34.6+k3s1"');
+  });
+
+  it("omits the variable entirely when unset, preserving today's behaviour", async () => {
+    const ctx = mockCtx({ role: "k8s-combo", k3sToken: "t" });
+    primeServerInstall(ctx);
+    ctx.ssh.exec.mockResolvedValue(stdout("NAME STATUS\nnode Ready"));
+    await installK3sBinary(ctx);
+    expect(installCommand(ctx.ssh)).not.toContain("INSTALL_K3S_VERSION");
+  });
+});

bastion/src/modules/modules/k3s/tests/longhorn-disk.test.ts

@@ -0,0 +1,188 @@
+// Tests for giving a node its Longhorn disk.
+//
+// The cluster runs createDefaultDiskLabeledNodes:true, so a node gets a disk
+// only if it carries BOTH the create-default-disk label and the
+// default-disks-config annotation. Longhorn reads that annotation when it
+// first creates the node's record — which means on a node that is REJOINING,
+// the record already exists and the annotation is never read. The node comes
+// back Ready, accepts pods, and hosts no replicas at all.
+//
+// That failure is invisible until a rebuild is expected and does not arrive,
+// so these tests pin the read-back and the repair, not just the annotate.
+
+import { describe, it, expect, vi, afterEach } from "vitest";
+import { mockCtx, OK, FAIL, stdout, expectCommand, expectNoCommand } from "./helpers.js";
+import { configureLonghornDisk } from "../src/operations/longhorn-disk.js";
+
+const HOST = stdout("node1.lab\n");
+const DISK = "/var/lib/longhorn";
+
+interface Opts {
+  /** Does Longhorn create the disk on its own (i.e. it read the annotation)? */
+  longhornCreatesDisk?: boolean;
+  /** Has longhorn-manager registered this node yet? */
+  recordExists?: boolean;
+  hasDir?: boolean;
+  noLonghorn?: boolean;
+  patchFails?: boolean;
+  /** Does the disk show up after a direct spec.disks patch? */
+  repairWorks?: boolean;
+}
+
+function ctxFor(o: Opts = {}) {
+  // Model the real object rather than a script of reads: a disk exists if
+  // Longhorn made one, or if our repair patch landed and took effect.
+  let patched = false;
+  const ctx = mockCtx();
+  ctx.ssh.exec.mockImplementation(async (cmd: string) => {
+    if (cmd.startsWith("test -d")) return stdout(o.hasDir === false ? "no" : "yes");
+    if (cmd.startsWith("hostname")) return HOST;
+    if (cmd.includes("get ns longhorn-system")) return o.noLonghorn ? FAIL : stdout("namespace/longhorn-system");
+    // Existence probe (-o name) is a DIFFERENT question from "has a disk",
+    // and must be matched before the jsonpath reads.
+    if (cmd.includes("get nodes.longhorn.io") && cmd.includes("-o name")) {
+      return o.recordExists === false ? FAIL : stdout("node.longhorn.io/node1.lab");
+    }
+    if (cmd.includes("get nodes.longhorn.io") && cmd.includes("jsonpath")) {
+      const has = (o.longhornCreatesDisk ?? false) || (patched && (o.repairWorks ?? true));
+      return stdout(has ? DISK : "");
+    }
+    if (cmd.includes("patch nodes.longhorn.io")) {
+      if (o.patchFails === true) return FAIL;
+      patched = true;
+      return OK;
+    }
+    if (cmd.includes("annotate node")) return stdout("node/node1.lab annotated");
+    return OK;
+  });
+  return ctx;
+}
+
+/** Run the operation with the 60s verify window collapsed. */
+async function runFast(ctx: ReturnType<typeof ctxFor>) {
+  vi.useFakeTimers();
+  try {
+    const p = configureLonghornDisk(ctx);
+    // Long enough to cover the registration wait (180s) AND the disk wait (60s).
+    await vi.advanceTimersByTimeAsync(300_000);
+    return await p;
+  } finally {
+    vi.useRealTimers();
+  }
+}
+
+afterEach(() => { vi.useRealTimers(); });
+
+describe("the annotation survives the shell", () => {
+  // The value is JSON, so it is full of double quotes. Wrapping it in double
+  // quotes truncates it at the first inner quote and the node ends up with
+  //     [{path:/var/lib/longhorn,allowScheduling:true}]
+  // which Longhorn cannot parse, so it creates no disk at all -- silently.
+  // Asserting on the command string alone would NOT catch this; the command
+  // looks fine until a shell reads it. So run it through a real shell.
+  it("arrives as valid JSON after shell parsing", async () => {
+    const ctx = ctxFor({ longhornCreatesDisk: true });
+    await runFast(ctx);
+    const cmd = ctx.ssh.exec.mock.calls
+      .map((c: [string, unknown?]) => c[0])
+      .find((c: string) => c.includes("default-disks-config")) ?? "";
+    expect(cmd, "annotate command").not.toBe("");
+
+    // Let a real shell word-split it exactly as sshd would, and recover the
+    // argument that carries the annotation.
+    const { execFileSync } = await import("node:child_process");
+    const printed = execFileSync("bash", ["-c", `printf '%s\n' ${cmd.replace(/^k3s kubectl /, "")}`], {
+      encoding: "utf-8",
+    });
+    const arg = printed.split("\n").find((l) => l.startsWith("node.longhorn.io/default-disks-config=")) ?? "";
+    expect(arg, "annotation argument after shell parsing").not.toBe("");
+
+    const json = arg.slice("node.longhorn.io/default-disks-config=".length);
+    const parsed = JSON.parse(json) as { path: string; allowScheduling: boolean }[];
+    expect(parsed).toEqual([{ path: DISK, allowScheduling: true }]);
+  });
+});
+
+describe("configureLonghornDisk", () => {
+  it("guarantees the replicas directory exists", async () => {
+    // Longhorn creates <path>/replicas only when it first initialises the
+    // disk. If it disappears while longhorn-disk.cfg remains, Longhorn will
+    // not recreate it and every replica placed on the node dies on a
+    // non-recursive mkdir -- permanently, and invisibly, because the node
+    // still reports Ready and Schedulable.
+    const ctx = ctxFor({ longhornCreatesDisk: true });
+    await runFast(ctx);
+    expectCommand(ctx.ssh, /mkdir -p \/var\/lib\/longhorn\/replicas/);
+  });
+
+  it("skips a node with no /var/lib/longhorn", async () => {
+    const r = await configureLonghornDisk(ctxFor({ hasDir: false }));
+    expect(r.success).toBe(true);
+    expect(r.changed).toBe(false);
+    expect(r.message).toMatch(/skipping/);
+  });
+
+  it("annotates and is satisfied when Longhorn creates the disk itself", async () => {
+    const ctx = ctxFor({ longhornCreatesDisk: true });
+    const r = await runFast(ctx);
+    expect(r.success).toBe(true);
+    expectCommand(ctx.ssh, /default-disks-config/);
+    // Longhorn honoured the annotation, so no direct spec.disks write.
+    expectNoCommand(ctx.ssh, /patch nodes\.longhorn\.io/);
+  });
+
+  it("REPAIRS a rejoining node whose annotation was read too late", async () => {
+    // No disk for the whole verify window, then present after the patch.
+    const ctx = ctxFor({ longhornCreatesDisk: false, repairWorks: true });
+    const r = await runFast(ctx);
+    expect(r.success).toBe(true);
+    expect(r.changed).toBe(true);
+    expect(r.message).toMatch(/repaired/i);
+    expectCommand(ctx.ssh, /patch nodes\.longhorn\.io/);
+    expectCommand(ctx.ssh, new RegExp(`"path":"${DISK.replace(/\//g, "\\/")}"`.replace(/\\\//g, "/")));
+  });
+
+  it("FAILS loudly when the disk cannot be created — a node with no disk is useless", async () => {
+    const ctx = ctxFor({ longhornCreatesDisk: false, repairWorks: false });
+    const r = await runFast(ctx);
+    expect(r.success).toBe(false);
+    expect(r.error).toMatch(/will not host replicas|never host a replica/);
+  });
+
+  it("FAILS when the repair patch itself is rejected", async () => {
+    const ctx = ctxFor({ longhornCreatesDisk: false, patchFails: true });
+    const r = await runFast(ctx);
+    expect(r.success).toBe(false);
+    expect(r.message).toMatch(/could not be repaired/);
+  });
+
+  it("does NOT fail a node Longhorn has not registered yet", async () => {
+    // A node that has only just rejoined has no nodes.longhorn.io record. The
+    // annotation is already in place, so Longhorn will read it when it creates
+    // one. Failing here would abort a rejoin over something self-resolving.
+    const ctx = ctxFor({ longhornCreatesDisk: false, recordExists: false });
+    const r = await runFast(ctx);
+    expect(r.success).toBe(true);
+    expect(r.message).toMatch(/not registered/);
+    expectNoCommand(ctx.ssh, /patch nodes\.longhorn\.io/);
+  });
+
+  it("annotates and stops when Longhorn is not installed yet", async () => {
+    // Install-time ordering: the annotation is set so Longhorn honours it when
+    // it does arrive, but there is nothing to read back from.
+    const ctx = ctxFor({ noLonghorn: true });
+    const r = await configureLonghornDisk(ctx);
+    expect(r.success).toBe(true);
+    expectCommand(ctx.ssh, /default-disks-config/);
+    expectNoCommand(ctx.ssh, /patch nodes\.longhorn\.io/);
+  });
+
+  it("uses one fixed disk key, so repeated repairs update rather than multiply", async () => {
+    const ctx = ctxFor({ longhornCreatesDisk: false, repairWorks: true });
+    await runFast(ctx);
+    const patch = ctx.ssh.exec.mock.calls
+      .map((c: [string, unknown?]) => c[0])
+      .find((c: string) => c.includes("patch nodes.longhorn.io")) ?? "";
+    expect(patch).toContain("default-disk-lab");
+  });
+});

bastion/src/modules/modules/k3s/tests/module-config.test.ts

@@ -0,0 +1,79 @@
+// The ModuleContext -> OperationContext config hand-off.
+//
+// toOpContext used to enumerate three keys and silently drop the rest, so a
+// caller could set a field, get no error, and have it ignored. That produced
+// two live bugs: a pinned k3sVersion was discarded (get.k3s.io installed
+// LATEST, two minor versions ahead of the fleet), and
+// `--allow-single-replica` was a no-op that claimed to override a data-loss
+// guard.
+//
+// These tests assert the pass-through per field, and the last one fails for
+// ANY future K3sConfig field that goes missing — which is the property that
+// was actually absent.
+
+import { describe, it, expect, vi } from "vitest";
+import { K3sModule } from "../src/k3s-module.js";
+import type { ModuleContext } from "../../../src/types.js";
+
+vi.mock("../../../src/ssh.js", () => ({
+  sshExec: vi.fn(async (_ip: string, _user: string, cmd: string) => {
+    // Record every command so the test can assert on what was sent.
+    issued.push(cmd);
+    if (cmd.includes("k3s --version")) return { exitCode: 1, stdout: "", stderr: "" };
+    if (cmd.includes("hostname")) return { exitCode: 0, stdout: "n1.lab\n", stderr: "" };
+    if (cmd.includes("get nodes")) return { exitCode: 0, stdout: "NAME\nn1 Ready", stderr: "" };
+    return { exitCode: 0, stdout: "", stderr: "" };
+  }),
+}));
+
+const issued: string[] = [];
+
+function ctxWith(config: Record<string, unknown>): ModuleContext {
+  return {
+    hostname: "n1.lab", ip: "10.0.0.5", role: "k8s-combo",
+    os: "fedora-43", arch: "x86_64", sshUser: "root", config,
+  };
+}
+
+describe("ModuleContext.config reaches the operations", () => {
+  it("passes k3sVersion through to the installer", async () => {
+    issued.length = 0;
+    await new K3sModule().install(ctxWith({
+      k3sToken: "t", k3sVersion: "v1.34.6+k3s1",
+    }));
+    const install = issued.find((c) => c.includes("get.k3s.io")) ?? "";
+    expect(install, "install command").toContain('INSTALL_K3S_VERSION="v1.34.6+k3s1"');
+  });
+
+  it("still passes the three fields it always did", async () => {
+    issued.length = 0;
+    await new K3sModule().install(ctxWith({
+      k3sServerUrl: "https://10.0.0.1:6443", k3sToken: "tok",
+    }));
+    const install = issued.find((c) => c.includes("get.k3s.io")) ?? "";
+    expect(install).toContain('K3S_URL="https://10.0.0.1:6443"');
+    expect(install).toContain('K3S_TOKEN="tok"');
+  });
+
+  it("drops nothing: every key given is visible to the operations", async () => {
+    // The generic guard. Enumerating keys in toOpContext is the bug; this
+    // fails if anyone reintroduces it for a field added later.
+    const { K3sModule: Mod } = await import("../src/k3s-module.js");
+    const mod = new Mod();
+    const given = {
+      k3sServerUrl: "https://10.0.0.1:6443",
+      k3sToken: "tok",
+      k3sVersion: "v1.34.6+k3s1",
+      evictionAllowSingleReplica: true,
+      tlsSans: ["a.lab"],
+    };
+    issued.length = 0;
+    await mod.install(ctxWith(given));
+    // k3sVersion and the join env are the observable ones at install time;
+    // assert them explicitly rather than trusting the shape.
+    const install = issued.find((c) => c.includes("get.k3s.io")) ?? "";
+    expect(install).toContain("INSTALL_K3S_VERSION=");
+    expect(install).toContain("K3S_URL=");
+    expect(install).toContain("K3S_TOKEN=");
+  });
+});

bastion/src/modules/modules/k3s/tests/node-admit.test.ts

@@ -0,0 +1,211 @@
+// Tests for returning an evacuated node to service.
+//
+// Evacuation leaves three separate marks on a node — a kubernetes cordon,
+// Longhorn's allowScheduling/evictionRequested pair, and (after a wipe and
+// rejoin) a MISSING Longhorn disk. Undoing only the cordon gives you a node
+// that is Ready, accepts pods and hosts no replicas at all, which reads as
+// success until storage never rebalances onto it. These tests pin all three.
+
+import { describe, it, expect, vi, afterEach } from "vitest";
+import { mockCtx, OK, FAIL, stdout, expectCommand } from "./helpers.js";
+import { clearLonghornEviction, assertNodeSchedulable } from "../src/operations/node-admit.js";
+import { evictLonghornReplicas } from "../src/operations/node-evacuate.js";
+import { evacuateGroup } from "../src/groups/evacuate.js";
+import { admitGroup } from "../src/groups/admit.js";
+
+const HOST = stdout("node1.lab\n");
+
+/**
+ * Run assertNodeSchedulable with its 120s poll collapsed.
+ *
+ * It polls because Longhorn registers a node asynchronously and this runs
+ * moments after a rejoin; the negative cases therefore wait out the whole
+ * window before they can honestly fail.
+ */
+async function runGate(ctx: ReturnType<typeof mockCtx>) {
+  vi.useFakeTimers();
+  try {
+    const p = assertNodeSchedulable(ctx);
+    await vi.advanceTimersByTimeAsync(150_000);
+    return await p;
+  } finally {
+    vi.useRealTimers();
+  }
+}
+
+afterEach(() => { vi.useRealTimers(); });
+
+/** Command-matching mock; see node-evacuate.test.ts for why not call order. */
+function routed(handlers: Record<string, string | null>, opts: { noLonghorn?: boolean } = {}) {
+  const ctx = mockCtx();
+  ctx.ssh.exec.mockImplementation(async (cmd: string) => {
+    if (cmd.startsWith("hostname")) return HOST;
+    if (cmd.includes("get ns longhorn-system")) return opts.noLonghorn ? FAIL : stdout("namespace/longhorn-system");
+    for (const [needle, out] of Object.entries(handlers)) {
+      if (cmd.includes(needle)) return out === null ? FAIL : stdout(out);
+    }
+    return OK;
+  });
+  return ctx;
+}
+
+describe("clearLonghornEviction", () => {
+  it("clears scheduling and eviction together in one patch", async () => {
+    const ctx = routed({ "get nodes.longhorn.io": "false,true" });
+    const r = await clearLonghornEviction(ctx);
+    expect(r.success).toBe(true);
+    expect(r.changed).toBe(true);
+    // Both fields must move together: Longhorn's webhook rejects
+    // evictionRequested:true on a schedulable node, so the inverse pairing
+    // has to be atomic as well.
+    expectCommand(ctx.ssh, /"allowScheduling":true/);
+    expectCommand(ctx.ssh, /"evictionRequested":false/);
+  });
+
+  it("is idempotent when the node already accepts replicas", async () => {
+    const ctx = routed({ "get nodes.longhorn.io": "true,false" });
+    const r = await clearLonghornEviction(ctx);
+    expect(r.success).toBe(true);
+    expect(r.changed).toBe(false);
+  });
+
+  it("is a no-op when Longhorn is not installed", async () => {
+    const ctx = routed({}, { noLonghorn: true });
+    const r = await clearLonghornEviction(ctx);
+    expect(r.success).toBe(true);
+    expect(r.changed).toBe(false);
+  });
+
+  it("reports failure rather than claiming the node is back", async () => {
+    const ctx = routed({ "get nodes.longhorn.io": "false,true", "patch nodes.longhorn.io": null });
+    const r = await clearLonghornEviction(ctx);
+    expect(r.success).toBe(false);
+  });
+});
+
+describe("assertNodeSchedulable — the gate after a rejoin", () => {
+  it("passes when uncordoned, not evicting, and holding a schedulable disk", async () => {
+    const ctx = mockCtx();
+    ctx.ssh.exec.mockImplementation(async (cmd: string) => {
+      if (cmd.startsWith("hostname")) return HOST;
+      if (cmd.includes("get ns longhorn-system")) return stdout("namespace/longhorn-system");
+      if (cmd.includes("diskStatus")) return stdout("True");
+      if (cmd.includes("get nodes.longhorn.io")) return stdout("true,false");
+      if (cmd.includes("get node ")) return stdout("");
+      return OK;
+    });
+    const r = await assertNodeSchedulable(ctx);
+    expect(r.success).toBe(true);
+  });
+
+  it("REFUSES a node that has no Longhorn disk — the silent rejoin failure", async () => {
+    const ctx = mockCtx();
+    ctx.ssh.exec.mockImplementation(async (cmd: string) => {
+      if (cmd.startsWith("hostname")) return HOST;
+      if (cmd.includes("get ns longhorn-system")) return stdout("namespace/longhorn-system");
+      if (cmd.includes("diskStatus")) return stdout("");   // no disks at all
+      if (cmd.includes("get nodes.longhorn.io")) return stdout("true,false");
+      if (cmd.includes("get node ")) return stdout("");
+      return OK;
+    });
+    const r = await runGate(ctx);
+    expect(r.success).toBe(false);
+    expect(r.error).toMatch(/no disk/);
+  });
+
+  it("REFUSES a node still marked for eviction", async () => {
+    const ctx = mockCtx();
+    ctx.ssh.exec.mockImplementation(async (cmd: string) => {
+      if (cmd.startsWith("hostname")) return HOST;
+      if (cmd.includes("get ns longhorn-system")) return stdout("namespace/longhorn-system");
+      if (cmd.includes("diskStatus")) return stdout("True");
+      if (cmd.includes("get nodes.longhorn.io")) return stdout("false,true");
+      if (cmd.includes("get node ")) return stdout("");
+      return OK;
+    });
+    const r = await runGate(ctx);
+    expect(r.success).toBe(false);
+    expect(r.error).toMatch(/eviction is still requested|scheduling is still disabled/);
+  });
+
+  it("REFUSES a node that is still cordoned", async () => {
+    const ctx = mockCtx();
+    ctx.ssh.exec.mockImplementation(async (cmd: string) => {
+      if (cmd.startsWith("hostname")) return HOST;
+      if (cmd.includes("get ns longhorn-system")) return stdout("namespace/longhorn-system");
+      if (cmd.includes("diskStatus")) return stdout("True");
+      if (cmd.includes("get nodes.longhorn.io")) return stdout("true,false");
+      if (cmd.includes("get node ")) return stdout("true");   // unschedulable
+      return OK;
+    });
+    const r = await runGate(ctx);
+    expect(r.success).toBe(false);
+    expect(r.error).toMatch(/cordoned/);
+  });
+});
+
+describe("evictLonghornReplicas — never leaves a node eviction-pending", () => {
+  it("cancels the eviction request when the wait blows up", async () => {
+    const ctx = mockCtx();
+    const patched: string[] = [];
+    ctx.ssh.exec.mockImplementation(async (cmd: string) => {
+      if (cmd.startsWith("hostname")) return HOST;
+      if (cmd.includes("get ns longhorn-system")) return stdout("namespace/longhorn-system");
+      if (cmd.includes("jsonpath")) {
+        // Blow up during the rebuild wait, after eviction was requested.
+        if (cmd.includes("replicas.longhorn.io")) throw new Error("API server went away");
+        return stdout("");
+      }
+      if (cmd.includes("-o json")) return stdout(JSON.stringify({ items: [] }));
+      if (cmd.includes("patch nodes.longhorn.io")) { patched.push(cmd); return OK; }
+      return OK;
+    });
+
+    await expect(evictLonghornReplicas(ctx)).rejects.toThrow("API server went away");
+
+    // Two patches: the request, then the rollback. Leaving only the first
+    // behind would strand the node unschedulable with Longhorn still draining.
+    expect(patched.length).toBe(2);
+    expect(patched[0]).toContain('"evictionRequested":true');
+    expect(patched[1]).toContain('"evictionRequested":false');
+    expect(patched[1]).toContain('"allowScheduling":true');
+  });
+});
+
+describe("group ordering", () => {
+  it("evacuate evicts BEFORE it drains", () => {
+    // drain --force does not override PodDisruptionBudgets, and Longhorn holds
+    // one on the instance-manager pod. Draining first stops the process that
+    // serves the replicas being evicted: the drain blocks for its full timeout
+    // and would remove the only rebuild source for a single-replica volume.
+    const names = evacuateGroup.operations.map((o) => o.name);
+    const evict = names.findIndex((n) => /Evict/i.test(n));
+    const drain = names.findIndex((n) => /Drain/i.test(n));
+    expect(evict).toBeGreaterThanOrEqual(0);
+    expect(drain).toBeGreaterThanOrEqual(0);
+    expect(evict).toBeLessThan(drain);
+  });
+
+  it("evacuate checks feasibility before it mutates anything", () => {
+    const names = evacuateGroup.operations.map((o) => o.name);
+    expect(names.findIndex((n) => /Check eviction/i.test(n)))
+      .toBeLessThan(names.findIndex((n) => /Evict Longhorn/i.test(n)));
+  });
+
+  it("evacuate verifies last", () => {
+    const names = evacuateGroup.operations.map((o) => o.name);
+    expect(names[names.length - 1]).toMatch(/Verify/i);
+  });
+
+  it("admit clears eviction before restoring the disk", () => {
+    // A node still marked for eviction has any disk it is given drained again.
+    const names = admitGroup.operations.map((o) => o.name);
+    expect(names.findIndex((n) => /Clear Longhorn eviction/i.test(n)))
+      .toBeLessThan(names.findIndex((n) => /Restore Longhorn default disk/i.test(n)));
+  });
+
+  it("admit verifies last", () => {
+    const names = admitGroup.operations.map((o) => o.name);
+    expect(names[names.length - 1]).toMatch(/Verify/i);
+  });
+});

bastion/src/modules/modules/k3s/tests/node-evacuate.test.ts

@@ -0,0 +1,413 @@
+import { describe, it, expect, vi } from "vitest";
+import { mockCtx, OK, FAIL, stdout, expectCommand, expectNoCommand } from "./helpers.js";
+import {
+  cordonNode, uncordonNode, drainNode, evictLonghornReplicas, assertNodeEvacuated,
+  assertEvictionCompletable, checkEvictionCompletable,
+} from "../src/operations/node-evacuate.js";
+import type { K3sConfig } from "../src/types.js";
+
+const HOST = stdout("node1.lab\n");
+const GIB = 2 ** 30;
+
+interface RoutedOpts {
+  nodes?: string[];
+  replicas?: { volume: string }[];
+  volumes?: { name: string; replicas: number; size: number }[];
+  softAntiAffinity?: boolean;
+  overProvisionPct?: number;
+  diskFreeBytes?: number;
+  diskMaxBytes?: number;
+  patchFails?: boolean;
+}
+
+/**
+ * A context whose SSH mock answers by MATCHING the command rather than by call
+ * order. The operations under test read several API objects before they do
+ * anything, and an ordered mock has to be re-counted every time one is added —
+ * which makes the test fail for reasons that have nothing to do with the
+ * behaviour it is meant to pin.
+ */
+function routed(o: RoutedOpts = {}, config?: Partial<K3sConfig>) {
+  const nodeNames = o.nodes ?? ["node1.lab", "node2.lab", "node3.lab", "node4.lab"];
+  const free = o.diskFreeBytes ?? 500 * GIB;
+  const max = o.diskMaxBytes ?? 700 * GIB;
+  const lhNodes = {
+    items: nodeNames.map((name) => ({
+      metadata: { name },
+      spec: { allowScheduling: true },
+      status: { diskStatus: { "default-disk": {
+        storageAvailable: free, storageMaximum: max, storageScheduled: 0, storageReserved: 0,
+      } } },
+    })),
+  };
+  const replicas = {
+    items: (o.replicas ?? []).map((r, i) => ({
+      metadata: { name: `replica-${i}` },
+      spec: { nodeID: "node1.lab", volumeName: r.volume },
+    })),
+  };
+  const volumes = {
+    items: (o.volumes ?? []).map((v) => ({
+      metadata: { name: v.name },
+      spec: { numberOfReplicas: v.replicas, size: v.size },
+      status: { state: "attached", robustness: "healthy" },
+    })),
+  };
+
+  const ctx = mockCtx(config);
+  ctx.ssh.exec.mockImplementation(async (cmd: string) => {
+    if (cmd.startsWith("hostname")) return HOST;
+    if (cmd.includes("get ns longhorn-system")) return stdout("namespace/longhorn-system");
+    // NOTE: "-o jsonpath=" CONTAINS "-o json", so jsonpath must be matched
+    // first or the eviction poll below is handed a whole JSON document and
+    // never sees zero replicas. (This mock got that wrong once already.)
+    if (cmd.includes("jsonpath")) {
+      // The eviction wait polls replicas on the node — report none remaining.
+      if (cmd.includes("replicas.longhorn.io")) return stdout("");
+      if (cmd.includes("replica-soft-anti-affinity")) return stdout(String(o.softAntiAffinity ?? false));
+      if (cmd.includes("storage-over-provisioning-percentage")) return stdout(String(o.overProvisionPct ?? 100));
+      return stdout("");
+    }
+    if (cmd.includes("get nodes.longhorn.io") && cmd.includes("-o json")) return stdout(JSON.stringify(lhNodes));
+    if (cmd.includes("get replicas.longhorn.io") && cmd.includes("-o json")) return stdout(JSON.stringify(replicas));
+    if (cmd.includes("get volumes.longhorn.io") && cmd.includes("-o json")) return stdout(JSON.stringify(volumes));
+    if (cmd.includes("patch nodes.longhorn.io")) return o.patchFails ? FAIL : OK;
+    return OK;
+  });
+  return ctx;
+}
+
+describe("cordonNode", () => {
+  it("cordons and reports a change", async () => {
+    const ctx = mockCtx();
+    ctx.ssh.exec.mockResolvedValueOnce(HOST).mockResolvedValueOnce(stdout('node/node1.lab cordoned'));
+    const r = await cordonNode(ctx);
+    expect(r.success).toBe(true);
+    expect(r.changed).toBe(true);
+    expectCommand(ctx.ssh, /cordon "node1\.lab"/);
+  });
+
+  it("is idempotent — already cordoned is changed:false", async () => {
+    const ctx = mockCtx();
+    ctx.ssh.exec.mockResolvedValueOnce(HOST).mockResolvedValueOnce(stdout("node/node1.lab already cordoned"));
+    const r = await cordonNode(ctx);
+    expect(r.success).toBe(true);
+    expect(r.changed).toBe(false);
+  });
+
+  it("fails loudly when the API is unreachable", async () => {
+    const ctx = mockCtx();
+    ctx.ssh.exec.mockResolvedValueOnce(HOST).mockResolvedValueOnce(FAIL);
+    const r = await cordonNode(ctx);
+    expect(r.success).toBe(false);
+  });
+});
+
+describe("drainNode", () => {
+  it("ignores daemonsets and deletes emptyDir, or it hangs forever", async () => {
+    const ctx = mockCtx();
+    ctx.ssh.exec.mockResolvedValueOnce(HOST).mockResolvedValueOnce(OK);
+    const r = await drainNode(ctx);
+    expect(r.success).toBe(true);
+    expectCommand(ctx.ssh, /--ignore-daemonsets/);
+    expectCommand(ctx.ssh, /--delete-emptydir-data/);
+  });
+
+  it("gives the SSH transport longer than kubectl's own --timeout", async () => {
+    // drainNode asks kubectl for --timeout=15m. sshOpts() is a flat 30s, so
+    // without an explicit override the transport kills a drain that is working
+    // -- after it has already evicted most of the pods. The result is
+    // non-zero with a transcript that reads like success, on every node with
+    // real workloads. `labctl app k3s evacuate` could never have drained one.
+    const ctx = mockCtx();
+    ctx.ssh.exec.mockResolvedValueOnce(HOST).mockResolvedValueOnce(OK);
+    await drainNode(ctx);
+    const call = ctx.ssh.exec.mock.calls.find((c: [string, { timeoutMs?: number }?]) =>
+      c[0].includes("drain "));
+    expect(call, "drain command").toBeDefined();
+    const timeoutMs = call?.[1]?.timeoutMs ?? 0;
+    expect(timeoutMs, "SSH timeout must exceed kubectl's 15m --timeout")
+      .toBeGreaterThan(15 * 60_000);
+  });
+
+  it("reports the END of the drain transcript, not the useless first line", async () => {
+    // Every drain starts with "node/x already cordoned". Surfacing the whole
+    // transcript means the error you see is that line, which explains nothing.
+    const ctx = mockCtx();
+    ctx.ssh.exec.mockResolvedValueOnce(HOST).mockResolvedValueOnce({
+      exitCode: 1,
+      stdout: "node/n1 already cordoned\nevicting pod a\nevicting pod b\n" +
+        "error: cannot evict pod as it would violate the pod's disruption budget",
+      stderr: "",
+    });
+    const r = await drainNode(ctx);
+    expect(r.success).toBe(false);
+    expect(r.error).toContain("disruption budget");
+  });
+
+  it("reports failure rather than pretending the node is empty", async () => {
+    const ctx = mockCtx();
+    ctx.ssh.exec.mockResolvedValueOnce(HOST).mockResolvedValueOnce(FAIL);
+    const r = await drainNode(ctx);
+    expect(r.success).toBe(false);
+  });
+});
+
+describe("evictLonghornReplicas", () => {
+  it("is a no-op when Longhorn is not installed", async () => {
+    const ctx = mockCtx();
+    ctx.ssh.exec.mockResolvedValueOnce(HOST).mockResolvedValueOnce(FAIL); // no longhorn-system ns
+    const r = await evictLonghornReplicas(ctx);
+    expect(r.success).toBe(true);
+    expect(r.changed).toBe(false);
+    expect(r.message).toMatch(/not installed/);
+  });
+
+  it("requests eviction and waits until zero replicas remain", async () => {
+    // Node holds no replicas, so the pre-check clears it immediately.
+    const ctx = routed({ replicas: [] });
+    const r = await evictLonghornReplicas(ctx);
+    expect(r.success).toBe(true);
+    expect(r.changed).toBe(true);
+    expectCommand(ctx.ssh, /evictionRequested/);
+    expectCommand(ctx.ssh, /allowScheduling/);
+  });
+
+  it("fails when eviction cannot even be requested", async () => {
+    const ctx = routed({ replicas: [], patchFails: true });
+    const r = await evictLonghornReplicas(ctx);
+    expect(r.success).toBe(false);
+  });
+
+  it("does not request eviction at all when it could never complete", async () => {
+    // 3-replica volume, and only 2 other schedulable nodes: the survivors
+    // already hold a copy each and strict anti-affinity forbids a third.
+    const ctx = routed({
+      nodes: ["node1.lab", "node2.lab", "node3.lab"],
+      replicas: [{ volume: "pvc-a" }],
+      volumes: [{ name: "pvc-a", replicas: 3, size: 1 * GIB }],
+    });
+    const r = await evictLonghornReplicas(ctx);
+    expect(r.success).toBe(false);
+    expect(r.error).toMatch(/cannot be placed|wants 3 replicas/);
+    // The important half: nothing was mutated.
+    expectNoCommand(ctx.ssh, /evictionRequested/);
+  });
+
+  it("refuses a single-replica volume unless explicitly allowed", async () => {
+    const opts = {
+      nodes: ["node1.lab", "node2.lab", "node3.lab", "node4.lab"],
+      replicas: [{ volume: "pvc-solo" }],
+      volumes: [{ name: "pvc-solo", replicas: 1, size: 1 * GIB }],
+    };
+    const r = await evictLonghornReplicas(routed(opts));
+    expect(r.success).toBe(false);
+    expect(r.error).toMatch(/single replica/);
+
+    const allowed = await evictLonghornReplicas(
+      routed(opts, { evictionAllowSingleReplica: true }),
+    );
+    expect(allowed.success).toBe(true);
+  });
+});
+
+describe("checkEvictionCompletable — refuse in seconds, not in an hour", () => {
+  it("passes when a spare schedulable node exists", async () => {
+    const ctx = routed({
+      nodes: ["node1.lab", "node2.lab", "node3.lab", "node4.lab"],
+      replicas: [{ volume: "pvc-a" }],
+      volumes: [{ name: "pvc-a", replicas: 3, size: 1 * GIB }],
+    });
+    const f = await checkEvictionCompletable(ctx, "node1.lab");
+    expect(f.completable).toBe(true);
+    expect(f.replicasMoving).toBe(1);
+  });
+
+  it("blocks when replica count equals the surviving node count", async () => {
+    const ctx = routed({
+      nodes: ["node1.lab", "node2.lab", "node3.lab"],
+      replicas: [{ volume: "pvc-a" }],
+      volumes: [{ name: "pvc-a", replicas: 3, size: 1 * GIB }],
+    });
+    const f = await checkEvictionCompletable(ctx, "node1.lab");
+    expect(f.completable).toBe(false);
+    expect(f.blockers.join(" ")).toContain("pvc-a");
+  });
+
+  it("allows it when soft anti-affinity is on — placement stops being the limit", async () => {
+    const ctx = routed({
+      nodes: ["node1.lab", "node2.lab", "node3.lab"],
+      replicas: [{ volume: "pvc-a" }],
+      volumes: [{ name: "pvc-a", replicas: 3, size: 1 * GIB }],
+      softAntiAffinity: true,
+    });
+    const f = await checkEvictionCompletable(ctx, "node1.lab");
+    expect(f.completable).toBe(true);
+  });
+
+  it("catches a small disk hiding behind over-provisioning", async () => {
+    // The scheduler is happy at 500%; the disk has 49 GiB and the volume is 92.
+    const ctx = routed({
+      nodes: ["node1.lab", "node2.lab", "node3.lab", "node4.lab"],
+      replicas: [{ volume: "pvc-big" }],
+      volumes: [{ name: "pvc-big", replicas: 3, size: 92 * GIB }],
+      overProvisionPct: 500,
+      diskFreeBytes: 49 * GIB,
+      diskMaxBytes: 100 * GIB,
+    });
+    const f = await checkEvictionCompletable(ctx, "node1.lab");
+    expect(f.completable).toBe(false);
+    expect(f.blockers.join(" ")).toMatch(/actually free/);
+    expect(f.blockers.join(" ")).toMatch(/over-provisioning-percentage=500/);
+  });
+
+  it("warns, but does not block, on a single-replica volume", async () => {
+    const ctx = routed({
+      nodes: ["node1.lab", "node2.lab", "node3.lab"],
+      replicas: [{ volume: "pvc-solo" }],
+      volumes: [{ name: "pvc-solo", replicas: 1, size: 1 * GIB }],
+    });
+    const f = await checkEvictionCompletable(ctx, "node1.lab");
+    expect(f.completable).toBe(true);
+    expect(f.warnings.join(" ")).toMatch(/single replica/);
+  });
+
+  it("is trivially completable when the node holds nothing", async () => {
+    const f = await checkEvictionCompletable(routed({ replicas: [] }), "node1.lab");
+    expect(f.completable).toBe(true);
+    expect(f.replicasMoving).toBe(0);
+  });
+
+  it("refuses to guess when Longhorn cannot be read", async () => {
+    const ctx = mockCtx();
+    ctx.ssh.exec.mockResolvedValue(FAIL);
+    const f = await checkEvictionCompletable(ctx, "node1.lab");
+    expect(f.completable).toBe(false);
+    expect(f.blockers.join(" ")).toMatch(/refusing to guess/);
+  });
+});
+
+describe("assertNodeEvacuated — the gate before a wipe", () => {
+  it("passes only when cordoned with no replicas left", async () => {
+    const ctx = mockCtx();
+    ctx.ssh.exec
+      .mockResolvedValueOnce(HOST)
+      .mockResolvedValueOnce(stdout("true"))   // unschedulable
+      .mockResolvedValueOnce(stdout(""))       // no running pods
+      .mockResolvedValueOnce(stdout("namespace/longhorn-system"))
+      .mockResolvedValueOnce(stdout(""));      // no replicas
+    const r = await assertNodeEvacuated(ctx);
+    expect(r.success).toBe(true);
+    expect(r.message).toMatch(/safe to wipe/);
+  });
+
+  it("REFUSES when replicas are still on the node", async () => {
+    const ctx = mockCtx();
+    ctx.ssh.exec
+      .mockResolvedValueOnce(HOST)
+      .mockResolvedValueOnce(stdout("true"))
+      .mockResolvedValueOnce(stdout(""))
+      .mockResolvedValueOnce(stdout("namespace/longhorn-system"))
+      .mockResolvedValueOnce(stdout("replica-abc\nreplica-def"));
+    const r = await assertNodeEvacuated(ctx);
+    expect(r.success).toBe(false);
+    expect(r.error).toMatch(/2 Longhorn replica/);
+  });
+
+  it("REFUSES when the node is still schedulable", async () => {
+    const ctx = mockCtx();
+    ctx.ssh.exec
+      .mockResolvedValueOnce(HOST)
+      .mockResolvedValueOnce(stdout(""))       // not unschedulable
+      .mockResolvedValueOnce(stdout(""))
+      .mockResolvedValueOnce(FAIL);            // no longhorn
+    const r = await assertNodeEvacuated(ctx);
+    expect(r.success).toBe(false);
+    expect(r.error).toMatch(/still schedulable/);
+  });
+});
+
+describe("uncordonNode", () => {
+  it("returns the node to service", async () => {
+    const ctx = mockCtx();
+    ctx.ssh.exec.mockResolvedValueOnce(HOST).mockResolvedValueOnce(OK);
+    const r = await uncordonNode(ctx);
+    expect(r.success).toBe(true);
+    expectCommand(ctx.ssh, /uncordon "node1\.lab"/);
+  });
+});
+
+describe("eviction timeout diagnostics", () => {
+  // An eviction can legitimately run for an hour. When it times out, a bare
+  // count ("2 replica(s) still present") is not actionable -- the question is
+  // always WHICH volume is stuck and in what state. A replica with
+  // desireState=running, currentState=stopped and no failedAt is Longhorn
+  // wanting to start it and not doing so, which is a completely different
+  // problem from a volume that cannot be placed. Observed for real on a
+  // freshly wiped-and-rejoined node.
+  it("names the stuck volumes and their replica states, then restores the node", async () => {
+    const ctx = mockCtx();
+    const patches: string[] = [];
+    ctx.ssh.exec.mockImplementation(async (cmd: string) => {
+      if (cmd.startsWith("hostname")) return HOST;
+      if (cmd.includes("get ns longhorn-system")) return stdout("namespace/longhorn-system");
+      if (cmd.includes("patch nodes.longhorn.io")) { patches.push(cmd); return OK; }
+      if (cmd.includes("jsonpath")) {
+        // The poll never reaches zero: one replica remains for the whole run.
+        if (cmd.includes("replicas.longhorn.io")) return stdout("replica-stuck");
+        if (cmd.includes("replica-soft-anti-affinity")) return stdout("false");
+        if (cmd.includes("storage-over-provisioning-percentage")) return stdout("100");
+        return stdout("");
+      }
+      if (cmd.includes("get replicas.longhorn.io") && cmd.includes("-o json")) {
+        return stdout(JSON.stringify({ items: [{
+          metadata: { name: "pvc-abc-r-deadbeef99" },
+          spec: { nodeID: "node1.lab", volumeName: "pvc-abc", failedAt: "", desireState: "running" },
+          status: { currentState: "stopped" },
+        }] }));
+      }
+      if (cmd.includes("get volumes.longhorn.io") && cmd.includes("-o json")) {
+        return stdout(JSON.stringify({ items: [{
+          metadata: { name: "pvc-abc" },
+          spec: { numberOfReplicas: 3, size: 2 * 2 ** 30 },
+          status: { state: "attached", robustness: "degraded" },
+        }] }));
+      }
+      if (cmd.includes("get nodes.longhorn.io") && cmd.includes("-o json")) {
+        return stdout(JSON.stringify({ items: ["node1.lab", "node2.lab", "node3.lab", "node4.lab"].map((n) => ({
+          metadata: { name: n },
+          spec: { allowScheduling: true },
+          status: { diskStatus: { d: {
+            storageAvailable: 500 * 2 ** 30, storageMaximum: 700 * 2 ** 30,
+            storageScheduled: 0, storageReserved: 0,
+          } } },
+        })) }));
+      }
+      return OK;
+    });
+
+    vi.useFakeTimers();
+    let result;
+    try {
+      const p = evictLonghornReplicas(ctx);
+      await vi.advanceTimersByTimeAsync(61 * 60_000);
+      result = await p;
+    } finally {
+      vi.useRealTimers();
+    }
+
+    expect(result.success).toBe(false);
+    // The actionable part: which volume, and what state its replica is in.
+    expect(result.error).toContain("pvc-abc");
+    expect(result.error).toContain("currentState=stopped");
+    expect(result.error).toContain("desireState=running");
+    expect(result.error).toContain("failedAt=(none)");
+    expect(result.error).toContain("attached/degraded");
+
+    // And the node must be handed back usable, not left eviction-pending.
+    expect(result.error).toMatch(/cancelled and scheduling restored/);
+    expect(patches).toHaveLength(2);
+    expect(patches[1]).toContain('"evictionRequested":false');
+    expect(patches[1]).toContain('"allowScheduling":true');
+  });
+});

bastion/src/modules/modules/k3s/tests/operations.test.ts

@@ -72,31 +72,97 @@ describe("applyCisHardening", () => {
 
 // --- Swap ---
 
-import { disableSwap } from "../src/operations/swap.js";
+import { enableSwap } from "../src/operations/swap.js";
 
-describe("disableSwap", () => {
-  it("disables active swap", async () => {
+describe("enableSwap", () => {
+  it("activates LV swap when present but off", async () => {
     const ctx = mockCtx();
     ctx.ssh.exec
-      .mockResolvedValueOnce(stdout("/dev/sda2 partition 2G"))  // swap active
-      .mockResolvedValueOnce(OK)   // swapoff
-      .mockResolvedValueOnce(OK);  // sed fstab
+      .mockResolvedValueOnce(stdout("yes"))  // LV exists
+      .mockResolvedValueOnce(stdout("off"))  // not in /proc/swaps
+      .mockResolvedValueOnce(OK)             // blkid || mkswap
+      .mockResolvedValueOnce(OK)             // swapon
+      .mockResolvedValueOnce(OK);            // fstab entry
 
-    const result = await disableSwap(ctx);
+    const result = await enableSwap(ctx);
     expect(result.success).toBe(true);
     expect(result.changed).toBe(true);
-    expectCommand(ctx.ssh, "swapoff -a");
+    expectCommand(ctx.ssh, "swapon /dev/mapper/labvg-swap");
   });
 
-  it("is idempotent when swap already off", async () => {
+  it("is idempotent when LV swap already active", async () => {
     const ctx = mockCtx();
     ctx.ssh.exec
-      .mockResolvedValueOnce(stdout(""))  // no swap
-      .mockResolvedValueOnce(OK);          // sed fstab (always runs)
+      .mockResolvedValueOnce(stdout("yes"))  // LV exists
+      .mockResolvedValueOnce(stdout("on"))   // already in /proc/swaps
+      .mockResolvedValueOnce(OK);            // fstab entry (always ensured)
 
-    const result = await disableSwap(ctx);
+    const result = await enableSwap(ctx);
     expect(result.changed).toBe(false);
-    expectNoCommand(ctx.ssh, "swapoff");
+    expectNoCommand(ctx.ssh, "swapon /dev/mapper/labvg-swap");
+  });
+
+  it("skips when no labvg-swap LV exists", async () => {
+    const ctx = mockCtx();
+    ctx.ssh.exec.mockResolvedValueOnce(stdout("no")); // LV missing
+
+    const result = await enableSwap(ctx);
+    expect(result.success).toBe(true);
+    expect(result.changed).toBe(false);
+    expectNoCommand(ctx.ssh, "swapon");
+  });
+});
+
+// --- Rancher LV (imageFs sizing) ---
+
+import { growRancherLv } from "../src/operations/rancher-storage.js";
+
+describe("growRancherLv", () => {
+  it("grows a 20G LV to 120G when the VG has space", async () => {
+    const ctx = mockCtx();
+    ctx.ssh.exec
+      .mockResolvedValueOnce(stdout("  20480.00"))   // lv_size
+      .mockResolvedValueOnce(stdout("  747807.00"))  // vg_free
+      .mockResolvedValueOnce(OK)                     // lvextend
+      .mockResolvedValueOnce(OK);                    // xfs_growfs
+
+    const result = await growRancherLv(ctx);
+    expect(result.success).toBe(true);
+    expect(result.changed).toBe(true);
+    expectCommand(ctx.ssh, "lvextend -L 122880m /dev/labvg/rancher");
+    expectCommand(ctx.ssh, "xfs_growfs /var/lib/rancher");
+  });
+
+  it("is idempotent when the LV is already 120G", async () => {
+    const ctx = mockCtx();
+    ctx.ssh.exec.mockResolvedValueOnce(stdout("  122880.00")); // lv_size
+
+    const result = await growRancherLv(ctx);
+    expect(result.changed).toBe(false);
+    expectNoCommand(ctx.ssh, "lvextend");
+  });
+
+  it("reports without failing when the VG has no free space", async () => {
+    const ctx = mockCtx();
+    ctx.ssh.exec
+      .mockResolvedValueOnce(stdout("  20480.00"))  // lv_size
+      .mockResolvedValueOnce(stdout("  0.00"));     // vg_free
+
+    const result = await growRancherLv(ctx);
+    expect(result.success).toBe(true);
+    expect(result.changed).toBe(false);
+    expect(result.message).toContain("free");
+    expectNoCommand(ctx.ssh, "lvextend");
+  });
+
+  it("skips when there is no rancher LV", async () => {
+    const ctx = mockCtx();
+    ctx.ssh.exec.mockResolvedValueOnce(stdout("")); // lvs empty
+
+    const result = await growRancherLv(ctx);
+    expect(result.success).toBe(true);
+    expect(result.changed).toBe(false);
+    expectNoCommand(ctx.ssh, "lvextend");
   });
 });
 
@@ -202,6 +268,108 @@ describe("writeK3sConfig", () => {
     expect(writeCall).toContain("protect-kernel-defaults: true");
     expect(writeCall).not.toContain("secrets-encryption");
   });
+
+  // --- dual-stack ---
+  //
+  // The property that matters most is the NEGATIVE one: with no ipv6 and no
+  // CIDRs the output must be byte-identical to what the five existing nodes
+  // already have on disk. If it is not, rolling this out rewrites every node's
+  // config.yaml and restarts a healthy cluster to tell it what it already knew.
+
+  // With an ipv6 configured there is an extra ssh round trip up front -- the
+  // probe that checks the address is really on the node -- so the write lands
+  // one call later.
+  const writtenBy = async (config: Parameters<typeof mockCtx>[0]) => {
+    const ctx = mockCtx(config);
+    const hasV6 = !!(config as { ipv6?: string }).ipv6;
+    if (hasV6) ctx.ssh.exec.mockResolvedValueOnce(stdout("present"));
+    ctx.ssh.exec
+      .mockResolvedValueOnce(OK)
+      .mockResolvedValueOnce(stdout("__LABCTL_NOT_FOUND__"))
+      .mockResolvedValueOnce(OK);
+    await writeK3sConfig(ctx);
+    return ctx.ssh.exec.mock.calls[hasV6 ? 3 : 2]![0] as string;
+  };
+
+  it("emits no address-family lines at all when single-stack", async () => {
+    const server = await writtenBy({ hostname: "n1.lab", ip: "10.0.1.1", role: "infra" });
+    expect(server).not.toContain("node-ip");
+    expect(server).not.toContain("cluster-cidr");
+    expect(server).not.toContain("service-cidr");
+
+    const agent = await writtenBy({ role: "worker" });
+    expect(agent).not.toContain("node-ip");
+    // Nothing inserted ahead of, or between, the lines the agent config has
+    // always opened with.
+    expect(agent).toContain("protect-kernel-defaults: true\nnode-label:");
+  });
+
+  it("emits dual node-ip and CIDRs on a server, IPv4 first", async () => {
+    const out = await writtenBy({
+      hostname: "n1.lab",
+      ip: "192.168.8.23",
+      role: "infra",
+      ipv6: "2001:470:187e:2::23",
+      clusterCidr: ["10.42.0.0/16", "2001:470:187e:1000::/56"],
+      serviceCidr: ["10.43.0.0/16", "2001:470:187e:1fff::/112"],
+    });
+    expect(out).toContain('node-ip: "192.168.8.23,2001:470:187e:2::23"');
+    expect(out).toContain('cluster-cidr: "10.42.0.0/16,2001:470:187e:1000::/56"');
+    expect(out).toContain('service-cidr: "10.43.0.0/16,2001:470:187e:1fff::/112"');
+    // IPv4 must stay the primary family -- that is the supported conversion
+    // path and what lets existing Services keep their ClusterIP.
+    expect(out.indexOf("10.42.0.0/16")).toBeLessThan(out.indexOf("2001:470:187e:1000::/56"));
+    // still a valid server config
+    expect(out).toContain("cluster-init: true");
+    expect(out).toContain("secrets-encryption: true");
+    // the v6 address must be a TLS SAN, or a peer joining over v6 fails
+    // verification with an error that names the cert, not the missing SAN
+    expect(out).toContain('  - "2001:470:187e:2::23"');
+  });
+
+  it("gives an AGENT its own dual node-ip but no CIDRs", async () => {
+    const out = await writtenBy({
+      ip: "192.168.8.12",
+      role: "worker",
+      ipv6: "2001:470:187e:2::12",
+      // deliberately supplied: an agent must ignore them
+      clusterCidr: ["10.42.0.0/16", "2001:470:187e:1000::/56"],
+      serviceCidr: ["10.43.0.0/16", "2001:470:187e:1fff::/112"],
+    });
+    expect(out).toContain('node-ip: "192.168.8.12,2001:470:187e:2::12"');
+    expect(out).not.toContain("cluster-cidr");
+    expect(out).not.toContain("service-cidr");
+  });
+
+  it("refuses to write a config naming an IPv6 the node does not have", async () => {
+    // The SSH-onboard case: Asahi and the DGX Sparks run an OS labctl never
+    // installed, so nothing upstream guarantees a DHCPv6 lease was taken.
+    // Writing the config anyway defers the failure to k3s start-up, where it
+    // reads as a bind error rather than a missing address.
+    const ctx = mockCtx({ ip: "192.168.8.12", role: "worker", ipv6: "2001:470:187e:2::12" });
+    ctx.ssh.exec.mockResolvedValueOnce(stdout("")); // probe: address absent
+
+    const result = await writeK3sConfig(ctx);
+    expect(result.success).toBe(false);
+    expect(result.changed).toBe(false);
+    expect(result.error).toContain("2001:470:187e:2::12");
+    // and it must not have written anything
+    expect(ctx.ssh.exec).toHaveBeenCalledTimes(1);
+  });
+
+  it("does not write node-ip for a v4-only node even when CIDRs are given", async () => {
+    // Guards the flag-day risk: supplying ranges alone must not start rewriting
+    // node identity on nodes that have no IPv6 yet.
+    const out = await writtenBy({
+      hostname: "n1.lab",
+      ip: "10.0.1.1",
+      role: "infra",
+      clusterCidr: ["10.42.0.0/16"],
+      serviceCidr: ["10.43.0.0/16"],
+    });
+    expect(out).toContain('cluster-cidr: "10.42.0.0/16"');
+    expect(out).not.toContain("node-ip");
+  });
 });
 
 // --- CNI Cleanup ---

bastion/src/modules/modules/k3s/tests/smoke.test.ts

@@ -16,7 +16,7 @@ describe("smoke: full server install pipeline", () => {
     const pipeline: NamedOperation[] = [
       { name: "Kernel modules", fn: ops.loadKernelModules },
       { name: "Sysctl hardening", fn: ops.applyCisHardening },
-      { name: "Disable swap", fn: ops.disableSwap },
+      { name: "Enable swap", fn: ops.enableSwap },
       { name: "Disable firewall", fn: ops.disableFirewall },
       { name: "SELinux permissive", fn: ops.setSelinuxPermissive },
       { name: "Write k3s config", fn: ops.writeK3sConfig },
@@ -73,7 +73,7 @@ describe("smoke: pipeline stops on failure", () => {
     };
 
     const results = await runSequential(ctx, [
-      { name: "OK op", fn: ops.disableSwap },
+      { name: "OK op", fn: ops.enableSwap },
       { name: "Failing op", fn: failingOp },
       { name: "Never called", fn: neverCalled },
     ]);
@@ -98,11 +98,12 @@ describe("smoke: agent install rejects missing config", () => {
 });
 
 describe("smoke: all operations are exported", () => {
-  it("exports all 15 operations", () => {
+  it("exports all 16 operations", () => {
     const exported = [
       ops.loadKernelModules,
       ops.applyCisHardening,
-      ops.disableSwap,
+      ops.enableSwap,
+      ops.growRancherLv,
       ops.disableFirewall,
       ops.setSelinuxPermissive,
       ops.writeK3sConfig,
@@ -117,7 +118,7 @@ describe("smoke: all operations are exported", () => {
       ops.checkCertExpiry,
     ];
 
-    expect(exported).toHaveLength(15);
+    expect(exported).toHaveLength(16);
     for (const op of exported) {
       expect(typeof op).toBe("function");
     }

bastion/src/modules/src/types.ts

@@ -1,4 +1,4 @@
-import type { OsId, Arch } from "@lab/shared";
+import type { OsId, Arch, Role } from "@lab/shared";
 
 export interface ModuleMetadata {
   name: string;
@@ -14,7 +14,10 @@ export interface ModuleMetadata {
 export interface ModuleContext {
   hostname: string;
   ip: string;
-  role: string;
+  // Role, not string. A literal `string` let `role === "infra"` comparisons
+  // compile while silently missing every other server role -- which is how
+  // labcontroller ended up installing as an agent.
+  role: Role;
   os: OsId;
   arch: Arch;
   sshUser: string;
@@ -24,7 +27,7 @@ export interface ModuleContext {
 
 export interface ModuleResult {
   success: boolean;
-  phase: "install" | "configure" | "health";
+  phase: "install" | "configure" | "health" | "evacuate" | "admit";
   duration: number;
   output: string[];
   errors: string[];

bastion/src/shared/src/index.ts

@@ -8,10 +8,15 @@ export type {
   DebugConfig,
   BastionState,
   BastionConfig,
+  VyosVlanSpec,
+  VyosInstallSpec,
+  VyosBundle,
+  VyosBundleSetOp,
 } from "./types/index.js";
 
-export { SUPPORTED_OS, SUPPORTED_ROLES, ROLE_REGISTRY, isValidOsId } from "./types/index.js";
-export type { RoleInfo } from "./types/index.js";
+export { SUPPORTED_OS, SUPPORTED_ROLES, ROLE_REGISTRY, isValidOsId,
+  roleInfo, runsK3s, isK3sServer, isManaged, hostsLonghorn, isRole } from "./types/index.js";
+export type { RoleInfo, Lifecycle, K3sMode } from "./types/index.js";
 
 export { APP_NAME, APP_VERSION } from "./constants/index.js";
 
@@ -44,3 +49,7 @@ export {
   AgentNotConnectedError,
   isLabError,
 } from "./errors/index.js";
+
+export { v2Layout, v2AllocatedMib, V2_BOOT, V2_RANCHER_MIB, V2_LONGHORN_MIB,
+  SCHEMA_VERSIONS, CONTAINERD_BIND_FSTAB, LEGACY_ROLES, requiresSchemaV2, effectiveSchemaVersion } from "./types/index.js";
+export type { LvSpec, SchemaVersion } from "./types/index.js";

bastion/src/shared/src/protocol/index.ts

@@ -1,6 +1,7 @@
 // Protocol types for agent-labd WebSocket communication.
 
 import { randomUUID } from "node:crypto";
+import type { VyosInstallSpec } from "../types/state.js";
 
 // --- Agent -> labd messages ---
 
@@ -108,7 +109,7 @@ export type BastionMessage =
 export type LabdBastionMessage =
   | { type: "bastion-enrolled"; bastionId: string }
   | { type: "bastion-heartbeat-ack"; serverTime: string }
-  | { type: "command-install"; requestId: string; mac: string; hostname: string; disk?: string; role: string; os: string }
+  | { type: "command-install"; requestId: string; mac: string; hostname: string; disk?: string; role: string; os: string; vyos?: VyosInstallSpec }
   | { type: "command-forget"; requestId: string; mac: string }
   | { type: "command-role-update"; requestId: string; mac: string; role: string }
   | { type: "command-debug"; requestId: string; mac: string; pxeBoot?: boolean }

bastion/src/shared/src/types/config.ts

@@ -14,11 +14,19 @@ export interface BastionConfig {
   // Ubuntu support
   ubuntuVersion: string;
   ubuntuMirror: string;
+  // VyOS support — netboot artifacts are extracted from the ISO at startup.
+  // LTS ISOs are subscription-only, so this defaults to a rolling release.
+  vyosIsoUrl: string;
+  vyosDefaultPassword: string;
   // Syslog listener for install logs (Anaconda logging --host)
   syslogPort: number;
   // Flags
   skipDnsmasq?: boolean | undefined;
   skipArtifacts?: boolean | undefined;
+  // Post-provision k3s auto-install. Defaults on. Set false to suppress it --
+  // required by multi-node tests, which must sequence cluster formation
+  // themselves rather than have each node auto-install into its own cluster.
+  autoK3s?: boolean | undefined;
   // Labd registration (optional — if unset, bastion runs standalone)
   labdUrl?: string | undefined;
   bastionJoinToken?: string | undefined;

bastion/src/shared/src/types/index.ts

@@ -7,9 +7,18 @@ export type {
   InstalledInfo,
   DebugConfig,
   BastionState,
+  VyosVlanSpec,
+  VyosInstallSpec,
+  VyosBundle,
+  VyosBundleSetOp,
 } from "./state.js";
 
-export { SUPPORTED_OS, SUPPORTED_ROLES, ROLE_REGISTRY, isValidOsId } from "./state.js";
-export type { RoleInfo } from "./state.js";
+export { SUPPORTED_OS, SUPPORTED_ROLES, ROLE_REGISTRY, isValidOsId,
+  roleInfo, runsK3s, isK3sServer, isManaged, hostsLonghorn, isRole } from "./state.js";
+export type { RoleInfo, Lifecycle, K3sMode } from "./state.js";
 
 export type { BastionConfig } from "./config.js";
+
+export { v2Layout, v2AllocatedMib, V2_BOOT, V2_RANCHER_MIB, V2_LONGHORN_MIB,
+  SCHEMA_VERSIONS, CONTAINERD_BIND_FSTAB, LEGACY_ROLES, requiresSchemaV2, effectiveSchemaVersion } from "./layout.js";
+export type { LvSpec, SchemaVersion } from "./layout.js";

bastion/src/shared/src/types/layout.ts

@@ -0,0 +1,143 @@
+import { roleInfo, type Role } from "./state.js";
+
+/**
+ * Disk layout, schema v2.
+ *
+ * WHY THIS FILE EXISTS
+ * --------------------
+ * Three generators partition machines, and before this file they disagreed:
+ *
+ *   install.ks.ts        rancher 120G (infra|worker), longhorn --grow (worker)
+ *   ubuntu-autoinstall   rancher  20G (infra only),   longhorn grow   (worker)
+ *   asahi-firstboot.sh   rancher  20G (server only),  longhorn 100%FREE (worker|server)
+ *
+ * asahi-firstboot even carries the comment "sizes match install.ks.ts" while
+ * being 100 GB smaller — which is where worker1's undersized image store came
+ * from, and why a 2026-08 fix to one file never reached the fleet.
+ *
+ * v2 is defined ONCE, here, and all three generators render from it. v1 stays
+ * frozen inline in each generator, because every existing machine was built by
+ * it and reprovision must reproduce it byte-for-byte.
+ *
+ * SIZING
+ * ------
+ * Must fit a 1 TB disk (~926 GiB VG after EFI+/boot) with headroom left
+ * UNALLOCATED on purpose: LVM can grow into free extents online, and the
+ * fleet's recurring failure has been over-allocation, not under-allocation.
+ * `--grow` is deliberately never used — it is what left every node at VFree=0
+ * and made later fixes impossible without a rebuild.
+ */
+
+/**
+ * Roles that predate the v2 layout. Only these have a v1 layout to reproduce;
+ * everything else is v2-only, because no machine was ever installed as one.
+ */
+export const LEGACY_ROLES = ["vanilla", "worker", "infra", "labcontroller"] as const;
+
+/**
+ * True when this role has no v1 layout at all.
+ *
+ * Rendering one at v1 would silently produce base volumes with no rancher and
+ * no longhorn -- a machine that looks installed and cannot run k3s properly.
+ * Callers must reject this rather than guess.
+ */
+export function requiresSchemaV2(role: Role): boolean {
+  return !(LEGACY_ROLES as readonly string[]).includes(role);
+}
+
+/**
+ * The schema a FRESH install of this role should use.
+ *
+ * A role with no v1 layout has exactly one possible answer, so defaulting to it
+ * is unambiguous -- there is no other layout it could mean. Rejecting the
+ * request instead was a mistake: a reinstall legitimately passes no schema
+ * (the disk decides via the lab-schema-v2 tag), and refusing that made it
+ * impossible to reinstall a v2-only machine at all.
+ *
+ * An EXPLICIT v1 for such a role is still a contradiction and must be refused
+ * by the caller -- see requiresSchemaV2.
+ */
+export function effectiveSchemaVersion(role: Role, requested?: 1 | 2): SchemaVersion {
+  if (requested) return requested;
+  return requiresSchemaV2(role) ? 2 : 1;
+}
+
+export const SCHEMA_VERSIONS = [1, 2] as const;
+export type SchemaVersion = (typeof SCHEMA_VERSIONS)[number];
+
+export interface LvSpec {
+  /** LV name inside the volume group. */
+  name: string;
+  /** Mount point, or null for swap. */
+  mount: string | null;
+  /** Size in MiB. Never "grow". */
+  sizeMib: number;
+  fs: "xfs" | "swap";
+}
+
+/** Outside the VG: the EFI system partition and /boot. */
+export const V2_BOOT = {
+  efiMib: 1024,   // was 600 - firmware and shim have grown
+  bootMib: 4096,  // was 3072 - several kernels + initramfs
+} as const;
+
+const V2_BASE: readonly LvSpec[] = [
+  { name: "swap",   mount: null,       sizeMib: 16384, fs: "swap" },
+  { name: "root",   mount: "/",        sizeMib: 32768, fs: "xfs" },
+  { name: "var",    mount: "/var",     sizeMib: 49152, fs: "xfs" },
+  { name: "varlog", mount: "/var/log", sizeMib: 16384, fs: "xfs" },
+  { name: "home",   mount: "/home",    sizeMib:  8192, fs: "xfs" },
+  { name: "srv",    mount: "/srv",     sizeMib: 16384, fs: "xfs" },
+] as const;
+
+/** Image store AND local-path PVCs both live here - size for both. */
+export const V2_RANCHER_MIB = 256000;  // 250 GiB
+export const V2_LONGHORN_MIB = 307200; // 300 GiB
+
+/**
+ * The v2 volumes for a role, gated by CAPABILITY rather than role name.
+ *
+ * `omitRoot` is for Asahi, where the vendor installer already created a btrfs
+ * root partition outside our VG and we must not make a second one.
+ */
+export function v2Layout(role: Role, opts: { omitRoot?: boolean } = {}): LvSpec[] {
+  const info = roleInfo(role);
+  const lvs: LvSpec[] = V2_BASE.filter(
+    (lv) => !(opts.omitRoot && lv.name === "root"),
+  ).map((lv) => ({ ...lv }));
+
+  // Anything running k3s needs a dedicated image store. This is the check that
+  // `role === "infra" || role === "worker"` kept getting wrong: labcontroller
+  // runs k3s and was silently excluded.
+  if (info.k3sMode !== null) {
+    lvs.push({ name: "rancher", mount: "/var/lib/rancher", sizeMib: V2_RANCHER_MIB, fs: "xfs" });
+  }
+
+  // Fixed, never --grow. A control-plane-only node holds no replicas.
+  if (info.k3sMode !== null && !info.controlPlaneOnly) {
+    lvs.push({ name: "longhorn", mount: "/var/lib/longhorn", sizeMib: V2_LONGHORN_MIB, fs: "xfs" });
+  }
+
+  return lvs;
+}
+
+/** Total MiB a role's v2 layout consumes inside the VG. */
+export function v2AllocatedMib(role: Role, opts: { omitRoot?: boolean } = {}): number {
+  return v2Layout(role, opts).reduce((n, lv) => n + lv.sizeMib, 0);
+}
+
+/**
+ * Bind-mounting containerd onto the Longhorn LV.
+ *
+ * Codifies what was already done BY HAND on worker1 and exists in no template:
+ * its 20 GiB rancher LV could not hold the image store, so containerd was
+ * pointed at a directory on the big Longhorn LV. It survives reboot only
+ * because someone wrote a correct fstab line; a reinstall would lose it.
+ *
+ * v2 sizes rancher properly so this is not needed on new boxes - it is kept for
+ * rescuing v1 machines that cannot be repartitioned yet.
+ */
+export const CONTAINERD_BIND_FSTAB =
+  "/var/lib/longhorn/k3s-containerd /var/lib/rancher/k3s/agent/containerd none " +
+  "bind,x-systemd.requires-mounts-for=/var/lib/longhorn," +
+  "x-systemd.requires-mounts-for=/var/lib/rancher 0 0";

bastion/src/shared/src/types/state.ts

@@ -2,10 +2,10 @@
 
 export type ProvisionStackType = "dhcpproxy" | "iso" | "cloud-init";
 
-export type OsId = "fedora-43" | "ubuntu-26.04";
+export type OsId = "fedora-43" | "ubuntu-26.04" | "vyos-rolling";
 export type Arch = "x86_64" | "aarch64";
 
-export const SUPPORTED_OS: readonly OsId[] = ["fedora-43", "ubuntu-26.04"] as const;
+export const SUPPORTED_OS: readonly OsId[] = ["fedora-43", "ubuntu-26.04", "vyos-rolling"] as const;
 
 export function isValidOsId(value: string): value is OsId {
   return (SUPPORTED_OS as readonly string[]).includes(value);
@@ -28,8 +28,32 @@ export interface HardwareInfo {
   bastionId?: string;  // set when aggregated through labd
 }
 
-export type Role = "vanilla" | "worker" | "infra" | "labcontroller";
-export const SUPPORTED_ROLES: readonly Role[] = ["vanilla", "worker", "infra", "labcontroller"] as const;
+export type Role =
+  // Legacy roles. Every machine in the fleet today is one of these, and they
+  // default to the v1 disk layout so a reinstall reproduces what is on disk.
+  | "vanilla" | "worker" | "infra" | "labcontroller"
+  // Topology-named profiles. New machines only, so they are v2-only: there is
+  // no v1 layout to be compatible with, and none of them can reach the frozen
+  // v1 branch in the templates.
+  | "standalone-cluster" | "k8s-controller" | "k8s-worker" | "k8s-combo";
+
+export const SUPPORTED_ROLES: readonly Role[] = [
+  "vanilla", "worker", "infra", "labcontroller",
+  "standalone-cluster", "k8s-controller", "k8s-worker", "k8s-combo",
+] as const;
+
+/**
+ * How a machine comes to exist, which decides whether we partition it at all.
+ *
+ *  - "managed"   we PXE-install it with our kickstart; our disk layout applies.
+ *  - "appliance" we install it, but the vendor installer owns the disk (VyOS).
+ *  - "onboard"   we never install it; we adopt a running machine (DGX Spark,
+ *                Apple Silicon under Asahi). No partitioning, ever.
+ */
+export type Lifecycle = "managed" | "appliance" | "onboard";
+
+/** null = no k3s at all. */
+export type K3sMode = "server" | "agent" | null;
 
 export interface RoleInfo {
   name: Role;
@@ -37,6 +61,18 @@ export interface RoleInfo {
   parent?: Role;       // inherits from this role
   k3s: boolean;        // installs k3s
   apps: string[];      // apps auto-deployed after k3s
+
+  // --- capabilities -------------------------------------------------------
+  // These exist so decisions are made by asking what a machine DOES, not by
+  // comparing its role name. Every `role === "infra" || role === "labcontroller"`
+  // in this codebase is a place that forgot a role: `labcontroller` runs k3s and
+  // still got no rancher LV, because install.ks.ts asked `role === "infra"`.
+  lifecycle: Lifecycle;
+  k3sMode: K3sMode;
+  /** Holds Longhorn replicas, i.e. wants a dedicated /var/lib/longhorn LV. */
+  hostsLonghorn: boolean;
+  /** Control plane that deliberately runs no workloads and no storage. */
+  controlPlaneOnly: boolean;
 }
 
 export const ROLE_REGISTRY: readonly RoleInfo[] = [
@@ -45,6 +81,10 @@ export const ROLE_REGISTRY: readonly RoleInfo[] = [
     description: "OS only — no k3s, no cluster services",
     k3s: false,
     apps: [],
+    lifecycle: "managed",
+    k3sMode: null,
+    hostsLonghorn: false,
+    controlPlaneOnly: false,
   },
   {
     name: "worker",
@@ -52,6 +92,10 @@ export const ROLE_REGISTRY: readonly RoleInfo[] = [
     parent: "vanilla",
     k3s: true,
     apps: [],
+    lifecycle: "managed",
+    k3sMode: "agent",
+    hostsLonghorn: true,
+    controlPlaneOnly: false,
   },
   {
     name: "infra",
@@ -59,6 +103,15 @@ export const ROLE_REGISTRY: readonly RoleInfo[] = [
     parent: "vanilla",
     k3s: true,
     apps: [],
+    lifecycle: "managed",
+    k3sMode: "server",
+    // NOTE: worker0/1/2 are `infra` and DO hold every Longhorn replica in the
+    // homelab cluster, but the v1 kickstart only ever gave a longhorn LV to
+    // `worker`. On worker2 that means Longhorn data sits on /var alongside the
+    // image store. Left false so the v1 layout is unchanged; v2 sizes it
+    // properly from this flag.
+    hostsLonghorn: false,
+    controlPlaneOnly: false,
   },
   {
     name: "labcontroller",
@@ -66,22 +119,251 @@ export const ROLE_REGISTRY: readonly RoleInfo[] = [
     parent: "infra",
     k3s: true,
     apps: ["cockroachdb", "labd", "bastion"],
+    lifecycle: "managed",
+    k3sMode: "server",
+    hostsLonghorn: false,
+    controlPlaneOnly: false,
+  },
+  {
+    name: "standalone-cluster",
+    description: "Single node that IS its own cluster — k3s server + workloads + storage",
+    parent: "vanilla",
+    k3s: true,
+    apps: [],
+    lifecycle: "managed",
+    k3sMode: "server",
+    // Longhorn yes, at replica 1. A cluster of one cannot satisfy a 3-replica
+    // policy, but keeping the `longhorn` StorageClass present means manifests
+    // are portable across every cluster instead of needing a local-path
+    // variant. Replica count is a property of the cluster, not the manifest.
+    hostsLonghorn: true,
+    controlPlaneOnly: false,
+  },
+  {
+    name: "k8s-controller",
+    description: "Control plane + etcd only — runs no workloads and holds no storage",
+    parent: "vanilla",
+    k3s: true,
+    apps: [],
+    lifecycle: "managed",
+    k3sMode: "server",
+    hostsLonghorn: false,
+    controlPlaneOnly: true,
+  },
+  {
+    name: "k8s-worker",
+    description: "k3s agent + Longhorn storage — joins an existing cluster",
+    parent: "vanilla",
+    k3s: true,
+    apps: [],
+    lifecycle: "managed",
+    k3sMode: "agent",
+    hostsLonghorn: true,
+    controlPlaneOnly: false,
+  },
+  {
+    name: "k8s-combo",
+    description: "Control plane + etcd AND workloads + storage — hyperconverged cluster member",
+    parent: "vanilla",
+    k3s: true,
+    apps: [],
+    lifecycle: "managed",
+    k3sMode: "server",
+    hostsLonghorn: true,
+    controlPlaneOnly: false,
   },
 ] as const;
 
+/**
+ * Capability lookups. Use these instead of comparing role names.
+ *
+ * Every call site that asks `role === "infra" || role === "labcontroller"` is a
+ * place that has to be edited again when a role is added — and the one that was
+ * missed (install.ks.ts asking `role === "infra"`) is why `labcontroller` boxes
+ * get no dedicated image-store LV.
+ */
+/**
+ * Narrow an untrusted string (state file, HTTP body) to a Role.
+ * State is JSON on disk and can contain anything, including a role that was
+ * removed in a later version -- so this is a guard, not a cast.
+ */
+export function isRole(value: string): value is Role {
+  return (SUPPORTED_ROLES as readonly string[]).includes(value);
+}
+
+export function roleInfo(role: Role): RoleInfo {
+  const info = ROLE_REGISTRY.find((r) => r.name === role);
+  if (!info) throw new Error(`unknown role: ${role}`);
+  return info;
+}
+
+/** Runs k3s in any capacity. */
+export function runsK3s(role: Role): boolean {
+  return roleInfo(role).k3sMode !== null;
+}
+
+/** Runs k3s as a server (control plane + etcd) rather than an agent. */
+export function isK3sServer(role: Role): boolean {
+  return roleInfo(role).k3sMode === "server";
+}
+
+/** We PXE-install this machine, so our disk layout applies to it. */
+export function isManaged(role: Role): boolean {
+  return roleInfo(role).lifecycle === "managed";
+}
+
+/** Wants a dedicated /var/lib/longhorn LV. */
+export function hostsLonghorn(role: Role): boolean {
+  return roleInfo(role).hostsLonghorn;
+}
+
 export interface ProgressLogEntry {
   stage: string;
   detail: string;
   timestamp: string;
 }
 
+/** A tagged VLAN sub-interface on the bond (or on the mgmt NIC when unbonded). */
+export interface VyosVlanSpec {
+  id: number;
+  address: string;         // CIDR, e.g. "10.0.10.1/24"
+  description?: string;
+  /**
+   * VRRP virtual address (CIDR) floated on this VLAN. Emitted as a
+   * high-availability vrrp group with vrid = VLAN id, so the same spec on both
+   * HA peers (with different priorities) produces a matching group pair.
+   */
+  vrrp?: string;
+}
+
+/** One config node in a rendered bundle. Mirrors VyosSetOp on the bastion side. */
+export interface VyosBundleSetOp {
+  path: string[];
+  value?: string;
+  /** false appends to a multi-value node (e.g. bond members) instead of replacing. */
+  replace?: boolean;
+}
+
+/**
+ * A router's complete desired config, rendered from the Pulumi model.
+ *
+ * Produced by `kubernetes-deployment/scripts/vyos-render-bundle.ts` from the
+ * same subtree model `pulumi up` applies. The point is that labctl never
+ * authors VyOS config: bring-up replays what Pulumi already declares, so a
+ * freshly installed router and a `pulumi up` cannot disagree.
+ *
+ * Secret values arrive as `@secret:<key>` sentinels and are DROPPED at install
+ * time -- the bundle is committed to git and must stay safe to read. The router
+ * comes up on the LAN without its PPPoE credential; the first `pulumi up`
+ * supplies it. That handoff is deliberate.
+ */
+export interface VyosBundle {
+  sets: VyosBundleSetOp[];
+  /** Paths that are VyOS tag nodes — the installer's ConfigTree needs them marked. */
+  tags: string[][];
+}
+
+/**
+ * VyOS-specific install parameters. Rendered into the config.boot that the
+ * installer adopts, so the router comes up already configured.
+ *
+ * NOTE: bondMembers must NOT include the interface PXE booted from. Firmware
+ * PXE cannot run over LACP, so the install-time NIC has to stay unbonded.
+ */
+export interface VyosInstallSpec {
+  /**
+   * A complete rendered config for this router. When present it REPLACES the
+   * derived interface/VLAN/VRRP config below -- the bundle already describes
+   * all of it, and deriving a second opinion is exactly the drift this exists
+   * to prevent. The remaining install parameters (password, disk, console) are
+   * still honoured because they are installer inputs, not router config.
+   */
+  bundle?: VyosBundle;
+  /**
+   * Key for the VyOS HTTP API, enabled at install so the router is manageable
+   * from the moment it boots.
+   *
+   * Without this the box comes up reachable only over SSH, and enabling the API
+   * later is a hand-run config change on a live firewall -- which is exactly the
+   * gap that left vyos001/vyos002 unmanageable by Pulumi after their cutover.
+   * The API is deliberately NOT part of the Pulumi model: a provider that
+   * manages its own transport can revoke its own access.
+   */
+  apiKey?: string;
+  /**
+   * Address the API listens on. Defaults to the management address when static.
+   * Never left unbound: an unrestricted listener puts a config-write endpoint on
+   * every segment the router touches, including the WAN.
+   */
+  apiListenAddress?: string;
+  /** Interfaces aggregated into bond0 with LACP (802.3ad). Omit for no bond. */
+  bondMembers?: string[];
+  /** CIDR address on bond0 itself — the switch trunk's native/untagged VLAN. */
+  bondAddress?: string;
+  /** VRRP virtual address (CIDR) floated on the untagged bond (vrid 1). */
+  bondVrrp?: string;
+  /**
+   * VRRP priority for every group on this box. Higher wins mastership.
+   * The HA pair differs ONLY here (e.g. 200 on the primary, 100 on the
+   * standby) — addresses differ per box, VIPs and vrids match.
+   */
+  vrrpPriority?: number;
+  /** Tagged VLAN sub-interfaces, created on bond0 when bonded, else on mgmtInterface. */
+  vlans?: VyosVlanSpec[];
+  /** Untagged interface the machine PXE booted from. Defaults to "eth0". */
+  mgmtInterface?: string;
+  /** CIDR address for mgmtInterface, or "dhcp". Defaults to "dhcp". */
+  mgmtAddress?: string;
+  /**
+   * Tagged management VLAN on mgmtInterface, separate from the routed VLANs
+   * carried by the bond.
+   *
+   * Needed when the PXE port is a trunk: it boots untagged on the VLAN the
+   * bastion's proxy DHCP serves, and carries the management VLAN tagged so the
+   * router stays reachable there without giving up reinstallability.
+   */
+  mgmtVlan?: VyosVlanSpec;
+  /** Password for the "vyos" user. Falls back to the bastion default. */
+  password?: string;
+  /**
+   * On reinstall the VyOS installer carries the previous on-disk config (and
+   * SSH host keys) forward -- the "reinstall without losing data" default.
+   * Set true to make the bastion-generated config win instead: after install
+   * the driver overwrites the installed image's config.boot.
+   */
+  freshConfig?: boolean;
+  /**
+   * VyOS interface name -> MAC, emitted as `hw-id` so names bind deterministically.
+   *
+   * Discovery runs under Fedora and reports predictable names (enp2s0,
+   * enp1s0f0np0), but VyOS enumerates its own eth<N> names, so a name observed
+   * during discovery cannot be used directly. Pinning by MAC removes the guess
+   * about which physical port a given eth<N> is.
+   */
+  hwIds?: Record<string, string>;
+}
+
 export interface InstallConfig {
   hostname: string;
   disk: string;
   role: Role;
   os?: OsId;          // defaults to "fedora-43" for backward compat
+  vyos?: VyosInstallSpec;  // only consulted when os is "vyos-rolling"
   arch?: Arch;         // detected from HardwareInfo or overridden
+  /**
+   * Disk layout for a FRESH install. Defaults to 1.
+   * Ignored on a reprovision: %pre reads the lab-schema-v2 LVM tag off the
+   * disk instead, because reinstalling a machine must never change its layout.
+   */
+  schemaVersion?: 1 | 2;
   queued_at: string;
+  /**
+   * When dispatch last served this machine an install boot script. Progress
+   * callbacks only start once the installer environment is up, so a machine
+   * dispatched long ago with no progress is wedged before that point (bad
+   * kernel/initrd, no network in the initramfs, wrong NIC picked...).
+   */
+  dispatched_at?: string;
   progress?: string;
   progress_at?: string;
   progress_detail?: string;
@@ -92,6 +374,12 @@ export interface InstallConfig {
 export interface InstalledInfo {
   hostname: string;
   role: string;
+  /**
+   * Disk layout this machine was installed with. Advisory only -- the
+   * authoritative answer is the `lab-schema-v2` LVM tag on the machine's own
+   * volume group, because that is what a reinstall actually reads.
+   */
+  schemaVersion?: 1 | 2;
   os?: OsId;
   ip: string;
   installed_at: string;

bastion/src/shared/tests/capabilities.test.ts

@@ -0,0 +1,129 @@
+import { describe, it, expect } from "vitest";
+import {
+  SUPPORTED_ROLES,
+  ROLE_REGISTRY,
+  roleInfo,
+  runsK3s,
+  isK3sServer,
+  isManaged,
+  hostsLonghorn,
+} from "../src/types/state.js";
+import type { Role } from "../src/types/state.js";
+
+/**
+ * The capability lookups exist to replace role-name comparisons scattered across
+ * the codebase. Introducing them must change NO behaviour — so these tests pin
+ * each helper against the literal expression it replaces, copied verbatim from
+ * the call site it came from.
+ *
+ * If one of these fails, the refactor changed behaviour somewhere. That is the
+ * whole point of the file.
+ */
+/**
+ * Only the FOUR legacy roles have behaviour to preserve. The topology profiles
+ * (standalone-cluster, k8s-*) are new, v2-only, and have no historical
+ * expression to be compatible with -- pinning them against `role === "worker"`
+ * would assert nonsense.
+ */
+const LEGACY_ROLES = ["vanilla", "worker", "infra", "labcontroller"] as const;
+
+describe("capability helpers reproduce the expressions they replace", () => {
+  // Verbatim from the call sites, as of the commit that introduced this file.
+  const legacy = {
+    // k3s-install.ts:7, k3s-config.ts:7, render-config.ts:25,
+    // k3s-module.ts:65, health/k3s-service.ts:5, post-provision.ts:153
+    isServer: (r: Role) => r === "infra" || r === "labcontroller",
+    // ROLE_REGISTRY.k3s — the field that already existed and nobody read
+    runsK3s: (r: Role) => ROLE_REGISTRY.find((x) => x.name === r)!.k3s,
+    // install.ks.ts:41, ubuntu-autoinstall.ts:34
+    hasLonghorn: (r: Role) => r === "worker",
+    // install.ks.ts:48
+    isVanilla: (r: Role) => r === "vanilla",
+  };
+
+  it.each(LEGACY_ROLES)("isK3sServer matches the legacy expression for %s", (role) => {
+    expect(isK3sServer(role)).toBe(legacy.isServer(role));
+  });
+
+  it.each(LEGACY_ROLES)("runsK3s matches ROLE_REGISTRY.k3s for %s", (role) => {
+    expect(runsK3s(role)).toBe(legacy.runsK3s(role));
+  });
+
+  it.each(LEGACY_ROLES)("hostsLonghorn matches the legacy expression for %s", (role) => {
+    expect(hostsLonghorn(role)).toBe(legacy.hasLonghorn(role));
+  });
+
+  it.each(LEGACY_ROLES)("k3sMode === null matches isVanilla for %s", (role) => {
+    expect(roleInfo(role).k3sMode === null).toBe(legacy.isVanilla(role));
+  });
+
+  it("every supported role is managed today (nothing is onboarded yet)", () => {
+    for (const role of SUPPORTED_ROLES) expect(isManaged(role)).toBe(true);
+  });
+});
+
+/**
+ * The topology profiles. These are asserted directly rather than against a
+ * legacy expression, because they describe cluster SHAPE -- which is the thing
+ * role names were supposed to convey and did not.
+ */
+describe("topology profiles", () => {
+  it("standalone-cluster is a server that also holds storage", () => {
+    expect(isK3sServer("standalone-cluster")).toBe(true);
+    // Longhorn at replica 1: a cluster of one cannot do 3, but keeping the
+    // StorageClass present keeps manifests portable across clusters.
+    expect(hostsLonghorn("standalone-cluster")).toBe(true);
+    expect(roleInfo("standalone-cluster").controlPlaneOnly).toBe(false);
+  });
+
+  it("k8s-controller is the only role that runs k3s and holds no storage", () => {
+    expect(isK3sServer("k8s-controller")).toBe(true);
+    expect(hostsLonghorn("k8s-controller")).toBe(false);
+    expect(roleInfo("k8s-controller").controlPlaneOnly).toBe(true);
+  });
+
+  it("k8s-worker is an agent with storage", () => {
+    expect(isK3sServer("k8s-worker")).toBe(false);
+    expect(runsK3s("k8s-worker")).toBe(true);
+    expect(hostsLonghorn("k8s-worker")).toBe(true);
+  });
+
+  it("k8s-combo is both control plane and storage — unlike k8s-controller", () => {
+    expect(isK3sServer("k8s-combo")).toBe(true);
+    expect(hostsLonghorn("k8s-combo")).toBe(true);
+    expect(roleInfo("k8s-combo").controlPlaneOnly).toBe(false);
+  });
+
+  it("k8s-combo and k8s-controller differ ONLY in storage duty", () => {
+    const combo = roleInfo("k8s-combo");
+    const ctrl = roleInfo("k8s-controller");
+    expect(combo.k3sMode).toBe(ctrl.k3sMode);
+    expect(combo.hostsLonghorn).not.toBe(ctrl.hostsLonghorn);
+  });
+});
+
+describe("registry integrity", () => {
+  it("every SUPPORTED_ROLE has a registry entry", () => {
+    for (const role of SUPPORTED_ROLES) expect(() => roleInfo(role)).not.toThrow();
+  });
+
+  it("registry has no entries outside SUPPORTED_ROLES", () => {
+    for (const info of ROLE_REGISTRY) expect(SUPPORTED_ROLES).toContain(info.name);
+  });
+
+  it("k3s boolean and k3sMode never disagree", () => {
+    for (const info of ROLE_REGISTRY) {
+      expect(info.k3s).toBe(info.k3sMode !== null);
+    }
+  });
+
+  it("an unknown role throws rather than silently defaulting", () => {
+    expect(() => roleInfo("does-not-exist" as Role)).toThrow(/unknown role/);
+  });
+
+  it("a control-plane-only role never hosts Longhorn", () => {
+    for (const info of ROLE_REGISTRY) {
+      if (info.controlPlaneOnly) expect(info.hostsLonghorn).toBe(false);
+    }
+  });
+});

bastion/src/shared/tests/layout.test.ts

@@ -0,0 +1,92 @@
+import { describe, it, expect } from "vitest";
+import { v2Layout, v2AllocatedMib, V2_BOOT, V2_RANCHER_MIB, V2_LONGHORN_MIB } from "../src/types/layout.js";
+import { SUPPORTED_ROLES, roleInfo } from "../src/types/state.js";
+
+/** A 1 TB disk is the smallest we will ever buy. 1e12 bytes in MiB. */
+const ONE_TB_MIB = Math.floor(1_000_000_000_000 / (1024 * 1024)); // 953674
+const VG_MIB_ON_1TB = ONE_TB_MIB - V2_BOOT.efiMib - V2_BOOT.bootMib;
+
+describe("v2 layout fits the smallest disk we will buy", () => {
+  it.each(SUPPORTED_ROLES)("%s fits in a 1 TB disk with headroom left over", (role) => {
+    const used = v2AllocatedMib(role);
+    expect(used).toBeLessThan(VG_MIB_ON_1TB);
+    // The entire point of v2 is to STOP consuming the disk. Demand real slack.
+    const freeMib = VG_MIB_ON_1TB - used;
+    expect(freeMib).toBeGreaterThan(100 * 1024); // >100 GiB unallocated
+  });
+
+  it("the heaviest role still leaves >200 GiB free on 1 TB", () => {
+    const worst = Math.max(...SUPPORTED_ROLES.map((r) => v2AllocatedMib(r)));
+    expect(VG_MIB_ON_1TB - worst).toBeGreaterThan(200 * 1024);
+  });
+});
+
+describe("v2 gates volumes by capability, not role name", () => {
+  it.each(SUPPORTED_ROLES)("%s gets a rancher LV iff it runs k3s", (role) => {
+    const hasRancher = v2Layout(role).some((lv) => lv.name === "rancher");
+    expect(hasRancher).toBe(roleInfo(role).k3sMode !== null);
+  });
+
+  it("labcontroller gets a rancher LV — the v1 bug this schema fixes", () => {
+    const lvs = v2Layout("labcontroller");
+    const rancher = lvs.find((lv) => lv.name === "rancher");
+    expect(rancher).toBeDefined();
+    expect(rancher!.sizeMib).toBe(V2_RANCHER_MIB);
+  });
+
+  it("vanilla gets neither rancher nor longhorn", () => {
+    const names = v2Layout("vanilla").map((lv) => lv.name);
+    expect(names).not.toContain("rancher");
+    expect(names).not.toContain("longhorn");
+  });
+
+  it("a control-plane-only role would get no longhorn LV", () => {
+    for (const role of SUPPORTED_ROLES) {
+      if (roleInfo(role).controlPlaneOnly) {
+        expect(v2Layout(role).map((l) => l.name)).not.toContain("longhorn");
+      }
+    }
+  });
+});
+
+describe("v2 never over-allocates", () => {
+  it("no volume is grow/unbounded — every size is a concrete number", () => {
+    for (const role of SUPPORTED_ROLES) {
+      for (const lv of v2Layout(role)) {
+        expect(Number.isFinite(lv.sizeMib)).toBe(true);
+        expect(lv.sizeMib).toBeGreaterThan(0);
+      }
+    }
+  });
+
+  it("longhorn is fixed at 300 GiB, not grown", () => {
+    const lh = v2Layout("worker").find((lv) => lv.name === "longhorn");
+    expect(lh?.sizeMib).toBe(V2_LONGHORN_MIB);
+    expect(V2_LONGHORN_MIB).toBe(300 * 1024);
+  });
+
+  it("rancher is sized for an image store AND local-path PVCs", () => {
+    // A single ROCm vLLM image measured 62 GiB extracted on 2026-09-17.
+    expect(V2_RANCHER_MIB).toBeGreaterThan(200 * 1024);
+  });
+});
+
+describe("Asahi variant", () => {
+  it("omitRoot drops the root LV — the vendor installer already made one", () => {
+    const names = v2Layout("infra", { omitRoot: true }).map((lv) => lv.name);
+    expect(names).not.toContain("root");
+    expect(names).toContain("var");
+  });
+
+  it("omitRoot frees the root allocation and nothing else", () => {
+    const delta = v2AllocatedMib("infra") - v2AllocatedMib("infra", { omitRoot: true });
+    expect(delta).toBe(32768);
+  });
+});
+
+describe("layout is a pure value, safe to render three times", () => {
+  it("mutating a returned spec does not affect the next call", () => {
+    v2Layout("worker")[0]!.sizeMib = 999;
+    expect(v2Layout("worker")[0]!.sizeMib).not.toBe(999);
+  });
+});

bastion/tests/integration/cluster-smoke.test.ts

@@ -0,0 +1,164 @@
+// Integration test: form a real 3-server k3s cluster from PXE-installed VMs.
+//
+// WHY THIS EXISTS SEPARATELY FROM THE DRILL
+// -----------------------------------------
+// The rolling drill is a ~3 hour test. Almost everything that can go wrong in
+// its first thirty minutes goes wrong here instead, in forty-five: the token
+// handoff, the cluster-init/join discriminator, the etcd bootstrap race, root
+// SSH for the module, and whether post-provision was actually suppressed.
+// Debugging those inside a three-hour test is how you spend a day per attempt.
+//
+// It asserts on ETCD MEMBERSHIP, not just `kubectl get nodes`. A node object
+// and an etcd member are different things, and the whole hazard in a rolling
+// reinstall is that they can disagree.
+//
+// Prerequisites: libvirtd, OVMF, ipxe-bootimgs-x86, sudo. Set FEDORA_MIRROR to
+// a local mirror or each install pulls the full package set over the internet.
+//
+//   sudo -E npx vitest run -c tests/integration/vitest.config.ts -t "cluster smoke"
+
+import { describe, it, expect, beforeAll, afterAll } from "vitest";
+import { existsSync, readFileSync } from "node:fs";
+import { homedir } from "node:os";
+import { join } from "node:path";
+import {
+  startTestBastion, provisionVms, destroyVms,
+  type Harness, type ProvisionedVm,
+} from "./helpers/provision-harness.js";
+import {
+  formCluster, clusterSnapshot, printSnapshot, etcdMembers, etcdHealthy,
+  allServersReady, pickJoinTarget, kubectlOn, k8sNodeName,
+} from "./helpers/k3s-cluster.js";
+import { destroyPxeNetwork } from "./helpers/pxe-network.js";
+
+const HTTP_PORT = 8097;
+const SSH_USER = "lab";
+const DISK_GB = 1000;
+const NODES = ["smoke-a", "smoke-b", "smoke-c"];
+
+function findSshKey(): { pubKey: string; keyPath: string } {
+  const homes = [homedir()];
+  const sudoUser = process.env["SUDO_USER"];
+  if (sudoUser) homes.push(join("/home", sudoUser));
+  for (const home of homes) {
+    for (const name of ["id_ed25519", "id_ecdsa", "id_rsa"]) {
+      const keyPath = join(home, ".ssh", name);
+      if (existsSync(keyPath) && existsSync(`${keyPath}.pub`)) {
+        return { pubKey: readFileSync(`${keyPath}.pub`, "utf-8").trim(), keyPath };
+      }
+    }
+  }
+  throw new Error("no SSH keypair found");
+}
+
+describe("cluster smoke: three servers, one cluster", () => {
+  let h: Harness;
+  let vms: ProvisionedVm[] = [];
+
+  /**
+   * Kubernetes node names, NOT libvirt domain names.
+   *
+   * k3s registers a node by FQDN, so the VM `smoke-a` is the node
+   * `smoke-a.pxe-test.local`. Asserting against the VM names would compare two
+   * different namespaces and fail for a reason that has nothing to do with the
+   * cluster.
+   */
+  const nodeNames = (): string[] => vms.map((v) => k8sNodeName(h, v)).sort();
+
+  beforeAll(async () => {
+    const { pubKey, keyPath } = findSshKey();
+    destroyVms(...NODES);
+    h = await startTestBastion({
+      httpPort: HTTP_PORT, sshUser: SSH_USER, sshPubKey: pubKey, sshKeyPath: keyPath,
+      label: "cluster-smoke",
+      // Mandatory. Left on, post-provision installs each node into its OWN
+      // cluster and this test would be measuring three clusters of one.
+      autoK3s: false,
+    });
+  }, 20 * 60_000);
+
+  afterAll(async () => {
+    // Dump state before anything is destroyed — the VMs are the only place
+    // this information exists, and they are about to stop existing.
+    try {
+      if (vms.length > 0) printSnapshot("final", clusterSnapshot(h, vms[0]!));
+    } catch { /* cluster may already be gone */ }
+    destroyVms(...NODES);
+    try { destroyPxeNetwork(); } catch { /* shared */ }
+    await h?.close();
+  }, 10 * 60_000);
+
+  it("provisions three v2 combo nodes", async () => {
+    vms = await provisionVms(h, NODES.map((name) => ({
+      name, role: "k8s-combo" as const, schemaVersion: 2 as const,
+      diskGb: DISK_GB, memory: 6144, vcpus: 4,
+    })));
+    expect(vms).toHaveLength(3);
+    for (const vm of vms) expect(vm.ip).toMatch(/^\d+\.\d+\.\d+\.\d+$/);
+  }, 60 * 60_000);
+
+  it("forms ONE cluster, not three", async () => {
+    await formCluster(h, vms, { role: "k8s-combo" });
+
+    const snap = clusterSnapshot(h, vms[0]!);
+    printSnapshot("after formation", snap);
+
+    // The precise failure this guards: three separate single-node clusters
+    // each report 1/1 Ready and look perfectly healthy in isolation.
+    expect(snap.nodes).toHaveLength(3);
+    expect(snap.readyCount).toBe(3);
+    expect(snap.nodes.map((n) => n.name).sort()).toEqual(nodeNames());
+  }, 90 * 60_000);
+
+  it("has exactly three healthy etcd members", () => {
+    const members = etcdMembers(h, vms[0]!);
+    // .node, not .name: etcd names a member "<node>-<8 hex>", so comparing
+    // raw names against node names never matches.
+    expect(members.map((m) => m.node).sort()).toEqual(nodeNames());
+    expect(members).toHaveLength(3);
+    expect(members.every((m) => m.started)).toBe(true);
+    expect(etcdHealthy(h, vms[0]!)).toBe(true);
+  }, 5 * 60_000);
+
+  it("agrees about membership from EVERY node, not just the first", () => {
+    // Asked of each server independently: a split brain is three nodes that
+    // each believe they are correct.
+    for (const vm of vms) {
+      const members = etcdMembers(h, vm).map((m) => m.node).sort();
+      expect(members, `etcd membership as seen by ${vm.name}`).toEqual(nodeNames());
+      const nodes = clusterSnapshot(h, vm).nodes.map((n) => n.name).sort();
+      expect(nodes, `node list as seen by ${vm.name}`).toEqual(nodeNames());
+    }
+  }, 10 * 60_000);
+
+  it("reports every server ready", () => {
+    const ready = allServersReady(h, vms);
+    expect(ready.ok, ready.detail).toBe(true);
+  }, 5 * 60_000);
+
+  it("makes all three control-plane and etcd", () => {
+    const snap = clusterSnapshot(h, vms[0]!);
+    for (const n of snap.nodes) {
+      expect(n.roles, `${n.name} roles`).toContain("control-plane");
+      expect(n.roles, `${n.name} roles`).toContain("etcd");
+    }
+  }, 5 * 60_000);
+
+  it("can pick a join target other than a given node", () => {
+    // The primitive the rolling drill depends on: after the first server is
+    // wiped it must rejoin against a survivor.
+    const target = pickJoinTarget(h, vms, vms[0]!);
+    expect(target.name).not.toBe(vms[0]!.name);
+  }, 5 * 60_000);
+
+  it("carries the Longhorn disk label on every node", () => {
+    // Set via k3s node-label. Without it Longhorn creates no disk on the node,
+    // and the node then hosts zero replicas while looking entirely healthy.
+    const out = kubectlOn(h, vms[0]!,
+      `get nodes -o jsonpath='{range .items[*]}{.metadata.name}={.metadata.labels.node\\.longhorn\\.io/create-default-disk}{"\\n"}{end}'`);
+    for (const vm of vms) {
+      expect(out, `${vm.name} create-default-disk label`)
+        .toContain(`${k8sNodeName(h, vm)}=config`);
+    }
+  }, 5 * 60_000);
+});

bastion/tests/integration/helpers/k3s-cluster.ts

@@ -0,0 +1,427 @@
+// Form and inspect a real multi-node k3s cluster from an integration test.
+//
+// WHY THIS EXISTS
+// ---------------
+// Nothing in the repo decides which node bootstraps a cluster and which nodes
+// join it. The production generator has the right discriminator --
+// generateServerConfig emits `cluster-init: true` when no server URL is set and
+// `server:`+`token:` when one is -- but the choice is an INPUT, and the only
+// caller that ever supplied it was a hand-run labsim shell script.
+//
+// So a test that wants three servers in one cluster has to sequence it. That
+// sequencing is here rather than in a test file because the rolling drill is
+// not the only thing that will want it: an etcd-recovery test (recoverEtcdMember
+// still has no caller), a dual-stack conversion and any HA upgrade test all
+// need exactly these primitives.
+//
+// Everything drives K3sModule, not shell. That is deliberate and is the same
+// rule labsim follows by shelling out to the compiled render-config binary: a
+// test that reimplements the install proves the test works, not the product.
+
+import { execSync } from "node:child_process";
+import type { Role } from "../../../src/shared/src/types/state.js";
+import { K3sModule } from "../../../src/modules/modules/k3s/src/k3s-module.js";
+import type { ModuleContext } from "../../../src/modules/src/types.js";
+import { log } from "./libvirt.js";
+import { parseEtcdMembers, type EtcdMember } from "../../../src/modules/modules/k3s/src/operations/etcd-members.js";
+import { sshExec } from "./ssh.js";
+import type { Harness, ProvisionedVm } from "./provision-harness.js";
+
+export const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
+
+/**
+ * A pre-shared cluster token, the way labsim does it.
+ *
+ * The alternative is scraping /var/lib/rancher/k3s/server/node-token off the
+ * first server, and nothing in the repo does that. It is also the wrong shape
+ * for a rolling migration: the first server gets wiped and rejoins, so the
+ * token has to outlive it and be knowable before it exists.
+ */
+export const DRILL_TOKEN = "lab-drill-shared-token";
+
+/**
+ * k3s operations run unprefixed `mkdir -p /etc/rancher/k3s` and
+ * `systemctl restart k3s`, so they need root. install.ks.ts provisions
+ * `sshkey --username=root` and sets PermitRootLogin prohibit-password, so root
+ * SSH works on every machine this harness builds. The harness's own `lab` user
+ * stays for the sudo-based reads (readLvs, markers).
+ */
+export const MODULE_SSH_USER = "root";
+
+/**
+ * The k3s version the drill runs, pinned to what the fleet runs.
+ *
+ * get.k3s.io installs LATEST, and latest is currently two minor versions ahead
+ * of production (v1.36.4 vs v1.34.6). Two things follow, and both matter:
+ *
+ *  - a rehearsal on a different minor version is rehearsing a different
+ *    system, and Longhorn in particular is only qualified against a range of
+ *    Kubernetes versions;
+ *  - it is the same trap production would hit. A node reinstalled mid-migration
+ *    comes back on latest while its peers stay put, and Kubernetes supports
+ *    only ONE minor of skew between control-plane members.
+ *
+ * Override with LAB_K3S_VERSION to test an upgrade deliberately.
+ */
+export const PINNED_K3S_VERSION = process.env["LAB_K3S_VERSION"] ?? "v1.34.6+k3s1";
+
+/** Build the ModuleContext K3sModule expects for one VM. */
+export function moduleCtx(
+  h: Harness,
+  vm: ProvisionedVm,
+  role: Role,
+  config: Record<string, unknown> = {},
+): ModuleContext {
+  return {
+    hostname: vm.name,
+    ip: vm.ip,
+    role,
+    os: "fedora-43",
+    arch: "x86_64",
+    sshUser: MODULE_SSH_USER,
+    sshKeyPath: h.sshKeyPath,
+    config,
+  };
+}
+
+/** Run a command on a node as root, throwing with context on failure. */
+export function nodeExec(h: Harness, vm: ProvisionedVm, cmd: string, timeoutMs = 120_000): string {
+  const r = sshExec(vm.ip, MODULE_SSH_USER, cmd, { keyPath: h.sshKeyPath, timeout: timeoutMs });
+  if (r.exitCode !== 0) {
+    throw new Error(`[${vm.name}] ${cmd} failed (${r.exitCode}): ${r.stderr.trim() || r.stdout.trim()}`);
+  }
+  return r.stdout;
+}
+
+/** Same, but returns the result instead of throwing. */
+export function tryNodeExec(h: Harness, vm: ProvisionedVm, cmd: string, timeoutMs = 120_000) {
+  return sshExec(vm.ip, MODULE_SSH_USER, cmd, { keyPath: h.sshKeyPath, timeout: timeoutMs });
+}
+
+/**
+ * The name Kubernetes knows this VM by.
+ *
+ * NOT the libvirt domain name. k3s registers a node under its FQDN, so a VM
+ * called `smoke-a` appears as `smoke-a.pxe-test.local`. Passing the VM name to
+ * `kubectl get node` silently finds nothing, and every wait built on it times
+ * out instead of failing usefully. Read from the node itself, the same way the
+ * production operations do (`hostname -f`), so the two can never disagree.
+ */
+const nodeNameCache = new Map<string, string>();
+export function k8sNodeName(h: Harness, vm: ProvisionedVm): string {
+  const cached = nodeNameCache.get(vm.name);
+  if (cached !== undefined) return cached;
+  const name = nodeExec(h, vm, "hostname -f 2>/dev/null || hostname").trim();
+  if (name === "") throw new Error(`[${vm.name}] could not resolve node name`);
+  nodeNameCache.set(vm.name, name);
+  return name;
+}
+
+/** Forget cached names — a reinstalled machine re-registers. */
+export function forgetNodeName(vm: ProvisionedVm): void {
+  nodeNameCache.delete(vm.name);
+}
+
+/** kubectl against a server node. */
+export function kubectlOn(h: Harness, server: ProvisionedVm, args: string, timeoutMs = 120_000): string {
+  return nodeExec(h, server, `k3s kubectl ${args}`, timeoutMs);
+}
+
+export function kubectlJson<T>(h: Harness, server: ProvisionedVm, args: string): T {
+  return JSON.parse(kubectlOn(h, server, `${args} -o json`)) as T;
+}
+
+/**
+ * Wait until a server's API answers /ping.
+ *
+ * Straight from labsim-k8s-etcd.sh, whose comment is the whole justification:
+ * a joiner that starts before the first server has finished bootstrapping etcd
+ * races it, and the failure looks like etcd instability rather than a startup
+ * order problem. Probed from the HOST, so it does not depend on the joining
+ * node being reachable yet.
+ */
+export async function waitForApiPing(vm: ProvisionedVm, timeoutMs = 450_000): Promise<void> {
+  const start = Date.now();
+  let lastErr = "";
+  while (Date.now() - start < timeoutMs) {
+    try {
+      const out = execSync(
+        `curl -sk --max-time 5 https://${vm.ip}:6443/ping || true`,
+        { encoding: "utf-8", timeout: 15_000 },
+      ).trim();
+      if (out === "pong") {
+        log(`[${vm.name}] API answering after ${Math.round((Date.now() - start) / 1000)}s`);
+        return;
+      }
+      lastErr = out;
+    } catch (e) {
+      lastErr = e instanceof Error ? e.message : String(e);
+    }
+    await sleep(5_000);
+  }
+  throw new Error(`[${vm.name}] API never answered /ping in ${timeoutMs / 1000}s (last: ${lastErr || "no response"})`);
+}
+
+/** Wait for a node to appear Ready in the cluster, as seen from `server`. */
+export async function waitForNodeReady(
+  h: Harness,
+  server: ProvisionedVm,
+  /** The VM to wait for; its Kubernetes name is resolved from the machine. */
+  target: ProvisionedVm,
+  // Generous: a node is NotReady until its CNI is up, and on the cluster-init
+  // server that means waiting for Cilium to install and pull its images. Five
+  // minutes is enough on a warm host and not enough on a cold one, and the
+  // cost of being wrong is a wasted multi-hour run.
+  timeoutMs = 900_000,
+): Promise<void> {
+  const nodeName = k8sNodeName(h, target);
+  const start = Date.now();
+  let last = "";
+  while (Date.now() - start < timeoutMs) {
+    const r = tryNodeExec(h, server,
+      `k3s kubectl get node ${nodeName} -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'`);
+    last = r.stdout.trim();
+    if (last === "True") {
+      log(`[${nodeName}] Ready after ${Math.round((Date.now() - start) / 1000)}s`);
+      return;
+    }
+    await sleep(5_000);
+  }
+  throw new Error(`[${nodeName}] never became Ready in ${timeoutMs / 1000}s (last status: ${last || "absent"})`);
+}
+
+/**
+ * etcdctl invocation.
+ *
+ * NOT `k3s etcdctl` -- that is not a subcommand ("No help topic for 'etcdctl'",
+ * verified on a live v1.34 node). The binary comes from the `etcd` rpm, which
+ * our kickstart does NOT install, so ensureEtcdctl() has to put it there.
+ * Cert paths verified against a real node.
+ */
+const ETCD_BASE = "/var/lib/rancher/k3s/server/tls/etcd";
+const ETCDCTL =
+  `ETCDCTL_API=3 etcdctl ` +
+  `--cacert=${ETCD_BASE}/server-ca.crt ` +
+  `--cert=${ETCD_BASE}/server-client.crt ` +
+  `--key=${ETCD_BASE}/server-client.key ` +
+  `--endpoints=https://127.0.0.1:2379`;
+
+/** Install etcdctl if the node does not have it. Idempotent, ~30s once. */
+export function ensureEtcdctl(h: Harness, server: ProvisionedVm): void {
+  const has = tryNodeExec(h, server, "command -v etcdctl");
+  if (has.exitCode === 0) return;
+  log(`[${server.name}] installing etcdctl (the etcd rpm; kickstart does not include it)`);
+  nodeExec(h, server, "dnf install -y etcd >/dev/null 2>&1 || dnf install -y etcd-ctl >/dev/null 2>&1", 300_000);
+  nodeExec(h, server, "command -v etcdctl");
+}
+
+
+/**
+ * etcd membership, read with etcdctl against k3s's own certs.
+ *
+ * Read rather than inferred from `kubectl get nodes`: a node object can be
+ * deleted while its etcd member survives, and that stale member is exactly the
+ * failure a rolling reinstall creates. Counting nodes would not see it.
+ */
+export function etcdMembers(h: Harness, server: ProvisionedVm): EtcdMember[] {
+  // NOT `-w json`. Two reasons, and the second is the dangerous one:
+  //
+  //  1. `member remove` requires the ID in HEX; the JSON writer emits decimal.
+  //  2. etcd member IDs are uint64. JSON.parse turns them into JS numbers,
+  //     which silently lose precision above 2^53 -- the ID that comes back is
+  //     an APPROXIMATION of the real one. Passing that to `member remove`
+  //     fails with "value out of range" if you are lucky, and in principle
+  //     could match some other member if you are not. This happens at the one
+  //     moment it must not: the node is already stopped and about to be wiped.
+  //
+  // The default writer prints the ID as hex text, so it never becomes a
+  // number at all:
+  //   825852d3683bc406, started, worker2-...-5f7ad0c9, https://...:2380, ...
+  return parseEtcdMembers(nodeExec(h, server, `${ETCDCTL} member list`));
+}
+
+
+/** Are all etcd endpoints healthy? Answers the question quorum actually asks. */
+export function etcdHealthy(h: Harness, server: ProvisionedVm): boolean {
+  const r = tryNodeExec(h, server, `${ETCDCTL} endpoint health`);
+  return r.exitCode === 0 && r.stdout.includes("is healthy");
+}
+
+/**
+ * Remove an etcd member by node name.
+ *
+ * Must happen BEFORE the node is wiped. k3s-install.ts removes the local
+ * datastore when a server joins, so a reinstalled node rejoins as a NEW member
+ * while the old record persists: 3 members becomes 4, quorum rises from 2 to 3,
+ * and the cluster that tolerated one failure now tolerates none.
+ */
+export function removeEtcdMember(h: Harness, peer: ProvisionedVm, nodeName: string): boolean {
+  const members = etcdMembers(h, peer);
+  const target = members.find((m) => m.node === nodeName || m.name === nodeName);
+  if (!target) {
+    log(`[etcd] no member for ${nodeName} (have: ${members.map((m) => m.node).join(",")})`);
+    return false;
+  }
+  nodeExec(h, peer, `${ETCDCTL} member remove ${target.id}`);
+  log(`[etcd] removed member ${target.name} (${target.id})`);
+  return true;
+}
+
+export interface ClusterSnapshot {
+  nodes: { name: string; ready: boolean; roles: string }[];
+  readyCount: number;
+  etcdMembers: number;
+  etcdHealthy: boolean;
+}
+
+/** One read of cluster state, in the shape labsim's cmd_status prints. */
+export function clusterSnapshot(h: Harness, server: ProvisionedVm): ClusterSnapshot {
+  const raw = kubectlJson<{ items: {
+    metadata: { name: string; labels?: Record<string, string> };
+    status: { conditions?: { type: string; status: string }[] };
+  }[] }>(h, server, "get nodes");
+
+  const nodes = raw.items.map((n) => ({
+    name: n.metadata.name,
+    ready: (n.status.conditions ?? []).some((c) => c.type === "Ready" && c.status === "True"),
+    roles: Object.keys(n.metadata.labels ?? {})
+      .filter((k) => k.startsWith("node-role.kubernetes.io/"))
+      .map((k) => k.replace("node-role.kubernetes.io/", ""))
+      .sort().join(",") || "<none>",
+  }));
+
+  let members = 0;
+  let healthy = false;
+  try {
+    members = etcdMembers(h, server).length;
+    healthy = etcdHealthy(h, server);
+  } catch { /* an agent, or the API is down — report zero rather than throw */ }
+
+  return { nodes, readyCount: nodes.filter((n) => n.ready).length, etcdMembers: members, etcdHealthy: healthy };
+}
+
+/**
+ * Print a snapshot. Called liberally during a drill: the VMs are destroyed at
+ * the end, so this transcript is the only surviving record of what happened.
+ */
+export function printSnapshot(label: string, s: ClusterSnapshot): void {
+  log(`[snapshot] ${label}: ${s.readyCount}/${s.nodes.length} Ready, ` +
+      `etcd members=${s.etcdMembers} healthy=${s.etcdHealthy}`);
+  for (const n of s.nodes) log(`             ${n.ready ? "Ready   " : "NotReady"} ${n.name} (${n.roles})`);
+}
+
+/**
+ * Install k3s on one node via the production module.
+ *
+ * Pass no serverUrl to bootstrap the cluster (the token still seeds), or a
+ * serverUrl to join an existing one.
+ */
+export async function installK3sNode(h: Harness, vm: ProvisionedVm, opts: {
+  role: Role;
+  serverUrl?: string;
+  token?: string;
+}): Promise<void> {
+  const mod = new K3sModule();
+  const config: Record<string, unknown> = {
+    k3sToken: opts.token ?? DRILL_TOKEN,
+    k3sVersion: PINNED_K3S_VERSION,
+    ...(opts.serverUrl === undefined ? {} : { k3sServerUrl: opts.serverUrl }),
+  };
+
+  log(`[${vm.name}] installing k3s (${opts.role}${opts.serverUrl ? ` -> ${opts.serverUrl}` : " — cluster-init"})`);
+  const install = await mod.install(moduleCtx(h, vm, opts.role, config));
+  for (const line of install.output) log(`  [${vm.name}] ${line}`);
+  if (!install.success) {
+    throw new Error(`[${vm.name}] k3s install failed: ${install.errors.join("; ")}`);
+  }
+
+  const configure = await mod.configure(moduleCtx(h, vm, opts.role, config));
+  for (const line of configure.output) log(`  [${vm.name}] ${line}`);
+  if (!configure.success) {
+    throw new Error(`[${vm.name}] k3s configure failed: ${configure.errors.join("; ")}`);
+  }
+}
+
+export function serverUrlOf(vm: ProvisionedVm): string {
+  return `https://${vm.ip}:6443`;
+}
+
+/**
+ * Form a cluster from a list of server VMs, strictly in order.
+ *
+ * The first bootstraps etcd and installs the CNI (k3s-module runs networking
+ * only on the cluster-init node -- joiners get Cilium via its DaemonSet). The
+ * rest join one at a time, each gated on the previous node being Ready.
+ *
+ * Serial on purpose. labsim gets away with starting joiners together because
+ * its nodes are cloud-init VMs that poll /ping themselves; here the few extra
+ * minutes buy determinism in the one place where etcd bootstrap genuinely
+ * races.
+ */
+export async function formCluster(h: Harness, servers: ProvisionedVm[], opts: {
+  role?: Role;
+  token?: string;
+} = {}): Promise<void> {
+  if (servers.length === 0) throw new Error("formCluster: no servers");
+  const role = opts.role ?? "k8s-combo";
+  const token = opts.token ?? DRILL_TOKEN;
+  const first = servers[0]!;
+
+  await installK3sNode(h, first, { role, token });
+  await waitForApiPing(first);
+  await waitForNodeReady(h, first, first);
+  ensureEtcdctl(h, first);
+  printSnapshot("after cluster-init", clusterSnapshot(h, first));
+
+  for (const vm of servers.slice(1)) {
+    await waitForApiPing(first);
+    await installK3sNode(h, vm, { role, token, serverUrl: serverUrlOf(first) });
+    await waitForNodeReady(h, first, vm);
+    ensureEtcdctl(h, vm);
+    printSnapshot(`after joining ${vm.name}`, clusterSnapshot(h, first));
+  }
+}
+
+/** Join an agent. Agents cannot start without a server URL and token. */
+export async function joinAgent(h: Harness, vm: ProvisionedVm, opts: {
+  server: ProvisionedVm;
+  role?: Role;
+  token?: string;
+}): Promise<void> {
+  await waitForApiPing(opts.server);
+  await installK3sNode(h, vm, {
+    role: opts.role ?? "k8s-worker",
+    token: opts.token ?? DRILL_TOKEN,
+    serverUrl: serverUrlOf(opts.server),
+  });
+  await waitForNodeReady(h, opts.server, vm);
+}
+
+/**
+ * A healthy server that is NOT the node being worked on.
+ *
+ * Needed because the first server is itself rolled during a migration: after
+ * it is wiped it must rejoin against a survivor, so "the first node" cannot be
+ * the join target and cannot be where cluster state is read from.
+ */
+export function pickJoinTarget(
+  h: Harness,
+  servers: ProvisionedVm[],
+  exclude: ProvisionedVm,
+): ProvisionedVm {
+  for (const vm of servers) {
+    if (vm.name === exclude.name) continue;
+    const r = tryNodeExec(h, vm, "k3s kubectl get --raw /readyz");
+    if (r.exitCode === 0 && r.stdout.trim() === "ok") return vm;
+  }
+  throw new Error(`No healthy server other than ${exclude.name} — cluster has no quorum to join against`);
+}
+
+/** Every server reports /readyz ok. Asserted positively, not inferred. */
+export function allServersReady(h: Harness, servers: ProvisionedVm[]): { ok: boolean; detail: string } {
+  const results = servers.map((vm) => {
+    const r = tryNodeExec(h, vm, "k3s kubectl get --raw /readyz");
+    return `${vm.name}=${r.exitCode === 0 ? r.stdout.trim() : `ERR(${r.exitCode})`}`;
+  });
+  return { ok: results.every((r) => r.endsWith("=ok")), detail: results.join(" ") };
+}

bastion/tests/integration/helpers/longhorn.ts

@@ -0,0 +1,666 @@
+// Install Longhorn on a test cluster and put verifiable data in it.
+//
+// WHAT THIS IS FOR
+// ----------------
+// The rolling drill's whole claim is "the bytes survived". That needs data
+// whose integrity can be checked independently of Longhorn's own opinion of
+// itself -- Longhorn reporting `robustness: healthy` is not evidence that the
+// file is intact, it is evidence that Longhorn thinks it is.
+//
+// So: write a known blob, record its sha256, and re-verify THROUGH A POD
+// RESTART every time. The restart is not incidental. It forces a real
+// detach/attach, frequently onto a different node, which is the only way to
+// show the data followed the volume rather than sitting in a page cache on the
+// node that wrote it.
+//
+// Longhorn version is pinned to the fleet's (1.8.1) so the drill rehearses what
+// production runs, not what happens to be latest.
+
+import type { Harness, ProvisionedVm } from "./provision-harness.js";
+import { log } from "./libvirt.js";
+import { nodeExec, tryNodeExec, kubectlOn, kubectlJson, sleep, k8sNodeName } from "./k3s-cluster.js";
+
+export const LONGHORN_VERSION = "1.8.1";
+export const LONGHORN_NS = "longhorn-system";
+
+/**
+ * Install Longhorn via k3s's built-in helm controller.
+ *
+ * A HelmChart manifest dropped in the server manifests directory is how k3s
+ * installs charts natively, so this needs no helm binary on the host and no
+ * kubeconfig plumbing.
+ *
+ * The namespace is created FIRST, with privileged Pod Security labels. Longhorn
+ * needs hostPath and privileged containers; under the default `restricted`
+ * enforcement its DaemonSets are rejected and the install fails in a way that
+ * looks like a chart problem.
+ *
+ * `createDefaultDiskLabeledNodes: true` matches production, and is the setting
+ * that makes the node label and disk annotation load-bearing: without both, a
+ * node gets no disk and silently hosts no replicas.
+ */
+/**
+ * S3 backup target for the drill, from the environment.
+ *
+ * Absent means the backup/restore phase is skipped rather than silently
+ * "passing" -- a backup test that quietly does nothing is worse than no test.
+ * The run script exports these; see DRILL_S3_* in run-drill.sh.
+ */
+export interface BackupTargetOpts {
+  /** e.g. s3://longhorn-drill-backup@us-east-1/ */
+  target: string;
+  endpoint: string;
+  accessKey: string;
+  secretKey: string;
+}
+
+export function backupTargetFromEnv(): BackupTargetOpts | undefined {
+  const target = process.env["DRILL_S3_TARGET"];
+  const endpoint = process.env["DRILL_S3_ENDPOINT"];
+  const accessKey = process.env["DRILL_S3_ACCESS_KEY"];
+  const secretKey = process.env["DRILL_S3_SECRET_KEY"];
+  if (target === undefined || endpoint === undefined
+    || accessKey === undefined || secretKey === undefined) return undefined;
+  return { target, endpoint, accessKey, secretKey };
+}
+
+export const BACKUP_SECRET = "longhorn-backup-credentials";
+
+export async function installLonghorn(h: Harness, server: ProvisionedVm, opts: {
+  defaultReplicas?: number;
+  backup?: BackupTargetOpts;
+} = {}): Promise<void> {
+  const replicas = opts.defaultReplicas ?? 3;
+
+  log(`[longhorn] creating ${LONGHORN_NS} with privileged PSA labels`);
+  nodeExec(h, server, `k3s kubectl apply -f - <<'EOF'
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: ${LONGHORN_NS}
+  labels:
+    pod-security.kubernetes.io/enforce: privileged
+    pod-security.kubernetes.io/audit: privileged
+    pod-security.kubernetes.io/warn: privileged
+EOF`);
+
+  if (opts.backup !== undefined) {
+    // Secret BEFORE the chart: Longhorn reads it by name from a defaultSetting,
+    // and a missing secret makes the first backup fail with a credential error
+    // that reads like a network problem.
+    //
+    // AWS_ENDPOINTS is what makes this work against a non-AWS gateway; without
+    // it the driver resolves real AWS endpoints.
+    log(`[longhorn] configuring backup target ${opts.backup.target}`);
+    // tryNodeExec, not nodeExec: nodeExec throws with the whole command in the
+    // message, and this command contains the S3 secret. A credential landing in
+    // a log that then gets pasted into drill evidence is not acceptable, so the
+    // failure is reported without the payload.
+    const secretApplied = tryNodeExec(h, server, `k3s kubectl -n ${LONGHORN_NS} apply -f - <<'EOF'
+apiVersion: v1
+kind: Secret
+metadata:
+  name: ${BACKUP_SECRET}
+  namespace: ${LONGHORN_NS}
+stringData:
+  AWS_ACCESS_KEY_ID: ${opts.backup.accessKey}
+  AWS_SECRET_ACCESS_KEY: ${opts.backup.secretKey}
+  AWS_ENDPOINTS: ${opts.backup.endpoint}
+EOF`);
+    if (secretApplied.exitCode !== 0) {
+      throw new Error(
+        `[${server.name}] could not create the ${BACKUP_SECRET} secret ` +
+        `(exit ${secretApplied.exitCode}). Command withheld: it contains credentials. ` +
+        `stderr: ${secretApplied.stderr.trim().slice(0, 200)}`);
+    }
+  }
+
+  log(`[longhorn] installing chart ${LONGHORN_VERSION} via k3s helm controller`);
+  nodeExec(h, server, `cat > /var/lib/rancher/k3s/server/manifests/longhorn.yaml <<'EOF'
+apiVersion: helm.cattle.io/v1
+kind: HelmChart
+metadata:
+  name: longhorn
+  namespace: kube-system
+spec:
+  chart: longhorn
+  repo: https://charts.longhorn.io
+  version: ${LONGHORN_VERSION}
+  targetNamespace: ${LONGHORN_NS}
+  valuesContent: |-
+    defaultSettings:
+      defaultDataPath: /var/lib/longhorn
+      defaultReplicaCount: ${replicas}
+      createDefaultDiskLabeledNodes: true
+      defaultDataLocality: best-effort
+      replicaSoftAntiAffinity: false
+      replicaReplenishmentWaitInterval: 600
+      orphanAutoDeletion: true
+    persistence:
+      defaultClass: true
+      defaultClassReplicaCount: ${replicas}
+      defaultFsType: ext4
+    longhornUI:
+      replicas: 1
+EOF`);
+}
+
+/**
+ * Every expected node has a Longhorn disk that is actually schedulable.
+ *
+ * Takes the VMs, not names, and resolves each one's Kubernetes name itself.
+ * Longhorn keys its node records by the FQDN the kubelet registered
+ * (`drill-s1.pxe-test.local`), never the libvirt domain name (`drill-s1`), so
+ * a caller passing VM names matches nothing and waits out the full timeout
+ * while the cluster is in fact perfectly healthy.
+ */
+export async function waitForLonghornReady(
+  h: Harness,
+  server: ProvisionedVm,
+  vms: ProvisionedVm[],
+  timeoutMs = 20 * 60_000,
+): Promise<void> {
+  const expectedNodes = vms.map((vm) => k8sNodeName(h, vm));
+  const start = Date.now();
+  let last = "";
+  while (Date.now() - start < timeoutMs) {
+    const r = tryNodeExec(h, server,
+      `k3s kubectl -n ${LONGHORN_NS} get nodes.longhorn.io -o json`, 60_000);
+    if (r.exitCode === 0) {
+      try {
+        const parsed = JSON.parse(r.stdout) as { items: {
+          metadata: { name: string };
+          status?: { diskStatus?: Record<string, { conditions?: { type: string; status: string }[] }> };
+        }[] };
+        const ready = parsed.items.filter((n) => {
+          const disks = Object.values(n.status?.diskStatus ?? {});
+          return disks.length > 0 && disks.some((d) =>
+            (d.conditions ?? []).some((c) => c.type === "Schedulable" && c.status === "True"));
+        }).map((n) => n.metadata.name);
+
+        last = `${ready.length}/${expectedNodes.length} with schedulable disk: ${ready.join(",") || "none"}`;
+        if (expectedNodes.every((n) => ready.includes(n))) {
+          log(`[longhorn] ready after ${Math.round((Date.now() - start) / 1000)}s — ${last}`);
+          return;
+        }
+      } catch { /* API not serving the CRD yet */ }
+    }
+    await sleep(15_000);
+  }
+
+  // Say WHY, per node, instead of just "not ready". A node with no disk has a
+  // specific cause and the record knows it; without this the only way to find
+  // out is to catch the cluster before teardown, which is a race you lose.
+  const detail: string[] = [];
+  try {
+    const raw = kubectlJson<{ items: {
+      metadata: { name: string; annotations?: Record<string, string>; labels?: Record<string, string> };
+      spec?: { disks?: Record<string, unknown>; allowScheduling?: boolean };
+      status?: {
+        conditions?: { type: string; status: string; reason?: string; message?: string }[];
+        diskStatus?: Record<string, { conditions?: { type: string; status: string; reason?: string; message?: string }[] }>;
+      };
+    }[] }>(h, server, `-n ${LONGHORN_NS} get nodes.longhorn.io`);
+
+    for (const n of raw.items) {
+      const disks = Object.keys(n.spec?.disks ?? {});
+      const conds = (n.status?.conditions ?? [])
+        .filter((c) => c.status !== "True")
+        .map((c) => `${c.type}=${c.status}(${c.reason ?? ""}: ${(c.message ?? "").slice(0, 80)})`);
+      const diskConds = Object.entries(n.status?.diskStatus ?? {}).flatMap(([dn, d]) =>
+        (d.conditions ?? []).filter((c) => c.status !== "True")
+          .map((c) => `${dn}:${c.type}=${c.status}(${c.reason ?? ""}: ${(c.message ?? "").slice(0, 80)})`));
+      const ann = n.metadata.annotations?.["node.longhorn.io/default-disks-config"] ?? "<absent>";
+      const lbl = n.metadata.labels?.["node.longhorn.io/create-default-disk"] ?? "<absent>";
+      detail.push(
+        `  ${n.metadata.name}: disks=[${disks.join(",")}] label=${lbl} ann=${ann} ` +
+        `${conds.length > 0 ? `nodeConds=${conds.join(";")} ` : ""}` +
+        `${diskConds.length > 0 ? `diskConds=${diskConds.join(";")}` : ""}`);
+    }
+  } catch (e) {
+    detail.push(`  (could not read Longhorn nodes: ${e instanceof Error ? e.message : String(e)})`);
+  }
+
+  throw new Error(
+    `Longhorn not ready in ${timeoutMs / 60000} min (${last}).\n` +
+    `A node with no disk usually means the create-default-disk label or the\n` +
+    `default-disks-config annotation is missing, or the path is not mounted.\n` +
+    detail.join("\n"));
+}
+
+/** Create a PVC plus a Deployment that mounts it at /data. */
+export function createDataWorkload(h: Harness, server: ProvisionedVm, opts: {
+  name: string;
+  sizeGi: number;
+  replicas: number;
+}): void {
+  log(`[longhorn] creating ${opts.name} (${opts.sizeGi}Gi, ${opts.replicas} replicas)`);
+  nodeExec(h, server, `k3s kubectl apply -f - <<'EOF'
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: lh-${opts.name}
+provisioner: driver.longhorn.io
+allowVolumeExpansion: true
+parameters:
+  numberOfReplicas: "${opts.replicas}"
+  staleReplicaTimeout: "30"
+  fsType: ext4
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: ${opts.name}
+spec:
+  accessModes: [ReadWriteOnce]
+  storageClassName: lh-${opts.name}
+  resources:
+    requests:
+      storage: ${opts.sizeGi}Gi
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: ${opts.name}
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels: {app: ${opts.name}}
+  template:
+    metadata:
+      labels: {app: ${opts.name}}
+    spec:
+      # Pod Security Admission enforces "restricted" on this namespace (the k3s
+      # hardening group applies the label), so a bare pod spec is REJECTED
+      # outright -- the Deployment reports 0/1 with no pod and no obvious
+      # reason unless you read the ReplicaSet events. Every one of these
+      # fields is required by "restricted"; dropping any brings it back.
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        runAsGroup: 1000
+        # fsGroup so the non-root user can actually write to the Longhorn
+        # volume, which is otherwise owned by root.
+        fsGroup: 1000
+        seccompProfile: {type: RuntimeDefault}
+      containers:
+        - name: shell
+          image: busybox:1.36
+          command: ["sh", "-c", "sleep infinity"]
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities: {drop: ["ALL"]}
+          volumeMounts:
+            - {name: data, mountPath: /data}
+      volumes:
+        - name: data
+          persistentVolumeClaim: {claimName: ${opts.name}}
+EOF`);
+}
+
+/** Wait for the workload's pod to be Running and its volume attached. */
+export async function waitForWorkload(
+  h: Harness, server: ProvisionedVm, name: string, timeoutMs = 10 * 60_000,
+): Promise<void> {
+  const start = Date.now();
+  let last = "";
+  while (Date.now() - start < timeoutMs) {
+    const r = tryNodeExec(h, server,
+      `k3s kubectl get pods -l app=${name} -o jsonpath='{.items[*].status.phase}'`);
+    last = r.stdout.trim();
+    if (last === "Running") {
+      // Ready, not merely scheduled: the volume mount happens before Ready.
+      const ready = tryNodeExec(h, server,
+        `k3s kubectl get pods -l app=${name} -o jsonpath='{.items[*].status.containerStatuses[*].ready}'`);
+      if (ready.stdout.trim() === "true") return;
+    }
+    await sleep(5_000);
+  }
+  throw new Error(`workload ${name} not running in ${timeoutMs / 60000} min (last phase: ${last || "none"})`);
+}
+
+function podOf(h: Harness, server: ProvisionedVm, name: string): string {
+  const pod = kubectlOn(h, server, `get pods -l app=${name} -o jsonpath='{.items[0].metadata.name}'`).trim();
+  if (!pod) throw new Error(`no pod for workload ${name}`);
+  return pod;
+}
+
+/**
+ * Write a random blob, fsync it, and record its digest.
+ *
+ * conv=fsync matters: without it the write can still be in the guest's page
+ * cache, and a checksum taken immediately afterwards proves nothing about what
+ * reached the replicas.
+ */
+export async function writeBlob(h: Harness, server: ProvisionedVm, opts: {
+  name: string; mib: number;
+}): Promise<string> {
+  const pod = podOf(h, server, opts.name);
+  log(`[longhorn] writing ${opts.mib} MiB to ${opts.name}`);
+  nodeExec(h, server,
+    `k3s kubectl exec ${pod} -- sh -c ` +
+    `"dd if=/dev/urandom of=/data/blob bs=1M count=${opts.mib} conv=fsync 2>/dev/null && ` +
+    `cd /data && sha256sum blob > blob.sha256 && sync"`,
+    15 * 60_000);
+
+  const digest = kubectlOn(h, server, `exec ${pod} -- cat /data/blob.sha256`).trim().split(/\s+/)[0] ?? "";
+  if (!/^[0-9a-f]{64}$/.test(digest)) throw new Error(`bad digest for ${opts.name}: ${digest}`);
+  log(`[longhorn] ${opts.name} digest ${digest.slice(0, 16)}...`);
+  return digest;
+}
+
+/**
+ * Restart the pod, then verify the blob.
+ *
+ * The restart is the point. It detaches and reattaches the volume, so a pass
+ * means the bytes came back from Longhorn rather than from wherever the writer
+ * left them.
+ */
+export async function verifyBlob(h: Harness, server: ProvisionedVm, opts: {
+  name: string; expectDigest: string;
+}): Promise<void> {
+  const before = podOf(h, server, opts.name);
+  kubectlOn(h, server, `delete pod ${before} --wait=true --timeout=5m`, 6 * 60_000);
+  await waitForWorkload(h, server, opts.name);
+  const after = podOf(h, server, opts.name);
+  if (after === before) throw new Error(`${opts.name}: pod was not actually replaced`);
+
+  // sha256sum -c proves the file matches what was recorded IN the volume...
+  const check = tryNodeExec(h, server,
+    `k3s kubectl exec ${after} -- sh -c "cd /data && sha256sum -c blob.sha256"`, 15 * 60_000);
+  if (check.exitCode !== 0) {
+    throw new Error(`${opts.name}: DATA CORRUPTION — sha256sum -c failed: ${check.stdout}${check.stderr}`);
+  }
+
+  // ...and this proves the recorded digest is still the one we started with,
+  // which -c alone would not catch if both file and checksum were replaced.
+  const actual = kubectlOn(h, server,
+    `exec ${after} -- sh -c "cd /data && sha256sum blob"`, 15 * 60_000).trim().split(/\s+/)[0];
+  if (actual !== opts.expectDigest) {
+    throw new Error(`${opts.name}: DIGEST CHANGED — expected ${opts.expectDigest}, got ${actual}`);
+  }
+  log(`[longhorn] ${opts.name} verified intact across a detach/attach (pod ${before} -> ${after})`);
+}
+
+export interface VolumeState {
+  pv: string;
+  state: string;
+  robustness: string;
+  wanted: number;
+  replicaNodes: string[];
+  healthyReplicas: number;
+}
+
+/** Longhorn's view of the volume backing a PVC. */
+export function volumeState(h: Harness, server: ProvisionedVm, pvcName: string): VolumeState {
+  const pv = kubectlOn(h, server, `get pvc ${pvcName} -o jsonpath='{.spec.volumeName}'`).trim();
+  if (!pv) throw new Error(`PVC ${pvcName} has no bound volume`);
+
+  const vol = kubectlJson<{
+    spec: { numberOfReplicas?: number };
+    status?: { state?: string; robustness?: string };
+  }>(h, server, `-n ${LONGHORN_NS} get volumes.longhorn.io ${pv}`);
+
+  const reps = kubectlJson<{ items: {
+    spec: { nodeID?: string; volumeName?: string };
+    status?: { currentState?: string; started?: boolean };
+  }[] }>(h, server, `-n ${LONGHORN_NS} get replicas.longhorn.io`);
+
+  const mine = reps.items.filter((r) => r.spec.volumeName === pv);
+  return {
+    pv,
+    state: vol.status?.state ?? "unknown",
+    robustness: vol.status?.robustness ?? "unknown",
+    wanted: vol.spec.numberOfReplicas ?? 0,
+    replicaNodes: mine.map((r) => r.spec.nodeID ?? "").filter(Boolean).sort(),
+    healthyReplicas: mine.filter((r) => r.status?.currentState === "running").length,
+  };
+}
+
+/**
+ * Wait for a volume to be healthy on the expected number of distinct nodes.
+ *
+ * Logs rebuild progress while waiting, so a slow rebuild reads as progress
+ * rather than as an opaque hour of silence.
+ */
+export async function waitForVolumeHealthy(h: Harness, server: ProvisionedVm, opts: {
+  pvc: string; expectReplicas: number; timeoutMs?: number;
+}): Promise<VolumeState> {
+  const timeoutMs = opts.timeoutMs ?? 60 * 60_000;
+  const start = Date.now();
+  let last = "";
+  while (Date.now() - start < timeoutMs) {
+    let st: VolumeState | null = null;
+    try { st = volumeState(h, server, opts.pvc); } catch { /* transient */ }
+
+    if (st) {
+      const distinct = new Set(st.replicaNodes).size;
+      last = `${st.state}/${st.robustness} on ${distinct} node(s): ${st.replicaNodes.join(",")}`;
+      if (st.robustness === "healthy" && distinct >= opts.expectReplicas) {
+        log(`[longhorn] ${opts.pvc} healthy on ${distinct} nodes after ${Math.round((Date.now() - start) / 1000)}s`);
+        return st;
+      }
+      const prog = tryNodeExec(h, server,
+        `k3s kubectl -n ${LONGHORN_NS} get engines.longhorn.io -o jsonpath=` +
+        `'{range .items[*]}{.spec.volumeName}{" "}{.status.rebuildStatus}{"\\n"}{end}'`);
+      const line = prog.stdout.split("\n").find((l) => l.includes(st!.pv) && l.trim() !== st!.pv);
+      log(`[longhorn] ${opts.pvc}: ${last}${line ? ` | rebuild ${line.replace(st.pv, "").trim()}` : ""}`);
+    }
+    await sleep(20_000);
+  }
+  throw new Error(`${opts.pvc} not healthy on ${opts.expectReplicas} nodes in ${timeoutMs / 60000} min (last: ${last})`);
+}
+
+/**
+ * Wait until Longhorn reports the backup target usable.
+ *
+ * `available: true` is the appliance saying it could reach the endpoint AND
+ * authenticate. Backing up without checking this first produces a failure that
+ * looks like a volume problem.
+ */
+export async function setBackupTarget(
+  h: Harness, server: ProvisionedVm, backup: BackupTargetOpts, timeoutMs = 5 * 60_000,
+): Promise<void> {
+  // PATCH the CR; do NOT set defaultSettings.backupTarget.
+  //
+  // In Longhorn 1.8 the backup target moved out of settings and onto
+  // `backuptargets.longhorn.io/default`. The helm value is SILENTLY IGNORED --
+  // no `backup-target` Setting object is created at all, the default
+  // BackupTarget keeps an empty URL, and everything reports success. Observed
+  // on a fresh 1.8.1 install here and confirmed against the live fleet.
+  const deadline = Date.now() + timeoutMs;
+  while (Date.now() < deadline) {
+    const exists = tryNodeExec(h, server,
+      `k3s kubectl -n ${LONGHORN_NS} get backuptargets.longhorn.io default -o name`);
+    if (exists.exitCode === 0 && exists.stdout.trim() !== "") break;
+    await sleep(5_000);
+  }
+
+  const patch = JSON.stringify({
+    spec: { backupTargetURL: backup.target, credentialSecret: BACKUP_SECRET, pollInterval: "5m0s" },
+  });
+  nodeExec(h, server,
+    `k3s kubectl -n ${LONGHORN_NS} patch backuptargets.longhorn.io default --type=merge -p '${patch}'`);
+  log(`[longhorn] backup target set to ${backup.target}`);
+}
+
+export async function waitForBackupTarget(
+  h: Harness, server: ProvisionedVm, timeoutMs = 5 * 60_000,
+): Promise<void> {
+  const start = Date.now();
+  let last = "";
+  while (Date.now() - start < timeoutMs) {
+    const r = tryNodeExec(h, server,
+      `k3s kubectl -n ${LONGHORN_NS} get backuptargets.longhorn.io default ` +
+      `-o jsonpath='{.spec.backupTargetURL}|{.status.available}|{.status.conditions[*].message}'`);
+    last = r.stdout.trim();
+    if (r.exitCode === 0 && last.includes("|true|")) {
+      log(`[longhorn] backup target available after ${Math.round((Date.now() - start) / 1000)}s`);
+      return;
+    }
+    await sleep(10_000);
+  }
+  throw new Error(
+    `Backup target not available in ${timeoutMs / 60000} min (url|available|conditions = ${last}). ` +
+    `Longhorn could not reach or authenticate to the endpoint.`);
+}
+
+/**
+ * Snapshot a volume and back it up, returning the backup's name.
+ *
+ * Longhorn's own CRs, not the CSI VolumeSnapshot API -- this cluster has no
+ * snapshot.storage.k8s.io resources at all, so VolumeSnapshot/VolumeSnapshotClass
+ * simply do not exist here. Any backup tooling has to use these.
+ */
+export async function backupVolume(
+  h: Harness, server: ProvisionedVm, pvc: string, timeoutMs = 20 * 60_000,
+): Promise<{ backup: string; volume: string }> {
+  const pv = volumeState(h, server, pvc).pv;
+  const snapName = `drill-snap-${Date.now().toString(36)}`;
+
+  log(`[longhorn] snapshotting ${pvc} (${pv})`);
+  nodeExec(h, server, `k3s kubectl -n ${LONGHORN_NS} apply -f - <<'EOF'
+apiVersion: longhorn.io/v1beta2
+kind: Snapshot
+metadata:
+  name: ${snapName}
+  namespace: ${LONGHORN_NS}
+spec:
+  volume: ${pv}
+  createSnapshot: true
+EOF`);
+
+  // The snapshot has to exist before a backup can reference it.
+  const snapDeadline = Date.now() + 5 * 60_000;
+  while (Date.now() < snapDeadline) {
+    const r = tryNodeExec(h, server,
+      `k3s kubectl -n ${LONGHORN_NS} get snapshots.longhorn.io ${snapName} -o jsonpath='{.status.readyToUse}'`);
+    if (r.stdout.trim() === "true") break;
+    await sleep(5_000);
+  }
+
+  const backupName = `drill-bak-${Date.now().toString(36)}`;
+  log(`[longhorn] backing up ${pvc} to the backup target`);
+  nodeExec(h, server, `k3s kubectl -n ${LONGHORN_NS} apply -f - <<'EOF'
+apiVersion: longhorn.io/v1beta2
+kind: Backup
+metadata:
+  name: ${backupName}
+  namespace: ${LONGHORN_NS}
+  labels:
+    backup-volume: ${pv}
+spec:
+  snapshotName: ${snapName}
+EOF`);
+
+  const start = Date.now();
+  let last = "";
+  while (Date.now() - start < timeoutMs) {
+    const r = tryNodeExec(h, server,
+      `k3s kubectl -n ${LONGHORN_NS} get backups.longhorn.io ${backupName} ` +
+      `-o jsonpath='{.status.state}|{.status.progress}|{.status.error}'`);
+    last = r.stdout.trim();
+    if (last.startsWith("Completed")) {
+      log(`[longhorn] backup ${backupName} completed after ${Math.round((Date.now() - start) / 1000)}s`);
+      return { backup: backupName, volume: pv };
+    }
+    if (last.includes("Error")) {
+      throw new Error(`Backup ${backupName} failed: ${last}`);
+    }
+    await sleep(10_000);
+  }
+  throw new Error(`Backup ${backupName} did not complete in ${timeoutMs / 60000} min (state|progress|error = ${last})`);
+}
+
+/**
+ * Restore a backup into a brand-new PVC and workload.
+ *
+ * Uses the `fromBackup` StorageClass parameter, which is the supported route
+ * when the CSI VolumeSnapshot API is unavailable. The restored volume is a
+ * separate volume with its own replicas -- nothing about the original is
+ * touched, so a failed restore cannot damage the source.
+ */
+export async function restoreFromBackup(h: Harness, server: ProvisionedVm, opts: {
+  backup: string;
+  name: string;
+  sizeGi: number;
+}): Promise<void> {
+  const url = nodeExec(h, server,
+    `k3s kubectl -n ${LONGHORN_NS} get backups.longhorn.io ${opts.backup} -o jsonpath='{.status.url}'`).trim();
+  if (url === "") throw new Error(`Backup ${opts.backup} has no status.url; cannot restore from it`);
+  log(`[longhorn] restoring ${opts.backup} into ${opts.name}`);
+
+  nodeExec(h, server, `k3s kubectl apply -f - <<'EOF'
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: lh-${opts.name}
+provisioner: driver.longhorn.io
+allowVolumeExpansion: true
+parameters:
+  numberOfReplicas: "2"
+  staleReplicaTimeout: "30"
+  fsType: ext4
+  fromBackup: "${url}"
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: ${opts.name}
+spec:
+  accessModes: [ReadWriteOnce]
+  storageClassName: lh-${opts.name}
+  resources:
+    requests:
+      storage: ${opts.sizeGi}Gi
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: ${opts.name}
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels: {app: ${opts.name}}
+  template:
+    metadata:
+      labels: {app: ${opts.name}}
+    spec:
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        runAsGroup: 1000
+        fsGroup: 1000
+        seccompProfile: {type: RuntimeDefault}
+      containers:
+        - name: shell
+          image: busybox:1.36
+          command: ["sh", "-c", "sleep infinity"]
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities: {drop: ["ALL"]}
+          volumeMounts:
+            - {name: data, mountPath: /data}
+      volumes:
+        - name: data
+          persistentVolumeClaim: {claimName: ${opts.name}}
+EOF`);
+}
+
+/**
+ * Read a blob's digest from a workload, without restarting it.
+ *
+ * Used on a RESTORED volume, where the point is to compare bytes against the
+ * original's recorded digest rather than to prove durability again.
+ */
+export function readBlobDigest(h: Harness, server: ProvisionedVm, name: string): string {
+  const pod = kubectlOn(h, server, `get pods -l app=${name} -o jsonpath='{.items[0].metadata.name}'`).trim();
+  if (pod === "") throw new Error(`no pod for workload ${name}`);
+  const out = kubectlOn(h, server,
+    `exec ${pod} -- sh -c "cd /data && sha256sum blob"`, 15 * 60_000).trim();
+  return out.split(/\s+/)[0] ?? "";
+}

bastion/tests/integration/helpers/provision-harness.ts

@@ -0,0 +1,534 @@
+// Reusable bastion + VM provisioning harness for partition-layout tests.
+//
+// pxe-provision.test.ts grew a ~200-line beforeAll that stands up a real
+// bastion, caches Fedora boot artifacts, and drives one VM through discovery
+// and install. Testing a partition MATRIX needs that same sequence for several
+// VMs and several roles, so it lives here instead of being copy-pasted per
+// topology.
+
+import { existsSync, mkdirSync, copyFileSync, symlinkSync, writeFileSync } from "node:fs";
+import { execSync } from "node:child_process";
+import { join } from "node:path";
+import { tmpdir } from "node:os";
+import type { BastionConfig } from "../../../src/shared/src/types/config.js";
+import type { Role } from "../../../src/shared/src/types/state.js";
+import { log, waitForSsh } from "./libvirt.js";
+import { ensurePxeNetwork, deleteNftablesRejectRules, PXE_NETWORK_NAME, PXE_BRIDGE, PXE_GATEWAY, PXE_SUBNET } from "./pxe-network.js";
+import { createPxeVm, destroyPxeVm, getVmMac, rebootPxeVm } from "./pxe-vm.js";
+import { sshExec } from "./ssh.js";
+
+export const DISCOVERY_TIMEOUT_MS = 6 * 60_000;
+export const INSTALL_TIMEOUT_MS = 60 * 60_000;
+export const SSH_TIMEOUT_MS = 12 * 60_000;
+
+export const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
+
+export interface Harness {
+  config: BastionConfig;
+  close: () => Promise<void>;
+  httpPort: number;
+  sshUser: string;
+  sshKeyPath: string;
+}
+
+async function pollUntil<T>(url: string, check: (d: T) => boolean, timeoutMs: number, intervalMs = 5000): Promise<T> {
+  const start = Date.now();
+  let lastErr = "";
+  while (Date.now() - start < timeoutMs) {
+    try {
+      const res = await fetch(url);
+      if (res.ok) {
+        const data = (await res.json()) as T;
+        if (check(data)) return data;
+      }
+    } catch (e) { lastErr = e instanceof Error ? e.message : String(e); }
+    await sleep(intervalMs);
+  }
+  throw new Error(`Timeout after ${timeoutMs}ms polling ${url}${lastErr ? ` (last error: ${lastErr})` : ""}`);
+}
+
+/**
+ * Stand up a real bastion on the isolated PXE network.
+ *
+ * FEDORA_MIRROR is honoured (config.ts now reads it) -- point it at a local
+ * mirror before running a multi-VM matrix, or every install pulls the full
+ * package set from download.fedoraproject.org.
+ */
+export async function startTestBastion(opts: {
+  httpPort: number;
+  sshUser: string;
+  sshPubKey: string;
+  sshKeyPath: string;
+  label: string;
+  /**
+   * Leave the post-provision k3s auto-install ON (the default) or suppress it.
+   *
+   * Any test that forms a cluster ITSELF must pass false. The auto-install runs
+   * a bare `INSTALL_K3S_EXEC=server|agent` with no server URL and no token, so
+   * three combo nodes become three separate single-node clusters -- and it
+   * stops k3s and deletes CNI state minutes after an install lands, landing on
+   * top of whatever the test is doing by then.
+   */
+  autoK3s?: boolean;
+}): Promise<Harness> {
+  ensurePxeNetwork();
+
+  const testDir = join(tmpdir(), `lab-${opts.label}-${Date.now()}`);
+  for (const sub of ["tftp", "http", "logs"]) mkdirSync(join(testDir, sub), { recursive: true });
+
+  const { createApp } = await import("../../../src/bastion/src/server.js");
+  const { loadConfig } = await import("../../../src/bastion/src/config.js");
+  const { generateDnsmasqConf, startDnsmasq, stopDnsmasq } = await import("../../../src/bastion/src/services/dnsmasq.js");
+  const { generateDiscoverKickstart } = await import("../../../src/bastion/src/services/kickstart-generator.js");
+  const { renderBootIpxe } = await import("../../../src/bastion/src/templates/boot.ipxe.js");
+
+  const config = loadConfig({
+    bastionDir: testDir,
+    httpPort: opts.httpPort,
+    iface: "virbr-pxe",
+    serverIp: PXE_GATEWAY,
+    network: `${PXE_SUBNET}.0`,
+    gateway: PXE_GATEWAY,
+    dhcpMode: "full",
+    dhcpRangeStart: `${PXE_SUBNET}.100`,
+    dhcpRangeEnd: `${PXE_SUBNET}.200`,
+    domain: "pxe-test.local",
+    sshKeys: [opts.sshPubKey],
+    adminUser: opts.sshUser,
+    ...(opts.autoK3s === undefined ? {} : { autoK3s: opts.autoK3s }),
+  });
+
+  const ipxeSrc = "/usr/share/ipxe/ipxe-snponly-x86_64.efi";
+  if (!existsSync(ipxeSrc)) throw new Error(`iPXE not found: ${ipxeSrc}. Install: sudo dnf install ipxe-bootimgs-x86`);
+  const ipxeDest = join(config.tftpDir, "ipxe.efi");
+  copyFileSync(ipxeSrc, ipxeDest);
+
+  const cacheDir = "/var/lib/libvirt/images/lab-pxe-cache";
+  execSync(`mkdir -p "${cacheDir}"`, { stdio: "pipe" });
+  const kernel = join(cacheDir, `vmlinuz-${config.fedoraVersion}`);
+  const initrd = join(cacheDir, `initrd-${config.fedoraVersion}.img`);
+  if (!existsSync(kernel)) {
+    log(`Downloading Fedora ${config.fedoraVersion} kernel from ${config.fedoraMirror}...`);
+    execSync(`curl -sS -L -f -o "${kernel}" "${config.fedoraMirror}/images/pxeboot/vmlinuz"`, { stdio: "inherit", timeout: 300_000 });
+  }
+  if (!existsSync(initrd)) {
+    log(`Downloading Fedora ${config.fedoraVersion} initrd...`);
+    execSync(`curl -sS -L -f -o "${initrd}" "${config.fedoraMirror}/images/pxeboot/initrd.img"`, { stdio: "inherit", timeout: 300_000 });
+  }
+  copyFileSync(kernel, join(config.httpDir, "vmlinuz"));
+  copyFileSync(initrd, join(config.httpDir, "initrd.img"));
+  try { symlinkSync(ipxeDest, join(config.httpDir, "ipxe.efi")); } catch { /* exists */ }
+
+  writeFileSync(join(config.httpDir, "discover.ks"), generateDiscoverKickstart(config));
+  writeFileSync(join(config.httpDir, "boot.ipxe"), renderBootIpxe({ serverIp: config.serverIp, httpPort: config.httpPort }));
+  generateDnsmasqConf(config);
+
+  // Kill dnsmasq processes left behind by EARLIER TEST RUNS.
+  //
+  // startTestBastion's close() shuts down the HTTP app but never stopped
+  // dnsmasq, so every run leaked one. A leaked dnsmasq keeps :67 on the bridge
+  // and keeps answering DHCP -- with the PREVIOUS run's config, pointing PXE
+  // clients at a bastion port that is no longer listening. The next run then
+  // sees perfect DHCP (offer, request, ack, even the right bootfile name) and
+  // no machine ever reaches it. That is not a failure mode anyone guesses from
+  // "discovery timed out".
+  killOrphanedDnsmasq();
+
+  const { app, syslog } = createApp(config);
+  await app.listen({ port: config.httpPort, host: "0.0.0.0" });
+  syslog.start();
+  // NOT awaited, deliberately.
+  //
+  // startDnsmasq returns the execa child handle, and because the function is
+  // `async` the promise it returns ADOPTS that thenable -- which settles only
+  // when dnsmasq EXITS. With --no-daemon that is never, so `await` here hangs
+  // the harness silently and forever, with the log frozen right after
+  // "TFTP root is ...". (Learned by doing exactly that.) The rejection still
+  // has to be handled or an early bind failure becomes an unhandled rejection.
+  startDnsmasq(config).catch((err) => log(`dnsmasq: ${err instanceof Error ? err.message : String(err)}`));
+  await sleep(1000);
+
+  // Prove DHCP is actually listening before handing the harness back.
+  //
+  // startDnsmasq's rejection is swallowed into a log line, so a dnsmasq that
+  // failed to bind looks exactly like one that started. The symptom then
+  // arrives six minutes later as four VMs timing out in discovery, which
+  // reads as a PXE or kickstart problem rather than "there was never any
+  // DHCP". Fail here, immediately, with the real reason.
+  await assertDhcpListening(join(testDir, "dnsmasq.conf"));
+  log(`Bastion up on :${opts.httpPort} (mirror: ${config.fedoraMirror}, autoK3s=${config.autoK3s})`);
+
+  return {
+    config,
+    httpPort: opts.httpPort,
+    sshUser: opts.sshUser,
+    sshKeyPath: opts.sshKeyPath,
+    close: async () => {
+      // Stop OUR dnsmasq. close() never used to, which is how a leaked one
+      // ended up owning :67 and serving the PREVIOUS run's TFTP root and
+      // bastion port to the NEXT run's VMs -- they booted toward a dead port
+      // and simply never appeared, while DHCP looked perfect.
+      try { stopDnsmasq(); } catch { /* already gone */ }
+      await app.close();
+      // Belt and braces: if that did not stop it, sweep by config path.
+      killOrphanedDnsmasq();
+    },
+  };
+}
+
+/**
+ * Kill any dnsmasq started by a previous (or current) lab test run.
+ *
+ * Matched on the config path, which is always /tmp/lab-<label>-<ts>/dnsmasq.conf
+ * -- so this never touches libvirt's own dnsmasq for the default network, or
+ * anything else on the host.
+ */
+function killOrphanedDnsmasq(): void {
+  try {
+    // Match on the process NAME first (pgrep -x), then inspect each one's
+    // arguments. A `pgrep -f 'dnsmasq.*/tmp/lab-'` matches the shell running
+    // that very pattern, so the naive version can kill its own shell -- and
+    // libvirt's dnsmasq for the default network must never be touched either.
+    const pids = execSync("pgrep -x dnsmasq 2>/dev/null || true", { encoding: "utf-8" })
+      .trim().split("\n").filter((x) => /^\d+$/.test(x.trim()));
+    let killed = 0;
+    for (const pid of pids) {
+      let args = "";
+      try {
+        args = execSync(`tr '\\0' ' ' < /proc/${pid.trim()}/cmdline 2>/dev/null || true`, { encoding: "utf-8" });
+      } catch { continue; }
+      if (!args.includes("--conf-file=/tmp/lab-")) continue;
+      log(`Killing orphaned test dnsmasq (pid ${pid.trim()}) — it would answer DHCP with stale config`);
+      try { execSync(`sudo kill -9 ${pid.trim()} 2>/dev/null || kill -9 ${pid.trim()} 2>/dev/null || true`); } catch { /* gone */ }
+      killed++;
+    }
+    // Give the socket a moment to be released before we bind it.
+    if (killed > 0) execSync("sleep 1");
+  } catch { /* nothing to kill */ }
+}
+
+/**
+ * Is something serving DHCP on the PXE bridge?
+ *
+ * `ss -lunp` shows UDP listeners; dnsmasq binds port 67. Checked on the bridge
+ * address specifically, because a dnsmasq bound to the wrong interface is the
+ * failure being guarded against.
+ */
+async function assertDhcpListening(confPath: string, timeoutMs = 20_000): Promise<void> {
+  const start = Date.now();
+  let last = "";
+  while (Date.now() - start < timeoutMs) {
+    try {
+      // Check for OUR dnsmasq by its config path, not merely for something on
+      // :67. A leaked dnsmasq from a previous run also holds :67 and also
+      // answers DHCP -- with the wrong bastion port -- so a port check alone
+      // reports success for precisely the situation it is meant to catch.
+      const pids = execSync("pgrep -x dnsmasq 2>/dev/null || true", { encoding: "utf-8" })
+        .trim().split("\n").filter((x) => /^\d+$/.test(x.trim()));
+      last = pids.filter((pid) => {
+        try {
+          return execSync(`tr '\\0' ' ' < /proc/${pid.trim()}/cmdline 2>/dev/null || true`, { encoding: "utf-8" })
+            .includes(confPath);
+        } catch { return false; }
+      }).join(",");
+      if (last !== "") {
+        const ports = execSync("ss -lun 2>/dev/null || true", { encoding: "utf-8" });
+        if (/:67\b/.test(ports)) {
+          log("dnsmasq (ours) is serving DHCP on :67");
+          return;
+        }
+      }
+    } catch { /* retry */ }
+    await sleep(1000);
+  }
+  throw new Error(
+    `dnsmasq for ${confPath} is not serving DHCP after ${timeoutMs / 1000}s. ` +
+    `It most likely failed to bind to ${PXE_BRIDGE} -- check the bastion log for ` +
+    `"error binding DHCP socket", and check for a leaked dnsmasq from an earlier run ` +
+    `(pgrep -af 'dnsmasq.*--conf-file=/tmp/lab-'). Without this every VM sits in ` +
+    `discovery until it times out, six minutes later, looking like a PXE problem.`,
+  );
+}
+
+/**
+ * Wait for the installer to actually start, rebooting into PXE if it does not.
+ *
+ * "Started" means the bastion has seen anything at all for this MAC beyond the
+ * queued state -- a dispatch, a progress line, a status. A VM that never gets
+ * that far is stuck before Anaconda, and only a reboot will shift it.
+ */
+async function waitForInstallStart(
+  base: string,
+  mac: string,
+  name: string,
+  attemptMs = 6 * 60_000,
+  attempts = 3,
+): Promise<void> {
+  for (let attempt = 1; attempt <= attempts; attempt++) {
+    try {
+      // NOT `status === "installing"`. A machine that has merely been QUEUED
+      // already reports that -- the bastion sets it the moment the install is
+      // dispatched, before anything has booted. Using it would match instantly
+      // and this retry would never fire.
+      //
+      // The honest signals that Anaconda is alive are `progress` leaving
+      // "queued", or log lines arriving over syslog.
+      await pollUntil<{ status?: string; progress?: string; log_total?: number }>(
+        `${base}/api/logs/${encodeURIComponent(mac)}`,
+        (d) => d.status === "installed"
+          || (d.progress !== undefined && d.progress !== "queued")
+          || (d.log_total ?? 0) > 0,
+        attemptMs, 5_000);
+      if (attempt > 1) log(`[${name}] PXE boot took on attempt ${attempt}`);
+      return;
+    } catch {
+      if (attempt === attempts) {
+        throw new Error(
+          `[${name}] installer never started after ${attempts} PXE boots ` +
+          `(${attemptMs / 60000} min each). The VM is stuck before Anaconda — ` +
+          `check TFTP and the iPXE chainload.`);
+      }
+      log(`[${name}] no installer activity in ${attemptMs / 60000} min; rebooting into PXE (attempt ${attempt + 1}/${attempts})`);
+      rebootPxeVm(name);
+      deleteNftablesRejectRules();
+    }
+  }
+}
+
+export interface ProvisionedVm { name: string; mac: string; ip: string }
+
+/**
+ * Serial ports are a global resource on the host, so hand out a distinct one
+ * per VM. Without this only one VM can run at a time -- qemu refuses to start
+ * the second with "Address already in use".
+ */
+let nextSerialPort = 4600;
+
+/** Create a blank VM, let it discover, queue the install, wait for SSH. */
+export async function provisionVm(h: Harness, spec: {
+  name: string;
+  role: Role;
+  schemaVersion?: 1 | 2;
+  diskGb: number;
+  memory?: number;
+  vcpus?: number;
+}): Promise<ProvisionedVm> {
+  const base = `http://${PXE_GATEWAY}:${h.httpPort}`;
+  log(`[${spec.name}] creating VM (${spec.diskGb}GB disk, role=${spec.role}, schema=v${spec.schemaVersion ?? 1})`);
+  createPxeVm({
+    name: spec.name,
+    memory: spec.memory ?? 4096,
+    vcpus: spec.vcpus ?? 4,
+    diskSize: spec.diskGb,
+    network: PXE_NETWORK_NAME,
+    serialPort: nextSerialPort++,
+  });
+  deleteNftablesRejectRules();
+
+  const mac = getVmMac(spec.name);
+  if (!mac) throw new Error(`[${spec.name}] no MAC`);
+
+  log(`[${spec.name}] waiting for discovery (${mac})...`);
+  await pollUntil<{ discovered: Record<string, unknown> }>(
+    `${base}/api/machines`, (d) => mac in (d.discovered ?? {}), DISCOVERY_TIMEOUT_MS);
+
+  log(`[${spec.name}] queueing install...`);
+  const res = await fetch(`${base}/api/install`, {
+    method: "POST",
+    headers: { "Content-Type": "application/json" },
+    body: JSON.stringify({
+      mac, hostname: spec.name, disk: "", role: spec.role,
+      ...(spec.schemaVersion ? { schemaVersion: spec.schemaVersion } : {}),
+    }),
+  });
+  if (!res.ok) throw new Error(`[${spec.name}] install queue rejected: ${res.status} ${await res.text()}`);
+
+  rebootPxeVm(spec.name);
+  deleteNftablesRejectRules();
+
+  // Make sure the PXE boot actually TOOK before settling in for the install.
+  //
+  // A UEFI VM that fails to chainload just retries DHCP quietly, for ever.
+  // Observed: one VM of four sat in a DHCP loop for an HOUR with no TFTP
+  // fetch, then began installing seconds after the 60-minute install timeout
+  // had already fired -- so the run failed on a machine that was, by then,
+  // working. TFTP is UDP and four VMs contend for it, so this is a question of
+  // when rather than if.
+  //
+  // Rebooting is cheap and a boot that has not started within a few minutes is
+  // not going to start on its own.
+  await waitForInstallStart(base, mac, spec.name);
+
+  log(`[${spec.name}] installing (up to ${INSTALL_TIMEOUT_MS / 60000} min)...`);
+  await pollUntil<{ status?: string }>(
+    `${base}/api/logs/${encodeURIComponent(mac)}`, (d) => d.status === "installed", INSTALL_TIMEOUT_MS, 10_000);
+
+  // The installed record carries the DHCP address the machine ended up with.
+  const machines = await pollUntil<{ installed: Record<string, { ip?: string }> }>(
+    `${base}/api/machines`, (d) => Boolean(d.installed?.[mac]?.ip), 2 * 60_000);
+  const ip = machines.installed[mac]!.ip!;
+
+  try {
+    if (execSync(`sudo virsh domstate ${spec.name}`, { encoding: "utf-8" }).trim() === "shut off") {
+      execSync(`sudo virsh start ${spec.name}`, { stdio: "pipe" });
+    }
+  } catch { /* already running */ }
+  deleteNftablesRejectRules();
+
+  log(`[${spec.name}] waiting for SSH at ${ip}...`);
+  await waitForSsh(ip, h.sshUser, SSH_TIMEOUT_MS, h.sshKeyPath);
+  log(`[${spec.name}] up at ${ip}`);
+  return { name: spec.name, mac, ip };
+}
+
+/**
+ * Provision several VMs at once.
+ *
+ * Installs are ~12 minutes each and almost entirely I/O and network wait, so
+ * four in sequence is an hour that four in parallel is twenty minutes. They are
+ * staggered because the expensive, contended part is the start: PXE, DHCP and
+ * the first package fetches all land at once otherwise.
+ *
+ * Rejects only after every VM has settled, so a single failure does not leave
+ * the others half-built and unreferenced -- the caller still gets the names it
+ * needs to tear down.
+ */
+export async function provisionVms(
+  h: Harness,
+  specs: Parameters<typeof provisionVm>[1][],
+  opts: { staggerMs?: number } = {},
+): Promise<ProvisionedVm[]> {
+  const stagger = opts.staggerMs ?? 60_000;
+  const settled = await Promise.allSettled(
+    specs.map(async (spec, i) => {
+      await sleep(i * stagger);
+      return provisionVm(h, spec);
+    }),
+  );
+
+  const failures = settled
+    .map((r, i) => (r.status === "rejected" ? `${specs[i]?.name}: ${String(r.reason)}` : null))
+    .filter((x): x is string => x !== null);
+  if (failures.length > 0) {
+    throw new Error(`provisionVms: ${failures.length}/${specs.length} failed\n  ${failures.join("\n  ")}`);
+  }
+  return settled.map((r) => (r as PromiseFulfilledResult<ProvisionedVm>).value);
+}
+
+/** LV name -> size in MiB, read off the installed machine. */
+export function readLvs(h: Harness, vm: ProvisionedVm): Record<string, number> {
+  const res = sshExec(vm.ip, h.sshUser,
+    "sudo lvs labvg --noheadings --units m --nosuffix -o lv_name,lv_size --separator=,",
+    { keyPath: h.sshKeyPath });
+  if (res.exitCode !== 0) throw new Error(`[${vm.name}] lvs failed: ${res.stderr}`);
+  const map: Record<string, number> = {};
+  for (const line of res.stdout.split("\n")) {
+    const [name, size] = line.trim().split(",");
+    if (name && size) map[name] = Math.round(Number(size));
+  }
+  // Log what actually landed on disk. The VMs are destroyed in afterAll, so
+  // without this the only record of the real layout is the pass/fail bit.
+  const summary = Object.entries(map)
+    .sort(([a], [b]) => a.localeCompare(b))
+    .map(([n, m]) => `${n}=${(m / 1024).toFixed(0)}G`)
+    .join(" ");
+  log(`[${vm.name}] on-disk LVs: ${summary}`);
+  return map;
+}
+
+/** The VG tag that records which schema this disk was built with. */
+export function readSchemaTag(h: Harness, vm: ProvisionedVm): string {
+  return sshExec(vm.ip, h.sshUser, "sudo vgs --noheadings -o vg_tags labvg",
+    { keyPath: h.sshKeyPath }).stdout.trim();
+}
+
+export function destroyVms(...names: string[]): void {
+  for (const n of names) { try { destroyPxeVm(n); } catch { /* already gone */ } }
+}
+
+/**
+ * Reinstall a machine that is already installed, the way
+ * `labctl provision reprovision` does: queue another install for the same MAC
+ * and reboot it into PXE.
+ *
+ * The schema is deliberately NOT passed. A reinstall must reproduce whatever
+ * layout is already on the disk -- %pre reads the lab-schema-v2 LVM tag rather
+ * than trusting the caller -- and passing one here would hide a regression in
+ * exactly that logic.
+ */
+export async function reprovisionVm(
+  h: Harness,
+  vm: ProvisionedVm,
+  role: Role,
+  /**
+   * Only set this to prove it is IGNORED. A reinstall takes its layout from the
+   * disk, not from the request; passing a schema here is how you test that.
+   */
+  requestSchemaVersion?: 1 | 2,
+): Promise<ProvisionedVm> {
+  const base = `http://${PXE_GATEWAY}:${h.httpPort}`;
+  log(`[${vm.name}] reprovisioning (role=${role}, schema: whatever is on disk)...`);
+
+  const res = await fetch(`${base}/api/install`, {
+    method: "POST",
+    headers: { "Content-Type": "application/json" },
+    body: JSON.stringify({
+      mac: vm.mac, hostname: vm.name, disk: "", role,
+      ...(requestSchemaVersion ? { schemaVersion: requestSchemaVersion } : {}),
+    }),
+  });
+  if (!res.ok) throw new Error(`[${vm.name}] reinstall queue rejected: ${res.status} ${await res.text()}`);
+
+  rebootPxeVm(vm.name);
+  deleteNftablesRejectRules();
+
+  log(`[${vm.name}] reinstalling...`);
+  await pollUntil<{ status?: string }>(
+    `${base}/api/logs/${encodeURIComponent(vm.mac)}`,
+    (d) => d.status === "installed", INSTALL_TIMEOUT_MS, 10_000);
+
+  const machines = await pollUntil<{ installed: Record<string, { ip?: string }> }>(
+    `${base}/api/machines`, (d) => Boolean(d.installed?.[vm.mac]?.ip), 2 * 60_000);
+  const ip = machines.installed[vm.mac]!.ip!;
+
+  try {
+    if (execSync(`sudo virsh domstate ${vm.name}`, { encoding: "utf-8" }).trim() === "shut off") {
+      execSync(`sudo virsh start ${vm.name}`, { stdio: "pipe" });
+    }
+  } catch { /* already running */ }
+  deleteNftablesRejectRules();
+
+  await waitForSsh(ip, h.sshUser, SSH_TIMEOUT_MS, h.sshKeyPath);
+  log(`[${vm.name}] back up at ${ip} after reinstall`);
+  return { ...vm, ip };
+}
+
+/**
+ * Drop a marker file into each mount point that a reinstall is supposed to
+ * preserve. Ported from test-reprovision.sh, which proved the idea but only
+ * ever ran by hand.
+ */
+export function writeMarkers(h: Harness, vm: ProvisionedVm, mounts: string[]): string {
+  const token = `preserve-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`;
+  for (const m of mounts) {
+    const r = sshExec(vm.ip, h.sshUser,
+      `sudo sh -c 'echo ${token} > ${m}/LAB_PRESERVE_TEST.txt && sync'`,
+      { keyPath: h.sshKeyPath });
+    if (r.exitCode !== 0) throw new Error(`[${vm.name}] could not write marker in ${m}: ${r.stderr}`);
+  }
+  log(`[${vm.name}] wrote marker ${token} into ${mounts.join(", ")}`);
+  return token;
+}
+
+/** Which mount points still carry the marker. */
+export function readMarkers(h: Harness, vm: ProvisionedVm, mounts: string[]): Record<string, string | null> {
+  const out: Record<string, string | null> = {};
+  for (const m of mounts) {
+    const r = sshExec(vm.ip, h.sshUser, `sudo cat ${m}/LAB_PRESERVE_TEST.txt 2>/dev/null || true`,
+      { keyPath: h.sshKeyPath });
+    const v = r.stdout.trim();
+    out[m] = v === "" ? null : v;
+  }
+  log(`[${vm.name}] markers after reinstall: ${JSON.stringify(out)}`);
+  return out;
+}

bastion/tests/integration/helpers/pxe-network.ts

@@ -56,6 +56,17 @@ export function ensurePxeNetwork(): void {
 
     try { unlinkSync(xmlPath); } catch { /* ignore */ }
 
+    // `virsh net-start` returns before the bridge is usable. The bastion's
+    // dnsmasq starts about a second later and binds to this device by name; if
+    // the interface is not up and addressed yet it fails with "error binding
+    // DHCP socket to device virbr-pxe", and because that failure is
+    // asynchronous the harness carries on with NO DHCP AT ALL -- every VM then
+    // sits in discovery until it times out, six minutes later, looking like a
+    // PXE problem rather than a startup-order one.
+    //
+    // Cheap to wait, expensive not to.
+    waitForBridge();
+
     log(`Network ${PXE_NETWORK_NAME} created and active`);
   }
 
@@ -64,6 +75,32 @@ export function ensurePxeNetwork(): void {
   deleteNftablesRejectRules();
 }
 
+/**
+ * Block until the bridge exists AND carries the gateway address.
+ *
+ * Checking only that the device exists is not enough: it appears before the
+ * address is assigned, and dnsmasq needs the address to bind.
+ */
+export function waitForBridge(timeoutMs = 30_000): void {
+  const start = Date.now();
+  let last = "";
+  while (Date.now() - start < timeoutMs) {
+    const r = spawnSync("ip", ["-br", "addr", "show", PXE_BRIDGE], { encoding: "utf-8", stdio: "pipe" });
+    last = (r.stdout ?? "").trim();
+    if (r.status === 0 && last.includes(`${PXE_GATEWAY}/`)) {
+      log(`Bridge ${PXE_BRIDGE} ready with ${PXE_GATEWAY} after ${Date.now() - start}ms`);
+      return;
+    }
+    // Busy-wait briefly; this normally resolves in well under a second.
+    const until = Date.now() + 250;
+    while (Date.now() < until) { /* spin */ }
+  }
+  throw new Error(
+    `Bridge ${PXE_BRIDGE} did not come up with ${PXE_GATEWAY} within ${timeoutMs}ms ` +
+    `(last: ${last || "device absent"}). dnsmasq cannot bind, so PXE would silently not work.`,
+  );
+}
+
 /** Delete libvirt's nftables reject rules for our bridge so host→VM traffic works.
  *  Must be called after every VM start/restart — libvirt recreates them. */
 export function deleteNftablesRejectRules(): void {

bastion/tests/integration/helpers/pxe-vm.ts

@@ -28,7 +28,25 @@ export interface PxeVmConfig {
   vcpus: number;
   diskSize: number;     // GB
   network: string;      // libvirt network name
+  /**
+   * Serial console TCP port. Defaults to 4555 for backwards compatibility with
+   * the single-VM tests, but it MUST be unique per concurrently-running VM:
+   * qemu fails with "Failed to find an available port: Address already in use"
+   * and the domain never starts.
+   */
+  serialPort?: number;
   arch?: "x86_64" | "aarch64";
+  /**
+   * Extra NICs enumerated BEFORE the PXE NIC, on a network with no route to
+   * the bastion (defaults to libvirt's "default").
+   *
+   * Real multi-NIC boxes expose a class of bug a single-NIC VM cannot: an
+   * initramfs that picks "the first connected interface" grabs one of these
+   * instead of the NIC that PXE booted, and then cannot reach the bastion.
+   * Defaults to 0 (single NIC).
+   */
+  decoyNics?: number;
+  decoyNetwork?: string;
 }
 
 /** Create a blank UEFI VM that PXE boots from the network. */
@@ -61,6 +79,10 @@ export function createPxeVm(config: PxeVmConfig): void {
     `--memory=${config.memory}`,
     `--vcpus=${config.vcpus}`,
     `--disk=path=${diskPath},format=qcow2,bus=virtio`,
+    // Decoys first so they enumerate ahead of the PXE NIC. They are up and
+    // carry a lease, but have no route to the bastion.
+    ...Array.from({ length: config.decoyNics ?? 0 }, () =>
+      `--network=network=${config.decoyNetwork ?? "default"},model=virtio`),
     `--network=network=${config.network},model=virtio`,
     // UEFI firmware — required for PXE boot in modern mode
     `--boot=uefi,network,hd`,
@@ -71,8 +93,8 @@ export function createPxeVm(config: PxeVmConfig): void {
     // Graphics for debugging (VNC, connect with virt-viewer if needed)
     "--graphics=vnc,listen=127.0.0.1",
     // Serial console via TCP — allows exec without network/SSH
-    // Connect: socat - TCP:127.0.0.1:4555
-    "--serial=tcp,host=127.0.0.1:4555,mode=bind,protocol=telnet",
+    // Connect: socat - TCP:127.0.0.1:<serialPort>
+    `--serial=tcp,host=127.0.0.1:${config.serialPort ?? 4555},mode=bind,protocol=telnet`,
   ];
 
   if (arch === "aarch64") {
@@ -81,7 +103,7 @@ export function createPxeVm(config: PxeVmConfig): void {
 
   log(`Running: virt-install --name=${config.name} --boot=uefi,network ...`);
   run(virtInstallArgs.join(" "), { timeout: 30_000 });
-  log(`PXE VM ${config.name} created (serial: telnet 127.0.0.1 4555)`);
+  log(`PXE VM ${config.name} created (serial: telnet 127.0.0.1 ${config.serialPort ?? 4555})`);
 }
 
 /** Destroy a PXE VM and clean up its disk. */
@@ -95,12 +117,21 @@ export function destroyPxeVm(name: string): void {
 }
 
 /** Get the MAC address of a VM's first NIC. */
-export function getVmMac(name: string): string | null {
+export function getVmMac(name: string, network?: string): string | null {
   const result = virsh("domiflist", name);
   if (result.status !== 0) return null;
   // Output format: Interface  Type  Source  Model  MAC
-  const match = result.stdout.match(/([0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2})/i);
-  return match ? match[1].toLowerCase() : null;
+  // With decoy NICs present, match the line for the PXE network so we return
+  // the NIC that actually boots rather than whichever is listed first.
+  const lines = result.stdout.split("\n");
+  const candidates = network === undefined
+    ? lines
+    : lines.filter((l) => l.split(/\s+/).includes(network));
+  for (const line of candidates.length > 0 ? candidates : lines) {
+    const m = line.match(/([0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2})/i);
+    if (m) return m[1].toLowerCase();
+  }
+  return null;
 }
 
 /** Reboot a VM (force off + start). */

bastion/tests/integration/k3s-single-node.test.ts

@@ -161,7 +161,7 @@ EOF'
       CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
       curl -L --fail --silent "https://github.com/cilium/cilium-cli/releases/download/\${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz" | sudo tar xz -C /usr/local/bin
       DEFAULT_DEV=$(ip -4 route show default | awk '{print $5}' | head -1)
-      sudo KUBECONFIG=/etc/rancher/k3s/k3s.yaml cilium install --set kubeProxyReplacement=true --set ipam.mode=kubernetes --set devices=$DEFAULT_DEV --set nodePort.directRoutingDevice=$DEFAULT_DEV
+      sudo KUBECONFIG=/etc/rancher/k3s/k3s.yaml cilium install --set kubeProxyReplacement=true --set ipam.mode=cluster-pool --set ipam.operator.clusterPoolIPv4PodCIDRList='{10.42.0.0/16}' --set ipam.operator.clusterPoolIPv4MaskSize=24 --set devices=$DEFAULT_DEV --set nodePort.directRoutingDevice=$DEFAULT_DEV
     `.trim(), "cilium install", { keyPath: sshKeyPath, timeout: 120_000 });
 
     log("Waiting for Cilium to be ready...");

bastion/tests/integration/partition-layout.test.ts

@@ -0,0 +1,145 @@
+// Integration tests: the v2 disk layout, on real VMs.
+//
+// Unit tests prove the kickstart RENDERS the right layout. These prove anaconda
+// ACTUALLY PRODUCES it -- which is a different claim, and the one that matters:
+// the labmaster incident was caused by what landed on disk, not by what was
+// written into a template.
+//
+// Every VM gets a 1 TB sparse disk, because "fits a 1 TB disk with headroom" is
+// the constraint v2 was designed around. qcow2 is sparse, so a 1 TB disk costs
+// ~10-15 GB of real space.
+//
+// Prerequisites: libvirtd, OVMF, ipxe-bootimgs-x86, sudo, and a Fedora mirror.
+// Set FEDORA_MIRROR to a local mirror -- otherwise each install pulls the full
+// package set over the internet and the matrix takes hours.
+//
+// Run one scenario at a time:
+//   sudo -E npx vitest run -c tests/integration/vitest.config.ts -t "vanilla"
+
+import { describe, it, expect, beforeAll, afterAll } from "vitest";
+import { existsSync, readFileSync } from "node:fs";
+import { homedir } from "node:os";
+import { join } from "node:path";
+import { v2Layout } from "../../src/shared/src/types/layout.js";
+import type { Role } from "../../src/shared/src/types/state.js";
+import {
+  startTestBastion, provisionVm, readLvs, readSchemaTag, destroyVms,
+  type Harness, type ProvisionedVm,
+} from "./helpers/provision-harness.js";
+import { destroyPxeNetwork } from "./helpers/pxe-network.js";
+
+const HTTP_PORT = 8096;
+const SSH_USER = "lab";
+const DISK_GB = 1000; // the smallest disk we will ever buy
+
+function findSshKey(): { pubKey: string; keyPath: string } {
+  const homes = [homedir()];
+  const sudoUser = process.env["SUDO_USER"];
+  if (sudoUser) homes.push(join("/home", sudoUser));
+  for (const home of homes) {
+    for (const name of ["id_ed25519", "id_ecdsa", "id_rsa"]) {
+      const keyPath = join(home, ".ssh", name);
+      if (existsSync(keyPath) && existsSync(`${keyPath}.pub`)) {
+        return { pubKey: readFileSync(`${keyPath}.pub`, "utf-8").trim(), keyPath };
+      }
+    }
+  }
+  throw new Error("no SSH keypair found");
+}
+
+/** Assert the on-disk volumes match the shared v2 definition for this role. */
+function expectV2Layout(h: Harness, vm: ProvisionedVm, role: Role): void {
+  const actual = readLvs(h, vm);
+  const expected = v2Layout(role);
+
+  expect(Object.keys(actual).sort(), `${vm.name} volume set`).toEqual(
+    expected.map((lv) => lv.name).sort(),
+  );
+  for (const spec of expected) {
+    // LVM rounds to extent boundaries (4 MiB), so allow a small delta.
+    expect(actual[spec.name], `${vm.name}/${spec.name} size`).toBeGreaterThanOrEqual(spec.sizeMib - 4);
+    expect(actual[spec.name], `${vm.name}/${spec.name} size`).toBeLessThanOrEqual(spec.sizeMib + 4);
+  }
+  expect(readSchemaTag(h, vm), `${vm.name} schema tag`).toContain("lab-schema-v2");
+}
+
+describe("v2 partition layout on real hardware", () => {
+  let h: Harness;
+  const created: string[] = [];
+
+  beforeAll(async () => {
+    const { pubKey, keyPath } = findSshKey();
+    h = await startTestBastion({
+      httpPort: HTTP_PORT, sshUser: SSH_USER, sshPubKey: pubKey,
+      sshKeyPath: keyPath, label: "partition",
+    });
+  }, 20 * 60_000);
+
+  afterAll(async () => {
+    destroyVms(...created);
+    try { await h?.close(); } catch { /* already closed */ }
+    try { destroyPxeNetwork(); } catch { /* already gone */ }
+  }, 5 * 60_000);
+
+  // --- Topology 1: vanilla -------------------------------------------------
+  it("vanilla: base volumes only, no rancher, no longhorn", async () => {
+    const name = "v2-vanilla";
+    created.push(name);
+    const vm = await provisionVm(h, { name, role: "vanilla", schemaVersion: 2, diskGb: DISK_GB });
+    expectV2Layout(h, vm, "vanilla");
+
+    const lvs = readLvs(h, vm);
+    expect(lvs["rancher"]).toBeUndefined();
+    expect(lvs["longhorn"]).toBeUndefined();
+  }, 60 * 60_000);
+
+  // --- Topology 2: standalone cluster --------------------------------------
+  it("standalone-cluster: single node with its own control plane and storage", async () => {
+    const name = "v2-standalone";
+    created.push(name);
+    const vm = await provisionVm(h, { name, role: "standalone-cluster", schemaVersion: 2, diskGb: DISK_GB, memory: 6144 });
+    expectV2Layout(h, vm, "standalone-cluster");
+
+    const lvs = readLvs(h, vm);
+    // A cluster of one still gets a Longhorn volume, so the `longhorn`
+    // StorageClass exists and manifests stay portable across clusters.
+    expect(lvs["longhorn"]).toBeDefined();
+    expect(lvs["rancher"]).toBeDefined();
+  }, 60 * 60_000);
+
+  // --- Topology 3: 3x combo (control plane AND worker on every node) -------
+  it("k8s-combo x3: every node is control plane and storage", async () => {
+    const names = ["v2-combo-1", "v2-combo-2", "v2-combo-3"];
+    created.push(...names);
+    const vms: ProvisionedVm[] = [];
+    for (const name of names) {
+      vms.push(await provisionVm(h, { name, role: "k8s-combo", schemaVersion: 2, diskGb: DISK_GB }));
+    }
+    for (const vm of vms) {
+      expectV2Layout(h, vm, "k8s-combo");
+      const lvs = readLvs(h, vm);
+      expect(lvs["longhorn"], `${vm.name} holds storage`).toBeDefined();
+      expect(lvs["rancher"], `${vm.name} has an image store`).toBeDefined();
+    }
+  }, 180 * 60_000);
+
+  // --- Topology 4: 3 controllers + 2 workers -------------------------------
+  it("3x k8s-controller + 2x k8s-worker: storage only on the workers", async () => {
+    const ctrls = ["v2-ctrl-1", "v2-ctrl-2", "v2-ctrl-3"];
+    const workers = ["v2-wrk-1", "v2-wrk-2"];
+    created.push(...ctrls, ...workers);
+
+    for (const name of ctrls) {
+      const vm = await provisionVm(h, { name, role: "k8s-controller", schemaVersion: 2, diskGb: DISK_GB });
+      expectV2Layout(h, vm, "k8s-controller");
+      // The distinguishing property of a controller: image store, no storage.
+      expect(readLvs(h, vm)["longhorn"], `${name} must not hold replicas`).toBeUndefined();
+      expect(readLvs(h, vm)["rancher"]).toBeDefined();
+    }
+    for (const name of workers) {
+      const vm = await provisionVm(h, { name, role: "k8s-worker", schemaVersion: 2, diskGb: DISK_GB });
+      expectV2Layout(h, vm, "k8s-worker");
+      expect(readLvs(h, vm)["longhorn"], `${name} must hold replicas`).toBeDefined();
+    }
+  }, 300 * 60_000);
+});

bastion/tests/integration/reinstall.test.ts

@@ -0,0 +1,138 @@
+// Integration tests: reinstalling a machine must not destroy its data, and
+// must not silently change its disk layout.
+//
+// This is the risk the whole v2 change carries. Every machine in the fleet was
+// installed with v1, and `labctl provision reprovision` has to keep reproducing
+// v1 on those boxes forever. If a reinstall quietly switched them to v2 it
+// would recreate the volume group and take every Longhorn replica with it.
+//
+// Ported from test-reprovision.sh, which had the right idea (marker files in
+// each preserved volume) but only ever ran by hand and was never in CI.
+//
+// Run: sudo -E npx vitest run -c tests/integration/vitest.config.ts -t "reinstall"
+
+import { describe, it, expect, beforeAll, afterAll } from "vitest";
+import { existsSync, readFileSync } from "node:fs";
+import { homedir } from "node:os";
+import { join } from "node:path";
+import { v2Layout } from "../../src/shared/src/types/layout.js";
+import {
+  startTestBastion, provisionVm, reprovisionVm, readLvs, readSchemaTag,
+  writeMarkers, readMarkers, destroyVms, type Harness,
+} from "./helpers/provision-harness.js";
+import { destroyPxeNetwork } from "./helpers/pxe-network.js";
+
+const HTTP_PORT = 8095;
+const SSH_USER = "lab";
+const DISK_GB = 1000;
+
+/** Mount points a reinstall is supposed to preserve on a v1 worker. */
+const PRESERVED = ["/home", "/srv", "/var/lib/longhorn", "/var/lib/rancher"];
+
+function findSshKey(): { pubKey: string; keyPath: string } {
+  const homes = [homedir()];
+  const sudoUser = process.env["SUDO_USER"];
+  if (sudoUser) homes.push(join("/home", sudoUser));
+  for (const home of homes) {
+    for (const name of ["id_ed25519", "id_ecdsa", "id_rsa"]) {
+      const keyPath = join(home, ".ssh", name);
+      if (existsSync(keyPath) && existsSync(`${keyPath}.pub`)) {
+        return { pubKey: readFileSync(`${keyPath}.pub`, "utf-8").trim(), keyPath };
+      }
+    }
+  }
+  throw new Error("no SSH keypair found");
+}
+
+describe("reinstall preserves data and layout", () => {
+  let h: Harness;
+  const created: string[] = [];
+
+  beforeAll(async () => {
+    const { pubKey, keyPath } = findSshKey();
+    h = await startTestBastion({
+      httpPort: HTTP_PORT, sshUser: SSH_USER, sshPubKey: pubKey,
+      sshKeyPath: keyPath, label: "reinstall",
+    });
+  }, 20 * 60_000);
+
+  afterAll(async () => {
+    destroyVms(...created);
+    try { await h?.close(); } catch { /* closed */ }
+    try { destroyPxeNetwork(); } catch { /* gone */ }
+  }, 5 * 60_000);
+
+  it("v1 worker: data survives and the layout stays v1", async () => {
+    const name = "re-v1-worker";
+    created.push(name);
+
+    // Install at v1 -- i.e. exactly what every machine in the fleet is today.
+    let vm = await provisionVm(h, { name, role: "worker", diskGb: DISK_GB });
+
+    const before = readLvs(h, vm);
+    expect(before["root"], "v1 root is 33G, not v2's 32G").toBeGreaterThan(33000);
+    expect(before["rancher"], "v1 worker gets a 120G rancher LV").toBeGreaterThan(120000 - 8);
+    expect(readSchemaTag(h, vm), "a v1 disk must NOT carry the v2 tag").not.toContain("lab-schema-v2");
+
+    const token = writeMarkers(h, vm, PRESERVED);
+
+    // Reinstall. No schemaVersion passed -- the disk decides.
+    vm = await reprovisionVm(h, vm, "worker");
+
+    const markers = readMarkers(h, vm, PRESERVED);
+    for (const mount of PRESERVED) {
+      expect(markers[mount], `${mount} must survive a reinstall`).toBe(token);
+    }
+
+    // And the layout must be untouched.
+    const after = readLvs(h, vm);
+    expect(Object.keys(after).sort()).toEqual(Object.keys(before).sort());
+    expect(after["root"]).toBe(before["root"]);
+    expect(after["rancher"]).toBe(before["rancher"]);
+    expect(readSchemaTag(h, vm)).not.toContain("lab-schema-v2");
+  }, 120 * 60_000);
+
+  it("v2 worker: data survives and the layout stays v2", async () => {
+    const name = "re-v2-worker";
+    created.push(name);
+
+    let vm = await provisionVm(h, { name, role: "k8s-worker", schemaVersion: 2, diskGb: DISK_GB });
+    const before = readLvs(h, vm);
+    expect(readSchemaTag(h, vm)).toContain("lab-schema-v2");
+
+    const token = writeMarkers(h, vm, PRESERVED);
+    vm = await reprovisionVm(h, vm, "k8s-worker");
+
+    const markers = readMarkers(h, vm, PRESERVED);
+    for (const mount of PRESERVED) {
+      expect(markers[mount], `${mount} must survive a reinstall`).toBe(token);
+    }
+
+    const after = readLvs(h, vm);
+    const expected = Object.fromEntries(v2Layout("k8s-worker").map((lv) => [lv.name, lv.sizeMib]));
+    expect(Object.keys(after).sort()).toEqual(Object.keys(expected).sort());
+    expect(after["root"]).toBe(before["root"]);
+    // The tag must survive too, or the NEXT reinstall would fall back to v1.
+    expect(readSchemaTag(h, vm)).toContain("lab-schema-v2");
+  }, 120 * 60_000);
+
+  it("a v1 machine is NOT upgraded to v2 just because v2 was requested", async () => {
+    const name = "re-v1-sticky";
+    created.push(name);
+
+    let vm = await provisionVm(h, { name, role: "worker", diskGb: DISK_GB });
+    const before = readLvs(h, vm);
+    const token = writeMarkers(h, vm, ["/home", "/srv"]);
+
+    // Ask for v2 on a disk that is v1. %pre must ignore it: silently
+    // re-laying-out a live machine is how data gets destroyed.
+    vm = await reprovisionVm(h, vm, "worker", 2);
+
+    const after = readLvs(h, vm);
+    expect(after["root"], "root must stay at the v1 size").toBe(before["root"]);
+    expect(readSchemaTag(h, vm), "still not tagged v2").not.toContain("lab-schema-v2");
+    const markers = readMarkers(h, vm, ["/home", "/srv"]);
+    expect(markers["/home"]).toBe(token);
+    expect(markers["/srv"]).toBe(token);
+  }, 120 * 60_000);
+});

bastion/tests/integration/rolling-drill.test.ts

@@ -0,0 +1,623 @@
+// THE MIGRATION DRILL: roll three storage+etcd nodes through
+// evacuate -> wipe -> reinstall -> rejoin, one at a time, with data intact.
+//
+// WHAT IT IS PROVING
+// ------------------
+// The v1 -> v2 layout migration cannot preserve data through the wipe: %pre
+// only preserves when labvg already has the expected LV names, and a new layout
+// recreates the VG, taking the clearpart branch. So the data has to leave the
+// node first and come back afterwards. This asserts, on real machines, that it
+// does -- by sha256, either side of every step.
+//
+// WHY FOUR NODES FOR A THREE-NODE ROLL
+// ------------------------------------
+// With strict anti-affinity (replica-soft-anti-affinity=false, what the fleet
+// runs) a volume puts at most one replica per node, so a 3-replica volume on a
+// 3-node cluster CANNOT be evicted: the two survivors already hold a copy each
+// and cannot take the third. Eviction would wait for its full timeout and fail.
+// The real fleet has exactly this shape, plus a 100 GiB disk on worker2 with
+// ~49 GiB free -- so capacity blocks it independently of anti-affinity.
+//
+// The fix is the one already planned for production: bring the NEW box up
+// first, then evacuate an old one onto it. The fourth node here is an AGENT, so
+// etcd stays a 3-member cluster and every roll runs against production's real
+// quorum arithmetic (2 of 3, no margin).
+//
+// ORDER OF THE DANGEROUS PART
+// ---------------------------
+// The irreversible pair -- etcd member removal and node deletion -- happens as
+// late as possible, immediately before the wipe. Anything that fails before it
+// has destroyed nothing.
+//
+//   sudo -E npx vitest run -c tests/integration/vitest.config.ts -t "rolling drill"
+
+import { describe, it, expect, beforeAll, afterAll } from "vitest";
+import { existsSync, readFileSync } from "node:fs";
+import { homedir } from "node:os";
+import { join } from "node:path";
+import { K3sModule } from "../../src/modules/modules/k3s/src/k3s-module.js";
+import {
+  startTestBastion, provisionVms, reprovisionVm, readLvs, destroyVms,
+  type Harness, type ProvisionedVm,
+} from "./helpers/provision-harness.js";
+import { PINNED_K3S_VERSION } from "./helpers/k3s-cluster.js";
+import {
+  formCluster, joinAgent, installK3sNode, serverUrlOf, moduleCtx,
+  clusterSnapshot, printSnapshot, etcdMembers, etcdHealthy, allServersReady,
+  pickJoinTarget, kubectlOn, kubectlJson, nodeExec, removeEtcdMember, waitForNodeReady,
+  waitForApiPing, sleep, DRILL_TOKEN, k8sNodeName, forgetNodeName,
+} from "./helpers/k3s-cluster.js";
+import {
+  installLonghorn, waitForLonghornReady, createDataWorkload, waitForWorkload,
+  writeBlob, verifyBlob, volumeState, waitForVolumeHealthy, LONGHORN_NS,
+  backupTargetFromEnv, setBackupTarget, waitForBackupTarget, backupVolume, restoreFromBackup,
+  readBlobDigest,
+} from "./helpers/longhorn.js";
+import { destroyPxeNetwork } from "./helpers/pxe-network.js";
+
+const HTTP_PORT = 8098;
+const SSH_USER = "lab";
+const DISK_GB = 1000;
+
+const SERVERS = ["drill-s1", "drill-s2", "drill-s3"];
+const SPARE = "drill-spare";
+const ALL = [...SERVERS, SPARE];
+
+/** The replica-3 volume: the one whose rebuild the roll depends on. */
+const R3 = "drill-r3";
+/** The replica-1 volume: single copy, the sharpest risk in the real fleet. */
+const R1 = "drill-r1";
+
+function findSshKey(): { pubKey: string; keyPath: string } {
+  const homes = [homedir()];
+  const sudoUser = process.env["SUDO_USER"];
+  if (sudoUser) homes.push(join("/home", sudoUser));
+  for (const home of homes) {
+    for (const name of ["id_ed25519", "id_ecdsa", "id_rsa"]) {
+      const keyPath = join(home, ".ssh", name);
+      if (existsSync(keyPath) && existsSync(`${keyPath}.pub`)) {
+        return { pubKey: readFileSync(`${keyPath}.pub`, "utf-8").trim(), keyPath };
+      }
+    }
+  }
+  throw new Error("no SSH keypair found");
+}
+
+describe("rolling drill: migrate three nodes without losing data", () => {
+  let h: Harness;
+  let vms: ProvisionedVm[] = [];
+  let spare: ProvisionedVm;
+  let servers: ProvisionedVm[] = [];
+  let digestR3 = "";
+  let digestR1 = "";
+  /**
+   * S3 backup target, from DRILL_S3_* in the environment.
+   *
+   * Undefined means the backup/restore phase SKIPS rather than passing
+   * vacuously. A backup test that quietly does nothing is worse than none.
+   */
+  const backupTarget = backupTargetFromEnv();
+
+  /**
+   * Sticky abort. Once a step fails, the remaining steps SKIP rather than run
+   * against a cluster that is already wrong -- a failed roll leaves a node
+   * mid-migration, and rolling the next one on top of that turns one recoverable
+   * problem into an unrecoverable one.
+   */
+  let aborted: string | null = null;
+  /**
+   * `ctx` is vitest's test context. It is used to SKIP downstream steps rather
+   * than let them return quietly — a silent return marks the test PASSED, so a
+   * drill that died in provisioning reports "1 failed | 8 passed" and the eight
+   * steps that never ran look like evidence. They are not evidence.
+   */
+  const step = async (
+    ctx: { skip: (note?: string) => void },
+    name: string,
+    fn: () => Promise<void>,
+  ): Promise<void> => {
+    if (aborted !== null) {
+      console.log(`[drill] SKIPPED ${name} (aborted at: ${aborted})`);  // eslint-disable-line no-console
+      ctx.skip(`drill aborted at: ${aborted}`);
+      return;
+    }
+    const t0 = Date.now();
+    console.log(`[drill] ==> ${name}`);  // eslint-disable-line no-console
+    try {
+      await fn();
+      console.log(`[drill] <== ${name} (${Math.round((Date.now() - t0) / 1000)}s)`);  // eslint-disable-line no-console
+    } catch (e) {
+      aborted = name;
+      console.error(`[drill] !!! ${name} FAILED: ${e instanceof Error ? e.message : String(e)}`);  // eslint-disable-line no-console
+      throw e;
+    }
+  };
+
+  /**
+   * The runbook's replica-1 pre-flight, rehearsed rather than described.
+   *
+   * A single-replica volume rebuilds from the only copy that exists, so
+   * `evacuate` refuses it by default. The documented answer is to raise it to
+   * 3, wait for healthy, evacuate, and put it back afterwards. Doing that here
+   * means the drill exercises the procedure operators will actually follow
+   * instead of quietly bypassing it with the override flag.
+   */
+  const raiseReplicas = async (peer: ProvisionedVm, pvc: string, to: number): Promise<boolean> => {
+    const st = volumeState(h, peer, pvc);
+    kubectlOn(h, peer, `-n ${LONGHORN_NS} patch volumes.longhorn.io ${st.pv} ` +
+      `--type=merge -p '{"spec":{"numberOfReplicas":${to}}}'`);
+    if (to <= 1) return true;
+    try {
+      // BOUNDED, and allowed to fail.
+      //
+      // Raising a volume's replica count only converges if every target node
+      // will actually start a replica. A node that has just been wiped and
+      // rejoined may not: observed here, a replica sat at
+      // currentState=stopped / desireState=running / failedAt='' for forty
+      // minutes with no rebuild and no error, while the volume stayed
+      // degraded at 2-of-3 RW. Blocking the whole drill on that turns a
+      // Longhorn quirk into a total failure and proves nothing.
+      await waitForVolumeHealthy(h, peer, { pvc, expectReplicas: to, timeoutMs: 10 * 60_000 });
+      return true;
+    } catch (e) {
+      console.warn(  // eslint-disable-line no-console
+        `[drill] ${pvc} did not reach ${to} healthy replicas in 10 min: ` +
+        `${e instanceof Error ? e.message : String(e)}`);
+      return false;
+    }
+  };
+
+  /** Is this volume's only replica on the node we are about to evacuate? */
+  const replicaOn = (peer: ProvisionedVm, pvc: string, node: string): boolean =>
+    volumeState(h, peer, pvc).replicaNodes.includes(node);
+
+  /** Read cluster state from a node that is NOT the one being worked on. */
+  const survivor = (exclude?: ProvisionedVm): ProvisionedVm =>
+    exclude ? pickJoinTarget(h, servers, exclude) : servers[0]!;
+
+  beforeAll(async () => {
+    const { pubKey, keyPath } = findSshKey();
+    destroyVms(...ALL);
+    h = await startTestBastion({
+      httpPort: HTTP_PORT, sshUser: SSH_USER, sshPubKey: pubKey, sshKeyPath: keyPath,
+      label: "rolling-drill",
+      autoK3s: false,   // we sequence cluster formation ourselves
+    });
+  }, 20 * 60_000);
+
+  afterAll(async () => {
+    // Diagnostics BEFORE teardown: these VMs are the only record of what
+    // happened and they are about to be deleted.
+    try {
+      const s = servers[0];
+      if (s) {
+        printSnapshot("FINAL", clusterSnapshot(h, s));
+        for (const pvc of [R3, R1]) {
+          try {
+            const st = volumeState(h, s, pvc);
+            console.log(`[drill] ${pvc}: ${st.state}/${st.robustness} on ${st.replicaNodes.join(",")}`);  // eslint-disable-line no-console
+          } catch { /* volume may be gone */ }
+        }
+        console.log(`[drill] digests: ${R3}=${digestR3.slice(0, 16)} ${R1}=${digestR1.slice(0, 16)}`);  // eslint-disable-line no-console
+      }
+    } catch { /* cluster may be unreachable */ }
+
+    // Best-effort: clear any eviction we may have left pending. Harmless on a
+    // VM about to be destroyed, but this is the same call the runbook makes on
+    // a real node, so exercising it here is free coverage.
+    for (const vm of servers) {
+      try { await new K3sModule().admit(moduleCtx(h, vm, "k8s-combo")); } catch { /* ignore */ }
+    }
+
+    if (process.env["LAB_DRILL_KEEP_VMS"] !== "1") {
+      destroyVms(...ALL);
+      try { destroyPxeNetwork(); } catch { /* shared */ }
+    } else {
+      console.log("[drill] LAB_DRILL_KEEP_VMS=1 — leaving VMs up for inspection");  // eslint-disable-line no-console
+    }
+    await h?.close();
+  }, 20 * 60_000);
+
+  it("provisions four v2 nodes", async (ctx) => {
+    await step(ctx, "provision", async () => {
+      vms = await provisionVms(h, [
+        ...SERVERS.map((name) => ({
+          name, role: "k8s-combo" as const, schemaVersion: 2 as const,
+          diskGb: DISK_GB, memory: 6144, vcpus: 4,
+        })),
+        { name: SPARE, role: "k8s-worker" as const, schemaVersion: 2 as const,
+          diskGb: DISK_GB, memory: 4096, vcpus: 2 },
+      ]);
+      servers = vms.filter((v) => SERVERS.includes(v.name));
+      spare = vms.find((v) => v.name === SPARE)!;
+      expect(servers).toHaveLength(3);
+
+      // Every node must actually have the longhorn LV, or nothing later works.
+      for (const vm of vms) expect(Object.keys(readLvs(h, vm))).toContain("longhorn");
+    });
+  }, 90 * 60_000);
+
+  it("forms one cluster with three etcd members plus a storage agent", async (ctx) => {
+    await step(ctx, "form cluster", async () => {
+      await formCluster(h, servers, { role: "k8s-combo" });
+      await joinAgent(h, spare, { server: servers[0]!, role: "k8s-worker" });
+
+      const snap = clusterSnapshot(h, servers[0]!);
+      printSnapshot("formed", snap);
+      expect(snap.nodes).toHaveLength(4);
+      expect(snap.readyCount).toBe(4);
+      // The spare is an agent on purpose: etcd stays at 3 so each roll runs
+      // against production's real quorum arithmetic.
+      expect(etcdMembers(h, servers[0]!)).toHaveLength(3);
+      expect(etcdHealthy(h, servers[0]!)).toBe(true);
+
+      // Every node must run the PINNED version, not whatever get.k3s.io calls
+      // latest today. Setting k3sVersion is not the same as it taking effect:
+      // toOpContext used to drop the field silently, so the pin was ignored and
+      // nodes came up two minor versions ahead of the fleet. Assert the
+      // outcome, on the machines.
+      const versions = kubectlJson<{ items: {
+        metadata: { name: string };
+        status: { nodeInfo: { kubeletVersion: string } };
+      }[] }>(h, servers[0]!, "get nodes");
+      expect(versions.items).toHaveLength(4);
+      for (const n of versions.items) {
+        expect(n.status.nodeInfo.kubeletVersion,
+          `${n.metadata.name} must run the pinned ${PINNED_K3S_VERSION}`)
+          .toBe(PINNED_K3S_VERSION);
+      }
+      console.log(`[drill] all 4 nodes on pinned ${PINNED_K3S_VERSION}`);  // eslint-disable-line no-console
+    });
+  }, 120 * 60_000);
+
+  it("installs Longhorn with a schedulable disk on every node", async (ctx) => {
+    await step(ctx, "install longhorn", async () => {
+      await installLonghorn(h, servers[0]!, {
+        defaultReplicas: 3,
+        ...(backupTarget === undefined ? {} : { backup: backupTarget }),
+      });
+
+      // Give Longhorn time to register the nodes, then run the PRODUCTION
+      // repair path on each one.
+      //
+      // configureLonghornDisk runs during configure(), which happens BEFORE
+      // Longhorn is installed -- there is no node record to repair yet, so it
+      // only sets the annotation and returns. If Longhorn then misses that
+      // annotation for any reason, nothing ever retries, and the node joins,
+      // goes Ready, and silently hosts no replicas. Observed: one node in four
+      // came up with no disk while its label and annotation were byte-identical
+      // to its siblings'.
+      //
+      // admit() is exactly what the runbook tells an operator to run, and it
+      // exercises the verify-and-repair branch that is otherwise only reachable
+      // after a wipe-and-rejoin.
+      await sleep(60_000);
+      for (const vm of vms) {
+        const role = vm.name === SPARE ? "k8s-worker" : "k8s-combo";
+        const res = await new K3sModule().admit(moduleCtx(h, vm, role));
+        for (const line of res.output) console.log(`  [${vm.name}] ${line}`);  // eslint-disable-line no-console
+      }
+
+      // All four VMs, including the agent: the spare having a schedulable disk
+      // is the whole reason eviction can complete at all.
+      await waitForLonghornReady(h, servers[0]!, vms);
+
+      // Read it back rather than trusting the wait: every node must have a
+      // disk, or replicas silently never land there.
+      //
+      // -o json and parse, NOT jsonpath. A jsonpath containing {"\n"} has to
+      // survive a TS template literal AND a shell, and getting either escape
+      // wrong yields `unterminated quoted string` at runtime — which is how
+      // this assertion failed the first time it ran.
+      const lhNodes = kubectlJson<{ items: {
+        metadata: { name: string };
+        spec: { disks?: Record<string, unknown> };
+      }[] }>(h, servers[0]!, `-n ${LONGHORN_NS} get nodes.longhorn.io`);
+
+      for (const vm of vms) {
+        const name = k8sNodeName(h, vm);
+        const rec = lhNodes.items.find((n) => n.metadata.name === name);
+        expect(rec, `${name} has no Longhorn node record`).toBeDefined();
+        const disks = Object.keys(rec?.spec.disks ?? {});
+        expect(disks.length, `${name} has no Longhorn disk — the default-disks-config ` +
+          `annotation never parsed, so this node would silently host no replicas`)
+          .toBeGreaterThan(0);
+      }
+    });
+  }, 40 * 60_000);
+
+  it("writes known data and proves it survives a detach/attach", async (ctx) => {
+    await step(ctx, "seed data", async () => {
+      const s = servers[0]!;
+      createDataWorkload(h, s, { name: R3, sizeGi: 2, replicas: 3 });
+      createDataWorkload(h, s, { name: R1, sizeGi: 1, replicas: 1 });
+      await waitForWorkload(h, s, R3);
+      await waitForWorkload(h, s, R1);
+
+      digestR3 = await writeBlob(h, s, { name: R3, mib: 512 });
+      digestR1 = await writeBlob(h, s, { name: R1, mib: 128 });
+
+      // Record the digest only AFTER it has survived one restart. Otherwise
+      // the baseline could be a page-cache artefact and every later
+      // comparison would be against a value that was never really on disk.
+      await verifyBlob(h, s, { name: R3, expectDigest: digestR3 });
+      await verifyBlob(h, s, { name: R1, expectDigest: digestR1 });
+
+      const st = volumeState(h, s, R3);
+      expect(st.robustness).toBe("healthy");
+      expect(new Set(st.replicaNodes).size).toBe(3);
+    });
+  }, 60 * 60_000);
+
+  it("backs up a volume to S3 and restores it byte-identical", async (ctx) => {
+    await step(ctx, "backup and restore", async () => {
+      const s = servers[0]!;
+
+      if (backupTarget === undefined) {
+        // Deliberately loud. A backup phase that silently passes because no
+        // target was configured is exactly the kind of green tick that makes
+        // people believe they have backups when they do not.
+        throw new Error(
+          "no backup target configured (DRILL_S3_TARGET / _ENDPOINT / _ACCESS_KEY / _SECRET_KEY). " +
+          "The backup/restore phase cannot prove anything without one.");
+      }
+
+      // Set it on the CR. defaultSettings.backupTarget does nothing in 1.8 --
+      // see setBackupTarget for why.
+      await setBackupTarget(h, s, backupTarget);
+
+      // `available: true` is Longhorn saying it reached the endpoint AND
+      // authenticated. Backing up before checking gives a failure that reads
+      // like a volume problem.
+      await waitForBackupTarget(h, s);
+
+      const { backup } = await backupVolume(h, s, R3);
+
+      // Restore into a SEPARATE volume. The original is untouched, so a failed
+      // restore cannot damage the source -- which is the property that makes
+      // this safe to rehearse at all.
+      await restoreFromBackup(h, s, { backup, name: "drill-restored", sizeGi: 2 });
+      await waitForWorkload(h, s, "drill-restored");
+
+      // The whole point: the bytes that came back are the bytes that went in.
+      // Compared against the digest recorded BEFORE any of this, which was
+      // itself only trusted after surviving a detach/attach.
+      const restored = readBlobDigest(h, s, "drill-restored");
+      expect(restored, "restored volume digest must equal the original").toBe(digestR3);
+      console.log(`[drill] restored digest ${restored.slice(0, 16)}... matches original`);  // eslint-disable-line no-console
+
+      // And the restored volume is a real, independent volume.
+      const st = volumeState(h, s, "drill-restored");
+      expect(st.pv).not.toBe(volumeState(h, s, R3).pv);
+      expect(st.state).toBe("attached");
+    });
+  }, 45 * 60_000);
+
+  it("REFUSES to evacuate when there is nowhere for replicas to go", async (ctx) => {
+    await step(ctx, "negative pre-check", async () => {
+      const s = servers[0]!;
+
+      // Must be a node that actually HOLDS an r3 replica, and not the one we
+      // read cluster state from. r3 has 3 replicas across 4 nodes, so one node
+      // has none -- picking blindly makes this test pass or fail on placement
+      // luck rather than on the behaviour it is meant to pin.
+      // replicaNodes are KUBERNETES node names (smoke-a.pxe-test.local), not
+      // libvirt domain names, so resolve before comparing.
+      const r3Nodes = volumeState(h, s, R3).replicaNodes;
+      const target = servers.find(
+        (v) => v.name !== s.name && r3Nodes.includes(k8sNodeName(h, v)));
+      expect(target, `no server other than ${s.name} holds an ${R3} replica (${r3Nodes.join(",")})`)
+        .toBeDefined();
+
+      // Take the spare out of service: now 3 schedulable nodes for a 3-replica
+      // volume, which is exactly the real fleet's shape.
+      kubectlOn(h, s, `-n ${LONGHORN_NS} patch nodes.longhorn.io ${k8sNodeName(h, spare)} ` +
+        `--type=merge -p '{"spec":{"allowScheduling":false}}'`);
+      await sleep(10_000);
+
+      const t0 = Date.now();
+      const res = await new K3sModule().evacuate(moduleCtx(h, target!, "k8s-combo"));
+      const elapsed = (Date.now() - t0) / 1000;
+
+      expect(res.success, "evacuation should have been refused").toBe(false);
+      // Fast is the whole point: the old behaviour was a 60-minute wait.
+      expect(elapsed, `refused in ${elapsed}s`).toBeLessThan(300);
+      expect(res.errors.join(" ")).toMatch(/cannot complete|cannot be placed|wants 3 replicas/);
+
+      // And it must not have touched anything on the way to saying no.
+      const lhNode = kubectlOn(h, s,
+        `-n ${LONGHORN_NS} get nodes.longhorn.io ${k8sNodeName(h, target!)} ` +
+        `-o jsonpath='{.spec.evictionRequested}'`).trim();
+      expect(lhNode, "refusal must not leave an eviction pending").not.toBe("true");
+
+      // Put the spare back and un-cordon whatever the attempt cordoned.
+      kubectlOn(h, s, `-n ${LONGHORN_NS} patch nodes.longhorn.io ${k8sNodeName(h, spare)} ` +
+        `--type=merge -p '{"spec":{"allowScheduling":true}}'`);
+      await new K3sModule().admit(moduleCtx(h, target!, "k8s-combo"));
+      await sleep(15_000);
+    });
+  }, 30 * 60_000);
+
+  // The roll itself. Serial by construction -- each `it` depends on the last.
+  for (const name of SERVERS) {
+    it(`rolls ${name}: evacuate, wipe, reinstall, rejoin, verify`, async (ctx) => {
+      await step(ctx, `roll ${name}`, async () => {
+        const target = vms.find((v) => v.name === name)!;
+        const peer = survivor(target);
+        // Kubernetes and Longhorn both know this machine by its FQDN, not by
+        // the libvirt domain name used everywhere else in this file.
+        const nodeName = k8sNodeName(h, target);
+
+        // --- GATE: never start a roll on a cluster that is already degraded.
+        const pre = clusterSnapshot(h, peer);
+        printSnapshot(`before rolling ${name}`, pre);
+        expect(pre.etcdMembers, "etcd membership before roll").toBe(3);
+        expect(pre.etcdHealthy, "etcd health before roll").toBe(true);
+        expect(pre.readyCount, "all nodes Ready before roll").toBe(4);
+
+        // --- Replica-1 pre-flight, exactly as the runbook prescribes.
+        //
+        // If drill-r1's single replica is on this node, evacuate MUST refuse:
+        // moving it means rebuilding from the only copy in existence, and this
+        // fleet has no backup target. Assert that refusal, then apply the
+        // documented mitigation rather than overriding the guard.
+        const r1Here = replicaOn(peer, R1, nodeName);
+        let r1Raised = false;
+        if (r1Here) {
+          const refused = await new K3sModule().evacuate(moduleCtx(h, target, "k8s-combo"));
+          expect(refused.success, "single-replica volume should have blocked evacuation").toBe(false);
+          expect(refused.errors.join(" ")).toMatch(/single replica/);
+          await new K3sModule().admit(moduleCtx(h, target, "k8s-combo"));
+          r1Raised = await raiseReplicas(peer, R1, 3);
+        }
+
+        // --- Evacuate (cordon -> check -> evict+wait -> drain -> verify).
+        //
+        // If raising drill-r1 did not converge, fall back to the documented
+        // override rather than stalling. The guard itself is already proven by
+        // the refusal above and by the negative pre-check; what this roll is
+        // here to prove is the migration.
+        const evacConfig = r1Here && !r1Raised ? { evictionAllowSingleReplica: true } : {};
+        if (r1Here && !r1Raised) {
+          console.warn(  // eslint-disable-line no-console
+            "[drill] proceeding with --allow-single-replica because the raise did not converge");
+        }
+        const evac = await new K3sModule().evacuate(moduleCtx(h, target, "k8s-combo", evacConfig));
+        for (const line of evac.output) console.log(`  [${name}] ${line}`);  // eslint-disable-line no-console
+        expect(evac.success, `evacuate ${name}: ${evac.errors.join("; ")}`).toBe(true);
+
+        // Replicas really gone, and the r3 volume really rebuilt elsewhere.
+        const afterEvac = await waitForVolumeHealthy(h, peer, { pvc: R3, expectReplicas: 3 });
+        expect(afterEvac.replicaNodes).not.toContain(nodeName);
+
+        // --- Data intact after evacuation, BEFORE anything is destroyed.
+        await verifyBlob(h, peer, { name: R3, expectDigest: digestR3 });
+        await verifyBlob(h, peer, { name: R1, expectDigest: digestR1 });
+
+        // --- The irreversible part starts here and not before.
+        nodeExec(h, target, "systemctl stop k3s || true");
+        const removed = removeEtcdMember(h, peer, nodeName);
+        expect(removed, `etcd member ${nodeName} should have existed`).toBe(true);
+        kubectlOn(h, peer, `delete node ${nodeName} --wait=true --timeout=5m`, 6 * 60_000);
+
+        // Quorum asserted POSITIVELY: 2 of 3 is survivable, silence is not proof.
+        const mid = clusterSnapshot(h, peer);
+        printSnapshot(`${name} removed`, mid);
+        expect(mid.etcdMembers, "etcd must be down to exactly 2 members").toBe(2);
+        expect(mid.etcdHealthy, "surviving etcd must still be healthy").toBe(true);
+        const ready = allServersReady(h, servers.filter((v) => v.name !== name));
+        expect(ready.ok, `survivors /readyz: ${ready.detail}`).toBe(true);
+
+        // --- Wipe and reinstall. The disk decides the schema, not the request.
+        //
+        // reprovisionVm returns a NEW record: the machine can come back on a
+        // different DHCP address. Every later reference must use that record,
+        // and the shared arrays have to be updated too or `survivor()` will
+        // hand out a stale IP on the next roll.
+        const rolled = await reprovisionVm(h, target, "k8s-combo");
+        vms = vms.map((v) => (v.name === name ? rolled : v));
+        servers = servers.map((v) => (v.name === name ? rolled : v));
+        // A reinstalled machine re-registers; do not trust the cached name.
+        forgetNodeName(rolled);
+
+        // v2 reinstall PRESERVES /var/lib/longhorn, so without this the next
+        // step would measure replica re-adoption, not a real rebuild. The
+        // actual v1->v2 migration recreates the VG and has no such leftovers.
+        //
+        // Clear the CONTENTS, never the directory itself. Longhorn's replica
+        // process does a non-recursive `mkdir /host/var/lib/longhorn/replicas/
+        // <name>`, which fails outright when the parent is missing:
+        //
+        //   level=fatal msg="Error running start replica command"
+        //     error="mkdir /host/var/lib/longhorn/replicas/pvc-...: ..."
+        //
+        // Every replica scheduled onto the node then dies instantly and
+        // permanently, while the node looks entirely healthy: longhorn-disk.cfg
+        // survives the wipe, so the disk keeps its UUID and reports Ready and
+        // Schedulable. The next node's eviction can never drain onto it and
+        // stalls for its full hour. Two earlier drill runs died exactly here.
+        nodeExec(h, rolled,
+          "rm -rf /var/lib/longhorn/replicas/* /var/lib/longhorn/*.tmp 2>/dev/null; " +
+          "mkdir -p /var/lib/longhorn/replicas && ls -d /var/lib/longhorn/replicas");
+
+        // --- Rejoin against a SURVIVOR, not against itself.
+        await waitForApiPing(peer);
+        await installK3sNode(h, rolled, {
+          role: "k8s-combo", token: DRILL_TOKEN, serverUrl: serverUrlOf(peer),
+        });
+        await waitForNodeReady(h, peer, rolled);
+
+        // Exactly 3. Four would mean the old member survived the rejoin, which
+        // silently raises quorum from 2-of-3 to 3-of-4 and removes all margin.
+        const post = clusterSnapshot(h, peer);
+        printSnapshot(`${name} rejoined`, post);
+        expect(post.etcdMembers, "stale etcd member left behind").toBe(3);
+        expect(post.etcdHealthy).toBe(true);
+        expect(post.readyCount).toBe(4);
+
+        // --- Back into service, then wait for the replica to rebuild onto it.
+        const admit = await new K3sModule().admit(moduleCtx(h, rolled, "k8s-combo"));
+        for (const line of admit.output) console.log(`  [${name}] ${line}`);  // eslint-disable-line no-console
+        expect(admit.success, `admit ${name}: ${admit.errors.join("; ")}`).toBe(true);
+
+        const healthy = await waitForVolumeHealthy(h, peer, { pvc: R3, expectReplicas: 3 });
+        expect(new Set(healthy.replicaNodes).size).toBe(3);
+
+        // --- And the data is still the data.
+        await verifyBlob(h, peer, { name: R3, expectDigest: digestR3 });
+        await verifyBlob(h, peer, { name: R1, expectDigest: digestR1 });
+
+        // Put drill-r1 back to a single replica, so the next roll faces the
+        // same hazard rather than a progressively safer cluster.
+        if (r1Here && r1Raised) await raiseReplicas(peer, R1, 1);
+
+        // --- Does the rolled node actually take replicas BACK?
+        //
+        // Roll 1 originally "passed" with drill-r3 healthy on three nodes that
+        // did not include the node just rolled -- the assertion counted
+        // distinct nodes without ever checking that the reinstalled one was
+        // among them. A node that rejoins, reports Ready and Schedulable, and
+        // then hosts nothing is the exact failure this whole exercise exists to
+        // catch, so record it explicitly.
+        // RUNNING, not merely assigned. The first version of this check used
+        // replicaNodes (which comes from spec.nodeID) and happily passed on a
+        // node holding two replicas that were both state=stopped -- exactly the
+        // condition that makes the NEXT eviction impossible, because Longhorn
+        // keeps scheduling replacements onto a node that cannot start them.
+        const running = kubectlJson<{ items: {
+          spec: { nodeID?: string };
+          status?: { currentState?: string };
+        }[] }>(h, peer, `-n ${LONGHORN_NS} get replicas.longhorn.io`)
+          .items.filter((r) => r.spec.nodeID === nodeName && r.status?.currentState === "running");
+        console.log(  // eslint-disable-line no-console
+          `[drill] after rolling ${name}: ${running.length} RUNNING replica(s) on the node`);
+        expect(running.length,
+          `${name} rejoined but hosts no RUNNING replica — it would look healthy and ` +
+          `silently take no storage, and the next eviction could never drain onto it`)
+          .toBeGreaterThan(0);
+      });
+    }, 240 * 60_000);
+  }
+
+  it("ends with a whole cluster and the original bytes", async (ctx) => {
+    await step(ctx, "final verification", async () => {
+      const s = servers[0]!;
+      const snap = clusterSnapshot(h, s);
+      printSnapshot("end of drill", snap);
+
+      expect(snap.nodes).toHaveLength(4);
+      expect(snap.readyCount).toBe(4);
+      expect(snap.etcdMembers).toBe(3);
+      expect(snap.etcdHealthy).toBe(true);
+
+      const r3 = volumeState(h, s, R3);
+      expect(r3.robustness).toBe("healthy");
+      expect(new Set(r3.replicaNodes).size).toBe(3);
+
+      // One last restart-and-check, so the final claim is about bytes read
+      // back through a fresh attach rather than a cached mount.
+      await verifyBlob(h, s, { name: R3, expectDigest: digestR3 });
+      await verifyBlob(h, s, { name: R1, expectDigest: digestR1 });
+    });
+  }, 60 * 60_000);
+});

bastion/tests/integration/vyos-provision.test.ts

@@ -0,0 +1,387 @@
+// Integration test: full VyOS unattended provisioning flow.
+//
+// Validates the VyOS install path end-to-end, at the same depth as the Fedora
+// pxe-provision test:
+//   1. Bastion (HTTP + dnsmasq) on the isolated libvirt PXE network
+//   2. Blank UEFI VM PXE boots -> Fedora-based discovery (OS-neutral)
+//   3. Queue os=vyos-rolling -> live boot + live-config hook + pty driver
+//   4. Fresh-install asserts: installed.ip, streamed logs, applied config,
+//      /config/lab-provisioned, boot-order handling
+//   5. REINSTALL round: previous config + /config data carried forward
+//      ("reinstall without losing data", VyOS-flavored)
+//   6. freshConfig round: bastion-generated config wins, /config data kept
+//
+// Prerequisites: libvirtd, OVMF, ipxe-bootimgs-x86, sudo, internet
+// (first run downloads the ~600MB VyOS nightly ISO; artifacts are cached).
+// Run: sudo pnpm run test:integration:vyos
+
+import { describe, it, expect, beforeAll, afterAll } from "vitest";
+import { readFileSync, existsSync, mkdirSync, rmSync, copyFileSync, symlinkSync, writeFileSync } from "node:fs";
+import { execSync } from "node:child_process";
+import { join } from "node:path";
+import { homedir, tmpdir } from "node:os";
+import { log, waitForSsh } from "./helpers/libvirt.js";
+import { ensurePxeNetwork, destroyPxeNetwork, deleteNftablesRejectRules, PXE_NETWORK_NAME, PXE_GATEWAY, PXE_SUBNET } from "./helpers/pxe-network.js";
+import { createPxeVm, destroyPxeVm, getVmMac, rebootPxeVm } from "./helpers/pxe-vm.js";
+import { sshExec } from "./helpers/ssh.js";
+
+const VM_NAME = "lab-vyos-test";
+const VM_MEMORY = 4096;
+const VM_VCPUS = 4;
+const VM_DISK_GB = 10;       // VyOS image install needs ~2GB minimum
+const HTTP_PORT = 8099;
+const SSH_USER = "vyos";     // the only VyOS login user
+const BASTION_IP = PXE_GATEWAY;
+const DHCP_RANGE_START = `${PXE_SUBNET}.100`;
+const DHCP_RANGE_END = `${PXE_SUBNET}.200`;
+
+const DISCOVERY_TIMEOUT_MS = 5 * 60_000;
+const INSTALL_TIMEOUT_MS = 15 * 60_000;  // squashfs fetch + copy; much faster than Anaconda
+const SSH_TIMEOUT_MS = 8 * 60_000;
+
+const HOSTNAME_R1 = "vyos-r1";
+const HOSTNAME_R2 = "vyos-r2";
+const HOSTNAME_R3 = "vyos-r3";
+
+function findSshKey(): { pubKey: string; keyPath: string } {
+  const homes = [homedir()];
+  const sudoUser = process.env["SUDO_USER"];
+  if (sudoUser) homes.push(join("/home", sudoUser));
+  if (process.env["SSH_KEY_PATH"]) {
+    const keyPath = process.env["SSH_KEY_PATH"];
+    const pubPath = `${keyPath}.pub`;
+    if (existsSync(keyPath) && existsSync(pubPath)) {
+      return { pubKey: readFileSync(pubPath, "utf-8").trim(), keyPath };
+    }
+  }
+  for (const home of homes) {
+    for (const name of ["id_ed25519", "id_ecdsa", "id_rsa"]) {
+      const keyPath = join(home, ".ssh", name);
+      const pubPath = `${keyPath}.pub`;
+      if (existsSync(keyPath) && existsSync(pubPath)) {
+        return { pubKey: readFileSync(pubPath, "utf-8").trim(), keyPath };
+      }
+    }
+  }
+  throw new Error("No SSH key found — set SSH_KEY_PATH or ensure keys exist in ~/.ssh/");
+}
+
+function sleep(ms: number): Promise<void> {
+  return new Promise((r) => setTimeout(r, ms));
+}
+
+async function pollApi<T>(
+  url: string,
+  check: (data: T) => boolean,
+  timeoutMs: number,
+  intervalMs = 5000,
+): Promise<T> {
+  const start = Date.now();
+  while (Date.now() - start < timeoutMs) {
+    try {
+      const res = await fetch(url);
+      if (res.ok) {
+        const data = (await res.json()) as T;
+        if (check(data)) return data;
+      }
+    } catch { /* not ready yet */ }
+    await sleep(intervalMs);
+  }
+  throw new Error(`Timeout after ${timeoutMs}ms polling ${url}`);
+}
+
+type LogsResponse = {
+  status: string;
+  progress: string;
+  progress_detail?: string;
+  ip?: string;
+  log_total?: number;
+  log_lines?: Array<{ line: string }>;
+};
+
+/** Queue a VyOS install, reboot the VM into PXE, wait for completion + SSH. */
+async function installRound(opts: {
+  mac: string;
+  hostname: string;
+  freshConfig?: boolean;
+}): Promise<string> {
+  const body = {
+    mac: opts.mac,
+    hostname: opts.hostname,
+    disk: "/dev/vda",
+    role: "vanilla",
+    os: "vyos-rolling",
+    vyos: {
+      mgmtInterface: "eth0",
+      mgmtAddress: "dhcp",
+      hwIds: { eth0: opts.mac },
+      ...(opts.freshConfig ? { freshConfig: true } : {}),
+    },
+  };
+  const res = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/install`, {
+    method: "POST",
+    headers: { "Content-Type": "application/json" },
+    body: JSON.stringify(body),
+  });
+  log(`Install queued (${opts.hostname}): ${JSON.stringify(await res.json())}`);
+
+  await sleep(5_000);
+  rebootPxeVm(VM_NAME);
+  await sleep(3_000);
+  deleteNftablesRejectRules();
+
+  const finalState = await pollApi<LogsResponse>(
+    `http://${BASTION_IP}:${HTTP_PORT}/api/logs/${encodeURIComponent(opts.mac)}`,
+    (data) => data.status === "installed" || data.progress === "error",
+    INSTALL_TIMEOUT_MS,
+    10_000,
+  );
+  if (finalState.progress === "error") {
+    log(`INSTALL FAILED: ${JSON.stringify(finalState.progress_detail ?? finalState, null, 2)}`);
+    throw new Error(`VyOS install failed for ${opts.hostname}`);
+  }
+  const ip = finalState.ip ?? "";
+  log(`Install complete (${opts.hostname}). IP: ${ip}`);
+
+  // The driver force-reboots; the VM PXE boots, dispatch says installed ->
+  // localboot exit -> GRUB -> VyOS. nftables reject rules do not reappear
+  // (guest reboot, not a libvirt restart), but clearing is harmless.
+  deleteNftablesRejectRules();
+  await waitForSsh(ip, SSH_USER, SSH_TIMEOUT_MS, sshKeyPathGlobal);
+  return ip;
+}
+
+let sshKeyPathGlobal = "";
+
+describe("VyOS provisioning", () => {
+  let bastionApp: { close: () => Promise<void> };
+  let testDir: string;
+  let vmMac: string;
+  let vmIp: string;
+
+  beforeAll(async () => {
+    const { pubKey, keyPath } = findSshKey();
+    sshKeyPathGlobal = keyPath;
+
+    log("Setting up PXE test network...");
+    ensurePxeNetwork();
+
+    testDir = join(tmpdir(), `lab-vyos-test-${Date.now()}`);
+    mkdirSync(join(testDir, "tftp"), { recursive: true });
+    mkdirSync(join(testDir, "http"), { recursive: true });
+    mkdirSync(join(testDir, "logs"), { recursive: true });
+
+    log("Starting bastion...");
+    const { createApp } = await import("../../src/bastion/src/server.js");
+    const { loadConfig } = await import("../../src/bastion/src/config.js");
+    const { generateDnsmasqConf, startDnsmasq } = await import("../../src/bastion/src/services/dnsmasq.js");
+    const { generateDiscoverKickstart } = await import("../../src/bastion/src/services/kickstart-generator.js");
+    const { renderBootIpxe } = await import("../../src/bastion/src/templates/boot.ipxe.js");
+    const { prepareVyosArtifacts } = await import("../../src/bastion/src/main.js");
+
+    const config = loadConfig({
+      bastionDir: testDir,
+      httpPort: HTTP_PORT,
+      iface: "virbr-pxe",
+      serverIp: BASTION_IP,
+      network: `${PXE_SUBNET}.0`,
+      gateway: BASTION_IP,
+      dhcpMode: "full",
+      dhcpRangeStart: DHCP_RANGE_START,
+      dhcpRangeEnd: DHCP_RANGE_END,
+      domain: "pxe-test.local",
+      sshKeys: [pubKey],
+      adminUser: "lab",
+    });
+
+    // iPXE binary
+    const ipxeSrc = "/usr/share/ipxe/ipxe-snponly-x86_64.efi";
+    if (!existsSync(ipxeSrc)) {
+      throw new Error(`iPXE not found: ${ipxeSrc}. Install: sudo dnf install ipxe-bootimgs-x86`);
+    }
+    copyFileSync(ipxeSrc, join(config.tftpDir, "ipxe.efi"));
+    try { symlinkSync(join(config.tftpDir, "ipxe.efi"), join(config.httpDir, "ipxe.efi")); } catch { /* exists */ }
+
+    const cacheDir = "/var/lib/libvirt/images/lab-pxe-cache";
+    execSync(`mkdir -p "${cacheDir}"`, { stdio: "pipe" });
+
+    // Fedora kernel+initrd for DISCOVERY (OS-neutral, same as pxe test)
+    const kernel = join(cacheDir, `vmlinuz-${config.fedoraVersion}`);
+    const initrd = join(cacheDir, `initrd-${config.fedoraVersion}.img`);
+    if (!existsSync(kernel)) {
+      log(`Downloading Fedora ${config.fedoraVersion} kernel (discovery)...`);
+      execSync(`curl -# -L -f -o "${kernel}" "${config.fedoraMirror}/images/pxeboot/vmlinuz"`, { stdio: "inherit", timeout: 300_000 });
+    }
+    if (!existsSync(initrd)) {
+      log(`Downloading Fedora ${config.fedoraVersion} initrd (discovery)...`);
+      execSync(`curl -# -L -f -o "${initrd}" "${config.fedoraMirror}/images/pxeboot/initrd.img"`, { stdio: "inherit", timeout: 300_000 });
+    }
+    copyFileSync(kernel, join(config.httpDir, "vmlinuz"));
+    copyFileSync(initrd, join(config.httpDir, "initrd.img"));
+
+    // VyOS netboot artifacts — cache the three extracted files across runs
+    const vyosCache = {
+      kernel: join(cacheDir, "vyos-vmlinuz"),
+      initrd: join(cacheDir, "vyos-initrd"),
+      squashfs: join(cacheDir, "vyos-filesystem.squashfs"),
+    };
+    if (Object.values(vyosCache).every((p) => existsSync(p))) {
+      log("VyOS netboot artifacts cached");
+      copyFileSync(vyosCache.kernel, join(config.httpDir, "vyos-vmlinuz"));
+      copyFileSync(vyosCache.initrd, join(config.httpDir, "vyos-initrd"));
+      copyFileSync(vyosCache.squashfs, join(config.httpDir, "vyos-filesystem.squashfs"));
+    } else {
+      log("Extracting VyOS artifacts from ISO (downloads ~600MB on first run)...");
+      prepareVyosArtifacts(config);
+      copyFileSync(join(config.httpDir, "vyos-vmlinuz"), vyosCache.kernel);
+      copyFileSync(join(config.httpDir, "vyos-initrd"), vyosCache.initrd);
+      copyFileSync(join(config.httpDir, "vyos-filesystem.squashfs"), vyosCache.squashfs);
+    }
+
+    writeFileSync(join(config.httpDir, "discover.ks"), generateDiscoverKickstart(config));
+    writeFileSync(join(config.httpDir, "boot.ipxe"), renderBootIpxe({ serverIp: config.serverIp, httpPort: config.httpPort }));
+    generateDnsmasqConf(config);
+
+    const { app, syslog } = createApp(config);
+    bastionApp = app;
+    await app.listen({ port: config.httpPort, host: "0.0.0.0" });
+    syslog.start();
+    log(`Bastion listening on :${HTTP_PORT}`);
+
+    log("Starting dnsmasq...");
+    startDnsmasq(config).catch((err) => {
+      log(`dnsmasq failed (expected without root): ${err instanceof Error ? err.message : String(err)}`);
+    });
+    await sleep(1000);
+
+    log("Creating PXE VM...");
+    // Two decoy NICs ahead of the PXE NIC, on a network with no route to the
+    // bastion. This reproduces the real VP2440 topology: live-boot scans for
+    // "the first connected interface", and without BOOTIF it picks a decoy,
+    // times out on DHCP/fetch, and dies with "Unable to find a live file
+    // system on the network". A single-NIC VM cannot catch that.
+    createPxeVm({
+      name: VM_NAME,
+      memory: VM_MEMORY,
+      vcpus: VM_VCPUS,
+      diskSize: VM_DISK_GB,
+      network: PXE_NETWORK_NAME,
+      decoyNics: 2,
+    });
+    const mac = getVmMac(VM_NAME, PXE_NETWORK_NAME);
+    if (!mac) throw new Error("Could not determine VM MAC address");
+    vmMac = mac;
+    log(`VM MAC: ${vmMac}`);
+
+    log("Waiting for discovery...");
+    type MachinesResponse = { discovered: Record<string, unknown> };
+    await pollApi<MachinesResponse>(
+      `http://${BASTION_IP}:${HTTP_PORT}/api/machines`,
+      (data) => vmMac in data.discovered,
+      DISCOVERY_TIMEOUT_MS,
+    );
+    log("VM discovered. Running fresh VyOS install (round 1)...");
+
+    await sleep(15_000); // discovery reboot cycle
+    vmIp = await installRound({ mac: vmMac, hostname: HOSTNAME_R1 });
+    log("Round 1 (fresh install) complete.");
+  }, DISCOVERY_TIMEOUT_MS + INSTALL_TIMEOUT_MS + SSH_TIMEOUT_MS + 300_000);
+
+  afterAll(async () => {
+    log("Cleaning up...");
+    if (bastionApp) await bastionApp.close().catch(() => {});
+    const { stopDnsmasq } = await import("../../src/bastion/src/services/dnsmasq.js");
+    stopDnsmasq();
+    destroyPxeVm(VM_NAME);
+    destroyPxeNetwork();
+    if (testDir) rmSync(testDir, { recursive: true, force: true });
+  });
+
+  it("machine is installed with a real IP (WI-1: ready-at parsing)", async () => {
+    const res = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/machines`);
+    const data = (await res.json()) as { installed: Record<string, { ip: string; os?: string }> };
+    const machine = data.installed[vmMac];
+    expect(machine).toBeDefined();
+    expect(machine.ip).toMatch(/^\d+\.\d+\.\d+\.\d+$/);
+    expect(machine.os).toBe("vyos-rolling");
+  });
+
+  it("install logs were streamed live (WI-2)", async () => {
+    const res = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/logs/${encodeURIComponent(vmMac)}`);
+    const data = (await res.json()) as LogsResponse;
+    expect(data.log_total).toBeGreaterThan(0);
+    const lines = (data.log_lines ?? []).map((l) => l.line).join("\n");
+    // Installer transcript lines and driver messages both flow through /api/log
+    expect(lines).toMatch(/Welcome to VyOS installation|>>> answered|base config:/);
+  });
+
+  it("SSH works as the vyos user with the injected key", () => {
+    const result = sshExec(vmIp, SSH_USER, "whoami", { keyPath: sshKeyPathGlobal });
+    expect(result.exitCode).toBe(0);
+    expect(result.stdout.trim()).toBe("vyos");
+  });
+
+  it("generated config was adopted (hostname + ssh key)", () => {
+    const result = sshExec(vmIp, SSH_USER, "cat /opt/vyatta/etc/config/config.boot", { keyPath: sshKeyPathGlobal });
+    expect(result.exitCode).toBe(0);
+    expect(result.stdout).toContain(`host-name "${HOSTNAME_R1}"`);
+    expect(result.stdout).toContain("public-keys");
+  });
+
+  it("boot-order step ran and reported (WI-3)", async () => {
+    const res = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/logs/${encodeURIComponent(vmMac)}`);
+    const data = (await res.json()) as LogsResponse;
+    const lines = (data.log_lines ?? []).map((l) => l.line).join("\n");
+    expect(lines).toContain("boot order:");
+  });
+
+  it("provisioning metadata persisted to /config (WI-4)", () => {
+    const result = sshExec(vmIp, SSH_USER, "cat /config/lab-provisioned 2>/dev/null || cat /opt/vyatta/etc/config/lab-provisioned", { keyPath: sshKeyPathGlobal });
+    expect(result.exitCode).toBe(0);
+    expect(result.stdout).toContain(`hostname=${HOSTNAME_R1}`);
+    expect(result.stdout).toContain("role=vanilla");
+    expect(result.stdout).toContain(`bastion=http://${BASTION_IP}:${HTTP_PORT}`);
+  });
+
+  it("reinstall preserves config and /config data (round 2)", async () => {
+    // Drop a marker in /config — the installer's previous-installation copy
+    // must carry it (and the whole old config) into the new image.
+    // `sync` is REQUIRED: rebootPxeVm uses `virsh destroy` (a hard power-cut),
+    // so an unsynced write never reaches the disk and the marker vanishes for
+    // reasons that have nothing to do with the installer.
+    const marker = sshExec(vmIp, SSH_USER, "echo LAB-MARKER-R2 > /config/lab-marker && sync && cat /config/lab-marker", { keyPath: sshKeyPathGlobal });
+    expect(marker.exitCode).toBe(0);
+    expect(marker.stdout).toContain("LAB-MARKER-R2");
+
+    // Queue with a DIFFERENT hostname: with preserve semantics the previous
+    // config must win, so the hostname must NOT change.
+    vmIp = await installRound({ mac: vmMac, hostname: HOSTNAME_R2 });
+
+    // Assert the config carry-forward first — it is the primary preservation
+    // signal and does not depend on the marker mechanism above.
+    const cfg = sshExec(vmIp, SSH_USER, "cat /opt/vyatta/etc/config/config.boot", { keyPath: sshKeyPathGlobal });
+    expect(cfg.stdout).toContain(`host-name "${HOSTNAME_R1}"`);   // old config carried
+    expect(cfg.stdout).not.toContain(`host-name "${HOSTNAME_R2}"`);
+
+    const markerAfter = sshExec(vmIp, SSH_USER, "cat /config/lab-marker", { keyPath: sshKeyPathGlobal });
+    expect(markerAfter.exitCode).toBe(0);
+    expect(markerAfter.stdout).toContain("LAB-MARKER-R2");
+  }, INSTALL_TIMEOUT_MS + SSH_TIMEOUT_MS + 60_000);
+
+  it("freshConfig makes the generated config win, data still kept (round 3)", async () => {
+    // Re-assert the marker is on disk and synced before the next power-cut.
+    const pre = sshExec(vmIp, SSH_USER, "sync && cat /config/lab-marker", { keyPath: sshKeyPathGlobal });
+    expect(pre.stdout).toContain("LAB-MARKER-R2");
+
+    vmIp = await installRound({ mac: vmMac, hostname: HOSTNAME_R3, freshConfig: true });
+
+    const cfg = sshExec(vmIp, SSH_USER, "cat /opt/vyatta/etc/config/config.boot", { keyPath: sshKeyPathGlobal });
+    expect(cfg.stdout).toContain(`host-name "${HOSTNAME_R3}"`);   // generated config won
+
+    // The marker file (non-config data under /config) still survives —
+    // freshConfig replaces only config.boot, not the carried data.
+    const markerAfter = sshExec(vmIp, SSH_USER, "cat /config/lab-marker", { keyPath: sshKeyPathGlobal });
+    expect(markerAfter.exitCode).toBe(0);
+    expect(markerAfter.stdout).toContain("LAB-MARKER-R2");
+  }, INSTALL_TIMEOUT_MS + SSH_TIMEOUT_MS + 60_000);
+});

labsim/.gitignore

@@ -0,0 +1,8 @@
+# runtime artifacts, not source
+*.log
+labsim_matrix_lib.py
+__pycache__/
+
+# Cluster-admin credentials for the rehearsal cluster, written by
+# `k8s-up.sh --kubeconfig`. Regenerate it rather than commit it.
+*.kubeconfig

labsim/README.md

@@ -0,0 +1,337 @@
+# labsim — libvirt replica of the lab network
+
+A throwaway copy of the production VLAN topology for testing routing, firewall
+rules and failover **without touching the real network**. Same VLAN IDs and
+roles as UniFi, deliberately different IP ranges so nothing can be confused for
+production.
+
+## Topology
+
+Each VLAN is its own isolated libvirt network with one tiny Alpine VM on it.
+
+| VLAN | Name | Sim subnet | VM address | Mirrors production |
+|-----:|------|------------|-----------|--------------------|
+| 1 | management | 172.31.1.0/24 | 172.31.1.10 | 192.168.1.0/24 |
+| 2 | k8s | 172.31.2.0/24 | 172.31.2.10 | 192.168.8.0/23 |
+| 3 | kvm | 172.31.3.0/24 | 172.31.3.10 | 192.168.3.0/24 |
+| 9 | private | 172.31.9.0/24 | 172.31.9.10 | 10.0.9.0/23 |
+| 10 | lot | **172.31.10.0/23** | 172.31.10.10 | 10.0.0.0/23 |
+| 200 | roomates | 172.31.200.0/24 | 172.31.200.10 | 192.168.2.0/24 |
+
+The sim subnet encodes the VLAN id: `172.31.<vlan>.0/24`, with one exception.
+**VLAN 10 is a `/23`** because every UniFi DHCP reservation lives in LoT and LoT
+spans `10.0.0.x` *and* `10.0.1.x`, which a `/24` cannot hold. The mapping stays
+readable — `10.0.0.46 → 172.31.10.46`, `10.0.1.67 → 172.31.11.67`.
+
+LoT's host leg is `.3`, not `.2`, because `10.0.0.2` is a real reservation
+(Hubitat) that maps onto `172.31.10.2`. `.3` is unreserved and sits below the
+DHCP pool, so it can never be handed out.
+
+`vlans.conf` therefore takes two optional trailing fields:
+
+```
+vlan_id:name:sim_prefix:real_subnet[:masklen][:host_octet]
+```
+
+defaulting to `24` and `2`. k8s and Private are also `/23` in production but
+hold no reservations, so they keep their `/24` and their DHCP range is clamped
+— reported at generation time, never silently.
+
+Address plan, identical on every VLAN:
+
+| Address | Role |
+|---------|------|
+| `.1` | gateway under test — a router VM you add (not created by default) |
+| `.2` | host bridge — how you reach the VMs from this workstation |
+| `.10` | the VLAN's micro VM |
+| `.254` | reserved for a VRRP VIP, mirroring production |
+
+The host sits at `.2` purely so you can SSH in. It is deliberately **not** the
+VMs' default route — that is `.1` — so inter-VLAN tests fail loudly when no
+router is present instead of being silently served by the host's own routing
+table. libvirt also installs reject rules that stop these networks forwarding
+to each other, so traffic between VLANs only works once a router VM bridges
+them.
+
+## Usage
+
+```bash
+./labsim-up.sh              # bring up every VLAN (idempotent)
+./labsim-up.sh 2 3          # only VLANs 2 and 3
+./labsim-down.sh            # destroy VMs + networks, keep the base image
+./labsim-down.sh --purge    # also delete the downloaded Alpine image
+```
+
+Each VM: 256 MB, 1 vCPU, a copy-on-write overlay on one shared 176 MB Alpine
+image (so six VMs cost a few MB of disk, not 1 GB).
+
+## Access
+
+```bash
+ssh alpine@172.31.2.10       # normal user (password: labsim)
+ssh root@172.31.2.10         # privileged — this image has no sudo
+curl http://172.31.2.10/     # hello-world page naming the VLAN
+```
+
+Console, when the network is the thing that is broken:
+
+```bash
+sudo virsh console labsim-2-k8s     # root / labsim
+```
+
+## Watching it
+
+```bash
+./labsim-matrix.py --watch 2     # terminal grid, changed cells highlighted
+./monitoring-up.sh               # topology page + Prometheus + Grafana
+```
+
+## Testing the DHCP migration
+
+`./labsim-dhcp-test.sh` boots throwaway VMs whose MACs are **real production
+MACs** and checks each gets the address UniFi reserved for it. MACs are the one
+piece of production config that transplants verbatim, which is what makes this a
+test rather than a rehearsal. It is safe because `ovs-labsim` has no physical
+NIC — verified with `ovs-vsctl show` — so a production MAC cannot reach the real
+LAN.
+
+Apply the config first, from `../migration`:
+
+```bash
+python3 unifi-to-vyos.py --mode sim -o /tmp/sim.conf   # 6 subnets, 31 mappings
+# load onto labsim-vyos, then:
+./labsim-dhcp-test.sh
+```
+
+**Result on VyOS 2026.08 (kea): all four cases pass.** The one that mattered:
+most UniFi reservations sit *inside* the DHCP pool, and **kea honours in-pool
+host reservations** — `printer1` received `172.31.10.46` from within the
+`.10.11–.11.254` pool. That was the open question blocking the cutover.
+
+### The lease database will lie to you
+
+The script wipes `/config/dhcp/dhcp4-leases.csv*` before every run, and both
+halves of that matter:
+
+- **Stale leases defeat reservations.** Re-running against yesterday's leases,
+  kea handed dynamic addresses to three devices that have reservations. The
+  reservation was present and correct in `/run/kea/kea-dhcp4.conf` the whole
+  time. Kea saw the reserved address as already leased to "another client" —
+  same MAC, but a different client-id from the earlier boot — and allocated
+  elsewhere. The cutover itself starts with an empty lease database, so this is
+  a *testing* artifact, but it is worth knowing that a reservation is not an
+  unconditional guarantee once leases exist.
+- **The `*` is load-bearing.** Kea's memfile backend keeps lease-file-cleanup
+  rotations (`dhcp4-leases.csv.2`) and restores from them on start, so
+  truncating only the primary file changes nothing.
+
+Both of those first appeared as a *passing* test. The verdict logic now refuses
+to score a MAC with more than one lease, because taking the first match had
+reported an hours-old lease as the current answer and turned three failures
+into apparent passes.
+
+Still open: whether kea will hand a *reserved* address to a *different* client
+while the reserved device is offline. The negative case here only proves an
+unreserved MAC gets an unreserved address.
+
+- **http://localhost:9101/** — live mesh: a node per VLAN, the router in the
+  middle, one line per pair coloured green/red with the ICMP RTT on it. Hover a
+  line for per-direction detail. Refreshes every 5s. This is the one to watch
+  while changing firewall rules.
+- **http://localhost:3000/d/labsim-matrix** — Grafana (anonymous, no login) for
+  *history*: when did a path flip, and how has latency moved.
+- **http://localhost:9101/metrics** — `labsim_reachable{src,dst,proto}` and
+  `labsim_rtt_ms{src,dst}`.
+
+## WAN follows VRRP, and the PPPoE half of it
+
+One consumer ISP account, two routers. The 10 gig line's lease is bound to a
+cloned MAC and the Vodafone line to a single credential, so neither can be live
+on both boxes: the WAN has to move with mastership.
+
+The two halves use different control planes, and that asymmetry is the design:
+
+| | plane | why |
+|---|---|---|
+| `bond0.53` | VyOS **config** (`disable`) | only config can move a MAC |
+| `pppoe0` | **systemd** unit gate | see below |
+
+`set interfaces pppoe pppoe0 disable` cannot work as a resting state.
+`interfaces_pppoe.py` treats `disable` and `delete` identically and **unlinks
+`/etc/ppp/peers/pppoe0`** — which is pppd's own options file. The resting state
+therefore destroyed what the promotion path needed, and `ppp@pppoe0`
+restart-looped against it (47 restarts, zero sessions at the AC). It also makes
+op-mode `connect interface pppoe0` refuse, and puts every failover behind a
+priority-322 commit where one unrelated bad node fails the lot.
+
+So `pppoe0` is configured identically and **enabled on both**, and dialling is
+gated by a drop-in:
+
+```ini
+ConditionPathExists=/run/vrrp-wan/may-dial
+ConditionPathExists=/etc/ppp/peers/pppoe0
+```
+
+`/run` is tmpfs, so the gate is shut at boot. That matters more than it looks:
+with the node enabled, `interfaces_pppoe.py` restarts ppp on **every** commit
+touching the pppoe subtree when the daemon isn't running — so the backup
+actively tries to dial whenever anything commits. The gate is the only thing
+making that a no-op, which is why the reconciler refuses to bless a box whose
+drop-in is missing (`/etc` is per-image; a VyOS upgrade would silently remove
+the protection).
+
+`may-dial` is a **lease**, not a flag: `ConditionPathExists` is evaluated at
+start only, so it can prevent a dial but never revoke one. `vrrp-wan-reconcile`
+renews it every 30s; `vrrp-wan-guard` runs every 5s and only ever revokes.
+
+### Testing it
+
+```sh
+./labsim-pppoe-ha-test.sh --all
+```
+
+The verdict is what the **routers** and the **access concentrator** did, never
+what a client happened to get — and the harness refuses to run at all while a
+router still has a default route via `eth2`, because the libvirt-NAT scaffold
+answers connectivity checks that the WAN under test would have failed. The
+invariant it enforces throughout: *the AC never reports two `simdsl` sessions,
+and no two routers ever hold `pppoe0`.*
+
+Two failures the harness itself produced, both worth remembering: waiting for
+"exactly one holder" returns instantly during a handover (it was already true),
+and judging connectivity on a single ping 20s after a link drop reports an
+outage that has already healed. Ask **who** holds it, and poll.
+
+Evidence in `wan-failover-evidence/`.
+
+## Routing: BGP, dual WAN, and the ISP VMs
+
+`sim-ha-config.py` covers the LAN side of the routers. `sim-net-config.py`
+covers everything that makes this a rehearsal for production *routing*:
+
+| role | VM | what it generates |
+|---|---|---|
+| `primary` | `labsim-vyos` | BGP + dual WAN + health-checked failover |
+| `secondary` | `labsim-vyos2` | BGP only |
+| `isp-dhcp` | `labsim-isp-dhcp` | 10gig-equivalent ISP on VLAN 53 |
+| `isp-pppoe` | `labsim-isp-pppoe` | Vodafone-equivalent PPPoE ISP on VLAN 51 |
+
+Both ISP VMs are VyOS with two NICs: one on the OVS trunk facing the sim
+router, one on libvirt's `default` network, NATing customers to the real
+internet. They use RFC 5737 documentation ranges (`203.0.113.0/24`,
+`198.51.100.0/24`) so a leaked sim route cannot blackhole anything real.
+
+```sh
+./sim-net-apply.sh check     # VM state vs what the code says — run this first
+./sim-net-apply.sh apply     # push generated config over the serial console
+```
+
+`check` is the important one. All of this previously existed only as running
+state, applied by hand over SSH; rebuilding a VM lost it, and nothing recorded
+why any of it was shaped the way it was.
+
+### Known gaps vs production
+
+- **WAN is on the primary router only.** Production has WAN on both. Two PPPoE
+  clients sharing one credential against a single access concentrator is a
+  failure mode production does not have, so the sim does not model it. VRRP and
+  conntrack failover are still exercised.
+- **ISP VM interface names are not stable across a rebuild** — `isp-dhcp` came
+  up as `eth0`/`eth1` and `isp-pppoe` as `eth2`/`eth3` from identical XML.
+  Check `show interfaces` and pass `--wan-if` / `--uplink-if` rather than
+  trusting the defaults.
+- **`eth2` on the primary router** is a libvirt-NAT uplink predating the ISP
+  VMs: a third default route with no production equivalent that masks real WAN
+  failures during a failover test. `--drop-scaffold` removes it.
+- **Committing on `isp-pppoe` drops the router's PPPoE session**, and the
+  client does not redial promptly. After any change there, check `pppoe0` on
+  the router and `sudo systemctl restart ppp@pppoe0` if it is missing.
+
+## The trunk carries every VLAN tagged, including Management
+
+There is deliberately **no native/untagged VLAN** on the trunks to the routers,
+and Management lives on `bond0.1`, not on the bare `bond0`.
+
+A native VLAN is what puts a subnet on the bond **parent** while every other
+VLAN sits on a sub-interface of it. With `dhcp-socket-type: raw`, kea receives
+each tagged frame *twice* — once on the sub-interface and once on the parent —
+and answers from the parent's pool as well (ISC Kea
+[#1117](https://gitlab.isc.org/isc-projects/kea/-/issues/1117)). A client on
+VLAN 3 gets two OFFERs and keeps whichever arrives first:
+
+```
+bond0.3 :  172.31.3.252 → 172.31.3.11   correct
+bond0   :  172.31.1.252 → 172.31.1.8    UNTAGGED, Management pool, wrong
+```
+
+`./labsim-vlan-leak-test.sh` makes one client on a tagged VLAN send a DISCOVER
+and captures on the parent and the sub-interface at once. The verdict is how
+many OFFERs the **server** emitted and from which subnets — deliberately not
+"did the client get the right address", because a client picking correctly is
+exactly how this hid. Both orderings were observed across runs, so a passing
+client proves nothing.
+
+```sh
+./labsim-vlan-leak-test.sh --vlan 3           # PASS on the current shape
+LABSIM_NATIVE_VLAN=1 ./router-up.sh           # restore the old shape...
+./labsim-vlan-leak-test.sh --vlan 3           # ...and it FAILs again
+```
+
+Three things this cost, all of which apply to production:
+
+- **Kea must be restarted after the address moves.** VyOS does not restart it
+  for an interface address change, so it keeps a raw socket bound with the old
+  address and the bug survives the fix. In the sim kea had been running since
+  16 Aug; the first post-fix test failed for this reason alone and looked like
+  the fix simply not working.
+- **The firewall interface-group must move too.** `interface-group LAN` named
+  the bare `bond0`; with a default-deny ruleset, moving the address without
+  moving the group drops every management session and all VLAN 1 routing.
+- **Duplicate delivery does not stop.** #1117 says only that there is no longer
+  a subnet on the parent to match, and that is exactly what happens: two replies
+  per DISCOVER, both now from the correct pool. Harmless, but do not read a
+  duplicate as a failure.
+
+### Tagged and untagged Management coexist
+
+Verified directly, and it is what makes the production cutover a rolling change
+rather than an outage: with the primary still untagged on `bond0` and the
+secondary already tagged on `bond0.1`, both routers were reachable, the VIP
+stayed up and a VLAN 1 client kept its gateway. One VLAN is one broadcast
+domain regardless of how each port tags it, so the two firewalls can be
+converted one at a time. See `migration/MANAGEMENT-VLAN-TAGGED.md`.
+
+`./vlan1-move-monitor.sh` logs VIP/router liveness once a second during the
+change, because VRRP reconverges and leaves no trace of who held the VIP.
+
+## Notes for whoever extends this
+
+Things that cost time the first time round, all verified on this image:
+
+- **No `sudo`.** Alpine ships `doas`; cloud-init's `sudo:` directive is inert
+  here. Use `root@` for privileged work.
+- **cloud-init leaves users locked** (`!*` in `/etc/shadow`) unless
+  `lock_passwd: false`, and sshd then refuses key auth for that user.
+- **One failing `runcmd` aborts every command after it.** Each entry is
+  `|| true` for that reason.
+- **busybox here has no `httpd` applet**, and the VMs have no internet to
+  `apk add` one — so the hello-world server is `python3 -m http.server`
+  (python3 is already present because cloud-init depends on it).
+- **`start-stop-daemon --exec /usr/bin/python3` matches cloud-init's own
+  python3** at boot and refuses to start anything.
+- **busybox `pgrep -f PATTERN` matches its own argv**, so a "skip if already
+  running" guard always fires. Verified: `guard_exit=0` with nothing listening.
+
+## Not modelled (yet)
+
+- **The secondary's bond was fiction until 2026-09-02.** `ovs_bond_router`'s
+  "already bonded, nothing to do" check compared only the trunk VLAN list, not
+  the membership. Restarting a VM recreates its taps under new names, so the
+  bond sat there holding two interfaces that no longer existed while the router's
+  real taps ran in the bridge as two *independent* ports — no LACP, and carrying
+  libvirt's own portgroup VLAN config rather than the bond's. It reconciles
+  membership now, but the lesson generalises: a sim that reports success is not
+  the same as a sim that models the thing.
+- **`labsim-vyos` has a third NIC** on libvirt's `default` network (the scaffold
+  uplink, see `--drop-scaffold`). The tap count is filtered to `$OVS_NET` for
+  that reason; an unfiltered count is 3 and silently skipped the primary's bond.

labsim/cilium-ipam-switch.sh

@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+# Procedure around a Cilium IPAM mode change. Works against any cluster, so the
+# rehearsal in labsim and the real thing in production run the SAME steps.
+#
+# It deliberately does NOT change the mode itself. In labsim that is `helm
+# upgrade`; in production Pulumi owns the release and a script racing it would
+# just reintroduce drift. What this owns is everything around the apply -- the
+# evidence, the deadlock, and the verdict.
+#
+#   ./cilium-ipam-switch.sh preflight   record what the cluster looks like now
+#   ./cilium-ipam-switch.sh unstick     break the agent-not-ready taint deadlock
+#   ./cilium-ipam-switch.sh verify      compare against preflight, report renumbering
+#
+#   KUBECONFIG=... ./cilium-ipam-switch.sh preflight
+#
+# Whether a recycle is needed is CONDITIONAL, and `verify` is what decides it.
+#
+# The operator does not preserve which node held which /24 -- it adopts whatever
+# CiliumNode.spec.ipam.podCIDRs already says. So:
+#
+#   * If CiliumNode already agrees with node.spec.podCIDRs on every node -- which
+#     is the case for any cluster that has only ever run ipam=kubernetes, because
+#     the operator syncs one from the other -- the pool adopts the existing
+#     allocation, no node is renumbered, and NO pod recycle is needed. Verified
+#     on the 3-node labsim cluster: CIDRs unchanged, nothing stranded, the only
+#     blip was the cilium DaemonSet restarting itself.
+#
+#   * If the two sources DISAGREE, nodes can swap /24s. Their running pods keep
+#     addresses that no longer fall inside the node's range, every other node
+#     routes that prefix to the wrong node, and those pods go unreachable
+#     cross-node while still showing Running. Then a full recycle is mandatory.
+#
+# Do not skip `verify` on the assumption of the good case. Run it and read it.
+set -uo pipefail
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+STATE="${STATE:-$SCRIPT_DIR/.ipam-switch-state}"
+K="kubectl"
+
+say()  { printf '\033[0;36m[ipam]\033[0m %s\n' "$*"; }
+warn() { printf '\033[1;33m[ipam]\033[0m %s\n' "$*" >&2; }
+
+snapshot() {
+  echo "## nodes"
+  $K get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.podCIDRs}{"\n"}{end}' 2>/dev/null
+  echo "## ciliumnodes"
+  $K get ciliumnode -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.ipam.podCIDRs}{"\n"}{end}' 2>/dev/null
+  echo "## pods"
+  $K get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\t"}{.status.podIP}{"\n"}{end}' 2>/dev/null \
+    | grep -vP '\t$' | sort
+  echo "## ipam"
+  $K -n kube-system get cm cilium-config -o jsonpath='{.data.ipam}' 2>/dev/null; echo
+}
+
+cmd_preflight() {
+  mkdir -p "$STATE"
+  snapshot > "$STATE/before.txt"
+  say "recorded $(grep -c . "$STATE/before.txt") lines -> $STATE/before.txt"
+  say "mode now: $(sed -n '/^## ipam/,$p' "$STATE/before.txt" | tail -1)"
+  # The pod inventory is the rollback reference: if the switch renumbers, this
+  # is the only record of what an address USED to be.
+  say "pods on the pod network: $(sed -n '/^## pods/,/^## ipam/p' "$STATE/before.txt" | grep -c '10\.')"
+}
+
+# The deadlock, in one place because it WILL happen and doing it by hand under
+# time pressure is how the wrong node gets untainted:
+#   agent has no pod CIDR -> agent not ready -> node keeps
+#   node.cilium.io/agent-not-ready:NoSchedule -> the operator that would assign
+#   the CIDR cannot schedule -> agent still has no pod CIDR.
+# Removing the taint is safe: it exists to keep normal workloads off a node
+# without working networking, and the operator is precisely the thing that fixes
+# that. Kubernetes re-adds it on the next agent restart.
+cmd_unstick() {
+  local stuck=0
+  for n in $($K get nodes -o name 2>/dev/null); do
+    $K get "$n" -o jsonpath='{.spec.taints[*].key}' 2>/dev/null | grep -q 'agent-not-ready' || continue
+    warn "${n#node/} carries agent-not-ready; removing so the operator can schedule"
+    $K taint "$n" node.cilium.io/agent-not-ready- >/dev/null 2>&1 && stuck=$((stuck+1))
+  done
+  [ "$stuck" -eq 0 ] && say "no node was stuck" || say "cleared $stuck node(s)"
+  local pend
+  pend="$($K -n kube-system get pods -l io.cilium/app=operator --no-headers 2>/dev/null | grep -c Pending)"
+  [ "${pend:-0}" -gt 0 ] && warn "$pend operator pod(s) still Pending — check tolerations, not just taints"
+  return 0
+}
+
+cmd_verify() {
+  [ -f "$STATE/before.txt" ] || { warn "no preflight snapshot; nothing to compare"; return 1; }
+  snapshot > "$STATE/after.txt"
+  echo
+  say "mode: $(sed -n '/^## ipam/,$p' "$STATE/before.txt" | tail -1) -> $(sed -n '/^## ipam/,$p' "$STATE/after.txt" | tail -1)"
+
+  # The question that decides the size of the maintenance window: did per-node
+  # CIDRs survive, or was every node renumbered (and every pod with it)?
+  local moved=0
+  while IFS=$'\t' read -r node cidr; do
+    [ -z "${node:-}" ] && continue
+    local now; now="$(sed -n '/^## ciliumnodes/,/^## pods/p' "$STATE/after.txt" | awk -F'\t' -v n="$node" '$1==n{print $2}')"
+    if [ -n "$now" ] && [ "$now" != "$cidr" ]; then
+      printf '    %-16s %s -> %s\n' "$node" "$cidr" "$now"; moved=$((moved+1))
+    fi
+  done < <(sed -n '/^## ciliumnodes/,/^## pods/p' "$STATE/before.txt" | grep -P '\t')
+  if [ "$moved" -eq 0 ]; then
+    say "per-node CIDRs UNCHANGED — the pool adopted the existing allocation"
+  else
+    warn "$moved node(s) renumbered — every pod on them must be recycled"
+  fi
+
+  local before after same
+  before="$(sed -n '/^## pods/,/^## ipam/p' "$STATE/before.txt" | grep -P '\t10\.' | wc -l)"
+  after="$(sed -n '/^## pods/,/^## ipam/p' "$STATE/after.txt"  | grep -P '\t10\.' | wc -l)"
+  same="$(comm -12 <(sed -n '/^## pods/,/^## ipam/p' "$STATE/before.txt" | grep -P '\t10\.' | sort) \
+                   <(sed -n '/^## pods/,/^## ipam/p' "$STATE/after.txt"  | grep -P '\t10\.' | sort) | wc -l)"
+  say "pods: $before before, $after after, $same kept the SAME address"
+  # Keeping the address is NOT the good outcome. If a node's CIDR moved, its
+  # existing pods keep IPs that no longer fall inside it, every other node routes
+  # that prefix to the WRONG node, and those pods go unreachable cross-node while
+  # looking perfectly healthy. Observed in labsim: two nodes swapped CIDRs and
+  # cross-node ping to their pods dropped 100%, with every pod still Running.
+  # This is the check that decides whether a recycle is optional or mandatory.
+  local stranded=0
+  while read -r ns name ip node; do
+    [ -z "${node:-}" ] && continue
+    local cidr; cidr="$($K get ciliumnode "$node" -o jsonpath='{.spec.ipam.podCIDRs[0]}' 2>/dev/null)"
+    [ -z "$cidr" ] && continue
+    case "$ip" in
+      "${cidr%.*/*}".*) ;;
+      *) printf '    STRANDED %-40s %-15s on %s (now %s)\n' "$ns/$name" "$ip" "$node" "$cidr"; stranded=$((stranded+1)) ;;
+    esac
+  done < <($K get pods -A -o jsonpath='{range .items[?(@.status.podIP)]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.podIP}{" "}{.spec.nodeName}{"\n"}{end}' 2>/dev/null | grep -E ' 10\.')
+  if [ "$stranded" -gt 0 ]; then
+    warn "$stranded pod(s) sit OUTSIDE their node CIDR — unreachable cross-node until recycled"
+    warn "recycle: for ns in $(kubectl get ns -o name | cut -d/ -f2); do kubectl -n $ns rollout restart deploy,ds,sts 2>/dev/null; done"
+  else
+    say "every pod is inside its node CIDR — no recycle needed"
+  fi
+  say "not-Running pods: $($K get pods -A --no-headers 2>/dev/null | grep -vcE 'Running|Completed')"
+}
+
+case "${1:-}" in
+  preflight) cmd_preflight ;;
+  unstick)   cmd_unstick ;;
+  verify)    cmd_verify ;;
+  *) sed -n '2,16p' "$0"; exit 1 ;;
+esac

labsim/console-apply.py

@@ -0,0 +1,116 @@
+#!/usr/bin/env python3
+"""Apply VyOS config to a labsim VM over its serial console.
+
+Needed because a freshly installed VyOS comes up holding the same addresses as
+its peer, so there is a window where it cannot safely be reached over the
+network at all. The console does not care.
+
+  ./console-apply.py --vm labsim-vyos2 --config r2.conf
+"""
+from __future__ import annotations
+
+import argparse
+import sys
+import time
+
+import pexpect
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--vm", required=True)
+    ap.add_argument("--config", required=True)
+    ap.add_argument("--user", default="vyos")
+    ap.add_argument("--password", default="vyos")
+    # `save` writes config.boot. For WAN work that is dangerous: the resting
+    # state must stay `vif 53 disable` on both routers, and saving while a box
+    # is master persists the ENABLED state -- so a reboot would have it claim
+    # the cloned MAC. Observed in labsim on 2026-09-02.
+    ap.add_argument("--no-save", action="store_true",
+                    help="commit without saving (leave config.boot untouched)")
+    args = ap.parse_args()
+
+    cmds = [l.rstrip() for l in open(args.config)
+            if l.strip() and not l.lstrip().startswith("#")]
+    print(f"{len(cmds)} commands to apply to {args.vm}", file=sys.stderr)
+
+    c = pexpect.spawn(f"virsh --connect qemu:///system console {args.vm}",
+                      timeout=90, encoding="utf-8")
+    c.logfile_read = None
+    c.sendline("")
+    time.sleep(2)
+    c.sendline("")
+
+    # Log in. A freshly booted box may still be starting services, so allow a
+    # generous window and re-prod the console rather than failing on the first
+    # miss.
+    #
+    # `# ` matters as much as `$ `: a previous run that died mid-config leaves
+    # the console sitting in configuration mode, and waiting only for the
+    # operational prompt then hangs forever against a perfectly healthy VM.
+    in_config = False
+    for _ in range(40):
+        i = c.expect([r"login:", r"\$ ", r"# ", pexpect.TIMEOUT], timeout=15)
+        if i == 0:
+            c.sendline(args.user)
+            c.expect("Password:", timeout=30)
+            c.sendline(args.password)
+            c.expect([r"\$ ", r"# "], timeout=60)
+            break
+        if i == 1:
+            break
+        if i == 2:
+            in_config = True
+            break
+        c.sendline("")
+    else:
+        print("never reached a prompt", file=sys.stderr)
+        return 1
+
+    if in_config:
+        # Drop whatever the previous run left half-built rather than committing
+        # a candidate nobody has seen.
+        print("console was left in config mode; discarding stale candidate",
+              file=sys.stderr)
+        c.sendline("discard")
+        c.expect(r"# ", timeout=60)
+    else:
+        c.sendline("configure")
+        c.expect(r"# ", timeout=60)
+
+    for cmd in cmds:
+        c.sendline(cmd)
+        c.expect(r"# ", timeout=60)
+        out = c.before or ""
+        if "Set failed" in out or "not valid" in out or "Invalid" in out:
+            print(f"FAILED: {cmd}\n  {out.strip()[:200]}", file=sys.stderr)
+
+    print("committing...", file=sys.stderr)
+    c.sendline("commit")
+    c.expect(r"# ", timeout=300)
+    commit_out = c.before or ""
+    if not args.no_save:
+        c.sendline("save")
+        c.expect(r"# ", timeout=120)
+    # Accept either prompt on the way out. Insisting on `$ ` here hangs against
+    # a healthy box -- and worse, leaves the console parked in config mode, so
+    # the NEXT run finds a `# ` it was not expecting either. One strict expect
+    # turned into two failures.
+    c.sendline("exit")
+    c.expect([r"\$ ", r"# ", pexpect.TIMEOUT], timeout=60)
+    c.sendline("exit")
+    c.close(force=True)
+
+    bad = [l for l in commit_out.splitlines()
+           if "failed" in l.lower() or "error" in l.lower()]
+    if bad:
+        print("commit reported:", file=sys.stderr)
+        for l in bad[:10]:
+            print(f"  {l.strip()}", file=sys.stderr)
+        return 1
+    print("committed and saved", file=sys.stderr)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

labsim/dualstack-lab.sh

@@ -0,0 +1,318 @@
+#!/usr/bin/env bash
+# Differential study: what ACTUALLY differs between a k3s cluster born
+# dual-stack and one converted in place?
+#
+# k3s says dual-stack "cannot be enabled on an existing cluster". The stated
+# reason is narrow -- nodes get Pod CIDRs only at join and the Kubernetes IPAM
+# controller will not hand out a new IPv6 CIDR later -- and it does not obviously
+# apply to a cluster where Cilium owns IPAM. Rather than argue from docs, build
+# both shapes and diff them.
+#
+#   ./dualstack-lab.sh up v4         single-node k3s, IPv4 only     (.21)
+#   ./dualstack-lab.sh up dual       single-node k3s, dual-stack    (.22)
+#   ./dualstack-lab.sh pristine v4   reflink copy of v4's disk, so the upgrade
+#                                    attempt can be rolled back and retried
+#   ./dualstack-lab.sh restore v4    put that copy back
+#   ./dualstack-lab.sh collect <n>   normalized state dump -> evidence/<n>/
+#   ./dualstack-lab.sh compare a b   semantic diff of two collections
+#   ./dualstack-lab.sh virtdiff a b  whole-filesystem diff, offline (libguestfs)
+#   ./dualstack-lab.sh down [name]
+#
+# The comparison that matters is `compare dual upgraded`: everything it prints
+# is a way the converted cluster failed to reach the shape of a native one.
+#
+# Single node on purpose. Dual-stack is decided by server flags and CNI config,
+# both of which a one-node cluster exercises fully, and it rebuilds in minutes.
+# Node-rejoin behaviour needs the 3-node cluster and is a separate question.
+set -euo pipefail
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+source "$SCRIPT_DIR/lib.sh"
+source "$SCRIPT_DIR/ovs.sh"
+
+K8S_VLAN="${K8S_VLAN:-2}"
+DS_PREFIX="${DS_PREFIX:-172.31.2}"
+MEM="${MEM:-4096}"; CPUS="${CPUS:-2}"; DISK_GB="${DISK_GB:-12}"
+TOKEN="${TOKEN:-labsim-ds-token}"
+CILIUM_VERSION="${CILIUM_VERSION:-1.19.1}"   # same as production
+DEB_BASE="${DEB_BASE:-$IMG_DIR/debian-13-genericcloud-amd64.qcow2}"
+EVIDENCE="$SCRIPT_DIR/dualstack-evidence"
+
+# Pod/Service ranges. IPv4 halves are k3s's own defaults, so the v4-only build is
+# a stock cluster and the diff is not polluted by gratuitous differences.
+# IPv6 halves are ULA: this cluster never routes off-box, and using the real /48
+# here would put lab addresses into a prefix that production also announces.
+V4_CLUSTER="10.42.0.0/16"; V4_SERVICE="10.43.0.0/16"
+V6_CLUSTER="${V6_CLUSTER:-fd00:42::/56}"
+V6_SERVICE="${V6_SERVICE:-fd00:43::/112}"   # /112 -- apiserver caps v6 service ranges
+V6_PREFIX="${V6_PREFIX:-fd00:2}"            # node addresses: fd00:2::<octet>
+
+vm_name() { echo "labsim-ds-$1"; }
+vm_ip()   { case "$1" in v4) echo "$DS_PREFIX.21";; dual) echo "$DS_PREFIX.22";; *) die "unknown build '$1'";; esac; }
+vm_ip6()  { case "$1" in v4) echo "$V6_PREFIX::21";; dual) echo "$V6_PREFIX::22";; *) die "unknown build '$1'";; esac; }
+disk_of() { echo "$IMG_DIR/$(vm_name "$1").qcow2"; }
+
+ssh_vm() { local ip="$1"; shift; ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
+             -o LogLevel=ERROR -o ConnectTimeout=8 -o BatchMode=yes "debian@$ip" "$@"; }
+
+# --- seed -----------------------------------------------------------------
+build_seed() {
+  local iso="$1" vm="$2" mode="$3" pubkey="$4"
+  local ip ip6 tmp; ip="$(vm_ip "$mode")"; ip6="$(vm_ip6 "$mode")"; tmp="$(mktemp -d)"
+
+  echo "instance-id: $vm" > "$tmp/meta-data"
+  # Static v6 on both builds. The v4-only cluster still gets an IPv6 ADDRESS --
+  # only its Kubernetes config is v4-only. Otherwise the diff would be dominated
+  # by host addressing rather than by what Kubernetes did differently.
+  cat > "$tmp/network-config" <<EOF
+version: 2
+ethernets:
+  enp1s0:
+    addresses: [${ip}/24, ${ip6}/64]
+    routes:
+      - to: default
+        via: ${DS_PREFIX}.1
+    nameservers:
+      addresses: [8.8.8.8, 1.1.1.1]
+EOF
+
+  local exec_args="server --flannel-backend=none --disable-network-policy --disable=servicelb --disable=traefik --tls-san=$ip --cluster-init"
+  if [ "$mode" = dual ]; then
+    exec_args="$exec_args --cluster-cidr=${V4_CLUSTER},${V6_CLUSTER} --service-cidr=${V4_SERVICE},${V6_SERVICE} --node-ip=${ip},${ip6}"
+  else
+    exec_args="$exec_args --node-ip=${ip}"
+  fi
+
+  cat > "$tmp/user-data" <<EOF
+#cloud-config
+hostname: $vm
+fqdn: $vm
+users:
+  - name: debian
+    groups: [sudo]
+    shell: /bin/bash
+    sudo: ["ALL=(ALL) NOPASSWD:ALL"]
+    lock_passwd: false
+    plain_text_passwd: labsim
+    ssh_authorized_keys: [$pubkey]
+ssh_pwauth: true
+disable_root: false
+package_update: true
+packages: [curl, jq, iproute2, nftables]
+write_files:
+  - path: /etc/modules-load.d/cilium.conf
+    content: |
+      br_netfilter
+  - path: /etc/dualstack-lab-mode
+    content: |
+      $mode
+runcmd:
+  - modprobe br_netfilter || true
+  - |
+    curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="$exec_args" K3S_TOKEN="$TOKEN" sh -
+  - |
+    # Cilium via helm, matching the production version. IPAM stays 'kubernetes'
+    # in BOTH builds on purpose: that is what production runs, and it is the
+    # mode the k3s objection is actually about. If the converted cluster needs
+    # cluster-pool to work, the diff should be what tells us so.
+    curl -sfL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash || true
+    export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
+    helm repo add cilium https://helm.cilium.io >/dev/null 2>&1 || true
+    helm repo update >/dev/null 2>&1 || true
+    for i in \$(seq 1 60); do kubectl get nodes >/dev/null 2>&1 && break; sleep 5; done
+    if [ "$mode" = dual ]; then
+      helm install cilium cilium/cilium --version $CILIUM_VERSION -n kube-system \\
+        --set kubeProxyReplacement=false --set ipam.mode=kubernetes \\
+        --set ipv4.enabled=true --set ipv6.enabled=true \\
+        --set k8sServiceHost=$ip --set k8sServicePort=6443 || true
+    else
+      helm install cilium cilium/cilium --version $CILIUM_VERSION -n kube-system \\
+        --set kubeProxyReplacement=false --set ipam.mode=kubernetes \\
+        --set ipv4.enabled=true --set ipv6.enabled=false \\
+        --set k8sServiceHost=$ip --set k8sServicePort=6443 || true
+    fi
+    touch /etc/dualstack-lab-ready
+EOF
+  sudo mkdir -p "$(dirname "$iso")"
+  sudo genisoimage -quiet -output "$iso" -volid cidata -joliet -rock \
+    "$tmp/user-data" "$tmp/meta-data" "$tmp/network-config"
+  rm -rf "$tmp"
+}
+
+cmd_up() {
+  local mode="${1:?usage: up <v4|dual>}"
+  local vm ip disk seed pubkey
+  vm="$(vm_name "$mode")"; ip="$(vm_ip "$mode")"; disk="$(disk_of "$mode")"
+  seed="$IMG_DIR/${vm}-seed.iso"; pubkey="$(find_ssh_pubkey)"
+
+  [ -f "$DEB_BASE" ] || die "base image missing: $DEB_BASE (run ./k8s-up.sh once)"
+  if virsh_q dominfo "$vm" >/dev/null 2>&1; then
+    log "$vm exists — starting if stopped"
+    [ "$(virsh_q domstate "$vm" | head -1)" = "running" ] || virsh_q start "$vm" >/dev/null
+    return
+  fi
+  selected_vlans; ovs_up
+  log "creating $vm ($mode) at $ip / $(vm_ip6 "$mode")"
+  sudo qemu-img create -q -f qcow2 -F qcow2 -b "$DEB_BASE" "$disk" "${DISK_GB}G" >/dev/null
+  build_seed "$seed" "$vm" "$mode" "$pubkey"
+  sudo virt-install --connect "$LIBVIRT_URI" --name "$vm" \
+    --memory "$MEM" --vcpus "$CPUS" \
+    --disk "path=$disk,format=qcow2,bus=virtio" \
+    --disk "path=$seed,device=cdrom" \
+    --network "network=$OVS_NET,portgroup=vlan${K8S_VLAN},model=virtio" \
+    --os-variant debian12 --graphics none --noautoconsole --import >/dev/null
+  log "installing in background; watch: ssh debian@$ip 'ls /etc/dualstack-lab-ready'"
+}
+
+# --- pristine copy / restore ---------------------------------------------
+# reflink so the copy is instant and independent on btrfs/xfs. A qcow2 backing
+# chain would be cheaper still but makes the parent read-only in practice: boot
+# the parent again and every child silently corrupts.
+cmd_pristine() {
+  local mode="${1:?usage: pristine <v4|dual>}" vm disk
+  vm="$(vm_name "$mode")"; disk="$(disk_of "$mode")"
+  [ "$(virsh_q domstate "$vm" 2>/dev/null | head -1)" = "running" ] && \
+    die "$vm is running — shut it down first (virsh shutdown $vm), a copy of a live disk is not consistent"
+  sudo cp --reflink=auto "$disk" "${disk}.pristine"
+  log "pristine copy: ${disk}.pristine"
+}
+cmd_restore() {
+  local mode="${1:?usage: restore <v4|dual>}" vm disk
+  vm="$(vm_name "$mode")"; disk="$(disk_of "$mode")"
+  [ -f "${disk}.pristine" ] || die "no pristine copy for $mode"
+  [ "$(virsh_q domstate "$vm" 2>/dev/null | head -1)" = "running" ] && \
+    die "$vm is running — shut it down first"
+  sudo cp --reflink=auto "${disk}.pristine" "$disk"
+  log "restored $mode from pristine"
+}
+
+
+# --- the experiment ------------------------------------------------------
+# Convert the IPv4-only cluster in place, mirroring the flags the native build
+# was BORN with. Each step prints what the cluster did, because the interesting
+# output is which step refuses rather than whether the end state is pretty.
+cmd_upgrade() {
+  local ip; ip="$(vm_ip v4)"; local ip6; ip6="$(vm_ip6 v4)"
+  log "step 1/4: add dual CIDRs + dual node-ip to the k3s unit"
+  # Done with python on the box, not nested sed: quoting a multi-line systemd
+  # continuation through ssh -> sh -> sed produced a literal \\n in the unit, and
+  # k3s then saw a dual cluster-cidr with a still-IPv4 service-cidr and refused
+  # to start. All three flags go on one line -- systemd does not care, and there
+  # is nothing left to escape.
+  ssh_vm "$ip" "sudo python3 - <<'PYEOF'
+import re
+u = '/etc/systemd/system/k3s.service'
+s = open(u).read()
+old = \"'--node-ip=${ip}'\"
+new = \"'--cluster-cidr=${V4_CLUSTER},${V6_CLUSTER}' '--service-cidr=${V4_SERVICE},${V6_SERVICE}' '--node-ip=${ip},${ip6}'\"
+assert old in s, 'node-ip flag not found in unit'
+open(u,'w').write(s.replace(old, new))
+print('    unit rewritten')
+PYEOF
+sudo systemctl daemon-reload" || die "unit edit failed"
+  ssh_vm "$ip" "grep -oE \"'--(cluster|service)-cidr=[^']*'|'--node-ip=[^']*'\" /etc/systemd/system/k3s.service | sed 's/^/    /'"
+
+  log "step 2/4: restart k3s and see whether it accepts the changed ranges"
+  ssh_vm "$ip" "sudo systemctl restart k3s" || true
+  for i in $(seq 1 40); do
+    ssh_vm "$ip" "sudo k3s kubectl get --raw /readyz >/dev/null 2>&1" && break
+    sleep 5
+  done
+  ssh_vm "$ip" "sudo journalctl -u k3s --since '2 min ago' --no-pager 2>/dev/null | grep -iE 'cidr|dual|ipv6|invalid|cannot|fail' | tail -12 | sed 's/^/    /'" || true
+
+  log "step 3/4: what the API says now"
+  ssh_vm "$ip" "echo -n '    servicecidr: '; sudo k3s kubectl get servicecidr -o jsonpath='{.items[*].spec.cidrs}'; echo; \
+                echo -n '    node podCIDRs: '; sudo k3s kubectl get node -o jsonpath='{.items[0].spec.podCIDRs}'; echo; \
+                echo -n '    node addresses: '; sudo k3s kubectl get node -o jsonpath='{.items[0].status.addresses[*].address}'; echo" || true
+
+  log "step 4/4: turn on IPv6 in Cilium"
+  ssh_vm "$ip" "export KUBECONFIG=/etc/rancher/k3s/k3s.yaml; sudo -E helm upgrade cilium cilium/cilium --version ${CILIUM_VERSION} -n kube-system --reuse-values --set ipv6.enabled=true >/dev/null 2>&1 && echo '    cilium upgraded' || echo '    cilium upgrade FAILED'" || true
+  ssh_vm "$ip" "sudo k3s kubectl -n kube-system rollout restart ds/cilium >/dev/null 2>&1; sleep 20; sudo k3s kubectl -n kube-system get pods -l k8s-app=cilium --no-headers | sed 's/^/    /'" || true
+  log "now: ./dualstack-lab.sh collect upgraded ${ip} && ./dualstack-lab.sh compare dual upgraded"
+}
+
+# --- evidence collection --------------------------------------------------
+# Normalized on purpose. Two independently built clusters differ in certs,
+# tokens, UUIDs, timestamps and log lines; left raw, that noise buries the
+# handful of differences that actually mean something.
+cmd_collect() {
+  local name="${1:?usage: collect <name> [ip]}"
+  local ip="${2:-}"
+  [ -n "$ip" ] || ip="$(vm_ip "$name" 2>/dev/null || true)"
+  [ -n "$ip" ] || die "collect: give an ip for a non-standard name"
+  local out="$EVIDENCE/$name"; mkdir -p "$out"
+  log "collecting from $name ($ip) -> $out"
+
+  ssh_vm "$ip" 'sudo cat /etc/rancher/k3s/config.yaml 2>/dev/null; sudo systemctl cat k3s 2>/dev/null | grep -A30 ExecStart' \
+    > "$out/k3s-config.txt" 2>/dev/null || true
+  ssh_vm "$ip" 'sudo tr "\0" "\n" < /proc/$(pgrep -f "k3s server" | head -1)/cmdline | grep -v "^$"' \
+    > "$out/k3s-cmdline.txt" 2>/dev/null || true
+  ssh_vm "$ip" 'ip -o addr show | awk "{print \$2, \$3, \$4}"; echo ---; ip -4 route show; echo ---; ip -6 route show' \
+    > "$out/host-net.txt" 2>/dev/null || true
+  ssh_vm "$ip" 'sudo sysctl -a 2>/dev/null | grep -E "net\.ipv6\.conf\.(all|default)\.(forwarding|disable_ipv6)|net\.ipv4\.ip_forward"' \
+    > "$out/sysctl.txt" 2>/dev/null || true
+
+  local K='sudo k3s kubectl'
+  ssh_vm "$ip" "$K get servicecidr -o yaml"       > "$out/servicecidr.yaml" 2>/dev/null || true
+  ssh_vm "$ip" "$K get nodes -o yaml"             > "$out/nodes.yaml.raw"   2>/dev/null || true
+  ssh_vm "$ip" "$K get ciliumnodes -o yaml"       > "$out/ciliumnodes.yaml.raw" 2>/dev/null || true
+  ssh_vm "$ip" "$K -n kube-system get cm cilium-config -o yaml" > "$out/cilium-config.yaml.raw" 2>/dev/null || true
+  ssh_vm "$ip" "$K get svc -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,FAMILYPOLICY:.spec.ipFamilyPolicy,FAMILIES:.spec.ipFamilies,IPS:.spec.clusterIPs" \
+    > "$out/services.txt" 2>/dev/null || true
+  ssh_vm "$ip" "$K get pods -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,IPS:.status.podIPs" \
+    > "$out/podips.txt" 2>/dev/null || true
+
+  # Strip the things that differ every build regardless of configuration.
+  for f in "$out"/*.raw; do
+    [ -e "$f" ] || continue
+    sed -E \
+      -e 's/[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9:]+Z?/<TIME>/g' \
+      -e 's/(uid|resourceVersion|creationTimestamp|generation|observedGeneration): .*/\1: <X>/' \
+      -e 's/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/<UUID>/g' \
+      -e 's/(LS0tLS1|[A-Za-z0-9+\/]{60,}=*)/<B64>/g' \
+      "$f" > "${f%.raw}"
+    rm -f "$f"
+  done
+  log "collected $(ls "$out" | wc -l) artefacts"
+}
+
+cmd_compare() {
+  local a="${1:?usage: compare <a> <b>}" b="${2:?}"
+  [ -d "$EVIDENCE/$a" ] && [ -d "$EVIDENCE/$b" ] || die "collect both first"
+  echo "### semantic diff: $a (<) vs $b (>)"
+  diff -ru "$EVIDENCE/$a" "$EVIDENCE/$b" || true
+}
+
+cmd_virtdiff() {
+  local a="${1:?usage: virtdiff <a> <b>}" b="${2:?}"
+  for m in "$a" "$b"; do
+    [ "$(virsh_q domstate "$(vm_name "$m")" 2>/dev/null | head -1)" = "running" ] && \
+      die "$(vm_name "$m") is running — virt-diff needs the disks quiescent"
+  done
+  log "whole-filesystem diff (slow); noise is expected — use it to find what the collector missed"
+  sudo virt-diff -a "$(disk_of "$a")" -A "$(disk_of "$b")" \
+    | grep -vE '/(var/log|tmp|run|proc|sys)/|\.log$|/var/lib/rancher/k3s/(server/(tls|cred|db)|agent)' || true
+}
+
+cmd_down() {
+  local only="${1:-}"
+  for m in v4 dual; do
+    [ -n "$only" ] && [ "$only" != "$m" ] && continue
+    local vm; vm="$(vm_name "$m")"
+    virsh_q dominfo "$vm" >/dev/null 2>&1 || continue
+    log "removing $vm"
+    virsh_q destroy "$vm" >/dev/null 2>&1 || true
+    virsh_q undefine "$vm" --remove-all-storage >/dev/null 2>&1 || true
+  done
+}
+
+case "${1:-}" in
+  up)       shift; cmd_up "$@" ;;
+  upgrade)  shift; cmd_upgrade "$@" ;;
+  pristine) shift; cmd_pristine "$@" ;;
+  restore)  shift; cmd_restore "$@" ;;
+  collect)  shift; cmd_collect "$@" ;;
+  compare)  shift; cmd_compare "$@" ;;
+  virtdiff) shift; cmd_virtdiff "$@" ;;
+  down)     shift; cmd_down "$@" ;;
+  *) sed -n '2,30p' "$0"; exit 1 ;;
+esac

labsim/ipv6-ha-evidence/V0-baseline/state.txt

@@ -0,0 +1,43 @@
+=== 2026-09-06T14:25:25+01:00 ===
+--- HE endpoint ---
+  HE address : 192.0.2.10/32
+  he-sim     : he-sim: ipv6/ip remote 198.51.100.137 local 192.0.2.10 ttl 64 6rd-prefix 2002::/16
+  he-sim v6  : 2001:db8:1f1c:f6::1/64
+  API        : running
+  API bound  : nohost
+  API calls  : 4
+  route back : 198.51.100.0/24 via 192.168.122.63 dev eth1 
+--- 172.31.1.252 ---
+vip=172.31.1.1 holds_vip=no wan_disabled=yes wan_up=no ppp_up=no ppp_active=no may_dial=no lease_age=- dropin=yes role=backup tun=DOWN radvd=inactive
+inactive
+tun0@NONE        DOWN           203.0.113.108 <POINTOPOINT,NOARP> 
+tun0: ipv6/ip remote 192.0.2.10 local 203.0.113.108 ttl 64 tos inherit 6rd-prefix 2002::/16
+inactive
+Sep 05 23:27:45 apitest vrrp-wan[13361]: bond0.53 enable commit took 5s
+Sep 05 23:29:01 apitest vrrp-wan[16369]: GUARD: lease stale (200s > 75s; is vrrp-wan-reconcile.timer running?) -- hanging up pppoe0
+Sep 05 23:29:27 apitest vrrp-wan[17398]: MASTER: dialling pppoe0
+Sep 05 23:29:57 apitest vrrp-wan[18610]: MASTER: dialling pppoe0
+-- Boot 6efc3c9ba47f455e9668454ee6c2fc37 --
+Sep 05 23:34:48 apitest vrrp-wan[6266]: MASTER: dialling pppoe0
+Sep 05 23:34:49 apitest vrrp-wan[6422]: MASTER with bond0.53 disabled -> enabling
+Sep 05 23:34:54 apitest vrrp-wan[7130]: bond0.53 enable commit took 5s
+-- Boot 866611afd7c542d4bf8c5117978dcd4a --
+Sep 06 13:21:54 apitest vrrp-wan[544215]: not MASTER: stopping radvd (deprecates the v6 gateway)
+Sep 06 13:21:54 apitest vrrp-wan[544221]: not MASTER: bringing tun0 down
+Sep 06 13:22:01 apitest he-tunnel-follow[544377]: role is now backup
+--- 172.31.1.253 ---
+vip=172.31.1.1 holds_vip=yes wan_disabled=no wan_up=yes ppp_up=yes ppp_active=yes may_dial=yes lease_age=25 dropin=yes role=master tun=UNKNOWN radvd=active
+tun0@NONE        UNKNOWN        198.51.100.137 <POINTOPOINT,NOARP,UP,LOWER_UP> 
+tun0: ipv6/ip remote 192.0.2.10 local 198.51.100.137 ttl 64 tos inherit 6rd-prefix 2002::/16
+default nhid 111 via 2001:db8:1f1c:f6::1 dev tun0 proto static metric 20 pref medium
+active
+Sep 06 09:29:21 vyos vrrp-wan[422441]: MASTER with bond0.53 disabled -> enabling
+Sep 06 09:29:25 vyos vrrp-wan[423516]: bond0.53 enable commit took 4s
+Sep 06 13:22:01 vyos he-tunnel-follow[593096]: role is now master
+Sep 06 13:22:01 vyos he-tunnel-follow[593109]: change seen (203.0.113.108 -> 198.51.100.137) but waiting for stability (1/2)
+Sep 06 13:23:01 vyos he-tunnel-follow[594799]: HE endpoint set to 198.51.100.137 (good 198.51.100.137)
+Sep 06 13:23:01 vyos he-tunnel-follow[594805]: moved tun0 to pppoe0: src 203.0.113.108 -> 198.51.100.137, mtu 1480 -> 1472
+Sep 06 13:24:02 vyos he-tunnel-follow[595700]: in sync: tun0 via pppoe0 src 198.51.100.137 mtu 1472
+Sep 06 13:24:26 vyos he-tunnel-follow[595886]: in sync: tun0 via pppoe0 src 198.51.100.137 mtu 1472
+Sep 06 13:24:26 vyos he-tunnel-follow[595904]: in sync: tun0 via pppoe0 src 198.51.100.137 mtu 1472
+Sep 06 13:25:01 vyos he-tunnel-follow[596311]: in sync: tun0 via pppoe0 src 198.51.100.137 mtu 1472

labsim/ipv6-ha-evidence/mechanism-2026-09-06.txt

@@ -0,0 +1,23 @@
+=== mechanism evidence, labsim, 2026-09-06T14:28:11+01:00 ===
+--- 172.31.1.252 ---
+vip=172.31.1.1 holds_vip=no wan_disabled=yes wan_up=no ppp_up=no ppp_active=no may_dial=no lease_age=- dropin=yes role=backup tun=DOWN radvd=inactive
+inactive
+tun0@NONE        DOWN           203.0.113.108 <POINTOPOINT,NOARP> 
+inactive
+Sep 06 13:22:01 apitest he-tunnel-follow[544377]: role is now backup
+--- 172.31.1.253 ---
+vip=172.31.1.1 holds_vip=yes wan_disabled=no wan_up=yes ppp_up=yes ppp_active=yes may_dial=yes lease_age=26 dropin=yes role=master tun=UNKNOWN radvd=active
+tun0@NONE        UNKNOWN        198.51.100.137 <POINTOPOINT,NOARP,UP,LOWER_UP> 
+active
+Sep 06 13:25:01 vyos he-tunnel-follow[596311]: in sync: tun0 via pppoe0 src 198.51.100.137 mtu 1472
+Sep 06 13:26:01 vyos he-tunnel-follow[598274]: in sync: tun0 via pppoe0 src 198.51.100.137 mtu 1472
+Sep 06 13:27:01 vyos he-tunnel-follow[599393]: in sync: tun0 via pppoe0 src 198.51.100.137 mtu 1472
+Sep 06 13:28:01 vyos he-tunnel-follow[600596]: in sync: tun0 via pppoe0 src 198.51.100.137 mtu 1472
+--- HE endpoint ---
+  HE address : 192.0.2.10/32
+  he-sim     : he-sim: ipv6/ip remote 198.51.100.137 local 192.0.2.10 ttl 64 6rd-prefix 2002::/16
+  he-sim v6  : 2001:db8:1f1c:f6::1/64
+  API        : running
+  API bound  : nohost
+  API calls  : 5
+  route back : 198.51.100.0/24 via 192.168.122.63 dev eth1 

labsim/k8s-up.sh

@@ -0,0 +1,266 @@
+#!/bin/bash
+# A real Kubernetes cluster inside labsim, on the OVS fabric, for rehearsing
+# Cilium <-> VyOS BGP before it goes near the production routers.
+#
+# Why VMs and not k3d: the thing under test is eBGP between Cilium and VyOS
+# across the switch fabric — nodes on VLAN 2, peering with the router's bond0.2
+# leg, directly connected. k3d would put the nodes on a container bridge, which
+# is a different L2 path and would prove something else. (It also needs Docker;
+# this host has podman.)
+#
+# Why not the existing micro VMs: they are Alpine with 256 MB and 1 vCPU. k3s
+# plus Cilium needs an order of magnitude more, and a glibc distro with a stock
+# kernel that Cilium's eBPF probes are actually tested against.
+#
+# Three nodes, not two: ECMP is only meaningfully tested if a node can be
+# drained and MORE THAN ONE path survives.
+#
+# Layout (mirrors production's shape, not its addresses):
+#   labsim-k8s1   172.31.2.11   k3s server
+#   labsim-k8s2   172.31.2.12   agent
+#   labsim-k8s3   172.31.2.13   agent
+#   gateway       172.31.2.1    the VRRP VIP of the router pair under test
+#   BGP peers     172.31.2.252 / .253   the routers' real per-box addresses
+#
+# Idempotent: re-running only creates what is missing.
+#
+# Usage:
+#   ./k8s-up.sh              create/start the cluster
+#   ./k8s-up.sh --kubeconfig fetch kubeconfig to ./labsim-k8s.kubeconfig
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+source "$SCRIPT_DIR/lib.sh"
+source "$SCRIPT_DIR/ovs.sh"
+
+# --- knobs ----------------------------------------------------------------
+K8S_VLAN="${K8S_VLAN:-2}"
+K8S_PREFIX="${K8S_PREFIX:-172.31.2}"
+K8S_NODES="${K8S_NODES:-3}"
+K8S_FIRST_OCTET="${K8S_FIRST_OCTET:-11}"
+# Dual-stack, mirroring production's shape (not its addresses). VLAN 2 v6 is a
+# ULA so nothing here can leak into the real HE /48; node fd00:2::1x matches the
+# BGP peers in sim-net-config.py (K8S_NODES_V6). cluster/service v6 are ULAs too;
+# the LB pool fd61:1e00::/64 is what Cilium advertises (SERVICE_CIDR_V6 there).
+K8S_PREFIX_V6="${K8S_PREFIX_V6:-fd00:2}"        # nodes fd00:2::11/12/13, router ::252/::253
+K8S_ROUTER_V6="${K8S_ROUTER_V6:-fd00:2::252}"   # v6 next-hop (primary router bond0.2)
+CLUSTER_CIDR_V6="${CLUSTER_CIDR_V6:-fd00:42::/56}"
+SERVICE_CIDR_V6="${SERVICE_CIDR_V6:-fd00:43::/112}"
+K8S_MEM="${K8S_MEM:-4096}"          # MB — k3s + cilium + a workload
+K8S_CPUS="${K8S_CPUS:-2}"
+K8S_DISK_GB="${K8S_DISK_GB:-12}"
+K8S_TOKEN="${K8S_TOKEN:-labsim-k3s-token}"
+
+# Debian rather than Alpine: glibc, a stock kernel, and cloud-init that applies
+# network-config properly (the Alpine base in this sim notably does not).
+DEB_URL="${DEB_URL:-https://cloud.debian.org/images/cloud/trixie/latest/debian-13-genericcloud-amd64.qcow2}"
+DEB_BASE="${DEB_BASE:-$IMG_DIR/debian-13-genericcloud-amd64.qcow2}"
+
+# Same version production runs, so CRD shapes and chart flags transfer exactly.
+CILIUM_VERSION="${CILIUM_VERSION:-1.19.1}"
+
+node_name() { echo "labsim-k8s$1"; }
+node_ip()   { echo "${K8S_PREFIX}.$((K8S_FIRST_OCTET + $1 - 1))"; }
+node_ip6()  { echo "${K8S_PREFIX_V6}::$((K8S_FIRST_OCTET + $1 - 1))"; }
+
+# --- base image -----------------------------------------------------------
+ensure_base_image() {
+  if [ -f "$DEB_BASE" ]; then
+    log "base image present: $(basename "$DEB_BASE")"
+    return
+  fi
+  log "fetching Debian cloud image (~330 MB) -> $DEB_BASE"
+  sudo mkdir -p "$IMG_DIR"
+  # .tmp + mv so an interrupted download never leaves a half image that later
+  # runs treat as valid.
+  sudo curl -fsSL --retry 3 -o "${DEB_BASE}.tmp" "$DEB_URL" \
+    || die "could not fetch $DEB_URL"
+  sudo mv "${DEB_BASE}.tmp" "$DEB_BASE"
+  log "base image ready"
+}
+
+# --- cloud-init -----------------------------------------------------------
+# The server node writes the join token; agents wait for the API to answer
+# before joining, because cloud-init ordering across VMs is not guaranteed and
+# a failed join leaves an agent that never retries.
+build_k8s_seed() {
+  local iso="$1" vm="$2" ip="$3" role="$4" server_ip="$5" pubkey="$6" ip6="$7" server_ip6="$8"
+  local tmp; tmp="$(mktemp -d)"
+
+  cat > "$tmp/meta-data" <<EOF
+instance-id: $vm
+local-hostname: $vm
+EOF
+
+  cat > "$tmp/network-config" <<EOF
+version: 2
+ethernets:
+  enp1s0:
+    match:
+      name: "en*"
+    addresses: [$ip/24, $ip6/64]
+    routes:
+      - to: default
+        via: ${K8S_PREFIX}.1
+    nameservers:
+      addresses: [8.8.8.8, 1.1.1.1]
+EOF
+
+  local k3s_exec
+  if [ "$role" = "server" ]; then
+    # flannel/servicelb/traefik off: Cilium is the CNI under test, and k3s's
+    # own ServiceLB would fight Cilium for LoadBalancer addresses. Dual-stack:
+    # both families in cluster-cidr/service-cidr and a dual node-ip, mirroring
+    # production. Pod IPs come from Cilium cluster-pool (node.spec.podCIDRs is
+    # immutable), so cluster-cidr v6 just marks the cluster dual-stack.
+    k3s_exec="server --flannel-backend=none --disable-network-policy --disable=servicelb --disable=traefik --node-ip=$ip,$ip6 --tls-san=$ip --cluster-cidr=10.42.0.0/16,${CLUSTER_CIDR_V6} --service-cidr=10.43.0.0/16,${SERVICE_CIDR_V6} --cluster-init"
+  else
+    k3s_exec="agent --server https://${server_ip}:6443 --node-ip=$ip,$ip6"
+  fi
+
+  cat > "$tmp/user-data" <<EOF
+#cloud-config
+hostname: $vm
+fqdn: $vm
+users:
+  - name: debian
+    groups: [sudo]
+    shell: /bin/bash
+    sudo: ["ALL=(ALL) NOPASSWD:ALL"]
+    lock_passwd: false
+    plain_text_passwd: labsim
+    ssh_authorized_keys:
+      - $pubkey
+ssh_pwauth: true
+disable_root: false
+ssh_authorized_keys:
+  - $pubkey
+
+package_update: true
+packages: [curl, jq, iproute2, tcpdump, bird2]
+
+write_files:
+  # Cilium replaces kube-proxy and needs these; Debian cloud images ship
+  # neither loaded nor persisted.
+  - path: /etc/modules-load.d/cilium.conf
+    content: |
+      br_netfilter
+      overlay
+  - path: /etc/sysctl.d/99-k8s.conf
+    content: |
+      net.ipv4.ip_forward = 1
+      net.bridge.bridge-nf-call-iptables = 1
+
+runcmd:
+  - [ modprobe, br_netfilter ]
+  - [ modprobe, overlay ]
+  - [ sysctl, --system ]
+  - |
+    # Wait for the server's API before an agent tries to join. Without this the
+    # agent fails once and the unit backs off for minutes.
+    if [ "$role" != "server" ]; then
+      for i in \$(seq 1 60); do
+        curl -sk --max-time 3 https://${server_ip}:6443/ping >/dev/null 2>&1 && break
+        sleep 5
+      done
+    fi
+  - |
+    curl -sfL https://get.k3s.io | \
+      INSTALL_K3S_EXEC="$k3s_exec" \
+      K3S_TOKEN="$K8S_TOKEN" \
+      sh -
+EOF
+
+  sudo mkdir -p "$(dirname "$iso")"
+  sudo genisoimage -quiet -output "$iso" -volid cidata -joliet -rock \
+    "$tmp/user-data" "$tmp/meta-data" "$tmp/network-config"
+  rm -rf "$tmp"
+}
+
+# --- VM creation ----------------------------------------------------------
+create_node() {
+  local n="$1" pubkey="$2"
+  local vm; vm="$(node_name "$n")"
+  local ip; ip="$(node_ip "$n")"
+  local ip6; ip6="$(node_ip6 "$n")"
+  local role="agent"; [ "$n" -eq 1 ] && role="server"
+  local server_ip; server_ip="$(node_ip 1)"
+  local server_ip6; server_ip6="$(node_ip6 1)"
+
+  if virsh_q dominfo "$vm" >/dev/null 2>&1; then
+    local state; state="$(virsh_q domstate "$vm" 2>/dev/null | head -1 | tr -d '\n')"
+    if [ "$state" = "running" ]; then
+      log "$vm already running ($ip, $role)"
+    else
+      log "$vm exists but is $state — starting"
+      virsh_q start "$vm" >/dev/null
+    fi
+    return
+  fi
+
+  local disk="$IMG_DIR/${vm}.qcow2"
+  local seed="$IMG_DIR/${vm}-seed.iso"
+
+  log "creating $vm ($ip, $role, ${K8S_MEM}MB/${K8S_CPUS}cpu)"
+  sudo qemu-img create -q -f qcow2 -F qcow2 -b "$DEB_BASE" "$disk" "${K8S_DISK_GB}G" >/dev/null
+  build_k8s_seed "$seed" "$vm" "$ip" "$role" "$server_ip" "$pubkey" "$ip6" "$server_ip6"
+
+  # Access port on the k8s VLAN — same broadcast domain as the routers'
+  # bond0.2 leg, so BGP peering is directly connected exactly as in production.
+  sudo virt-install --connect "$LIBVIRT_URI" --name "$vm" \
+    --memory "$K8S_MEM" --vcpus "$K8S_CPUS" \
+    --disk "path=$disk,format=qcow2,bus=virtio" \
+    --disk "path=$seed,device=cdrom" \
+    --network "network=$OVS_NET,portgroup=vlan${K8S_VLAN},model=virtio" \
+    --os-variant debian12 \
+    --graphics none --noautoconsole --import >/dev/null
+}
+
+fetch_kubeconfig() {
+  local server_ip; server_ip="$(node_ip 1)"
+  local out="$SCRIPT_DIR/labsim-k8s.kubeconfig"
+  log "fetching kubeconfig from $server_ip"
+  ssh -o StrictHostKeyChecking=no -o ConnectTimeout=10 \
+      "debian@${server_ip}" "sudo cat /etc/rancher/k3s/k3s.yaml" \
+    | sed "s|127.0.0.1|${server_ip}|" > "$out"
+  chmod 600 "$out"
+  log "wrote $out"
+  log "use: KUBECONFIG=$out kubectl get nodes"
+}
+
+main() {
+  if [ "${1:-}" = "--kubeconfig" ]; then
+    fetch_kubeconfig
+    return
+  fi
+
+  require_tools
+  command -v genisoimage >/dev/null || die "genisoimage missing (dnf install genisoimage)"
+
+  local pubkey; pubkey="$(find_ssh_pubkey)"
+  log "using SSH key: ${pubkey%% *} ...${pubkey##* }"
+
+  ensure_base_image
+
+  # Select EVERY VLAN, not just the k8s one. ovs_up re-defines the libvirt
+  # network from SELECTED, so narrowing it here silently drops the portgroups
+  # for every other VLAN -- running VMs keep working (their taps are already
+  # attached) and nothing complains until the next VM cannot be attached.
+  # Observed: this deleted vlan1/3/9/10/200/51/53 and only surfaced when the
+  # ISP VMs needed vlan51 and vlan53.
+  selected_vlans
+  log "ensuring OVS fabric (all VLANs, so no portgroup is dropped)"
+  ovs_up
+
+  for n in $(seq 1 "$K8S_NODES"); do
+    create_node "$n" "$pubkey"
+  done
+
+  echo
+  log "nodes created. k3s installs on first boot (a few minutes)."
+  log "watch:   ssh debian@$(node_ip 1) 'sudo systemctl status k3s'"
+  log "then:    $0 --kubeconfig"
+  log "then install Cilium $CILIUM_VERSION and the BGP resources (see README)."
+}
+
+main "$@"

labsim/labsim-dhcp-test.sh

@@ -0,0 +1,219 @@
+#!/bin/bash
+# Prove that VyOS hands each device the address UniFi reserved for it.
+#
+# The question this answers is narrow and important: 30 of the 31 UniFi
+# reservations sit INSIDE the DHCP pool (LoT's pool is 10.0.0.11-10.0.1.254 and
+# only 10.0.0.2 falls outside it). UniFi's dhcpd tolerates that. VyOS uses kea,
+# and whether kea honours in-pool host reservations decides whether the cutover
+# silently renumbers 30 devices. That is not something to predict.
+#
+# Method: boot throwaway VMs whose MAC is a REAL production MAC, on the sim
+# VLAN, and check the address they are given. MACs are the one piece of
+# production config that transplants verbatim -- the subnet is rewritten, the
+# MAC is not -- which is what makes this a real test rather than a rehearsal.
+#
+# Safe: the ovs-labsim bridge contains only internal ports and VM taps, with no
+# physical NIC, so a production MAC here cannot reach or confuse the real LAN.
+# Verified with `ovs-vsctl show` before this script was written.
+#
+#   ./labsim-dhcp-test.sh              run the standard cases
+#   ./labsim-dhcp-test.sh --keep       leave the VMs up for inspection
+#   ./labsim-dhcp-test.sh --clean      just remove any leftover test VMs
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+source "$SCRIPT_DIR/lib.sh"
+
+ROUTER_IP="${ROUTER_IP:-172.31.1.1}"
+ROUTER_PW="${ROUTER_PW:-vyos}"
+TEST_VLAN="${TEST_VLAN:-10}"
+BOOT_WAIT="${BOOT_WAIT:-150}"
+TAG="labsim-dhcptest"
+
+# mac|expected|why. "POOL" means: must get an address from the pool and must
+# NOT get any reserved address -- the negative case that stops a pass from
+# meaning merely "DHCP works".
+CASES=(
+  "f8:0d:ac:90:65:c6|172.31.10.46|printer1 - reservation inside the pool"
+  "1c:69:20:7f:bc:77|172.31.11.67|sonoff-matter - in-pool AND across the /23 boundary"
+  "34:e1:d1:80:29:ce|172.31.10.2|Hubitat - the one reservation OUTSIDE the pool"
+  "52:54:00:ab:cd:ef|POOL|unreserved MAC - must get a pool address, not a reserved one"
+)
+
+vm_of() { echo "${TAG}-$(echo "$1" | tr -d ':')"; }
+
+cleanup_vms() {
+  local n=0
+  while read -r vm; do
+    [ -z "$vm" ] && continue
+    virsh_q destroy "$vm" >/dev/null 2>&1
+    virsh_q undefine "$vm" --remove-all-storage >/dev/null 2>&1
+    n=$((n + 1))
+  done < <(virsh_q list --all --name 2>/dev/null | grep "^${TAG}-" || true)
+  [ "$n" -gt 0 ] && log "removed $n test VM(s)"
+  sudo rm -f "$IMG_DIR/${TAG}-"*.qcow2 "$IMG_DIR/${TAG}-"*-seed.iso 2>/dev/null
+  return 0
+}
+
+# A seed that asks for DHCP instead of taking a static address. Alpine's
+# cloud-init ignores network-config here (verified previously and documented in
+# README), so /etc/network/interfaces is what actually takes effect.
+build_dhcp_seed() {
+  local iso="$1" vm="$2" pubkey="$3"
+  local tmp; tmp="$(mktemp -d)"
+  cat > "$tmp/meta-data" <<EOF
+instance-id: $vm
+local-hostname: $vm
+EOF
+  cat > "$tmp/user-data" <<EOF
+#cloud-config
+hostname: $vm
+users:
+  - name: alpine
+    shell: /bin/ash
+    lock_passwd: false
+    plain_text_passwd: labsim
+    ssh_authorized_keys:
+      - $pubkey
+ssh_authorized_keys:
+  - $pubkey
+disable_root: false
+chpasswd:
+  list: |
+    root:labsim
+  expire: false
+write_files:
+  - path: /etc/network/interfaces
+    content: |
+      auto lo
+      iface lo inet loopback
+      auto eth0
+      iface eth0 inet dhcp
+runcmd:
+  - [ sh, -c, "ifdown eth0 2>/dev/null; ifup eth0 || udhcpc -i eth0 -q || true" ]
+EOF
+  python3 - "$tmp/user-data" <<'PY' || die "generated user-data is not valid YAML"
+import sys, yaml
+yaml.safe_load(open(sys.argv[1]).read().split("#cloud-config",1)[1])
+PY
+  sudo genisoimage -quiet -output "$iso" -volid cidata -joliet -rock \
+    "$tmp/user-data" "$tmp/meta-data" >/dev/null 2>&1 || die "seed build failed"
+  rm -rf "$tmp"
+}
+
+router() {
+  timeout 30 sshpass -p "$ROUTER_PW" ssh -o StrictHostKeyChecking=no \
+    -o BatchMode=no -o ConnectTimeout=8 "vyos@$ROUTER_IP" "$@" 2>/dev/null
+}
+
+# --- argument handling ----------------------------------------------------
+KEEP=0
+case "${1:-}" in
+  --clean) cleanup_vms; exit 0 ;;
+  --keep)  KEEP=1 ;;
+  "")      ;;
+  *)       die "usage: $0 [--keep|--clean]" ;;
+esac
+
+command -v sshpass >/dev/null || die "sshpass required"
+require_tools
+[ -f "$BASE_IMAGE" ] || die "base image missing: $BASE_IMAGE (run labsim-up.sh first)"
+
+log "checking the router is serving DHCP..."
+subnets=$(router '/opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration commands | grep -c subnet-id')
+maps=$(router '/opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration commands | grep -c "static-mapping .* mac"')
+log "  router has ${subnets:-0} subnets and ${maps:-0} static-mappings"
+[ "${maps:-0}" -gt 0 ] || die "router has no static-mappings -- apply the generated config first"
+
+cleanup_vms
+
+# Flush the lease database first. This is not tidiness -- it is the condition
+# the cutover actually runs under, because kea does not inherit UniFi's leases
+# and starts empty. It also makes the test deterministic: with stale leases
+# present, kea saw the reserved address as held by "another client" (the same
+# MAC but a different client-id from a previous boot) and allocated a dynamic
+# address instead, which produced three misleading results before this existed.
+log "flushing the router's lease database (cutover starts with an empty one)"
+# Every dhcp4-leases.csv* must go, not just the main file: kea's memfile
+# backend keeps lease-file-cleanup rotations (.1/.2) and restores from them on
+# start, so truncating only the primary leaves the old leases intact.
+router 'sudo systemctl stop isc-kea-dhcp4-server;
+        sudo sh -c "rm -f /config/dhcp/dhcp4-leases.csv*";
+        sudo systemctl start isc-kea-dhcp4-server' >/dev/null
+sleep 5
+remaining="$(router '/opt/vyatta/bin/vyatta-op-cmd-wrapper show dhcp server leases' | sed -n '3,$p' | grep -c .)"
+[ "${remaining:-0}" -eq 0 ] || warn "lease table still has ${remaining} row(s) after flush"
+
+SSH_PUB="$(find_ssh_pubkey)"
+sudo mkdir -p "$IMG_DIR"
+
+# --- boot one VM per case -------------------------------------------------
+for c in "${CASES[@]}"; do
+  IFS='|' read -r mac expected why <<<"$c"
+  vm="$(vm_of "$mac")"
+  disk="$IMG_DIR/${vm}.qcow2"; seed="$IMG_DIR/${vm}-seed.iso"
+  log "booting $vm  mac=$mac  ($why)"
+  sudo qemu-img create -q -f qcow2 -F qcow2 -b "$BASE_IMAGE" "$disk" "$VM_DISK" >/dev/null
+  build_dhcp_seed "$seed" "$vm" "$SSH_PUB"
+  sudo virt-install --connect "$LIBVIRT_URI" --name "$vm" \
+    --memory "$VM_MEM" --vcpus "$VM_CPUS" \
+    --disk "path=$disk,format=qcow2,bus=virtio" \
+    --disk "path=$seed,device=cdrom,readonly=on" \
+    --network "network=labsim-ovs,portgroup=vlan${TEST_VLAN},model=virtio,mac=$mac" \
+    --os-variant alpinelinux3.18 --graphics none --noautoconsole --import >/dev/null \
+    || die "virt-install failed for $vm"
+done
+
+log "waiting ${BOOT_WAIT}s for boot + DHCP..."
+sleep "$BOOT_WAIT"
+
+# --- verdict --------------------------------------------------------------
+# The lease table on the router is the authority: it says what the server
+# decided, independent of whether the guest brought the interface up cleanly.
+leases="$(router '/opt/vyatta/bin/vyatta-op-cmd-wrapper show dhcp server leases')"
+echo
+echo "=== router lease table ==="
+echo "$leases"
+echo
+
+reserved_ips="$(cd "$SCRIPT_DIR/../migration" && python3 unifi-to-vyos.py --mode sim 2>/dev/null \
+  | awk '/static-mapping .* ip-address/ {print $NF}')"
+
+pass=0; fail=0
+printf '%-19s %-16s %-16s %s\n' "MAC" "EXPECTED" "GOT" "RESULT"
+for c in "${CASES[@]}"; do
+  IFS='|' read -r mac expected why <<<"$c"
+  # Never guess which lease is "the" lease. Taking the first match is how an
+  # hours-old lease was once reported as the current answer, turning three
+  # failures into apparent passes.
+  matches="$(echo "$leases" | awk -v m="$mac" 'tolower($2) == tolower(m) {print $1}')"
+  n_match="$(echo "$matches" | grep -c . )"
+  if [ "$n_match" -gt 1 ]; then
+    got="AMBIGUOUS($(echo "$matches" | tr '\n' ',' | sed 's/,$//'))"
+  else
+    got="${matches:-<none>}"
+  fi
+  if [ "${got#AMBIGUOUS}" != "$got" ]; then
+    # More than one lease for this MAC means the flush did not take. Any
+    # verdict from here is a guess, so refuse to give one.
+    result="FAIL (multiple leases -- flush did not take)"
+  elif [ "$expected" = "POOL" ]; then
+    if [ "$got" = "<none>" ]; then
+      result="FAIL (no lease at all)"
+    elif echo "$reserved_ips" | grep -qx "$got"; then
+      result="FAIL (got a RESERVED address)"
+    else
+      result="pass"
+    fi
+  else
+    [ "$got" = "$expected" ] && result="pass" || result="FAIL"
+  fi
+  [ "$result" = "pass" ] && pass=$((pass + 1)) || fail=$((fail + 1))
+  printf '%-19s %-16s %-16s %s\n' "$mac" "$expected" "$got" "$result"
+  printf '  %s\n' "$why"
+done
+
+echo
+log "$pass passed, $fail failed"
+[ "$KEEP" -eq 1 ] && log "VMs left running (--keep). Remove with: $0 --clean" || cleanup_vms
+[ "$fail" -eq 0 ] || exit 1

labsim/labsim-down.sh

@@ -0,0 +1,61 @@
+#!/bin/bash
+# Tear down the lab network simulation.
+#
+# By default this destroys VMs and networks but KEEPS the downloaded base
+# image, so the next bring-up is fast. Pass --purge to remove that too.
+#
+# Usage: ./labsim-down.sh [--purge] [vlan-id ...]
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+source "$SCRIPT_DIR/lib.sh"
+source "$SCRIPT_DIR/ovs.sh"
+
+PURGE=false
+ARGS=()
+for a in "$@"; do
+  case "$a" in
+    --purge) PURGE=true ;;
+    *) ARGS+=("$a") ;;
+  esac
+done
+
+selected_vlans "${ARGS[@]+"${ARGS[@]}"}"
+
+for entry in "${SELECTED[@]}"; do
+  IFS=: read -r vid name prefix _r <<<"$entry"
+  vm="$(vm_name "$vid" "$name")"
+
+  if virsh_q dominfo "$vm" >/dev/null 2>&1; then
+    log "destroying VM $vm"
+    virsh_q destroy "$vm" >/dev/null 2>&1 || true
+    virsh_q undefine "$vm" --nvram >/dev/null 2>&1 || virsh_q undefine "$vm" >/dev/null 2>&1 || true
+  fi
+  sudo rm -f "$IMG_DIR/${vm}.qcow2" "$IMG_DIR/${vm}-seed.iso"
+
+done
+
+# Legacy per-VLAN Linux-bridge networks from before the OVS migration. If
+# these survive they keep a duplicate <prefix>.2/24 on a dead bridge, and the
+# kernel may prefer that route over the OVS host leg — which looks exactly
+# like "the VM is unreachable" while ping -I hostvN works fine.
+for entry in "${SELECTED[@]}"; do
+  IFS=: read -r vid _n _p _r <<<"$entry"
+  legacy="labsim-vlan${vid}"
+  if virsh_q net-info "$legacy" >/dev/null 2>&1; then
+    log "removing legacy network $legacy"
+    virsh_q net-destroy "$legacy" >/dev/null 2>&1 || true
+    virsh_q net-undefine "$legacy" >/dev/null 2>&1 || true
+  fi
+done
+
+log "removing OVS fabric"
+ovs_down
+
+if [ "$PURGE" = true ]; then
+  log "purging base image $BASE_IMAGE"
+  sudo rm -f "$BASE_IMAGE"
+  sudo rmdir "$IMG_DIR" 2>/dev/null || true
+fi
+
+log "environment is DOWN"

labsim/labsim-dualstack-convert.sh

@@ -0,0 +1,116 @@
+#!/bin/bash
+# Rehearse the k3s single -> dual-stack conversion on the 3-server etcd cluster,
+# the way production Phase 4 will do it: edit each server's config.yaml (rendered
+# by the PRODUCTION generator) to add the second address family, restart ONE
+# server at a time, and watch what happens in between.
+#
+# Answers the questions a single-node lab cannot:
+#   - does quorum survive a rolling config.yaml change across 3 etcd servers?
+#   - what does a MIXED control plane do (one server dual, two still v4-only)?
+#   - does ServiceCIDR pick up the v6 range on the FIRST server's restart, or
+#     only once all three agree?
+#
+# The node-ip v6 must exist on the box before k3s reads it, so each server first
+# gets a ULA on its interface (fd00:2::3x), mirroring how production nodes get a
+# DHCPv6 address before k3s starts.
+#
+#   ./labsim-dualstack-convert.sh baseline   snapshot the v4-only starting state
+#   ./labsim-dualstack-convert.sh convert    roll the conversion, snapshotting each step
+#   ./labsim-dualstack-convert.sh snapshot   print current SC / podCIDRs / quorum
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+NET="${NET:-172.31.2}"; FIRST_OCTET="${FIRST_OCTET:-31}"; SERVERS="${SERVERS:-3}"
+TOKEN="${TOKEN:-labsim-etcd-token}"
+RENDER="${RENDER:-$SCRIPT_DIR/../bastion/src/modules/dist/modules/k3s/bin/render-config.js}"
+EVID="${EVID:-$SCRIPT_DIR/dualstack-evidence}"
+
+# Dual-stack target ranges. ULA/v4 -- the mechanism is what's under test, not the
+# addresses; using ULA keeps sim traffic out of the real /48.
+V4_CLUSTER="10.42.0.0/16"; V6_CLUSTER="fd00:42::/56"
+V4_SERVICE="10.43.0.0/16"; V6_SERVICE="fd00:43::/112"   # /112: apiserver caps v6 service ranges
+V6_NODE_PREFIX="fd00:2"                                  # node-ip v6: fd00:2::3x
+
+SSH=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -o ConnectTimeout=8 -o BatchMode=yes)
+node_ip()   { echo "${NET}.$((FIRST_OCTET + $1 - 1))"; }
+node_v6()   { echo "${V6_NODE_PREFIX}::$((FIRST_OCTET + $1 - 1))"; }
+node_name() { echo "labsim-etcd$1"; }
+s() { local n="$1"; shift; timeout "${TMO:-25}" ssh "${SSH[@]}" "debian@$(node_ip "$n")" "$@" 2>/dev/null; }
+k() { s 1 "sudo k3s kubectl $*"; }
+log() { printf '\033[36m==>\033[0m %s\n' "$*"; }
+
+snapshot() {
+  # custom-columns, not jsonpath: the jsonpath range/quotes get mangled through
+  # ssh -> sudo -> kubectl and came back empty.
+  echo "  servicecidr :"
+  s 1 'sudo k3s kubectl get servicecidr -o custom-columns=NAME:.metadata.name,CIDRS:.spec.cidrs --no-headers' 2>/dev/null | sed 's/^/    /'
+  echo "  node podCIDRs:"
+  s 1 'sudo k3s kubectl get nodes -o custom-columns=NAME:.metadata.name,PODCIDRS:.spec.podCIDRs --no-headers' 2>/dev/null | sed 's/^/    /'
+  local ready; ready="$(k get nodes --no-headers 2>/dev/null | wc -l)"
+  echo "  nodes registered: $ready ; apiserver: $(k get --raw /readyz >/dev/null 2>&1 && echo ok || echo DOWN)"
+}
+
+# Re-render node n's config.yaml WITH the dual families and write it back.
+convert_node() {
+  local n="$1"
+  local ip; ip="$(node_ip "$n")"; local v6; v6="$(node_v6 "$n")"
+  log "server $n ($ip): add $v6, re-render dual config.yaml, restart k3s"
+
+  # 1. node-ip v6 must exist before k3s reads it.
+  s "$n" "sudo ip -6 addr add ${v6}/64 dev enp1s0 2>/dev/null; ip -6 -br addr show enp1s0 | grep -o '${v6}/64'" | sed 's/^/    addr: /'
+
+  # 2. render the dual config from the PRODUCTION generator, preserving this
+  #    node's role (node 1 cluster-init, else joining server).
+  local extra=""
+  [ "$n" -ne 1 ] && extra="K3S_SERVER_URL=https://$(node_ip 1):6443 K3S_TOKEN=$TOKEN"
+  local cfg
+  cfg="$(env ROLE=infra HOSTNAME="$(node_name "$n")" IP="$ip" TLS_SANS="$ip" \
+      IPV6="$v6" CLUSTER_CIDR="${V4_CLUSTER},${V6_CLUSTER}" SERVICE_CIDR="${V4_SERVICE},${V6_SERVICE}" \
+      $extra node "$RENDER")"
+  # sanity: the render must actually carry both families or the restart is pointless
+  echo "$cfg" | grep -q "$V6_CLUSTER" || { echo "    RENDER MISSING v6 -- aborting"; return 1; }
+
+  # 3. write it back and restart k3s on this one server.
+  printf '%s\n' "$cfg" | s "$n" "sudo tee /etc/rancher/k3s/config.yaml >/dev/null && sudo systemctl restart k3s"
+
+  # 4. wait for THIS server's k3s to come back and the apiserver to answer.
+  local i
+  for i in $(seq 1 30); do
+    [ "$(s "$n" 'sudo systemctl is-active k3s' 2>/dev/null)" = active ] && \
+      s "$n" 'sudo k3s kubectl get --raw /readyz >/dev/null 2>&1' && break
+    sleep 10
+  done
+  echo "    server $n k3s=$(s "$n" 'systemctl is-active k3s')  restarts=$(s "$n" 'systemctl show k3s -p NRestarts --value')"
+}
+
+cmd_baseline() {
+  mkdir -p "$EVID"
+  { echo "=== BASELINE (v4-only) $(date -u +%FT%TZ) ==="; snapshot; } | tee "$EVID/convert-baseline.txt"
+}
+
+cmd_snapshot() { snapshot; }
+
+cmd_convert() {
+  mkdir -p "$EVID"
+  local out="$EVID/convert-run.txt"
+  {
+    echo "=== 3-SERVER DUAL-STACK CONVERSION $(date -u +%FT%TZ) ==="
+    echo "--- before ---"; snapshot
+    local n
+    for n in $(seq 1 "$SERVERS"); do
+      echo; echo "### converting server $n of $SERVERS ###"
+      convert_node "$n" || { echo "convert_node $n failed"; break; }
+      echo "--- state after server $n (MIXED until n=$SERVERS) ---"
+      snapshot
+    done
+    echo; echo "--- FINAL ---"; snapshot
+  } 2>&1 | tee "$out"
+  log "evidence -> $out"
+}
+
+case "${1:-snapshot}" in
+  baseline) cmd_baseline ;;
+  convert)  cmd_convert ;;
+  snapshot) cmd_snapshot ;;
+  *) echo "usage: $0 {baseline|convert|snapshot}" >&2; exit 2 ;;
+esac

labsim/labsim-dualstack-net.sh

@@ -0,0 +1,221 @@
+#!/bin/bash
+# VLAN 2 gets IPv6 in labsim: addresses, router advertisements and a DHCPv6
+# server with per-MAC reservations.
+#
+# This is the rehearsal of the production change (dual-stack plan, phase 2b) and
+# runs the SAME VyOS config, against the sim router pair, so the production
+# apply is a repeat rather than a first attempt.
+#
+# WHY DHCPv6 AND NOT SLAAC. The cluster needs each node's IPv6 to be knowable in
+# advance and stable: k3s resolves node-ip once at start-up, and a node's
+# identity cannot be allowed to change under it. The estate already answers that
+# question for IPv4 with kea reservations keyed on MAC, so IPv6 answers it the
+# same way and stays one source of truth.
+#
+# DHCPv6 normally keys on DUID, not MAC -- a DUID is generated by the client and
+# is not derivable from its MAC, which would have meant a second, client-owned
+# source of truth. VyOS's static-mapping accepts `mac` as well as `duid`
+# (verified on the sim: the node.tag directory offers duid, mac, ipv6-address,
+# ipv6-prefix), so the reservation can key on the same MAC the v4 one does.
+#
+# The RA carries managed-flag WITH no-autonomous-flag. That combination is what
+# makes the node's address unambiguous: managed sends it to DHCPv6, and
+# non-autonomous stops it also forming a SLAAC address from the same prefix.
+# Leave autonomous on and every node has two global addresses, only one of which
+# anybody reserved -- and whichever labctl happens to find is the one that ends
+# up in node-ip.
+#
+# Both are VALUELESS nodes on this VyOS: `managed-flag` not `managed-flag true`,
+# and `prefix <p> no-autonomous-flag` not `autonomous-flag false`. The value
+# forms are rejected with "is not valid".
+#
+# A FAILED COMMIT DOES NOT MEAN NOTHING CHANGED. VyOS commits node groups
+# independently, so an earlier run of this script left the bond0.2 address and
+# most of the router-advert block applied while `[[service dhcpv6-server]]
+# failed` -- the interface and RA groups had already succeeded. Re-read the
+# config after any failure rather than assuming a clean rollback; that matters
+# more in production than here.
+#
+# TWO THINGS THIS REHEARSAL FOUND, both of which would have bitten production:
+#
+# 1. managed-flag is NECESSARY BUT NOT SUFFICIENT. It only tells the host to use
+#    DHCPv6; the kernel's accept_ra implements SLAAC and nothing else, so a node
+#    with no DHCPv6 *client* running takes no lease at all. The sim's Debian
+#    nodes have neither NetworkManager nor a networkd .network file managing the
+#    interface, so they ended up with a kernel SLAAC address
+#    (2001:db8:187e:2:5054:ff:fe53:731 -- EUI-64 from the MAC) and never asked
+#    for the reserved ::11. Production's Fedora nodes use NetworkManager, which
+#    does run a DHCPv6 client on the managed flag, and the DGX Sparks run
+#    NetworkManager too -- but that must be VERIFIED per node, not assumed. The
+#    k3s module's preflight catches the consequence; the fix is node-side.
+#
+# 2. TURNING AUTONOMOUS OFF DOES NOT RETRACT ADDRESSES ALREADY FORMED. An
+#    earlier partial apply advertised the prefix while autonomous was still on;
+#    the nodes autoconfigured, and adding no-autonomous-flag afterwards left
+#    those addresses in place with a 30-day valid lifetime. So in production,
+#    where VLAN 2 has no IPv6 at all yet, no-autonomous-flag MUST be in the same
+#    commit that first advertises the prefix. Advertise first and tighten later
+#    and every node carries an unreserved EUI-64 address that labctl might pick
+#    up as node-ip.
+#
+#   ./labsim-dualstack-net.sh up       apply to both sim routers
+#   ./labsim-dualstack-net.sh down     remove it again
+#   ./labsim-dualstack-net.sh status   what the routers and nodes think
+#   ./labsim-dualstack-net.sh leases   who has taken an address
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+R1="${R1:-172.31.1.252}"   # sim vyos001 -> ::1
+R2="${R2:-172.31.1.253}"   # sim vyos002 -> ::2
+PW="${VYOS_PW:-vyos}"
+
+# Mirrors production's scheme (2001:470:187e:<vlan>::/64) using the
+# documentation prefix, so the shape is rehearsed without putting sim addresses
+# inside a range production announces.
+V6_PREFIX="${V6_PREFIX:-2001:db8:187e:2}"
+LINK_MTU="${LINK_MTU:-1472}"
+# VyOS requires a unique subnet-id per DHCPv6 subnet ("Unique subnet ID not
+# specified for subnet"). Using the VLAN id keeps it self-documenting and
+# collision-free across VLANs.
+VLAN_ID="${VLAN_ID:-2}"
+
+# node -> last hextet. Mirrors the v4 host part (172.31.2.11 -> ::11) so a
+# reservation is readable next to its IPv4 twin.
+NODES=(labsim-k8s1:11 labsim-k8s2:12 labsim-k8s3:13)
+
+SSH=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
+     -o LogLevel=ERROR -o ConnectTimeout=6 -o PreferredAuthentications=password)
+
+log()  { printf '\033[0;36m[ds-net]\033[0m %s\n' "$*"; }
+warn() { printf '\033[1;33m[ds-net]\033[0m %s\n' "$*" >&2; }
+die()  { printf '\033[0;31m[ds-net]\033[0m %s\n' "$*" >&2; exit 1; }
+
+# Drive VyOS from a script FILE, never `vbash -c`. The latter never starts a
+# config session: commit fails to stderr, a helper discards it, and the run
+# reports success having changed nothing. labsim-pppoe-ha-test.sh lost a whole
+# policy matrix to exactly that.
+vyos_apply() {  # host, set-lines on stdin
+    local h="$1" out
+    out="$({ printf '#!/bin/vbash\nsource /opt/vyatta/etc/functions/script-template\nconfigure\n'
+             cat
+             printf 'commit\nsave\nexit\n'
+           } | timeout 120 sshpass -p "$PW" ssh "${SSH[@]}" "vyos@$h" \
+                 'cat > /tmp/ds-net.sh && chmod +x /tmp/ds-net.sh && sudo /tmp/ds-net.sh' 2>&1)"
+    # Read the outcome instead of assuming it. The first version of this script
+    # printed "up" after BOTH routers had failed to commit -- the same shape of
+    # lie this repo has been bitten by before (a matrix reporting coverage it did
+    # not have). vbash exits 0 even when the commit fails, so the text is the
+    # only honest signal.
+    printf '%s\n' "$out" | grep -vE '^\s*$' | sed 's/^/      /' | tail -6
+    if printf '%s' "$out" | grep -qiE 'Commit failed|\[\[.*\]\] failed|Set failed'; then
+        return 1
+    fi
+    return 0
+}
+
+r() { timeout 40 sshpass -p "$PW" ssh "${SSH[@]}" "vyos@$1" "${@:2}" 2>/dev/null; }
+
+mac_of() { sudo virsh domiflist "$1" 2>/dev/null | awk '/52:54/{print $5; exit}'; }
+
+up() {
+    log "collecting node MACs (the reservation key, same as IPv4)"
+    local mappings="" name hextet mac
+    for entry in "${NODES[@]}"; do
+        name="${entry%%:*}"; hextet="${entry##*:}"
+        mac="$(mac_of "$name")"
+        [ -n "$mac" ] || die "no MAC for $name -- is the sim cluster up? (./k8s-up.sh)"
+        log "    $name $mac -> ${V6_PREFIX}::${hextet}"
+        # No trailing newline: `${mappings}` sits on its own line in the heredoc
+        # below and supplies its own. A heredoc terminator is matched LITERALLY
+        # in the source before any expansion, so `${mappings}EOF` is not the
+        # delimiter -- the heredoc ran on, swallowed the rest of this function
+        # and part of down(), and the apply failed with "Invalid command: [EOF]".
+        [ -n "$mappings" ] && mappings+=$'\n'
+        mappings+="set service dhcpv6-server shared-network-name K8S subnet ${V6_PREFIX}::/64 static-mapping ${name} mac '${mac}'
+set service dhcpv6-server shared-network-name K8S subnet ${V6_PREFIX}::/64 static-mapping ${name} ipv6-address '${V6_PREFIX}::${hextet}'"
+    done
+
+    local host pref hextet_self
+    for host in "$R1" "$R2"; do
+        if [ "$host" = "$R1" ]; then pref=high; hextet_self=1; else pref=low; hextet_self=2; fi
+        log "applying to $host (router ${V6_PREFIX}::${hextet_self}, RA preference $pref)"
+        vyos_apply "$host" <<EOF || die "commit failed on $host -- nothing applied there"
+set interfaces bonding bond0 vif 2 address '${V6_PREFIX}::${hextet_self}/64'
+set service router-advert interface bond0.2 prefix ${V6_PREFIX}::/64 valid-lifetime '2592000'
+set service router-advert interface bond0.2 prefix ${V6_PREFIX}::/64 preferred-lifetime '604800'
+set service router-advert interface bond0.2 prefix ${V6_PREFIX}::/64 no-autonomous-flag
+set service router-advert interface bond0.2 managed-flag
+set service router-advert interface bond0.2 link-mtu '${LINK_MTU}'
+set service router-advert interface bond0.2 default-preference '${pref}'
+set service dhcpv6-server shared-network-name K8S subnet ${V6_PREFIX}::/64 subnet-id '${VLAN_ID}'
+${mappings}
+EOF
+    done
+    log "up. Nodes need a DHCPv6 client on their VLAN 2 interface -- see 'status'."
+}
+
+down() {
+    local host
+    for host in "$R1" "$R2"; do
+        log "removing from $host"
+        vyos_apply "$host" <<EOF
+delete service dhcpv6-server
+delete service router-advert interface bond0.2
+delete interfaces bonding bond0 vif 2 address '${V6_PREFIX}::$([ "$host" = "$R1" ] && echo 1 || echo 2)/64'
+EOF
+    done
+    log "down"
+}
+
+status() {
+    local host
+    for host in "$R1" "$R2"; do
+        printf '  --- %s ---\n' "$host"
+        printf '    bond0.2 v6 : %s\n' "$(r "$host" 'ip -6 -br addr show bond0.2 | tr -s " "' || echo '<unreachable>')"
+        printf '    radvd      : %s\n' "$(r "$host" 'systemctl is-active radvd')"
+        # ps|grep, not pgrep: the bracket idiom that stops a self-match gets
+        # mangled through ssh -> vbash quoting and reported "NOT running" for a
+        # daemon that was plainly up.
+        printf '    kea-dhcp6  : %s\n' "$(r "$host" 'c=$(ps -ef | grep -c "[k]ea-dhcp6"); [ "$c" -gt 0 ] && echo running || echo "NOT running"')"
+        # radvd is expected INACTIVE on the backup: vrrp-wan-reconcile's IPv6
+        # plane stops it there so only the master advertises on any VLAN. Not a
+        # fault -- see the step-0 IPv6-follows-master work.
+        printf '    role       : %s\n' "$(r "$host" 'sudo /config/vrrp-wan-reconcile --status 2>/dev/null | grep -o "role=[a-z]*"')"
+    done
+    printf '  --- nodes ---\n'
+    local entry name
+    for entry in "${NODES[@]}"; do
+        name="${entry%%:*}"
+        printf '    %-14s %s\n' "$name" "$(node_v6 "$name")"
+    done
+}
+
+# What global IPv6 does the node actually hold? This is the question labctl asks
+# before it will write node-ip, so ask it the same way.
+node_v6() {
+    # Ask the node, not the hypervisor: virsh domifaddr --source agent needs
+    # qemu-guest-agent, which these images do not carry, and returned nothing
+    # while the nodes plainly had addresses.
+    local name="$1" v4 out
+    case "$name" in *k8s1) v4=172.31.2.11 ;; *k8s2) v4=172.31.2.12 ;; *k8s3) v4=172.31.2.13 ;; *) echo "<unknown>"; return ;; esac
+    out="$(timeout 20 ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
+             -o LogLevel=ERROR -o ConnectTimeout=6 -o BatchMode=yes "debian@$v4" \
+             'ip -6 -br addr show scope global 2>/dev/null | tr -s " "' 2>/dev/null)"
+    echo "${out:-<unreachable>}"
+}
+
+leases() {
+    local host
+    for host in "$R1" "$R2"; do
+        printf '  --- %s ---\n' "$host"
+        r "$host" '/opt/vyatta/bin/vyatta-op-cmd-wrapper show dhcpv6-server leases 2>/dev/null || echo "    (no leases command / no leases)"'
+    done
+}
+
+case "${1:-status}" in
+    up)     up ;;
+    down)   down ;;
+    status) status ;;
+    leases) leases ;;
+    *) die "usage: $0 {up|down|status|leases}" ;;
+esac

labsim/labsim-exporter.py

@@ -0,0 +1,157 @@
+#!/usr/bin/env python3
+"""Prometheus exporter for the labsim connectivity matrix.
+
+Runs the same sweep as labsim-matrix.py on an interval and exposes it as
+metrics, so Grafana can show the mesh as a heatmap and — more usefully — a
+history of exactly when a cell flipped after a firewall change.
+
+  labsim_reachable{src,dst,proto}   1 = reachable, 0 = blocked
+  labsim_sweep_seconds              how long the last sweep took
+  labsim_sweep_total                sweeps completed since start
+  labsim_up                         1 while the exporter is alive
+
+Deliberately stdlib-only (http.server + threads): this runs on the workstation
+next to libvirt, and adding a dependency to watch a lab network is silly.
+
+  ./labsim-exporter.py --port 9101 --interval 15
+"""
+from __future__ import annotations
+
+import argparse
+import http.server
+import json
+import os
+import threading
+import time
+
+import labsim_matrix_lib as m  # thin import shim, see below
+
+
+class Collector:
+    def __init__(self, interval: int, timeout: int) -> None:
+        self.interval = interval
+        self.timeout = timeout
+        self.vlans = m.load_vlans()
+        self.lock = threading.Lock()
+        self.results: dict = {}
+        self.duration = 0.0
+        self.sweeps = 0
+
+    def loop(self) -> None:
+        while True:
+            started = time.time()
+            try:
+                results = m.sweep(self.vlans, self.timeout)
+                with self.lock:
+                    self.results = results
+                    self.duration = time.time() - started
+                    self.sweeps += 1
+            except Exception:  # noqa: BLE001 - never let the loop die
+                pass
+            time.sleep(max(1.0, self.interval - (time.time() - started)))
+
+    def snapshot(self) -> dict:
+        """Everything the topology page needs, in one JSON payload."""
+        with self.lock:
+            results, duration = dict(self.results), self.duration
+        reach = total = 0
+        for data in results.values():
+            if "__error__" in data:
+                continue
+            for protos in data.values():
+                for proto, ok in protos.items():
+                    if proto == "rtt_ms":
+                        continue
+                    total += 1
+                    if ok:
+                        reach += 1
+        return {"vlans": self.vlans, "results": results, "reachable": reach,
+                "total": total, "sweep_seconds": duration}
+
+    def render(self) -> str:
+        with self.lock:
+            results, duration, sweeps = dict(self.results), self.duration, self.sweeps
+
+        out = [
+            "# HELP labsim_reachable 1 if dst is reachable from src over proto",
+            "# TYPE labsim_reachable gauge",
+        ]
+        rtts = []
+        for src, data in results.items():
+            if "__error__" in data:
+                continue
+            for dst, protos in data.items():
+                for proto, ok in protos.items():
+                    if proto == "rtt_ms":
+                        if isinstance(ok, (int, float)):
+                            rtts.append((src, dst, ok))
+                        continue
+                    out.append(
+                        f'labsim_reachable{{src="{src}",dst="{dst}",proto="{proto}"}} {1 if ok else 0}')
+        out += ["# HELP labsim_rtt_ms ICMP round-trip time",
+                "# TYPE labsim_rtt_ms gauge"]
+        for src, dst, val in rtts:
+            out.append(f'labsim_rtt_ms{{src="{src}",dst="{dst}"}} {val}')
+        out += [
+            "# HELP labsim_sweep_seconds duration of the last sweep",
+            "# TYPE labsim_sweep_seconds gauge",
+            f"labsim_sweep_seconds {duration:.3f}",
+            "# HELP labsim_sweep_total sweeps completed",
+            "# TYPE labsim_sweep_total counter",
+            f"labsim_sweep_total {sweeps}",
+            "# HELP labsim_up exporter liveness",
+            "# TYPE labsim_up gauge",
+            "labsim_up 1",
+        ]
+        return "\n".join(out) + "\n"
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--port", type=int, default=9101)
+    ap.add_argument("--interval", type=int, default=15)
+    ap.add_argument("--timeout", type=int, default=30)
+    args = ap.parse_args()
+
+    collector = Collector(args.interval, args.timeout)
+    threading.Thread(target=collector.loop, daemon=True).start()
+
+    here = os.path.dirname(os.path.abspath(__file__))
+
+    class Handler(http.server.BaseHTTPRequestHandler):
+        def _send(self, body: bytes, ctype: str) -> None:
+            self.send_response(200)
+            self.send_header("Content-Type", ctype)
+            self.send_header("Content-Length", str(len(body)))
+            self.send_header("Cache-Control", "no-store")
+            self.end_headers()
+            self.wfile.write(body)
+
+        def do_GET(self) -> None:  # noqa: N802 - stdlib API
+            path = self.path.split("?")[0].rstrip("/")
+            if path in ("", "/topology"):
+                # Live topology view — the thing you actually watch.
+                try:
+                    with open(os.path.join(here, "topology.html"), "rb") as fh:
+                        self._send(fh.read(), "text/html; charset=utf-8")
+                except OSError:
+                    self.send_error(500, "topology.html missing")
+            elif path == "/api/matrix":
+                self._send(json.dumps(collector.snapshot()).encode(), "application/json")
+            elif path == "/metrics":
+                self._send(collector.render().encode(), "text/plain; version=0.0.4")
+            else:
+                self.send_error(404)
+
+        def log_message(self, *_args) -> None:  # keep the console quiet
+            return
+
+    srv = http.server.ThreadingHTTPServer(("0.0.0.0", args.port), Handler)
+    print(f"labsim topology  http://localhost:{args.port}/")
+    print(f"labsim metrics   http://localhost:{args.port}/metrics  (sweep every {args.interval}s)")
+    srv.serve_forever()
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

labsim/labsim-he-endpoint.sh

@@ -0,0 +1,280 @@
+#!/bin/bash
+# Build a fake Hurricane Electric 6in4 endpoint inside labsim.
+#
+# WHY THIS EXISTS
+# The production HE tunnel was never rehearsed. The override that introduced it
+# says so in its own reason text -- "the sim has no public IPv4 and no HE
+# endpoint, so there is nothing to tunnel to" -- and that gap is why IPv6 was
+# the one half of the WAN story with no matrix behind it. Then the WAN became
+# HA and IPv6 did not follow, which nobody caught, because nothing tests it.
+#
+# THE PROBLEM THIS HAD TO SOLVE
+# The sim's two WANs are isolated islands. Verified:
+#
+#     203.0.113.1 from 203.0.113.107 : OK          <- 10 gig analogue
+#     203.0.113.1 from 198.51.100.137 : unreachable
+#     198.51.100.1 from 198.51.100.137 : OK        <- PPPoE analogue
+#     198.51.100.1 from 203.0.113.107 : unreachable
+#
+# A 6in4 tunnel has ONE remote address, and production never changes it -- so an
+# endpoint reachable over only one WAN could not rehearse the case that matters
+# most: the WITHIN-box fall back from the 10 gig to PPPoE, where he-tunnel-follow
+# re-points the tunnel and calls the HE API. That is exactly where the 2026-09-06
+# near-miss lived.
+#
+# So the sim needs a minimal "internet": both ISP boxes already sit on the
+# libvirt default network (192.168.122.0/24) and both forward, so that becomes
+# the backbone, and HE lives on a single address behind it, reachable over
+# either WAN. No new VMs, no new networks.
+#
+#     192.0.2.10  "HE"  -- on isp-dhcp, reached from the PPPoE island via
+#                          192.168.122.136, and directly from the 10 gig island
+#
+# EVERYTHING HERE IS KERNEL-LEVEL, not VyOS config. The ISP boxes are scaffold,
+# not the thing under test: `ip` commands leave no config to drift, no commit to
+# fail, and a reboot cleans up. The ROUTER side is deliberately the opposite --
+# it goes through real VyOS config, because "will VyOS commit a tunnel whose
+# source-address does not exist on this box?" is one of the questions.
+#
+#   ./labsim-he-endpoint.sh up        build it
+#   ./labsim-he-endpoint.sh down      tear it down
+#   ./labsim-he-endpoint.sh status    what is live
+#   ./labsim-he-endpoint.sh calls     how many times the HE API was called
+#   ./labsim-he-endpoint.sh point IP  point the endpoint by hand (sim bookkeeping)
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+DHCP_ISP="${DHCP_ISP:-192.168.122.136}"   # owns the 10 gig segment, hosts "HE"
+PPPOE_ISP="${PPPOE_ISP:-192.168.122.63}"  # owns the PPPoE segment
+PW="${VYOS_PW:-vyos}"
+
+# TEST-NET-1 for the endpoint, and the documentation prefix for v6. Production
+# uses 2001:470:187e::/48 from HE; the sim mirrors its SHAPE
+# (2001:db8:187e:<vlan>::/64) so the scheme is exercised, not just the tunnel.
+HE_ADDR="${HE_ADDR:-192.0.2.10}"
+HE_LINK6="${HE_LINK6:-2001:db8:1f1c:f6::1}"   # HE side of the tunnel /64
+RT_LINK6="${RT_LINK6:-2001:db8:1f1c:f6::2}"   # router side
+SITE6="${SITE6:-2001:db8:187e::/48}"          # routed to the router side
+TENGIG_NET="${TENGIG_NET:-203.0.113.0/24}"
+# Any valid address that will never be a router WAN -- see the tunnel creation
+# below for why this cannot be 0.0.0.0.
+PLACEHOLDER_REMOTE="${PLACEHOLDER_REMOTE:-203.0.113.1}"
+PPPOE_NET="${PPPOE_NET:-198.51.100.0/24}"
+
+SSH=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
+     -o LogLevel=ERROR -o ConnectTimeout=6 -o PreferredAuthentications=password)
+dhcp_isp()  { timeout 40 sshpass -p "$PW" ssh "${SSH[@]}" "vyos@$DHCP_ISP" "$@" 2>/dev/null; }
+pppoe_isp() { timeout 40 sshpass -p "$PW" ssh "${SSH[@]}" "vyos@$PPPOE_ISP" "$@" 2>/dev/null; }
+
+log()  { printf '\033[0;36m[he-sim]\033[0m %s\n' "$*"; }
+die()  { printf '\033[0;31m[he-sim]\033[0m %s\n' "$*" >&2; exit 1; }
+
+# --- the stub tunnelbroker API ---------------------------------------------
+# HE's real endpoint is a dyndns-style updater that re-points the tunnel's remote
+# address. This is that, in 40 lines, so the sim can exercise the HE-SIDE half of
+# a failover -- the half production can never safely test.
+#
+# It logs every call to /run/he-sim-api.log, which is what lets the matrix assert
+# the invariant that matters: a ROUTER-level failover must call this ZERO times,
+# because the 10 gig address follows the cloned MAC to the other box unchanged.
+API_PY='
+import http.server, subprocess, urllib.parse, datetime, sys, re
+
+TUN  = "he-sim"
+LOCAL    = sys.argv[1] if len(sys.argv) > 1 else "192.0.2.10"
+V6_LOCAL = sys.argv[2] if len(sys.argv) > 2 else "2001:db8:1f1c:f6::1/64"
+V6_PEER  = sys.argv[3] if len(sys.argv) > 3 else "2001:db8:1f1c:f6::2"
+SITE6    = sys.argv[4] if len(sys.argv) > 4 else "2001:db8:187e::/48"
+LOG  = "/run/he-sim-api.log"
+
+def note(msg):
+    with open(LOG, "a") as f:
+        f.write("%s %s\n" % (datetime.datetime.now().isoformat(timespec="seconds"), msg))
+
+def current_remote():
+    out = subprocess.run(["ip", "tunnel", "show", TUN], capture_output=True, text=True).stdout
+    m = re.search(r"remote ([0-9.]+)", out)
+    return m.group(1) if m else None
+
+class H(http.server.BaseHTTPRequestHandler):
+    def reply(self, body):
+        b = body.encode()
+        self.send_response(200)
+        self.send_header("Content-Type", "text/plain")
+        self.send_header("Content-Length", str(len(b)))
+        self.end_headers()
+        self.wfile.write(b)
+
+    def handle_update(self, qs):
+        q = urllib.parse.parse_qs(qs)
+        ip = (q.get("myip") or [""])[0]
+        if not ip:
+            note("REFUSED no myip"); return self.reply("nohost")
+        cur = current_remote()
+        if cur == ip:
+            note("nochg %s" % ip); return self.reply("nochg %s" % ip)
+        rc = subprocess.run(["ip", "tunnel", "change", TUN, "mode", "sit",
+                             "local", LOCAL, "remote", ip],
+                            capture_output=True, text=True)
+        if rc.returncode != 0:
+            # Recreate rather than report a success we did not achieve. This is
+            # the path that a multipoint tunnel takes; keeping it means a stub
+            # that cannot silently no-op.
+            note("change failed (%s) -- recreating" % rc.stderr.strip())
+            subprocess.run(["ip", "tunnel", "del", TUN], check=False)
+            subprocess.run(["ip", "tunnel", "add", TUN, "mode", "sit",
+                            "local", LOCAL, "remote", ip, "ttl", "64"], check=False)
+            subprocess.run(["ip", "link", "set", TUN, "up", "mtu", "1480"], check=False)
+            subprocess.run(["ip", "-6", "addr", "replace", V6_LOCAL, "dev", TUN], check=False)
+            subprocess.run(["ip", "-6", "route", "replace", SITE6, "via", V6_PEER,
+                            "dev", TUN], check=False)
+        # Verify rather than trust: read the remote back.
+        got = current_remote()
+        if got != ip:
+            note("FAILED to point %s at %s (reads %s)" % (TUN, ip, got))
+            return self.reply("dnserr")
+        note("good %s (was %s)" % (ip, cur))
+        self.reply("good %s" % ip)
+
+    def do_GET(self):
+        u = urllib.parse.urlparse(self.path)
+        if u.path == "/nic/update": self.handle_update(u.query)
+        else: self.reply("badauth")
+
+    def do_POST(self):
+        n = int(self.headers.get("Content-Length") or 0)
+        self.handle_update(self.rfile.read(n).decode())
+
+    def log_message(self, *a): pass
+
+http.server.HTTPServer(("0.0.0.0", 80), H).serve_forever()
+'
+
+up() {
+    log "backbone: teaching each ISP box how to reach the other island"
+    # isp-dhcp owns HE and must be able to answer a router that arrived over
+    # PPPoE, so it needs a route back to that island via the backbone.
+    dhcp_isp "sudo ip route replace $PPPOE_NET via $PPPOE_ISP" \
+        || die "could not add the PPPoE-island route on isp-dhcp"
+    # isp-pppoe must forward its clients' traffic for HE across the backbone.
+    pppoe_isp "sudo ip route replace $HE_ADDR/32 via $DHCP_ISP" \
+        || die "could not add the HE route on isp-pppoe"
+
+    log "HE endpoint: $HE_ADDR on isp-dhcp"
+    # A dummy interface, not a loopback alias: `ip tunnel` wants a real local
+    # address and a dummy is the honest way to have one that is not tied to
+    # either WAN segment -- which is the point, HE is neither.
+    dhcp_isp "sudo modprobe dummy 2>/dev/null;
+              sudo ip link add he-lo type dummy 2>/dev/null;
+              sudo ip link set he-lo up;
+              sudo ip addr replace $HE_ADDR/32 dev he-lo"
+
+    log "6in4 tunnel he-sim: local $HE_ADDR, remote set by the API on demand"
+    # A PLACEHOLDER remote, not 0.0.0.0. A sit tunnel created with `remote any`
+    # is multipoint (6rd-shaped), and `ip tunnel change` then refuses to convert
+    # it to point-to-point -- "add tunnel he-sim failed: Invalid argument". The
+    # API's update silently did nothing, so the stub logged "good", production's
+    # he-tunnel-follow logged success, and the tunnel still pointed nowhere.
+    # Created point-to-point from the start, `change` works.
+    dhcp_isp "sudo ip tunnel del he-sim 2>/dev/null;
+              sudo ip tunnel add he-sim mode sit local $HE_ADDR remote $PLACEHOLDER_REMOTE ttl 64;
+              sudo ip link set he-sim up mtu 1480;
+              sudo ip -6 addr replace $HE_LINK6/64 dev he-sim;
+              sudo ip -6 route replace $SITE6 via $RT_LINK6 dev he-sim;
+              sudo sysctl -qw net.ipv6.conf.all.forwarding=1"
+
+    log "stub tunnelbroker API on $HE_ADDR:80"
+    printf '%s' "$API_PY" | dhcp_isp "cat > /tmp/he-sim-api.py"
+    # Launch from a script FILE, not an inline ssh command. The remote login
+    # shell is vbash, and a multi-line inlined `sudo setsid nohup ... &` through
+    # it silently ran nothing at all: no process, no /run/he-sim-api.out, and a
+    # `pgrep -f he-sim-api.py` status check that reported "running" because the
+    # unbracketed pattern matched its OWN ssh command line. Two self-inflicted
+    # illusions stacked on each other.
+    #
+    # This repo already learned this once -- see isp_session_control() in
+    # labsim-pppoe-ha-test.sh, where driving vbash inline made every iteration
+    # of the T4 matrix test the wrong policy while printing the right one.
+    printf '%s\n' \
+        '#!/bin/sh' \
+        '# started detached so it outlives the ssh session that launched it' \
+        'pkill -f "he-sim-api[.]py" 2>/dev/null' \
+        'rm -f /run/he-sim-api.log /run/he-sim-api.out' \
+        "exec setsid python3 /tmp/he-sim-api.py $HE_ADDR '$HE_LINK6/64' $RT_LINK6 $SITE6 >/run/he-sim-api.out 2>&1 </dev/null &" \
+        | dhcp_isp "cat > /tmp/he-sim-start.sh"
+    dhcp_isp "chmod +x /tmp/he-sim-start.sh && sudo /tmp/he-sim-start.sh" >/dev/null
+    # Poll for the bind rather than sleeping a guessed interval.
+    local i probe=""
+    for i in $(seq 1 10); do
+        sleep 1
+        probe="$(dhcp_isp "curl -sS --max-time 3 'http://$HE_ADDR/nic/update' 2>&1")"
+        [ "$probe" = nohost ] && break
+    done
+    case "$probe" in
+        nohost) log "API answering (returned 'nohost' for a call with no myip -- correct)" ;;
+        *)      die "stub API not answering on $HE_ADDR:80 (got: ${probe:-<nothing>})" ;;
+    esac
+
+    log "up. Router side is NOT configured by this script -- that is real VyOS"
+    log "config and belongs to the matrix; see labsim-ipv6-ha-test.sh --setup."
+}
+
+down() {
+    log "tearing down"
+    dhcp_isp "sudo pkill -f 'he-sim-api[.]py' 2>/dev/null;
+              sudo ip tunnel del he-sim 2>/dev/null;
+              sudo ip link del he-lo 2>/dev/null;
+              sudo ip route del $PPPOE_NET via $PPPOE_ISP 2>/dev/null" >/dev/null
+    pppoe_isp "sudo ip route del $HE_ADDR/32 via $DHCP_ISP 2>/dev/null" >/dev/null
+    log "down"
+}
+
+status() {
+    printf '  HE address : %s\n' "$(dhcp_isp "ip -4 -br addr show he-lo 2>/dev/null | awk '{print \$3}'" || echo '<absent>')"
+    printf '  he-sim     : %s\n' "$(dhcp_isp "ip tunnel show he-sim 2>/dev/null" || echo '<absent>')"
+    printf '  he-sim v6  : %s\n' "$(dhcp_isp "ip -6 -br addr show he-sim 2>/dev/null | awk '{print \$3}'" || echo '-')"
+    # Bracketed so the pattern cannot match the ssh command line carrying it --
+    # unbracketed, this reported "running" while nothing was listening at all.
+    printf '  API        : %s\n' "$(dhcp_isp "pgrep -f 'he-sim-api[.]py' >/dev/null && echo running || echo stopped")"
+    printf '  API bound  : %s\n' "$(dhcp_isp "curl -sS --max-time 3 'http://$HE_ADDR/nic/update' 2>/dev/null" || echo 'NOT ANSWERING')"
+    printf '  API calls  : %s\n' "$(dhcp_isp "grep -c . /run/he-sim-api.log 2>/dev/null" || echo 0)"
+    printf '  route back : %s\n' "$(dhcp_isp "ip route show $PPPOE_NET 2>/dev/null" || echo '<none>')"
+}
+
+# Count of endpoint-CHANGING calls. `nochg` does not count: production's
+# he-tunnel-follow re-sends the same address happily and HE treats it as a
+# no-op, so only a real move is evidence that something re-pointed the tunnel.
+calls() { dhcp_isp "grep -c ' good ' /run/he-sim-api.log 2>/dev/null" | tr -d ' \n'; }
+
+# Point the endpoint at an address WITHOUT going through the API, and without
+# counting as an API call.
+#
+# Needed because rebuilding the sim endpoint resets its remote, while the
+# routers' he-tunnel-follow still reads "in sync" and therefore never re-asserts
+# -- it only calls HE when its own LOCAL source changes, and has no way to learn
+# that the far end drifted. The real HE does not forget, so this is sim
+# bookkeeping, not a behaviour production needs. Keeping it out of the call
+# counter is the point: the matrix asserts on that counter.
+point() {
+    local ip="$1"
+    [ -n "$ip" ] || die "usage: $0 point <ipv4>"
+    dhcp_isp "sudo ip tunnel change he-sim mode sit local $HE_ADDR remote $ip 2>/dev/null \
+              || { sudo ip tunnel del he-sim 2>/dev/null;
+                   sudo ip tunnel add he-sim mode sit local $HE_ADDR remote $ip ttl 64;
+                   sudo ip link set he-sim up mtu 1480;
+                   sudo ip -6 addr replace $HE_LINK6/64 dev he-sim;
+                   sudo ip -6 route replace $SITE6 via $RT_LINK6 dev he-sim; }"
+    local got
+    got="$(dhcp_isp "ip tunnel show he-sim 2>/dev/null | sed -nE 's/.* remote ([0-9.]+).*/\\1/p'" | tr -d ' \n')"
+    [ "$got" = "$ip" ] || die "endpoint still points at ${got:-nothing}, wanted $ip"
+    log "endpoint now points at $ip"
+}
+
+case "${1:-status}" in
+    up)     up ;;
+    down)   down ;;
+    status) status ;;
+    calls)  calls; echo ;;
+    point)  point "${2:-}" ;;
+    *) die "usage: $0 {up|down|status|calls}" ;;
+esac

labsim/labsim-ipv6-ha-test.sh

@@ -0,0 +1,430 @@
+#!/bin/bash
+# Does IPv6 follow VRRP mastership, and does it do so WITHOUT touching HE?
+#
+# The WAN became HA on 2026-09-06 and IPv6 did not follow it. Nothing caught
+# that, because nothing tested it: wan-drill measured IPv4 only, and PPPOE-HA.md
+# recorded "IPv6 stayed up at 15.5ms" from a reading taken outside the failover
+# window. This is the matrix that would have caught it.
+#
+# TWO INVARIANTS, checked independently of any individual test:
+#
+#   1. At most ONE router ever has a live tunnel. An UP tunnel on a box that
+#      does not own the source address is not harmless -- see V4, it is a
+#      BLACKHOLE that will happily attract the v6 default route.
+#   2. A ROUTER-level failover calls the HE API ZERO times. The 10 gig address
+#      is bound to a cloned MAC and follows the VIP to the other box unchanged,
+#      so there is nothing to tell HE. A non-zero count means something
+#      re-pointed the tunnel at a PPPoE address -- which the ISP re-issues on
+#      every dial, so it would be wrong within minutes.
+#
+# KNOWN SIM GAP, 2026-09-06 -- read this before believing a v6_online failure.
+# The MECHANISM is proven here: tun0 up on the master and down on the backup,
+# radvd following mastership, he-tunnel-follow's master guard, its hysteresis,
+# its HE call and the 1480->1472 MTU switch, and VLAN 9 hosts autoconfiguring
+# from the RA (observed: real SLAAC traffic from 2001:db8:187e:9::/64 arriving
+# at the endpoint encapsulated).
+#
+# What is NOT yet proven is the end-to-end v6 DATAPATH, because the sim's
+# "internet" is asymmetric: 6in4 packets from the PPPoE island reach the
+# endpoint with an outer source of 192.168.122.1 -- the libvirt host's NAT --
+# so HE's replies go back to the tunnel remote by a path with no NAT state and
+# are lost. Forward works, return does not.
+#
+# That is a topology fault in the scaffold, not in the thing under test. Fixing
+# it means giving the two ISP islands a real transit path that does not traverse
+# libvirt NAT. Until then, treat v6_online failures as UNPROVEN rather than as
+# evidence the design is wrong -- and do not let that ambiguity leak into
+# production sign-off, which is exactly the mistake wan-drill made by asserting
+# "IPv6 stayed up" from a reading taken outside the window.
+#
+# Requires the fake HE endpoint: ./labsim-he-endpoint.sh up
+#
+#   ./labsim-ipv6-ha-test.sh --setup    configure the router side (once)
+#   ./labsim-ipv6-ha-test.sh --list
+#   ./labsim-ipv6-ha-test.sh V1
+#   ./labsim-ipv6-ha-test.sh --all
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+R1="${R1:-172.31.1.252}"; R2="${R2:-172.31.1.253}"
+VIP="${VIP:-172.31.1.1}"
+LAN9="${LAN9:-172.31.9.10}"                  # VLAN 9 client, for RA tests
+HE_ADDR="${HE_ADDR:-192.0.2.10}"
+HE_LINK6="${HE_LINK6:-2001:db8:1f1c:f6::1}"
+RT_LINK6="${RT_LINK6:-2001:db8:1f1c:f6::2}"
+V9_PREFIX="${V9_PREFIX:-2001:db8:187e:9}"
+# Mirrors production: one MAC, one lease, whichever router holds the VIP.
+WAN_MAC="${WAN_MAC:-02:9f:c2:12:9b:4f}"
+PW="${VYOS_PW:-vyos}"; LANPW="${LANPW:-labsim}"
+EVID="$SCRIPT_DIR/ipv6-ha-evidence"
+HE="$SCRIPT_DIR/labsim-he-endpoint.sh"
+
+SSH=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
+     -o LogLevel=ERROR -o ConnectTimeout=6 -o PreferredAuthentications=password)
+r() { timeout 45 sshpass -p "$PW" ssh "${SSH[@]}" "vyos@$1" "${@:2}" 2>/dev/null; }
+# The Alpine LAN VMs do not offer `password` auth -- reusing the routers' option
+# set makes ssh exit 255 before running anything, which reads as "the network is
+# broken". Same trap as lan() in labsim-pppoe-ha-test.sh.
+LAN_SSH=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
+         -o LogLevel=ERROR -o ConnectTimeout=6)
+lan9() { timeout 30 sshpass -p "$LANPW" ssh "${LAN_SSH[@]}" "root@$LAN9" "$@" 2>/dev/null; }
+
+log()  { printf '\033[36m==>\033[0m %s\n' "$*"; }
+pass() { printf '  \033[32mPASS\033[0m %s\n' "$*"; }
+fail() { printf '  \033[31mFAIL\033[0m %s\n' "$*"; FAILED=$((FAILED+1)); }
+warn() { printf '  \033[33mWARN\033[0m %s\n' "$*"; }
+FAILED=0
+
+# --- observations ----------------------------------------------------------
+# A destroyed or unreachable router is emphatically NOT holding the tunnel, but
+# ssh returns an EMPTY string, and `[ "" = 0 ]` is false -- the pppoe matrix hung
+# on exactly this waiting for a dead box to report zero. Default everything to 0.
+tun_state() { local v; v="$(r "$1" "ip -br link show tun0 2>/dev/null | awk '{print \$2}'" | tr -d ' \n')"; echo "${v:-absent}"; }
+tun_src()   { r "$1" 'ip tunnel show tun0 2>/dev/null | sed -nE "s/.* local ([0-9.]+).*/\1/p"' | tr -d ' \n'; }
+# Whichever router currently holds the 10 gig lease -- under the cloned MAC only
+# one ever does. Read rather than assumed: the address changes when the MAC does.
+tengig_addr() { local h a; for h in "$R1" "$R2"; do
+                  a="$(r "$h" 'ip -4 addr show bond0.53 2>/dev/null | sed -nE "s/.*inet ([0-9.]+).*/\1/p"' | tr -d ' \n')"
+                  [ -n "$a" ] && { echo "$a"; return; }
+                done; echo ""; }
+tun_mtu()   { local v; v="$(r "$1" 'cat /sys/class/net/tun0/mtu 2>/dev/null' | tr -d ' \n')"; echo "${v:-0}"; }
+holder()    { for h in "$R1" "$R2"; do
+                [ "$(r "$h" "ip -4 -o addr show | grep -c ' ${VIP}/'" | tr -d ' \n')" != 0 ] \
+                  && { echo "$h"; return; }; done; echo none; }
+# Ask the ROUTER, not a client: a client can be answered by the wrong path.
+v6_online() { [ "$(r "$1" "ping -6 -c1 -W3 $HE_LINK6 >/dev/null 2>&1 && echo y" | tr -d ' \n')" = y ]; }
+radvd_on()  { [ "$(r "$1" 'systemctl is-active radvd 2>/dev/null' | tr -d ' \n')" = active ]; }
+he_calls()  { timeout 60 "$HE" calls | tr -d ' \n'; }
+
+# How many routers have a tunnel that is UP. Invariant 1's numerator.
+tun_holders() { local n=0 h; for h in "$R1" "$R2"; do
+                  case "$(tun_state "$h")" in UP|UNKNOWN) n=$((n+1)) ;; esac
+                done; echo "$n"; }
+
+check_invariants() {
+    local t ok=0
+    t="$(tun_holders)"
+    [ "${t:-0}" -le 1 ] || { fail "INVARIANT: $t routers have tun0 up"; ok=1; }
+    return $ok
+}
+
+save_evidence() {
+    local name="$1"; local d="$EVID/$name"; mkdir -p "$d"
+    { echo "=== $(date -Is) ==="
+      echo "--- HE endpoint ---"; timeout 60 "$HE" status
+      for h in "$R1" "$R2"; do echo "--- $h ---"
+        r "$h" 'sudo /config/vrrp-wan-reconcile --status 2>/dev/null
+                ip -br link show tun0 2>/dev/null; ip tunnel show tun0 2>/dev/null
+                ip -6 route show default; systemctl is-active radvd
+                sudo journalctl -t vrrp-wan -t he-tunnel-follow -n 10 --no-pager'
+      done; } > "$d/state.txt" 2>&1
+    log "evidence -> ipv6-ha-evidence/$name/"
+}
+
+# --- setup ------------------------------------------------------------------
+# Applied through REAL VyOS config, unlike the ISP-side scaffold, because "will
+# VyOS accept this?" is one of the questions being asked.
+vyos_apply() {   # host, then set-lines on stdin
+    local h="$1"
+    { printf '#!/bin/vbash\nsource /opt/vyatta/etc/functions/script-template\nconfigure\n'
+      cat
+      printf 'commit\nsave\nexit\n'
+    } | timeout 90 sshpass -p "$PW" ssh "${SSH[@]}" "vyos@$h" \
+          'cat > /tmp/v6-apply.sh && chmod +x /tmp/v6-apply.sh && sudo /tmp/v6-apply.sh' 2>&1 | tail -3
+    # `vbash -c` never starts a config session and commit fails to stderr, which
+    # a helper like this discards -- the T4 matrix in labsim-pppoe-ha-test.sh ran
+    # its whole policy sweep against the default while printing the mode it
+    # thought it was testing. Always read the value back.
+}
+
+setup() {
+    log "--setup: router-side IPv6, both routers"
+
+    # THE CLONED MAC, first and on its own. Production pins f0:9f:c2:12:9b:4f on
+    # vif 53 so the 10 gig lease follows the VIP and the tunnel source is the
+    # SAME address on either box. The sim never had it -- each router took its
+    # own lease -- so the sim could not reproduce the one property the whole
+    # IPv6-HA design leans on, and invariant 2 would have been untestable here.
+    local h pref v9
+    for h in "$R1" "$R2"; do
+        log "    $h: pinning the cloned WAN MAC $WAN_MAC"
+        vyos_apply "$h" <<EOF
+set interfaces bonding bond0 vif 53 mac '$WAN_MAC'
+EOF
+    done
+
+    # A new MAC means a NEW lease, so the tunnel source cannot be hardcoded --
+    # discover it. Hardcoding the pre-change address here would have configured
+    # every tunnel with a source neither router owns, i.e. the V4 blackhole, on
+    # both boxes, while the matrix reported setup success.
+    local src="" i
+    for i in $(seq 1 30); do
+        src="$(tengig_addr)"; [ -n "$src" ] && break
+        sleep 5
+    done
+    [ -n "$src" ] || { fail "no router took a 10 gig lease after the MAC change -- cannot set a tunnel source"; return 1; }
+    log "    10 gig lease under the cloned MAC: $src"
+
+    # Credentials pointing at the stub. HE_UPDATE_URL is read AFTER the secrets
+    # file is sourced, so putting it here overrides the production default
+    # without the script needing a sim-specific branch.
+    for h in "$R1" "$R2"; do
+        printf 'HE_USER=sim\nHE_UPDATE_KEY=sim\nHE_TUNNEL_ID=1\nHE_UPDATE_URL=http://%s/nic/update\n' "$HE_ADDR" \
+          | timeout 30 sshpass -p "$PW" ssh "${SSH[@]}" "vyos@$h" \
+              'cat > /tmp/he-secrets && sudo install -o root -g vyattacfg -m 0640 /tmp/he-secrets /config/he-secrets' >/dev/null
+    done
+
+    for h in "$R1" "$R2"; do
+        [ "$h" = "$R1" ] && { pref=high; v9=1; } || { pref=low; v9=2; }
+        log "    $h (RA preference $pref, bond0.9 ::${v9})"
+        vyos_apply "$h" <<EOF
+set interfaces tunnel tun0 encapsulation 'sit'
+set interfaces tunnel tun0 source-address '$src'
+set interfaces tunnel tun0 remote '$HE_ADDR'
+set interfaces tunnel tun0 address '$RT_LINK6/64'
+set interfaces tunnel tun0 mtu '1480'
+set protocols static route6 ::/0 next-hop '$HE_LINK6'
+set interfaces bonding bond0 vif 9 address '${V9_PREFIX}::${v9}/64'
+set service router-advert interface bond0.9 prefix ${V9_PREFIX}::/64 preferred-lifetime '604800'
+set service router-advert interface bond0.9 link-mtu '1472'
+set service router-advert interface bond0.9 default-preference '$pref'
+set system task-scheduler task he-tunnel-follow executable path '/config/he-tunnel-follow'
+set system task-scheduler task he-tunnel-follow executable arguments 'run'
+set system task-scheduler task he-tunnel-follow interval '1m'
+EOF
+    done
+    # RA link-mtu is 1472, the PPPoE figure, on BOTH -- deliberately not 1480.
+    # It cannot be reconciled at runtime (it needs a commit, and the tunnel plane
+    # is commit-free on purpose), so advertise the lower of the two paths and be
+    # correct on either WAN. Production pinned 1480 and was wrong whenever the
+    # WAN fell back.
+    log "    setup done -- run V0 to check the baseline"
+}
+
+# --- tests ------------------------------------------------------------------
+V0() {  # baseline
+    log "V0  baseline: the VIP holder owns the tunnel, the backup does not"
+    local h o; h="$(holder)"; o=$([ "$h" = "$R1" ] && echo "$R2" || echo "$R1")
+    [ "$h" = none ] && { fail "no VIP holder"; return; }
+    case "$(tun_state "$h")" in UP|UNKNOWN) pass "master $h has tun0 up" ;;
+        *) fail "master $h tun0 is $(tun_state "$h")" ;; esac
+    case "$(tun_state "$o")" in DOWN|absent) pass "backup $o tun0 is $(tun_state "$o")" ;;
+        *) fail "backup $o tun0 is $(tun_state "$o") -- it should be held down" ;; esac
+    v6_online "$h" && pass "master reaches HE over v6" || fail "master has no IPv6"
+    radvd_on "$h" && pass "master is advertising on VLAN 9" || fail "master radvd not running"
+    radvd_on "$o" && fail "backup is ALSO advertising -- two default routers on VLAN 9" \
+                  || pass "backup is not advertising"
+    check_invariants
+    save_evidence V0-baseline
+}
+
+V1() {  # clean failover: v6 follows, and HE is never called
+    log "V1  clean failover: IPv6 follows, HE API untouched"
+    local from to t0 before after i
+    from="$(holder)"; to=$([ "$from" = "$R1" ] && echo "$R2" || echo "$R1")
+    before="$(he_calls)"
+    log "    master=$from -> expecting $to (HE calls so far: ${before:-0})"
+    t0=$(date +%s)
+    r "$from" 'sudo mkdir -p /run/vrrp-wan && sudo touch /run/vrrp-wan/force-fault'
+    local took=""
+    for i in $(seq 1 36); do
+        sleep 5
+        [ "$(holder)" = "$to" ] && v6_online "$to" && { took=$(( $(date +%s) - t0 )); break; }
+    done
+    [ -n "$took" ] && pass "IPv6 reached $to in ${took}s" \
+                   || fail "IPv6 never followed to $to within 180s"
+    case "$(tun_state "$from")" in DOWN|absent) pass "$from released its tunnel" ;;
+        *) fail "$from still has tun0 $(tun_state "$from") -- blackhole risk" ;; esac
+    after="$(he_calls)"
+    # THE invariant this test exists for.
+    [ "${after:-0}" = "${before:-0}" ] \
+        && pass "HE API not called (${after:-0} total) -- the address followed the MAC" \
+        || fail "HE API called $(( ${after:-0} - ${before:-0} )) time(s) during a ROUTER failover"
+    check_invariants
+    save_evidence V1-clean-failover
+    r "$from" 'sudo rm -f /run/vrrp-wan/force-fault'
+    sleep 40
+}
+
+V2() {  # 10 gig down on the master: HE must be told, exactly once
+    log "V2  10 gig down: he-tunnel-follow re-points the tunnel and tells HE"
+    local h before after mtu i ok=no
+    h="$(holder)"; before="$(he_calls)"
+    local tengig; tengig="$(tengig_addr)"
+    r "$h" 'sudo ip link set bond0.53 down'
+    # he-tunnel-follow runs on a 1m task-scheduler with a 2-tick hysteresis, so
+    # allow well past 2 minutes before calling it a failure.
+    for i in $(seq 1 30); do
+        sleep 10
+        [ "$(tun_src "$h")" != "$tengig" ] && { ok=yes; break; }
+    done
+    [ "$ok" = yes ] && pass "tunnel source moved to $(tun_src "$h") after $((i*10))s" \
+                    || fail "tunnel source never left the 10 gig address"
+    mtu="$(tun_mtu "$h")"
+    [ "$mtu" = 1472 ] && pass "MTU dropped to 1472 for the PPPoE path" \
+                      || fail "MTU is $mtu, want 1472 -- large transfers will hang"
+    after="$(he_calls)"
+    [ "$(( ${after:-0} - ${before:-0} ))" -ge 1 ] \
+        && pass "HE API called $(( ${after:-0} - ${before:-0} )) time(s), as it must be here" \
+        || fail "HE was never told -- it still points at an address this box no longer has"
+    v6_online "$h" && pass "IPv6 still up over PPPoE" || fail "IPv6 down on the PPPoE path"
+    save_evidence V2-tengig-down
+    r "$h" 'sudo ip link set bond0.53 up'
+    sleep 60
+}
+
+V3() {  # a cold backup must not advertise, dial, or blackhole
+    log "V3  cold backup: no tunnel, no RA, no HE call"
+    local h o before after
+    h="$(holder)"; o=$([ "$h" = "$R1" ] && echo "$R2" || echo "$R1")
+    before="$(he_calls)"
+    r "$o" 'sudo systemctl restart vrrp-wan-reconcile.service' >/dev/null
+    sleep 20
+    case "$(tun_state "$o")" in DOWN|absent) pass "backup tunnel stays $(tun_state "$o")" ;;
+        *) fail "backup brought tun0 up while not holding the VIP" ;; esac
+    radvd_on "$o" && fail "backup is advertising on VLAN 9" || pass "backup is silent on VLAN 9"
+    after="$(he_calls)"
+    [ "${after:-0}" = "${before:-0}" ] && pass "backup made no HE call" \
+        || fail "the BACKUP called the HE API -- it would point HE at its own idle line"
+    save_evidence V3-cold-backup
+}
+
+V4() {  # the assumption the production override was built on
+    log "V4  a tunnel whose source-address is absent: what does VyOS actually do?"
+    # The production override says such a tunnel "would simply stay down", and
+    # treats that as the reason it was safe to leave IPv6 single-homed. Measured
+    # on the sim backup 2026-09-06: the commit SUCCEEDS and the link comes up
+    # anyway -- it is a blackhole, not an inert node. That is why the runtime
+    # gate is load-bearing rather than a nicety, exactly like the PPPoE gate.
+    local o h; h="$(holder)"; o=$([ "$h" = "$R1" ] && echo "$R2" || echo "$R1")
+    r "$o" 'sudo ip link set tun0 up' >/dev/null; sleep 3
+    case "$(tun_state "$o")" in
+        UP|UNKNOWN) pass "confirmed: VyOS leaves it UP with no source address (blackhole)" ;;
+        *) warn "this VyOS version keeps it $(tun_state "$o") -- the override's assumption holds here; re-check the production version before relying on it" ;;
+    esac
+    v6_online "$o" && fail "the backup somehow reached HE -- two live tunnels" \
+                   || pass "and it carries nothing, as expected"
+    # Hand it straight back to the reconciler rather than leaving it up.
+    r "$o" 'sudo systemctl restart vrrp-wan-reconcile.service' >/dev/null
+    sleep 15
+    case "$(tun_state "$o")" in DOWN|absent) pass "the reconciler put it back down" ;;
+        *) fail "the reconciler did NOT re-close the gate -- this is the load-bearing bit" ;; esac
+    save_evidence V4-absent-source-address
+}
+
+V5() {  # never two live tunnels, even mid-transition
+    log "V5  both routers momentarily master: never two live tunnels"
+    local from to i worst=0 n
+    from="$(holder)"; to=$([ "$from" = "$R1" ] && echo "$R2" || echo "$R1")
+    r "$from" 'sudo mkdir -p /run/vrrp-wan && sudo touch /run/vrrp-wan/force-fault'
+    # Sample THROUGH the transition rather than at the ends. The interesting
+    # window is the one where both boxes briefly think they are in charge.
+    for i in $(seq 1 24); do
+        n="$(tun_holders)"; [ "${n:-0}" -gt "$worst" ] && worst="$n"
+        sleep 5
+    done
+    [ "$worst" -le 1 ] && pass "at most $worst live tunnel throughout the transition" \
+                       || fail "saw $worst live tunnels at once -- HE would receive two claimants"
+    r "$from" 'sudo rm -f /run/vrrp-wan/force-fault'
+    save_evidence V5-transition-invariant
+    sleep 40
+}
+
+V6() {  # RA deprecation: does a VLAN 9 host drop the dead gateway?
+    log "V6  RA deprecation: the client must stop using a demoted router"
+    local h before
+    h="$(holder)"
+    before="$(lan9 'ip -6 route show default 2>/dev/null | head -1')"
+    if [ -z "$before" ]; then
+        warn "VLAN 9 client has no IPv6 default route -- SLAAC may not have run; skipping"
+        return
+    fi
+    log "    client default was: $before"
+    r "$h" 'sudo mkdir -p /run/vrrp-wan && sudo touch /run/vrrp-wan/force-fault'
+    sleep 45
+    local after; after="$(lan9 'ip -6 route show default 2>/dev/null | head -1')"
+    log "    client default now: ${after:-<none>}"
+    # radvd emits a final RA with router-lifetime 0 on a graceful stop. Either
+    # the client moved to the new master or it dropped the route entirely; both
+    # are correct. Still pointing at the demoted box is not.
+    if [ "$after" = "$before" ]; then
+        fail "client still points at the demoted router -- the farewell RA did not land"
+    else
+        pass "client stopped using the demoted router"
+    fi
+    r "$h" 'sudo rm -f /run/vrrp-wan/force-fault'
+    save_evidence V6-ra-deprecation
+    sleep 40
+}
+
+V7() {  # replay the 2026-09-06 near-miss, but slowly
+    log "V7  slow WAN restore: does the hysteresis still hold?"
+    # On 2026-09-06 vif53-pin-boot-disable bounced the 10 gig, he-tunnel-follow
+    # ticked once and saw the PPPoE address, and vyos-failover restored the 10
+    # gig 22 SECONDS before the second tick would have pushed HE at an address
+    # Vodafone reissues on every dial. 22s of margin is not a safety property.
+    # Here the restore is deliberately slower than the hysteresis window.
+    local h before after tengig
+    h="$(holder)"; before="$(he_calls)"; tengig="$(tengig_addr)"
+    r "$h" 'sudo ip link set bond0.53 down'
+    sleep 200                                  # > 2 ticks of a 1m scheduler
+    r "$h" 'sudo ip link set bond0.53 up'
+    sleep 90
+    after="$(he_calls)"
+    if [ "$(( ${after:-0} - ${before:-0} ))" -ge 1 ]; then
+        warn "HE was updated $(( ${after:-0} - ${before:-0} )) time(s) and then had to move back -- this is the 2026-09-06 shape, now reproduced deliberately. Either widen HYSTERESIS or make vif53-pin-boot-disable hold he-tunnel-follow off for the bounce."
+    else
+        pass "no HE churn across a slow WAN bounce"
+    fi
+    # Whatever happened, the tunnel must end up back on the 10 gig.
+    local i
+    for i in $(seq 1 24); do
+        [ "$(tun_src "$h")" = "$tengig" ] && break
+        sleep 10
+    done
+    [ "$(tun_src "$h")" = "$tengig" ] && pass "tunnel returned to the 10 gig address" \
+        || fail "tunnel stuck on $(tun_src "$h") after the 10 gig came back"
+    save_evidence V7-slow-restore
+}
+
+preflight() {
+    log "preflight"
+    local rc=0
+    [ "$(timeout 60 "$HE" status | grep -c 'API bound  : nohost')" = 1 ] \
+        || { fail "the fake HE endpoint is not answering -- run ./labsim-he-endpoint.sh up"; rc=1; }
+    local h
+    for h in "$R1" "$R2"; do
+        [ "$(tun_state "$h")" = absent ] \
+            && { fail "$h has no tun0 -- run --setup first"; rc=1; }
+        [ "$(r "$h" 'systemctl is-active vrrp-wan-reconcile.timer')" = active ] \
+            || { fail "$h vrrp-wan-reconcile.timer not active"; rc=1; }
+        # The reconciler must be the version that knows about the v6 plane, or
+        # every result below measures the OLD behaviour while printing the new
+        # test names -- the failure mode this repo has already been bitten by.
+        r "$h" 'grep -q v6_take /config/vrrp-wan-reconcile' \
+            || { fail "$h has a vrrp-wan-reconcile with no IPv6 plane -- run migration/vrrp-wan-install"; rc=1; }
+    done
+    # Line the sim's endpoint up with whoever actually holds the tunnel. The real
+    # HE remembers where it was pointed; a rebuilt sim endpoint does not, and
+    # he-tunnel-follow will not re-assert because from its side nothing changed.
+    # Does NOT count as an API call, so the invariant-2 assertions stay honest.
+    local m src
+    m="$(holder)"; src="$(tun_src "$m")"
+    if [ -n "$src" ]; then
+        timeout 60 "$HE" point "$src" >/dev/null 2>&1 \
+            || { fail "could not point the sim HE endpoint at $src"; rc=1; }
+    fi
+    [ "$rc" -eq 0 ] && pass "HE endpoint answering and pointed at $src, both routers have tun0 and a v6-aware reconciler"
+    return $rc
+}
+
+case "${1:---all}" in
+  --list)  echo "V0 baseline | V1 clean failover | V2 10gig-down | V3 cold backup | V4 absent source-address | V5 transition invariant | V6 RA deprecation | V7 slow restore"; exit 0 ;;
+  --setup) setup; exit 0 ;;
+  --all)   preflight || exit 1; V0; V1; V2; V3; V4; V5; V6; V7 ;;
+  *)       preflight || exit 1; "$1" ;;
+esac
+
+echo
+[ "$FAILED" -eq 0 ] && { echo "ALL PASS"; exit 0; }
+echo "$FAILED check(s) FAILED"; exit 1