vyos: WAN follows VRRP mastership, rehearsed in labsim

The ISP is consumer with one static IP, so "both routers hold WAN" is not
available: the 10 gig lease is anchored to a cloned MAC and the PPPoE line to a
single credential. The WAN therefore has to move with mastership.

Proven end to end in the sim: the secondary was promoted, took the WAN, got a
lease, installed a default route, and a LAN VM reached the internet through it
(3/3, 9ms). Demoting released it -- link down, no address. Rebooting the master
converged correctly too: it came back BACKUP with the WAN disabled while the
peer kept it.

The rehearsal earned its keep four times over, and none of these were visible
from reading the docs:

  - `transition-script` alone is NOT safe to hang internet on. VyOS delivers it
    through keepalived-fifo.py, and on one promotion that helper logged NOTHING
    while Keepalived_vrrp logged all six instances entering MASTER and the
    built-in notify_master for conntrack-sync ran normally. The result was a
    router holding every VIP with no WAN -- the 2026-09-02 outage, recreated by
    the mechanism meant to prevent it. Hence vrrp-wan-reconcile on a 30s timer:
    the scripts give speed, the timer gives correctness.

  - The health check must ask REALITY, not a marker. Keying "am I master" on a
    /run file written by the transition script meant that when the script did
    not run, the router believed it was backup, passed the check, and kept the
    VIPs it could not serve. It now asks whether the VIP is actually on the box.

  - The old address-based check DEADLOCKED this design: may-I-be-master required
    already having WAN, and only the master gets WAN. That is why vyos002 sat in
    FAULT for ever -- the safety check had silently removed the redundancy it
    existed to protect.

  - script-template must be the FIRST thing a script does. Sourced after an if,
    an exec and a mkdir it terminated the script inside the source, rc=0, no
    output: the reconciler reported success having done nothing. Hence the split
    into vrrp-wan-apply, matching the shape /config/vyos-known-good already uses.

Two hazards found and handled rather than discovered in production:

  - A `configure` session whose process dies leaks a unionfs mount under
    /opt/vyatta/config/tmp, and one of those holds the commit lock -- after
    which every commit fails, including the manual one you try to fix it with.
    A 30s job that can leak one per failure wedges the box on its own, so the
    reconciler reaps dead sessions before it starts. It cleared 8 on the sim.

  - Any `save` while a box is master persists the enabled WAN into config.boot,
    so a reboot would claim the shared MAC regardless of VRRP state. Observed:
    an ordinary console-apply did exactly this. config.boot must keep `disable`
    on BOTH routers; the model asserts it and vyos:verify reports it as drift.

NOT yet applied to production, and it should not be until the remaining item is
settled: a clean, deliberately-triggered failover has been seen via reboot, but
`restart vrrp` twice failed to move mastership at all, so the trigger for a
planned failover is still unproven.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH

.gitignore

@@ -31,3 +31,9 @@ node_modules/
 # Asahi build artifacts (large)
 bastion/.asahi-cache/
 bastion/asahi-repo/*.zip
+
+# Regenerated by labsim/dualstack-lab.sh; derived state, not source.
+labsim/dualstack-evidence/
+
+# Runtime snapshots from labsim/cilium-ipam-switch.sh
+labsim/.ipam-switch-state/

bastion/src/bastion/src/templates/vyos-config-spec.ts

@@ -52,6 +52,134 @@ function normalizeDiskPath(value: string | undefined): string {
   return raw.startsWith("/dev/") ? raw : `/dev/${raw}`;
 }
 
+/** Sentinel marking a value that lives in Pulumi config, not in the bundle. */
+const SECRET_PREFIX = "@secret:";
+
+/**
+ * Enable the VyOS HTTP API so the router is manageable the moment it boots.
+ *
+ * This belongs at install time rather than in the Pulumi model: the model is
+ * applied THROUGH this API, so a router that lacks it cannot be brought under
+ * management without a hand-run change on a live firewall. It is also why the
+ * model excludes `service https` outright -- a provider able to rewrite its own
+ * transport can lock itself out permanently.
+ *
+ * `listen-address` is always set. Leaving it unbound would expose a
+ * config-write endpoint on every segment the router touches, the WAN included.
+ */
+function apiSets(apiKey: string, listenAddress: string): VyosSetOp[] {
+  const sets: VyosSetOp[] = [
+    { path: ["service", "https", "api", "keys", "id", "pulumi", "key"], value: apiKey },
+    { path: ["service", "https", "api", "rest"] },
+  ];
+  if (listenAddress !== "") {
+    sets.push({ path: ["service", "https", "listen-address"], value: listenAddress });
+  }
+  return sets;
+}
+
+/** Tag nodes introduced by the API config, needed by the installer's ConfigTree. */
+const API_TAGS: string[][] = [["service", "https", "api", "keys", "id"]];
+
+/**
+ * The address to bind the API to: an explicit choice, else the management
+ * address with its prefix length stripped. Under DHCP there is no address to
+ * bind at build time, so the caller must pass one or the listener stays unbound
+ * and the API is not enabled at all.
+ */
+function apiListenAddress(spec: VyosInstallSpec, mgmtAddress: string): string {
+  if (spec.apiListenAddress !== undefined && spec.apiListenAddress !== "") {
+    return spec.apiListenAddress;
+  }
+  return mgmtAddress.includes("/") ? (mgmtAddress.split("/")[0] ?? "") : "";
+}
+
+/**
+ * Use a Pulumi-rendered bundle as the router's config verbatim.
+ *
+ * Secret-valued nodes are dropped rather than installed with their sentinel
+ * text: writing `@secret:pppoePassword` into config.boot would look configured
+ * while being wrong, which is worse than being absent. The router comes up
+ * without those values and the first `pulumi up` fills them in.
+ *
+ * `system host-name` is forced to the hostname the install was asked for. The
+ * bundle carries the name of whichever router it was exported from, and
+ * installing vyos001's hostname onto vyos002 would collide on the network.
+ */
+function buildFromBundle(
+  params: { hostname: string; defaultPassword: string; disk?: string | undefined },
+  spec: VyosInstallSpec,
+  bundle: NonNullable<VyosInstallSpec["bundle"]>,
+  mgmtAddress: string,
+): VyosConfigSpec {
+  const sets: VyosSetOp[] = [];
+  const dropped: string[] = [];
+  for (const op of bundle.sets) {
+    if (op.value !== undefined && op.value.startsWith(SECRET_PREFIX)) {
+      dropped.push(op.path.join(" "));
+      continue;
+    }
+    if (op.path.length === 2 && op.path[0] === "system" && op.path[1] === "host-name") {
+      continue;
+    }
+    sets.push({
+      path: op.path,
+      ...(op.value === undefined ? {} : { value: op.value }),
+      ...(op.replace === undefined ? {} : { replace: op.replace }),
+    });
+  }
+  sets.unshift({ path: ["system", "host-name"], value: params.hostname });
+
+  if (dropped.length > 0) {
+    console.warn(
+      `vyos ${params.hostname}: ${dropped.length} secret-valued node(s) left unset by the ` +
+        `bundle; run \`pulumi up\` to supply them: ${dropped.join(", ")}`,
+    );
+  }
+
+  const tags = [...bundle.tags];
+  const api = enableApi(spec, params.hostname, mgmtAddress);
+  if (api.length > 0) {
+    sets.push(...api);
+    tags.push(...API_TAGS);
+  }
+
+  return {
+    hostname: params.hostname,
+    imageName: "",
+    password: spec.password ?? params.defaultPassword,
+    console: "K",
+    disk: normalizeDiskPath(params.disk),
+    reportAddress: mgmtAddress.includes("/") ? (mgmtAddress.split("/")[0] ?? "") : "",
+    raid: false,
+    freshConfig: spec.freshConfig ?? false,
+    sets,
+    tags,
+  };
+}
+
+/**
+ * The API config for this install, or nothing when it cannot be enabled safely.
+ *
+ * Refusing to enable it unbound is deliberate. Under DHCP there is no address
+ * known at build time, and the alternative -- binding to every interface --
+ * would publish a config-write endpoint on the WAN. Better to leave the router
+ * SSH-only and say so than to open it everywhere.
+ */
+function enableApi(spec: VyosInstallSpec, hostname: string, mgmtAddress: string): VyosSetOp[] {
+  if (spec.apiKey === undefined || spec.apiKey === "") return [];
+  const listen = apiListenAddress(spec, mgmtAddress);
+  if (listen === "") {
+    console.warn(
+      `vyos ${hostname}: --vyos-api-key given but no address to bind to ` +
+        `(management is "${mgmtAddress}"). Pass --vyos-api-listen <addr>; the HTTP API ` +
+        `has NOT been enabled, so Pulumi cannot manage this router yet.`,
+    );
+    return [];
+  }
+  return apiSets(spec.apiKey, listen);
+}
+
 export function buildVyosConfigSpec(params: {
   hostname: string;
   spec?: VyosInstallSpec | undefined;
@@ -62,6 +190,14 @@ export function buildVyosConfigSpec(params: {
   const spec = params.spec ?? {};
   const mgmt = spec.mgmtInterface ?? "eth0";
   const mgmtAddress = spec.mgmtAddress ?? "dhcp";
+
+  // A rendered bundle replaces the derived config entirely. Deriving a second
+  // opinion alongside it is the drift the bundle exists to prevent: Pulumi and
+  // labctl would each believe they knew the router's config, and the box would
+  // end up with whichever ran last.
+  if (spec.bundle !== undefined) {
+    return buildFromBundle(params, spec, spec.bundle, mgmtAddress);
+  }
   const bondMembers = spec.bondMembers ?? [];
   const vlans = spec.vlans ?? [];
 
@@ -192,6 +328,14 @@ export function buildVyosConfigSpec(params: {
     });
   }
 
+  // Enabled here too, not just for bundle installs: every VyOS this bastion
+  // provisions should be manageable from first boot.
+  const api = enableApi(spec, params.hostname, mgmtAddress);
+  if (api.length > 0) {
+    sets.push(...api);
+    tags.push(...API_TAGS);
+  }
+
   return {
     hostname: params.hostname,
     imageName: "",

bastion/src/bastion/tests/vyos-bundle.test.ts

@@ -0,0 +1,155 @@
+import { describe, it, expect, vi } from "vitest";
+import type { VyosBundle } from "@lab/shared";
+import { buildVyosConfigSpec } from "../src/templates/vyos-config-spec.js";
+
+/**
+ * A bundle is what makes "one config, two apply paths" true rather than
+ * aspirational: `pulumi up` POSTs the subtree model to a running router, labctl
+ * writes the same model into config.boot during a PXE install. These tests pin
+ * the properties that keep the two honest.
+ */
+const bundle: VyosBundle = {
+  sets: [
+    { path: ["system", "host-name"], value: "vyos001" },
+    { path: ["interfaces", "bonding", "bond0", "address"], value: "192.168.1.252/24" },
+    { path: ["interfaces", "bonding", "bond0", "member", "interface"], value: "eth1", replace: false },
+    { path: ["interfaces", "bonding", "bond0", "vif", "53", "disable"] },
+    { path: ["interfaces", "pppoe", "pppoe0", "authentication", "password"], value: "@secret:pppoePassword" },
+    { path: ["interfaces", "pppoe", "pppoe0", "mtu"], value: "1492" },
+  ],
+  tags: [["interfaces", "bonding", "bond0"], ["interfaces", "ethernet"]],
+};
+
+const build = (hostname: string, extra: Record<string, unknown> = {}) =>
+  buildVyosConfigSpec({
+    hostname,
+    spec: { bundle, ...extra },
+    defaultPassword: "changeme",
+  });
+
+describe("vyos config spec from a Pulumi bundle", () => {
+  it("applies non-secret nodes verbatim, preserving valuelessness and replace:false", () => {
+    const spec = build("vyos001");
+
+    expect(spec.sets).toContainEqual({
+      path: ["interfaces", "bonding", "bond0", "address"],
+      value: "192.168.1.252/24",
+    });
+    // A multi-value node must keep replace:false or the second bond member
+    // overwrites the first.
+    expect(spec.sets).toContainEqual({
+      path: ["interfaces", "bonding", "bond0", "member", "interface"],
+      value: "eth1",
+      replace: false,
+    });
+    // A valueless node must not acquire a value on the way through.
+    expect(spec.sets).toContainEqual({
+      path: ["interfaces", "bonding", "bond0", "vif", "53", "disable"],
+    });
+    expect(spec.tags).toEqual(bundle.tags);
+  });
+
+  it("drops secret-valued nodes instead of installing the sentinel text", () => {
+    const warn = vi.spyOn(console, "warn").mockImplementation(() => {});
+    const spec = build("vyos001");
+
+    const values = spec.sets.map((s) => s.value ?? "");
+    expect(values.some((v) => v.startsWith("@secret:"))).toBe(false);
+    expect(spec.sets.some((s) => s.path.includes("authentication"))).toBe(false);
+    // Silently dropping the WAN credential would leave someone debugging a dead
+    // PPPoE link, so it has to be said out loud.
+    expect(warn).toHaveBeenCalledWith(expect.stringContaining("pulumi up"));
+    warn.mockRestore();
+  });
+
+  it("forces the hostname the install was asked for, not the bundle's", () => {
+    // The bundle is exported from one router and reused for its peer; taking the
+    // hostname from it would put two vyos001s on the network.
+    const spec = build("vyos002");
+    const hostnames = spec.sets.filter(
+      (s) => s.path.length === 2 && s.path[0] === "system" && s.path[1] === "host-name",
+    );
+    expect(hostnames).toEqual([{ path: ["system", "host-name"], value: "vyos002" }]);
+  });
+
+  it("still honours installer inputs, which are not router config", () => {
+    const spec = buildVyosConfigSpec({
+      hostname: "vyos001",
+      spec: { bundle, password: "s3cret", freshConfig: true },
+      defaultPassword: "changeme",
+      disk: "nvme0n1",
+    });
+    expect(spec.password).toBe("s3cret");
+    expect(spec.freshConfig).toBe(true);
+    expect(spec.disk).toBe("/dev/nvme0n1");
+  });
+
+  it("enables the HTTP API at install so Pulumi can manage the router from first boot", () => {
+    const warn = vi.spyOn(console, "warn").mockImplementation(() => {});
+    const spec = buildVyosConfigSpec({
+      hostname: "vyos001",
+      spec: { bundle, apiKey: "k3y", apiListenAddress: "10.0.1.252" },
+      defaultPassword: "changeme",
+    });
+
+    expect(spec.sets).toContainEqual({
+      path: ["service", "https", "api", "keys", "id", "pulumi", "key"],
+      value: "k3y",
+    });
+    expect(spec.sets).toContainEqual({ path: ["service", "https", "api", "rest"] });
+    expect(spec.sets).toContainEqual({
+      path: ["service", "https", "listen-address"],
+      value: "10.0.1.252",
+    });
+    // The key id is a tag node; without this the installer's ConfigTree rejects it.
+    expect(spec.tags).toContainEqual(["service", "https", "api", "keys", "id"]);
+    warn.mockRestore();
+  });
+
+  it("binds the API to the static management address when none is given", () => {
+    const spec = buildVyosConfigSpec({
+      hostname: "vyos001",
+      spec: { apiKey: "k3y", mgmtAddress: "192.168.1.252/24" },
+      defaultPassword: "changeme",
+    });
+    expect(spec.sets).toContainEqual({
+      path: ["service", "https", "listen-address"],
+      value: "192.168.1.252",
+    });
+  });
+
+  it("refuses to enable the API unbound rather than exposing it on the WAN", () => {
+    const warn = vi.spyOn(console, "warn").mockImplementation(() => {});
+    // Management is DHCP, so there is no address to bind at build time. Binding
+    // to everything would put a config-write endpoint on the WAN.
+    const spec = buildVyosConfigSpec({
+      hostname: "vyos001",
+      spec: { apiKey: "k3y", mgmtAddress: "dhcp" },
+      defaultPassword: "changeme",
+    });
+    expect(spec.sets.some((s) => s.path[0] === "service" && s.path[1] === "https")).toBe(false);
+    expect(warn).toHaveBeenCalledWith(expect.stringContaining("has NOT been enabled"));
+    warn.mockRestore();
+  });
+
+  it("does not enable the API when no key is supplied", () => {
+    const spec = buildVyosConfigSpec({
+      hostname: "vyos001",
+      spec: { mgmtAddress: "192.168.1.252/24" },
+      defaultPassword: "changeme",
+    });
+    expect(spec.sets.some((s) => s.path[0] === "service" && s.path[1] === "https")).toBe(false);
+  });
+
+  it("ignores the derived path entirely when a bundle is present", () => {
+    // Belt and braces: even if topology flags reach this far (the CLI rejects
+    // them), the bundle must win rather than merge.
+    const spec = buildVyosConfigSpec({
+      hostname: "vyos001",
+      spec: { bundle, bondMembers: ["eth2", "eth3"], vlans: [{ id: 99, address: "10.9.9.1/24" }] },
+      defaultPassword: "changeme",
+    });
+    expect(spec.sets.some((s) => s.path.includes("99"))).toBe(false);
+    expect(spec.sets.filter((s) => s.value === "eth2" || s.value === "eth3")).toEqual([]);
+  });
+});

bastion/src/cli/src/commands/install.ts

@@ -1,11 +1,42 @@
 // CLI command: provision install
 // Queue a discovered machine for OS installation via labd.
 
+import { readFileSync } from "node:fs";
 import { Command, Option, InvalidArgumentError } from "commander";
 import { isValidOsId, SUPPORTED_OS, SUPPORTED_ROLES, ROLE_REGISTRY } from "@lab/shared";
-import type { VyosInstallSpec, VyosVlanSpec } from "@lab/shared";
+import type { VyosBundle, VyosInstallSpec, VyosVlanSpec } from "@lab/shared";
 import { getLabdClient } from "../api/config.js";
 
+/**
+ * Load one router's config out of a Pulumi-rendered bundle.
+ *
+ * The bundle is produced by `kubernetes-deployment` (npm run vyos:bundle) and
+ * holds every router it manages, keyed by name. Selecting by hostname here is
+ * what keeps bring-up and `pulumi up` describing the same box: labctl replays
+ * the declared config rather than deriving its own.
+ */
+export function loadVyosBundle(path: string, hostname: string): VyosBundle {
+  let parsed: { version?: number; routers?: Record<string, VyosBundle> };
+  try {
+    parsed = JSON.parse(readFileSync(path, "utf8"));
+  } catch (e) {
+    throw new InvalidArgumentError(`Cannot read VyOS bundle ${path}: ${(e as Error).message}`);
+  }
+  if (parsed.version !== 1) {
+    throw new InvalidArgumentError(
+      `VyOS bundle ${path} has version ${parsed.version ?? "<none>"}; this labctl understands 1`,
+    );
+  }
+  const router = parsed.routers?.[hostname];
+  if (router === undefined) {
+    const known = Object.keys(parsed.routers ?? {}).join(", ") || "<none>";
+    throw new InvalidArgumentError(
+      `VyOS bundle ${path} has no entry for "${hostname}" (has: ${known})`,
+    );
+  }
+  return router;
+}
+
 /** Parse a repeated --vlan flag: "<id>:<cidr>[:<description>]". */
 export function parseVlan(value: string, previous: VyosVlanSpec[] = []): VyosVlanSpec[] {
   const parts = value.split(":");
@@ -88,6 +119,20 @@ export function registerInstallCommand(parent: Command): void {
     .option("--vyos-password <password>", "VyOS: password for the 'vyos' user")
     .option("--vyos-hwid <iface=mac>", "VyOS: pin an interface name to a MAC via hw-id (repeatable)", parseHwId)
     .option("--vyos-fresh-config", "VyOS: on reinstall, overwrite the preserved config with the generated one")
+    .option(
+      "--vyos-bundle <path>",
+      "VyOS: apply a Pulumi-rendered bundle verbatim (kubernetes-deployment/infra/vyos/vyos-bundle.json). " +
+        "Replaces the derived --vyos-bond/--vlan/... config; secret values are left unset for `pulumi up`.",
+    )
+    .option(
+      "--vyos-api-key <key>",
+      "VyOS: enable the HTTP API with this key so Pulumi can manage the router from first boot",
+    )
+    .option(
+      "--vyos-api-listen <addr>",
+      "VyOS: address the HTTP API binds to (default: the static management address). " +
+        "Required when management is DHCP; the API is never bound to all interfaces.",
+    )
     .action(async (mac: string, hostname: string, opts: {
       role: string;
       os: string;
@@ -104,6 +149,9 @@ export function registerInstallCommand(parent: Command): void {
       vyosPassword?: string;
       vyosHwid?: Record<string, string>;
       vyosFreshConfig?: boolean;
+      vyosBundle?: string;
+      vyosApiKey?: string;
+      vyosApiListen?: string;
     }) => {
       if (!isValidOsId(opts.os)) {
         console.error(`Unknown OS: ${opts.os}. Supported: ${SUPPORTED_OS.join(", ")}`);
@@ -159,9 +207,31 @@ export function registerInstallCommand(parent: Command): void {
         ...(opts.vyosMgmtVlan !== undefined && opts.vyosMgmtVlan !== ""
           ? { mgmtVlan: parseVlan(opts.vyosMgmtVlan)[0] as VyosVlanSpec } : {}),
         ...(opts.vyosFreshConfig === true ? { freshConfig: true } : {}),
+        ...(opts.vyosBundle !== undefined && opts.vyosBundle !== ""
+          ? { bundle: loadVyosBundle(opts.vyosBundle, hostname) } : {}),
+        ...(opts.vyosApiKey !== undefined && opts.vyosApiKey !== ""
+          ? { apiKey: opts.vyosApiKey } : {}),
+        ...(opts.vyosApiListen !== undefined && opts.vyosApiListen !== ""
+          ? { apiListenAddress: opts.vyosApiListen } : {}),
       };
       const hasVyosOptions = Object.keys(vyos).length > 0;
 
+      // A bundle already describes the whole router. Accepting derived topology
+      // flags alongside it would silently discard them (the bundle wins in
+      // buildVyosConfigSpec), so say so rather than appear to honour both.
+      if (vyos.bundle !== undefined) {
+        const derived = ["mgmtInterface", "mgmtAddress", "bondMembers", "bondAddress",
+                         "bondVrrp", "vrrpPriority", "vlans", "mgmtVlan"] as const;
+        const conflicting = derived.filter((k) => vyos[k] !== undefined);
+        if (conflicting.length > 0) {
+          console.error(
+            `--vyos-bundle describes the whole router; these would be ignored: ${conflicting.join(", ")}`,
+          );
+          console.error("Remove them, or change the bundle in kubernetes-deployment and re-render.");
+          process.exit(1);
+        }
+      }
+
       if (hasVyosOptions && !opts.os.startsWith("vyos")) {
         console.error(`VyOS options require --os vyos-rolling (got --os ${opts.os})`);
         process.exit(1);

bastion/src/modules/modules/k3s/src/install.ts

@@ -215,7 +215,9 @@ echo "  Using network device: \$DEFAULT_DEV"
 
 KUBECONFIG=/etc/rancher/k3s/k3s.yaml cilium install \\
   --set kubeProxyReplacement=true \\
-  --set ipam.mode=kubernetes \\
+  --set ipam.mode=cluster-pool \\
+  --set ipam.operator.clusterPoolIPv4PodCIDRList='{10.42.0.0/16}' \\
+  --set ipam.operator.clusterPoolIPv4MaskSize=24 \\
   --set devices="\$DEFAULT_DEV" \\
   --set nodePort.directRoutingDevice="\$DEFAULT_DEV"
 

bastion/src/modules/modules/k3s/src/operations/cilium.ts

@@ -47,7 +47,9 @@ export const installCilium: Operation = async (ctx): Promise<OperationResult> =>
   const installResult = await ctx.ssh.exec(
     `KUBECONFIG=/etc/rancher/k3s/k3s.yaml cilium install \
       --set kubeProxyReplacement=true \
-      --set ipam.mode=kubernetes \
+      --set ipam.mode=cluster-pool \
+      --set ipam.operator.clusterPoolIPv4PodCIDRList='{10.42.0.0/16}' \
+      --set ipam.operator.clusterPoolIPv4MaskSize=24 \
       --set k8sServiceHost=127.0.0.1 \
       --set k8sServicePort=6444 \
       --set cni.exclusive=false \

bastion/src/shared/src/index.ts

@@ -10,6 +10,8 @@ export type {
   BastionConfig,
   VyosVlanSpec,
   VyosInstallSpec,
+  VyosBundle,
+  VyosBundleSetOp,
 } from "./types/index.js";
 
 export { SUPPORTED_OS, SUPPORTED_ROLES, ROLE_REGISTRY, isValidOsId } from "./types/index.js";

bastion/src/shared/src/types/index.ts

@@ -9,6 +9,8 @@ export type {
   BastionState,
   VyosVlanSpec,
   VyosInstallSpec,
+  VyosBundle,
+  VyosBundleSetOp,
 } from "./state.js";
 
 export { SUPPORTED_OS, SUPPORTED_ROLES, ROLE_REGISTRY, isValidOsId } from "./state.js";

bastion/src/shared/src/types/state.ts

@@ -88,6 +88,33 @@ export interface VyosVlanSpec {
   vrrp?: string;
 }
 
+/** One config node in a rendered bundle. Mirrors VyosSetOp on the bastion side. */
+export interface VyosBundleSetOp {
+  path: string[];
+  value?: string;
+  /** false appends to a multi-value node (e.g. bond members) instead of replacing. */
+  replace?: boolean;
+}
+
+/**
+ * A router's complete desired config, rendered from the Pulumi model.
+ *
+ * Produced by `kubernetes-deployment/scripts/vyos-render-bundle.ts` from the
+ * same subtree model `pulumi up` applies. The point is that labctl never
+ * authors VyOS config: bring-up replays what Pulumi already declares, so a
+ * freshly installed router and a `pulumi up` cannot disagree.
+ *
+ * Secret values arrive as `@secret:<key>` sentinels and are DROPPED at install
+ * time -- the bundle is committed to git and must stay safe to read. The router
+ * comes up on the LAN without its PPPoE credential; the first `pulumi up`
+ * supplies it. That handoff is deliberate.
+ */
+export interface VyosBundle {
+  sets: VyosBundleSetOp[];
+  /** Paths that are VyOS tag nodes — the installer's ConfigTree needs them marked. */
+  tags: string[][];
+}
+
 /**
  * VyOS-specific install parameters. Rendered into the config.boot that the
  * installer adopts, so the router comes up already configured.
@@ -96,6 +123,31 @@ export interface VyosVlanSpec {
  * PXE cannot run over LACP, so the install-time NIC has to stay unbonded.
  */
 export interface VyosInstallSpec {
+  /**
+   * A complete rendered config for this router. When present it REPLACES the
+   * derived interface/VLAN/VRRP config below -- the bundle already describes
+   * all of it, and deriving a second opinion is exactly the drift this exists
+   * to prevent. The remaining install parameters (password, disk, console) are
+   * still honoured because they are installer inputs, not router config.
+   */
+  bundle?: VyosBundle;
+  /**
+   * Key for the VyOS HTTP API, enabled at install so the router is manageable
+   * from the moment it boots.
+   *
+   * Without this the box comes up reachable only over SSH, and enabling the API
+   * later is a hand-run config change on a live firewall -- which is exactly the
+   * gap that left vyos001/vyos002 unmanageable by Pulumi after their cutover.
+   * The API is deliberately NOT part of the Pulumi model: a provider that
+   * manages its own transport can revoke its own access.
+   */
+  apiKey?: string;
+  /**
+   * Address the API listens on. Defaults to the management address when static.
+   * Never left unbound: an unrestricted listener puts a config-write endpoint on
+   * every segment the router touches, including the WAN.
+   */
+  apiListenAddress?: string;
   /** Interfaces aggregated into bond0 with LACP (802.3ad). Omit for no bond. */
   bondMembers?: string[];
   /** CIDR address on bond0 itself — the switch trunk's native/untagged VLAN. */

bastion/tests/integration/k3s-single-node.test.ts

@@ -161,7 +161,7 @@ EOF'
       CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
       curl -L --fail --silent "https://github.com/cilium/cilium-cli/releases/download/\${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz" | sudo tar xz -C /usr/local/bin
       DEFAULT_DEV=$(ip -4 route show default | awk '{print $5}' | head -1)
-      sudo KUBECONFIG=/etc/rancher/k3s/k3s.yaml cilium install --set kubeProxyReplacement=true --set ipam.mode=kubernetes --set devices=$DEFAULT_DEV --set nodePort.directRoutingDevice=$DEFAULT_DEV
+      sudo KUBECONFIG=/etc/rancher/k3s/k3s.yaml cilium install --set kubeProxyReplacement=true --set ipam.mode=cluster-pool --set ipam.operator.clusterPoolIPv4PodCIDRList='{10.42.0.0/16}' --set ipam.operator.clusterPoolIPv4MaskSize=24 --set devices=$DEFAULT_DEV --set nodePort.directRoutingDevice=$DEFAULT_DEV
     `.trim(), "cilium install", { keyPath: sshKeyPath, timeout: 120_000 });
 
     log("Waiting for Cilium to be ready...");

labsim/.gitignore

@@ -2,3 +2,7 @@
 *.log
 labsim_matrix_lib.py
 __pycache__/
+
+# Cluster-admin credentials for the rehearsal cluster, written by
+# `k8s-up.sh --kubeconfig`. Regenerate it rather than commit it.
+*.kubeconfig

labsim/README.md

@@ -15,10 +15,27 @@ Each VLAN is its own isolated libvirt network with one tiny Alpine VM on it.
 | 2 | k8s | 172.31.2.0/24 | 172.31.2.10 | 192.168.8.0/23 |
 | 3 | kvm | 172.31.3.0/24 | 172.31.3.10 | 192.168.3.0/24 |
 | 9 | private | 172.31.9.0/24 | 172.31.9.10 | 10.0.9.0/23 |
-| 10 | lot | 172.31.10.0/24 | 172.31.10.10 | 10.0.0.0/23 |
+| 10 | lot | **172.31.10.0/23** | 172.31.10.10 | 10.0.0.0/23 |
 | 200 | roomates | 172.31.200.0/24 | 172.31.200.10 | 192.168.2.0/24 |
 
-The sim subnet always encodes the VLAN id: `172.31.<vlan>.0/24`.
+The sim subnet encodes the VLAN id: `172.31.<vlan>.0/24`, with one exception.
+**VLAN 10 is a `/23`** because every UniFi DHCP reservation lives in LoT and LoT
+spans `10.0.0.x` *and* `10.0.1.x`, which a `/24` cannot hold. The mapping stays
+readable — `10.0.0.46 → 172.31.10.46`, `10.0.1.67 → 172.31.11.67`.
+
+LoT's host leg is `.3`, not `.2`, because `10.0.0.2` is a real reservation
+(Hubitat) that maps onto `172.31.10.2`. `.3` is unreserved and sits below the
+DHCP pool, so it can never be handed out.
+
+`vlans.conf` therefore takes two optional trailing fields:
+
+```
+vlan_id:name:sim_prefix:real_subnet[:masklen][:host_octet]
+```
+
+defaulting to `24` and `2`. k8s and Private are also `/23` in production but
+hold no reservations, so they keep their `/24` and their DHCP range is clamped
+— reported at generation time, never silently.
 
 Address plan, identical on every VLAN:
 
@@ -69,6 +86,54 @@ sudo virsh console labsim-2-k8s     # root / labsim
 ./monitoring-up.sh               # topology page + Prometheus + Grafana
 ```
 
+## Testing the DHCP migration
+
+`./labsim-dhcp-test.sh` boots throwaway VMs whose MACs are **real production
+MACs** and checks each gets the address UniFi reserved for it. MACs are the one
+piece of production config that transplants verbatim, which is what makes this a
+test rather than a rehearsal. It is safe because `ovs-labsim` has no physical
+NIC — verified with `ovs-vsctl show` — so a production MAC cannot reach the real
+LAN.
+
+Apply the config first, from `../migration`:
+
+```bash
+python3 unifi-to-vyos.py --mode sim -o /tmp/sim.conf   # 6 subnets, 31 mappings
+# load onto labsim-vyos, then:
+./labsim-dhcp-test.sh
+```
+
+**Result on VyOS 2026.08 (kea): all four cases pass.** The one that mattered:
+most UniFi reservations sit *inside* the DHCP pool, and **kea honours in-pool
+host reservations** — `printer1` received `172.31.10.46` from within the
+`.10.11–.11.254` pool. That was the open question blocking the cutover.
+
+### The lease database will lie to you
+
+The script wipes `/config/dhcp/dhcp4-leases.csv*` before every run, and both
+halves of that matter:
+
+- **Stale leases defeat reservations.** Re-running against yesterday's leases,
+  kea handed dynamic addresses to three devices that have reservations. The
+  reservation was present and correct in `/run/kea/kea-dhcp4.conf` the whole
+  time. Kea saw the reserved address as already leased to "another client" —
+  same MAC, but a different client-id from the earlier boot — and allocated
+  elsewhere. The cutover itself starts with an empty lease database, so this is
+  a *testing* artifact, but it is worth knowing that a reservation is not an
+  unconditional guarantee once leases exist.
+- **The `*` is load-bearing.** Kea's memfile backend keeps lease-file-cleanup
+  rotations (`dhcp4-leases.csv.2`) and restores from them on start, so
+  truncating only the primary file changes nothing.
+
+Both of those first appeared as a *passing* test. The verdict logic now refuses
+to score a MAC with more than one lease, because taking the first match had
+reported an hours-old lease as the current answer and turned three failures
+into apparent passes.
+
+Still open: whether kea will hand a *reserved* address to a *different* client
+while the reserved device is offline. The negative case here only proves an
+unreserved MAC gets an unreserved address.
+
 - **http://localhost:9101/** — live mesh: a node per VLAN, the router in the
   middle, one line per pair coloured green/red with the ICMP RTT on it. Hover a
   line for per-direction detail. Refreshes every 5s. This is the one to watch
@@ -78,6 +143,106 @@ sudo virsh console labsim-2-k8s     # root / labsim
 - **http://localhost:9101/metrics** — `labsim_reachable{src,dst,proto}` and
   `labsim_rtt_ms{src,dst}`.
 
+## Routing: BGP, dual WAN, and the ISP VMs
+
+`sim-ha-config.py` covers the LAN side of the routers. `sim-net-config.py`
+covers everything that makes this a rehearsal for production *routing*:
+
+| role | VM | what it generates |
+|---|---|---|
+| `primary` | `labsim-vyos` | BGP + dual WAN + health-checked failover |
+| `secondary` | `labsim-vyos2` | BGP only |
+| `isp-dhcp` | `labsim-isp-dhcp` | 10gig-equivalent ISP on VLAN 53 |
+| `isp-pppoe` | `labsim-isp-pppoe` | Vodafone-equivalent PPPoE ISP on VLAN 51 |
+
+Both ISP VMs are VyOS with two NICs: one on the OVS trunk facing the sim
+router, one on libvirt's `default` network, NATing customers to the real
+internet. They use RFC 5737 documentation ranges (`203.0.113.0/24`,
+`198.51.100.0/24`) so a leaked sim route cannot blackhole anything real.
+
+```sh
+./sim-net-apply.sh check     # VM state vs what the code says — run this first
+./sim-net-apply.sh apply     # push generated config over the serial console
+```
+
+`check` is the important one. All of this previously existed only as running
+state, applied by hand over SSH; rebuilding a VM lost it, and nothing recorded
+why any of it was shaped the way it was.
+
+### Known gaps vs production
+
+- **WAN is on the primary router only.** Production has WAN on both. Two PPPoE
+  clients sharing one credential against a single access concentrator is a
+  failure mode production does not have, so the sim does not model it. VRRP and
+  conntrack failover are still exercised.
+- **ISP VM interface names are not stable across a rebuild** — `isp-dhcp` came
+  up as `eth0`/`eth1` and `isp-pppoe` as `eth2`/`eth3` from identical XML.
+  Check `show interfaces` and pass `--wan-if` / `--uplink-if` rather than
+  trusting the defaults.
+- **`eth2` on the primary router** is a libvirt-NAT uplink predating the ISP
+  VMs: a third default route with no production equivalent that masks real WAN
+  failures during a failover test. `--drop-scaffold` removes it.
+- **Committing on `isp-pppoe` drops the router's PPPoE session**, and the
+  client does not redial promptly. After any change there, check `pppoe0` on
+  the router and `sudo systemctl restart ppp@pppoe0` if it is missing.
+
+## The trunk carries every VLAN tagged, including Management
+
+There is deliberately **no native/untagged VLAN** on the trunks to the routers,
+and Management lives on `bond0.1`, not on the bare `bond0`.
+
+A native VLAN is what puts a subnet on the bond **parent** while every other
+VLAN sits on a sub-interface of it. With `dhcp-socket-type: raw`, kea receives
+each tagged frame *twice* — once on the sub-interface and once on the parent —
+and answers from the parent's pool as well (ISC Kea
+[#1117](https://gitlab.isc.org/isc-projects/kea/-/issues/1117)). A client on
+VLAN 3 gets two OFFERs and keeps whichever arrives first:
+
+```
+bond0.3 :  172.31.3.252 → 172.31.3.11   correct
+bond0   :  172.31.1.252 → 172.31.1.8    UNTAGGED, Management pool, wrong
+```
+
+`./labsim-vlan-leak-test.sh` makes one client on a tagged VLAN send a DISCOVER
+and captures on the parent and the sub-interface at once. The verdict is how
+many OFFERs the **server** emitted and from which subnets — deliberately not
+"did the client get the right address", because a client picking correctly is
+exactly how this hid. Both orderings were observed across runs, so a passing
+client proves nothing.
+
+```sh
+./labsim-vlan-leak-test.sh --vlan 3           # PASS on the current shape
+LABSIM_NATIVE_VLAN=1 ./router-up.sh           # restore the old shape...
+./labsim-vlan-leak-test.sh --vlan 3           # ...and it FAILs again
+```
+
+Three things this cost, all of which apply to production:
+
+- **Kea must be restarted after the address moves.** VyOS does not restart it
+  for an interface address change, so it keeps a raw socket bound with the old
+  address and the bug survives the fix. In the sim kea had been running since
+  16 Aug; the first post-fix test failed for this reason alone and looked like
+  the fix simply not working.
+- **The firewall interface-group must move too.** `interface-group LAN` named
+  the bare `bond0`; with a default-deny ruleset, moving the address without
+  moving the group drops every management session and all VLAN 1 routing.
+- **Duplicate delivery does not stop.** #1117 says only that there is no longer
+  a subnet on the parent to match, and that is exactly what happens: two replies
+  per DISCOVER, both now from the correct pool. Harmless, but do not read a
+  duplicate as a failure.
+
+### Tagged and untagged Management coexist
+
+Verified directly, and it is what makes the production cutover a rolling change
+rather than an outage: with the primary still untagged on `bond0` and the
+secondary already tagged on `bond0.1`, both routers were reachable, the VIP
+stayed up and a VLAN 1 client kept its gateway. One VLAN is one broadcast
+domain regardless of how each port tags it, so the two firewalls can be
+converted one at a time. See `migration/MANAGEMENT-VLAN-TAGGED.md`.
+
+`./vlan1-move-monitor.sh` logs VIP/router liveness once a second during the
+change, because VRRP reconverges and leaves no trace of who held the VIP.
+
 ## Notes for whoever extends this
 
 Things that cost time the first time round, all verified on this image:
@@ -98,7 +263,14 @@ Things that cost time the first time round, all verified on this image:
 
 ## Not modelled (yet)
 
-VLANs are separate L2 segments rather than one 802.1Q trunk, so this exercises
+- **The secondary's bond was fiction until 2026-09-02.** `ovs_bond_router`'s
-inter-VLAN routing but not a `bond0.<vif>` trunk config specifically. A router
+  "already bonded, nothing to do" check compared only the trunk VLAN list, not
-VM would attach one NIC per VLAN. Adding a tagged-trunk variant is the obvious
+  the membership. Restarting a VM recreates its taps under new names, so the
-next step if the bond/vif config itself needs testing.
+  bond sat there holding two interfaces that no longer existed while the router's
+  real taps ran in the bridge as two *independent* ports — no LACP, and carrying
+  libvirt's own portgroup VLAN config rather than the bond's. It reconciles
+  membership now, but the lesson generalises: a sim that reports success is not
+  the same as a sim that models the thing.
+- **`labsim-vyos` has a third NIC** on libvirt's `default` network (the scaffold
+  uplink, see `--drop-scaffold`). The tap count is filtered to `$OVS_NET` for
+  that reason; an unfiltered count is 3 and silently skipped the primary's bond.

labsim/cilium-ipam-switch.sh

@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+# Procedure around a Cilium IPAM mode change. Works against any cluster, so the
+# rehearsal in labsim and the real thing in production run the SAME steps.
+#
+# It deliberately does NOT change the mode itself. In labsim that is `helm
+# upgrade`; in production Pulumi owns the release and a script racing it would
+# just reintroduce drift. What this owns is everything around the apply -- the
+# evidence, the deadlock, and the verdict.
+#
+#   ./cilium-ipam-switch.sh preflight   record what the cluster looks like now
+#   ./cilium-ipam-switch.sh unstick     break the agent-not-ready taint deadlock
+#   ./cilium-ipam-switch.sh verify      compare against preflight, report renumbering
+#
+#   KUBECONFIG=... ./cilium-ipam-switch.sh preflight
+#
+# Whether a recycle is needed is CONDITIONAL, and `verify` is what decides it.
+#
+# The operator does not preserve which node held which /24 -- it adopts whatever
+# CiliumNode.spec.ipam.podCIDRs already says. So:
+#
+#   * If CiliumNode already agrees with node.spec.podCIDRs on every node -- which
+#     is the case for any cluster that has only ever run ipam=kubernetes, because
+#     the operator syncs one from the other -- the pool adopts the existing
+#     allocation, no node is renumbered, and NO pod recycle is needed. Verified
+#     on the 3-node labsim cluster: CIDRs unchanged, nothing stranded, the only
+#     blip was the cilium DaemonSet restarting itself.
+#
+#   * If the two sources DISAGREE, nodes can swap /24s. Their running pods keep
+#     addresses that no longer fall inside the node's range, every other node
+#     routes that prefix to the wrong node, and those pods go unreachable
+#     cross-node while still showing Running. Then a full recycle is mandatory.
+#
+# Do not skip `verify` on the assumption of the good case. Run it and read it.
+set -uo pipefail
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+STATE="${STATE:-$SCRIPT_DIR/.ipam-switch-state}"
+K="kubectl"
+
+say()  { printf '\033[0;36m[ipam]\033[0m %s\n' "$*"; }
+warn() { printf '\033[1;33m[ipam]\033[0m %s\n' "$*" >&2; }
+
+snapshot() {
+  echo "## nodes"
+  $K get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.podCIDRs}{"\n"}{end}' 2>/dev/null
+  echo "## ciliumnodes"
+  $K get ciliumnode -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.ipam.podCIDRs}{"\n"}{end}' 2>/dev/null
+  echo "## pods"
+  $K get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\t"}{.status.podIP}{"\n"}{end}' 2>/dev/null \
+    | grep -vP '\t$' | sort
+  echo "## ipam"
+  $K -n kube-system get cm cilium-config -o jsonpath='{.data.ipam}' 2>/dev/null; echo
+}
+
+cmd_preflight() {
+  mkdir -p "$STATE"
+  snapshot > "$STATE/before.txt"
+  say "recorded $(grep -c . "$STATE/before.txt") lines -> $STATE/before.txt"
+  say "mode now: $(sed -n '/^## ipam/,$p' "$STATE/before.txt" | tail -1)"
+  # The pod inventory is the rollback reference: if the switch renumbers, this
+  # is the only record of what an address USED to be.
+  say "pods on the pod network: $(sed -n '/^## pods/,/^## ipam/p' "$STATE/before.txt" | grep -c '10\.')"
+}
+
+# The deadlock, in one place because it WILL happen and doing it by hand under
+# time pressure is how the wrong node gets untainted:
+#   agent has no pod CIDR -> agent not ready -> node keeps
+#   node.cilium.io/agent-not-ready:NoSchedule -> the operator that would assign
+#   the CIDR cannot schedule -> agent still has no pod CIDR.
+# Removing the taint is safe: it exists to keep normal workloads off a node
+# without working networking, and the operator is precisely the thing that fixes
+# that. Kubernetes re-adds it on the next agent restart.
+cmd_unstick() {
+  local stuck=0
+  for n in $($K get nodes -o name 2>/dev/null); do
+    $K get "$n" -o jsonpath='{.spec.taints[*].key}' 2>/dev/null | grep -q 'agent-not-ready' || continue
+    warn "${n#node/} carries agent-not-ready; removing so the operator can schedule"
+    $K taint "$n" node.cilium.io/agent-not-ready- >/dev/null 2>&1 && stuck=$((stuck+1))
+  done
+  [ "$stuck" -eq 0 ] && say "no node was stuck" || say "cleared $stuck node(s)"
+  local pend
+  pend="$($K -n kube-system get pods -l io.cilium/app=operator --no-headers 2>/dev/null | grep -c Pending)"
+  [ "${pend:-0}" -gt 0 ] && warn "$pend operator pod(s) still Pending — check tolerations, not just taints"
+  return 0
+}
+
+cmd_verify() {
+  [ -f "$STATE/before.txt" ] || { warn "no preflight snapshot; nothing to compare"; return 1; }
+  snapshot > "$STATE/after.txt"
+  echo
+  say "mode: $(sed -n '/^## ipam/,$p' "$STATE/before.txt" | tail -1) -> $(sed -n '/^## ipam/,$p' "$STATE/after.txt" | tail -1)"
+
+  # The question that decides the size of the maintenance window: did per-node
+  # CIDRs survive, or was every node renumbered (and every pod with it)?
+  local moved=0
+  while IFS=$'\t' read -r node cidr; do
+    [ -z "${node:-}" ] && continue
+    local now; now="$(sed -n '/^## ciliumnodes/,/^## pods/p' "$STATE/after.txt" | awk -F'\t' -v n="$node" '$1==n{print $2}')"
+    if [ -n "$now" ] && [ "$now" != "$cidr" ]; then
+      printf '    %-16s %s -> %s\n' "$node" "$cidr" "$now"; moved=$((moved+1))
+    fi
+  done < <(sed -n '/^## ciliumnodes/,/^## pods/p' "$STATE/before.txt" | grep -P '\t')
+  if [ "$moved" -eq 0 ]; then
+    say "per-node CIDRs UNCHANGED — the pool adopted the existing allocation"
+  else
+    warn "$moved node(s) renumbered — every pod on them must be recycled"
+  fi
+
+  local before after same
+  before="$(sed -n '/^## pods/,/^## ipam/p' "$STATE/before.txt" | grep -P '\t10\.' | wc -l)"
+  after="$(sed -n '/^## pods/,/^## ipam/p' "$STATE/after.txt"  | grep -P '\t10\.' | wc -l)"
+  same="$(comm -12 <(sed -n '/^## pods/,/^## ipam/p' "$STATE/before.txt" | grep -P '\t10\.' | sort) \
+                   <(sed -n '/^## pods/,/^## ipam/p' "$STATE/after.txt"  | grep -P '\t10\.' | sort) | wc -l)"
+  say "pods: $before before, $after after, $same kept the SAME address"
+  # Keeping the address is NOT the good outcome. If a node's CIDR moved, its
+  # existing pods keep IPs that no longer fall inside it, every other node routes
+  # that prefix to the WRONG node, and those pods go unreachable cross-node while
+  # looking perfectly healthy. Observed in labsim: two nodes swapped CIDRs and
+  # cross-node ping to their pods dropped 100%, with every pod still Running.
+  # This is the check that decides whether a recycle is optional or mandatory.
+  local stranded=0
+  while read -r ns name ip node; do
+    [ -z "${node:-}" ] && continue
+    local cidr; cidr="$($K get ciliumnode "$node" -o jsonpath='{.spec.ipam.podCIDRs[0]}' 2>/dev/null)"
+    [ -z "$cidr" ] && continue
+    case "$ip" in
+      "${cidr%.*/*}".*) ;;
+      *) printf '    STRANDED %-40s %-15s on %s (now %s)\n' "$ns/$name" "$ip" "$node" "$cidr"; stranded=$((stranded+1)) ;;
+    esac
+  done < <($K get pods -A -o jsonpath='{range .items[?(@.status.podIP)]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.podIP}{" "}{.spec.nodeName}{"\n"}{end}' 2>/dev/null | grep -E ' 10\.')
+  if [ "$stranded" -gt 0 ]; then
+    warn "$stranded pod(s) sit OUTSIDE their node CIDR — unreachable cross-node until recycled"
+    warn "recycle: for ns in $(kubectl get ns -o name | cut -d/ -f2); do kubectl -n $ns rollout restart deploy,ds,sts 2>/dev/null; done"
+  else
+    say "every pod is inside its node CIDR — no recycle needed"
+  fi
+  say "not-Running pods: $($K get pods -A --no-headers 2>/dev/null | grep -vcE 'Running|Completed')"
+}
+
+case "${1:-}" in
+  preflight) cmd_preflight ;;
+  unstick)   cmd_unstick ;;
+  verify)    cmd_verify ;;
+  *) sed -n '2,16p' "$0"; exit 1 ;;
+esac

labsim/console-apply.py

@@ -0,0 +1,109 @@
+#!/usr/bin/env python3
+"""Apply VyOS config to a labsim VM over its serial console.
+
+Needed because a freshly installed VyOS comes up holding the same addresses as
+its peer, so there is a window where it cannot safely be reached over the
+network at all. The console does not care.
+
+  ./console-apply.py --vm labsim-vyos2 --config r2.conf
+"""
+from __future__ import annotations
+
+import argparse
+import sys
+import time
+
+import pexpect
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--vm", required=True)
+    ap.add_argument("--config", required=True)
+    ap.add_argument("--user", default="vyos")
+    ap.add_argument("--password", default="vyos")
+    args = ap.parse_args()
+
+    cmds = [l.rstrip() for l in open(args.config)
+            if l.strip() and not l.lstrip().startswith("#")]
+    print(f"{len(cmds)} commands to apply to {args.vm}", file=sys.stderr)
+
+    c = pexpect.spawn(f"virsh --connect qemu:///system console {args.vm}",
+                      timeout=90, encoding="utf-8")
+    c.logfile_read = None
+    c.sendline("")
+    time.sleep(2)
+    c.sendline("")
+
+    # Log in. A freshly booted box may still be starting services, so allow a
+    # generous window and re-prod the console rather than failing on the first
+    # miss.
+    #
+    # `# ` matters as much as `$ `: a previous run that died mid-config leaves
+    # the console sitting in configuration mode, and waiting only for the
+    # operational prompt then hangs forever against a perfectly healthy VM.
+    in_config = False
+    for _ in range(40):
+        i = c.expect([r"login:", r"\$ ", r"# ", pexpect.TIMEOUT], timeout=15)
+        if i == 0:
+            c.sendline(args.user)
+            c.expect("Password:", timeout=30)
+            c.sendline(args.password)
+            c.expect([r"\$ ", r"# "], timeout=60)
+            break
+        if i == 1:
+            break
+        if i == 2:
+            in_config = True
+            break
+        c.sendline("")
+    else:
+        print("never reached a prompt", file=sys.stderr)
+        return 1
+
+    if in_config:
+        # Drop whatever the previous run left half-built rather than committing
+        # a candidate nobody has seen.
+        print("console was left in config mode; discarding stale candidate",
+              file=sys.stderr)
+        c.sendline("discard")
+        c.expect(r"# ", timeout=60)
+    else:
+        c.sendline("configure")
+        c.expect(r"# ", timeout=60)
+
+    for cmd in cmds:
+        c.sendline(cmd)
+        c.expect(r"# ", timeout=60)
+        out = c.before or ""
+        if "Set failed" in out or "not valid" in out or "Invalid" in out:
+            print(f"FAILED: {cmd}\n  {out.strip()[:200]}", file=sys.stderr)
+
+    print("committing...", file=sys.stderr)
+    c.sendline("commit")
+    c.expect(r"# ", timeout=300)
+    commit_out = c.before or ""
+    c.sendline("save")
+    c.expect(r"# ", timeout=120)
+    # Accept either prompt on the way out. Insisting on `$ ` here hangs against
+    # a healthy box -- and worse, leaves the console parked in config mode, so
+    # the NEXT run finds a `# ` it was not expecting either. One strict expect
+    # turned into two failures.
+    c.sendline("exit")
+    c.expect([r"\$ ", r"# ", pexpect.TIMEOUT], timeout=60)
+    c.sendline("exit")
+    c.close(force=True)
+
+    bad = [l for l in commit_out.splitlines()
+           if "failed" in l.lower() or "error" in l.lower()]
+    if bad:
+        print("commit reported:", file=sys.stderr)
+        for l in bad[:10]:
+            print(f"  {l.strip()}", file=sys.stderr)
+        return 1
+    print("committed and saved", file=sys.stderr)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

labsim/dualstack-lab.sh

@@ -0,0 +1,318 @@
+#!/usr/bin/env bash
+# Differential study: what ACTUALLY differs between a k3s cluster born
+# dual-stack and one converted in place?
+#
+# k3s says dual-stack "cannot be enabled on an existing cluster". The stated
+# reason is narrow -- nodes get Pod CIDRs only at join and the Kubernetes IPAM
+# controller will not hand out a new IPv6 CIDR later -- and it does not obviously
+# apply to a cluster where Cilium owns IPAM. Rather than argue from docs, build
+# both shapes and diff them.
+#
+#   ./dualstack-lab.sh up v4         single-node k3s, IPv4 only     (.21)
+#   ./dualstack-lab.sh up dual       single-node k3s, dual-stack    (.22)
+#   ./dualstack-lab.sh pristine v4   reflink copy of v4's disk, so the upgrade
+#                                    attempt can be rolled back and retried
+#   ./dualstack-lab.sh restore v4    put that copy back
+#   ./dualstack-lab.sh collect <n>   normalized state dump -> evidence/<n>/
+#   ./dualstack-lab.sh compare a b   semantic diff of two collections
+#   ./dualstack-lab.sh virtdiff a b  whole-filesystem diff, offline (libguestfs)
+#   ./dualstack-lab.sh down [name]
+#
+# The comparison that matters is `compare dual upgraded`: everything it prints
+# is a way the converted cluster failed to reach the shape of a native one.
+#
+# Single node on purpose. Dual-stack is decided by server flags and CNI config,
+# both of which a one-node cluster exercises fully, and it rebuilds in minutes.
+# Node-rejoin behaviour needs the 3-node cluster and is a separate question.
+set -euo pipefail
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+source "$SCRIPT_DIR/lib.sh"
+source "$SCRIPT_DIR/ovs.sh"
+
+K8S_VLAN="${K8S_VLAN:-2}"
+DS_PREFIX="${DS_PREFIX:-172.31.2}"
+MEM="${MEM:-4096}"; CPUS="${CPUS:-2}"; DISK_GB="${DISK_GB:-12}"
+TOKEN="${TOKEN:-labsim-ds-token}"
+CILIUM_VERSION="${CILIUM_VERSION:-1.19.1}"   # same as production
+DEB_BASE="${DEB_BASE:-$IMG_DIR/debian-13-genericcloud-amd64.qcow2}"
+EVIDENCE="$SCRIPT_DIR/dualstack-evidence"
+
+# Pod/Service ranges. IPv4 halves are k3s's own defaults, so the v4-only build is
+# a stock cluster and the diff is not polluted by gratuitous differences.
+# IPv6 halves are ULA: this cluster never routes off-box, and using the real /48
+# here would put lab addresses into a prefix that production also announces.
+V4_CLUSTER="10.42.0.0/16"; V4_SERVICE="10.43.0.0/16"
+V6_CLUSTER="${V6_CLUSTER:-fd00:42::/56}"
+V6_SERVICE="${V6_SERVICE:-fd00:43::/112}"   # /112 -- apiserver caps v6 service ranges
+V6_PREFIX="${V6_PREFIX:-fd00:2}"            # node addresses: fd00:2::<octet>
+
+vm_name() { echo "labsim-ds-$1"; }
+vm_ip()   { case "$1" in v4) echo "$DS_PREFIX.21";; dual) echo "$DS_PREFIX.22";; *) die "unknown build '$1'";; esac; }
+vm_ip6()  { case "$1" in v4) echo "$V6_PREFIX::21";; dual) echo "$V6_PREFIX::22";; *) die "unknown build '$1'";; esac; }
+disk_of() { echo "$IMG_DIR/$(vm_name "$1").qcow2"; }
+
+ssh_vm() { local ip="$1"; shift; ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
+             -o LogLevel=ERROR -o ConnectTimeout=8 -o BatchMode=yes "debian@$ip" "$@"; }
+
+# --- seed -----------------------------------------------------------------
+build_seed() {
+  local iso="$1" vm="$2" mode="$3" pubkey="$4"
+  local ip ip6 tmp; ip="$(vm_ip "$mode")"; ip6="$(vm_ip6 "$mode")"; tmp="$(mktemp -d)"
+
+  echo "instance-id: $vm" > "$tmp/meta-data"
+  # Static v6 on both builds. The v4-only cluster still gets an IPv6 ADDRESS --
+  # only its Kubernetes config is v4-only. Otherwise the diff would be dominated
+  # by host addressing rather than by what Kubernetes did differently.
+  cat > "$tmp/network-config" <<EOF
+version: 2
+ethernets:
+  enp1s0:
+    addresses: [${ip}/24, ${ip6}/64]
+    routes:
+      - to: default
+        via: ${DS_PREFIX}.1
+    nameservers:
+      addresses: [8.8.8.8, 1.1.1.1]
+EOF
+
+  local exec_args="server --flannel-backend=none --disable-network-policy --disable=servicelb --disable=traefik --tls-san=$ip --cluster-init"
+  if [ "$mode" = dual ]; then
+    exec_args="$exec_args --cluster-cidr=${V4_CLUSTER},${V6_CLUSTER} --service-cidr=${V4_SERVICE},${V6_SERVICE} --node-ip=${ip},${ip6}"
+  else
+    exec_args="$exec_args --node-ip=${ip}"
+  fi
+
+  cat > "$tmp/user-data" <<EOF
+#cloud-config
+hostname: $vm
+fqdn: $vm
+users:
+  - name: debian
+    groups: [sudo]
+    shell: /bin/bash
+    sudo: ["ALL=(ALL) NOPASSWD:ALL"]
+    lock_passwd: false
+    plain_text_passwd: labsim
+    ssh_authorized_keys: [$pubkey]
+ssh_pwauth: true
+disable_root: false
+package_update: true
+packages: [curl, jq, iproute2, nftables]
+write_files:
+  - path: /etc/modules-load.d/cilium.conf
+    content: |
+      br_netfilter
+  - path: /etc/dualstack-lab-mode
+    content: |
+      $mode
+runcmd:
+  - modprobe br_netfilter || true
+  - |
+    curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="$exec_args" K3S_TOKEN="$TOKEN" sh -
+  - |
+    # Cilium via helm, matching the production version. IPAM stays 'kubernetes'
+    # in BOTH builds on purpose: that is what production runs, and it is the
+    # mode the k3s objection is actually about. If the converted cluster needs
+    # cluster-pool to work, the diff should be what tells us so.
+    curl -sfL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash || true
+    export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
+    helm repo add cilium https://helm.cilium.io >/dev/null 2>&1 || true
+    helm repo update >/dev/null 2>&1 || true
+    for i in \$(seq 1 60); do kubectl get nodes >/dev/null 2>&1 && break; sleep 5; done
+    if [ "$mode" = dual ]; then
+      helm install cilium cilium/cilium --version $CILIUM_VERSION -n kube-system \\
+        --set kubeProxyReplacement=false --set ipam.mode=kubernetes \\
+        --set ipv4.enabled=true --set ipv6.enabled=true \\
+        --set k8sServiceHost=$ip --set k8sServicePort=6443 || true
+    else
+      helm install cilium cilium/cilium --version $CILIUM_VERSION -n kube-system \\
+        --set kubeProxyReplacement=false --set ipam.mode=kubernetes \\
+        --set ipv4.enabled=true --set ipv6.enabled=false \\
+        --set k8sServiceHost=$ip --set k8sServicePort=6443 || true
+    fi
+    touch /etc/dualstack-lab-ready
+EOF
+  sudo mkdir -p "$(dirname "$iso")"
+  sudo genisoimage -quiet -output "$iso" -volid cidata -joliet -rock \
+    "$tmp/user-data" "$tmp/meta-data" "$tmp/network-config"
+  rm -rf "$tmp"
+}
+
+cmd_up() {
+  local mode="${1:?usage: up <v4|dual>}"
+  local vm ip disk seed pubkey
+  vm="$(vm_name "$mode")"; ip="$(vm_ip "$mode")"; disk="$(disk_of "$mode")"
+  seed="$IMG_DIR/${vm}-seed.iso"; pubkey="$(find_ssh_pubkey)"
+
+  [ -f "$DEB_BASE" ] || die "base image missing: $DEB_BASE (run ./k8s-up.sh once)"
+  if virsh_q dominfo "$vm" >/dev/null 2>&1; then
+    log "$vm exists — starting if stopped"
+    [ "$(virsh_q domstate "$vm" | head -1)" = "running" ] || virsh_q start "$vm" >/dev/null
+    return
+  fi
+  selected_vlans; ovs_up
+  log "creating $vm ($mode) at $ip / $(vm_ip6 "$mode")"
+  sudo qemu-img create -q -f qcow2 -F qcow2 -b "$DEB_BASE" "$disk" "${DISK_GB}G" >/dev/null
+  build_seed "$seed" "$vm" "$mode" "$pubkey"
+  sudo virt-install --connect "$LIBVIRT_URI" --name "$vm" \
+    --memory "$MEM" --vcpus "$CPUS" \
+    --disk "path=$disk,format=qcow2,bus=virtio" \
+    --disk "path=$seed,device=cdrom" \
+    --network "network=$OVS_NET,portgroup=vlan${K8S_VLAN},model=virtio" \
+    --os-variant debian12 --graphics none --noautoconsole --import >/dev/null
+  log "installing in background; watch: ssh debian@$ip 'ls /etc/dualstack-lab-ready'"
+}
+
+# --- pristine copy / restore ---------------------------------------------
+# reflink so the copy is instant and independent on btrfs/xfs. A qcow2 backing
+# chain would be cheaper still but makes the parent read-only in practice: boot
+# the parent again and every child silently corrupts.
+cmd_pristine() {
+  local mode="${1:?usage: pristine <v4|dual>}" vm disk
+  vm="$(vm_name "$mode")"; disk="$(disk_of "$mode")"
+  [ "$(virsh_q domstate "$vm" 2>/dev/null | head -1)" = "running" ] && \
+    die "$vm is running — shut it down first (virsh shutdown $vm), a copy of a live disk is not consistent"
+  sudo cp --reflink=auto "$disk" "${disk}.pristine"
+  log "pristine copy: ${disk}.pristine"
+}
+cmd_restore() {
+  local mode="${1:?usage: restore <v4|dual>}" vm disk
+  vm="$(vm_name "$mode")"; disk="$(disk_of "$mode")"
+  [ -f "${disk}.pristine" ] || die "no pristine copy for $mode"
+  [ "$(virsh_q domstate "$vm" 2>/dev/null | head -1)" = "running" ] && \
+    die "$vm is running — shut it down first"
+  sudo cp --reflink=auto "${disk}.pristine" "$disk"
+  log "restored $mode from pristine"
+}
+
+
+# --- the experiment ------------------------------------------------------
+# Convert the IPv4-only cluster in place, mirroring the flags the native build
+# was BORN with. Each step prints what the cluster did, because the interesting
+# output is which step refuses rather than whether the end state is pretty.
+cmd_upgrade() {
+  local ip; ip="$(vm_ip v4)"; local ip6; ip6="$(vm_ip6 v4)"
+  log "step 1/4: add dual CIDRs + dual node-ip to the k3s unit"
+  # Done with python on the box, not nested sed: quoting a multi-line systemd
+  # continuation through ssh -> sh -> sed produced a literal \\n in the unit, and
+  # k3s then saw a dual cluster-cidr with a still-IPv4 service-cidr and refused
+  # to start. All three flags go on one line -- systemd does not care, and there
+  # is nothing left to escape.
+  ssh_vm "$ip" "sudo python3 - <<'PYEOF'
+import re
+u = '/etc/systemd/system/k3s.service'
+s = open(u).read()
+old = \"'--node-ip=${ip}'\"
+new = \"'--cluster-cidr=${V4_CLUSTER},${V6_CLUSTER}' '--service-cidr=${V4_SERVICE},${V6_SERVICE}' '--node-ip=${ip},${ip6}'\"
+assert old in s, 'node-ip flag not found in unit'
+open(u,'w').write(s.replace(old, new))
+print('    unit rewritten')
+PYEOF
+sudo systemctl daemon-reload" || die "unit edit failed"
+  ssh_vm "$ip" "grep -oE \"'--(cluster|service)-cidr=[^']*'|'--node-ip=[^']*'\" /etc/systemd/system/k3s.service | sed 's/^/    /'"
+
+  log "step 2/4: restart k3s and see whether it accepts the changed ranges"
+  ssh_vm "$ip" "sudo systemctl restart k3s" || true
+  for i in $(seq 1 40); do
+    ssh_vm "$ip" "sudo k3s kubectl get --raw /readyz >/dev/null 2>&1" && break
+    sleep 5
+  done
+  ssh_vm "$ip" "sudo journalctl -u k3s --since '2 min ago' --no-pager 2>/dev/null | grep -iE 'cidr|dual|ipv6|invalid|cannot|fail' | tail -12 | sed 's/^/    /'" || true
+
+  log "step 3/4: what the API says now"
+  ssh_vm "$ip" "echo -n '    servicecidr: '; sudo k3s kubectl get servicecidr -o jsonpath='{.items[*].spec.cidrs}'; echo; \
+                echo -n '    node podCIDRs: '; sudo k3s kubectl get node -o jsonpath='{.items[0].spec.podCIDRs}'; echo; \
+                echo -n '    node addresses: '; sudo k3s kubectl get node -o jsonpath='{.items[0].status.addresses[*].address}'; echo" || true
+
+  log "step 4/4: turn on IPv6 in Cilium"
+  ssh_vm "$ip" "export KUBECONFIG=/etc/rancher/k3s/k3s.yaml; sudo -E helm upgrade cilium cilium/cilium --version ${CILIUM_VERSION} -n kube-system --reuse-values --set ipv6.enabled=true >/dev/null 2>&1 && echo '    cilium upgraded' || echo '    cilium upgrade FAILED'" || true
+  ssh_vm "$ip" "sudo k3s kubectl -n kube-system rollout restart ds/cilium >/dev/null 2>&1; sleep 20; sudo k3s kubectl -n kube-system get pods -l k8s-app=cilium --no-headers | sed 's/^/    /'" || true
+  log "now: ./dualstack-lab.sh collect upgraded ${ip} && ./dualstack-lab.sh compare dual upgraded"
+}
+
+# --- evidence collection --------------------------------------------------
+# Normalized on purpose. Two independently built clusters differ in certs,
+# tokens, UUIDs, timestamps and log lines; left raw, that noise buries the
+# handful of differences that actually mean something.
+cmd_collect() {
+  local name="${1:?usage: collect <name> [ip]}"
+  local ip="${2:-}"
+  [ -n "$ip" ] || ip="$(vm_ip "$name" 2>/dev/null || true)"
+  [ -n "$ip" ] || die "collect: give an ip for a non-standard name"
+  local out="$EVIDENCE/$name"; mkdir -p "$out"
+  log "collecting from $name ($ip) -> $out"
+
+  ssh_vm "$ip" 'sudo cat /etc/rancher/k3s/config.yaml 2>/dev/null; sudo systemctl cat k3s 2>/dev/null | grep -A30 ExecStart' \
+    > "$out/k3s-config.txt" 2>/dev/null || true
+  ssh_vm "$ip" 'sudo tr "\0" "\n" < /proc/$(pgrep -f "k3s server" | head -1)/cmdline | grep -v "^$"' \
+    > "$out/k3s-cmdline.txt" 2>/dev/null || true
+  ssh_vm "$ip" 'ip -o addr show | awk "{print \$2, \$3, \$4}"; echo ---; ip -4 route show; echo ---; ip -6 route show' \
+    > "$out/host-net.txt" 2>/dev/null || true
+  ssh_vm "$ip" 'sudo sysctl -a 2>/dev/null | grep -E "net\.ipv6\.conf\.(all|default)\.(forwarding|disable_ipv6)|net\.ipv4\.ip_forward"' \
+    > "$out/sysctl.txt" 2>/dev/null || true
+
+  local K='sudo k3s kubectl'
+  ssh_vm "$ip" "$K get servicecidr -o yaml"       > "$out/servicecidr.yaml" 2>/dev/null || true
+  ssh_vm "$ip" "$K get nodes -o yaml"             > "$out/nodes.yaml.raw"   2>/dev/null || true
+  ssh_vm "$ip" "$K get ciliumnodes -o yaml"       > "$out/ciliumnodes.yaml.raw" 2>/dev/null || true
+  ssh_vm "$ip" "$K -n kube-system get cm cilium-config -o yaml" > "$out/cilium-config.yaml.raw" 2>/dev/null || true
+  ssh_vm "$ip" "$K get svc -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,FAMILYPOLICY:.spec.ipFamilyPolicy,FAMILIES:.spec.ipFamilies,IPS:.spec.clusterIPs" \
+    > "$out/services.txt" 2>/dev/null || true
+  ssh_vm "$ip" "$K get pods -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,IPS:.status.podIPs" \
+    > "$out/podips.txt" 2>/dev/null || true
+
+  # Strip the things that differ every build regardless of configuration.
+  for f in "$out"/*.raw; do
+    [ -e "$f" ] || continue
+    sed -E \
+      -e 's/[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9:]+Z?/<TIME>/g' \
+      -e 's/(uid|resourceVersion|creationTimestamp|generation|observedGeneration): .*/\1: <X>/' \
+      -e 's/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/<UUID>/g' \
+      -e 's/(LS0tLS1|[A-Za-z0-9+\/]{60,}=*)/<B64>/g' \
+      "$f" > "${f%.raw}"
+    rm -f "$f"
+  done
+  log "collected $(ls "$out" | wc -l) artefacts"
+}
+
+cmd_compare() {
+  local a="${1:?usage: compare <a> <b>}" b="${2:?}"
+  [ -d "$EVIDENCE/$a" ] && [ -d "$EVIDENCE/$b" ] || die "collect both first"
+  echo "### semantic diff: $a (<) vs $b (>)"
+  diff -ru "$EVIDENCE/$a" "$EVIDENCE/$b" || true
+}
+
+cmd_virtdiff() {
+  local a="${1:?usage: virtdiff <a> <b>}" b="${2:?}"
+  for m in "$a" "$b"; do
+    [ "$(virsh_q domstate "$(vm_name "$m")" 2>/dev/null | head -1)" = "running" ] && \
+      die "$(vm_name "$m") is running — virt-diff needs the disks quiescent"
+  done
+  log "whole-filesystem diff (slow); noise is expected — use it to find what the collector missed"
+  sudo virt-diff -a "$(disk_of "$a")" -A "$(disk_of "$b")" \
+    | grep -vE '/(var/log|tmp|run|proc|sys)/|\.log$|/var/lib/rancher/k3s/(server/(tls|cred|db)|agent)' || true
+}
+
+cmd_down() {
+  local only="${1:-}"
+  for m in v4 dual; do
+    [ -n "$only" ] && [ "$only" != "$m" ] && continue
+    local vm; vm="$(vm_name "$m")"
+    virsh_q dominfo "$vm" >/dev/null 2>&1 || continue
+    log "removing $vm"
+    virsh_q destroy "$vm" >/dev/null 2>&1 || true
+    virsh_q undefine "$vm" --remove-all-storage >/dev/null 2>&1 || true
+  done
+}
+
+case "${1:-}" in
+  up)       shift; cmd_up "$@" ;;
+  upgrade)  shift; cmd_upgrade "$@" ;;
+  pristine) shift; cmd_pristine "$@" ;;
+  restore)  shift; cmd_restore "$@" ;;
+  collect)  shift; cmd_collect "$@" ;;
+  compare)  shift; cmd_compare "$@" ;;
+  virtdiff) shift; cmd_virtdiff "$@" ;;
+  down)     shift; cmd_down "$@" ;;
+  *) sed -n '2,30p' "$0"; exit 1 ;;
+esac

labsim/k8s-up.sh

@@ -0,0 +1,252 @@
+#!/bin/bash
+# A real Kubernetes cluster inside labsim, on the OVS fabric, for rehearsing
+# Cilium <-> VyOS BGP before it goes near the production routers.
+#
+# Why VMs and not k3d: the thing under test is eBGP between Cilium and VyOS
+# across the switch fabric — nodes on VLAN 2, peering with the router's bond0.2
+# leg, directly connected. k3d would put the nodes on a container bridge, which
+# is a different L2 path and would prove something else. (It also needs Docker;
+# this host has podman.)
+#
+# Why not the existing micro VMs: they are Alpine with 256 MB and 1 vCPU. k3s
+# plus Cilium needs an order of magnitude more, and a glibc distro with a stock
+# kernel that Cilium's eBPF probes are actually tested against.
+#
+# Three nodes, not two: ECMP is only meaningfully tested if a node can be
+# drained and MORE THAN ONE path survives.
+#
+# Layout (mirrors production's shape, not its addresses):
+#   labsim-k8s1   172.31.2.11   k3s server
+#   labsim-k8s2   172.31.2.12   agent
+#   labsim-k8s3   172.31.2.13   agent
+#   gateway       172.31.2.1    the VRRP VIP of the router pair under test
+#   BGP peers     172.31.2.252 / .253   the routers' real per-box addresses
+#
+# Idempotent: re-running only creates what is missing.
+#
+# Usage:
+#   ./k8s-up.sh              create/start the cluster
+#   ./k8s-up.sh --kubeconfig fetch kubeconfig to ./labsim-k8s.kubeconfig
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+source "$SCRIPT_DIR/lib.sh"
+source "$SCRIPT_DIR/ovs.sh"
+
+# --- knobs ----------------------------------------------------------------
+K8S_VLAN="${K8S_VLAN:-2}"
+K8S_PREFIX="${K8S_PREFIX:-172.31.2}"
+K8S_NODES="${K8S_NODES:-3}"
+K8S_FIRST_OCTET="${K8S_FIRST_OCTET:-11}"
+K8S_MEM="${K8S_MEM:-4096}"          # MB — k3s + cilium + a workload
+K8S_CPUS="${K8S_CPUS:-2}"
+K8S_DISK_GB="${K8S_DISK_GB:-12}"
+K8S_TOKEN="${K8S_TOKEN:-labsim-k3s-token}"
+
+# Debian rather than Alpine: glibc, a stock kernel, and cloud-init that applies
+# network-config properly (the Alpine base in this sim notably does not).
+DEB_URL="${DEB_URL:-https://cloud.debian.org/images/cloud/trixie/latest/debian-13-genericcloud-amd64.qcow2}"
+DEB_BASE="${DEB_BASE:-$IMG_DIR/debian-13-genericcloud-amd64.qcow2}"
+
+# Same version production runs, so CRD shapes and chart flags transfer exactly.
+CILIUM_VERSION="${CILIUM_VERSION:-1.19.1}"
+
+node_name() { echo "labsim-k8s$1"; }
+node_ip()   { echo "${K8S_PREFIX}.$((K8S_FIRST_OCTET + $1 - 1))"; }
+
+# --- base image -----------------------------------------------------------
+ensure_base_image() {
+  if [ -f "$DEB_BASE" ]; then
+    log "base image present: $(basename "$DEB_BASE")"
+    return
+  fi
+  log "fetching Debian cloud image (~330 MB) -> $DEB_BASE"
+  sudo mkdir -p "$IMG_DIR"
+  # .tmp + mv so an interrupted download never leaves a half image that later
+  # runs treat as valid.
+  sudo curl -fsSL --retry 3 -o "${DEB_BASE}.tmp" "$DEB_URL" \
+    || die "could not fetch $DEB_URL"
+  sudo mv "${DEB_BASE}.tmp" "$DEB_BASE"
+  log "base image ready"
+}
+
+# --- cloud-init -----------------------------------------------------------
+# The server node writes the join token; agents wait for the API to answer
+# before joining, because cloud-init ordering across VMs is not guaranteed and
+# a failed join leaves an agent that never retries.
+build_k8s_seed() {
+  local iso="$1" vm="$2" ip="$3" role="$4" server_ip="$5" pubkey="$6"
+  local tmp; tmp="$(mktemp -d)"
+
+  cat > "$tmp/meta-data" <<EOF
+instance-id: $vm
+local-hostname: $vm
+EOF
+
+  cat > "$tmp/network-config" <<EOF
+version: 2
+ethernets:
+  enp1s0:
+    match:
+      name: "en*"
+    addresses: [$ip/24]
+    routes:
+      - to: default
+        via: ${K8S_PREFIX}.1
+    nameservers:
+      addresses: [8.8.8.8, 1.1.1.1]
+EOF
+
+  local k3s_exec
+  if [ "$role" = "server" ]; then
+    # flannel/servicelb/traefik off: Cilium is the CNI under test, and k3s's
+    # own ServiceLB would fight Cilium for LoadBalancer addresses.
+    k3s_exec="server --flannel-backend=none --disable-network-policy --disable=servicelb --disable=traefik --node-ip=$ip --tls-san=$ip --cluster-init"
+  else
+    k3s_exec="agent --server https://${server_ip}:6443 --node-ip=$ip"
+  fi
+
+  cat > "$tmp/user-data" <<EOF
+#cloud-config
+hostname: $vm
+fqdn: $vm
+users:
+  - name: debian
+    groups: [sudo]
+    shell: /bin/bash
+    sudo: ["ALL=(ALL) NOPASSWD:ALL"]
+    lock_passwd: false
+    plain_text_passwd: labsim
+    ssh_authorized_keys:
+      - $pubkey
+ssh_pwauth: true
+disable_root: false
+ssh_authorized_keys:
+  - $pubkey
+
+package_update: true
+packages: [curl, jq, iproute2, tcpdump, bird2]
+
+write_files:
+  # Cilium replaces kube-proxy and needs these; Debian cloud images ship
+  # neither loaded nor persisted.
+  - path: /etc/modules-load.d/cilium.conf
+    content: |
+      br_netfilter
+      overlay
+  - path: /etc/sysctl.d/99-k8s.conf
+    content: |
+      net.ipv4.ip_forward = 1
+      net.bridge.bridge-nf-call-iptables = 1
+
+runcmd:
+  - [ modprobe, br_netfilter ]
+  - [ modprobe, overlay ]
+  - [ sysctl, --system ]
+  - |
+    # Wait for the server's API before an agent tries to join. Without this the
+    # agent fails once and the unit backs off for minutes.
+    if [ "$role" != "server" ]; then
+      for i in \$(seq 1 60); do
+        curl -sk --max-time 3 https://${server_ip}:6443/ping >/dev/null 2>&1 && break
+        sleep 5
+      done
+    fi
+  - |
+    curl -sfL https://get.k3s.io | \
+      INSTALL_K3S_EXEC="$k3s_exec" \
+      K3S_TOKEN="$K8S_TOKEN" \
+      sh -
+EOF
+
+  sudo mkdir -p "$(dirname "$iso")"
+  sudo genisoimage -quiet -output "$iso" -volid cidata -joliet -rock \
+    "$tmp/user-data" "$tmp/meta-data" "$tmp/network-config"
+  rm -rf "$tmp"
+}
+
+# --- VM creation ----------------------------------------------------------
+create_node() {
+  local n="$1" pubkey="$2"
+  local vm; vm="$(node_name "$n")"
+  local ip; ip="$(node_ip "$n")"
+  local role="agent"; [ "$n" -eq 1 ] && role="server"
+  local server_ip; server_ip="$(node_ip 1)"
+
+  if virsh_q dominfo "$vm" >/dev/null 2>&1; then
+    local state; state="$(virsh_q domstate "$vm" 2>/dev/null | head -1 | tr -d '\n')"
+    if [ "$state" = "running" ]; then
+      log "$vm already running ($ip, $role)"
+    else
+      log "$vm exists but is $state — starting"
+      virsh_q start "$vm" >/dev/null
+    fi
+    return
+  fi
+
+  local disk="$IMG_DIR/${vm}.qcow2"
+  local seed="$IMG_DIR/${vm}-seed.iso"
+
+  log "creating $vm ($ip, $role, ${K8S_MEM}MB/${K8S_CPUS}cpu)"
+  sudo qemu-img create -q -f qcow2 -F qcow2 -b "$DEB_BASE" "$disk" "${K8S_DISK_GB}G" >/dev/null
+  build_k8s_seed "$seed" "$vm" "$ip" "$role" "$server_ip" "$pubkey"
+
+  # Access port on the k8s VLAN — same broadcast domain as the routers'
+  # bond0.2 leg, so BGP peering is directly connected exactly as in production.
+  sudo virt-install --connect "$LIBVIRT_URI" --name "$vm" \
+    --memory "$K8S_MEM" --vcpus "$K8S_CPUS" \
+    --disk "path=$disk,format=qcow2,bus=virtio" \
+    --disk "path=$seed,device=cdrom" \
+    --network "network=$OVS_NET,portgroup=vlan${K8S_VLAN},model=virtio" \
+    --os-variant debian12 \
+    --graphics none --noautoconsole --import >/dev/null
+}
+
+fetch_kubeconfig() {
+  local server_ip; server_ip="$(node_ip 1)"
+  local out="$SCRIPT_DIR/labsim-k8s.kubeconfig"
+  log "fetching kubeconfig from $server_ip"
+  ssh -o StrictHostKeyChecking=no -o ConnectTimeout=10 \
+      "debian@${server_ip}" "sudo cat /etc/rancher/k3s/k3s.yaml" \
+    | sed "s|127.0.0.1|${server_ip}|" > "$out"
+  chmod 600 "$out"
+  log "wrote $out"
+  log "use: KUBECONFIG=$out kubectl get nodes"
+}
+
+main() {
+  if [ "${1:-}" = "--kubeconfig" ]; then
+    fetch_kubeconfig
+    return
+  fi
+
+  require_tools
+  command -v genisoimage >/dev/null || die "genisoimage missing (dnf install genisoimage)"
+
+  local pubkey; pubkey="$(find_ssh_pubkey)"
+  log "using SSH key: ${pubkey%% *} ...${pubkey##* }"
+
+  ensure_base_image
+
+  # Select EVERY VLAN, not just the k8s one. ovs_up re-defines the libvirt
+  # network from SELECTED, so narrowing it here silently drops the portgroups
+  # for every other VLAN -- running VMs keep working (their taps are already
+  # attached) and nothing complains until the next VM cannot be attached.
+  # Observed: this deleted vlan1/3/9/10/200/51/53 and only surfaced when the
+  # ISP VMs needed vlan51 and vlan53.
+  selected_vlans
+  log "ensuring OVS fabric (all VLANs, so no portgroup is dropped)"
+  ovs_up
+
+  for n in $(seq 1 "$K8S_NODES"); do
+    create_node "$n" "$pubkey"
+  done
+
+  echo
+  log "nodes created. k3s installs on first boot (a few minutes)."
+  log "watch:   ssh debian@$(node_ip 1) 'sudo systemctl status k3s'"
+  log "then:    $0 --kubeconfig"
+  log "then install Cilium $CILIUM_VERSION and the BGP resources (see README)."
+}
+
+main "$@"

labsim/labsim-dhcp-test.sh

@@ -0,0 +1,219 @@
+#!/bin/bash
+# Prove that VyOS hands each device the address UniFi reserved for it.
+#
+# The question this answers is narrow and important: 30 of the 31 UniFi
+# reservations sit INSIDE the DHCP pool (LoT's pool is 10.0.0.11-10.0.1.254 and
+# only 10.0.0.2 falls outside it). UniFi's dhcpd tolerates that. VyOS uses kea,
+# and whether kea honours in-pool host reservations decides whether the cutover
+# silently renumbers 30 devices. That is not something to predict.
+#
+# Method: boot throwaway VMs whose MAC is a REAL production MAC, on the sim
+# VLAN, and check the address they are given. MACs are the one piece of
+# production config that transplants verbatim -- the subnet is rewritten, the
+# MAC is not -- which is what makes this a real test rather than a rehearsal.
+#
+# Safe: the ovs-labsim bridge contains only internal ports and VM taps, with no
+# physical NIC, so a production MAC here cannot reach or confuse the real LAN.
+# Verified with `ovs-vsctl show` before this script was written.
+#
+#   ./labsim-dhcp-test.sh              run the standard cases
+#   ./labsim-dhcp-test.sh --keep       leave the VMs up for inspection
+#   ./labsim-dhcp-test.sh --clean      just remove any leftover test VMs
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+source "$SCRIPT_DIR/lib.sh"
+
+ROUTER_IP="${ROUTER_IP:-172.31.1.1}"
+ROUTER_PW="${ROUTER_PW:-vyos}"
+TEST_VLAN="${TEST_VLAN:-10}"
+BOOT_WAIT="${BOOT_WAIT:-150}"
+TAG="labsim-dhcptest"
+
+# mac|expected|why. "POOL" means: must get an address from the pool and must
+# NOT get any reserved address -- the negative case that stops a pass from
+# meaning merely "DHCP works".
+CASES=(
+  "f8:0d:ac:90:65:c6|172.31.10.46|printer1 - reservation inside the pool"
+  "1c:69:20:7f:bc:77|172.31.11.67|sonoff-matter - in-pool AND across the /23 boundary"
+  "34:e1:d1:80:29:ce|172.31.10.2|Hubitat - the one reservation OUTSIDE the pool"
+  "52:54:00:ab:cd:ef|POOL|unreserved MAC - must get a pool address, not a reserved one"
+)
+
+vm_of() { echo "${TAG}-$(echo "$1" | tr -d ':')"; }
+
+cleanup_vms() {
+  local n=0
+  while read -r vm; do
+    [ -z "$vm" ] && continue
+    virsh_q destroy "$vm" >/dev/null 2>&1
+    virsh_q undefine "$vm" --remove-all-storage >/dev/null 2>&1
+    n=$((n + 1))
+  done < <(virsh_q list --all --name 2>/dev/null | grep "^${TAG}-" || true)
+  [ "$n" -gt 0 ] && log "removed $n test VM(s)"
+  sudo rm -f "$IMG_DIR/${TAG}-"*.qcow2 "$IMG_DIR/${TAG}-"*-seed.iso 2>/dev/null
+  return 0
+}
+
+# A seed that asks for DHCP instead of taking a static address. Alpine's
+# cloud-init ignores network-config here (verified previously and documented in
+# README), so /etc/network/interfaces is what actually takes effect.
+build_dhcp_seed() {
+  local iso="$1" vm="$2" pubkey="$3"
+  local tmp; tmp="$(mktemp -d)"
+  cat > "$tmp/meta-data" <<EOF
+instance-id: $vm
+local-hostname: $vm
+EOF
+  cat > "$tmp/user-data" <<EOF
+#cloud-config
+hostname: $vm
+users:
+  - name: alpine
+    shell: /bin/ash
+    lock_passwd: false
+    plain_text_passwd: labsim
+    ssh_authorized_keys:
+      - $pubkey
+ssh_authorized_keys:
+  - $pubkey
+disable_root: false
+chpasswd:
+  list: |
+    root:labsim
+  expire: false
+write_files:
+  - path: /etc/network/interfaces
+    content: |
+      auto lo
+      iface lo inet loopback
+      auto eth0
+      iface eth0 inet dhcp
+runcmd:
+  - [ sh, -c, "ifdown eth0 2>/dev/null; ifup eth0 || udhcpc -i eth0 -q || true" ]
+EOF
+  python3 - "$tmp/user-data" <<'PY' || die "generated user-data is not valid YAML"
+import sys, yaml
+yaml.safe_load(open(sys.argv[1]).read().split("#cloud-config",1)[1])
+PY
+  sudo genisoimage -quiet -output "$iso" -volid cidata -joliet -rock \
+    "$tmp/user-data" "$tmp/meta-data" >/dev/null 2>&1 || die "seed build failed"
+  rm -rf "$tmp"
+}
+
+router() {
+  timeout 30 sshpass -p "$ROUTER_PW" ssh -o StrictHostKeyChecking=no \
+    -o BatchMode=no -o ConnectTimeout=8 "vyos@$ROUTER_IP" "$@" 2>/dev/null
+}
+
+# --- argument handling ----------------------------------------------------
+KEEP=0
+case "${1:-}" in
+  --clean) cleanup_vms; exit 0 ;;
+  --keep)  KEEP=1 ;;
+  "")      ;;
+  *)       die "usage: $0 [--keep|--clean]" ;;
+esac
+
+command -v sshpass >/dev/null || die "sshpass required"
+require_tools
+[ -f "$BASE_IMAGE" ] || die "base image missing: $BASE_IMAGE (run labsim-up.sh first)"
+
+log "checking the router is serving DHCP..."
+subnets=$(router '/opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration commands | grep -c subnet-id')
+maps=$(router '/opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration commands | grep -c "static-mapping .* mac"')
+log "  router has ${subnets:-0} subnets and ${maps:-0} static-mappings"
+[ "${maps:-0}" -gt 0 ] || die "router has no static-mappings -- apply the generated config first"
+
+cleanup_vms
+
+# Flush the lease database first. This is not tidiness -- it is the condition
+# the cutover actually runs under, because kea does not inherit UniFi's leases
+# and starts empty. It also makes the test deterministic: with stale leases
+# present, kea saw the reserved address as held by "another client" (the same
+# MAC but a different client-id from a previous boot) and allocated a dynamic
+# address instead, which produced three misleading results before this existed.
+log "flushing the router's lease database (cutover starts with an empty one)"
+# Every dhcp4-leases.csv* must go, not just the main file: kea's memfile
+# backend keeps lease-file-cleanup rotations (.1/.2) and restores from them on
+# start, so truncating only the primary leaves the old leases intact.
+router 'sudo systemctl stop isc-kea-dhcp4-server;
+        sudo sh -c "rm -f /config/dhcp/dhcp4-leases.csv*";
+        sudo systemctl start isc-kea-dhcp4-server' >/dev/null
+sleep 5
+remaining="$(router '/opt/vyatta/bin/vyatta-op-cmd-wrapper show dhcp server leases' | sed -n '3,$p' | grep -c .)"
+[ "${remaining:-0}" -eq 0 ] || warn "lease table still has ${remaining} row(s) after flush"
+
+SSH_PUB="$(find_ssh_pubkey)"
+sudo mkdir -p "$IMG_DIR"
+
+# --- boot one VM per case -------------------------------------------------
+for c in "${CASES[@]}"; do
+  IFS='|' read -r mac expected why <<<"$c"
+  vm="$(vm_of "$mac")"
+  disk="$IMG_DIR/${vm}.qcow2"; seed="$IMG_DIR/${vm}-seed.iso"
+  log "booting $vm  mac=$mac  ($why)"
+  sudo qemu-img create -q -f qcow2 -F qcow2 -b "$BASE_IMAGE" "$disk" "$VM_DISK" >/dev/null
+  build_dhcp_seed "$seed" "$vm" "$SSH_PUB"
+  sudo virt-install --connect "$LIBVIRT_URI" --name "$vm" \
+    --memory "$VM_MEM" --vcpus "$VM_CPUS" \
+    --disk "path=$disk,format=qcow2,bus=virtio" \
+    --disk "path=$seed,device=cdrom,readonly=on" \
+    --network "network=labsim-ovs,portgroup=vlan${TEST_VLAN},model=virtio,mac=$mac" \
+    --os-variant alpinelinux3.18 --graphics none --noautoconsole --import >/dev/null \
+    || die "virt-install failed for $vm"
+done
+
+log "waiting ${BOOT_WAIT}s for boot + DHCP..."
+sleep "$BOOT_WAIT"
+
+# --- verdict --------------------------------------------------------------
+# The lease table on the router is the authority: it says what the server
+# decided, independent of whether the guest brought the interface up cleanly.
+leases="$(router '/opt/vyatta/bin/vyatta-op-cmd-wrapper show dhcp server leases')"
+echo
+echo "=== router lease table ==="
+echo "$leases"
+echo
+
+reserved_ips="$(cd "$SCRIPT_DIR/../migration" && python3 unifi-to-vyos.py --mode sim 2>/dev/null \
+  | awk '/static-mapping .* ip-address/ {print $NF}')"
+
+pass=0; fail=0
+printf '%-19s %-16s %-16s %s\n' "MAC" "EXPECTED" "GOT" "RESULT"
+for c in "${CASES[@]}"; do
+  IFS='|' read -r mac expected why <<<"$c"
+  # Never guess which lease is "the" lease. Taking the first match is how an
+  # hours-old lease was once reported as the current answer, turning three
+  # failures into apparent passes.
+  matches="$(echo "$leases" | awk -v m="$mac" 'tolower($2) == tolower(m) {print $1}')"
+  n_match="$(echo "$matches" | grep -c . )"
+  if [ "$n_match" -gt 1 ]; then
+    got="AMBIGUOUS($(echo "$matches" | tr '\n' ',' | sed 's/,$//'))"
+  else
+    got="${matches:-<none>}"
+  fi
+  if [ "${got#AMBIGUOUS}" != "$got" ]; then
+    # More than one lease for this MAC means the flush did not take. Any
+    # verdict from here is a guess, so refuse to give one.
+    result="FAIL (multiple leases -- flush did not take)"
+  elif [ "$expected" = "POOL" ]; then
+    if [ "$got" = "<none>" ]; then
+      result="FAIL (no lease at all)"
+    elif echo "$reserved_ips" | grep -qx "$got"; then
+      result="FAIL (got a RESERVED address)"
+    else
+      result="pass"
+    fi
+  else
+    [ "$got" = "$expected" ] && result="pass" || result="FAIL"
+  fi
+  [ "$result" = "pass" ] && pass=$((pass + 1)) || fail=$((fail + 1))
+  printf '%-19s %-16s %-16s %s\n' "$mac" "$expected" "$got" "$result"
+  printf '  %s\n' "$why"
+done
+
+echo
+log "$pass passed, $fail failed"
+[ "$KEEP" -eq 1 ] && log "VMs left running (--keep). Remove with: $0 --clean" || cleanup_vms
+[ "$fail" -eq 0 ] || exit 1

labsim/labsim-matrix.py

@@ -78,9 +78,15 @@ def load_vlans() -> list[dict]:
             line = line.strip()
             if not line or line.startswith("#"):
                 continue
-            vid, name, prefix, real = line.split(":", 3)
+            # masklen and host_octet are optional trailing fields; VLAN 10 sets
+            # both because it must be a /23 (see vlans.conf).
+            parts = line.split(":")
+            vid, name, prefix, real = parts[0], parts[1], parts[2], parts[3]
+            masklen = int(parts[4]) if len(parts) > 4 and parts[4] else 24
+            host = parts[5] if len(parts) > 5 and parts[5] else "2"
             vlans.append({"vid": vid, "name": name, "ip": f"{prefix}.10",
-                          "label": f"{vid}:{name}", "real": real})
+                          "label": f"{vid}:{name}", "real": real,
+                          "masklen": masklen, "host_ip": f"{prefix}.{host}"})
     return vlans
 
 

labsim/labsim-up.sh

@@ -25,7 +25,8 @@ ovs_up
 
 # --- VMs ------------------------------------------------------------------
 for entry in "${SELECTED[@]}"; do
-  IFS=: read -r vid name prefix real <<<"$entry"
+  parse_vlan_entry "$entry"
+  vid="$V_VID"; name="$V_NAME"; prefix="$V_PREFIX"; real="$V_REAL"
   vm="$(vm_name "$vid" "$name")"
   ip="${prefix}.10"
 
@@ -48,7 +49,7 @@ for entry in "${SELECTED[@]}"; do
   # Copy-on-write overlay: each VM costs a few MB, not 176.
   sudo qemu-img create -q -f qcow2 -F qcow2 -b "$BASE_IMAGE" "$disk" "$VM_DISK" >/dev/null
 
-  build_seed "$seed" "$vm" "$vid" "$name" "$prefix" "$ip" "$real" "$SSH_PUB"
+  build_seed "$seed" "$vm" "$vid" "$name" "$prefix" "$ip" "$real" "$SSH_PUB" "$V_MASK"
 
   sudo virt-install \
     --connect "$LIBVIRT_URI" \

labsim/labsim-vlan-leak-test.sh

@@ -0,0 +1,178 @@
+#!/bin/bash
+# Does the router offer an address from the WRONG VLAN's pool?
+#
+# The fault (ISC Kea #1117, "Mix of physical and virtual interfaces (VLAN) does
+# not work"): with `dhcp-socket-type: raw`, a frame tagged for a sub-interface is
+# ALSO delivered to the PARENT's AF_PACKET socket. Kea then selects a subnet from
+# the parent's own address and answers a second time from the wrong pool. Both
+# offers race to the client and the CLIENT decides which one wins -- which is why
+# the symptom looks device-dependent and unreproducible.
+#
+# Production and this sim have the identical shape that triggers it: Management
+# is the NATIVE/untagged VLAN on `bond0` and therefore has a subnet on the
+# parent, while every other VLAN is a `bond0.<vif>` sub-interface of that same
+# bond.
+#
+# Method: make one DHCP client on a TAGGED VLAN send a DISCOVER, and capture
+# simultaneously on the parent and on the sub-interface. The verdict is not
+# "did the client get the right address" -- the client picking correctly is
+# exactly how this hid for weeks. The verdict is how many OFFERs the SERVER
+# emitted and which source addresses they carried.
+#
+#   ./labsim-vlan-leak-test.sh                 test VLAN 3
+#   ./labsim-vlan-leak-test.sh --vlan 9        test another VLAN
+#   ./labsim-vlan-leak-test.sh --save before   also write the raw captures to
+#                                              vlan-leak-evidence/before/
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+
+ROUTER_IP="${ROUTER_IP:-172.31.1.1}"
+ROUTER_PW="${ROUTER_PW:-vyos}"
+CLIENT_PW="${CLIENT_PW:-labsim}"
+VLAN=3
+CLIENT=""
+SAVE=""
+
+while [ $# -gt 0 ]; do
+  case "$1" in
+    --vlan)   VLAN="$2"; shift 2 ;;
+    --client) CLIENT="$2"; shift 2 ;;
+    --save)   SAVE="$2"; shift 2 ;;
+    *) echo "usage: $0 [--vlan N] [--client IP] [--save LABEL]" >&2; exit 2 ;;
+  esac
+done
+: "${CLIENT:=172.31.${VLAN}.10}"
+
+log() { printf '\033[36m==>\033[0m %s\n' "$*"; }
+die() { printf '\033[31merror:\033[0m %s\n' "$*" >&2; exit 1; }
+
+command -v sshpass >/dev/null || die "sshpass required"
+
+# A silent router is the one verdict worth double-checking before reporting.
+#
+# Kea can be `is-active` and answering nothing -- it reopens sockets on a retry
+# loop, and some configurations (`listen-interface`, notably) leave individual
+# VLANs dead while the rest work. Both look identical to a one-shot test: "no
+# reply at all". Two opposite and equally wrong conclusions about
+# `listen-interface` came out of believing a single negative run, in both
+# directions, before a retry made the real pattern obvious.
+#
+# Kea's fallback UDP socket appearing is NOT a readiness signal -- it is bound
+# well before the server actually answers. Checked, and it does not work.
+RETRIED="${RETRIED:-0}"
+
+router() {
+  timeout 40 sshpass -p "$ROUTER_PW" ssh -o StrictHostKeyChecking=no \
+    -o ConnectTimeout=8 "vyos@$ROUTER_IP" "$@" 2>/dev/null
+}
+# VyOS's login shell is vbash, which returns 255 on anything it does not like --
+# in particular a backgrounded job. Feeding the script to `bash -s` on stdin
+# sidesteps vbash entirely and is the only reliable way to leave a daemon behind.
+router_sh() {
+  timeout 40 sshpass -p "$ROUTER_PW" ssh -o StrictHostKeyChecking=no \
+    -o ConnectTimeout=8 "vyos@$ROUTER_IP" 'bash -s' 2>/dev/null
+}
+client() {
+  timeout 60 sshpass -p "$CLIENT_PW" ssh -o StrictHostKeyChecking=no \
+    -o ConnectTimeout=8 "root@$CLIENT" "$@" 2>/dev/null
+}
+
+# Which interfaces to watch. The parent is the whole point: after the fix it
+# should carry no DHCP traffic of its own at all.
+PARENT="bond0"
+VIF="bond0.${VLAN}"
+
+log "router $ROUTER_IP -- capturing on $PARENT and $VIF"
+# Kill EVERY tcpdump first, not just ones matching this run's pattern, and count
+# only afterwards. Counting `pgrep -f 'tcpdump -i bond0'` while a stray tcpdump
+# from an earlier session was still running satisfied the >=2 guard with zero of
+# THIS run's captures alive -- and a capture that records nothing reports
+# "the router sent no reply at all", which reads as a DHCP outage. That sent me
+# chasing a fault in the router that was entirely in the test harness.
+started="$(router_sh <<EOF
+sudo pkill -x tcpdump >/dev/null 2>&1
+sleep 1
+sudo rm -f /tmp/leak-*.txt
+sudo nohup tcpdump -i $PARENT -e -nn -l 'udp port 67 or udp port 68' > /tmp/leak-parent.txt 2>/dev/null &
+sudo nohup tcpdump -i $VIF -e -nn -l 'udp port 67 or udp port 68' > /tmp/leak-vif.txt 2>/dev/null &
+sleep 3
+pgrep -c -x tcpdump
+EOF
+)"
+[ "${started:-0}" -eq 2 ] || die "capture did not start on the router (got ${started:-0}, expected exactly 2)"
+
+# -s /bin/true: ask, observe the answer, apply nothing. The client's existing
+# static address is left alone, so this is safe to run against a live sim VM.
+log "client $CLIENT -- sending DISCOVER on VLAN $VLAN"
+client_out="$(client "udhcpc -n -q -f -i eth0 -s /bin/true -t 3 -T 3 2>&1")"
+[ -n "$client_out" ] || die "no response from client $CLIENT"
+
+sleep 2
+router "sudo pkill -x tcpdump" >/dev/null
+parent="$(router 'sudo cat /tmp/leak-parent.txt')"
+vif="$(router 'sudo cat /tmp/leak-vif.txt')"
+
+echo
+echo "--- client ---"
+echo "$client_out" | sed 's/^/  /'
+echo
+echo "--- $PARENT (parent) ---"
+echo "${parent:-  (nothing)}" | sed 's/^/  /'
+echo
+echo "--- $VIF (sub-interface) ---"
+echo "${vif:-  (nothing)}" | sed 's/^/  /'
+echo
+
+if [ -n "$SAVE" ]; then
+  d="$SCRIPT_DIR/vlan-leak-evidence/$SAVE"
+  mkdir -p "$d"
+  printf '%s\n' "$client_out" > "$d/client.txt"
+  printf '%s\n' "$parent"     > "$d/capture-parent.txt"
+  printf '%s\n' "$vif"        > "$d/capture-vif.txt"
+  router '/opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration commands' \
+    | grep -E 'interfaces bonding|vrrp group' > "$d/router-config.txt"
+  log "evidence saved to vlan-leak-evidence/$SAVE/"
+fi
+
+# --- verdict ---------------------------------------------------------------
+# Every BOOTP Reply seen anywhere, reduced to its source address. A reply whose
+# source is not this VLAN's router leg is an offer from the wrong subnet.
+replies="$(printf '%s\n%s\n' "$parent" "$vif" \
+  | grep -o '[0-9.]*\.67 > [0-9.]*\.68' | awk '{print $1}' | sed 's/\.67$//' \
+  | sort -u)"
+want_prefix="172.31.${VLAN}."
+
+echo "=== verdict ==="
+if [ -z "$replies" ]; then
+  if [ "$RETRIED" -eq 0 ]; then
+    log "no reply -- retrying once in 20s before calling DHCP down"
+    sleep 20; RETRIED=1 exec "$0" --vlan "$VLAN" --client "$CLIENT" ${SAVE:+--save "$SAVE"}
+  fi
+  echo "INCONCLUSIVE: the router sent no reply at all, twice -- DHCP is down on VLAN $VLAN"
+  exit 2
+fi
+
+bad=0
+while read -r src; do
+  [ -z "$src" ] && continue
+  case "$src" in
+    "$want_prefix"*) printf '  ok    offer from %s (this VLAN)\n' "$src" ;;
+    *)               printf '  LEAK  offer from %s (WRONG subnet)\n' "$src"; bad=1 ;;
+  esac
+done <<<"$replies"
+
+# The parent carrying any DHCP of its own is the mechanism, not just a symptom:
+# it means the parent still has a subnet kea can match a tagged frame against.
+if printf '%s' "$parent" | grep -q 'ethertype IPv4' \
+   && printf '%s' "$parent" | grep -v 'vlan ' | grep -q '\.67 > '; then
+  echo "  note  $PARENT emitted an UNTAGGED reply -- the parent still serves a subnet"
+fi
+
+echo
+if [ "$bad" -eq 0 ]; then
+  echo "PASS: only this VLAN's pool answered."
+  exit 0
+fi
+echo "FAIL: the router answered from another VLAN's pool (kea #1117)."
+exit 1

labsim/lib.sh

@@ -58,9 +58,32 @@ selected_vlans() {
   [ ${#SELECTED[@]} -gt 0 ] || die "no VLANs selected (checked $CONF)"
 }
 
+# Split one vlans.conf line, applying defaults for the two optional trailing
+# fields. Sets V_VID V_NAME V_PREFIX V_REAL V_MASK V_HOST.
+parse_vlan_entry() {
+  IFS=: read -r V_VID V_NAME V_PREFIX V_REAL V_MASK V_HOST <<<"$1"
+  V_MASK="${V_MASK:-24}"
+  V_HOST="${V_HOST:-2}"
+}
+
+# Dotted netmask for a prefix length — cloud-init's network-config v1 wants the
+# dotted form, not a /len. /24 -> 255.255.255.0, /23 -> 255.255.254.0.
+netmask_for() {
+  local len="$1" i bits out=()
+  for i in 0 1 2 3; do
+    bits=$(( len - i * 8 ))
+    (( bits > 8 )) && bits=8
+    (( bits < 0 )) && bits=0
+    out+=( $(( 256 - 2 ** (8 - bits) )) )
+  done
+  local IFS=.; echo "${out[*]}"
+}
+
 # cloud-init NoCloud seed: static addressing + SSH key + hello-world HTTP.
 build_seed() {
   local iso="$1" vm="$2" vid="$3" name="$4" prefix="$5" ip="$6" real="$7" pubkey="$8"
+  local masklen="${9:-24}"
+  local netmask; netmask="$(netmask_for "$masklen")"
   local tmp; tmp="$(mktemp -d)"
 
   cat > "$tmp/meta-data" <<EOF
@@ -85,7 +108,7 @@ config:
     subnets:
       - type: static
         address: $ip
-        netmask: 255.255.255.0
+        netmask: $netmask
         # Default route via the router under test. Without this the VMs can
         # reach their own /24 and their gateway, but nothing beyond it — which
         # looks exactly like "the router is broken" in the matrix.
@@ -122,14 +145,14 @@ write_files:
       auto eth0
       iface eth0 inet static
         address $ip
-        netmask 255.255.255.0
+        netmask $netmask
         post-up ip route add default via ${prefix}.1 || true
   - path: /var/www/index.html
     content: |
       <html><body>
       <h1>labsim vlan $vid — $name</h1>
       <p>host: $vm</p>
-      <p>address: $ip/24</p>
+      <p>address: $ip/$masklen</p>
       <p>gateway under test: ${prefix}.1</p>
       <p>mirrors production: $real</p>
       </body></html>

labsim/ovs.sh

@@ -17,6 +17,21 @@ OVS_BR="${OVS_BR:-ovs-labsim}"
 OVS_NET="${OVS_NET:-labsim-ovs}"          # libvirt network wrapping the bridge
 LAG_NAME="${LAG_NAME:-lag-vyos}"
 
+# Native (untagged) VLAN on the trunks to the routers. Empty means NONE: every
+# VLAN, Management included, is tagged.
+#
+# This is not a style choice. A native VLAN is what puts a subnet on the bond
+# PARENT (`bond0`) while every other VLAN lives on a sub-interface of it. With
+# `dhcp-socket-type: raw`, kea then receives each tagged frame TWICE -- once on
+# the sub-interface and once on the parent -- and answers from the parent's pool
+# as well, so a client on VLAN 3 is offered a Management address and picks
+# whichever reply arrives first (ISC Kea #1117).
+#
+# Set LABSIM_NATIVE_VLAN=1 to restore the old shape and reproduce the bug:
+#   LABSIM_NATIVE_VLAN=1 ./router-up.sh && ./labsim-vlan-leak-test.sh   # FAIL
+#                        ./router-up.sh && ./labsim-vlan-leak-test.sh   # PASS
+NATIVE_VLAN="${LABSIM_NATIVE_VLAN:-}"
+
 ovs() { sudo ovs-vsctl "$@"; }
 
 ovs_require() {
@@ -25,6 +40,10 @@ ovs_require() {
     || die "could not start openvswitch"
 }
 
+# A comma-separated VLAN list, numerically sorted, for comparing two lists that
+# came from different places and need not agree on order.
+vlan_sorted() { echo "$1" | tr ',' '\n' | grep -v '^$' | sort -n | paste -sd, -; }
+
 # All VLAN ids from the config, comma separated — used for trunk ports.
 vlan_id_list() {
   local ids=()
@@ -41,12 +60,20 @@ ovs_up() {
   # default route (.1 is), so inter-VLAN tests exercise the router, not the
   # host's routing table.
   for entry in "${SELECTED[@]}"; do
-    IFS=: read -r vid _name prefix _real <<<"$entry"
+    parse_vlan_entry "$entry"
-    local port="hostv${vid}"
+    local port="hostv${V_VID}"
-    ovs --may-exist add-port "$OVS_BR" "$port" tag="$vid" \
+    ovs --may-exist add-port "$OVS_BR" "$port" tag="$V_VID" \
       -- set interface "$port" type=internal
     sudo ip link set "$port" up 2>/dev/null || true
-    sudo ip addr replace "${prefix}.2/24" dev "$port"
+    # Drop any address from a previous mask/octet so a changed vlans.conf does
+    # not leave a stale second address on the port.
+    sudo ip -4 addr flush dev "$port" 2>/dev/null || true
+    # host_octet 0 means "no host leg": the WAN transport VLANs belong to the
+    # fake ISPs, and giving the host an address there would misrepresent the
+    # segment -- the whole point is that VyOS reaches an ISP, not the host.
+    if [ "$V_HOST" != "0" ]; then
+      sudo ip addr replace "${V_PREFIX}.${V_HOST}/${V_MASK}" dev "$port"
+    fi
   done
 
   ovs_define_libvirt_net
@@ -64,18 +91,23 @@ ovs_define_libvirt_net() {
 "
   done
 
-  # Trunk: VLAN 1 native/untagged, everything else tagged — the production
+  # Trunk: every VLAN tagged, and by default NO native VLAN (see NATIVE_VLAN at
-  # shape. libvirt expresses this declaratively via nativeMode='untagged'
+  # the top of this file for why -- it is the kea #1117 fix, not tidiness).
-  # (see libvirt formatnetwork.html), so it does not need fixing up by hand.
+  # libvirt expresses a native VLAN declaratively via nativeMode='untagged'
-  # It also matters functionally: LACPDUs are untagged, and a trunk with no
+  # (see libvirt formatnetwork.html), so it needs no fixing up by hand.
-  # native VLAN has nowhere to put them.
+  #
+  # The worry that a trunk with no native VLAN has nowhere to put LACPDUs is
+  # unfounded, and was tested rather than reasoned about: with vlan_mode=trunk
+  # and no tag, `ovs-appctl bond/show` still reports lacp_status: negotiated
+  # with both members enabled. LACPDUs are slow-protocol frames handled per
+  # member, below the VLAN layer.
   local trunk="  <portgroup name='trunk'>
     <vlan trunk='yes'>
 "
   for entry in "${SELECTED[@]}"; do
     IFS=: read -r vid _n _p _r <<<"$entry"
-    if [ "$vid" = "1" ]; then
+    if [ -n "$NATIVE_VLAN" ] && [ "$vid" = "$NATIVE_VLAN" ]; then
-      trunk+="      <tag id='1' nativeMode='untagged'/>
+      trunk+="      <tag id='${vid}' nativeMode='untagged'/>
 "
     else
       trunk+="      <tag id='${vid}'/>
@@ -108,19 +140,59 @@ ${pg}${trunk}</network>"
 ovs_bond_router() {
   local vm="$1"
   local taps
-  # NB: domiflist indents its rows, so anchor on the FIELD not the line —
+  # Two filters, both load-bearing:
-  # /^vnet/ silently matches nothing and the bond never gets built.
+  #
-  taps="$(virsh_q domiflist "$vm" 2>/dev/null | awk '$1 ~ /^vnet/ {print $1}')"
+  #   $1 ~ /^vnet/ -- domiflist indents its rows, so anchor on the FIELD, not
+  #   the line. /^vnet/ silently matches nothing and the bond never gets built.
+  #
+  #   $3 == OVS_NET -- count only the taps on the sim fabric. The primary also
+  #   carries a libvirt-NAT scaffold NIC (see --drop-scaffold in the README), so
+  #   an unfiltered count is 3, and this function's "expected 2" guard then
+  #   skipped the primary's bond entirely while reporting only a warning.
+  taps="$(virsh_q domiflist "$vm" 2>/dev/null \
+          | awk -v net="$OVS_NET" '$1 ~ /^vnet/ && $3 == net {print $1}')"
   local count; count="$(echo "$taps" | grep -c .)"
-  [ "$count" -eq 2 ] || { warn "router $vm has $count tap(s), expected 2 — skipping bond"; return 1; }
+  [ "$count" -eq 2 ] \
-
+    || { warn "router $vm has $count tap(s) on $OVS_NET, expected 2 — skipping bond"; return 1; }
-  # Already bonded? (idempotent re-runs)
-  if ovs list-ports "$OVS_BR" 2>/dev/null | grep -qx "$LAG_NAME"; then
-    log "LACP bond $LAG_NAME already present"
-    return 0
-  fi
 
   local t1 t2; t1="$(echo "$taps" | sed -n 1p)"; t2="$(echo "$taps" | sed -n 2p)"
+  local want; want="$(vlan_id_list)"
+
+  # Already bonded? Re-runs must still reconcile BOTH the VLAN list and the
+  # membership, and each has drawn blood:
+  #
+  #   VLANs -- adding a VLAN to vlans.conf and finding the bond unchanged is how
+  #   a VLAN silently fails to reach a router: interface present, tag missing,
+  #   frames dropped by the switch.
+  #
+  #   MEMBERS -- restarting the VM recreates its taps with NEW names, leaving the
+  #   bond holding two interfaces that no longer exist. `list-ports` still shows
+  #   the bond, so this early return declared success while the router's real
+  #   taps sat in the bridge as two INDEPENDENT ports, each carrying libvirt's
+  #   own portgroup VLAN config. That is how labsim-vyos2 ran for weeks with no
+  #   LACP at all and a native VLAN nobody had asked for -- and it is invisible
+  #   until you change the trunk and only one router follows.
+  if ovs list-ports "$OVS_BR" 2>/dev/null | grep -qx "$LAG_NAME"; then
+    local members; members="$(ovs-appctl-members)"
+    if [ "$members" != "$(printf '%s\n%s' "$t1" "$t2" | sort | paste -sd, -)" ]; then
+      warn "bond $LAG_NAME holds stale members [$members], VM has [$t1,$t2] — rebuilding"
+      ovs --if-exists del-port "$OVS_BR" "$LAG_NAME"
+    else
+      # Compare as SETS. vlan_id_list yields config order (1,2,3,9,10,200,51,53)
+      # while OVS returns its own sorted order, so a raw string compare reports
+      # drift on every run and rewrites a trunk that was already correct.
+      local have; have="$(ovs get port "$LAG_NAME" trunks 2>/dev/null | tr -d '[] ')"
+      if [ "$(vlan_sorted "$want")" != "$(vlan_sorted "$have")" ]; then
+        log "bond $LAG_NAME trunk drift: [$have] -> [$want]; updating"
+        ovs set port "$LAG_NAME" trunks="$want"
+      else
+        log "LACP bond $LAG_NAME already present, trunk correct"
+      fi
+      ovs_set_native "$LAG_NAME"
+      return 0
+    fi
+  fi
+
   log "bonding $t1 + $t2 into $LAG_NAME (LACP active, balance-tcp)"
   ovs del-port "$OVS_BR" "$t1" 2>/dev/null || true
   ovs del-port "$OVS_BR" "$t2" 2>/dev/null || true
@@ -134,15 +206,36 @@ ovs_bond_router() {
   # LACPDUs. Falling back to active-backup brings the links up so negotiation
   # can start.
   #
-  # native-untagged + tag=1 carries the untagged LACPDUs and the management
+  # The VLAN mode is set inline: libvirt's portgroup config does NOT apply here,
-  # VLAN, matching production. libvirt's portgroup VLAN config does NOT apply
+  # because the bond is a port libvirt never created.
-  # here — the bond is a port libvirt never created — so set it inline.
-  local tagged; tagged="$(vlan_id_list | tr ',' '\n' | grep -vx 1 | paste -sd, -)"
   ovs add-bond "$OVS_BR" "$LAG_NAME" "$t1" "$t2" \
-      lacp=active bond_mode=balance-tcp \
+      lacp=active bond_mode=balance-tcp trunks="$want" \
-      vlan_mode=native-untagged tag=1 trunks="$tagged" \
     -- set port "$LAG_NAME" other_config:lacp-time=fast \
     -- set port "$LAG_NAME" other_config:lacp-fallback-ab=true
+  ovs_set_native "$LAG_NAME"
+}
+
+# The bond's current members, sorted and comma-joined, or empty if the bond does
+# not resolve at all (which is itself the stale case worth rebuilding for).
+ovs-appctl-members() {
+  sudo ovs-appctl bond/show "$LAG_NAME" 2>/dev/null \
+    | awk '/^member /{gsub(/:/,"",$2); print $2}' | sort | paste -sd, -
+}
+
+# Apply NATIVE_VLAN to a trunk port.
+#
+# `tag` MUST be removed, not merely left alone, when there is no native VLAN.
+# Setting vlan_mode=trunk while a stale `tag` remains looks correct in
+# `ovs-vsctl list port` -- it prints vlan_mode: trunk right next to tag: 1 --
+# but the port keeps egressing that VLAN untagged. Half an hour went into
+# "the router is ignoring the trunk change" before the tag was the answer.
+ovs_set_native() {
+  local port="$1"
+  if [ -n "$NATIVE_VLAN" ]; then
+    ovs set port "$port" vlan_mode=native-untagged tag="$NATIVE_VLAN"
+  else
+    ovs set port "$port" vlan_mode=trunk -- clear port "$port" tag
+  fi
 }
 
 ovs_bond_status() {

labsim/sim-ha-config.py

@@ -0,0 +1,115 @@
+#!/usr/bin/env python3
+"""Generate the HA config for the labsim VyOS pair.
+
+Exists to answer one question that cannot be answered on a single router, and
+that would otherwise only be discovered at cutover: with kea HA active-passive,
+does exactly ONE box answer a DHCP request?
+
+Mirrors the production shape so the answer transfers:
+
+    router1  172.31.<v>.252   priority 200   DHCP HA primary
+    router2  172.31.<v>.253   priority 100   DHCP HA secondary
+    VIP      172.31.<v>.1     (what clients use as their gateway)
+
+Note the sim's LoT VLAN is a /23 like production, so the VIP prefix differs
+there -- getting that wrong produces a config that commits and then behaves
+subtly wrongly, which is worse than a failure.
+
+  ./sim-ha-config.py --role primary   > r1.conf
+  ./sim-ha-config.py --role secondary > r2.conf
+"""
+from __future__ import annotations
+
+import argparse
+import importlib.util
+import json
+import os
+import sys
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+MIG = os.path.join(HERE, "..", "migration")
+
+# Reuse the DHCP/DNS generator rather than hand-writing subnets: the whole
+# point is that what is proven here and what production gets share a code path.
+_spec = importlib.util.spec_from_file_location(
+    "unifi_to_vyos", os.path.join(MIG, "unifi-to-vyos.py"))
+unifi_to_vyos = importlib.util.module_from_spec(_spec)
+_spec.loader.exec_module(unifi_to_vyos)
+
+# vlan -> (prefix, cidr). LoT is a /23 in the sim, matching production.
+VLANS = {
+    1:   ("172.31.1",   24),
+    2:   ("172.31.2",   24),
+    3:   ("172.31.3",   24),
+    9:   ("172.31.9",   24),
+    10:  ("172.31.10",  23),
+    200: ("172.31.200", 24),
+}
+DHCP_HA_NAME = "labsim-dhcp-pair"     # must not equal either host-name
+
+
+def group(vlan: int) -> str:
+    return "native" if vlan == 1 else f"vlan{vlan}"
+
+
+def build(role: str) -> list[str]:
+    primary = role == "primary"
+    self_o, peer_o = (252, 253) if primary else (253, 252)
+    prio = 200 if primary else 100
+    out = [f"# labsim VyOS HA -- {role}", ""]
+
+    for vlan, (pfx, cidr) in VLANS.items():
+        g = group(vlan)
+        # EVERY VLAN is a sub-interface, Management (VLAN 1) included. Putting
+        # Management on the bare `bond0` is what gives the parent a subnet, and
+        # kea then answers tagged frames from it as well as from the correct
+        # sub-interface -- clients on other VLANs get offered a Management
+        # address (ISC Kea #1117). See NATIVE_VLAN in ovs.sh; proven by
+        # labsim-vlan-leak-test.sh.
+        iface = f"bond0 vif {vlan}"
+        out += [
+            f"# VLAN {vlan}",
+            # The node's own address replaces the .1 it used to hold directly;
+            # .1 becomes the floating VIP, exactly as production will be.
+            f"delete interfaces bonding {iface} address",
+            f"set interfaces bonding {iface} address '{pfx}.{self_o}/{cidr}'",
+            f"set high-availability vrrp group {g} interface bond0.{vlan}",
+            f"set high-availability vrrp group {g} vrid {vlan}",
+            f"set high-availability vrrp group {g} address {pfx}.1/{cidr}",
+            f"set high-availability vrrp group {g} priority {prio}",
+            f"set high-availability vrrp group {g} hello-source-address {pfx}.{self_o}",
+            f"set high-availability vrrp group {g} peer-address {pfx}.{peer_o}",
+            f"set high-availability vrrp group {g} no-preempt",
+            f"set high-availability vrrp sync-group MAIN member {g}",
+            "",
+        ]
+
+    out += [
+        "# --- DHCP high-availability ---",
+        "# The thing under test: active-passive should mean exactly one OFFER.",
+        "set service dhcp-server high-availability mode active-passive",
+        f"set service dhcp-server high-availability status {role}",
+        f"set service dhcp-server high-availability name {DHCP_HA_NAME}",
+        f"set service dhcp-server high-availability source-address 172.31.10.{self_o}",
+        f"set service dhcp-server high-availability remote 172.31.10.{peer_o}",
+        "",
+    ]
+
+    inv = json.load(open(os.path.join(MIG, "export", "inventory.json")))
+    dhcp, stats = unifi_to_vyos.build(inv, "sim")
+    out += [l for l in dhcp if l.strip() and not l.startswith("#")]
+    print(f"{role}: {stats['subnets']} subnets, {stats['mappings']} mappings",
+          file=sys.stderr)
+    return out
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--role", choices=("primary", "secondary"), required=True)
+    args = ap.parse_args()
+    sys.stdout.write("\n".join(build(args.role)) + "\n")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

labsim/sim-net-apply.sh

@@ -0,0 +1,59 @@
+#!/usr/bin/env bash
+# Apply -- or drift-check -- the labsim routing config on all four VMs.
+#
+#   ./sim-net-apply.sh check    what the VMs run vs what sim-net-config.py says
+#   ./sim-net-apply.sh apply    push the generated config over the serial console
+#
+# `check` is the one you want most of the time. The whole failure mode this
+# guards against is somebody (including me) fixing something on a VM over SSH
+# and never writing it down, so the next rebuild silently loses it.
+#
+# Applied over the serial console rather than SSH because a freshly installed
+# sim router holds the same addresses as its peer -- there is a window where it
+# is not safely reachable over the network at all. See console-apply.py.
+set -uo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ACTION="${1:-check}"
+WORK="$(mktemp -d)"; trap 'rm -rf "$WORK"' EXIT
+
+# role : vm : address : regex selecting the subtrees this generator owns
+TARGETS=(
+  "primary:labsim-vyos:172.31.1.252:^set (protocols (bgp|failover|static)|policy (prefix-list|route-map)|nat source rule 1[12]0|interfaces (pppoe|bonding bond0 vif 5[13])|firewall (group interface-group LAN|ipv4|ipv6))"
+  "secondary:labsim-vyos2:172.31.1.253:^set (protocols bgp|policy (prefix-list|route-map)|firewall (group interface-group LAN|ipv4|ipv6))"
+  "isp-dhcp:labsim-isp-dhcp:192.168.122.136:^set (interfaces ethernet|nat source|service dhcp-server|firewall ipv4 forward|system host-name)"
+  "isp-pppoe:labsim-isp-pppoe:192.168.122.63:^set (interfaces ethernet|nat source|service pppoe-server|firewall ipv4 forward|system host-name)"
+)
+# Sim-only credential; these VMs hold nothing real and are not reachable from
+# outside the hypervisor.
+SSH_OPTS=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
+          -o LogLevel=ERROR -o PreferredAuthentications=password -o ConnectTimeout=5)
+live() { timeout 30 sshpass -p vyos ssh "${SSH_OPTS[@]}" "vyos@$1" \
+           "/opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration commands" 2>/dev/null; }
+norm() { sed "s/'//g" | grep -v 'hw-id\|offload' | sort -u; }
+
+rc=0
+for t in "${TARGETS[@]}"; do
+  IFS=: read -r role vm addr rx <<<"$t"
+  "$HERE/sim-net-config.py" --role "$role" >"$WORK/$role.conf" 2>/dev/null || {
+    printf '  %-11s GENERATE FAILED\n' "$role"; rc=1; continue; }
+
+  if [ "$ACTION" = apply ]; then
+    printf '  %-11s applying to %s over console...\n' "$role" "$vm"
+    "$HERE/console-apply.py" --vm "$vm" --config "$WORK/$role.conf" || rc=1
+    continue
+  fi
+
+  if ! live "$addr" >"$WORK/$role.live" || [ ! -s "$WORK/$role.live" ]; then
+    printf '  %-11s UNREACHABLE (%s)\n' "$role" "$addr"; rc=1; continue
+  fi
+  grep -E '^set ' "$WORK/$role.conf" | norm >"$WORK/$role.g"
+  grep -E "$rx" "$WORK/$role.live" | norm >"$WORK/$role.l"
+  if d="$(diff "$WORK/$role.g" "$WORK/$role.l")" && [ -z "$d" ]; then
+    printf '  %-11s in sync (%s commands)\n' "$role" "$(wc -l <"$WORK/$role.g")"
+  else
+    printf '  %-11s DRIFT — "<" only in code, ">" only on the VM:\n' "$role"
+    printf '%s\n' "$d" | sed 's/^/      /'
+    rc=1
+  fi
+done
+exit $rc

labsim/sim-net-config.py

@@ -0,0 +1,417 @@
+#!/usr/bin/env python3
+"""Generate the labsim routing + WAN config: BGP, dual WAN, and the two ISP VMs.
+
+`sim-ha-config.py` covers the LAN side of the sim routers (addresses, VRRP,
+conntrack-sync, DHCP). This covers everything that makes the sim a rehearsal for
+production routing rather than just a LAN:
+
+  * eBGP between the sim routers and the k3s nodes, carrying the service range
+  * dual WAN -- DHCP on VLAN 53, PPPoE on VLAN 51 -- with health-checked failover
+  * the two ISP VMs that terminate those WANs and NAT to the real internet
+
+All of it previously existed only as running state on the VMs, applied by hand
+over SSH. Rebuilding a VM lost the rehearsal, and nothing recorded *why* any of
+it was shaped the way it is. That is the entire reason this file exists.
+
+  ./sim-net-config.py --role primary   > r1-net.conf
+  ./console-apply.py --vm labsim-vyos  --config r1-net.conf
+
+or apply all four at once with ./sim-net-apply.sh.
+"""
+from __future__ import annotations
+
+import argparse
+import sys
+
+# ---------------------------------------------------------------------------
+# BGP. Numbers match production so what is proven here ports over unchanged.
+# ---------------------------------------------------------------------------
+ROUTER_AS = 65000
+CLUSTER_AS = 65001
+# The service range Cilium advertises. Chosen against a survey of third-party
+# RFC1918 defaults (docker-desktop, tailscale, k3s, EKS...) so it cannot collide
+# with something we adopt later. Production uses the same /22 -- keep them equal.
+SERVICE_CIDR = "10.61.0.0/22"
+K8S_VLAN = 2
+K8S_NODES = ["172.31.2.11", "172.31.2.12", "172.31.2.13"]
+PEER_GROUP = "K8S"
+PFX_LIST = "K8S-SERVICE-IPS"
+RM_IN, RM_OUT = "K8S-IN", "K8S-OUT"
+# One route per node; ECMP across all three. 4 leaves headroom for a fourth node
+# without a config change.
+MAX_PATHS = 4
+# A safety valve, not a capacity plan: a misconfigured Cilium that starts
+# advertising pod CIDRs should tear the session down, not quietly fill the FIB.
+MAX_PREFIX = 100
+
+# ---------------------------------------------------------------------------
+# Dual WAN. The sim ISPs deliberately use TEST-NET-3 (203.0.113.0/24) and
+# TEST-NET-2 (198.51.100.0/24) from RFC 5737: documentation ranges that are
+# guaranteed never to be real destinations, so a leaked sim route cannot
+# blackhole something that matters.
+# ---------------------------------------------------------------------------
+WAN_DHCP_VLAN = 53          # "10gig-equivalent" -- the primary in production
+WAN_PPPOE_VLAN = 51         # "Vodafone-equivalent" -- the backup
+ISP_DHCP_NET = "203.0.113.0/24"
+ISP_DHCP_GW = "203.0.113.1"
+ISP_DHCP_POOL = ("203.0.113.100", "203.0.113.150")
+ISP_PPPOE_NET = "198.51.100.0/24"
+ISP_PPPOE_GW = "198.51.100.1"
+ISP_PPPOE_POOL = ("198.51.100.100", "198.51.100.150")
+# Sim-only fake credentials. Both ends are in this file on purpose: they
+# authenticate nothing real, and splitting them across a secret store would make
+# the sim unreproducible for no security gain. The PRODUCTION PPPoE password
+# lives in /config/wan-secrets on the router and is never in git.
+PPPOE_USER, PPPOE_PASS = "simdsl", "simpass"
+PPPOE_MTU = 1492            # 1500 - 8 bytes of PPPoE header
+PPPOE_AC = "sim-isp"
+
+# Failover probe targets. NOT 8.8.8.8/8.8.4.4: those are `system name-server`,
+# so a probe failure and a DNS failure would be the same event and the router
+# would flap the WAN every time DNS hiccuped.
+PROBE_TARGETS = ["9.9.9.9", "208.67.222.222"]
+# The bug this shape fixes (WI-8, found here, fixed in production): `ping -I
+# bond0.53` binds the SOURCE address but does not make the kernel use that
+# interface's gateway. On a cold boot where PPPoE won the default route, probes
+# for the 10 gig egressed via PPPoE, succeeded, and the 10 gig was still never
+# selected -- the house ran on the backup line silently. Pinning each target as
+# a /32 via `dhcp-interface` forces the probe onto the line being tested.
+WAN_DHCP_DISTANCE = 210     # NOT `no-default-route`, which blanks new_routers
+PPPOE_DISTANCE = 10         #   in the lease file, leaving failover no gateway
+                            #   to install and silently handing the default
+                            #   route to the backup line.
+SIM_LAN = "172.31.0.0/16"
+
+# The sim routers' own libvirt-NAT uplink, from before the ISP VMs existed. It
+# is a third default route that does not exist in production and quietly masks
+# WAN failures during a failover test. `--drop-scaffold` removes it.
+SCAFFOLD_IF = "eth2"
+SCAFFOLD_NAT_RULE = 100
+
+
+def bgp(role: str) -> list[str]:
+    """eBGP toward the k3s nodes. Identical on both routers except router-id."""
+    octet = 252 if role == "primary" else 253
+    out = [
+        f"# --- BGP: AS{ROUTER_AS} <-> AS{CLUSTER_AS} (k3s/Cilium) ---",
+        # FRR enforces RFC 8212: an eBGP session with no policy establishes but
+        # exchanges ZERO prefixes, silently. Both directions need a policy or
+        # the session looks perfectly healthy and carries nothing.
+        f"set policy prefix-list {PFX_LIST} rule 10 action permit",
+        f"set policy prefix-list {PFX_LIST} rule 10 prefix {SERVICE_CIDR}",
+        # `le 32` because Cilium advertises individual /32 service addresses out
+        # of the pool, not the aggregate.
+        f"set policy prefix-list {PFX_LIST} rule 10 le 32",
+        f"set policy route-map {RM_IN} rule 10 action permit",
+        f"set policy route-map {RM_IN} rule 10 match ip address prefix-list {PFX_LIST}",
+        # Deny everything outbound. The cluster must never learn a default route
+        # from us -- Cilium would install it and blackhole pod egress.
+        f"set policy route-map {RM_OUT} rule 10 action deny",
+        f"set protocols bgp system-as {ROUTER_AS}",
+        f"set protocols bgp parameters router-id 172.31.{K8S_VLAN}.{octet}",
+        f"set protocols bgp address-family ipv4-unicast maximum-paths ebgp {MAX_PATHS}",
+        f"set protocols bgp peer-group {PEER_GROUP} remote-as {CLUSTER_AS}",
+        f"set protocols bgp peer-group {PEER_GROUP} address-family ipv4-unicast route-map import {RM_IN}",
+        f"set protocols bgp peer-group {PEER_GROUP} address-family ipv4-unicast route-map export {RM_OUT}",
+        f"set protocols bgp peer-group {PEER_GROUP} address-family ipv4-unicast maximum-prefix {MAX_PREFIX}",
+    ]
+    out += [f"set protocols bgp neighbor {n} peer-group {PEER_GROUP}" for n in K8S_NODES]
+    out.append("")
+    return out
+
+
+def wan(drop_scaffold: bool) -> list[str]:
+    """Dual WAN + health-checked failover. Primary router only -- see README."""
+    out = [
+        "# --- WAN: DHCP (primary) + PPPoE (backup), health-checked ---",
+        f"set interfaces bonding bond0 vif {WAN_PPPOE_VLAN} description "
+        f"'WAN1 Vodafone-equivalent (sim ISP PPPoE)'",
+        f"set interfaces bonding bond0 vif {WAN_DHCP_VLAN} address dhcp",
+        f"set interfaces bonding bond0 vif {WAN_DHCP_VLAN} description "
+        f"'WAN3 10gig-equivalent (sim ISP DHCP)'",
+        f"set interfaces bonding bond0 vif {WAN_DHCP_VLAN} dhcp-options "
+        f"default-route-distance {WAN_DHCP_DISTANCE}",
+        f"set interfaces pppoe pppoe0 source-interface bond0.{WAN_PPPOE_VLAN}",
+        f"set interfaces pppoe pppoe0 authentication username {PPPOE_USER}",
+        f"set interfaces pppoe pppoe0 authentication password {PPPOE_PASS}",
+        f"set interfaces pppoe pppoe0 default-route-distance {PPPOE_DISTANCE}",
+        f"set interfaces pppoe pppoe0 mtu {PPPOE_MTU}",
+        # The ISP's resolvers would otherwise overwrite ours in resolv.conf every
+        # time the session comes up.
+        "set interfaces pppoe pppoe0 no-peer-dns",
+        "",
+        "# Failover: prefer the DHCP WAN, fall back to PPPoE when probes fail.",
+        f"set protocols failover route 0.0.0.0/0 dhcp-interface bond0.{WAN_DHCP_VLAN} metric 1",
+        f"set protocols failover route 0.0.0.0/0 dhcp-interface bond0.{WAN_DHCP_VLAN} check type icmp",
+        f"set protocols failover route 0.0.0.0/0 dhcp-interface bond0.{WAN_DHCP_VLAN} check timeout 5",
+        # any-available, not all: one unreachable public resolver is a normal
+        # internet event, not a reason to abandon a working 10 gig line.
+        f"set protocols failover route 0.0.0.0/0 dhcp-interface bond0.{WAN_DHCP_VLAN} check policy any-available",
+    ]
+    for t in PROBE_TARGETS:
+        out.append(f"set protocols failover route 0.0.0.0/0 dhcp-interface "
+                   f"bond0.{WAN_DHCP_VLAN} check target {t}")
+    out.append("")
+    out.append("# Pin the probe targets to the line under test (WI-8 -- see above).")
+    for t in PROBE_TARGETS:
+        out.append(f"set protocols static route {t}/32 dhcp-interface bond0.{WAN_DHCP_VLAN}")
+    out += [
+        "",
+        "# Masquerade out of whichever WAN currently holds the default route.",
+        f"set nat source rule 110 outbound-interface name bond0.{WAN_DHCP_VLAN}",
+        f"set nat source rule 110 source address {SIM_LAN}",
+        "set nat source rule 110 translation address masquerade",
+        "set nat source rule 120 outbound-interface name pppoe0",
+        f"set nat source rule 120 source address {SIM_LAN}",
+        "set nat source rule 120 translation address masquerade",
+        "",
+    ]
+    if drop_scaffold:
+        out += [
+            "# Remove the pre-ISP-VM libvirt-NAT uplink: a third default route",
+            "# that has no production equivalent and hides real WAN failures.",
+            f"delete interfaces ethernet {SCAFFOLD_IF} address",
+            f"delete nat source rule {SCAFFOLD_NAT_RULE}",
+            "",
+        ]
+    return out
+
+
+# ---------------------------------------------------------------------------
+# Firewall. The policy is: internal VLANs talk to each other and to the
+# internet; the internet initiates nothing inward.
+#
+# That was already the *effect* of the previous IPv4 ruleset, but it was built
+# as a blacklist -- `default-action accept` plus explicit drops on each WAN
+# interface. The result is identical right up until someone adds a WAN, at
+# which point it is wide open and nothing looks wrong. This is the same policy
+# expressed as a whitelist, so a new interface is closed until it is named.
+# ---------------------------------------------------------------------------
+# Management is `bond0.1`, NOT the bare `bond0`. Every VLAN is tagged and the
+# bond parent carries no subnet at all -- see NATIVE_VLAN in ovs.sh for why.
+#
+# This line is the trap in that change. The address move is the visible part and
+# the part you remember; leaving `bond0` here instead of `bond0.1` means the
+# whole Management VLAN falls outside the LAN group, and with a default-deny
+# ruleset that is every management session and all inter-VLAN routing for VLAN 1
+# dropped the instant the commit lands -- on a router you reach through itself.
+LAN_IFACES = ["bond0.1", "bond0.2", "bond0.3", "bond0.9", "bond0.10", "bond0.200"]
+LAN_GROUP = "LAN"
+
+
+def firewall(wan_dhcp_if: str | None = f"bond0.{WAN_DHCP_VLAN}") -> list[str]:
+    """wan_dhcp_if=None on a router with no DHCP WAN -- a firewall rule naming
+    an interface that does not exist is rejected at commit."""
+    out = [f"# --- firewall: LAN-to-anywhere, internet-to-nothing ---"]
+    # Delete each filter before rebuilding it. `set` on a rule number is
+    # ADDITIVE: if a rule 10 already exists carrying an inbound-interface
+    # constraint, `set ... rule 10 state established` silently ANDs onto it,
+    # and you get a stateful-accept rule that only applies to one interface
+    # pair. Observed in labsim: return traffic from the internet matched
+    # neither that rule nor the LAN rule and hit the default drop, so LAN
+    # hosts could reach nothing outbound. Everything here is one commit, so
+    # nftables is rebuilt atomically -- there is no window with no firewall.
+    out += [f"delete firewall {fam} {hook} filter"
+            for fam in ("ipv4", "ipv6") for hook in ("forward", "input")]
+    out += [f"set firewall group interface-group {LAN_GROUP} interface {i}"
+            for i in LAN_IFACES]
+    out += [
+        "",
+        # INPUT -- traffic terminating ON the router.
+        # Loopback first. Under a default-drop input policy, services talking to
+        # 127.0.0.1 are filtered like anything else, and the failures are
+        # bizarre and hard to attribute. Nothing off-box can forge iif lo.
+        "set firewall ipv4 input filter rule 5 action accept",
+        "set firewall ipv4 input filter rule 5 description 'loopback'",
+        "set firewall ipv4 input filter rule 5 inbound-interface name lo",
+        "set firewall ipv4 input filter rule 10 action accept",
+        "set firewall ipv4 input filter rule 10 description 'established/related'",
+        "set firewall ipv4 input filter rule 10 state established",
+        "set firewall ipv4 input filter rule 10 state related",
+        # One rule covers VRRP, conntrack-sync, kea HA, SSH, DNS and BGP,
+        # because every one of them arrives on a LAN interface. Enumerating the
+        # protocols instead would mean a new firewall rule every time the pair
+        # gains a feature -- and a lockout the day someone forgets.
+        f"set firewall ipv4 input filter rule 20 action accept",
+        f"set firewall ipv4 input filter rule 20 description 'trusted LAN to the router'",
+        f"set firewall ipv4 input filter rule 20 inbound-interface group {LAN_GROUP}",
+        # DHCP client. Lease RENEWAL is unicast UDP to port 68 and conntrack
+        # does not reliably cover it, so without this the WAN keeps working
+        # until the lease expires and then dies -- a delayed failure that looks
+        # nothing like a firewall change.
+        "set firewall ipv4 input filter default-action drop",
+        "",
+        # FORWARD -- traffic passing THROUGH the router.
+        "set firewall ipv4 forward filter rule 10 action accept",
+        "set firewall ipv4 forward filter rule 10 description 'established/related'",
+        "set firewall ipv4 forward filter rule 10 state established",
+        "set firewall ipv4 forward filter rule 10 state related",
+        # Inter-VLAN *and* LAN-to-internet in one rule: both are "came in on a
+        # LAN interface". Deliberately no restriction between internal VLANs --
+        # segmenting them is a separate decision, not a side effect of this one.
+        f"set firewall ipv4 forward filter rule 20 action accept",
+        f"set firewall ipv4 forward filter rule 20 description 'LAN to anywhere (inter-VLAN + internet)'",
+        f"set firewall ipv4 forward filter rule 20 inbound-interface group {LAN_GROUP}",
+        "set firewall ipv4 forward filter default-action drop",
+        "",
+        # IPv6 already runs default-deny. It only lacks the loopback rule.
+        "set firewall ipv6 input filter rule 5 action accept",
+        "set firewall ipv6 input filter rule 5 description 'loopback'",
+        "set firewall ipv6 input filter rule 5 inbound-interface name lo",
+        "set firewall ipv6 input filter rule 10 action accept",
+        "set firewall ipv6 input filter rule 10 description 'replies to our own traffic'",
+        "set firewall ipv6 input filter rule 10 state established",
+        "set firewall ipv6 input filter rule 10 state related",
+        # RFC 4890: filtering ICMPv6 wholesale breaks ND and PMTUD, which
+        # presents as "IPv6 works until something large", not as a block.
+        "set firewall ipv6 input filter rule 20 action accept",
+        "set firewall ipv6 input filter rule 20 description 'ICMPv6 - ND/RA/PMTUD'",
+        "set firewall ipv6 input filter rule 20 protocol icmpv6",
+        f"set firewall ipv6 input filter rule 30 action accept",
+        f"set firewall ipv6 input filter rule 30 description 'trusted LAN to the router'",
+        f"set firewall ipv6 input filter rule 30 inbound-interface group {LAN_GROUP}",
+        "set firewall ipv6 input filter default-action drop",
+        "set firewall ipv6 forward filter rule 10 action accept",
+        "set firewall ipv6 forward filter rule 10 description 'replies to our own traffic'",
+        "set firewall ipv6 forward filter rule 10 state established",
+        "set firewall ipv6 forward filter rule 10 state related",
+        "set firewall ipv6 forward filter rule 20 action accept",
+        "set firewall ipv6 forward filter rule 20 description 'ICMPv6 - ND/RA/PMTUD'",
+        "set firewall ipv6 forward filter rule 20 protocol icmpv6",
+        f"set firewall ipv6 forward filter rule 30 action accept",
+        f"set firewall ipv6 forward filter rule 30 description 'trusted LAN interfaces only'",
+        f"set firewall ipv6 forward filter rule 30 inbound-interface group {LAN_GROUP}",
+        "set firewall ipv6 forward filter default-action drop",
+        "",
+    ]
+    if wan_dhcp_if:
+        dhcp = [
+            "set firewall ipv4 input filter rule 140 action accept",
+            "set firewall ipv4 input filter rule 140 description 'DHCP client lease renewal'",
+            "set firewall ipv4 input filter rule 140 protocol udp",
+            "set firewall ipv4 input filter rule 140 destination port 68",
+            f"set firewall ipv4 input filter rule 140 inbound-interface name {wan_dhcp_if}",
+        ]
+        i = out.index("set firewall ipv4 input filter default-action drop")
+        out[i:i] = dhcp
+    return out
+
+
+def isp_dhcp(wan_if: str, uplink_if: str) -> list[str]:
+    """The 10gig-equivalent ISP: hands out a lease, NATs to the real internet."""
+    return [
+        f"# --- sim ISP: DHCP WAN on VLAN {WAN_DHCP_VLAN} ---",
+        "set system host-name isp-dhcp",
+        f"set interfaces ethernet {wan_if} address {ISP_DHCP_GW}/24",
+        f"set interfaces ethernet {wan_if} description "
+        f"'sim ISP - 10gig-equivalent WAN on VLAN{WAN_DHCP_VLAN}'",
+        f"set interfaces ethernet {uplink_if} address dhcp",
+        f"set interfaces ethernet {uplink_if} description 'uplink to the real internet'",
+        f"set service dhcp-server shared-network-name WAN{WAN_DHCP_VLAN} "
+        f"subnet {ISP_DHCP_NET} subnet-id 1",
+        f"set service dhcp-server shared-network-name WAN{WAN_DHCP_VLAN} "
+        f"subnet {ISP_DHCP_NET} option default-router {ISP_DHCP_GW}",
+        f"set service dhcp-server shared-network-name WAN{WAN_DHCP_VLAN} "
+        f"subnet {ISP_DHCP_NET} option name-server 8.8.8.8",
+        f"set service dhcp-server shared-network-name WAN{WAN_DHCP_VLAN} "
+        f"subnet {ISP_DHCP_NET} range CUST start {ISP_DHCP_POOL[0]}",
+        f"set service dhcp-server shared-network-name WAN{WAN_DHCP_VLAN} "
+        f"subnet {ISP_DHCP_NET} range CUST stop {ISP_DHCP_POOL[1]}",
+        "",
+    ] + _isp_common(uplink_if, ISP_DHCP_NET, "sim ISP: NAT customers to the real internet")
+
+
+def isp_pppoe(wan_if: str, uplink_if: str) -> list[str]:
+    """The Vodafone-equivalent ISP: terminates PPPoE, NATs to the real internet."""
+    return [
+        f"# --- sim ISP: PPPoE WAN on VLAN {WAN_PPPOE_VLAN} ---",
+        "set system host-name isp-pppoe",
+        f"set interfaces ethernet {wan_if} description "
+        f"'sim ISP - Vodafone-equivalent WAN on VLAN{WAN_PPPOE_VLAN} (PPPoE)'",
+        f"set interfaces ethernet {uplink_if} address dhcp",
+        f"set interfaces ethernet {uplink_if} description 'uplink to the real internet'",
+        f"set service pppoe-server access-concentrator {PPPOE_AC}",
+        f"set service pppoe-server interface {wan_if}",
+        f"set service pppoe-server gateway-address {ISP_PPPOE_GW}",
+        "set service pppoe-server authentication mode local",
+        f"set service pppoe-server authentication local-users username {PPPOE_USER} "
+        f"password {PPPOE_PASS}",
+        f"set service pppoe-server client-ip-pool CUST range "
+        f"{ISP_PPPOE_POOL[0]}-{ISP_PPPOE_POOL[1]}",
+        "set service pppoe-server default-pool CUST",
+        "set service pppoe-server name-server 8.8.8.8",
+        "",
+    ] + _isp_common(uplink_if, ISP_PPPOE_NET, "sim ISP: NAT PPPoE customers to the real internet")
+
+
+def _isp_common(uplink_if: str, customer_net: str, desc: str) -> list[str]:
+    return [
+        "set nat source rule 100 description " + f"'{desc}'",
+        f"set nat source rule 100 outbound-interface name {uplink_if}",
+        f"set nat source rule 100 source address {customer_net}",
+        "set nat source rule 100 translation address masquerade",
+        "",
+        # An ISP that drops return traffic is not simulating an ISP. The forward
+        # chain defaults to accept here on purpose -- these VMs model the
+        # internet, and the thing under test is the router's firewall, not this.
+        "set firewall ipv4 forward filter default-action accept",
+        "set firewall ipv4 forward filter rule 10 action accept",
+        "set firewall ipv4 forward filter rule 10 state established",
+        "set firewall ipv4 forward filter rule 10 state related",
+        "set firewall ipv4 forward filter rule 10 description conntrack-engage",
+        "",
+    ]
+
+
+def build(role: str, drop_scaffold: bool, wan_if: str, uplink_if: str) -> list[str]:
+    if role == "primary":
+        # WAN lives on the primary only. Production has WAN on both routers;
+        # the sim does not, because two PPPoE clients sharing one credential
+        # against a single access concentrator is a different failure mode than
+        # anything production has. VRRP/conntrack failover is still exercised --
+        # see README, "known gaps".
+        return ([f"# labsim routing -- {role}", ""]
+                + bgp(role) + wan(drop_scaffold) + firewall())
+    if role == "secondary":
+        # The backup has no WAN in the sim, so it has no DHCP client to
+        # exempt -- but it gets the same policy otherwise, because after a VRRP
+        # failover it IS the router and a divergent ruleset would only be
+        # discovered during the failover.
+        return ([f"# labsim routing -- {role}", ""]
+                + bgp(role) + firewall(wan_dhcp_if=None))
+    if role == "isp-dhcp":
+        return isp_dhcp(wan_if, uplink_if)
+    return isp_pppoe(wan_if, uplink_if)
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--role", required=True,
+                    choices=("primary", "secondary", "isp-dhcp", "isp-pppoe"))
+    ap.add_argument("--drop-scaffold", action="store_true",
+                    help="also remove the pre-ISP-VM libvirt-NAT uplink (primary only)")
+    # The ISP VMs' interface names depend on PCI enumeration order, which is not
+    # stable across a rebuild: isp-dhcp came up as eth0/eth1 and isp-pppoe as
+    # eth2/eth3 from identical XML. Check with `show interfaces` before applying
+    # rather than trusting these defaults.
+    ap.add_argument("--wan-if", default=None, help="ISP VM: customer-facing NIC")
+    ap.add_argument("--uplink-if", default=None, help="ISP VM: internet-facing NIC")
+    args = ap.parse_args()
+
+    defaults = {"isp-dhcp": ("eth0", "eth1"), "isp-pppoe": ("eth2", "eth3")}
+    w, u = defaults.get(args.role, ("", ""))
+    w, u = args.wan_if or w, args.uplink_if or u
+
+    if args.drop_scaffold and args.role != "primary":
+        print("--drop-scaffold only applies to --role primary", file=sys.stderr)
+        return 2
+
+    lines = build(args.role, args.drop_scaffold, w, u)
+    sys.stdout.write("\n".join(lines) + "\n")
+    n = len([l for l in lines if l.startswith(("set ", "delete "))])
+    print(f"{args.role}: {n} commands", file=sys.stderr)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

labsim/vlan-leak-evidence/after-vlan1/capture-parent.txt

@@ -0,0 +1,6 @@
+12:52:35.919490 52:54:00:6d:71:e7 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 1, p 0, ethertype IPv4 (0x0800), 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:6d:71:e7, length 300
+12:52:35.920089 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype 802.1Q (0x8100), length 329: vlan 1, p 0, ethertype IPv4 (0x0800), 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283
+12:52:35.920307 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype 802.1Q (0x8100), length 329: vlan 1, p 0, ethertype IPv4 (0x0800), 172.31.1.252.67 > 172.31.1.7.68: BOOTP/DHCP, Reply, length 283
+12:52:35.922052 52:54:00:6d:71:e7 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 1, p 0, ethertype IPv4 (0x0800), 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:6d:71:e7, length 300
+12:52:35.922509 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype 802.1Q (0x8100), length 329: vlan 1, p 0, ethertype IPv4 (0x0800), 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283
+12:52:35.923327 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype 802.1Q (0x8100), length 329: vlan 1, p 0, ethertype IPv4 (0x0800), 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283

labsim/vlan-leak-evidence/after-vlan1/capture-vif.txt

@@ -0,0 +1,6 @@
+12:52:35.919490 52:54:00:6d:71:e7 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:6d:71:e7, length 300
+12:52:35.920081 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype IPv4 (0x0800), length 325: 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283
+12:52:35.920305 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype IPv4 (0x0800), length 325: 172.31.1.252.67 > 172.31.1.7.68: BOOTP/DHCP, Reply, length 283
+12:52:35.922052 52:54:00:6d:71:e7 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:6d:71:e7, length 300
+12:52:35.922507 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype IPv4 (0x0800), length 325: 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283
+12:52:35.923326 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype IPv4 (0x0800), length 325: 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283

labsim/vlan-leak-evidence/after-vlan1/client.txt

@@ -0,0 +1,4 @@
+udhcpc: started, v1.37.0
+udhcpc: broadcasting discover
+udhcpc: broadcasting select for 172.31.1.6, server 172.31.1.252
+udhcpc: lease of 172.31.1.6 obtained from 172.31.1.252, lease time 86400

labsim/vlan-leak-evidence/after-vlan1/router-config.txt

@@ -0,0 +1,64 @@
+set high-availability vrrp group native address 172.31.1.1/24
+set high-availability vrrp group native hello-source-address '172.31.1.252'
+set high-availability vrrp group native interface 'bond0.1'
+set high-availability vrrp group native no-preempt
+set high-availability vrrp group native peer-address '172.31.1.253'
+set high-availability vrrp group native priority '200'
+set high-availability vrrp group native vrid '1'
+set high-availability vrrp group vlan2 address 172.31.2.1/24
+set high-availability vrrp group vlan2 hello-source-address '172.31.2.252'
+set high-availability vrrp group vlan2 interface 'bond0.2'
+set high-availability vrrp group vlan2 no-preempt
+set high-availability vrrp group vlan2 peer-address '172.31.2.253'
+set high-availability vrrp group vlan2 priority '200'
+set high-availability vrrp group vlan2 vrid '2'
+set high-availability vrrp group vlan3 address 172.31.3.1/24
+set high-availability vrrp group vlan3 hello-source-address '172.31.3.252'
+set high-availability vrrp group vlan3 interface 'bond0.3'
+set high-availability vrrp group vlan3 no-preempt
+set high-availability vrrp group vlan3 peer-address '172.31.3.253'
+set high-availability vrrp group vlan3 priority '200'
+set high-availability vrrp group vlan3 vrid '3'
+set high-availability vrrp group vlan9 address 172.31.9.1/24
+set high-availability vrrp group vlan9 hello-source-address '172.31.9.252'
+set high-availability vrrp group vlan9 interface 'bond0.9'
+set high-availability vrrp group vlan9 no-preempt
+set high-availability vrrp group vlan9 peer-address '172.31.9.253'
+set high-availability vrrp group vlan9 priority '200'
+set high-availability vrrp group vlan9 vrid '9'
+set high-availability vrrp group vlan10 address 172.31.10.1/23
+set high-availability vrrp group vlan10 hello-source-address '172.31.10.252'
+set high-availability vrrp group vlan10 interface 'bond0.10'
+set high-availability vrrp group vlan10 no-preempt
+set high-availability vrrp group vlan10 peer-address '172.31.10.253'
+set high-availability vrrp group vlan10 priority '200'
+set high-availability vrrp group vlan10 vrid '10'
+set high-availability vrrp group vlan200 address 172.31.200.1/24
+set high-availability vrrp group vlan200 hello-source-address '172.31.200.252'
+set high-availability vrrp group vlan200 interface 'bond0.200'
+set high-availability vrrp group vlan200 no-preempt
+set high-availability vrrp group vlan200 peer-address '172.31.200.253'
+set high-availability vrrp group vlan200 priority '200'
+set high-availability vrrp group vlan200 vrid '200'
+set interfaces bonding bond0 description 'api-batch-test'
+set interfaces bonding bond0 hash-policy 'layer2+3'
+set interfaces bonding bond0 lacp-rate 'fast'
+set interfaces bonding bond0 member interface 'eth0'
+set interfaces bonding bond0 member interface 'eth1'
+set interfaces bonding bond0 mode '802.3ad'
+set interfaces bonding bond0 vif 1 address '172.31.1.252/24'
+set interfaces bonding bond0 vif 1 description 'management'
+set interfaces bonding bond0 vif 2 address '172.31.2.252/24'
+set interfaces bonding bond0 vif 2 description 'k8s'
+set interfaces bonding bond0 vif 3 address '172.31.3.252/24'
+set interfaces bonding bond0 vif 3 description 'kvm'
+set interfaces bonding bond0 vif 9 address '172.31.9.252/24'
+set interfaces bonding bond0 vif 9 description 'private'
+set interfaces bonding bond0 vif 10 address '172.31.10.252/23'
+set interfaces bonding bond0 vif 10 description 'lot'
+set interfaces bonding bond0 vif 51 description 'WAN1 Vodafone-equivalent (sim ISP PPPoE)'
+set interfaces bonding bond0 vif 53 address 'dhcp'
+set interfaces bonding bond0 vif 53 description 'WAN3 10gig-equivalent (sim ISP DHCP)'
+set interfaces bonding bond0 vif 53 dhcp-options default-route-distance '210'
+set interfaces bonding bond0 vif 200 address '172.31.200.252/24'
+set interfaces bonding bond0 vif 200 description 'roomates'

labsim/vlan-leak-evidence/after/capture-parent.txt

@@ -0,0 +1,6 @@
+12:52:14.639170 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 3, p 0, ethertype IPv4 (0x0800), 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
+12:52:14.640467 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype 802.1Q (0x8100), length 329: vlan 3, p 0, ethertype IPv4 (0x0800), 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
+12:52:14.640846 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype 802.1Q (0x8100), length 329: vlan 3, p 0, ethertype IPv4 (0x0800), 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
+12:52:14.642554 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 3, p 0, ethertype IPv4 (0x0800), 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
+12:52:14.642766 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype 802.1Q (0x8100), length 329: vlan 3, p 0, ethertype IPv4 (0x0800), 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
+12:52:14.643056 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype 802.1Q (0x8100), length 329: vlan 3, p 0, ethertype IPv4 (0x0800), 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283

labsim/vlan-leak-evidence/after/capture-vif.txt

@@ -0,0 +1,6 @@
+12:52:14.639170 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
+12:52:14.640465 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
+12:52:14.640845 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
+12:52:14.642554 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
+12:52:14.642764 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
+12:52:14.643055 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283

labsim/vlan-leak-evidence/after/client.txt

@@ -0,0 +1,4 @@
+udhcpc: started, v1.37.0
+udhcpc: broadcasting discover
+udhcpc: broadcasting select for 172.31.3.11, server 172.31.3.252
+udhcpc: lease of 172.31.3.11 obtained from 172.31.3.252, lease time 85374

labsim/vlan-leak-evidence/after/router-config.txt

@@ -0,0 +1,64 @@
+set high-availability vrrp group native address 172.31.1.1/24
+set high-availability vrrp group native hello-source-address '172.31.1.252'
+set high-availability vrrp group native interface 'bond0.1'
+set high-availability vrrp group native no-preempt
+set high-availability vrrp group native peer-address '172.31.1.253'
+set high-availability vrrp group native priority '200'
+set high-availability vrrp group native vrid '1'
+set high-availability vrrp group vlan2 address 172.31.2.1/24
+set high-availability vrrp group vlan2 hello-source-address '172.31.2.252'
+set high-availability vrrp group vlan2 interface 'bond0.2'
+set high-availability vrrp group vlan2 no-preempt
+set high-availability vrrp group vlan2 peer-address '172.31.2.253'
+set high-availability vrrp group vlan2 priority '200'
+set high-availability vrrp group vlan2 vrid '2'
+set high-availability vrrp group vlan3 address 172.31.3.1/24
+set high-availability vrrp group vlan3 hello-source-address '172.31.3.252'
+set high-availability vrrp group vlan3 interface 'bond0.3'
+set high-availability vrrp group vlan3 no-preempt
+set high-availability vrrp group vlan3 peer-address '172.31.3.253'
+set high-availability vrrp group vlan3 priority '200'
+set high-availability vrrp group vlan3 vrid '3'
+set high-availability vrrp group vlan9 address 172.31.9.1/24
+set high-availability vrrp group vlan9 hello-source-address '172.31.9.252'
+set high-availability vrrp group vlan9 interface 'bond0.9'
+set high-availability vrrp group vlan9 no-preempt
+set high-availability vrrp group vlan9 peer-address '172.31.9.253'
+set high-availability vrrp group vlan9 priority '200'
+set high-availability vrrp group vlan9 vrid '9'
+set high-availability vrrp group vlan10 address 172.31.10.1/23
+set high-availability vrrp group vlan10 hello-source-address '172.31.10.252'
+set high-availability vrrp group vlan10 interface 'bond0.10'
+set high-availability vrrp group vlan10 no-preempt
+set high-availability vrrp group vlan10 peer-address '172.31.10.253'
+set high-availability vrrp group vlan10 priority '200'
+set high-availability vrrp group vlan10 vrid '10'
+set high-availability vrrp group vlan200 address 172.31.200.1/24
+set high-availability vrrp group vlan200 hello-source-address '172.31.200.252'
+set high-availability vrrp group vlan200 interface 'bond0.200'
+set high-availability vrrp group vlan200 no-preempt
+set high-availability vrrp group vlan200 peer-address '172.31.200.253'
+set high-availability vrrp group vlan200 priority '200'
+set high-availability vrrp group vlan200 vrid '200'
+set interfaces bonding bond0 description 'api-batch-test'
+set interfaces bonding bond0 hash-policy 'layer2+3'
+set interfaces bonding bond0 lacp-rate 'fast'
+set interfaces bonding bond0 member interface 'eth0'
+set interfaces bonding bond0 member interface 'eth1'
+set interfaces bonding bond0 mode '802.3ad'
+set interfaces bonding bond0 vif 1 address '172.31.1.252/24'
+set interfaces bonding bond0 vif 1 description 'management'
+set interfaces bonding bond0 vif 2 address '172.31.2.252/24'
+set interfaces bonding bond0 vif 2 description 'k8s'
+set interfaces bonding bond0 vif 3 address '172.31.3.252/24'
+set interfaces bonding bond0 vif 3 description 'kvm'
+set interfaces bonding bond0 vif 9 address '172.31.9.252/24'
+set interfaces bonding bond0 vif 9 description 'private'
+set interfaces bonding bond0 vif 10 address '172.31.10.252/23'
+set interfaces bonding bond0 vif 10 description 'lot'
+set interfaces bonding bond0 vif 51 description 'WAN1 Vodafone-equivalent (sim ISP PPPoE)'
+set interfaces bonding bond0 vif 53 address 'dhcp'
+set interfaces bonding bond0 vif 53 description 'WAN3 10gig-equivalent (sim ISP DHCP)'
+set interfaces bonding bond0 vif 53 dhcp-options default-route-distance '210'
+set interfaces bonding bond0 vif 200 address '172.31.200.252/24'
+set interfaces bonding bond0 vif 200 description 'roomates'

labsim/vlan-leak-evidence/before/capture-parent.txt

@@ -0,0 +1,5 @@
+12:37:08.491910 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 3, p 0, ethertype IPv4 (0x0800), 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
+12:37:08.492629 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.1.252.67 > 172.31.1.9.68: BOOTP/DHCP, Reply, length 283
+12:37:08.493628 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype 802.1Q (0x8100), length 329: vlan 3, p 0, ethertype IPv4 (0x0800), 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
+12:37:08.495587 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 3, p 0, ethertype IPv4 (0x0800), 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
+12:37:08.495946 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype 802.1Q (0x8100), length 329: vlan 3, p 0, ethertype IPv4 (0x0800), 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283

labsim/vlan-leak-evidence/before/capture-vif.txt

@@ -0,0 +1,4 @@
+12:37:08.491910 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
+12:37:08.493625 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
+12:37:08.495587 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
+12:37:08.495944 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283

labsim/vlan-leak-evidence/before/client.txt

@@ -0,0 +1,4 @@
+udhcpc: started, v1.37.0
+udhcpc: broadcasting discover
+udhcpc: broadcasting select for 172.31.3.11, server 172.31.3.252
+udhcpc: lease of 172.31.3.11 obtained from 172.31.3.252, lease time 86280

labsim/vlan-leak-evidence/before/router-config.txt

@@ -0,0 +1,63 @@
+set high-availability vrrp group native address 172.31.1.1/24
+set high-availability vrrp group native hello-source-address '172.31.1.252'
+set high-availability vrrp group native interface 'bond0'
+set high-availability vrrp group native no-preempt
+set high-availability vrrp group native peer-address '172.31.1.253'
+set high-availability vrrp group native priority '200'
+set high-availability vrrp group native vrid '1'
+set high-availability vrrp group vlan2 address 172.31.2.1/24
+set high-availability vrrp group vlan2 hello-source-address '172.31.2.252'
+set high-availability vrrp group vlan2 interface 'bond0.2'
+set high-availability vrrp group vlan2 no-preempt
+set high-availability vrrp group vlan2 peer-address '172.31.2.253'
+set high-availability vrrp group vlan2 priority '200'
+set high-availability vrrp group vlan2 vrid '2'
+set high-availability vrrp group vlan3 address 172.31.3.1/24
+set high-availability vrrp group vlan3 hello-source-address '172.31.3.252'
+set high-availability vrrp group vlan3 interface 'bond0.3'
+set high-availability vrrp group vlan3 no-preempt
+set high-availability vrrp group vlan3 peer-address '172.31.3.253'
+set high-availability vrrp group vlan3 priority '200'
+set high-availability vrrp group vlan3 vrid '3'
+set high-availability vrrp group vlan9 address 172.31.9.1/24
+set high-availability vrrp group vlan9 hello-source-address '172.31.9.252'
+set high-availability vrrp group vlan9 interface 'bond0.9'
+set high-availability vrrp group vlan9 no-preempt
+set high-availability vrrp group vlan9 peer-address '172.31.9.253'
+set high-availability vrrp group vlan9 priority '200'
+set high-availability vrrp group vlan9 vrid '9'
+set high-availability vrrp group vlan10 address 172.31.10.1/23
+set high-availability vrrp group vlan10 hello-source-address '172.31.10.252'
+set high-availability vrrp group vlan10 interface 'bond0.10'
+set high-availability vrrp group vlan10 no-preempt
+set high-availability vrrp group vlan10 peer-address '172.31.10.253'
+set high-availability vrrp group vlan10 priority '200'
+set high-availability vrrp group vlan10 vrid '10'
+set high-availability vrrp group vlan200 address 172.31.200.1/24
+set high-availability vrrp group vlan200 hello-source-address '172.31.200.252'
+set high-availability vrrp group vlan200 interface 'bond0.200'
+set high-availability vrrp group vlan200 no-preempt
+set high-availability vrrp group vlan200 peer-address '172.31.200.253'
+set high-availability vrrp group vlan200 priority '200'
+set high-availability vrrp group vlan200 vrid '200'
+set interfaces bonding bond0 address '172.31.1.252/24'
+set interfaces bonding bond0 description 'api-batch-test'
+set interfaces bonding bond0 hash-policy 'layer2+3'
+set interfaces bonding bond0 lacp-rate 'fast'
+set interfaces bonding bond0 member interface 'eth0'
+set interfaces bonding bond0 member interface 'eth1'
+set interfaces bonding bond0 mode '802.3ad'
+set interfaces bonding bond0 vif 2 address '172.31.2.252/24'
+set interfaces bonding bond0 vif 2 description 'k8s'
+set interfaces bonding bond0 vif 3 address '172.31.3.252/24'
+set interfaces bonding bond0 vif 3 description 'kvm'
+set interfaces bonding bond0 vif 9 address '172.31.9.252/24'
+set interfaces bonding bond0 vif 9 description 'private'
+set interfaces bonding bond0 vif 10 address '172.31.10.252/23'
+set interfaces bonding bond0 vif 10 description 'lot'
+set interfaces bonding bond0 vif 51 description 'WAN1 Vodafone-equivalent (sim ISP PPPoE)'
+set interfaces bonding bond0 vif 53 address 'dhcp'
+set interfaces bonding bond0 vif 53 description 'WAN3 10gig-equivalent (sim ISP DHCP)'
+set interfaces bonding bond0 vif 53 dhcp-options default-route-distance '210'
+set interfaces bonding bond0 vif 200 address '172.31.200.252/24'
+set interfaces bonding bond0 vif 200 description 'roomates'

labsim/vlan1-move-monitor.sh

@@ -0,0 +1,17 @@
+#!/bin/bash
+# Timestamped liveness log for the Management VLAN during the bond0 -> bond0.1 move.
+#
+# The question this answers is not "did it work" but "for how long was it not
+# working, and what held the VIP while it was not". Both are invisible after the
+# fact: VRRP reconverges and leaves no trace of who was master during the gap.
+#
+#   ./vlan1-move-monitor.sh > /tmp/move.log &
+# Columns: time  VIP-ping  R1-ping  R2-ping  VIP-mac
+VIP="${VIP:-172.31.1.1}"; R1="${R1:-172.31.1.252}"; R2="${R2:-172.31.1.253}"
+p() { ping -c1 -W1 -n "$1" >/dev/null 2>&1 && echo up || echo DOWN; }
+while :; do
+  mac="$(ip neigh show "$VIP" 2>/dev/null | awk '{for(i=1;i<=NF;i++) if($i=="lladdr") print $(i+1)}')"
+  printf '%s  vip=%-4s r1=%-4s r2=%-4s vipmac=%s\n' \
+    "$(date +%H:%M:%S)" "$(p "$VIP")" "$(p "$R1")" "$(p "$R2")" "${mac:-none}"
+  sleep 1
+done

labsim/vlans.conf

@@ -12,10 +12,32 @@
 #   .10   the micro VM for this VLAN
 #   .254  VRRP VIP             (reserved, mirrors production)
 #
-# Format: vlan_id:name:sim_subnet_prefix:real_subnet(for reference)
+# Format: vlan_id:name:sim_subnet_prefix:real_subnet:[masklen]:[host_octet]
+#
+# masklen defaults to 24 and host_octet to 2. Both exist for VLAN 10, which is
+# the one VLAN that has to be a /23 here: every UniFi DHCP reservation lives in
+# LoT, and LoT spans 10.0.0.x AND 10.0.1.x, which a /24 cannot represent. With
+# /23 the mapping stays readable — 10.0.0.46 -> 172.31.10.46 and
+# 10.0.1.67 -> 172.31.11.67.
+#
+# LoT's host leg is .3 rather than .2 because 10.0.0.2 is a real reservation
+# (Hubitat) and would map straight onto the host's own address. .3 is free in
+# production and sits below the DHCP pool (which starts at .11), so it can
+# never be handed out.
 1:management:172.31.1:192.168.1.0/24
 2:k8s:172.31.2:192.168.8.0/23
 3:kvm:172.31.3:192.168.3.0/24
 9:private:172.31.9:10.0.9.0/23
-10:lot:172.31.10:10.0.0.0/23
+10:lot:172.31.10:10.0.0.0/23:23:3
 200:roomates:172.31.200:192.168.2.0/24
+
+# WAN transport VLANs, mirroring production. These exist so the sim can run a
+# fake ISP on each and the switch script's WAN health checks actually execute
+# instead of printing "this delta configures no WAN -- skipping". A cutover
+# attempt failed on the WAN with nothing having tested it, because the sim
+# modelled every LAN VLAN faithfully and omitted the WAN entirely.
+#
+# No host leg is wanted here (host_octet 0 means "skip"): the ISP VMs own these
+# segments, and a host address on a WAN transport VLAN would be a lie.
+51:wan1:172.31.51:vodafone-pppoe(VLAN 51):24:0
+53:wan3:172.31.53:10gig-dhcp(VLAN 53):24:0

migration/.gitignore

@@ -0,0 +1,4 @@
+# Raw UniFi export: contains WiFi passphrases (wlanconf) and controller auth
+# material (setting). The inventory is regenerable — never commit it.
+export/
+__pycache__/

migration/CUTOVER.md

@@ -0,0 +1,209 @@
+# Cutover runbook — USG to VyOS
+
+**Print this.** During the cutover there is no internet, so there is no
+assistant and no web search. Everything you need is on this page and on the
+boxes themselves.
+
+## Use these addresses. Not the other ones.
+
+| | use this | do NOT use |
+|---|---|---|
+| vyos001 (MASTER) | **`10.0.1.252`** | ~~192.168.8.143~~ |
+| vyos002 (BACKUP) | **`10.0.1.253`** | ~~192.168.8.144~~ |
+
+`ssh vyos@10.0.1.252` — by IP, not by name.
+
+**The `192.168.8.x` addresses stop working the instant the USG is unplugged.**
+That is not a maybe. Your workstation is on LoT (`10.0.0.210/23`) and reaching
+`192.168.8.x` requires routing *through the USG*:
+
+```
+ip route get 192.168.8.143  ->  via 10.0.0.1     <- the USG. Gone.
+ip route get 10.0.1.252     ->  dev lanbr0       <- same L2. Survives.
+```
+
+`10.0.1.252` and `.253` are on the LoT VLAN, the same broadcast domain as your
+workstation, so they need no gateway at all. They are the only remote path that
+survives the cutover.
+
+**Between unplugging the USG and finishing the switch there is no inter-VLAN
+routing.** In that window:
+
+- the **JetKVMs are unreachable** from your workstation (they are on Management
+  and kvm) — they are *not* a fallback during the gap
+- **Tailscale is down** with the internet
+- your workstation keeps `10.0.0.210` (86400s lease) and can still resolve via
+  `10.0.0.194`, which is also link-scope
+
+If LoT SSH fails, the next step is physical console, not the network.
+
+| | |
+|---|---|
+| JetKVMs (after routing is restored) | `192.168.1.28`, `192.168.1.29`, `192.168.3.6` |
+| Switch script | `/config/vyos-unifi-switch` on each box |
+| Peer link | `eth3` ↔ `eth3` direct cable, 2.5 GbE, `10.255.255.0/30` — conntrack state sync |
+| Login | user `vyos` |
+
+---
+
+## If something is wrong, do this
+
+```
+sudo /config/vyos-unifi-switch unifi
+```
+
+Then reconnect the USG. That command runs no health checks, asks nothing and
+cannot refuse. It restores a byte-exact copy of the configuration the box had
+before the cutover — verified by diff, not by assumption.
+
+**You do not have to be quick.** If you do nothing at all after
+`vyos-unifi-switch vyos`, the box reverts by itself within 10 minutes. Verified:
+config returns to the previous state and the box does **not** reboot
+(`uptime` and boot-id unchanged across an auto-revert).
+
+---
+
+## What has actually been tested
+
+Proven on the labsim router (same VyOS version, isolated OVS bridge with no
+physical NIC), by loading **vyos001's real running config** and applying the
+**real production delta**:
+
+- All 317 commands accepted, and the whole delta **commits** (`COMMIT OK`).
+- `unifi` mode restores the previous config **byte-exact** (diff clean).
+- Auto-revert fires when the commit is not confirmed: config returns to the
+  saved state and the box does **not** reboot — `uptime` and boot-id unchanged
+  across the revert.
+- Failed health checks trigger an immediate revert rather than waiting out the
+  timer.
+
+Two bugs were found this way and would each have failed the entire switch,
+since the delta commits as one unit: `bond0.51` did not exist for PPPoE to
+reference, and `translation port` rejects a port list.
+
+**Not tested, and untestable in advance:**
+
+- **PPPoE.** The line permits one session and the USG holds it. The first real
+  attempt is during the cutover.
+- **The commit on the real boxes.** The rehearsal ran with vyos001's `eth2` and
+  `eth3` stanzas stripped, because the sim VM has only two NICs. Those are
+  plain interface configs that already work on the real hardware, but they were
+  not part of what committed.
+
+## Before you unplug anything
+
+1. Tether your workstation to your phone if you want the assistant available.
+   Cutting the USG cuts your internet, not your LAN.
+2. On **both** boxes, confirm the machinery is present:
+   ```
+   sudo /config/vyos-unifi-switch status
+   ls -la /config/modes/          # unifi.boot + to-vyos.commands
+   ls -la /config/wan-secrets     # must be 0600
+   ```
+   `status` must report `mode: unifi`. If `unifi.boot` is missing, **stop** —
+   there is no way back without it.
+3. Confirm the revert action is `reload`, not `reboot`:
+   ```
+   show configuration commands | match commit-confirm
+   ```
+   Must show `action 'reload'`. Without it a failed switch **reboots** the
+   firewall instead of reverting it. The switch script refuses to run if this
+   is missing, but check anyway.
+
+## The cutover
+
+0. **Open both SSH sessions BEFORE you unplug anything**, and leave them open:
+   ```
+   ssh vyos@10.0.1.253      # vyos002, BACKUP
+   ssh vyos@10.0.1.252      # vyos001, MASTER
+   ```
+   If either will not connect, stop. Do not unplug the USG.
+
+1. **Physically disconnect the USG.** Not just powered off — disconnected. The
+   switch script refuses to run while anything still answers on a gateway
+   address, because two devices on `.1` is the worst available outcome.
+   You cannot switch first and unplug after, for exactly that reason.
+
+2. In the **vyos002 (BACKUP)** session, first:
+   ```
+   sudo /config/vyos-unifi-switch vyos
+   ```
+3. Watch the health checks. They cover PPPoE, the default route, kea, the DNS
+   forwarder and reachability. On failure the script reverts immediately and
+   tells you so.
+4. If vyos002 came up clean, repeat on **vyos001 (MASTER)**.
+5. Check a real client: does it get an address, and is it the *same* address as
+   before? Every active client has a reservation, so it should be.
+
+## What will probably go wrong first
+
+**The WAN.** There are two, and they behave differently:
+
+| | line | VLAN | transport | notes |
+|---|---|---|---|---|
+| WAN2 | 10 gig ISP | **53** | DHCP, public `87.192.101.48/21` | primary, distance 1 |
+| WAN1 | Vodafone | **51** | PPPoE, ~900/700 Mbit | failover, distance 10 |
+
+VyOS clones the USG's WAN2 MAC (`f0:9f:c2:12:9b:4f`) on `bond0.53`, which is how
+it keeps the existing public lease rather than asking for a new one.
+
+**Both boxes carry the identical WAN and NAT config.** vyos002's WAN interfaces
+are simply held administratively down, so the cloned MAC is never live on two
+boxes at once. To move the internet path to vyos002:
+
+```
+configure
+delete interfaces bonding bond0 vif 53 disable
+delete interfaces pppoe pppoe0 disable
+commit; save
+```
+
+Two lines. Do it only when vyos001 is genuinely down or disconnected — two boxes
+holding that MAC at once is exactly what the disable prevents.
+
+PPPoE is no longer an unknown: it was proven on the USG before cutover
+(`pppoe0` came up with `90.241.226.213`, MTU 1492). What remains untested is
+VyOS dialling it, and whether the ISP hands the same lease to the cloned MAC.
+
+If the WAN check fails:
+
+```
+show interfaces pppoe pppoe0
+sudo journalctl -u ppp@pppoe0 -n 50 --no-pager
+```
+
+Check the credential in `/config/wan-secrets` and that VLAN 51 actually reaches
+the box. If it will not come up, run `vyos-unifi-switch unifi`, reconnect the
+USG, and debug with the internet back on.
+
+## Things that are true and easy to forget
+
+- **WiFi keeps working, but through VyOS.** The SSIDs stay in UniFi and the APs
+  are untouched, but 37 of 83 active clients are wireless and every one is on
+  LoT — they get their addresses from VyOS now.
+- **DHCP leases last 24h (86400s).** A device that does not renew promptly keeps
+  its old address for a while. That is fine, not a symptom.
+- **The firewalls resolve via `8.8.8.8` / `8.8.4.4`** — matching the DNS the USG
+  used on its WAN. This means their own name resolution now depends on the
+  *internet* being up, so between unplugging the USG and PPPoE establishing,
+  the boxes have no DNS at all. That is expected and harmless: they only need
+  DNS for NTP hostnames, and the switch's own health checks use it precisely to
+  prove the WAN came up. Nothing in the switch itself resolves a name.
+- Internal `ad.itaz.eu` names still resolve through Google, because that zone is
+  published publicly with private addresses in it (`nas001` → `10.0.0.194`,
+  `kvm-macstudio1` → `192.168.3.8`). Convenient here; worth knowing it is public.
+- **The USG was a DNS resolver** for every VLAN except LoT. VyOS now runs
+  `dns forwarding` in its place. If names stop resolving but IPs still work,
+  that is where to look.
+- **`eth2` and `bond0.2` are both in `192.168.8.0/23`.** It works, but if you
+  see odd source-address behaviour on the management NIC, that is why.
+
+## Afterwards
+
+Once it has been stable for a day:
+
+- Re-run `migration/unifi-export.py` — the UniFi controller is no longer the
+  source of truth for DHCP, and the export will drift.
+- The VPN rules (ESP, UDP 500/4500) are carried over but the VPN itself still
+  terminated on the USG. Decide whether it moves.
+- `labsim` still holds a deliberate `kvm→k8s` drop rule from earlier testing.

migration/MANAGEMENT-VLAN-TAGGED.md

@@ -0,0 +1,137 @@
+# Moving Management onto a tagged VLAN
+
+Rehearsed end to end in labsim on 2026-09-02. This is the fix for kea serving
+addresses from the wrong VLAN's pool.
+
+## Why
+
+ISC Kea [#1117](https://gitlab.isc.org/isc-projects/kea/-/issues/1117): with
+`dhcp-socket-type: raw`, a frame tagged for a sub-interface is **also** delivered
+to the parent's `AF_PACKET` socket. If the parent serves a subnet, kea answers
+from it too. Ours does — Management is the native/untagged VLAN on `bond0` while
+VLANs 2/3/9/10/200 are sub-interfaces of that same bond — so one DISCOVER on
+VLAN 3 produces two OFFERs and the *client* decides which to keep:
+
+```
+bond0.3 :  192.168.3.14  correct
+bond0   :  192.168.1.28  UNTAGGED, Management pool, wrong
+```
+
+The fix is to leave **no subnet on the parent**: every VLAN tagged, Management
+included, moved from `bond0` to `bond0.1`.
+
+Confirmed in labsim across all six LAN VLANs: fails before, passes after.
+`labsim/labsim-vlan-leak-test.sh` is the test; evidence in
+`labsim/vlan-leak-evidence/`.
+
+## What must change together
+
+Per router:
+
+| | from | to |
+|---|---|---|
+| address | `interfaces bonding bond0 address` | `interfaces bonding bond0 vif 1 address` |
+| firewall | `interface-group LAN interface bond0` | `... interface bond0.1` |
+| VRRP | `vrrp group native interface bond0` | `... interface bond0.1` |
+| kea | — | **restart it** (see traps) |
+
+On the switch: Native VLAN = **None** on the trunk to that firewall, with VLAN 1
+added to the tagged set.
+
+## The ordering constraint
+
+**There is no overlap state.** An 802.1Q port always egresses its native VLAN
+untagged, so while VLAN 1 is native the router can *send* tagged VLAN 1 but can
+never *receive* it. Verified: a tagged VLAN 1 ARP sent from the switch arrived on
+`bond0` untagged and never on `bond0.1`. Configuring "native VLAN 1 **and** VLAN 1
+tagged" as a make-before-break does not work; the switch and router changes for a
+given firewall are strictly simultaneous, and that router loses Management in
+between.
+
+What makes this safe anyway: **tagged and untagged Management coexist on the
+same VLAN.** One VLAN is one broadcast domain no matter how each port tags it, so
+the firewalls can be converted one at a time — verified with the primary untagged
+and the secondary already tagged, both reachable, VIP up, VLAN 1 clients fine.
+
+Access ports are untouched throughout. The UniFi controller at 192.168.1.5 and
+your workstation are on access ports and never traverse the firewall trunks, so
+you keep the controller you are making the change from. Only the router being
+converted goes dark, and only until its own config lands.
+
+## Procedure
+
+Do the **backup** router first, then fail the VIPs over and do the other. You
+need console (JetKVM) on the router being converted — its Management SSH dies the
+moment the switch port changes.
+
+For each router in turn:
+
+1. Confirm the *other* router is MASTER and healthy:
+   `show vrrp` and `sudo /config/vrrp-wan-health; echo $?` (must be 0).
+2. Start the monitor from a workstation on an access port:
+   `labsim/vlan1-move-monitor.sh` (edit the three addresses for production).
+3. UniFi: on this firewall's trunk ports, Native VLAN → None, VLAN 1 → tagged.
+   This router's Management drops now.
+4. Over the console, in one commit:
+   ```
+   set interfaces bonding bond0 vif 1 address '192.168.1.252/24'   # .253 on vyos002
+   set interfaces bonding bond0 vif 1 description 'management'
+   delete interfaces bonding bond0 address
+   set firewall group interface-group LAN interface 'bond0.1'
+   delete firewall group interface-group LAN interface 'bond0'
+   set high-availability vrrp group native interface 'bond0.1'
+   commit
+   save
+   ```
+5. `sudo systemctl restart isc-kea-dhcp4-server` — see traps.
+6. Verify: Management SSH back, `show vrrp` shows `native` on `bond0.1`, and the
+   leak test passes.
+
+Then fail back if the VIPs moved (below), and repeat for the other router.
+
+### Measured windows (labsim)
+
+| | |
+|---|---|
+| this router's own Management unreachable | ~27 s (the console apply) |
+| VIP `.1` unreachable, peer already converted | **0 s** |
+| VIP `.1` unreachable, converting the current MASTER | ~6 s (VRRP failover) |
+| VIP unreachable if you convert both routers before the switch | **5 min 30 s** |
+
+That last row is the failure mode to avoid: with both routers untagged and the
+trunks already changed, the VIP is a black hole and **the healthy BACKUP does not
+take over**. Its `native` group stays BACKUP because the *other* VLANs still hear
+the master, and the sync group holds them together. Redundancy does not help you
+here; only ordering does.
+
+## Traps
+
+- **Restart kea.** VyOS does not restart it for an interface address change, so
+  it keeps a raw socket bound with the old address and keeps emitting the wrong
+  offers. The first post-fix test in the sim failed for this reason alone and
+  looked exactly like the fix not working.
+- **`interface-group LAN`.** Moving the address without moving the group means
+  Management falls outside the group, and with default-deny that is every
+  management session and all VLAN 1 inter-VLAN routing, gone on commit — on a
+  router you reach through itself. Use `commit-confirm` if you are not on console.
+- **The VIPs may move, and `no-preempt` keeps them moved.** Converting a router
+  restarts keepalived and re-initialises *every* group, not just `native`. In one
+  rehearsal the priority-100 secondary took all six VIPs and held them while the
+  priority-200 primary sat at BACKUP; in another the restart was quick enough that
+  nothing moved. It is non-deterministic — check afterwards, every time.
+  Fail back with `restart vrrp` **on the router currently holding them**.
+- **Duplicate delivery does not stop**, and should not be read as failure. #1117
+  only promises there is no longer a subnet on the parent to match. Expect two
+  identical replies per DISCOVER, both from the correct pool.
+- **Both firewalls' trunks must end up the same.** If UniFi shares one port
+  profile between them, changing it converts both at once and you get the 5m30s
+  row above. Check before you start; use per-port overrides if it does.
+
+## Not covered by the rehearsal
+
+- Whether UniFi's port profile can express "no native VLAN" the way OVS can, and
+  whether the two firewalls share a profile. Unverified — check on the controller.
+- Why the JetKVM consoles specifically accepted the wrong OFFER when a VLAN 3
+  access port should not receive an untagged VLAN 1 frame at all. Their port
+  profile likely passes VLAN 1 untagged. Worth confirming, though it does not
+  change the fix.

migration/RECOVERY-CARD-vlan1-move.md

@@ -0,0 +1,194 @@
+# Recovery card — moving Management to tagged VLAN 1
+
+Print or keep open. **During this change there is no internet, so no Claude.**
+Everything you need is on this page.
+
+---
+
+## The one thing that matters
+
+```
+ssh vyos@10.0.1.252
+```
+
+Your workstation is `10.0.0.210/23`; vyos001's LoT leg is `10.0.1.252/23`. Same
+subnet, same VLAN, **direct L2** — verified: `ip route get` returns
+`dev lanbr0 src 10.0.0.210` with no `via`, MAC `64:62:66:25:96:45`.
+
+It therefore does **not** depend on: the Management VLAN, VRRP, the VIPs,
+inter-VLAN routing, DNS, or the switch trunk config. If the router is up and its
+bond has link, this works. `bond0.10` is untouched by the change and stays in the
+firewall `LAN` group throughout.
+
+vyos002, once it is up, is `10.0.1.253` the same way.
+
+Other legs that also survive: `192.168.3.4` (kvm), `192.168.2.252` (Roomates).
+
+---
+
+## Before you touch anything
+
+```
+ssh vyos@10.0.1.252
+sudo /config/vyos-known-good save
+```
+
+The existing snapshot is from **2026-08-24** and predates today's fixes
+(eth2 removal, VRRP health-check) — restoring that one would undo them. Take a
+fresh one first. Check with `sudo /config/vyos-known-good status`.
+
+---
+
+## Order: switch FIRST, router SECOND
+
+This matters and is easy to get backwards.
+
+The UniFi controller is `192.168.1.5`, on the **Management** VLAN. Your
+workstation is on LoT and reaches it *through vyos001*. The moment the router
+has Management on `bond0.1` while the switch is still sending it untagged, that
+routing is dead — **and you lose the controller**, which is the thing you still
+need in order to change the switch.
+
+So:
+
+1. **UniFi first**, while everything still works:
+   USW Aggregation → port 1 `firewall001` (LAG, members 1+2) →
+   Native VLAN: Management → **None**, and make sure VLAN 1 is tagged/allowed.
+   *vyos001 loses Management the instant this lands. That is expected.*
+   Do **not** touch port 3 `firewall002` — that is vyos002, and it is down.
+2. **Router second**, over `ssh vyos@10.0.1.252` (still works — L2 direct).
+
+If UniFi will not offer "no native VLAN", stop and read *"If UniFi cannot do it"*
+below rather than improvising.
+
+---
+
+## The router change
+
+```
+ssh vyos@10.0.1.252
+configure
+set interfaces bonding bond0 vif 1 address '192.168.1.252/24'
+set interfaces bonding bond0 vif 1 description 'management'
+delete interfaces bonding bond0 address
+set firewall group interface-group LAN interface 'bond0.1'
+delete firewall group interface-group LAN interface 'bond0'
+set high-availability vrrp group native interface 'bond0.1'
+commit-confirm 10
+save
+exit
+```
+
+**Use `commit-confirm 10`, not `commit`.** If it goes wrong and you cannot get
+back in, the router reverts itself after 10 minutes and comes back on its own.
+That is your safety net with no internet and no help.
+
+Once you have confirmed it works (below), run:
+
+```
+configure
+confirm
+save
+exit
+```
+
+`save` after `confirm`, or a reboot loses it.
+
+### Then, and this is the step that gets forgotten
+
+```
+sudo systemctl restart isc-kea-dhcp4-server
+```
+
+VyOS does **not** restart kea for an interface address change. Without this it
+keeps a raw socket bound to the old address and keeps handing out wrong-VLAN
+addresses — the fix looks like it did nothing. Give it ~60s before judging;
+kea reopens sockets on a retry loop and answers nothing for a while after a
+restart (measured: still silent at 55s in the sim, then fine).
+
+Also check DNS came back, since the forwarder binds the VIP `192.168.1.1`:
+
+```
+sudo systemctl status pdns-recursor --no-pager | head -3
+dig @192.168.1.1 google.com +short
+```
+
+---
+
+## Verify
+
+```
+ssh vyos@192.168.1.252          # Management back, now tagged
+show vrrp                        # native should be on bond0.1
+show dhcp server leases | head
+```
+
+Then from a machine on VLAN 3, force a DHCP renew and confirm it gets a
+`192.168.3.x` address and not a `192.168.1.x` one.
+
+---
+
+## If you are locked out
+
+In order:
+
+1. **Wait 10 minutes.** `commit-confirm` reverts by itself. This is the answer
+   most of the time. Do not power-cycle during this — you will lose the revert.
+2. `ssh vyos@10.0.1.252` — the LoT leg. Then `configure` / `rollback 1` / `commit`.
+3. Other legs: `ssh vyos@192.168.3.4`, `ssh vyos@192.168.2.252`.
+4. `sudo /config/vyos-known-good restore` — back to the snapshot you took at the
+   start. It is itself commit-confirmed, so even this cannot strand you.
+5. Put the UniFi port back: Native VLAN → Management on USW Aggregation port 1.
+   That alone restores the old shape and Management comes back untagged.
+
+**Do not** power-cycle vyos001 as a first move. Everything above is faster and
+non-destructive, and a reboot loses an unsaved `commit-confirm` revert.
+
+---
+
+## Do NOT power on vyos002 yet
+
+It still has `interfaces ethernet eth2 address 192.168.8.144/23` on the box — the
+same subnet as `bond0.2`. That is what ARP-poisoned `192.168.8.1` and took the
+cluster down. It also has no `/config/vrrp-wan-health`, so it can take the
+floating IPs with no WAN.
+
+Its console (`kvm - vyos002`, US24 port 9) is currently **unreachable** — it sits
+on a VLAN 3 port holding a Management lease `192.168.1.28`, which is the very bug
+being fixed here. Fixing DHCP first is what gets that console back.
+
+---
+
+## If UniFi cannot do it
+
+Classic UniFi (this is a classic controller, 10.4.57) may not offer
+"Native VLAN = None" — every switch port has a PVID. Two things make this awkward
+here: Management is UniFi's *default* network with **no VLAN ID at all**
+(`vlan: null`), so there may be nothing to "tag VLAN 1" with.
+
+If so, **stop and change nothing.** The workaround is to point the trunk's native
+VLAN at a VLAN the router does not serve (so `bond0` still ends up with no
+subnet), which needs a throwaway VLAN-only network created first. That is a
+design decision, not something to improvise at 1am with no internet. Put the port
+back to Native = Management and everything returns to today's working state.
+
+---
+
+## Facts worth having on paper
+
+| | |
+|---|---|
+| vyos001 Management | `192.168.1.252` → becomes `bond0.1` |
+| vyos001 LoT (recovery) | `10.0.1.252`, L2-direct from your workstation |
+| vyos002 Management | `192.168.1.253` (down) |
+| VIP Management | `192.168.1.1` |
+| UniFi controller | `192.168.1.5` (on Management — you lose it mid-change) |
+| firewall001 trunk | USW Aggregation port 1, LAG members 1+2 |
+| firewall002 trunk | USW Aggregation port 3, LAG members 3+4 |
+| SSH user / pass | `vyos` / `vyos` |
+| vyos001 bond MAC | `64:62:66:25:96:45` |
+
+Measured in labsim: converting the router while the peer is already converted
+costs **0s** of VIP downtime; converting it while it holds the VIPs costs about
+**6s**. vyos002 is down, so vyos001 holds everything — expect the ~6s, and expect
+Management to be gone from the UniFi change until the router change lands.

migration/_unifi.py

@@ -0,0 +1,84 @@
+"""Shared UniFi API client for the migration tooling.
+
+The controller is a CLASSIC self-hosted UniFi Network app (server_version
+10.4.x), not UniFi OS: login is /api/login and data lives under
+/api/s/<site>/... . UniFi OS would use /api/auth/login + /proxy/network/api.
+Credentials come from the mcpctl server definition so they are not duplicated
+here.
+"""
+from __future__ import annotations
+
+import json, re, ssl, subprocess, urllib.request, http.cookiejar
+
+
+def client():
+    raw = subprocess.run(["mcpctl", "describe", "server", "unifi-network"],
+                         capture_output=True, text=True).stdout
+    m = re.search(r"UNIFI_TARGETS\s+(\[.*)", raw)
+    if not m:
+        raise SystemExit("could not read UNIFI_TARGETS from mcpctl")
+    blob = m.group(1).strip()
+    try:
+        targets = json.loads(blob)
+    except json.JSONDecodeError:
+        targets = json.loads(blob + "}" * (blob.count("{") - blob.count("}")))
+    t = targets[0]
+    base = t["base_url"].rstrip("/")
+    auth = t.get("auth", {})
+    site = t.get("default_site", "default")
+
+    ctx = ssl.create_default_context()
+    ctx.check_hostname = False
+    ctx.verify_mode = ssl.CERT_NONE
+    opener = urllib.request.build_opener(
+        urllib.request.HTTPCookieProcessor(http.cookiejar.CookieJar()),
+        urllib.request.HTTPSHandler(context=ctx))
+    req = urllib.request.Request(
+        f"{base}/api/login",
+        data=json.dumps({"username": auth.get("username"),
+                         "password": auth.get("password")}).encode(),
+        headers={"Content-Type": "application/json"})
+    opener.open(req, timeout=20).read()
+    return opener, base, site
+
+
+def get(opener, base, site, path):
+    """GET /api/s/<site>/<path>, returning the `data` list (never raising)."""
+    try:
+        body = opener.open(f"{base}/api/s/{site}/{path}", timeout=30).read()
+        return json.loads(body).get("data", [])
+    except Exception as exc:
+        return {"__error__": f"{type(exc).__name__}: {exc}"}
+
+
+def post(opener, base, site, path, payload):
+    """POST to /api/s/<site>/<path> -- used for device commands (cmd/devmgr)."""
+    req = urllib.request.Request(
+        f"{base}/api/s/{site}/{path}",
+        data=json.dumps(payload).encode(),
+        headers={"Content-Type": "application/json"}, method="POST")
+    try:
+        return json.loads(opener.open(req, timeout=30).read()).get("data", [])
+    except urllib.error.HTTPError as exc:
+        return {"__error__": f"HTTP {exc.code}: {exc.read()[:300].decode(errors='replace')}"}
+    except Exception as exc:
+        return {"__error__": f"{type(exc).__name__}: {exc}"}
+
+
+def put(opener, base, site, path, payload):
+    """PUT to /api/s/<site>/<path>. Returns the `data` list or an __error__ dict.
+
+    Classic controllers accept the session cookie alone -- no CSRF token, which
+    UniFi OS would require. Errors are returned rather than raised so a caller
+    changing production config can report and stop rather than traceback.
+    """
+    req = urllib.request.Request(
+        f"{base}/api/s/{site}/{path}",
+        data=json.dumps(payload).encode(),
+        headers={"Content-Type": "application/json"}, method="PUT")
+    try:
+        return json.loads(opener.open(req, timeout=30).read()).get("data", [])
+    except urllib.error.HTTPError as exc:
+        return {"__error__": f"HTTP {exc.code}: {exc.read()[:300].decode(errors='replace')}"}
+    except Exception as exc:
+        return {"__error__": f"{type(exc).__name__}: {exc}"}

migration/he-tunnel-follow

@@ -0,0 +1,121 @@
+#!/bin/bash
+# Keep the Hurricane Electric 6in4 tunnel pointed at whichever WAN is live.
+#
+# The tunnel is anchored to a source IPv4. When failover moves the default route
+# from the 10 gig to PPPoE, 6in4 packets keep leaving with the old source, HE
+# drops them, and IPv6 goes dark while IPv4 keeps working -- a partial outage
+# that presents as "some sites are broken", which is far worse to diagnose than
+# a clean one.
+#
+# Changes are made at KERNEL level (`ip tunnel change`), not in VyOS config, on
+# purpose:
+#   - no commit per WAN flip, so a flapping line cannot churn the config;
+#   - no drift against the Pulumi model, so `vyos-verify` stays meaningful;
+#   - a reboot restores config.boot, which pins the 10 gig -- the correct
+#     default -- so the wrong state cannot survive a restart.
+#
+#   he-tunnel-follow status     what is live vs what should be (read-only)
+#   he-tunnel-follow run        reconcile, updating HE if the source changed
+#   he-tunnel-follow run --dry  say what it would do, change nothing
+#
+# Credentials in /config/he-secrets (0600), NOT in git:
+#   HE_USER=<tunnelbroker username>
+#   HE_UPDATE_KEY=<from the tunnel's Advanced tab -- replaces the account password>
+#   HE_TUNNEL_ID=<numeric tunnel id>
+set -uo pipefail
+
+TUNNEL="${TUNNEL:-tun0}"
+SECRETS="${SECRETS:-/config/he-secrets}"
+STATE="${STATE:-/run/he-tunnel-follow.state}"
+# 6in4 costs 20 bytes. The 10 gig path is 1500 -> 1480; PPPoE is 1492 -> 1472.
+# Getting this wrong is the classic "IPv6 works until something large" failure.
+declare -A WAN_MTU=( ["bond0.53"]=1480 ["pppoe0"]=1472 )
+# Require the same answer twice before acting. HE rate-limits updates, and a
+# flapping WAN would otherwise hammer the API exactly when it is needed most.
+HYSTERESIS="${HYSTERESIS:-2}"
+
+log() { logger -t he-tunnel-follow -- "$*"; printf '  %s\n' "$*"; }
+die() { logger -t he-tunnel-follow -p user.err -- "$*"; printf '  ERROR: %s\n' "$*" >&2; exit 1; }
+
+active_wan() { ip -4 route show default 2>/dev/null | awk '/^default/{for(i=1;i<=NF;i++) if($i=="dev") print $(i+1); exit}'; }
+addr_of()    { ip -4 -br addr show "$1" 2>/dev/null | awk '{print $3}' | cut -d/ -f1; }
+tunnel_src() { ip tunnel show "$TUNNEL" 2>/dev/null | sed -nE 's/.* local ([0-9.]+).*/\1/p'; }
+tunnel_mtu() { cat "/sys/class/net/$TUNNEL/mtu" 2>/dev/null; }
+
+# HE's dyndns-style endpoint. `myip` is passed EXPLICITLY rather than letting HE
+# infer it from the request source: mid-failover the request itself may egress
+# either line, and inferring would happily point the tunnel at the WAN we just
+# left.
+he_update() {
+  local ip="$1"
+  [ -r "$SECRETS" ] || die "no $SECRETS -- create it with HE_USER / HE_UPDATE_KEY / HE_TUNNEL_ID (0600)"
+  # shellcheck disable=SC1090
+  . "$SECRETS"
+  [ -n "${HE_USER:-}" ] && [ -n "${HE_UPDATE_KEY:-}" ] && [ -n "${HE_TUNNEL_ID:-}" ] \
+    || die "$SECRETS is missing HE_USER, HE_UPDATE_KEY or HE_TUNNEL_ID"
+
+  local out
+  out="$(curl -sS --max-time 25 \
+      --data-urlencode "username=$HE_USER" \
+      --data-urlencode "password=$HE_UPDATE_KEY" \
+      --data-urlencode "hostname=$HE_TUNNEL_ID" \
+      --data-urlencode "myip=$ip" \
+      "https://ipv4.tunnelbroker.net/nic/update" 2>&1)"
+  # dyndns protocol: "good <ip>" or "nochg <ip>" are both success.
+  case "$out" in
+    good*|nochg*) log "HE endpoint set to $ip ($out)"; return 0 ;;
+    *)            die "HE update refused: $out" ;;
+  esac
+}
+
+reconcile() {
+  local dry="${1:-}"
+  local wan src want_mtu cur_src cur_mtu
+  wan="$(active_wan)"; [ -n "$wan" ] || die "no default route; refusing to guess"
+  src="$(addr_of "$wan")"; [ -n "$src" ] || die "no IPv4 address on $wan"
+  want_mtu="${WAN_MTU[$wan]:-}"
+  [ -n "$want_mtu" ] || die "unknown WAN '$wan' -- add it to WAN_MTU rather than guessing an MTU"
+  cur_src="$(tunnel_src)"; cur_mtu="$(tunnel_mtu)"
+
+  if [ "$cur_src" = "$src" ] && [ "$cur_mtu" = "$want_mtu" ]; then
+    rm -f "$STATE"
+    log "in sync: $TUNNEL via $wan src $src mtu $cur_mtu"
+    return 0
+  fi
+
+  # Hysteresis: count consecutive runs agreeing on the same target.
+  local seen=0 last=""
+  [ -r "$STATE" ] && { read -r last seen < "$STATE"; }
+  if [ "$last" = "$src" ]; then seen=$((seen + 1)); else seen=1; fi
+  echo "$src $seen" > "$STATE"
+  if [ "$seen" -lt "$HYSTERESIS" ]; then
+    log "change seen ($cur_src -> $src) but waiting for stability ($seen/$HYSTERESIS)"
+    return 0
+  fi
+
+  if [ "$dry" = "--dry" ]; then
+    log "DRY RUN: would set HE endpoint to $src, then $TUNNEL local $src mtu $want_mtu"
+    return 0
+  fi
+
+  # HE first, then local. Either order costs a brief drop, but changing locally
+  # first guarantees HE discards our packets for the whole window.
+  he_update "$src" || return 1
+  sudo ip tunnel change "$TUNNEL" mode sit local "$src" || die "failed to set tunnel local address"
+  sudo ip link set "$TUNNEL" mtu "$want_mtu"            || die "failed to set tunnel MTU"
+  rm -f "$STATE"
+  log "moved $TUNNEL to $wan: src $cur_src -> $src, mtu $cur_mtu -> $want_mtu"
+}
+
+case "${1:-status}" in
+  status)
+    wan="$(active_wan)"
+    printf '  active WAN : %s\n' "${wan:-<none>}"
+    printf '  wan addr   : %s\n' "$(addr_of "${wan:-lo}")"
+    printf '  tunnel src : %s\n' "$(tunnel_src)"
+    printf '  tunnel mtu : %s (want %s)\n' "$(tunnel_mtu)" "${WAN_MTU[${wan:-}]:-?}"
+    [ -r "$SECRETS" ] && printf '  credentials: present\n' || printf '  credentials: MISSING (%s)\n' "$SECRETS"
+    ;;
+  run) reconcile "${2:-}" ;;
+  *)   die "usage: he-tunnel-follow {status|run [--dry]}" ;;
+esac

migration/unifi-export.py

@@ -0,0 +1,335 @@
+#!/usr/bin/env python3
+"""Export everything from the UniFi controller that the VyOS cutover must preserve.
+
+The point is that nothing quietly stops working after the switch. That means
+capturing not just the networks but every DHCP reservation, every port forward
+and every firewall rule — the things nobody remembers configuring until they
+break.
+
+Writes one JSON file per endpoint into ./export/ (raw, unmodified — the source
+of truth) plus inventory.json, a normalised view used by the VyOS generator.
+
+  ./unifi-export.py                 # export to ./export/
+  ./unifi-export.py --out /tmp/x    # elsewhere
+  ./unifi-export.py --summary       # print a human summary of what was found
+
+WARNING: the raw export contains secrets (wlanconf holds WiFi passphrases,
+setting holds RADIUS/auth material). ./export/ is gitignored — keep it that way.
+"""
+from __future__ import annotations
+
+import argparse
+import ipaddress
+import json
+import os
+import sys
+
+import _unifi
+
+# endpoint -> why it matters for the cutover
+ENDPOINTS = {
+    "rest/networkconf": "networks: VLANs, subnets, DHCP ranges, DNS, lease time",
+    "rest/user":        "known clients — this is where fixed DHCP reservations live",
+    "stat/sta":         "currently active clients and their live IPs",
+    "rest/firewallrule": "firewall rules",
+    "rest/firewallgroup": "address/port groups referenced by rules",
+    "rest/portforward": "port forwards (inbound NAT)",
+    "rest/routing":     "static routes",
+    "rest/dhcpoption":  "custom DHCP options",
+    "rest/wlanconf":    "wireless networks (VLAN bindings)",
+    "stat/device":      "switches/APs incl. per-port VLAN config",
+    "rest/setting":     "controller settings (incl. USG/gateway config)",
+    "rest/usergroup":   "bandwidth groups referenced by clients",
+    "rest/dynamicdns":  "dynamic DNS",
+}
+
+
+def build_inventory(raw: dict) -> dict:
+    """Normalise the parts a migration actually has to reproduce."""
+    nets_by_id = {n["_id"]: n for n in raw.get("rest/networkconf", []) if isinstance(n, dict)}
+
+    networks = []
+    for n in raw.get("rest/networkconf", []):
+        if not isinstance(n, dict):
+            continue
+        networks.append({
+            "id": n.get("_id"),
+            "name": n.get("name"),
+            "purpose": n.get("purpose"),
+            "vlan": n.get("vlan"),
+            "vlan_enabled": n.get("vlan_enabled"),
+            "subnet": n.get("ip_subnet"),
+            "domain_name": n.get("domain_name"),
+            "dhcp_enabled": n.get("dhcpd_enabled"),
+            "dhcp_start": n.get("dhcpd_start"),
+            "dhcp_stop": n.get("dhcpd_stop"),
+            "dhcp_lease": n.get("dhcpd_leasetime"),
+            "dhcp_dns": [n.get(f"dhcpd_dns_{i}") for i in (1, 2, 3, 4) if n.get(f"dhcpd_dns_{i}")],
+            "dhcp_gateway": n.get("dhcpd_gateway") or n.get("dhcpd_gateway_enabled"),
+            "dhcp_ntp": [n.get(f"dhcpd_ntp_{i}") for i in (1, 2) if n.get(f"dhcpd_ntp_{i}")],
+            "igmp_snooping": n.get("igmp_snooping"),
+            "enabled": n.get("enabled", True),
+        })
+
+    # ip_subnet is the GATEWAY address with a prefix ("10.0.0.1/23"), not the
+    # network address — so derive the real subnet before matching against it.
+    subnets = []
+    for n in networks:
+        if not n["subnet"]:
+            continue
+        try:
+            iface = ipaddress.ip_interface(n["subnet"])
+        except ValueError:
+            continue
+        subnets.append((iface.network, n))
+
+    def resolve_net(ip: str | None, network_id: str | None) -> dict:
+        """Which network does this reservation belong to?
+
+        Most reservations here (23 of 31 as of the first export) carry no
+        network_id at all — UniFi simply does not bind them. VyOS needs the
+        subnet to place a static-mapping, so fall back to containment.
+        """
+        if network_id and network_id in nets_by_id:
+            nid = nets_by_id[network_id]
+            return {"name": nid.get("name"), "vlan": nid.get("vlan"), "by": "network_id"}
+        if ip:
+            try:
+                addr = ipaddress.ip_address(ip)
+            except ValueError:
+                return {"name": None, "vlan": None, "by": "unresolved"}
+            for net, meta in subnets:
+                if addr in net:
+                    return {"name": meta["name"], "vlan": meta["vlan"], "by": "subnet"}
+        return {"name": None, "vlan": None, "by": "unresolved"}
+
+    # Fixed reservations: the single most important thing to carry over, and
+    # the easiest to lose — nobody has these written down anywhere else.
+    reservations = []
+    for u in raw.get("rest/user", []):
+        if not isinstance(u, dict) or not u.get("use_fixedip"):
+            continue
+        net = resolve_net(u.get("fixed_ip"), u.get("network_id"))
+        reservations.append({
+            "mac": (u.get("mac") or "").lower(),
+            "ip": u.get("fixed_ip"),
+            "name": u.get("name") or u.get("hostname") or "",
+            "hostname": u.get("hostname") or "",
+            "network_id": u.get("network_id"),
+            "network_name": net["name"],
+            "network_vlan": net["vlan"],
+            "resolved_by": net["by"],
+            "note": (u.get("note") or "").strip(),
+        })
+    reservations.sort(key=lambda r: tuple(int(p) for p in r["ip"].split(".")) if r["ip"] else (0,))
+
+    # Active leases without a reservation: these devices work today by luck of
+    # the lease database. After a DHCP server swap they get a NEW address.
+    reserved_macs = {r["mac"] for r in reservations}
+    dynamic = []
+    for c in raw.get("stat/sta", []):
+        if not isinstance(c, dict):
+            continue
+        mac = (c.get("mac") or "").lower()
+        if mac in reserved_macs or not c.get("ip"):
+            continue
+        dynamic.append({
+            "mac": mac,
+            "ip": c.get("ip"),
+            "name": c.get("name") or c.get("hostname") or "",
+            "network": c.get("network"),
+        })
+    dynamic.sort(key=lambda r: tuple(int(p) for p in r["ip"].split(".")) if r["ip"] else (0,))
+
+    port_forwards = [{
+        "name": p.get("name"), "enabled": p.get("enabled"),
+        "proto": p.get("proto"), "src": p.get("src"),
+        "dst_port": p.get("dst_port"), "fwd": p.get("fwd"),
+        "fwd_port": p.get("fwd_port"), "log": p.get("log"),
+    } for p in raw.get("rest/portforward", []) if isinstance(p, dict)]
+
+    firewall_rules = [{
+        "name": r.get("name"), "enabled": r.get("enabled"), "action": r.get("action"),
+        "ruleset": r.get("ruleset"), "rule_index": r.get("rule_index"),
+        "protocol": r.get("protocol"),
+        "src_address": r.get("src_address"), "dst_address": r.get("dst_address"),
+        "src_firewallgroup_ids": r.get("src_firewallgroup_ids"),
+        "dst_firewallgroup_ids": r.get("dst_firewallgroup_ids"),
+        "src_networkconf_id": r.get("src_networkconf_id"),
+        "dst_networkconf_id": r.get("dst_networkconf_id"),
+    } for r in raw.get("rest/firewallrule", []) if isinstance(r, dict)]
+
+    firewall_groups = [{
+        "id": g.get("_id"), "name": g.get("name"),
+        "type": g.get("group_type"), "members": g.get("group_members"),
+    } for g in raw.get("rest/firewallgroup", []) if isinstance(g, dict)]
+
+    static_routes = [{
+        "name": r.get("name"), "enabled": r.get("enabled"),
+        "network": r.get("static-route_network"),
+        "nexthop": r.get("static-route_nexthop"),
+        "distance": r.get("static-route_distance"),
+        "type": r.get("static-route_type"),
+    } for r in raw.get("rest/routing", []) if isinstance(r, dict)]
+
+    return {
+        "networks": networks,
+        "reservations": reservations,
+        "dynamic_clients": dynamic,
+        "port_forwards": port_forwards,
+        "firewall_rules": firewall_rules,
+        "firewall_groups": firewall_groups,
+        "static_routes": static_routes,
+        "warnings": find_warnings(networks, reservations),
+    }
+
+
+def find_warnings(networks: list, reservations: list) -> list:
+    """Things that are fine under UniFi but bite when rebuilt on VyOS."""
+    warns = []
+
+    for n in networks:
+        if not n["subnet"]:
+            continue
+        try:
+            iface = ipaddress.ip_interface(n["subnet"])
+        except ValueError:
+            warns.append({"kind": "bad_subnet", "network": n["name"], "detail": n["subnet"]})
+            continue
+        # UniFi stores the gateway in ip_subnet. A gateway equal to the network
+        # address is legal in a /23 but plenty of tooling rejects it, so it must
+        # not be discovered during the cutover window.
+        if iface.ip == iface.network.network_address:
+            warns.append({
+                "kind": "gateway_is_network_address", "network": n["name"],
+                "detail": f"gateway {iface.ip} is the network address of {iface.network}",
+            })
+
+    # Reservations that sit inside the dynamic pool. UniFi's dhcpd tolerates
+    # this; whether VyOS does depends on its DHCP backend, so every one of these
+    # is a config that must be proven on the sim before cutover.
+    ranges = []
+    for n in networks:
+        if n["dhcp_enabled"] and n["dhcp_start"] and n["dhcp_stop"]:
+            try:
+                ranges.append((n["name"], ipaddress.ip_address(n["dhcp_start"]),
+                               ipaddress.ip_address(n["dhcp_stop"])))
+            except ValueError:
+                pass
+    inside = []
+    for r in reservations:
+        if not r["ip"]:
+            continue
+        try:
+            addr = ipaddress.ip_address(r["ip"])
+        except ValueError:
+            continue
+        for name, lo, hi in ranges:
+            if lo <= addr <= hi:
+                inside.append(f"{r['ip']} ({r['name'] or r['mac']}) in {name} pool")
+                break
+    if inside:
+        warns.append({"kind": "reservation_inside_dhcp_pool",
+                      "count": len(inside), "detail": inside})
+
+    unresolved = [f"{r['ip']} {r['mac']} {r['name']}"
+                  for r in reservations if r["resolved_by"] == "unresolved"]
+    if unresolved:
+        warns.append({"kind": "reservation_matches_no_subnet",
+                      "count": len(unresolved), "detail": unresolved})
+    return warns
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--out", default=os.path.join(os.path.dirname(os.path.abspath(__file__)), "export"))
+    ap.add_argument("--summary", action="store_true")
+    args = ap.parse_args()
+
+    os.makedirs(args.out, exist_ok=True)
+    opener, base, site = _unifi.client()
+    print(f"controller {base}  site {site}")
+
+    raw: dict = {}
+    errors = []
+    for path, why in ENDPOINTS.items():
+        data = _unifi.get(opener, base, site, path)
+        if isinstance(data, dict) and "__error__" in data:
+            errors.append((path, data["__error__"]))
+            print(f"  {path:22} FAILED  {data['__error__'][:50]}")
+            continue
+        raw[path] = data
+        fname = path.replace("/", "_") + ".json"
+        with open(os.path.join(args.out, fname), "w") as fh:
+            json.dump(data, fh, indent=2, sort_keys=True)
+        print(f"  {path:22} {len(data):4d}  -> {fname}")
+
+    inventory = build_inventory(raw)
+    with open(os.path.join(args.out, "inventory.json"), "w") as fh:
+        json.dump(inventory, fh, indent=2, sort_keys=True)
+
+    print(f"\nwrote {args.out}/inventory.json")
+    print(f"  networks           {len(inventory['networks'])}")
+    print(f"  DHCP reservations  {len(inventory['reservations'])}")
+    print(f"  dynamic clients    {len(inventory['dynamic_clients'])}  (no reservation — see summary)")
+    print(f"  port forwards      {len(inventory['port_forwards'])}")
+    print(f"  firewall rules     {len(inventory['firewall_rules'])}")
+    print(f"  static routes      {len(inventory['static_routes'])}")
+    if errors:
+        print(f"  endpoints failed   {len(errors)}: {[e[0] for e in errors]}")
+
+    if args.summary:
+        print_summary(inventory)
+    return 0
+
+
+def print_summary(inv: dict) -> None:
+    print("\n=== networks ===")
+    print(f"{'name':22} {'vlan':>5}  {'subnet':20} {'dhcp range':32} lease")
+    for n in sorted(inv["networks"], key=lambda x: (x["vlan"] or 0)):
+        rng = f"{n['dhcp_start']} - {n['dhcp_stop']}" if n["dhcp_enabled"] else "(dhcp off)"
+        print(f"{(n['name'] or '')[:22]:22} {str(n['vlan'] or '-'):>5}  "
+              f"{(n['subnet'] or '-'):20} {rng:32} {n['dhcp_lease'] or '-'}")
+
+    print(f"\n=== DHCP reservations ({len(inv['reservations'])}) ===")
+    for r in inv["reservations"]:
+        print(f"  {r['ip']:16} {r['mac']:18} {(r['network_name'] or '?')[:14]:14} {r['name'][:30]}")
+
+    if inv["port_forwards"]:
+        print(f"\n=== port forwards ({len(inv['port_forwards'])}) ===")
+        for p in inv["port_forwards"]:
+            state = "" if p["enabled"] else "  [DISABLED]"
+            print(f"  {p['proto']:6} {str(p['src']):16}:{str(p['dst_port']):11} -> "
+                  f"{p['fwd']}:{p['fwd_port']}   {p['name']}{state}")
+
+    if inv["firewall_rules"]:
+        print(f"\n=== firewall rules ({len(inv['firewall_rules'])}) ===")
+        for r in inv["firewall_rules"]:
+            state = "" if r["enabled"] else "  [DISABLED]"
+            print(f"  {str(r['ruleset']):22} {str(r['action']):8} {r['name']}{state}")
+
+    if inv["warnings"]:
+        print(f"\n=== {len(inv['warnings'])} things to settle before cutover ===")
+        for w in inv["warnings"]:
+            n = w.get("count")
+            print(f"  [{w['kind']}]" + (f"  x{n}" if n else ""))
+            det = w["detail"]
+            for line in (det if isinstance(det, list) else [det])[:6]:
+                print(f"      {line}")
+            if isinstance(det, list) and len(det) > 6:
+                print(f"      ... and {len(det) - 6} more (see inventory.json)")
+
+    n_dyn = len(inv["dynamic_clients"])
+    if n_dyn:
+        print(f"\n=== {n_dyn} active clients WITHOUT a reservation ===")
+        print("  These hold their address only via the current lease database. A DHCP")
+        print("  server swap hands them a different one — fine for phones, not fine for")
+        print("  anything another host reaches by IP. Review before cutover:")
+        for c in inv["dynamic_clients"][:40]:
+            print(f"  {c['ip']:16} {c['mac']:18} {(c['network'] or '')[:14]:14} {c['name'][:30]}")
+        if n_dyn > 40:
+            print(f"  ... and {n_dyn - 40} more (see inventory.json)")
+
+
+if __name__ == "__main__":
+    sys.exit(main())

migration/unifi-reserve-all.py

@@ -0,0 +1,236 @@
+#!/usr/bin/env python3
+"""Reserve every active client at the address it already has.
+
+Why this exists: kea does not inherit UniFi's lease database. At cutover it
+starts with an empty view of who holds what, so it can hand an address that is
+currently in use to a different device. Reservations are what carry "this
+device has this address" across the switch, because they live in config rather
+than in lease state.
+
+Dry run by default -- this writes to the live controller, and 40-odd writes is
+not something to trigger by accident.
+
+  ./unifi-reserve-all.py                  # show the plan, change nothing
+  ./unifi-reserve-all.py --apply          # write them
+  ./unifi-reserve-all.py --skip-random    # omit randomised/private MACs
+
+Only clients on networks that actually run DHCP are considered, which
+automatically excludes WAN transit VLANs where a reservation is meaningless.
+Anything already reserved is left alone, and an address already reserved to a
+different MAC is reported and skipped rather than stolen.
+"""
+from __future__ import annotations
+
+import argparse
+import ipaddress
+import sys
+
+import _unifi
+
+# The VRRP virtual addresses, read from `show configuration commands` on
+# vyos001. UniFi sees these as ordinary client addresses because the firewalls'
+# bond MACs answer for them, and their reported IP flips between the real
+# interface address and the VIP. Reserving one would put a DHCP reservation on
+# the gateway address itself.
+VIPS = {
+    "192.168.1.254", "192.168.9.254", "192.168.3.254",
+    "10.0.9.254", "10.0.1.254", "192.168.2.254",
+}
+
+# Every MAC the two firewalls own (bond0/eth0/eth1 share one, eth2 and eth3
+# have their own). These interfaces are statically configured routers, not DHCP
+# clients -- except eth2, which is deliberately reserved and already handled.
+ROUTER_MACS = {
+    "64:62:66:25:96:45", "64:62:66:25:96:46", "64:62:66:25:96:48",  # vyos001
+    "64:62:66:25:96:51", "64:62:66:25:96:52", "64:62:66:25:96:54",  # vyos002
+}
+
+
+def is_random_mac(mac: str) -> bool:
+    """Locally-administered bit set => a privacy/randomised MAC.
+
+    Worth calling out: such a device re-randomises periodically, so the
+    reservation stops matching it and becomes dead config. Harmless, but it
+    will never do what it looks like it does.
+    """
+    try:
+        return bool(int(mac.split(":")[0], 16) & 0x02)
+    except (ValueError, IndexError):
+        return False
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--apply", action="store_true", help="actually write (default: dry run)")
+    ap.add_argument("--skip-random", action="store_true", help="omit randomised MACs")
+    ap.add_argument("--plan", default="reservation-plan.tsv",
+                    help="dry run WRITES this file; --apply READS it and applies "
+                         "exactly what it contains")
+    args = ap.parse_args()
+
+    opener, base, site = _unifi.client()
+    users = _unifi.get(opener, base, site, "rest/user")
+    nets = _unifi.get(opener, base, site, "rest/networkconf")
+    sta = _unifi.get(opener, base, site, "stat/sta")
+    for blob in (users, nets, sta):
+        if isinstance(blob, dict):
+            print(f"error reading controller: {blob['__error__']}", file=sys.stderr)
+            return 1
+
+    # Only networks that serve DHCP: a reservation on a WAN transit VLAN means
+    # nothing, and those are exactly the ones without dhcpd_enabled.
+    serving = []
+    for n in nets:
+        if not (n.get("dhcpd_enabled") and n.get("ip_subnet")):
+            continue
+        try:
+            serving.append((ipaddress.ip_interface(n["ip_subnet"]).network, n))
+        except ValueError:
+            continue
+
+    by_mac = {(u.get("mac") or "").lower(): u for u in users}
+    taken = {u.get("fixed_ip"): (u.get("mac") or "").lower()
+             for u in users if u.get("use_fixedip")}
+
+    plan, skipped = [], []
+    for c in sta:
+        mac, ip = (c.get("mac") or "").lower(), c.get("ip")
+        if not mac or not ip:
+            continue
+        label = c.get("name") or c.get("hostname") or "?"
+        user = by_mac.get(mac)
+
+        if ip in VIPS:
+            skipped.append((ip, mac, label, "VRRP virtual address - not a client"))
+            continue
+        if mac in ROUTER_MACS:
+            skipped.append((ip, mac, label, "firewall's own interface - statically configured"))
+            continue
+
+        net = next((n for netw, n in serving if ipaddress.ip_address(ip) in netw), None)
+        if net is None:
+            skipped.append((ip, mac, label, "not on a DHCP-serving network"))
+            continue
+        # The gateway is the router, not a lease.
+        if ip == str(ipaddress.ip_interface(net["ip_subnet"]).ip):
+            skipped.append((ip, mac, label, "network gateway address"))
+            continue
+        if user is None:
+            skipped.append((ip, mac, label, "not a known client on the controller"))
+            continue
+        if user.get("use_fixedip"):
+            if user.get("fixed_ip") != ip:
+                skipped.append((ip, mac, label,
+                                f"already reserved at {user.get('fixed_ip')} - left alone"))
+            continue
+        if ip in taken and taken[ip] != mac:
+            skipped.append((ip, mac, label,
+                            f"address already reserved to {taken[ip]}"))
+            continue
+        if args.skip_random and is_random_mac(mac):
+            skipped.append((ip, mac, label, "randomised MAC (--skip-random)"))
+            continue
+        plan.append((ip, mac, label, user, net))
+
+    # Generic safety net: if two MACs report the same current address, at most
+    # one of them can legitimately keep it and we cannot tell which. Drop both
+    # and say so -- this is exactly how the VRRP VIPs first showed up.
+    counts: dict[str, int] = {}
+    for ip, *_ in plan:
+        counts[ip] = counts.get(ip, 0) + 1
+    contested = {ip for ip, n in counts.items() if n > 1}
+    if contested:
+        for ip, mac, label, _u, _n in [p for p in plan if p[0] in contested]:
+            skipped.append((ip, mac, label, "address claimed by more than one MAC"))
+        plan = [p for p in plan if p[0] not in contested]
+
+    plan.sort(key=lambda r: ipaddress.ip_address(r[0]))
+
+    print(f"=== plan: {len(plan)} new reservation(s) ===")
+    for ip, mac, label, _u, net in plan:
+        flag = "  [randomised MAC]" if is_random_mac(mac) else ""
+        print(f"  {ip:16} {mac:18} {label[:28]:28} {net.get('name')}{flag}")
+    if skipped:
+        print(f"\n=== skipped ({len(skipped)}) ===")
+        for ip, mac, label, why in sorted(skipped):
+            print(f"  {ip:16} {mac:18} {label[:24]:24} {why}")
+
+    n_rand = sum(1 for p in plan if is_random_mac(p[1]))
+    if n_rand:
+        print(f"\nnote: {n_rand} of these use randomised MACs. The reservation "
+              f"stops matching once the device re-randomises.")
+
+    if not args.apply:
+        with open(args.plan, "w") as fh:
+            for ip, mac, label, _u, _n in plan:
+                fh.write(f"{mac}\t{ip}\t{label}\n")
+        print(f"\ndry run -- nothing written to the controller.")
+        print(f"plan saved to {args.plan}; re-run with --apply to apply exactly that.")
+        return 0
+
+    # Apply the plan that was REVIEWED, not one recomputed now.
+    #
+    # This cost a k8s node an outage. The apply used to re-read stat/sta, and a
+    # client that renewed between the dry run and the apply got pinned to
+    # whatever transient address it happened to hold at that instant -- worker1
+    # was reviewed at .13 and written as .242. A plan you looked at and a plan
+    # that gets applied must be the same object.
+    try:
+        with open(args.plan) as fh:
+            reviewed = {}
+            for line in fh:
+                parts = line.rstrip("\n").split("\t")
+                if len(parts) >= 2:
+                    reviewed[parts[0].lower()] = parts[1]
+    except OSError:
+        print(f"no plan at {args.plan}. Run without --apply first and review it.",
+              file=sys.stderr)
+        return 1
+
+    drifted = [(ip, mac) for ip, mac, _l, _u, _n in plan
+               if mac in reviewed and reviewed[mac] != ip]
+    for ip, mac in drifted:
+        print(f"  note: {mac} now reports {ip}, plan says {reviewed[mac]} -- "
+              f"applying the plan", file=sys.stderr)
+
+    plan = [(reviewed[mac], mac, label, user, net)
+            for ip, mac, label, user, net in plan if mac in reviewed]
+    print(f"applying {len(plan)} reservation(s) from {args.plan}")
+
+    print()
+    ok = fail = 0
+    for ip, mac, label, user, net in plan:
+        res = _unifi.put(opener, base, site, f"rest/user/{user['_id']}",
+                         {"use_fixedip": True, "fixed_ip": ip, "network_id": net["_id"]})
+        if isinstance(res, dict):
+            print(f"  FAILED {ip:16} {mac}  {res['__error__'][:70]}")
+            fail += 1
+        else:
+            ok += 1
+
+    # Read back rather than trusting the write responses.
+    after = _unifi.get(opener, base, site, "rest/user")
+    live = {(u.get("mac") or "").lower() for u in after if u.get("use_fixedip")}
+    verified = sum(1 for _ip, mac, _l, _u, _n in plan if mac in live)
+    print(f"\nwrote {ok}, failed {fail}, verified live {verified}/{len(plan)}")
+    print(f"total reservations on the controller now: "
+          f"{sum(1 for u in after if u.get('use_fixedip'))}")
+
+    # Writing the controller is only half the job. The gateway applies config
+    # on its own schedule, and a device running config from before these
+    # changes will hand out addresses that disagree with what the controller
+    # shows -- which is how a k8s node ended up unable to get any lease at all
+    # while the controller looked perfectly correct.
+    print("\nThe controller now disagrees with what the gateway is running.")
+    print("Push it to the device and wait for state to return to 'connected':")
+    print("  python3 -c \"import _unifi; o,b,s=_unifi.client(); "
+          "print(_unifi.post(o,b,s,'cmd/devmgr',"
+          "{'cmd':'force-provision','mac':'<gateway-mac>'}))\"")
+    print("Then verify a real DISCOVER is answered before trusting it:")
+    print("  ssh vyos@<fw> 'sudo nmap --script broadcast-dhcp-discover -e eth2 "
+          "--script-args broadcast-dhcp-discover.mac=<client-mac>'")
+    return 0 if fail == 0 and verified == len(plan) else 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())

migration/unifi-reserve.py

@@ -0,0 +1,106 @@
+#!/usr/bin/env python3
+"""Set a fixed-IP reservation in UniFi, so it cannot lease that address away.
+
+Written for the firewalls' management NICs: their addresses are pinned static
+on the VyOS side, but UniFi still owns the pool they sit in and would happily
+hand the same address to something else. A reservation closes that gap while
+the USG is still the DHCP server, and it keeps the management address identical
+in both cutover modes.
+
+  ./unifi-reserve.py 64:62:66:25:96:47 192.168.8.143
+  ./unifi-reserve.py --dry-run <mac> <ip>
+
+Idempotent: an existing, matching reservation is reported and left alone. This
+writes to the live controller, so it verifies by reading the record back rather
+than trusting the response.
+"""
+from __future__ import annotations
+
+import argparse
+import ipaddress
+import sys
+
+import _unifi
+
+
+def find_network(nets: list, ip: str) -> dict | None:
+    """Which configured network contains this address?
+
+    ip_subnet holds the gateway address with a prefix ("192.168.8.1/23"), so
+    the network has to be derived from it rather than compared directly.
+    """
+    addr = ipaddress.ip_address(ip)
+    for n in nets:
+        raw = n.get("ip_subnet")
+        if not raw:
+            continue
+        try:
+            if addr in ipaddress.ip_interface(raw).network:
+                return n
+        except ValueError:
+            continue
+    return None
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("mac")
+    ap.add_argument("ip")
+    ap.add_argument("--dry-run", action="store_true")
+    args = ap.parse_args()
+
+    mac = args.mac.lower().replace("-", ":")
+    ipaddress.ip_address(args.ip)  # fail early on a typo
+
+    opener, base, site = _unifi.client()
+    users = _unifi.get(opener, base, site, "rest/user")
+    nets = _unifi.get(opener, base, site, "rest/networkconf")
+    for blob in (users, nets):
+        if isinstance(blob, dict):
+            print(f"error reading controller: {blob['__error__']}", file=sys.stderr)
+            return 1
+
+    user = next((u for u in users if (u.get("mac") or "").lower() == mac), None)
+    if user is None:
+        print(f"{mac} is not a known client -- connect it once, or create it "
+              f"in the UI first", file=sys.stderr)
+        return 1
+
+    net = find_network(nets, args.ip)
+    if net is None:
+        print(f"no configured network contains {args.ip}", file=sys.stderr)
+        return 1
+
+    label = user.get("name") or user.get("hostname") or mac
+    if user.get("use_fixedip") and user.get("fixed_ip") == args.ip:
+        print(f"{label} ({mac}) already reserved at {args.ip} -- nothing to do")
+        return 0
+    if user.get("use_fixedip"):
+        print(f"WARNING: {label} currently reserved at {user.get('fixed_ip')}, "
+              f"changing to {args.ip}", file=sys.stderr)
+
+    print(f"{label} ({mac}) -> {args.ip} on '{net.get('name')}' (VLAN {net.get('vlan') or 'native'})")
+    if args.dry_run:
+        print("  --dry-run: not writing")
+        return 0
+
+    payload = {"use_fixedip": True, "fixed_ip": args.ip, "network_id": net["_id"]}
+    res = _unifi.put(opener, base, site, f"rest/user/{user['_id']}", payload)
+    if isinstance(res, dict):
+        print(f"  write failed: {res['__error__']}", file=sys.stderr)
+        return 1
+
+    # Read it back: the controller accepting a PUT is not proof it stored what
+    # we asked for.
+    after = _unifi.get(opener, base, site, "rest/user")
+    check = next((u for u in after if (u.get("mac") or "").lower() == mac), {})
+    if check.get("use_fixedip") and check.get("fixed_ip") == args.ip:
+        print(f"  verified: reservation is live")
+        return 0
+    print(f"  VERIFY FAILED: controller reports use_fixedip="
+          f"{check.get('use_fixedip')} fixed_ip={check.get('fixed_ip')}", file=sys.stderr)
+    return 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())

migration/unifi-to-vyos.py

@@ -0,0 +1,251 @@
+#!/usr/bin/env python3
+"""Turn the UniFi export into the VyOS config that replaces it.
+
+Scope is deliberately narrow: DHCP and DNS. Those are the services the USG owns
+that VyOS must reproduce byte-for-byte in behaviour, because getting them wrong
+means clients lose their addresses or their name resolution. Everything else
+either stays on UniFi (wireless), has no VyOS equivalent (user groups), or is
+hand-written because it is not in the export (WAN, NAT, VRRP).
+
+This is not a general UniFi-to-VyOS converter and should not grow into one.
+
+  ./unifi-to-vyos.py --mode prod          # the cutover artifact
+  ./unifi-to-vyos.py --mode sim           # same MACs, labsim addresses
+  ./unifi-to-vyos.py --mode prod --check  # counts only, no output
+
+Both modes come from one code path on purpose: the config proven in labsim and
+the config applied to the firewalls must not be able to drift apart.
+
+DNS note: UniFi hands out the gateway's own IP as resolver whenever a network
+has no explicit dhcpd_dns -- true for 5 of the 6 VLANs, verified by labmaster
+resolving against 192.168.8.1. So VyOS must run `service dns forwarding` or
+those VLANs lose DNS entirely at cutover. LoT's explicit 10.0.0.194 is preserved
+as-is.
+"""
+from __future__ import annotations
+
+import argparse
+import ipaddress
+import json
+import os
+import re
+import sys
+
+# Upstream resolvers for VyOS's own forwarder -- the same pair the USG used on
+# its WAN (wan_dns1/wan_dns2). The NAS at 10.0.0.194 is deliberately NOT here:
+# it is legacy for ad.itaz.eu, and those records now live in Cloudflare, so the
+# zone resolves publicly (verified: nas001.ad.itaz.eu and kvm-macstudio1 both
+# answer from 8.8.8.8). That means no conditional forward is needed and the NAS
+# is out of the DNS path entirely.
+UPSTREAM_DNS = ["8.8.8.8", "8.8.4.4"]
+
+# labsim equivalents, keyed by VLAN id. Only VLAN 10 needs a /23: every one of
+# the 31 reservations is in LoT, which spans 10.0.0.x and 10.0.1.x, and a /24
+# cannot represent that. k8s and Private are also /23 in production but hold no
+# reservations, so they keep their existing /24 and their DHCP range is clamped
+# (reported at generation time -- never silently).
+SIM_SUBNETS = {
+    1:   "172.31.1.0/24",
+    2:   "172.31.2.0/24",
+    3:   "172.31.3.0/24",
+    9:   "172.31.9.0/24",
+    10:  "172.31.10.0/23",
+    200: "172.31.200.0/24",
+}
+
+
+def vlan_of(net: dict) -> int:
+    """VLAN id, treating the untagged Management network as 1.
+
+    UniFi stores vlan=None for the native network; the VyOS side already uses
+    vrid 1 for it (high-availability group 'native'), so 1 is the consistent id.
+    """
+    return int(net["vlan"]) if net.get("vlan") else 1
+
+
+def sanitize(name: str, fallback: str) -> str:
+    """Reduce a UniFi client name to something VyOS will accept as a node name.
+
+    VyOS validates static-mapping names as *hostnames*, so underscores are
+    rejected outright -- verified: `Dongle-M_C0D4` fails with "Invalid static
+    mapping hostname". Letters, digits and hyphens only, no leading digit or
+    hyphen, no trailing hyphen.
+    """
+    cleaned = re.sub(r"[^A-Za-z0-9-]", "-", (name or "").strip())
+    cleaned = re.sub(r"-{2,}", "-", cleaned).strip("-")
+    if cleaned and cleaned[0].isdigit():
+        cleaned = "h" + cleaned
+    return cleaned or fallback
+
+
+class Mapper:
+    """Translates production addresses into the target mode's address space.
+
+    In prod mode this is the identity. In sim mode an address is mapped by its
+    offset from the network address, so the host part is preserved: 10.0.0.46
+    -> 172.31.10.46 and 10.0.1.67 -> 172.31.11.67. That is what makes the sim
+    test meaningful -- the MAC is identical and the host octet is recognisable.
+    """
+
+    def __init__(self, mode: str, networks: list) -> None:
+        self.mode = mode
+        self.clamped: list[str] = []
+        self.map: dict[int, tuple] = {}
+        for n in networks:
+            prod = ipaddress.ip_network(
+                ipaddress.ip_interface(n["subnet"]).network)
+            if mode == "sim":
+                sim = ipaddress.ip_network(SIM_SUBNETS[vlan_of(n)])
+            else:
+                sim = prod
+            self.map[vlan_of(n)] = (prod, sim)
+
+    def net(self, vlan: int) -> ipaddress.IPv4Network:
+        return self.map[vlan][1]
+
+    def addr(self, vlan: int, ip: str, what: str) -> str | None:
+        """Map one address, or None if it does not fit the target subnet."""
+        prod, sim = self.map[vlan]
+        offset = int(ipaddress.ip_address(ip)) - int(prod.network_address)
+        if offset < 0 or offset >= sim.num_addresses:
+            self.clamped.append(f"{what}: {ip} does not fit {sim}")
+            return None
+        return str(ipaddress.ip_address(int(sim.network_address) + offset))
+
+    def gateway(self, vlan: int, net: dict) -> str:
+        """The address clients are told to use as their default route.
+
+        Production: the USG's current address (VIPs move .254 -> .1 at cutover),
+        so no client has to change anything. Sim: the sim router at .1.
+        """
+        if self.mode == "sim":
+            return str(self.net(vlan).network_address + 1)
+        return str(ipaddress.ip_interface(net["subnet"]).ip)
+
+
+def build(inv: dict, mode: str) -> tuple[list[str], dict]:
+    nets = [n for n in inv["networks"] if n["dhcp_enabled"] and n["subnet"]]
+    nets.sort(key=vlan_of)
+    m = Mapper(mode, nets)
+
+    out: list[str] = []
+    used_tags: set[str] = set()
+    stats = {"subnets": 0, "mappings": 0, "dropped": []}
+    by_vlan: dict[int, list] = {}
+    for r in inv["reservations"]:
+        if r["network_vlan"] is None and r["network_name"] != "Management":
+            # resolved_by == "unresolved"; cannot place it without a subnet
+            stats["dropped"].append(f"{r['ip']} {r['mac']} (no network)")
+            continue
+        by_vlan.setdefault(r["network_vlan"] or 1, []).append(r)
+
+    out.append("# --- DHCP ---------------------------------------------------")
+    for n in nets:
+        vlan = vlan_of(n)
+        sub = m.net(vlan)
+        base = f"set service dhcp-server shared-network-name {sanitize(n['name'], f'vlan{vlan}')} subnet {sub}"
+        gw = m.gateway(vlan, n)
+
+        out.append("")
+        out.append(f"# {n['name']} (VLAN {vlan}) <- {n['subnet']}")
+        # subnet-id is required by kea and must be stable across regenerations;
+        # the VLAN id is already the unique per-network number in this lab.
+        out.append(f"{base} subnet-id {vlan}")
+        out.append(f"{base} option default-router {gw}")
+
+        # Every VLAN is handed the gateway as its resolver, so all lookups go
+        # through VyOS and out to the upstreams above. UniFi set an explicit
+        # resolver on LoT only (the NAS); that is deliberately not carried over
+        # -- the NAS is legacy and pointing clients at it would keep it in the
+        # path for one VLAN and not the others.
+        out.append(f"{base} option name-server {gw}")
+
+        if n["domain_name"]:
+            out.append(f"{base} option domain-name '{n['domain_name']}'")
+        if n["dhcp_lease"]:
+            out.append(f"{base} lease {n['dhcp_lease']}")
+
+        start = m.addr(vlan, n["dhcp_start"], f"{n['name']} range start")
+        stop = m.addr(vlan, n["dhcp_stop"], f"{n['name']} range stop")
+        if start is None:
+            start = str(sub.network_address + 11)
+        if stop is None:
+            # Clamp to the last usable address rather than dropping the pool.
+            stop = str(sub.broadcast_address - 1)
+        out.append(f"{base} range LAN start {start}")
+        out.append(f"{base} range LAN stop {stop}")
+        stats["subnets"] += 1
+
+        for r in sorted(by_vlan.get(vlan, []), key=lambda x: ipaddress.ip_address(x["ip"])):
+            ip = m.addr(vlan, r["ip"], f"reservation {r['name']}")
+            if ip is None:
+                stats["dropped"].append(f"{r['ip']} {r['mac']} ({r['name']})")
+                continue
+            tag = sanitize(r["name"] or r["hostname"], "host-" + r["mac"].replace(":", ""))
+            # Distinct clients can sanitize to the same name; a collision would
+            # silently overwrite one reservation with another's address.
+            if tag in used_tags:
+                tag = f"{tag}-{r['mac'].replace(':', '')[-4:]}"
+            used_tags.add(tag)
+            out.append(f"{base} static-mapping {tag} mac {r['mac']}")
+            out.append(f"{base} static-mapping {tag} ip-address {ip}")
+            stats["mappings"] += 1
+
+    out.append("")
+    out.append("# --- DNS ----------------------------------------------------")
+    out.append("# The USG resolves for 5 of 6 VLANs today (it hands out its own")
+    out.append("# address when dhcpd_dns is empty). Without this, they lose DNS.")
+    for n in nets:
+        vlan = vlan_of(n)
+        out.append(f"set service dns forwarding listen-address {m.gateway(vlan, n)}")
+        out.append(f"set service dns forwarding allow-from {m.net(vlan)}")
+    for ns in UPSTREAM_DNS:
+        out.append(f"set service dns forwarding name-server {ns}")
+    out.append("set service dns forwarding cache-size 10000")
+
+    stats["clamped"] = m.clamped
+    return out, stats
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    here = os.path.dirname(os.path.abspath(__file__))
+    ap.add_argument("--mode", choices=("prod", "sim"), required=True)
+    ap.add_argument("--inventory", default=os.path.join(here, "export", "inventory.json"))
+    ap.add_argument("-o", "--out")
+    ap.add_argument("--check", action="store_true", help="counts only, no config")
+    args = ap.parse_args()
+
+    with open(args.inventory) as fh:
+        inv = json.load(fh)
+
+    lines, stats = build(inv, args.mode)
+
+    expected = len(inv["reservations"])
+    print(f"mode={args.mode}  subnets={stats['subnets']}  "
+          f"static-mappings={stats['mappings']}/{expected}", file=sys.stderr)
+    for c in stats["clamped"]:
+        print(f"  clamped: {c}", file=sys.stderr)
+    for d in stats["dropped"]:
+        print(f"  DROPPED: {d}", file=sys.stderr)
+
+    if stats["mappings"] != expected and args.mode == "prod":
+        print(f"ERROR: {expected - stats['mappings']} reservation(s) missing from "
+              f"prod output -- every one must survive the cutover", file=sys.stderr)
+        return 1
+
+    if args.check:
+        return 0
+
+    text = "\n".join(lines) + "\n"
+    if args.out:
+        with open(args.out, "w") as fh:
+            fh.write(text)
+        print(f"wrote {args.out}", file=sys.stderr)
+    else:
+        sys.stdout.write(text)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

migration/vrrp-wan-apply

@@ -0,0 +1,32 @@
+#!/bin/vbash
+# Enable or disable the WAN. Split out from vrrp-wan-reconcile for one reason:
+# `source /opt/vyatta/etc/functions/script-template` must be the FIRST thing the
+# script does. Sourced after a few statements -- an if, an exec, a mkdir -- it
+# silently terminated the script; `set -x` showed execution stopping inside the
+# source with no error and rc=0, so the reconciler reported success having done
+# nothing. Only a single assignment may precede it (the template resets the
+# positional parameters, so the mode is captured first), which is the same shape
+# /config/vyos-known-good uses.
+#
+#   vrrp-wan-apply enable    take the WAN
+#   vrrp-wan-apply disable   release it
+MODE="${1:-}"
+source /opt/vyatta/etc/functions/script-template
+
+WAN_VIF=53
+cfg() { /opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration commands 2>/dev/null; }
+wan_disabled(){ cfg | grep -q "vif ${WAN_VIF} disable"; }
+ppp_disabled(){ cfg | grep -q "pppoe pppoe0 disable"; }
+
+configure
+if [ "$MODE" = enable ]; then
+    # Guarded: `delete` of an absent node aborts the whole batch with
+    # "Nothing to delete", which left the box detected-but-unfixed.
+    wan_disabled && delete interfaces bonding bond0 vif ${WAN_VIF} disable
+    ppp_disabled && delete interfaces pppoe pppoe0 disable
+else
+    wan_disabled || set interfaces bonding bond0 vif ${WAN_VIF} disable
+    ppp_disabled || set interfaces pppoe pppoe0 disable
+fi
+commit
+exit

migration/vrrp-wan-health

@@ -0,0 +1,68 @@
+#!/bin/sh
+# VRRP health check: may THIS router hold the floating IPs?
+#
+# It may only if it can actually carry the WAN. Without this, VRRP decides
+# mastership purely on whether the peer is still advertising -- so a router with
+# no WAN at all happily takes the VIPs and blackholes the entire LAN's internet
+# while looking perfectly healthy. That is not hypothetical: it is the outage of
+# 2026-09-02, reproduced in labsim.
+#
+# ---------------------------------------------------------------------------
+# The first version of this script asked one question: "do I have an address on
+# a WAN interface". That is correct for a pair where both routers hold WAN all
+# the time. Ours cannot: the 10 gig lease is bound to a cloned MAC and the
+# PPPoE line to a single credential, so the WAN follows mastership (see
+# vrrp-wan-take). Against that design the old check DEADLOCKS --
+#
+#     may I be master?  ->  only if I already have WAN
+#     do I have WAN?    ->  only if I am master
+#
+# -- and the backup sits in FAULT for ever. vyos002 sat exactly there, which
+# meant the pair could not fail over at all: the safety check had quietly
+# removed the redundancy it was protecting.
+#
+# So the question is now asked in the right order: enforce "must have WAN" only
+# on the router that is actually HOLDING the VIPs, and give a new master time to
+# bring the WAN up before judging it.
+# ---------------------------------------------------------------------------
+#
+# exit 0 = eligible for MASTER, non-zero = release and let the peer have it.
+
+STATE=/run/vrrp-wan
+GRACE=90          # seconds a new master gets to complete DHCP / PPPoE dial-up
+VIP="${VRRP_WAN_VIP:-192.168.1.1}"
+
+# Am I holding the VIPs? Asked of REALITY -- is the management VIP actually on
+# this box -- and not of a /run marker.
+#
+# The marker was the first design and it is unsafe: it is written by the VRRP
+# transition script, and in labsim that script silently failed to run on a
+# promotion (VyOS's keepalived-fifo.py helper stopped delivering while
+# keepalived's own notifies kept working). The router then believed it was
+# backup, passed this check, and sat holding every VIP with no WAN -- the exact
+# outage this script exists to prevent, re-created by trusting the reporter
+# instead of the fact.
+[ -n "$(ip -4 -o addr show 2>/dev/null | grep " ${VIP}/")" ] || exit 0
+
+# Master with an address on a WAN interface: healthy.
+#
+# Deliberately NOT "can I reach the internet" and NOT "do I have a default
+# route". During a real ISP outage the route disappears on BOTH routers; a check
+# keyed on that would put both into FAULT, nobody would hold the VIPs, and the
+# LAN would lose inter-VLAN routing too -- turning an internet outage into a
+# total one. A DHCP lease survives an ISP outage, so an address still
+# distinguishes "this box structurally cannot route" from "the internet is down
+# right now", which is the distinction that matters.
+for ifc in bond0.53 pppoe0; do
+    ip -4 addr show dev "$ifc" 2>/dev/null | grep -q 'inet ' && exit 0
+done
+
+# Master, no WAN yet, still within the grace window: DHCP negotiation and PPPoE
+# dial-up take real time, and the ISP has to accept the cloned MAC arriving on a
+# different port. Failing here would demote the new master before it ever had a
+# chance, and hand the VIPs straight back -- a flap, not a failover.
+since=$(cat "$STATE/since" 2>/dev/null || echo 0)
+[ $(( $(date +%s) - since )) -lt "$GRACE" ] && exit 0
+
+# Master, past grace, still no WAN: release. This is the 2026-09-02 case.
+exit 1

migration/vrrp-wan-reconcile

@@ -0,0 +1,106 @@
+#!/bin/sh
+# Make the WAN match VRRP mastership. Idempotent; safe to run every 30s and on
+# every VRRP transition.
+#
+# Why a reconciler and not just transition scripts
+# ------------------------------------------------
+# VyOS delivers `transition-script` through a helper process,
+# /usr/libexec/vyos/system/keepalived-fifo.py, fed by keepalived's notify_fifo.
+# Observed in labsim on 2026-09-02: the primary's Keepalived_vrrp logged
+# "(native) Entering MASTER STATE" for all six instances and the built-in
+# notify_master for conntrack-sync ran -- while the fifo helper logged NOTHING
+# and the master transition script never ran. The helper process was still
+# alive. The result was a router holding every VIP with no WAN at all: the exact
+# 2026-09-02 outage, re-created by the mechanism meant to prevent it.
+#
+# So transition scripts are kept for speed but nothing is trusted to them: this
+# also runs on a timer, and derives everything from ground truth rather than
+# from a marker that only exists if the script it depends on ran.
+#
+#   vrrp-wan-reconcile          reconcile once
+#   vrrp-wan-reconcile --status what it thinks, changing nothing
+#
+# Ground truth for "am I master" is whether the management VIP is really on this
+# box. It is what VRRP actually does, it is observable, and it cannot silently
+# disagree with reality.
+
+VIP="${VRRP_WAN_VIP:-192.168.1.1}"      # management VIP; sim overrides via env
+WAN_VIF=53                               # bond0.53, the DHCP WAN
+STATE=/run/vrrp-wan
+LOCK=/run/vrrp-wan.lock
+
+cfg() { /opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration commands 2>/dev/null; }
+holds_vip()   { ip -4 -o addr show 2>/dev/null | grep -q " ${VIP}/"; }
+wan_up()      { ip -4 addr show "bond0.${WAN_VIF}" 2>/dev/null | grep -q 'inet '; }
+wan_disabled(){ cfg | grep -q "vif ${WAN_VIF} disable"; }
+ppp_disabled(){ cfg | grep -q "pppoe pppoe0 disable"; }
+
+# --status must answer WITHOUT sourcing script-template. The template's `exit`
+# is a function that leaves configuration mode, not the shell builtin, so a
+# status run that had sourced it opened and closed a config session on every
+# call -- which is how a read-only query started colliding with the timer and
+# logging "Configuration system temporarily locked due to another commit".
+if [ "${1:-}" = "--status" ]; then
+    printf 'vip=%s holds_vip=%s wan_disabled=%s wan_up=%s role=%s\n' \
+        "$VIP" "$(holds_vip && echo yes || echo no)" \
+        "$(wan_disabled && echo yes || echo no)" \
+        "$(wan_up && echo yes || echo no)" \
+        "$(cat "$STATE/role" 2>/dev/null || echo unset)"
+    exit 0
+fi
+
+# One writer. The 30s timer and a VRRP transition can fire together, and two
+# VyOS commits in flight on one box do not queue -- the second fails outright.
+exec 9>"$LOCK"
+flock -n 9 || exit 0
+
+mkdir -p "$STATE"
+
+# Reap config sessions whose owning process is gone. VyOS creates
+# /opt/vyatta/config/tmp/new_config_<pid> (a unionfs mount) per `configure`, and
+# a script that dies inside a session never removes it. One of those holds the
+# commit lock, and from then on EVERY commit fails with "Configuration system
+# temporarily locked due to another commit in progress" -- including the manual
+# one you try in order to fix it. A job on a 30s timer that can leak a session
+# per failure will wedge the router's config system on its own, so it cleans up
+# before it starts. `umount -l` first: the directory is a mount point and plain
+# rm returns "Device or resource busy".
+for d in /opt/vyatta/config/tmp/new_config_*; do
+    [ -d "$d" ] || continue
+    pid=${d##*_}
+    kill -0 "$pid" 2>/dev/null && continue
+    umount -l "$d" 2>/dev/null
+    rm -rf "$d" 2>/dev/null
+done
+
+# The config edit lives in vrrp-wan-apply, because script-template must be the
+# first thing its script does -- sourced any later it terminates the script
+# silently with rc=0. See the header there.
+APPLY=/config/vrrp-wan-apply
+
+if holds_vip; then
+    echo master > "$STATE/role"
+    # Stamp only on entry to master, so the health check's grace window measures
+    # time-since-promotion rather than time-since-last-tick.
+    [ -f "$STATE/since" ] || date +%s > "$STATE/since"
+    wan_disabled || ppp_disabled || exit 0
+    logger -t vrrp-wan "MASTER with WAN disabled -> enabling bond0.${WAN_VIF} + pppoe0"
+    "$APPLY" enable
+else
+    echo backup > "$STATE/role"
+    rm -f "$STATE/since"
+    { wan_disabled && ppp_disabled; } && exit 0
+    # Releasing matters more than taking. A demoted router that keeps the WAN up
+    # holds the cloned MAC f0:9f:c2:12:9b:4f on VLAN 53 at the same time as the
+    # new master, and the switch sends the ISP's replies to whichever port spoke
+    # last -- the WAN-side twin of the eth2 incident.
+    logger -t vrrp-wan "not MASTER but WAN enabled -> releasing bond0.${WAN_VIF} + pppoe0"
+    "$APPLY" disable
+fi
+
+# Deliberately no `save`. config.boot keeps `disable` on BOTH routers, so a
+# reboot in any order comes up unable to claim the shared MAC, and only holding
+# the VIP re-enables it. NOTE: any `save` while this box is master (a hand
+# commit, or `pulumi up`) WILL persist the enabled state -- observed in labsim.
+# The Pulumi model asserts `disable` on both routers so an apply puts it back,
+# and vyos:verify reports it as drift if it does not.

migration/vrrp-wan-reconcile.service

@@ -0,0 +1,10 @@
+[Unit]
+# Belt to the transition scripts' braces. VyOS's keepalived-fifo.py helper was
+# observed dropping a MASTER transition silently, leaving a router holding every
+# VIP with no WAN. A timer cannot be dropped the same way.
+Description=Reconcile WAN interface state with VRRP mastership
+After=keepalived.service
+
+[Service]
+Type=oneshot
+ExecStart=/config/vrrp-wan-reconcile

migration/vrrp-wan-reconcile.timer

@@ -0,0 +1,13 @@
+[Unit]
+Description=Reconcile WAN with VRRP mastership every 30s
+
+[Timer]
+# 30s: fast enough that a dropped transition is a blip rather than an outage,
+# slow enough that it is never the thing generating load. It only commits when
+# state actually disagrees, so a steady-state tick is two `ip` calls and a grep.
+OnBootSec=60
+OnUnitActiveSec=30
+AccuracySec=5
+
+[Install]
+WantedBy=timers.target

migration/vrrp-wan-release

@@ -0,0 +1,5 @@
+#!/bin/sh
+# VRRP transition hook. One code path: the reconciler derives everything from
+# ground truth, so take and release are the same operation asked at different
+# moments. Speed comes from here; correctness comes from the timer.
+exec /config/vrrp-wan-reconcile

migration/vrrp-wan-take

@@ -0,0 +1,5 @@
+#!/bin/sh
+# VRRP transition hook. One code path: the reconciler derives everything from
+# ground truth, so take and release are the same operation asked at different
+# moments. Speed comes from here; correctness comes from the timer.
+exec /config/vrrp-wan-reconcile

migration/vyos-known-good

@@ -0,0 +1,120 @@
+#!/bin/vbash
+# Pin a config you have SEEN working, and get back to it with one command.
+#
+# Why this exists when VyOS already has rollback: `rollback 1` returns you to the
+# previous revision, which may itself be broken -- you can walk backwards through
+# several bad commits looking for the one that worked. This pins a state you
+# explicitly confirmed was good, so recovery is one step and does not require
+# remembering how many changes ago things last worked.
+#
+# It is deliberately NOT automatic. A config is only "known good" once a human
+# has used the network and found it working; a script cannot judge that, and a
+# snapshot taken automatically after every commit would faithfully preserve the
+# broken one.
+#
+#   vyos-known-good save      mark the running config as known-good
+#   vyos-known-good status    when it was taken, and how it differs from running
+#   vyos-known-good restore   go back to it (commit-confirmed, so even this is safe)
+#   vyos-known-good diff      what would change if you restored
+#
+# Lives in /config so it survives image upgrades, like vyos-unifi-switch.
+
+# Capture the arguments BEFORE sourcing script-template: sourcing it resets the
+# positional parameters, so $1 is empty by the time the case statement runs and
+# every invocation silently falls through to the usage message.
+ACTION="${1:-status}"
+
+source /opt/vyatta/etc/functions/script-template
+
+GOOD="/config/known-good.boot"
+META="/config/known-good.meta"
+RUNNING="/config/config.boot"
+CONFIRM_MINUTES="${CONFIRM_MINUTES:-5}"
+
+say()  { printf '\033[0;36m[known-good]\033[0m %s\n' "$*"; }
+warn() { printf '\033[1;33m[known-good]\033[0m %s\n' "$*" >&2; }
+die()  { printf '\033[0;31m[known-good]\033[0m %s\n' "$*" >&2; exit 1; }
+
+# The running config on disk is only current if nothing is uncommitted-and-unsaved.
+# Saving a snapshot that does not match what is actually running would be worse
+# than having no snapshot at all -- it would look like a safety net and not be one.
+require_saved() {
+  if ! cli-shell-api sessionChanged >/dev/null 2>&1; then
+    return 0
+  fi
+  die "there are uncommitted changes; commit and save first, or this snapshot would not match reality"
+}
+
+cmd_save() {
+  require_saved
+  [ -r "$RUNNING" ] || die "cannot read $RUNNING"
+  sudo cp "$RUNNING" "$GOOD"
+  # 0660 root:vyattacfg, matching /config/config.boot. 0600 would make the
+  # snapshot unreadable to the vyos user, so `status` and `diff` -- the two you
+  # run while deciding whether to restore -- would silently show nothing.
+  sudo chmod 0660 "$GOOD"; sudo chgrp vyattacfg "$GOOD"
+  sudo chmod 0660 "$META" 2>/dev/null; sudo chgrp vyattacfg "$META" 2>/dev/null
+  {
+    echo "saved_at=$(date -Is)"
+    echo "saved_by=${SUDO_USER:-$USER}"
+    echo "hostname=$(hostname)"
+    echo "lines=$(wc -l < "$RUNNING")"
+  } | sudo tee "$META" >/dev/null
+  say "pinned $(wc -l < "$GOOD") lines as known-good on $(hostname)"
+  say "restore with: /config/vyos-known-good restore"
+}
+
+cmd_status() {
+  [ -r "$GOOD" ] || { warn "no known-good snapshot on $(hostname) -- run 'save' while things work"; return 1; }
+  say "known-good on $(hostname):"
+  sed 's/^/    /' "$META" 2>/dev/null
+  local n
+  n="$(diff <(grep -vE '^\s*$' "$GOOD") <(grep -vE '^\s*$' "$RUNNING") 2>/dev/null | grep -c '^[<>]')"
+  if [ "${n:-0}" -eq 0 ]; then
+    say "running config MATCHES known-good"
+  else
+    warn "running config differs from known-good by $n line(s) -- 'diff' to see them"
+  fi
+}
+
+cmd_diff() {
+  [ -r "$GOOD" ] || die "no known-good snapshot"
+  diff -u "$GOOD" "$RUNNING" | sed -E "s/(password|key|secret)[[:space:]]+\S+/\1 <REDACTED>/I" || true
+}
+
+cmd_restore() {
+  [ -r "$GOOD" ] || die "no known-good snapshot to restore"
+  say "restoring known-good on $(hostname) (taken $(grep -m1 saved_at "$META" 2>/dev/null | cut -d= -f2-))"
+
+  # Commit-confirmed even here. If the known-good snapshot is itself somehow
+  # wrong, or the restore cannot be confirmed because access is still broken,
+  # the router undoes it rather than leaving you worse off. Silence reverts.
+  local script; script="$(mktemp)"
+  {
+    echo 'source /opt/vyatta/etc/functions/script-template'
+    echo 'configure'
+    echo "load $GOOD"
+    printf 'sudo sg vyattacfg "/usr/bin/config-mgmt commit_confirm -y -t=%s"\n' "$CONFIRM_MINUTES"
+    echo 'export IN_COMMIT_CONFIRM=t'
+    echo 'commit'
+    echo 'unset IN_COMMIT_CONFIRM'
+    echo 'exit'
+  } > "$script"
+  vbash "$script"; local rc=$?
+  rm -f "$script"
+
+  [ $rc -eq 0 ] || die "restore failed (rc=$rc) -- nothing was committed"
+  say ""
+  say "RESTORED under a ${CONFIRM_MINUTES} minute timer."
+  say "Check the network NOW. If it works, confirm it:"
+  say "    sudo sg vyattacfg '/usr/bin/config-mgmt confirm'"
+  say "If you do nothing, the router reverts on its own."
+}
+
+case "$ACTION" in
+  save)    cmd_save ;;
+  status)  cmd_status ;;
+  diff)    cmd_diff ;;
+  restore) cmd_restore ;;
+  *) die "usage: vyos-known-good {save|status|diff|restore}" ;;
+esac

migration/vyos-mode-delta.py

@@ -0,0 +1,491 @@
+#!/usr/bin/env python3
+"""Generate the delta that turns a passive VyOS pair into the gateway.
+
+The switch works as: load the known-good `unifi.boot` snapshot, apply this
+delta, commit-confirm. Deriving the gateway mode from base+delta every time
+means there is no inverse to maintain and no drift between two hand-kept
+configs -- the revert is just loading the snapshot again.
+
+  ./vyos-mode-delta.py --priority 200 -o to-vyos.commands     # vyos001 (master)
+  ./vyos-mode-delta.py --priority 100 -o to-vyos.commands     # vyos002 (backup)
+  ./vyos-mode-delta.py --emit-secrets /path/wan-secrets       # credentials, 0600
+
+The PPPoE password is NOT written into the delta. The delta carries the
+placeholder @@WAN_PASSWORD@@ and the switch script substitutes it at apply time
+from /config/wan-secrets, so the generated artifact can be read, diffed and
+copied around without carrying a credential.
+"""
+from __future__ import annotations
+
+import argparse
+import importlib.util
+import ipaddress
+import json
+import os
+import sys
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+
+# unifi-to-vyos.py has hyphens, so it cannot be imported by name. Reuse it
+# rather than duplicating the DHCP/DNS generation -- the whole point is that
+# what labsim proved and what production gets come from one code path.
+_spec = importlib.util.spec_from_file_location(
+    "unifi_to_vyos", os.path.join(HERE, "unifi-to-vyos.py"))
+unifi_to_vyos = importlib.util.module_from_spec(_spec)
+_spec.loader.exec_module(unifi_to_vyos)
+
+# Two WANs, established by reading the live USG rather than the UniFi fields
+# (which report wan_type=dhcp for both and are simply wrong):
+#
+#   WAN1  Vodafone, PPPoE on the USG's eth0, ~900/700 Mbit. Verified working:
+#         pppoe0 came up with 90.241.226.213 peer 84.65.128.1, MTU 1492.
+#   WAN2  10 gig ISP, plain DHCP on the USG's eth2, public 87.192.101.48/21
+#         gw 87.192.96.1. This is what carries traffic today.
+#
+# Both reach the USG as untagged access ports but are carried across the switch
+# fabric as vlan-only networks 51 and 53, so VyOS picks them up as bond vifs.
+WAN_PPPOE_VIF = "bond0.51"      # Vodafone
+WAN_PPPOE_IF = "pppoe0"
+WAN_DHCP_VIF = "bond0.53"       # 10 gig ISP
+
+# The DHCP lease is bound to the MAC, so cloning the USG's WAN2 MAC is how VyOS
+# keeps 87.192.101.48 instead of negotiating a fresh lease -- or getting none,
+# if the ISP hands out one per line. Only ONE box may carry this at a time.
+WAN_DHCP_MAC = "f0:9f:c2:12:9b:4f"
+
+# Route distances: the 10 gig line wins, Vodafone is failover.
+#
+# The live 10 gig default route is owned by `protocols failover`, so that losing
+# the ISP *without* losing carrier withdraws it instead of black-holing every
+# packet -- a DHCP-installed route never withdraws on a dead upstream.
+#
+# The vif still needs default-route-distance rather than no-default-route:
+# vyos-failover resolves a dhcp-interface gateway by reading new_routers out of
+# /run/dhclient/dhclient_<if>.lease, and no-default-route leaves that field
+# EMPTY, so the daemon finds no next hop and installs nothing. Verified on
+# vyos001: with no-default-route the default route fell through to Vodafone.
+#
+# So DHCP keeps a route, deliberately demoted BELOW Vodafone. Order of
+# preference: failover's kernel route (distance 0) > pppoe (10) > DHCP (210).
+# The demoted route is never selected while pppoe is up, so it cannot re-create
+# the black-hole it exists to avoid.
+DIST_PPPOE = 10
+DIST_DHCP_FALLBACK = 210
+
+# Health-checked primary. Two targets, any-available, so one resolver having a
+# bad day is not read as "the line is down". Verified on the sim: failover and
+# failback both inside 5s with the router's own interface still UP.
+FAILOVER_METRIC = 1
+FAILOVER_TARGETS = ["8.8.8.8", "1.1.1.1"]
+FAILOVER_TIMEOUT = 5
+
+PLACEHOLDER = "@@WAN_PASSWORD@@"
+
+# Per-VLAN interface addresses of each node, read from the live boxes. VRRP
+# unicast (hello-source-address/peer-address) needs both ends explicitly, and
+# these are NOT derivable from the subnet -- VLAN 3 is .4/.5 while everything
+# else is .252/.253.
+#                vlan: (vyos001,          vyos002)
+NODE_ADDRS = {
+    1:   ("192.168.1.252", "192.168.1.253"),
+    2:   ("192.168.9.252", "192.168.9.253"),
+    3:   ("192.168.3.4",   "192.168.3.5"),
+    9:   ("10.8.0.252",    "10.8.0.253"),
+    10:  ("10.0.1.252",    "10.0.1.253"),
+    200: ("192.168.2.252", "192.168.2.253"),
+}
+
+# Dedicated point-to-point link for conntrack state sync (eth3 <-> eth3).
+CONNTRACK_ADDRS = ("10.255.255.1/30", "10.255.255.2/30")
+CONNTRACK_IF = "eth3"
+
+# kea HA talks over TCP 647. The LoT addresses are used because they are stable
+# and reachable today without the conntrack cable being plugged in.
+DHCP_HA_NAME = "vyos-dhcp-pair"     # must NOT equal either system host-name
+
+
+
+def vrrp_group(vlan: int) -> str:
+    """VRRP group names as configured on the boxes: 'native' for the untagged
+    VLAN, 'vlan<id>' otherwise."""
+    return "native" if vlan == 1 else f"vlan{vlan}"
+
+
+def build_delta(inv: dict, priority: int, wan_user: str, with_wan: bool,
+                conntrack_link: bool) -> list[str]:
+    out: list[str] = []
+    primary = priority >= 200          # vyos001 is the master/primary
+    self_i, peer_i = (0, 1) if primary else (1, 0)
+    nets = [n for n in inv["networks"] if n["dhcp_enabled"] and n["subnet"]]
+    nets.sort(key=unifi_to_vyos.vlan_of)
+
+    out += [
+        "# ==========================================================",
+        "# Delta: passive VyOS pair -> gateway. Applied on top of a",
+        "# freshly loaded unifi.boot, never on top of itself.",
+        "# ==========================================================",
+        "",
+        "# An unconfirmed commit must reload the previous config, NOT reboot.",
+        "# 'reboot' is the VyOS default and would turn a failed switch into a",
+        "# real outage on the box that is meant to be carrying the network.",
+        "set system config-management commit-confirm action reload",
+        "",
+        "# --- gateway addresses ------------------------------------",
+        "# The VIP takes over the address the USG holds today, so no client",
+        "# changes anything: no renewal needed, hardcoded gateways keep working.",
+    ]
+    for n in nets:
+        vlan = unifi_to_vyos.vlan_of(n)
+        grp = vrrp_group(vlan)
+        iface = ipaddress.ip_interface(n["subnet"])
+        out.append(f"# {n['name']} (VLAN {vlan}) -> {iface.with_prefixlen}")
+        # Delete the whole address node rather than a computed old value.
+        # `address` is multi-value, and the current VIPs are NOT at
+        # network+254 on the /23 networks -- they are 192.168.9.254,
+        # 10.0.9.254 and 10.0.1.254, in the upper half. A delete naming the
+        # wrong address fails quietly and leaves the group holding two VIPs.
+        out.append(f"delete high-availability vrrp group {grp} address")
+        out.append(f"set high-availability vrrp group {grp} address {iface.with_prefixlen}")
+        out.append(f"set high-availability vrrp group {grp} priority {priority}")
+        own, peer = NODE_ADDRS[vlan] if primary else NODE_ADDRS[vlan][::-1]
+        # Unicast VRRP: the walkthrough sets both ends explicitly rather than
+        # relying on multicast, which is more predictable across a switch fabric.
+        out.append(f"set high-availability vrrp group {grp} hello-source-address {own}")
+        out.append(f"set high-availability vrrp group {grp} peer-address {peer}")
+        # Without no-preempt a recovered box reclaims the VIP immediately --
+        # before conntrack state has synced -- and drops every established
+        # connection. If preemption is ever wanted, preempt-delay must be >=
+        # the conntrack-sync purge-timeout.
+        out.append(f"set high-availability vrrp group {grp} no-preempt")
+
+    out += [
+        "",
+    ]
+
+
+    out += [
+        "",
+        "# --- stateful tracking (BOTH boxes) ------------------------",
+        "# VyOS only engages conntrack when a firewall or NAT exists. The",
+        "# backup has no WAN and therefore no NAT, so without this rule it",
+        "# tracks nothing -- and conntrack-sync entries replicated to a box",
+        "# whose conntrack is not engaged cannot be used when it takes over.",
+        "# Verified in labsim: zero conntrack entries until a state-matching",
+        "# rule was present, then replication began immediately.",
+        "set firewall ipv4 forward filter default-action accept",
+        "set firewall ipv4 forward filter rule 10 action accept",
+        "set firewall ipv4 forward filter rule 10 state established",
+        "set firewall ipv4 forward filter rule 10 state related",
+        "set firewall ipv4 forward filter rule 10 description 'stateful tracking'",
+    ]
+
+    if True:  # WAN config on BOTH boxes; see the disable block below
+        out += [
+            "# --- WAN -----------------------------------------------",
+            "# Both vifs must be created before anything references them.",
+            "# Neither firewall has vif 51 or 53 today (only 2, 3, 9, 10, 200),",
+            "# and pppoe source-interface points at an interface that must",
+            "# already exist -- without this the commit fails and, since the",
+            "# delta commits as one unit, takes the whole switch with it.",
+            f"set interfaces bonding bond0 vif {WAN_PPPOE_VIF.split('.')[1]} description 'WAN1 Vodafone (PPPoE)'",
+            f"set interfaces bonding bond0 vif {WAN_DHCP_VIF.split('.')[1]} description 'WAN2 10gig ISP (DHCP)'",
+            "",
+            "# WAN2, the 10 gig line -- primary. The cloned MAC is what keeps",
+            "# the existing public lease (87.192.101.48) instead of asking for",
+            "# a new one. Only the box carrying the WAN may set this.",
+            f"set interfaces bonding bond0 vif {WAN_DHCP_VIF.split('.')[1]} mac '{WAN_DHCP_MAC}'",
+            f"set interfaces bonding bond0 vif {WAN_DHCP_VIF.split('.')[1]} address dhcp",
+            # Demoted below Vodafone; `protocols failover` owns the live route.
+            # NOT no-default-route -- that blanks new_routers in the lease and
+            # leaves the failover daemon with no gateway to install.
+            f"set interfaces bonding bond0 vif {WAN_DHCP_VIF.split('.')[1]} dhcp-options default-route-distance {DIST_DHCP_FALLBACK}",
+            "",
+            "# WAN1, Vodafone -- failover at a higher distance. Verified working",
+            "# on the USG: pppoe0 came up with a public address, MTU 1492.",
+            f"set interfaces pppoe {WAN_PPPOE_IF} source-interface {WAN_PPPOE_VIF}",
+            f"set interfaces pppoe {WAN_PPPOE_IF} authentication username '{wan_user}'",
+            f"set interfaces pppoe {WAN_PPPOE_IF} authentication password '{PLACEHOLDER}'",
+            f"set interfaces pppoe {WAN_PPPOE_IF} mtu 1492",
+            f"set interfaces pppoe {WAN_PPPOE_IF} default-route-distance {DIST_PPPOE}",
+            # The peer's resolvers would otherwise overwrite resolv.conf.
+            f"set interfaces pppoe {WAN_PPPOE_IF} no-peer-dns",
+            "",
+            "# The static default route exists only for unifi mode, where the",
+            "# USG is the next hop. Both WANs supply one here.",
+            "delete protocols static route 0.0.0.0/0",
+            "",
+            "# --- Health-checked primary ----------------------------",
+            "# Without this, failover only fires when bond0.53 loses carrier",
+            "# or its lease. An ISP that keeps the link up while dropping",
+            "# traffic -- the common failure -- would black-hole everything,",
+            "# because a DHCP-installed route has nothing to withdraw it.",
+            "#",
+            "# vyos-failover pings each target bound to the interface",
+            "# (`ping -I bond0.53`), so the backup can never be validated",
+            "# through the primary's path and vice versa. On withdrawal the",
+            "# kernel falls through to Vodafone's distance-10 route.",
+            f"set protocols failover route 0.0.0.0/0 dhcp-interface {WAN_DHCP_VIF} check type icmp",
+            f"set protocols failover route 0.0.0.0/0 dhcp-interface {WAN_DHCP_VIF} check policy any-available",
+            f"set protocols failover route 0.0.0.0/0 dhcp-interface {WAN_DHCP_VIF} check timeout {FAILOVER_TIMEOUT}",
+            f"set protocols failover route 0.0.0.0/0 dhcp-interface {WAN_DHCP_VIF} metric {FAILOVER_METRIC}",
+            *[
+                f"set protocols failover route 0.0.0.0/0 dhcp-interface {WAN_DHCP_VIF} check target {t}"
+                for t in FAILOVER_TARGETS
+            ],
+            "",
+            "# --- NAT -----------------------------------------------",
+            f"set nat source rule 100 outbound-interface name {WAN_DHCP_VIF}",
+            "set nat source rule 100 translation address masquerade",
+            "set nat source rule 100 description 'LAN out via the 10gig line'",
+            f"set nat source rule 110 outbound-interface name {WAN_PPPOE_IF}",
+            "set nat source rule 110 translation address masquerade",
+            "set nat source rule 110 description 'LAN out via Vodafone (failover)'",
+        ]
+
+    if not with_wan:
+        # The backup carries the identical WAN and NAT config but with the
+        # interfaces administratively DOWN. The cloned MAC is therefore never
+        # live on two boxes at once, while everything needed to route and
+        # masquerade is already present -- taking over is enabling two
+        # interfaces, not rebuilding a config under pressure.
+        #
+        # NAT rules naming a down interface are harmless: VyOS warns at commit
+        # ("Interface ... does not exist!") and commits anyway, verified.
+        out += [
+            "",
+            "# --- WAN held DOWN on this box -----------------------------",
+            "# Enable these two to take over the internet path:",
+            f"#   set interfaces bonding bond0 vif {WAN_DHCP_VIF.split('.')[1]} disable  <- delete this",
+            f"#   set interfaces pppoe {WAN_PPPOE_IF} disable                <- and this",
+            f"set interfaces bonding bond0 vif {WAN_DHCP_VIF.split('.')[1]} disable",
+            f"set interfaces pppoe {WAN_PPPOE_IF} disable",
+        ]
+
+    if True:
+        # Port forwards and the WAN firewall go on BOTH boxes. They name
+        # interfaces that are present-but-disabled on the backup, which VyOS
+        # accepts (it warns and commits). Putting them here means a failover is
+        # enabling an interface, not reconstructing NAT under pressure.
+        # Port forwards, straight from UniFi.
+        for i, p in enumerate(inv["port_forwards"]):
+            if not p.get("enabled"):
+                continue
+            rule = 100 + i * 10
+            proto = p["proto"]  # tcp | udp | tcp_udp -- all valid VyOS values
+            out += [
+                "",
+                f"set nat destination rule {rule} description '{p['name']}'",
+                f"set nat destination rule {rule} inbound-interface name {WAN_DHCP_VIF}",
+                f"set nat destination rule {rule} protocol {proto}",
+                f"set nat destination rule {rule} destination port '{p['dst_port']}'",
+                f"set nat destination rule {rule} translation address {p['fwd']}",
+            ]
+            # `destination port` accepts a comma list but `translation port` does
+            # NOT -- "16881,6881 is not a valid service name" -- because mapping a
+            # list onto a list is ambiguous. Every forward here maps a port to
+            # itself, and omitting translation port makes VyOS preserve the
+            # original, which is exactly right. Only emit it when it genuinely
+            # differs, and refuse rather than guess when a differing list appears.
+            if p["fwd_port"] != p["dst_port"]:
+                if "," in str(p["fwd_port"]) or "," in str(p["dst_port"]):
+                    raise SystemExit(
+                        f"port forward '{p['name']}' remaps a LIST of ports "
+                        f"({p['dst_port']} -> {p['fwd_port']}). VyOS cannot express "
+                        f"that in one rule; split it into one rule per port by hand.")
+                out.append(f"set nat destination rule {rule} translation port '{p['fwd_port']}'")
+
+        out += [
+            "",
+            "# --- firewall ----------------------------------------------",
+            "# VyOS defaults to accepting everything. The USG has an implicit",
+            "# WAN drop, so migrating the port forwards alone would leave the",
+            "# router's own services and the whole LAN reachable from the WAN.",
+            "#",
+            "# Scoped to the WAN interface rather than a global default-action",
+            "# drop: that way a mistake here cannot lock anyone out over the LAN,",
+            "# which is the only path back in during a cutover.",
+            "",
+            "# Traffic TO the router.",
+            "set firewall ipv4 input filter default-action accept",
+            "set firewall ipv4 input filter rule 100 action accept",
+            "set firewall ipv4 input filter rule 100 state established",
+            "set firewall ipv4 input filter rule 100 state related",
+            "set firewall ipv4 input filter rule 100 description 'established/related'",
+        ]
+        # The two WAN_LOCAL accepts carried over from UniFi.
+        out += [
+            "",
+            "set firewall ipv4 input filter rule 110 action accept",
+            "set firewall ipv4 input filter rule 110 protocol esp",
+            f"set firewall ipv4 input filter rule 110 inbound-interface name {WAN_DHCP_VIF}",
+            "set firewall ipv4 input filter rule 110 description 'VPN accept ESP (from UniFi WAN_LOCAL)'",
+            "",
+            "set firewall ipv4 input filter rule 120 action accept",
+            "set firewall ipv4 input filter rule 120 protocol udp",
+            "set firewall ipv4 input filter rule 120 destination port '500,4500'",
+            f"set firewall ipv4 input filter rule 120 inbound-interface name {WAN_DHCP_VIF}",
+            "set firewall ipv4 input filter rule 120 description 'VPN accept UDP500/4500 (from UniFi WAN_LOCAL)'",
+            "",
+            "set firewall ipv4 input filter rule 130 action accept",
+            "set firewall ipv4 input filter rule 130 protocol icmp",
+            f"set firewall ipv4 input filter rule 130 inbound-interface name {WAN_DHCP_VIF}",
+            "set firewall ipv4 input filter rule 130 description 'ICMP to the router (path MTU discovery)'",
+            "",
+            "# Everything else arriving from the WAN is dropped. LAN is untouched.",
+            "set firewall ipv4 input filter rule 900 action drop",
+            f"set firewall ipv4 input filter rule 900 inbound-interface name {WAN_DHCP_VIF}",
+            f"set firewall ipv4 input filter rule 910 action drop",
+            f"set firewall ipv4 input filter rule 910 inbound-interface name {WAN_PPPOE_IF}",
+            "set firewall ipv4 input filter rule 910 description 'drop all other WAN-to-router (Vodafone)'",
+            "set firewall ipv4 input filter rule 900 description 'drop all other WAN-to-router'",
+            "",
+            "# Traffic THROUGH the router.",
+            "set firewall ipv4 forward filter default-action accept",
+            "set firewall ipv4 forward filter rule 100 action accept",
+            "set firewall ipv4 forward filter rule 100 state established",
+            "set firewall ipv4 forward filter rule 100 state related",
+            "set firewall ipv4 forward filter rule 100 description 'established/related'",
+        ]
+
+        # Destination NAT happens before the forward filter, so these rules must
+        # match the translated destination, not the WAN address.
+        for i, p in enumerate(inv["port_forwards"]):
+            if not p.get("enabled"):
+                continue
+            rule = 200 + i * 10
+            out += [
+                "",
+                f"set firewall ipv4 forward filter rule {rule} action accept",
+                f"set firewall ipv4 forward filter rule {rule} inbound-interface name {WAN_DHCP_VIF}",
+                f"set firewall ipv4 forward filter rule {rule} protocol {p['proto']}",
+                f"set firewall ipv4 forward filter rule {rule} destination address {p['fwd']}",
+                f"set firewall ipv4 forward filter rule {rule} destination port '{p['fwd_port']}'",
+                f"set firewall ipv4 forward filter rule {rule} description 'port forward: {p['name']}'",
+            ]
+
+        out += [
+            "",
+            "# New inbound connections from the WAN that are not a port forward.",
+            "set firewall ipv4 forward filter rule 900 action drop",
+            f"set firewall ipv4 forward filter rule 900 inbound-interface name {WAN_DHCP_VIF}",
+            f"set firewall ipv4 forward filter rule 910 action drop",
+            f"set firewall ipv4 forward filter rule 910 inbound-interface name {WAN_PPPOE_IF}",
+            "set firewall ipv4 forward filter rule 910 description 'drop unsolicited WAN-to-LAN (Vodafone)'",
+            "set firewall ipv4 forward filter rule 900 description 'drop unsolicited WAN-to-LAN'",
+            "",
+        ]
+
+
+    # --- DHCP high-availability -------------------------------------------
+    # Without this BOTH boxes run kea on the same VLANs and race to answer the
+    # same broadcasts, handing different pool addresses to the same client.
+    # active-passive so only the primary serves, matching the VRRP shape.
+    dhcp_self, dhcp_peer = NODE_ADDRS[10][self_i], NODE_ADDRS[10][peer_i]
+    out += [
+        "",
+        "# --- DHCP high-availability --------------------------------",
+        "# Peers sync leases over TCP 647. Each subnet already carries a",
+        "# unique subnet-id (keyed on VLAN id), which kea HA requires.",
+        "set service dhcp-server high-availability mode active-passive",
+        f"set service dhcp-server high-availability status {'primary' if primary else 'secondary'}",
+        # The peer name must not collide with either system host-name.
+        f"set service dhcp-server high-availability name {DHCP_HA_NAME}",
+        f"set service dhcp-server high-availability source-address {dhcp_self}",
+        f"set service dhcp-server high-availability remote {dhcp_peer}",
+    ]
+
+    if conntrack_link:
+        # Stateful failover. Without it VRRP moves the address but every
+        # established connection dies, because the backup has no conntrack
+        # table. Needs the eth3 <-> eth3 cable physically present.
+        out += [
+            "",
+            "# --- conntrack-sync ----------------------------------------",
+            "# Dedicated point-to-point link: sync traffic must not compete",
+            "# with production, and must not die when the LAN does.",
+            f"set interfaces ethernet {CONNTRACK_IF} address {CONNTRACK_ADDRS[self_i]}",
+            f"set interfaces ethernet {CONNTRACK_IF} description 'conntrack-sync peer link'",
+            f"set service conntrack-sync interface {CONNTRACK_IF}",
+            "set service conntrack-sync failover-mechanism vrrp sync-group MAIN",
+            "set service conntrack-sync accept-protocol tcp",
+            "set service conntrack-sync accept-protocol udp",
+            "set service conntrack-sync accept-protocol icmp",
+            "set service conntrack-sync mcast-group 225.0.0.50",
+        ]
+
+    # DHCP + DNS, from the same generator labsim proved.
+    dhcp_lines, stats = unifi_to_vyos.build(inv, "prod")
+    expected = len(inv["reservations"])
+    if stats["mappings"] != expected:
+        raise SystemExit(
+            f"refusing to generate: {expected - stats['mappings']} reservation(s) "
+            f"missing -- every one must survive the cutover")
+    out += dhcp_lines
+    return out
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--priority", type=int, required=True,
+                    help="VRRP priority: 200 for the master, 100 for the backup")
+    ap.add_argument("--inventory", default=os.path.join(HERE, "export", "inventory.json"))
+    ap.add_argument("--raw-networkconf", default=os.path.join(HERE, "export", "rest_networkconf.json"))
+    ap.add_argument("--with-wan", action="store_true",
+                help="configure the WAN on this box. Only ONE of the pair may have\n                      it, because the cloned WAN MAC must be unique.")
+    ap.add_argument("--conntrack-link", action="store_true",
+                    help="emit conntrack-sync over the eth3 peer link. Requires the\n                          cable to be physically present on both boxes.")
+    ap.add_argument("-o", "--out")
+    ap.add_argument("--emit-secrets", metavar="PATH",
+                    help="write the PPPoE credential to PATH with mode 0600 and exit")
+    args = ap.parse_args()
+
+    with open(args.inventory) as fh:
+        inv = json.load(fh)
+    with open(args.raw_networkconf) as fh:
+        raw_nets = json.load(fh)
+
+    wan = next((n for n in raw_nets
+                if n.get("purpose") == "wan" and n.get("wan_username")), None)
+    if wan is None:
+        print("no WAN network with credentials found in the export", file=sys.stderr)
+        return 1
+
+    if args.emit_secrets:
+        fd = os.open(args.emit_secrets, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
+        with os.fdopen(fd, "w") as fh:
+            fh.write(f"WAN_PASSWORD='{wan.get('x_wan_password', '')}'\n")
+        # Re-assert the mode in case the file already existed with a wider one.
+        os.chmod(args.emit_secrets, 0o600)
+        mode = oct(os.stat(args.emit_secrets).st_mode & 0o777)
+        print(f"wrote {args.emit_secrets} (mode {mode}) for user {wan['wan_username']}",
+              file=sys.stderr)
+        return 0
+
+    lines = build_delta(inv, args.priority, wan["wan_username"], args.with_wan,
+                        args.conntrack_link)
+    text = "\n".join(lines) + "\n"
+
+    # Only a WAN-carrying delta has a credential to placeholder-substitute.
+    if args.with_wan and PLACEHOLDER not in text:
+        print("BUG: password placeholder missing from a WAN delta", file=sys.stderr)
+        return 1
+    if wan.get("x_wan_password") and wan["x_wan_password"] in text:
+        print("BUG: the WAN password leaked into the delta", file=sys.stderr)
+        return 1
+
+    n_set = sum(1 for l in lines if l.startswith("set "))
+    n_del = sum(1 for l in lines if l.startswith("delete "))
+    print(f"delta: {n_set} set, {n_del} delete, priority {args.priority}, "
+          f"{len(inv['reservations'])} reservations", file=sys.stderr)
+
+    if args.out:
+        with open(args.out, "w") as fh:
+            fh.write(text)
+        print(f"wrote {args.out}", file=sys.stderr)
+    else:
+        sys.stdout.write(text)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

migration/vyos-unifi-switch

@@ -0,0 +1,304 @@
+#!/bin/bash
+# Switch this VyOS box between passive (USG is the gateway) and active
+# (VyOS is the gateway). Installed at /config/vyos-unifi-switch, which
+# survives image upgrades, so it can be run from a local terminal or the
+# JetKVM console with no workstation and no internet.
+#
+#   vyos-unifi-switch            report the current mode
+#   vyos-unifi-switch vyos       take over: VIPs to .1, DHCP, DNS, PPPoE, NAT
+#   vyos-unifi-switch unifi      revert; the USG can then be reconnected
+#
+# READ THIS BEFORE THE CUTOVER
+#
+# `unifi` is the escape hatch. It runs no health checks, asks no questions and
+# has nothing that can refuse. If anything at all looks wrong, run it, then
+# plug the USG back in.
+#
+# `vyos` commits with commit-confirm. If it is not confirmed -- because the
+# health checks failed, or because you lost access, or because you walked away
+# -- the box returns to the saved configuration on its own. That requires
+# `system config-management commit-confirm action reload`; without it VyOS
+# REBOOTS instead, which on a gateway is an outage rather than an undo. The
+# script refuses to run if that setting is missing.
+set -uo pipefail
+
+MODES=/config/modes
+UNIFI_BOOT="$MODES/unifi.boot"
+DELTA="$MODES/to-vyos.commands"
+SECRETS=/config/wan-secrets
+MARKER="$MODES/current-mode"
+CONFIRM_MINUTES="${CONFIRM_MINUTES:-10}"
+OPRUN=/opt/vyatta/bin/vyatta-op-cmd-wrapper
+
+say()  { printf '[switch] %s\n' "$*"; }
+warn() { printf '[switch] WARNING: %s\n' "$*" >&2; }
+die()  { printf '[switch] ERROR: %s\n' "$*" >&2; exit 1; }
+
+# Run configuration commands in a real config session. Everything the caller
+# feeds in runs between `configure` and `exit`.
+run_cfg() {
+  local script rc
+  script="$(mktemp)"
+  {
+    echo 'source /opt/vyatta/etc/functions/script-template'
+    echo 'configure'
+    cat
+    echo 'exit'
+  } > "$script"
+  vbash "$script"; rc=$?
+  rm -f "$script"
+  return $rc
+}
+
+# Two traps live in this one function, both of which produced wrong answers
+# rather than errors:
+#   1. `show configuration commands` quotes values ("action 'reload'"), so the
+#      quotes have to go before matching or every value-bearing check fails.
+#   2. `... | grep -q` under `set -o pipefail` reports FAILURE even on a match:
+#      grep exits at the first hit, the producer takes SIGPIPE, and pipefail
+#      surfaces that. Whether it triggers depends on output size, so it fails
+#      intermittently. Match against a captured string instead of a pipeline.
+cfg_has() {
+  local out
+  out="$($OPRUN show configuration commands 2>/dev/null | tr -d "'")"
+  case "$out" in *"$1"*) return 0 ;; *) return 1 ;; esac
+}
+
+# ---------------------------------------------------------------- status ----
+current_mode() {
+  # The marker records intent; the running config is the truth. Report the
+  # config, and complain if the two disagree.
+  local live="unknown"
+  if cfg_has "set service dhcp-server"; then live="vyos"; else live="unifi"; fi
+  echo "$live"
+}
+
+show_status() {
+  local live marked
+  live="$(current_mode)"
+  marked="$(cat "$MARKER" 2>/dev/null || echo "never set")"
+  echo "host:          $(hostname)"
+  echo "mode:          $live   (marker: $marked)"
+  [ "$live" != "$marked" ] && [ "$marked" != "never set" ] && \
+    warn "marker disagrees with the running config -- trust the config"
+  echo "VRRP:"
+  $OPRUN show vrrp 2>/dev/null | sed -n '3,$p' | awk '{printf "               %-9s %-12s %s\n", $1, $2, $4}'
+  echo "DHCP server:   $(systemctl is-active isc-kea-dhcp4-server 2>/dev/null)"
+  echo "DNS forwarder: $(systemctl is-active pdns-recursor 2>/dev/null)"
+  echo "WAN (pppoe0):  $(ip -4 -br addr show pppoe0 2>/dev/null | awk '{print $2, $3}' || echo 'not present')"
+  echo "default route: $(ip -4 route show default 2>/dev/null | head -1 || echo none)"
+  echo "unsaved changes: $(cfg_unsaved)"
+}
+
+cfg_unsaved() {
+  if /usr/bin/config-mgmt compare >/dev/null 2>&1; then echo "no"; else echo "possibly - check 'compare saved'"; fi
+}
+
+# ------------------------------------------------------------- preflight ----
+require_files() {
+  [ -r "$UNIFI_BOOT" ] || die "missing $UNIFI_BOOT -- capture it while the USG is still the gateway"
+  [ -s "$UNIFI_BOOT" ] || die "$UNIFI_BOOT is empty"
+}
+
+require_reload_action() {
+  cfg_has "set system config-management commit-confirm action reload" && return 0
+  die "commit-confirm action is not 'reload'. Without it an unconfirmed switch
+       REBOOTS this box instead of reverting. Fix first:
+         configure
+         set system config-management commit-confirm action reload
+         commit; save"
+}
+
+# The single worst outcome available is two devices answering on the gateway
+# address. It costs one ARP probe to make that impossible.
+# Returns 0 (success) when something IS answering on a gateway address we are
+# about to claim -- i.e. "yes, still alive, do not proceed".
+usg_still_alive() {
+  local found=0 ip dev targets
+  targets="$(grep -oE "vrrp group [a-z0-9]+ address [0-9.]+" "$DELTA" 2>/dev/null | awk '{print $NF}')"
+  if [ -z "$targets" ]; then
+    # A delta with no VIPs cannot be probed, which means the single guard
+    # against two devices sharing a gateway address is inert. Refuse by
+    # default: an unprobeable delta on the real boxes is a broken delta, and
+    # "warn and continue" would let the one failure this script exists to
+    # prevent through unnoticed. The override is for the lab only.
+    if [ "${ALLOW_NO_VIP_DELTA:-0}" = "1" ]; then
+      warn "delta claims no VIPs; proceeding because ALLOW_NO_VIP_DELTA=1 (lab only)"
+      return 1
+    fi
+    die "this delta claims no VIPs, so the gateway-address guard cannot run.
+       On the real boxes that means a broken delta. If this really is a lab
+       run, re-invoke with ALLOW_NO_VIP_DELTA=1."
+  fi
+  for ip in $targets; do
+    dev="$(ip -4 route get "$ip" 2>/dev/null | sed -n 's/.* dev \([^ ]*\).*/\1/p' | head -1)"
+    if [ -n "$dev" ] && command -v arping >/dev/null 2>&1; then
+      # ARP is the right probe: it answers even when the host filters ICMP.
+      if arping -c 2 -w 3 -f -I "$dev" "$ip" >/dev/null 2>&1; then
+        warn "something already answers ARP on $ip (via $dev)"; found=1
+      fi
+    elif ping -c 2 -W 2 "$ip" >/dev/null 2>&1; then
+      warn "something already answers ICMP on $ip"; found=1
+    fi
+  done
+  [ "$found" -eq 1 ]
+}
+
+# ------------------------------------------------------------- to unifi -----
+to_unifi() {
+  require_files
+  say "reverting to unifi mode (USG is the gateway)"
+  run_cfg <<EOF || die "load/commit failed -- the box is unchanged, use the console"
+load $UNIFI_BOOT
+commit
+save
+EOF
+  echo "unifi" > "$MARKER"
+  say "done. The USG can be reconnected."
+  say "If it was already connected during this, nothing was disturbed."
+}
+
+# -------------------------------------------------------------- to vyos -----
+health_checks() {
+  local fails=0
+  _chk() { # name, command
+    if eval "$2" >/dev/null 2>&1; then say "  ok    $1"; else say "  FAIL  $1"; fails=$((fails+1)); fi
+  }
+  _chk "kea (DHCP) is running"      "systemctl is-active --quiet isc-kea-dhcp4-server"
+  _chk "DNS forwarder is running"   "systemctl is-active --quiet pdns-recursor"
+
+  # Only assert on the WAN if this delta actually brings one up. A delta with
+  # no PPPoE stanza is a lab/partial delta, and failing it on a missing
+  # pppoe0 would make the script untestable anywhere but the live cutover.
+  # Announced loudly, because a quietly skipped check is worse than no check.
+  # A box whose delta holds its WAN interfaces DOWN is the backup: it has no
+  # route out by design, and demanding one reverts a perfectly correct config.
+  # This tore down vyos002 on the first successful cutover -- vyos001 went live
+  # and its backup was judged unhealthy for lacking the WAN it is deliberately
+  # not carrying. Same mistake as requiring the failover line: checks that do
+  # not apply to the box being checked.
+  if grep -qE "^set interfaces (pppoe pppoe0|bonding bond0 vif 53) disable$" "$DELTA"; then
+    say "  note  this box holds its WAN down (backup); skipping WAN checks"
+  elif grep -qE "^set interfaces (pppoe|bonding bond0 vif 5)" "$DELTA"; then
+    # What matters is that SOME WAN works, not that every WAN works.
+    #
+    # This reverted a cutover that had genuinely succeeded. The 10 gig line came
+    # up on bond0.53 and the cloned MAC was handed the same public address the
+    # USG had (87.192.101.48); kea was serving real LAN clients at the same
+    # moment. The only failure was pppoe0 -- the Vodafone FAILOVER line -- and
+    # requiring it undid a working gateway.
+    #
+    # Written as [ -n "$(...)" ] rather than `... | grep -q` for the pipefail
+    # reason above: a pipeline ending in grep -q cannot be trusted here.
+    _chk "a default route exists"   '[ -n "$(ip -4 route show default)" ]'
+    _chk "internet reachable"       "ping -c2 -W3 8.8.8.8"
+    _chk "DNS resolves through us"  "getent hosts vyos.net"
+
+    # Informational only: report each WAN, fail on neither. A failover line
+    # being down is worth seeing, not worth reverting for.
+    for _w in pppoe0 bond0.53; do
+      if [ -n "$(ip -4 -br addr show "$_w" 2>/dev/null | awk '{print $3}')" ]; then
+        say "  ok    WAN $_w has an address (informational)"
+      else
+        say "  note  WAN $_w has no address (informational, not fatal)"
+      fi
+    done
+  else
+    warn "this delta configures no WAN -- skipping all WAN health checks."
+    warn "That is expected in the lab and WRONG for the real cutover."
+  fi
+  return $fails
+}
+
+to_vyos() {
+  require_files
+  require_reload_action
+  [ -r "$DELTA" ] || die "missing $DELTA"
+
+  if usg_still_alive; then
+    die "refusing: something is still answering on a gateway address.
+       Disconnect the USG first. Two devices on the same gateway IP is the
+       one failure this script exists to prevent."
+  fi
+
+  local pw="" tmp
+  if [ -r "$SECRETS" ]; then
+    # shellcheck disable=SC1090
+    . "$SECRETS"; pw="${WAN_PASSWORD:-}"
+  fi
+  [ -n "$pw" ] || warn "no WAN_PASSWORD in $SECRETS -- PPPoE will not authenticate"
+
+  tmp="$(mktemp)"; chmod 600 "$tmp"
+  sed "s|@@WAN_PASSWORD@@|${pw}|g" "$DELTA" | grep -vE '^\s*(#|$)' > "$tmp"
+
+  say "switching to vyos mode (this box becomes the gateway)"
+  say "commit-confirm: ${CONFIRM_MINUTES} min to confirm, else it reverts itself"
+
+  # commit-confirm is TWO steps, and doing only the first commits nothing:
+  # `config-mgmt commit_confirm` arms the revert timer, then a normal `commit`
+  # applies the candidate config. The interactive prompt lives in the first
+  # step, which is why it is invoked directly with -y instead of via the
+  # `commit-confirm` alias. IN_COMMIT_CONFIRM is what the real CLI sets, and
+  # the commit hooks look at it.
+  if ! run_cfg < <(printf 'load %s\n' "$UNIFI_BOOT"; cat "$tmp";
+                   printf 'sudo sg vyattacfg "/usr/bin/config-mgmt commit_confirm -y -t=%s"\n' "$CONFIRM_MINUTES";
+                   printf 'export IN_COMMIT_CONFIRM=t\ncommit\nunset IN_COMMIT_CONFIRM\n'); then
+    rm -f "$tmp"
+    die "commit-confirm failed. Nothing was applied; the box is still in its
+       previous mode. Run '$0 unifi' if you are unsure."
+  fi
+  rm -f "$tmp"
+
+  # Poll, do not sample once.
+  #
+  # A cutover attempt failed here on a fixed 25s wait. That is far too short for
+  # a WAN: PPPoE is PADI/PADO/PADR/PADS then LCP, auth and IPCP, routinely 15-30s
+  # by itself, and both lines had just been released by the USG seconds earlier.
+  # ISPs commonly hold the previous session and MAC binding for minutes before
+  # leasing to the "same" CPE again -- which is precisely what a cloned MAC looks
+  # like to them. One sample at 25s reported a healthy setup as broken and
+  # reverted it.
+  #
+  # There is still a deadline, because commit-confirm is running: stop well
+  # before it so the decision is ours rather than the timer's.
+  local budget="${HEALTH_BUDGET:-180}" waited=0 step=15
+  say "committed. Polling health for up to ${budget}s (commit-confirm has ${CONFIRM_MINUTES} min)..."
+  while :; do
+    sleep "$step"; waited=$(( waited + step ))
+    if health_checks >/dev/null 2>&1; then
+      say "healthy after ${waited}s"
+      break
+    fi
+    if [ "$waited" -ge "$budget" ]; then
+      say "still unhealthy after ${waited}s -- final check:"
+      break
+    fi
+    say "  not healthy yet at ${waited}s, still waiting..."
+  done
+
+  say "health checks:"
+  if health_checks; then
+    say "all checks passed -- confirming"
+    /usr/bin/config-mgmt confirm >/dev/null 2>&1 || die "confirm failed; it will revert on its own shortly"
+    run_cfg <<'EOF' || warn "save failed -- config is live but will not survive a reboot"
+save
+EOF
+    echo "vyos" > "$MARKER"
+    say "vyos mode is live and saved."
+  else
+    warn "health checks FAILED -- reverting now rather than waiting out the timer"
+    /usr/bin/config-mgmt revert_soft >/dev/null 2>&1 \
+      || warn "revert_soft failed; the commit-confirm timer will still fire within ${CONFIRM_MINUTES} min"
+    echo "unifi" > "$MARKER"
+    die "reverted to the previous configuration. Reconnect the USG.
+       Check: ip addr show pppoe0; journalctl -u pppd; $0 status"
+  fi
+}
+
+# ----------------------------------------------------------------- main -----
+case "${1:-status}" in
+  vyos)   to_vyos ;;
+  unifi)  to_unifi ;;
+  status) show_status ;;
+  *)      echo "usage: $(basename "$0") [vyos|unifi|status]" >&2; exit 2 ;;
+esac

migration/vyos002-catch.sh

@@ -0,0 +1,85 @@
+#!/bin/bash
+# Arm BEFORE powering vyos002 on. Strips the eth2 address the moment the box is
+# reachable, then installs the VRRP health-check.
+#
+# Why this exists: vyos002 boots with `interfaces ethernet eth2 address
+# 192.168.8.144/23` still in config.boot -- the same subnet as bond0.2. Linux
+# answers ARP for any local address out of any interface on that L2, so eth2
+# answers for addresses that bond0.2 is supposed to route, and traffic lands on
+# a port that does not route it. That is the 2026-09-02 cluster outage.
+#
+# A human "jumping on it fast" loses this race more often than not; the box is
+# reachable within a second or two of the interfaces coming up. This polls at
+# 1s and commits the moment it gets in.
+#
+#   ./vyos002-catch.sh            arm and wait (Ctrl-C to disarm)
+#
+# Bounded risk while you wait, worth knowing: vyos002 comes up BACKUP (priority
+# 100, no-preempt, vyos001 healthy MASTER), so it does NOT hold 192.168.8.1 and
+# the GATEWAY cannot be poisoned. The exposure is its own bond0.2 address, which
+# is survivable. The unbounded case is it becoming MASTER while eth2 is present
+# -- which is exactly what the health-check in step 2 prevents.
+set -uo pipefail
+
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PW="${VYOS_PW:-vyos}"
+# Management first because it comes up with the box; LoT is the fallback and is
+# L2-direct from this workstation (see RECOVERY-CARD-vlan1-move.md).
+TARGETS=("192.168.1.253" "10.0.1.253")
+SSH_OPTS=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
+          -o LogLevel=ERROR -o ConnectTimeout=2 -o PreferredAuthentications=password)
+
+log() { printf '\033[36m[catch %s]\033[0m %s\n' "$(date +%T)" "$*"; }
+
+on() { timeout 12 sshpass -p "$PW" ssh "${SSH_OPTS[@]}" "vyos@$1" "$@"; }
+
+log "armed -- polling ${TARGETS[*]} every 1s. Power on vyos002 now."
+HOST=""
+while [ -z "$HOST" ]; do
+  for t in "${TARGETS[@]}"; do
+    if timeout 4 sshpass -p "$PW" ssh "${SSH_OPTS[@]}" "vyos@$t" true 2>/dev/null; then
+      HOST="$t"; break
+    fi
+  done
+  [ -z "$HOST" ] && sleep 1
+done
+log "CAUGHT on $HOST -- stripping eth2 address"
+
+# Step 1, on its own commit: get the address off eth2 before anything else. Any
+# extra command in this commit is extra seconds of exposure.
+timeout 90 sshpass -p "$PW" ssh "${SSH_OPTS[@]}" "vyos@$HOST" 'vbash -s' <<'EOF' 2>&1 | tail -3
+source /opt/vyatta/etc/functions/script-template
+delete interfaces ethernet eth2 address
+commit
+echo "ETH2_RC=$?"
+save
+exit
+EOF
+log "eth2 address removed"
+
+# Step 2: the health-check. Without it this box can hold every floating IP while
+# having no WAN -- the outage itself. Copy the script BEFORE referencing it, or
+# the commit succeeds and the check silently never passes.
+timeout 30 sshpass -p "$PW" scp "${SSH_OPTS[@]}" \
+  "$HERE/vrrp-wan-health" "vyos@$HOST:/tmp/vrrp-wan-health" >/dev/null 2>&1 \
+  && log "health-check script copied" || log "WARN: scp failed -- step 2 will be skipped"
+
+timeout 90 sshpass -p "$PW" ssh "${SSH_OPTS[@]}" "vyos@$HOST" 'vbash -s' <<'EOF' 2>&1 | tail -3
+sudo install -o root -g vyattacfg -m 0775 /tmp/vrrp-wan-health /config/vrrp-wan-health
+source /opt/vyatta/etc/functions/script-template
+set high-availability vrrp sync-group MAIN health-check script '/config/vrrp-wan-health'
+set high-availability vrrp sync-group MAIN health-check interval '5'
+set high-availability vrrp sync-group MAIN health-check failure-count '3'
+commit
+echo "HEALTH_RC=$?"
+save
+exit
+EOF
+log "health-check installed"
+
+echo
+log "=== state ==="
+timeout 30 sshpass -p "$PW" ssh "${SSH_OPTS[@]}" "vyos@$HOST" \
+  'echo "-- eth2 (must show no inet) --"; ip -4 addr show eth2 2>/dev/null | grep inet || echo "   none"
+   echo "-- health-check --"; sudo /config/vrrp-wan-health; echo "   exit=$? (non-zero = no WAN = refuses the VIPs, which is CORRECT and safe)"
+   echo "-- vrrp --"; /opt/vyatta/bin/vyatta-op-cmd-wrapper show vrrp' 2>&1

migration/vyos002-return.conf

@@ -0,0 +1,37 @@
+# vyos002 — everything that must land before it is trusted on the network.
+#
+# Apply order matters only in that this is ONE commit: the eth2 address and the
+# missing health-check are the two defects that caused the 2026-09-02 outage, and
+# neither should survive a single reboot window.
+#
+#   ssh vyos@192.168.1.253   (or 10.0.1.253 — L2-direct, see RECOVERY-CARD)
+#   configure; <paste>; commit; save
+
+# 1. The ARP poisoner. eth2 is the 1G copper NIC on US24 port 16, native VLAN 2,
+#    and it held 192.168.8.144/23 -- the same subnet as bond0.2. Two interfaces
+#    answering for one subnet is what hijacked 192.168.8.1 and took the cluster
+#    down: eth2's MAC answered while bond0.2's MAC routed.
+#    Origin: eth2 was `address dhcp`, a kea reservation for the ROUTER'S OWN NIC
+#    handed it .144, and a CLI commit froze it static.
+delete interfaces ethernet eth2 address
+
+# 2. The health-check. Without it this box can hold every floating IP while
+#    having no WAN at all -- the outage itself. It goes on the SYNC GROUP; VyOS
+#    rejects it per-group.
+#    /config/vrrp-wan-health must be copied over FIRST (from migration/) and be
+#    chmod +x, or the commit succeeds and the check silently never passes.
+set high-availability vrrp sync-group MAIN health-check script '/config/vrrp-wan-health'
+set high-availability vrrp sync-group MAIN health-check interval '5'
+set high-availability vrrp sync-group MAIN health-check failure-count '3'
+
+# 3. Management onto a tagged sub-interface, matching vyos001 (kea #1117).
+#    Pair this with USW Aggregation port 3 (LAG 3+4) -> Native VLAN = None.
+#    Until that switch change lands, leave these three commented out: vyos002
+#    can run untagged on bond0 while vyos001 runs tagged -- one VLAN is one
+#    broadcast domain, and the coexistence was proven in labsim.
+# set interfaces bonding bond0 vif 1 address '192.168.1.253/24'
+# set interfaces bonding bond0 vif 1 description 'management'
+# delete interfaces bonding bond0 address
+# set firewall group interface-group LAN interface 'bond0.1'
+# delete firewall group interface-group LAN interface 'bond0'
+# set high-availability vrrp group native interface 'bond0.1'

pulumi-vyos/Pulumi.labsim.yaml

@@ -0,0 +1 @@
+encryptionsalt: v1:tMNG4q79HiI=:v1:K35iEOw3pgyukCr6:LmKO37/jghjyT2sLuoLmCQIDPOIsgA==

pulumi-vyos/Pulumi.yaml

@@ -0,0 +1,3 @@
+name: vyos-proto
+runtime: nodejs
+description: Prototype — VyOS config subtrees as Pulumi resources, with commit-confirm

pulumi-vyos/README.md

@@ -0,0 +1,72 @@
+# VyOS as Pulumi resources — prototype
+
+Proves that VyOS config can be managed from the same Pulumi plan as the
+Kubernetes side, **without** giving up the safety property that matters on a
+gateway: a config push that breaks your access undoes itself.
+
+## Why not the community Terraform providers
+
+They are better than expected. `foltik/vyos`'s `vyos_config_block_tree` flattens
+an entire subtree into one payload and sends **one POST to `/configure`** —
+so one resource is one commit, not one commit per config line. That worry was
+unfounded.
+
+What they do *not* do is send `confirm_time`. The payload is only
+`op`/`path`/`value`, so every change is an unprotected commit. On a router you
+reach *through* the router, that is the difference between a mistake and an
+outage.
+
+## What the VyOS API actually supports
+
+Read from `rest/models.py` and `rest/routers.py` on the box, then verified by
+hand against a live router:
+
+- **Batching**: a list of operations in one request, applied as one commit.
+- **commit-confirm**: `confirm_time` on the request starts the revert timer.
+  Response says `Initialized commit-confirm; N minutes to confirm before reload`.
+
+Three details that cost time and are easy to get wrong:
+
+1. **`confirm_time` is only read when the body parses as `ConfigureListModel`** —
+   i.e. `{"commands": [...], "confirm_time": N}`. A **bare array** is accepted
+   and committed happily with **no timer armed**. It looks identical to success.
+   The resource therefore checks the response actually mentions commit-confirm
+   and refuses to continue if it does not.
+2. **There is no `/confirm` endpoint.** Confirming is an op on `/configure`.
+3. **Confirm still requires a `path` field**, even though it ignores it — the
+   Union resolves to `ConfigureModel`, which mandates `path`. Without it you get
+   `missing 'path' field` and the timer keeps running.
+
+## Shape
+
+One resource per **subtree**, not per line:
+
+```ts
+new VyosConfigTree("dns-forwarding", {
+    host, apiKey,
+    path: ["service", "dns", "forwarding"],
+    commands: [["cache-size", "20000"], ["name-server", "8.8.8.8"]],
+    confirmMinutes: 2,
+});
+```
+
+Apply is `delete <path>` + all the `set`s in one request, so the result is the
+declared state rather than a merge with whatever was there — which is what makes
+`pulumi up` converge instead of accumulate.
+
+## Verified on labsim
+
+- `pulumi up` create and update both land in ~6s, each a single
+  commit-confirmed transaction; update shows `[diff: ~commands]`.
+- Auto-revert observed: an unconfirmed commit returned the router to its saved
+  config on its own.
+- `pulumi destroy` removes the subtree.
+
+## Not done
+
+- The API is HTTP with a self-signed certificate and `rejectUnauthorized: false`.
+  Bind it to the management VLAN or the peer link and install a real
+  certificate before this goes near production.
+- No `refresh`/drift detection yet: `read` is not implemented, so out-of-band
+  changes are not noticed until the next `up` overwrites them.
+- The cutover itself should still use `vyos-unifi-switch`. This is for day-2.

pulumi-vyos/index.ts

@@ -0,0 +1,27 @@
+import * as pulumi from "@pulumi/pulumi";
+import { VyosConfigTree } from "./vyosConfigTree";
+
+const cfg = new pulumi.Config();
+const host = cfg.get("host") ?? "172.31.1.252";
+const apiKey = cfg.get("apiKey") ?? "labsim-proto-key";
+
+// One subtree, one resource, one commit. This is the shape a BGP change would
+// take: edit the commands array, `pulumi up`, and it lands as a single
+// commit-confirmed transaction alongside whatever Kubernetes resources changed
+// in the same plan.
+const dnsForwarding = new VyosConfigTree("dns-forwarding", {
+    host, apiKey,
+    path: ["service", "dns", "forwarding"],
+    commands: [
+        ["cache-size", "20000"],
+        ["listen-address", "172.31.10.1"],
+        ["allow-from", "172.31.10.0/23"],
+        ["name-server", "8.8.8.8"],
+        ["name-server", "8.8.4.4"],
+        ["name-server", "1.1.1.1"],
+    ],
+    confirmMinutes: 2,
+    save: true,
+});
+
+export const managed = dnsForwarding.id;

pulumi-vyos/package.json

@@ -0,0 +1,11 @@
+{
+  "name": "vyos-proto",
+  "main": "index.ts",
+  "devDependencies": {
+    "@types/node": "^22",
+    "typescript": "^5.9.3"
+  },
+  "dependencies": {
+    "@pulumi/pulumi": "^3.140.0"
+  }
+}

pulumi-vyos/tsconfig.json

@@ -0,0 +1,2 @@
+{ "compilerOptions": { "strict": true, "target": "es2020", "module": "commonjs",
+  "moduleResolution": "node", "skipLibCheck": true, "esModuleInterop": true } }

pulumi-vyos/vyosConfigTree.ts

@@ -0,0 +1,205 @@
+import * as pulumi from "@pulumi/pulumi";
+import * as https from "https";
+
+/**
+ * A VyOS config subtree, managed as ONE Pulumi resource.
+ *
+ * Why a dynamic provider rather than the community Terraform providers: they
+ * batch correctly (one `vyos_config_block_tree` becomes a single POST to
+ * /configure, so one commit) but they never send `confirm_time`. VyOS's own API
+ * supports it -- `ConfigureListModel.confirm_time`, and "A non-zero confirm_time
+ * will start commit-confirm timer on commit" in rest/routers.py -- so a config
+ * push that breaks your access to the router can undo itself. On a gateway that
+ * is the difference between a mistake and an outage, and it is worth ~150 lines
+ * to keep.
+ *
+ * The unit of change is a SUBTREE, not a line. `protocols bgp` is one resource;
+ * so is `service dhcp-server`. That keeps one Pulumi resource == one commit,
+ * rather than turning 300 config lines into 300 commits with no ordering
+ * guarantee and no way to revert them as a unit.
+ */
+
+export interface VyosConfigTreeArgs {
+    /** Router address, e.g. "10.0.1.252". */
+    host: pulumi.Input<string>;
+    /** API key from `set service https api keys id <n> key <k>`. */
+    apiKey: pulumi.Input<string>;
+    /** Subtree root as a path array, e.g. ["protocols", "bgp"]. */
+    path: pulumi.Input<string[]>;
+    /**
+     * Desired state of the subtree: `set` command suffixes relative to `path`,
+     * each already split into path components with the value last.
+     */
+    commands: pulumi.Input<string[][]>;
+    /**
+     * Minutes before an unconfirmed commit reverts itself. 0 disables it.
+     * Non-zero is strongly preferred for anything reachable only through the
+     * router being changed.
+     */
+    confirmMinutes?: pulumi.Input<number>;
+    /** Persist to config.boot after a successful confirm. */
+    save?: pulumi.Input<boolean>;
+}
+
+interface Inputs {
+    host: string;
+    apiKey: string;
+    path: string[];
+    commands: string[][];
+    confirmMinutes: number;
+    save: boolean;
+}
+
+// Kept inside the module so the dynamic provider can serialise it.
+function apiCall(
+    host: string, apiKey: string, endpoint: string,
+    form: Record<string, string>,
+): Promise<any> {
+    const body = Object.entries(form)
+        .map(([k, v]) => `${encodeURIComponent(k)}=${encodeURIComponent(v)}`)
+        .join("&");
+    return new Promise((resolve, reject) => {
+        const req = https.request({
+            host, port: 443, path: `/${endpoint}`, method: "POST",
+            // VyOS ships a self-signed certificate by default. Verification is
+            // disabled deliberately; if this ever leaves a trusted segment,
+            // install a real certificate and turn it back on.
+            rejectUnauthorized: false,
+            headers: {
+                "Content-Type": "application/x-www-form-urlencoded",
+                "Content-Length": Buffer.byteLength(body),
+            },
+            timeout: 120_000,
+        }, (res) => {
+            let data = "";
+            res.on("data", (c) => (data += c));
+            res.on("end", () => {
+                try {
+                    const parsed = JSON.parse(data);
+                    if (parsed.success === false) {
+                        reject(new Error(`VyOS API: ${parsed.error ?? data}`));
+                    } else {
+                        resolve(parsed);
+                    }
+                } catch {
+                    reject(new Error(`VyOS API returned non-JSON: ${data.slice(0, 300)}`));
+                }
+            });
+        });
+        req.on("error", reject);
+        req.on("timeout", () => { req.destroy(); reject(new Error("VyOS API timed out")); });
+        req.write(body);
+        req.end();
+    });
+}
+
+/**
+ * Replace a subtree: delete it, then set the desired state, in ONE request so
+ * it is a single commit. Deleting first makes the result the declared state
+ * rather than a merge with whatever was there -- which is what makes `pulumi
+ * up` converge instead of accumulating.
+ */
+async function applyTree(i: Inputs, desired: string[][]): Promise<void> {
+    const commands: any[] = [{ op: "delete", path: i.path }];
+    for (const c of desired) {
+        commands.push({ op: "set", path: [...i.path, ...c] });
+    }
+
+    // The payload MUST be {commands: [...], confirm_time: N}, not a bare array.
+    // VyOS only reads confirm_time when the body parses as ConfigureListModel
+    // (`if isinstance(data, (ConfigureModel, ConfigureListModel, ...))` in
+    // rest/routers.py). A bare array is accepted and committed happily -- with
+    // NO timer armed, silently discarding the safety net. Verified both ways:
+    // bare array gives no "Initialized commit-confirm" in the response, the
+    // object form returns "Initialized commit-confirm; N minutes to confirm".
+    const payload: any = { commands };
+    if (i.confirmMinutes > 0) payload.confirm_time = i.confirmMinutes;
+
+    const res = await apiCall(i.host, i.apiKey, "configure",
+                              { key: i.apiKey, data: JSON.stringify(payload) });
+
+    if (i.confirmMinutes > 0) {
+        // Refuse to continue if the timer did not actually arm -- otherwise a
+        // silent fallback to an unprotected commit is exactly the failure this
+        // resource exists to prevent.
+        const said = String((res && res.data) || "");
+        if (!said.includes("commit-confirm")) {
+            throw new Error(
+                "commit-confirm was requested but VyOS did not arm a timer " +
+                `(response: ${said.trim() || "<empty>"}). Refusing to proceed.`);
+        }
+    }
+
+    if (i.confirmMinutes > 0) {
+        // The commit landed but is on a timer. Reaching the API again proves
+        // the box is still answering *after* the change -- the only evidence
+        // worth confirming on. If this throws we deliberately do NOT confirm,
+        // and the router reverts itself.
+        await apiCall(i.host, i.apiKey, "retrieve", {
+            key: i.apiKey,
+            data: JSON.stringify({ op: "showConfig", path: [] }),
+        });
+        // There is no /confirm endpoint -- confirm is an op on /configure, and
+        // it requires a `path` field even though it ignores it (the Union
+        // resolves to ConfigureModel, which mandates path). Without it the API
+        // answers "missing 'path' field" and the timer keeps running.
+        await apiCall(i.host, i.apiKey, "configure", {
+            key: i.apiKey,
+            data: JSON.stringify({ op: "confirm", path: [] }),
+        });
+    }
+
+    if (i.save) {
+        await apiCall(i.host, i.apiKey, "config-file", {
+            key: i.apiKey, data: JSON.stringify({ op: "save" }),
+        });
+    }
+}
+
+const provider: pulumi.dynamic.ResourceProvider = {
+    async create(inputs: Inputs) {
+        await applyTree(inputs, inputs.commands);
+        return { id: `${inputs.host}:${inputs.path.join("/")}`, outs: inputs };
+    },
+
+    async update(_id, _old: Inputs, news: Inputs) {
+        await applyTree(news, news.commands);
+        return { outs: news };
+    },
+
+    async delete(_id, props: Inputs) {
+        const form: Record<string, string> = {
+            key: props.apiKey,
+            data: JSON.stringify([{ op: "delete", path: props.path }]),
+        };
+        if (props.confirmMinutes > 0) form.confirm_time = String(props.confirmMinutes);
+        await apiCall(props.host, props.apiKey, "configure", form);
+        if (props.confirmMinutes > 0) {
+            await apiCall(props.host, props.apiKey, "configure",
+                { key: props.apiKey, data: JSON.stringify({ op: "confirm", path: [] }) });
+        }
+    },
+
+    async diff(_id, olds: Inputs, news: Inputs) {
+        const changed =
+            JSON.stringify(olds.commands) !== JSON.stringify(news.commands) ||
+            JSON.stringify(olds.path) !== JSON.stringify(news.path) ||
+            olds.host !== news.host;
+        return {
+            changes: changed,
+            // Changing which subtree or which router is a different resource.
+            replaces: olds.host !== news.host ||
+                      JSON.stringify(olds.path) !== JSON.stringify(news.path) ? ["path"] : [],
+        };
+    },
+};
+
+export class VyosConfigTree extends pulumi.dynamic.Resource {
+    constructor(name: string, args: VyosConfigTreeArgs, opts?: pulumi.CustomResourceOptions) {
+        super(provider, name, {
+            confirmMinutes: 5,
+            save: true,
+            ...args,
+        }, opts);
+    }
+}