vyos: WAN follows VRRP mastership, rehearsed in labsim

The ISP is consumer with one static IP, so "both routers hold WAN" is not
available: the 10 gig lease is anchored to a cloned MAC and the PPPoE line to a
single credential. The WAN therefore has to move with mastership.

Proven end to end in the sim: the secondary was promoted, took the WAN, got a
lease, installed a default route, and a LAN VM reached the internet through it
(3/3, 9ms). Demoting released it -- link down, no address. Rebooting the master
converged correctly too: it came back BACKUP with the WAN disabled while the
peer kept it.

The rehearsal earned its keep four times over, and none of these were visible
from reading the docs:

  - `transition-script` alone is NOT safe to hang internet on. VyOS delivers it
    through keepalived-fifo.py, and on one promotion that helper logged NOTHING
    while Keepalived_vrrp logged all six instances entering MASTER and the
    built-in notify_master for conntrack-sync ran normally. The result was a
    router holding every VIP with no WAN -- the 2026-09-02 outage, recreated by
    the mechanism meant to prevent it. Hence vrrp-wan-reconcile on a 30s timer:
    the scripts give speed, the timer gives correctness.

  - The health check must ask REALITY, not a marker. Keying "am I master" on a
    /run file written by the transition script meant that when the script did
    not run, the router believed it was backup, passed the check, and kept the
    VIPs it could not serve. It now asks whether the VIP is actually on the box.

  - The old address-based check DEADLOCKED this design: may-I-be-master required
    already having WAN, and only the master gets WAN. That is why vyos002 sat in
    FAULT for ever -- the safety check had silently removed the redundancy it
    existed to protect.

  - script-template must be the FIRST thing a script does. Sourced after an if,
    an exec and a mkdir it terminated the script inside the source, rc=0, no
    output: the reconciler reported success having done nothing. Hence the split
    into vrrp-wan-apply, matching the shape /config/vyos-known-good already uses.

Two hazards found and handled rather than discovered in production:

  - A `configure` session whose process dies leaks a unionfs mount under
    /opt/vyatta/config/tmp, and one of those holds the commit lock -- after
    which every commit fails, including the manual one you try to fix it with.
    A 30s job that can leak one per failure wedges the box on its own, so the
    reconciler reaps dead sessions before it starts. It cleared 8 on the sim.

  - Any `save` while a box is master persists the enabled WAN into config.boot,
    so a reboot would claim the shared MAC regardless of VRRP state. Observed:
    an ordinary console-apply did exactly this. config.boot must keep `disable`
    on BOTH routers; the model asserts it and vyos:verify reports it as drift.

NOT yet applied to production, and it should not be until the remaining item is
settled: a clean, deliberately-triggered failover has been seen via reboot, but
`restart vrrp` twice failed to move mastership at all, so the trigger for a
planned failover is still unproven.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH

.gitignore

@@ -31,3 +31,9 @@ node_modules/
 # Asahi build artifacts (large)
 bastion/.asahi-cache/
 bastion/asahi-repo/*.zip
+
+# Regenerated by labsim/dualstack-lab.sh; derived state, not source.
+labsim/dualstack-evidence/
+
+# Runtime snapshots from labsim/cilium-ipam-switch.sh
+labsim/.ipam-switch-state/

bastion/src/modules/modules/k3s/src/install.ts

@@ -215,7 +215,9 @@ echo "  Using network device: \$DEFAULT_DEV"
 
 KUBECONFIG=/etc/rancher/k3s/k3s.yaml cilium install \\
   --set kubeProxyReplacement=true \\
-  --set ipam.mode=kubernetes \\
+  --set ipam.mode=cluster-pool \\
+  --set ipam.operator.clusterPoolIPv4PodCIDRList='{10.42.0.0/16}' \\
+  --set ipam.operator.clusterPoolIPv4MaskSize=24 \\
   --set devices="\$DEFAULT_DEV" \\
   --set nodePort.directRoutingDevice="\$DEFAULT_DEV"
 

bastion/src/modules/modules/k3s/src/operations/cilium.ts

@@ -47,7 +47,9 @@ export const installCilium: Operation = async (ctx): Promise<OperationResult> =>
   const installResult = await ctx.ssh.exec(
     `KUBECONFIG=/etc/rancher/k3s/k3s.yaml cilium install \
       --set kubeProxyReplacement=true \
-      --set ipam.mode=kubernetes \
+      --set ipam.mode=cluster-pool \
+      --set ipam.operator.clusterPoolIPv4PodCIDRList='{10.42.0.0/16}' \
+      --set ipam.operator.clusterPoolIPv4MaskSize=24 \
       --set k8sServiceHost=127.0.0.1 \
       --set k8sServicePort=6444 \
       --set cni.exclusive=false \

bastion/tests/integration/k3s-single-node.test.ts

@@ -161,7 +161,7 @@ EOF'
       CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
       curl -L --fail --silent "https://github.com/cilium/cilium-cli/releases/download/\${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz" | sudo tar xz -C /usr/local/bin
       DEFAULT_DEV=$(ip -4 route show default | awk '{print $5}' | head -1)
-      sudo KUBECONFIG=/etc/rancher/k3s/k3s.yaml cilium install --set kubeProxyReplacement=true --set ipam.mode=kubernetes --set devices=$DEFAULT_DEV --set nodePort.directRoutingDevice=$DEFAULT_DEV
+      sudo KUBECONFIG=/etc/rancher/k3s/k3s.yaml cilium install --set kubeProxyReplacement=true --set ipam.mode=cluster-pool --set ipam.operator.clusterPoolIPv4PodCIDRList='{10.42.0.0/16}' --set ipam.operator.clusterPoolIPv4MaskSize=24 --set devices=$DEFAULT_DEV --set nodePort.directRoutingDevice=$DEFAULT_DEV
     `.trim(), "cilium install", { keyPath: sshKeyPath, timeout: 120_000 });
 
     log("Waiting for Cilium to be ready...");

labsim/README.md

@@ -186,6 +186,63 @@ why any of it was shaped the way it was.
   client does not redial promptly. After any change there, check `pppoe0` on
   the router and `sudo systemctl restart ppp@pppoe0` if it is missing.
 
+## The trunk carries every VLAN tagged, including Management
+
+There is deliberately **no native/untagged VLAN** on the trunks to the routers,
+and Management lives on `bond0.1`, not on the bare `bond0`.
+
+A native VLAN is what puts a subnet on the bond **parent** while every other
+VLAN sits on a sub-interface of it. With `dhcp-socket-type: raw`, kea receives
+each tagged frame *twice* — once on the sub-interface and once on the parent —
+and answers from the parent's pool as well (ISC Kea
+[#1117](https://gitlab.isc.org/isc-projects/kea/-/issues/1117)). A client on
+VLAN 3 gets two OFFERs and keeps whichever arrives first:
+
+```
+bond0.3 :  172.31.3.252 → 172.31.3.11   correct
+bond0   :  172.31.1.252 → 172.31.1.8    UNTAGGED, Management pool, wrong
+```
+
+`./labsim-vlan-leak-test.sh` makes one client on a tagged VLAN send a DISCOVER
+and captures on the parent and the sub-interface at once. The verdict is how
+many OFFERs the **server** emitted and from which subnets — deliberately not
+"did the client get the right address", because a client picking correctly is
+exactly how this hid. Both orderings were observed across runs, so a passing
+client proves nothing.
+
+```sh
+./labsim-vlan-leak-test.sh --vlan 3           # PASS on the current shape
+LABSIM_NATIVE_VLAN=1 ./router-up.sh           # restore the old shape...
+./labsim-vlan-leak-test.sh --vlan 3           # ...and it FAILs again
+```
+
+Three things this cost, all of which apply to production:
+
+- **Kea must be restarted after the address moves.** VyOS does not restart it
+  for an interface address change, so it keeps a raw socket bound with the old
+  address and the bug survives the fix. In the sim kea had been running since
+  16 Aug; the first post-fix test failed for this reason alone and looked like
+  the fix simply not working.
+- **The firewall interface-group must move too.** `interface-group LAN` named
+  the bare `bond0`; with a default-deny ruleset, moving the address without
+  moving the group drops every management session and all VLAN 1 routing.
+- **Duplicate delivery does not stop.** #1117 says only that there is no longer
+  a subnet on the parent to match, and that is exactly what happens: two replies
+  per DISCOVER, both now from the correct pool. Harmless, but do not read a
+  duplicate as a failure.
+
+### Tagged and untagged Management coexist
+
+Verified directly, and it is what makes the production cutover a rolling change
+rather than an outage: with the primary still untagged on `bond0` and the
+secondary already tagged on `bond0.1`, both routers were reachable, the VIP
+stayed up and a VLAN 1 client kept its gateway. One VLAN is one broadcast
+domain regardless of how each port tags it, so the two firewalls can be
+converted one at a time. See `migration/MANAGEMENT-VLAN-TAGGED.md`.
+
+`./vlan1-move-monitor.sh` logs VIP/router liveness once a second during the
+change, because VRRP reconverges and leaves no trace of who held the VIP.
+
 ## Notes for whoever extends this
 
 Things that cost time the first time round, all verified on this image:
@@ -206,7 +263,14 @@ Things that cost time the first time round, all verified on this image:
 
 ## Not modelled (yet)
 
-VLANs are separate L2 segments rather than one 802.1Q trunk, so this exercises
-inter-VLAN routing but not a `bond0.<vif>` trunk config specifically. A router
-VM would attach one NIC per VLAN. Adding a tagged-trunk variant is the obvious
-next step if the bond/vif config itself needs testing.
+- **The secondary's bond was fiction until 2026-09-02.** `ovs_bond_router`'s
+  "already bonded, nothing to do" check compared only the trunk VLAN list, not
+  the membership. Restarting a VM recreates its taps under new names, so the
+  bond sat there holding two interfaces that no longer existed while the router's
+  real taps ran in the bridge as two *independent* ports — no LACP, and carrying
+  libvirt's own portgroup VLAN config rather than the bond's. It reconciles
+  membership now, but the lesson generalises: a sim that reports success is not
+  the same as a sim that models the thing.
+- **`labsim-vyos` has a third NIC** on libvirt's `default` network (the scaffold
+  uplink, see `--drop-scaffold`). The tap count is filtered to `$OVS_NET` for
+  that reason; an unfiltered count is 3 and silently skipped the primary's bond.

labsim/cilium-ipam-switch.sh

@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+# Procedure around a Cilium IPAM mode change. Works against any cluster, so the
+# rehearsal in labsim and the real thing in production run the SAME steps.
+#
+# It deliberately does NOT change the mode itself. In labsim that is `helm
+# upgrade`; in production Pulumi owns the release and a script racing it would
+# just reintroduce drift. What this owns is everything around the apply -- the
+# evidence, the deadlock, and the verdict.
+#
+#   ./cilium-ipam-switch.sh preflight   record what the cluster looks like now
+#   ./cilium-ipam-switch.sh unstick     break the agent-not-ready taint deadlock
+#   ./cilium-ipam-switch.sh verify      compare against preflight, report renumbering
+#
+#   KUBECONFIG=... ./cilium-ipam-switch.sh preflight
+#
+# Whether a recycle is needed is CONDITIONAL, and `verify` is what decides it.
+#
+# The operator does not preserve which node held which /24 -- it adopts whatever
+# CiliumNode.spec.ipam.podCIDRs already says. So:
+#
+#   * If CiliumNode already agrees with node.spec.podCIDRs on every node -- which
+#     is the case for any cluster that has only ever run ipam=kubernetes, because
+#     the operator syncs one from the other -- the pool adopts the existing
+#     allocation, no node is renumbered, and NO pod recycle is needed. Verified
+#     on the 3-node labsim cluster: CIDRs unchanged, nothing stranded, the only
+#     blip was the cilium DaemonSet restarting itself.
+#
+#   * If the two sources DISAGREE, nodes can swap /24s. Their running pods keep
+#     addresses that no longer fall inside the node's range, every other node
+#     routes that prefix to the wrong node, and those pods go unreachable
+#     cross-node while still showing Running. Then a full recycle is mandatory.
+#
+# Do not skip `verify` on the assumption of the good case. Run it and read it.
+set -uo pipefail
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+STATE="${STATE:-$SCRIPT_DIR/.ipam-switch-state}"
+K="kubectl"
+
+say()  { printf '\033[0;36m[ipam]\033[0m %s\n' "$*"; }
+warn() { printf '\033[1;33m[ipam]\033[0m %s\n' "$*" >&2; }
+
+snapshot() {
+  echo "## nodes"
+  $K get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.podCIDRs}{"\n"}{end}' 2>/dev/null
+  echo "## ciliumnodes"
+  $K get ciliumnode -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.ipam.podCIDRs}{"\n"}{end}' 2>/dev/null
+  echo "## pods"
+  $K get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\t"}{.status.podIP}{"\n"}{end}' 2>/dev/null \
+    | grep -vP '\t$' | sort
+  echo "## ipam"
+  $K -n kube-system get cm cilium-config -o jsonpath='{.data.ipam}' 2>/dev/null; echo
+}
+
+cmd_preflight() {
+  mkdir -p "$STATE"
+  snapshot > "$STATE/before.txt"
+  say "recorded $(grep -c . "$STATE/before.txt") lines -> $STATE/before.txt"
+  say "mode now: $(sed -n '/^## ipam/,$p' "$STATE/before.txt" | tail -1)"
+  # The pod inventory is the rollback reference: if the switch renumbers, this
+  # is the only record of what an address USED to be.
+  say "pods on the pod network: $(sed -n '/^## pods/,/^## ipam/p' "$STATE/before.txt" | grep -c '10\.')"
+}
+
+# The deadlock, in one place because it WILL happen and doing it by hand under
+# time pressure is how the wrong node gets untainted:
+#   agent has no pod CIDR -> agent not ready -> node keeps
+#   node.cilium.io/agent-not-ready:NoSchedule -> the operator that would assign
+#   the CIDR cannot schedule -> agent still has no pod CIDR.
+# Removing the taint is safe: it exists to keep normal workloads off a node
+# without working networking, and the operator is precisely the thing that fixes
+# that. Kubernetes re-adds it on the next agent restart.
+cmd_unstick() {
+  local stuck=0
+  for n in $($K get nodes -o name 2>/dev/null); do
+    $K get "$n" -o jsonpath='{.spec.taints[*].key}' 2>/dev/null | grep -q 'agent-not-ready' || continue
+    warn "${n#node/} carries agent-not-ready; removing so the operator can schedule"
+    $K taint "$n" node.cilium.io/agent-not-ready- >/dev/null 2>&1 && stuck=$((stuck+1))
+  done
+  [ "$stuck" -eq 0 ] && say "no node was stuck" || say "cleared $stuck node(s)"
+  local pend
+  pend="$($K -n kube-system get pods -l io.cilium/app=operator --no-headers 2>/dev/null | grep -c Pending)"
+  [ "${pend:-0}" -gt 0 ] && warn "$pend operator pod(s) still Pending — check tolerations, not just taints"
+  return 0
+}
+
+cmd_verify() {
+  [ -f "$STATE/before.txt" ] || { warn "no preflight snapshot; nothing to compare"; return 1; }
+  snapshot > "$STATE/after.txt"
+  echo
+  say "mode: $(sed -n '/^## ipam/,$p' "$STATE/before.txt" | tail -1) -> $(sed -n '/^## ipam/,$p' "$STATE/after.txt" | tail -1)"
+
+  # The question that decides the size of the maintenance window: did per-node
+  # CIDRs survive, or was every node renumbered (and every pod with it)?
+  local moved=0
+  while IFS=$'\t' read -r node cidr; do
+    [ -z "${node:-}" ] && continue
+    local now; now="$(sed -n '/^## ciliumnodes/,/^## pods/p' "$STATE/after.txt" | awk -F'\t' -v n="$node" '$1==n{print $2}')"
+    if [ -n "$now" ] && [ "$now" != "$cidr" ]; then
+      printf '    %-16s %s -> %s\n' "$node" "$cidr" "$now"; moved=$((moved+1))
+    fi
+  done < <(sed -n '/^## ciliumnodes/,/^## pods/p' "$STATE/before.txt" | grep -P '\t')
+  if [ "$moved" -eq 0 ]; then
+    say "per-node CIDRs UNCHANGED — the pool adopted the existing allocation"
+  else
+    warn "$moved node(s) renumbered — every pod on them must be recycled"
+  fi
+
+  local before after same
+  before="$(sed -n '/^## pods/,/^## ipam/p' "$STATE/before.txt" | grep -P '\t10\.' | wc -l)"
+  after="$(sed -n '/^## pods/,/^## ipam/p' "$STATE/after.txt"  | grep -P '\t10\.' | wc -l)"
+  same="$(comm -12 <(sed -n '/^## pods/,/^## ipam/p' "$STATE/before.txt" | grep -P '\t10\.' | sort) \
+                   <(sed -n '/^## pods/,/^## ipam/p' "$STATE/after.txt"  | grep -P '\t10\.' | sort) | wc -l)"
+  say "pods: $before before, $after after, $same kept the SAME address"
+  # Keeping the address is NOT the good outcome. If a node's CIDR moved, its
+  # existing pods keep IPs that no longer fall inside it, every other node routes
+  # that prefix to the WRONG node, and those pods go unreachable cross-node while
+  # looking perfectly healthy. Observed in labsim: two nodes swapped CIDRs and
+  # cross-node ping to their pods dropped 100%, with every pod still Running.
+  # This is the check that decides whether a recycle is optional or mandatory.
+  local stranded=0
+  while read -r ns name ip node; do
+    [ -z "${node:-}" ] && continue
+    local cidr; cidr="$($K get ciliumnode "$node" -o jsonpath='{.spec.ipam.podCIDRs[0]}' 2>/dev/null)"
+    [ -z "$cidr" ] && continue
+    case "$ip" in
+      "${cidr%.*/*}".*) ;;
+      *) printf '    STRANDED %-40s %-15s on %s (now %s)\n' "$ns/$name" "$ip" "$node" "$cidr"; stranded=$((stranded+1)) ;;
+    esac
+  done < <($K get pods -A -o jsonpath='{range .items[?(@.status.podIP)]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.podIP}{" "}{.spec.nodeName}{"\n"}{end}' 2>/dev/null | grep -E ' 10\.')
+  if [ "$stranded" -gt 0 ]; then
+    warn "$stranded pod(s) sit OUTSIDE their node CIDR — unreachable cross-node until recycled"
+    warn "recycle: for ns in $(kubectl get ns -o name | cut -d/ -f2); do kubectl -n $ns rollout restart deploy,ds,sts 2>/dev/null; done"
+  else
+    say "every pod is inside its node CIDR — no recycle needed"
+  fi
+  say "not-Running pods: $($K get pods -A --no-headers 2>/dev/null | grep -vcE 'Running|Completed')"
+}
+
+case "${1:-}" in
+  preflight) cmd_preflight ;;
+  unstick)   cmd_unstick ;;
+  verify)    cmd_verify ;;
+  *) sed -n '2,16p' "$0"; exit 1 ;;
+esac

labsim/dualstack-lab.sh

@@ -0,0 +1,318 @@
+#!/usr/bin/env bash
+# Differential study: what ACTUALLY differs between a k3s cluster born
+# dual-stack and one converted in place?
+#
+# k3s says dual-stack "cannot be enabled on an existing cluster". The stated
+# reason is narrow -- nodes get Pod CIDRs only at join and the Kubernetes IPAM
+# controller will not hand out a new IPv6 CIDR later -- and it does not obviously
+# apply to a cluster where Cilium owns IPAM. Rather than argue from docs, build
+# both shapes and diff them.
+#
+#   ./dualstack-lab.sh up v4         single-node k3s, IPv4 only     (.21)
+#   ./dualstack-lab.sh up dual       single-node k3s, dual-stack    (.22)
+#   ./dualstack-lab.sh pristine v4   reflink copy of v4's disk, so the upgrade
+#                                    attempt can be rolled back and retried
+#   ./dualstack-lab.sh restore v4    put that copy back
+#   ./dualstack-lab.sh collect <n>   normalized state dump -> evidence/<n>/
+#   ./dualstack-lab.sh compare a b   semantic diff of two collections
+#   ./dualstack-lab.sh virtdiff a b  whole-filesystem diff, offline (libguestfs)
+#   ./dualstack-lab.sh down [name]
+#
+# The comparison that matters is `compare dual upgraded`: everything it prints
+# is a way the converted cluster failed to reach the shape of a native one.
+#
+# Single node on purpose. Dual-stack is decided by server flags and CNI config,
+# both of which a one-node cluster exercises fully, and it rebuilds in minutes.
+# Node-rejoin behaviour needs the 3-node cluster and is a separate question.
+set -euo pipefail
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+source "$SCRIPT_DIR/lib.sh"
+source "$SCRIPT_DIR/ovs.sh"
+
+K8S_VLAN="${K8S_VLAN:-2}"
+DS_PREFIX="${DS_PREFIX:-172.31.2}"
+MEM="${MEM:-4096}"; CPUS="${CPUS:-2}"; DISK_GB="${DISK_GB:-12}"
+TOKEN="${TOKEN:-labsim-ds-token}"
+CILIUM_VERSION="${CILIUM_VERSION:-1.19.1}"   # same as production
+DEB_BASE="${DEB_BASE:-$IMG_DIR/debian-13-genericcloud-amd64.qcow2}"
+EVIDENCE="$SCRIPT_DIR/dualstack-evidence"
+
+# Pod/Service ranges. IPv4 halves are k3s's own defaults, so the v4-only build is
+# a stock cluster and the diff is not polluted by gratuitous differences.
+# IPv6 halves are ULA: this cluster never routes off-box, and using the real /48
+# here would put lab addresses into a prefix that production also announces.
+V4_CLUSTER="10.42.0.0/16"; V4_SERVICE="10.43.0.0/16"
+V6_CLUSTER="${V6_CLUSTER:-fd00:42::/56}"
+V6_SERVICE="${V6_SERVICE:-fd00:43::/112}"   # /112 -- apiserver caps v6 service ranges
+V6_PREFIX="${V6_PREFIX:-fd00:2}"            # node addresses: fd00:2::<octet>
+
+vm_name() { echo "labsim-ds-$1"; }
+vm_ip()   { case "$1" in v4) echo "$DS_PREFIX.21";; dual) echo "$DS_PREFIX.22";; *) die "unknown build '$1'";; esac; }
+vm_ip6()  { case "$1" in v4) echo "$V6_PREFIX::21";; dual) echo "$V6_PREFIX::22";; *) die "unknown build '$1'";; esac; }
+disk_of() { echo "$IMG_DIR/$(vm_name "$1").qcow2"; }
+
+ssh_vm() { local ip="$1"; shift; ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
+             -o LogLevel=ERROR -o ConnectTimeout=8 -o BatchMode=yes "debian@$ip" "$@"; }
+
+# --- seed -----------------------------------------------------------------
+build_seed() {
+  local iso="$1" vm="$2" mode="$3" pubkey="$4"
+  local ip ip6 tmp; ip="$(vm_ip "$mode")"; ip6="$(vm_ip6 "$mode")"; tmp="$(mktemp -d)"
+
+  echo "instance-id: $vm" > "$tmp/meta-data"
+  # Static v6 on both builds. The v4-only cluster still gets an IPv6 ADDRESS --
+  # only its Kubernetes config is v4-only. Otherwise the diff would be dominated
+  # by host addressing rather than by what Kubernetes did differently.
+  cat > "$tmp/network-config" <<EOF
+version: 2
+ethernets:
+  enp1s0:
+    addresses: [${ip}/24, ${ip6}/64]
+    routes:
+      - to: default
+        via: ${DS_PREFIX}.1
+    nameservers:
+      addresses: [8.8.8.8, 1.1.1.1]
+EOF
+
+  local exec_args="server --flannel-backend=none --disable-network-policy --disable=servicelb --disable=traefik --tls-san=$ip --cluster-init"
+  if [ "$mode" = dual ]; then
+    exec_args="$exec_args --cluster-cidr=${V4_CLUSTER},${V6_CLUSTER} --service-cidr=${V4_SERVICE},${V6_SERVICE} --node-ip=${ip},${ip6}"
+  else
+    exec_args="$exec_args --node-ip=${ip}"
+  fi
+
+  cat > "$tmp/user-data" <<EOF
+#cloud-config
+hostname: $vm
+fqdn: $vm
+users:
+  - name: debian
+    groups: [sudo]
+    shell: /bin/bash
+    sudo: ["ALL=(ALL) NOPASSWD:ALL"]
+    lock_passwd: false
+    plain_text_passwd: labsim
+    ssh_authorized_keys: [$pubkey]
+ssh_pwauth: true
+disable_root: false
+package_update: true
+packages: [curl, jq, iproute2, nftables]
+write_files:
+  - path: /etc/modules-load.d/cilium.conf
+    content: |
+      br_netfilter
+  - path: /etc/dualstack-lab-mode
+    content: |
+      $mode
+runcmd:
+  - modprobe br_netfilter || true
+  - |
+    curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="$exec_args" K3S_TOKEN="$TOKEN" sh -
+  - |
+    # Cilium via helm, matching the production version. IPAM stays 'kubernetes'
+    # in BOTH builds on purpose: that is what production runs, and it is the
+    # mode the k3s objection is actually about. If the converted cluster needs
+    # cluster-pool to work, the diff should be what tells us so.
+    curl -sfL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash || true
+    export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
+    helm repo add cilium https://helm.cilium.io >/dev/null 2>&1 || true
+    helm repo update >/dev/null 2>&1 || true
+    for i in \$(seq 1 60); do kubectl get nodes >/dev/null 2>&1 && break; sleep 5; done
+    if [ "$mode" = dual ]; then
+      helm install cilium cilium/cilium --version $CILIUM_VERSION -n kube-system \\
+        --set kubeProxyReplacement=false --set ipam.mode=kubernetes \\
+        --set ipv4.enabled=true --set ipv6.enabled=true \\
+        --set k8sServiceHost=$ip --set k8sServicePort=6443 || true
+    else
+      helm install cilium cilium/cilium --version $CILIUM_VERSION -n kube-system \\
+        --set kubeProxyReplacement=false --set ipam.mode=kubernetes \\
+        --set ipv4.enabled=true --set ipv6.enabled=false \\
+        --set k8sServiceHost=$ip --set k8sServicePort=6443 || true
+    fi
+    touch /etc/dualstack-lab-ready
+EOF
+  sudo mkdir -p "$(dirname "$iso")"
+  sudo genisoimage -quiet -output "$iso" -volid cidata -joliet -rock \
+    "$tmp/user-data" "$tmp/meta-data" "$tmp/network-config"
+  rm -rf "$tmp"
+}
+
+cmd_up() {
+  local mode="${1:?usage: up <v4|dual>}"
+  local vm ip disk seed pubkey
+  vm="$(vm_name "$mode")"; ip="$(vm_ip "$mode")"; disk="$(disk_of "$mode")"
+  seed="$IMG_DIR/${vm}-seed.iso"; pubkey="$(find_ssh_pubkey)"
+
+  [ -f "$DEB_BASE" ] || die "base image missing: $DEB_BASE (run ./k8s-up.sh once)"
+  if virsh_q dominfo "$vm" >/dev/null 2>&1; then
+    log "$vm exists — starting if stopped"
+    [ "$(virsh_q domstate "$vm" | head -1)" = "running" ] || virsh_q start "$vm" >/dev/null
+    return
+  fi
+  selected_vlans; ovs_up
+  log "creating $vm ($mode) at $ip / $(vm_ip6 "$mode")"
+  sudo qemu-img create -q -f qcow2 -F qcow2 -b "$DEB_BASE" "$disk" "${DISK_GB}G" >/dev/null
+  build_seed "$seed" "$vm" "$mode" "$pubkey"
+  sudo virt-install --connect "$LIBVIRT_URI" --name "$vm" \
+    --memory "$MEM" --vcpus "$CPUS" \
+    --disk "path=$disk,format=qcow2,bus=virtio" \
+    --disk "path=$seed,device=cdrom" \
+    --network "network=$OVS_NET,portgroup=vlan${K8S_VLAN},model=virtio" \
+    --os-variant debian12 --graphics none --noautoconsole --import >/dev/null
+  log "installing in background; watch: ssh debian@$ip 'ls /etc/dualstack-lab-ready'"
+}
+
+# --- pristine copy / restore ---------------------------------------------
+# reflink so the copy is instant and independent on btrfs/xfs. A qcow2 backing
+# chain would be cheaper still but makes the parent read-only in practice: boot
+# the parent again and every child silently corrupts.
+cmd_pristine() {
+  local mode="${1:?usage: pristine <v4|dual>}" vm disk
+  vm="$(vm_name "$mode")"; disk="$(disk_of "$mode")"
+  [ "$(virsh_q domstate "$vm" 2>/dev/null | head -1)" = "running" ] && \
+    die "$vm is running — shut it down first (virsh shutdown $vm), a copy of a live disk is not consistent"
+  sudo cp --reflink=auto "$disk" "${disk}.pristine"
+  log "pristine copy: ${disk}.pristine"
+}
+cmd_restore() {
+  local mode="${1:?usage: restore <v4|dual>}" vm disk
+  vm="$(vm_name "$mode")"; disk="$(disk_of "$mode")"
+  [ -f "${disk}.pristine" ] || die "no pristine copy for $mode"
+  [ "$(virsh_q domstate "$vm" 2>/dev/null | head -1)" = "running" ] && \
+    die "$vm is running — shut it down first"
+  sudo cp --reflink=auto "${disk}.pristine" "$disk"
+  log "restored $mode from pristine"
+}
+
+
+# --- the experiment ------------------------------------------------------
+# Convert the IPv4-only cluster in place, mirroring the flags the native build
+# was BORN with. Each step prints what the cluster did, because the interesting
+# output is which step refuses rather than whether the end state is pretty.
+cmd_upgrade() {
+  local ip; ip="$(vm_ip v4)"; local ip6; ip6="$(vm_ip6 v4)"
+  log "step 1/4: add dual CIDRs + dual node-ip to the k3s unit"
+  # Done with python on the box, not nested sed: quoting a multi-line systemd
+  # continuation through ssh -> sh -> sed produced a literal \\n in the unit, and
+  # k3s then saw a dual cluster-cidr with a still-IPv4 service-cidr and refused
+  # to start. All three flags go on one line -- systemd does not care, and there
+  # is nothing left to escape.
+  ssh_vm "$ip" "sudo python3 - <<'PYEOF'
+import re
+u = '/etc/systemd/system/k3s.service'
+s = open(u).read()
+old = \"'--node-ip=${ip}'\"
+new = \"'--cluster-cidr=${V4_CLUSTER},${V6_CLUSTER}' '--service-cidr=${V4_SERVICE},${V6_SERVICE}' '--node-ip=${ip},${ip6}'\"
+assert old in s, 'node-ip flag not found in unit'
+open(u,'w').write(s.replace(old, new))
+print('    unit rewritten')
+PYEOF
+sudo systemctl daemon-reload" || die "unit edit failed"
+  ssh_vm "$ip" "grep -oE \"'--(cluster|service)-cidr=[^']*'|'--node-ip=[^']*'\" /etc/systemd/system/k3s.service | sed 's/^/    /'"
+
+  log "step 2/4: restart k3s and see whether it accepts the changed ranges"
+  ssh_vm "$ip" "sudo systemctl restart k3s" || true
+  for i in $(seq 1 40); do
+    ssh_vm "$ip" "sudo k3s kubectl get --raw /readyz >/dev/null 2>&1" && break
+    sleep 5
+  done
+  ssh_vm "$ip" "sudo journalctl -u k3s --since '2 min ago' --no-pager 2>/dev/null | grep -iE 'cidr|dual|ipv6|invalid|cannot|fail' | tail -12 | sed 's/^/    /'" || true
+
+  log "step 3/4: what the API says now"
+  ssh_vm "$ip" "echo -n '    servicecidr: '; sudo k3s kubectl get servicecidr -o jsonpath='{.items[*].spec.cidrs}'; echo; \
+                echo -n '    node podCIDRs: '; sudo k3s kubectl get node -o jsonpath='{.items[0].spec.podCIDRs}'; echo; \
+                echo -n '    node addresses: '; sudo k3s kubectl get node -o jsonpath='{.items[0].status.addresses[*].address}'; echo" || true
+
+  log "step 4/4: turn on IPv6 in Cilium"
+  ssh_vm "$ip" "export KUBECONFIG=/etc/rancher/k3s/k3s.yaml; sudo -E helm upgrade cilium cilium/cilium --version ${CILIUM_VERSION} -n kube-system --reuse-values --set ipv6.enabled=true >/dev/null 2>&1 && echo '    cilium upgraded' || echo '    cilium upgrade FAILED'" || true
+  ssh_vm "$ip" "sudo k3s kubectl -n kube-system rollout restart ds/cilium >/dev/null 2>&1; sleep 20; sudo k3s kubectl -n kube-system get pods -l k8s-app=cilium --no-headers | sed 's/^/    /'" || true
+  log "now: ./dualstack-lab.sh collect upgraded ${ip} && ./dualstack-lab.sh compare dual upgraded"
+}
+
+# --- evidence collection --------------------------------------------------
+# Normalized on purpose. Two independently built clusters differ in certs,
+# tokens, UUIDs, timestamps and log lines; left raw, that noise buries the
+# handful of differences that actually mean something.
+cmd_collect() {
+  local name="${1:?usage: collect <name> [ip]}"
+  local ip="${2:-}"
+  [ -n "$ip" ] || ip="$(vm_ip "$name" 2>/dev/null || true)"
+  [ -n "$ip" ] || die "collect: give an ip for a non-standard name"
+  local out="$EVIDENCE/$name"; mkdir -p "$out"
+  log "collecting from $name ($ip) -> $out"
+
+  ssh_vm "$ip" 'sudo cat /etc/rancher/k3s/config.yaml 2>/dev/null; sudo systemctl cat k3s 2>/dev/null | grep -A30 ExecStart' \
+    > "$out/k3s-config.txt" 2>/dev/null || true
+  ssh_vm "$ip" 'sudo tr "\0" "\n" < /proc/$(pgrep -f "k3s server" | head -1)/cmdline | grep -v "^$"' \
+    > "$out/k3s-cmdline.txt" 2>/dev/null || true
+  ssh_vm "$ip" 'ip -o addr show | awk "{print \$2, \$3, \$4}"; echo ---; ip -4 route show; echo ---; ip -6 route show' \
+    > "$out/host-net.txt" 2>/dev/null || true
+  ssh_vm "$ip" 'sudo sysctl -a 2>/dev/null | grep -E "net\.ipv6\.conf\.(all|default)\.(forwarding|disable_ipv6)|net\.ipv4\.ip_forward"' \
+    > "$out/sysctl.txt" 2>/dev/null || true
+
+  local K='sudo k3s kubectl'
+  ssh_vm "$ip" "$K get servicecidr -o yaml"       > "$out/servicecidr.yaml" 2>/dev/null || true
+  ssh_vm "$ip" "$K get nodes -o yaml"             > "$out/nodes.yaml.raw"   2>/dev/null || true
+  ssh_vm "$ip" "$K get ciliumnodes -o yaml"       > "$out/ciliumnodes.yaml.raw" 2>/dev/null || true
+  ssh_vm "$ip" "$K -n kube-system get cm cilium-config -o yaml" > "$out/cilium-config.yaml.raw" 2>/dev/null || true
+  ssh_vm "$ip" "$K get svc -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,FAMILYPOLICY:.spec.ipFamilyPolicy,FAMILIES:.spec.ipFamilies,IPS:.spec.clusterIPs" \
+    > "$out/services.txt" 2>/dev/null || true
+  ssh_vm "$ip" "$K get pods -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,IPS:.status.podIPs" \
+    > "$out/podips.txt" 2>/dev/null || true
+
+  # Strip the things that differ every build regardless of configuration.
+  for f in "$out"/*.raw; do
+    [ -e "$f" ] || continue
+    sed -E \
+      -e 's/[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9:]+Z?/<TIME>/g' \
+      -e 's/(uid|resourceVersion|creationTimestamp|generation|observedGeneration): .*/\1: <X>/' \
+      -e 's/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/<UUID>/g' \
+      -e 's/(LS0tLS1|[A-Za-z0-9+\/]{60,}=*)/<B64>/g' \
+      "$f" > "${f%.raw}"
+    rm -f "$f"
+  done
+  log "collected $(ls "$out" | wc -l) artefacts"
+}
+
+cmd_compare() {
+  local a="${1:?usage: compare <a> <b>}" b="${2:?}"
+  [ -d "$EVIDENCE/$a" ] && [ -d "$EVIDENCE/$b" ] || die "collect both first"
+  echo "### semantic diff: $a (<) vs $b (>)"
+  diff -ru "$EVIDENCE/$a" "$EVIDENCE/$b" || true
+}
+
+cmd_virtdiff() {
+  local a="${1:?usage: virtdiff <a> <b>}" b="${2:?}"
+  for m in "$a" "$b"; do
+    [ "$(virsh_q domstate "$(vm_name "$m")" 2>/dev/null | head -1)" = "running" ] && \
+      die "$(vm_name "$m") is running — virt-diff needs the disks quiescent"
+  done
+  log "whole-filesystem diff (slow); noise is expected — use it to find what the collector missed"
+  sudo virt-diff -a "$(disk_of "$a")" -A "$(disk_of "$b")" \
+    | grep -vE '/(var/log|tmp|run|proc|sys)/|\.log$|/var/lib/rancher/k3s/(server/(tls|cred|db)|agent)' || true
+}
+
+cmd_down() {
+  local only="${1:-}"
+  for m in v4 dual; do
+    [ -n "$only" ] && [ "$only" != "$m" ] && continue
+    local vm; vm="$(vm_name "$m")"
+    virsh_q dominfo "$vm" >/dev/null 2>&1 || continue
+    log "removing $vm"
+    virsh_q destroy "$vm" >/dev/null 2>&1 || true
+    virsh_q undefine "$vm" --remove-all-storage >/dev/null 2>&1 || true
+  done
+}
+
+case "${1:-}" in
+  up)       shift; cmd_up "$@" ;;
+  upgrade)  shift; cmd_upgrade "$@" ;;
+  pristine) shift; cmd_pristine "$@" ;;
+  restore)  shift; cmd_restore "$@" ;;
+  collect)  shift; cmd_collect "$@" ;;
+  compare)  shift; cmd_compare "$@" ;;
+  virtdiff) shift; cmd_virtdiff "$@" ;;
+  down)     shift; cmd_down "$@" ;;
+  *) sed -n '2,30p' "$0"; exit 1 ;;
+esac

labsim/labsim-vlan-leak-test.sh

@@ -0,0 +1,178 @@
+#!/bin/bash
+# Does the router offer an address from the WRONG VLAN's pool?
+#
+# The fault (ISC Kea #1117, "Mix of physical and virtual interfaces (VLAN) does
+# not work"): with `dhcp-socket-type: raw`, a frame tagged for a sub-interface is
+# ALSO delivered to the PARENT's AF_PACKET socket. Kea then selects a subnet from
+# the parent's own address and answers a second time from the wrong pool. Both
+# offers race to the client and the CLIENT decides which one wins -- which is why
+# the symptom looks device-dependent and unreproducible.
+#
+# Production and this sim have the identical shape that triggers it: Management
+# is the NATIVE/untagged VLAN on `bond0` and therefore has a subnet on the
+# parent, while every other VLAN is a `bond0.<vif>` sub-interface of that same
+# bond.
+#
+# Method: make one DHCP client on a TAGGED VLAN send a DISCOVER, and capture
+# simultaneously on the parent and on the sub-interface. The verdict is not
+# "did the client get the right address" -- the client picking correctly is
+# exactly how this hid for weeks. The verdict is how many OFFERs the SERVER
+# emitted and which source addresses they carried.
+#
+#   ./labsim-vlan-leak-test.sh                 test VLAN 3
+#   ./labsim-vlan-leak-test.sh --vlan 9        test another VLAN
+#   ./labsim-vlan-leak-test.sh --save before   also write the raw captures to
+#                                              vlan-leak-evidence/before/
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+
+ROUTER_IP="${ROUTER_IP:-172.31.1.1}"
+ROUTER_PW="${ROUTER_PW:-vyos}"
+CLIENT_PW="${CLIENT_PW:-labsim}"
+VLAN=3
+CLIENT=""
+SAVE=""
+
+while [ $# -gt 0 ]; do
+  case "$1" in
+    --vlan)   VLAN="$2"; shift 2 ;;
+    --client) CLIENT="$2"; shift 2 ;;
+    --save)   SAVE="$2"; shift 2 ;;
+    *) echo "usage: $0 [--vlan N] [--client IP] [--save LABEL]" >&2; exit 2 ;;
+  esac
+done
+: "${CLIENT:=172.31.${VLAN}.10}"
+
+log() { printf '\033[36m==>\033[0m %s\n' "$*"; }
+die() { printf '\033[31merror:\033[0m %s\n' "$*" >&2; exit 1; }
+
+command -v sshpass >/dev/null || die "sshpass required"
+
+# A silent router is the one verdict worth double-checking before reporting.
+#
+# Kea can be `is-active` and answering nothing -- it reopens sockets on a retry
+# loop, and some configurations (`listen-interface`, notably) leave individual
+# VLANs dead while the rest work. Both look identical to a one-shot test: "no
+# reply at all". Two opposite and equally wrong conclusions about
+# `listen-interface` came out of believing a single negative run, in both
+# directions, before a retry made the real pattern obvious.
+#
+# Kea's fallback UDP socket appearing is NOT a readiness signal -- it is bound
+# well before the server actually answers. Checked, and it does not work.
+RETRIED="${RETRIED:-0}"
+
+router() {
+  timeout 40 sshpass -p "$ROUTER_PW" ssh -o StrictHostKeyChecking=no \
+    -o ConnectTimeout=8 "vyos@$ROUTER_IP" "$@" 2>/dev/null
+}
+# VyOS's login shell is vbash, which returns 255 on anything it does not like --
+# in particular a backgrounded job. Feeding the script to `bash -s` on stdin
+# sidesteps vbash entirely and is the only reliable way to leave a daemon behind.
+router_sh() {
+  timeout 40 sshpass -p "$ROUTER_PW" ssh -o StrictHostKeyChecking=no \
+    -o ConnectTimeout=8 "vyos@$ROUTER_IP" 'bash -s' 2>/dev/null
+}
+client() {
+  timeout 60 sshpass -p "$CLIENT_PW" ssh -o StrictHostKeyChecking=no \
+    -o ConnectTimeout=8 "root@$CLIENT" "$@" 2>/dev/null
+}
+
+# Which interfaces to watch. The parent is the whole point: after the fix it
+# should carry no DHCP traffic of its own at all.
+PARENT="bond0"
+VIF="bond0.${VLAN}"
+
+log "router $ROUTER_IP -- capturing on $PARENT and $VIF"
+# Kill EVERY tcpdump first, not just ones matching this run's pattern, and count
+# only afterwards. Counting `pgrep -f 'tcpdump -i bond0'` while a stray tcpdump
+# from an earlier session was still running satisfied the >=2 guard with zero of
+# THIS run's captures alive -- and a capture that records nothing reports
+# "the router sent no reply at all", which reads as a DHCP outage. That sent me
+# chasing a fault in the router that was entirely in the test harness.
+started="$(router_sh <<EOF
+sudo pkill -x tcpdump >/dev/null 2>&1
+sleep 1
+sudo rm -f /tmp/leak-*.txt
+sudo nohup tcpdump -i $PARENT -e -nn -l 'udp port 67 or udp port 68' > /tmp/leak-parent.txt 2>/dev/null &
+sudo nohup tcpdump -i $VIF -e -nn -l 'udp port 67 or udp port 68' > /tmp/leak-vif.txt 2>/dev/null &
+sleep 3
+pgrep -c -x tcpdump
+EOF
+)"
+[ "${started:-0}" -eq 2 ] || die "capture did not start on the router (got ${started:-0}, expected exactly 2)"
+
+# -s /bin/true: ask, observe the answer, apply nothing. The client's existing
+# static address is left alone, so this is safe to run against a live sim VM.
+log "client $CLIENT -- sending DISCOVER on VLAN $VLAN"
+client_out="$(client "udhcpc -n -q -f -i eth0 -s /bin/true -t 3 -T 3 2>&1")"
+[ -n "$client_out" ] || die "no response from client $CLIENT"
+
+sleep 2
+router "sudo pkill -x tcpdump" >/dev/null
+parent="$(router 'sudo cat /tmp/leak-parent.txt')"
+vif="$(router 'sudo cat /tmp/leak-vif.txt')"
+
+echo
+echo "--- client ---"
+echo "$client_out" | sed 's/^/  /'
+echo
+echo "--- $PARENT (parent) ---"
+echo "${parent:-  (nothing)}" | sed 's/^/  /'
+echo
+echo "--- $VIF (sub-interface) ---"
+echo "${vif:-  (nothing)}" | sed 's/^/  /'
+echo
+
+if [ -n "$SAVE" ]; then
+  d="$SCRIPT_DIR/vlan-leak-evidence/$SAVE"
+  mkdir -p "$d"
+  printf '%s\n' "$client_out" > "$d/client.txt"
+  printf '%s\n' "$parent"     > "$d/capture-parent.txt"
+  printf '%s\n' "$vif"        > "$d/capture-vif.txt"
+  router '/opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration commands' \
+    | grep -E 'interfaces bonding|vrrp group' > "$d/router-config.txt"
+  log "evidence saved to vlan-leak-evidence/$SAVE/"
+fi
+
+# --- verdict ---------------------------------------------------------------
+# Every BOOTP Reply seen anywhere, reduced to its source address. A reply whose
+# source is not this VLAN's router leg is an offer from the wrong subnet.
+replies="$(printf '%s\n%s\n' "$parent" "$vif" \
+  | grep -o '[0-9.]*\.67 > [0-9.]*\.68' | awk '{print $1}' | sed 's/\.67$//' \
+  | sort -u)"
+want_prefix="172.31.${VLAN}."
+
+echo "=== verdict ==="
+if [ -z "$replies" ]; then
+  if [ "$RETRIED" -eq 0 ]; then
+    log "no reply -- retrying once in 20s before calling DHCP down"
+    sleep 20; RETRIED=1 exec "$0" --vlan "$VLAN" --client "$CLIENT" ${SAVE:+--save "$SAVE"}
+  fi
+  echo "INCONCLUSIVE: the router sent no reply at all, twice -- DHCP is down on VLAN $VLAN"
+  exit 2
+fi
+
+bad=0
+while read -r src; do
+  [ -z "$src" ] && continue
+  case "$src" in
+    "$want_prefix"*) printf '  ok    offer from %s (this VLAN)\n' "$src" ;;
+    *)               printf '  LEAK  offer from %s (WRONG subnet)\n' "$src"; bad=1 ;;
+  esac
+done <<<"$replies"
+
+# The parent carrying any DHCP of its own is the mechanism, not just a symptom:
+# it means the parent still has a subnet kea can match a tagged frame against.
+if printf '%s' "$parent" | grep -q 'ethertype IPv4' \
+   && printf '%s' "$parent" | grep -v 'vlan ' | grep -q '\.67 > '; then
+  echo "  note  $PARENT emitted an UNTAGGED reply -- the parent still serves a subnet"
+fi
+
+echo
+if [ "$bad" -eq 0 ]; then
+  echo "PASS: only this VLAN's pool answered."
+  exit 0
+fi
+echo "FAIL: the router answered from another VLAN's pool (kea #1117)."
+exit 1

labsim/ovs.sh

@@ -17,6 +17,21 @@ OVS_BR="${OVS_BR:-ovs-labsim}"
 OVS_NET="${OVS_NET:-labsim-ovs}"          # libvirt network wrapping the bridge
 LAG_NAME="${LAG_NAME:-lag-vyos}"
 
+# Native (untagged) VLAN on the trunks to the routers. Empty means NONE: every
+# VLAN, Management included, is tagged.
+#
+# This is not a style choice. A native VLAN is what puts a subnet on the bond
+# PARENT (`bond0`) while every other VLAN lives on a sub-interface of it. With
+# `dhcp-socket-type: raw`, kea then receives each tagged frame TWICE -- once on
+# the sub-interface and once on the parent -- and answers from the parent's pool
+# as well, so a client on VLAN 3 is offered a Management address and picks
+# whichever reply arrives first (ISC Kea #1117).
+#
+# Set LABSIM_NATIVE_VLAN=1 to restore the old shape and reproduce the bug:
+#   LABSIM_NATIVE_VLAN=1 ./router-up.sh && ./labsim-vlan-leak-test.sh   # FAIL
+#                        ./router-up.sh && ./labsim-vlan-leak-test.sh   # PASS
+NATIVE_VLAN="${LABSIM_NATIVE_VLAN:-}"
+
 ovs() { sudo ovs-vsctl "$@"; }
 
 ovs_require() {
@@ -25,6 +40,10 @@ ovs_require() {
     || die "could not start openvswitch"
 }
 
+# A comma-separated VLAN list, numerically sorted, for comparing two lists that
+# came from different places and need not agree on order.
+vlan_sorted() { echo "$1" | tr ',' '\n' | grep -v '^$' | sort -n | paste -sd, -; }
+
 # All VLAN ids from the config, comma separated — used for trunk ports.
 vlan_id_list() {
   local ids=()
@@ -72,18 +91,23 @@ ovs_define_libvirt_net() {
 "
   done
 
-  # Trunk: VLAN 1 native/untagged, everything else tagged — the production
-  # shape. libvirt expresses this declaratively via nativeMode='untagged'
-  # (see libvirt formatnetwork.html), so it does not need fixing up by hand.
-  # It also matters functionally: LACPDUs are untagged, and a trunk with no
-  # native VLAN has nowhere to put them.
+  # Trunk: every VLAN tagged, and by default NO native VLAN (see NATIVE_VLAN at
+  # the top of this file for why -- it is the kea #1117 fix, not tidiness).
+  # libvirt expresses a native VLAN declaratively via nativeMode='untagged'
+  # (see libvirt formatnetwork.html), so it needs no fixing up by hand.
+  #
+  # The worry that a trunk with no native VLAN has nowhere to put LACPDUs is
+  # unfounded, and was tested rather than reasoned about: with vlan_mode=trunk
+  # and no tag, `ovs-appctl bond/show` still reports lacp_status: negotiated
+  # with both members enabled. LACPDUs are slow-protocol frames handled per
+  # member, below the VLAN layer.
   local trunk="  <portgroup name='trunk'>
     <vlan trunk='yes'>
 "
   for entry in "${SELECTED[@]}"; do
     IFS=: read -r vid _n _p _r <<<"$entry"
-    if [ "$vid" = "1" ]; then
-      trunk+="      <tag id='1' nativeMode='untagged'/>
+    if [ -n "$NATIVE_VLAN" ] && [ "$vid" = "$NATIVE_VLAN" ]; then
+      trunk+="      <tag id='${vid}' nativeMode='untagged'/>
 "
     else
       trunk+="      <tag id='${vid}'/>
@@ -116,29 +140,59 @@ ${pg}${trunk}</network>"
 ovs_bond_router() {
   local vm="$1"
   local taps
-  # NB: domiflist indents its rows, so anchor on the FIELD not the line —
-  # /^vnet/ silently matches nothing and the bond never gets built.
-  taps="$(virsh_q domiflist "$vm" 2>/dev/null | awk '$1 ~ /^vnet/ {print $1}')"
+  # Two filters, both load-bearing:
+  #
+  #   $1 ~ /^vnet/ -- domiflist indents its rows, so anchor on the FIELD, not
+  #   the line. /^vnet/ silently matches nothing and the bond never gets built.
+  #
+  #   $3 == OVS_NET -- count only the taps on the sim fabric. The primary also
+  #   carries a libvirt-NAT scaffold NIC (see --drop-scaffold in the README), so
+  #   an unfiltered count is 3, and this function's "expected 2" guard then
+  #   skipped the primary's bond entirely while reporting only a warning.
+  taps="$(virsh_q domiflist "$vm" 2>/dev/null \
+          | awk -v net="$OVS_NET" '$1 ~ /^vnet/ && $3 == net {print $1}')"
   local count; count="$(echo "$taps" | grep -c .)"
-  [ "$count" -eq 2 ] || { warn "router $vm has $count tap(s), expected 2 — skipping bond"; return 1; }
+  [ "$count" -eq 2 ] \
+    || { warn "router $vm has $count tap(s) on $OVS_NET, expected 2 — skipping bond"; return 1; }
 
-  # Already bonded? Re-runs must still reconcile the VLAN list: adding a VLAN to
-  # vlans.conf and finding the bond unchanged is exactly how a VLAN silently
-  # fails to reach a router -- interface present, tag missing, frames dropped by
-  # the switch. Returning early here once cost real debugging time.
+  local t1 t2; t1="$(echo "$taps" | sed -n 1p)"; t2="$(echo "$taps" | sed -n 2p)"
+  local want; want="$(vlan_id_list)"
+
+  # Already bonded? Re-runs must still reconcile BOTH the VLAN list and the
+  # membership, and each has drawn blood:
+  #
+  #   VLANs -- adding a VLAN to vlans.conf and finding the bond unchanged is how
+  #   a VLAN silently fails to reach a router: interface present, tag missing,
+  #   frames dropped by the switch.
+  #
+  #   MEMBERS -- restarting the VM recreates its taps with NEW names, leaving the
+  #   bond holding two interfaces that no longer exist. `list-ports` still shows
+  #   the bond, so this early return declared success while the router's real
+  #   taps sat in the bridge as two INDEPENDENT ports, each carrying libvirt's
+  #   own portgroup VLAN config. That is how labsim-vyos2 ran for weeks with no
+  #   LACP at all and a native VLAN nobody had asked for -- and it is invisible
+  #   until you change the trunk and only one router follows.
   if ovs list-ports "$OVS_BR" 2>/dev/null | grep -qx "$LAG_NAME"; then
-    local want; want="$(vlan_id_list | tr ',' '\n' | grep -vx 1 | paste -sd, -)"
+    local members; members="$(ovs-appctl-members)"
+    if [ "$members" != "$(printf '%s\n%s' "$t1" "$t2" | sort | paste -sd, -)" ]; then
+      warn "bond $LAG_NAME holds stale members [$members], VM has [$t1,$t2] — rebuilding"
+      ovs --if-exists del-port "$OVS_BR" "$LAG_NAME"
+    else
+      # Compare as SETS. vlan_id_list yields config order (1,2,3,9,10,200,51,53)
+      # while OVS returns its own sorted order, so a raw string compare reports
+      # drift on every run and rewrites a trunk that was already correct.
       local have; have="$(ovs get port "$LAG_NAME" trunks 2>/dev/null | tr -d '[] ')"
-    if [ "$want" != "$have" ]; then
+      if [ "$(vlan_sorted "$want")" != "$(vlan_sorted "$have")" ]; then
         log "bond $LAG_NAME trunk drift: [$have] -> [$want]; updating"
         ovs set port "$LAG_NAME" trunks="$want"
       else
         log "LACP bond $LAG_NAME already present, trunk correct"
       fi
+      ovs_set_native "$LAG_NAME"
       return 0
     fi
+  fi
 
-  local t1 t2; t1="$(echo "$taps" | sed -n 1p)"; t2="$(echo "$taps" | sed -n 2p)"
   log "bonding $t1 + $t2 into $LAG_NAME (LACP active, balance-tcp)"
   ovs del-port "$OVS_BR" "$t1" 2>/dev/null || true
   ovs del-port "$OVS_BR" "$t2" 2>/dev/null || true
@@ -152,15 +206,36 @@ ovs_bond_router() {
   # LACPDUs. Falling back to active-backup brings the links up so negotiation
   # can start.
   #
-  # native-untagged + tag=1 carries the untagged LACPDUs and the management
-  # VLAN, matching production. libvirt's portgroup VLAN config does NOT apply
-  # here — the bond is a port libvirt never created — so set it inline.
-  local tagged; tagged="$(vlan_id_list | tr ',' '\n' | grep -vx 1 | paste -sd, -)"
+  # The VLAN mode is set inline: libvirt's portgroup config does NOT apply here,
+  # because the bond is a port libvirt never created.
   ovs add-bond "$OVS_BR" "$LAG_NAME" "$t1" "$t2" \
-      lacp=active bond_mode=balance-tcp \
-      vlan_mode=native-untagged tag=1 trunks="$tagged" \
+      lacp=active bond_mode=balance-tcp trunks="$want" \
     -- set port "$LAG_NAME" other_config:lacp-time=fast \
     -- set port "$LAG_NAME" other_config:lacp-fallback-ab=true
+  ovs_set_native "$LAG_NAME"
+}
+
+# The bond's current members, sorted and comma-joined, or empty if the bond does
+# not resolve at all (which is itself the stale case worth rebuilding for).
+ovs-appctl-members() {
+  sudo ovs-appctl bond/show "$LAG_NAME" 2>/dev/null \
+    | awk '/^member /{gsub(/:/,"",$2); print $2}' | sort | paste -sd, -
+}
+
+# Apply NATIVE_VLAN to a trunk port.
+#
+# `tag` MUST be removed, not merely left alone, when there is no native VLAN.
+# Setting vlan_mode=trunk while a stale `tag` remains looks correct in
+# `ovs-vsctl list port` -- it prints vlan_mode: trunk right next to tag: 1 --
+# but the port keeps egressing that VLAN untagged. Half an hour went into
+# "the router is ignoring the trunk change" before the tag was the answer.
+ovs_set_native() {
+  local port="$1"
+  if [ -n "$NATIVE_VLAN" ]; then
+    ovs set port "$port" vlan_mode=native-untagged tag="$NATIVE_VLAN"
+  else
+    ovs set port "$port" vlan_mode=trunk -- clear port "$port" tag
+  fi
 }
 
 ovs_bond_status() {

labsim/sim-ha-config.py

@@ -60,14 +60,20 @@ def build(role: str) -> list[str]:
 
     for vlan, (pfx, cidr) in VLANS.items():
         g = group(vlan)
-        iface = "bond0" if vlan == 1 else f"bond0 vif {vlan}"
+        # EVERY VLAN is a sub-interface, Management (VLAN 1) included. Putting
+        # Management on the bare `bond0` is what gives the parent a subnet, and
+        # kea then answers tagged frames from it as well as from the correct
+        # sub-interface -- clients on other VLANs get offered a Management
+        # address (ISC Kea #1117). See NATIVE_VLAN in ovs.sh; proven by
+        # labsim-vlan-leak-test.sh.
+        iface = f"bond0 vif {vlan}"
         out += [
             f"# VLAN {vlan}",
             # The node's own address replaces the .1 it used to hold directly;
             # .1 becomes the floating VIP, exactly as production will be.
             f"delete interfaces bonding {iface} address",
             f"set interfaces bonding {iface} address '{pfx}.{self_o}/{cidr}'",
-            f"set high-availability vrrp group {g} interface bond0{'' if vlan == 1 else f'.{vlan}'}",
+            f"set high-availability vrrp group {g} interface bond0.{vlan}",
             f"set high-availability vrrp group {g} vrid {vlan}",
             f"set high-availability vrrp group {g} address {pfx}.1/{cidr}",
             f"set high-availability vrrp group {g} priority {prio}",

labsim/sim-net-config.py

@@ -187,7 +187,15 @@ def wan(drop_scaffold: bool) -> list[str]:
 # which point it is wide open and nothing looks wrong. This is the same policy
 # expressed as a whitelist, so a new interface is closed until it is named.
 # ---------------------------------------------------------------------------
-LAN_IFACES = ["bond0", "bond0.2", "bond0.3", "bond0.9", "bond0.10", "bond0.200"]
+# Management is `bond0.1`, NOT the bare `bond0`. Every VLAN is tagged and the
+# bond parent carries no subnet at all -- see NATIVE_VLAN in ovs.sh for why.
+#
+# This line is the trap in that change. The address move is the visible part and
+# the part you remember; leaving `bond0` here instead of `bond0.1` means the
+# whole Management VLAN falls outside the LAN group, and with a default-deny
+# ruleset that is every management session and all inter-VLAN routing for VLAN 1
+# dropped the instant the commit lands -- on a router you reach through itself.
+LAN_IFACES = ["bond0.1", "bond0.2", "bond0.3", "bond0.9", "bond0.10", "bond0.200"]
 LAN_GROUP = "LAN"
 
 

labsim/vlan-leak-evidence/after-vlan1/capture-parent.txt

@@ -0,0 +1,6 @@
+12:52:35.919490 52:54:00:6d:71:e7 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 1, p 0, ethertype IPv4 (0x0800), 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:6d:71:e7, length 300
+12:52:35.920089 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype 802.1Q (0x8100), length 329: vlan 1, p 0, ethertype IPv4 (0x0800), 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283
+12:52:35.920307 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype 802.1Q (0x8100), length 329: vlan 1, p 0, ethertype IPv4 (0x0800), 172.31.1.252.67 > 172.31.1.7.68: BOOTP/DHCP, Reply, length 283
+12:52:35.922052 52:54:00:6d:71:e7 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 1, p 0, ethertype IPv4 (0x0800), 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:6d:71:e7, length 300
+12:52:35.922509 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype 802.1Q (0x8100), length 329: vlan 1, p 0, ethertype IPv4 (0x0800), 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283
+12:52:35.923327 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype 802.1Q (0x8100), length 329: vlan 1, p 0, ethertype IPv4 (0x0800), 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283

labsim/vlan-leak-evidence/after-vlan1/capture-vif.txt

@@ -0,0 +1,6 @@
+12:52:35.919490 52:54:00:6d:71:e7 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:6d:71:e7, length 300
+12:52:35.920081 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype IPv4 (0x0800), length 325: 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283
+12:52:35.920305 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype IPv4 (0x0800), length 325: 172.31.1.252.67 > 172.31.1.7.68: BOOTP/DHCP, Reply, length 283
+12:52:35.922052 52:54:00:6d:71:e7 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:6d:71:e7, length 300
+12:52:35.922507 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype IPv4 (0x0800), length 325: 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283
+12:52:35.923326 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype IPv4 (0x0800), length 325: 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283

labsim/vlan-leak-evidence/after-vlan1/client.txt

@@ -0,0 +1,4 @@
+udhcpc: started, v1.37.0
+udhcpc: broadcasting discover
+udhcpc: broadcasting select for 172.31.1.6, server 172.31.1.252
+udhcpc: lease of 172.31.1.6 obtained from 172.31.1.252, lease time 86400

labsim/vlan-leak-evidence/after-vlan1/router-config.txt

@@ -0,0 +1,64 @@
+set high-availability vrrp group native address 172.31.1.1/24
+set high-availability vrrp group native hello-source-address '172.31.1.252'
+set high-availability vrrp group native interface 'bond0.1'
+set high-availability vrrp group native no-preempt
+set high-availability vrrp group native peer-address '172.31.1.253'
+set high-availability vrrp group native priority '200'
+set high-availability vrrp group native vrid '1'
+set high-availability vrrp group vlan2 address 172.31.2.1/24
+set high-availability vrrp group vlan2 hello-source-address '172.31.2.252'
+set high-availability vrrp group vlan2 interface 'bond0.2'
+set high-availability vrrp group vlan2 no-preempt
+set high-availability vrrp group vlan2 peer-address '172.31.2.253'
+set high-availability vrrp group vlan2 priority '200'
+set high-availability vrrp group vlan2 vrid '2'
+set high-availability vrrp group vlan3 address 172.31.3.1/24
+set high-availability vrrp group vlan3 hello-source-address '172.31.3.252'
+set high-availability vrrp group vlan3 interface 'bond0.3'
+set high-availability vrrp group vlan3 no-preempt
+set high-availability vrrp group vlan3 peer-address '172.31.3.253'
+set high-availability vrrp group vlan3 priority '200'
+set high-availability vrrp group vlan3 vrid '3'
+set high-availability vrrp group vlan9 address 172.31.9.1/24
+set high-availability vrrp group vlan9 hello-source-address '172.31.9.252'
+set high-availability vrrp group vlan9 interface 'bond0.9'
+set high-availability vrrp group vlan9 no-preempt
+set high-availability vrrp group vlan9 peer-address '172.31.9.253'
+set high-availability vrrp group vlan9 priority '200'
+set high-availability vrrp group vlan9 vrid '9'
+set high-availability vrrp group vlan10 address 172.31.10.1/23
+set high-availability vrrp group vlan10 hello-source-address '172.31.10.252'
+set high-availability vrrp group vlan10 interface 'bond0.10'
+set high-availability vrrp group vlan10 no-preempt
+set high-availability vrrp group vlan10 peer-address '172.31.10.253'
+set high-availability vrrp group vlan10 priority '200'
+set high-availability vrrp group vlan10 vrid '10'
+set high-availability vrrp group vlan200 address 172.31.200.1/24
+set high-availability vrrp group vlan200 hello-source-address '172.31.200.252'
+set high-availability vrrp group vlan200 interface 'bond0.200'
+set high-availability vrrp group vlan200 no-preempt
+set high-availability vrrp group vlan200 peer-address '172.31.200.253'
+set high-availability vrrp group vlan200 priority '200'
+set high-availability vrrp group vlan200 vrid '200'
+set interfaces bonding bond0 description 'api-batch-test'
+set interfaces bonding bond0 hash-policy 'layer2+3'
+set interfaces bonding bond0 lacp-rate 'fast'
+set interfaces bonding bond0 member interface 'eth0'
+set interfaces bonding bond0 member interface 'eth1'
+set interfaces bonding bond0 mode '802.3ad'
+set interfaces bonding bond0 vif 1 address '172.31.1.252/24'
+set interfaces bonding bond0 vif 1 description 'management'
+set interfaces bonding bond0 vif 2 address '172.31.2.252/24'
+set interfaces bonding bond0 vif 2 description 'k8s'
+set interfaces bonding bond0 vif 3 address '172.31.3.252/24'
+set interfaces bonding bond0 vif 3 description 'kvm'
+set interfaces bonding bond0 vif 9 address '172.31.9.252/24'
+set interfaces bonding bond0 vif 9 description 'private'
+set interfaces bonding bond0 vif 10 address '172.31.10.252/23'
+set interfaces bonding bond0 vif 10 description 'lot'
+set interfaces bonding bond0 vif 51 description 'WAN1 Vodafone-equivalent (sim ISP PPPoE)'
+set interfaces bonding bond0 vif 53 address 'dhcp'
+set interfaces bonding bond0 vif 53 description 'WAN3 10gig-equivalent (sim ISP DHCP)'
+set interfaces bonding bond0 vif 53 dhcp-options default-route-distance '210'
+set interfaces bonding bond0 vif 200 address '172.31.200.252/24'
+set interfaces bonding bond0 vif 200 description 'roomates'

labsim/vlan-leak-evidence/after/capture-parent.txt

@@ -0,0 +1,6 @@
+12:52:14.639170 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 3, p 0, ethertype IPv4 (0x0800), 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
+12:52:14.640467 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype 802.1Q (0x8100), length 329: vlan 3, p 0, ethertype IPv4 (0x0800), 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
+12:52:14.640846 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype 802.1Q (0x8100), length 329: vlan 3, p 0, ethertype IPv4 (0x0800), 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
+12:52:14.642554 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 3, p 0, ethertype IPv4 (0x0800), 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
+12:52:14.642766 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype 802.1Q (0x8100), length 329: vlan 3, p 0, ethertype IPv4 (0x0800), 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
+12:52:14.643056 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype 802.1Q (0x8100), length 329: vlan 3, p 0, ethertype IPv4 (0x0800), 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283

labsim/vlan-leak-evidence/after/capture-vif.txt

@@ -0,0 +1,6 @@
+12:52:14.639170 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
+12:52:14.640465 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
+12:52:14.640845 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
+12:52:14.642554 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
+12:52:14.642764 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
+12:52:14.643055 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283

labsim/vlan-leak-evidence/after/client.txt

@@ -0,0 +1,4 @@
+udhcpc: started, v1.37.0
+udhcpc: broadcasting discover
+udhcpc: broadcasting select for 172.31.3.11, server 172.31.3.252
+udhcpc: lease of 172.31.3.11 obtained from 172.31.3.252, lease time 85374

labsim/vlan-leak-evidence/after/router-config.txt

@@ -0,0 +1,64 @@
+set high-availability vrrp group native address 172.31.1.1/24
+set high-availability vrrp group native hello-source-address '172.31.1.252'
+set high-availability vrrp group native interface 'bond0.1'
+set high-availability vrrp group native no-preempt
+set high-availability vrrp group native peer-address '172.31.1.253'
+set high-availability vrrp group native priority '200'
+set high-availability vrrp group native vrid '1'
+set high-availability vrrp group vlan2 address 172.31.2.1/24
+set high-availability vrrp group vlan2 hello-source-address '172.31.2.252'
+set high-availability vrrp group vlan2 interface 'bond0.2'
+set high-availability vrrp group vlan2 no-preempt
+set high-availability vrrp group vlan2 peer-address '172.31.2.253'
+set high-availability vrrp group vlan2 priority '200'
+set high-availability vrrp group vlan2 vrid '2'
+set high-availability vrrp group vlan3 address 172.31.3.1/24
+set high-availability vrrp group vlan3 hello-source-address '172.31.3.252'
+set high-availability vrrp group vlan3 interface 'bond0.3'
+set high-availability vrrp group vlan3 no-preempt
+set high-availability vrrp group vlan3 peer-address '172.31.3.253'
+set high-availability vrrp group vlan3 priority '200'
+set high-availability vrrp group vlan3 vrid '3'
+set high-availability vrrp group vlan9 address 172.31.9.1/24
+set high-availability vrrp group vlan9 hello-source-address '172.31.9.252'
+set high-availability vrrp group vlan9 interface 'bond0.9'
+set high-availability vrrp group vlan9 no-preempt
+set high-availability vrrp group vlan9 peer-address '172.31.9.253'
+set high-availability vrrp group vlan9 priority '200'
+set high-availability vrrp group vlan9 vrid '9'
+set high-availability vrrp group vlan10 address 172.31.10.1/23
+set high-availability vrrp group vlan10 hello-source-address '172.31.10.252'
+set high-availability vrrp group vlan10 interface 'bond0.10'
+set high-availability vrrp group vlan10 no-preempt
+set high-availability vrrp group vlan10 peer-address '172.31.10.253'
+set high-availability vrrp group vlan10 priority '200'
+set high-availability vrrp group vlan10 vrid '10'
+set high-availability vrrp group vlan200 address 172.31.200.1/24
+set high-availability vrrp group vlan200 hello-source-address '172.31.200.252'
+set high-availability vrrp group vlan200 interface 'bond0.200'
+set high-availability vrrp group vlan200 no-preempt
+set high-availability vrrp group vlan200 peer-address '172.31.200.253'
+set high-availability vrrp group vlan200 priority '200'
+set high-availability vrrp group vlan200 vrid '200'
+set interfaces bonding bond0 description 'api-batch-test'
+set interfaces bonding bond0 hash-policy 'layer2+3'
+set interfaces bonding bond0 lacp-rate 'fast'
+set interfaces bonding bond0 member interface 'eth0'
+set interfaces bonding bond0 member interface 'eth1'
+set interfaces bonding bond0 mode '802.3ad'
+set interfaces bonding bond0 vif 1 address '172.31.1.252/24'
+set interfaces bonding bond0 vif 1 description 'management'
+set interfaces bonding bond0 vif 2 address '172.31.2.252/24'
+set interfaces bonding bond0 vif 2 description 'k8s'
+set interfaces bonding bond0 vif 3 address '172.31.3.252/24'
+set interfaces bonding bond0 vif 3 description 'kvm'
+set interfaces bonding bond0 vif 9 address '172.31.9.252/24'
+set interfaces bonding bond0 vif 9 description 'private'
+set interfaces bonding bond0 vif 10 address '172.31.10.252/23'
+set interfaces bonding bond0 vif 10 description 'lot'
+set interfaces bonding bond0 vif 51 description 'WAN1 Vodafone-equivalent (sim ISP PPPoE)'
+set interfaces bonding bond0 vif 53 address 'dhcp'
+set interfaces bonding bond0 vif 53 description 'WAN3 10gig-equivalent (sim ISP DHCP)'
+set interfaces bonding bond0 vif 53 dhcp-options default-route-distance '210'
+set interfaces bonding bond0 vif 200 address '172.31.200.252/24'
+set interfaces bonding bond0 vif 200 description 'roomates'

labsim/vlan-leak-evidence/before/capture-parent.txt

@@ -0,0 +1,5 @@
+12:37:08.491910 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 3, p 0, ethertype IPv4 (0x0800), 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
+12:37:08.492629 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.1.252.67 > 172.31.1.9.68: BOOTP/DHCP, Reply, length 283
+12:37:08.493628 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype 802.1Q (0x8100), length 329: vlan 3, p 0, ethertype IPv4 (0x0800), 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
+12:37:08.495587 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 3, p 0, ethertype IPv4 (0x0800), 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
+12:37:08.495946 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype 802.1Q (0x8100), length 329: vlan 3, p 0, ethertype IPv4 (0x0800), 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283

labsim/vlan-leak-evidence/before/capture-vif.txt

@@ -0,0 +1,4 @@
+12:37:08.491910 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
+12:37:08.493625 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
+12:37:08.495587 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
+12:37:08.495944 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283

labsim/vlan-leak-evidence/before/client.txt

@@ -0,0 +1,4 @@
+udhcpc: started, v1.37.0
+udhcpc: broadcasting discover
+udhcpc: broadcasting select for 172.31.3.11, server 172.31.3.252
+udhcpc: lease of 172.31.3.11 obtained from 172.31.3.252, lease time 86280

labsim/vlan-leak-evidence/before/router-config.txt

@@ -0,0 +1,63 @@
+set high-availability vrrp group native address 172.31.1.1/24
+set high-availability vrrp group native hello-source-address '172.31.1.252'
+set high-availability vrrp group native interface 'bond0'
+set high-availability vrrp group native no-preempt
+set high-availability vrrp group native peer-address '172.31.1.253'
+set high-availability vrrp group native priority '200'
+set high-availability vrrp group native vrid '1'
+set high-availability vrrp group vlan2 address 172.31.2.1/24
+set high-availability vrrp group vlan2 hello-source-address '172.31.2.252'
+set high-availability vrrp group vlan2 interface 'bond0.2'
+set high-availability vrrp group vlan2 no-preempt
+set high-availability vrrp group vlan2 peer-address '172.31.2.253'
+set high-availability vrrp group vlan2 priority '200'
+set high-availability vrrp group vlan2 vrid '2'
+set high-availability vrrp group vlan3 address 172.31.3.1/24
+set high-availability vrrp group vlan3 hello-source-address '172.31.3.252'
+set high-availability vrrp group vlan3 interface 'bond0.3'
+set high-availability vrrp group vlan3 no-preempt
+set high-availability vrrp group vlan3 peer-address '172.31.3.253'
+set high-availability vrrp group vlan3 priority '200'
+set high-availability vrrp group vlan3 vrid '3'
+set high-availability vrrp group vlan9 address 172.31.9.1/24
+set high-availability vrrp group vlan9 hello-source-address '172.31.9.252'
+set high-availability vrrp group vlan9 interface 'bond0.9'
+set high-availability vrrp group vlan9 no-preempt
+set high-availability vrrp group vlan9 peer-address '172.31.9.253'
+set high-availability vrrp group vlan9 priority '200'
+set high-availability vrrp group vlan9 vrid '9'
+set high-availability vrrp group vlan10 address 172.31.10.1/23
+set high-availability vrrp group vlan10 hello-source-address '172.31.10.252'
+set high-availability vrrp group vlan10 interface 'bond0.10'
+set high-availability vrrp group vlan10 no-preempt
+set high-availability vrrp group vlan10 peer-address '172.31.10.253'
+set high-availability vrrp group vlan10 priority '200'
+set high-availability vrrp group vlan10 vrid '10'
+set high-availability vrrp group vlan200 address 172.31.200.1/24
+set high-availability vrrp group vlan200 hello-source-address '172.31.200.252'
+set high-availability vrrp group vlan200 interface 'bond0.200'
+set high-availability vrrp group vlan200 no-preempt
+set high-availability vrrp group vlan200 peer-address '172.31.200.253'
+set high-availability vrrp group vlan200 priority '200'
+set high-availability vrrp group vlan200 vrid '200'
+set interfaces bonding bond0 address '172.31.1.252/24'
+set interfaces bonding bond0 description 'api-batch-test'
+set interfaces bonding bond0 hash-policy 'layer2+3'
+set interfaces bonding bond0 lacp-rate 'fast'
+set interfaces bonding bond0 member interface 'eth0'
+set interfaces bonding bond0 member interface 'eth1'
+set interfaces bonding bond0 mode '802.3ad'
+set interfaces bonding bond0 vif 2 address '172.31.2.252/24'
+set interfaces bonding bond0 vif 2 description 'k8s'
+set interfaces bonding bond0 vif 3 address '172.31.3.252/24'
+set interfaces bonding bond0 vif 3 description 'kvm'
+set interfaces bonding bond0 vif 9 address '172.31.9.252/24'
+set interfaces bonding bond0 vif 9 description 'private'
+set interfaces bonding bond0 vif 10 address '172.31.10.252/23'
+set interfaces bonding bond0 vif 10 description 'lot'
+set interfaces bonding bond0 vif 51 description 'WAN1 Vodafone-equivalent (sim ISP PPPoE)'
+set interfaces bonding bond0 vif 53 address 'dhcp'
+set interfaces bonding bond0 vif 53 description 'WAN3 10gig-equivalent (sim ISP DHCP)'
+set interfaces bonding bond0 vif 53 dhcp-options default-route-distance '210'
+set interfaces bonding bond0 vif 200 address '172.31.200.252/24'
+set interfaces bonding bond0 vif 200 description 'roomates'

labsim/vlan1-move-monitor.sh

@@ -0,0 +1,17 @@
+#!/bin/bash
+# Timestamped liveness log for the Management VLAN during the bond0 -> bond0.1 move.
+#
+# The question this answers is not "did it work" but "for how long was it not
+# working, and what held the VIP while it was not". Both are invisible after the
+# fact: VRRP reconverges and leaves no trace of who was master during the gap.
+#
+#   ./vlan1-move-monitor.sh > /tmp/move.log &
+# Columns: time  VIP-ping  R1-ping  R2-ping  VIP-mac
+VIP="${VIP:-172.31.1.1}"; R1="${R1:-172.31.1.252}"; R2="${R2:-172.31.1.253}"
+p() { ping -c1 -W1 -n "$1" >/dev/null 2>&1 && echo up || echo DOWN; }
+while :; do
+  mac="$(ip neigh show "$VIP" 2>/dev/null | awk '{for(i=1;i<=NF;i++) if($i=="lladdr") print $(i+1)}')"
+  printf '%s  vip=%-4s r1=%-4s r2=%-4s vipmac=%s\n' \
+    "$(date +%H:%M:%S)" "$(p "$VIP")" "$(p "$R1")" "$(p "$R2")" "${mac:-none}"
+  sleep 1
+done

migration/MANAGEMENT-VLAN-TAGGED.md

@@ -0,0 +1,137 @@
+# Moving Management onto a tagged VLAN
+
+Rehearsed end to end in labsim on 2026-09-02. This is the fix for kea serving
+addresses from the wrong VLAN's pool.
+
+## Why
+
+ISC Kea [#1117](https://gitlab.isc.org/isc-projects/kea/-/issues/1117): with
+`dhcp-socket-type: raw`, a frame tagged for a sub-interface is **also** delivered
+to the parent's `AF_PACKET` socket. If the parent serves a subnet, kea answers
+from it too. Ours does — Management is the native/untagged VLAN on `bond0` while
+VLANs 2/3/9/10/200 are sub-interfaces of that same bond — so one DISCOVER on
+VLAN 3 produces two OFFERs and the *client* decides which to keep:
+
+```
+bond0.3 :  192.168.3.14  correct
+bond0   :  192.168.1.28  UNTAGGED, Management pool, wrong
+```
+
+The fix is to leave **no subnet on the parent**: every VLAN tagged, Management
+included, moved from `bond0` to `bond0.1`.
+
+Confirmed in labsim across all six LAN VLANs: fails before, passes after.
+`labsim/labsim-vlan-leak-test.sh` is the test; evidence in
+`labsim/vlan-leak-evidence/`.
+
+## What must change together
+
+Per router:
+
+| | from | to |
+|---|---|---|
+| address | `interfaces bonding bond0 address` | `interfaces bonding bond0 vif 1 address` |
+| firewall | `interface-group LAN interface bond0` | `... interface bond0.1` |
+| VRRP | `vrrp group native interface bond0` | `... interface bond0.1` |
+| kea | — | **restart it** (see traps) |
+
+On the switch: Native VLAN = **None** on the trunk to that firewall, with VLAN 1
+added to the tagged set.
+
+## The ordering constraint
+
+**There is no overlap state.** An 802.1Q port always egresses its native VLAN
+untagged, so while VLAN 1 is native the router can *send* tagged VLAN 1 but can
+never *receive* it. Verified: a tagged VLAN 1 ARP sent from the switch arrived on
+`bond0` untagged and never on `bond0.1`. Configuring "native VLAN 1 **and** VLAN 1
+tagged" as a make-before-break does not work; the switch and router changes for a
+given firewall are strictly simultaneous, and that router loses Management in
+between.
+
+What makes this safe anyway: **tagged and untagged Management coexist on the
+same VLAN.** One VLAN is one broadcast domain no matter how each port tags it, so
+the firewalls can be converted one at a time — verified with the primary untagged
+and the secondary already tagged, both reachable, VIP up, VLAN 1 clients fine.
+
+Access ports are untouched throughout. The UniFi controller at 192.168.1.5 and
+your workstation are on access ports and never traverse the firewall trunks, so
+you keep the controller you are making the change from. Only the router being
+converted goes dark, and only until its own config lands.
+
+## Procedure
+
+Do the **backup** router first, then fail the VIPs over and do the other. You
+need console (JetKVM) on the router being converted — its Management SSH dies the
+moment the switch port changes.
+
+For each router in turn:
+
+1. Confirm the *other* router is MASTER and healthy:
+   `show vrrp` and `sudo /config/vrrp-wan-health; echo $?` (must be 0).
+2. Start the monitor from a workstation on an access port:
+   `labsim/vlan1-move-monitor.sh` (edit the three addresses for production).
+3. UniFi: on this firewall's trunk ports, Native VLAN → None, VLAN 1 → tagged.
+   This router's Management drops now.
+4. Over the console, in one commit:
+   ```
+   set interfaces bonding bond0 vif 1 address '192.168.1.252/24'   # .253 on vyos002
+   set interfaces bonding bond0 vif 1 description 'management'
+   delete interfaces bonding bond0 address
+   set firewall group interface-group LAN interface 'bond0.1'
+   delete firewall group interface-group LAN interface 'bond0'
+   set high-availability vrrp group native interface 'bond0.1'
+   commit
+   save
+   ```
+5. `sudo systemctl restart isc-kea-dhcp4-server` — see traps.
+6. Verify: Management SSH back, `show vrrp` shows `native` on `bond0.1`, and the
+   leak test passes.
+
+Then fail back if the VIPs moved (below), and repeat for the other router.
+
+### Measured windows (labsim)
+
+| | |
+|---|---|
+| this router's own Management unreachable | ~27 s (the console apply) |
+| VIP `.1` unreachable, peer already converted | **0 s** |
+| VIP `.1` unreachable, converting the current MASTER | ~6 s (VRRP failover) |
+| VIP unreachable if you convert both routers before the switch | **5 min 30 s** |
+
+That last row is the failure mode to avoid: with both routers untagged and the
+trunks already changed, the VIP is a black hole and **the healthy BACKUP does not
+take over**. Its `native` group stays BACKUP because the *other* VLANs still hear
+the master, and the sync group holds them together. Redundancy does not help you
+here; only ordering does.
+
+## Traps
+
+- **Restart kea.** VyOS does not restart it for an interface address change, so
+  it keeps a raw socket bound with the old address and keeps emitting the wrong
+  offers. The first post-fix test in the sim failed for this reason alone and
+  looked exactly like the fix not working.
+- **`interface-group LAN`.** Moving the address without moving the group means
+  Management falls outside the group, and with default-deny that is every
+  management session and all VLAN 1 inter-VLAN routing, gone on commit — on a
+  router you reach through itself. Use `commit-confirm` if you are not on console.
+- **The VIPs may move, and `no-preempt` keeps them moved.** Converting a router
+  restarts keepalived and re-initialises *every* group, not just `native`. In one
+  rehearsal the priority-100 secondary took all six VIPs and held them while the
+  priority-200 primary sat at BACKUP; in another the restart was quick enough that
+  nothing moved. It is non-deterministic — check afterwards, every time.
+  Fail back with `restart vrrp` **on the router currently holding them**.
+- **Duplicate delivery does not stop**, and should not be read as failure. #1117
+  only promises there is no longer a subnet on the parent to match. Expect two
+  identical replies per DISCOVER, both from the correct pool.
+- **Both firewalls' trunks must end up the same.** If UniFi shares one port
+  profile between them, changing it converts both at once and you get the 5m30s
+  row above. Check before you start; use per-port overrides if it does.
+
+## Not covered by the rehearsal
+
+- Whether UniFi's port profile can express "no native VLAN" the way OVS can, and
+  whether the two firewalls share a profile. Unverified — check on the controller.
+- Why the JetKVM consoles specifically accepted the wrong OFFER when a VLAN 3
+  access port should not receive an untagged VLAN 1 frame at all. Their port
+  profile likely passes VLAN 1 untagged. Worth confirming, though it does not
+  change the fix.

migration/RECOVERY-CARD-vlan1-move.md

@@ -0,0 +1,194 @@
+# Recovery card — moving Management to tagged VLAN 1
+
+Print or keep open. **During this change there is no internet, so no Claude.**
+Everything you need is on this page.
+
+---
+
+## The one thing that matters
+
+```
+ssh vyos@10.0.1.252
+```
+
+Your workstation is `10.0.0.210/23`; vyos001's LoT leg is `10.0.1.252/23`. Same
+subnet, same VLAN, **direct L2** — verified: `ip route get` returns
+`dev lanbr0 src 10.0.0.210` with no `via`, MAC `64:62:66:25:96:45`.
+
+It therefore does **not** depend on: the Management VLAN, VRRP, the VIPs,
+inter-VLAN routing, DNS, or the switch trunk config. If the router is up and its
+bond has link, this works. `bond0.10` is untouched by the change and stays in the
+firewall `LAN` group throughout.
+
+vyos002, once it is up, is `10.0.1.253` the same way.
+
+Other legs that also survive: `192.168.3.4` (kvm), `192.168.2.252` (Roomates).
+
+---
+
+## Before you touch anything
+
+```
+ssh vyos@10.0.1.252
+sudo /config/vyos-known-good save
+```
+
+The existing snapshot is from **2026-08-24** and predates today's fixes
+(eth2 removal, VRRP health-check) — restoring that one would undo them. Take a
+fresh one first. Check with `sudo /config/vyos-known-good status`.
+
+---
+
+## Order: switch FIRST, router SECOND
+
+This matters and is easy to get backwards.
+
+The UniFi controller is `192.168.1.5`, on the **Management** VLAN. Your
+workstation is on LoT and reaches it *through vyos001*. The moment the router
+has Management on `bond0.1` while the switch is still sending it untagged, that
+routing is dead — **and you lose the controller**, which is the thing you still
+need in order to change the switch.
+
+So:
+
+1. **UniFi first**, while everything still works:
+   USW Aggregation → port 1 `firewall001` (LAG, members 1+2) →
+   Native VLAN: Management → **None**, and make sure VLAN 1 is tagged/allowed.
+   *vyos001 loses Management the instant this lands. That is expected.*
+   Do **not** touch port 3 `firewall002` — that is vyos002, and it is down.
+2. **Router second**, over `ssh vyos@10.0.1.252` (still works — L2 direct).
+
+If UniFi will not offer "no native VLAN", stop and read *"If UniFi cannot do it"*
+below rather than improvising.
+
+---
+
+## The router change
+
+```
+ssh vyos@10.0.1.252
+configure
+set interfaces bonding bond0 vif 1 address '192.168.1.252/24'
+set interfaces bonding bond0 vif 1 description 'management'
+delete interfaces bonding bond0 address
+set firewall group interface-group LAN interface 'bond0.1'
+delete firewall group interface-group LAN interface 'bond0'
+set high-availability vrrp group native interface 'bond0.1'
+commit-confirm 10
+save
+exit
+```
+
+**Use `commit-confirm 10`, not `commit`.** If it goes wrong and you cannot get
+back in, the router reverts itself after 10 minutes and comes back on its own.
+That is your safety net with no internet and no help.
+
+Once you have confirmed it works (below), run:
+
+```
+configure
+confirm
+save
+exit
+```
+
+`save` after `confirm`, or a reboot loses it.
+
+### Then, and this is the step that gets forgotten
+
+```
+sudo systemctl restart isc-kea-dhcp4-server
+```
+
+VyOS does **not** restart kea for an interface address change. Without this it
+keeps a raw socket bound to the old address and keeps handing out wrong-VLAN
+addresses — the fix looks like it did nothing. Give it ~60s before judging;
+kea reopens sockets on a retry loop and answers nothing for a while after a
+restart (measured: still silent at 55s in the sim, then fine).
+
+Also check DNS came back, since the forwarder binds the VIP `192.168.1.1`:
+
+```
+sudo systemctl status pdns-recursor --no-pager | head -3
+dig @192.168.1.1 google.com +short
+```
+
+---
+
+## Verify
+
+```
+ssh vyos@192.168.1.252          # Management back, now tagged
+show vrrp                        # native should be on bond0.1
+show dhcp server leases | head
+```
+
+Then from a machine on VLAN 3, force a DHCP renew and confirm it gets a
+`192.168.3.x` address and not a `192.168.1.x` one.
+
+---
+
+## If you are locked out
+
+In order:
+
+1. **Wait 10 minutes.** `commit-confirm` reverts by itself. This is the answer
+   most of the time. Do not power-cycle during this — you will lose the revert.
+2. `ssh vyos@10.0.1.252` — the LoT leg. Then `configure` / `rollback 1` / `commit`.
+3. Other legs: `ssh vyos@192.168.3.4`, `ssh vyos@192.168.2.252`.
+4. `sudo /config/vyos-known-good restore` — back to the snapshot you took at the
+   start. It is itself commit-confirmed, so even this cannot strand you.
+5. Put the UniFi port back: Native VLAN → Management on USW Aggregation port 1.
+   That alone restores the old shape and Management comes back untagged.
+
+**Do not** power-cycle vyos001 as a first move. Everything above is faster and
+non-destructive, and a reboot loses an unsaved `commit-confirm` revert.
+
+---
+
+## Do NOT power on vyos002 yet
+
+It still has `interfaces ethernet eth2 address 192.168.8.144/23` on the box — the
+same subnet as `bond0.2`. That is what ARP-poisoned `192.168.8.1` and took the
+cluster down. It also has no `/config/vrrp-wan-health`, so it can take the
+floating IPs with no WAN.
+
+Its console (`kvm - vyos002`, US24 port 9) is currently **unreachable** — it sits
+on a VLAN 3 port holding a Management lease `192.168.1.28`, which is the very bug
+being fixed here. Fixing DHCP first is what gets that console back.
+
+---
+
+## If UniFi cannot do it
+
+Classic UniFi (this is a classic controller, 10.4.57) may not offer
+"Native VLAN = None" — every switch port has a PVID. Two things make this awkward
+here: Management is UniFi's *default* network with **no VLAN ID at all**
+(`vlan: null`), so there may be nothing to "tag VLAN 1" with.
+
+If so, **stop and change nothing.** The workaround is to point the trunk's native
+VLAN at a VLAN the router does not serve (so `bond0` still ends up with no
+subnet), which needs a throwaway VLAN-only network created first. That is a
+design decision, not something to improvise at 1am with no internet. Put the port
+back to Native = Management and everything returns to today's working state.
+
+---
+
+## Facts worth having on paper
+
+| | |
+|---|---|
+| vyos001 Management | `192.168.1.252` → becomes `bond0.1` |
+| vyos001 LoT (recovery) | `10.0.1.252`, L2-direct from your workstation |
+| vyos002 Management | `192.168.1.253` (down) |
+| VIP Management | `192.168.1.1` |
+| UniFi controller | `192.168.1.5` (on Management — you lose it mid-change) |
+| firewall001 trunk | USW Aggregation port 1, LAG members 1+2 |
+| firewall002 trunk | USW Aggregation port 3, LAG members 3+4 |
+| SSH user / pass | `vyos` / `vyos` |
+| vyos001 bond MAC | `64:62:66:25:96:45` |
+
+Measured in labsim: converting the router while the peer is already converted
+costs **0s** of VIP downtime; converting it while it holds the VIPs costs about
+**6s**. vyos002 is down, so vyos001 holds everything — expect the ~6s, and expect
+Management to be gone from the UniFi change until the router change lands.

migration/vrrp-wan-apply

@@ -0,0 +1,32 @@
+#!/bin/vbash
+# Enable or disable the WAN. Split out from vrrp-wan-reconcile for one reason:
+# `source /opt/vyatta/etc/functions/script-template` must be the FIRST thing the
+# script does. Sourced after a few statements -- an if, an exec, a mkdir -- it
+# silently terminated the script; `set -x` showed execution stopping inside the
+# source with no error and rc=0, so the reconciler reported success having done
+# nothing. Only a single assignment may precede it (the template resets the
+# positional parameters, so the mode is captured first), which is the same shape
+# /config/vyos-known-good uses.
+#
+#   vrrp-wan-apply enable    take the WAN
+#   vrrp-wan-apply disable   release it
+MODE="${1:-}"
+source /opt/vyatta/etc/functions/script-template
+
+WAN_VIF=53
+cfg() { /opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration commands 2>/dev/null; }
+wan_disabled(){ cfg | grep -q "vif ${WAN_VIF} disable"; }
+ppp_disabled(){ cfg | grep -q "pppoe pppoe0 disable"; }
+
+configure
+if [ "$MODE" = enable ]; then
+    # Guarded: `delete` of an absent node aborts the whole batch with
+    # "Nothing to delete", which left the box detected-but-unfixed.
+    wan_disabled && delete interfaces bonding bond0 vif ${WAN_VIF} disable
+    ppp_disabled && delete interfaces pppoe pppoe0 disable
+else
+    wan_disabled || set interfaces bonding bond0 vif ${WAN_VIF} disable
+    ppp_disabled || set interfaces pppoe pppoe0 disable
+fi
+commit
+exit

migration/vrrp-wan-health

@@ -0,0 +1,68 @@
+#!/bin/sh
+# VRRP health check: may THIS router hold the floating IPs?
+#
+# It may only if it can actually carry the WAN. Without this, VRRP decides
+# mastership purely on whether the peer is still advertising -- so a router with
+# no WAN at all happily takes the VIPs and blackholes the entire LAN's internet
+# while looking perfectly healthy. That is not hypothetical: it is the outage of
+# 2026-09-02, reproduced in labsim.
+#
+# ---------------------------------------------------------------------------
+# The first version of this script asked one question: "do I have an address on
+# a WAN interface". That is correct for a pair where both routers hold WAN all
+# the time. Ours cannot: the 10 gig lease is bound to a cloned MAC and the
+# PPPoE line to a single credential, so the WAN follows mastership (see
+# vrrp-wan-take). Against that design the old check DEADLOCKS --
+#
+#     may I be master?  ->  only if I already have WAN
+#     do I have WAN?    ->  only if I am master
+#
+# -- and the backup sits in FAULT for ever. vyos002 sat exactly there, which
+# meant the pair could not fail over at all: the safety check had quietly
+# removed the redundancy it was protecting.
+#
+# So the question is now asked in the right order: enforce "must have WAN" only
+# on the router that is actually HOLDING the VIPs, and give a new master time to
+# bring the WAN up before judging it.
+# ---------------------------------------------------------------------------
+#
+# exit 0 = eligible for MASTER, non-zero = release and let the peer have it.
+
+STATE=/run/vrrp-wan
+GRACE=90          # seconds a new master gets to complete DHCP / PPPoE dial-up
+VIP="${VRRP_WAN_VIP:-192.168.1.1}"
+
+# Am I holding the VIPs? Asked of REALITY -- is the management VIP actually on
+# this box -- and not of a /run marker.
+#
+# The marker was the first design and it is unsafe: it is written by the VRRP
+# transition script, and in labsim that script silently failed to run on a
+# promotion (VyOS's keepalived-fifo.py helper stopped delivering while
+# keepalived's own notifies kept working). The router then believed it was
+# backup, passed this check, and sat holding every VIP with no WAN -- the exact
+# outage this script exists to prevent, re-created by trusting the reporter
+# instead of the fact.
+[ -n "$(ip -4 -o addr show 2>/dev/null | grep " ${VIP}/")" ] || exit 0
+
+# Master with an address on a WAN interface: healthy.
+#
+# Deliberately NOT "can I reach the internet" and NOT "do I have a default
+# route". During a real ISP outage the route disappears on BOTH routers; a check
+# keyed on that would put both into FAULT, nobody would hold the VIPs, and the
+# LAN would lose inter-VLAN routing too -- turning an internet outage into a
+# total one. A DHCP lease survives an ISP outage, so an address still
+# distinguishes "this box structurally cannot route" from "the internet is down
+# right now", which is the distinction that matters.
+for ifc in bond0.53 pppoe0; do
+    ip -4 addr show dev "$ifc" 2>/dev/null | grep -q 'inet ' && exit 0
+done
+
+# Master, no WAN yet, still within the grace window: DHCP negotiation and PPPoE
+# dial-up take real time, and the ISP has to accept the cloned MAC arriving on a
+# different port. Failing here would demote the new master before it ever had a
+# chance, and hand the VIPs straight back -- a flap, not a failover.
+since=$(cat "$STATE/since" 2>/dev/null || echo 0)
+[ $(( $(date +%s) - since )) -lt "$GRACE" ] && exit 0
+
+# Master, past grace, still no WAN: release. This is the 2026-09-02 case.
+exit 1

migration/vrrp-wan-reconcile

@@ -0,0 +1,106 @@
+#!/bin/sh
+# Make the WAN match VRRP mastership. Idempotent; safe to run every 30s and on
+# every VRRP transition.
+#
+# Why a reconciler and not just transition scripts
+# ------------------------------------------------
+# VyOS delivers `transition-script` through a helper process,
+# /usr/libexec/vyos/system/keepalived-fifo.py, fed by keepalived's notify_fifo.
+# Observed in labsim on 2026-09-02: the primary's Keepalived_vrrp logged
+# "(native) Entering MASTER STATE" for all six instances and the built-in
+# notify_master for conntrack-sync ran -- while the fifo helper logged NOTHING
+# and the master transition script never ran. The helper process was still
+# alive. The result was a router holding every VIP with no WAN at all: the exact
+# 2026-09-02 outage, re-created by the mechanism meant to prevent it.
+#
+# So transition scripts are kept for speed but nothing is trusted to them: this
+# also runs on a timer, and derives everything from ground truth rather than
+# from a marker that only exists if the script it depends on ran.
+#
+#   vrrp-wan-reconcile          reconcile once
+#   vrrp-wan-reconcile --status what it thinks, changing nothing
+#
+# Ground truth for "am I master" is whether the management VIP is really on this
+# box. It is what VRRP actually does, it is observable, and it cannot silently
+# disagree with reality.
+
+VIP="${VRRP_WAN_VIP:-192.168.1.1}"      # management VIP; sim overrides via env
+WAN_VIF=53                               # bond0.53, the DHCP WAN
+STATE=/run/vrrp-wan
+LOCK=/run/vrrp-wan.lock
+
+cfg() { /opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration commands 2>/dev/null; }
+holds_vip()   { ip -4 -o addr show 2>/dev/null | grep -q " ${VIP}/"; }
+wan_up()      { ip -4 addr show "bond0.${WAN_VIF}" 2>/dev/null | grep -q 'inet '; }
+wan_disabled(){ cfg | grep -q "vif ${WAN_VIF} disable"; }
+ppp_disabled(){ cfg | grep -q "pppoe pppoe0 disable"; }
+
+# --status must answer WITHOUT sourcing script-template. The template's `exit`
+# is a function that leaves configuration mode, not the shell builtin, so a
+# status run that had sourced it opened and closed a config session on every
+# call -- which is how a read-only query started colliding with the timer and
+# logging "Configuration system temporarily locked due to another commit".
+if [ "${1:-}" = "--status" ]; then
+    printf 'vip=%s holds_vip=%s wan_disabled=%s wan_up=%s role=%s\n' \
+        "$VIP" "$(holds_vip && echo yes || echo no)" \
+        "$(wan_disabled && echo yes || echo no)" \
+        "$(wan_up && echo yes || echo no)" \
+        "$(cat "$STATE/role" 2>/dev/null || echo unset)"
+    exit 0
+fi
+
+# One writer. The 30s timer and a VRRP transition can fire together, and two
+# VyOS commits in flight on one box do not queue -- the second fails outright.
+exec 9>"$LOCK"
+flock -n 9 || exit 0
+
+mkdir -p "$STATE"
+
+# Reap config sessions whose owning process is gone. VyOS creates
+# /opt/vyatta/config/tmp/new_config_<pid> (a unionfs mount) per `configure`, and
+# a script that dies inside a session never removes it. One of those holds the
+# commit lock, and from then on EVERY commit fails with "Configuration system
+# temporarily locked due to another commit in progress" -- including the manual
+# one you try in order to fix it. A job on a 30s timer that can leak a session
+# per failure will wedge the router's config system on its own, so it cleans up
+# before it starts. `umount -l` first: the directory is a mount point and plain
+# rm returns "Device or resource busy".
+for d in /opt/vyatta/config/tmp/new_config_*; do
+    [ -d "$d" ] || continue
+    pid=${d##*_}
+    kill -0 "$pid" 2>/dev/null && continue
+    umount -l "$d" 2>/dev/null
+    rm -rf "$d" 2>/dev/null
+done
+
+# The config edit lives in vrrp-wan-apply, because script-template must be the
+# first thing its script does -- sourced any later it terminates the script
+# silently with rc=0. See the header there.
+APPLY=/config/vrrp-wan-apply
+
+if holds_vip; then
+    echo master > "$STATE/role"
+    # Stamp only on entry to master, so the health check's grace window measures
+    # time-since-promotion rather than time-since-last-tick.
+    [ -f "$STATE/since" ] || date +%s > "$STATE/since"
+    wan_disabled || ppp_disabled || exit 0
+    logger -t vrrp-wan "MASTER with WAN disabled -> enabling bond0.${WAN_VIF} + pppoe0"
+    "$APPLY" enable
+else
+    echo backup > "$STATE/role"
+    rm -f "$STATE/since"
+    { wan_disabled && ppp_disabled; } && exit 0
+    # Releasing matters more than taking. A demoted router that keeps the WAN up
+    # holds the cloned MAC f0:9f:c2:12:9b:4f on VLAN 53 at the same time as the
+    # new master, and the switch sends the ISP's replies to whichever port spoke
+    # last -- the WAN-side twin of the eth2 incident.
+    logger -t vrrp-wan "not MASTER but WAN enabled -> releasing bond0.${WAN_VIF} + pppoe0"
+    "$APPLY" disable
+fi
+
+# Deliberately no `save`. config.boot keeps `disable` on BOTH routers, so a
+# reboot in any order comes up unable to claim the shared MAC, and only holding
+# the VIP re-enables it. NOTE: any `save` while this box is master (a hand
+# commit, or `pulumi up`) WILL persist the enabled state -- observed in labsim.
+# The Pulumi model asserts `disable` on both routers so an apply puts it back,
+# and vyos:verify reports it as drift if it does not.

migration/vrrp-wan-reconcile.service

@@ -0,0 +1,10 @@
+[Unit]
+# Belt to the transition scripts' braces. VyOS's keepalived-fifo.py helper was
+# observed dropping a MASTER transition silently, leaving a router holding every
+# VIP with no WAN. A timer cannot be dropped the same way.
+Description=Reconcile WAN interface state with VRRP mastership
+After=keepalived.service
+
+[Service]
+Type=oneshot
+ExecStart=/config/vrrp-wan-reconcile

migration/vrrp-wan-reconcile.timer

@@ -0,0 +1,13 @@
+[Unit]
+Description=Reconcile WAN with VRRP mastership every 30s
+
+[Timer]
+# 30s: fast enough that a dropped transition is a blip rather than an outage,
+# slow enough that it is never the thing generating load. It only commits when
+# state actually disagrees, so a steady-state tick is two `ip` calls and a grep.
+OnBootSec=60
+OnUnitActiveSec=30
+AccuracySec=5
+
+[Install]
+WantedBy=timers.target

migration/vrrp-wan-release

@@ -0,0 +1,5 @@
+#!/bin/sh
+# VRRP transition hook. One code path: the reconciler derives everything from
+# ground truth, so take and release are the same operation asked at different
+# moments. Speed comes from here; correctness comes from the timer.
+exec /config/vrrp-wan-reconcile

migration/vrrp-wan-take

@@ -0,0 +1,5 @@
+#!/bin/sh
+# VRRP transition hook. One code path: the reconciler derives everything from
+# ground truth, so take and release are the same operation asked at different
+# moments. Speed comes from here; correctness comes from the timer.
+exec /config/vrrp-wan-reconcile

migration/vyos002-catch.sh

@@ -0,0 +1,85 @@
+#!/bin/bash
+# Arm BEFORE powering vyos002 on. Strips the eth2 address the moment the box is
+# reachable, then installs the VRRP health-check.
+#
+# Why this exists: vyos002 boots with `interfaces ethernet eth2 address
+# 192.168.8.144/23` still in config.boot -- the same subnet as bond0.2. Linux
+# answers ARP for any local address out of any interface on that L2, so eth2
+# answers for addresses that bond0.2 is supposed to route, and traffic lands on
+# a port that does not route it. That is the 2026-09-02 cluster outage.
+#
+# A human "jumping on it fast" loses this race more often than not; the box is
+# reachable within a second or two of the interfaces coming up. This polls at
+# 1s and commits the moment it gets in.
+#
+#   ./vyos002-catch.sh            arm and wait (Ctrl-C to disarm)
+#
+# Bounded risk while you wait, worth knowing: vyos002 comes up BACKUP (priority
+# 100, no-preempt, vyos001 healthy MASTER), so it does NOT hold 192.168.8.1 and
+# the GATEWAY cannot be poisoned. The exposure is its own bond0.2 address, which
+# is survivable. The unbounded case is it becoming MASTER while eth2 is present
+# -- which is exactly what the health-check in step 2 prevents.
+set -uo pipefail
+
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PW="${VYOS_PW:-vyos}"
+# Management first because it comes up with the box; LoT is the fallback and is
+# L2-direct from this workstation (see RECOVERY-CARD-vlan1-move.md).
+TARGETS=("192.168.1.253" "10.0.1.253")
+SSH_OPTS=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
+          -o LogLevel=ERROR -o ConnectTimeout=2 -o PreferredAuthentications=password)
+
+log() { printf '\033[36m[catch %s]\033[0m %s\n' "$(date +%T)" "$*"; }
+
+on() { timeout 12 sshpass -p "$PW" ssh "${SSH_OPTS[@]}" "vyos@$1" "$@"; }
+
+log "armed -- polling ${TARGETS[*]} every 1s. Power on vyos002 now."
+HOST=""
+while [ -z "$HOST" ]; do
+  for t in "${TARGETS[@]}"; do
+    if timeout 4 sshpass -p "$PW" ssh "${SSH_OPTS[@]}" "vyos@$t" true 2>/dev/null; then
+      HOST="$t"; break
+    fi
+  done
+  [ -z "$HOST" ] && sleep 1
+done
+log "CAUGHT on $HOST -- stripping eth2 address"
+
+# Step 1, on its own commit: get the address off eth2 before anything else. Any
+# extra command in this commit is extra seconds of exposure.
+timeout 90 sshpass -p "$PW" ssh "${SSH_OPTS[@]}" "vyos@$HOST" 'vbash -s' <<'EOF' 2>&1 | tail -3
+source /opt/vyatta/etc/functions/script-template
+delete interfaces ethernet eth2 address
+commit
+echo "ETH2_RC=$?"
+save
+exit
+EOF
+log "eth2 address removed"
+
+# Step 2: the health-check. Without it this box can hold every floating IP while
+# having no WAN -- the outage itself. Copy the script BEFORE referencing it, or
+# the commit succeeds and the check silently never passes.
+timeout 30 sshpass -p "$PW" scp "${SSH_OPTS[@]}" \
+  "$HERE/vrrp-wan-health" "vyos@$HOST:/tmp/vrrp-wan-health" >/dev/null 2>&1 \
+  && log "health-check script copied" || log "WARN: scp failed -- step 2 will be skipped"
+
+timeout 90 sshpass -p "$PW" ssh "${SSH_OPTS[@]}" "vyos@$HOST" 'vbash -s' <<'EOF' 2>&1 | tail -3
+sudo install -o root -g vyattacfg -m 0775 /tmp/vrrp-wan-health /config/vrrp-wan-health
+source /opt/vyatta/etc/functions/script-template
+set high-availability vrrp sync-group MAIN health-check script '/config/vrrp-wan-health'
+set high-availability vrrp sync-group MAIN health-check interval '5'
+set high-availability vrrp sync-group MAIN health-check failure-count '3'
+commit
+echo "HEALTH_RC=$?"
+save
+exit
+EOF
+log "health-check installed"
+
+echo
+log "=== state ==="
+timeout 30 sshpass -p "$PW" ssh "${SSH_OPTS[@]}" "vyos@$HOST" \
+  'echo "-- eth2 (must show no inet) --"; ip -4 addr show eth2 2>/dev/null | grep inet || echo "   none"
+   echo "-- health-check --"; sudo /config/vrrp-wan-health; echo "   exit=$? (non-zero = no WAN = refuses the VIPs, which is CORRECT and safe)"
+   echo "-- vrrp --"; /opt/vyatta/bin/vyatta-op-cmd-wrapper show vrrp' 2>&1

migration/vyos002-return.conf

@@ -0,0 +1,37 @@
+# vyos002 — everything that must land before it is trusted on the network.
+#
+# Apply order matters only in that this is ONE commit: the eth2 address and the
+# missing health-check are the two defects that caused the 2026-09-02 outage, and
+# neither should survive a single reboot window.
+#
+#   ssh vyos@192.168.1.253   (or 10.0.1.253 — L2-direct, see RECOVERY-CARD)
+#   configure; <paste>; commit; save
+
+# 1. The ARP poisoner. eth2 is the 1G copper NIC on US24 port 16, native VLAN 2,
+#    and it held 192.168.8.144/23 -- the same subnet as bond0.2. Two interfaces
+#    answering for one subnet is what hijacked 192.168.8.1 and took the cluster
+#    down: eth2's MAC answered while bond0.2's MAC routed.
+#    Origin: eth2 was `address dhcp`, a kea reservation for the ROUTER'S OWN NIC
+#    handed it .144, and a CLI commit froze it static.
+delete interfaces ethernet eth2 address
+
+# 2. The health-check. Without it this box can hold every floating IP while
+#    having no WAN at all -- the outage itself. It goes on the SYNC GROUP; VyOS
+#    rejects it per-group.
+#    /config/vrrp-wan-health must be copied over FIRST (from migration/) and be
+#    chmod +x, or the commit succeeds and the check silently never passes.
+set high-availability vrrp sync-group MAIN health-check script '/config/vrrp-wan-health'
+set high-availability vrrp sync-group MAIN health-check interval '5'
+set high-availability vrrp sync-group MAIN health-check failure-count '3'
+
+# 3. Management onto a tagged sub-interface, matching vyos001 (kea #1117).
+#    Pair this with USW Aggregation port 3 (LAG 3+4) -> Native VLAN = None.
+#    Until that switch change lands, leave these three commented out: vyos002
+#    can run untagged on bond0 while vyos001 runs tagged -- one VLAN is one
+#    broadcast domain, and the coexistence was proven in labsim.
+# set interfaces bonding bond0 vif 1 address '192.168.1.253/24'
+# set interfaces bonding bond0 vif 1 description 'management'
+# delete interfaces bonding bond0 address
+# set firewall group interface-group LAN interface 'bond0.1'
+# delete firewall group interface-group LAN interface 'bond0'
+# set high-availability vrrp group native interface 'bond0.1'