test(labsim): second VyOS router proves DHCP active-passive HA

Answers the question a single router could not, and that would otherwise only
have been discovered at cutover: with kea high-availability active-passive, does
exactly ONE box answer a DHCP request?

Yes. Probing sim VLAN 10 with broadcast-dhcp-discover returns offers from a
single distinct Server Identifier -- 172.31.10.252, the primary. The secondary
runs kea but stays silent. Without this the delta would have put 6 subnets and
84 static-mappings on both boxes with nothing to arbitrate them, and two kea
instances would have raced on every broadcast domain.

Worth noting the raw response count is misleading: nmap reports "Response 1 of
2" because it sends several discovers, and both replies carry the same server
identifier. Counting responses says "2 servers"; counting distinct server
identifiers says "1". The second number is the true one.

labsim now runs a real pair, mirroring production:

    router1  172.31.<v>.252  priority 200  DHCP HA primary
    router2  172.31.<v>.253  priority 100  DHCP HA secondary
    VIP      172.31.<v>.1    floating, held by the master

That required converting router1, which held .1 directly, to .252 plus a
floating VIP -- otherwise it is two routers, not a pair. All six VRRP groups
show MASTER on router1 and BACKUP on router2.

New tooling:

  - sim-ha-config.py generates each role's config, reusing unifi-to-vyos.py
    --mode sim for the DHCP half so what is proven here and what production
    gets share a code path. VLAN 10 correctly carries /23.
  - console-apply.py applies config over the serial console, which is necessary
    because a freshly installed VyOS holds the same addresses as its peer and
    cannot safely be reached over the network at all until reconfigured.

Known sim-only quirk, deliberately not chased: router1 cannot ARP router2 on
the untagged VLAN 1 while every tagged VLAN works, and VRRP forms correctly on
all six groups regardless. Both OVS bonds carry identical vlan_mode/tag/trunks
and the bond MACs differ, so this is OVS bond behaviour on the native VLAN with
two bonds on one bridge -- not a VyOS config problem, and not present in
production, which uses a real switch.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH

labsim/console-apply.py

@@ -0,0 +1,89 @@
+#!/usr/bin/env python3
+"""Apply VyOS config to a labsim VM over its serial console.
+
+Needed because a freshly installed VyOS comes up holding the same addresses as
+its peer, so there is a window where it cannot safely be reached over the
+network at all. The console does not care.
+
+  ./console-apply.py --vm labsim-vyos2 --config r2.conf
+"""
+from __future__ import annotations
+
+import argparse
+import sys
+import time
+
+import pexpect
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--vm", required=True)
+    ap.add_argument("--config", required=True)
+    ap.add_argument("--user", default="vyos")
+    ap.add_argument("--password", default="vyos")
+    args = ap.parse_args()
+
+    cmds = [l.rstrip() for l in open(args.config)
+            if l.strip() and not l.lstrip().startswith("#")]
+    print(f"{len(cmds)} commands to apply to {args.vm}", file=sys.stderr)
+
+    c = pexpect.spawn(f"virsh --connect qemu:///system console {args.vm}",
+                      timeout=90, encoding="utf-8")
+    c.logfile_read = None
+    c.sendline("")
+    time.sleep(2)
+    c.sendline("")
+
+    # Log in. A freshly booted box may still be starting services, so allow a
+    # generous window and re-prod the console rather than failing on the first
+    # miss.
+    for _ in range(40):
+        i = c.expect([r"login:", r"\$ ", r"# ", pexpect.TIMEOUT], timeout=15)
+        if i == 0:
+            c.sendline(args.user)
+            c.expect("Password:", timeout=30)
+            c.sendline(args.password)
+            c.expect(r"\$ ", timeout=60)
+            break
+        if i in (1, 2):
+            break
+        c.sendline("")
+    else:
+        print("never reached a prompt", file=sys.stderr)
+        return 1
+
+    c.sendline("configure")
+    c.expect(r"# ", timeout=60)
+
+    for cmd in cmds:
+        c.sendline(cmd)
+        c.expect(r"# ", timeout=60)
+        out = c.before or ""
+        if "Set failed" in out or "not valid" in out or "Invalid" in out:
+            print(f"FAILED: {cmd}\n  {out.strip()[:200]}", file=sys.stderr)
+
+    print("committing...", file=sys.stderr)
+    c.sendline("commit")
+    c.expect(r"# ", timeout=300)
+    commit_out = c.before or ""
+    c.sendline("save")
+    c.expect(r"# ", timeout=120)
+    c.sendline("exit")
+    c.expect(r"\$ ", timeout=60)
+    c.sendline("exit")
+    c.close()
+
+    bad = [l for l in commit_out.splitlines()
+           if "failed" in l.lower() or "error" in l.lower()]
+    if bad:
+        print("commit reported:", file=sys.stderr)
+        for l in bad[:10]:
+            print(f"  {l.strip()}", file=sys.stderr)
+        return 1
+    print("committed and saved", file=sys.stderr)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

labsim/sim-ha-config.py

@@ -0,0 +1,109 @@
+#!/usr/bin/env python3
+"""Generate the HA config for the labsim VyOS pair.
+
+Exists to answer one question that cannot be answered on a single router, and
+that would otherwise only be discovered at cutover: with kea HA active-passive,
+does exactly ONE box answer a DHCP request?
+
+Mirrors the production shape so the answer transfers:
+
+    router1  172.31.<v>.252   priority 200   DHCP HA primary
+    router2  172.31.<v>.253   priority 100   DHCP HA secondary
+    VIP      172.31.<v>.1     (what clients use as their gateway)
+
+Note the sim's LoT VLAN is a /23 like production, so the VIP prefix differs
+there -- getting that wrong produces a config that commits and then behaves
+subtly wrongly, which is worse than a failure.
+
+  ./sim-ha-config.py --role primary   > r1.conf
+  ./sim-ha-config.py --role secondary > r2.conf
+"""
+from __future__ import annotations
+
+import argparse
+import importlib.util
+import json
+import os
+import sys
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+MIG = os.path.join(HERE, "..", "migration")
+
+# Reuse the DHCP/DNS generator rather than hand-writing subnets: the whole
+# point is that what is proven here and what production gets share a code path.
+_spec = importlib.util.spec_from_file_location(
+    "unifi_to_vyos", os.path.join(MIG, "unifi-to-vyos.py"))
+unifi_to_vyos = importlib.util.module_from_spec(_spec)
+_spec.loader.exec_module(unifi_to_vyos)
+
+# vlan -> (prefix, cidr). LoT is a /23 in the sim, matching production.
+VLANS = {
+    1:   ("172.31.1",   24),
+    2:   ("172.31.2",   24),
+    3:   ("172.31.3",   24),
+    9:   ("172.31.9",   24),
+    10:  ("172.31.10",  23),
+    200: ("172.31.200", 24),
+}
+DHCP_HA_NAME = "labsim-dhcp-pair"     # must not equal either host-name
+
+
+def group(vlan: int) -> str:
+    return "native" if vlan == 1 else f"vlan{vlan}"
+
+
+def build(role: str) -> list[str]:
+    primary = role == "primary"
+    self_o, peer_o = (252, 253) if primary else (253, 252)
+    prio = 200 if primary else 100
+    out = [f"# labsim VyOS HA -- {role}", ""]
+
+    for vlan, (pfx, cidr) in VLANS.items():
+        g = group(vlan)
+        iface = "bond0" if vlan == 1 else f"bond0 vif {vlan}"
+        out += [
+            f"# VLAN {vlan}",
+            # The node's own address replaces the .1 it used to hold directly;
+            # .1 becomes the floating VIP, exactly as production will be.
+            f"delete interfaces bonding {iface} address",
+            f"set interfaces bonding {iface} address '{pfx}.{self_o}/{cidr}'",
+            f"set high-availability vrrp group {g} interface bond0{'' if vlan == 1 else f'.{vlan}'}",
+            f"set high-availability vrrp group {g} vrid {vlan}",
+            f"set high-availability vrrp group {g} address {pfx}.1/{cidr}",
+            f"set high-availability vrrp group {g} priority {prio}",
+            f"set high-availability vrrp group {g} hello-source-address {pfx}.{self_o}",
+            f"set high-availability vrrp group {g} peer-address {pfx}.{peer_o}",
+            f"set high-availability vrrp group {g} no-preempt",
+            f"set high-availability vrrp sync-group MAIN member {g}",
+            "",
+        ]
+
+    out += [
+        "# --- DHCP high-availability ---",
+        "# The thing under test: active-passive should mean exactly one OFFER.",
+        "set service dhcp-server high-availability mode active-passive",
+        f"set service dhcp-server high-availability status {role}",
+        f"set service dhcp-server high-availability name {DHCP_HA_NAME}",
+        f"set service dhcp-server high-availability source-address 172.31.10.{self_o}",
+        f"set service dhcp-server high-availability remote 172.31.10.{peer_o}",
+        "",
+    ]
+
+    inv = json.load(open(os.path.join(MIG, "export", "inventory.json")))
+    dhcp, stats = unifi_to_vyos.build(inv, "sim")
+    out += [l for l in dhcp if l.strip() and not l.startswith("#")]
+    print(f"{role}: {stats['subnets']} subnets, {stats['mappings']} mappings",
+          file=sys.stderr)
+    return out
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--role", choices=("primary", "secondary"), required=True)
+    args = ap.parse_args()
+    sys.stdout.write("\n".join(build(args.role)) + "\n")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())