vyos: move PPPoE off the config plane onto a gated systemd unit

PPPoE HA could not work as written, and the reason is structural rather than a
bug: `set interfaces pppoe pppoe0 disable` and `delete` are handled identically
by interfaces_pppoe.py -- both UNLINK /etc/ppp/peers/pppoe0. That path is pppd's
own options file, so the resting state destroyed exactly what the promotion path
needed, and `ppp@pppoe0` restart-looped against it (observed: 47 restarts, zero
sessions at the access concentrator). It also made op-mode `connect interface
pppoe0` unusable, and put every failover behind a priority-322 commit where one
unrelated invalid node fails the whole thing -- which has already taken the
10 gig down once.

pppoe0 is now configured identically and ENABLED on both routers, so the peers
file always exists, and dialling is gated by a drop-in on the unit:

    ConditionPathExists=/run/vrrp-wan/may-dial
    ConditionPathExists=/etc/ppp/peers/pppoe0

/run is tmpfs, so the gate is shut at boot and neither box can dial before VRRP
has decided. That matters more than it first appears: with the node enabled,
interfaces_pppoe.py restarts ppp on EVERY commit touching the pppoe subtree when
the daemon is not running -- so the backup actively tries to dial whenever
anything commits. The gate is the only thing making that a no-op, which is why
vrrp-wan-reconcile now refuses to bless a box whose drop-in is missing: /etc is
per-image, and a VyOS upgrade would otherwise silently remove the protection.

may-dial is a LEASE, not a flag. ConditionPathExists is evaluated at start only
-- it can prevent a dial, never revoke one -- so a reconciler that stops running
while its box is demoted would keep the one ISP session for ever. The reconciler
renews the lease; a new 5s vrrp-wan-guard revokes it, and only ever revokes. It
fired correctly first time: "GUARD: lease stale (81s > 75s)".

Also: remove-then-stop on release (the file's absence blocks a NEW start that a
concurrent commit would trigger); a flap damper, because two routers that both
believe they hold the VIP will both dial and each dial kills the other's session
-- against a real ISP that is how an account gets rate-limited; and a guard on
`cfg` returning empty under commit-lock contention, which had already produced
one spurious "releasing" on a box that needed nothing.

GRACE 90 -> 180. accel-ppp's dead-peer budget is lcp-echo-interval(30) x
failure(3) = 90s, so the old value sat exactly on the boundary: a hard failover
into an AC that does not replace the stale session would fail its own check,
shed the VIPs, and leave both routers in FAULT.

The sim could not have tested any of this. Both routers now get the identical
WAN -- the secondary had none "because two PPPoE clients sharing one credential
is a different failure mode than anything production has", which is backwards:
that IS production. It also left the pair incomparable, ten NAT rules against
none. Safety now comes from resting state, not asymmetry.

Three more things the sim was hiding:
  - the drift check's secondary regex omitted interfaces pppoe/bonding, nat
    source and protocols failover, so it reported "in sync" for a box with no
    WAN at all;
  - the VRRP health-check and transition-script hooks existed on both live VMs
    and in NEITHER generator -- the mechanism under test was pure undetected
    drift;
  - labsim-vyos's only default route was the libvirt-NAT scaffold, so every
    "the LAN still has internet" verdict on it was answered by eth2 rather than
    the WAN. --drop-scaffold applied; the earlier DHCP-failover proof is being
    re-run because of it.

vrrp-wan-install ends the other half of that: the sim's previous proof came from
scripts hand-`sed`-ed in place, so the tested behaviour was not the committed
behaviour. `--check` now makes that a hard failure.

First green run: master holds both WANs, backup released, and the AC reports
exactly ONE session. sim-net-apply.sh check: all four in sync.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH

labsim/console-apply.py

@@ -22,6 +22,12 @@ def main() -> int:
     ap.add_argument("--config", required=True)
     ap.add_argument("--user", default="vyos")
     ap.add_argument("--password", default="vyos")
+    # `save` writes config.boot. For WAN work that is dangerous: the resting
+    # state must stay `vif 53 disable` on both routers, and saving while a box
+    # is master persists the ENABLED state -- so a reboot would have it claim
+    # the cloned MAC. Observed in labsim on 2026-09-02.
+    ap.add_argument("--no-save", action="store_true",
+                    help="commit without saving (leave config.boot untouched)")
     args = ap.parse_args()
 
     cmds = [l.rstrip() for l in open(args.config)
@@ -83,8 +89,9 @@ def main() -> int:
     c.sendline("commit")
     c.expect(r"# ", timeout=300)
     commit_out = c.before or ""
-    c.sendline("save")
-    c.expect(r"# ", timeout=120)
+    if not args.no_save:
+        c.sendline("save")
+        c.expect(r"# ", timeout=120)
     # Accept either prompt on the way out. Insisting on `$ ` here hangs against
     # a healthy box -- and worse, leaves the console parked in config mode, so
     # the NEXT run finds a `# ` it was not expecting either. One strict expect

labsim/sim-ha-config.py

@@ -85,6 +85,27 @@ def build(role: str) -> list[str]:
         ]
 
     out += [
+        "# --- WAN follows VRRP mastership ---",
+        # These four hooks and the health check existed on both live sim VMs but
+        # in NEITHER generator, so `sim-net-apply.sh check` reported "in sync"
+        # while the mechanism under test was pure undetected drift -- exactly
+        # the failure mode this file was written to end.
+        #
+        # The check goes on the SYNC GROUP, not per group: VyOS rejects a
+        # per-group check while the group is in a sync group ("Only sync group
+        # health check will be used").
+        "set high-availability vrrp sync-group MAIN health-check script '/config/vrrp-wan-health'",
+        "set high-availability vrrp sync-group MAIN health-check interval '5'",
+        "set high-availability vrrp sync-group MAIN health-check failure-count '3'",
+        # take and release both exec vrrp-wan-reconcile: one code path, asked at
+        # different moments. `stop` matters as much as `backup` -- a stopped
+        # keepalived is a demotion too, and without it the box would keep the
+        # WAN while holding no VIPs.
+        "set high-availability vrrp sync-group MAIN transition-script master '/config/vrrp-wan-take'",
+        "set high-availability vrrp sync-group MAIN transition-script backup '/config/vrrp-wan-release'",
+        "set high-availability vrrp sync-group MAIN transition-script fault '/config/vrrp-wan-release'",
+        "set high-availability vrrp sync-group MAIN transition-script stop '/config/vrrp-wan-release'",
+        "",
         "# --- DHCP high-availability ---",
         "# The thing under test: active-passive should mean exactly one OFFER.",
         "set service dhcp-server high-availability mode active-passive",

labsim/sim-net-apply.sh

@@ -17,9 +17,15 @@ ACTION="${1:-check}"
 WORK="$(mktemp -d)"; trap 'rm -rf "$WORK"' EXIT
 
 # role : vm : address : regex selecting the subtrees this generator owns
+#
+# primary and secondary now own the SAME subtrees. The secondary's used to omit
+# `interfaces pppoe`, `interfaces bonding`, `nat source` and `protocols
+# failover|static`, so `check` was blind to precisely the WAN config the
+# failover mechanism depends on -- it reported "in sync" for a box that had no
+# WAN at all.
 TARGETS=(
   "primary:labsim-vyos:172.31.1.252:^set (protocols (bgp|failover|static)|policy (prefix-list|route-map)|nat source rule 1[12]0|interfaces (pppoe|bonding bond0 vif 5[13])|firewall (group interface-group LAN|ipv4|ipv6))"
-  "secondary:labsim-vyos2:172.31.1.253:^set (protocols bgp|policy (prefix-list|route-map)|firewall (group interface-group LAN|ipv4|ipv6))"
+  "secondary:labsim-vyos2:172.31.1.253:^set (protocols (bgp|failover|static)|policy (prefix-list|route-map)|nat source rule 1[12]0|interfaces (pppoe|bonding bond0 vif 5[13])|firewall (group interface-group LAN|ipv4|ipv6))"
   "isp-dhcp:labsim-isp-dhcp:192.168.122.136:^set (interfaces ethernet|nat source|service dhcp-server|firewall ipv4 forward|system host-name)"
   "isp-pppoe:labsim-isp-pppoe:192.168.122.63:^set (interfaces ethernet|nat source|service pppoe-server|firewall ipv4 forward|system host-name)"
 )
@@ -29,7 +35,13 @@ SSH_OPTS=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
           -o LogLevel=ERROR -o PreferredAuthentications=password -o ConnectTimeout=5)
 live() { timeout 30 sshpass -p vyos ssh "${SSH_OPTS[@]}" "vyos@$1" \
            "/opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration commands" 2>/dev/null; }
-norm() { sed "s/'//g" | grep -v 'hw-id\|offload' | sort -u; }
+# `vif 53 disable` is RUNTIME state, not config drift. The generators declare it
+# on both routers as the safe resting state (only one box may hold the cloned
+# MAC), and vrrp-wan-reconcile removes it on whichever box currently holds the
+# VIP. Comparing it would therefore report drift on the master for ever, and a
+# check that always cries wolf is a check nobody reads.
+norm() { sed "s/'//g" | grep -v 'hw-id\|offload' \
+           | grep -v 'interfaces bonding bond0 vif 53 disable' | sort -u; }
 
 rc=0
 for t in "${TARGETS[@]}"; do

labsim/sim-net-config.py

@@ -80,6 +80,13 @@ WAN_DHCP_DISTANCE = 210     # NOT `no-default-route`, which blanks new_routers
 PPPOE_DISTANCE = 10         #   in the lease file, leaving failover no gateway
                             #   to install and silently handing the default
                             #   route to the backup line.
+
+# One MAC, cloned onto BOTH routers' bond0.53, mirroring production's use of the
+# retired USG's WAN2 MAC to keep its DHCP lease. The sim did not model a shared
+# MAC at all, which is exactly why bond0.53 has to stay on the config plane --
+# only VyOS config can move a MAC between boxes. Locally-administered, sim-only.
+WAN_DHCP_MAC = "02:53:10:61:00:53"
+
 SIM_LAN = "172.31.0.0/16"
 
 # The sim routers' own libvirt-NAT uplink, from before the ISP VMs existed. It
@@ -120,8 +127,27 @@ def bgp(role: str) -> list[str]:
     return out
 
 
-def wan(drop_scaffold: bool) -> list[str]:
-    """Dual WAN + health-checked failover. Primary router only -- see README."""
+def wan(drop_scaffold: bool, role: str = "primary") -> list[str]:
+    """Dual WAN + health-checked failover. IDENTICAL on both routers.
+
+    It used to be primary-only, on the grounds that "two PPPoE clients sharing
+    one credential against a single access concentrator is a different failure
+    mode than anything production has". That was backwards: production has
+    exactly that, and by omitting it the sim could not test the one thing most
+    likely to go wrong. The secondary having no WAN is also why it ended up with
+    zero NAT rules while the primary had ten -- the pair was not comparable.
+
+    Both routers therefore get the same WAN config. What differs is the RESTING
+    STATE, and only for the DHCP line:
+
+      bond0.53  `disable` on BOTH. Its lease is bound to a cloned MAC, and two
+                boxes claiming one MAC is the fault this whole design exists to
+                prevent. vrrp-wan-reconcile removes `disable` on the master.
+      pppoe0    enabled on BOTH, never `disable`d. `disable` unlinks
+                /etc/ppp/peers/pppoe0, which is pppd's own options file, so the
+                promotion path destroyed what it needed. Dialling is gated at
+                the systemd unit instead -- see migration/ppp-vrrp-gate.conf.
+    """
     out = [
         "# --- WAN: DHCP (primary) + PPPoE (backup), health-checked ---",
         f"set interfaces bonding bond0 vif {WAN_PPPOE_VLAN} description "
@@ -131,6 +157,13 @@ def wan(drop_scaffold: bool) -> list[str]:
         f"'WAN3 10gig-equivalent (sim ISP DHCP)'",
         f"set interfaces bonding bond0 vif {WAN_DHCP_VLAN} dhcp-options "
         f"default-route-distance {WAN_DHCP_DISTANCE}",
+        # The cloned MAC. Production clones the old USG's WAN2 MAC so the ISP
+        # keeps handing back the same lease; the sim did not model a shared MAC
+        # at all, which is precisely why bond0.53 must stay on the config plane.
+        # Modelling it lets the sim prove the lease returns to the new master.
+        f"set interfaces bonding bond0 vif {WAN_DHCP_VLAN} mac {WAN_DHCP_MAC}",
+        # Safe at rest on BOTH routers: only the VIP holder enables it.
+        f"set interfaces bonding bond0 vif {WAN_DHCP_VLAN} disable",
         f"set interfaces pppoe pppoe0 source-interface bond0.{WAN_PPPOE_VLAN}",
         f"set interfaces pppoe pppoe0 authentication username {PPPOE_USER}",
         f"set interfaces pppoe pppoe0 authentication password {PPPOE_PASS}",
@@ -364,21 +397,20 @@ def _isp_common(uplink_if: str, customer_net: str, desc: str) -> list[str]:
 
 
 def build(role: str, drop_scaffold: bool, wan_if: str, uplink_if: str) -> list[str]:
-    if role == "primary":
-        # WAN lives on the primary only. Production has WAN on both routers;
-        # the sim does not, because two PPPoE clients sharing one credential
-        # against a single access concentrator is a different failure mode than
-        # anything production has. VRRP/conntrack failover is still exercised --
-        # see README, "known gaps".
+    if role in ("primary", "secondary"):
+        # BOTH routers get the identical WAN. The sim used to give it to the
+        # primary only, reasoning that two PPPoE clients sharing one credential
+        # was "a different failure mode than anything production has" -- but
+        # that IS production, and omitting it meant the failover path was the
+        # one path the sim could not exercise. It also left the pair
+        # incomparable: ten NAT rules on one box, none on the other.
+        #
+        # Safety comes from resting state, not from asymmetry: bond0.53 is
+        # `disable`d on both (cloned MAC), pppoe0 is enabled on both but gated
+        # at the systemd unit. See wan() and migration/ppp-vrrp-gate.conf.
         return ([f"# labsim routing -- {role}", ""]
-                + bgp(role) + wan(drop_scaffold) + firewall())
-    if role == "secondary":
-        # The backup has no WAN in the sim, so it has no DHCP client to
-        # exempt -- but it gets the same policy otherwise, because after a VRRP
-        # failover it IS the router and a divergent ruleset would only be
-        # discovered during the failover.
-        return ([f"# labsim routing -- {role}", ""]
-                + bgp(role) + firewall(wan_dhcp_if=None))
+                + bgp(role) + wan(drop_scaffold, role)
+                + firewall(wan_dhcp_if=f"bond0.{WAN_DHCP_VLAN}"))
     if role == "isp-dhcp":
         return isp_dhcp(wan_if, uplink_if)
     return isp_pppoe(wan_if, uplink_if)