migration/vrrp-wan-reconcile.service

[Unit]
# Belt to the transition scripts' braces. VyOS's keepalived-fifo.py helper was
# observed dropping a MASTER transition silently, leaving a router holding every
# VIP with no WAN. A timer cannot be dropped the same way.
Description=Reconcile WAN interface state with VRRP mastership
# vyos-router loads config at boot and its pppoe handler will try to dial; the
# gate drop-in blocks that, but ordering after it keeps the logs readable.
After=keepalived.service vyos-router.service

[Service]
Type=oneshot
ExecStart=/config/vrrp-wan-reconcile
vyos: WAN follows VRRP mastership, rehearsed in labsim The ISP is consumer with one static IP, so "both routers hold WAN" is not available: the 10 gig lease is anchored to a cloned MAC and the PPPoE line to a single credential. The WAN therefore has to move with mastership. Proven end to end in the sim: the secondary was promoted, took the WAN, got a lease, installed a default route, and a LAN VM reached the internet through it (3/3, 9ms). Demoting released it -- link down, no address. Rebooting the master converged correctly too: it came back BACKUP with the WAN disabled while the peer kept it. The rehearsal earned its keep four times over, and none of these were visible from reading the docs: - `transition-script` alone is NOT safe to hang internet on. VyOS delivers it through keepalived-fifo.py, and on one promotion that helper logged NOTHING while Keepalived_vrrp logged all six instances entering MASTER and the built-in notify_master for conntrack-sync ran normally. The result was a router holding every VIP with no WAN -- the 2026-09-02 outage, recreated by the mechanism meant to prevent it. Hence vrrp-wan-reconcile on a 30s timer: the scripts give speed, the timer gives correctness. - The health check must ask REALITY, not a marker. Keying "am I master" on a /run file written by the transition script meant that when the script did not run, the router believed it was backup, passed the check, and kept the VIPs it could not serve. It now asks whether the VIP is actually on the box. - The old address-based check DEADLOCKED this design: may-I-be-master required already having WAN, and only the master gets WAN. That is why vyos002 sat in FAULT for ever -- the safety check had silently removed the redundancy it existed to protect. - script-template must be the FIRST thing a script does. Sourced after an if, an exec and a mkdir it terminated the script inside the source, rc=0, no output: the reconciler reported success having done nothing. Hence the split into vrrp-wan-apply, matching the shape /config/vyos-known-good already uses. Two hazards found and handled rather than discovered in production: - A `configure` session whose process dies leaks a unionfs mount under /opt/vyatta/config/tmp, and one of those holds the commit lock -- after which every commit fails, including the manual one you try to fix it with. A 30s job that can leak one per failure wedges the box on its own, so the reconciler reaps dead sessions before it starts. It cleared 8 on the sim. - Any `save` while a box is master persists the enabled WAN into config.boot, so a reboot would claim the shared MAC regardless of VRRP state. Observed: an ordinary console-apply did exactly this. config.boot must keep `disable` on BOTH routers; the model asserts it and vyos:verify reports it as drift. NOT yet applied to production, and it should not be until the remaining item is settled: a clean, deliberately-triggered failover has been seen via reboot, but `restart vrrp` twice failed to move mastership at all, so the trigger for a planned failover is still unproven. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH 2026-09-02 18:02:58 +01:00			`[Unit]`
			`# Belt to the transition scripts' braces. VyOS's keepalived-fifo.py helper was`
			`# observed dropping a MASTER transition silently, leaving a router holding every`
			`# VIP with no WAN. A timer cannot be dropped the same way.`
			`Description=Reconcile WAN interface state with VRRP mastership`
vyos: move PPPoE off the config plane onto a gated systemd unit PPPoE HA could not work as written, and the reason is structural rather than a bug: `set interfaces pppoe pppoe0 disable` and `delete` are handled identically by interfaces_pppoe.py -- both UNLINK /etc/ppp/peers/pppoe0. That path is pppd's own options file, so the resting state destroyed exactly what the promotion path needed, and `ppp@pppoe0` restart-looped against it (observed: 47 restarts, zero sessions at the access concentrator). It also made op-mode `connect interface pppoe0` unusable, and put every failover behind a priority-322 commit where one unrelated invalid node fails the whole thing -- which has already taken the 10 gig down once. pppoe0 is now configured identically and ENABLED on both routers, so the peers file always exists, and dialling is gated by a drop-in on the unit: ConditionPathExists=/run/vrrp-wan/may-dial ConditionPathExists=/etc/ppp/peers/pppoe0 /run is tmpfs, so the gate is shut at boot and neither box can dial before VRRP has decided. That matters more than it first appears: with the node enabled, interfaces_pppoe.py restarts ppp on EVERY commit touching the pppoe subtree when the daemon is not running -- so the backup actively tries to dial whenever anything commits. The gate is the only thing making that a no-op, which is why vrrp-wan-reconcile now refuses to bless a box whose drop-in is missing: /etc is per-image, and a VyOS upgrade would otherwise silently remove the protection. may-dial is a LEASE, not a flag. ConditionPathExists is evaluated at start only -- it can prevent a dial, never revoke one -- so a reconciler that stops running while its box is demoted would keep the one ISP session for ever. The reconciler renews the lease; a new 5s vrrp-wan-guard revokes it, and only ever revokes. It fired correctly first time: "GUARD: lease stale (81s > 75s)". Also: remove-then-stop on release (the file's absence blocks a NEW start that a concurrent commit would trigger); a flap damper, because two routers that both believe they hold the VIP will both dial and each dial kills the other's session -- against a real ISP that is how an account gets rate-limited; and a guard on `cfg` returning empty under commit-lock contention, which had already produced one spurious "releasing" on a box that needed nothing. GRACE 90 -> 180. accel-ppp's dead-peer budget is lcp-echo-interval(30) x failure(3) = 90s, so the old value sat exactly on the boundary: a hard failover into an AC that does not replace the stale session would fail its own check, shed the VIPs, and leave both routers in FAULT. The sim could not have tested any of this. Both routers now get the identical WAN -- the secondary had none "because two PPPoE clients sharing one credential is a different failure mode than anything production has", which is backwards: that IS production. It also left the pair incomparable, ten NAT rules against none. Safety now comes from resting state, not asymmetry. Three more things the sim was hiding: - the drift check's secondary regex omitted interfaces pppoe/bonding, nat source and protocols failover, so it reported "in sync" for a box with no WAN at all; - the VRRP health-check and transition-script hooks existed on both live VMs and in NEITHER generator -- the mechanism under test was pure undetected drift; - labsim-vyos's only default route was the libvirt-NAT scaffold, so every "the LAN still has internet" verdict on it was answered by eth2 rather than the WAN. --drop-scaffold applied; the earlier DHCP-failover proof is being re-run because of it. vrrp-wan-install ends the other half of that: the sim's previous proof came from scripts hand-`sed`-ed in place, so the tested behaviour was not the committed behaviour. `--check` now makes that a hard failure. First green run: master holds both WANs, backup released, and the AC reports exactly ONE session. sim-net-apply.sh check: all four in sync. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH 2026-09-05 18:50:16 +01:00			`# vyos-router loads config at boot and its pppoe handler will try to dial; the`
			`# gate drop-in blocks that, but ordering after it keeps the logs readable.`
			`After=keepalived.service vyos-router.service`
vyos: WAN follows VRRP mastership, rehearsed in labsim The ISP is consumer with one static IP, so "both routers hold WAN" is not available: the 10 gig lease is anchored to a cloned MAC and the PPPoE line to a single credential. The WAN therefore has to move with mastership. Proven end to end in the sim: the secondary was promoted, took the WAN, got a lease, installed a default route, and a LAN VM reached the internet through it (3/3, 9ms). Demoting released it -- link down, no address. Rebooting the master converged correctly too: it came back BACKUP with the WAN disabled while the peer kept it. The rehearsal earned its keep four times over, and none of these were visible from reading the docs: - `transition-script` alone is NOT safe to hang internet on. VyOS delivers it through keepalived-fifo.py, and on one promotion that helper logged NOTHING while Keepalived_vrrp logged all six instances entering MASTER and the built-in notify_master for conntrack-sync ran normally. The result was a router holding every VIP with no WAN -- the 2026-09-02 outage, recreated by the mechanism meant to prevent it. Hence vrrp-wan-reconcile on a 30s timer: the scripts give speed, the timer gives correctness. - The health check must ask REALITY, not a marker. Keying "am I master" on a /run file written by the transition script meant that when the script did not run, the router believed it was backup, passed the check, and kept the VIPs it could not serve. It now asks whether the VIP is actually on the box. - The old address-based check DEADLOCKED this design: may-I-be-master required already having WAN, and only the master gets WAN. That is why vyos002 sat in FAULT for ever -- the safety check had silently removed the redundancy it existed to protect. - script-template must be the FIRST thing a script does. Sourced after an if, an exec and a mkdir it terminated the script inside the source, rc=0, no output: the reconciler reported success having done nothing. Hence the split into vrrp-wan-apply, matching the shape /config/vyos-known-good already uses. Two hazards found and handled rather than discovered in production: - A `configure` session whose process dies leaks a unionfs mount under /opt/vyatta/config/tmp, and one of those holds the commit lock -- after which every commit fails, including the manual one you try to fix it with. A 30s job that can leak one per failure wedges the box on its own, so the reconciler reaps dead sessions before it starts. It cleared 8 on the sim. - Any `save` while a box is master persists the enabled WAN into config.boot, so a reboot would claim the shared MAC regardless of VRRP state. Observed: an ordinary console-apply did exactly this. config.boot must keep `disable` on BOTH routers; the model asserts it and vyos:verify reports it as drift. NOT yet applied to production, and it should not be until the remaining item is settled: a clean, deliberately-triggered failover has been seen via reboot, but `restart vrrp` twice failed to move mastership at all, so the trigger for a planned failover is still unproven. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH 2026-09-02 18:02:58 +01:00
			`[Service]`
			`Type=oneshot`
			`ExecStart=/config/vrrp-wan-reconcile`