vyos: WAN follows VRRP mastership, rehearsed in labsim
The ISP is consumer with one static IP, so "both routers hold WAN" is not
available: the 10 gig lease is anchored to a cloned MAC and the PPPoE line to a
single credential. The WAN therefore has to move with mastership.
Proven end to end in the sim: the secondary was promoted, took the WAN, got a
lease, installed a default route, and a LAN VM reached the internet through it
(3/3, 9ms). Demoting released it -- link down, no address. Rebooting the master
converged correctly too: it came back BACKUP with the WAN disabled while the
peer kept it.
The rehearsal earned its keep four times over, and none of these were visible
from reading the docs:
- `transition-script` alone is NOT safe to hang internet on. VyOS delivers it
through keepalived-fifo.py, and on one promotion that helper logged NOTHING
while Keepalived_vrrp logged all six instances entering MASTER and the
built-in notify_master for conntrack-sync ran normally. The result was a
router holding every VIP with no WAN -- the 2026-09-02 outage, recreated by
the mechanism meant to prevent it. Hence vrrp-wan-reconcile on a 30s timer:
the scripts give speed, the timer gives correctness.
- The health check must ask REALITY, not a marker. Keying "am I master" on a
/run file written by the transition script meant that when the script did
not run, the router believed it was backup, passed the check, and kept the
VIPs it could not serve. It now asks whether the VIP is actually on the box.
- The old address-based check DEADLOCKED this design: may-I-be-master required
already having WAN, and only the master gets WAN. That is why vyos002 sat in
FAULT for ever -- the safety check had silently removed the redundancy it
existed to protect.
- script-template must be the FIRST thing a script does. Sourced after an if,
an exec and a mkdir it terminated the script inside the source, rc=0, no
output: the reconciler reported success having done nothing. Hence the split
into vrrp-wan-apply, matching the shape /config/vyos-known-good already uses.
Two hazards found and handled rather than discovered in production:
- A `configure` session whose process dies leaks a unionfs mount under
/opt/vyatta/config/tmp, and one of those holds the commit lock -- after
which every commit fails, including the manual one you try to fix it with.
A 30s job that can leak one per failure wedges the box on its own, so the
reconciler reaps dead sessions before it starts. It cleared 8 on the sim.
- Any `save` while a box is master persists the enabled WAN into config.boot,
so a reboot would claim the shared MAC regardless of VRRP state. Observed:
an ordinary console-apply did exactly this. config.boot must keep `disable`
on BOTH routers; the model asserts it and vyos:verify reports it as drift.
NOT yet applied to production, and it should not be until the remaining item is
settled: a clean, deliberately-triggered failover has been seen via reboot, but
`restart vrrp` twice failed to move mastership at all, so the trigger for a
planned failover is still unproven.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-02 18:02:58 +01:00
|
|
|
#!/bin/vbash
|
vyos: move PPPoE off the config plane onto a gated systemd unit
PPPoE HA could not work as written, and the reason is structural rather than a
bug: `set interfaces pppoe pppoe0 disable` and `delete` are handled identically
by interfaces_pppoe.py -- both UNLINK /etc/ppp/peers/pppoe0. That path is pppd's
own options file, so the resting state destroyed exactly what the promotion path
needed, and `ppp@pppoe0` restart-looped against it (observed: 47 restarts, zero
sessions at the access concentrator). It also made op-mode `connect interface
pppoe0` unusable, and put every failover behind a priority-322 commit where one
unrelated invalid node fails the whole thing -- which has already taken the
10 gig down once.
pppoe0 is now configured identically and ENABLED on both routers, so the peers
file always exists, and dialling is gated by a drop-in on the unit:
ConditionPathExists=/run/vrrp-wan/may-dial
ConditionPathExists=/etc/ppp/peers/pppoe0
/run is tmpfs, so the gate is shut at boot and neither box can dial before VRRP
has decided. That matters more than it first appears: with the node enabled,
interfaces_pppoe.py restarts ppp on EVERY commit touching the pppoe subtree when
the daemon is not running -- so the backup actively tries to dial whenever
anything commits. The gate is the only thing making that a no-op, which is why
vrrp-wan-reconcile now refuses to bless a box whose drop-in is missing: /etc is
per-image, and a VyOS upgrade would otherwise silently remove the protection.
may-dial is a LEASE, not a flag. ConditionPathExists is evaluated at start only
-- it can prevent a dial, never revoke one -- so a reconciler that stops running
while its box is demoted would keep the one ISP session for ever. The reconciler
renews the lease; a new 5s vrrp-wan-guard revokes it, and only ever revokes. It
fired correctly first time: "GUARD: lease stale (81s > 75s)".
Also: remove-then-stop on release (the file's absence blocks a NEW start that a
concurrent commit would trigger); a flap damper, because two routers that both
believe they hold the VIP will both dial and each dial kills the other's session
-- against a real ISP that is how an account gets rate-limited; and a guard on
`cfg` returning empty under commit-lock contention, which had already produced
one spurious "releasing" on a box that needed nothing.
GRACE 90 -> 180. accel-ppp's dead-peer budget is lcp-echo-interval(30) x
failure(3) = 90s, so the old value sat exactly on the boundary: a hard failover
into an AC that does not replace the stale session would fail its own check,
shed the VIPs, and leave both routers in FAULT.
The sim could not have tested any of this. Both routers now get the identical
WAN -- the secondary had none "because two PPPoE clients sharing one credential
is a different failure mode than anything production has", which is backwards:
that IS production. It also left the pair incomparable, ten NAT rules against
none. Safety now comes from resting state, not asymmetry.
Three more things the sim was hiding:
- the drift check's secondary regex omitted interfaces pppoe/bonding, nat
source and protocols failover, so it reported "in sync" for a box with no
WAN at all;
- the VRRP health-check and transition-script hooks existed on both live VMs
and in NEITHER generator -- the mechanism under test was pure undetected
drift;
- labsim-vyos's only default route was the libvirt-NAT scaffold, so every
"the LAN still has internet" verdict on it was answered by eth2 rather than
the WAN. --drop-scaffold applied; the earlier DHCP-failover proof is being
re-run because of it.
vrrp-wan-install ends the other half of that: the sim's previous proof came from
scripts hand-`sed`-ed in place, so the tested behaviour was not the committed
behaviour. `--check` now makes that a hard failure.
First green run: master holds both WANs, backup released, and the AC reports
exactly ONE session. sim-net-apply.sh check: all four in sync.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-05 18:50:16 +01:00
|
|
|
# Enable or disable the DHCP WAN (bond0.53). The ONLY part of the failover that
|
|
|
|
|
# touches VyOS configuration.
|
|
|
|
|
#
|
|
|
|
|
# PPPoE deliberately does NOT appear here any more, and should not be added back
|
|
|
|
|
# for symmetry. `set interfaces pppoe pppoe0 disable` unlinks
|
|
|
|
|
# /etc/ppp/peers/pppoe0 -- interfaces_pppoe.py treats `disable` and `delete`
|
|
|
|
|
# identically -- and that path is pppd's options file, so the resting state
|
|
|
|
|
# destroyed what the promotion path needed and the unit restart-looped (47 times,
|
|
|
|
|
# zero sessions at the AC). It also made a pppoe node able to fail this commit
|
|
|
|
|
# and take the 10 gig down with it: `interfaces pppoe` is priority 322 and one
|
|
|
|
|
# invalid node fails the whole commit. PPPoE is now gated at the systemd unit
|
|
|
|
|
# instead; see migration/ppp-vrrp-gate.conf and vrrp-wan-reconcile.
|
|
|
|
|
#
|
|
|
|
|
# bond0.53 stays here because its lease is bound to a cloned MAC and only VyOS
|
|
|
|
|
# config can move a MAC between boxes.
|
|
|
|
|
#
|
vyos: WAN follows VRRP mastership, rehearsed in labsim
The ISP is consumer with one static IP, so "both routers hold WAN" is not
available: the 10 gig lease is anchored to a cloned MAC and the PPPoE line to a
single credential. The WAN therefore has to move with mastership.
Proven end to end in the sim: the secondary was promoted, took the WAN, got a
lease, installed a default route, and a LAN VM reached the internet through it
(3/3, 9ms). Demoting released it -- link down, no address. Rebooting the master
converged correctly too: it came back BACKUP with the WAN disabled while the
peer kept it.
The rehearsal earned its keep four times over, and none of these were visible
from reading the docs:
- `transition-script` alone is NOT safe to hang internet on. VyOS delivers it
through keepalived-fifo.py, and on one promotion that helper logged NOTHING
while Keepalived_vrrp logged all six instances entering MASTER and the
built-in notify_master for conntrack-sync ran normally. The result was a
router holding every VIP with no WAN -- the 2026-09-02 outage, recreated by
the mechanism meant to prevent it. Hence vrrp-wan-reconcile on a 30s timer:
the scripts give speed, the timer gives correctness.
- The health check must ask REALITY, not a marker. Keying "am I master" on a
/run file written by the transition script meant that when the script did
not run, the router believed it was backup, passed the check, and kept the
VIPs it could not serve. It now asks whether the VIP is actually on the box.
- The old address-based check DEADLOCKED this design: may-I-be-master required
already having WAN, and only the master gets WAN. That is why vyos002 sat in
FAULT for ever -- the safety check had silently removed the redundancy it
existed to protect.
- script-template must be the FIRST thing a script does. Sourced after an if,
an exec and a mkdir it terminated the script inside the source, rc=0, no
output: the reconciler reported success having done nothing. Hence the split
into vrrp-wan-apply, matching the shape /config/vyos-known-good already uses.
Two hazards found and handled rather than discovered in production:
- A `configure` session whose process dies leaks a unionfs mount under
/opt/vyatta/config/tmp, and one of those holds the commit lock -- after
which every commit fails, including the manual one you try to fix it with.
A 30s job that can leak one per failure wedges the box on its own, so the
reconciler reaps dead sessions before it starts. It cleared 8 on the sim.
- Any `save` while a box is master persists the enabled WAN into config.boot,
so a reboot would claim the shared MAC regardless of VRRP state. Observed:
an ordinary console-apply did exactly this. config.boot must keep `disable`
on BOTH routers; the model asserts it and vyos:verify reports it as drift.
NOT yet applied to production, and it should not be until the remaining item is
settled: a clean, deliberately-triggered failover has been seen via reboot, but
`restart vrrp` twice failed to move mastership at all, so the trigger for a
planned failover is still unproven.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-02 18:02:58 +01:00
|
|
|
# `source /opt/vyatta/etc/functions/script-template` must be the FIRST thing the
|
vyos: move PPPoE off the config plane onto a gated systemd unit
PPPoE HA could not work as written, and the reason is structural rather than a
bug: `set interfaces pppoe pppoe0 disable` and `delete` are handled identically
by interfaces_pppoe.py -- both UNLINK /etc/ppp/peers/pppoe0. That path is pppd's
own options file, so the resting state destroyed exactly what the promotion path
needed, and `ppp@pppoe0` restart-looped against it (observed: 47 restarts, zero
sessions at the access concentrator). It also made op-mode `connect interface
pppoe0` unusable, and put every failover behind a priority-322 commit where one
unrelated invalid node fails the whole thing -- which has already taken the
10 gig down once.
pppoe0 is now configured identically and ENABLED on both routers, so the peers
file always exists, and dialling is gated by a drop-in on the unit:
ConditionPathExists=/run/vrrp-wan/may-dial
ConditionPathExists=/etc/ppp/peers/pppoe0
/run is tmpfs, so the gate is shut at boot and neither box can dial before VRRP
has decided. That matters more than it first appears: with the node enabled,
interfaces_pppoe.py restarts ppp on EVERY commit touching the pppoe subtree when
the daemon is not running -- so the backup actively tries to dial whenever
anything commits. The gate is the only thing making that a no-op, which is why
vrrp-wan-reconcile now refuses to bless a box whose drop-in is missing: /etc is
per-image, and a VyOS upgrade would otherwise silently remove the protection.
may-dial is a LEASE, not a flag. ConditionPathExists is evaluated at start only
-- it can prevent a dial, never revoke one -- so a reconciler that stops running
while its box is demoted would keep the one ISP session for ever. The reconciler
renews the lease; a new 5s vrrp-wan-guard revokes it, and only ever revokes. It
fired correctly first time: "GUARD: lease stale (81s > 75s)".
Also: remove-then-stop on release (the file's absence blocks a NEW start that a
concurrent commit would trigger); a flap damper, because two routers that both
believe they hold the VIP will both dial and each dial kills the other's session
-- against a real ISP that is how an account gets rate-limited; and a guard on
`cfg` returning empty under commit-lock contention, which had already produced
one spurious "releasing" on a box that needed nothing.
GRACE 90 -> 180. accel-ppp's dead-peer budget is lcp-echo-interval(30) x
failure(3) = 90s, so the old value sat exactly on the boundary: a hard failover
into an AC that does not replace the stale session would fail its own check,
shed the VIPs, and leave both routers in FAULT.
The sim could not have tested any of this. Both routers now get the identical
WAN -- the secondary had none "because two PPPoE clients sharing one credential
is a different failure mode than anything production has", which is backwards:
that IS production. It also left the pair incomparable, ten NAT rules against
none. Safety now comes from resting state, not asymmetry.
Three more things the sim was hiding:
- the drift check's secondary regex omitted interfaces pppoe/bonding, nat
source and protocols failover, so it reported "in sync" for a box with no
WAN at all;
- the VRRP health-check and transition-script hooks existed on both live VMs
and in NEITHER generator -- the mechanism under test was pure undetected
drift;
- labsim-vyos's only default route was the libvirt-NAT scaffold, so every
"the LAN still has internet" verdict on it was answered by eth2 rather than
the WAN. --drop-scaffold applied; the earlier DHCP-failover proof is being
re-run because of it.
vrrp-wan-install ends the other half of that: the sim's previous proof came from
scripts hand-`sed`-ed in place, so the tested behaviour was not the committed
behaviour. `--check` now makes that a hard failure.
First green run: master holds both WANs, backup released, and the AC reports
exactly ONE session. sim-net-apply.sh check: all four in sync.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-05 18:50:16 +01:00
|
|
|
# script does. Sourced after an if, an exec and a mkdir it terminated the script
|
|
|
|
|
# inside the source, rc=0, no output -- the caller reported success having done
|
vyos: WAN follows VRRP mastership, rehearsed in labsim
The ISP is consumer with one static IP, so "both routers hold WAN" is not
available: the 10 gig lease is anchored to a cloned MAC and the PPPoE line to a
single credential. The WAN therefore has to move with mastership.
Proven end to end in the sim: the secondary was promoted, took the WAN, got a
lease, installed a default route, and a LAN VM reached the internet through it
(3/3, 9ms). Demoting released it -- link down, no address. Rebooting the master
converged correctly too: it came back BACKUP with the WAN disabled while the
peer kept it.
The rehearsal earned its keep four times over, and none of these were visible
from reading the docs:
- `transition-script` alone is NOT safe to hang internet on. VyOS delivers it
through keepalived-fifo.py, and on one promotion that helper logged NOTHING
while Keepalived_vrrp logged all six instances entering MASTER and the
built-in notify_master for conntrack-sync ran normally. The result was a
router holding every VIP with no WAN -- the 2026-09-02 outage, recreated by
the mechanism meant to prevent it. Hence vrrp-wan-reconcile on a 30s timer:
the scripts give speed, the timer gives correctness.
- The health check must ask REALITY, not a marker. Keying "am I master" on a
/run file written by the transition script meant that when the script did
not run, the router believed it was backup, passed the check, and kept the
VIPs it could not serve. It now asks whether the VIP is actually on the box.
- The old address-based check DEADLOCKED this design: may-I-be-master required
already having WAN, and only the master gets WAN. That is why vyos002 sat in
FAULT for ever -- the safety check had silently removed the redundancy it
existed to protect.
- script-template must be the FIRST thing a script does. Sourced after an if,
an exec and a mkdir it terminated the script inside the source, rc=0, no
output: the reconciler reported success having done nothing. Hence the split
into vrrp-wan-apply, matching the shape /config/vyos-known-good already uses.
Two hazards found and handled rather than discovered in production:
- A `configure` session whose process dies leaks a unionfs mount under
/opt/vyatta/config/tmp, and one of those holds the commit lock -- after
which every commit fails, including the manual one you try to fix it with.
A 30s job that can leak one per failure wedges the box on its own, so the
reconciler reaps dead sessions before it starts. It cleared 8 on the sim.
- Any `save` while a box is master persists the enabled WAN into config.boot,
so a reboot would claim the shared MAC regardless of VRRP state. Observed:
an ordinary console-apply did exactly this. config.boot must keep `disable`
on BOTH routers; the model asserts it and vyos:verify reports it as drift.
NOT yet applied to production, and it should not be until the remaining item is
settled: a clean, deliberately-triggered failover has been seen via reboot, but
`restart vrrp` twice failed to move mastership at all, so the trigger for a
planned failover is still unproven.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-02 18:02:58 +01:00
|
|
|
# nothing. Only a single assignment may precede it (the template resets the
|
vyos: move PPPoE off the config plane onto a gated systemd unit
PPPoE HA could not work as written, and the reason is structural rather than a
bug: `set interfaces pppoe pppoe0 disable` and `delete` are handled identically
by interfaces_pppoe.py -- both UNLINK /etc/ppp/peers/pppoe0. That path is pppd's
own options file, so the resting state destroyed exactly what the promotion path
needed, and `ppp@pppoe0` restart-looped against it (observed: 47 restarts, zero
sessions at the access concentrator). It also made op-mode `connect interface
pppoe0` unusable, and put every failover behind a priority-322 commit where one
unrelated invalid node fails the whole thing -- which has already taken the
10 gig down once.
pppoe0 is now configured identically and ENABLED on both routers, so the peers
file always exists, and dialling is gated by a drop-in on the unit:
ConditionPathExists=/run/vrrp-wan/may-dial
ConditionPathExists=/etc/ppp/peers/pppoe0
/run is tmpfs, so the gate is shut at boot and neither box can dial before VRRP
has decided. That matters more than it first appears: with the node enabled,
interfaces_pppoe.py restarts ppp on EVERY commit touching the pppoe subtree when
the daemon is not running -- so the backup actively tries to dial whenever
anything commits. The gate is the only thing making that a no-op, which is why
vrrp-wan-reconcile now refuses to bless a box whose drop-in is missing: /etc is
per-image, and a VyOS upgrade would otherwise silently remove the protection.
may-dial is a LEASE, not a flag. ConditionPathExists is evaluated at start only
-- it can prevent a dial, never revoke one -- so a reconciler that stops running
while its box is demoted would keep the one ISP session for ever. The reconciler
renews the lease; a new 5s vrrp-wan-guard revokes it, and only ever revokes. It
fired correctly first time: "GUARD: lease stale (81s > 75s)".
Also: remove-then-stop on release (the file's absence blocks a NEW start that a
concurrent commit would trigger); a flap damper, because two routers that both
believe they hold the VIP will both dial and each dial kills the other's session
-- against a real ISP that is how an account gets rate-limited; and a guard on
`cfg` returning empty under commit-lock contention, which had already produced
one spurious "releasing" on a box that needed nothing.
GRACE 90 -> 180. accel-ppp's dead-peer budget is lcp-echo-interval(30) x
failure(3) = 90s, so the old value sat exactly on the boundary: a hard failover
into an AC that does not replace the stale session would fail its own check,
shed the VIPs, and leave both routers in FAULT.
The sim could not have tested any of this. Both routers now get the identical
WAN -- the secondary had none "because two PPPoE clients sharing one credential
is a different failure mode than anything production has", which is backwards:
that IS production. It also left the pair incomparable, ten NAT rules against
none. Safety now comes from resting state, not asymmetry.
Three more things the sim was hiding:
- the drift check's secondary regex omitted interfaces pppoe/bonding, nat
source and protocols failover, so it reported "in sync" for a box with no
WAN at all;
- the VRRP health-check and transition-script hooks existed on both live VMs
and in NEITHER generator -- the mechanism under test was pure undetected
drift;
- labsim-vyos's only default route was the libvirt-NAT scaffold, so every
"the LAN still has internet" verdict on it was answered by eth2 rather than
the WAN. --drop-scaffold applied; the earlier DHCP-failover proof is being
re-run because of it.
vrrp-wan-install ends the other half of that: the sim's previous proof came from
scripts hand-`sed`-ed in place, so the tested behaviour was not the committed
behaviour. `--check` now makes that a hard failure.
First green run: master holds both WANs, backup released, and the AC reports
exactly ONE session. sim-net-apply.sh check: all four in sync.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-05 18:50:16 +01:00
|
|
|
# positional parameters), which is the shape /config/vyos-known-good uses.
|
vyos: WAN follows VRRP mastership, rehearsed in labsim
The ISP is consumer with one static IP, so "both routers hold WAN" is not
available: the 10 gig lease is anchored to a cloned MAC and the PPPoE line to a
single credential. The WAN therefore has to move with mastership.
Proven end to end in the sim: the secondary was promoted, took the WAN, got a
lease, installed a default route, and a LAN VM reached the internet through it
(3/3, 9ms). Demoting released it -- link down, no address. Rebooting the master
converged correctly too: it came back BACKUP with the WAN disabled while the
peer kept it.
The rehearsal earned its keep four times over, and none of these were visible
from reading the docs:
- `transition-script` alone is NOT safe to hang internet on. VyOS delivers it
through keepalived-fifo.py, and on one promotion that helper logged NOTHING
while Keepalived_vrrp logged all six instances entering MASTER and the
built-in notify_master for conntrack-sync ran normally. The result was a
router holding every VIP with no WAN -- the 2026-09-02 outage, recreated by
the mechanism meant to prevent it. Hence vrrp-wan-reconcile on a 30s timer:
the scripts give speed, the timer gives correctness.
- The health check must ask REALITY, not a marker. Keying "am I master" on a
/run file written by the transition script meant that when the script did
not run, the router believed it was backup, passed the check, and kept the
VIPs it could not serve. It now asks whether the VIP is actually on the box.
- The old address-based check DEADLOCKED this design: may-I-be-master required
already having WAN, and only the master gets WAN. That is why vyos002 sat in
FAULT for ever -- the safety check had silently removed the redundancy it
existed to protect.
- script-template must be the FIRST thing a script does. Sourced after an if,
an exec and a mkdir it terminated the script inside the source, rc=0, no
output: the reconciler reported success having done nothing. Hence the split
into vrrp-wan-apply, matching the shape /config/vyos-known-good already uses.
Two hazards found and handled rather than discovered in production:
- A `configure` session whose process dies leaks a unionfs mount under
/opt/vyatta/config/tmp, and one of those holds the commit lock -- after
which every commit fails, including the manual one you try to fix it with.
A 30s job that can leak one per failure wedges the box on its own, so the
reconciler reaps dead sessions before it starts. It cleared 8 on the sim.
- Any `save` while a box is master persists the enabled WAN into config.boot,
so a reboot would claim the shared MAC regardless of VRRP state. Observed:
an ordinary console-apply did exactly this. config.boot must keep `disable`
on BOTH routers; the model asserts it and vyos:verify reports it as drift.
NOT yet applied to production, and it should not be until the remaining item is
settled: a clean, deliberately-triggered failover has been seen via reboot, but
`restart vrrp` twice failed to move mastership at all, so the trigger for a
planned failover is still unproven.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-02 18:02:58 +01:00
|
|
|
#
|
vyos: move PPPoE off the config plane onto a gated systemd unit
PPPoE HA could not work as written, and the reason is structural rather than a
bug: `set interfaces pppoe pppoe0 disable` and `delete` are handled identically
by interfaces_pppoe.py -- both UNLINK /etc/ppp/peers/pppoe0. That path is pppd's
own options file, so the resting state destroyed exactly what the promotion path
needed, and `ppp@pppoe0` restart-looped against it (observed: 47 restarts, zero
sessions at the access concentrator). It also made op-mode `connect interface
pppoe0` unusable, and put every failover behind a priority-322 commit where one
unrelated invalid node fails the whole thing -- which has already taken the
10 gig down once.
pppoe0 is now configured identically and ENABLED on both routers, so the peers
file always exists, and dialling is gated by a drop-in on the unit:
ConditionPathExists=/run/vrrp-wan/may-dial
ConditionPathExists=/etc/ppp/peers/pppoe0
/run is tmpfs, so the gate is shut at boot and neither box can dial before VRRP
has decided. That matters more than it first appears: with the node enabled,
interfaces_pppoe.py restarts ppp on EVERY commit touching the pppoe subtree when
the daemon is not running -- so the backup actively tries to dial whenever
anything commits. The gate is the only thing making that a no-op, which is why
vrrp-wan-reconcile now refuses to bless a box whose drop-in is missing: /etc is
per-image, and a VyOS upgrade would otherwise silently remove the protection.
may-dial is a LEASE, not a flag. ConditionPathExists is evaluated at start only
-- it can prevent a dial, never revoke one -- so a reconciler that stops running
while its box is demoted would keep the one ISP session for ever. The reconciler
renews the lease; a new 5s vrrp-wan-guard revokes it, and only ever revokes. It
fired correctly first time: "GUARD: lease stale (81s > 75s)".
Also: remove-then-stop on release (the file's absence blocks a NEW start that a
concurrent commit would trigger); a flap damper, because two routers that both
believe they hold the VIP will both dial and each dial kills the other's session
-- against a real ISP that is how an account gets rate-limited; and a guard on
`cfg` returning empty under commit-lock contention, which had already produced
one spurious "releasing" on a box that needed nothing.
GRACE 90 -> 180. accel-ppp's dead-peer budget is lcp-echo-interval(30) x
failure(3) = 90s, so the old value sat exactly on the boundary: a hard failover
into an AC that does not replace the stale session would fail its own check,
shed the VIPs, and leave both routers in FAULT.
The sim could not have tested any of this. Both routers now get the identical
WAN -- the secondary had none "because two PPPoE clients sharing one credential
is a different failure mode than anything production has", which is backwards:
that IS production. It also left the pair incomparable, ten NAT rules against
none. Safety now comes from resting state, not asymmetry.
Three more things the sim was hiding:
- the drift check's secondary regex omitted interfaces pppoe/bonding, nat
source and protocols failover, so it reported "in sync" for a box with no
WAN at all;
- the VRRP health-check and transition-script hooks existed on both live VMs
and in NEITHER generator -- the mechanism under test was pure undetected
drift;
- labsim-vyos's only default route was the libvirt-NAT scaffold, so every
"the LAN still has internet" verdict on it was answered by eth2 rather than
the WAN. --drop-scaffold applied; the earlier DHCP-failover proof is being
re-run because of it.
vrrp-wan-install ends the other half of that: the sim's previous proof came from
scripts hand-`sed`-ed in place, so the tested behaviour was not the committed
behaviour. `--check` now makes that a hard failure.
First green run: master holds both WANs, backup released, and the AC reports
exactly ONE session. sim-net-apply.sh check: all four in sync.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-05 18:50:16 +01:00
|
|
|
# vrrp-wan-apply enable take the DHCP WAN
|
vyos: WAN follows VRRP mastership, rehearsed in labsim
The ISP is consumer with one static IP, so "both routers hold WAN" is not
available: the 10 gig lease is anchored to a cloned MAC and the PPPoE line to a
single credential. The WAN therefore has to move with mastership.
Proven end to end in the sim: the secondary was promoted, took the WAN, got a
lease, installed a default route, and a LAN VM reached the internet through it
(3/3, 9ms). Demoting released it -- link down, no address. Rebooting the master
converged correctly too: it came back BACKUP with the WAN disabled while the
peer kept it.
The rehearsal earned its keep four times over, and none of these were visible
from reading the docs:
- `transition-script` alone is NOT safe to hang internet on. VyOS delivers it
through keepalived-fifo.py, and on one promotion that helper logged NOTHING
while Keepalived_vrrp logged all six instances entering MASTER and the
built-in notify_master for conntrack-sync ran normally. The result was a
router holding every VIP with no WAN -- the 2026-09-02 outage, recreated by
the mechanism meant to prevent it. Hence vrrp-wan-reconcile on a 30s timer:
the scripts give speed, the timer gives correctness.
- The health check must ask REALITY, not a marker. Keying "am I master" on a
/run file written by the transition script meant that when the script did
not run, the router believed it was backup, passed the check, and kept the
VIPs it could not serve. It now asks whether the VIP is actually on the box.
- The old address-based check DEADLOCKED this design: may-I-be-master required
already having WAN, and only the master gets WAN. That is why vyos002 sat in
FAULT for ever -- the safety check had silently removed the redundancy it
existed to protect.
- script-template must be the FIRST thing a script does. Sourced after an if,
an exec and a mkdir it terminated the script inside the source, rc=0, no
output: the reconciler reported success having done nothing. Hence the split
into vrrp-wan-apply, matching the shape /config/vyos-known-good already uses.
Two hazards found and handled rather than discovered in production:
- A `configure` session whose process dies leaks a unionfs mount under
/opt/vyatta/config/tmp, and one of those holds the commit lock -- after
which every commit fails, including the manual one you try to fix it with.
A 30s job that can leak one per failure wedges the box on its own, so the
reconciler reaps dead sessions before it starts. It cleared 8 on the sim.
- Any `save` while a box is master persists the enabled WAN into config.boot,
so a reboot would claim the shared MAC regardless of VRRP state. Observed:
an ordinary console-apply did exactly this. config.boot must keep `disable`
on BOTH routers; the model asserts it and vyos:verify reports it as drift.
NOT yet applied to production, and it should not be until the remaining item is
settled: a clean, deliberately-triggered failover has been seen via reboot, but
`restart vrrp` twice failed to move mastership at all, so the trigger for a
planned failover is still unproven.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-02 18:02:58 +01:00
|
|
|
# vrrp-wan-apply disable release it
|
|
|
|
|
MODE="${1:-}"
|
|
|
|
|
source /opt/vyatta/etc/functions/script-template
|
|
|
|
|
|
vyos: move PPPoE off the config plane onto a gated systemd unit
PPPoE HA could not work as written, and the reason is structural rather than a
bug: `set interfaces pppoe pppoe0 disable` and `delete` are handled identically
by interfaces_pppoe.py -- both UNLINK /etc/ppp/peers/pppoe0. That path is pppd's
own options file, so the resting state destroyed exactly what the promotion path
needed, and `ppp@pppoe0` restart-looped against it (observed: 47 restarts, zero
sessions at the access concentrator). It also made op-mode `connect interface
pppoe0` unusable, and put every failover behind a priority-322 commit where one
unrelated invalid node fails the whole thing -- which has already taken the
10 gig down once.
pppoe0 is now configured identically and ENABLED on both routers, so the peers
file always exists, and dialling is gated by a drop-in on the unit:
ConditionPathExists=/run/vrrp-wan/may-dial
ConditionPathExists=/etc/ppp/peers/pppoe0
/run is tmpfs, so the gate is shut at boot and neither box can dial before VRRP
has decided. That matters more than it first appears: with the node enabled,
interfaces_pppoe.py restarts ppp on EVERY commit touching the pppoe subtree when
the daemon is not running -- so the backup actively tries to dial whenever
anything commits. The gate is the only thing making that a no-op, which is why
vrrp-wan-reconcile now refuses to bless a box whose drop-in is missing: /etc is
per-image, and a VyOS upgrade would otherwise silently remove the protection.
may-dial is a LEASE, not a flag. ConditionPathExists is evaluated at start only
-- it can prevent a dial, never revoke one -- so a reconciler that stops running
while its box is demoted would keep the one ISP session for ever. The reconciler
renews the lease; a new 5s vrrp-wan-guard revokes it, and only ever revokes. It
fired correctly first time: "GUARD: lease stale (81s > 75s)".
Also: remove-then-stop on release (the file's absence blocks a NEW start that a
concurrent commit would trigger); a flap damper, because two routers that both
believe they hold the VIP will both dial and each dial kills the other's session
-- against a real ISP that is how an account gets rate-limited; and a guard on
`cfg` returning empty under commit-lock contention, which had already produced
one spurious "releasing" on a box that needed nothing.
GRACE 90 -> 180. accel-ppp's dead-peer budget is lcp-echo-interval(30) x
failure(3) = 90s, so the old value sat exactly on the boundary: a hard failover
into an AC that does not replace the stale session would fail its own check,
shed the VIPs, and leave both routers in FAULT.
The sim could not have tested any of this. Both routers now get the identical
WAN -- the secondary had none "because two PPPoE clients sharing one credential
is a different failure mode than anything production has", which is backwards:
that IS production. It also left the pair incomparable, ten NAT rules against
none. Safety now comes from resting state, not asymmetry.
Three more things the sim was hiding:
- the drift check's secondary regex omitted interfaces pppoe/bonding, nat
source and protocols failover, so it reported "in sync" for a box with no
WAN at all;
- the VRRP health-check and transition-script hooks existed on both live VMs
and in NEITHER generator -- the mechanism under test was pure undetected
drift;
- labsim-vyos's only default route was the libvirt-NAT scaffold, so every
"the LAN still has internet" verdict on it was answered by eth2 rather than
the WAN. --drop-scaffold applied; the earlier DHCP-failover proof is being
re-run because of it.
vrrp-wan-install ends the other half of that: the sim's previous proof came from
scripts hand-`sed`-ed in place, so the tested behaviour was not the committed
behaviour. `--check` now makes that a hard failure.
First green run: master holds both WANs, backup released, and the AC reports
exactly ONE session. sim-net-apply.sh check: all four in sync.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-05 18:50:16 +01:00
|
|
|
CONF=/config/vrrp-wan.conf
|
|
|
|
|
[ -r "$CONF" ] && . "$CONF"
|
|
|
|
|
WAN_VIF="${WAN_VIF:-53}"
|
|
|
|
|
|
vyos: WAN follows VRRP mastership, rehearsed in labsim
The ISP is consumer with one static IP, so "both routers hold WAN" is not
available: the 10 gig lease is anchored to a cloned MAC and the PPPoE line to a
single credential. The WAN therefore has to move with mastership.
Proven end to end in the sim: the secondary was promoted, took the WAN, got a
lease, installed a default route, and a LAN VM reached the internet through it
(3/3, 9ms). Demoting released it -- link down, no address. Rebooting the master
converged correctly too: it came back BACKUP with the WAN disabled while the
peer kept it.
The rehearsal earned its keep four times over, and none of these were visible
from reading the docs:
- `transition-script` alone is NOT safe to hang internet on. VyOS delivers it
through keepalived-fifo.py, and on one promotion that helper logged NOTHING
while Keepalived_vrrp logged all six instances entering MASTER and the
built-in notify_master for conntrack-sync ran normally. The result was a
router holding every VIP with no WAN -- the 2026-09-02 outage, recreated by
the mechanism meant to prevent it. Hence vrrp-wan-reconcile on a 30s timer:
the scripts give speed, the timer gives correctness.
- The health check must ask REALITY, not a marker. Keying "am I master" on a
/run file written by the transition script meant that when the script did
not run, the router believed it was backup, passed the check, and kept the
VIPs it could not serve. It now asks whether the VIP is actually on the box.
- The old address-based check DEADLOCKED this design: may-I-be-master required
already having WAN, and only the master gets WAN. That is why vyos002 sat in
FAULT for ever -- the safety check had silently removed the redundancy it
existed to protect.
- script-template must be the FIRST thing a script does. Sourced after an if,
an exec and a mkdir it terminated the script inside the source, rc=0, no
output: the reconciler reported success having done nothing. Hence the split
into vrrp-wan-apply, matching the shape /config/vyos-known-good already uses.
Two hazards found and handled rather than discovered in production:
- A `configure` session whose process dies leaks a unionfs mount under
/opt/vyatta/config/tmp, and one of those holds the commit lock -- after
which every commit fails, including the manual one you try to fix it with.
A 30s job that can leak one per failure wedges the box on its own, so the
reconciler reaps dead sessions before it starts. It cleared 8 on the sim.
- Any `save` while a box is master persists the enabled WAN into config.boot,
so a reboot would claim the shared MAC regardless of VRRP state. Observed:
an ordinary console-apply did exactly this. config.boot must keep `disable`
on BOTH routers; the model asserts it and vyos:verify reports it as drift.
NOT yet applied to production, and it should not be until the remaining item is
settled: a clean, deliberately-triggered failover has been seen via reboot, but
`restart vrrp` twice failed to move mastership at all, so the trigger for a
planned failover is still unproven.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-02 18:02:58 +01:00
|
|
|
cfg() { /opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration commands 2>/dev/null; }
|
|
|
|
|
wan_disabled(){ cfg | grep -q "vif ${WAN_VIF} disable"; }
|
|
|
|
|
|
|
|
|
|
configure
|
|
|
|
|
if [ "$MODE" = enable ]; then
|
|
|
|
|
# Guarded: `delete` of an absent node aborts the whole batch with
|
vyos: move PPPoE off the config plane onto a gated systemd unit
PPPoE HA could not work as written, and the reason is structural rather than a
bug: `set interfaces pppoe pppoe0 disable` and `delete` are handled identically
by interfaces_pppoe.py -- both UNLINK /etc/ppp/peers/pppoe0. That path is pppd's
own options file, so the resting state destroyed exactly what the promotion path
needed, and `ppp@pppoe0` restart-looped against it (observed: 47 restarts, zero
sessions at the access concentrator). It also made op-mode `connect interface
pppoe0` unusable, and put every failover behind a priority-322 commit where one
unrelated invalid node fails the whole thing -- which has already taken the
10 gig down once.
pppoe0 is now configured identically and ENABLED on both routers, so the peers
file always exists, and dialling is gated by a drop-in on the unit:
ConditionPathExists=/run/vrrp-wan/may-dial
ConditionPathExists=/etc/ppp/peers/pppoe0
/run is tmpfs, so the gate is shut at boot and neither box can dial before VRRP
has decided. That matters more than it first appears: with the node enabled,
interfaces_pppoe.py restarts ppp on EVERY commit touching the pppoe subtree when
the daemon is not running -- so the backup actively tries to dial whenever
anything commits. The gate is the only thing making that a no-op, which is why
vrrp-wan-reconcile now refuses to bless a box whose drop-in is missing: /etc is
per-image, and a VyOS upgrade would otherwise silently remove the protection.
may-dial is a LEASE, not a flag. ConditionPathExists is evaluated at start only
-- it can prevent a dial, never revoke one -- so a reconciler that stops running
while its box is demoted would keep the one ISP session for ever. The reconciler
renews the lease; a new 5s vrrp-wan-guard revokes it, and only ever revokes. It
fired correctly first time: "GUARD: lease stale (81s > 75s)".
Also: remove-then-stop on release (the file's absence blocks a NEW start that a
concurrent commit would trigger); a flap damper, because two routers that both
believe they hold the VIP will both dial and each dial kills the other's session
-- against a real ISP that is how an account gets rate-limited; and a guard on
`cfg` returning empty under commit-lock contention, which had already produced
one spurious "releasing" on a box that needed nothing.
GRACE 90 -> 180. accel-ppp's dead-peer budget is lcp-echo-interval(30) x
failure(3) = 90s, so the old value sat exactly on the boundary: a hard failover
into an AC that does not replace the stale session would fail its own check,
shed the VIPs, and leave both routers in FAULT.
The sim could not have tested any of this. Both routers now get the identical
WAN -- the secondary had none "because two PPPoE clients sharing one credential
is a different failure mode than anything production has", which is backwards:
that IS production. It also left the pair incomparable, ten NAT rules against
none. Safety now comes from resting state, not asymmetry.
Three more things the sim was hiding:
- the drift check's secondary regex omitted interfaces pppoe/bonding, nat
source and protocols failover, so it reported "in sync" for a box with no
WAN at all;
- the VRRP health-check and transition-script hooks existed on both live VMs
and in NEITHER generator -- the mechanism under test was pure undetected
drift;
- labsim-vyos's only default route was the libvirt-NAT scaffold, so every
"the LAN still has internet" verdict on it was answered by eth2 rather than
the WAN. --drop-scaffold applied; the earlier DHCP-failover proof is being
re-run because of it.
vrrp-wan-install ends the other half of that: the sim's previous proof came from
scripts hand-`sed`-ed in place, so the tested behaviour was not the committed
behaviour. `--check` now makes that a hard failure.
First green run: master holds both WANs, backup released, and the AC reports
exactly ONE session. sim-net-apply.sh check: all four in sync.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-05 18:50:16 +01:00
|
|
|
# "Nothing to delete", which once left the box detected-but-unfixed.
|
vyos: WAN follows VRRP mastership, rehearsed in labsim
The ISP is consumer with one static IP, so "both routers hold WAN" is not
available: the 10 gig lease is anchored to a cloned MAC and the PPPoE line to a
single credential. The WAN therefore has to move with mastership.
Proven end to end in the sim: the secondary was promoted, took the WAN, got a
lease, installed a default route, and a LAN VM reached the internet through it
(3/3, 9ms). Demoting released it -- link down, no address. Rebooting the master
converged correctly too: it came back BACKUP with the WAN disabled while the
peer kept it.
The rehearsal earned its keep four times over, and none of these were visible
from reading the docs:
- `transition-script` alone is NOT safe to hang internet on. VyOS delivers it
through keepalived-fifo.py, and on one promotion that helper logged NOTHING
while Keepalived_vrrp logged all six instances entering MASTER and the
built-in notify_master for conntrack-sync ran normally. The result was a
router holding every VIP with no WAN -- the 2026-09-02 outage, recreated by
the mechanism meant to prevent it. Hence vrrp-wan-reconcile on a 30s timer:
the scripts give speed, the timer gives correctness.
- The health check must ask REALITY, not a marker. Keying "am I master" on a
/run file written by the transition script meant that when the script did
not run, the router believed it was backup, passed the check, and kept the
VIPs it could not serve. It now asks whether the VIP is actually on the box.
- The old address-based check DEADLOCKED this design: may-I-be-master required
already having WAN, and only the master gets WAN. That is why vyos002 sat in
FAULT for ever -- the safety check had silently removed the redundancy it
existed to protect.
- script-template must be the FIRST thing a script does. Sourced after an if,
an exec and a mkdir it terminated the script inside the source, rc=0, no
output: the reconciler reported success having done nothing. Hence the split
into vrrp-wan-apply, matching the shape /config/vyos-known-good already uses.
Two hazards found and handled rather than discovered in production:
- A `configure` session whose process dies leaks a unionfs mount under
/opt/vyatta/config/tmp, and one of those holds the commit lock -- after
which every commit fails, including the manual one you try to fix it with.
A 30s job that can leak one per failure wedges the box on its own, so the
reconciler reaps dead sessions before it starts. It cleared 8 on the sim.
- Any `save` while a box is master persists the enabled WAN into config.boot,
so a reboot would claim the shared MAC regardless of VRRP state. Observed:
an ordinary console-apply did exactly this. config.boot must keep `disable`
on BOTH routers; the model asserts it and vyos:verify reports it as drift.
NOT yet applied to production, and it should not be until the remaining item is
settled: a clean, deliberately-triggered failover has been seen via reboot, but
`restart vrrp` twice failed to move mastership at all, so the trigger for a
planned failover is still unproven.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-02 18:02:58 +01:00
|
|
|
wan_disabled && delete interfaces bonding bond0 vif ${WAN_VIF} disable
|
|
|
|
|
else
|
|
|
|
|
wan_disabled || set interfaces bonding bond0 vif ${WAN_VIF} disable
|
vyos: a failover you can actually trigger, and four bugs found triggering it
A planned failover had no reliable lever. VyOS offers only `restart vrrp`, and
neither that nor `systemctl restart keepalived` is dependable: with advert_int 1
the peer declares the master dead after ~3.6s and a restart usually finishes
inside that window. Measured -- the same command moved mastership on one run and
not on the next three. A fail-back step you cannot trigger on purpose is not a
procedure, and the recovery card depended on one.
The lever is now `touch /run/vrrp-wan/force-fault`: the health check fails, the
sync group sheds every VIP, and the peer takes over. It exercises the same path
a real WAN loss takes rather than a special case, and it lives in /run so a
reboot cannot leave a router permanently ineligible.
Proven end to end in labsim: lever -> mastership moves -> WAN follows -> the old
master releases -> a LAN VM has internet -> the faulted router returns to BACKUP
and is eligible again.
Getting there exposed four real bugs, two of which would have broken a GENUINE
failover, not just the drill:
- The grace stamp was written only by the 30s reconciler, so a freshly
promoted master reached the 5s health check with no stamp, scored grace = 0,
failed instantly and went FAULT. With the peer already faulted that left
BOTH routers in FAULT and the LAN with no gateway at all -- worse than the
outage the check exists to prevent. The check now stamps on promotion.
- And it inherited STALE stamps from an earlier mastership, failing ~5s after
passing. The stamp is now cleared on the way down, by the health check
itself, not only by the reconciler.
- The lock fd leaked into VyOS's config session: `exec 9>` is inherited by the
long-lived unionfs-fuse the session spawns, which never closes it. From the
first config change on, every later reconciler run lost the flock and exited
0 having done nothing -- healthy-looking journal, silently stopped
reconciling. That is how a demoted router kept the WAN. Children now get 9>&-.
- vrrp-wan-apply touched pppoe0 unconditionally. On a box where pppoe0 has no
source-interface VyOS rejects the whole commit ("Physical source-interface
required"), taking the bond0.53 change down with it -- and the script still
returned 0, so the reconciler logged a release that never happened. pppoe0 is
now guarded on existence and the commit's verdict is propagated.
Still NOT applied to production. The pair is single-homed on WAN until it is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-02 23:27:17 +01:00
|
|
|
fi
|
vyos: move PPPoE off the config plane onto a gated systemd unit
PPPoE HA could not work as written, and the reason is structural rather than a
bug: `set interfaces pppoe pppoe0 disable` and `delete` are handled identically
by interfaces_pppoe.py -- both UNLINK /etc/ppp/peers/pppoe0. That path is pppd's
own options file, so the resting state destroyed exactly what the promotion path
needed, and `ppp@pppoe0` restart-looped against it (observed: 47 restarts, zero
sessions at the access concentrator). It also made op-mode `connect interface
pppoe0` unusable, and put every failover behind a priority-322 commit where one
unrelated invalid node fails the whole thing -- which has already taken the
10 gig down once.
pppoe0 is now configured identically and ENABLED on both routers, so the peers
file always exists, and dialling is gated by a drop-in on the unit:
ConditionPathExists=/run/vrrp-wan/may-dial
ConditionPathExists=/etc/ppp/peers/pppoe0
/run is tmpfs, so the gate is shut at boot and neither box can dial before VRRP
has decided. That matters more than it first appears: with the node enabled,
interfaces_pppoe.py restarts ppp on EVERY commit touching the pppoe subtree when
the daemon is not running -- so the backup actively tries to dial whenever
anything commits. The gate is the only thing making that a no-op, which is why
vrrp-wan-reconcile now refuses to bless a box whose drop-in is missing: /etc is
per-image, and a VyOS upgrade would otherwise silently remove the protection.
may-dial is a LEASE, not a flag. ConditionPathExists is evaluated at start only
-- it can prevent a dial, never revoke one -- so a reconciler that stops running
while its box is demoted would keep the one ISP session for ever. The reconciler
renews the lease; a new 5s vrrp-wan-guard revokes it, and only ever revokes. It
fired correctly first time: "GUARD: lease stale (81s > 75s)".
Also: remove-then-stop on release (the file's absence blocks a NEW start that a
concurrent commit would trigger); a flap damper, because two routers that both
believe they hold the VIP will both dial and each dial kills the other's session
-- against a real ISP that is how an account gets rate-limited; and a guard on
`cfg` returning empty under commit-lock contention, which had already produced
one spurious "releasing" on a box that needed nothing.
GRACE 90 -> 180. accel-ppp's dead-peer budget is lcp-echo-interval(30) x
failure(3) = 90s, so the old value sat exactly on the boundary: a hard failover
into an AC that does not replace the stale session would fail its own check,
shed the VIPs, and leave both routers in FAULT.
The sim could not have tested any of this. Both routers now get the identical
WAN -- the secondary had none "because two PPPoE clients sharing one credential
is a different failure mode than anything production has", which is backwards:
that IS production. It also left the pair incomparable, ten NAT rules against
none. Safety now comes from resting state, not asymmetry.
Three more things the sim was hiding:
- the drift check's secondary regex omitted interfaces pppoe/bonding, nat
source and protocols failover, so it reported "in sync" for a box with no
WAN at all;
- the VRRP health-check and transition-script hooks existed on both live VMs
and in NEITHER generator -- the mechanism under test was pure undetected
drift;
- labsim-vyos's only default route was the libvirt-NAT scaffold, so every
"the LAN still has internet" verdict on it was answered by eth2 rather than
the WAN. --drop-scaffold applied; the earlier DHCP-failover proof is being
re-run because of it.
vrrp-wan-install ends the other half of that: the sim's previous proof came from
scripts hand-`sed`-ed in place, so the tested behaviour was not the committed
behaviour. `--check` now makes that a hard failure.
First green run: master holds both WANs, backup released, and the AC reports
exactly ONE session. sim-net-apply.sh check: all four in sync.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-05 18:50:16 +01:00
|
|
|
# Report the commit's verdict. This script previously ended on `exit` (a
|
vyos: a failover you can actually trigger, and four bugs found triggering it
A planned failover had no reliable lever. VyOS offers only `restart vrrp`, and
neither that nor `systemctl restart keepalived` is dependable: with advert_int 1
the peer declares the master dead after ~3.6s and a restart usually finishes
inside that window. Measured -- the same command moved mastership on one run and
not on the next three. A fail-back step you cannot trigger on purpose is not a
procedure, and the recovery card depended on one.
The lever is now `touch /run/vrrp-wan/force-fault`: the health check fails, the
sync group sheds every VIP, and the peer takes over. It exercises the same path
a real WAN loss takes rather than a special case, and it lives in /run so a
reboot cannot leave a router permanently ineligible.
Proven end to end in labsim: lever -> mastership moves -> WAN follows -> the old
master releases -> a LAN VM has internet -> the faulted router returns to BACKUP
and is eligible again.
Getting there exposed four real bugs, two of which would have broken a GENUINE
failover, not just the drill:
- The grace stamp was written only by the 30s reconciler, so a freshly
promoted master reached the 5s health check with no stamp, scored grace = 0,
failed instantly and went FAULT. With the peer already faulted that left
BOTH routers in FAULT and the LAN with no gateway at all -- worse than the
outage the check exists to prevent. The check now stamps on promotion.
- And it inherited STALE stamps from an earlier mastership, failing ~5s after
passing. The stamp is now cleared on the way down, by the health check
itself, not only by the reconciler.
- The lock fd leaked into VyOS's config session: `exec 9>` is inherited by the
long-lived unionfs-fuse the session spawns, which never closes it. From the
first config change on, every later reconciler run lost the flock and exited
0 having done nothing -- healthy-looking journal, silently stopped
reconciling. That is how a demoted router kept the WAN. Children now get 9>&-.
- vrrp-wan-apply touched pppoe0 unconditionally. On a box where pppoe0 has no
source-interface VyOS rejects the whole commit ("Physical source-interface
required"), taking the bond0.53 change down with it -- and the script still
returned 0, so the reconciler logged a release that never happened. pppoe0 is
now guarded on existence and the commit's verdict is propagated.
Still NOT applied to production. The pair is single-homed on WAN until it is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-02 23:27:17 +01:00
|
|
|
# script-template function) and returned 0 even after "Commit failed", so the
|
vyos: move PPPoE off the config plane onto a gated systemd unit
PPPoE HA could not work as written, and the reason is structural rather than a
bug: `set interfaces pppoe pppoe0 disable` and `delete` are handled identically
by interfaces_pppoe.py -- both UNLINK /etc/ppp/peers/pppoe0. That path is pppd's
own options file, so the resting state destroyed exactly what the promotion path
needed, and `ppp@pppoe0` restart-looped against it (observed: 47 restarts, zero
sessions at the access concentrator). It also made op-mode `connect interface
pppoe0` unusable, and put every failover behind a priority-322 commit where one
unrelated invalid node fails the whole thing -- which has already taken the
10 gig down once.
pppoe0 is now configured identically and ENABLED on both routers, so the peers
file always exists, and dialling is gated by a drop-in on the unit:
ConditionPathExists=/run/vrrp-wan/may-dial
ConditionPathExists=/etc/ppp/peers/pppoe0
/run is tmpfs, so the gate is shut at boot and neither box can dial before VRRP
has decided. That matters more than it first appears: with the node enabled,
interfaces_pppoe.py restarts ppp on EVERY commit touching the pppoe subtree when
the daemon is not running -- so the backup actively tries to dial whenever
anything commits. The gate is the only thing making that a no-op, which is why
vrrp-wan-reconcile now refuses to bless a box whose drop-in is missing: /etc is
per-image, and a VyOS upgrade would otherwise silently remove the protection.
may-dial is a LEASE, not a flag. ConditionPathExists is evaluated at start only
-- it can prevent a dial, never revoke one -- so a reconciler that stops running
while its box is demoted would keep the one ISP session for ever. The reconciler
renews the lease; a new 5s vrrp-wan-guard revokes it, and only ever revokes. It
fired correctly first time: "GUARD: lease stale (81s > 75s)".
Also: remove-then-stop on release (the file's absence blocks a NEW start that a
concurrent commit would trigger); a flap damper, because two routers that both
believe they hold the VIP will both dial and each dial kills the other's session
-- against a real ISP that is how an account gets rate-limited; and a guard on
`cfg` returning empty under commit-lock contention, which had already produced
one spurious "releasing" on a box that needed nothing.
GRACE 90 -> 180. accel-ppp's dead-peer budget is lcp-echo-interval(30) x
failure(3) = 90s, so the old value sat exactly on the boundary: a hard failover
into an AC that does not replace the stale session would fail its own check,
shed the VIPs, and leave both routers in FAULT.
The sim could not have tested any of this. Both routers now get the identical
WAN -- the secondary had none "because two PPPoE clients sharing one credential
is a different failure mode than anything production has", which is backwards:
that IS production. It also left the pair incomparable, ten NAT rules against
none. Safety now comes from resting state, not asymmetry.
Three more things the sim was hiding:
- the drift check's secondary regex omitted interfaces pppoe/bonding, nat
source and protocols failover, so it reported "in sync" for a box with no
WAN at all;
- the VRRP health-check and transition-script hooks existed on both live VMs
and in NEITHER generator -- the mechanism under test was pure undetected
drift;
- labsim-vyos's only default route was the libvirt-NAT scaffold, so every
"the LAN still has internet" verdict on it was answered by eth2 rather than
the WAN. --drop-scaffold applied; the earlier DHCP-failover proof is being
re-run because of it.
vrrp-wan-install ends the other half of that: the sim's previous proof came from
scripts hand-`sed`-ed in place, so the tested behaviour was not the committed
behaviour. `--check` now makes that a hard failure.
First green run: master holds both WANs, backup released, and the AC reports
exactly ONE session. sim-net-apply.sh check: all four in sync.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-05 18:50:16 +01:00
|
|
|
# reconciler logged a release that had not happened -- the worst kind of failure
|
|
|
|
|
# for something whose job is to keep two routers from holding one WAN.
|
vyos: a failover you can actually trigger, and four bugs found triggering it
A planned failover had no reliable lever. VyOS offers only `restart vrrp`, and
neither that nor `systemctl restart keepalived` is dependable: with advert_int 1
the peer declares the master dead after ~3.6s and a restart usually finishes
inside that window. Measured -- the same command moved mastership on one run and
not on the next three. A fail-back step you cannot trigger on purpose is not a
procedure, and the recovery card depended on one.
The lever is now `touch /run/vrrp-wan/force-fault`: the health check fails, the
sync group sheds every VIP, and the peer takes over. It exercises the same path
a real WAN loss takes rather than a special case, and it lives in /run so a
reboot cannot leave a router permanently ineligible.
Proven end to end in labsim: lever -> mastership moves -> WAN follows -> the old
master releases -> a LAN VM has internet -> the faulted router returns to BACKUP
and is eligible again.
Getting there exposed four real bugs, two of which would have broken a GENUINE
failover, not just the drill:
- The grace stamp was written only by the 30s reconciler, so a freshly
promoted master reached the 5s health check with no stamp, scored grace = 0,
failed instantly and went FAULT. With the peer already faulted that left
BOTH routers in FAULT and the LAN with no gateway at all -- worse than the
outage the check exists to prevent. The check now stamps on promotion.
- And it inherited STALE stamps from an earlier mastership, failing ~5s after
passing. The stamp is now cleared on the way down, by the health check
itself, not only by the reconciler.
- The lock fd leaked into VyOS's config session: `exec 9>` is inherited by the
long-lived unionfs-fuse the session spawns, which never closes it. From the
first config change on, every later reconciler run lost the flock and exited
0 having done nothing -- healthy-looking journal, silently stopped
reconciling. That is how a demoted router kept the WAN. Children now get 9>&-.
- vrrp-wan-apply touched pppoe0 unconditionally. On a box where pppoe0 has no
source-interface VyOS rejects the whole commit ("Physical source-interface
required"), taking the bond0.53 change down with it -- and the script still
returned 0, so the reconciler logged a release that never happened. pppoe0 is
now guarded on existence and the commit's verdict is propagated.
Still NOT applied to production. The pair is single-homed on WAN until it is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-02 23:27:17 +01:00
|
|
|
if commit 2>&1 | tee /tmp/vrrp-wan-commit.log | grep -qi "commit failed"; then
|
|
|
|
|
logger -t vrrp-wan "COMMIT FAILED applying '$MODE' -- see /tmp/vrrp-wan-commit.log"
|
|
|
|
|
exit 1
|
vyos: WAN follows VRRP mastership, rehearsed in labsim
The ISP is consumer with one static IP, so "both routers hold WAN" is not
available: the 10 gig lease is anchored to a cloned MAC and the PPPoE line to a
single credential. The WAN therefore has to move with mastership.
Proven end to end in the sim: the secondary was promoted, took the WAN, got a
lease, installed a default route, and a LAN VM reached the internet through it
(3/3, 9ms). Demoting released it -- link down, no address. Rebooting the master
converged correctly too: it came back BACKUP with the WAN disabled while the
peer kept it.
The rehearsal earned its keep four times over, and none of these were visible
from reading the docs:
- `transition-script` alone is NOT safe to hang internet on. VyOS delivers it
through keepalived-fifo.py, and on one promotion that helper logged NOTHING
while Keepalived_vrrp logged all six instances entering MASTER and the
built-in notify_master for conntrack-sync ran normally. The result was a
router holding every VIP with no WAN -- the 2026-09-02 outage, recreated by
the mechanism meant to prevent it. Hence vrrp-wan-reconcile on a 30s timer:
the scripts give speed, the timer gives correctness.
- The health check must ask REALITY, not a marker. Keying "am I master" on a
/run file written by the transition script meant that when the script did
not run, the router believed it was backup, passed the check, and kept the
VIPs it could not serve. It now asks whether the VIP is actually on the box.
- The old address-based check DEADLOCKED this design: may-I-be-master required
already having WAN, and only the master gets WAN. That is why vyos002 sat in
FAULT for ever -- the safety check had silently removed the redundancy it
existed to protect.
- script-template must be the FIRST thing a script does. Sourced after an if,
an exec and a mkdir it terminated the script inside the source, rc=0, no
output: the reconciler reported success having done nothing. Hence the split
into vrrp-wan-apply, matching the shape /config/vyos-known-good already uses.
Two hazards found and handled rather than discovered in production:
- A `configure` session whose process dies leaks a unionfs mount under
/opt/vyatta/config/tmp, and one of those holds the commit lock -- after
which every commit fails, including the manual one you try to fix it with.
A 30s job that can leak one per failure wedges the box on its own, so the
reconciler reaps dead sessions before it starts. It cleared 8 on the sim.
- Any `save` while a box is master persists the enabled WAN into config.boot,
so a reboot would claim the shared MAC regardless of VRRP state. Observed:
an ordinary console-apply did exactly this. config.boot must keep `disable`
on BOTH routers; the model asserts it and vyos:verify reports it as drift.
NOT yet applied to production, and it should not be until the remaining item is
settled: a clean, deliberately-triggered failover has been seen via reboot, but
`restart vrrp` twice failed to move mastership at all, so the trigger for a
planned failover is still unproven.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-02 18:02:58 +01:00
|
|
|
fi
|
|
|
|
|
exit
|