The cutover succeeded on vyos001 -- gateway live, bond0.53 holding
87.192.101.48 with the cloned MAC, kea serving, clients routing out through NAT.
vyos002 then ran the same script and was judged unhealthy, because the mandatory
checks are "default route exists / internet reachable / DNS resolves" and the
backup deliberately holds its WAN interfaces DOWN. Its config was correct; the
check did not apply to it. Confirmed by hand before the timer could revert a
good config.
This is the third instance of one mistake: asserting a condition that is not
true of the box being checked. First requiring every WAN when one suffices, now
requiring a WAN on the box that is configured not to have one.
A delta containing `interfaces ... disable` for the WAN now identifies the
backup, and the WAN-dependent checks are skipped with a note. kea and the DNS
forwarder remain mandatory on both -- those are what the backup must actually
be able to do.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
This reverted a cutover that had actually succeeded.
The evidence, from the revert tearing it down:
dhclient: DHCPRELEASE of 87.192.101.48 on bond0.53 to 185.232.119.244
vtysh: "no ip route 0.0.0.0/0 87.192.96.1 bond0.53 tag 210 1"
netlinkd: RTM_NEWLINK -> bond0.53, mac=f0:9f:c2:12:9b:4f
bond0.53 came up with the cloned MAC and was handed 87.192.101.48 -- the exact
public address the USG holds -- with a default route via the real ISP gateway.
kea was serving live LAN clients at the same moment (10.0.0.12, 10.0.0.13,
192.168.8.28). The gateway was working.
The only failure was pppoe0: ppp@pppoe0.service exited 5/NOTINSTALLED. That is
the Vodafone FAILOVER line, and the health check listed "pppoe0 has an address"
as mandatory, so a working gateway was torn down because its backup WAN was
down. The check encoded "every WAN must work" when the requirement is "the box
must reach the internet".
Now: default route, reachability and DNS are mandatory; each WAN interface is
reported individually but fatal on neither. A failover line being down is worth
seeing, not worth reverting for.
This also incidentally settles the last genuine unknown in the migration, which
could not be tested any other way: the ISP does hand the same lease to the
cloned MAC. That was the one thing I had said was unknowable until the USG let
go of it.
Note the earlier polling fix (54b21fa) addressed a real weakness but not this
failure -- no amount of waiting would have satisfied a check that required a
line which was never going to come up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
A real cutover attempt reported failure and reverted a configuration that may
well have been fine. The health check waited a fixed 25 seconds and then judged:
[switch] committed. Waiting 25s for PPPoE and services to settle...
[switch] FAIL pppoe0 has an address
25s is far too short for a WAN. PPPoE alone is PADI/PADO/PADR/PADS followed by
LCP, authentication and IPCP -- routinely 15-30s on its own. Both lines had also
just been released by the USG seconds earlier, and ISPs commonly hold the
previous session and MAC binding for minutes before leasing to the "same" CPE
again, which is exactly what a cloned MAC looks like from their side. The one
thing the design could not tolerate was being impatient, and it was.
Now polls every 15s up to HEALTH_BUDGET (default 180s), reporting progress, and
stops early the moment everything is healthy. The budget deliberately finishes
long before commit-confirm fires -- 180s against a 10 minute timer leaves 420s
of margin -- so the decision to confirm or revert stays ours rather than being
made by the timer.
Also recorded while chasing this: the earlier claim that VLANs 51/53 are not
trunked to the firewalls was WRONG, and the UniFi port settings disprove it --
those LAG ports are Native VLAN Management (1) with Tagged VLAN Management set
to Allow All. My evidence never supported the claim: a passive RX count cannot
distinguish an absent VLAN from a quiet one, because switches do not flood
unicast, and the active DHCP probe used a random MAC that an ISP binding to its
registered CPE would ignore regardless. Both observations fit a perfectly
healthy trunk.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
The ARP guard against two devices holding the same gateway address is the one
check that prevents this script's worst outcome. It reads the addresses to
probe out of the delta -- so a delta with no VIP lines made the guard inert,
and it previously warned and carried on. That was a testing convenience (the
lab delta has no VIPs) weakening a production safety check, which is backwards.
It now refuses by default. ALLOW_NO_VIP_DELTA=1 is the explicit lab override.
The guard's probing path had never actually executed before this: every sim
run took the no-VIPs branch. Verified against the live USG from vyos001:
arping is present on VyOS, the regex extracts all six gateway addresses from
the real delta (192.168.1.1, 192.168.8.1, 192.168.3.1, 10.0.9.0, 10.0.0.1,
192.168.2.1), and every one of them answers ARP right now -- so on the real
boxes, with the USG connected, the guard fires.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
The cutover is a switch, not a migration: unplug the USG, run one command, and
if anything is wrong run the other one and plug it back in. The operator will
have no internet during this and therefore no assistant, so the machinery has
to live on the boxes and the failure paths have to be proven in advance.
vyos-mode-delta.py generates the delta that turns the passive pair into the
gateway. Only one artifact is authored: gateway mode is always derived from
`load unifi.boot` + delta, so there is no inverse to maintain and no drift
between two hand-kept configs. It reuses unifi-to-vyos.py rather than
duplicating it, so what labsim proved and what production gets are one code
path. The PPPoE password is never written into the delta -- it carries a
placeholder the switch substitutes at apply time from /config/wan-secrets --
and generation fails if the real password appears in the output.
Two things the delta covers that the plan had underweighted:
- VyOS defaults to ACCEPT while the USG has an implicit WAN drop. Migrating
the port forwards alone would have left the router's own services and the
whole LAN reachable from the WAN. Added a stateful baseline scoped to the
WAN interface rather than a global default-action drop, so a mistake there
cannot lock anyone out over the LAN -- the only way back during a cutover.
- The old VIPs are NOT at network+254 on the /23 networks; they are
192.168.9.254, 10.0.9.254 and 10.0.1.254, in the upper half. A delete
naming a computed address fails quietly and leaves the group holding two
VIPs. The delta deletes the whole address node instead of guessing.
vyos-unifi-switch runs on the box from /config, which survives image upgrades,
so it works from a local terminal or the JetKVM with no workstation.
Proven on labsim, not assumed:
- unifi mode restores the previous config BYTE-EXACT (138 lines, diff clean).
- Auto-revert fires when the commit is not confirmed: 85 static-mappings ->
0, kea stopped, hostname restored, and uptime plus boot-id UNCHANGED, so
it reloaded rather than rebooted. That distinction is the whole reason
`commit-confirm action reload` is a prerequisite.
- Health-check failure triggers an immediate revert_soft rather than waiting
out the timer.
Four bugs found while doing it, each of which produced a wrong answer rather
than an error:
- commit-confirm is TWO steps. `config-mgmt commit_confirm` only arms the
revert timer; a normal `commit` still has to follow. Arming alone committed
nothing while reporting success.
- `sudo sg vyattacfg "config-mgmt ..."` loses the config-session environment,
so it reported "No configuration changes to commit" against a candidate
that plainly had 446 added lines.
- `... | grep -q` under `set -o pipefail` reports FAILURE on a match: grep
exits early, the producer takes SIGPIPE. Whether it triggers depends on
output size, so `status` misreported the mode intermittently.
- `show configuration commands` quotes values, so a fixed-string match for
`action reload` never matched `action 'reload'`.
CUTOVER.md is the printable runbook: both reachable addresses per box, the
escape hatch first, and the note that PPPoE is the one thing that could not be
tested beforehand.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
