Matches the DNS the USG used on its WAN (wan_dns1/wan_dns2), replacing the
10.0.0.194 I had set earlier. Applied to both boxes and saved; VRRP unchanged
(MASTER/BACKUP), NTP still synced, no config drift.
/config/modes/unifi.boot was RE-CAPTURED on both afterwards. It had been taken
before this change, so the escape hatch would have quietly reverted the
resolver on any rollback -- a snapshot is only an escape hatch for the state it
was taken from.
Two things recorded in the runbook:
- The boxes' name resolution now depends on the internet, so between
unplugging the USG and PPPoE establishing they have no DNS. Harmless:
nothing in the switch resolves a name, and the health checks use DNS
precisely to prove the WAN came up.
- Internal ad.itaz.eu names still resolve via Google, because that zone is
published publicly with private addresses in it (nas001 -> 10.0.0.194,
kvm-macstudio1 -> 192.168.3.8). So no conditional forward was needed --
though it is worth knowing the internal topology is public.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
The access table led with 192.168.8.143/.144 and offered the LoT addresses as a
fallback ("if unreachable, try"). That is backwards and would have stranded the
operator at the worst moment: the workstation sits on LoT, and reaching
192.168.8.x routes *through the USG*, so those addresses are guaranteed dead the
instant it is unplugged. Measured:
ip route get 192.168.8.143 -> via 10.0.0.1 (the USG)
ip route get 10.0.1.252 -> dev lanbr0 (same L2, no gateway)
10.0.1.252 and .253 are on the LoT VLAN, same broadcast domain as the
workstation, and both answer SSH. They are now the only addresses the runbook
gives, with the k8s ones struck through.
Also recorded: the switch cannot be run before unplugging the USG (two devices
on every gateway address; the guard refuses), so the order is forced. And
during the gap between unplugging and completing the switch there is no
inter-VLAN routing at all -- which means the JetKVMs (Management and kvm) and
Tailscale are NOT fallbacks in that window. LoT SSH is the only remote path;
below it is physical console. Added a step 0: open both SSH sessions and leave
them open before touching anything.
Both boxes are now installed and pass the pre-flight gate: mode unifi,
unifi.boot 231 lines including the reload action, delta at the right priority
(200/100), wan-secrets 0600, script executable, no config drift, VRRP still
MASTER/BACKUP. `vyos-unifi-switch vyos` refuses on both -- all six gateway
addresses detected answering ARP -- and neither box has gained dhcp-server, dns
or nat, so nothing about their behaviour has changed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
Ran vyos001's real running config plus the real production delta on the labsim
router -- same VyOS version, isolated OVS bridge with no physical NIC, so the
sim briefly holding vyos001's actual addresses could not reach the real LAN.
Result: all 317 commands accepted and the whole delta commits (COMMIT OK), the
revert is byte-exact, and auto-revert fires without rebooting (uptime and
boot-id unchanged across it).
Also written down are the two things this did NOT establish, because a runbook
that overstates its own coverage is worse than one that admits the gap: PPPoE
cannot be tried while the USG holds the single available session, and the
rehearsal ran with vyos001's eth2/eth3 stanzas stripped because the sim VM has
two NICs rather than four.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
The cutover is a switch, not a migration: unplug the USG, run one command, and
if anything is wrong run the other one and plug it back in. The operator will
have no internet during this and therefore no assistant, so the machinery has
to live on the boxes and the failure paths have to be proven in advance.
vyos-mode-delta.py generates the delta that turns the passive pair into the
gateway. Only one artifact is authored: gateway mode is always derived from
`load unifi.boot` + delta, so there is no inverse to maintain and no drift
between two hand-kept configs. It reuses unifi-to-vyos.py rather than
duplicating it, so what labsim proved and what production gets are one code
path. The PPPoE password is never written into the delta -- it carries a
placeholder the switch substitutes at apply time from /config/wan-secrets --
and generation fails if the real password appears in the output.
Two things the delta covers that the plan had underweighted:
- VyOS defaults to ACCEPT while the USG has an implicit WAN drop. Migrating
the port forwards alone would have left the router's own services and the
whole LAN reachable from the WAN. Added a stateful baseline scoped to the
WAN interface rather than a global default-action drop, so a mistake there
cannot lock anyone out over the LAN -- the only way back during a cutover.
- The old VIPs are NOT at network+254 on the /23 networks; they are
192.168.9.254, 10.0.9.254 and 10.0.1.254, in the upper half. A delete
naming a computed address fails quietly and leaves the group holding two
VIPs. The delta deletes the whole address node instead of guessing.
vyos-unifi-switch runs on the box from /config, which survives image upgrades,
so it works from a local terminal or the JetKVM with no workstation.
Proven on labsim, not assumed:
- unifi mode restores the previous config BYTE-EXACT (138 lines, diff clean).
- Auto-revert fires when the commit is not confirmed: 85 static-mappings ->
0, kea stopped, hostname restored, and uptime plus boot-id UNCHANGED, so
it reloaded rather than rebooted. That distinction is the whole reason
`commit-confirm action reload` is a prerequisite.
- Health-check failure triggers an immediate revert_soft rather than waiting
out the timer.
Four bugs found while doing it, each of which produced a wrong answer rather
than an error:
- commit-confirm is TWO steps. `config-mgmt commit_confirm` only arms the
revert timer; a normal `commit` still has to follow. Arming alone committed
nothing while reporting success.
- `sudo sg vyattacfg "config-mgmt ..."` loses the config-session environment,
so it reported "No configuration changes to commit" against a candidate
that plainly had 446 added lines.
- `... | grep -q` under `set -o pipefail` reports FAILURE on a match: grep
exits early, the producer takes SIGPIPE. Whether it triggers depends on
output size, so `status` misreported the mode intermittently.
- `show configuration commands` quotes values, so a fixed-string match for
`action reload` never matched `action 'reload'`.
CUTOVER.md is the printable runbook: both reachable addresses per box, the
escape hatch first, and the note that PPPoE is the one thing that could not be
tested beforehand.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
