lab/migration/CUTOVER.md

Cutover runbook — USG to VyOS

Print this. During the cutover there is no internet, so there is no assistant and no web search. Everything you need is on this page and on the boxes themselves.

vyos001 (MASTER)	192.168.8.143 — also 10.0.1.252 on LoT
vyos002 (BACKUP)	192.168.8.144 — also 10.0.1.253 on LoT
JetKVMs	192.168.1.28, 192.168.1.29, 192.168.3.6
Switch script	/config/vyos-unifi-switch on each box
Login	user vyos

Both boxes have two reachable addresses on different interfaces. If eth2 (192.168.8.x) is unreachable, try the LoT address (10.0.1.x), and only then the JetKVM.

---

If something is wrong, do this

sudo /config/vyos-unifi-switch unifi

Then reconnect the USG. That command runs no health checks, asks nothing and cannot refuse. It restores a byte-exact copy of the configuration the box had before the cutover — verified by diff, not by assumption.

You do not have to be quick. If you do nothing at all after vyos-unifi-switch vyos, the box reverts by itself within 10 minutes. Verified: config returns to the previous state and the box does not reboot (uptime and boot-id unchanged across an auto-revert).

---

What has actually been tested

Proven on the labsim router (same VyOS version, isolated OVS bridge with no physical NIC), by loading vyos001's real running config and applying the real production delta:

• All 317 commands accepted, and the whole delta commits (COMMIT OK).
• unifi mode restores the previous config byte-exact (diff clean).
• Auto-revert fires when the commit is not confirmed: config returns to the saved state and the box does not reboot — uptime and boot-id unchanged across the revert.
• Failed health checks trigger an immediate revert rather than waiting out the timer.

Two bugs were found this way and would each have failed the entire switch, since the delta commits as one unit: bond0.51 did not exist for PPPoE to reference, and translation port rejects a port list.

Not tested, and untestable in advance:

• PPPoE. The line permits one session and the USG holds it. The first real attempt is during the cutover.
• The commit on the real boxes. The rehearsal ran with vyos001's eth2 and eth3 stanzas stripped, because the sim VM has only two NICs. Those are plain interface configs that already work on the real hardware, but they were not part of what committed.

Before you unplug anything

1. Tether your workstation to your phone if you want the assistant available. Cutting the USG cuts your internet, not your LAN.
2. On both boxes, confirm the machinery is present:

   sudo /config/vyos-unifi-switch status
   ls -la /config/modes/          # unifi.boot + to-vyos.commands
   ls -la /config/wan-secrets     # must be 0600
   

   status must report mode: unifi. If unifi.boot is missing, stop — there is no way back without it.
3. Confirm the revert action is reload, not reboot:

   show configuration commands | match commit-confirm
   

   Must show action 'reload'. Without it a failed switch reboots the firewall instead of reverting it. The switch script refuses to run if this is missing, but check anyway.

The cutover

1. Physically disconnect the USG. Not just powered off — disconnected. The switch script refuses to run while anything still answers on a gateway address, because two devices on .1 is the worst available outcome.
2. On vyos002 (BACKUP) first:

   sudo /config/vyos-unifi-switch vyos
   

3. Watch the health checks. They cover PPPoE, the default route, kea, the DNS forwarder and reachability. On failure the script reverts immediately and tells you so.
4. If vyos002 came up clean, repeat on vyos001 (MASTER).
5. Check a real client: does it get an address, and is it the same address as before? Every active client has a reservation, so it should be.

What will probably go wrong first

PPPoE. It is the one thing that could not be tested in advance — the line permits a single session and the USG held it until you unplugged it. If the WAN check fails:

show interfaces pppoe pppoe0
sudo journalctl -u ppp@pppoe0 -n 50 --no-pager

Check the credential in /config/wan-secrets and that VLAN 51 actually reaches the box. If it will not come up, run vyos-unifi-switch unifi, reconnect the USG, and debug with the internet back on.

Things that are true and easy to forget

• WiFi keeps working, but through VyOS. The SSIDs stay in UniFi and the APs are untouched, but 37 of 83 active clients are wireless and every one is on LoT — they get their addresses from VyOS now.
• DHCP leases last 24h (86400s). A device that does not renew promptly keeps its old address for a while. That is fine, not a symptom.
• The firewalls resolve via 10.0.0.194, not the USG. That was changed ahead of time precisely so they keep DNS when the USG goes away.
• The USG was a DNS resolver for every VLAN except LoT. VyOS now runs dns forwarding in its place. If names stop resolving but IPs still work, that is where to look.
• eth2 and bond0.2 are both in 192.168.8.0/23. It works, but if you see odd source-address behaviour on the management NIC, that is why.

Afterwards

Once it has been stable for a day:

• Re-run migration/unifi-export.py — the UniFi controller is no longer the source of truth for DHCP, and the export will drift.
• The VPN rules (ESP, UDP 500/4500) are carried over but the VPN itself still terminated on the USG. Decide whether it moves.
• labsim still holds a deliberate kvm→k8s drop rule from earlier testing.