lab/migration/CUTOVER.md

# Cutover runbook — USG to VyOS

**Print this.** During the cutover there is no internet, so there is no
assistant and no web search. Everything you need is on this page and on the
boxes themselves.

| | |
|---|---|
| vyos001 (MASTER) | `192.168.8.143` — also `10.0.1.252` on LoT |
| vyos002 (BACKUP) | `192.168.8.144` — also `10.0.1.253` on LoT |
| JetKVMs | `192.168.1.28`, `192.168.1.29`, `192.168.3.6` |
| Switch script | `/config/vyos-unifi-switch` on each box |
| Login | user `vyos` |

Both boxes have **two** reachable addresses on different interfaces. If `eth2`
(`192.168.8.x`) is unreachable, try the LoT address (`10.0.1.x`), and only then
the JetKVM.

---

## If something is wrong, do this

```
sudo /config/vyos-unifi-switch unifi
```

Then reconnect the USG. That command runs no health checks, asks nothing and
cannot refuse. It restores a byte-exact copy of the configuration the box had
before the cutover — verified by diff, not by assumption.

**You do not have to be quick.** If you do nothing at all after
`vyos-unifi-switch vyos`, the box reverts by itself within 10 minutes. Verified:
config returns to the previous state and the box does **not** reboot
(`uptime` and boot-id unchanged across an auto-revert).

---

## What has actually been tested

Proven on the labsim router (same VyOS version, isolated OVS bridge with no
physical NIC), by loading **vyos001's real running config** and applying the
**real production delta**:

- All 317 commands accepted, and the whole delta **commits** (`COMMIT OK`).
- `unifi` mode restores the previous config **byte-exact** (diff clean).
- Auto-revert fires when the commit is not confirmed: config returns to the
  saved state and the box does **not** reboot — `uptime` and boot-id unchanged
  across the revert.
- Failed health checks trigger an immediate revert rather than waiting out the
  timer.

Two bugs were found this way and would each have failed the entire switch,
since the delta commits as one unit: `bond0.51` did not exist for PPPoE to
reference, and `translation port` rejects a port list.

**Not tested, and untestable in advance:**

- **PPPoE.** The line permits one session and the USG holds it. The first real
  attempt is during the cutover.
- **The commit on the real boxes.** The rehearsal ran with vyos001's `eth2` and
  `eth3` stanzas stripped, because the sim VM has only two NICs. Those are
  plain interface configs that already work on the real hardware, but they were
  not part of what committed.

## Before you unplug anything

1. Tether your workstation to your phone if you want the assistant available.
   Cutting the USG cuts your internet, not your LAN.
2. On **both** boxes, confirm the machinery is present:
   ```
   sudo /config/vyos-unifi-switch status
   ls -la /config/modes/          # unifi.boot + to-vyos.commands
   ls -la /config/wan-secrets     # must be 0600
   ```
   `status` must report `mode: unifi`. If `unifi.boot` is missing, **stop** —
   there is no way back without it.
3. Confirm the revert action is `reload`, not `reboot`:
   ```
   show configuration commands | match commit-confirm
   ```
   Must show `action 'reload'`. Without it a failed switch **reboots** the
   firewall instead of reverting it. The switch script refuses to run if this
   is missing, but check anyway.

## The cutover

1. **Physically disconnect the USG.** Not just powered off — disconnected. The
   switch script refuses to run while anything still answers on a gateway
   address, because two devices on `.1` is the worst available outcome.
2. On **vyos002 (BACKUP) first**:
   ```
   sudo /config/vyos-unifi-switch vyos
   ```
3. Watch the health checks. They cover PPPoE, the default route, kea, the DNS
   forwarder and reachability. On failure the script reverts immediately and
   tells you so.
4. If vyos002 came up clean, repeat on **vyos001 (MASTER)**.
5. Check a real client: does it get an address, and is it the *same* address as
   before? Every active client has a reservation, so it should be.

## What will probably go wrong first

**PPPoE.** It is the one thing that could not be tested in advance — the line
permits a single session and the USG held it until you unplugged it. If the
WAN check fails:

```
show interfaces pppoe pppoe0
sudo journalctl -u ppp@pppoe0 -n 50 --no-pager
```

Check the credential in `/config/wan-secrets` and that VLAN 51 actually reaches
the box. If it will not come up, run `vyos-unifi-switch unifi`, reconnect the
USG, and debug with the internet back on.

## Things that are true and easy to forget

- **WiFi keeps working, but through VyOS.** The SSIDs stay in UniFi and the APs
  are untouched, but 37 of 83 active clients are wireless and every one is on
  LoT — they get their addresses from VyOS now.
- **DHCP leases last 24h (86400s).** A device that does not renew promptly keeps
  its old address for a while. That is fine, not a symptom.
- **The firewalls resolve via `10.0.0.194`**, not the USG. That was changed
  ahead of time precisely so they keep DNS when the USG goes away.
- **The USG was a DNS resolver** for every VLAN except LoT. VyOS now runs
  `dns forwarding` in its place. If names stop resolving but IPs still work,
  that is where to look.
- **`eth2` and `bond0.2` are both in `192.168.8.0/23`.** It works, but if you
  see odd source-address behaviour on the management NIC, that is why.

## Afterwards

Once it has been stable for a day:

- Re-run `migration/unifi-export.py` — the UniFi controller is no longer the
  source of truth for DHCP, and the export will drift.
- The VPN rules (ESP, UDP 500/4500) are carried over but the VPN itself still
  terminated on the USG. Decide whether it moves.
- `labsim` still holds a deliberate `kvm→k8s` drop rule from earlier testing.