feat(migration): reversible USG->VyOS switch, proven on the sim
The cutover is a switch, not a migration: unplug the USG, run one command, and
if anything is wrong run the other one and plug it back in. The operator will
have no internet during this and therefore no assistant, so the machinery has
to live on the boxes and the failure paths have to be proven in advance.
vyos-mode-delta.py generates the delta that turns the passive pair into the
gateway. Only one artifact is authored: gateway mode is always derived from
`load unifi.boot` + delta, so there is no inverse to maintain and no drift
between two hand-kept configs. It reuses unifi-to-vyos.py rather than
duplicating it, so what labsim proved and what production gets are one code
path. The PPPoE password is never written into the delta -- it carries a
placeholder the switch substitutes at apply time from /config/wan-secrets --
and generation fails if the real password appears in the output.
Two things the delta covers that the plan had underweighted:
- VyOS defaults to ACCEPT while the USG has an implicit WAN drop. Migrating
the port forwards alone would have left the router's own services and the
whole LAN reachable from the WAN. Added a stateful baseline scoped to the
WAN interface rather than a global default-action drop, so a mistake there
cannot lock anyone out over the LAN -- the only way back during a cutover.
- The old VIPs are NOT at network+254 on the /23 networks; they are
192.168.9.254, 10.0.9.254 and 10.0.1.254, in the upper half. A delete
naming a computed address fails quietly and leaves the group holding two
VIPs. The delta deletes the whole address node instead of guessing.
vyos-unifi-switch runs on the box from /config, which survives image upgrades,
so it works from a local terminal or the JetKVM with no workstation.
Proven on labsim, not assumed:
- unifi mode restores the previous config BYTE-EXACT (138 lines, diff clean).
- Auto-revert fires when the commit is not confirmed: 85 static-mappings ->
0, kea stopped, hostname restored, and uptime plus boot-id UNCHANGED, so
it reloaded rather than rebooted. That distinction is the whole reason
`commit-confirm action reload` is a prerequisite.
- Health-check failure triggers an immediate revert_soft rather than waiting
out the timer.
Four bugs found while doing it, each of which produced a wrong answer rather
than an error:
- commit-confirm is TWO steps. `config-mgmt commit_confirm` only arms the
revert timer; a normal `commit` still has to follow. Arming alone committed
nothing while reporting success.
- `sudo sg vyattacfg "config-mgmt ..."` loses the config-session environment,
so it reported "No configuration changes to commit" against a candidate
that plainly had 446 added lines.
- `... | grep -q` under `set -o pipefail` reports FAILURE on a match: grep
exits early, the producer takes SIGPIPE. Whether it triggers depends on
output size, so `status` misreported the mode intermittently.
- `show configuration commands` quotes values, so a fixed-string match for
`action reload` never matched `action 'reload'`.
CUTOVER.md is the printable runbook: both reachable addresses per box, the
escape hatch first, and the note that PPPoE is the one thing that could not be
tested beforehand.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-15 23:30:28 +01:00
|
|
|
# Cutover runbook — USG to VyOS
|
|
|
|
|
|
|
|
|
|
**Print this.** During the cutover there is no internet, so there is no
|
|
|
|
|
assistant and no web search. Everything you need is on this page and on the
|
|
|
|
|
boxes themselves.
|
|
|
|
|
|
fix(migration): the runbook pointed at addresses that die with the USG
The access table led with 192.168.8.143/.144 and offered the LoT addresses as a
fallback ("if unreachable, try"). That is backwards and would have stranded the
operator at the worst moment: the workstation sits on LoT, and reaching
192.168.8.x routes *through the USG*, so those addresses are guaranteed dead the
instant it is unplugged. Measured:
ip route get 192.168.8.143 -> via 10.0.0.1 (the USG)
ip route get 10.0.1.252 -> dev lanbr0 (same L2, no gateway)
10.0.1.252 and .253 are on the LoT VLAN, same broadcast domain as the
workstation, and both answer SSH. They are now the only addresses the runbook
gives, with the k8s ones struck through.
Also recorded: the switch cannot be run before unplugging the USG (two devices
on every gateway address; the guard refuses), so the order is forced. And
during the gap between unplugging and completing the switch there is no
inter-VLAN routing at all -- which means the JetKVMs (Management and kvm) and
Tailscale are NOT fallbacks in that window. LoT SSH is the only remote path;
below it is physical console. Added a step 0: open both SSH sessions and leave
them open before touching anything.
Both boxes are now installed and pass the pre-flight gate: mode unifi,
unifi.boot 231 lines including the reload action, delta at the right priority
(200/100), wan-secrets 0600, script executable, no config drift, VRRP still
MASTER/BACKUP. `vyos-unifi-switch vyos` refuses on both -- all six gateway
addresses detected answering ARP -- and neither box has gained dhcp-server, dns
or nat, so nothing about their behaviour has changed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-16 16:25:04 +01:00
|
|
|
## Use these addresses. Not the other ones.
|
|
|
|
|
|
|
|
|
|
| | use this | do NOT use |
|
|
|
|
|
|---|---|---|
|
|
|
|
|
| vyos001 (MASTER) | **`10.0.1.252`** | ~~192.168.8.143~~ |
|
|
|
|
|
| vyos002 (BACKUP) | **`10.0.1.253`** | ~~192.168.8.144~~ |
|
|
|
|
|
|
|
|
|
|
`ssh vyos@10.0.1.252` — by IP, not by name.
|
|
|
|
|
|
|
|
|
|
**The `192.168.8.x` addresses stop working the instant the USG is unplugged.**
|
|
|
|
|
That is not a maybe. Your workstation is on LoT (`10.0.0.210/23`) and reaching
|
|
|
|
|
`192.168.8.x` requires routing *through the USG*:
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
ip route get 192.168.8.143 -> via 10.0.0.1 <- the USG. Gone.
|
|
|
|
|
ip route get 10.0.1.252 -> dev lanbr0 <- same L2. Survives.
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
`10.0.1.252` and `.253` are on the LoT VLAN, the same broadcast domain as your
|
|
|
|
|
workstation, so they need no gateway at all. They are the only remote path that
|
|
|
|
|
survives the cutover.
|
|
|
|
|
|
|
|
|
|
**Between unplugging the USG and finishing the switch there is no inter-VLAN
|
|
|
|
|
routing.** In that window:
|
|
|
|
|
|
|
|
|
|
- the **JetKVMs are unreachable** from your workstation (they are on Management
|
|
|
|
|
and kvm) — they are *not* a fallback during the gap
|
|
|
|
|
- **Tailscale is down** with the internet
|
|
|
|
|
- your workstation keeps `10.0.0.210` (86400s lease) and can still resolve via
|
|
|
|
|
`10.0.0.194`, which is also link-scope
|
|
|
|
|
|
|
|
|
|
If LoT SSH fails, the next step is physical console, not the network.
|
|
|
|
|
|
feat(migration): reversible USG->VyOS switch, proven on the sim
The cutover is a switch, not a migration: unplug the USG, run one command, and
if anything is wrong run the other one and plug it back in. The operator will
have no internet during this and therefore no assistant, so the machinery has
to live on the boxes and the failure paths have to be proven in advance.
vyos-mode-delta.py generates the delta that turns the passive pair into the
gateway. Only one artifact is authored: gateway mode is always derived from
`load unifi.boot` + delta, so there is no inverse to maintain and no drift
between two hand-kept configs. It reuses unifi-to-vyos.py rather than
duplicating it, so what labsim proved and what production gets are one code
path. The PPPoE password is never written into the delta -- it carries a
placeholder the switch substitutes at apply time from /config/wan-secrets --
and generation fails if the real password appears in the output.
Two things the delta covers that the plan had underweighted:
- VyOS defaults to ACCEPT while the USG has an implicit WAN drop. Migrating
the port forwards alone would have left the router's own services and the
whole LAN reachable from the WAN. Added a stateful baseline scoped to the
WAN interface rather than a global default-action drop, so a mistake there
cannot lock anyone out over the LAN -- the only way back during a cutover.
- The old VIPs are NOT at network+254 on the /23 networks; they are
192.168.9.254, 10.0.9.254 and 10.0.1.254, in the upper half. A delete
naming a computed address fails quietly and leaves the group holding two
VIPs. The delta deletes the whole address node instead of guessing.
vyos-unifi-switch runs on the box from /config, which survives image upgrades,
so it works from a local terminal or the JetKVM with no workstation.
Proven on labsim, not assumed:
- unifi mode restores the previous config BYTE-EXACT (138 lines, diff clean).
- Auto-revert fires when the commit is not confirmed: 85 static-mappings ->
0, kea stopped, hostname restored, and uptime plus boot-id UNCHANGED, so
it reloaded rather than rebooted. That distinction is the whole reason
`commit-confirm action reload` is a prerequisite.
- Health-check failure triggers an immediate revert_soft rather than waiting
out the timer.
Four bugs found while doing it, each of which produced a wrong answer rather
than an error:
- commit-confirm is TWO steps. `config-mgmt commit_confirm` only arms the
revert timer; a normal `commit` still has to follow. Arming alone committed
nothing while reporting success.
- `sudo sg vyattacfg "config-mgmt ..."` loses the config-session environment,
so it reported "No configuration changes to commit" against a candidate
that plainly had 446 added lines.
- `... | grep -q` under `set -o pipefail` reports FAILURE on a match: grep
exits early, the producer takes SIGPIPE. Whether it triggers depends on
output size, so `status` misreported the mode intermittently.
- `show configuration commands` quotes values, so a fixed-string match for
`action reload` never matched `action 'reload'`.
CUTOVER.md is the printable runbook: both reachable addresses per box, the
escape hatch first, and the note that PPPoE is the one thing that could not be
tested beforehand.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-15 23:30:28 +01:00
|
|
|
| | |
|
|
|
|
|
|---|---|
|
fix(migration): the runbook pointed at addresses that die with the USG
The access table led with 192.168.8.143/.144 and offered the LoT addresses as a
fallback ("if unreachable, try"). That is backwards and would have stranded the
operator at the worst moment: the workstation sits on LoT, and reaching
192.168.8.x routes *through the USG*, so those addresses are guaranteed dead the
instant it is unplugged. Measured:
ip route get 192.168.8.143 -> via 10.0.0.1 (the USG)
ip route get 10.0.1.252 -> dev lanbr0 (same L2, no gateway)
10.0.1.252 and .253 are on the LoT VLAN, same broadcast domain as the
workstation, and both answer SSH. They are now the only addresses the runbook
gives, with the k8s ones struck through.
Also recorded: the switch cannot be run before unplugging the USG (two devices
on every gateway address; the guard refuses), so the order is forced. And
during the gap between unplugging and completing the switch there is no
inter-VLAN routing at all -- which means the JetKVMs (Management and kvm) and
Tailscale are NOT fallbacks in that window. LoT SSH is the only remote path;
below it is physical console. Added a step 0: open both SSH sessions and leave
them open before touching anything.
Both boxes are now installed and pass the pre-flight gate: mode unifi,
unifi.boot 231 lines including the reload action, delta at the right priority
(200/100), wan-secrets 0600, script executable, no config drift, VRRP still
MASTER/BACKUP. `vyos-unifi-switch vyos` refuses on both -- all six gateway
addresses detected answering ARP -- and neither box has gained dhcp-server, dns
or nat, so nothing about their behaviour has changed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-16 16:25:04 +01:00
|
|
|
| JetKVMs (after routing is restored) | `192.168.1.28`, `192.168.1.29`, `192.168.3.6` |
|
feat(migration): reversible USG->VyOS switch, proven on the sim
The cutover is a switch, not a migration: unplug the USG, run one command, and
if anything is wrong run the other one and plug it back in. The operator will
have no internet during this and therefore no assistant, so the machinery has
to live on the boxes and the failure paths have to be proven in advance.
vyos-mode-delta.py generates the delta that turns the passive pair into the
gateway. Only one artifact is authored: gateway mode is always derived from
`load unifi.boot` + delta, so there is no inverse to maintain and no drift
between two hand-kept configs. It reuses unifi-to-vyos.py rather than
duplicating it, so what labsim proved and what production gets are one code
path. The PPPoE password is never written into the delta -- it carries a
placeholder the switch substitutes at apply time from /config/wan-secrets --
and generation fails if the real password appears in the output.
Two things the delta covers that the plan had underweighted:
- VyOS defaults to ACCEPT while the USG has an implicit WAN drop. Migrating
the port forwards alone would have left the router's own services and the
whole LAN reachable from the WAN. Added a stateful baseline scoped to the
WAN interface rather than a global default-action drop, so a mistake there
cannot lock anyone out over the LAN -- the only way back during a cutover.
- The old VIPs are NOT at network+254 on the /23 networks; they are
192.168.9.254, 10.0.9.254 and 10.0.1.254, in the upper half. A delete
naming a computed address fails quietly and leaves the group holding two
VIPs. The delta deletes the whole address node instead of guessing.
vyos-unifi-switch runs on the box from /config, which survives image upgrades,
so it works from a local terminal or the JetKVM with no workstation.
Proven on labsim, not assumed:
- unifi mode restores the previous config BYTE-EXACT (138 lines, diff clean).
- Auto-revert fires when the commit is not confirmed: 85 static-mappings ->
0, kea stopped, hostname restored, and uptime plus boot-id UNCHANGED, so
it reloaded rather than rebooted. That distinction is the whole reason
`commit-confirm action reload` is a prerequisite.
- Health-check failure triggers an immediate revert_soft rather than waiting
out the timer.
Four bugs found while doing it, each of which produced a wrong answer rather
than an error:
- commit-confirm is TWO steps. `config-mgmt commit_confirm` only arms the
revert timer; a normal `commit` still has to follow. Arming alone committed
nothing while reporting success.
- `sudo sg vyattacfg "config-mgmt ..."` loses the config-session environment,
so it reported "No configuration changes to commit" against a candidate
that plainly had 446 added lines.
- `... | grep -q` under `set -o pipefail` reports FAILURE on a match: grep
exits early, the producer takes SIGPIPE. Whether it triggers depends on
output size, so `status` misreported the mode intermittently.
- `show configuration commands` quotes values, so a fixed-string match for
`action reload` never matched `action 'reload'`.
CUTOVER.md is the printable runbook: both reachable addresses per box, the
escape hatch first, and the note that PPPoE is the one thing that could not be
tested beforehand.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-15 23:30:28 +01:00
|
|
|
| Switch script | `/config/vyos-unifi-switch` on each box |
|
|
|
|
|
| Login | user `vyos` |
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## If something is wrong, do this
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
sudo /config/vyos-unifi-switch unifi
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Then reconnect the USG. That command runs no health checks, asks nothing and
|
|
|
|
|
cannot refuse. It restores a byte-exact copy of the configuration the box had
|
|
|
|
|
before the cutover — verified by diff, not by assumption.
|
|
|
|
|
|
|
|
|
|
**You do not have to be quick.** If you do nothing at all after
|
|
|
|
|
`vyos-unifi-switch vyos`, the box reverts by itself within 10 minutes. Verified:
|
|
|
|
|
config returns to the previous state and the box does **not** reboot
|
|
|
|
|
(`uptime` and boot-id unchanged across an auto-revert).
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
2026-08-16 00:08:13 +01:00
|
|
|
## What has actually been tested
|
|
|
|
|
|
|
|
|
|
Proven on the labsim router (same VyOS version, isolated OVS bridge with no
|
|
|
|
|
physical NIC), by loading **vyos001's real running config** and applying the
|
|
|
|
|
**real production delta**:
|
|
|
|
|
|
|
|
|
|
- All 317 commands accepted, and the whole delta **commits** (`COMMIT OK`).
|
|
|
|
|
- `unifi` mode restores the previous config **byte-exact** (diff clean).
|
|
|
|
|
- Auto-revert fires when the commit is not confirmed: config returns to the
|
|
|
|
|
saved state and the box does **not** reboot — `uptime` and boot-id unchanged
|
|
|
|
|
across the revert.
|
|
|
|
|
- Failed health checks trigger an immediate revert rather than waiting out the
|
|
|
|
|
timer.
|
|
|
|
|
|
|
|
|
|
Two bugs were found this way and would each have failed the entire switch,
|
|
|
|
|
since the delta commits as one unit: `bond0.51` did not exist for PPPoE to
|
|
|
|
|
reference, and `translation port` rejects a port list.
|
|
|
|
|
|
|
|
|
|
**Not tested, and untestable in advance:**
|
|
|
|
|
|
|
|
|
|
- **PPPoE.** The line permits one session and the USG holds it. The first real
|
|
|
|
|
attempt is during the cutover.
|
|
|
|
|
- **The commit on the real boxes.** The rehearsal ran with vyos001's `eth2` and
|
|
|
|
|
`eth3` stanzas stripped, because the sim VM has only two NICs. Those are
|
|
|
|
|
plain interface configs that already work on the real hardware, but they were
|
|
|
|
|
not part of what committed.
|
|
|
|
|
|
feat(migration): reversible USG->VyOS switch, proven on the sim
The cutover is a switch, not a migration: unplug the USG, run one command, and
if anything is wrong run the other one and plug it back in. The operator will
have no internet during this and therefore no assistant, so the machinery has
to live on the boxes and the failure paths have to be proven in advance.
vyos-mode-delta.py generates the delta that turns the passive pair into the
gateway. Only one artifact is authored: gateway mode is always derived from
`load unifi.boot` + delta, so there is no inverse to maintain and no drift
between two hand-kept configs. It reuses unifi-to-vyos.py rather than
duplicating it, so what labsim proved and what production gets are one code
path. The PPPoE password is never written into the delta -- it carries a
placeholder the switch substitutes at apply time from /config/wan-secrets --
and generation fails if the real password appears in the output.
Two things the delta covers that the plan had underweighted:
- VyOS defaults to ACCEPT while the USG has an implicit WAN drop. Migrating
the port forwards alone would have left the router's own services and the
whole LAN reachable from the WAN. Added a stateful baseline scoped to the
WAN interface rather than a global default-action drop, so a mistake there
cannot lock anyone out over the LAN -- the only way back during a cutover.
- The old VIPs are NOT at network+254 on the /23 networks; they are
192.168.9.254, 10.0.9.254 and 10.0.1.254, in the upper half. A delete
naming a computed address fails quietly and leaves the group holding two
VIPs. The delta deletes the whole address node instead of guessing.
vyos-unifi-switch runs on the box from /config, which survives image upgrades,
so it works from a local terminal or the JetKVM with no workstation.
Proven on labsim, not assumed:
- unifi mode restores the previous config BYTE-EXACT (138 lines, diff clean).
- Auto-revert fires when the commit is not confirmed: 85 static-mappings ->
0, kea stopped, hostname restored, and uptime plus boot-id UNCHANGED, so
it reloaded rather than rebooted. That distinction is the whole reason
`commit-confirm action reload` is a prerequisite.
- Health-check failure triggers an immediate revert_soft rather than waiting
out the timer.
Four bugs found while doing it, each of which produced a wrong answer rather
than an error:
- commit-confirm is TWO steps. `config-mgmt commit_confirm` only arms the
revert timer; a normal `commit` still has to follow. Arming alone committed
nothing while reporting success.
- `sudo sg vyattacfg "config-mgmt ..."` loses the config-session environment,
so it reported "No configuration changes to commit" against a candidate
that plainly had 446 added lines.
- `... | grep -q` under `set -o pipefail` reports FAILURE on a match: grep
exits early, the producer takes SIGPIPE. Whether it triggers depends on
output size, so `status` misreported the mode intermittently.
- `show configuration commands` quotes values, so a fixed-string match for
`action reload` never matched `action 'reload'`.
CUTOVER.md is the printable runbook: both reachable addresses per box, the
escape hatch first, and the note that PPPoE is the one thing that could not be
tested beforehand.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-15 23:30:28 +01:00
|
|
|
## Before you unplug anything
|
|
|
|
|
|
|
|
|
|
1. Tether your workstation to your phone if you want the assistant available.
|
|
|
|
|
Cutting the USG cuts your internet, not your LAN.
|
|
|
|
|
2. On **both** boxes, confirm the machinery is present:
|
|
|
|
|
```
|
|
|
|
|
sudo /config/vyos-unifi-switch status
|
|
|
|
|
ls -la /config/modes/ # unifi.boot + to-vyos.commands
|
|
|
|
|
ls -la /config/wan-secrets # must be 0600
|
|
|
|
|
```
|
|
|
|
|
`status` must report `mode: unifi`. If `unifi.boot` is missing, **stop** —
|
|
|
|
|
there is no way back without it.
|
|
|
|
|
3. Confirm the revert action is `reload`, not `reboot`:
|
|
|
|
|
```
|
|
|
|
|
show configuration commands | match commit-confirm
|
|
|
|
|
```
|
|
|
|
|
Must show `action 'reload'`. Without it a failed switch **reboots** the
|
|
|
|
|
firewall instead of reverting it. The switch script refuses to run if this
|
|
|
|
|
is missing, but check anyway.
|
|
|
|
|
|
|
|
|
|
## The cutover
|
|
|
|
|
|
fix(migration): the runbook pointed at addresses that die with the USG
The access table led with 192.168.8.143/.144 and offered the LoT addresses as a
fallback ("if unreachable, try"). That is backwards and would have stranded the
operator at the worst moment: the workstation sits on LoT, and reaching
192.168.8.x routes *through the USG*, so those addresses are guaranteed dead the
instant it is unplugged. Measured:
ip route get 192.168.8.143 -> via 10.0.0.1 (the USG)
ip route get 10.0.1.252 -> dev lanbr0 (same L2, no gateway)
10.0.1.252 and .253 are on the LoT VLAN, same broadcast domain as the
workstation, and both answer SSH. They are now the only addresses the runbook
gives, with the k8s ones struck through.
Also recorded: the switch cannot be run before unplugging the USG (two devices
on every gateway address; the guard refuses), so the order is forced. And
during the gap between unplugging and completing the switch there is no
inter-VLAN routing at all -- which means the JetKVMs (Management and kvm) and
Tailscale are NOT fallbacks in that window. LoT SSH is the only remote path;
below it is physical console. Added a step 0: open both SSH sessions and leave
them open before touching anything.
Both boxes are now installed and pass the pre-flight gate: mode unifi,
unifi.boot 231 lines including the reload action, delta at the right priority
(200/100), wan-secrets 0600, script executable, no config drift, VRRP still
MASTER/BACKUP. `vyos-unifi-switch vyos` refuses on both -- all six gateway
addresses detected answering ARP -- and neither box has gained dhcp-server, dns
or nat, so nothing about their behaviour has changed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-16 16:25:04 +01:00
|
|
|
0. **Open both SSH sessions BEFORE you unplug anything**, and leave them open:
|
|
|
|
|
```
|
|
|
|
|
ssh vyos@10.0.1.253 # vyos002, BACKUP
|
|
|
|
|
ssh vyos@10.0.1.252 # vyos001, MASTER
|
|
|
|
|
```
|
|
|
|
|
If either will not connect, stop. Do not unplug the USG.
|
|
|
|
|
|
feat(migration): reversible USG->VyOS switch, proven on the sim
The cutover is a switch, not a migration: unplug the USG, run one command, and
if anything is wrong run the other one and plug it back in. The operator will
have no internet during this and therefore no assistant, so the machinery has
to live on the boxes and the failure paths have to be proven in advance.
vyos-mode-delta.py generates the delta that turns the passive pair into the
gateway. Only one artifact is authored: gateway mode is always derived from
`load unifi.boot` + delta, so there is no inverse to maintain and no drift
between two hand-kept configs. It reuses unifi-to-vyos.py rather than
duplicating it, so what labsim proved and what production gets are one code
path. The PPPoE password is never written into the delta -- it carries a
placeholder the switch substitutes at apply time from /config/wan-secrets --
and generation fails if the real password appears in the output.
Two things the delta covers that the plan had underweighted:
- VyOS defaults to ACCEPT while the USG has an implicit WAN drop. Migrating
the port forwards alone would have left the router's own services and the
whole LAN reachable from the WAN. Added a stateful baseline scoped to the
WAN interface rather than a global default-action drop, so a mistake there
cannot lock anyone out over the LAN -- the only way back during a cutover.
- The old VIPs are NOT at network+254 on the /23 networks; they are
192.168.9.254, 10.0.9.254 and 10.0.1.254, in the upper half. A delete
naming a computed address fails quietly and leaves the group holding two
VIPs. The delta deletes the whole address node instead of guessing.
vyos-unifi-switch runs on the box from /config, which survives image upgrades,
so it works from a local terminal or the JetKVM with no workstation.
Proven on labsim, not assumed:
- unifi mode restores the previous config BYTE-EXACT (138 lines, diff clean).
- Auto-revert fires when the commit is not confirmed: 85 static-mappings ->
0, kea stopped, hostname restored, and uptime plus boot-id UNCHANGED, so
it reloaded rather than rebooted. That distinction is the whole reason
`commit-confirm action reload` is a prerequisite.
- Health-check failure triggers an immediate revert_soft rather than waiting
out the timer.
Four bugs found while doing it, each of which produced a wrong answer rather
than an error:
- commit-confirm is TWO steps. `config-mgmt commit_confirm` only arms the
revert timer; a normal `commit` still has to follow. Arming alone committed
nothing while reporting success.
- `sudo sg vyattacfg "config-mgmt ..."` loses the config-session environment,
so it reported "No configuration changes to commit" against a candidate
that plainly had 446 added lines.
- `... | grep -q` under `set -o pipefail` reports FAILURE on a match: grep
exits early, the producer takes SIGPIPE. Whether it triggers depends on
output size, so `status` misreported the mode intermittently.
- `show configuration commands` quotes values, so a fixed-string match for
`action reload` never matched `action 'reload'`.
CUTOVER.md is the printable runbook: both reachable addresses per box, the
escape hatch first, and the note that PPPoE is the one thing that could not be
tested beforehand.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-15 23:30:28 +01:00
|
|
|
1. **Physically disconnect the USG.** Not just powered off — disconnected. The
|
|
|
|
|
switch script refuses to run while anything still answers on a gateway
|
|
|
|
|
address, because two devices on `.1` is the worst available outcome.
|
fix(migration): the runbook pointed at addresses that die with the USG
The access table led with 192.168.8.143/.144 and offered the LoT addresses as a
fallback ("if unreachable, try"). That is backwards and would have stranded the
operator at the worst moment: the workstation sits on LoT, and reaching
192.168.8.x routes *through the USG*, so those addresses are guaranteed dead the
instant it is unplugged. Measured:
ip route get 192.168.8.143 -> via 10.0.0.1 (the USG)
ip route get 10.0.1.252 -> dev lanbr0 (same L2, no gateway)
10.0.1.252 and .253 are on the LoT VLAN, same broadcast domain as the
workstation, and both answer SSH. They are now the only addresses the runbook
gives, with the k8s ones struck through.
Also recorded: the switch cannot be run before unplugging the USG (two devices
on every gateway address; the guard refuses), so the order is forced. And
during the gap between unplugging and completing the switch there is no
inter-VLAN routing at all -- which means the JetKVMs (Management and kvm) and
Tailscale are NOT fallbacks in that window. LoT SSH is the only remote path;
below it is physical console. Added a step 0: open both SSH sessions and leave
them open before touching anything.
Both boxes are now installed and pass the pre-flight gate: mode unifi,
unifi.boot 231 lines including the reload action, delta at the right priority
(200/100), wan-secrets 0600, script executable, no config drift, VRRP still
MASTER/BACKUP. `vyos-unifi-switch vyos` refuses on both -- all six gateway
addresses detected answering ARP -- and neither box has gained dhcp-server, dns
or nat, so nothing about their behaviour has changed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-16 16:25:04 +01:00
|
|
|
You cannot switch first and unplug after, for exactly that reason.
|
|
|
|
|
|
|
|
|
|
2. In the **vyos002 (BACKUP)** session, first:
|
feat(migration): reversible USG->VyOS switch, proven on the sim
The cutover is a switch, not a migration: unplug the USG, run one command, and
if anything is wrong run the other one and plug it back in. The operator will
have no internet during this and therefore no assistant, so the machinery has
to live on the boxes and the failure paths have to be proven in advance.
vyos-mode-delta.py generates the delta that turns the passive pair into the
gateway. Only one artifact is authored: gateway mode is always derived from
`load unifi.boot` + delta, so there is no inverse to maintain and no drift
between two hand-kept configs. It reuses unifi-to-vyos.py rather than
duplicating it, so what labsim proved and what production gets are one code
path. The PPPoE password is never written into the delta -- it carries a
placeholder the switch substitutes at apply time from /config/wan-secrets --
and generation fails if the real password appears in the output.
Two things the delta covers that the plan had underweighted:
- VyOS defaults to ACCEPT while the USG has an implicit WAN drop. Migrating
the port forwards alone would have left the router's own services and the
whole LAN reachable from the WAN. Added a stateful baseline scoped to the
WAN interface rather than a global default-action drop, so a mistake there
cannot lock anyone out over the LAN -- the only way back during a cutover.
- The old VIPs are NOT at network+254 on the /23 networks; they are
192.168.9.254, 10.0.9.254 and 10.0.1.254, in the upper half. A delete
naming a computed address fails quietly and leaves the group holding two
VIPs. The delta deletes the whole address node instead of guessing.
vyos-unifi-switch runs on the box from /config, which survives image upgrades,
so it works from a local terminal or the JetKVM with no workstation.
Proven on labsim, not assumed:
- unifi mode restores the previous config BYTE-EXACT (138 lines, diff clean).
- Auto-revert fires when the commit is not confirmed: 85 static-mappings ->
0, kea stopped, hostname restored, and uptime plus boot-id UNCHANGED, so
it reloaded rather than rebooted. That distinction is the whole reason
`commit-confirm action reload` is a prerequisite.
- Health-check failure triggers an immediate revert_soft rather than waiting
out the timer.
Four bugs found while doing it, each of which produced a wrong answer rather
than an error:
- commit-confirm is TWO steps. `config-mgmt commit_confirm` only arms the
revert timer; a normal `commit` still has to follow. Arming alone committed
nothing while reporting success.
- `sudo sg vyattacfg "config-mgmt ..."` loses the config-session environment,
so it reported "No configuration changes to commit" against a candidate
that plainly had 446 added lines.
- `... | grep -q` under `set -o pipefail` reports FAILURE on a match: grep
exits early, the producer takes SIGPIPE. Whether it triggers depends on
output size, so `status` misreported the mode intermittently.
- `show configuration commands` quotes values, so a fixed-string match for
`action reload` never matched `action 'reload'`.
CUTOVER.md is the printable runbook: both reachable addresses per box, the
escape hatch first, and the note that PPPoE is the one thing that could not be
tested beforehand.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-15 23:30:28 +01:00
|
|
|
```
|
|
|
|
|
sudo /config/vyos-unifi-switch vyos
|
|
|
|
|
```
|
|
|
|
|
3. Watch the health checks. They cover PPPoE, the default route, kea, the DNS
|
|
|
|
|
forwarder and reachability. On failure the script reverts immediately and
|
|
|
|
|
tells you so.
|
|
|
|
|
4. If vyos002 came up clean, repeat on **vyos001 (MASTER)**.
|
|
|
|
|
5. Check a real client: does it get an address, and is it the *same* address as
|
|
|
|
|
before? Every active client has a reservation, so it should be.
|
|
|
|
|
|
|
|
|
|
## What will probably go wrong first
|
|
|
|
|
|
|
|
|
|
**PPPoE.** It is the one thing that could not be tested in advance — the line
|
|
|
|
|
permits a single session and the USG held it until you unplugged it. If the
|
|
|
|
|
WAN check fails:
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
show interfaces pppoe pppoe0
|
|
|
|
|
sudo journalctl -u ppp@pppoe0 -n 50 --no-pager
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Check the credential in `/config/wan-secrets` and that VLAN 51 actually reaches
|
|
|
|
|
the box. If it will not come up, run `vyos-unifi-switch unifi`, reconnect the
|
|
|
|
|
USG, and debug with the internet back on.
|
|
|
|
|
|
|
|
|
|
## Things that are true and easy to forget
|
|
|
|
|
|
|
|
|
|
- **WiFi keeps working, but through VyOS.** The SSIDs stay in UniFi and the APs
|
|
|
|
|
are untouched, but 37 of 83 active clients are wireless and every one is on
|
|
|
|
|
LoT — they get their addresses from VyOS now.
|
|
|
|
|
- **DHCP leases last 24h (86400s).** A device that does not renew promptly keeps
|
|
|
|
|
its old address for a while. That is fine, not a symptom.
|
2026-08-16 16:49:27 +01:00
|
|
|
- **The firewalls resolve via `8.8.8.8` / `8.8.4.4`** — matching the DNS the USG
|
|
|
|
|
used on its WAN. This means their own name resolution now depends on the
|
|
|
|
|
*internet* being up, so between unplugging the USG and PPPoE establishing,
|
|
|
|
|
the boxes have no DNS at all. That is expected and harmless: they only need
|
|
|
|
|
DNS for NTP hostnames, and the switch's own health checks use it precisely to
|
|
|
|
|
prove the WAN came up. Nothing in the switch itself resolves a name.
|
|
|
|
|
- Internal `ad.itaz.eu` names still resolve through Google, because that zone is
|
|
|
|
|
published publicly with private addresses in it (`nas001` → `10.0.0.194`,
|
|
|
|
|
`kvm-macstudio1` → `192.168.3.8`). Convenient here; worth knowing it is public.
|
feat(migration): reversible USG->VyOS switch, proven on the sim
The cutover is a switch, not a migration: unplug the USG, run one command, and
if anything is wrong run the other one and plug it back in. The operator will
have no internet during this and therefore no assistant, so the machinery has
to live on the boxes and the failure paths have to be proven in advance.
vyos-mode-delta.py generates the delta that turns the passive pair into the
gateway. Only one artifact is authored: gateway mode is always derived from
`load unifi.boot` + delta, so there is no inverse to maintain and no drift
between two hand-kept configs. It reuses unifi-to-vyos.py rather than
duplicating it, so what labsim proved and what production gets are one code
path. The PPPoE password is never written into the delta -- it carries a
placeholder the switch substitutes at apply time from /config/wan-secrets --
and generation fails if the real password appears in the output.
Two things the delta covers that the plan had underweighted:
- VyOS defaults to ACCEPT while the USG has an implicit WAN drop. Migrating
the port forwards alone would have left the router's own services and the
whole LAN reachable from the WAN. Added a stateful baseline scoped to the
WAN interface rather than a global default-action drop, so a mistake there
cannot lock anyone out over the LAN -- the only way back during a cutover.
- The old VIPs are NOT at network+254 on the /23 networks; they are
192.168.9.254, 10.0.9.254 and 10.0.1.254, in the upper half. A delete
naming a computed address fails quietly and leaves the group holding two
VIPs. The delta deletes the whole address node instead of guessing.
vyos-unifi-switch runs on the box from /config, which survives image upgrades,
so it works from a local terminal or the JetKVM with no workstation.
Proven on labsim, not assumed:
- unifi mode restores the previous config BYTE-EXACT (138 lines, diff clean).
- Auto-revert fires when the commit is not confirmed: 85 static-mappings ->
0, kea stopped, hostname restored, and uptime plus boot-id UNCHANGED, so
it reloaded rather than rebooted. That distinction is the whole reason
`commit-confirm action reload` is a prerequisite.
- Health-check failure triggers an immediate revert_soft rather than waiting
out the timer.
Four bugs found while doing it, each of which produced a wrong answer rather
than an error:
- commit-confirm is TWO steps. `config-mgmt commit_confirm` only arms the
revert timer; a normal `commit` still has to follow. Arming alone committed
nothing while reporting success.
- `sudo sg vyattacfg "config-mgmt ..."` loses the config-session environment,
so it reported "No configuration changes to commit" against a candidate
that plainly had 446 added lines.
- `... | grep -q` under `set -o pipefail` reports FAILURE on a match: grep
exits early, the producer takes SIGPIPE. Whether it triggers depends on
output size, so `status` misreported the mode intermittently.
- `show configuration commands` quotes values, so a fixed-string match for
`action reload` never matched `action 'reload'`.
CUTOVER.md is the printable runbook: both reachable addresses per box, the
escape hatch first, and the note that PPPoE is the one thing that could not be
tested beforehand.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-15 23:30:28 +01:00
|
|
|
- **The USG was a DNS resolver** for every VLAN except LoT. VyOS now runs
|
|
|
|
|
`dns forwarding` in its place. If names stop resolving but IPs still work,
|
|
|
|
|
that is where to look.
|
|
|
|
|
- **`eth2` and `bond0.2` are both in `192.168.8.0/23`.** It works, but if you
|
|
|
|
|
see odd source-address behaviour on the management NIC, that is why.
|
|
|
|
|
|
|
|
|
|
## Afterwards
|
|
|
|
|
|
|
|
|
|
Once it has been stable for a day:
|
|
|
|
|
|
|
|
|
|
- Re-run `migration/unifi-export.py` — the UniFi controller is no longer the
|
|
|
|
|
source of truth for DHCP, and the export will drift.
|
|
|
|
|
- The VPN rules (ESP, UDP 500/4500) are carried over but the VPN itself still
|
|
|
|
|
terminated on the USG. Decide whether it moves.
|
|
|
|
|
- `labsim` still holds a deliberate `kvm→k8s` drop rule from earlier testing.
|