feat(migration): reversible USG->VyOS switch, proven on the sim
The cutover is a switch, not a migration: unplug the USG, run one command, and
if anything is wrong run the other one and plug it back in. The operator will
have no internet during this and therefore no assistant, so the machinery has
to live on the boxes and the failure paths have to be proven in advance.
vyos-mode-delta.py generates the delta that turns the passive pair into the
gateway. Only one artifact is authored: gateway mode is always derived from
`load unifi.boot` + delta, so there is no inverse to maintain and no drift
between two hand-kept configs. It reuses unifi-to-vyos.py rather than
duplicating it, so what labsim proved and what production gets are one code
path. The PPPoE password is never written into the delta -- it carries a
placeholder the switch substitutes at apply time from /config/wan-secrets --
and generation fails if the real password appears in the output.
Two things the delta covers that the plan had underweighted:
- VyOS defaults to ACCEPT while the USG has an implicit WAN drop. Migrating
the port forwards alone would have left the router's own services and the
whole LAN reachable from the WAN. Added a stateful baseline scoped to the
WAN interface rather than a global default-action drop, so a mistake there
cannot lock anyone out over the LAN -- the only way back during a cutover.
- The old VIPs are NOT at network+254 on the /23 networks; they are
192.168.9.254, 10.0.9.254 and 10.0.1.254, in the upper half. A delete
naming a computed address fails quietly and leaves the group holding two
VIPs. The delta deletes the whole address node instead of guessing.
vyos-unifi-switch runs on the box from /config, which survives image upgrades,
so it works from a local terminal or the JetKVM with no workstation.
Proven on labsim, not assumed:
- unifi mode restores the previous config BYTE-EXACT (138 lines, diff clean).
- Auto-revert fires when the commit is not confirmed: 85 static-mappings ->
0, kea stopped, hostname restored, and uptime plus boot-id UNCHANGED, so
it reloaded rather than rebooted. That distinction is the whole reason
`commit-confirm action reload` is a prerequisite.
- Health-check failure triggers an immediate revert_soft rather than waiting
out the timer.
Four bugs found while doing it, each of which produced a wrong answer rather
than an error:
- commit-confirm is TWO steps. `config-mgmt commit_confirm` only arms the
revert timer; a normal `commit` still has to follow. Arming alone committed
nothing while reporting success.
- `sudo sg vyattacfg "config-mgmt ..."` loses the config-session environment,
so it reported "No configuration changes to commit" against a candidate
that plainly had 446 added lines.
- `... | grep -q` under `set -o pipefail` reports FAILURE on a match: grep
exits early, the producer takes SIGPIPE. Whether it triggers depends on
output size, so `status` misreported the mode intermittently.
- `show configuration commands` quotes values, so a fixed-string match for
`action reload` never matched `action 'reload'`.
CUTOVER.md is the printable runbook: both reachable addresses per box, the
escape hatch first, and the note that PPPoE is the one thing that could not be
tested beforehand.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
This commit is contained in:
Michal
112
migration/CUTOVER.md
Normal file
112
migration/CUTOVER.md
Normal file
@@ -0,0 +1,112 @@
|
||||
# Cutover runbook — USG to VyOS
|
||||
|
||||
**Print this.** During the cutover there is no internet, so there is no
|
||||
assistant and no web search. Everything you need is on this page and on the
|
||||
boxes themselves.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| vyos001 (MASTER) | `192.168.8.143` — also `10.0.1.252` on LoT |
|
||||
| vyos002 (BACKUP) | `192.168.8.144` — also `10.0.1.253` on LoT |
|
||||
| JetKVMs | `192.168.1.28`, `192.168.1.29`, `192.168.3.6` |
|
||||
| Switch script | `/config/vyos-unifi-switch` on each box |
|
||||
| Login | user `vyos` |
|
||||
|
||||
Both boxes have **two** reachable addresses on different interfaces. If `eth2`
|
||||
(`192.168.8.x`) is unreachable, try the LoT address (`10.0.1.x`), and only then
|
||||
the JetKVM.
|
||||
|
||||
---
|
||||
|
||||
## If something is wrong, do this
|
||||
|
||||
```
|
||||
sudo /config/vyos-unifi-switch unifi
|
||||
```
|
||||
|
||||
Then reconnect the USG. That command runs no health checks, asks nothing and
|
||||
cannot refuse. It restores a byte-exact copy of the configuration the box had
|
||||
before the cutover — verified by diff, not by assumption.
|
||||
|
||||
**You do not have to be quick.** If you do nothing at all after
|
||||
`vyos-unifi-switch vyos`, the box reverts by itself within 10 minutes. Verified:
|
||||
config returns to the previous state and the box does **not** reboot
|
||||
(`uptime` and boot-id unchanged across an auto-revert).
|
||||
|
||||
---
|
||||
|
||||
## Before you unplug anything
|
||||
|
||||
1. Tether your workstation to your phone if you want the assistant available.
|
||||
Cutting the USG cuts your internet, not your LAN.
|
||||
2. On **both** boxes, confirm the machinery is present:
|
||||
```
|
||||
sudo /config/vyos-unifi-switch status
|
||||
ls -la /config/modes/ # unifi.boot + to-vyos.commands
|
||||
ls -la /config/wan-secrets # must be 0600
|
||||
```
|
||||
`status` must report `mode: unifi`. If `unifi.boot` is missing, **stop** —
|
||||
there is no way back without it.
|
||||
3. Confirm the revert action is `reload`, not `reboot`:
|
||||
```
|
||||
show configuration commands | match commit-confirm
|
||||
```
|
||||
Must show `action 'reload'`. Without it a failed switch **reboots** the
|
||||
firewall instead of reverting it. The switch script refuses to run if this
|
||||
is missing, but check anyway.
|
||||
|
||||
## The cutover
|
||||
|
||||
1. **Physically disconnect the USG.** Not just powered off — disconnected. The
|
||||
switch script refuses to run while anything still answers on a gateway
|
||||
address, because two devices on `.1` is the worst available outcome.
|
||||
2. On **vyos002 (BACKUP) first**:
|
||||
```
|
||||
sudo /config/vyos-unifi-switch vyos
|
||||
```
|
||||
3. Watch the health checks. They cover PPPoE, the default route, kea, the DNS
|
||||
forwarder and reachability. On failure the script reverts immediately and
|
||||
tells you so.
|
||||
4. If vyos002 came up clean, repeat on **vyos001 (MASTER)**.
|
||||
5. Check a real client: does it get an address, and is it the *same* address as
|
||||
before? Every active client has a reservation, so it should be.
|
||||
|
||||
## What will probably go wrong first
|
||||
|
||||
**PPPoE.** It is the one thing that could not be tested in advance — the line
|
||||
permits a single session and the USG held it until you unplugged it. If the
|
||||
WAN check fails:
|
||||
|
||||
```
|
||||
show interfaces pppoe pppoe0
|
||||
sudo journalctl -u ppp@pppoe0 -n 50 --no-pager
|
||||
```
|
||||
|
||||
Check the credential in `/config/wan-secrets` and that VLAN 51 actually reaches
|
||||
the box. If it will not come up, run `vyos-unifi-switch unifi`, reconnect the
|
||||
USG, and debug with the internet back on.
|
||||
|
||||
## Things that are true and easy to forget
|
||||
|
||||
- **WiFi keeps working, but through VyOS.** The SSIDs stay in UniFi and the APs
|
||||
are untouched, but 37 of 83 active clients are wireless and every one is on
|
||||
LoT — they get their addresses from VyOS now.
|
||||
- **DHCP leases last 24h (86400s).** A device that does not renew promptly keeps
|
||||
its old address for a while. That is fine, not a symptom.
|
||||
- **The firewalls resolve via `10.0.0.194`**, not the USG. That was changed
|
||||
ahead of time precisely so they keep DNS when the USG goes away.
|
||||
- **The USG was a DNS resolver** for every VLAN except LoT. VyOS now runs
|
||||
`dns forwarding` in its place. If names stop resolving but IPs still work,
|
||||
that is where to look.
|
||||
- **`eth2` and `bond0.2` are both in `192.168.8.0/23`.** It works, but if you
|
||||
see odd source-address behaviour on the management NIC, that is why.
|
||||
|
||||
## Afterwards
|
||||
|
||||
Once it has been stable for a day:
|
||||
|
||||
- Re-run `migration/unifi-export.py` — the UniFi controller is no longer the
|
||||
source of truth for DHCP, and the export will drift.
|
||||
- The VPN rules (ESP, UDP 500/4500) are carried over but the VPN itself still
|
||||
terminated on the USG. Decide whether it moves.
|
||||
- `labsim` still holds a deliberate `kvm→k8s` drop rule from earlier testing.
|
||||
267
migration/vyos-mode-delta.py
Executable file
267
migration/vyos-mode-delta.py
Executable file
@@ -0,0 +1,267 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Generate the delta that turns a passive VyOS pair into the gateway.
|
||||
|
||||
The switch works as: load the known-good `unifi.boot` snapshot, apply this
|
||||
delta, commit-confirm. Deriving the gateway mode from base+delta every time
|
||||
means there is no inverse to maintain and no drift between two hand-kept
|
||||
configs -- the revert is just loading the snapshot again.
|
||||
|
||||
./vyos-mode-delta.py --priority 200 -o to-vyos.commands # vyos001 (master)
|
||||
./vyos-mode-delta.py --priority 100 -o to-vyos.commands # vyos002 (backup)
|
||||
./vyos-mode-delta.py --emit-secrets /path/wan-secrets # credentials, 0600
|
||||
|
||||
The PPPoE password is NOT written into the delta. The delta carries the
|
||||
placeholder @@WAN_PASSWORD@@ and the switch script substitutes it at apply time
|
||||
from /config/wan-secrets, so the generated artifact can be read, diffed and
|
||||
copied around without carrying a credential.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import importlib.util
|
||||
import ipaddress
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
|
||||
# unifi-to-vyos.py has hyphens, so it cannot be imported by name. Reuse it
|
||||
# rather than duplicating the DHCP/DNS generation -- the whole point is that
|
||||
# what labsim proved and what production gets come from one code path.
|
||||
_spec = importlib.util.spec_from_file_location(
|
||||
"unifi_to_vyos", os.path.join(HERE, "unifi-to-vyos.py"))
|
||||
unifi_to_vyos = importlib.util.module_from_spec(_spec)
|
||||
_spec.loader.exec_module(unifi_to_vyos)
|
||||
|
||||
WAN_VIF = "bond0.51" # WAN arrives trunked as vlan-only network 51
|
||||
WAN_IF = "pppoe0"
|
||||
PLACEHOLDER = "@@WAN_PASSWORD@@"
|
||||
|
||||
|
||||
def vrrp_group(vlan: int) -> str:
|
||||
"""VRRP group names as configured on the boxes: 'native' for the untagged
|
||||
VLAN, 'vlan<id>' otherwise."""
|
||||
return "native" if vlan == 1 else f"vlan{vlan}"
|
||||
|
||||
|
||||
def build_delta(inv: dict, priority: int, wan_user: str) -> list[str]:
|
||||
out: list[str] = []
|
||||
nets = [n for n in inv["networks"] if n["dhcp_enabled"] and n["subnet"]]
|
||||
nets.sort(key=unifi_to_vyos.vlan_of)
|
||||
|
||||
out += [
|
||||
"# ==========================================================",
|
||||
"# Delta: passive VyOS pair -> gateway. Applied on top of a",
|
||||
"# freshly loaded unifi.boot, never on top of itself.",
|
||||
"# ==========================================================",
|
||||
"",
|
||||
"# An unconfirmed commit must reload the previous config, NOT reboot.",
|
||||
"# 'reboot' is the VyOS default and would turn a failed switch into a",
|
||||
"# real outage on the box that is meant to be carrying the network.",
|
||||
"set system config-management commit-confirm action reload",
|
||||
"",
|
||||
"# --- gateway addresses ------------------------------------",
|
||||
"# The VIP takes over the address the USG holds today, so no client",
|
||||
"# changes anything: no renewal needed, hardcoded gateways keep working.",
|
||||
]
|
||||
for n in nets:
|
||||
vlan = unifi_to_vyos.vlan_of(n)
|
||||
grp = vrrp_group(vlan)
|
||||
iface = ipaddress.ip_interface(n["subnet"])
|
||||
out.append(f"# {n['name']} (VLAN {vlan}) -> {iface.with_prefixlen}")
|
||||
# Delete the whole address node rather than a computed old value.
|
||||
# `address` is multi-value, and the current VIPs are NOT at
|
||||
# network+254 on the /23 networks -- they are 192.168.9.254,
|
||||
# 10.0.9.254 and 10.0.1.254, in the upper half. A delete naming the
|
||||
# wrong address fails quietly and leaves the group holding two VIPs.
|
||||
out.append(f"delete high-availability vrrp group {grp} address")
|
||||
out.append(f"set high-availability vrrp group {grp} address {iface.with_prefixlen}")
|
||||
out.append(f"set high-availability vrrp group {grp} priority {priority}")
|
||||
|
||||
out += [
|
||||
"",
|
||||
"# --- WAN ---------------------------------------------------",
|
||||
"# Vodafone line, PPPoE over the trunked WAN VLAN. This is the one",
|
||||
"# part that cannot be tested before cutover: the line permits a",
|
||||
"# single session and the USG holds it until it is unplugged.",
|
||||
f"set interfaces pppoe {WAN_IF} source-interface {WAN_VIF}",
|
||||
f"set interfaces pppoe {WAN_IF} authentication username '{wan_user}'",
|
||||
f"set interfaces pppoe {WAN_IF} authentication password '{PLACEHOLDER}'",
|
||||
f"set interfaces pppoe {WAN_IF} mtu 1492",
|
||||
# The peer's resolvers would otherwise overwrite /etc/resolv.conf and
|
||||
# cost the box its ability to resolve internal ad.itaz.eu names.
|
||||
f"set interfaces pppoe {WAN_IF} no-peer-dns",
|
||||
"",
|
||||
"# The static default route exists only for unifi mode, where the USG",
|
||||
"# is the next hop. PPPoE supplies the default here; leaving both at",
|
||||
"# the same distance would be ambiguous.",
|
||||
"delete protocols static route 0.0.0.0/0",
|
||||
"",
|
||||
"# --- NAT ---------------------------------------------------",
|
||||
f"set nat source rule 100 outbound-interface name {WAN_IF}",
|
||||
"set nat source rule 100 translation address masquerade",
|
||||
"set nat source rule 100 description 'LAN out to the internet'",
|
||||
]
|
||||
|
||||
# Port forwards, straight from UniFi.
|
||||
for i, p in enumerate(inv["port_forwards"]):
|
||||
if not p.get("enabled"):
|
||||
continue
|
||||
rule = 100 + i * 10
|
||||
proto = p["proto"] # tcp | udp | tcp_udp -- all valid VyOS values
|
||||
out += [
|
||||
"",
|
||||
f"set nat destination rule {rule} description '{p['name']}'",
|
||||
f"set nat destination rule {rule} inbound-interface name {WAN_IF}",
|
||||
f"set nat destination rule {rule} protocol {proto}",
|
||||
f"set nat destination rule {rule} destination port '{p['dst_port']}'",
|
||||
f"set nat destination rule {rule} translation address {p['fwd']}",
|
||||
f"set nat destination rule {rule} translation port '{p['fwd_port']}'",
|
||||
]
|
||||
|
||||
out += [
|
||||
"",
|
||||
"# --- firewall ----------------------------------------------",
|
||||
"# VyOS defaults to accepting everything. The USG has an implicit",
|
||||
"# WAN drop, so migrating the port forwards alone would leave the",
|
||||
"# router's own services and the whole LAN reachable from the WAN.",
|
||||
"#",
|
||||
"# Scoped to the WAN interface rather than a global default-action",
|
||||
"# drop: that way a mistake here cannot lock anyone out over the LAN,",
|
||||
"# which is the only path back in during a cutover.",
|
||||
"",
|
||||
"# Traffic TO the router.",
|
||||
"set firewall ipv4 input filter default-action accept",
|
||||
"set firewall ipv4 input filter rule 100 action accept",
|
||||
"set firewall ipv4 input filter rule 100 state established",
|
||||
"set firewall ipv4 input filter rule 100 state related",
|
||||
"set firewall ipv4 input filter rule 100 description 'established/related'",
|
||||
]
|
||||
# The two WAN_LOCAL accepts carried over from UniFi.
|
||||
out += [
|
||||
"",
|
||||
"set firewall ipv4 input filter rule 110 action accept",
|
||||
"set firewall ipv4 input filter rule 110 protocol esp",
|
||||
f"set firewall ipv4 input filter rule 110 inbound-interface name {WAN_IF}",
|
||||
"set firewall ipv4 input filter rule 110 description 'VPN accept ESP (from UniFi WAN_LOCAL)'",
|
||||
"",
|
||||
"set firewall ipv4 input filter rule 120 action accept",
|
||||
"set firewall ipv4 input filter rule 120 protocol udp",
|
||||
"set firewall ipv4 input filter rule 120 destination port '500,4500'",
|
||||
f"set firewall ipv4 input filter rule 120 inbound-interface name {WAN_IF}",
|
||||
"set firewall ipv4 input filter rule 120 description 'VPN accept UDP500/4500 (from UniFi WAN_LOCAL)'",
|
||||
"",
|
||||
"set firewall ipv4 input filter rule 130 action accept",
|
||||
"set firewall ipv4 input filter rule 130 protocol icmp",
|
||||
f"set firewall ipv4 input filter rule 130 inbound-interface name {WAN_IF}",
|
||||
"set firewall ipv4 input filter rule 130 description 'ICMP to the router (path MTU discovery)'",
|
||||
"",
|
||||
"# Everything else arriving from the WAN is dropped. LAN is untouched.",
|
||||
"set firewall ipv4 input filter rule 900 action drop",
|
||||
f"set firewall ipv4 input filter rule 900 inbound-interface name {WAN_IF}",
|
||||
"set firewall ipv4 input filter rule 900 description 'drop all other WAN-to-router'",
|
||||
"",
|
||||
"# Traffic THROUGH the router.",
|
||||
"set firewall ipv4 forward filter default-action accept",
|
||||
"set firewall ipv4 forward filter rule 100 action accept",
|
||||
"set firewall ipv4 forward filter rule 100 state established",
|
||||
"set firewall ipv4 forward filter rule 100 state related",
|
||||
"set firewall ipv4 forward filter rule 100 description 'established/related'",
|
||||
]
|
||||
|
||||
# Destination NAT happens before the forward filter, so these rules must
|
||||
# match the translated destination, not the WAN address.
|
||||
for i, p in enumerate(inv["port_forwards"]):
|
||||
if not p.get("enabled"):
|
||||
continue
|
||||
rule = 200 + i * 10
|
||||
out += [
|
||||
"",
|
||||
f"set firewall ipv4 forward filter rule {rule} action accept",
|
||||
f"set firewall ipv4 forward filter rule {rule} inbound-interface name {WAN_IF}",
|
||||
f"set firewall ipv4 forward filter rule {rule} protocol {p['proto']}",
|
||||
f"set firewall ipv4 forward filter rule {rule} destination address {p['fwd']}",
|
||||
f"set firewall ipv4 forward filter rule {rule} destination port '{p['fwd_port']}'",
|
||||
f"set firewall ipv4 forward filter rule {rule} description 'port forward: {p['name']}'",
|
||||
]
|
||||
|
||||
out += [
|
||||
"",
|
||||
"# New inbound connections from the WAN that are not a port forward.",
|
||||
"set firewall ipv4 forward filter rule 900 action drop",
|
||||
f"set firewall ipv4 forward filter rule 900 inbound-interface name {WAN_IF}",
|
||||
"set firewall ipv4 forward filter rule 900 description 'drop unsolicited WAN-to-LAN'",
|
||||
"",
|
||||
]
|
||||
|
||||
# DHCP + DNS, from the same generator labsim proved.
|
||||
dhcp_lines, stats = unifi_to_vyos.build(inv, "prod")
|
||||
expected = len(inv["reservations"])
|
||||
if stats["mappings"] != expected:
|
||||
raise SystemExit(
|
||||
f"refusing to generate: {expected - stats['mappings']} reservation(s) "
|
||||
f"missing -- every one must survive the cutover")
|
||||
out += dhcp_lines
|
||||
return out
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--priority", type=int, required=True,
|
||||
help="VRRP priority: 200 for the master, 100 for the backup")
|
||||
ap.add_argument("--inventory", default=os.path.join(HERE, "export", "inventory.json"))
|
||||
ap.add_argument("--raw-networkconf", default=os.path.join(HERE, "export", "rest_networkconf.json"))
|
||||
ap.add_argument("-o", "--out")
|
||||
ap.add_argument("--emit-secrets", metavar="PATH",
|
||||
help="write the PPPoE credential to PATH with mode 0600 and exit")
|
||||
args = ap.parse_args()
|
||||
|
||||
with open(args.inventory) as fh:
|
||||
inv = json.load(fh)
|
||||
with open(args.raw_networkconf) as fh:
|
||||
raw_nets = json.load(fh)
|
||||
|
||||
wan = next((n for n in raw_nets
|
||||
if n.get("purpose") == "wan" and n.get("wan_username")), None)
|
||||
if wan is None:
|
||||
print("no WAN network with credentials found in the export", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
if args.emit_secrets:
|
||||
fd = os.open(args.emit_secrets, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
|
||||
with os.fdopen(fd, "w") as fh:
|
||||
fh.write(f"WAN_PASSWORD='{wan.get('x_wan_password', '')}'\n")
|
||||
# Re-assert the mode in case the file already existed with a wider one.
|
||||
os.chmod(args.emit_secrets, 0o600)
|
||||
mode = oct(os.stat(args.emit_secrets).st_mode & 0o777)
|
||||
print(f"wrote {args.emit_secrets} (mode {mode}) for user {wan['wan_username']}",
|
||||
file=sys.stderr)
|
||||
return 0
|
||||
|
||||
lines = build_delta(inv, args.priority, wan["wan_username"])
|
||||
text = "\n".join(lines) + "\n"
|
||||
|
||||
if PLACEHOLDER not in text:
|
||||
print("BUG: password placeholder missing from the delta", file=sys.stderr)
|
||||
return 1
|
||||
if wan.get("x_wan_password") and wan["x_wan_password"] in text:
|
||||
print("BUG: the WAN password leaked into the delta", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
n_set = sum(1 for l in lines if l.startswith("set "))
|
||||
n_del = sum(1 for l in lines if l.startswith("delete "))
|
||||
print(f"delta: {n_set} set, {n_del} delete, priority {args.priority}, "
|
||||
f"{len(inv['reservations'])} reservations", file=sys.stderr)
|
||||
|
||||
if args.out:
|
||||
with open(args.out, "w") as fh:
|
||||
fh.write(text)
|
||||
print(f"wrote {args.out}", file=sys.stderr)
|
||||
else:
|
||||
sys.stdout.write(text)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
246
migration/vyos-unifi-switch
Executable file
246
migration/vyos-unifi-switch
Executable file
@@ -0,0 +1,246 @@
|
||||
#!/bin/bash
|
||||
# Switch this VyOS box between passive (USG is the gateway) and active
|
||||
# (VyOS is the gateway). Installed at /config/vyos-unifi-switch, which
|
||||
# survives image upgrades, so it can be run from a local terminal or the
|
||||
# JetKVM console with no workstation and no internet.
|
||||
#
|
||||
# vyos-unifi-switch report the current mode
|
||||
# vyos-unifi-switch vyos take over: VIPs to .1, DHCP, DNS, PPPoE, NAT
|
||||
# vyos-unifi-switch unifi revert; the USG can then be reconnected
|
||||
#
|
||||
# READ THIS BEFORE THE CUTOVER
|
||||
#
|
||||
# `unifi` is the escape hatch. It runs no health checks, asks no questions and
|
||||
# has nothing that can refuse. If anything at all looks wrong, run it, then
|
||||
# plug the USG back in.
|
||||
#
|
||||
# `vyos` commits with commit-confirm. If it is not confirmed -- because the
|
||||
# health checks failed, or because you lost access, or because you walked away
|
||||
# -- the box returns to the saved configuration on its own. That requires
|
||||
# `system config-management commit-confirm action reload`; without it VyOS
|
||||
# REBOOTS instead, which on a gateway is an outage rather than an undo. The
|
||||
# script refuses to run if that setting is missing.
|
||||
set -uo pipefail
|
||||
|
||||
MODES=/config/modes
|
||||
UNIFI_BOOT="$MODES/unifi.boot"
|
||||
DELTA="$MODES/to-vyos.commands"
|
||||
SECRETS=/config/wan-secrets
|
||||
MARKER="$MODES/current-mode"
|
||||
CONFIRM_MINUTES="${CONFIRM_MINUTES:-10}"
|
||||
OPRUN=/opt/vyatta/bin/vyatta-op-cmd-wrapper
|
||||
|
||||
say() { printf '[switch] %s\n' "$*"; }
|
||||
warn() { printf '[switch] WARNING: %s\n' "$*" >&2; }
|
||||
die() { printf '[switch] ERROR: %s\n' "$*" >&2; exit 1; }
|
||||
|
||||
# Run configuration commands in a real config session. Everything the caller
|
||||
# feeds in runs between `configure` and `exit`.
|
||||
run_cfg() {
|
||||
local script rc
|
||||
script="$(mktemp)"
|
||||
{
|
||||
echo 'source /opt/vyatta/etc/functions/script-template'
|
||||
echo 'configure'
|
||||
cat
|
||||
echo 'exit'
|
||||
} > "$script"
|
||||
vbash "$script"; rc=$?
|
||||
rm -f "$script"
|
||||
return $rc
|
||||
}
|
||||
|
||||
# Two traps live in this one function, both of which produced wrong answers
|
||||
# rather than errors:
|
||||
# 1. `show configuration commands` quotes values ("action 'reload'"), so the
|
||||
# quotes have to go before matching or every value-bearing check fails.
|
||||
# 2. `... | grep -q` under `set -o pipefail` reports FAILURE even on a match:
|
||||
# grep exits at the first hit, the producer takes SIGPIPE, and pipefail
|
||||
# surfaces that. Whether it triggers depends on output size, so it fails
|
||||
# intermittently. Match against a captured string instead of a pipeline.
|
||||
cfg_has() {
|
||||
local out
|
||||
out="$($OPRUN show configuration commands 2>/dev/null | tr -d "'")"
|
||||
case "$out" in *"$1"*) return 0 ;; *) return 1 ;; esac
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------- status ----
|
||||
current_mode() {
|
||||
# The marker records intent; the running config is the truth. Report the
|
||||
# config, and complain if the two disagree.
|
||||
local live="unknown"
|
||||
if cfg_has "set service dhcp-server"; then live="vyos"; else live="unifi"; fi
|
||||
echo "$live"
|
||||
}
|
||||
|
||||
show_status() {
|
||||
local live marked
|
||||
live="$(current_mode)"
|
||||
marked="$(cat "$MARKER" 2>/dev/null || echo "never set")"
|
||||
echo "host: $(hostname)"
|
||||
echo "mode: $live (marker: $marked)"
|
||||
[ "$live" != "$marked" ] && [ "$marked" != "never set" ] && \
|
||||
warn "marker disagrees with the running config -- trust the config"
|
||||
echo "VRRP:"
|
||||
$OPRUN show vrrp 2>/dev/null | sed -n '3,$p' | awk '{printf " %-9s %-12s %s\n", $1, $2, $4}'
|
||||
echo "DHCP server: $(systemctl is-active isc-kea-dhcp4-server 2>/dev/null)"
|
||||
echo "DNS forwarder: $(systemctl is-active pdns-recursor 2>/dev/null)"
|
||||
echo "WAN (pppoe0): $(ip -4 -br addr show pppoe0 2>/dev/null | awk '{print $2, $3}' || echo 'not present')"
|
||||
echo "default route: $(ip -4 route show default 2>/dev/null | head -1 || echo none)"
|
||||
echo "unsaved changes: $(cfg_unsaved)"
|
||||
}
|
||||
|
||||
cfg_unsaved() {
|
||||
if /usr/bin/config-mgmt compare >/dev/null 2>&1; then echo "no"; else echo "possibly - check 'compare saved'"; fi
|
||||
}
|
||||
|
||||
# ------------------------------------------------------------- preflight ----
|
||||
require_files() {
|
||||
[ -r "$UNIFI_BOOT" ] || die "missing $UNIFI_BOOT -- capture it while the USG is still the gateway"
|
||||
[ -s "$UNIFI_BOOT" ] || die "$UNIFI_BOOT is empty"
|
||||
}
|
||||
|
||||
require_reload_action() {
|
||||
cfg_has "set system config-management commit-confirm action reload" && return 0
|
||||
die "commit-confirm action is not 'reload'. Without it an unconfirmed switch
|
||||
REBOOTS this box instead of reverting. Fix first:
|
||||
configure
|
||||
set system config-management commit-confirm action reload
|
||||
commit; save"
|
||||
}
|
||||
|
||||
# The single worst outcome available is two devices answering on the gateway
|
||||
# address. It costs one ARP probe to make that impossible.
|
||||
# Returns 0 (success) when something IS answering on a gateway address we are
|
||||
# about to claim -- i.e. "yes, still alive, do not proceed".
|
||||
usg_still_alive() {
|
||||
local found=0 ip dev targets
|
||||
targets="$(grep -oE "vrrp group [a-z0-9]+ address [0-9.]+" "$DELTA" 2>/dev/null | awk '{print $NF}')"
|
||||
if [ -z "$targets" ]; then
|
||||
warn "this delta claims no VIPs, so there is nothing to probe."
|
||||
warn "That is expected in the lab and WRONG for the real cutover."
|
||||
return 1
|
||||
fi
|
||||
for ip in $targets; do
|
||||
dev="$(ip -4 route get "$ip" 2>/dev/null | sed -n 's/.* dev \([^ ]*\).*/\1/p' | head -1)"
|
||||
if [ -n "$dev" ] && command -v arping >/dev/null 2>&1; then
|
||||
# ARP is the right probe: it answers even when the host filters ICMP.
|
||||
if arping -c 2 -w 3 -f -I "$dev" "$ip" >/dev/null 2>&1; then
|
||||
warn "something already answers ARP on $ip (via $dev)"; found=1
|
||||
fi
|
||||
elif ping -c 2 -W 2 "$ip" >/dev/null 2>&1; then
|
||||
warn "something already answers ICMP on $ip"; found=1
|
||||
fi
|
||||
done
|
||||
[ "$found" -eq 1 ]
|
||||
}
|
||||
|
||||
# ------------------------------------------------------------- to unifi -----
|
||||
to_unifi() {
|
||||
require_files
|
||||
say "reverting to unifi mode (USG is the gateway)"
|
||||
run_cfg <<EOF || die "load/commit failed -- the box is unchanged, use the console"
|
||||
load $UNIFI_BOOT
|
||||
commit
|
||||
save
|
||||
EOF
|
||||
echo "unifi" > "$MARKER"
|
||||
say "done. The USG can be reconnected."
|
||||
say "If it was already connected during this, nothing was disturbed."
|
||||
}
|
||||
|
||||
# -------------------------------------------------------------- to vyos -----
|
||||
health_checks() {
|
||||
local fails=0
|
||||
_chk() { # name, command
|
||||
if eval "$2" >/dev/null 2>&1; then say " ok $1"; else say " FAIL $1"; fails=$((fails+1)); fi
|
||||
}
|
||||
_chk "kea (DHCP) is running" "systemctl is-active --quiet isc-kea-dhcp4-server"
|
||||
_chk "DNS forwarder is running" "systemctl is-active --quiet pdns-recursor"
|
||||
|
||||
# Only assert on the WAN if this delta actually brings one up. A delta with
|
||||
# no PPPoE stanza is a lab/partial delta, and failing it on a missing
|
||||
# pppoe0 would make the script untestable anywhere but the live cutover.
|
||||
# Announced loudly, because a quietly skipped check is worse than no check.
|
||||
if grep -q "^set interfaces pppoe" "$DELTA"; then
|
||||
# Written as [ -n "$(...)" ] rather than `... | grep -q` for the pipefail
|
||||
# reason above: a pipeline ending in grep -q cannot be trusted here.
|
||||
_chk "pppoe0 has an address" '[ -n "$(ip -4 -br addr show pppoe0 2>/dev/null | awk "{print \$3}")" ]'
|
||||
_chk "a default route exists" '[ -n "$(ip -4 route show default)" ]'
|
||||
_chk "internet reachable" "ping -c2 -W3 8.8.8.8"
|
||||
_chk "DNS resolves through us" "getent hosts vyos.net"
|
||||
else
|
||||
warn "this delta configures no WAN -- skipping all WAN health checks."
|
||||
warn "That is expected in the lab and WRONG for the real cutover."
|
||||
fi
|
||||
return $fails
|
||||
}
|
||||
|
||||
to_vyos() {
|
||||
require_files
|
||||
require_reload_action
|
||||
[ -r "$DELTA" ] || die "missing $DELTA"
|
||||
|
||||
if usg_still_alive; then
|
||||
die "refusing: something is still answering on a gateway address.
|
||||
Disconnect the USG first. Two devices on the same gateway IP is the
|
||||
one failure this script exists to prevent."
|
||||
fi
|
||||
|
||||
local pw="" tmp
|
||||
if [ -r "$SECRETS" ]; then
|
||||
# shellcheck disable=SC1090
|
||||
. "$SECRETS"; pw="${WAN_PASSWORD:-}"
|
||||
fi
|
||||
[ -n "$pw" ] || warn "no WAN_PASSWORD in $SECRETS -- PPPoE will not authenticate"
|
||||
|
||||
tmp="$(mktemp)"; chmod 600 "$tmp"
|
||||
sed "s|@@WAN_PASSWORD@@|${pw}|g" "$DELTA" | grep -vE '^\s*(#|$)' > "$tmp"
|
||||
|
||||
say "switching to vyos mode (this box becomes the gateway)"
|
||||
say "commit-confirm: ${CONFIRM_MINUTES} min to confirm, else it reverts itself"
|
||||
|
||||
# commit-confirm is TWO steps, and doing only the first commits nothing:
|
||||
# `config-mgmt commit_confirm` arms the revert timer, then a normal `commit`
|
||||
# applies the candidate config. The interactive prompt lives in the first
|
||||
# step, which is why it is invoked directly with -y instead of via the
|
||||
# `commit-confirm` alias. IN_COMMIT_CONFIRM is what the real CLI sets, and
|
||||
# the commit hooks look at it.
|
||||
if ! run_cfg < <(printf 'load %s\n' "$UNIFI_BOOT"; cat "$tmp";
|
||||
printf 'sudo sg vyattacfg "/usr/bin/config-mgmt commit_confirm -y -t=%s"\n' "$CONFIRM_MINUTES";
|
||||
printf 'export IN_COMMIT_CONFIRM=t\ncommit\nunset IN_COMMIT_CONFIRM\n'); then
|
||||
rm -f "$tmp"
|
||||
die "commit-confirm failed. Nothing was applied; the box is still in its
|
||||
previous mode. Run '$0 unifi' if you are unsure."
|
||||
fi
|
||||
rm -f "$tmp"
|
||||
|
||||
say "committed. Waiting 25s for PPPoE and services to settle..."
|
||||
sleep 25
|
||||
|
||||
say "health checks:"
|
||||
if health_checks; then
|
||||
say "all checks passed -- confirming"
|
||||
/usr/bin/config-mgmt confirm >/dev/null 2>&1 || die "confirm failed; it will revert on its own shortly"
|
||||
run_cfg <<'EOF' || warn "save failed -- config is live but will not survive a reboot"
|
||||
save
|
||||
EOF
|
||||
echo "vyos" > "$MARKER"
|
||||
say "vyos mode is live and saved."
|
||||
else
|
||||
warn "health checks FAILED -- reverting now rather than waiting out the timer"
|
||||
/usr/bin/config-mgmt revert_soft >/dev/null 2>&1 \
|
||||
|| warn "revert_soft failed; the commit-confirm timer will still fire within ${CONFIRM_MINUTES} min"
|
||||
echo "unifi" > "$MARKER"
|
||||
die "reverted to the previous configuration. Reconnect the USG.
|
||||
Check: ip addr show pppoe0; journalctl -u pppd; $0 status"
|
||||
fi
|
||||
}
|
||||
|
||||
# ----------------------------------------------------------------- main -----
|
||||
case "${1:-status}" in
|
||||
vyos) to_vyos ;;
|
||||
unifi) to_unifi ;;
|
||||
status) show_status ;;
|
||||
*) echo "usage: $(basename "$0") [vyos|unifi|status]" >&2; exit 2 ;;
|
||||
esac
|
||||
Reference in New Issue
Block a user