Files
lab/migration/_unifi.py

85 lines
3.4 KiB
Python
Raw Permalink Normal View History

feat(migration): export UniFi config and generate VyOS DHCP+DNS from it Groundwork for replacing the USG with the VyOS pair without anything on the network noticing. Three pieces: migration/unifi-export.py pulls 13 endpoints off the classic controller into timestamped JSON plus a normalised inventory: 11 networks, 31 DHCP reservations, 4 port forwards, 2 firewall rules, 0 static routes. Two things this turned up that a naive export would have lost: - 23 of the 31 reservations carry no network_id at all -- UniFi simply does not store the binding -- so they are resolved by subnet containment instead. Without that, three quarters of the reservations have no subnet to be placed in. - 30 of the 31 sit INSIDE the DHCP pool, which UniFi's dhcpd tolerates and which is flagged as a warning rather than discovered at cutover. migration/unifi-to-vyos.py turns that inventory into VyOS `set` commands for DHCP and DNS only -- the services the USG owns that VyOS must reproduce. Not a general converter. --prod and --sim come from one code path so the config proven in the sim and the config applied to the firewalls cannot drift. Prod mode hard-fails if any reservation is missing, since a silent drop is the failure mode that matters. DNS is included because the USG resolves for 5 of 6 VLANs today: UniFi hands out the gateway's own address whenever dhcpd_dns is empty, verified by labmaster resolving against 192.168.8.1. Replacing the USG without a forwarder would take DNS away from those VLANs entirely. labsim/labsim-dhcp-test.sh proves it by booting throwaway VMs with real production MACs -- the one piece of production config that transplants verbatim. Safe because ovs-labsim has no physical NIC, so those MACs cannot reach the real LAN. Result on VyOS 2026.08 (kea), 4/4: printer1 got 172.31.10.46 from inside the pool, sonoff-matter got 172.31.11.67 across the /23 boundary, Hubitat got its out-of-pool .2, and an unreserved MAC got an unreserved address. kea honours in-pool host reservations -- the open question blocking the cutover. Supporting changes to labsim: - VLAN 10 widened to /23. Every reservation is in LoT and LoT spans 10.0.0.x and 10.0.1.x, which a /24 cannot represent. - LoT's host leg moved to .3, because 10.0.0.2 is a real reservation (Hubitat) that maps onto the host's own address. - vlans.conf gained optional masklen and host_octet fields, defaulting to 24 and 2 so the other five VLANs are untouched. - Fixed /etc/network/interfaces hardcoding 255.255.255.0. That file is what actually takes effect on these Alpine guests -- cloud-init's network-config is ignored -- so any non-/24 VLAN was silently wrong. Two generator bugs found by VyOS rejecting the output: static-mapping names are validated as hostnames, so underscores fail; and two devices named "espressif" plus two named "thebeast" collided into single names, which would have overwritten one reservation with another's address. The raw export holds WiFi passphrases and the WAN PPPoE credentials and is gitignored. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-15 01:33:00 +01:00
"""Shared UniFi API client for the migration tooling.
The controller is a CLASSIC self-hosted UniFi Network app (server_version
10.4.x), not UniFi OS: login is /api/login and data lives under
/api/s/<site>/... . UniFi OS would use /api/auth/login + /proxy/network/api.
Credentials come from the mcpctl server definition so they are not duplicated
here.
"""
from __future__ import annotations
import json, re, ssl, subprocess, urllib.request, http.cookiejar
def client():
raw = subprocess.run(["mcpctl", "describe", "server", "unifi-network"],
capture_output=True, text=True).stdout
m = re.search(r"UNIFI_TARGETS\s+(\[.*)", raw)
if not m:
raise SystemExit("could not read UNIFI_TARGETS from mcpctl")
blob = m.group(1).strip()
try:
targets = json.loads(blob)
except json.JSONDecodeError:
targets = json.loads(blob + "}" * (blob.count("{") - blob.count("}")))
t = targets[0]
base = t["base_url"].rstrip("/")
auth = t.get("auth", {})
site = t.get("default_site", "default")
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
opener = urllib.request.build_opener(
urllib.request.HTTPCookieProcessor(http.cookiejar.CookieJar()),
urllib.request.HTTPSHandler(context=ctx))
req = urllib.request.Request(
f"{base}/api/login",
data=json.dumps({"username": auth.get("username"),
"password": auth.get("password")}).encode(),
headers={"Content-Type": "application/json"})
opener.open(req, timeout=20).read()
return opener, base, site
def get(opener, base, site, path):
"""GET /api/s/<site>/<path>, returning the `data` list (never raising)."""
try:
body = opener.open(f"{base}/api/s/{site}/{path}", timeout=30).read()
return json.loads(body).get("data", [])
except Exception as exc:
return {"__error__": f"{type(exc).__name__}: {exc}"}
feat(migration): pin firewall management NICs and reserve them in UniFi Prerequisite for the cutover. eth2 on both firewalls was DHCP-served by the USG, and both addresses (192.168.8.143/.144) sit inside the pool VyOS will serve -- with no reservation for either MAC. After the switch they would renew from kea, get no mapping, and the management addresses could move. That is the worst possible moment for the address you SSH to to change, because losing the USG also means losing internet and any outside help. On both boxes, driven over the LoT path (bond0.10) so the interface being changed was never the one carrying the session: - eth2 pinned static at its current address, so it no longer depends on DHCP - `system name-server eth2` replaced with 10.0.0.194. That setting inherited resolvers from the DHCP lease, i.e. the boxes were resolving via the USG and would have lost DNS with it. 10.0.0.194 is reachable directly over bond0.10 and is authoritative for ad.itaz.eu, so internal names now resolve on the firewalls -- they did not before. - static default route via 192.168.8.1, replacing the one the lease provided. Superseded by PPPoE in vyos mode; this keeps unifi mode as it was. Verified after each: SSH on the pinned address, external and internal DNS, NTP still synced, VRRP unchanged (vyos001 MASTER, vyos002 BACKUP). migration/unifi-reserve.py adds the matching UniFi reservations so the controller cannot lease those addresses to anything else, keeping the management address identical in both modes. It reads the record back after writing, because a controller accepting a PUT is not proof it stored what was asked for, and it is idempotent. Also noted while doing this: VyOS `commit-confirm` REBOOTS the box if not confirmed -- "Minutes until reboot, unless 'confirm'" -- it does not roll the config back in place. For a gateway that means a real outage window, which changes how the switch script must use it. `config-mgmt commit_confirm -y` executes without the interactive prompt. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-15 22:12:42 +01:00
fix(migration): apply the reviewed reservation plan, not a recomputed one This caused a real outage. unifi-reserve-all.py recomputed its plan at --apply time by re-reading stat/sta, so a client that renewed between the dry run and the apply was pinned to whatever transient address it happened to hold at that instant. worker1-k8s0 was reviewed at 192.168.8.13 and written as 192.168.8.242. On its next reboot it could not get an address at all, taking a k8s node down. A plan that gets reviewed and a plan that gets applied must be the same object. The dry run now WRITES the plan to a file and --apply READS it and applies exactly that, reporting any client whose current address has since drifted rather than silently preferring the new value. 1 of 51 diverged; the rest were verified against the reviewed list and were correct. worker1 has been restored to .13 and confirmed: DHCPOFFER for its own MAC returns 192.168.8.13, and the node is up with a full lease and working internet. The second half of the outage was drift between controller and device: the USG was still running config from ~16h before these changes, so the controller looked perfectly correct while the gateway handed out something else. Writing the controller is only half the job, so the script now says so explicitly and gives the force-provision and DHCP-probe commands to verify with. `nmap --script broadcast-dhcp-discover --script-args broadcast-dhcp-discover.mac=...` is the way to prove a specific reservation is live without disturbing the client -- it elicits an OFFER without ever sending a REQUEST. _unifi.py gains post() for device commands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-16 14:31:30 +01:00
def post(opener, base, site, path, payload):
"""POST to /api/s/<site>/<path> -- used for device commands (cmd/devmgr)."""
req = urllib.request.Request(
f"{base}/api/s/{site}/{path}",
data=json.dumps(payload).encode(),
headers={"Content-Type": "application/json"}, method="POST")
try:
return json.loads(opener.open(req, timeout=30).read()).get("data", [])
except urllib.error.HTTPError as exc:
return {"__error__": f"HTTP {exc.code}: {exc.read()[:300].decode(errors='replace')}"}
except Exception as exc:
return {"__error__": f"{type(exc).__name__}: {exc}"}
feat(migration): pin firewall management NICs and reserve them in UniFi Prerequisite for the cutover. eth2 on both firewalls was DHCP-served by the USG, and both addresses (192.168.8.143/.144) sit inside the pool VyOS will serve -- with no reservation for either MAC. After the switch they would renew from kea, get no mapping, and the management addresses could move. That is the worst possible moment for the address you SSH to to change, because losing the USG also means losing internet and any outside help. On both boxes, driven over the LoT path (bond0.10) so the interface being changed was never the one carrying the session: - eth2 pinned static at its current address, so it no longer depends on DHCP - `system name-server eth2` replaced with 10.0.0.194. That setting inherited resolvers from the DHCP lease, i.e. the boxes were resolving via the USG and would have lost DNS with it. 10.0.0.194 is reachable directly over bond0.10 and is authoritative for ad.itaz.eu, so internal names now resolve on the firewalls -- they did not before. - static default route via 192.168.8.1, replacing the one the lease provided. Superseded by PPPoE in vyos mode; this keeps unifi mode as it was. Verified after each: SSH on the pinned address, external and internal DNS, NTP still synced, VRRP unchanged (vyos001 MASTER, vyos002 BACKUP). migration/unifi-reserve.py adds the matching UniFi reservations so the controller cannot lease those addresses to anything else, keeping the management address identical in both modes. It reads the record back after writing, because a controller accepting a PUT is not proof it stored what was asked for, and it is idempotent. Also noted while doing this: VyOS `commit-confirm` REBOOTS the box if not confirmed -- "Minutes until reboot, unless 'confirm'" -- it does not roll the config back in place. For a gateway that means a real outage window, which changes how the switch script must use it. `config-mgmt commit_confirm -y` executes without the interactive prompt. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-15 22:12:42 +01:00
def put(opener, base, site, path, payload):
"""PUT to /api/s/<site>/<path>. Returns the `data` list or an __error__ dict.
Classic controllers accept the session cookie alone -- no CSRF token, which
UniFi OS would require. Errors are returned rather than raised so a caller
changing production config can report and stop rather than traceback.
"""
req = urllib.request.Request(
f"{base}/api/s/{site}/{path}",
data=json.dumps(payload).encode(),
headers={"Content-Type": "application/json"}, method="PUT")
try:
return json.loads(opener.open(req, timeout=30).read()).get("data", [])
except urllib.error.HTTPError as exc:
return {"__error__": f"HTTP {exc.code}: {exc.read()[:300].decode(errors='replace')}"}
except Exception as exc:
return {"__error__": f"{type(exc).__name__}: {exc}"}