michal/lab
1
0
Fork 0
Code Issues Pull Requests 3 Actions Packages Projects Releases Wiki Activity

fix(migration): apply the reviewed reservation plan, not a recomputed one

Browse Source 
This caused a real outage. unifi-reserve-all.py recomputed its plan at --apply
time by re-reading stat/sta, so a client that renewed between the dry run and
the apply was pinned to whatever transient address it happened to hold at that
instant. worker1-k8s0 was reviewed at 192.168.8.13 and written as
192.168.8.242. On its next reboot it could not get an address at all, taking a
k8s node down.

A plan that gets reviewed and a plan that gets applied must be the same object.
The dry run now WRITES the plan to a file and --apply READS it and applies
exactly that, reporting any client whose current address has since drifted
rather than silently preferring the new value.

1 of 51 diverged; the rest were verified against the reviewed list and were
correct. worker1 has been restored to .13 and confirmed: DHCPOFFER for its own
MAC returns 192.168.8.13, and the node is up with a full lease and working
internet.

The second half of the outage was drift between controller and device: the USG
was still running config from ~16h before these changes, so the controller
looked perfectly correct while the gateway handed out something else. Writing
the controller is only half the job, so the script now says so explicitly and
gives the force-provision and DHCP-probe commands to verify with. `nmap
--script broadcast-dhcp-discover --script-args broadcast-dhcp-discover.mac=...`
is the way to prove a specific reservation is live without disturbing the
client -- it elicits an OFFER without ever sending a REQUEST.

_unifi.py gains post() for device commands.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
This commit is contained in:
Michal
2026-08-16 14:31:30 +01:00
parent b37cd79432
commit fc31013ceb
2 changed files with 65 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
migration/_unifi.py
View File

@@ -51,6 +51,20 @@ def get(opener, base, site, path):
return {"__error__": f"{type(exc).__name__}: {exc}"}
def post(opener, base, site, path, payload):
"""POST to /api/s/<site>/<path> -- used for device commands (cmd/devmgr)."""
req = urllib.request.Request(
f"{base}/api/s/{site}/{path}",
data=json.dumps(payload).encode(),
headers={"Content-Type": "application/json"}, method="POST")
try:
return json.loads(opener.open(req, timeout=30).read()).get("data", [])
except urllib.error.HTTPError as exc:
return {"__error__": f"HTTP {exc.code}: {exc.read()[:300].decode(errors='replace')}"}
except Exception as exc:
return {"__error__": f"{type(exc).__name__}: {exc}"}
def put(opener, base, site, path, payload):
"""PUT to /api/s/<site>/<path>. Returns the `data` list or an __error__ dict.

52
migration/unifi-reserve-all.py
View File

@@ -63,6 +63,9 @@ def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--apply", action="store_true", help="actually write (default: dry run)")
ap.add_argument("--skip-random", action="store_true", help="omit randomised MACs")
ap.add_argument("--plan", default="reservation-plan.tsv",
help="dry run WRITES this file; --apply READS it and applies "
"exactly what it contains")
args = ap.parse_args()
opener, base, site = _unifi.client()
@@ -158,9 +161,42 @@ def main() -> int:
f"stops matching once the device re-randomises.")
if not args.apply:
print(f"\ndry run -- nothing written. Re-run with --apply to create {len(plan)}.")
with open(args.plan, "w") as fh:
for ip, mac, label, _u, _n in plan:
fh.write(f"{mac}\t{ip}\t{label}\n")
print(f"\ndry run -- nothing written to the controller.")
print(f"plan saved to {args.plan}; re-run with --apply to apply exactly that.")
return 0
# Apply the plan that was REVIEWED, not one recomputed now.
#
# This cost a k8s node an outage. The apply used to re-read stat/sta, and a
# client that renewed between the dry run and the apply got pinned to
# whatever transient address it happened to hold at that instant -- worker1
# was reviewed at .13 and written as .242. A plan you looked at and a plan
# that gets applied must be the same object.
try:
with open(args.plan) as fh:
reviewed = {}
for line in fh:
parts = line.rstrip("\n").split("\t")
if len(parts) >= 2:
reviewed[parts[0].lower()] = parts[1]
except OSError:
print(f"no plan at {args.plan}. Run without --apply first and review it.",
file=sys.stderr)
return 1
drifted = [(ip, mac) for ip, mac, _l, _u, _n in plan
if mac in reviewed and reviewed[mac] != ip]
for ip, mac in drifted:
print(f" note: {mac} now reports {ip}, plan says {reviewed[mac]} -- "
f"applying the plan", file=sys.stderr)
plan = [(reviewed[mac], mac, label, user, net)
for ip, mac, label, user, net in plan if mac in reviewed]
print(f"applying {len(plan)} reservation(s) from {args.plan}")
print()
ok = fail = 0
for ip, mac, label, user, net in plan:
@@ -179,6 +215,20 @@ def main() -> int:
print(f"\nwrote {ok}, failed {fail}, verified live {verified}/{len(plan)}")
print(f"total reservations on the controller now: "
f"{sum(1 for u in after if u.get('use_fixedip'))}")
# Writing the controller is only half the job. The gateway applies config
# on its own schedule, and a device running config from before these
# changes will hand out addresses that disagree with what the controller
# shows -- which is how a k8s node ended up unable to get any lease at all
# while the controller looked perfectly correct.
print("\nThe controller now disagrees with what the gateway is running.")
print("Push it to the device and wait for state to return to 'connected':")
print(" python3 -c \"import _unifi; o,b,s=_unifi.client(); "
"print(_unifi.post(o,b,s,'cmd/devmgr',"
"{'cmd':'force-provision','mac':'<gateway-mac>'}))\"")
print("Then verify a real DISCOVER is answered before trusting it:")
print(" ssh vyos@<fw> 'sudo nmap --script broadcast-dhcp-discover -e eth2 "
"--script-args broadcast-dhcp-discover.mac=<client-mac>'")
return 0 if fail == 0 and verified == len(plan) else 1