diff --git a/labsim/labsim-vlan-leak-test.sh b/labsim/labsim-vlan-leak-test.sh index 6d6abd8..4b25d27 100755 --- a/labsim/labsim-vlan-leak-test.sh +++ b/labsim/labsim-vlan-leak-test.sh @@ -49,6 +49,19 @@ die() { printf '\033[31merror:\033[0m %s\n' "$*" >&2; exit 1; } command -v sshpass >/dev/null || die "sshpass required" +# A silent router is the one verdict worth double-checking before reporting. +# +# Kea can be `is-active` and answering nothing -- it reopens sockets on a retry +# loop, and some configurations (`listen-interface`, notably) leave individual +# VLANs dead while the rest work. Both look identical to a one-shot test: "no +# reply at all". Two opposite and equally wrong conclusions about +# `listen-interface` came out of believing a single negative run, in both +# directions, before a retry made the real pattern obvious. +# +# Kea's fallback UDP socket appearing is NOT a readiness signal -- it is bound +# well before the server actually answers. Checked, and it does not work. +RETRIED="${RETRIED:-0}" + router() { timeout 40 sshpass -p "$ROUTER_PW" ssh -o StrictHostKeyChecking=no \ -o ConnectTimeout=8 "vyos@$ROUTER_IP" "$@" 2>/dev/null @@ -71,16 +84,23 @@ PARENT="bond0" VIF="bond0.${VLAN}" log "router $ROUTER_IP -- capturing on $PARENT and $VIF" +# Kill EVERY tcpdump first, not just ones matching this run's pattern, and count +# only afterwards. Counting `pgrep -f 'tcpdump -i bond0'` while a stray tcpdump +# from an earlier session was still running satisfied the >=2 guard with zero of +# THIS run's captures alive -- and a capture that records nothing reports +# "the router sent no reply at all", which reads as a DHCP outage. That sent me +# chasing a fault in the router that was entirely in the test harness. started="$(router_sh </dev/null 2>&1 +sudo pkill -x tcpdump >/dev/null 2>&1 +sleep 1 sudo rm -f /tmp/leak-*.txt sudo nohup tcpdump -i $PARENT -e -nn -l 'udp port 67 or udp port 68' > /tmp/leak-parent.txt 2>/dev/null & sudo nohup tcpdump -i $VIF -e -nn -l 'udp port 67 or udp port 68' > /tmp/leak-vif.txt 2>/dev/null & sleep 3 -pgrep -c -f 'tcpdump -i bond0' +pgrep -c -x tcpdump EOF )" -[ "${started:-0}" -ge 2 ] || die "capture did not start on the router (got ${started:-0} of 2)" +[ "${started:-0}" -eq 2 ] || die "capture did not start on the router (got ${started:-0}, expected exactly 2)" # -s /bin/true: ask, observe the answer, apply nothing. The client's existing # static address is left alone, so this is safe to run against a live sim VM. @@ -89,7 +109,7 @@ client_out="$(client "udhcpc -n -q -f -i eth0 -s /bin/true -t 3 -T 3 2>&1")" [ -n "$client_out" ] || die "no response from client $CLIENT" sleep 2 -router "sudo pkill -f 'tcpdump -i bond0'" >/dev/null +router "sudo pkill -x tcpdump" >/dev/null parent="$(router 'sudo cat /tmp/leak-parent.txt')" vif="$(router 'sudo cat /tmp/leak-vif.txt')" @@ -125,7 +145,11 @@ want_prefix="172.31.${VLAN}." echo "=== verdict ===" if [ -z "$replies" ]; then - echo "INCONCLUSIVE: the router sent no reply at all -- is DHCP running?" + if [ "$RETRIED" -eq 0 ]; then + log "no reply -- retrying once in 20s before calling DHCP down" + sleep 20; RETRIED=1 exec "$0" --vlan "$VLAN" --client "$CLIENT" ${SAVE:+--save "$SAVE"} + fi + echo "INCONCLUSIVE: the router sent no reply at all, twice -- DHCP is down on VLAN $VLAN" exit 2 fi diff --git a/migration/RECOVERY-CARD-vlan1-move.md b/migration/RECOVERY-CARD-vlan1-move.md new file mode 100644 index 0000000..63bb39b --- /dev/null +++ b/migration/RECOVERY-CARD-vlan1-move.md @@ -0,0 +1,194 @@ +# Recovery card — moving Management to tagged VLAN 1 + +Print or keep open. **During this change there is no internet, so no Claude.** +Everything you need is on this page. + +--- + +## The one thing that matters + +``` +ssh vyos@10.0.1.252 +``` + +Your workstation is `10.0.0.210/23`; vyos001's LoT leg is `10.0.1.252/23`. Same +subnet, same VLAN, **direct L2** — verified: `ip route get` returns +`dev lanbr0 src 10.0.0.210` with no `via`, MAC `64:62:66:25:96:45`. + +It therefore does **not** depend on: the Management VLAN, VRRP, the VIPs, +inter-VLAN routing, DNS, or the switch trunk config. If the router is up and its +bond has link, this works. `bond0.10` is untouched by the change and stays in the +firewall `LAN` group throughout. + +vyos002, once it is up, is `10.0.1.253` the same way. + +Other legs that also survive: `192.168.3.4` (kvm), `192.168.2.252` (Roomates). + +--- + +## Before you touch anything + +``` +ssh vyos@10.0.1.252 +sudo /config/vyos-known-good save +``` + +The existing snapshot is from **2026-08-24** and predates today's fixes +(eth2 removal, VRRP health-check) — restoring that one would undo them. Take a +fresh one first. Check with `sudo /config/vyos-known-good status`. + +--- + +## Order: switch FIRST, router SECOND + +This matters and is easy to get backwards. + +The UniFi controller is `192.168.1.5`, on the **Management** VLAN. Your +workstation is on LoT and reaches it *through vyos001*. The moment the router +has Management on `bond0.1` while the switch is still sending it untagged, that +routing is dead — **and you lose the controller**, which is the thing you still +need in order to change the switch. + +So: + +1. **UniFi first**, while everything still works: + USW Aggregation → port 1 `firewall001` (LAG, members 1+2) → + Native VLAN: Management → **None**, and make sure VLAN 1 is tagged/allowed. + *vyos001 loses Management the instant this lands. That is expected.* + Do **not** touch port 3 `firewall002` — that is vyos002, and it is down. +2. **Router second**, over `ssh vyos@10.0.1.252` (still works — L2 direct). + +If UniFi will not offer "no native VLAN", stop and read *"If UniFi cannot do it"* +below rather than improvising. + +--- + +## The router change + +``` +ssh vyos@10.0.1.252 +configure +set interfaces bonding bond0 vif 1 address '192.168.1.252/24' +set interfaces bonding bond0 vif 1 description 'management' +delete interfaces bonding bond0 address +set firewall group interface-group LAN interface 'bond0.1' +delete firewall group interface-group LAN interface 'bond0' +set high-availability vrrp group native interface 'bond0.1' +commit-confirm 10 +save +exit +``` + +**Use `commit-confirm 10`, not `commit`.** If it goes wrong and you cannot get +back in, the router reverts itself after 10 minutes and comes back on its own. +That is your safety net with no internet and no help. + +Once you have confirmed it works (below), run: + +``` +configure +confirm +save +exit +``` + +`save` after `confirm`, or a reboot loses it. + +### Then, and this is the step that gets forgotten + +``` +sudo systemctl restart isc-kea-dhcp4-server +``` + +VyOS does **not** restart kea for an interface address change. Without this it +keeps a raw socket bound to the old address and keeps handing out wrong-VLAN +addresses — the fix looks like it did nothing. Give it ~60s before judging; +kea reopens sockets on a retry loop and answers nothing for a while after a +restart (measured: still silent at 55s in the sim, then fine). + +Also check DNS came back, since the forwarder binds the VIP `192.168.1.1`: + +``` +sudo systemctl status pdns-recursor --no-pager | head -3 +dig @192.168.1.1 google.com +short +``` + +--- + +## Verify + +``` +ssh vyos@192.168.1.252 # Management back, now tagged +show vrrp # native should be on bond0.1 +show dhcp server leases | head +``` + +Then from a machine on VLAN 3, force a DHCP renew and confirm it gets a +`192.168.3.x` address and not a `192.168.1.x` one. + +--- + +## If you are locked out + +In order: + +1. **Wait 10 minutes.** `commit-confirm` reverts by itself. This is the answer + most of the time. Do not power-cycle during this — you will lose the revert. +2. `ssh vyos@10.0.1.252` — the LoT leg. Then `configure` / `rollback 1` / `commit`. +3. Other legs: `ssh vyos@192.168.3.4`, `ssh vyos@192.168.2.252`. +4. `sudo /config/vyos-known-good restore` — back to the snapshot you took at the + start. It is itself commit-confirmed, so even this cannot strand you. +5. Put the UniFi port back: Native VLAN → Management on USW Aggregation port 1. + That alone restores the old shape and Management comes back untagged. + +**Do not** power-cycle vyos001 as a first move. Everything above is faster and +non-destructive, and a reboot loses an unsaved `commit-confirm` revert. + +--- + +## Do NOT power on vyos002 yet + +It still has `interfaces ethernet eth2 address 192.168.8.144/23` on the box — the +same subnet as `bond0.2`. That is what ARP-poisoned `192.168.8.1` and took the +cluster down. It also has no `/config/vrrp-wan-health`, so it can take the +floating IPs with no WAN. + +Its console (`kvm - vyos002`, US24 port 9) is currently **unreachable** — it sits +on a VLAN 3 port holding a Management lease `192.168.1.28`, which is the very bug +being fixed here. Fixing DHCP first is what gets that console back. + +--- + +## If UniFi cannot do it + +Classic UniFi (this is a classic controller, 10.4.57) may not offer +"Native VLAN = None" — every switch port has a PVID. Two things make this awkward +here: Management is UniFi's *default* network with **no VLAN ID at all** +(`vlan: null`), so there may be nothing to "tag VLAN 1" with. + +If so, **stop and change nothing.** The workaround is to point the trunk's native +VLAN at a VLAN the router does not serve (so `bond0` still ends up with no +subnet), which needs a throwaway VLAN-only network created first. That is a +design decision, not something to improvise at 1am with no internet. Put the port +back to Native = Management and everything returns to today's working state. + +--- + +## Facts worth having on paper + +| | | +|---|---| +| vyos001 Management | `192.168.1.252` → becomes `bond0.1` | +| vyos001 LoT (recovery) | `10.0.1.252`, L2-direct from your workstation | +| vyos002 Management | `192.168.1.253` (down) | +| VIP Management | `192.168.1.1` | +| UniFi controller | `192.168.1.5` (on Management — you lose it mid-change) | +| firewall001 trunk | USW Aggregation port 1, LAG members 1+2 | +| firewall002 trunk | USW Aggregation port 3, LAG members 3+4 | +| SSH user / pass | `vyos` / `vyos` | +| vyos001 bond MAC | `64:62:66:25:96:45` | + +Measured in labsim: converting the router while the peer is already converted +costs **0s** of VIP downtime; converting it while it holds the VIPs costs about +**6s**. vyos002 is down, so vyos001 holds everything — expect the ~6s, and expect +Management to be gone from the UniFi change until the router change lands.