diff --git a/labsim/README.md b/labsim/README.md index 292954c..ecd6e40 100644 --- a/labsim/README.md +++ b/labsim/README.md @@ -186,6 +186,63 @@ why any of it was shaped the way it was. client does not redial promptly. After any change there, check `pppoe0` on the router and `sudo systemctl restart ppp@pppoe0` if it is missing. +## The trunk carries every VLAN tagged, including Management + +There is deliberately **no native/untagged VLAN** on the trunks to the routers, +and Management lives on `bond0.1`, not on the bare `bond0`. + +A native VLAN is what puts a subnet on the bond **parent** while every other +VLAN sits on a sub-interface of it. With `dhcp-socket-type: raw`, kea receives +each tagged frame *twice* — once on the sub-interface and once on the parent — +and answers from the parent's pool as well (ISC Kea +[#1117](https://gitlab.isc.org/isc-projects/kea/-/issues/1117)). A client on +VLAN 3 gets two OFFERs and keeps whichever arrives first: + +``` +bond0.3 : 172.31.3.252 → 172.31.3.11 correct +bond0 : 172.31.1.252 → 172.31.1.8 UNTAGGED, Management pool, wrong +``` + +`./labsim-vlan-leak-test.sh` makes one client on a tagged VLAN send a DISCOVER +and captures on the parent and the sub-interface at once. The verdict is how +many OFFERs the **server** emitted and from which subnets — deliberately not +"did the client get the right address", because a client picking correctly is +exactly how this hid. Both orderings were observed across runs, so a passing +client proves nothing. + +```sh +./labsim-vlan-leak-test.sh --vlan 3 # PASS on the current shape +LABSIM_NATIVE_VLAN=1 ./router-up.sh # restore the old shape... +./labsim-vlan-leak-test.sh --vlan 3 # ...and it FAILs again +``` + +Three things this cost, all of which apply to production: + +- **Kea must be restarted after the address moves.** VyOS does not restart it + for an interface address change, so it keeps a raw socket bound with the old + address and the bug survives the fix. In the sim kea had been running since + 16 Aug; the first post-fix test failed for this reason alone and looked like + the fix simply not working. +- **The firewall interface-group must move too.** `interface-group LAN` named + the bare `bond0`; with a default-deny ruleset, moving the address without + moving the group drops every management session and all VLAN 1 routing. +- **Duplicate delivery does not stop.** #1117 says only that there is no longer + a subnet on the parent to match, and that is exactly what happens: two replies + per DISCOVER, both now from the correct pool. Harmless, but do not read a + duplicate as a failure. + +### Tagged and untagged Management coexist + +Verified directly, and it is what makes the production cutover a rolling change +rather than an outage: with the primary still untagged on `bond0` and the +secondary already tagged on `bond0.1`, both routers were reachable, the VIP +stayed up and a VLAN 1 client kept its gateway. One VLAN is one broadcast +domain regardless of how each port tags it, so the two firewalls can be +converted one at a time. See `migration/MANAGEMENT-VLAN-TAGGED.md`. + +`./vlan1-move-monitor.sh` logs VIP/router liveness once a second during the +change, because VRRP reconverges and leaves no trace of who held the VIP. + ## Notes for whoever extends this Things that cost time the first time round, all verified on this image: @@ -206,7 +263,14 @@ Things that cost time the first time round, all verified on this image: ## Not modelled (yet) -VLANs are separate L2 segments rather than one 802.1Q trunk, so this exercises -inter-VLAN routing but not a `bond0.` trunk config specifically. A router -VM would attach one NIC per VLAN. Adding a tagged-trunk variant is the obvious -next step if the bond/vif config itself needs testing. +- **The secondary's bond was fiction until 2026-09-02.** `ovs_bond_router`'s + "already bonded, nothing to do" check compared only the trunk VLAN list, not + the membership. Restarting a VM recreates its taps under new names, so the + bond sat there holding two interfaces that no longer existed while the router's + real taps ran in the bridge as two *independent* ports — no LACP, and carrying + libvirt's own portgroup VLAN config rather than the bond's. It reconciles + membership now, but the lesson generalises: a sim that reports success is not + the same as a sim that models the thing. +- **`labsim-vyos` has a third NIC** on libvirt's `default` network (the scaffold + uplink, see `--drop-scaffold`). The tap count is filtered to `$OVS_NET` for + that reason; an unfiltered count is 3 and silently skipped the primary's bond. diff --git a/labsim/labsim-vlan-leak-test.sh b/labsim/labsim-vlan-leak-test.sh new file mode 100755 index 0000000..6d6abd8 --- /dev/null +++ b/labsim/labsim-vlan-leak-test.sh @@ -0,0 +1,154 @@ +#!/bin/bash +# Does the router offer an address from the WRONG VLAN's pool? +# +# The fault (ISC Kea #1117, "Mix of physical and virtual interfaces (VLAN) does +# not work"): with `dhcp-socket-type: raw`, a frame tagged for a sub-interface is +# ALSO delivered to the PARENT's AF_PACKET socket. Kea then selects a subnet from +# the parent's own address and answers a second time from the wrong pool. Both +# offers race to the client and the CLIENT decides which one wins -- which is why +# the symptom looks device-dependent and unreproducible. +# +# Production and this sim have the identical shape that triggers it: Management +# is the NATIVE/untagged VLAN on `bond0` and therefore has a subnet on the +# parent, while every other VLAN is a `bond0.` sub-interface of that same +# bond. +# +# Method: make one DHCP client on a TAGGED VLAN send a DISCOVER, and capture +# simultaneously on the parent and on the sub-interface. The verdict is not +# "did the client get the right address" -- the client picking correctly is +# exactly how this hid for weeks. The verdict is how many OFFERs the SERVER +# emitted and which source addresses they carried. +# +# ./labsim-vlan-leak-test.sh test VLAN 3 +# ./labsim-vlan-leak-test.sh --vlan 9 test another VLAN +# ./labsim-vlan-leak-test.sh --save before also write the raw captures to +# vlan-leak-evidence/before/ +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" + +ROUTER_IP="${ROUTER_IP:-172.31.1.1}" +ROUTER_PW="${ROUTER_PW:-vyos}" +CLIENT_PW="${CLIENT_PW:-labsim}" +VLAN=3 +CLIENT="" +SAVE="" + +while [ $# -gt 0 ]; do + case "$1" in + --vlan) VLAN="$2"; shift 2 ;; + --client) CLIENT="$2"; shift 2 ;; + --save) SAVE="$2"; shift 2 ;; + *) echo "usage: $0 [--vlan N] [--client IP] [--save LABEL]" >&2; exit 2 ;; + esac +done +: "${CLIENT:=172.31.${VLAN}.10}" + +log() { printf '\033[36m==>\033[0m %s\n' "$*"; } +die() { printf '\033[31merror:\033[0m %s\n' "$*" >&2; exit 1; } + +command -v sshpass >/dev/null || die "sshpass required" + +router() { + timeout 40 sshpass -p "$ROUTER_PW" ssh -o StrictHostKeyChecking=no \ + -o ConnectTimeout=8 "vyos@$ROUTER_IP" "$@" 2>/dev/null +} +# VyOS's login shell is vbash, which returns 255 on anything it does not like -- +# in particular a backgrounded job. Feeding the script to `bash -s` on stdin +# sidesteps vbash entirely and is the only reliable way to leave a daemon behind. +router_sh() { + timeout 40 sshpass -p "$ROUTER_PW" ssh -o StrictHostKeyChecking=no \ + -o ConnectTimeout=8 "vyos@$ROUTER_IP" 'bash -s' 2>/dev/null +} +client() { + timeout 60 sshpass -p "$CLIENT_PW" ssh -o StrictHostKeyChecking=no \ + -o ConnectTimeout=8 "root@$CLIENT" "$@" 2>/dev/null +} + +# Which interfaces to watch. The parent is the whole point: after the fix it +# should carry no DHCP traffic of its own at all. +PARENT="bond0" +VIF="bond0.${VLAN}" + +log "router $ROUTER_IP -- capturing on $PARENT and $VIF" +started="$(router_sh </dev/null 2>&1 +sudo rm -f /tmp/leak-*.txt +sudo nohup tcpdump -i $PARENT -e -nn -l 'udp port 67 or udp port 68' > /tmp/leak-parent.txt 2>/dev/null & +sudo nohup tcpdump -i $VIF -e -nn -l 'udp port 67 or udp port 68' > /tmp/leak-vif.txt 2>/dev/null & +sleep 3 +pgrep -c -f 'tcpdump -i bond0' +EOF +)" +[ "${started:-0}" -ge 2 ] || die "capture did not start on the router (got ${started:-0} of 2)" + +# -s /bin/true: ask, observe the answer, apply nothing. The client's existing +# static address is left alone, so this is safe to run against a live sim VM. +log "client $CLIENT -- sending DISCOVER on VLAN $VLAN" +client_out="$(client "udhcpc -n -q -f -i eth0 -s /bin/true -t 3 -T 3 2>&1")" +[ -n "$client_out" ] || die "no response from client $CLIENT" + +sleep 2 +router "sudo pkill -f 'tcpdump -i bond0'" >/dev/null +parent="$(router 'sudo cat /tmp/leak-parent.txt')" +vif="$(router 'sudo cat /tmp/leak-vif.txt')" + +echo +echo "--- client ---" +echo "$client_out" | sed 's/^/ /' +echo +echo "--- $PARENT (parent) ---" +echo "${parent:- (nothing)}" | sed 's/^/ /' +echo +echo "--- $VIF (sub-interface) ---" +echo "${vif:- (nothing)}" | sed 's/^/ /' +echo + +if [ -n "$SAVE" ]; then + d="$SCRIPT_DIR/vlan-leak-evidence/$SAVE" + mkdir -p "$d" + printf '%s\n' "$client_out" > "$d/client.txt" + printf '%s\n' "$parent" > "$d/capture-parent.txt" + printf '%s\n' "$vif" > "$d/capture-vif.txt" + router '/opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration commands' \ + | grep -E 'interfaces bonding|vrrp group' > "$d/router-config.txt" + log "evidence saved to vlan-leak-evidence/$SAVE/" +fi + +# --- verdict --------------------------------------------------------------- +# Every BOOTP Reply seen anywhere, reduced to its source address. A reply whose +# source is not this VLAN's router leg is an offer from the wrong subnet. +replies="$(printf '%s\n%s\n' "$parent" "$vif" \ + | grep -o '[0-9.]*\.67 > [0-9.]*\.68' | awk '{print $1}' | sed 's/\.67$//' \ + | sort -u)" +want_prefix="172.31.${VLAN}." + +echo "=== verdict ===" +if [ -z "$replies" ]; then + echo "INCONCLUSIVE: the router sent no reply at all -- is DHCP running?" + exit 2 +fi + +bad=0 +while read -r src; do + [ -z "$src" ] && continue + case "$src" in + "$want_prefix"*) printf ' ok offer from %s (this VLAN)\n' "$src" ;; + *) printf ' LEAK offer from %s (WRONG subnet)\n' "$src"; bad=1 ;; + esac +done <<<"$replies" + +# The parent carrying any DHCP of its own is the mechanism, not just a symptom: +# it means the parent still has a subnet kea can match a tagged frame against. +if printf '%s' "$parent" | grep -q 'ethertype IPv4' \ + && printf '%s' "$parent" | grep -v 'vlan ' | grep -q '\.67 > '; then + echo " note $PARENT emitted an UNTAGGED reply -- the parent still serves a subnet" +fi + +echo +if [ "$bad" -eq 0 ]; then + echo "PASS: only this VLAN's pool answered." + exit 0 +fi +echo "FAIL: the router answered from another VLAN's pool (kea #1117)." +exit 1 diff --git a/labsim/ovs.sh b/labsim/ovs.sh index d4067a9..4a79a8b 100644 --- a/labsim/ovs.sh +++ b/labsim/ovs.sh @@ -17,6 +17,21 @@ OVS_BR="${OVS_BR:-ovs-labsim}" OVS_NET="${OVS_NET:-labsim-ovs}" # libvirt network wrapping the bridge LAG_NAME="${LAG_NAME:-lag-vyos}" +# Native (untagged) VLAN on the trunks to the routers. Empty means NONE: every +# VLAN, Management included, is tagged. +# +# This is not a style choice. A native VLAN is what puts a subnet on the bond +# PARENT (`bond0`) while every other VLAN lives on a sub-interface of it. With +# `dhcp-socket-type: raw`, kea then receives each tagged frame TWICE -- once on +# the sub-interface and once on the parent -- and answers from the parent's pool +# as well, so a client on VLAN 3 is offered a Management address and picks +# whichever reply arrives first (ISC Kea #1117). +# +# Set LABSIM_NATIVE_VLAN=1 to restore the old shape and reproduce the bug: +# LABSIM_NATIVE_VLAN=1 ./router-up.sh && ./labsim-vlan-leak-test.sh # FAIL +# ./router-up.sh && ./labsim-vlan-leak-test.sh # PASS +NATIVE_VLAN="${LABSIM_NATIVE_VLAN:-}" + ovs() { sudo ovs-vsctl "$@"; } ovs_require() { @@ -25,6 +40,10 @@ ovs_require() { || die "could not start openvswitch" } +# A comma-separated VLAN list, numerically sorted, for comparing two lists that +# came from different places and need not agree on order. +vlan_sorted() { echo "$1" | tr ',' '\n' | grep -v '^$' | sort -n | paste -sd, -; } + # All VLAN ids from the config, comma separated — used for trunk ports. vlan_id_list() { local ids=() @@ -72,18 +91,23 @@ ovs_define_libvirt_net() { " done - # Trunk: VLAN 1 native/untagged, everything else tagged — the production - # shape. libvirt expresses this declaratively via nativeMode='untagged' - # (see libvirt formatnetwork.html), so it does not need fixing up by hand. - # It also matters functionally: LACPDUs are untagged, and a trunk with no - # native VLAN has nowhere to put them. + # Trunk: every VLAN tagged, and by default NO native VLAN (see NATIVE_VLAN at + # the top of this file for why -- it is the kea #1117 fix, not tidiness). + # libvirt expresses a native VLAN declaratively via nativeMode='untagged' + # (see libvirt formatnetwork.html), so it needs no fixing up by hand. + # + # The worry that a trunk with no native VLAN has nowhere to put LACPDUs is + # unfounded, and was tested rather than reasoned about: with vlan_mode=trunk + # and no tag, `ovs-appctl bond/show` still reports lacp_status: negotiated + # with both members enabled. LACPDUs are slow-protocol frames handled per + # member, below the VLAN layer. local trunk=" " for entry in "${SELECTED[@]}"; do IFS=: read -r vid _n _p _r <<<"$entry" - if [ "$vid" = "1" ]; then - trunk+=" + if [ -n "$NATIVE_VLAN" ] && [ "$vid" = "$NATIVE_VLAN" ]; then + trunk+=" " else trunk+=" @@ -116,29 +140,59 @@ ${pg}${trunk}" ovs_bond_router() { local vm="$1" local taps - # NB: domiflist indents its rows, so anchor on the FIELD not the line — - # /^vnet/ silently matches nothing and the bond never gets built. - taps="$(virsh_q domiflist "$vm" 2>/dev/null | awk '$1 ~ /^vnet/ {print $1}')" + # Two filters, both load-bearing: + # + # $1 ~ /^vnet/ -- domiflist indents its rows, so anchor on the FIELD, not + # the line. /^vnet/ silently matches nothing and the bond never gets built. + # + # $3 == OVS_NET -- count only the taps on the sim fabric. The primary also + # carries a libvirt-NAT scaffold NIC (see --drop-scaffold in the README), so + # an unfiltered count is 3, and this function's "expected 2" guard then + # skipped the primary's bond entirely while reporting only a warning. + taps="$(virsh_q domiflist "$vm" 2>/dev/null \ + | awk -v net="$OVS_NET" '$1 ~ /^vnet/ && $3 == net {print $1}')" local count; count="$(echo "$taps" | grep -c .)" - [ "$count" -eq 2 ] || { warn "router $vm has $count tap(s), expected 2 — skipping bond"; return 1; } - - # Already bonded? Re-runs must still reconcile the VLAN list: adding a VLAN to - # vlans.conf and finding the bond unchanged is exactly how a VLAN silently - # fails to reach a router -- interface present, tag missing, frames dropped by - # the switch. Returning early here once cost real debugging time. - if ovs list-ports "$OVS_BR" 2>/dev/null | grep -qx "$LAG_NAME"; then - local want; want="$(vlan_id_list | tr ',' '\n' | grep -vx 1 | paste -sd, -)" - local have; have="$(ovs get port "$LAG_NAME" trunks 2>/dev/null | tr -d '[] ')" - if [ "$want" != "$have" ]; then - log "bond $LAG_NAME trunk drift: [$have] -> [$want]; updating" - ovs set port "$LAG_NAME" trunks="$want" - else - log "LACP bond $LAG_NAME already present, trunk correct" - fi - return 0 - fi + [ "$count" -eq 2 ] \ + || { warn "router $vm has $count tap(s) on $OVS_NET, expected 2 — skipping bond"; return 1; } local t1 t2; t1="$(echo "$taps" | sed -n 1p)"; t2="$(echo "$taps" | sed -n 2p)" + local want; want="$(vlan_id_list)" + + # Already bonded? Re-runs must still reconcile BOTH the VLAN list and the + # membership, and each has drawn blood: + # + # VLANs -- adding a VLAN to vlans.conf and finding the bond unchanged is how + # a VLAN silently fails to reach a router: interface present, tag missing, + # frames dropped by the switch. + # + # MEMBERS -- restarting the VM recreates its taps with NEW names, leaving the + # bond holding two interfaces that no longer exist. `list-ports` still shows + # the bond, so this early return declared success while the router's real + # taps sat in the bridge as two INDEPENDENT ports, each carrying libvirt's + # own portgroup VLAN config. That is how labsim-vyos2 ran for weeks with no + # LACP at all and a native VLAN nobody had asked for -- and it is invisible + # until you change the trunk and only one router follows. + if ovs list-ports "$OVS_BR" 2>/dev/null | grep -qx "$LAG_NAME"; then + local members; members="$(ovs-appctl-members)" + if [ "$members" != "$(printf '%s\n%s' "$t1" "$t2" | sort | paste -sd, -)" ]; then + warn "bond $LAG_NAME holds stale members [$members], VM has [$t1,$t2] — rebuilding" + ovs --if-exists del-port "$OVS_BR" "$LAG_NAME" + else + # Compare as SETS. vlan_id_list yields config order (1,2,3,9,10,200,51,53) + # while OVS returns its own sorted order, so a raw string compare reports + # drift on every run and rewrites a trunk that was already correct. + local have; have="$(ovs get port "$LAG_NAME" trunks 2>/dev/null | tr -d '[] ')" + if [ "$(vlan_sorted "$want")" != "$(vlan_sorted "$have")" ]; then + log "bond $LAG_NAME trunk drift: [$have] -> [$want]; updating" + ovs set port "$LAG_NAME" trunks="$want" + else + log "LACP bond $LAG_NAME already present, trunk correct" + fi + ovs_set_native "$LAG_NAME" + return 0 + fi + fi + log "bonding $t1 + $t2 into $LAG_NAME (LACP active, balance-tcp)" ovs del-port "$OVS_BR" "$t1" 2>/dev/null || true ovs del-port "$OVS_BR" "$t2" 2>/dev/null || true @@ -152,15 +206,36 @@ ovs_bond_router() { # LACPDUs. Falling back to active-backup brings the links up so negotiation # can start. # - # native-untagged + tag=1 carries the untagged LACPDUs and the management - # VLAN, matching production. libvirt's portgroup VLAN config does NOT apply - # here — the bond is a port libvirt never created — so set it inline. - local tagged; tagged="$(vlan_id_list | tr ',' '\n' | grep -vx 1 | paste -sd, -)" + # The VLAN mode is set inline: libvirt's portgroup config does NOT apply here, + # because the bond is a port libvirt never created. ovs add-bond "$OVS_BR" "$LAG_NAME" "$t1" "$t2" \ - lacp=active bond_mode=balance-tcp \ - vlan_mode=native-untagged tag=1 trunks="$tagged" \ + lacp=active bond_mode=balance-tcp trunks="$want" \ -- set port "$LAG_NAME" other_config:lacp-time=fast \ -- set port "$LAG_NAME" other_config:lacp-fallback-ab=true + ovs_set_native "$LAG_NAME" +} + +# The bond's current members, sorted and comma-joined, or empty if the bond does +# not resolve at all (which is itself the stale case worth rebuilding for). +ovs-appctl-members() { + sudo ovs-appctl bond/show "$LAG_NAME" 2>/dev/null \ + | awk '/^member /{gsub(/:/,"",$2); print $2}' | sort | paste -sd, - +} + +# Apply NATIVE_VLAN to a trunk port. +# +# `tag` MUST be removed, not merely left alone, when there is no native VLAN. +# Setting vlan_mode=trunk while a stale `tag` remains looks correct in +# `ovs-vsctl list port` -- it prints vlan_mode: trunk right next to tag: 1 -- +# but the port keeps egressing that VLAN untagged. Half an hour went into +# "the router is ignoring the trunk change" before the tag was the answer. +ovs_set_native() { + local port="$1" + if [ -n "$NATIVE_VLAN" ]; then + ovs set port "$port" vlan_mode=native-untagged tag="$NATIVE_VLAN" + else + ovs set port "$port" vlan_mode=trunk -- clear port "$port" tag + fi } ovs_bond_status() { diff --git a/labsim/sim-ha-config.py b/labsim/sim-ha-config.py index 62389e4..0e87ec3 100755 --- a/labsim/sim-ha-config.py +++ b/labsim/sim-ha-config.py @@ -60,14 +60,20 @@ def build(role: str) -> list[str]: for vlan, (pfx, cidr) in VLANS.items(): g = group(vlan) - iface = "bond0" if vlan == 1 else f"bond0 vif {vlan}" + # EVERY VLAN is a sub-interface, Management (VLAN 1) included. Putting + # Management on the bare `bond0` is what gives the parent a subnet, and + # kea then answers tagged frames from it as well as from the correct + # sub-interface -- clients on other VLANs get offered a Management + # address (ISC Kea #1117). See NATIVE_VLAN in ovs.sh; proven by + # labsim-vlan-leak-test.sh. + iface = f"bond0 vif {vlan}" out += [ f"# VLAN {vlan}", # The node's own address replaces the .1 it used to hold directly; # .1 becomes the floating VIP, exactly as production will be. f"delete interfaces bonding {iface} address", f"set interfaces bonding {iface} address '{pfx}.{self_o}/{cidr}'", - f"set high-availability vrrp group {g} interface bond0{'' if vlan == 1 else f'.{vlan}'}", + f"set high-availability vrrp group {g} interface bond0.{vlan}", f"set high-availability vrrp group {g} vrid {vlan}", f"set high-availability vrrp group {g} address {pfx}.1/{cidr}", f"set high-availability vrrp group {g} priority {prio}", diff --git a/labsim/sim-net-config.py b/labsim/sim-net-config.py index 2c27dd3..12f3215 100755 --- a/labsim/sim-net-config.py +++ b/labsim/sim-net-config.py @@ -187,7 +187,15 @@ def wan(drop_scaffold: bool) -> list[str]: # which point it is wide open and nothing looks wrong. This is the same policy # expressed as a whitelist, so a new interface is closed until it is named. # --------------------------------------------------------------------------- -LAN_IFACES = ["bond0", "bond0.2", "bond0.3", "bond0.9", "bond0.10", "bond0.200"] +# Management is `bond0.1`, NOT the bare `bond0`. Every VLAN is tagged and the +# bond parent carries no subnet at all -- see NATIVE_VLAN in ovs.sh for why. +# +# This line is the trap in that change. The address move is the visible part and +# the part you remember; leaving `bond0` here instead of `bond0.1` means the +# whole Management VLAN falls outside the LAN group, and with a default-deny +# ruleset that is every management session and all inter-VLAN routing for VLAN 1 +# dropped the instant the commit lands -- on a router you reach through itself. +LAN_IFACES = ["bond0.1", "bond0.2", "bond0.3", "bond0.9", "bond0.10", "bond0.200"] LAN_GROUP = "LAN" diff --git a/labsim/vlan-leak-evidence/after-vlan1/capture-parent.txt b/labsim/vlan-leak-evidence/after-vlan1/capture-parent.txt new file mode 100644 index 0000000..b29b06d --- /dev/null +++ b/labsim/vlan-leak-evidence/after-vlan1/capture-parent.txt @@ -0,0 +1,6 @@ +12:52:35.919490 52:54:00:6d:71:e7 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 1, p 0, ethertype IPv4 (0x0800), 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:6d:71:e7, length 300 +12:52:35.920089 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype 802.1Q (0x8100), length 329: vlan 1, p 0, ethertype IPv4 (0x0800), 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283 +12:52:35.920307 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype 802.1Q (0x8100), length 329: vlan 1, p 0, ethertype IPv4 (0x0800), 172.31.1.252.67 > 172.31.1.7.68: BOOTP/DHCP, Reply, length 283 +12:52:35.922052 52:54:00:6d:71:e7 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 1, p 0, ethertype IPv4 (0x0800), 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:6d:71:e7, length 300 +12:52:35.922509 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype 802.1Q (0x8100), length 329: vlan 1, p 0, ethertype IPv4 (0x0800), 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283 +12:52:35.923327 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype 802.1Q (0x8100), length 329: vlan 1, p 0, ethertype IPv4 (0x0800), 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283 diff --git a/labsim/vlan-leak-evidence/after-vlan1/capture-vif.txt b/labsim/vlan-leak-evidence/after-vlan1/capture-vif.txt new file mode 100644 index 0000000..fe24f7d --- /dev/null +++ b/labsim/vlan-leak-evidence/after-vlan1/capture-vif.txt @@ -0,0 +1,6 @@ +12:52:35.919490 52:54:00:6d:71:e7 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:6d:71:e7, length 300 +12:52:35.920081 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype IPv4 (0x0800), length 325: 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283 +12:52:35.920305 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype IPv4 (0x0800), length 325: 172.31.1.252.67 > 172.31.1.7.68: BOOTP/DHCP, Reply, length 283 +12:52:35.922052 52:54:00:6d:71:e7 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:6d:71:e7, length 300 +12:52:35.922507 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype IPv4 (0x0800), length 325: 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283 +12:52:35.923326 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype IPv4 (0x0800), length 325: 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283 diff --git a/labsim/vlan-leak-evidence/after-vlan1/client.txt b/labsim/vlan-leak-evidence/after-vlan1/client.txt new file mode 100644 index 0000000..68e9e04 --- /dev/null +++ b/labsim/vlan-leak-evidence/after-vlan1/client.txt @@ -0,0 +1,4 @@ +udhcpc: started, v1.37.0 +udhcpc: broadcasting discover +udhcpc: broadcasting select for 172.31.1.6, server 172.31.1.252 +udhcpc: lease of 172.31.1.6 obtained from 172.31.1.252, lease time 86400 diff --git a/labsim/vlan-leak-evidence/after-vlan1/router-config.txt b/labsim/vlan-leak-evidence/after-vlan1/router-config.txt new file mode 100644 index 0000000..aa7e083 --- /dev/null +++ b/labsim/vlan-leak-evidence/after-vlan1/router-config.txt @@ -0,0 +1,64 @@ +set high-availability vrrp group native address 172.31.1.1/24 +set high-availability vrrp group native hello-source-address '172.31.1.252' +set high-availability vrrp group native interface 'bond0.1' +set high-availability vrrp group native no-preempt +set high-availability vrrp group native peer-address '172.31.1.253' +set high-availability vrrp group native priority '200' +set high-availability vrrp group native vrid '1' +set high-availability vrrp group vlan2 address 172.31.2.1/24 +set high-availability vrrp group vlan2 hello-source-address '172.31.2.252' +set high-availability vrrp group vlan2 interface 'bond0.2' +set high-availability vrrp group vlan2 no-preempt +set high-availability vrrp group vlan2 peer-address '172.31.2.253' +set high-availability vrrp group vlan2 priority '200' +set high-availability vrrp group vlan2 vrid '2' +set high-availability vrrp group vlan3 address 172.31.3.1/24 +set high-availability vrrp group vlan3 hello-source-address '172.31.3.252' +set high-availability vrrp group vlan3 interface 'bond0.3' +set high-availability vrrp group vlan3 no-preempt +set high-availability vrrp group vlan3 peer-address '172.31.3.253' +set high-availability vrrp group vlan3 priority '200' +set high-availability vrrp group vlan3 vrid '3' +set high-availability vrrp group vlan9 address 172.31.9.1/24 +set high-availability vrrp group vlan9 hello-source-address '172.31.9.252' +set high-availability vrrp group vlan9 interface 'bond0.9' +set high-availability vrrp group vlan9 no-preempt +set high-availability vrrp group vlan9 peer-address '172.31.9.253' +set high-availability vrrp group vlan9 priority '200' +set high-availability vrrp group vlan9 vrid '9' +set high-availability vrrp group vlan10 address 172.31.10.1/23 +set high-availability vrrp group vlan10 hello-source-address '172.31.10.252' +set high-availability vrrp group vlan10 interface 'bond0.10' +set high-availability vrrp group vlan10 no-preempt +set high-availability vrrp group vlan10 peer-address '172.31.10.253' +set high-availability vrrp group vlan10 priority '200' +set high-availability vrrp group vlan10 vrid '10' +set high-availability vrrp group vlan200 address 172.31.200.1/24 +set high-availability vrrp group vlan200 hello-source-address '172.31.200.252' +set high-availability vrrp group vlan200 interface 'bond0.200' +set high-availability vrrp group vlan200 no-preempt +set high-availability vrrp group vlan200 peer-address '172.31.200.253' +set high-availability vrrp group vlan200 priority '200' +set high-availability vrrp group vlan200 vrid '200' +set interfaces bonding bond0 description 'api-batch-test' +set interfaces bonding bond0 hash-policy 'layer2+3' +set interfaces bonding bond0 lacp-rate 'fast' +set interfaces bonding bond0 member interface 'eth0' +set interfaces bonding bond0 member interface 'eth1' +set interfaces bonding bond0 mode '802.3ad' +set interfaces bonding bond0 vif 1 address '172.31.1.252/24' +set interfaces bonding bond0 vif 1 description 'management' +set interfaces bonding bond0 vif 2 address '172.31.2.252/24' +set interfaces bonding bond0 vif 2 description 'k8s' +set interfaces bonding bond0 vif 3 address '172.31.3.252/24' +set interfaces bonding bond0 vif 3 description 'kvm' +set interfaces bonding bond0 vif 9 address '172.31.9.252/24' +set interfaces bonding bond0 vif 9 description 'private' +set interfaces bonding bond0 vif 10 address '172.31.10.252/23' +set interfaces bonding bond0 vif 10 description 'lot' +set interfaces bonding bond0 vif 51 description 'WAN1 Vodafone-equivalent (sim ISP PPPoE)' +set interfaces bonding bond0 vif 53 address 'dhcp' +set interfaces bonding bond0 vif 53 description 'WAN3 10gig-equivalent (sim ISP DHCP)' +set interfaces bonding bond0 vif 53 dhcp-options default-route-distance '210' +set interfaces bonding bond0 vif 200 address '172.31.200.252/24' +set interfaces bonding bond0 vif 200 description 'roomates' diff --git a/labsim/vlan-leak-evidence/after/capture-parent.txt b/labsim/vlan-leak-evidence/after/capture-parent.txt new file mode 100644 index 0000000..19eb796 --- /dev/null +++ b/labsim/vlan-leak-evidence/after/capture-parent.txt @@ -0,0 +1,6 @@ +12:52:14.639170 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 3, p 0, ethertype IPv4 (0x0800), 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300 +12:52:14.640467 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype 802.1Q (0x8100), length 329: vlan 3, p 0, ethertype IPv4 (0x0800), 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283 +12:52:14.640846 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype 802.1Q (0x8100), length 329: vlan 3, p 0, ethertype IPv4 (0x0800), 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283 +12:52:14.642554 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 3, p 0, ethertype IPv4 (0x0800), 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300 +12:52:14.642766 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype 802.1Q (0x8100), length 329: vlan 3, p 0, ethertype IPv4 (0x0800), 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283 +12:52:14.643056 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype 802.1Q (0x8100), length 329: vlan 3, p 0, ethertype IPv4 (0x0800), 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283 diff --git a/labsim/vlan-leak-evidence/after/capture-vif.txt b/labsim/vlan-leak-evidence/after/capture-vif.txt new file mode 100644 index 0000000..9cd7ef9 --- /dev/null +++ b/labsim/vlan-leak-evidence/after/capture-vif.txt @@ -0,0 +1,6 @@ +12:52:14.639170 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300 +12:52:14.640465 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283 +12:52:14.640845 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283 +12:52:14.642554 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300 +12:52:14.642764 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283 +12:52:14.643055 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283 diff --git a/labsim/vlan-leak-evidence/after/client.txt b/labsim/vlan-leak-evidence/after/client.txt new file mode 100644 index 0000000..29d4f04 --- /dev/null +++ b/labsim/vlan-leak-evidence/after/client.txt @@ -0,0 +1,4 @@ +udhcpc: started, v1.37.0 +udhcpc: broadcasting discover +udhcpc: broadcasting select for 172.31.3.11, server 172.31.3.252 +udhcpc: lease of 172.31.3.11 obtained from 172.31.3.252, lease time 85374 diff --git a/labsim/vlan-leak-evidence/after/router-config.txt b/labsim/vlan-leak-evidence/after/router-config.txt new file mode 100644 index 0000000..aa7e083 --- /dev/null +++ b/labsim/vlan-leak-evidence/after/router-config.txt @@ -0,0 +1,64 @@ +set high-availability vrrp group native address 172.31.1.1/24 +set high-availability vrrp group native hello-source-address '172.31.1.252' +set high-availability vrrp group native interface 'bond0.1' +set high-availability vrrp group native no-preempt +set high-availability vrrp group native peer-address '172.31.1.253' +set high-availability vrrp group native priority '200' +set high-availability vrrp group native vrid '1' +set high-availability vrrp group vlan2 address 172.31.2.1/24 +set high-availability vrrp group vlan2 hello-source-address '172.31.2.252' +set high-availability vrrp group vlan2 interface 'bond0.2' +set high-availability vrrp group vlan2 no-preempt +set high-availability vrrp group vlan2 peer-address '172.31.2.253' +set high-availability vrrp group vlan2 priority '200' +set high-availability vrrp group vlan2 vrid '2' +set high-availability vrrp group vlan3 address 172.31.3.1/24 +set high-availability vrrp group vlan3 hello-source-address '172.31.3.252' +set high-availability vrrp group vlan3 interface 'bond0.3' +set high-availability vrrp group vlan3 no-preempt +set high-availability vrrp group vlan3 peer-address '172.31.3.253' +set high-availability vrrp group vlan3 priority '200' +set high-availability vrrp group vlan3 vrid '3' +set high-availability vrrp group vlan9 address 172.31.9.1/24 +set high-availability vrrp group vlan9 hello-source-address '172.31.9.252' +set high-availability vrrp group vlan9 interface 'bond0.9' +set high-availability vrrp group vlan9 no-preempt +set high-availability vrrp group vlan9 peer-address '172.31.9.253' +set high-availability vrrp group vlan9 priority '200' +set high-availability vrrp group vlan9 vrid '9' +set high-availability vrrp group vlan10 address 172.31.10.1/23 +set high-availability vrrp group vlan10 hello-source-address '172.31.10.252' +set high-availability vrrp group vlan10 interface 'bond0.10' +set high-availability vrrp group vlan10 no-preempt +set high-availability vrrp group vlan10 peer-address '172.31.10.253' +set high-availability vrrp group vlan10 priority '200' +set high-availability vrrp group vlan10 vrid '10' +set high-availability vrrp group vlan200 address 172.31.200.1/24 +set high-availability vrrp group vlan200 hello-source-address '172.31.200.252' +set high-availability vrrp group vlan200 interface 'bond0.200' +set high-availability vrrp group vlan200 no-preempt +set high-availability vrrp group vlan200 peer-address '172.31.200.253' +set high-availability vrrp group vlan200 priority '200' +set high-availability vrrp group vlan200 vrid '200' +set interfaces bonding bond0 description 'api-batch-test' +set interfaces bonding bond0 hash-policy 'layer2+3' +set interfaces bonding bond0 lacp-rate 'fast' +set interfaces bonding bond0 member interface 'eth0' +set interfaces bonding bond0 member interface 'eth1' +set interfaces bonding bond0 mode '802.3ad' +set interfaces bonding bond0 vif 1 address '172.31.1.252/24' +set interfaces bonding bond0 vif 1 description 'management' +set interfaces bonding bond0 vif 2 address '172.31.2.252/24' +set interfaces bonding bond0 vif 2 description 'k8s' +set interfaces bonding bond0 vif 3 address '172.31.3.252/24' +set interfaces bonding bond0 vif 3 description 'kvm' +set interfaces bonding bond0 vif 9 address '172.31.9.252/24' +set interfaces bonding bond0 vif 9 description 'private' +set interfaces bonding bond0 vif 10 address '172.31.10.252/23' +set interfaces bonding bond0 vif 10 description 'lot' +set interfaces bonding bond0 vif 51 description 'WAN1 Vodafone-equivalent (sim ISP PPPoE)' +set interfaces bonding bond0 vif 53 address 'dhcp' +set interfaces bonding bond0 vif 53 description 'WAN3 10gig-equivalent (sim ISP DHCP)' +set interfaces bonding bond0 vif 53 dhcp-options default-route-distance '210' +set interfaces bonding bond0 vif 200 address '172.31.200.252/24' +set interfaces bonding bond0 vif 200 description 'roomates' diff --git a/labsim/vlan-leak-evidence/before/capture-parent.txt b/labsim/vlan-leak-evidence/before/capture-parent.txt new file mode 100644 index 0000000..f26cf94 --- /dev/null +++ b/labsim/vlan-leak-evidence/before/capture-parent.txt @@ -0,0 +1,5 @@ +12:37:08.491910 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 3, p 0, ethertype IPv4 (0x0800), 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300 +12:37:08.492629 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.1.252.67 > 172.31.1.9.68: BOOTP/DHCP, Reply, length 283 +12:37:08.493628 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype 802.1Q (0x8100), length 329: vlan 3, p 0, ethertype IPv4 (0x0800), 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283 +12:37:08.495587 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 3, p 0, ethertype IPv4 (0x0800), 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300 +12:37:08.495946 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype 802.1Q (0x8100), length 329: vlan 3, p 0, ethertype IPv4 (0x0800), 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283 diff --git a/labsim/vlan-leak-evidence/before/capture-vif.txt b/labsim/vlan-leak-evidence/before/capture-vif.txt new file mode 100644 index 0000000..17922b9 --- /dev/null +++ b/labsim/vlan-leak-evidence/before/capture-vif.txt @@ -0,0 +1,4 @@ +12:37:08.491910 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300 +12:37:08.493625 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283 +12:37:08.495587 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300 +12:37:08.495944 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283 diff --git a/labsim/vlan-leak-evidence/before/client.txt b/labsim/vlan-leak-evidence/before/client.txt new file mode 100644 index 0000000..b1d25dd --- /dev/null +++ b/labsim/vlan-leak-evidence/before/client.txt @@ -0,0 +1,4 @@ +udhcpc: started, v1.37.0 +udhcpc: broadcasting discover +udhcpc: broadcasting select for 172.31.3.11, server 172.31.3.252 +udhcpc: lease of 172.31.3.11 obtained from 172.31.3.252, lease time 86280 diff --git a/labsim/vlan-leak-evidence/before/router-config.txt b/labsim/vlan-leak-evidence/before/router-config.txt new file mode 100644 index 0000000..156258f --- /dev/null +++ b/labsim/vlan-leak-evidence/before/router-config.txt @@ -0,0 +1,63 @@ +set high-availability vrrp group native address 172.31.1.1/24 +set high-availability vrrp group native hello-source-address '172.31.1.252' +set high-availability vrrp group native interface 'bond0' +set high-availability vrrp group native no-preempt +set high-availability vrrp group native peer-address '172.31.1.253' +set high-availability vrrp group native priority '200' +set high-availability vrrp group native vrid '1' +set high-availability vrrp group vlan2 address 172.31.2.1/24 +set high-availability vrrp group vlan2 hello-source-address '172.31.2.252' +set high-availability vrrp group vlan2 interface 'bond0.2' +set high-availability vrrp group vlan2 no-preempt +set high-availability vrrp group vlan2 peer-address '172.31.2.253' +set high-availability vrrp group vlan2 priority '200' +set high-availability vrrp group vlan2 vrid '2' +set high-availability vrrp group vlan3 address 172.31.3.1/24 +set high-availability vrrp group vlan3 hello-source-address '172.31.3.252' +set high-availability vrrp group vlan3 interface 'bond0.3' +set high-availability vrrp group vlan3 no-preempt +set high-availability vrrp group vlan3 peer-address '172.31.3.253' +set high-availability vrrp group vlan3 priority '200' +set high-availability vrrp group vlan3 vrid '3' +set high-availability vrrp group vlan9 address 172.31.9.1/24 +set high-availability vrrp group vlan9 hello-source-address '172.31.9.252' +set high-availability vrrp group vlan9 interface 'bond0.9' +set high-availability vrrp group vlan9 no-preempt +set high-availability vrrp group vlan9 peer-address '172.31.9.253' +set high-availability vrrp group vlan9 priority '200' +set high-availability vrrp group vlan9 vrid '9' +set high-availability vrrp group vlan10 address 172.31.10.1/23 +set high-availability vrrp group vlan10 hello-source-address '172.31.10.252' +set high-availability vrrp group vlan10 interface 'bond0.10' +set high-availability vrrp group vlan10 no-preempt +set high-availability vrrp group vlan10 peer-address '172.31.10.253' +set high-availability vrrp group vlan10 priority '200' +set high-availability vrrp group vlan10 vrid '10' +set high-availability vrrp group vlan200 address 172.31.200.1/24 +set high-availability vrrp group vlan200 hello-source-address '172.31.200.252' +set high-availability vrrp group vlan200 interface 'bond0.200' +set high-availability vrrp group vlan200 no-preempt +set high-availability vrrp group vlan200 peer-address '172.31.200.253' +set high-availability vrrp group vlan200 priority '200' +set high-availability vrrp group vlan200 vrid '200' +set interfaces bonding bond0 address '172.31.1.252/24' +set interfaces bonding bond0 description 'api-batch-test' +set interfaces bonding bond0 hash-policy 'layer2+3' +set interfaces bonding bond0 lacp-rate 'fast' +set interfaces bonding bond0 member interface 'eth0' +set interfaces bonding bond0 member interface 'eth1' +set interfaces bonding bond0 mode '802.3ad' +set interfaces bonding bond0 vif 2 address '172.31.2.252/24' +set interfaces bonding bond0 vif 2 description 'k8s' +set interfaces bonding bond0 vif 3 address '172.31.3.252/24' +set interfaces bonding bond0 vif 3 description 'kvm' +set interfaces bonding bond0 vif 9 address '172.31.9.252/24' +set interfaces bonding bond0 vif 9 description 'private' +set interfaces bonding bond0 vif 10 address '172.31.10.252/23' +set interfaces bonding bond0 vif 10 description 'lot' +set interfaces bonding bond0 vif 51 description 'WAN1 Vodafone-equivalent (sim ISP PPPoE)' +set interfaces bonding bond0 vif 53 address 'dhcp' +set interfaces bonding bond0 vif 53 description 'WAN3 10gig-equivalent (sim ISP DHCP)' +set interfaces bonding bond0 vif 53 dhcp-options default-route-distance '210' +set interfaces bonding bond0 vif 200 address '172.31.200.252/24' +set interfaces bonding bond0 vif 200 description 'roomates' diff --git a/labsim/vlan1-move-monitor.sh b/labsim/vlan1-move-monitor.sh new file mode 100755 index 0000000..04006ec --- /dev/null +++ b/labsim/vlan1-move-monitor.sh @@ -0,0 +1,17 @@ +#!/bin/bash +# Timestamped liveness log for the Management VLAN during the bond0 -> bond0.1 move. +# +# The question this answers is not "did it work" but "for how long was it not +# working, and what held the VIP while it was not". Both are invisible after the +# fact: VRRP reconverges and leaves no trace of who was master during the gap. +# +# ./vlan1-move-monitor.sh > /tmp/move.log & +# Columns: time VIP-ping R1-ping R2-ping VIP-mac +VIP="${VIP:-172.31.1.1}"; R1="${R1:-172.31.1.252}"; R2="${R2:-172.31.1.253}" +p() { ping -c1 -W1 -n "$1" >/dev/null 2>&1 && echo up || echo DOWN; } +while :; do + mac="$(ip neigh show "$VIP" 2>/dev/null | awk '{for(i=1;i<=NF;i++) if($i=="lladdr") print $(i+1)}')" + printf '%s vip=%-4s r1=%-4s r2=%-4s vipmac=%s\n' \ + "$(date +%H:%M:%S)" "$(p "$VIP")" "$(p "$R1")" "$(p "$R2")" "${mac:-none}" + sleep 1 +done diff --git a/migration/MANAGEMENT-VLAN-TAGGED.md b/migration/MANAGEMENT-VLAN-TAGGED.md new file mode 100644 index 0000000..96d32dd --- /dev/null +++ b/migration/MANAGEMENT-VLAN-TAGGED.md @@ -0,0 +1,137 @@ +# Moving Management onto a tagged VLAN + +Rehearsed end to end in labsim on 2026-09-02. This is the fix for kea serving +addresses from the wrong VLAN's pool. + +## Why + +ISC Kea [#1117](https://gitlab.isc.org/isc-projects/kea/-/issues/1117): with +`dhcp-socket-type: raw`, a frame tagged for a sub-interface is **also** delivered +to the parent's `AF_PACKET` socket. If the parent serves a subnet, kea answers +from it too. Ours does — Management is the native/untagged VLAN on `bond0` while +VLANs 2/3/9/10/200 are sub-interfaces of that same bond — so one DISCOVER on +VLAN 3 produces two OFFERs and the *client* decides which to keep: + +``` +bond0.3 : 192.168.3.14 correct +bond0 : 192.168.1.28 UNTAGGED, Management pool, wrong +``` + +The fix is to leave **no subnet on the parent**: every VLAN tagged, Management +included, moved from `bond0` to `bond0.1`. + +Confirmed in labsim across all six LAN VLANs: fails before, passes after. +`labsim/labsim-vlan-leak-test.sh` is the test; evidence in +`labsim/vlan-leak-evidence/`. + +## What must change together + +Per router: + +| | from | to | +|---|---|---| +| address | `interfaces bonding bond0 address` | `interfaces bonding bond0 vif 1 address` | +| firewall | `interface-group LAN interface bond0` | `... interface bond0.1` | +| VRRP | `vrrp group native interface bond0` | `... interface bond0.1` | +| kea | — | **restart it** (see traps) | + +On the switch: Native VLAN = **None** on the trunk to that firewall, with VLAN 1 +added to the tagged set. + +## The ordering constraint + +**There is no overlap state.** An 802.1Q port always egresses its native VLAN +untagged, so while VLAN 1 is native the router can *send* tagged VLAN 1 but can +never *receive* it. Verified: a tagged VLAN 1 ARP sent from the switch arrived on +`bond0` untagged and never on `bond0.1`. Configuring "native VLAN 1 **and** VLAN 1 +tagged" as a make-before-break does not work; the switch and router changes for a +given firewall are strictly simultaneous, and that router loses Management in +between. + +What makes this safe anyway: **tagged and untagged Management coexist on the +same VLAN.** One VLAN is one broadcast domain no matter how each port tags it, so +the firewalls can be converted one at a time — verified with the primary untagged +and the secondary already tagged, both reachable, VIP up, VLAN 1 clients fine. + +Access ports are untouched throughout. The UniFi controller at 192.168.1.5 and +your workstation are on access ports and never traverse the firewall trunks, so +you keep the controller you are making the change from. Only the router being +converted goes dark, and only until its own config lands. + +## Procedure + +Do the **backup** router first, then fail the VIPs over and do the other. You +need console (JetKVM) on the router being converted — its Management SSH dies the +moment the switch port changes. + +For each router in turn: + +1. Confirm the *other* router is MASTER and healthy: + `show vrrp` and `sudo /config/vrrp-wan-health; echo $?` (must be 0). +2. Start the monitor from a workstation on an access port: + `labsim/vlan1-move-monitor.sh` (edit the three addresses for production). +3. UniFi: on this firewall's trunk ports, Native VLAN → None, VLAN 1 → tagged. + This router's Management drops now. +4. Over the console, in one commit: + ``` + set interfaces bonding bond0 vif 1 address '192.168.1.252/24' # .253 on vyos002 + set interfaces bonding bond0 vif 1 description 'management' + delete interfaces bonding bond0 address + set firewall group interface-group LAN interface 'bond0.1' + delete firewall group interface-group LAN interface 'bond0' + set high-availability vrrp group native interface 'bond0.1' + commit + save + ``` +5. `sudo systemctl restart isc-kea-dhcp4-server` — see traps. +6. Verify: Management SSH back, `show vrrp` shows `native` on `bond0.1`, and the + leak test passes. + +Then fail back if the VIPs moved (below), and repeat for the other router. + +### Measured windows (labsim) + +| | | +|---|---| +| this router's own Management unreachable | ~27 s (the console apply) | +| VIP `.1` unreachable, peer already converted | **0 s** | +| VIP `.1` unreachable, converting the current MASTER | ~6 s (VRRP failover) | +| VIP unreachable if you convert both routers before the switch | **5 min 30 s** | + +That last row is the failure mode to avoid: with both routers untagged and the +trunks already changed, the VIP is a black hole and **the healthy BACKUP does not +take over**. Its `native` group stays BACKUP because the *other* VLANs still hear +the master, and the sync group holds them together. Redundancy does not help you +here; only ordering does. + +## Traps + +- **Restart kea.** VyOS does not restart it for an interface address change, so + it keeps a raw socket bound with the old address and keeps emitting the wrong + offers. The first post-fix test in the sim failed for this reason alone and + looked exactly like the fix not working. +- **`interface-group LAN`.** Moving the address without moving the group means + Management falls outside the group, and with default-deny that is every + management session and all VLAN 1 inter-VLAN routing, gone on commit — on a + router you reach through itself. Use `commit-confirm` if you are not on console. +- **The VIPs may move, and `no-preempt` keeps them moved.** Converting a router + restarts keepalived and re-initialises *every* group, not just `native`. In one + rehearsal the priority-100 secondary took all six VIPs and held them while the + priority-200 primary sat at BACKUP; in another the restart was quick enough that + nothing moved. It is non-deterministic — check afterwards, every time. + Fail back with `restart vrrp` **on the router currently holding them**. +- **Duplicate delivery does not stop**, and should not be read as failure. #1117 + only promises there is no longer a subnet on the parent to match. Expect two + identical replies per DISCOVER, both from the correct pool. +- **Both firewalls' trunks must end up the same.** If UniFi shares one port + profile between them, changing it converts both at once and you get the 5m30s + row above. Check before you start; use per-port overrides if it does. + +## Not covered by the rehearsal + +- Whether UniFi's port profile can express "no native VLAN" the way OVS can, and + whether the two firewalls share a profile. Unverified — check on the controller. +- Why the JetKVM consoles specifically accepted the wrong OFFER when a VLAN 3 + access port should not receive an untagged VLAN 1 frame at all. Their port + profile likely passes VLAN 1 untagged. Worth confirming, though it does not + change the fix.