labsim: prove the tagged-Management fix for kea's wrong-pool offers
Kea #1117: with dhcp-socket-type raw, a frame tagged for a sub-interface is also delivered to the parent's AF_PACKET socket, and if the parent serves a subnet kea answers from it too. Management being the native VLAN on bond0 is what gives the parent that subnet. One DISCOVER on VLAN 3 produced two OFFERs, and in the captures here the WRONG one arrives first as often as not -- which is why this looked device-dependent rather than like a server bug. labsim-vlan-leak-test.sh reproduces it and scores the SERVER's offers, not the client's choice; a client picking correctly is how this hid. Fails on the old shape, passes on the new one across all six LAN VLANs. Three things the rehearsal caught that reasoning had not: - kea keeps its old raw socket. VyOS does not restart it for an interface address change, so the first post-fix test failed and looked exactly like the fix not working. - interface-group LAN names the bare bond0. Moving the address without moving the group drops every management session under default-deny. - there is no make-before-break. A port always egresses its native VLAN untagged, so while VLAN 1 is native the router can send tagged VLAN 1 but never receive it -- verified, the ARP landed on bond0 untagged. What makes the cutover safe anyway is that tagged and untagged Management coexist, so the firewalls convert one at a time: 0s of VIP downtime, versus 5m30s if both routers go before the switch does. In that state the healthy BACKUP does NOT take over -- the sync group holds native BACKUP because the other VLANs still hear the master. Also fixes two ways the sim was lying. ovs_bond_router compared only the trunk VLAN list on re-runs, so a VM restart left the bond holding taps that no longer existed while the real ones sat in the bridge unbonded -- labsim-vyos2 had no LACP at all. And the tap count included the primary's libvirt-NAT scaffold NIC, so the primary's bond was skipped outright. Runbook: migration/MANAGEMENT-VLAN-TAGGED.md Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
This commit is contained in:
Michal
@@ -186,6 +186,63 @@ why any of it was shaped the way it was.
|
||||
client does not redial promptly. After any change there, check `pppoe0` on
|
||||
the router and `sudo systemctl restart ppp@pppoe0` if it is missing.
|
||||
|
||||
## The trunk carries every VLAN tagged, including Management
|
||||
|
||||
There is deliberately **no native/untagged VLAN** on the trunks to the routers,
|
||||
and Management lives on `bond0.1`, not on the bare `bond0`.
|
||||
|
||||
A native VLAN is what puts a subnet on the bond **parent** while every other
|
||||
VLAN sits on a sub-interface of it. With `dhcp-socket-type: raw`, kea receives
|
||||
each tagged frame *twice* — once on the sub-interface and once on the parent —
|
||||
and answers from the parent's pool as well (ISC Kea
|
||||
[#1117](https://gitlab.isc.org/isc-projects/kea/-/issues/1117)). A client on
|
||||
VLAN 3 gets two OFFERs and keeps whichever arrives first:
|
||||
|
||||
```
|
||||
bond0.3 : 172.31.3.252 → 172.31.3.11 correct
|
||||
bond0 : 172.31.1.252 → 172.31.1.8 UNTAGGED, Management pool, wrong
|
||||
```
|
||||
|
||||
`./labsim-vlan-leak-test.sh` makes one client on a tagged VLAN send a DISCOVER
|
||||
and captures on the parent and the sub-interface at once. The verdict is how
|
||||
many OFFERs the **server** emitted and from which subnets — deliberately not
|
||||
"did the client get the right address", because a client picking correctly is
|
||||
exactly how this hid. Both orderings were observed across runs, so a passing
|
||||
client proves nothing.
|
||||
|
||||
```sh
|
||||
./labsim-vlan-leak-test.sh --vlan 3 # PASS on the current shape
|
||||
LABSIM_NATIVE_VLAN=1 ./router-up.sh # restore the old shape...
|
||||
./labsim-vlan-leak-test.sh --vlan 3 # ...and it FAILs again
|
||||
```
|
||||
|
||||
Three things this cost, all of which apply to production:
|
||||
|
||||
- **Kea must be restarted after the address moves.** VyOS does not restart it
|
||||
for an interface address change, so it keeps a raw socket bound with the old
|
||||
address and the bug survives the fix. In the sim kea had been running since
|
||||
16 Aug; the first post-fix test failed for this reason alone and looked like
|
||||
the fix simply not working.
|
||||
- **The firewall interface-group must move too.** `interface-group LAN` named
|
||||
the bare `bond0`; with a default-deny ruleset, moving the address without
|
||||
moving the group drops every management session and all VLAN 1 routing.
|
||||
- **Duplicate delivery does not stop.** #1117 says only that there is no longer
|
||||
a subnet on the parent to match, and that is exactly what happens: two replies
|
||||
per DISCOVER, both now from the correct pool. Harmless, but do not read a
|
||||
duplicate as a failure.
|
||||
|
||||
### Tagged and untagged Management coexist
|
||||
|
||||
Verified directly, and it is what makes the production cutover a rolling change
|
||||
rather than an outage: with the primary still untagged on `bond0` and the
|
||||
secondary already tagged on `bond0.1`, both routers were reachable, the VIP
|
||||
stayed up and a VLAN 1 client kept its gateway. One VLAN is one broadcast
|
||||
domain regardless of how each port tags it, so the two firewalls can be
|
||||
converted one at a time. See `migration/MANAGEMENT-VLAN-TAGGED.md`.
|
||||
|
||||
`./vlan1-move-monitor.sh` logs VIP/router liveness once a second during the
|
||||
change, because VRRP reconverges and leaves no trace of who held the VIP.
|
||||
|
||||
## Notes for whoever extends this
|
||||
|
||||
Things that cost time the first time round, all verified on this image:
|
||||
@@ -206,7 +263,14 @@ Things that cost time the first time round, all verified on this image:
|
||||
|
||||
## Not modelled (yet)
|
||||
|
||||
VLANs are separate L2 segments rather than one 802.1Q trunk, so this exercises
|
||||
inter-VLAN routing but not a `bond0.<vif>` trunk config specifically. A router
|
||||
VM would attach one NIC per VLAN. Adding a tagged-trunk variant is the obvious
|
||||
next step if the bond/vif config itself needs testing.
|
||||
- **The secondary's bond was fiction until 2026-09-02.** `ovs_bond_router`'s
|
||||
"already bonded, nothing to do" check compared only the trunk VLAN list, not
|
||||
the membership. Restarting a VM recreates its taps under new names, so the
|
||||
bond sat there holding two interfaces that no longer existed while the router's
|
||||
real taps ran in the bridge as two *independent* ports — no LACP, and carrying
|
||||
libvirt's own portgroup VLAN config rather than the bond's. It reconciles
|
||||
membership now, but the lesson generalises: a sim that reports success is not
|
||||
the same as a sim that models the thing.
|
||||
- **`labsim-vyos` has a third NIC** on libvirt's `default` network (the scaffold
|
||||
uplink, see `--drop-scaffold`). The tap count is filtered to `$OVS_NET` for
|
||||
that reason; an unfiltered count is 3 and silently skipped the primary's bond.
|
||||
|
||||
154
labsim/labsim-vlan-leak-test.sh
Executable file
154
labsim/labsim-vlan-leak-test.sh
Executable file
@@ -0,0 +1,154 @@
|
||||
#!/bin/bash
|
||||
# Does the router offer an address from the WRONG VLAN's pool?
|
||||
#
|
||||
# The fault (ISC Kea #1117, "Mix of physical and virtual interfaces (VLAN) does
|
||||
# not work"): with `dhcp-socket-type: raw`, a frame tagged for a sub-interface is
|
||||
# ALSO delivered to the PARENT's AF_PACKET socket. Kea then selects a subnet from
|
||||
# the parent's own address and answers a second time from the wrong pool. Both
|
||||
# offers race to the client and the CLIENT decides which one wins -- which is why
|
||||
# the symptom looks device-dependent and unreproducible.
|
||||
#
|
||||
# Production and this sim have the identical shape that triggers it: Management
|
||||
# is the NATIVE/untagged VLAN on `bond0` and therefore has a subnet on the
|
||||
# parent, while every other VLAN is a `bond0.<vif>` sub-interface of that same
|
||||
# bond.
|
||||
#
|
||||
# Method: make one DHCP client on a TAGGED VLAN send a DISCOVER, and capture
|
||||
# simultaneously on the parent and on the sub-interface. The verdict is not
|
||||
# "did the client get the right address" -- the client picking correctly is
|
||||
# exactly how this hid for weeks. The verdict is how many OFFERs the SERVER
|
||||
# emitted and which source addresses they carried.
|
||||
#
|
||||
# ./labsim-vlan-leak-test.sh test VLAN 3
|
||||
# ./labsim-vlan-leak-test.sh --vlan 9 test another VLAN
|
||||
# ./labsim-vlan-leak-test.sh --save before also write the raw captures to
|
||||
# vlan-leak-evidence/before/
|
||||
set -uo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
|
||||
ROUTER_IP="${ROUTER_IP:-172.31.1.1}"
|
||||
ROUTER_PW="${ROUTER_PW:-vyos}"
|
||||
CLIENT_PW="${CLIENT_PW:-labsim}"
|
||||
VLAN=3
|
||||
CLIENT=""
|
||||
SAVE=""
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--vlan) VLAN="$2"; shift 2 ;;
|
||||
--client) CLIENT="$2"; shift 2 ;;
|
||||
--save) SAVE="$2"; shift 2 ;;
|
||||
*) echo "usage: $0 [--vlan N] [--client IP] [--save LABEL]" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
: "${CLIENT:=172.31.${VLAN}.10}"
|
||||
|
||||
log() { printf '\033[36m==>\033[0m %s\n' "$*"; }
|
||||
die() { printf '\033[31merror:\033[0m %s\n' "$*" >&2; exit 1; }
|
||||
|
||||
command -v sshpass >/dev/null || die "sshpass required"
|
||||
|
||||
router() {
|
||||
timeout 40 sshpass -p "$ROUTER_PW" ssh -o StrictHostKeyChecking=no \
|
||||
-o ConnectTimeout=8 "vyos@$ROUTER_IP" "$@" 2>/dev/null
|
||||
}
|
||||
# VyOS's login shell is vbash, which returns 255 on anything it does not like --
|
||||
# in particular a backgrounded job. Feeding the script to `bash -s` on stdin
|
||||
# sidesteps vbash entirely and is the only reliable way to leave a daemon behind.
|
||||
router_sh() {
|
||||
timeout 40 sshpass -p "$ROUTER_PW" ssh -o StrictHostKeyChecking=no \
|
||||
-o ConnectTimeout=8 "vyos@$ROUTER_IP" 'bash -s' 2>/dev/null
|
||||
}
|
||||
client() {
|
||||
timeout 60 sshpass -p "$CLIENT_PW" ssh -o StrictHostKeyChecking=no \
|
||||
-o ConnectTimeout=8 "root@$CLIENT" "$@" 2>/dev/null
|
||||
}
|
||||
|
||||
# Which interfaces to watch. The parent is the whole point: after the fix it
|
||||
# should carry no DHCP traffic of its own at all.
|
||||
PARENT="bond0"
|
||||
VIF="bond0.${VLAN}"
|
||||
|
||||
log "router $ROUTER_IP -- capturing on $PARENT and $VIF"
|
||||
started="$(router_sh <<EOF
|
||||
sudo pkill -f 'tcpdump -i bond0' >/dev/null 2>&1
|
||||
sudo rm -f /tmp/leak-*.txt
|
||||
sudo nohup tcpdump -i $PARENT -e -nn -l 'udp port 67 or udp port 68' > /tmp/leak-parent.txt 2>/dev/null &
|
||||
sudo nohup tcpdump -i $VIF -e -nn -l 'udp port 67 or udp port 68' > /tmp/leak-vif.txt 2>/dev/null &
|
||||
sleep 3
|
||||
pgrep -c -f 'tcpdump -i bond0'
|
||||
EOF
|
||||
)"
|
||||
[ "${started:-0}" -ge 2 ] || die "capture did not start on the router (got ${started:-0} of 2)"
|
||||
|
||||
# -s /bin/true: ask, observe the answer, apply nothing. The client's existing
|
||||
# static address is left alone, so this is safe to run against a live sim VM.
|
||||
log "client $CLIENT -- sending DISCOVER on VLAN $VLAN"
|
||||
client_out="$(client "udhcpc -n -q -f -i eth0 -s /bin/true -t 3 -T 3 2>&1")"
|
||||
[ -n "$client_out" ] || die "no response from client $CLIENT"
|
||||
|
||||
sleep 2
|
||||
router "sudo pkill -f 'tcpdump -i bond0'" >/dev/null
|
||||
parent="$(router 'sudo cat /tmp/leak-parent.txt')"
|
||||
vif="$(router 'sudo cat /tmp/leak-vif.txt')"
|
||||
|
||||
echo
|
||||
echo "--- client ---"
|
||||
echo "$client_out" | sed 's/^/ /'
|
||||
echo
|
||||
echo "--- $PARENT (parent) ---"
|
||||
echo "${parent:- (nothing)}" | sed 's/^/ /'
|
||||
echo
|
||||
echo "--- $VIF (sub-interface) ---"
|
||||
echo "${vif:- (nothing)}" | sed 's/^/ /'
|
||||
echo
|
||||
|
||||
if [ -n "$SAVE" ]; then
|
||||
d="$SCRIPT_DIR/vlan-leak-evidence/$SAVE"
|
||||
mkdir -p "$d"
|
||||
printf '%s\n' "$client_out" > "$d/client.txt"
|
||||
printf '%s\n' "$parent" > "$d/capture-parent.txt"
|
||||
printf '%s\n' "$vif" > "$d/capture-vif.txt"
|
||||
router '/opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration commands' \
|
||||
| grep -E 'interfaces bonding|vrrp group' > "$d/router-config.txt"
|
||||
log "evidence saved to vlan-leak-evidence/$SAVE/"
|
||||
fi
|
||||
|
||||
# --- verdict ---------------------------------------------------------------
|
||||
# Every BOOTP Reply seen anywhere, reduced to its source address. A reply whose
|
||||
# source is not this VLAN's router leg is an offer from the wrong subnet.
|
||||
replies="$(printf '%s\n%s\n' "$parent" "$vif" \
|
||||
| grep -o '[0-9.]*\.67 > [0-9.]*\.68' | awk '{print $1}' | sed 's/\.67$//' \
|
||||
| sort -u)"
|
||||
want_prefix="172.31.${VLAN}."
|
||||
|
||||
echo "=== verdict ==="
|
||||
if [ -z "$replies" ]; then
|
||||
echo "INCONCLUSIVE: the router sent no reply at all -- is DHCP running?"
|
||||
exit 2
|
||||
fi
|
||||
|
||||
bad=0
|
||||
while read -r src; do
|
||||
[ -z "$src" ] && continue
|
||||
case "$src" in
|
||||
"$want_prefix"*) printf ' ok offer from %s (this VLAN)\n' "$src" ;;
|
||||
*) printf ' LEAK offer from %s (WRONG subnet)\n' "$src"; bad=1 ;;
|
||||
esac
|
||||
done <<<"$replies"
|
||||
|
||||
# The parent carrying any DHCP of its own is the mechanism, not just a symptom:
|
||||
# it means the parent still has a subnet kea can match a tagged frame against.
|
||||
if printf '%s' "$parent" | grep -q 'ethertype IPv4' \
|
||||
&& printf '%s' "$parent" | grep -v 'vlan ' | grep -q '\.67 > '; then
|
||||
echo " note $PARENT emitted an UNTAGGED reply -- the parent still serves a subnet"
|
||||
fi
|
||||
|
||||
echo
|
||||
if [ "$bad" -eq 0 ]; then
|
||||
echo "PASS: only this VLAN's pool answered."
|
||||
exit 0
|
||||
fi
|
||||
echo "FAIL: the router answered from another VLAN's pool (kea #1117)."
|
||||
exit 1
|
||||
141
labsim/ovs.sh
141
labsim/ovs.sh
@@ -17,6 +17,21 @@ OVS_BR="${OVS_BR:-ovs-labsim}"
|
||||
OVS_NET="${OVS_NET:-labsim-ovs}" # libvirt network wrapping the bridge
|
||||
LAG_NAME="${LAG_NAME:-lag-vyos}"
|
||||
|
||||
# Native (untagged) VLAN on the trunks to the routers. Empty means NONE: every
|
||||
# VLAN, Management included, is tagged.
|
||||
#
|
||||
# This is not a style choice. A native VLAN is what puts a subnet on the bond
|
||||
# PARENT (`bond0`) while every other VLAN lives on a sub-interface of it. With
|
||||
# `dhcp-socket-type: raw`, kea then receives each tagged frame TWICE -- once on
|
||||
# the sub-interface and once on the parent -- and answers from the parent's pool
|
||||
# as well, so a client on VLAN 3 is offered a Management address and picks
|
||||
# whichever reply arrives first (ISC Kea #1117).
|
||||
#
|
||||
# Set LABSIM_NATIVE_VLAN=1 to restore the old shape and reproduce the bug:
|
||||
# LABSIM_NATIVE_VLAN=1 ./router-up.sh && ./labsim-vlan-leak-test.sh # FAIL
|
||||
# ./router-up.sh && ./labsim-vlan-leak-test.sh # PASS
|
||||
NATIVE_VLAN="${LABSIM_NATIVE_VLAN:-}"
|
||||
|
||||
ovs() { sudo ovs-vsctl "$@"; }
|
||||
|
||||
ovs_require() {
|
||||
@@ -25,6 +40,10 @@ ovs_require() {
|
||||
|| die "could not start openvswitch"
|
||||
}
|
||||
|
||||
# A comma-separated VLAN list, numerically sorted, for comparing two lists that
|
||||
# came from different places and need not agree on order.
|
||||
vlan_sorted() { echo "$1" | tr ',' '\n' | grep -v '^$' | sort -n | paste -sd, -; }
|
||||
|
||||
# All VLAN ids from the config, comma separated — used for trunk ports.
|
||||
vlan_id_list() {
|
||||
local ids=()
|
||||
@@ -72,18 +91,23 @@ ovs_define_libvirt_net() {
|
||||
"
|
||||
done
|
||||
|
||||
# Trunk: VLAN 1 native/untagged, everything else tagged — the production
|
||||
# shape. libvirt expresses this declaratively via nativeMode='untagged'
|
||||
# (see libvirt formatnetwork.html), so it does not need fixing up by hand.
|
||||
# It also matters functionally: LACPDUs are untagged, and a trunk with no
|
||||
# native VLAN has nowhere to put them.
|
||||
# Trunk: every VLAN tagged, and by default NO native VLAN (see NATIVE_VLAN at
|
||||
# the top of this file for why -- it is the kea #1117 fix, not tidiness).
|
||||
# libvirt expresses a native VLAN declaratively via nativeMode='untagged'
|
||||
# (see libvirt formatnetwork.html), so it needs no fixing up by hand.
|
||||
#
|
||||
# The worry that a trunk with no native VLAN has nowhere to put LACPDUs is
|
||||
# unfounded, and was tested rather than reasoned about: with vlan_mode=trunk
|
||||
# and no tag, `ovs-appctl bond/show` still reports lacp_status: negotiated
|
||||
# with both members enabled. LACPDUs are slow-protocol frames handled per
|
||||
# member, below the VLAN layer.
|
||||
local trunk=" <portgroup name='trunk'>
|
||||
<vlan trunk='yes'>
|
||||
"
|
||||
for entry in "${SELECTED[@]}"; do
|
||||
IFS=: read -r vid _n _p _r <<<"$entry"
|
||||
if [ "$vid" = "1" ]; then
|
||||
trunk+=" <tag id='1' nativeMode='untagged'/>
|
||||
if [ -n "$NATIVE_VLAN" ] && [ "$vid" = "$NATIVE_VLAN" ]; then
|
||||
trunk+=" <tag id='${vid}' nativeMode='untagged'/>
|
||||
"
|
||||
else
|
||||
trunk+=" <tag id='${vid}'/>
|
||||
@@ -116,29 +140,59 @@ ${pg}${trunk}</network>"
|
||||
ovs_bond_router() {
|
||||
local vm="$1"
|
||||
local taps
|
||||
# NB: domiflist indents its rows, so anchor on the FIELD not the line —
|
||||
# /^vnet/ silently matches nothing and the bond never gets built.
|
||||
taps="$(virsh_q domiflist "$vm" 2>/dev/null | awk '$1 ~ /^vnet/ {print $1}')"
|
||||
# Two filters, both load-bearing:
|
||||
#
|
||||
# $1 ~ /^vnet/ -- domiflist indents its rows, so anchor on the FIELD, not
|
||||
# the line. /^vnet/ silently matches nothing and the bond never gets built.
|
||||
#
|
||||
# $3 == OVS_NET -- count only the taps on the sim fabric. The primary also
|
||||
# carries a libvirt-NAT scaffold NIC (see --drop-scaffold in the README), so
|
||||
# an unfiltered count is 3, and this function's "expected 2" guard then
|
||||
# skipped the primary's bond entirely while reporting only a warning.
|
||||
taps="$(virsh_q domiflist "$vm" 2>/dev/null \
|
||||
| awk -v net="$OVS_NET" '$1 ~ /^vnet/ && $3 == net {print $1}')"
|
||||
local count; count="$(echo "$taps" | grep -c .)"
|
||||
[ "$count" -eq 2 ] || { warn "router $vm has $count tap(s), expected 2 — skipping bond"; return 1; }
|
||||
|
||||
# Already bonded? Re-runs must still reconcile the VLAN list: adding a VLAN to
|
||||
# vlans.conf and finding the bond unchanged is exactly how a VLAN silently
|
||||
# fails to reach a router -- interface present, tag missing, frames dropped by
|
||||
# the switch. Returning early here once cost real debugging time.
|
||||
if ovs list-ports "$OVS_BR" 2>/dev/null | grep -qx "$LAG_NAME"; then
|
||||
local want; want="$(vlan_id_list | tr ',' '\n' | grep -vx 1 | paste -sd, -)"
|
||||
local have; have="$(ovs get port "$LAG_NAME" trunks 2>/dev/null | tr -d '[] ')"
|
||||
if [ "$want" != "$have" ]; then
|
||||
log "bond $LAG_NAME trunk drift: [$have] -> [$want]; updating"
|
||||
ovs set port "$LAG_NAME" trunks="$want"
|
||||
else
|
||||
log "LACP bond $LAG_NAME already present, trunk correct"
|
||||
fi
|
||||
return 0
|
||||
fi
|
||||
[ "$count" -eq 2 ] \
|
||||
|| { warn "router $vm has $count tap(s) on $OVS_NET, expected 2 — skipping bond"; return 1; }
|
||||
|
||||
local t1 t2; t1="$(echo "$taps" | sed -n 1p)"; t2="$(echo "$taps" | sed -n 2p)"
|
||||
local want; want="$(vlan_id_list)"
|
||||
|
||||
# Already bonded? Re-runs must still reconcile BOTH the VLAN list and the
|
||||
# membership, and each has drawn blood:
|
||||
#
|
||||
# VLANs -- adding a VLAN to vlans.conf and finding the bond unchanged is how
|
||||
# a VLAN silently fails to reach a router: interface present, tag missing,
|
||||
# frames dropped by the switch.
|
||||
#
|
||||
# MEMBERS -- restarting the VM recreates its taps with NEW names, leaving the
|
||||
# bond holding two interfaces that no longer exist. `list-ports` still shows
|
||||
# the bond, so this early return declared success while the router's real
|
||||
# taps sat in the bridge as two INDEPENDENT ports, each carrying libvirt's
|
||||
# own portgroup VLAN config. That is how labsim-vyos2 ran for weeks with no
|
||||
# LACP at all and a native VLAN nobody had asked for -- and it is invisible
|
||||
# until you change the trunk and only one router follows.
|
||||
if ovs list-ports "$OVS_BR" 2>/dev/null | grep -qx "$LAG_NAME"; then
|
||||
local members; members="$(ovs-appctl-members)"
|
||||
if [ "$members" != "$(printf '%s\n%s' "$t1" "$t2" | sort | paste -sd, -)" ]; then
|
||||
warn "bond $LAG_NAME holds stale members [$members], VM has [$t1,$t2] — rebuilding"
|
||||
ovs --if-exists del-port "$OVS_BR" "$LAG_NAME"
|
||||
else
|
||||
# Compare as SETS. vlan_id_list yields config order (1,2,3,9,10,200,51,53)
|
||||
# while OVS returns its own sorted order, so a raw string compare reports
|
||||
# drift on every run and rewrites a trunk that was already correct.
|
||||
local have; have="$(ovs get port "$LAG_NAME" trunks 2>/dev/null | tr -d '[] ')"
|
||||
if [ "$(vlan_sorted "$want")" != "$(vlan_sorted "$have")" ]; then
|
||||
log "bond $LAG_NAME trunk drift: [$have] -> [$want]; updating"
|
||||
ovs set port "$LAG_NAME" trunks="$want"
|
||||
else
|
||||
log "LACP bond $LAG_NAME already present, trunk correct"
|
||||
fi
|
||||
ovs_set_native "$LAG_NAME"
|
||||
return 0
|
||||
fi
|
||||
fi
|
||||
|
||||
log "bonding $t1 + $t2 into $LAG_NAME (LACP active, balance-tcp)"
|
||||
ovs del-port "$OVS_BR" "$t1" 2>/dev/null || true
|
||||
ovs del-port "$OVS_BR" "$t2" 2>/dev/null || true
|
||||
@@ -152,15 +206,36 @@ ovs_bond_router() {
|
||||
# LACPDUs. Falling back to active-backup brings the links up so negotiation
|
||||
# can start.
|
||||
#
|
||||
# native-untagged + tag=1 carries the untagged LACPDUs and the management
|
||||
# VLAN, matching production. libvirt's portgroup VLAN config does NOT apply
|
||||
# here — the bond is a port libvirt never created — so set it inline.
|
||||
local tagged; tagged="$(vlan_id_list | tr ',' '\n' | grep -vx 1 | paste -sd, -)"
|
||||
# The VLAN mode is set inline: libvirt's portgroup config does NOT apply here,
|
||||
# because the bond is a port libvirt never created.
|
||||
ovs add-bond "$OVS_BR" "$LAG_NAME" "$t1" "$t2" \
|
||||
lacp=active bond_mode=balance-tcp \
|
||||
vlan_mode=native-untagged tag=1 trunks="$tagged" \
|
||||
lacp=active bond_mode=balance-tcp trunks="$want" \
|
||||
-- set port "$LAG_NAME" other_config:lacp-time=fast \
|
||||
-- set port "$LAG_NAME" other_config:lacp-fallback-ab=true
|
||||
ovs_set_native "$LAG_NAME"
|
||||
}
|
||||
|
||||
# The bond's current members, sorted and comma-joined, or empty if the bond does
|
||||
# not resolve at all (which is itself the stale case worth rebuilding for).
|
||||
ovs-appctl-members() {
|
||||
sudo ovs-appctl bond/show "$LAG_NAME" 2>/dev/null \
|
||||
| awk '/^member /{gsub(/:/,"",$2); print $2}' | sort | paste -sd, -
|
||||
}
|
||||
|
||||
# Apply NATIVE_VLAN to a trunk port.
|
||||
#
|
||||
# `tag` MUST be removed, not merely left alone, when there is no native VLAN.
|
||||
# Setting vlan_mode=trunk while a stale `tag` remains looks correct in
|
||||
# `ovs-vsctl list port` -- it prints vlan_mode: trunk right next to tag: 1 --
|
||||
# but the port keeps egressing that VLAN untagged. Half an hour went into
|
||||
# "the router is ignoring the trunk change" before the tag was the answer.
|
||||
ovs_set_native() {
|
||||
local port="$1"
|
||||
if [ -n "$NATIVE_VLAN" ]; then
|
||||
ovs set port "$port" vlan_mode=native-untagged tag="$NATIVE_VLAN"
|
||||
else
|
||||
ovs set port "$port" vlan_mode=trunk -- clear port "$port" tag
|
||||
fi
|
||||
}
|
||||
|
||||
ovs_bond_status() {
|
||||
|
||||
@@ -60,14 +60,20 @@ def build(role: str) -> list[str]:
|
||||
|
||||
for vlan, (pfx, cidr) in VLANS.items():
|
||||
g = group(vlan)
|
||||
iface = "bond0" if vlan == 1 else f"bond0 vif {vlan}"
|
||||
# EVERY VLAN is a sub-interface, Management (VLAN 1) included. Putting
|
||||
# Management on the bare `bond0` is what gives the parent a subnet, and
|
||||
# kea then answers tagged frames from it as well as from the correct
|
||||
# sub-interface -- clients on other VLANs get offered a Management
|
||||
# address (ISC Kea #1117). See NATIVE_VLAN in ovs.sh; proven by
|
||||
# labsim-vlan-leak-test.sh.
|
||||
iface = f"bond0 vif {vlan}"
|
||||
out += [
|
||||
f"# VLAN {vlan}",
|
||||
# The node's own address replaces the .1 it used to hold directly;
|
||||
# .1 becomes the floating VIP, exactly as production will be.
|
||||
f"delete interfaces bonding {iface} address",
|
||||
f"set interfaces bonding {iface} address '{pfx}.{self_o}/{cidr}'",
|
||||
f"set high-availability vrrp group {g} interface bond0{'' if vlan == 1 else f'.{vlan}'}",
|
||||
f"set high-availability vrrp group {g} interface bond0.{vlan}",
|
||||
f"set high-availability vrrp group {g} vrid {vlan}",
|
||||
f"set high-availability vrrp group {g} address {pfx}.1/{cidr}",
|
||||
f"set high-availability vrrp group {g} priority {prio}",
|
||||
|
||||
@@ -187,7 +187,15 @@ def wan(drop_scaffold: bool) -> list[str]:
|
||||
# which point it is wide open and nothing looks wrong. This is the same policy
|
||||
# expressed as a whitelist, so a new interface is closed until it is named.
|
||||
# ---------------------------------------------------------------------------
|
||||
LAN_IFACES = ["bond0", "bond0.2", "bond0.3", "bond0.9", "bond0.10", "bond0.200"]
|
||||
# Management is `bond0.1`, NOT the bare `bond0`. Every VLAN is tagged and the
|
||||
# bond parent carries no subnet at all -- see NATIVE_VLAN in ovs.sh for why.
|
||||
#
|
||||
# This line is the trap in that change. The address move is the visible part and
|
||||
# the part you remember; leaving `bond0` here instead of `bond0.1` means the
|
||||
# whole Management VLAN falls outside the LAN group, and with a default-deny
|
||||
# ruleset that is every management session and all inter-VLAN routing for VLAN 1
|
||||
# dropped the instant the commit lands -- on a router you reach through itself.
|
||||
LAN_IFACES = ["bond0.1", "bond0.2", "bond0.3", "bond0.9", "bond0.10", "bond0.200"]
|
||||
LAN_GROUP = "LAN"
|
||||
|
||||
|
||||
|
||||
6
labsim/vlan-leak-evidence/after-vlan1/capture-parent.txt
Normal file
6
labsim/vlan-leak-evidence/after-vlan1/capture-parent.txt
Normal file
@@ -0,0 +1,6 @@
|
||||
12:52:35.919490 52:54:00:6d:71:e7 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 1, p 0, ethertype IPv4 (0x0800), 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:6d:71:e7, length 300
|
||||
12:52:35.920089 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype 802.1Q (0x8100), length 329: vlan 1, p 0, ethertype IPv4 (0x0800), 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283
|
||||
12:52:35.920307 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype 802.1Q (0x8100), length 329: vlan 1, p 0, ethertype IPv4 (0x0800), 172.31.1.252.67 > 172.31.1.7.68: BOOTP/DHCP, Reply, length 283
|
||||
12:52:35.922052 52:54:00:6d:71:e7 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 1, p 0, ethertype IPv4 (0x0800), 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:6d:71:e7, length 300
|
||||
12:52:35.922509 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype 802.1Q (0x8100), length 329: vlan 1, p 0, ethertype IPv4 (0x0800), 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283
|
||||
12:52:35.923327 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype 802.1Q (0x8100), length 329: vlan 1, p 0, ethertype IPv4 (0x0800), 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283
|
||||
6
labsim/vlan-leak-evidence/after-vlan1/capture-vif.txt
Normal file
6
labsim/vlan-leak-evidence/after-vlan1/capture-vif.txt
Normal file
@@ -0,0 +1,6 @@
|
||||
12:52:35.919490 52:54:00:6d:71:e7 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:6d:71:e7, length 300
|
||||
12:52:35.920081 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype IPv4 (0x0800), length 325: 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283
|
||||
12:52:35.920305 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype IPv4 (0x0800), length 325: 172.31.1.252.67 > 172.31.1.7.68: BOOTP/DHCP, Reply, length 283
|
||||
12:52:35.922052 52:54:00:6d:71:e7 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:6d:71:e7, length 300
|
||||
12:52:35.922507 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype IPv4 (0x0800), length 325: 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283
|
||||
12:52:35.923326 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype IPv4 (0x0800), length 325: 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283
|
||||
4
labsim/vlan-leak-evidence/after-vlan1/client.txt
Normal file
4
labsim/vlan-leak-evidence/after-vlan1/client.txt
Normal file
@@ -0,0 +1,4 @@
|
||||
udhcpc: started, v1.37.0
|
||||
udhcpc: broadcasting discover
|
||||
udhcpc: broadcasting select for 172.31.1.6, server 172.31.1.252
|
||||
udhcpc: lease of 172.31.1.6 obtained from 172.31.1.252, lease time 86400
|
||||
64
labsim/vlan-leak-evidence/after-vlan1/router-config.txt
Normal file
64
labsim/vlan-leak-evidence/after-vlan1/router-config.txt
Normal file
@@ -0,0 +1,64 @@
|
||||
set high-availability vrrp group native address 172.31.1.1/24
|
||||
set high-availability vrrp group native hello-source-address '172.31.1.252'
|
||||
set high-availability vrrp group native interface 'bond0.1'
|
||||
set high-availability vrrp group native no-preempt
|
||||
set high-availability vrrp group native peer-address '172.31.1.253'
|
||||
set high-availability vrrp group native priority '200'
|
||||
set high-availability vrrp group native vrid '1'
|
||||
set high-availability vrrp group vlan2 address 172.31.2.1/24
|
||||
set high-availability vrrp group vlan2 hello-source-address '172.31.2.252'
|
||||
set high-availability vrrp group vlan2 interface 'bond0.2'
|
||||
set high-availability vrrp group vlan2 no-preempt
|
||||
set high-availability vrrp group vlan2 peer-address '172.31.2.253'
|
||||
set high-availability vrrp group vlan2 priority '200'
|
||||
set high-availability vrrp group vlan2 vrid '2'
|
||||
set high-availability vrrp group vlan3 address 172.31.3.1/24
|
||||
set high-availability vrrp group vlan3 hello-source-address '172.31.3.252'
|
||||
set high-availability vrrp group vlan3 interface 'bond0.3'
|
||||
set high-availability vrrp group vlan3 no-preempt
|
||||
set high-availability vrrp group vlan3 peer-address '172.31.3.253'
|
||||
set high-availability vrrp group vlan3 priority '200'
|
||||
set high-availability vrrp group vlan3 vrid '3'
|
||||
set high-availability vrrp group vlan9 address 172.31.9.1/24
|
||||
set high-availability vrrp group vlan9 hello-source-address '172.31.9.252'
|
||||
set high-availability vrrp group vlan9 interface 'bond0.9'
|
||||
set high-availability vrrp group vlan9 no-preempt
|
||||
set high-availability vrrp group vlan9 peer-address '172.31.9.253'
|
||||
set high-availability vrrp group vlan9 priority '200'
|
||||
set high-availability vrrp group vlan9 vrid '9'
|
||||
set high-availability vrrp group vlan10 address 172.31.10.1/23
|
||||
set high-availability vrrp group vlan10 hello-source-address '172.31.10.252'
|
||||
set high-availability vrrp group vlan10 interface 'bond0.10'
|
||||
set high-availability vrrp group vlan10 no-preempt
|
||||
set high-availability vrrp group vlan10 peer-address '172.31.10.253'
|
||||
set high-availability vrrp group vlan10 priority '200'
|
||||
set high-availability vrrp group vlan10 vrid '10'
|
||||
set high-availability vrrp group vlan200 address 172.31.200.1/24
|
||||
set high-availability vrrp group vlan200 hello-source-address '172.31.200.252'
|
||||
set high-availability vrrp group vlan200 interface 'bond0.200'
|
||||
set high-availability vrrp group vlan200 no-preempt
|
||||
set high-availability vrrp group vlan200 peer-address '172.31.200.253'
|
||||
set high-availability vrrp group vlan200 priority '200'
|
||||
set high-availability vrrp group vlan200 vrid '200'
|
||||
set interfaces bonding bond0 description 'api-batch-test'
|
||||
set interfaces bonding bond0 hash-policy 'layer2+3'
|
||||
set interfaces bonding bond0 lacp-rate 'fast'
|
||||
set interfaces bonding bond0 member interface 'eth0'
|
||||
set interfaces bonding bond0 member interface 'eth1'
|
||||
set interfaces bonding bond0 mode '802.3ad'
|
||||
set interfaces bonding bond0 vif 1 address '172.31.1.252/24'
|
||||
set interfaces bonding bond0 vif 1 description 'management'
|
||||
set interfaces bonding bond0 vif 2 address '172.31.2.252/24'
|
||||
set interfaces bonding bond0 vif 2 description 'k8s'
|
||||
set interfaces bonding bond0 vif 3 address '172.31.3.252/24'
|
||||
set interfaces bonding bond0 vif 3 description 'kvm'
|
||||
set interfaces bonding bond0 vif 9 address '172.31.9.252/24'
|
||||
set interfaces bonding bond0 vif 9 description 'private'
|
||||
set interfaces bonding bond0 vif 10 address '172.31.10.252/23'
|
||||
set interfaces bonding bond0 vif 10 description 'lot'
|
||||
set interfaces bonding bond0 vif 51 description 'WAN1 Vodafone-equivalent (sim ISP PPPoE)'
|
||||
set interfaces bonding bond0 vif 53 address 'dhcp'
|
||||
set interfaces bonding bond0 vif 53 description 'WAN3 10gig-equivalent (sim ISP DHCP)'
|
||||
set interfaces bonding bond0 vif 53 dhcp-options default-route-distance '210'
|
||||
set interfaces bonding bond0 vif 200 address '172.31.200.252/24'
|
||||
set interfaces bonding bond0 vif 200 description 'roomates'
|
||||
6
labsim/vlan-leak-evidence/after/capture-parent.txt
Normal file
6
labsim/vlan-leak-evidence/after/capture-parent.txt
Normal file
@@ -0,0 +1,6 @@
|
||||
12:52:14.639170 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 3, p 0, ethertype IPv4 (0x0800), 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
|
||||
12:52:14.640467 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype 802.1Q (0x8100), length 329: vlan 3, p 0, ethertype IPv4 (0x0800), 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
|
||||
12:52:14.640846 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype 802.1Q (0x8100), length 329: vlan 3, p 0, ethertype IPv4 (0x0800), 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
|
||||
12:52:14.642554 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 3, p 0, ethertype IPv4 (0x0800), 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
|
||||
12:52:14.642766 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype 802.1Q (0x8100), length 329: vlan 3, p 0, ethertype IPv4 (0x0800), 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
|
||||
12:52:14.643056 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype 802.1Q (0x8100), length 329: vlan 3, p 0, ethertype IPv4 (0x0800), 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
|
||||
6
labsim/vlan-leak-evidence/after/capture-vif.txt
Normal file
6
labsim/vlan-leak-evidence/after/capture-vif.txt
Normal file
@@ -0,0 +1,6 @@
|
||||
12:52:14.639170 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
|
||||
12:52:14.640465 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
|
||||
12:52:14.640845 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
|
||||
12:52:14.642554 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
|
||||
12:52:14.642764 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
|
||||
12:52:14.643055 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
|
||||
4
labsim/vlan-leak-evidence/after/client.txt
Normal file
4
labsim/vlan-leak-evidence/after/client.txt
Normal file
@@ -0,0 +1,4 @@
|
||||
udhcpc: started, v1.37.0
|
||||
udhcpc: broadcasting discover
|
||||
udhcpc: broadcasting select for 172.31.3.11, server 172.31.3.252
|
||||
udhcpc: lease of 172.31.3.11 obtained from 172.31.3.252, lease time 85374
|
||||
64
labsim/vlan-leak-evidence/after/router-config.txt
Normal file
64
labsim/vlan-leak-evidence/after/router-config.txt
Normal file
@@ -0,0 +1,64 @@
|
||||
set high-availability vrrp group native address 172.31.1.1/24
|
||||
set high-availability vrrp group native hello-source-address '172.31.1.252'
|
||||
set high-availability vrrp group native interface 'bond0.1'
|
||||
set high-availability vrrp group native no-preempt
|
||||
set high-availability vrrp group native peer-address '172.31.1.253'
|
||||
set high-availability vrrp group native priority '200'
|
||||
set high-availability vrrp group native vrid '1'
|
||||
set high-availability vrrp group vlan2 address 172.31.2.1/24
|
||||
set high-availability vrrp group vlan2 hello-source-address '172.31.2.252'
|
||||
set high-availability vrrp group vlan2 interface 'bond0.2'
|
||||
set high-availability vrrp group vlan2 no-preempt
|
||||
set high-availability vrrp group vlan2 peer-address '172.31.2.253'
|
||||
set high-availability vrrp group vlan2 priority '200'
|
||||
set high-availability vrrp group vlan2 vrid '2'
|
||||
set high-availability vrrp group vlan3 address 172.31.3.1/24
|
||||
set high-availability vrrp group vlan3 hello-source-address '172.31.3.252'
|
||||
set high-availability vrrp group vlan3 interface 'bond0.3'
|
||||
set high-availability vrrp group vlan3 no-preempt
|
||||
set high-availability vrrp group vlan3 peer-address '172.31.3.253'
|
||||
set high-availability vrrp group vlan3 priority '200'
|
||||
set high-availability vrrp group vlan3 vrid '3'
|
||||
set high-availability vrrp group vlan9 address 172.31.9.1/24
|
||||
set high-availability vrrp group vlan9 hello-source-address '172.31.9.252'
|
||||
set high-availability vrrp group vlan9 interface 'bond0.9'
|
||||
set high-availability vrrp group vlan9 no-preempt
|
||||
set high-availability vrrp group vlan9 peer-address '172.31.9.253'
|
||||
set high-availability vrrp group vlan9 priority '200'
|
||||
set high-availability vrrp group vlan9 vrid '9'
|
||||
set high-availability vrrp group vlan10 address 172.31.10.1/23
|
||||
set high-availability vrrp group vlan10 hello-source-address '172.31.10.252'
|
||||
set high-availability vrrp group vlan10 interface 'bond0.10'
|
||||
set high-availability vrrp group vlan10 no-preempt
|
||||
set high-availability vrrp group vlan10 peer-address '172.31.10.253'
|
||||
set high-availability vrrp group vlan10 priority '200'
|
||||
set high-availability vrrp group vlan10 vrid '10'
|
||||
set high-availability vrrp group vlan200 address 172.31.200.1/24
|
||||
set high-availability vrrp group vlan200 hello-source-address '172.31.200.252'
|
||||
set high-availability vrrp group vlan200 interface 'bond0.200'
|
||||
set high-availability vrrp group vlan200 no-preempt
|
||||
set high-availability vrrp group vlan200 peer-address '172.31.200.253'
|
||||
set high-availability vrrp group vlan200 priority '200'
|
||||
set high-availability vrrp group vlan200 vrid '200'
|
||||
set interfaces bonding bond0 description 'api-batch-test'
|
||||
set interfaces bonding bond0 hash-policy 'layer2+3'
|
||||
set interfaces bonding bond0 lacp-rate 'fast'
|
||||
set interfaces bonding bond0 member interface 'eth0'
|
||||
set interfaces bonding bond0 member interface 'eth1'
|
||||
set interfaces bonding bond0 mode '802.3ad'
|
||||
set interfaces bonding bond0 vif 1 address '172.31.1.252/24'
|
||||
set interfaces bonding bond0 vif 1 description 'management'
|
||||
set interfaces bonding bond0 vif 2 address '172.31.2.252/24'
|
||||
set interfaces bonding bond0 vif 2 description 'k8s'
|
||||
set interfaces bonding bond0 vif 3 address '172.31.3.252/24'
|
||||
set interfaces bonding bond0 vif 3 description 'kvm'
|
||||
set interfaces bonding bond0 vif 9 address '172.31.9.252/24'
|
||||
set interfaces bonding bond0 vif 9 description 'private'
|
||||
set interfaces bonding bond0 vif 10 address '172.31.10.252/23'
|
||||
set interfaces bonding bond0 vif 10 description 'lot'
|
||||
set interfaces bonding bond0 vif 51 description 'WAN1 Vodafone-equivalent (sim ISP PPPoE)'
|
||||
set interfaces bonding bond0 vif 53 address 'dhcp'
|
||||
set interfaces bonding bond0 vif 53 description 'WAN3 10gig-equivalent (sim ISP DHCP)'
|
||||
set interfaces bonding bond0 vif 53 dhcp-options default-route-distance '210'
|
||||
set interfaces bonding bond0 vif 200 address '172.31.200.252/24'
|
||||
set interfaces bonding bond0 vif 200 description 'roomates'
|
||||
5
labsim/vlan-leak-evidence/before/capture-parent.txt
Normal file
5
labsim/vlan-leak-evidence/before/capture-parent.txt
Normal file
@@ -0,0 +1,5 @@
|
||||
12:37:08.491910 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 3, p 0, ethertype IPv4 (0x0800), 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
|
||||
12:37:08.492629 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.1.252.67 > 172.31.1.9.68: BOOTP/DHCP, Reply, length 283
|
||||
12:37:08.493628 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype 802.1Q (0x8100), length 329: vlan 3, p 0, ethertype IPv4 (0x0800), 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
|
||||
12:37:08.495587 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 3, p 0, ethertype IPv4 (0x0800), 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
|
||||
12:37:08.495946 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype 802.1Q (0x8100), length 329: vlan 3, p 0, ethertype IPv4 (0x0800), 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
|
||||
4
labsim/vlan-leak-evidence/before/capture-vif.txt
Normal file
4
labsim/vlan-leak-evidence/before/capture-vif.txt
Normal file
@@ -0,0 +1,4 @@
|
||||
12:37:08.491910 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
|
||||
12:37:08.493625 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
|
||||
12:37:08.495587 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
|
||||
12:37:08.495944 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
|
||||
4
labsim/vlan-leak-evidence/before/client.txt
Normal file
4
labsim/vlan-leak-evidence/before/client.txt
Normal file
@@ -0,0 +1,4 @@
|
||||
udhcpc: started, v1.37.0
|
||||
udhcpc: broadcasting discover
|
||||
udhcpc: broadcasting select for 172.31.3.11, server 172.31.3.252
|
||||
udhcpc: lease of 172.31.3.11 obtained from 172.31.3.252, lease time 86280
|
||||
63
labsim/vlan-leak-evidence/before/router-config.txt
Normal file
63
labsim/vlan-leak-evidence/before/router-config.txt
Normal file
@@ -0,0 +1,63 @@
|
||||
set high-availability vrrp group native address 172.31.1.1/24
|
||||
set high-availability vrrp group native hello-source-address '172.31.1.252'
|
||||
set high-availability vrrp group native interface 'bond0'
|
||||
set high-availability vrrp group native no-preempt
|
||||
set high-availability vrrp group native peer-address '172.31.1.253'
|
||||
set high-availability vrrp group native priority '200'
|
||||
set high-availability vrrp group native vrid '1'
|
||||
set high-availability vrrp group vlan2 address 172.31.2.1/24
|
||||
set high-availability vrrp group vlan2 hello-source-address '172.31.2.252'
|
||||
set high-availability vrrp group vlan2 interface 'bond0.2'
|
||||
set high-availability vrrp group vlan2 no-preempt
|
||||
set high-availability vrrp group vlan2 peer-address '172.31.2.253'
|
||||
set high-availability vrrp group vlan2 priority '200'
|
||||
set high-availability vrrp group vlan2 vrid '2'
|
||||
set high-availability vrrp group vlan3 address 172.31.3.1/24
|
||||
set high-availability vrrp group vlan3 hello-source-address '172.31.3.252'
|
||||
set high-availability vrrp group vlan3 interface 'bond0.3'
|
||||
set high-availability vrrp group vlan3 no-preempt
|
||||
set high-availability vrrp group vlan3 peer-address '172.31.3.253'
|
||||
set high-availability vrrp group vlan3 priority '200'
|
||||
set high-availability vrrp group vlan3 vrid '3'
|
||||
set high-availability vrrp group vlan9 address 172.31.9.1/24
|
||||
set high-availability vrrp group vlan9 hello-source-address '172.31.9.252'
|
||||
set high-availability vrrp group vlan9 interface 'bond0.9'
|
||||
set high-availability vrrp group vlan9 no-preempt
|
||||
set high-availability vrrp group vlan9 peer-address '172.31.9.253'
|
||||
set high-availability vrrp group vlan9 priority '200'
|
||||
set high-availability vrrp group vlan9 vrid '9'
|
||||
set high-availability vrrp group vlan10 address 172.31.10.1/23
|
||||
set high-availability vrrp group vlan10 hello-source-address '172.31.10.252'
|
||||
set high-availability vrrp group vlan10 interface 'bond0.10'
|
||||
set high-availability vrrp group vlan10 no-preempt
|
||||
set high-availability vrrp group vlan10 peer-address '172.31.10.253'
|
||||
set high-availability vrrp group vlan10 priority '200'
|
||||
set high-availability vrrp group vlan10 vrid '10'
|
||||
set high-availability vrrp group vlan200 address 172.31.200.1/24
|
||||
set high-availability vrrp group vlan200 hello-source-address '172.31.200.252'
|
||||
set high-availability vrrp group vlan200 interface 'bond0.200'
|
||||
set high-availability vrrp group vlan200 no-preempt
|
||||
set high-availability vrrp group vlan200 peer-address '172.31.200.253'
|
||||
set high-availability vrrp group vlan200 priority '200'
|
||||
set high-availability vrrp group vlan200 vrid '200'
|
||||
set interfaces bonding bond0 address '172.31.1.252/24'
|
||||
set interfaces bonding bond0 description 'api-batch-test'
|
||||
set interfaces bonding bond0 hash-policy 'layer2+3'
|
||||
set interfaces bonding bond0 lacp-rate 'fast'
|
||||
set interfaces bonding bond0 member interface 'eth0'
|
||||
set interfaces bonding bond0 member interface 'eth1'
|
||||
set interfaces bonding bond0 mode '802.3ad'
|
||||
set interfaces bonding bond0 vif 2 address '172.31.2.252/24'
|
||||
set interfaces bonding bond0 vif 2 description 'k8s'
|
||||
set interfaces bonding bond0 vif 3 address '172.31.3.252/24'
|
||||
set interfaces bonding bond0 vif 3 description 'kvm'
|
||||
set interfaces bonding bond0 vif 9 address '172.31.9.252/24'
|
||||
set interfaces bonding bond0 vif 9 description 'private'
|
||||
set interfaces bonding bond0 vif 10 address '172.31.10.252/23'
|
||||
set interfaces bonding bond0 vif 10 description 'lot'
|
||||
set interfaces bonding bond0 vif 51 description 'WAN1 Vodafone-equivalent (sim ISP PPPoE)'
|
||||
set interfaces bonding bond0 vif 53 address 'dhcp'
|
||||
set interfaces bonding bond0 vif 53 description 'WAN3 10gig-equivalent (sim ISP DHCP)'
|
||||
set interfaces bonding bond0 vif 53 dhcp-options default-route-distance '210'
|
||||
set interfaces bonding bond0 vif 200 address '172.31.200.252/24'
|
||||
set interfaces bonding bond0 vif 200 description 'roomates'
|
||||
17
labsim/vlan1-move-monitor.sh
Executable file
17
labsim/vlan1-move-monitor.sh
Executable file
@@ -0,0 +1,17 @@
|
||||
#!/bin/bash
|
||||
# Timestamped liveness log for the Management VLAN during the bond0 -> bond0.1 move.
|
||||
#
|
||||
# The question this answers is not "did it work" but "for how long was it not
|
||||
# working, and what held the VIP while it was not". Both are invisible after the
|
||||
# fact: VRRP reconverges and leaves no trace of who was master during the gap.
|
||||
#
|
||||
# ./vlan1-move-monitor.sh > /tmp/move.log &
|
||||
# Columns: time VIP-ping R1-ping R2-ping VIP-mac
|
||||
VIP="${VIP:-172.31.1.1}"; R1="${R1:-172.31.1.252}"; R2="${R2:-172.31.1.253}"
|
||||
p() { ping -c1 -W1 -n "$1" >/dev/null 2>&1 && echo up || echo DOWN; }
|
||||
while :; do
|
||||
mac="$(ip neigh show "$VIP" 2>/dev/null | awk '{for(i=1;i<=NF;i++) if($i=="lladdr") print $(i+1)}')"
|
||||
printf '%s vip=%-4s r1=%-4s r2=%-4s vipmac=%s\n' \
|
||||
"$(date +%H:%M:%S)" "$(p "$VIP")" "$(p "$R1")" "$(p "$R2")" "${mac:-none}"
|
||||
sleep 1
|
||||
done
|
||||
137
migration/MANAGEMENT-VLAN-TAGGED.md
Normal file
137
migration/MANAGEMENT-VLAN-TAGGED.md
Normal file
@@ -0,0 +1,137 @@
|
||||
# Moving Management onto a tagged VLAN
|
||||
|
||||
Rehearsed end to end in labsim on 2026-09-02. This is the fix for kea serving
|
||||
addresses from the wrong VLAN's pool.
|
||||
|
||||
## Why
|
||||
|
||||
ISC Kea [#1117](https://gitlab.isc.org/isc-projects/kea/-/issues/1117): with
|
||||
`dhcp-socket-type: raw`, a frame tagged for a sub-interface is **also** delivered
|
||||
to the parent's `AF_PACKET` socket. If the parent serves a subnet, kea answers
|
||||
from it too. Ours does — Management is the native/untagged VLAN on `bond0` while
|
||||
VLANs 2/3/9/10/200 are sub-interfaces of that same bond — so one DISCOVER on
|
||||
VLAN 3 produces two OFFERs and the *client* decides which to keep:
|
||||
|
||||
```
|
||||
bond0.3 : 192.168.3.14 correct
|
||||
bond0 : 192.168.1.28 UNTAGGED, Management pool, wrong
|
||||
```
|
||||
|
||||
The fix is to leave **no subnet on the parent**: every VLAN tagged, Management
|
||||
included, moved from `bond0` to `bond0.1`.
|
||||
|
||||
Confirmed in labsim across all six LAN VLANs: fails before, passes after.
|
||||
`labsim/labsim-vlan-leak-test.sh` is the test; evidence in
|
||||
`labsim/vlan-leak-evidence/`.
|
||||
|
||||
## What must change together
|
||||
|
||||
Per router:
|
||||
|
||||
| | from | to |
|
||||
|---|---|---|
|
||||
| address | `interfaces bonding bond0 address` | `interfaces bonding bond0 vif 1 address` |
|
||||
| firewall | `interface-group LAN interface bond0` | `... interface bond0.1` |
|
||||
| VRRP | `vrrp group native interface bond0` | `... interface bond0.1` |
|
||||
| kea | — | **restart it** (see traps) |
|
||||
|
||||
On the switch: Native VLAN = **None** on the trunk to that firewall, with VLAN 1
|
||||
added to the tagged set.
|
||||
|
||||
## The ordering constraint
|
||||
|
||||
**There is no overlap state.** An 802.1Q port always egresses its native VLAN
|
||||
untagged, so while VLAN 1 is native the router can *send* tagged VLAN 1 but can
|
||||
never *receive* it. Verified: a tagged VLAN 1 ARP sent from the switch arrived on
|
||||
`bond0` untagged and never on `bond0.1`. Configuring "native VLAN 1 **and** VLAN 1
|
||||
tagged" as a make-before-break does not work; the switch and router changes for a
|
||||
given firewall are strictly simultaneous, and that router loses Management in
|
||||
between.
|
||||
|
||||
What makes this safe anyway: **tagged and untagged Management coexist on the
|
||||
same VLAN.** One VLAN is one broadcast domain no matter how each port tags it, so
|
||||
the firewalls can be converted one at a time — verified with the primary untagged
|
||||
and the secondary already tagged, both reachable, VIP up, VLAN 1 clients fine.
|
||||
|
||||
Access ports are untouched throughout. The UniFi controller at 192.168.1.5 and
|
||||
your workstation are on access ports and never traverse the firewall trunks, so
|
||||
you keep the controller you are making the change from. Only the router being
|
||||
converted goes dark, and only until its own config lands.
|
||||
|
||||
## Procedure
|
||||
|
||||
Do the **backup** router first, then fail the VIPs over and do the other. You
|
||||
need console (JetKVM) on the router being converted — its Management SSH dies the
|
||||
moment the switch port changes.
|
||||
|
||||
For each router in turn:
|
||||
|
||||
1. Confirm the *other* router is MASTER and healthy:
|
||||
`show vrrp` and `sudo /config/vrrp-wan-health; echo $?` (must be 0).
|
||||
2. Start the monitor from a workstation on an access port:
|
||||
`labsim/vlan1-move-monitor.sh` (edit the three addresses for production).
|
||||
3. UniFi: on this firewall's trunk ports, Native VLAN → None, VLAN 1 → tagged.
|
||||
This router's Management drops now.
|
||||
4. Over the console, in one commit:
|
||||
```
|
||||
set interfaces bonding bond0 vif 1 address '192.168.1.252/24' # .253 on vyos002
|
||||
set interfaces bonding bond0 vif 1 description 'management'
|
||||
delete interfaces bonding bond0 address
|
||||
set firewall group interface-group LAN interface 'bond0.1'
|
||||
delete firewall group interface-group LAN interface 'bond0'
|
||||
set high-availability vrrp group native interface 'bond0.1'
|
||||
commit
|
||||
save
|
||||
```
|
||||
5. `sudo systemctl restart isc-kea-dhcp4-server` — see traps.
|
||||
6. Verify: Management SSH back, `show vrrp` shows `native` on `bond0.1`, and the
|
||||
leak test passes.
|
||||
|
||||
Then fail back if the VIPs moved (below), and repeat for the other router.
|
||||
|
||||
### Measured windows (labsim)
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| this router's own Management unreachable | ~27 s (the console apply) |
|
||||
| VIP `.1` unreachable, peer already converted | **0 s** |
|
||||
| VIP `.1` unreachable, converting the current MASTER | ~6 s (VRRP failover) |
|
||||
| VIP unreachable if you convert both routers before the switch | **5 min 30 s** |
|
||||
|
||||
That last row is the failure mode to avoid: with both routers untagged and the
|
||||
trunks already changed, the VIP is a black hole and **the healthy BACKUP does not
|
||||
take over**. Its `native` group stays BACKUP because the *other* VLANs still hear
|
||||
the master, and the sync group holds them together. Redundancy does not help you
|
||||
here; only ordering does.
|
||||
|
||||
## Traps
|
||||
|
||||
- **Restart kea.** VyOS does not restart it for an interface address change, so
|
||||
it keeps a raw socket bound with the old address and keeps emitting the wrong
|
||||
offers. The first post-fix test in the sim failed for this reason alone and
|
||||
looked exactly like the fix not working.
|
||||
- **`interface-group LAN`.** Moving the address without moving the group means
|
||||
Management falls outside the group, and with default-deny that is every
|
||||
management session and all VLAN 1 inter-VLAN routing, gone on commit — on a
|
||||
router you reach through itself. Use `commit-confirm` if you are not on console.
|
||||
- **The VIPs may move, and `no-preempt` keeps them moved.** Converting a router
|
||||
restarts keepalived and re-initialises *every* group, not just `native`. In one
|
||||
rehearsal the priority-100 secondary took all six VIPs and held them while the
|
||||
priority-200 primary sat at BACKUP; in another the restart was quick enough that
|
||||
nothing moved. It is non-deterministic — check afterwards, every time.
|
||||
Fail back with `restart vrrp` **on the router currently holding them**.
|
||||
- **Duplicate delivery does not stop**, and should not be read as failure. #1117
|
||||
only promises there is no longer a subnet on the parent to match. Expect two
|
||||
identical replies per DISCOVER, both from the correct pool.
|
||||
- **Both firewalls' trunks must end up the same.** If UniFi shares one port
|
||||
profile between them, changing it converts both at once and you get the 5m30s
|
||||
row above. Check before you start; use per-port overrides if it does.
|
||||
|
||||
## Not covered by the rehearsal
|
||||
|
||||
- Whether UniFi's port profile can express "no native VLAN" the way OVS can, and
|
||||
whether the two firewalls share a profile. Unverified — check on the controller.
|
||||
- Why the JetKVM consoles specifically accepted the wrong OFFER when a VLAN 3
|
||||
access port should not receive an untagged VLAN 1 frame at all. Their port
|
||||
profile likely passes VLAN 1 untagged. Worth confirming, though it does not
|
||||
change the fix.
|
||||
Reference in New Issue
Block a user