labsim: prove the tagged-Management fix for kea's wrong-pool offers
Kea #1117: with dhcp-socket-type raw, a frame tagged for a sub-interface is also delivered to the parent's AF_PACKET socket, and if the parent serves a subnet kea answers from it too. Management being the native VLAN on bond0 is what gives the parent that subnet. One DISCOVER on VLAN 3 produced two OFFERs, and in the captures here the WRONG one arrives first as often as not -- which is why this looked device-dependent rather than like a server bug. labsim-vlan-leak-test.sh reproduces it and scores the SERVER's offers, not the client's choice; a client picking correctly is how this hid. Fails on the old shape, passes on the new one across all six LAN VLANs. Three things the rehearsal caught that reasoning had not: - kea keeps its old raw socket. VyOS does not restart it for an interface address change, so the first post-fix test failed and looked exactly like the fix not working. - interface-group LAN names the bare bond0. Moving the address without moving the group drops every management session under default-deny. - there is no make-before-break. A port always egresses its native VLAN untagged, so while VLAN 1 is native the router can send tagged VLAN 1 but never receive it -- verified, the ARP landed on bond0 untagged. What makes the cutover safe anyway is that tagged and untagged Management coexist, so the firewalls convert one at a time: 0s of VIP downtime, versus 5m30s if both routers go before the switch does. In that state the healthy BACKUP does NOT take over -- the sync group holds native BACKUP because the other VLANs still hear the master. Also fixes two ways the sim was lying. ovs_bond_router compared only the trunk VLAN list on re-runs, so a VM restart left the bond holding taps that no longer existed while the real ones sat in the bridge unbonded -- labsim-vyos2 had no LACP at all. And the tap count included the primary's libvirt-NAT scaffold NIC, so the primary's bond was skipped outright. Runbook: migration/MANAGEMENT-VLAN-TAGGED.md Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
This commit is contained in:
Michal
141
labsim/ovs.sh
141
labsim/ovs.sh
@@ -17,6 +17,21 @@ OVS_BR="${OVS_BR:-ovs-labsim}"
|
||||
OVS_NET="${OVS_NET:-labsim-ovs}" # libvirt network wrapping the bridge
|
||||
LAG_NAME="${LAG_NAME:-lag-vyos}"
|
||||
|
||||
# Native (untagged) VLAN on the trunks to the routers. Empty means NONE: every
|
||||
# VLAN, Management included, is tagged.
|
||||
#
|
||||
# This is not a style choice. A native VLAN is what puts a subnet on the bond
|
||||
# PARENT (`bond0`) while every other VLAN lives on a sub-interface of it. With
|
||||
# `dhcp-socket-type: raw`, kea then receives each tagged frame TWICE -- once on
|
||||
# the sub-interface and once on the parent -- and answers from the parent's pool
|
||||
# as well, so a client on VLAN 3 is offered a Management address and picks
|
||||
# whichever reply arrives first (ISC Kea #1117).
|
||||
#
|
||||
# Set LABSIM_NATIVE_VLAN=1 to restore the old shape and reproduce the bug:
|
||||
# LABSIM_NATIVE_VLAN=1 ./router-up.sh && ./labsim-vlan-leak-test.sh # FAIL
|
||||
# ./router-up.sh && ./labsim-vlan-leak-test.sh # PASS
|
||||
NATIVE_VLAN="${LABSIM_NATIVE_VLAN:-}"
|
||||
|
||||
ovs() { sudo ovs-vsctl "$@"; }
|
||||
|
||||
ovs_require() {
|
||||
@@ -25,6 +40,10 @@ ovs_require() {
|
||||
|| die "could not start openvswitch"
|
||||
}
|
||||
|
||||
# A comma-separated VLAN list, numerically sorted, for comparing two lists that
|
||||
# came from different places and need not agree on order.
|
||||
vlan_sorted() { echo "$1" | tr ',' '\n' | grep -v '^$' | sort -n | paste -sd, -; }
|
||||
|
||||
# All VLAN ids from the config, comma separated — used for trunk ports.
|
||||
vlan_id_list() {
|
||||
local ids=()
|
||||
@@ -72,18 +91,23 @@ ovs_define_libvirt_net() {
|
||||
"
|
||||
done
|
||||
|
||||
# Trunk: VLAN 1 native/untagged, everything else tagged — the production
|
||||
# shape. libvirt expresses this declaratively via nativeMode='untagged'
|
||||
# (see libvirt formatnetwork.html), so it does not need fixing up by hand.
|
||||
# It also matters functionally: LACPDUs are untagged, and a trunk with no
|
||||
# native VLAN has nowhere to put them.
|
||||
# Trunk: every VLAN tagged, and by default NO native VLAN (see NATIVE_VLAN at
|
||||
# the top of this file for why -- it is the kea #1117 fix, not tidiness).
|
||||
# libvirt expresses a native VLAN declaratively via nativeMode='untagged'
|
||||
# (see libvirt formatnetwork.html), so it needs no fixing up by hand.
|
||||
#
|
||||
# The worry that a trunk with no native VLAN has nowhere to put LACPDUs is
|
||||
# unfounded, and was tested rather than reasoned about: with vlan_mode=trunk
|
||||
# and no tag, `ovs-appctl bond/show` still reports lacp_status: negotiated
|
||||
# with both members enabled. LACPDUs are slow-protocol frames handled per
|
||||
# member, below the VLAN layer.
|
||||
local trunk=" <portgroup name='trunk'>
|
||||
<vlan trunk='yes'>
|
||||
"
|
||||
for entry in "${SELECTED[@]}"; do
|
||||
IFS=: read -r vid _n _p _r <<<"$entry"
|
||||
if [ "$vid" = "1" ]; then
|
||||
trunk+=" <tag id='1' nativeMode='untagged'/>
|
||||
if [ -n "$NATIVE_VLAN" ] && [ "$vid" = "$NATIVE_VLAN" ]; then
|
||||
trunk+=" <tag id='${vid}' nativeMode='untagged'/>
|
||||
"
|
||||
else
|
||||
trunk+=" <tag id='${vid}'/>
|
||||
@@ -116,29 +140,59 @@ ${pg}${trunk}</network>"
|
||||
ovs_bond_router() {
|
||||
local vm="$1"
|
||||
local taps
|
||||
# NB: domiflist indents its rows, so anchor on the FIELD not the line —
|
||||
# /^vnet/ silently matches nothing and the bond never gets built.
|
||||
taps="$(virsh_q domiflist "$vm" 2>/dev/null | awk '$1 ~ /^vnet/ {print $1}')"
|
||||
# Two filters, both load-bearing:
|
||||
#
|
||||
# $1 ~ /^vnet/ -- domiflist indents its rows, so anchor on the FIELD, not
|
||||
# the line. /^vnet/ silently matches nothing and the bond never gets built.
|
||||
#
|
||||
# $3 == OVS_NET -- count only the taps on the sim fabric. The primary also
|
||||
# carries a libvirt-NAT scaffold NIC (see --drop-scaffold in the README), so
|
||||
# an unfiltered count is 3, and this function's "expected 2" guard then
|
||||
# skipped the primary's bond entirely while reporting only a warning.
|
||||
taps="$(virsh_q domiflist "$vm" 2>/dev/null \
|
||||
| awk -v net="$OVS_NET" '$1 ~ /^vnet/ && $3 == net {print $1}')"
|
||||
local count; count="$(echo "$taps" | grep -c .)"
|
||||
[ "$count" -eq 2 ] || { warn "router $vm has $count tap(s), expected 2 — skipping bond"; return 1; }
|
||||
|
||||
# Already bonded? Re-runs must still reconcile the VLAN list: adding a VLAN to
|
||||
# vlans.conf and finding the bond unchanged is exactly how a VLAN silently
|
||||
# fails to reach a router -- interface present, tag missing, frames dropped by
|
||||
# the switch. Returning early here once cost real debugging time.
|
||||
if ovs list-ports "$OVS_BR" 2>/dev/null | grep -qx "$LAG_NAME"; then
|
||||
local want; want="$(vlan_id_list | tr ',' '\n' | grep -vx 1 | paste -sd, -)"
|
||||
local have; have="$(ovs get port "$LAG_NAME" trunks 2>/dev/null | tr -d '[] ')"
|
||||
if [ "$want" != "$have" ]; then
|
||||
log "bond $LAG_NAME trunk drift: [$have] -> [$want]; updating"
|
||||
ovs set port "$LAG_NAME" trunks="$want"
|
||||
else
|
||||
log "LACP bond $LAG_NAME already present, trunk correct"
|
||||
fi
|
||||
return 0
|
||||
fi
|
||||
[ "$count" -eq 2 ] \
|
||||
|| { warn "router $vm has $count tap(s) on $OVS_NET, expected 2 — skipping bond"; return 1; }
|
||||
|
||||
local t1 t2; t1="$(echo "$taps" | sed -n 1p)"; t2="$(echo "$taps" | sed -n 2p)"
|
||||
local want; want="$(vlan_id_list)"
|
||||
|
||||
# Already bonded? Re-runs must still reconcile BOTH the VLAN list and the
|
||||
# membership, and each has drawn blood:
|
||||
#
|
||||
# VLANs -- adding a VLAN to vlans.conf and finding the bond unchanged is how
|
||||
# a VLAN silently fails to reach a router: interface present, tag missing,
|
||||
# frames dropped by the switch.
|
||||
#
|
||||
# MEMBERS -- restarting the VM recreates its taps with NEW names, leaving the
|
||||
# bond holding two interfaces that no longer exist. `list-ports` still shows
|
||||
# the bond, so this early return declared success while the router's real
|
||||
# taps sat in the bridge as two INDEPENDENT ports, each carrying libvirt's
|
||||
# own portgroup VLAN config. That is how labsim-vyos2 ran for weeks with no
|
||||
# LACP at all and a native VLAN nobody had asked for -- and it is invisible
|
||||
# until you change the trunk and only one router follows.
|
||||
if ovs list-ports "$OVS_BR" 2>/dev/null | grep -qx "$LAG_NAME"; then
|
||||
local members; members="$(ovs-appctl-members)"
|
||||
if [ "$members" != "$(printf '%s\n%s' "$t1" "$t2" | sort | paste -sd, -)" ]; then
|
||||
warn "bond $LAG_NAME holds stale members [$members], VM has [$t1,$t2] — rebuilding"
|
||||
ovs --if-exists del-port "$OVS_BR" "$LAG_NAME"
|
||||
else
|
||||
# Compare as SETS. vlan_id_list yields config order (1,2,3,9,10,200,51,53)
|
||||
# while OVS returns its own sorted order, so a raw string compare reports
|
||||
# drift on every run and rewrites a trunk that was already correct.
|
||||
local have; have="$(ovs get port "$LAG_NAME" trunks 2>/dev/null | tr -d '[] ')"
|
||||
if [ "$(vlan_sorted "$want")" != "$(vlan_sorted "$have")" ]; then
|
||||
log "bond $LAG_NAME trunk drift: [$have] -> [$want]; updating"
|
||||
ovs set port "$LAG_NAME" trunks="$want"
|
||||
else
|
||||
log "LACP bond $LAG_NAME already present, trunk correct"
|
||||
fi
|
||||
ovs_set_native "$LAG_NAME"
|
||||
return 0
|
||||
fi
|
||||
fi
|
||||
|
||||
log "bonding $t1 + $t2 into $LAG_NAME (LACP active, balance-tcp)"
|
||||
ovs del-port "$OVS_BR" "$t1" 2>/dev/null || true
|
||||
ovs del-port "$OVS_BR" "$t2" 2>/dev/null || true
|
||||
@@ -152,15 +206,36 @@ ovs_bond_router() {
|
||||
# LACPDUs. Falling back to active-backup brings the links up so negotiation
|
||||
# can start.
|
||||
#
|
||||
# native-untagged + tag=1 carries the untagged LACPDUs and the management
|
||||
# VLAN, matching production. libvirt's portgroup VLAN config does NOT apply
|
||||
# here — the bond is a port libvirt never created — so set it inline.
|
||||
local tagged; tagged="$(vlan_id_list | tr ',' '\n' | grep -vx 1 | paste -sd, -)"
|
||||
# The VLAN mode is set inline: libvirt's portgroup config does NOT apply here,
|
||||
# because the bond is a port libvirt never created.
|
||||
ovs add-bond "$OVS_BR" "$LAG_NAME" "$t1" "$t2" \
|
||||
lacp=active bond_mode=balance-tcp \
|
||||
vlan_mode=native-untagged tag=1 trunks="$tagged" \
|
||||
lacp=active bond_mode=balance-tcp trunks="$want" \
|
||||
-- set port "$LAG_NAME" other_config:lacp-time=fast \
|
||||
-- set port "$LAG_NAME" other_config:lacp-fallback-ab=true
|
||||
ovs_set_native "$LAG_NAME"
|
||||
}
|
||||
|
||||
# The bond's current members, sorted and comma-joined, or empty if the bond does
|
||||
# not resolve at all (which is itself the stale case worth rebuilding for).
|
||||
ovs-appctl-members() {
|
||||
sudo ovs-appctl bond/show "$LAG_NAME" 2>/dev/null \
|
||||
| awk '/^member /{gsub(/:/,"",$2); print $2}' | sort | paste -sd, -
|
||||
}
|
||||
|
||||
# Apply NATIVE_VLAN to a trunk port.
|
||||
#
|
||||
# `tag` MUST be removed, not merely left alone, when there is no native VLAN.
|
||||
# Setting vlan_mode=trunk while a stale `tag` remains looks correct in
|
||||
# `ovs-vsctl list port` -- it prints vlan_mode: trunk right next to tag: 1 --
|
||||
# but the port keeps egressing that VLAN untagged. Half an hour went into
|
||||
# "the router is ignoring the trunk change" before the tag was the answer.
|
||||
ovs_set_native() {
|
||||
local port="$1"
|
||||
if [ -n "$NATIVE_VLAN" ]; then
|
||||
ovs set port "$port" vlan_mode=native-untagged tag="$NATIVE_VLAN"
|
||||
else
|
||||
ovs set port "$port" vlan_mode=trunk -- clear port "$port" tag
|
||||
fi
|
||||
}
|
||||
|
||||
ovs_bond_status() {
|
||||
|
||||
Reference in New Issue
Block a user