ipng/vpp-maglev
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Restart-neutral VPP LB sync; deterministic AS ordering; maglevt cadence; v0.9.5

Browse Source 
Three reliability fixes bundled with docs updates.

Restart-neutral VPP LB sync via a startup warmup window
(internal/vpp/warmup.go). Before this, a maglevd restart would
immediately issue SyncLBStateAll with every backend still in
StateUnknown — mapped through BackendEffectiveWeight to weight
0 — and VPP would black-hole all new flows until the checker's
rise counters caught up, several seconds later. The new warmup
tracker owns a process-wide state machine gated by two config
knobs: vpp.lb.startup-min-delay (default 5s) is an absolute
hands-off window during which neither the periodic sync loop
nor the per-transition reconciler touches VPP; vpp.lb.
startup-max-delay (default 30s) is the watchdog for a per-VIP
release phase that runs between the two, releasing each frontend
as soon as every backend it references reaches a non-Unknown
state. At max-delay a final SyncLBStateAll runs for any stragglers
still in Unknown. Config reload does not reset the clock. Both
delays can be set to 0 to disable the warmup entirely. The
reconciler's suppressed-during-warmup events log at DEBUG so
operators can still see them with --log-level debug. Unit tests
cover the tracker state machine, allBackendsKnown precondition,
and the zero-delay escape hatch.

Deterministic AS iteration in VPP LB sync. reconcileVIP and
recreateVIP now issue their lb_as_add_del / lb_as_set_weight
calls in numeric IP order (IPv4 before IPv6, ascending within
each family) via a new sortedIPKeys helper, instead of Go map
iteration order. VPP's LB plugin breaks per-bucket ties in the
Maglev lookup table by insertion position in its internal AS
vec, so without a stable call order two maglevd instances on
the same config could push identical AS sets into VPP in
different orders and produce divergent new-flow tables. Numeric
sort is used in preference to lexicographic so the sync log
stays human-readable: string order would place 10.0.0.10 before
10.0.0.2, and the same problem in v6. Unit tests cover empty,
single, v4/v6 numeric vs lexicographic, v4-before-v6 grouping,
a 1000-iteration stability loop against Go's randomised map
iteration, insertion-order invariance, and the desiredAS
call-site type.

maglevt interval fix. runProbeLoop used to sleep the full
jittered interval after every probe, so a 100ms --interval
with a 30ms probe actually produced a 130ms period. The sleep
now subtracts result.Duration so cadence matches the flag.
Probes that overrun clamp sleep to zero and fire the next
probe immediately without trying to catch up on missed cycles
— a slow backend doesn't get flooded with back-to-back probes
at the moment it's already struggling.

Docs. config-guide now documents flush-on-down and the new
startup-min-delay / startup-max-delay knobs; user-guide's
maglevd section explains the restart-neutrality property, the
three warmup phases, and the relevant slog lines operators
should watch for during a bounce.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Pim van Pelt
2026-04-15 11:25:53 +02:00
parent 695ebc4bd1
commit 6d78921edd
10 changed files with 1257 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files

204
internal/vpp/lbsync_test.go
View File

@@ -438,3 +438,207 @@ func TestDesiredFromFrontendSharedBackend(t *testing.T) {
})
}
}
// TestSortedIPKeysDeterministic pins the iteration-order helper that
// reconcileVIP and recreateVIP use to sequence their lb_as_add_del
// calls. The Maglev lookup table in VPP's LB plugin breaks per-bucket
// ties by the order ASes sit in its internal vec, which is just the
// order maglevd issued add calls — so if this helper ever stops
// returning a total, stable ordering, two independent maglevd
// instances on the same config can silently program different
// new-flow tables.
//
// Sort order is numeric (by the parsed net.IP), not lexicographic.
// The specific cases that a string sort would get wrong and a
// numeric sort must get right:
//
// - 10.0.0.2 < 10.0.0.10 (string sort puts "10" before "2")
// - 2001:db8::2 < 2001:db8::10 (same issue in v6)
// - all IPv4 before all IPv6 (operator-friendly grouping)
func TestSortedIPKeysDeterministic(t *testing.T) {
t.Run("empty", func(t *testing.T) {
got := sortedIPKeys(map[string]int{})
if len(got) != 0 {
t.Errorf("empty map: got %v, want []", got)
}
})
t.Run("single entry", func(t *testing.T) {
got := sortedIPKeys(map[string]int{"10.0.0.1": 1})
if len(got) != 1 || got[0] != "10.0.0.1" {
t.Errorf("got %v, want [10.0.0.1]", got)
}
})
t.Run("v4 numeric order beats string order", func(t *testing.T) {
// The headline bug: "10.0.0.10" < "10.0.0.2" lexicographically
// because '1' < '2'. Numeric sort must place 2 before 10.
m := map[string]int{
"10.0.0.10": 1,
"10.0.0.2": 2,
"10.0.0.1": 3,
"10.0.0.11": 4,
}
got := sortedIPKeys(m)
want := []string{"10.0.0.1", "10.0.0.2", "10.0.0.10", "10.0.0.11"}
if len(got) != len(want) {
t.Fatalf("got %v, want %v", got, want)
}
for i := range want {
if got[i] != want[i] {
t.Errorf("pos %d: got %q, want %q", i, got[i], want[i])
}
}
})
t.Run("v6 numeric order beats string order", func(t *testing.T) {
// Same bug in v6: "2001:db8::10" < "2001:db8::2" lexicographically.
// The To16() canonical byte form handles both compressed and
// expanded forms correctly.
m := map[string]int{
"2001:db8::10": 1,
"2001:db8::2": 2,
"2001:db8::1": 3,
}
got := sortedIPKeys(m)
want := []string{"2001:db8::1", "2001:db8::2", "2001:db8::10"}
for i := range want {
if got[i] != want[i] {
t.Errorf("pos %d: got %q, want %q", i, got[i], want[i])
}
}
})
t.Run("v4 before v6", func(t *testing.T) {
// Mixed-family frontends: the operator-friendly order is
// the v4 block before the v6 block, each sorted numerically
// within its family.
m := map[string]int{
"2001:db8::1": 1,
"10.0.0.2": 2,
"10.0.0.1": 3,
"fe80::1": 4,
"192.168.0.1": 5,
}
got := sortedIPKeys(m)
want := []string{
"10.0.0.1", "10.0.0.2", "192.168.0.1",
"2001:db8::1", "fe80::1",
}
if len(got) != len(want) {
t.Fatalf("got %v, want %v", got, want)
}
for i := range want {
if got[i] != want[i] {
t.Errorf("pos %d: got %q, want %q", i, got[i], want[i])
}
}
})
t.Run("repeated calls produce identical sequence", func(t *testing.T) {
// Core determinism property: Go's map iteration is randomised,
// but sortedIPKeys must normalise it. Run the helper many
// times and compare every result to the first — if the
// normalisation ever breaks we'll see a divergence well within
// the loop count.
m := map[string]int{
"10.0.0.5": 1, "10.0.0.3": 2, "10.0.0.11": 3,
"10.0.0.2": 4, "10.0.0.4": 5, "10.0.0.20": 6,
}
first := sortedIPKeys(m)
for i := 0; i < 1000; i++ {
got := sortedIPKeys(m)
if len(got) != len(first) {
t.Fatalf("iter %d: length drift: got %v, first %v", i, got, first)
}
for j := range first {
if got[j] != first[j] {
t.Fatalf("iter %d pos %d: got %q, first %q", i, j, got[j], first[j])
}
}
}
})
t.Run("insertion order does not matter", func(t *testing.T) {
// A map built by inserting keys in ascending order must
// produce the same result as one built in descending order.
// Both go through the same normalisation.
asc := map[string]int{}
for _, k := range []string{"10.0.0.1", "10.0.0.2", "10.0.0.3", "10.0.0.10", "10.0.0.11"} {
asc[k] = 0
}
desc := map[string]int{}
for _, k := range []string{"10.0.0.11", "10.0.0.10", "10.0.0.3", "10.0.0.2", "10.0.0.1"} {
desc[k] = 0
}
gotAsc := sortedIPKeys(asc)
gotDesc := sortedIPKeys(desc)
if len(gotAsc) != len(gotDesc) {
t.Fatalf("length mismatch: asc %v, desc %v", gotAsc, gotDesc)
}
for i := range gotAsc {
if gotAsc[i] != gotDesc[i] {
t.Errorf("pos %d: asc %q, desc %q", i, gotAsc[i], gotDesc[i])
}
}
})
t.Run("desiredAS map", func(t *testing.T) {
// Exercise the actual call-site type: map[string]desiredAS.
// If the generic helper ever loses its type parameterisation
// this catches it at compile time (the call would fail).
m := map[string]desiredAS{
"10.0.0.9": {Address: net.ParseIP("10.0.0.9"), Weight: 100},
"10.0.0.11": {Address: net.ParseIP("10.0.0.11"), Weight: 100},
"10.0.0.5": {Address: net.ParseIP("10.0.0.5"), Weight: 50},
"10.0.0.1": {Address: net.ParseIP("10.0.0.1"), Weight: 25},
}
got := sortedIPKeys(m)
want := []string{"10.0.0.1", "10.0.0.5", "10.0.0.9", "10.0.0.11"}
for i := range want {
if got[i] != want[i] {
t.Errorf("pos %d: got %q, want %q", i, got[i], want[i])
}
}
})
}
// TestCompareIPNumeric pins the ordering comparator that sortedIPKeys
// delegates to. Split out so the v4/v6 boundary and nil-safety logic
// have named failure modes rather than being buried in the map-based
// subtests.
func TestCompareIPNumeric(t *testing.T) {
cases := []struct {
name string
a, b net.IP
want int // -1, 0, +1 (sign of compareIPNumeric)
}{
{"v4 numeric asc", net.ParseIP("10.0.0.2"), net.ParseIP("10.0.0.10"), -1},
{"v4 numeric desc", net.ParseIP("10.0.0.10"), net.ParseIP("10.0.0.2"), 1},
{"v4 equal", net.ParseIP("10.0.0.1"), net.ParseIP("10.0.0.1"), 0},
{"v6 numeric asc", net.ParseIP("2001:db8::2"), net.ParseIP("2001:db8::10"), -1},
{"v6 numeric desc", net.ParseIP("2001:db8::10"), net.ParseIP("2001:db8::2"), 1},
{"v4 before v6", net.ParseIP("192.168.0.1"), net.ParseIP("2001:db8::1"), -1},
{"v6 after v4", net.ParseIP("2001:db8::1"), net.ParseIP("192.168.0.1"), 1},
{"nil before v4", nil, net.ParseIP("10.0.0.1"), -1},
{"v4 after nil", net.ParseIP("10.0.0.1"), nil, 1},
{"nil equal nil", nil, nil, 0},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
got := compareIPNumeric(tc.a, tc.b)
sign := func(x int) int {
switch {
case x < 0:
return -1
case x > 0:
return 1
}
return 0
}
if sign(got) != tc.want {
t.Errorf("got %d (sign %d), want sign %d", got, sign(got), tc.want)
}
})
}
}