Restart-neutral VPP LB sync; deterministic AS ordering; maglevt cadence; v0.9.5

Three reliability fixes bundled with docs updates.

Restart-neutral VPP LB sync via a startup warmup window
(internal/vpp/warmup.go). Before this, a maglevd restart would
immediately issue SyncLBStateAll with every backend still in
StateUnknown — mapped through BackendEffectiveWeight to weight
0 — and VPP would black-hole all new flows until the checker's
rise counters caught up, several seconds later. The new warmup
tracker owns a process-wide state machine gated by two config
knobs: vpp.lb.startup-min-delay (default 5s) is an absolute
hands-off window during which neither the periodic sync loop
nor the per-transition reconciler touches VPP; vpp.lb.
startup-max-delay (default 30s) is the watchdog for a per-VIP
release phase that runs between the two, releasing each frontend
as soon as every backend it references reaches a non-Unknown
state. At max-delay a final SyncLBStateAll runs for any stragglers
still in Unknown. Config reload does not reset the clock. Both
delays can be set to 0 to disable the warmup entirely. The
reconciler's suppressed-during-warmup events log at DEBUG so
operators can still see them with --log-level debug. Unit tests
cover the tracker state machine, allBackendsKnown precondition,
and the zero-delay escape hatch.

Deterministic AS iteration in VPP LB sync. reconcileVIP and
recreateVIP now issue their lb_as_add_del / lb_as_set_weight
calls in numeric IP order (IPv4 before IPv6, ascending within
each family) via a new sortedIPKeys helper, instead of Go map
iteration order. VPP's LB plugin breaks per-bucket ties in the
Maglev lookup table by insertion position in its internal AS
vec, so without a stable call order two maglevd instances on
the same config could push identical AS sets into VPP in
different orders and produce divergent new-flow tables. Numeric
sort is used in preference to lexicographic so the sync log
stays human-readable: string order would place 10.0.0.10 before
10.0.0.2, and the same problem in v6. Unit tests cover empty,
single, v4/v6 numeric vs lexicographic, v4-before-v6 grouping,
a 1000-iteration stability loop against Go's randomised map
iteration, insertion-order invariance, and the desiredAS
call-site type.

maglevt interval fix. runProbeLoop used to sleep the full
jittered interval after every probe, so a 100ms --interval
with a 30ms probe actually produced a 130ms period. The sleep
now subtracts result.Duration so cadence matches the flag.
Probes that overrun clamp sleep to zero and fire the next
probe immediately without trying to catch up on missed cycles
— a slow backend doesn't get flooded with back-to-back probes
at the moment it's already struggling.

Docs. config-guide now documents flush-on-down and the new
startup-min-delay / startup-max-delay knobs; user-guide's
maglevd section explains the restart-neutrality property, the
three warmup phases, and the relevant slog lines operators
should watch for during a bounce.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

internal/vpp/lbsync.go

@@ -3,11 +3,13 @@
 package vpp
 
 import (
+	"bytes"
 	"errors"
 	"fmt"
 	"log/slog"
 	"net"
 	"regexp"
+	"sort"
 	"strconv"
 	"strings"
 
@@ -21,6 +23,80 @@ import (
 	lb_types "git.ipng.ch/ipng/vpp-maglev/internal/vpp/binapi/lb_types"
 )
 
+// sortedIPKeys returns the keys of m in ascending numeric IP order. Used
+// by the AS iteration sites in reconcileVIP and recreateVIP to make the
+// sequence of lb_as_add_del calls deterministic across maglevd instances
+// on the same config.
+//
+// Why this matters for Maglev: VPP's LB plugin stores ASes in a vec in
+// insertion order and rebuilds the new-flow lookup table by walking that
+// vec. The per-AS permutation is a pure function of the AS address (so it
+// matches across instances), but when two ASes want the same bucket on
+// the same pass the tie is broken by whichever one comes first in the
+// vec. If maglevd issues its add calls in Go map iteration order — which
+// is randomised on every run — two independent maglevd instances with
+// bit-identical configs can push the same ASes into VPP in different
+// orders, and their lookup tables end up differing in tie-broken buckets.
+// Sorting here is the first half of the fix; a matching sort inside VPP's
+// lb_vip_update_new_flow_table closes the flap-history hole (where a
+// remove+re-add after steady state reuses freed slots in the as_pool in
+// locally-visited order) and is landing in a separate commit.
+//
+// The keys at every call site are IP literals from net.IP.String(). Sort
+// order is numeric, not lexicographic: lexicographic would put 10.0.0.10
+// before 10.0.0.2 (and 2001:db8::10 before 2001:db8::2), which is
+// correct for determinism but confusing to operators reading the sync
+// log. We parse each key back into a net.IP once, compare the canonical
+// 16-byte form, and group IPv4 before IPv6 so mixed-family frontends
+// read as "v4 block, v6 block" top-to-bottom. ParseIP always succeeds
+// here because the caller built the map key via net.IP.String() in the
+// first place.
+func sortedIPKeys[V any](m map[string]V) []string {
+	type kv struct {
+		key string
+		ip  net.IP
+	}
+	pairs := make([]kv, 0, len(m))
+	for k := range m {
+		pairs = append(pairs, kv{k, net.ParseIP(k)})
+	}
+	sort.Slice(pairs, func(i, j int) bool {
+		return compareIPNumeric(pairs[i].ip, pairs[j].ip) < 0
+	})
+	out := make([]string, len(pairs))
+	for i, p := range pairs {
+		out[i] = p.key
+	}
+	return out
+}
+
+// compareIPNumeric returns <0, 0, >0 in the three-way convention. IPv4
+// sorts before IPv6. Within each family the comparison runs against the
+// canonical fixed-width byte form (4 bytes for v4, 16 bytes for v6),
+// which makes byte ordering match numeric ordering. A nil input — which
+// should not happen given sortedIPKeys's call contract — sorts before
+// any non-nil address to keep the comparator total.
+func compareIPNumeric(a, b net.IP) int {
+	a4 := a.To4()
+	b4 := b.To4()
+	switch {
+	case a == nil && b == nil:
+		return 0
+	case a == nil:
+		return -1
+	case b == nil:
+		return 1
+	case a4 != nil && b4 == nil:
+		return -1
+	case a4 == nil && b4 != nil:
+		return 1
+	case a4 != nil: // both v4
+		return bytes.Compare(a4, b4)
+	default: // both v6
+		return bytes.Compare(a.To16(), b.To16())
+	}
+}
+
 // ErrFrontendNotFound is returned by SyncLBStateVIP when the caller asks for
 // a frontend name that does not exist in the config.
 var ErrFrontendNotFound = errors.New("frontend not found in config")
@@ -241,8 +317,8 @@ func reconcileVIP(ch *loggedChannel, d desiredVIP, cur *LBVIP, curSticky bool, f
 			return err
 		}
 		st.vipAdd++
-		for _, as := range d.ASes {
-			if err := addAS(ch, d.Prefix, d.Protocol, d.Port, as); err != nil {
+		for _, addr := range sortedIPKeys(d.ASes) {
+			if err := addAS(ch, d.Prefix, d.Protocol, d.Port, d.ASes[addr]); err != nil {
 				return err
 			}
 			st.asAdd++
@@ -279,7 +355,8 @@ func reconcileVIP(ch *loggedChannel, d desiredVIP, cur *LBVIP, curSticky bool, f
 	}
 
 	// Remove ASes that are in VPP but not desired.
-	for addr, a := range curASes {
+	for _, addr := range sortedIPKeys(curASes) {
+		a := curASes[addr]
 		if _, keep := d.ASes[addr]; keep {
 			continue
 		}
@@ -290,7 +367,8 @@ func reconcileVIP(ch *loggedChannel, d desiredVIP, cur *LBVIP, curSticky bool, f
 	}
 
 	// Add new ASes, update weights on existing ones.
-	for addr, a := range d.ASes {
+	for _, addr := range sortedIPKeys(d.ASes) {
+		a := d.ASes[addr]
 		c, hit := curASes[addr]
 		if !hit {
 			if err := addAS(ch, d.Prefix, d.Protocol, d.Port, a); err != nil {
@@ -362,8 +440,8 @@ func recreateVIP(ch *loggedChannel, d desiredVIP, cur LBVIP, st *syncStats, reas
 		return err
 	}
 	st.vipAdd++
-	for _, as := range d.ASes {
-		if err := addAS(ch, d.Prefix, d.Protocol, d.Port, as); err != nil {
+	for _, addr := range sortedIPKeys(d.ASes) {
+		if err := addAS(ch, d.Prefix, d.Protocol, d.Port, d.ASes[addr]); err != nil {
 			return err
 		}
 		st.asAdd++