Restart-neutral VPP LB sync; deterministic AS ordering; maglevt cadence; v0.9.5

Three reliability fixes bundled with docs updates.

Restart-neutral VPP LB sync via a startup warmup window
(internal/vpp/warmup.go). Before this, a maglevd restart would
immediately issue SyncLBStateAll with every backend still in
StateUnknown — mapped through BackendEffectiveWeight to weight
0 — and VPP would black-hole all new flows until the checker's
rise counters caught up, several seconds later. The new warmup
tracker owns a process-wide state machine gated by two config
knobs: vpp.lb.startup-min-delay (default 5s) is an absolute
hands-off window during which neither the periodic sync loop
nor the per-transition reconciler touches VPP; vpp.lb.
startup-max-delay (default 30s) is the watchdog for a per-VIP
release phase that runs between the two, releasing each frontend
as soon as every backend it references reaches a non-Unknown
state. At max-delay a final SyncLBStateAll runs for any stragglers
still in Unknown. Config reload does not reset the clock. Both
delays can be set to 0 to disable the warmup entirely. The
reconciler's suppressed-during-warmup events log at DEBUG so
operators can still see them with --log-level debug. Unit tests
cover the tracker state machine, allBackendsKnown precondition,
and the zero-delay escape hatch.

Deterministic AS iteration in VPP LB sync. reconcileVIP and
recreateVIP now issue their lb_as_add_del / lb_as_set_weight
calls in numeric IP order (IPv4 before IPv6, ascending within
each family) via a new sortedIPKeys helper, instead of Go map
iteration order. VPP's LB plugin breaks per-bucket ties in the
Maglev lookup table by insertion position in its internal AS
vec, so without a stable call order two maglevd instances on
the same config could push identical AS sets into VPP in
different orders and produce divergent new-flow tables. Numeric
sort is used in preference to lexicographic so the sync log
stays human-readable: string order would place 10.0.0.10 before
10.0.0.2, and the same problem in v6. Unit tests cover empty,
single, v4/v6 numeric vs lexicographic, v4-before-v6 grouping,
a 1000-iteration stability loop against Go's randomised map
iteration, insertion-order invariance, and the desiredAS
call-site type.

maglevt interval fix. runProbeLoop used to sleep the full
jittered interval after every probe, so a 100ms --interval
with a 30ms probe actually produced a 130ms period. The sleep
now subtracts result.Duration so cadence matches the flag.
Probes that overrun clamp sleep to zero and fire the next
probe immediately without trying to catch up on missed cycles
— a slow backend doesn't get flooded with back-to-back probes
at the moment it's already struggling.

Docs. config-guide now documents flush-on-down and the new
startup-min-delay / startup-max-delay knobs; user-guide's
maglevd section explains the restart-neutrality property, the
three warmup phases, and the relevant slog lines operators
should watch for during a bounce.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

internal/vpp/client.go

@@ -66,6 +66,12 @@ type Client struct {
 	// lbStatsLoop. Published as an immutable slice via atomic.Pointer so
 	// Prometheus scrapes (metrics.Collector.Collect) don't take any lock.
 	lbStatsSnap atomic.Pointer[[]metrics.VIPStatEntry]
+
+	// warmup gates every VPP LB sync call (both periodic and event-
+	// driven) during the first StartupMaxDelay seconds after Client
+	// construction. See warmup.go for the state machine. Process-wide,
+	// not per-connection: reconnects do not re-enter warmup.
+	warmup *warmupTracker
 }
 
 // SetStateSource attaches a live config + health state source. When set, the
@@ -86,9 +92,18 @@ func (c *Client) getStateSource() StateSource {
 	return c.stateSrc
 }
 
-// New creates a Client for the given socket paths.
+// New creates a Client for the given socket paths. The warmup tracker's
+// clock starts here — the restart-neutrality window is measured from the
+// moment the Client is constructed, which in practice is a few tens of
+// milliseconds after process start (see cmd/maglevd/main.go startup
+// sequence). If main.go ever grows a long-running initialisation step
+// before vpp.New(), the warmup clock should be moved accordingly.
 func New(apiAddr, statsAddr string) *Client {
-	return &Client{apiAddr: apiAddr, statsAddr: statsAddr}
+	return &Client{
+		apiAddr:   apiAddr,
+		statsAddr: statsAddr,
+		warmup:    newWarmupTracker(),
+	}
 }
 
 // Run connects to VPP and maintains the connection until ctx is cancelled.
@@ -165,19 +180,48 @@ func (c *Client) Run(ctx context.Context) {
 	}
 }
 
-// lbSyncLoop periodically runs SyncLBStateAll to catch drift between the
-// maglev config and the VPP dataplane. The first run happens immediately
-// on loop start (VPP has just connected, so any pre-existing state needs
-// reconciliation). Subsequent runs fire every cfg.VPP.LB.SyncInterval.
-// Exits when ctx is cancelled.
+// lbSyncLoop drives the periodic VPP LB sync. On first entry (after the
+// first successful VPP connect) it runs the warmup phase via runWarmup,
+// which enforces the restart-neutrality window and handles the first full
+// sync itself. Subsequent reconnect entries find warmup.allDone == true
+// and skip straight to the periodic ticker. Exits when ctx is cancelled.
+//
+// The warmup phase is intentionally run from inside this loop rather
+// than from Run: it needs the state source registered (which happens
+// only after SetStateSource) and it wants to be cancelled by the same
+// connCtx that cancels the stats loop on disconnect, so a VPP drop
+// during warmup doesn't leak a goroutine.
 func (c *Client) lbSyncLoop(ctx context.Context) {
 	src := c.getStateSource()
 	if src == nil {
 		return // no state source registered; nothing to sync
 	}
 
-	// next-run timestamp starts at "now" so the first tick is immediate.
-	next := time.Now()
+	// Warmup phase: runs once per process. On the first successful
+	// VPP connect, runWarmup handles the entire window (min-delay
+	// hands-off, per-VIP release phase, final SyncLBStateAll at
+	// max-delay) and calls finishAll before returning. On any
+	// reconnect after that, the gate is already open and we skip
+	// straight to the periodic ticker. A VPP drop mid-warmup
+	// returns from runWarmup via ctx.Done without closing the gate;
+	// the next reconnect re-enters runWarmup, which re-reads the
+	// process-relative clock and picks up wherever it left off.
+	if !c.warmup.isAllDone() {
+		c.runWarmup(ctx)
+		if ctx.Err() != nil {
+			return
+		}
+	}
+
+	cfg := src.Config()
+	if cfg == nil {
+		return
+	}
+	interval := cfg.VPP.LB.SyncInterval
+	if interval <= 0 {
+		interval = defaultLBSyncInterval
+	}
+	next := time.Now().Add(interval)
 	for {
 		wait := time.Until(next)
 		if wait < 0 {
@@ -189,12 +233,12 @@ func (c *Client) lbSyncLoop(ctx context.Context) {
 		case <-time.After(wait):
 		}
 
-		cfg := src.Config()
+		cfg = src.Config()
 		if cfg == nil {
 			next = time.Now().Add(defaultLBSyncInterval)
 			continue
 		}
-		interval := cfg.VPP.LB.SyncInterval
+		interval = cfg.VPP.LB.SyncInterval
 		if interval <= 0 {
 			interval = defaultLBSyncInterval
 		}