Restart-neutral VPP LB sync; deterministic AS ordering; maglevt cadence; v0.9.5

Three reliability fixes bundled with docs updates.

Restart-neutral VPP LB sync via a startup warmup window
(internal/vpp/warmup.go). Before this, a maglevd restart would
immediately issue SyncLBStateAll with every backend still in
StateUnknown — mapped through BackendEffectiveWeight to weight
0 — and VPP would black-hole all new flows until the checker's
rise counters caught up, several seconds later. The new warmup
tracker owns a process-wide state machine gated by two config
knobs: vpp.lb.startup-min-delay (default 5s) is an absolute
hands-off window during which neither the periodic sync loop
nor the per-transition reconciler touches VPP; vpp.lb.
startup-max-delay (default 30s) is the watchdog for a per-VIP
release phase that runs between the two, releasing each frontend
as soon as every backend it references reaches a non-Unknown
state. At max-delay a final SyncLBStateAll runs for any stragglers
still in Unknown. Config reload does not reset the clock. Both
delays can be set to 0 to disable the warmup entirely. The
reconciler's suppressed-during-warmup events log at DEBUG so
operators can still see them with --log-level debug. Unit tests
cover the tracker state machine, allBackendsKnown precondition,
and the zero-delay escape hatch.

Deterministic AS iteration in VPP LB sync. reconcileVIP and
recreateVIP now issue their lb_as_add_del / lb_as_set_weight
calls in numeric IP order (IPv4 before IPv6, ascending within
each family) via a new sortedIPKeys helper, instead of Go map
iteration order. VPP's LB plugin breaks per-bucket ties in the
Maglev lookup table by insertion position in its internal AS
vec, so without a stable call order two maglevd instances on
the same config could push identical AS sets into VPP in
different orders and produce divergent new-flow tables. Numeric
sort is used in preference to lexicographic so the sync log
stays human-readable: string order would place 10.0.0.10 before
10.0.0.2, and the same problem in v6. Unit tests cover empty,
single, v4/v6 numeric vs lexicographic, v4-before-v6 grouping,
a 1000-iteration stability loop against Go's randomised map
iteration, insertion-order invariance, and the desiredAS
call-site type.

maglevt interval fix. runProbeLoop used to sleep the full
jittered interval after every probe, so a 100ms --interval
with a 30ms probe actually produced a 130ms period. The sleep
now subtracts result.Duration so cadence matches the flag.
Probes that overrun clamp sleep to zero and fire the next
probe immediately without trying to catch up on missed cycles
— a slow backend doesn't get flooded with back-to-back probes
at the moment it's already struggling.

Docs. config-guide now documents flush-on-down and the new
startup-min-delay / startup-max-delay knobs; user-guide's
maglevd section explains the restart-neutrality property, the
three warmup phases, and the relevant slog lines operators
should watch for during a bounce.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

internal/config/config.go

@@ -59,6 +59,30 @@ type VPPLBConfig struct {
 	// removed from the table. Must be between 1 and 120 seconds inclusive.
 	// Defaults to 40s.
 	FlowTimeout time.Duration
+
+	// StartupMinDelay is the absolute hands-off window at the start of
+	// the maglevd process. For the first StartupMinDelay seconds after
+	// startup, the VPP LB sync path makes no calls to VPP at all —
+	// neither the periodic SyncLBStateAll loop nor the per-transition
+	// SyncLBStateVIP path from the reconciler. This gives a restarting
+	// maglevd a chance to complete its first few probes before any VPP
+	// state is touched, so a bounce does not black-hole traffic while
+	// the checker is still warming up. Default 5s. Set to 0 together
+	// with StartupMaxDelay to disable the warmup entirely and sync VPP
+	// immediately on startup.
+	StartupMinDelay time.Duration
+
+	// StartupMaxDelay is the watchdog for the per-VIP release phase.
+	// Between StartupMinDelay and StartupMaxDelay, each VIP is released
+	// (and one SyncLBStateVIP runs against it) as soon as every backend
+	// it references has reached a non-Unknown state. At StartupMaxDelay
+	// the warmup driver unconditionally runs SyncLBStateAll to handle
+	// any stragglers whose backends are still Unknown — those get
+	// programmed with whatever weight their current state maps to,
+	// which for a still-Unknown backend is 0. Must be >= StartupMinDelay.
+	// Default 30s. Set to 0 together with StartupMinDelay to disable
+	// the warmup.
+	StartupMaxDelay time.Duration
 }
 
 // HealthCheck describes how to probe a backend.
@@ -161,6 +185,8 @@ type rawVPPLBCfg struct {
 	IPv6SrcAddress       string  `yaml:"ipv6-src-address"`
 	StickyBucketsPerCore *uint32 `yaml:"sticky-buckets-per-core"` // default 65536
 	FlowTimeout          string  `yaml:"flow-timeout"`            // Go duration; default 40s, [1-120]s
+	StartupMinDelay      *string `yaml:"startup-min-delay"`       // Go duration; default 5s; 0 disables
+	StartupMaxDelay      *string `yaml:"startup-max-delay"`       // Go duration; default 30s; must be >= startup-min-delay
 }
 
 type rawHealthCheck struct {
@@ -403,6 +429,41 @@ func convertVPP(r *rawVPPCfg, cfg *VPPConfig) error {
 		cfg.LB.FlowTimeout = 40 * time.Second
 	}
 
+	// startup-min-delay: absolute hands-off window at process start.
+	// Default 5s. May be 0 (no gate) but must not be negative.
+	if r.LB.StartupMinDelay != nil {
+		d, err := time.ParseDuration(*r.LB.StartupMinDelay)
+		if err != nil {
+			return fmt.Errorf("vpp.lb.startup-min-delay: %w", err)
+		}
+		if d < 0 {
+			return fmt.Errorf("vpp.lb.startup-min-delay must be >= 0")
+		}
+		cfg.LB.StartupMinDelay = d
+	} else {
+		cfg.LB.StartupMinDelay = 5 * time.Second
+	}
+
+	// startup-max-delay: watchdog for the per-VIP release phase. Default
+	// 30s. May be 0 (no warmup at all, together with min-delay=0), but
+	// must be >= min-delay so the per-VIP release phase is well-formed.
+	if r.LB.StartupMaxDelay != nil {
+		d, err := time.ParseDuration(*r.LB.StartupMaxDelay)
+		if err != nil {
+			return fmt.Errorf("vpp.lb.startup-max-delay: %w", err)
+		}
+		if d < 0 {
+			return fmt.Errorf("vpp.lb.startup-max-delay must be >= 0")
+		}
+		cfg.LB.StartupMaxDelay = d
+	} else {
+		cfg.LB.StartupMaxDelay = 30 * time.Second
+	}
+	if cfg.LB.StartupMaxDelay < cfg.LB.StartupMinDelay {
+		return fmt.Errorf("vpp.lb.startup-max-delay (%s) must be >= startup-min-delay (%s)",
+			cfg.LB.StartupMaxDelay, cfg.LB.StartupMinDelay)
+	}
+
 	// A missing src address is a hard error: VPP's GRE encap needs a source,
 	// and every VIP we program uses GRE. Fail the config check so the
 	// operator cannot start maglevd with a broken setup.