Files
vpp-maglev/internal/health/state.go
Pim van Pelt 0049c2ae73 VPP reconciler: event-driven sync, pool failover, bug fixes
This commit wires the checker's state machine through to the VPP dataplane:
every backend state transition flows through a single code path that
recomputes the effective per-backend weight (with pool failover) and pushes
the result to VPP. Along the way several latent bugs in the state machine
and the sync path were fixed.

internal/vpp/reconciler.go (new)
- New Reconciler type subscribes to checker.Checker events and, on every
  transition, calls Client.SyncLBStateVIP for the affected frontend. This
  is the ONLY place in the codebase where backend state changes cause VPP
  calls — the "single path" discipline requested during design.
- Defines an EventSource interface (checker.Checker satisfies it) so the
  dependency direction stays vpp → checker; the checker never imports vpp.

internal/vpp/client.go
- Renamed ConfigSource → StateSource. The interface now has two methods:
  Config() and BackendState(name) — the reconciler and the desired-state
  builder both need live health state to compute effective weights.
- SetConfigSource → SetStateSource; internal cfgSrc field → stateSrc.
- New getStateSource() helper for internal locked access.
- lbSyncLoop still uses the state source for its periodic drift
  reconciliation; it's fully idempotent and runs the same code path as
  event-driven syncs.

internal/vpp/lbsync.go
- desiredAS grows a Flush bool so the mapping function can signal "on
  transition to weight 0, flush existing flow-table entries".
- asFromBackend is now the single source of truth for the state →
  (weight, flush) rule. Documented with a full truth table. Takes an
  activePool parameter so it can distinguish "up in active pool" from
  "up but standby".
- activePoolIndex(fe, states) implements priority failover: returns the
  index of the first pool containing any StateUp backend. pool[0] wins
  when at least one member is up; pool[1] takes over when pool[0] is
  empty; and so on. Defaults to 0 (unobservable, since all backends map
  to weight 0 when nothing is up).
- desiredFromFrontend snapshots backend states once, computes activePool,
  then walks every backend through asFromBackend. No more filtering on
  b.Enabled — disabled backends stay in the desired set so they keep
  their AS entry in VPP with weight=0. The previous filter caused delAS
  on disable, which destroyed the entry and broke enable afterwards.
- EffectiveWeights(fe, src) exported helper that returns the per-pool
  per-backend weight map for one frontend. Used by the gRPC GetFrontend
  handler and robot tests to observe failover without touching VPP.
- reconcileVIP computes flush at the weight-change call site:
    flush = desired.Flush && cur.Weight > 0 && desired.Weight == 0
  This ensures only the *transition* to disabled flushes sessions —
  steady-state syncs with already-zero weight skip the call entirely.
- setASWeight now plumbs IsFlush into lb_as_set_weight.

internal/vpp/lbsync_test.go (new)
- TestAsFromBackend: 15 cases locking down the truth table, including
  failover scenarios (up in standby pool, up promoted in pool[1]).
- TestActivePoolIndex: 8 cases covering pool[0]-has-up, pool[0]-all-down,
  all-disabled, all-paused, all-unknown, nothing-up-anywhere, and
  three-tier failover.
- TestDesiredFromFrontendFailover: 5 end-to-end scenarios wiring a fake
  StateSource through desiredFromFrontend and asserting the final
  per-IP weight map. Exercises the complete pipeline without VPP.

internal/checker/checker.go
- Added BackendState(name) (health.State, bool) — one-line method that
  satisfies vpp.StateSource. The checker is otherwise unchanged.
- EnableBackend rewritten to reuse the existing worker (parallel to
  ResumeBackend). The old code called startWorker which constructed a
  brand-new Backend via health.New, throwing away the transition
  history; the resulting 'backend-transition' log showed the bogus
  from=unknown,to=unknown. Now uses w.backend.Enable() to record a
  proper disabled→unknown transition and launches a fresh goroutine.
- Static (no-healthcheck) backends now fire their synthetic 'always up'
  pass on the first iteration of runProbe instead of sleeping 30s
  first. Previously static backends sat in StateUnknown for 30s after
  startup — useless for deterministic testing and surprising for
  operators. The fix is a simple first-iteration flag.

internal/health/state.go
- New Enable(maxHistory) method parallel to Disable. Transitions the
  backend from whatever state it's in (typically StateDisabled) to
  StateUnknown, resets the health counter to rise-1 so the expedited
  resolution kicks in on the first probe result, and emits a transition
  with code 'enabled'.

proto/maglev.proto
- PoolBackendInfo gains effective_weight: the state-aware weight that
  would be programmed into VPP (distinct from the configured weight in
  the YAML). Exposed via GetFrontend.

internal/grpcapi/server.go
- frontendToProto takes a vpp.StateSource, computes effective weights
  via vpp.EffectiveWeights, and populates PoolBackendInfo.EffectiveWeight.
- GetFrontend and SetFrontendPoolBackendWeight updated to pass the
  checker in.

cmd/maglevc/commands.go
- 'show frontends <name>' now renders every pool backend row as
    <name>  weight <cfg>  effective <eff>  [disabled]?
  so both values are always visible. The VPP-style key/value format
  avoids the ANSI-alignment pitfall we hit earlier and makes the output
  regex-parseable for robot tests.

cmd/maglevd/main.go
- Construct and start the Reconciler alongside the VPP client. Two
  extra lines, no other changes to startup.

tests/01-maglevd/maglevd-lab/maglev.yaml
- Two new static backends (static-primary, static-fallback) and a new
  failover-vip frontend with one backend per pool. No healthcheck, so
  the state machine resolves them to 'up' immediately via the synthetic
  pass. Used by the failover robot tests.

tests/01-maglevd/01-healthcheck.robot
- Three new test cases exercising pool failover end-to-end:
  1. primary up, secondary standby (initial state)
  2. disable primary → fallback takes over (effective weight flips)
  3. enable primary → fallback steps back
  All run without VPP: they scrape 'maglevc show frontends <name>' and
  regex-match the effective weight in the output. Deterministic and
  fast (~2s total) because the static backends don't probe.
- Two helper keywords: Static Backend Should Be Up and
  Effective Weight Should Be.

Net result: 16/16 robot tests pass. Backend state transitions now
flow through a single documented path (checker event → reconciler →
SyncLBStateVIP → desiredFromFrontend → asFromBackend → reconcileVIP →
setASWeight), and the pool failover / enable-after-disable / static-
backend-startup bugs are all fixed.
2026-04-12 12:40:09 +02:00

234 lines
7.0 KiB
Go

// Copyright (c) 2026, Pim van Pelt <pim@ipng.ch>
package health
import (
"net"
"time"
)
// CheckLayer indicates at which network layer a probe stopped.
type CheckLayer int
const (
LayerUnknown CheckLayer = iota
LayerL4 // TCP connect
LayerL6 // TLS handshake
LayerL7 // Application (HTTP response, ICMP reply)
)
// ProbeResult is the outcome of a single probe execution.
type ProbeResult struct {
OK bool
Layer CheckLayer
Code string // "L4OK", "L4TOUT", "L4CON", "L7OK", "L7TOUT", "L7RSP", "L7STS"
Detail string // human-readable, e.g. "HTTP 503", "connection refused"
}
// State represents the health state of a backend.
type State int
const (
StateUnknown State = iota // initial state before first probe
StateUp // backend is healthy
StateDown // backend has failed enough probes
StatePaused // operator paused health checking
StateDisabled // operator disabled the backend
StateRemoved // backend removed from configuration by reload
)
func (s State) String() string {
switch s {
case StateUnknown:
return "unknown"
case StateUp:
return "up"
case StateDown:
return "down"
case StatePaused:
return "paused"
case StateDisabled:
return "disabled"
case StateRemoved:
return "removed"
default:
return "unknown"
}
}
// Transition records a single state change event.
type Transition struct {
From State
To State
At time.Time
Result ProbeResult
}
// HealthCounter is HAProxy's single-integer rise/fall model.
//
// Health ∈ [0, Rise+Fall-1]. Server is UP when Health >= Rise, DOWN when
// Health < Rise. On success Health increments (ceiling Rise+Fall-1); on
// failure Health decrements (floor 0). This gives hysteresis: a flapping
// backend stays in the degraded range without bouncing between UP and DOWN.
type HealthCounter struct {
Health int
Rise int
Fall int
}
func (h *HealthCounter) Max() int { return h.Rise + h.Fall - 1 }
func (h *HealthCounter) IsUp() bool { return h.Health >= h.Rise }
func (h *HealthCounter) IsDegraded() bool { return h.Health > 0 && h.Health < h.Max() }
// RecordPass increments the counter. Returns true if the server just became UP.
func (h *HealthCounter) RecordPass() bool {
wasUp := h.IsUp()
if h.Health < h.Max() {
h.Health++
}
return !wasUp && h.IsUp()
}
// RecordFail decrements the counter. Returns true if the server just went DOWN.
func (h *HealthCounter) RecordFail() bool {
wasDown := !h.IsUp()
if h.Health > 0 {
h.Health--
}
return !wasDown && !h.IsUp()
}
// Backend tracks the health state of a named backend.
type Backend struct {
Name string
Address net.IP
State State
Counter HealthCounter
Transitions []Transition // newest first, capped at maxHistory
}
// New creates a Backend in StateUnknown with the health counter pre-loaded to
// Rise-1, so the very first probe resolves the state: one pass → Up, any
// fail → Down (via the StateUnknown shortcut in Record).
func New(name string, addr net.IP, rise, fall int) *Backend {
return &Backend{
Name: name,
Address: addr,
State: StateUnknown,
Counter: HealthCounter{Rise: rise, Fall: fall, Health: rise - 1},
}
}
// Record applies a probe result to the health counter and transitions state if
// needed. Returns true if the state changed.
//
// StateUnknown transitions to StateDown on the first failure (any evidence of
// failure means the backend is not yet confirmed reachable), and to StateUp
// once the counter reaches Rise consecutive passes.
func (b *Backend) Record(r ProbeResult, maxHistory int) bool {
if b.State == StatePaused || b.State == StateDisabled || b.State == StateRemoved {
return false
}
if r.OK {
if b.Counter.RecordPass() {
b.transition(StateUp, r, maxHistory)
return true
}
} else {
if b.Counter.RecordFail() || b.State == StateUnknown {
b.transition(StateDown, r, maxHistory)
return true
}
}
return false
}
// Pause transitions the backend to StatePaused. Returns true if the state changed.
func (b *Backend) Pause(maxHistory int) bool {
if b.State == StatePaused {
return false
}
b.transition(StatePaused, ProbeResult{}, maxHistory)
b.Counter.Health = 0
return true
}
// Resume transitions a paused backend back to StateUnknown, resetting the
// counter. Returns true if the state changed.
func (b *Backend) Resume(maxHistory int) bool {
if b.State != StatePaused {
return false
}
b.transition(StateUnknown, ProbeResult{}, maxHistory)
b.Counter.Health = b.Counter.Rise - 1
return true
}
// NextInterval returns the appropriate probe interval based on state and counter:
// - Unknown (initial / post-resume): fastInterval (falls back to interval) — probe quickly to establish state
// - Fully healthy (counter at max): interval
// - Fully down (counter at 0): downInterval (falls back to interval)
// - Degraded (anywhere in between): fastInterval (falls back to interval)
func (b *Backend) NextInterval(interval, fastInterval, downInterval time.Duration) time.Duration {
if b.State == StateUnknown {
if fastInterval > 0 {
return fastInterval
}
return interval
}
if b.Counter.Health == b.Counter.Max() {
return interval
}
if b.Counter.Health == 0 {
if downInterval > 0 {
return downInterval
}
return interval
}
if fastInterval > 0 {
return fastInterval
}
return interval
}
// Start records the initial StateUnknown transition when a backend is first
// created or restarted. It exists solely to populate the transition history
// and fire a reload event; the state does not change.
func (b *Backend) Start(maxHistory int) Transition {
b.transition(StateUnknown, ProbeResult{Code: "start"}, maxHistory)
return b.Transitions[0]
}
// Disable transitions the backend to StateDisabled. Returns the transition.
// After this call no further probe results are accepted.
func (b *Backend) Disable(maxHistory int) Transition {
b.transition(StateDisabled, ProbeResult{Code: "disabled"}, maxHistory)
return b.Transitions[0]
}
// Enable transitions a disabled backend back to StateUnknown, resetting the
// counter so the first probe result resolves state (rise-1 preload gives
// 1-pass → Up, 1-fail → Down). Returns the transition.
func (b *Backend) Enable(maxHistory int) Transition {
b.transition(StateUnknown, ProbeResult{Code: "enabled"}, maxHistory)
b.Counter.Health = b.Counter.Rise - 1
return b.Transitions[0]
}
// Remove transitions the backend to StateRemoved. Returns the transition.
// After this call no further probe results are accepted.
func (b *Backend) Remove(maxHistory int) Transition {
b.transition(StateRemoved, ProbeResult{Code: "removed"}, maxHistory)
return b.Transitions[0]
}
// transition appends a new Transition and updates State.
func (b *Backend) transition(to State, r ProbeResult, maxHistory int) {
t := Transition{From: b.State, To: to, At: time.Now(), Result: r}
b.Transitions = append([]Transition{t}, b.Transitions...)
if len(b.Transitions) > maxHistory {
b.Transitions = b.Transitions[:maxHistory]
}
b.State = to
}