bc6ccaa84411a2176182ff849152c2a566175f7b
9 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Pim van Pelt
|
bc6ccaa844 |
v1.0.0 — first release
Bump VERSION to 1.0.0 and cut the first tagged release of vpp-maglev. Also in this commit: - maglevc: MAGLEV_SERVER env var as an alternative to the --server flag, matching the MAGLEV_CONFIG / MAGLEV_GRPC_ADDR convention on the other binaries. The flag takes precedence when both are set. - Rename cmd/maglevd -> cmd/server and cmd/maglevc -> cmd/client so the source directory names are decoupled from binary names (the frontend and tester commands already followed this convention). Build outputs and the Debian packages are unchanged. |
||
|
Pim van Pelt
|
6d78921edd |
Restart-neutral VPP LB sync; deterministic AS ordering; maglevt cadence; v0.9.5
Three reliability fixes bundled with docs updates. Restart-neutral VPP LB sync via a startup warmup window (internal/vpp/warmup.go). Before this, a maglevd restart would immediately issue SyncLBStateAll with every backend still in StateUnknown — mapped through BackendEffectiveWeight to weight 0 — and VPP would black-hole all new flows until the checker's rise counters caught up, several seconds later. The new warmup tracker owns a process-wide state machine gated by two config knobs: vpp.lb.startup-min-delay (default 5s) is an absolute hands-off window during which neither the periodic sync loop nor the per-transition reconciler touches VPP; vpp.lb. startup-max-delay (default 30s) is the watchdog for a per-VIP release phase that runs between the two, releasing each frontend as soon as every backend it references reaches a non-Unknown state. At max-delay a final SyncLBStateAll runs for any stragglers still in Unknown. Config reload does not reset the clock. Both delays can be set to 0 to disable the warmup entirely. The reconciler's suppressed-during-warmup events log at DEBUG so operators can still see them with --log-level debug. Unit tests cover the tracker state machine, allBackendsKnown precondition, and the zero-delay escape hatch. Deterministic AS iteration in VPP LB sync. reconcileVIP and recreateVIP now issue their lb_as_add_del / lb_as_set_weight calls in numeric IP order (IPv4 before IPv6, ascending within each family) via a new sortedIPKeys helper, instead of Go map iteration order. VPP's LB plugin breaks per-bucket ties in the Maglev lookup table by insertion position in its internal AS vec, so without a stable call order two maglevd instances on the same config could push identical AS sets into VPP in different orders and produce divergent new-flow tables. Numeric sort is used in preference to lexicographic so the sync log stays human-readable: string order would place 10.0.0.10 before 10.0.0.2, and the same problem in v6. Unit tests cover empty, single, v4/v6 numeric vs lexicographic, v4-before-v6 grouping, a 1000-iteration stability loop against Go's randomised map iteration, insertion-order invariance, and the desiredAS call-site type. maglevt interval fix. runProbeLoop used to sleep the full jittered interval after every probe, so a 100ms --interval with a 30ms probe actually produced a 130ms period. The sleep now subtracts result.Duration so cadence matches the flag. Probes that overrun clamp sleep to zero and fire the next probe immediately without trying to catch up on missed cycles — a slow backend doesn't get flooded with back-to-back probes at the moment it's already struggling. Docs. config-guide now documents flush-on-down and the new startup-min-delay / startup-max-delay knobs; user-guide's maglevd section explains the restart-neutrality property, the three warmup phases, and the relevant slog lines operators should watch for during a bounce. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Pim van Pelt
|
224167ce39 |
Dataplane reconcile fixes; LB counters cleanup; SPA scope cookie
Checker / reload:
- Reload's update-in-place branch now mirrors b.Address onto the
runtime health.Backend. Without this, GetBackend kept returning
the pre-reload address indefinitely after a config edit that
touched addresses but not healthcheck settings — the VPP sync
path reads cfg.Backends directly so the dataplane moved on
while the gRPC and SPA view stayed wedged on the old IPv4/IPv6.
Sync (internal/vpp/lbsync.go):
- reconcileVIP now detects encap mismatch in addition to
src-ip-sticky mismatch and takes the full tear-down / re-add
path via a new shared recreateVIP helper. Triggered when every
backend flips address family (gre4 <-> gre6) and the existing
VIP can no longer accept new ASes — previously the sync wedged
with 'Invalid address family' until a full maglevd restart.
- setASWeight is issued whenever the state machine requests
flush (a.Flush=true), not only on the weight-value transition
edge. Fixes the case where a backend reached StateDisabled
after its effective weight had already been drained to 0 by
pool failover — the sticky-cache entries pointing at it were
previously never cleared.
maglev-frontend:
- signal.Ignore(SIGHUP) so a controlling-terminal disconnect
doesn't kill the daemon.
- debian/vpp-maglev.service grants CAP_SYS_ADMIN in addition to
CAP_NET_RAW so setns(CLONE_NEWNET) can join the healthcheck
netns. Comment documents the 'operation not permitted' symptom
and notes the knob can be dropped if the deployment doesn't use
the 'netns:' healthcheck option.
LB plugin counters (internal/vpp/lbstats.go + friends):
- Fix the VIP counter regex: the LB plugin registers
vlib_simple_counter_main_t names without a leading '/'
(vlib_validate_simple_counter in counter.c:50 uses cm->name
verbatim; only entries that set cm->stat_segment_name get a
slash). first/next/untracked/no-server now read through as
live values instead of zero.
- Drop the per-backend FIB counter block end-to-end (proto,
grpcapi, metrics, vpp.Client, lbstats, maglevc). Traced from
lb/node.c:558 into ip{4,6}_forward.h:141 — the LB plugin
forwards by writing adj_index[VLIB_TX] directly and bypassing
ip{4,6}_lookup_inline, which is the only path that increments
lbm_to_counters. The backend's FIB load_balance stats_index
literally never ticks for LB-forwarded traffic, so the column
was always zero and misleading. docs/implementation/TODO
records the full investigation and the recommended upstream
path (new lb_as_stats_dump API message) for when we're ready
to carry that VPP patch.
- maglevc show vpp lb counters: plain-text tabular headers.
label() wraps strings in ANSI escapes (~11 bytes of overhead),
but tabwriter counts bytes, not rendered width — so a header
row with label()'d cells and data rows with plain cells drifts
column alignment on every row. color.go comment now spells
out the constraint: label() only works when column N is
wrapped identically in every row (key-value layouts are fine,
multi-column tables with header-only labelling are not).
SPA:
- stores/scope.ts is cookie-backed (maglev_scope, 1 year,
SameSite=Lax). App.tsx hydrates from the cookie then validates
against the fetched snapshots: a cookie referencing a maglevd
that no longer exists falls through to snaps[0] instead of
leaving the user on a ghost selection.
- components/Flash.tsx wraps props.value in createMemo. Solid's
on() fires its callback on every dep notification, not on
value change — source is right in solid-js/dist/solid.js:460,
no equality check. Without the memo, flipping scope between
two 'connected' maglevds (or any other cross-store reactive
re-eval that doesn't actually change the concrete string)
replays the animation every time. createMemo's default ===
dedupe fixes it in one place for every Flash consumer,
superseding the local createMemo workaround we'd added in
BackendRow earlier.
|
||
|
Pim van Pelt
|
284b4cc9a4 |
New maglev-frontend component; promote LB sync events to INFO
Introduces maglev-frontend, a responsive, real-time web dashboard for one
or more running maglevd instances. Source lives at cmd/frontend/; the
built binary is maglev-frontend. It is a single Go process with the
SolidJS SPA embedded via //go:embed — no runtime file dependencies.
Architecture
- One persistent gRPC connection per configured maglevd (-server A,B,C).
Each connection runs three background loops: a WatchEvents stream
subscribed at log_level=debug for live events, a 30s refresh loop as
a safety net for drift, and a 5s health loop that surfaces connection
drops quickly.
- In-process pub/sub broker with a 30s / 2000-event replay ring using
<epoch>-<seq> monotonic IDs. Short browser reconnects (nginx idle,
wifi flap, laptop wake) silently replay buffered events via the
EventSource Last-Event-ID header; longer outages or frontend restarts
fall through to a "resync" event that triggers a full state refetch.
- HTTP surface: /view/ (SPA), /view/api/state, /view/api/state/{name},
/view/api/maglevds, /view/api/version, /view/api/events (SSE),
/healthz, and an /admin/* placeholder returning 501 for a future
basic-auth mutation surface.
- SSE handler follows the full operational checklist: retry hint, 15s
: ping heartbeat, Flush after every write, r.Context().Done() teardown,
X-Accel-Buffering: no, and no gzip.
SolidJS SPA (cmd/frontend/web/, Vite + TypeScript)
- solid-js/store for a reactive per-maglevd state tree; reducers apply
backend transitions, maglevd-status flips, and resync refetches.
- Scope selector tabs for multi-maglevd support, per-maglevd frontend
cards with pool tables showing state, configured weight, effective
weight, and last-transition age.
- ProbeHeartbeat component turns a middle-dot into ❤️ on probe-start and
back on probe-done, driven by real log events; fixed-size wrapper so
the emoji swap doesn't jiggle the row.
- Flash wrapper animates any primitive on change (1s yellow fade via
Web Animations API, skipped on first mount). Wired into the state
badge, configured weight, and effective weight columns.
- DebugPanel: chronological rolling event tail with tail-style auto-
scroll, pause/resume, and scope/firehose filter. Syntactic highlight
for vpp-lb-sync-* events with fixed-order attribute formatting.
- Live effective_weight updates: vpp-lb-sync-as-added/removed/weight-
updated log events are routed through a reducer that walks the
snapshot's pool rows and sets effective_weight on every match
without waiting for the 30s refresh.
- Header shows build version + commit with build date in a tooltip,
fetched once from /view/api/version on mount.
- Prettier wired in as the web-side fixstyle; make fixstyle now tidies
both Go and web in one shot via a new fixstyle-web target.
Per-mutation VPP LB sync logging
- Promotes the addVIP/delVIP/addAS/delAS/setASWeight helpers from
slog.Debug to slog.Info and renames them from vpp-lbsync-* to
vpp-lb-sync-{vip-added,vip-removed,as-added,as-removed,as-weight-
updated}. Matching rename for vpp-lb-sync-start / -done / -error /
-vip-recreate. The Prometheus metric name (maglev_vpp_lbsync_total)
is left alone to preserve dashboards.
- setASWeight now takes the prior weight so the event can emit
from=X to=Y and the UI can show the delta.
- The vip field in every event is the bare address (no /32 or /128
mask), matching the CLI output style.
- Any listener on the gRPC WatchEvents stream — CLI watch events or
maglev-frontend — now sees every VIP/AS dataplane change in real
time without needing to raise the log level.
Build and tooling
- Makefile: maglev-frontend added to BINARIES; build / build-amd64 /
build-arm64 emit the binary alongside maglevd and maglevc. A new
maglev-frontend-web target rebuilds the SolidJS bundle via npm.
- web/dist/ is tracked so a bare `go build` keeps working for Go-only
contributors and CI.
- .gitignore skips cmd/frontend/web/node_modules/.
Stability fixes
- maglevd's WatchEvents synthetic replay events (from==to, at_unix_ns=0)
were corrupting the frontend's LastTransition cache with at=0,
rendering as "20555d ago" in the browser. Client now skips synthetic
events: the cache comes from refreshAll and doesn't need them.
- Frontends, Backends, and HealthChecks are now served in the order
returned by the corresponding List* RPC instead of Go map iteration
order, so reloads and refreshes keep the SPA stable.
|
||
|
Pim van Pelt
|
fb62532fd5 |
VPP LB counters, src-ip-sticky, and frontend state aggregation
New feature: per-VIP / per-backend runtime counters
* New GetVPPLBCounters RPC serving an in-process snapshot refreshed
by a 5s scrape loop (internal/vpp/lbstats.go). Each cycle pulls
the LB plugin's four SimpleCounters (next, first, untracked,
no-server) plus the FIB /net/route/to CombinedCounter for every
VIP and every backend host prefix via a single DumpStats call.
* FIB stats-index discovery via ip_route_lookup (internal/vpp/
fibstats.go); per-worker reduction happens in the collector.
* Prometheus collector exports vip_packets_total (kind label),
vip_route_{packets,bytes}_total, and backend_route_{packets,
bytes}_total. Metrics source interface extended with VIPStats /
BackendRouteStats; vpp.Client publishes snapshots via
atomic.Pointer and clears them on disconnect.
* New 'show vpp lb counters' CLI command. The 'show vpp lbstate'
and 'sync vpp lbstate' commands are restructured under 'show
vpp lb {state,counters}' / 'sync vpp lb state' to make room
for the new verb.
New feature: src-ip-sticky frontends
* New frontend YAML key 'src-ip-sticky' (bool). Plumbed through
config.Frontend, desiredVIP, and the lb_add_del_vip_v2 call.
* Reflected in gRPC FrontendInfo.src_ip_sticky and VPPLBVIP.
src_ip_sticky, and shown in 'show vpp lb state' output.
* Scraped back from VPP by parsing 'show lb vips verbose' through
cli_inband — lb_vip_details does not expose the flag. The same
scrape also recovers the LB pool index for each VIP, which the
stats-segment counters are keyed on. This is a documented
temporary workaround until VPP ships an lb_vip_v2_dump.
* src_ip_sticky cannot be mutated on a live VIP, so a flipped flag
triggers a tear-down-and-recreate in reconcileVIP (ASes deleted
with flush, VIP deleted, then re-added). Flip is logged.
New feature: frontend state aggregation and events
* New health.FrontendState (unknown/up/down) and FrontendTransition
types. A frontend is 'up' iff at least one backend has a nonzero
effective weight, 'unknown' iff no backend has real state yet,
and 'down' otherwise.
* Checker tracks per-frontend aggregate state, recomputing after
each backend transition and emitting a frontend-transition Event
on change. Reload drops entries for removed frontends.
* checker.Event gains an optional FrontendTransition pointer;
backend- vs. frontend-transition events are demultiplexed on
that field.
* WatchEvents now sends an initial snapshot of frontend state on
connect (mirroring the existing backend snapshot), subscribes
once to the checker stream, and fans out to backend/frontend
handlers based on the client's filter flags. The proto
FrontendEvent message grows name + transition fields.
* New Checker.FrontendState accessor.
Refactor: pure health helpers
* Moved the priority-failover selector and the (pool idx, active
pool, state, cfg weight) → (vpp weight, flush) mapping out of
internal/vpp/lbsync.go into a new internal/health/weights.go so
the checker can reuse them for frontend-state computation
without importing internal/vpp.
* New functions: health.ActivePoolIndex, BackendEffectiveWeight,
EffectiveWeights, ComputeFrontendState. lbsync.go now calls
these directly; vpp.EffectiveWeights is a thin wrapper over
health.EffectiveWeights retained for the gRPC observability
path. Fully unit-tested in internal/health/weights_test.go.
maglevc polish
* --color default is now mode-aware: on in the interactive shell,
off in one-shot mode so piped output is script-safe. Explicit
--color=true/false still overrides.
* New stripHostMask helper drops /32 and /128 from VIP display;
non-host prefixes pass through unchanged.
* Counter table column order fixed (first before next) and
packets/bytes columns renamed to fib-packets/fib-bytes to
clarify they come from the FIB, not the LB plugin.
Docs
* config-guide: document src-ip-sticky, including the VIP
recreate-on-change caveat.
* user-guide, maglevc.1, maglevd.8: updated command tree, new
counters command, color defaults, and the src-ip-sticky field.
|
||
|
Pim van Pelt
|
d5fbf5c640 |
Prometheus: add VPP, LB sync, and gRPC metrics; expand docs
New metrics plus the corresponding documentation for everything that's
accumulated since the last Prometheus pass.
internal/metrics/metrics.go
- New VPPSource interface (IsConnected, VPPInfo) plus a metrics-local
VPPInfo struct that mirrors vpp.Info. Decoupling via interface +
struct-mirror keeps the dependency direction one-way (vpp → metrics),
so vpp can import metrics to update inline counters without a cycle.
- New Collector gauges scraped on demand: maglev_vpp_connected,
maglev_vpp_uptime_seconds (from /sys/boottime), maglev_vpp_connected_seconds
(time since maglevd connected), and maglev_vpp_info (static 1-gauge
carrying version, build_date, and pid as labels).
- New inline counters:
- maglev_vpp_api_total{msg, direction, result} — bumped from the
loggedChannel wrapper on every VPP binary-API send/recv. Gives full
visibility into what maglevd is doing with VPP, broken down by
message name, direction (send/recv), and result (success/failure).
- maglev_vpp_lbsync_total{scope, kind} — bumped from the reconciler
at the end of each SyncLBStateAll/SyncLBStateVIP run. kind ∈
{vip_added, vip_removed, as_added, as_removed, as_weight_updated};
scope ∈ {all, vip}. Zero-valued kinds are not emitted so noise
stays low.
- Register() signature now takes a VPPSource (may be nil) alongside
the existing StateSource.
internal/vpp/client.go
- New VPPInfo() (metrics.VPPInfo, bool) shim method on *Client that
satisfies metrics.VPPSource. Returns (_, false) when disconnected so
the collector skips the vpp_* gauges cleanly.
internal/vpp/apilog.go
- The loggedChannel's SendRequest / SendMultiRequest / ReceiveReply
paths now call metrics.VPPAPITotal.WithLabelValues(...).Inc() in
addition to slog.Debug. Since every VPP API call in the codebase
must go through loggedChannel (NewAPIChannel is unexported), this
one instrumentation point catches everything.
internal/vpp/lbsync.go
- New recordSyncStats(scope, st) helper called once at the end of
SyncLBStateAll and SyncLBStateVIP to bump maglev_vpp_lbsync_total.
Zero-valued stats are skipped.
cmd/maglevd/main.go
- Added github.com/grpc-ecosystem/go-grpc-middleware/providers/prometheus
for the standard gRPC server metrics (grpc_server_started_total,
grpc_server_handled_total, grpc_server_handling_seconds, etc.,
labelled by service/method/type/code).
- Constructs grpcprom.NewServerMetrics(WithServerHandlingTimeHistogram())
before creating the grpc.Server, installs it as UnaryInterceptor +
StreamInterceptor, then calls InitializeMetrics(srv) after service
registration so every method appears at 0 on the first scrape
instead of materialising lazily on first RPC.
- Passes the vppClient (or nil) as a metrics.VPPSource to
metrics.Register so the vpp_* gauges are emitted when integration
is enabled and silently omitted otherwise.
docs/user-guide.md
- New 'Prometheus metrics' section in the maglevd chapter,
tabulating every metric family: backend state gauges, probe
counters/histogram, transition counters, the new VPP gauges and
counters, and the standard gRPC server metrics.
- 'show frontends <name>' description updated to mention the two
weight columns ('weight' = configured from YAML, 'effective' =
state-aware after pool-failover logic).
- Pause / disable descriptions clarified: transition history is
preserved across these operator actions.
docs/healthchecks.md
- New 'Static (no-healthcheck) backends' section explaining that
backends without a healthcheck use rise/fall=1, fire a synthetic
passing probe immediately on startup (no 30s wait), and idle at
30s between iterations thereafter.
- New 'Pool failover' section documenting the priority-tier model,
the active-pool definition, when promotion happens, cascading to
further tiers, and graceful drain on demotion. Points readers at
'maglevc show frontends <name>' as the inspection interface.
docs/config-guide.md
- healthcheck field doc now describes static-backend behavior and
cross-references healthchecks.md.
- pools field doc now explains failover semantics at a high level
and cross-references the detailed healthchecks.md section.
|
||
|
Pim van Pelt
|
0049c2ae73 |
VPP reconciler: event-driven sync, pool failover, bug fixes
This commit wires the checker's state machine through to the VPP dataplane:
every backend state transition flows through a single code path that
recomputes the effective per-backend weight (with pool failover) and pushes
the result to VPP. Along the way several latent bugs in the state machine
and the sync path were fixed.
internal/vpp/reconciler.go (new)
- New Reconciler type subscribes to checker.Checker events and, on every
transition, calls Client.SyncLBStateVIP for the affected frontend. This
is the ONLY place in the codebase where backend state changes cause VPP
calls — the "single path" discipline requested during design.
- Defines an EventSource interface (checker.Checker satisfies it) so the
dependency direction stays vpp → checker; the checker never imports vpp.
internal/vpp/client.go
- Renamed ConfigSource → StateSource. The interface now has two methods:
Config() and BackendState(name) — the reconciler and the desired-state
builder both need live health state to compute effective weights.
- SetConfigSource → SetStateSource; internal cfgSrc field → stateSrc.
- New getStateSource() helper for internal locked access.
- lbSyncLoop still uses the state source for its periodic drift
reconciliation; it's fully idempotent and runs the same code path as
event-driven syncs.
internal/vpp/lbsync.go
- desiredAS grows a Flush bool so the mapping function can signal "on
transition to weight 0, flush existing flow-table entries".
- asFromBackend is now the single source of truth for the state →
(weight, flush) rule. Documented with a full truth table. Takes an
activePool parameter so it can distinguish "up in active pool" from
"up but standby".
- activePoolIndex(fe, states) implements priority failover: returns the
index of the first pool containing any StateUp backend. pool[0] wins
when at least one member is up; pool[1] takes over when pool[0] is
empty; and so on. Defaults to 0 (unobservable, since all backends map
to weight 0 when nothing is up).
- desiredFromFrontend snapshots backend states once, computes activePool,
then walks every backend through asFromBackend. No more filtering on
b.Enabled — disabled backends stay in the desired set so they keep
their AS entry in VPP with weight=0. The previous filter caused delAS
on disable, which destroyed the entry and broke enable afterwards.
- EffectiveWeights(fe, src) exported helper that returns the per-pool
per-backend weight map for one frontend. Used by the gRPC GetFrontend
handler and robot tests to observe failover without touching VPP.
- reconcileVIP computes flush at the weight-change call site:
flush = desired.Flush && cur.Weight > 0 && desired.Weight == 0
This ensures only the *transition* to disabled flushes sessions —
steady-state syncs with already-zero weight skip the call entirely.
- setASWeight now plumbs IsFlush into lb_as_set_weight.
internal/vpp/lbsync_test.go (new)
- TestAsFromBackend: 15 cases locking down the truth table, including
failover scenarios (up in standby pool, up promoted in pool[1]).
- TestActivePoolIndex: 8 cases covering pool[0]-has-up, pool[0]-all-down,
all-disabled, all-paused, all-unknown, nothing-up-anywhere, and
three-tier failover.
- TestDesiredFromFrontendFailover: 5 end-to-end scenarios wiring a fake
StateSource through desiredFromFrontend and asserting the final
per-IP weight map. Exercises the complete pipeline without VPP.
internal/checker/checker.go
- Added BackendState(name) (health.State, bool) — one-line method that
satisfies vpp.StateSource. The checker is otherwise unchanged.
- EnableBackend rewritten to reuse the existing worker (parallel to
ResumeBackend). The old code called startWorker which constructed a
brand-new Backend via health.New, throwing away the transition
history; the resulting 'backend-transition' log showed the bogus
from=unknown,to=unknown. Now uses w.backend.Enable() to record a
proper disabled→unknown transition and launches a fresh goroutine.
- Static (no-healthcheck) backends now fire their synthetic 'always up'
pass on the first iteration of runProbe instead of sleeping 30s
first. Previously static backends sat in StateUnknown for 30s after
startup — useless for deterministic testing and surprising for
operators. The fix is a simple first-iteration flag.
internal/health/state.go
- New Enable(maxHistory) method parallel to Disable. Transitions the
backend from whatever state it's in (typically StateDisabled) to
StateUnknown, resets the health counter to rise-1 so the expedited
resolution kicks in on the first probe result, and emits a transition
with code 'enabled'.
proto/maglev.proto
- PoolBackendInfo gains effective_weight: the state-aware weight that
would be programmed into VPP (distinct from the configured weight in
the YAML). Exposed via GetFrontend.
internal/grpcapi/server.go
- frontendToProto takes a vpp.StateSource, computes effective weights
via vpp.EffectiveWeights, and populates PoolBackendInfo.EffectiveWeight.
- GetFrontend and SetFrontendPoolBackendWeight updated to pass the
checker in.
cmd/maglevc/commands.go
- 'show frontends <name>' now renders every pool backend row as
<name> weight <cfg> effective <eff> [disabled]?
so both values are always visible. The VPP-style key/value format
avoids the ANSI-alignment pitfall we hit earlier and makes the output
regex-parseable for robot tests.
cmd/maglevd/main.go
- Construct and start the Reconciler alongside the VPP client. Two
extra lines, no other changes to startup.
tests/01-maglevd/maglevd-lab/maglev.yaml
- Two new static backends (static-primary, static-fallback) and a new
failover-vip frontend with one backend per pool. No healthcheck, so
the state machine resolves them to 'up' immediately via the synthetic
pass. Used by the failover robot tests.
tests/01-maglevd/01-healthcheck.robot
- Three new test cases exercising pool failover end-to-end:
1. primary up, secondary standby (initial state)
2. disable primary → fallback takes over (effective weight flips)
3. enable primary → fallback steps back
All run without VPP: they scrape 'maglevc show frontends <name>' and
regex-match the effective weight in the output. Deterministic and
fast (~2s total) because the static backends don't probe.
- Two helper keywords: Static Backend Should Be Up and
Effective Weight Should Be.
Net result: 16/16 robot tests pass. Backend state transitions now
flow through a single documented path (checker event → reconciler →
SyncLBStateVIP → desiredFromFrontend → asFromBackend → reconcileVIP →
setASWeight), and the pool failover / enable-after-disable / static-
backend-startup bugs are all fixed.
|
||
|
Pim van Pelt
|
d3c5c86037 |
VPP load-balancer dataplane integration: state, sync, and global conf
This commit wires maglevd through to VPP's LB plugin end-to-end, using
locally-generated GoVPP bindings for the newer v2 API messages.
VPP binapi (vendored)
- New package internal/vpp/binapi/ containing lb, lb_types, ip_types, and
interface_types, generated from a local VPP build (~/src/vpp) via a new
'make vpp-binapi' target. GoVPP v0.12.0 upstream lacks the v2 messages we
need (lb_conf_get, lb_add_del_vip_v2, lb_add_del_as_v2, lb_as_v2_dump,
lb_as_set_weight), so we commit the generated output in-tree.
- All generated files go through our loggedChannel wrapper; every VPP API
send/receive is recorded at DEBUG via slog (vpp-api-send / vpp-api-recv /
vpp-api-send-multi / vpp-api-recv-multi) so the full wire-level trail is
auditable. NewAPIChannel is unexported — callers must use c.apiChannel().
Read path: GetLBState{All,VIP}
- GetLBStateAll returns a full snapshot (global conf + every VIP with its
attached application servers).
- GetLBStateVIP looks up a single VIP by (prefix, protocol, port) and
returns (nil, nil) when the VIP doesn't exist in VPP. This is the
efficient path for targeted updates on a busy LB.
- Helpers factored out: getLBConf, dumpAllVIPs, dumpASesForVIP, lookupVIP,
vipFromDetails.
Write path: SyncLBState{All,VIP}
- SyncLBStateAll reconciles every configured frontend with VPP: creates
missing VIPs, removes stale ones (with AS flush), and reconciles AS
membership and weights within VIPs that exist on both sides.
- SyncLBStateVIP targets a single frontend by name. Never removes VIPs.
Returns ErrFrontendNotFound (wrapped with the name) when the frontend
isn't in config, so callers can use errors.Is.
- Shared reconcileVIP helper does the per-VIP AS diff; removeVIP is used
only by the full-sync pass.
- LbAddDelVipV2 requests always set NewFlowsTableLength=1024. The .api
default=1024 annotation is only applied by VAT/CLI parsers, not wire-
level marshalling — sending 0 caused VPP to vec_validate with mask
0xFFFFFFFF and OOM-panic.
- Pool semantics: backends in the primary (first) pool of a frontend get
their configured weight; backends in secondary pools get weight 0. All
backends are installed so higher layers can flip weights on failover
without add/remove churn.
- Every individual change emits a DEBUG slog (vpp-lbsync-vip-add/del,
vpp-lbsync-as-add/del, vpp-lbsync-as-weight). Start/done INFO logs
carry a scope=all|vip label plus aggregate counts.
Global conf push: SetLBConf
- New SetLBConf(cfg) sends lb_conf with ipv4-src, ipv6-src, sticky-buckets,
and flow-timeout. Called automatically on VPP (re)connect and after
every config reload (via doReloadConfig). Results are cached on the
Client so redundant pushes are silently skipped — only actual changes
produce a vpp-lb-conf-set INFO log line.
Periodic drift reconciliation
- vpp.Client.lbSyncLoop runs in a goroutine tied to each VPP connection's
lifetime. Its first tick is immediate (startup and post-reconnect
sync quickly); subsequent ticks fire every vpp.lb.sync-interval from
config (default 30s). Purpose: catch drift if something/someone
modifies VPP state by hand. The loop uses a ConfigSource interface
(satisfied by checker.Checker via its new Config() accessor) to avoid
an import cycle with the checker package.
Config schema additions (maglev.vpp.lb)
- sync-interval: positive Go duration, default 30s.
- ipv4-src-address: REQUIRED. Used as the outer source for GRE4 encap
to application servers. Missing this is a hard semantic error —
maglevd --check exits 2 and the daemon refuses to start. VPP GRE
needs a source address and every VIP we program uses GRE, so there
is no meaningful config without it.
- ipv6-src-address: REQUIRED. Same treatment as ipv4-src-address.
- sticky-buckets-per-core: default 65536, must be a power of 2.
- flow-timeout: default 40s, must be a whole number of seconds in [1s, 120s].
- VPP validation runs at the end of convert() so structural errors in
healthchecks/backends/frontends surface first — operators fix those,
then get the VPP-specific requirements.
gRPC API
- New GetVPPLBState RPC returning VPPLBState: global conf + VIPs with
ASes. Mirrors the read-path but strips fields irrelevant to our
GRE-only deployment (srv_type, dscp, target_port).
- New SyncVPPLBState RPC with optional frontend_name. Unset → full sync
(may remove stale VIPs). Set → single-VIP sync (never removes).
Returns codes.NotFound for unknown frontends, codes.Unavailable when
VPP integration is disabled or disconnected.
maglevc (CLI)
- New 'show vpp lbstate' command displaying the LB plugin state. VPP-only
fields the dataplane irrelevant to GRE are suppressed. Per-AS lines use
a key-value format ("address X weight Y flow-table-buckets Z")
instead of a tabwriter column, which avoids the ANSI-color alignment
issue we hit with mixed label/data rows.
- New 'sync vpp lbstate [<name>]' command. Without a name, triggers a
full reconciliation; with a name, targets one frontend.
- Previous 'show vpp lb' renamed to 'show vpp lbstate' for consistency
with the new sync command.
Test fixtures
- validConfig and all ad-hoc config_test.go fixtures that reach the end
of convert() now include the two required vpp.lb src addresses.
- tests/01-maglevd/maglevd-lab/maglev.yaml gains a vpp.lb section so the
robot integration tests can still load the config.
- cmd/maglevc/tree_test.go gains expected paths for the new commands.
Docs
- config-guide.md: new 'vpp' section in the basic structure, detailed
vpp.lb field reference, noting ipv4/ipv6 src addresses as REQUIRED
(hard error) with no defaults; example config updated.
- user-guide.md: documented 'show vpp info', 'show vpp lbstate',
'sync vpp lbstate [<name>]', new --vpp-api-addr and --vpp-stats-addr
flags, the vpp-lb-conf-set log line, and corrected the pause/resume
description to reflect that pause cancels the probe goroutine.
- debian/maglev.yaml: example config gains a vpp.lb block with src
addresses and commented optional overrides.
|
||
|
Pim van Pelt
|
3227263d68 |
Add GoVPP integration and GetVPPInfo gRPC call
VPP client (internal/vpp/) - New package managing connections to both VPP API and stats sockets, treated as a unit: if either drops, both are torn down and re-established together. - Run() loop: connect, fetch version via vpe.ShowVersion, read /sys/boottime from the stats segment, log vpp-connect, then monitor with control_ping every 10s. On failure, disconnect both, retry after 5s. - Registers as client name "vpp-maglev" (visible in VPP's "show api clients"). - Flags: --vpp-api-addr (default /run/vpp/api.sock) and --vpp-stats-addr (default /run/vpp/stats.sock). Empty api addr disables VPP integration entirely. gRPC / proto - Add GetVPPInfo RPC returning VPPInfo: version, build_date, build_directory, pid, boottime_ns, connecttime_ns. Both times are unix timestamps in nanoseconds — the client computes durations locally for display. - Returns codes.Unavailable if VPP is disabled or not connected. maglevc - Add 'show vpp info' command displaying version, build-date, build-dir, vpp-pid, vpp-boottime (with duration), and connected time (with duration). |