695ebc4bd11b395cd806e1b829c6e8b99b7306a1
16 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Pim van Pelt
|
6b2b04b2d1 |
Frontend flush-on-down policy; v0.9.3
Adds a per-frontend flush-on-down flag (default true) that causes maglevd to set is_flush=true on lb_as_set_weight when a backend transitions to StateDown, tearing down existing flows pinned to the dead AS instead of just draining them. rise/fall debouncing in the health checker already absorbs single-probe flaps, so a fall-counted down is almost always a real outage — and during a real outage the client-visible "connection refused" oscillation window (where VPP keeps steering existing flows at a dead AS until retry) is a reliability regression worth closing by default. Operators who want the pre-flag drain-only behaviour can set flush-on-down: false per frontend. BackendEffectiveWeight's truth table grows one axis: StateDown now returns (0, flushOnDown); StateDisabled still unconditionally flushes; StateUnknown / StatePaused still never flush. The unit test pins all four combinations. The flag surfaces in the gRPC FrontendInfo message and in `maglevc show frontend <name>` right next to src-ip-sticky. |
||
|
Pim van Pelt
|
744b1cb3d2 |
install-deps Makefile target; docs refresh; golangci-lint v2 clean
Makefile:
- New install-deps umbrella target split into three sub-targets:
install-deps-apt — Debian/Trixie-packaged build deps
(nodejs, npm, protobuf-compiler, git, make,
dpkg-dev, ca-certificates, curl, tar). Uses
sudo when not already root.
install-deps-go — ensures a Go toolchain >= GO_VERSION (go.mod
floor, default 1.25.0). Short-circuits when
the system Go is already recent enough;
otherwise downloads the upstream tarball
from go.dev/dl/ into /usr/local/go. Trixie
only ships 1.24 so this step is load-bearing.
install-deps-go-tools — go install protoc-gen-go, protoc-gen-go-grpc,
and golangci-lint/v2/cmd/golangci-lint. Then
asserts the installed golangci-lint version
parses as >= GOLANGCI_LINT_VERSION (default
1.64.0, the floor that supports Go 1.25
syntax) to catch stale binaries in $GOPATH
/bin before they silently run against Go
1.25 code.
- Parser bug fixed: golangci-lint v1.x prints "has version v1.64.8" but
v2.x dropped the 'v' prefix and prints "has version 2.11.4". The
original sed regex required the 'v' and returned an empty match on
v2.x, making the assertion explode with "could not parse version
output". Fixed by switching to extended regex (sed -En) with 'v?' so
both forms parse cleanly.
- GO_VERSION and GOLANGCI_LINT_VERSION exposed as Makefile variables
so operators can override on the command line, e.g.
make install-deps GO_VERSION=1.25.5 GOLANGCI_LINT_VERSION=2.0.0
- .PHONY extended with the four new target names.
Docs:
- README.md: capability note rewritten to cover CAP_NET_RAW (ICMP) and
the new CAP_SYS_ADMIN requirement when healthchecker.netns is set,
plus a paragraph explaining that the Debian systemd unit grants both
automatically. Docker example gained a second variant that shows the
additional --cap-add SYS_ADMIN and /var/run/netns bind mount for
netns-scoped deployments. Also notes that maglevd-frontend ignores
SIGHUP so controlling-terminal disconnects don't kill it.
- docs/user-guide.md: Capabilities section rewritten as a bulleted
list covering both caps, with the EPERM error string and three
different ways to grant them (systemd unit, setcap, systemd-run);
'show vpp lb counters' command description updated to explain that
per-backend packet counts are no longer shown (LB plugin's
forwarding node bypasses ip{4,6}_lookup_inline, so /net/route/to at
the backend's FIB entry never ticks for LB-forwarded traffic); new
~75-line "What the SPA shows" subsection covering the scope
selector + maglev_scope cookie, the per-maglevd frontend cards, the
health-cascade icon table (ok / bug-buckets / primary-drained /
degraded / unknown), the lb buckets column semantics, the
maglev_zippy_open cookie, the admin-mode lifecycle dialogs with
their plain-English consequence text, and the debug panel.
- docs/config-guide.md: healthchecker.netns field gains a capability-
requirement note spelling out setns(CLONE_NEWNET), the EPERM
symptom string, and the /var/run/netns/ readability requirement.
- docs/healthchecks.md: new "Jitter" subsection explaining the +/-10%
scaling on every computed interval, and a "Probe timing while a
probe is in flight" subsection that explains why fast-interval alone
doesn't give fast fault detection against hanging backends (the
probe loop is synchronous, so each iteration is timeout +
fast-interval; the advice is to lower timeout, not fast-interval).
- docs/maglevd.8: description paragraph corrected (dropped the
per-backend stats claim and added a short note pointing at the LB
plugin forwarding-path bypass); new CAPABILITIES section between
SIGNALS and FILES covering both CAP_NET_RAW and CAP_SYS_ADMIN with
the drop-in-override hint.
- docs/maglevd-frontend.8: new SIGNALS section documenting the
explicit SIGHUP ignore (so a controlling-terminal disconnect doesn't
kill the daemon); description extended with paragraphs on the two
persistence cookies (maglev_scope, maglev_zippy_open) and on the
health-cascade icon + lb buckets column.
- docs/maglevc.1: left untouched — intentionally minimal and delegates
to docs/user-guide.md.
Lint (26 issues across 12 files, all errcheck / ineffassign / S1021):
- cmd/frontend/handlers.go: _, _ = fmt.Fprintf(...) for the SSE retry
hint and resync control-event writes.
- cmd/maglevc/commands.go: bulk-prefix every fmt.Fprintf(w, ...) with
_, _ =; also merged 'var watchEventsOptSlot *Node; ... = &Node{...}'
into a single := declaration (staticcheck S1021) — the self-
referencing pattern still works because the Children back-ref is
assigned on the next statement, not inside the struct literal.
- cmd/maglevc/complete.go: _, _ = fmt.Fprintf(ql.rl.Stderr(), ...)
for the banner and help writes; removed the ineffectual
'partial = ""' assignment (nothing downstream reads partial after
that branch, so setting it was dead code flagged by ineffassign).
- cmd/maglevc/shell.go: defer func() { _ = rl.Close() }() for the
readline instance; _, _ = fmt.Fprintf(rl.Stderr(), ...) for error
display in the REPL loop.
- cmd/maglevc/main.go: defer func() { _ = conn.Close() }() for the
gRPC client connection.
- internal/grpcapi/server_test.go: _ = conn.Close() in the test
teardown closure.
- internal/prober/http.go: _ = c.Close() in the TLS-handshake-failed
path; defer func() { _ = conn.Close() }() and defer func() { _ =
resp.Body.Close() }() for the two deferred cleanups.
- internal/prober/http_test.go: defer func() { _ = resp.Body.Close()
}() plus three _, _ = fmt.Fprint(w, ...) in the httptest.Server
handlers and _, _ = fmt.Sscanf(...) when parsing the test listener's
port.
- internal/prober/icmp.go: defer func() { _ = pc.Close() }() for the
ICMP packet conn.
- internal/prober/netns.go: defer func() { _ = origNs.Close() }(),
defer func() { _ = netns.Set(origNs) }(), defer func() { _ =
targetNs.Close() }() — also dropped a stray //nolint:errcheck that
was no longer needed once the closure wrapping handled the discard.
- internal/prober/tcp.go: _ = conn.Close() in the L4-only path,
_ = tlsConn.Close() in the failed and succeeded handshake branches,
_ = tlsConn.SetDeadline(...) (also dropped a //nolint:errcheck
previously covering it).
Iterative 'make lint' runs were needed because golangci-lint v2.x
caps same-linter reports per pass, so the first pass reported 21,
then 4, then 3, then 1, then 0. Final pass: 0 issues. make test is
green across every package, and make build produces all three
binaries cleanly.
|
||
|
Pim van Pelt
|
224167ce39 |
Dataplane reconcile fixes; LB counters cleanup; SPA scope cookie
Checker / reload:
- Reload's update-in-place branch now mirrors b.Address onto the
runtime health.Backend. Without this, GetBackend kept returning
the pre-reload address indefinitely after a config edit that
touched addresses but not healthcheck settings — the VPP sync
path reads cfg.Backends directly so the dataplane moved on
while the gRPC and SPA view stayed wedged on the old IPv4/IPv6.
Sync (internal/vpp/lbsync.go):
- reconcileVIP now detects encap mismatch in addition to
src-ip-sticky mismatch and takes the full tear-down / re-add
path via a new shared recreateVIP helper. Triggered when every
backend flips address family (gre4 <-> gre6) and the existing
VIP can no longer accept new ASes — previously the sync wedged
with 'Invalid address family' until a full maglevd restart.
- setASWeight is issued whenever the state machine requests
flush (a.Flush=true), not only on the weight-value transition
edge. Fixes the case where a backend reached StateDisabled
after its effective weight had already been drained to 0 by
pool failover — the sticky-cache entries pointing at it were
previously never cleared.
maglev-frontend:
- signal.Ignore(SIGHUP) so a controlling-terminal disconnect
doesn't kill the daemon.
- debian/vpp-maglev.service grants CAP_SYS_ADMIN in addition to
CAP_NET_RAW so setns(CLONE_NEWNET) can join the healthcheck
netns. Comment documents the 'operation not permitted' symptom
and notes the knob can be dropped if the deployment doesn't use
the 'netns:' healthcheck option.
LB plugin counters (internal/vpp/lbstats.go + friends):
- Fix the VIP counter regex: the LB plugin registers
vlib_simple_counter_main_t names without a leading '/'
(vlib_validate_simple_counter in counter.c:50 uses cm->name
verbatim; only entries that set cm->stat_segment_name get a
slash). first/next/untracked/no-server now read through as
live values instead of zero.
- Drop the per-backend FIB counter block end-to-end (proto,
grpcapi, metrics, vpp.Client, lbstats, maglevc). Traced from
lb/node.c:558 into ip{4,6}_forward.h:141 — the LB plugin
forwards by writing adj_index[VLIB_TX] directly and bypassing
ip{4,6}_lookup_inline, which is the only path that increments
lbm_to_counters. The backend's FIB load_balance stats_index
literally never ticks for LB-forwarded traffic, so the column
was always zero and misleading. docs/implementation/TODO
records the full investigation and the recommended upstream
path (new lb_as_stats_dump API message) for when we're ready
to carry that VPP patch.
- maglevc show vpp lb counters: plain-text tabular headers.
label() wraps strings in ANSI escapes (~11 bytes of overhead),
but tabwriter counts bytes, not rendered width — so a header
row with label()'d cells and data rows with plain cells drifts
column alignment on every row. color.go comment now spells
out the constraint: label() only works when column N is
wrapped identically in every row (key-value layouts are fine,
multi-column tables with header-only labelling are not).
SPA:
- stores/scope.ts is cookie-backed (maglev_scope, 1 year,
SameSite=Lax). App.tsx hydrates from the cookie then validates
against the fetched snapshots: a cookie referencing a maglevd
that no longer exists falls through to snaps[0] instead of
leaving the user on a ghost selection.
- components/Flash.tsx wraps props.value in createMemo. Solid's
on() fires its callback on every dep notification, not on
value change — source is right in solid-js/dist/solid.js:460,
no equality check. Without the memo, flipping scope between
two 'connected' maglevds (or any other cross-store reactive
re-eval that doesn't actually change the concrete string)
replays the animation every time. createMemo's default ===
dedupe fixes it in one place for every Flash consumer,
superseding the local createMemo workaround we'd added in
BackendRow earlier.
|
||
|
Pim van Pelt
|
1a1c48ef54 |
LB buckets column + health cascade; VPP dump fix; maglevc strictness
SPA (cmd/frontend/web): - New "lb buckets" column backed by a 1s-debounced GetVPPLBState fetch loop with leading+trailing edge coalesce. - Per-frontend health icon (✅/⚠️/❗/‼️/❓) in the Zippy header, gated by a settling flag that suppresses ‼️ until the next lb-state reconciliation after a backend transition or weight change. - In-place leaf merge on lb-state so stable bucket values (e.g. "0") don't retrigger the Flash animation on every refresh. - Zippy cards remember open state in a cookie, default closed on fresh load; fixed-width frontend-title-name + reserved icon slot so headers line up across all cards. - Clock-drift watchdog in sse.ts that forces a fresh EventSource on laptop-wake so the broker emits a resync instead of hanging on a dead half-open socket. Frontend service (cmd/frontend): - maglevClient.lbStateLoop, trigger on backend transitions + vpp-connect, best-effort fetch on refreshAll. - Admin handlers explicitly wake the lb-state loop after lifecycle ops and set-weight (the latter emits no transition event on the maglevd side, so the WatchEvents path wouldn't have caught it). - /favicon.ico served from embedded web/public IPng logo. VPP integration: - internal/vpp/lbstate.go: dumpASesForVIP drops Pfx from the dump request (setting it silently wipes IPv4 replies in the LB plugin) and filters results by prefix on the response side instead, which also demuxes multi-VIP-on-same-port cases correctly. maglevc: - Walk now returns the unconsumed token tail; dispatch and the question listener reject unknown commands with a targeted error instead of dumping the full command tree prefixed with garbage. - On '?', echo the current line (including the '?') before the help list so the output reads like birdc. Checker / prober: - internal/checker: ±10% jitter on NextInterval so probes across restart don't all fire on the same tick. - internal/prober: HTTP User-Agent now carries the build version and project URL. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Pim van Pelt
|
4347bb9b05 |
Bug fixes, config validation, SPA tightening, set-weight UI
This session covers three distinct arcs: correctness bug fixes in the
VPP sync path and frontend reducers, new config validation, and a
large polish pass on the web frontend (tighter layout, backend kebab
dialogs, live grouped-table, live config-reload re-sync).
- encap for a VIP is now derived from the backend address family,
not the VIP's. A v6 VIP with v4 backends is programmed as IP6_GRE4
(not the buggy IP6_GRE6), matching the VPP LB plugin's
requirement that encap reflects the tunnel inner family. desiredVIP
gained an Encap field populated in desiredFromFrontend.
- ActivePoolIndex now requires at least one backend in a pool to be
BOTH in StateUp AND pb.Weight>0 before the pool counts as active.
Previously a primary pool with every backend manually zeroed would
still win over a fallback with weight=100, so fallback traffic
never materialized. New TestActivePoolIndexWeightedFailover table
pins the rule in five subcases.
- SyncLBStateVIP gained a flushAddress parameter threaded through
reconcileVIP; it forces flush=true on the setASWeight call for a
specific backend regardless of the usual 0→N heuristic. Wires up
the explicit [flush] knob the CLI exposes.
- convertFrontend already enforced that backends within one frontend
share a family. New cross-frontend pass validateVIPFamilyConsistency
rejects configs where two frontends share a VIP address but carry
backends in different families — VPP's LB plugin requires every
VIP on a prefix to have the same encap type, so such a config
would fail at lb_add_del_vip_v2 time with VNET_API_ERROR_INVALID
_ARGUMENT (-73). Catching it at config load turns a silent
runtime failure into a clear startup error.
- Two new TestValidationErrors cases pin the behavior: mismatched
families reject, same-family frontends on one VIP address allowed.
- Proto adds `bool flush = 5` to SetWeightRequest. The RPC now
drives a VIP sync immediately after mutating config (fixing the
latent "weight change only takes effect at the next 30s periodic
reconcile" gap), passing flushAddress = backend IP when req.Flush
is true.
- maglevc grows an optional [flush] token: `set frontend F pool P
backend B weight N [flush]`. Implementation uses two Run closures
(runSetFrontendPoolBackendWeight and -Flush) because the tree
walker only puts slot tokens in args — literal keywords like
`flush` advance the node but don't appear in the arg list.
- docs/user-guide.md updated with the [flush] optional and a
three-paragraph explainer of the graceful-drain vs. flush
semantics at the VPP level.
- checker.ListFrontends now sorts alphabetically to match the
existing sort in ListBackends / ListHealthChecks — RPC responses
no longer shuffle VIPs per call. cmd/frontend/client.go also
sorts defensively in refreshAll so an old maglevd build renders
alphabetically too.
- backendFromProto was returning out.Transitions[n-1] as the
LastTransition, but maglevd stores (and the proto carries)
transitions newest-first, so [n-1] was actually the oldest.
Reverse on read, which normalizes the client's Transitions slice
to oldest-first and makes [n-1] genuinely the newest. LastTransition
now points at the actual latest transition record.
- applyBackendTransition (Go and TS) derives Enabled = state!="disabled"
so the two fields stay in lockstep — closed a drift window where
a recently re-enabled backend still rendered with a stuck
[disabled] tag. The tag was later removed entirely since state
and enabled carry the same information.
- Layout tightened substantially: "FRONTENDS" panel header removed,
zippy-summary and zippy-body paddings cut, backend-table row
padding dropped to 2px, per-pool <h3> removed. Pools now live in
a single consolidated table per frontend with a dedicated "pool"
column that shows the pool name only on the first row of each
group — classic grouped-table layout, maximally dense.
- Description moved inline into the Zippy summary as muted italic
text, freeing a vertical line per frontend card.
- formatVIPAddress() helper renders IPv6 VIPs as [addr]:port and
IPv4 as addr:port, matching RFC 3986 authority syntax.
- Pools with effective_weight=0 on every backend (standby
fallbacks, fully-drained primaries) render at opacity 0.35 on
their non-actions cells; the kebab column stays at full contrast
because its menu is still fully functional on standby backends.
- Config-reload propagation: a maglevd config-reload-done log
event triggers triggerConfigResync() on the frontend side —
refreshAll() runs off the event-dispatch goroutine, then a
BrowserEvent{Type:"resync"} is published through the broker.
writeEvent emits type="resync" as a named SSE frame so the
SPA's existing addEventListener("resync") handler picks it up
and calls fetchAllState → replaceAll.
- recomputeEffectiveWeights in stores/state.ts mirrors the
server-side health.EffectiveWeights logic so the SPA keeps
pool.effective_weight correct the moment a backend transitions,
without waiting for the 30s refresh. Fixed a nasty bug where
applyBackendEffectiveWeight wrote VIP-scoped vpp-lb-sync-as-*
event weights into every frontend sharing the backend,
corrupting frontends with different per-pool configured weights.
The old log-event reducer was removed; applyConfiguredWeight is
the narrower replacement used by the kebab set-weight flow.
- applyBackendTransition calls recomputeEffectiveWeights after
state updates so pool-failover transitions (primary ⇌ fallback)
reflect instantly in the UI.
- Confirmation dialogs via a new Modal primitive
(Portal-mounted to document.body, escape/click-outside close,
click-outside debounced on mousedown so mid-row-text-selection
drags don't dismiss).
- pause/resume/enable/disable each show a Modal with a consequence
paragraph explaining what hits live traffic ("will keep existing
flows", "will flush VPP's flow table", etc.). The disable commit
button is styled btn-danger red.
- set-weight action shows a Modal with a range slider (0-100,
seeded from the current configured weight, accent-colored live
numeric readout via <output>) plus a flush checkbox and a live-
swapping note/warn paragraph describing what will happen. On
commit, the SPA also updates its local store via
applyConfiguredWeight so the operator sees the new weight
immediately without waiting for the next refresh.
- ProbeHeartbeat is now state-aware: ▶ (play) at rest for up/
down/unknown backends, ⏸ (pause) for paused, ⏹ (stop) for
disabled/removed, ❤️ (heart) during an in-flight probe.
- Drop the probe-done event listener — fast probes (<10ms)
could fire probe-done in the same render tick as probe-start
and the heart would never visibly paint. Each probe-start now
runs a fixed 400ms scale-pop animation on a timer; subsequent
probe-start events reset the timer, so fast cadences produce a
continuous heart pulse.
- Fixed wrapper box (16x14 px, overflow hidden) so the row
doesn't jiggle when the glyph swaps between the narrow ▶/⏸/⏹
text glyphs and the wider ❤️ emoji.
- Brand wordmark changed from "maglev" to "vpp-maglev" and wrapped
in an <a> linking to https://git.ipng.ch/ipng/vpp-maglev. Logo
link changed to https://ipng.ch/. Both open in a new tab with
rel="noopener".
- .gitignore fix: `frontend`, `maglevc`, `maglevd` were matching
ANY file or directory with those names anywhere in the tree,
silently ignoring cmd/frontend and friends. Anchored with
leading slashes so only repo-root build artifacts match.
|
||
|
Pim van Pelt
|
fb62532fd5 |
VPP LB counters, src-ip-sticky, and frontend state aggregation
New feature: per-VIP / per-backend runtime counters
* New GetVPPLBCounters RPC serving an in-process snapshot refreshed
by a 5s scrape loop (internal/vpp/lbstats.go). Each cycle pulls
the LB plugin's four SimpleCounters (next, first, untracked,
no-server) plus the FIB /net/route/to CombinedCounter for every
VIP and every backend host prefix via a single DumpStats call.
* FIB stats-index discovery via ip_route_lookup (internal/vpp/
fibstats.go); per-worker reduction happens in the collector.
* Prometheus collector exports vip_packets_total (kind label),
vip_route_{packets,bytes}_total, and backend_route_{packets,
bytes}_total. Metrics source interface extended with VIPStats /
BackendRouteStats; vpp.Client publishes snapshots via
atomic.Pointer and clears them on disconnect.
* New 'show vpp lb counters' CLI command. The 'show vpp lbstate'
and 'sync vpp lbstate' commands are restructured under 'show
vpp lb {state,counters}' / 'sync vpp lb state' to make room
for the new verb.
New feature: src-ip-sticky frontends
* New frontend YAML key 'src-ip-sticky' (bool). Plumbed through
config.Frontend, desiredVIP, and the lb_add_del_vip_v2 call.
* Reflected in gRPC FrontendInfo.src_ip_sticky and VPPLBVIP.
src_ip_sticky, and shown in 'show vpp lb state' output.
* Scraped back from VPP by parsing 'show lb vips verbose' through
cli_inband — lb_vip_details does not expose the flag. The same
scrape also recovers the LB pool index for each VIP, which the
stats-segment counters are keyed on. This is a documented
temporary workaround until VPP ships an lb_vip_v2_dump.
* src_ip_sticky cannot be mutated on a live VIP, so a flipped flag
triggers a tear-down-and-recreate in reconcileVIP (ASes deleted
with flush, VIP deleted, then re-added). Flip is logged.
New feature: frontend state aggregation and events
* New health.FrontendState (unknown/up/down) and FrontendTransition
types. A frontend is 'up' iff at least one backend has a nonzero
effective weight, 'unknown' iff no backend has real state yet,
and 'down' otherwise.
* Checker tracks per-frontend aggregate state, recomputing after
each backend transition and emitting a frontend-transition Event
on change. Reload drops entries for removed frontends.
* checker.Event gains an optional FrontendTransition pointer;
backend- vs. frontend-transition events are demultiplexed on
that field.
* WatchEvents now sends an initial snapshot of frontend state on
connect (mirroring the existing backend snapshot), subscribes
once to the checker stream, and fans out to backend/frontend
handlers based on the client's filter flags. The proto
FrontendEvent message grows name + transition fields.
* New Checker.FrontendState accessor.
Refactor: pure health helpers
* Moved the priority-failover selector and the (pool idx, active
pool, state, cfg weight) → (vpp weight, flush) mapping out of
internal/vpp/lbsync.go into a new internal/health/weights.go so
the checker can reuse them for frontend-state computation
without importing internal/vpp.
* New functions: health.ActivePoolIndex, BackendEffectiveWeight,
EffectiveWeights, ComputeFrontendState. lbsync.go now calls
these directly; vpp.EffectiveWeights is a thin wrapper over
health.EffectiveWeights retained for the gRPC observability
path. Fully unit-tested in internal/health/weights_test.go.
maglevc polish
* --color default is now mode-aware: on in the interactive shell,
off in one-shot mode so piped output is script-safe. Explicit
--color=true/false still overrides.
* New stripHostMask helper drops /32 and /128 from VIP display;
non-host prefixes pass through unchanged.
* Counter table column order fixed (first before next) and
packets/bytes columns renamed to fib-packets/fib-bytes to
clarify they come from the FIB, not the LB plugin.
Docs
* config-guide: document src-ip-sticky, including the VIP
recreate-on-change caveat.
* user-guide, maglevc.1, maglevd.8: updated command tree, new
counters command, color defaults, and the src-ip-sticky field.
|
||
|
Pim van Pelt
|
0049c2ae73 |
VPP reconciler: event-driven sync, pool failover, bug fixes
This commit wires the checker's state machine through to the VPP dataplane:
every backend state transition flows through a single code path that
recomputes the effective per-backend weight (with pool failover) and pushes
the result to VPP. Along the way several latent bugs in the state machine
and the sync path were fixed.
internal/vpp/reconciler.go (new)
- New Reconciler type subscribes to checker.Checker events and, on every
transition, calls Client.SyncLBStateVIP for the affected frontend. This
is the ONLY place in the codebase where backend state changes cause VPP
calls — the "single path" discipline requested during design.
- Defines an EventSource interface (checker.Checker satisfies it) so the
dependency direction stays vpp → checker; the checker never imports vpp.
internal/vpp/client.go
- Renamed ConfigSource → StateSource. The interface now has two methods:
Config() and BackendState(name) — the reconciler and the desired-state
builder both need live health state to compute effective weights.
- SetConfigSource → SetStateSource; internal cfgSrc field → stateSrc.
- New getStateSource() helper for internal locked access.
- lbSyncLoop still uses the state source for its periodic drift
reconciliation; it's fully idempotent and runs the same code path as
event-driven syncs.
internal/vpp/lbsync.go
- desiredAS grows a Flush bool so the mapping function can signal "on
transition to weight 0, flush existing flow-table entries".
- asFromBackend is now the single source of truth for the state →
(weight, flush) rule. Documented with a full truth table. Takes an
activePool parameter so it can distinguish "up in active pool" from
"up but standby".
- activePoolIndex(fe, states) implements priority failover: returns the
index of the first pool containing any StateUp backend. pool[0] wins
when at least one member is up; pool[1] takes over when pool[0] is
empty; and so on. Defaults to 0 (unobservable, since all backends map
to weight 0 when nothing is up).
- desiredFromFrontend snapshots backend states once, computes activePool,
then walks every backend through asFromBackend. No more filtering on
b.Enabled — disabled backends stay in the desired set so they keep
their AS entry in VPP with weight=0. The previous filter caused delAS
on disable, which destroyed the entry and broke enable afterwards.
- EffectiveWeights(fe, src) exported helper that returns the per-pool
per-backend weight map for one frontend. Used by the gRPC GetFrontend
handler and robot tests to observe failover without touching VPP.
- reconcileVIP computes flush at the weight-change call site:
flush = desired.Flush && cur.Weight > 0 && desired.Weight == 0
This ensures only the *transition* to disabled flushes sessions —
steady-state syncs with already-zero weight skip the call entirely.
- setASWeight now plumbs IsFlush into lb_as_set_weight.
internal/vpp/lbsync_test.go (new)
- TestAsFromBackend: 15 cases locking down the truth table, including
failover scenarios (up in standby pool, up promoted in pool[1]).
- TestActivePoolIndex: 8 cases covering pool[0]-has-up, pool[0]-all-down,
all-disabled, all-paused, all-unknown, nothing-up-anywhere, and
three-tier failover.
- TestDesiredFromFrontendFailover: 5 end-to-end scenarios wiring a fake
StateSource through desiredFromFrontend and asserting the final
per-IP weight map. Exercises the complete pipeline without VPP.
internal/checker/checker.go
- Added BackendState(name) (health.State, bool) — one-line method that
satisfies vpp.StateSource. The checker is otherwise unchanged.
- EnableBackend rewritten to reuse the existing worker (parallel to
ResumeBackend). The old code called startWorker which constructed a
brand-new Backend via health.New, throwing away the transition
history; the resulting 'backend-transition' log showed the bogus
from=unknown,to=unknown. Now uses w.backend.Enable() to record a
proper disabled→unknown transition and launches a fresh goroutine.
- Static (no-healthcheck) backends now fire their synthetic 'always up'
pass on the first iteration of runProbe instead of sleeping 30s
first. Previously static backends sat in StateUnknown for 30s after
startup — useless for deterministic testing and surprising for
operators. The fix is a simple first-iteration flag.
internal/health/state.go
- New Enable(maxHistory) method parallel to Disable. Transitions the
backend from whatever state it's in (typically StateDisabled) to
StateUnknown, resets the health counter to rise-1 so the expedited
resolution kicks in on the first probe result, and emits a transition
with code 'enabled'.
proto/maglev.proto
- PoolBackendInfo gains effective_weight: the state-aware weight that
would be programmed into VPP (distinct from the configured weight in
the YAML). Exposed via GetFrontend.
internal/grpcapi/server.go
- frontendToProto takes a vpp.StateSource, computes effective weights
via vpp.EffectiveWeights, and populates PoolBackendInfo.EffectiveWeight.
- GetFrontend and SetFrontendPoolBackendWeight updated to pass the
checker in.
cmd/maglevc/commands.go
- 'show frontends <name>' now renders every pool backend row as
<name> weight <cfg> effective <eff> [disabled]?
so both values are always visible. The VPP-style key/value format
avoids the ANSI-alignment pitfall we hit earlier and makes the output
regex-parseable for robot tests.
cmd/maglevd/main.go
- Construct and start the Reconciler alongside the VPP client. Two
extra lines, no other changes to startup.
tests/01-maglevd/maglevd-lab/maglev.yaml
- Two new static backends (static-primary, static-fallback) and a new
failover-vip frontend with one backend per pool. No healthcheck, so
the state machine resolves them to 'up' immediately via the synthetic
pass. Used by the failover robot tests.
tests/01-maglevd/01-healthcheck.robot
- Three new test cases exercising pool failover end-to-end:
1. primary up, secondary standby (initial state)
2. disable primary → fallback takes over (effective weight flips)
3. enable primary → fallback steps back
All run without VPP: they scrape 'maglevc show frontends <name>' and
regex-match the effective weight in the output. Deterministic and
fast (~2s total) because the static backends don't probe.
- Two helper keywords: Static Backend Should Be Up and
Effective Weight Should Be.
Net result: 16/16 robot tests pass. Backend state transitions now
flow through a single documented path (checker event → reconciler →
SyncLBStateVIP → desiredFromFrontend → asFromBackend → reconcileVIP →
setASWeight), and the pool failover / enable-after-disable / static-
backend-startup bugs are all fixed.
|
||
|
Pim van Pelt
|
d3c5c86037 |
VPP load-balancer dataplane integration: state, sync, and global conf
This commit wires maglevd through to VPP's LB plugin end-to-end, using
locally-generated GoVPP bindings for the newer v2 API messages.
VPP binapi (vendored)
- New package internal/vpp/binapi/ containing lb, lb_types, ip_types, and
interface_types, generated from a local VPP build (~/src/vpp) via a new
'make vpp-binapi' target. GoVPP v0.12.0 upstream lacks the v2 messages we
need (lb_conf_get, lb_add_del_vip_v2, lb_add_del_as_v2, lb_as_v2_dump,
lb_as_set_weight), so we commit the generated output in-tree.
- All generated files go through our loggedChannel wrapper; every VPP API
send/receive is recorded at DEBUG via slog (vpp-api-send / vpp-api-recv /
vpp-api-send-multi / vpp-api-recv-multi) so the full wire-level trail is
auditable. NewAPIChannel is unexported — callers must use c.apiChannel().
Read path: GetLBState{All,VIP}
- GetLBStateAll returns a full snapshot (global conf + every VIP with its
attached application servers).
- GetLBStateVIP looks up a single VIP by (prefix, protocol, port) and
returns (nil, nil) when the VIP doesn't exist in VPP. This is the
efficient path for targeted updates on a busy LB.
- Helpers factored out: getLBConf, dumpAllVIPs, dumpASesForVIP, lookupVIP,
vipFromDetails.
Write path: SyncLBState{All,VIP}
- SyncLBStateAll reconciles every configured frontend with VPP: creates
missing VIPs, removes stale ones (with AS flush), and reconciles AS
membership and weights within VIPs that exist on both sides.
- SyncLBStateVIP targets a single frontend by name. Never removes VIPs.
Returns ErrFrontendNotFound (wrapped with the name) when the frontend
isn't in config, so callers can use errors.Is.
- Shared reconcileVIP helper does the per-VIP AS diff; removeVIP is used
only by the full-sync pass.
- LbAddDelVipV2 requests always set NewFlowsTableLength=1024. The .api
default=1024 annotation is only applied by VAT/CLI parsers, not wire-
level marshalling — sending 0 caused VPP to vec_validate with mask
0xFFFFFFFF and OOM-panic.
- Pool semantics: backends in the primary (first) pool of a frontend get
their configured weight; backends in secondary pools get weight 0. All
backends are installed so higher layers can flip weights on failover
without add/remove churn.
- Every individual change emits a DEBUG slog (vpp-lbsync-vip-add/del,
vpp-lbsync-as-add/del, vpp-lbsync-as-weight). Start/done INFO logs
carry a scope=all|vip label plus aggregate counts.
Global conf push: SetLBConf
- New SetLBConf(cfg) sends lb_conf with ipv4-src, ipv6-src, sticky-buckets,
and flow-timeout. Called automatically on VPP (re)connect and after
every config reload (via doReloadConfig). Results are cached on the
Client so redundant pushes are silently skipped — only actual changes
produce a vpp-lb-conf-set INFO log line.
Periodic drift reconciliation
- vpp.Client.lbSyncLoop runs in a goroutine tied to each VPP connection's
lifetime. Its first tick is immediate (startup and post-reconnect
sync quickly); subsequent ticks fire every vpp.lb.sync-interval from
config (default 30s). Purpose: catch drift if something/someone
modifies VPP state by hand. The loop uses a ConfigSource interface
(satisfied by checker.Checker via its new Config() accessor) to avoid
an import cycle with the checker package.
Config schema additions (maglev.vpp.lb)
- sync-interval: positive Go duration, default 30s.
- ipv4-src-address: REQUIRED. Used as the outer source for GRE4 encap
to application servers. Missing this is a hard semantic error —
maglevd --check exits 2 and the daemon refuses to start. VPP GRE
needs a source address and every VIP we program uses GRE, so there
is no meaningful config without it.
- ipv6-src-address: REQUIRED. Same treatment as ipv4-src-address.
- sticky-buckets-per-core: default 65536, must be a power of 2.
- flow-timeout: default 40s, must be a whole number of seconds in [1s, 120s].
- VPP validation runs at the end of convert() so structural errors in
healthchecks/backends/frontends surface first — operators fix those,
then get the VPP-specific requirements.
gRPC API
- New GetVPPLBState RPC returning VPPLBState: global conf + VIPs with
ASes. Mirrors the read-path but strips fields irrelevant to our
GRE-only deployment (srv_type, dscp, target_port).
- New SyncVPPLBState RPC with optional frontend_name. Unset → full sync
(may remove stale VIPs). Set → single-VIP sync (never removes).
Returns codes.NotFound for unknown frontends, codes.Unavailable when
VPP integration is disabled or disconnected.
maglevc (CLI)
- New 'show vpp lbstate' command displaying the LB plugin state. VPP-only
fields the dataplane irrelevant to GRE are suppressed. Per-AS lines use
a key-value format ("address X weight Y flow-table-buckets Z")
instead of a tabwriter column, which avoids the ANSI-color alignment
issue we hit with mixed label/data rows.
- New 'sync vpp lbstate [<name>]' command. Without a name, triggers a
full reconciliation; with a name, targets one frontend.
- Previous 'show vpp lb' renamed to 'show vpp lbstate' for consistency
with the new sync command.
Test fixtures
- validConfig and all ad-hoc config_test.go fixtures that reach the end
of convert() now include the two required vpp.lb src addresses.
- tests/01-maglevd/maglevd-lab/maglev.yaml gains a vpp.lb section so the
robot integration tests can still load the config.
- cmd/maglevc/tree_test.go gains expected paths for the new commands.
Docs
- config-guide.md: new 'vpp' section in the basic structure, detailed
vpp.lb field reference, noting ipv4/ipv6 src addresses as REQUIRED
(hard error) with no defaults; example config updated.
- user-guide.md: documented 'show vpp info', 'show vpp lbstate',
'sync vpp lbstate [<name>]', new --vpp-api-addr and --vpp-stats-addr
flags, the vpp-lb-conf-set log line, and corrected the pause/resume
description to reflect that pause cancels the probe goroutine.
- debian/maglev.yaml: example config gains a vpp.lb block with src
addresses and commented optional overrides.
|
||
|
Pim van Pelt
|
3227263d68 |
Add GoVPP integration and GetVPPInfo gRPC call
VPP client (internal/vpp/) - New package managing connections to both VPP API and stats sockets, treated as a unit: if either drops, both are torn down and re-established together. - Run() loop: connect, fetch version via vpe.ShowVersion, read /sys/boottime from the stats segment, log vpp-connect, then monitor with control_ping every 10s. On failure, disconnect both, retry after 5s. - Registers as client name "vpp-maglev" (visible in VPP's "show api clients"). - Flags: --vpp-api-addr (default /run/vpp/api.sock) and --vpp-stats-addr (default /run/vpp/stats.sock). Empty api addr disables VPP integration entirely. gRPC / proto - Add GetVPPInfo RPC returning VPPInfo: version, build_date, build_directory, pid, boottime_ns, connecttime_ns. Both times are unix timestamps in nanoseconds — the client computes durations locally for display. - Returns codes.Unavailable if VPP is disabled or not connected. maglevc - Add 'show vpp info' command displaying version, build-date, build-dir, vpp-pid, vpp-boottime (with duration), and connected time (with duration). |
||
|
Pim van Pelt
|
74448cf6d0 |
Guard pause/resume against disabled backends; clean up CLI errors
- PauseBackend and ResumeBackend return an error (not bool) when the
backend is disabled, preventing an inconsistent state where the
health state says "paused" but enabled=false.
- DisableBackend and EnableBackend now log uniform backend-transition
lines with from/to instead of separate backend-disable/backend-enable
messages.
- CLI errors strip gRPC boilerplate ("rpc error: code = ... desc = ")
and display the server message in red (when color is enabled). Both
the interactive shell and one-shot mode use the same formatError path.
|
||
|
Pim van Pelt
|
3bd30b69f4 |
Refactor CLI: birdc-style help, collapsed nouns, ReloadConfig, bug fixes
maglevc - Rewrite '?' handler (birdc-style): show full command paths from current position to every leaf, right-aligned help column, dynamic slot values displayed as an indented block when cursor is at a slot position. - Collapse show frontends/frontend, backends/backend, healthchecks/healthcheck into single plural-noun nodes with an optional <name> slot. Allows 'sh ba' (list all) and 'sh ba nginx0' (show one) without ambiguity. - Add 'config reload' command. - Fix tabwriter ANSI alignment: continuation lines in transition output now carry the same label() byte overhead as the header line. - Fix broken Walk for 'set frontend' command: setFrontendPoolName and setWeightValue were fixed-word nodes that couldn't capture user input; mark them as slot nodes with dynNone. - Add tree_test.go covering expandPaths, cycle detection, prefix matching, and the full weight-command walk. gRPC / proto - Add ReloadConfig RPC: checks config then applies it to the running checker, returning ok/parse_error/semantic_error/reload_error. - Add logging to CheckConfig (config-check-start/config-check-done at INFO level). maglevd - SIGHUP handler now calls maglevServer.TriggerReload(), sharing the same code path as the gRPC ReloadConfig RPC. docs - Collapse show command documentation to use [<name>] optional syntax. - Remove developer-facing 'Command tree and parser' section. - Document 'config reload'. |
||
|
Pim van Pelt
|
58391f5463 |
Add WatchEvents, enable/disable/weight RPCs, and config check
gRPC / proto - Rename WatchBackendEvents → WatchEvents; return a stream of Event oneof (LogEvent, BackendEvent, FrontendEvent) with optional filter flags (log, log_level, backend, frontend) - Add EnableBackend, DisableBackend, SetFrontendPoolBackendWeight RPCs - Rename PauseResumeRequest → BackendRequest - Add CheckConfig RPC returning ok/parse_error/semantic_error maglevd - Route slog through a LogBroadcaster (slog.Handler) so WatchEvents subscribers can receive structured log records independently of the daemon's own --log-level - Add --reflection flag (default true) to toggle gRPC server reflection - Add --check flag: validates config file and exits 0/1/2 - SIGHUP: use config.Check before applying reload; log parse vs semantic error separately; refuse reload on any error - Rename default config path /etc/maglev → /etc/vpp-maglev maglevc - Add 'watch events [num <n>] [log [level <level>]] [backend] [frontend]' command; prints compact protojson, stops on any keypress or Ctrl-C; uses cbreak mode (not raw) so output post-processing is preserved - Add 'set backend <name> enable|disable' - Add 'set frontend <name> pool <pool> backend <name> weight <0-100>' - Add 'config check' command Debian packaging - Rename service unit to vpp-maglevd.service - Rename conffiles to /etc/default/vpp-maglev and /etc/vpp-maglev/ - Create maglevd system user/group in postinst; add to vpp group if present - Add postrm; add adduser to Depends |
||
|
Pim van Pelt
|
d612086a5f |
Pools, CLI, versioning, Debian packaging, HTTPS fix
- Replaced flat `backends: [...]` list on frontends with an ordered `pools:`
list; each pool has a name and a map of backends with per-pool weights (0–100,
default 100). Pools express priority: first pool with a healthy backend wins.
- Removed global backend weight (was on the backend, now lives in the pool).
- Config validation enforces non-empty pools, non-empty pool names, weight
range, and consistent address families across all pools of a frontend.
- Added `PoolBackendInfo { name, weight }` and changed `PoolInfo.backends` from
`repeated string` to `repeated PoolBackendInfo` so weights are visible over
the API.
- Full interactive shell with readline, tab completion, and `?` inline help.
- Command tree parser (Walk) handles fixed keywords and dynamic slot nodes;
prefix matching with exact-match priority.
- Commands: `show version/frontends/frontend/backends/backend/healthchecks/
healthcheck`, `set backend <name> pause|resume`, `quit`/`exit`.
- `show frontend` output is hierarchical (pools → backends) with per-backend
weights and `[disabled]` notation; pool section uses fixed-width formatting
so ANSI color codes don't corrupt tabwriter alignment.
- `-color` flag (default true) wraps static field labels in dark-blue ANSI;
works correctly with tabwriter because all labels carry identical-length
escape sequences.
- `cmd/version.go` package holds `version`, `commit`, `date` vars set at build
time via `-ldflags -X`.
- `make build` / `make build-amd64` / `make build-arm64` all inject
`VERSION=0.1.1`, `COMMIT_HASH` (from `git rev-parse --short HEAD`), and
`DATE` (UTC ISO-8601).
- `maglevc` prints version on interactive startup and exposes `show version`.
- `maglevd` logs version/commit/date at startup; `-version` flag prints and exits.
- `doHTTPProbe` was building a `https://` target URL even though TLS was already
applied to the connection inside `inNetns`. `http.Transport` then wrapped the
connection in a second TLS layer, producing "http: server gave HTTP response
to HTTPS client". Fixed by always using `http://` in the target URL.
- Added `TestHTTPSProbe` using `httptest.NewTLSServer` to cover the full path.
- New `docs/user-guide.md`: maglevd flags/signals, maglevc commands, shell
completion, and command-tree parser walkthrough.
- New `docs/healthchecks.md`: state machine, rise/fall model, probe intervals,
all transition events with log examples.
- Updated `docs/config-guide.md`: pools design, removed global weight from
backends, updated all examples.
- Updated `README.md`: packaging table, build paths, corrected binary locations
(`/usr/sbin/maglevd`), config filename (`.yaml`).
- `debian/` directory contains `control.in`, `maglevd.service`, `default.maglev`,
`maglev.yaml` (example config), `conffiles`, `postinst`, `prerm`.
- `debian/build-deb.sh` stages a package tree and calls `dpkg-deb`; emits
`build/vpp-maglev_<version>~<commit>_<arch>.deb`.
- Cross-compiles for amd64 and arm64 in one `make pkg-deb` invocation.
- `maglevd` installed to `/usr/sbin/`, `maglevc` to `/usr/bin/`.
- Service reads `MAGLEV_CONFIG` from `/etc/default/maglev`
(default: `/etc/maglev/maglev.yaml`).
- Man pages `maglevd(8)` and `maglevc(1)` live in `docs/` and are gzip'd into
the package.
- All build output goes to `build/<arch>/`; `build/` is gitignored.
|
||
|
Pim van Pelt
|
ad7d7e20fc | go fmt | ||
|
Pim van Pelt
|
56a4a6ba25 | Some output tweaks; and some additional transition events upon resume (paused->unknown->{up|down}) | ||
|
Pim van Pelt
|
46e78ec36f | First stab at maglevc |