Pim van Pelt
d3c5c86037
This commit wires maglevd through to VPP's LB plugin end-to-end, using
locally-generated GoVPP bindings for the newer v2 API messages.
VPP binapi (vendored)
- New package internal/vpp/binapi/ containing lb, lb_types, ip_types, and
interface_types, generated from a local VPP build (~/src/vpp) via a new
'make vpp-binapi' target. GoVPP v0.12.0 upstream lacks the v2 messages we
need (lb_conf_get, lb_add_del_vip_v2, lb_add_del_as_v2, lb_as_v2_dump,
lb_as_set_weight), so we commit the generated output in-tree.
- All generated files go through our loggedChannel wrapper; every VPP API
send/receive is recorded at DEBUG via slog (vpp-api-send / vpp-api-recv /
vpp-api-send-multi / vpp-api-recv-multi) so the full wire-level trail is
auditable. NewAPIChannel is unexported — callers must use c.apiChannel().
Read path: GetLBState{All,VIP}
- GetLBStateAll returns a full snapshot (global conf + every VIP with its
attached application servers).
- GetLBStateVIP looks up a single VIP by (prefix, protocol, port) and
returns (nil, nil) when the VIP doesn't exist in VPP. This is the
efficient path for targeted updates on a busy LB.
- Helpers factored out: getLBConf, dumpAllVIPs, dumpASesForVIP, lookupVIP,
vipFromDetails.
Write path: SyncLBState{All,VIP}
- SyncLBStateAll reconciles every configured frontend with VPP: creates
missing VIPs, removes stale ones (with AS flush), and reconciles AS
membership and weights within VIPs that exist on both sides.
- SyncLBStateVIP targets a single frontend by name. Never removes VIPs.
Returns ErrFrontendNotFound (wrapped with the name) when the frontend
isn't in config, so callers can use errors.Is.
- Shared reconcileVIP helper does the per-VIP AS diff; removeVIP is used
only by the full-sync pass.
- LbAddDelVipV2 requests always set NewFlowsTableLength=1024. The .api
default=1024 annotation is only applied by VAT/CLI parsers, not wire-
level marshalling — sending 0 caused VPP to vec_validate with mask
0xFFFFFFFF and OOM-panic.
- Pool semantics: backends in the primary (first) pool of a frontend get
their configured weight; backends in secondary pools get weight 0. All
backends are installed so higher layers can flip weights on failover
without add/remove churn.
- Every individual change emits a DEBUG slog (vpp-lbsync-vip-add/del,
vpp-lbsync-as-add/del, vpp-lbsync-as-weight). Start/done INFO logs
carry a scope=all|vip label plus aggregate counts.
Global conf push: SetLBConf
- New SetLBConf(cfg) sends lb_conf with ipv4-src, ipv6-src, sticky-buckets,
and flow-timeout. Called automatically on VPP (re)connect and after
every config reload (via doReloadConfig). Results are cached on the
Client so redundant pushes are silently skipped — only actual changes
produce a vpp-lb-conf-set INFO log line.
Periodic drift reconciliation
- vpp.Client.lbSyncLoop runs in a goroutine tied to each VPP connection's
lifetime. Its first tick is immediate (startup and post-reconnect
sync quickly); subsequent ticks fire every vpp.lb.sync-interval from
config (default 30s). Purpose: catch drift if something/someone
modifies VPP state by hand. The loop uses a ConfigSource interface
(satisfied by checker.Checker via its new Config() accessor) to avoid
an import cycle with the checker package.
Config schema additions (maglev.vpp.lb)
- sync-interval: positive Go duration, default 30s.
- ipv4-src-address: REQUIRED. Used as the outer source for GRE4 encap
to application servers. Missing this is a hard semantic error —
maglevd --check exits 2 and the daemon refuses to start. VPP GRE
needs a source address and every VIP we program uses GRE, so there
is no meaningful config without it.
- ipv6-src-address: REQUIRED. Same treatment as ipv4-src-address.
- sticky-buckets-per-core: default 65536, must be a power of 2.
- flow-timeout: default 40s, must be a whole number of seconds in [1s, 120s].
- VPP validation runs at the end of convert() so structural errors in
healthchecks/backends/frontends surface first — operators fix those,
then get the VPP-specific requirements.
gRPC API
- New GetVPPLBState RPC returning VPPLBState: global conf + VIPs with
ASes. Mirrors the read-path but strips fields irrelevant to our
GRE-only deployment (srv_type, dscp, target_port).
- New SyncVPPLBState RPC with optional frontend_name. Unset → full sync
(may remove stale VIPs). Set → single-VIP sync (never removes).
Returns codes.NotFound for unknown frontends, codes.Unavailable when
VPP integration is disabled or disconnected.
maglevc (CLI)
- New 'show vpp lbstate' command displaying the LB plugin state. VPP-only
fields the dataplane irrelevant to GRE are suppressed. Per-AS lines use
a key-value format ("address X weight Y flow-table-buckets Z")
instead of a tabwriter column, which avoids the ANSI-color alignment
issue we hit with mixed label/data rows.
- New 'sync vpp lbstate [<name>]' command. Without a name, triggers a
full reconciliation; with a name, targets one frontend.
- Previous 'show vpp lb' renamed to 'show vpp lbstate' for consistency
with the new sync command.
Test fixtures
- validConfig and all ad-hoc config_test.go fixtures that reach the end
of convert() now include the two required vpp.lb src addresses.
- tests/01-maglevd/maglevd-lab/maglev.yaml gains a vpp.lb section so the
robot integration tests can still load the config.
- cmd/maglevc/tree_test.go gains expected paths for the new commands.
Docs
- config-guide.md: new 'vpp' section in the basic structure, detailed
vpp.lb field reference, noting ipv4/ipv6 src addresses as REQUIRED
(hard error) with no defaults; example config updated.
- user-guide.md: documented 'show vpp info', 'show vpp lbstate',
'sync vpp lbstate [<name>]', new --vpp-api-addr and --vpp-stats-addr
flags, the vpp-lb-conf-set log line, and corrected the pause/resume
description to reflect that pause cancels the probe goroutine.
- debian/maglev.yaml: example config gains a vpp.lb block with src
addresses and commented optional overrides.
8.9 KiB
8.9 KiB
User Guide
maglevd
maglevd is the health-checker daemon. It probes backends according to the
configuration file, maintains their health state, and exposes a gRPC API for
inspection and control.
Flags
| Flag | Environment variable | Default | Description |
|---|---|---|---|
--config |
MAGLEV_CONFIG |
/etc/vpp-maglev/maglev.yaml |
Path to the YAML configuration file. |
--grpc-addr |
MAGLEV_GRPC_ADDR |
:9090 |
TCP address on which the gRPC server listens. |
--metrics-addr |
MAGLEV_METRICS_ADDR |
:9091 |
TCP address for the Prometheus /metrics HTTP endpoint. Set to empty to disable. |
--vpp-api-addr |
MAGLEV_VPP_API_ADDR |
/run/vpp/api.sock |
VPP binary API socket path. Set to empty to disable VPP integration. |
--vpp-stats-addr |
MAGLEV_VPP_STATS_ADDR |
/run/vpp/stats.sock |
VPP stats socket path. |
--log-level |
MAGLEV_LOG_LEVEL |
info |
Log verbosity: debug, info, warn, or error. |
--check |
— | — | Read and validate the config file, then exit. Exits 0 if the config is valid, 1 on YAML parse error, 2 on semantic error. |
--reflection |
— | true |
Enable gRPC server reflection. Allows grpcurl to introspect the API without the .proto file. Set to false to disable. |
--version |
— | — | Print version, commit hash, and build date, then exit. |
Flags take precedence over environment variables. Both are optional; defaults are used for anything not set.
Signals
| Signal | Effect |
|---|---|
SIGHUP |
Reload the configuration file (same code path as config reload in maglevc). The file is checked before applying; if there is a parse or semantic error the reload is aborted and the error is logged (the daemon continues running with its current config). New backends are started, removed backends are stopped, backends whose health-check config is unchanged continue probing without interruption. |
SIGTERM / SIGINT |
Graceful shutdown. Active gRPC streams are closed, the server drains, then the process exits. |
Capabilities
maglevd requires CAP_NET_RAW when any health check uses type: icmp.
All other check types (tcp, http) use normal TCP sockets and require no
special capabilities.
Logging
All log output is written to stdout as JSON using Go's log/slog. The first
line logged after the logger is configured is a starting record that includes
version, commit, and date. Every state change emits a backend-transition
line at INFO level. Set --log-level debug to see individual probe attempts
and their outcomes.
maglevc
maglevc is the interactive control-plane client. It connects to a running
maglevd over gRPC and either executes a single command or drops into an
interactive shell.
Usage
maglevc [--server host:port] [--color[=bool]] [command...]
| Flag | Default | Description |
|---|---|---|
--server |
localhost:9090 |
Address of the maglevd gRPC server. |
--color |
true |
Colorize static field labels in output (dark blue ANSI). Pass --color=false to disable, e.g. when piping. |
When command arguments are supplied the command is executed and maglevc
exits. When no arguments are given an interactive shell is started and the
build version is printed on entry.
Commands
show version Print build version, commit hash, and build date.
show frontends [<name>] Without name: list all frontend names.
With name: show address, protocol, port, description,
and pools. Each pool lists its backends with weights
(if != 100) and marks disabled backends with [disabled].
show backends [<name>] Without name: list all backend names.
With name: show address, current state (with duration),
enabled flag, health check, and recent state transitions
with timestamps and how long ago each occurred.
show healthchecks [<name>] Without name: list all health-check names.
With name: show full health-check configuration.
show vpp info Show VPP version, build date, PID, uptime, and when
maglevd connected. Returns an error if VPP is not
connected.
show vpp lbstate Show the VPP load-balancer plugin state: global
configuration, configured VIPs, and their attached
application servers (address, weight, bucket count).
Returns an error if VPP is not connected.
sync vpp lbstate [<name>] Reconcile the VPP load-balancer dataplane from the
running config. Without a name: runs a full sync —
creates missing VIPs, removes stale VIPs, and adjusts
application-server membership and weights across all
frontends. With a name: only the named frontend's VIP
is reconciled, and no VIPs are removed. A full sync
also runs automatically every
maglev.vpp.lb.sync-interval (default 30s) to catch
drift, and once on startup.
set backend <name> pause Stop health checking for a backend. Cancels the probe
goroutine so no further traffic is sent, and freezes
the state at whatever it was when paused.
set backend <name> resume Resume health checking. A fresh probe goroutine is
started and the backend re-enters unknown state.
set backend <name> disable Stop probing entirely and remove the backend from rotation.
The backend remains visible (state: disabled) and can be
re-enabled without reloading configuration.
set backend <name> enable Re-enable a disabled backend. A fresh probe goroutine is
started and the backend re-enters unknown state.
set frontend <name> pool <pool> backend <name> weight <0-100>
Set the weight of a backend within a pool. Weight 0 keeps
the backend in the pool but assigns it no traffic.
Takes effect immediately without reloading configuration.
watch events Stream all events (log, backend transitions, frontend)
[num <n>] Stop after receiving n events.
[log [level <level>]] Include log events. level is debug|info|warn|error
(default: info). Omitting log/backend/frontend enables all.
[backend] Include backend transition events.
[frontend] Include frontend events (reserved for future use).
Each event is printed as compact JSON on its own line.
Press any key or Ctrl-C to stop. Examples:
watch events
watch events num 20
watch events log level debug
watch events backend num 100
watch events log level debug backend
config check Ask maglevd to read and validate its current config file.
Prints "config ok" on success, or the error (parse or
semantic) returned by the daemon.
config reload Check and reload the configuration file. Equivalent to
sending SIGHUP to maglevd. Prints "config reloaded" on
success, or the specific error (parse, semantic, or
reload) that prevented the reload.
quit / exit Leave the interactive shell.
Interactive shell
The shell prompt is maglev> . Two completion mechanisms are available:
Tab completion — pressing <Tab> at any point completes the current token.
Fixed keywords (commands and subcommands) are completed from the command tree.
Backend, frontend, and health-check names are fetched live from the server with
a 1-second timeout. If the partial token is unambiguous the word is completed
in place; if multiple candidates exist they are listed and the prompt is
restored.
Inline help (?) — typing ? at any point prints the available
completions for the current position, with a short description next to each
keyword. The ? character is not added to the input line.
Commands and keywords support prefix matching: typing sh ba is equivalent
to show backends, and sh ba nginx0 is equivalent to show backends nginx0.