New metrics plus the corresponding documentation for everything that's
accumulated since the last Prometheus pass.
internal/metrics/metrics.go
- New VPPSource interface (IsConnected, VPPInfo) plus a metrics-local
VPPInfo struct that mirrors vpp.Info. Decoupling via interface +
struct-mirror keeps the dependency direction one-way (vpp → metrics),
so vpp can import metrics to update inline counters without a cycle.
- New Collector gauges scraped on demand: maglev_vpp_connected,
maglev_vpp_uptime_seconds (from /sys/boottime), maglev_vpp_connected_seconds
(time since maglevd connected), and maglev_vpp_info (static 1-gauge
carrying version, build_date, and pid as labels).
- New inline counters:
- maglev_vpp_api_total{msg, direction, result} — bumped from the
loggedChannel wrapper on every VPP binary-API send/recv. Gives full
visibility into what maglevd is doing with VPP, broken down by
message name, direction (send/recv), and result (success/failure).
- maglev_vpp_lbsync_total{scope, kind} — bumped from the reconciler
at the end of each SyncLBStateAll/SyncLBStateVIP run. kind ∈
{vip_added, vip_removed, as_added, as_removed, as_weight_updated};
scope ∈ {all, vip}. Zero-valued kinds are not emitted so noise
stays low.
- Register() signature now takes a VPPSource (may be nil) alongside
the existing StateSource.
internal/vpp/client.go
- New VPPInfo() (metrics.VPPInfo, bool) shim method on *Client that
satisfies metrics.VPPSource. Returns (_, false) when disconnected so
the collector skips the vpp_* gauges cleanly.
internal/vpp/apilog.go
- The loggedChannel's SendRequest / SendMultiRequest / ReceiveReply
paths now call metrics.VPPAPITotal.WithLabelValues(...).Inc() in
addition to slog.Debug. Since every VPP API call in the codebase
must go through loggedChannel (NewAPIChannel is unexported), this
one instrumentation point catches everything.
internal/vpp/lbsync.go
- New recordSyncStats(scope, st) helper called once at the end of
SyncLBStateAll and SyncLBStateVIP to bump maglev_vpp_lbsync_total.
Zero-valued stats are skipped.
cmd/maglevd/main.go
- Added github.com/grpc-ecosystem/go-grpc-middleware/providers/prometheus
for the standard gRPC server metrics (grpc_server_started_total,
grpc_server_handled_total, grpc_server_handling_seconds, etc.,
labelled by service/method/type/code).
- Constructs grpcprom.NewServerMetrics(WithServerHandlingTimeHistogram())
before creating the grpc.Server, installs it as UnaryInterceptor +
StreamInterceptor, then calls InitializeMetrics(srv) after service
registration so every method appears at 0 on the first scrape
instead of materialising lazily on first RPC.
- Passes the vppClient (or nil) as a metrics.VPPSource to
metrics.Register so the vpp_* gauges are emitted when integration
is enabled and silently omitted otherwise.
docs/user-guide.md
- New 'Prometheus metrics' section in the maglevd chapter,
tabulating every metric family: backend state gauges, probe
counters/histogram, transition counters, the new VPP gauges and
counters, and the standard gRPC server metrics.
- 'show frontends <name>' description updated to mention the two
weight columns ('weight' = configured from YAML, 'effective' =
state-aware after pool-failover logic).
- Pause / disable descriptions clarified: transition history is
preserved across these operator actions.
docs/healthchecks.md
- New 'Static (no-healthcheck) backends' section explaining that
backends without a healthcheck use rise/fall=1, fire a synthetic
passing probe immediately on startup (no 30s wait), and idle at
30s between iterations thereafter.
- New 'Pool failover' section documenting the priority-tier model,
the active-pool definition, when promotion happens, cascading to
further tiers, and graceful drain on demotion. Points readers at
'maglevc show frontends <name>' as the inspection interface.
docs/config-guide.md
- healthcheck field doc now describes static-backend behavior and
cross-references healthchecks.md.
- pools field doc now explains failover semantics at a high level
and cross-references the detailed healthchecks.md section.
12 KiB
User Guide
maglevd
maglevd is the health-checker daemon. It probes backends according to the
configuration file, maintains their health state, and exposes a gRPC API for
inspection and control.
Flags
| Flag | Environment variable | Default | Description |
|---|---|---|---|
--config |
MAGLEV_CONFIG |
/etc/vpp-maglev/maglev.yaml |
Path to the YAML configuration file. |
--grpc-addr |
MAGLEV_GRPC_ADDR |
:9090 |
TCP address on which the gRPC server listens. |
--metrics-addr |
MAGLEV_METRICS_ADDR |
:9091 |
TCP address for the Prometheus /metrics HTTP endpoint. Set to empty to disable. |
--vpp-api-addr |
MAGLEV_VPP_API_ADDR |
/run/vpp/api.sock |
VPP binary API socket path. Set to empty to disable VPP integration. |
--vpp-stats-addr |
MAGLEV_VPP_STATS_ADDR |
/run/vpp/stats.sock |
VPP stats socket path. |
--log-level |
MAGLEV_LOG_LEVEL |
info |
Log verbosity: debug, info, warn, or error. |
--check |
— | — | Read and validate the config file, then exit. Exits 0 if the config is valid, 1 on YAML parse error, 2 on semantic error. |
--reflection |
— | true |
Enable gRPC server reflection. Allows grpcurl to introspect the API without the .proto file. Set to false to disable. |
--version |
— | — | Print version, commit hash, and build date, then exit. |
Flags take precedence over environment variables. Both are optional; defaults are used for anything not set.
Signals
| Signal | Effect |
|---|---|
SIGHUP |
Reload the configuration file (same code path as config reload in maglevc). The file is checked before applying; if there is a parse or semantic error the reload is aborted and the error is logged (the daemon continues running with its current config). New backends are started, removed backends are stopped, backends whose health-check config is unchanged continue probing without interruption. |
SIGTERM / SIGINT |
Graceful shutdown. Active gRPC streams are closed, the server drains, then the process exits. |
Capabilities
maglevd requires CAP_NET_RAW when any health check uses type: icmp.
All other check types (tcp, http) use normal TCP sockets and require no
special capabilities.
Logging
All log output is written to stdout as JSON using Go's log/slog. The first
line logged after the logger is configured is a starting record that includes
version, commit, and date. Every state change emits a backend-transition
line at INFO level. Set --log-level debug to see individual probe attempts,
every VPP binary-API call (vpp-api-send / vpp-api-recv with full payload),
and the per-VIP sync operations (vpp-lbsync-vip-add, vpp-lbsync-as-weight,
etc.) as they happen.
Prometheus metrics
maglevd exposes Prometheus metrics on --metrics-addr (default :9091) at
the /metrics path. Metric families:
Health-check and backend state (gauges, on-demand):
| Metric | Labels | Description |
|---|---|---|
maglev_backend_state |
backend, address, healthcheck, state |
1 for the current state row per backend, 0 otherwise. |
maglev_backend_health |
backend |
Current rise/fall counter value. |
maglev_backend_enabled |
backend |
1 if enabled, 0 if disabled. |
maglev_frontend_pool_backend_weight |
frontend, pool, backend |
Configured weight from YAML. |
Probe counters and latency (inline):
| Metric | Labels | Description |
|---|---|---|
maglev_probe_total |
backend, type, result, code |
Probes executed. result is success or failure. |
maglev_probe_duration_seconds |
backend, type |
Histogram of probe wall time. |
maglev_backend_transitions_total |
backend, from, to |
State machine transitions. |
VPP integration (when enabled):
| Metric | Labels | Description |
|---|---|---|
maglev_vpp_connected |
— | 1 if maglevd currently has a live VPP connection. |
maglev_vpp_uptime_seconds |
— | Seconds since VPP started (from /sys/boottime). |
maglev_vpp_connected_seconds |
— | Seconds since maglevd established the current VPP connection. |
maglev_vpp_info |
version, build_date, pid |
Static VPP build metadata; always 1. |
maglev_vpp_api_total |
msg, direction, result |
VPP binary-API calls. direction is send or recv; result is success or failure. |
maglev_vpp_lbsync_total |
scope, kind |
Per-mutation sync counters. scope is all or vip; kind is one of vip_added, vip_removed, as_added, as_removed, as_weight_updated. |
gRPC server (standard go-grpc-middleware/prometheus metrics):
grpc_server_started_total, grpc_server_handled_total,
grpc_server_msg_received_total, grpc_server_msg_sent_total, and
grpc_server_handling_seconds — all labelled by grpc_service,
grpc_method, grpc_type, and grpc_code. Every method is
pre-registered at zero so time series exist on the first scrape.
maglevc
maglevc is the interactive control-plane client. It connects to a running
maglevd over gRPC and either executes a single command or drops into an
interactive shell.
Usage
maglevc [--server host:port] [--color[=bool]] [command...]
| Flag | Default | Description |
|---|---|---|
--server |
localhost:9090 |
Address of the maglevd gRPC server. |
--color |
true |
Colorize static field labels in output (dark blue ANSI). Pass --color=false to disable, e.g. when piping. |
When command arguments are supplied the command is executed and maglevc
exits. When no arguments are given an interactive shell is started and the
build version is printed on entry.
Commands
show version Print build version, commit hash, and build date.
show frontends [<name>] Without name: list all frontend names.
With name: show address, protocol, port, description,
and pools. Each pool lists its backends with two
weight columns:
weight — configured weight from the YAML
effective — state-aware weight after pool failover
(what gets programmed into VPP)
Disabled backends are marked with [disabled].
show backends [<name>] Without name: list all backend names.
With name: show address, current state (with duration),
enabled flag, health check, and recent state transitions
with timestamps and how long ago each occurred.
show healthchecks [<name>] Without name: list all health-check names.
With name: show full health-check configuration.
show vpp info Show VPP version, build date, PID, uptime, and when
maglevd connected. Returns an error if VPP is not
connected.
show vpp lbstate Show the VPP load-balancer plugin state: global
configuration, configured VIPs, and their attached
application servers (address, weight, bucket count).
Returns an error if VPP is not connected.
sync vpp lbstate [<name>] Reconcile the VPP load-balancer dataplane from the
running config. Without a name: runs a full sync —
creates missing VIPs, removes stale VIPs, and adjusts
application-server membership and weights across all
frontends. With a name: only the named frontend's VIP
is reconciled, and no VIPs are removed. A full sync
also runs automatically every
maglev.vpp.lb.sync-interval (default 30s) to catch
drift, and once on startup.
set backend <name> pause Stop health checking for a backend. Cancels the probe
goroutine so no further traffic is sent, and sets the
state to 'paused'. The backend's transition history is
preserved, so 'show backend <name>' still shows where
it came from.
set backend <name> resume Resume health checking. A fresh probe goroutine is
started and the backend re-enters unknown state.
set backend <name> disable Stop probing entirely and remove the backend from
rotation. The backend remains visible (state: disabled)
with its transition history intact and can be re-enabled
without reloading configuration.
set backend <name> enable Re-enable a disabled backend. A fresh probe goroutine is
started and the backend re-enters unknown state.
set frontend <name> pool <pool> backend <name> weight <0-100>
Set the weight of a backend within a pool. Weight 0 keeps
the backend in the pool but assigns it no traffic.
Takes effect immediately without reloading configuration.
watch events Stream all events (log, backend transitions, frontend)
[num <n>] Stop after receiving n events.
[log [level <level>]] Include log events. level is debug|info|warn|error
(default: info). Omitting log/backend/frontend enables all.
[backend] Include backend transition events.
[frontend] Include frontend events (reserved for future use).
Each event is printed as compact JSON on its own line.
Press any key or Ctrl-C to stop. Examples:
watch events
watch events num 20
watch events log level debug
watch events backend num 100
watch events log level debug backend
config check Ask maglevd to read and validate its current config file.
Prints "config ok" on success, or the error (parse or
semantic) returned by the daemon.
config reload Check and reload the configuration file. Equivalent to
sending SIGHUP to maglevd. Prints "config reloaded" on
success, or the specific error (parse, semantic, or
reload) that prevented the reload.
quit / exit Leave the interactive shell.
Interactive shell
The shell prompt is maglev> . Two completion mechanisms are available:
Tab completion — pressing <Tab> at any point completes the current token.
Fixed keywords (commands and subcommands) are completed from the command tree.
Backend, frontend, and health-check names are fetched live from the server with
a 1-second timeout. If the partial token is unambiguous the word is completed
in place; if multiple candidates exist they are listed and the prompt is
restored.
Inline help (?) — typing ? at any point prints the available
completions for the current position, with a short description next to each
keyword. The ? character is not added to the input line.
Commands and keywords support prefix matching: typing sh ba is equivalent
to show backends, and sh ba nginx0 is equivalent to show backends nginx0.