Files
vpp-maglev/docs/user-guide.md
Pim van Pelt d5fbf5c640 Prometheus: add VPP, LB sync, and gRPC metrics; expand docs
New metrics plus the corresponding documentation for everything that's
accumulated since the last Prometheus pass.

internal/metrics/metrics.go
- New VPPSource interface (IsConnected, VPPInfo) plus a metrics-local
  VPPInfo struct that mirrors vpp.Info. Decoupling via interface +
  struct-mirror keeps the dependency direction one-way (vpp → metrics),
  so vpp can import metrics to update inline counters without a cycle.
- New Collector gauges scraped on demand: maglev_vpp_connected,
  maglev_vpp_uptime_seconds (from /sys/boottime), maglev_vpp_connected_seconds
  (time since maglevd connected), and maglev_vpp_info (static 1-gauge
  carrying version, build_date, and pid as labels).
- New inline counters:
  - maglev_vpp_api_total{msg, direction, result} — bumped from the
    loggedChannel wrapper on every VPP binary-API send/recv. Gives full
    visibility into what maglevd is doing with VPP, broken down by
    message name, direction (send/recv), and result (success/failure).
  - maglev_vpp_lbsync_total{scope, kind} — bumped from the reconciler
    at the end of each SyncLBStateAll/SyncLBStateVIP run. kind ∈
    {vip_added, vip_removed, as_added, as_removed, as_weight_updated};
    scope ∈ {all, vip}. Zero-valued kinds are not emitted so noise
    stays low.
- Register() signature now takes a VPPSource (may be nil) alongside
  the existing StateSource.

internal/vpp/client.go
- New VPPInfo() (metrics.VPPInfo, bool) shim method on *Client that
  satisfies metrics.VPPSource. Returns (_, false) when disconnected so
  the collector skips the vpp_* gauges cleanly.

internal/vpp/apilog.go
- The loggedChannel's SendRequest / SendMultiRequest / ReceiveReply
  paths now call metrics.VPPAPITotal.WithLabelValues(...).Inc() in
  addition to slog.Debug. Since every VPP API call in the codebase
  must go through loggedChannel (NewAPIChannel is unexported), this
  one instrumentation point catches everything.

internal/vpp/lbsync.go
- New recordSyncStats(scope, st) helper called once at the end of
  SyncLBStateAll and SyncLBStateVIP to bump maglev_vpp_lbsync_total.
  Zero-valued stats are skipped.

cmd/maglevd/main.go
- Added github.com/grpc-ecosystem/go-grpc-middleware/providers/prometheus
  for the standard gRPC server metrics (grpc_server_started_total,
  grpc_server_handled_total, grpc_server_handling_seconds, etc.,
  labelled by service/method/type/code).
- Constructs grpcprom.NewServerMetrics(WithServerHandlingTimeHistogram())
  before creating the grpc.Server, installs it as UnaryInterceptor +
  StreamInterceptor, then calls InitializeMetrics(srv) after service
  registration so every method appears at 0 on the first scrape
  instead of materialising lazily on first RPC.
- Passes the vppClient (or nil) as a metrics.VPPSource to
  metrics.Register so the vpp_* gauges are emitted when integration
  is enabled and silently omitted otherwise.

docs/user-guide.md
- New 'Prometheus metrics' section in the maglevd chapter,
  tabulating every metric family: backend state gauges, probe
  counters/histogram, transition counters, the new VPP gauges and
  counters, and the standard gRPC server metrics.
- 'show frontends <name>' description updated to mention the two
  weight columns ('weight' = configured from YAML, 'effective' =
  state-aware after pool-failover logic).
- Pause / disable descriptions clarified: transition history is
  preserved across these operator actions.

docs/healthchecks.md
- New 'Static (no-healthcheck) backends' section explaining that
  backends without a healthcheck use rise/fall=1, fire a synthetic
  passing probe immediately on startup (no 30s wait), and idle at
  30s between iterations thereafter.
- New 'Pool failover' section documenting the priority-tier model,
  the active-pool definition, when promotion happens, cascading to
  further tiers, and graceful drain on demotion. Points readers at
  'maglevc show frontends <name>' as the inspection interface.

docs/config-guide.md
- healthcheck field doc now describes static-backend behavior and
  cross-references healthchecks.md.
- pools field doc now explains failover semantics at a high level
  and cross-references the detailed healthchecks.md section.
2026-04-12 13:00:35 +02:00

12 KiB

User Guide

maglevd

maglevd is the health-checker daemon. It probes backends according to the configuration file, maintains their health state, and exposes a gRPC API for inspection and control.

Flags

Flag Environment variable Default Description
--config MAGLEV_CONFIG /etc/vpp-maglev/maglev.yaml Path to the YAML configuration file.
--grpc-addr MAGLEV_GRPC_ADDR :9090 TCP address on which the gRPC server listens.
--metrics-addr MAGLEV_METRICS_ADDR :9091 TCP address for the Prometheus /metrics HTTP endpoint. Set to empty to disable.
--vpp-api-addr MAGLEV_VPP_API_ADDR /run/vpp/api.sock VPP binary API socket path. Set to empty to disable VPP integration.
--vpp-stats-addr MAGLEV_VPP_STATS_ADDR /run/vpp/stats.sock VPP stats socket path.
--log-level MAGLEV_LOG_LEVEL info Log verbosity: debug, info, warn, or error.
--check Read and validate the config file, then exit. Exits 0 if the config is valid, 1 on YAML parse error, 2 on semantic error.
--reflection true Enable gRPC server reflection. Allows grpcurl to introspect the API without the .proto file. Set to false to disable.
--version Print version, commit hash, and build date, then exit.

Flags take precedence over environment variables. Both are optional; defaults are used for anything not set.

Signals

Signal Effect
SIGHUP Reload the configuration file (same code path as config reload in maglevc). The file is checked before applying; if there is a parse or semantic error the reload is aborted and the error is logged (the daemon continues running with its current config). New backends are started, removed backends are stopped, backends whose health-check config is unchanged continue probing without interruption.
SIGTERM / SIGINT Graceful shutdown. Active gRPC streams are closed, the server drains, then the process exits.

Capabilities

maglevd requires CAP_NET_RAW when any health check uses type: icmp. All other check types (tcp, http) use normal TCP sockets and require no special capabilities.

Logging

All log output is written to stdout as JSON using Go's log/slog. The first line logged after the logger is configured is a starting record that includes version, commit, and date. Every state change emits a backend-transition line at INFO level. Set --log-level debug to see individual probe attempts, every VPP binary-API call (vpp-api-send / vpp-api-recv with full payload), and the per-VIP sync operations (vpp-lbsync-vip-add, vpp-lbsync-as-weight, etc.) as they happen.

Prometheus metrics

maglevd exposes Prometheus metrics on --metrics-addr (default :9091) at the /metrics path. Metric families:

Health-check and backend state (gauges, on-demand):

Metric Labels Description
maglev_backend_state backend, address, healthcheck, state 1 for the current state row per backend, 0 otherwise.
maglev_backend_health backend Current rise/fall counter value.
maglev_backend_enabled backend 1 if enabled, 0 if disabled.
maglev_frontend_pool_backend_weight frontend, pool, backend Configured weight from YAML.

Probe counters and latency (inline):

Metric Labels Description
maglev_probe_total backend, type, result, code Probes executed. result is success or failure.
maglev_probe_duration_seconds backend, type Histogram of probe wall time.
maglev_backend_transitions_total backend, from, to State machine transitions.

VPP integration (when enabled):

Metric Labels Description
maglev_vpp_connected 1 if maglevd currently has a live VPP connection.
maglev_vpp_uptime_seconds Seconds since VPP started (from /sys/boottime).
maglev_vpp_connected_seconds Seconds since maglevd established the current VPP connection.
maglev_vpp_info version, build_date, pid Static VPP build metadata; always 1.
maglev_vpp_api_total msg, direction, result VPP binary-API calls. direction is send or recv; result is success or failure.
maglev_vpp_lbsync_total scope, kind Per-mutation sync counters. scope is all or vip; kind is one of vip_added, vip_removed, as_added, as_removed, as_weight_updated.

gRPC server (standard go-grpc-middleware/prometheus metrics): grpc_server_started_total, grpc_server_handled_total, grpc_server_msg_received_total, grpc_server_msg_sent_total, and grpc_server_handling_seconds — all labelled by grpc_service, grpc_method, grpc_type, and grpc_code. Every method is pre-registered at zero so time series exist on the first scrape.


maglevc

maglevc is the interactive control-plane client. It connects to a running maglevd over gRPC and either executes a single command or drops into an interactive shell.

Usage

maglevc [--server host:port] [--color[=bool]] [command...]
Flag Default Description
--server localhost:9090 Address of the maglevd gRPC server.
--color true Colorize static field labels in output (dark blue ANSI). Pass --color=false to disable, e.g. when piping.

When command arguments are supplied the command is executed and maglevc exits. When no arguments are given an interactive shell is started and the build version is printed on entry.

Commands

show version                     Print build version, commit hash, and build date.

show frontends [<name>]          Without name: list all frontend names.
                                 With name: show address, protocol, port, description,
                                 and pools. Each pool lists its backends with two
                                 weight columns:
                                   weight     — configured weight from the YAML
                                   effective  — state-aware weight after pool failover
                                                (what gets programmed into VPP)
                                 Disabled backends are marked with [disabled].

show backends [<name>]           Without name: list all backend names.
                                 With name: show address, current state (with duration),
                                 enabled flag, health check, and recent state transitions
                                 with timestamps and how long ago each occurred.

show healthchecks [<name>]       Without name: list all health-check names.
                                 With name: show full health-check configuration.

show vpp info                    Show VPP version, build date, PID, uptime, and when
                                 maglevd connected. Returns an error if VPP is not
                                 connected.
show vpp lbstate                 Show the VPP load-balancer plugin state: global
                                 configuration, configured VIPs, and their attached
                                 application servers (address, weight, bucket count).
                                 Returns an error if VPP is not connected.

sync vpp lbstate [<name>]        Reconcile the VPP load-balancer dataplane from the
                                 running config. Without a name: runs a full sync —
                                 creates missing VIPs, removes stale VIPs, and adjusts
                                 application-server membership and weights across all
                                 frontends. With a name: only the named frontend's VIP
                                 is reconciled, and no VIPs are removed. A full sync
                                 also runs automatically every
                                 maglev.vpp.lb.sync-interval (default 30s) to catch
                                 drift, and once on startup.

set backend <name> pause         Stop health checking for a backend. Cancels the probe
                                 goroutine so no further traffic is sent, and sets the
                                 state to 'paused'. The backend's transition history is
                                 preserved, so 'show backend <name>' still shows where
                                 it came from.
set backend <name> resume        Resume health checking. A fresh probe goroutine is
                                 started and the backend re-enters unknown state.
set backend <name> disable       Stop probing entirely and remove the backend from
                                 rotation. The backend remains visible (state: disabled)
                                 with its transition history intact and can be re-enabled
                                 without reloading configuration.
set backend <name> enable        Re-enable a disabled backend. A fresh probe goroutine is
                                 started and the backend re-enters unknown state.

set frontend <name> pool <pool> backend <name> weight <0-100>
                                 Set the weight of a backend within a pool. Weight 0 keeps
                                 the backend in the pool but assigns it no traffic.
                                 Takes effect immediately without reloading configuration.

watch events                     Stream all events (log, backend transitions, frontend)
  [num <n>]                      Stop after receiving n events.
  [log [level <level>]]          Include log events. level is debug|info|warn|error
                                 (default: info). Omitting log/backend/frontend enables all.
  [backend]                      Include backend transition events.
  [frontend]                     Include frontend events (reserved for future use).

                                 Each event is printed as compact JSON on its own line.
                                 Press any key or Ctrl-C to stop. Examples:

                                   watch events
                                   watch events num 20
                                   watch events log level debug
                                   watch events backend num 100
                                   watch events log level debug backend

config check                     Ask maglevd to read and validate its current config file.
                                 Prints "config ok" on success, or the error (parse or
                                 semantic) returned by the daemon.
config reload                    Check and reload the configuration file. Equivalent to
                                 sending SIGHUP to maglevd. Prints "config reloaded" on
                                 success, or the specific error (parse, semantic, or
                                 reload) that prevented the reload.

quit / exit                      Leave the interactive shell.

Interactive shell

The shell prompt is maglev> . Two completion mechanisms are available:

Tab completion — pressing <Tab> at any point completes the current token. Fixed keywords (commands and subcommands) are completed from the command tree. Backend, frontend, and health-check names are fetched live from the server with a 1-second timeout. If the partial token is unambiguous the word is completed in place; if multiple candidates exist they are listed and the prompt is restored.

Inline help (?) — typing ? at any point prints the available completions for the current position, with a short description next to each keyword. The ? character is not added to the input line.

Commands and keywords support prefix matching: typing sh ba is equivalent to show backends, and sh ba nginx0 is equivalent to show backends nginx0.