Pim van Pelt
fb62532fd5
New feature: per-VIP / per-backend runtime counters
* New GetVPPLBCounters RPC serving an in-process snapshot refreshed
by a 5s scrape loop (internal/vpp/lbstats.go). Each cycle pulls
the LB plugin's four SimpleCounters (next, first, untracked,
no-server) plus the FIB /net/route/to CombinedCounter for every
VIP and every backend host prefix via a single DumpStats call.
* FIB stats-index discovery via ip_route_lookup (internal/vpp/
fibstats.go); per-worker reduction happens in the collector.
* Prometheus collector exports vip_packets_total (kind label),
vip_route_{packets,bytes}_total, and backend_route_{packets,
bytes}_total. Metrics source interface extended with VIPStats /
BackendRouteStats; vpp.Client publishes snapshots via
atomic.Pointer and clears them on disconnect.
* New 'show vpp lb counters' CLI command. The 'show vpp lbstate'
and 'sync vpp lbstate' commands are restructured under 'show
vpp lb {state,counters}' / 'sync vpp lb state' to make room
for the new verb.
New feature: src-ip-sticky frontends
* New frontend YAML key 'src-ip-sticky' (bool). Plumbed through
config.Frontend, desiredVIP, and the lb_add_del_vip_v2 call.
* Reflected in gRPC FrontendInfo.src_ip_sticky and VPPLBVIP.
src_ip_sticky, and shown in 'show vpp lb state' output.
* Scraped back from VPP by parsing 'show lb vips verbose' through
cli_inband — lb_vip_details does not expose the flag. The same
scrape also recovers the LB pool index for each VIP, which the
stats-segment counters are keyed on. This is a documented
temporary workaround until VPP ships an lb_vip_v2_dump.
* src_ip_sticky cannot be mutated on a live VIP, so a flipped flag
triggers a tear-down-and-recreate in reconcileVIP (ASes deleted
with flush, VIP deleted, then re-added). Flip is logged.
New feature: frontend state aggregation and events
* New health.FrontendState (unknown/up/down) and FrontendTransition
types. A frontend is 'up' iff at least one backend has a nonzero
effective weight, 'unknown' iff no backend has real state yet,
and 'down' otherwise.
* Checker tracks per-frontend aggregate state, recomputing after
each backend transition and emitting a frontend-transition Event
on change. Reload drops entries for removed frontends.
* checker.Event gains an optional FrontendTransition pointer;
backend- vs. frontend-transition events are demultiplexed on
that field.
* WatchEvents now sends an initial snapshot of frontend state on
connect (mirroring the existing backend snapshot), subscribes
once to the checker stream, and fans out to backend/frontend
handlers based on the client's filter flags. The proto
FrontendEvent message grows name + transition fields.
* New Checker.FrontendState accessor.
Refactor: pure health helpers
* Moved the priority-failover selector and the (pool idx, active
pool, state, cfg weight) → (vpp weight, flush) mapping out of
internal/vpp/lbsync.go into a new internal/health/weights.go so
the checker can reuse them for frontend-state computation
without importing internal/vpp.
* New functions: health.ActivePoolIndex, BackendEffectiveWeight,
EffectiveWeights, ComputeFrontendState. lbsync.go now calls
these directly; vpp.EffectiveWeights is a thin wrapper over
health.EffectiveWeights retained for the gRPC observability
path. Fully unit-tested in internal/health/weights_test.go.
maglevc polish
* --color default is now mode-aware: on in the interactive shell,
off in one-shot mode so piped output is script-safe. Explicit
--color=true/false still overrides.
* New stripHostMask helper drops /32 and /128 from VIP display;
non-host prefixes pass through unchanged.
* Counter table column order fixed (first before next) and
packets/bytes columns renamed to fib-packets/fib-bytes to
clarify they come from the FIB, not the LB plugin.
Docs
* config-guide: document src-ip-sticky, including the VIP
recreate-on-change caveat.
* user-guide, maglevc.1, maglevd.8: updated command tree, new
counters command, color defaults, and the src-ip-sticky field.
221 lines
13 KiB
Markdown
221 lines
13 KiB
Markdown
# User Guide
|
|
|
|
## maglevd
|
|
|
|
`maglevd` is the health-checker daemon. It probes backends according to the
|
|
configuration file, maintains their health state, and exposes a gRPC API for
|
|
inspection and control.
|
|
|
|
### Flags
|
|
|
|
| Flag | Environment variable | Default | Description |
|
|
|---|---|---|---|
|
|
| `--config` | `MAGLEV_CONFIG` | `/etc/vpp-maglev/maglev.yaml` | Path to the YAML configuration file. |
|
|
| `--grpc-addr` | `MAGLEV_GRPC_ADDR` | `:9090` | TCP address on which the gRPC server listens. |
|
|
| `--metrics-addr` | `MAGLEV_METRICS_ADDR` | `:9091` | TCP address for the Prometheus `/metrics` HTTP endpoint. Set to empty to disable. |
|
|
| `--vpp-api-addr` | `MAGLEV_VPP_API_ADDR` | `/run/vpp/api.sock` | VPP binary API socket path. Set to empty to disable VPP integration. |
|
|
| `--vpp-stats-addr` | `MAGLEV_VPP_STATS_ADDR` | `/run/vpp/stats.sock` | VPP stats socket path. |
|
|
| `--log-level` | `MAGLEV_LOG_LEVEL` | `info` | Log verbosity: `debug`, `info`, `warn`, or `error`. |
|
|
| `--check` | — | — | Read and validate the config file, then exit. Exits 0 if the config is valid, 1 on YAML parse error, 2 on semantic error. |
|
|
| `--reflection` | — | `true` | Enable gRPC server reflection. Allows `grpcurl` to introspect the API without the `.proto` file. Set to `false` to disable. |
|
|
| `--version` | — | — | Print version, commit hash, and build date, then exit. |
|
|
|
|
Flags take precedence over environment variables. Both are optional; defaults
|
|
are used for anything not set.
|
|
|
|
### Signals
|
|
|
|
| Signal | Effect |
|
|
|---|---|
|
|
| `SIGHUP` | Reload the configuration file (same code path as `config reload` in `maglevc`). The file is checked before applying; if there is a parse or semantic error the reload is aborted and the error is logged (the daemon continues running with its current config). New backends are started, removed backends are stopped, backends whose health-check config is unchanged continue probing without interruption. |
|
|
| `SIGTERM` / `SIGINT` | Graceful shutdown. Active gRPC streams are closed, the server drains, then the process exits. |
|
|
|
|
### Capabilities
|
|
|
|
`maglevd` requires `CAP_NET_RAW` when any health check uses `type: icmp`.
|
|
All other check types (`tcp`, `http`) use normal TCP sockets and require no
|
|
special capabilities.
|
|
|
|
### Logging
|
|
|
|
All log output is written to stdout as JSON using Go's `log/slog`. The first
|
|
line logged after the logger is configured is a `starting` record that includes
|
|
`version`, `commit`, and `date`. Every state change emits a `backend-transition`
|
|
line at `INFO` level. Set `--log-level debug` to see individual probe attempts,
|
|
every VPP binary-API call (`vpp-api-send` / `vpp-api-recv` with full payload),
|
|
and the per-VIP sync operations (`vpp-lbsync-vip-add`, `vpp-lbsync-as-weight`,
|
|
etc.) as they happen.
|
|
|
|
### Prometheus metrics
|
|
|
|
`maglevd` exposes Prometheus metrics on `--metrics-addr` (default `:9091`) at
|
|
the `/metrics` path. Metric families:
|
|
|
|
**Health-check and backend state (gauges, on-demand):**
|
|
| Metric | Labels | Description |
|
|
|---|---|---|
|
|
| `maglev_backend_state` | `backend`, `address`, `healthcheck`, `state` | 1 for the current state row per backend, 0 otherwise. |
|
|
| `maglev_backend_health` | `backend` | Current rise/fall counter value. |
|
|
| `maglev_backend_enabled` | `backend` | 1 if enabled, 0 if disabled. |
|
|
| `maglev_frontend_pool_backend_weight` | `frontend`, `pool`, `backend` | Configured weight from YAML. |
|
|
|
|
**Probe counters and latency (inline):**
|
|
| Metric | Labels | Description |
|
|
|---|---|---|
|
|
| `maglev_probe_total` | `backend`, `type`, `result`, `code` | Probes executed. `result` is `success` or `failure`. |
|
|
| `maglev_probe_duration_seconds` | `backend`, `type` | Histogram of probe wall time. |
|
|
| `maglev_backend_transitions_total` | `backend`, `from`, `to` | State machine transitions. |
|
|
|
|
**VPP integration (when enabled):**
|
|
| Metric | Labels | Description |
|
|
|---|---|---|
|
|
| `maglev_vpp_connected` | — | 1 if maglevd currently has a live VPP connection. |
|
|
| `maglev_vpp_uptime_seconds` | — | Seconds since VPP started (from `/sys/boottime`). |
|
|
| `maglev_vpp_connected_seconds` | — | Seconds since maglevd established the current VPP connection. |
|
|
| `maglev_vpp_info` | `version`, `build_date`, `pid` | Static VPP build metadata; always 1. |
|
|
| `maglev_vpp_api_total` | `msg`, `direction`, `result` | VPP binary-API calls. `direction` is `send` or `recv`; `result` is `success` or `failure`. |
|
|
| `maglev_vpp_lbsync_total` | `scope`, `kind` | Per-mutation sync counters. `scope` is `all` or `vip`; `kind` is one of `vip_added`, `vip_removed`, `as_added`, `as_removed`, `as_weight_updated`. |
|
|
|
|
**gRPC server (standard `go-grpc-middleware/prometheus` metrics):**
|
|
`grpc_server_started_total`, `grpc_server_handled_total`,
|
|
`grpc_server_msg_received_total`, `grpc_server_msg_sent_total`, and
|
|
`grpc_server_handling_seconds` — all labelled by `grpc_service`,
|
|
`grpc_method`, `grpc_type`, and `grpc_code`. Every method is
|
|
pre-registered at zero so time series exist on the first scrape.
|
|
|
|
---
|
|
|
|
## maglevc
|
|
|
|
`maglevc` is the interactive control-plane client. It connects to a running
|
|
`maglevd` over gRPC and either executes a single command or drops into an
|
|
interactive shell.
|
|
|
|
### Usage
|
|
|
|
```sh
|
|
maglevc [--server host:port] [--color[=bool]] [command...]
|
|
```
|
|
|
|
| Flag | Default | Description |
|
|
|---|---|---|
|
|
| `--server` | `localhost:9090` | Address of the `maglevd` gRPC server. |
|
|
| `--color` | mode-aware | Colorize static field labels (dark blue ANSI). Defaults to `true` in the interactive shell and `false` in one-shot mode, so output piped into scripts stays free of escape codes. Pass `--color=true` or `--color=false` explicitly to override either default. |
|
|
|
|
When `command` arguments are supplied the command is executed and `maglevc`
|
|
exits; in this mode ANSI color is off by default so the output is script-safe.
|
|
When no arguments are given an interactive shell is started, the build version
|
|
is printed on entry, and color is on by default.
|
|
|
|
### Commands
|
|
|
|
```
|
|
show version Print build version, commit hash, and build date.
|
|
|
|
show frontends [<name>] Without name: list all frontend names.
|
|
With name: show address, protocol, port, src-ip-sticky,
|
|
description, and pools. Each pool lists its backends
|
|
with two weight columns:
|
|
weight — configured weight from the YAML
|
|
effective — state-aware weight after pool failover
|
|
(what gets programmed into VPP)
|
|
Disabled backends are marked with [disabled].
|
|
|
|
show backends [<name>] Without name: list all backend names.
|
|
With name: show address, current state (with duration),
|
|
enabled flag, health check, and recent state transitions
|
|
with timestamps and how long ago each occurred.
|
|
|
|
show healthchecks [<name>] Without name: list all health-check names.
|
|
With name: show full health-check configuration.
|
|
|
|
show vpp info Show VPP version, build date, PID, uptime, and when
|
|
maglevd connected. Returns an error if VPP is not
|
|
connected.
|
|
show vpp lb state Show the VPP load-balancer plugin state: global
|
|
configuration, configured VIPs, and their attached
|
|
application servers (address, weight, bucket count).
|
|
Returns an error if VPP is not connected.
|
|
show vpp lb counters Show per-VIP and per-backend packet/byte counters
|
|
from the VPP stats segment, refreshed roughly every
|
|
five seconds by maglevd. Each VIP row reports the LB
|
|
plugin counters (next, first, untracked, no-server)
|
|
and the FIB packets/bytes at the VIP's host prefix.
|
|
Each backend row reports FIB packets/bytes at the
|
|
backend's /32 or /128 prefix. Use Prometheus for
|
|
live rates; this command shows absolute values.
|
|
|
|
sync vpp lb state [<name>] Reconcile the VPP load-balancer dataplane from the
|
|
running config. Without a name: runs a full sync —
|
|
creates missing VIPs, removes stale VIPs, and adjusts
|
|
application-server membership and weights across all
|
|
frontends. With a name: only the named frontend's VIP
|
|
is reconciled, and no VIPs are removed. A full sync
|
|
also runs automatically every
|
|
maglev.vpp.lb.sync-interval (default 30s) to catch
|
|
drift, and once on startup.
|
|
|
|
set backend <name> pause Stop health checking for a backend. Cancels the probe
|
|
goroutine so no further traffic is sent, and sets the
|
|
state to 'paused'. The backend's transition history is
|
|
preserved, so 'show backend <name>' still shows where
|
|
it came from.
|
|
set backend <name> resume Resume health checking. A fresh probe goroutine is
|
|
started and the backend re-enters unknown state.
|
|
set backend <name> disable Stop probing entirely and remove the backend from
|
|
rotation. The backend remains visible (state: disabled)
|
|
with its transition history intact and can be re-enabled
|
|
without reloading configuration.
|
|
set backend <name> enable Re-enable a disabled backend. A fresh probe goroutine is
|
|
started and the backend re-enters unknown state.
|
|
|
|
set frontend <name> pool <pool> backend <name> weight <0-100>
|
|
Set the weight of a backend within a pool. Weight 0 keeps
|
|
the backend in the pool but assigns it no traffic.
|
|
Takes effect immediately without reloading configuration.
|
|
|
|
watch events Stream all events (log, backend transitions, frontend)
|
|
[num <n>] Stop after receiving n events.
|
|
[log [level <level>]] Include log events. level is debug|info|warn|error
|
|
(default: info). Omitting log/backend/frontend enables all.
|
|
[backend] Include backend transition events.
|
|
[frontend] Include frontend events (reserved for future use).
|
|
|
|
Each event is printed as compact JSON on its own line.
|
|
Press any key or Ctrl-C to stop. Examples:
|
|
|
|
watch events
|
|
watch events num 20
|
|
watch events log level debug
|
|
watch events backend num 100
|
|
watch events log level debug backend
|
|
|
|
config check Ask maglevd to read and validate its current config file.
|
|
Prints "config ok" on success, or the error (parse or
|
|
semantic) returned by the daemon.
|
|
config reload Check and reload the configuration file. Equivalent to
|
|
sending SIGHUP to maglevd. Prints "config reloaded" on
|
|
success, or the specific error (parse, semantic, or
|
|
reload) that prevented the reload.
|
|
|
|
quit / exit Leave the interactive shell.
|
|
```
|
|
|
|
### Interactive shell
|
|
|
|
The shell prompt is `maglev> `. Two completion mechanisms are available:
|
|
|
|
**Tab completion** — pressing `<Tab>` at any point completes the current token.
|
|
Fixed keywords (commands and subcommands) are completed from the command tree.
|
|
Backend, frontend, and health-check names are fetched live from the server with
|
|
a 1-second timeout. If the partial token is unambiguous the word is completed
|
|
in place; if multiple candidates exist they are listed and the prompt is
|
|
restored.
|
|
|
|
**Inline help (`?`)** — typing `?` at any point prints the available
|
|
completions for the current position, with a short description next to each
|
|
keyword. The `?` character is not added to the input line.
|
|
|
|
Commands and keywords support **prefix matching**: typing `sh ba` is equivalent
|
|
to `show backends`, and `sh ba nginx0` is equivalent to `show backends nginx0`.
|