vpp-maglev/docs/user-guide.md

# User Guide

## maglevd

`maglevd` is the health-checker daemon. It probes backends according to the
configuration file, maintains their health state, and exposes a gRPC API for
inspection and control.

### Flags

| Flag | Environment variable | Default | Description |
|---|---|---|---|
| `--config` | `MAGLEV_CONFIG` | `/etc/vpp-maglev/maglev.yaml` | Path to the YAML configuration file. |
| `--grpc-addr` | `MAGLEV_GRPC_ADDR` | `:9090` | TCP address on which the gRPC server listens. |
| `--metrics-addr` | `MAGLEV_METRICS_ADDR` | `:9091` | TCP address for the Prometheus `/metrics` HTTP endpoint. Set to empty to disable. |
| `--vpp-api-addr` | `MAGLEV_VPP_API_ADDR` | `/run/vpp/api.sock` | VPP binary API socket path. Set to empty to disable VPP integration. |
| `--vpp-stats-addr` | `MAGLEV_VPP_STATS_ADDR` | `/run/vpp/stats.sock` | VPP stats socket path. |
| `--log-level` | `MAGLEV_LOG_LEVEL` | `info` | Log verbosity: `debug`, `info`, `warn`, or `error`. |
| `--check` | — | — | Read and validate the config file, then exit. Exits 0 if the config is valid, 1 on YAML parse error, 2 on semantic error. |
| `--reflection` | — | `true` | Enable gRPC server reflection. Allows `grpcurl` to introspect the API without the `.proto` file. Set to `false` to disable. |
| `--version` | — | — | Print version, commit hash, and build date, then exit. |

Flags take precedence over environment variables. Both are optional; defaults
are used for anything not set.

### Signals

| Signal | Effect |
|---|---|
| `SIGHUP` | Reload the configuration file (same code path as `config reload` in `maglevc`). The file is checked before applying; if there is a parse or semantic error the reload is aborted and the error is logged (the daemon continues running with its current config). New backends are started, removed backends are stopped, backends whose health-check config is unchanged continue probing without interruption. |
| `SIGTERM` / `SIGINT` | Graceful shutdown. Active gRPC streams are closed, the server drains, then the process exits. |

### Capabilities

`maglevd` requires `CAP_NET_RAW` when any health check uses `type: icmp`.
All other check types (`tcp`, `http`) use normal TCP sockets and require no
special capabilities.

### Logging

All log output is written to stdout as JSON using Go's `log/slog`. The first
line logged after the logger is configured is a `starting` record that includes
`version`, `commit`, and `date`. Every state change emits a `backend-transition`
line at `INFO` level. Per-mutation VPP LB sync events
(`vpp-lb-sync-vip-added`, `vpp-lb-sync-vip-removed`, `vpp-lb-sync-as-added`,
`vpp-lb-sync-as-removed`, `vpp-lb-sync-as-weight-updated`) are also emitted
at `INFO` so the CLI `watch events` stream and the web frontend see every
dataplane change without raising the log level. Set `--log-level debug` to
see individual probe attempts and every VPP binary-API call
(`vpp-api-send` / `vpp-api-recv` with full payload) as they happen.

### Prometheus metrics

`maglevd` exposes Prometheus metrics on `--metrics-addr` (default `:9091`) at
the `/metrics` path. Metric families:

**Health-check and backend state (gauges, on-demand):**
| Metric | Labels | Description |
|---|---|---|
| `maglev_backend_state` | `backend`, `address`, `healthcheck`, `state` | 1 for the current state row per backend, 0 otherwise. |
| `maglev_backend_health` | `backend` | Current rise/fall counter value. |
| `maglev_backend_enabled` | `backend` | 1 if enabled, 0 if disabled. |
| `maglev_frontend_pool_backend_weight` | `frontend`, `pool`, `backend` | Configured weight from YAML. |

**Probe counters and latency (inline):**
| Metric | Labels | Description |
|---|---|---|
| `maglev_probe_total` | `backend`, `type`, `result`, `code` | Probes executed. `result` is `success` or `failure`. |
| `maglev_probe_duration_seconds` | `backend`, `type` | Histogram of probe wall time. |
| `maglev_backend_transitions_total` | `backend`, `from`, `to` | State machine transitions. |

**VPP integration (when enabled):**
| Metric | Labels | Description |
|---|---|---|
| `maglev_vpp_connected` | — | 1 if maglevd currently has a live VPP connection. |
| `maglev_vpp_uptime_seconds` | — | Seconds since VPP started (from `/sys/boottime`). |
| `maglev_vpp_connected_seconds` | — | Seconds since maglevd established the current VPP connection. |
| `maglev_vpp_info` | `version`, `build_date`, `pid` | Static VPP build metadata; always 1. |
| `maglev_vpp_api_total` | `msg`, `direction`, `result` | VPP binary-API calls. `direction` is `send` or `recv`; `result` is `success` or `failure`. |
| `maglev_vpp_lbsync_total` | `scope`, `kind` | Per-mutation sync counters. `scope` is `all` or `vip`; `kind` is one of `vip_added`, `vip_removed`, `as_added`, `as_removed`, `as_weight_updated`. |

**gRPC server (standard `go-grpc-middleware/prometheus` metrics):**
`grpc_server_started_total`, `grpc_server_handled_total`,
`grpc_server_msg_received_total`, `grpc_server_msg_sent_total`, and
`grpc_server_handling_seconds` — all labelled by `grpc_service`,
`grpc_method`, `grpc_type`, and `grpc_code`. Every method is
pre-registered at zero so time series exist on the first scrape.

---

## maglevc

`maglevc` is the interactive control-plane client. It connects to a running
`maglevd` over gRPC and either executes a single command or drops into an
interactive shell.

### Usage

```sh
maglevc [--server host:port] [--color[=bool]] [command...]
```

| Flag | Default | Description |
|---|---|---|
| `--server` | `localhost:9090` | Address of the `maglevd` gRPC server. |
| `--color` | mode-aware | Colorize static field labels (dark blue ANSI). Defaults to `true` in the interactive shell and `false` in one-shot mode, so output piped into scripts stays free of escape codes. Pass `--color=true` or `--color=false` explicitly to override either default. |

When `command` arguments are supplied the command is executed and `maglevc`
exits; in this mode ANSI color is off by default so the output is script-safe.
When no arguments are given an interactive shell is started, the build version
is printed on entry, and color is on by default.

### Commands

```
show version                     Print build version, commit hash, and build date.

show frontends [<name>]          Without name: list all frontend names.
                                 With name: show address, protocol, port, src-ip-sticky,
                                 description, and pools. Each pool lists its backends
                                 with two weight columns:
                                   weight     — configured weight from the YAML
                                   effective  — state-aware weight after pool failover
                                                (what gets programmed into VPP)
                                 Disabled backends are marked with [disabled].

show backends [<name>]           Without name: list all backend names.
                                 With name: show address, current state (with duration),
                                 enabled flag, health check, and recent state transitions
                                 with timestamps and how long ago each occurred.

show healthchecks [<name>]       Without name: list all health-check names.
                                 With name: show full health-check configuration.

show vpp info                    Show VPP version, build date, PID, uptime, and when
                                 maglevd connected. Returns an error if VPP is not
                                 connected.
show vpp lb state                Show the VPP load-balancer plugin state: global
                                 configuration, configured VIPs, and their attached
                                 application servers (address, weight, bucket count).
                                 Returns an error if VPP is not connected.
show vpp lb counters             Show per-VIP and per-backend packet/byte counters
                                 from the VPP stats segment, refreshed roughly every
                                 five seconds by maglevd. Each VIP row reports the LB
                                 plugin counters (next, first, untracked, no-server)
                                 and the FIB packets/bytes at the VIP's host prefix.
                                 Each backend row reports FIB packets/bytes at the
                                 backend's /32 or /128 prefix. Use Prometheus for
                                 live rates; this command shows absolute values.

sync vpp lb state [<name>]       Reconcile the VPP load-balancer dataplane from the
                                 running config. Without a name: runs a full sync —
                                 creates missing VIPs, removes stale VIPs, and adjusts
                                 application-server membership and weights across all
                                 frontends. With a name: only the named frontend's VIP
                                 is reconciled, and no VIPs are removed. A full sync
                                 also runs automatically every
                                 maglev.vpp.lb.sync-interval (default 30s) to catch
                                 drift, and once on startup.

set backend <name> pause         Stop health checking for a backend. Cancels the probe
                                 goroutine so no further traffic is sent, and sets the
                                 state to 'paused'. The backend's transition history is
                                 preserved, so 'show backend <name>' still shows where
                                 it came from.
set backend <name> resume        Resume health checking. A fresh probe goroutine is
                                 started and the backend re-enters unknown state.
set backend <name> disable       Stop probing entirely and remove the backend from
                                 rotation. The backend remains visible (state: disabled)
                                 with its transition history intact and can be re-enabled
                                 without reloading configuration.
set backend <name> enable        Re-enable a disabled backend. A fresh probe goroutine is
                                 started and the backend re-enters unknown state.

set frontend <name> pool <pool> backend <name> weight <0-100> [flush]
                                 Set the weight of a backend within a pool. Weight 0 keeps
                                 the backend in the pool but assigns it no traffic. Takes
                                 effect immediately: maglevd pushes the change into VPP
                                 via a targeted single-VIP reconcile, so there's no need
                                 to wait for the periodic sync tick.

                                 Without `flush`, the new weight is installed in Maglev's
                                 new-bucket mapping but VPP's flow table is left alone.
                                 Existing sessions keep reaching this backend until they
                                 naturally drain — useful for graceful draining where
                                 you want new connections to land elsewhere but don't
                                 want to reset any in-flight traffic.

                                 With `flush`, the corresponding application-server row
                                 is rewritten with `lb_as_set_weight(is_flush=true)`,
                                 which clears VPP's flow table entries for this backend.
                                 Existing sessions are dropped immediately — useful when
                                 the backend is being taken out of service for emergency
                                 reasons and you don't want to wait for flows to drain.

                                 Examples:
                                   set frontend web pool primary backend nginx0 weight 50
                                   set frontend web pool primary backend nginx0 weight 0 flush

watch events                     Stream all events (log, backend transitions, frontend)
  [num <n>]                      Stop after receiving n events.
  [log [level <level>]]          Include log events. level is debug|info|warn|error
                                 (default: info). Omitting log/backend/frontend enables all.
  [backend]                      Include backend transition events.
  [frontend]                     Include frontend events (reserved for future use).

                                 Each event is printed as compact JSON on its own line.
                                 Press any key or Ctrl-C to stop. Examples:

                                   watch events
                                   watch events num 20
                                   watch events log level debug
                                   watch events backend num 100
                                   watch events log level debug backend

config check                     Ask maglevd to read and validate its current config file.
                                 Prints "config ok" on success, or the error (parse or
                                 semantic) returned by the daemon.
config reload                    Check and reload the configuration file. Equivalent to
                                 sending SIGHUP to maglevd. Prints "config reloaded" on
                                 success, or the specific error (parse, semantic, or
                                 reload) that prevented the reload.

quit / exit                      Leave the interactive shell.
```

### Interactive shell

The shell prompt is `maglev> `. Two completion mechanisms are available:

**Tab completion** — pressing `<Tab>` at any point completes the current token.
Fixed keywords (commands and subcommands) are completed from the command tree.
Backend, frontend, and health-check names are fetched live from the server with
a 1-second timeout. If the partial token is unambiguous the word is completed
in place; if multiple candidates exist they are listed and the prompt is
restored.

**Inline help (`?`)** — typing `?` at any point prints the available
completions for the current position, with a short description next to each
keyword. The `?` character is not added to the input line.

Commands and keywords support **prefix matching**: typing `sh ba` is equivalent
to `show backends`, and `sh ba nginx0` is equivalent to `show backends nginx0`.