Pim van Pelt
284b4cc9a4
Introduces maglev-frontend, a responsive, real-time web dashboard for one
or more running maglevd instances. Source lives at cmd/frontend/; the
built binary is maglev-frontend. It is a single Go process with the
SolidJS SPA embedded via //go:embed — no runtime file dependencies.
Architecture
- One persistent gRPC connection per configured maglevd (-server A,B,C).
Each connection runs three background loops: a WatchEvents stream
subscribed at log_level=debug for live events, a 30s refresh loop as
a safety net for drift, and a 5s health loop that surfaces connection
drops quickly.
- In-process pub/sub broker with a 30s / 2000-event replay ring using
<epoch>-<seq> monotonic IDs. Short browser reconnects (nginx idle,
wifi flap, laptop wake) silently replay buffered events via the
EventSource Last-Event-ID header; longer outages or frontend restarts
fall through to a "resync" event that triggers a full state refetch.
- HTTP surface: /view/ (SPA), /view/api/state, /view/api/state/{name},
/view/api/maglevds, /view/api/version, /view/api/events (SSE),
/healthz, and an /admin/* placeholder returning 501 for a future
basic-auth mutation surface.
- SSE handler follows the full operational checklist: retry hint, 15s
: ping heartbeat, Flush after every write, r.Context().Done() teardown,
X-Accel-Buffering: no, and no gzip.
SolidJS SPA (cmd/frontend/web/, Vite + TypeScript)
- solid-js/store for a reactive per-maglevd state tree; reducers apply
backend transitions, maglevd-status flips, and resync refetches.
- Scope selector tabs for multi-maglevd support, per-maglevd frontend
cards with pool tables showing state, configured weight, effective
weight, and last-transition age.
- ProbeHeartbeat component turns a middle-dot into ❤️ on probe-start and
back on probe-done, driven by real log events; fixed-size wrapper so
the emoji swap doesn't jiggle the row.
- Flash wrapper animates any primitive on change (1s yellow fade via
Web Animations API, skipped on first mount). Wired into the state
badge, configured weight, and effective weight columns.
- DebugPanel: chronological rolling event tail with tail-style auto-
scroll, pause/resume, and scope/firehose filter. Syntactic highlight
for vpp-lb-sync-* events with fixed-order attribute formatting.
- Live effective_weight updates: vpp-lb-sync-as-added/removed/weight-
updated log events are routed through a reducer that walks the
snapshot's pool rows and sets effective_weight on every match
without waiting for the 30s refresh.
- Header shows build version + commit with build date in a tooltip,
fetched once from /view/api/version on mount.
- Prettier wired in as the web-side fixstyle; make fixstyle now tidies
both Go and web in one shot via a new fixstyle-web target.
Per-mutation VPP LB sync logging
- Promotes the addVIP/delVIP/addAS/delAS/setASWeight helpers from
slog.Debug to slog.Info and renames them from vpp-lbsync-* to
vpp-lb-sync-{vip-added,vip-removed,as-added,as-removed,as-weight-
updated}. Matching rename for vpp-lb-sync-start / -done / -error /
-vip-recreate. The Prometheus metric name (maglev_vpp_lbsync_total)
is left alone to preserve dashboards.
- setASWeight now takes the prior weight so the event can emit
from=X to=Y and the UI can show the delta.
- The vip field in every event is the bare address (no /32 or /128
mask), matching the CLI output style.
- Any listener on the gRPC WatchEvents stream — CLI watch events or
maglev-frontend — now sees every VIP/AS dataplane change in real
time without needing to raise the log level.
Build and tooling
- Makefile: maglev-frontend added to BINARIES; build / build-amd64 /
build-arm64 emit the binary alongside maglevd and maglevc. A new
maglev-frontend-web target rebuilds the SolidJS bundle via npm.
- web/dist/ is tracked so a bare `go build` keeps working for Go-only
contributors and CI.
- .gitignore skips cmd/frontend/web/node_modules/.
Stability fixes
- maglevd's WatchEvents synthetic replay events (from==to, at_unix_ns=0)
were corrupting the frontend's LastTransition cache with at=0,
rendering as "20555d ago" in the browser. Client now skips synthetic
events: the cache comes from refreshAll and doesn't need them.
- Frontends, Backends, and HealthChecks are now served in the order
returned by the corresponding List* RPC instead of Go map iteration
order, so reloads and refreshes keep the SPA stable.
224 lines
13 KiB
Markdown
224 lines
13 KiB
Markdown
# User Guide
|
|
|
|
## maglevd
|
|
|
|
`maglevd` is the health-checker daemon. It probes backends according to the
|
|
configuration file, maintains their health state, and exposes a gRPC API for
|
|
inspection and control.
|
|
|
|
### Flags
|
|
|
|
| Flag | Environment variable | Default | Description |
|
|
|---|---|---|---|
|
|
| `--config` | `MAGLEV_CONFIG` | `/etc/vpp-maglev/maglev.yaml` | Path to the YAML configuration file. |
|
|
| `--grpc-addr` | `MAGLEV_GRPC_ADDR` | `:9090` | TCP address on which the gRPC server listens. |
|
|
| `--metrics-addr` | `MAGLEV_METRICS_ADDR` | `:9091` | TCP address for the Prometheus `/metrics` HTTP endpoint. Set to empty to disable. |
|
|
| `--vpp-api-addr` | `MAGLEV_VPP_API_ADDR` | `/run/vpp/api.sock` | VPP binary API socket path. Set to empty to disable VPP integration. |
|
|
| `--vpp-stats-addr` | `MAGLEV_VPP_STATS_ADDR` | `/run/vpp/stats.sock` | VPP stats socket path. |
|
|
| `--log-level` | `MAGLEV_LOG_LEVEL` | `info` | Log verbosity: `debug`, `info`, `warn`, or `error`. |
|
|
| `--check` | — | — | Read and validate the config file, then exit. Exits 0 if the config is valid, 1 on YAML parse error, 2 on semantic error. |
|
|
| `--reflection` | — | `true` | Enable gRPC server reflection. Allows `grpcurl` to introspect the API without the `.proto` file. Set to `false` to disable. |
|
|
| `--version` | — | — | Print version, commit hash, and build date, then exit. |
|
|
|
|
Flags take precedence over environment variables. Both are optional; defaults
|
|
are used for anything not set.
|
|
|
|
### Signals
|
|
|
|
| Signal | Effect |
|
|
|---|---|
|
|
| `SIGHUP` | Reload the configuration file (same code path as `config reload` in `maglevc`). The file is checked before applying; if there is a parse or semantic error the reload is aborted and the error is logged (the daemon continues running with its current config). New backends are started, removed backends are stopped, backends whose health-check config is unchanged continue probing without interruption. |
|
|
| `SIGTERM` / `SIGINT` | Graceful shutdown. Active gRPC streams are closed, the server drains, then the process exits. |
|
|
|
|
### Capabilities
|
|
|
|
`maglevd` requires `CAP_NET_RAW` when any health check uses `type: icmp`.
|
|
All other check types (`tcp`, `http`) use normal TCP sockets and require no
|
|
special capabilities.
|
|
|
|
### Logging
|
|
|
|
All log output is written to stdout as JSON using Go's `log/slog`. The first
|
|
line logged after the logger is configured is a `starting` record that includes
|
|
`version`, `commit`, and `date`. Every state change emits a `backend-transition`
|
|
line at `INFO` level. Per-mutation VPP LB sync events
|
|
(`vpp-lb-sync-vip-added`, `vpp-lb-sync-vip-removed`, `vpp-lb-sync-as-added`,
|
|
`vpp-lb-sync-as-removed`, `vpp-lb-sync-as-weight-updated`) are also emitted
|
|
at `INFO` so the CLI `watch events` stream and the web frontend see every
|
|
dataplane change without raising the log level. Set `--log-level debug` to
|
|
see individual probe attempts and every VPP binary-API call
|
|
(`vpp-api-send` / `vpp-api-recv` with full payload) as they happen.
|
|
|
|
### Prometheus metrics
|
|
|
|
`maglevd` exposes Prometheus metrics on `--metrics-addr` (default `:9091`) at
|
|
the `/metrics` path. Metric families:
|
|
|
|
**Health-check and backend state (gauges, on-demand):**
|
|
| Metric | Labels | Description |
|
|
|---|---|---|
|
|
| `maglev_backend_state` | `backend`, `address`, `healthcheck`, `state` | 1 for the current state row per backend, 0 otherwise. |
|
|
| `maglev_backend_health` | `backend` | Current rise/fall counter value. |
|
|
| `maglev_backend_enabled` | `backend` | 1 if enabled, 0 if disabled. |
|
|
| `maglev_frontend_pool_backend_weight` | `frontend`, `pool`, `backend` | Configured weight from YAML. |
|
|
|
|
**Probe counters and latency (inline):**
|
|
| Metric | Labels | Description |
|
|
|---|---|---|
|
|
| `maglev_probe_total` | `backend`, `type`, `result`, `code` | Probes executed. `result` is `success` or `failure`. |
|
|
| `maglev_probe_duration_seconds` | `backend`, `type` | Histogram of probe wall time. |
|
|
| `maglev_backend_transitions_total` | `backend`, `from`, `to` | State machine transitions. |
|
|
|
|
**VPP integration (when enabled):**
|
|
| Metric | Labels | Description |
|
|
|---|---|---|
|
|
| `maglev_vpp_connected` | — | 1 if maglevd currently has a live VPP connection. |
|
|
| `maglev_vpp_uptime_seconds` | — | Seconds since VPP started (from `/sys/boottime`). |
|
|
| `maglev_vpp_connected_seconds` | — | Seconds since maglevd established the current VPP connection. |
|
|
| `maglev_vpp_info` | `version`, `build_date`, `pid` | Static VPP build metadata; always 1. |
|
|
| `maglev_vpp_api_total` | `msg`, `direction`, `result` | VPP binary-API calls. `direction` is `send` or `recv`; `result` is `success` or `failure`. |
|
|
| `maglev_vpp_lbsync_total` | `scope`, `kind` | Per-mutation sync counters. `scope` is `all` or `vip`; `kind` is one of `vip_added`, `vip_removed`, `as_added`, `as_removed`, `as_weight_updated`. |
|
|
|
|
**gRPC server (standard `go-grpc-middleware/prometheus` metrics):**
|
|
`grpc_server_started_total`, `grpc_server_handled_total`,
|
|
`grpc_server_msg_received_total`, `grpc_server_msg_sent_total`, and
|
|
`grpc_server_handling_seconds` — all labelled by `grpc_service`,
|
|
`grpc_method`, `grpc_type`, and `grpc_code`. Every method is
|
|
pre-registered at zero so time series exist on the first scrape.
|
|
|
|
---
|
|
|
|
## maglevc
|
|
|
|
`maglevc` is the interactive control-plane client. It connects to a running
|
|
`maglevd` over gRPC and either executes a single command or drops into an
|
|
interactive shell.
|
|
|
|
### Usage
|
|
|
|
```sh
|
|
maglevc [--server host:port] [--color[=bool]] [command...]
|
|
```
|
|
|
|
| Flag | Default | Description |
|
|
|---|---|---|
|
|
| `--server` | `localhost:9090` | Address of the `maglevd` gRPC server. |
|
|
| `--color` | mode-aware | Colorize static field labels (dark blue ANSI). Defaults to `true` in the interactive shell and `false` in one-shot mode, so output piped into scripts stays free of escape codes. Pass `--color=true` or `--color=false` explicitly to override either default. |
|
|
|
|
When `command` arguments are supplied the command is executed and `maglevc`
|
|
exits; in this mode ANSI color is off by default so the output is script-safe.
|
|
When no arguments are given an interactive shell is started, the build version
|
|
is printed on entry, and color is on by default.
|
|
|
|
### Commands
|
|
|
|
```
|
|
show version Print build version, commit hash, and build date.
|
|
|
|
show frontends [<name>] Without name: list all frontend names.
|
|
With name: show address, protocol, port, src-ip-sticky,
|
|
description, and pools. Each pool lists its backends
|
|
with two weight columns:
|
|
weight — configured weight from the YAML
|
|
effective — state-aware weight after pool failover
|
|
(what gets programmed into VPP)
|
|
Disabled backends are marked with [disabled].
|
|
|
|
show backends [<name>] Without name: list all backend names.
|
|
With name: show address, current state (with duration),
|
|
enabled flag, health check, and recent state transitions
|
|
with timestamps and how long ago each occurred.
|
|
|
|
show healthchecks [<name>] Without name: list all health-check names.
|
|
With name: show full health-check configuration.
|
|
|
|
show vpp info Show VPP version, build date, PID, uptime, and when
|
|
maglevd connected. Returns an error if VPP is not
|
|
connected.
|
|
show vpp lb state Show the VPP load-balancer plugin state: global
|
|
configuration, configured VIPs, and their attached
|
|
application servers (address, weight, bucket count).
|
|
Returns an error if VPP is not connected.
|
|
show vpp lb counters Show per-VIP and per-backend packet/byte counters
|
|
from the VPP stats segment, refreshed roughly every
|
|
five seconds by maglevd. Each VIP row reports the LB
|
|
plugin counters (next, first, untracked, no-server)
|
|
and the FIB packets/bytes at the VIP's host prefix.
|
|
Each backend row reports FIB packets/bytes at the
|
|
backend's /32 or /128 prefix. Use Prometheus for
|
|
live rates; this command shows absolute values.
|
|
|
|
sync vpp lb state [<name>] Reconcile the VPP load-balancer dataplane from the
|
|
running config. Without a name: runs a full sync —
|
|
creates missing VIPs, removes stale VIPs, and adjusts
|
|
application-server membership and weights across all
|
|
frontends. With a name: only the named frontend's VIP
|
|
is reconciled, and no VIPs are removed. A full sync
|
|
also runs automatically every
|
|
maglev.vpp.lb.sync-interval (default 30s) to catch
|
|
drift, and once on startup.
|
|
|
|
set backend <name> pause Stop health checking for a backend. Cancels the probe
|
|
goroutine so no further traffic is sent, and sets the
|
|
state to 'paused'. The backend's transition history is
|
|
preserved, so 'show backend <name>' still shows where
|
|
it came from.
|
|
set backend <name> resume Resume health checking. A fresh probe goroutine is
|
|
started and the backend re-enters unknown state.
|
|
set backend <name> disable Stop probing entirely and remove the backend from
|
|
rotation. The backend remains visible (state: disabled)
|
|
with its transition history intact and can be re-enabled
|
|
without reloading configuration.
|
|
set backend <name> enable Re-enable a disabled backend. A fresh probe goroutine is
|
|
started and the backend re-enters unknown state.
|
|
|
|
set frontend <name> pool <pool> backend <name> weight <0-100>
|
|
Set the weight of a backend within a pool. Weight 0 keeps
|
|
the backend in the pool but assigns it no traffic.
|
|
Takes effect immediately without reloading configuration.
|
|
|
|
watch events Stream all events (log, backend transitions, frontend)
|
|
[num <n>] Stop after receiving n events.
|
|
[log [level <level>]] Include log events. level is debug|info|warn|error
|
|
(default: info). Omitting log/backend/frontend enables all.
|
|
[backend] Include backend transition events.
|
|
[frontend] Include frontend events (reserved for future use).
|
|
|
|
Each event is printed as compact JSON on its own line.
|
|
Press any key or Ctrl-C to stop. Examples:
|
|
|
|
watch events
|
|
watch events num 20
|
|
watch events log level debug
|
|
watch events backend num 100
|
|
watch events log level debug backend
|
|
|
|
config check Ask maglevd to read and validate its current config file.
|
|
Prints "config ok" on success, or the error (parse or
|
|
semantic) returned by the daemon.
|
|
config reload Check and reload the configuration file. Equivalent to
|
|
sending SIGHUP to maglevd. Prints "config reloaded" on
|
|
success, or the specific error (parse, semantic, or
|
|
reload) that prevented the reload.
|
|
|
|
quit / exit Leave the interactive shell.
|
|
```
|
|
|
|
### Interactive shell
|
|
|
|
The shell prompt is `maglev> `. Two completion mechanisms are available:
|
|
|
|
**Tab completion** — pressing `<Tab>` at any point completes the current token.
|
|
Fixed keywords (commands and subcommands) are completed from the command tree.
|
|
Backend, frontend, and health-check names are fetched live from the server with
|
|
a 1-second timeout. If the partial token is unambiguous the word is completed
|
|
in place; if multiple candidates exist they are listed and the prompt is
|
|
restored.
|
|
|
|
**Inline help (`?`)** — typing `?` at any point prints the available
|
|
completions for the current position, with a short description next to each
|
|
keyword. The `?` character is not added to the input line.
|
|
|
|
Commands and keywords support **prefix matching**: typing `sh ba` is equivalent
|
|
to `show backends`, and `sh ba nginx0` is equivalent to `show backends nginx0`.
|