Pim van Pelt
d5fbf5c640
New metrics plus the corresponding documentation for everything that's
accumulated since the last Prometheus pass.
internal/metrics/metrics.go
- New VPPSource interface (IsConnected, VPPInfo) plus a metrics-local
VPPInfo struct that mirrors vpp.Info. Decoupling via interface +
struct-mirror keeps the dependency direction one-way (vpp → metrics),
so vpp can import metrics to update inline counters without a cycle.
- New Collector gauges scraped on demand: maglev_vpp_connected,
maglev_vpp_uptime_seconds (from /sys/boottime), maglev_vpp_connected_seconds
(time since maglevd connected), and maglev_vpp_info (static 1-gauge
carrying version, build_date, and pid as labels).
- New inline counters:
- maglev_vpp_api_total{msg, direction, result} — bumped from the
loggedChannel wrapper on every VPP binary-API send/recv. Gives full
visibility into what maglevd is doing with VPP, broken down by
message name, direction (send/recv), and result (success/failure).
- maglev_vpp_lbsync_total{scope, kind} — bumped from the reconciler
at the end of each SyncLBStateAll/SyncLBStateVIP run. kind ∈
{vip_added, vip_removed, as_added, as_removed, as_weight_updated};
scope ∈ {all, vip}. Zero-valued kinds are not emitted so noise
stays low.
- Register() signature now takes a VPPSource (may be nil) alongside
the existing StateSource.
internal/vpp/client.go
- New VPPInfo() (metrics.VPPInfo, bool) shim method on *Client that
satisfies metrics.VPPSource. Returns (_, false) when disconnected so
the collector skips the vpp_* gauges cleanly.
internal/vpp/apilog.go
- The loggedChannel's SendRequest / SendMultiRequest / ReceiveReply
paths now call metrics.VPPAPITotal.WithLabelValues(...).Inc() in
addition to slog.Debug. Since every VPP API call in the codebase
must go through loggedChannel (NewAPIChannel is unexported), this
one instrumentation point catches everything.
internal/vpp/lbsync.go
- New recordSyncStats(scope, st) helper called once at the end of
SyncLBStateAll and SyncLBStateVIP to bump maglev_vpp_lbsync_total.
Zero-valued stats are skipped.
cmd/maglevd/main.go
- Added github.com/grpc-ecosystem/go-grpc-middleware/providers/prometheus
for the standard gRPC server metrics (grpc_server_started_total,
grpc_server_handled_total, grpc_server_handling_seconds, etc.,
labelled by service/method/type/code).
- Constructs grpcprom.NewServerMetrics(WithServerHandlingTimeHistogram())
before creating the grpc.Server, installs it as UnaryInterceptor +
StreamInterceptor, then calls InitializeMetrics(srv) after service
registration so every method appears at 0 on the first scrape
instead of materialising lazily on first RPC.
- Passes the vppClient (or nil) as a metrics.VPPSource to
metrics.Register so the vpp_* gauges are emitted when integration
is enabled and silently omitted otherwise.
docs/user-guide.md
- New 'Prometheus metrics' section in the maglevd chapter,
tabulating every metric family: backend state gauges, probe
counters/histogram, transition counters, the new VPP gauges and
counters, and the standard gRPC server metrics.
- 'show frontends <name>' description updated to mention the two
weight columns ('weight' = configured from YAML, 'effective' =
state-aware after pool-failover logic).
- Pause / disable descriptions clarified: transition history is
preserved across these operator actions.
docs/healthchecks.md
- New 'Static (no-healthcheck) backends' section explaining that
backends without a healthcheck use rise/fall=1, fire a synthetic
passing probe immediately on startup (no 30s wait), and idle at
30s between iterations thereafter.
- New 'Pool failover' section documenting the priority-tier model,
the active-pool definition, when promotion happens, cascading to
further tiers, and graceful drain on demotion. Points readers at
'maglevc show frontends <name>' as the inspection interface.
docs/config-guide.md
- healthcheck field doc now describes static-backend behavior and
cross-references healthchecks.md.
- pools field doc now explains failover semantics at a high level
and cross-references the detailed healthchecks.md section.
340 lines
12 KiB
Markdown
340 lines
12 KiB
Markdown
# maglevd Configuration Guide
|
|
|
|
## Overview
|
|
|
|
`maglevd` consumes a YAML configuration file of a specific format. Validation is performed
|
|
in two stages:
|
|
|
|
1. **Structural parsing**: the YAML is unmarshalled into typed Go structs. Unknown fields and
|
|
type mismatches are rejected immediately.
|
|
1. **Semantic validation**: cross-field and cross-object rules are enforced, for example
|
|
ensuring that every backend referenced by a frontend exists, that address families are
|
|
consistent within a frontend, and that IP source addresses are the correct family.
|
|
|
|
If you want to get started quickly, take a look at the [example config](../debian/maglev.yaml).
|
|
|
|
## Basic structure
|
|
|
|
The YAML configuration file has the following top-level structure:
|
|
|
|
```yaml
|
|
maglev:
|
|
healthchecker:
|
|
[ Global health checker settings ]
|
|
|
|
vpp:
|
|
lb:
|
|
[ VPP load-balancer integration settings ]
|
|
|
|
healthchecks:
|
|
my-check:
|
|
[ Health check definition ]
|
|
|
|
backends:
|
|
my-backend:
|
|
[ Backend definition ]
|
|
|
|
frontends:
|
|
my-frontend:
|
|
[ Frontend (VIP) definition ]
|
|
```
|
|
|
|
All five sections live under the top-level `maglev:` key. The `healthchecks`, `backends`,
|
|
and `frontends` sections are maps keyed by an arbitrary name of your choosing. Names must be
|
|
unique within their section and are case-sensitive. The `vpp` section is required when
|
|
`maglevd` has a working VPP connection — its `lb.ipv4-src-address` and `lb.ipv6-src-address`
|
|
fields are mandatory and `maglevd` will refuse to start without them.
|
|
|
|
---
|
|
|
|
## healthchecker
|
|
|
|
Global settings for the health checker engine.
|
|
|
|
* ***transition-history***: An integer >= 1 that controls how many state transitions are
|
|
retained per backend for display via the gRPC API. Defaults to `5`.
|
|
* ***netns***: The name of a Linux network namespace in which probes are executed. When
|
|
empty or omitted, probes run in the current (default) network namespace. Useful when
|
|
backends are reachable only through a dedicated dataplane namespace.
|
|
|
|
Example:
|
|
```yaml
|
|
maglev:
|
|
healthchecker:
|
|
transition-history: 10
|
|
netns: dataplane
|
|
```
|
|
|
|
---
|
|
|
|
## vpp
|
|
|
|
Settings controlling the integration with a locally running VPP instance. The
|
|
`vpp` section is a map with a single sub-section, `lb`. Both `lb.ipv4-src-address`
|
|
and `lb.ipv6-src-address` are **required** — `maglevd --check` exits with a
|
|
semantic error and the daemon refuses to start when either is missing, because
|
|
VPP's GRE encap needs a source address and every VIP `maglevd` programs uses GRE.
|
|
|
|
* ***lb.ipv4-src-address***: Required. The IPv4 source address VPP uses when
|
|
encapsulating IPv4 traffic into GRE4 tunnels to application servers. Must
|
|
be a valid IPv4 address. No default.
|
|
* ***lb.ipv6-src-address***: Required. The IPv6 source address VPP uses when
|
|
encapsulating IPv6 traffic into GRE6 tunnels. Must be a valid IPv6 address.
|
|
No default.
|
|
* ***lb.sync-interval***: A positive Go duration (e.g. `30s`, `1m`) controlling
|
|
how often `maglevd` reconciles the VPP load-balancer dataplane against its
|
|
running configuration. On startup, an immediate full sync runs; subsequent
|
|
syncs fire at this interval as long as the VPP connection is up. Defaults
|
|
to `30s`. The purpose is to catch drift — for example, a VIP added to VPP
|
|
by hand — and bring VPP back in line with the maglev config.
|
|
* ***lb.sticky-buckets-per-core***: The number of buckets per worker thread in
|
|
the established-flow table. Must be a power of 2. Defaults to `65536` (64k).
|
|
* ***lb.flow-timeout***: Idle time after which an established flow is removed
|
|
from the table. Must be a whole number of seconds between `1s` and `120s`
|
|
inclusive. Defaults to `40s`.
|
|
|
|
These four values are pushed to VPP via `lb_conf` when `maglevd` connects to
|
|
VPP and again after every config reload (whenever they change). A log line
|
|
`vpp-lb-conf-set` records the effective values.
|
|
|
|
Example:
|
|
```yaml
|
|
maglev:
|
|
vpp:
|
|
lb:
|
|
sync-interval: 60s
|
|
ipv4-src-address: 10.0.0.1
|
|
ipv6-src-address: 2001:db8::1
|
|
sticky-buckets-per-core: 65536
|
|
flow-timeout: 40s
|
|
```
|
|
|
|
---
|
|
|
|
## healthchecks
|
|
|
|
A named map of health check definitions. Each health check describes *how* to probe a backend.
|
|
Backends reference health checks by name. The same health check can be reused across any number
|
|
of backends; each backend is probed exactly once regardless of how many frontends reference it.
|
|
|
|
Common fields (all types):
|
|
|
|
* ***type***: Required. One of `icmp`, `tcp`, `http`, or `https`.
|
|
* ***port***: The destination port to probe. Required for `tcp`, `http`, and `https`.
|
|
Must be omitted for `icmp`.
|
|
* ***probe-ipv4-src***: An optional IPv4 source address used when probing IPv4 backends.
|
|
Must be an IPv4 address. When omitted, the OS chooses the source address.
|
|
* ***probe-ipv6-src***: An optional IPv6 source address used when probing IPv6 backends.
|
|
Must be an IPv6 address. When omitted, the OS chooses the source address.
|
|
* ***interval***: Required. A positive Go duration string (e.g. `2s`, `500ms`) controlling
|
|
how often a probe is sent when the backend is fully healthy (counter at maximum).
|
|
* ***fast-interval***: Optional. A positive duration used instead of `interval` while the
|
|
backend's health counter is degraded (between down and up) or in `unknown` state. When
|
|
omitted, `interval` is used.
|
|
* ***down-interval***: Optional. A positive duration used instead of `interval` while the
|
|
backend is fully down (counter at zero). When omitted, `interval` is used. Setting this to
|
|
a longer value reduces probe traffic to backends that are known to be offline.
|
|
* ***timeout***: Required. A positive duration after which an in-flight probe is abandoned
|
|
and counted as a failure.
|
|
* ***rise***: The number of consecutive successes required to transition from down to up.
|
|
Defaults to `2`. Must be >= 1.
|
|
* ***fall***: The number of consecutive failures required to transition from up to down.
|
|
Defaults to `3`. Must be >= 1.
|
|
|
|
### type: icmp
|
|
|
|
Sends an ICMP echo request (ping) to the backend address. Requires `CAP_NET_RAW`. No `port`
|
|
may be specified. No `params` block is used.
|
|
|
|
```yaml
|
|
healthchecks:
|
|
ping:
|
|
type: icmp
|
|
probe-ipv4-src: 10.0.0.1
|
|
probe-ipv6-src: 2001:db8::1
|
|
interval: 2s
|
|
timeout: 1s
|
|
rise: 2
|
|
fall: 3
|
|
```
|
|
|
|
### type: tcp
|
|
|
|
Opens a TCP connection to the backend and immediately closes it upon success. Use `params` to
|
|
optionally wrap the connection in TLS.
|
|
|
|
* ***params.ssl***: A boolean. When `true`, a TLS handshake is performed after the TCP
|
|
connection is established. Defaults to `false`.
|
|
* ***params.server-name***: The TLS SNI hostname sent during the handshake. When omitted,
|
|
the backend IP address is used.
|
|
* ***params.insecure-skip-verify***: A boolean. When `true`, the TLS certificate presented
|
|
by the server is not verified. Defaults to `false`.
|
|
|
|
```yaml
|
|
healthchecks:
|
|
imaps-check:
|
|
type: tcp
|
|
port: 993
|
|
params:
|
|
ssl: true
|
|
server-name: imaps.example.com
|
|
interval: 5s
|
|
timeout: 3s
|
|
rise: 2
|
|
fall: 3
|
|
```
|
|
|
|
### type: http / https
|
|
|
|
Opens a TCP (or TLS for `https`) connection, sends an HTTP request, and evaluates the response
|
|
code. An optional regexp can additionally match against the response body.
|
|
|
|
* ***params.path***: Required. The HTTP request path, e.g. `/healthz`.
|
|
* ***params.host***: The `Host` header value sent in the request. When omitted, the backend
|
|
IP address is used.
|
|
* ***params.response-code***: The expected HTTP response code. Can be a single value (`"200"`)
|
|
or an inclusive range (`"200-299"`). Defaults to `"200"`.
|
|
* ***params.response-regexp***: An optional Go regular expression matched against the response
|
|
body. If specified, the body must match for the probe to succeed.
|
|
* ***params.server-name***: The TLS SNI hostname (`https` only). Defaults to the value of
|
|
`params.host` if not set.
|
|
* ***params.insecure-skip-verify***: A boolean. Skip TLS certificate verification (`https`
|
|
only). Defaults to `false`.
|
|
|
|
```yaml
|
|
healthchecks:
|
|
nginx-http:
|
|
type: http
|
|
port: 80
|
|
params:
|
|
path: /healthz
|
|
host: nginx.example.com
|
|
response-code: "200-204"
|
|
interval: 2s
|
|
fast-interval: 500ms
|
|
down-interval: 30s
|
|
timeout: 3s
|
|
rise: 2
|
|
fall: 3
|
|
|
|
nginx-https:
|
|
type: https
|
|
port: 443
|
|
params:
|
|
path: /healthz
|
|
host: nginx.example.com
|
|
server-name: nginx.example.com
|
|
insecure-skip-verify: false
|
|
interval: 5s
|
|
timeout: 3s
|
|
```
|
|
|
|
---
|
|
|
|
## backends
|
|
|
|
A named map of individual backend servers. Each backend has a single IP address and optionally
|
|
references a health check by name. Backends are probed exactly once, even if they appear in
|
|
multiple frontends.
|
|
|
|
* ***address***: Required. The IPv4 or IPv6 address of this backend server.
|
|
* ***healthcheck***: The name of a health check defined in the `healthchecks` section.
|
|
When empty or omitted, the backend is static: no probing is performed and the backend
|
|
enters `StateUp` immediately on startup (via a synthetic pass, rise/fall forced to 1/1).
|
|
This is useful for backends that are always available or managed by other means. See
|
|
[healthchecks.md](healthchecks.md) for details on the static-backend behavior.
|
|
* ***enabled***: A boolean controlling whether this backend participates in any frontend.
|
|
When `false`, the backend is excluded entirely and no probe goroutine is started.
|
|
Defaults to `true`.
|
|
|
|
Examples:
|
|
```yaml
|
|
backends:
|
|
nginx0-ams:
|
|
address: 198.51.100.10
|
|
healthcheck: nginx-http
|
|
nginx0-lon:
|
|
address: 198.51.100.11
|
|
healthcheck: nginx-http
|
|
nginx0-draining:
|
|
address: 198.51.100.12
|
|
healthcheck: nginx-http
|
|
enabled: false
|
|
static-backend:
|
|
address: 198.51.100.20
|
|
# no healthcheck: assumed always healthy
|
|
```
|
|
|
|
---
|
|
|
|
## frontends
|
|
|
|
A named map of virtual IPs (VIPs). Each frontend ties together a listener address with an
|
|
ordered list of backend pools. The gRPC API exposes frontends by name.
|
|
|
|
* ***description***: An optional free-text string for documentation purposes.
|
|
* ***address***: Required. The IPv4 or IPv6 address of the VIP.
|
|
* ***protocol***: The IP protocol, either `tcp` or `udp`. When omitted, the frontend matches
|
|
all traffic to the VIP address regardless of protocol. If `port` is specified, `protocol`
|
|
must also be set.
|
|
* ***port***: The destination port of the VIP, an integer between 1 and 65535. Requires
|
|
`protocol` to be set. When omitted, the frontend matches all ports. Note that the
|
|
frontend port is independent of the healthcheck port: a frontend on port 443 may use
|
|
a healthcheck that probes port 80.
|
|
* ***pools***: Required. A non-empty ordered list of pool objects. Pools express priority:
|
|
the first pool is preferred; subsequent pools act as fallbacks. When every backend in
|
|
pool[0] leaves `StateUp` (down, paused, disabled, or not yet probed), pool[1] is
|
|
automatically promoted — its up backends take over serving traffic. The promotion
|
|
cascades across further tiers. See [healthchecks.md](healthchecks.md#pool-failover)
|
|
for the full failover semantics. All backends across all pools in a frontend must
|
|
have addresses of the same address family (all IPv4 or all IPv6).
|
|
|
|
Each pool has:
|
|
|
|
* ***name***: Required. A non-empty string identifying the pool (e.g. `primary`, `fallback`).
|
|
* ***backends***: A map of backend names to per-pool backend options. Every name must refer
|
|
to an existing entry in the `backends` section.
|
|
|
|
Per-pool backend options:
|
|
|
|
* ***weight***: An integer between 0 and 100 (inclusive) expressing the relative weight of
|
|
this backend within the pool. `0` keeps the backend in the pool but assigns it no traffic.
|
|
Defaults to `100`. Weight is per-pool, not global — the same backend can appear with
|
|
different weights in different frontends.
|
|
|
|
Examples:
|
|
```yaml
|
|
frontends:
|
|
nginx-v4-http:
|
|
description: "IPv4 HTTP VIP with fallback"
|
|
address: 198.51.100.1
|
|
protocol: tcp
|
|
port: 80
|
|
pools:
|
|
- name: primary
|
|
backends:
|
|
nginx0-ams: { weight: 10 }
|
|
nginx0-lon: {}
|
|
- name: fallback
|
|
backends:
|
|
nginx0-fra: {}
|
|
|
|
maildrop-imaps:
|
|
description: "IMAPS VIP"
|
|
address: 2001:db8::1
|
|
protocol: tcp
|
|
port: 993
|
|
pools:
|
|
- name: primary
|
|
backends:
|
|
maildrop0-ams: {}
|
|
maildrop0-lon: {}
|
|
```
|
|
|
|
---
|
|
|
|
For a detailed description of the health state machine, probe intervals, and all transition events,
|
|
see [healthchecks.md](healthchecks.md). For a user guide on how to use the maglev daemon and client,
|
|
see the [user-guide.md](user-guide.md).
|
|
|