vpp-maglev/docs/design.md

# vpp-maglev Design Document

## Metadata

| | |
| --- | --- |
| **Status** | Retrofit — describes shipped behavior as of `v1.0.0` |
| **Author** | Pim van Pelt `<pim@ipng.ch>` |
| **Last updated** | 2026-04-15 |
| **Audience** | Operators and contributors who will read the source tree next |

The key words **MUST**, **MUST NOT**, **SHOULD**, **SHOULD NOT**, and
**MAY** are used as described in
[RFC 2119](https://datatracker.ietf.org/doc/html/rfc2119), and are
reserved in this document for requirements that are actually enforced
in code or by an external dependency. Plain-language descriptions of
what the system or an operator can do are written in lowercase —
"can", "will", "does" — and should not be read as normative.

## Summary

`vpp-maglev` is a control plane for the VPP `lb` (Maglev load
balancer) plugin. A single daemon — `maglevd` — probes a fleet of
backends, maintains an authoritative view of their health, and
programs the VPP dataplane so that traffic hashed to a given VIP
lands only on healthy backends. Operators drive the system through
`maglevc` (an interactive CLI) or `maglevd-frontend` (a read-only
web dashboard with an optional authenticated admin surface). A small
companion binary, `maglevt`, validates VIPs from outside the control
plane by sending live HTTP probes and reporting failover behavior.

## Background

VPP's `lb` plugin implements Maglev consistent hashing inside the
dataplane: a VIP is backed by a pool of Application Servers (ASes),
each with an integer weight in `[0, 100]`, and incoming flows are
hashed onto a bucket ring so that weight changes disturb as few
existing flows as possible. The plugin knows nothing about backend
health; if an AS dies while it holds buckets, traffic to those
buckets is black-holed until something external tells `lb` to remove
or re-weight the AS.

`vpp-maglev` is that external thing. Before `vpp-maglev`, operators
maintained VIP configurations by hand and reacted to incidents with
`vppctl`. The project replaces that loop with a daemon that owns the
health story, reconciles it with the dataplane, and exposes the
result through a uniform gRPC API so that CLIs, dashboards, and
scripts all read the same source of truth.

## Goals and Non-Goals

### Product Goals

1. **Accurate backend health.** Detect that a backend is up,
   degraded, or down quickly enough to keep user-visible error rates
   low, and avoid flapping under transient faults.
2. **Correct VPP state.** The set of VIPs and per-AS weights in VPP
   converges to the configured intent, filtered by current health,
   for every supported failure mode.
3. **Restart neutrality.** Restarting `maglevd` with VPP already up
   MUST NOT cause traffic to be black-holed while health probes warm
   up.
4. **Operator control.** A human can pause, drain, or weight-shift
   a backend in seconds without editing config files.
5. **Uniform observability.** Every state transition, VPP API call,
   and probe result is emitted as a structured log, a Prometheus
   metric, or a streaming event — ideally all three.
6. **One source of truth.** Every other component (CLI, web
   frontend, scripts) reads `maglevd` through one typed interface.
   There is no secondary control plane.

### Non-Goals

- `vpp-maglev` is not a VPP installer or packaging layer. It assumes
  VPP is already running with the `lb` plugin loaded.
- It does not implement its own dataplane fast path. All forwarding
  stays in VPP; `maglevd` only programs the plugin.
- It is not a generic service mesh. There is no L7 routing, cert
  issuance, service discovery, or east-west policy — only VIPs,
  pools, and backends.
- It is not a config store. Configuration is a YAML file on disk;
  the gRPC API can check and reload it but cannot author it.
- It does not secure its own transport. gRPC runs insecure by
  default; TLS, mTLS, or firewalls are the operator's
  responsibility.

## Requirements

Each requirement carries a unique identifier (`FR-X.Y` or `NFR-X.Y`)
so that later sections can cite it.

### Functional Requirements

**FR-1 Health checking**

- **FR-1.1** The system supports ICMP, TCP, HTTP, and HTTPS health
  checks, each with its own protocol-specific success criteria.
- **FR-1.2** Each health check MUST apply HAProxy rise/fall
  semantics with operator-configurable thresholds.
- **FR-1.3** A health check MAY declare distinct `interval`,
  `fast-interval`, and `down-interval` values so that recovery from
  a degraded or down state is faster than steady-state polling.
- **FR-1.4** Each probe attempt is bounded by a configurable
  per-probe timeout, independent of the scheduling interval.
- **FR-1.5** If the configuration sets `healthchecker.netns`, every
  probe MUST execute inside the named Linux network namespace.
- **FR-1.6** The first probe result against a newly-created backend
  forces an immediate transition out of `Unknown`, without waiting
  for `rise` or `fall` consecutive results.
- **FR-1.7** A backend MAY omit its `healthcheck` reference to
  declare itself **static**. A static backend is not probed and is
  treated as permanently Up; it still participates in pool failover
  and still honors operator Pause and Disable overrides.

**FR-2 Aggregation and pool failover**

- **FR-2.1** A frontend MAY reference one or more named pools.
  Each referenced pool MUST contain at least one
  `(backend, configured-weight)` tuple; an empty pool is a
  configuration error and is rejected at load time.
- **FR-2.2** At any time, exactly one pool — the first, in
  configuration order, that contains a healthy backend with
  non-zero configured weight — is active; backends in other pools
  contribute zero effective weight.
- **FR-2.3** The effective weight of a `(frontend, pool, backend)`
  tuple is the configured weight when the backend is Up **and** the
  pool is active, and zero in every other case.
- **FR-2.4** A frontend's aggregate state is Up when at least one
  backend has non-zero effective weight, Unknown when every
  referenced backend is still Unknown (or the frontend references
  no backends), and Down otherwise.

**FR-3 Operator control**

- **FR-3.1** Operators can pause and resume individual backends at
  runtime. Pausing stops the probe worker, freezes the rise/fall
  counter, and drives effective weight to zero in **every** pool
  and **every** frontend that references the backend. Existing
  flows are not torn down; this is a soft drain.
- **FR-3.2** Operators can disable and re-enable individual
  backends at runtime. Disabling drives effective weight to zero in
  **every** pool and **every** frontend that references the
  backend, and MUST cause existing flows to be torn down on the
  next VPP sync.
- **FR-3.3** Operators can set the configured weight of a specific
  `(frontend, pool, backend)` tuple at runtime.
- **FR-3.4** Operator overrides (Pause, Disable) and operator
  weight mutations survive a configuration **reload** (`SIGHUP`)
  as long as the underlying backend and tuple still exist in the
  new configuration.
- **FR-3.5** Operator overrides and operator weight mutations do
  **not** survive a `maglevd` **restart**. After a restart, the
  YAML configuration file is authoritative for every backend and
  every tuple: paused backends come back unpaused, disabled
  backends come back enabled, mutated weights revert to the
  configured value. Operators who need persistent changes must
  edit the config file.

**FR-4 VPP reconciliation**

- **FR-4.1** For every backend state transition that changes an
  effective weight, `maglevd` pushes the resulting AS state into
  VPP for every affected VIP.
- **FR-4.2** `maglevd` runs a periodic full reconciliation on a
  configurable cadence (default thirty seconds) as a safety net
  against missed events and VPP restarts.
- **FR-4.3** Weight-to-zero is communicated to VPP as a graceful
  drain by default; transitions to Disabled and transitions to
  Down while `flush-on-down` is true MUST tear existing flows down
  on the next sync.
- **FR-4.4** `maglevd` tolerates VPP disconnects by auto-reconnecting
  and resuming reconciliation once the connection is
  re-established.

**FR-5 Configuration**

- **FR-5.1** Configuration is loaded from a single YAML file
  specified at startup and referenced by all later operations.
- **FR-5.2** Configuration validation distinguishes **parse
  errors** (malformed YAML) from **semantic errors** (structural
  invariants) and MUST report each with its own exit code from
  `--check`: 0 (OK), 1 (parse), 2 (semantic).
- **FR-5.3** `maglevd` reloads its configuration on `SIGHUP`
  without restarting the process, without restarting unchanged
  probe workers, and without losing operator overrides (see
  FR-3.4).
- **FR-5.4** A parse or semantic error encountered during reload
  MUST leave the running configuration in place.
- **FR-5.5** The same validation and reload semantics are also
  reachable through gRPC (`CheckConfig`, `ReloadConfig`).

**FR-6 Observability**

- **FR-6.1** All logs are emitted as structured JSON on stdout at
  a configurable level.
- **FR-6.2** `maglevd` exposes Prometheus metrics for probe
  outcomes, probe latency, backend state transitions, VPP API
  traffic, and VPP LB sync mutations.
- **FR-6.3** A streaming gRPC API multiplexes log entries, backend
  transitions, and frontend aggregate transitions to any number of
  subscribers with per-subscriber filters.
- **FR-6.4** Per-VIP packet counters from VPP's stats segment are
  surfaced through both the gRPC API and the Prometheus surface.

**FR-7 Clients and peripheral tools**

- **FR-7.1** An interactive CLI (`maglevc`) provides a
  tab-completing shell and a one-shot command mode, both backed
  by the same command tree.
- **FR-7.2** A web frontend (`maglevd-frontend`) can multiplex more
  than one `maglevd` in a single process and present their
  combined state.
- **FR-7.3** The web frontend partitions its HTTP surface into a
  public read-only path (`/view/`) and an authenticated mutating
  path (`/admin/`). If credentials are not configured, `/admin/`
  MUST NOT be advertised (the path returns 404).
- **FR-7.4** An out-of-band tester (`maglevt`) probes configured
  VIPs from outside the control plane, measures latency, and
  tallies a configurable response header.

### Non-Functional Requirements

**NFR-1 Availability and reliability**

- **NFR-1.1** A `maglevd` outage MUST NOT stop the dataplane.
  While `maglevd` is absent, VPP continues to forward traffic
  with its last-programmed state.
- **NFR-1.2** Restarting `maglevd` with VPP up MUST NOT black-hole
  new flows during the probe warm-up window; this is enforced by
  the startup warmup state machine described under `maglevd`.
- **NFR-1.3** The warmup clock is tied to process start and MUST
  NOT be reset by VPP reconnects or configuration reloads.
- **NFR-1.4** A `maglevd`-side reload with a broken file MUST NOT
  interrupt any running probe.

**NFR-2 Determinism and correctness**

- **NFR-2.1** Two `maglevd` instances given the same configuration
  and the same backend state MUST issue the same sequence of
  `lb_as_add_del` calls to VPP, so that VPP's bucket assignment is
  stable across process swaps. This is the job of the
  deterministic AS ordering rule.
- **NFR-2.2** Configuration reload MUST be atomic: either every
  change in the new file takes effect, or none of them do.
- **NFR-2.3** Probe scheduling SHOULD apply bounded jitter so
  that, after a daemon restart or a configuration reload, probes
  do not phase-lock to the wall clock.
- **NFR-2.4** Operator mutations, event-driven syncs, and
  periodic full syncs against VPP MUST be serialized with respect
  to one another; they MUST NOT interleave.

**NFR-3 Performance and scalability**

- **NFR-3.1** Probing N backends costs roughly N goroutines doing
  mostly idle waits; there is no central probe scheduler.
- **NFR-3.2** Event fan-out, transition history, and
  per-subscriber event queues MUST all be bounded; no structure
  grows without limit under sustained load.
- **NFR-3.3** VPP stats snapshots are published as an atomic
  pointer so that Prometheus scrapes and gRPC counter reads are
  wait-free.
- **NFR-3.4** A gRPC subscriber that cannot keep up MUST be
  dropped rather than blocking the central fan-out.

**NFR-4 Security**

- **NFR-4.1** `maglevd` runs with only the Linux capabilities it
  actually needs: `CAP_NET_RAW` only when ICMP probes are in use,
  `CAP_SYS_ADMIN` only when `healthchecker.netns` is set.
- **NFR-4.2** gRPC transport security is explicitly out of scope;
  the daemon runs insecure by default and deployments SHOULD
  front it with a firewall, a trusted network, or a
  TLS-terminating sidecar.
- **NFR-4.3** The web frontend's mutating surface MUST be hidden
  entirely (HTTP 404) when either of its basic-auth environment
  variables is unset.

**NFR-5 Operability**

- **NFR-5.1** Every CLI flag on every binary SHOULD have an
  environment-variable equivalent so that the binaries can be
  driven purely through env in container deployments.
- **NFR-5.2** `maglevd --check` MUST provide a stable exit-code
  contract (0 / 1 / 2) for use by packaging scripts and
  `ExecStartPre` handlers.
- **NFR-5.3** Dashboards can track state in real time through the
  streaming event interface rather than by tight polling.
- **NFR-5.4** `maglevc` and `maglevd-frontend` MUST NOT maintain
  any authoritative state of their own; all truth lives in
  `maglevd`.

## Architecture Overview

### Process Model

The system ships as three independent executables plus one optional
companion tester:

- **`maglevd`** — the long-running daemon. Hosts both the health
  checker and the VPP control plane.
- **`maglevc`** — short-lived CLI client.
- **`maglevd-frontend`** — long-running web dashboard (optional).
- **`maglevt`** — short-lived out-of-band probe TUI (optional).

VPP itself is a fourth moving part, but it is an external
dependency, not part of the `vpp-maglev` codebase.

### Data Flow

Configuration flows **in** from a YAML file on disk (read by
`maglevd`) and from runtime mutations issued over gRPC by `maglevc`
or `maglevd-frontend`. Health state flows **out** of `maglevd` in
three directions: into VPP (as AS weight changes), into Prometheus
(as metrics), and into gRPC clients (as streaming events and
snapshot reads). Traffic counters flow **back in** from VPP's stats
segment and are surfaced through the same gRPC and Prometheus
channels. No component writes to VPP except `maglevd`. No component
serves `maglevd`'s state except `maglevd` itself.

## Components

### maglevd

`maglevd` is the entire control plane. It is a single Go process
that bundles three internal concerns — a fleet of probe workers, a
VPP reconciler, and a gRPC server — around one shared, versioned
view of `(config, backend state, frontend state)`.

#### Responsibilities

- Load and validate configuration; accept reloads on `SIGHUP`
  (FR-5.3, FR-5.4).
- Run one health-check worker per backend defined in config
  (NFR-3.1).
- Maintain each backend's rise/fall counter and derive its state
  (FR-1.2, FR-1.6).
- Aggregate backend state into per-frontend state, honoring
  pool-based failover and per-backend operator overrides
  (FR-2.x, FR-3.x).
- Connect to VPP's binary API and stats socket, reconnecting
  automatically on disconnect (FR-4.4).
- Compute a desired VPP `lb` state from current configuration and
  health, and drive VPP to match it (FR-4.1, FR-4.2).
- Expose the whole picture through a gRPC service and a Prometheus
  `/metrics` endpoint (FR-6.x).

#### Probe Types and Intervals

Four probe types are supported (FR-1.1):

- **ICMP** — sends an echo request, expects a matching reply. This
  probe type MUST have access to a raw socket, which requires
  `CAP_NET_RAW` (NFR-4.1).
- **TCP** — establishes a TCP connection and immediately closes
  it. No payload is exchanged.
- **HTTP** — issues a request against a configured path, matches
  the response code against a configured numeric range, and
  optionally matches the response body against a regular
  expression.
- **HTTPS** — HTTP over TLS with configurable SNI and an option to
  skip certificate verification.

Each health check configures three candidate intervals (FR-1.3):
the nominal `interval`, an optional faster `fast-interval` used
while the counter is in its degraded zone, and an optional slower
`down-interval` used while the backend is fully down. If an
optional interval is not set, the nominal interval is used. Every
scheduled sleep receives bounded random jitter; this is the
mechanism that satisfies NFR-2.3.

Each probe also has a `timeout` (FR-1.4). The probe-level timeout
bounds a single attempt; the interval bounds the time between the
**start** of consecutive attempts, with the actual probe latency
deducted from the next sleep so that slow probes do not push the
schedule later and later.

If the configuration sets `healthchecker.netns`, every probe of
every type MUST run inside that Linux network namespace (FR-1.5).
Entering a netns requires `CAP_SYS_ADMIN`; without it, probes will
fail and the backend will go down. This is a deliberate deployment
choice, not a bug — see the security subsection below.

#### Rise/Fall State Machine

Each backend carries a single integer counter in the closed range
`[0, rise + fall − 1]`. A backend is considered **Up** when the
counter is at or above `rise`, and **Down** otherwise. A successful
probe increments the counter, saturating at the maximum; a failing
probe decrements it, saturating at zero. This is the HAProxy
hysteresis model adapted to a single scalar (FR-1.2).

Four additional states overlay the rise/fall logic:

- **Unknown** — the backend has not yet produced any probe result
  since `maglevd` started (or since it was re-added by a reload).
  An Unknown backend contributes zero effective weight and the
  transition to Up or Down is taken on the *first* result rather
  than after `rise` or `fall` consecutive results (FR-1.6). This
  asymmetric rule lets fresh daemons discover the world quickly
  while still requiring hysteresis for steady-state flaps.
- **Paused** — operator override (FR-3.1). The probe worker is
  stopped and the counter is frozen. Effective weight is zero in
  every pool and every frontend that references the backend, but
  existing flows are not torn down; this is a soft drain.
- **Disabled** — operator override (FR-3.2). The probe worker is
  stopped and effective weight is zero in every pool and every
  frontend that references the backend. Unlike Paused, Disabled
  causes existing flows to be torn down on the next VPP sync
  (FR-4.3).
- **Removed** — the backend was deleted by a configuration reload.
  Its final transition is emitted on the event stream and then
  all references are dropped.

Backends declared **static** (no `healthcheck` reference in
config, FR-1.7) bypass the rise/fall machinery entirely. They are
not probed, their counter is not maintained, and they enter Up on
startup via a single synthetic pass. They still participate in
pool-failover weight computation like any other backend and still
honor operator Pause and Disable overrides.

Operator overrides and operator weight mutations are held in
process memory only. They survive a `SIGHUP` reload (FR-3.4) but
do **not** survive a daemon restart (FR-3.5): when `maglevd`
starts, the YAML file is the sole source of truth, and any
earlier runtime mutation is gone. Operators who need durable
changes must commit them to the configuration file.

#### Aggregation to Frontend State

A frontend references one or more named pools. Each referenced
pool contains one or more backends with a per-reference configured
weight in `[0, 100]` (FR-2.1). The effective weight that `maglevd`
computes for a given `(frontend, pool, backend)` tuple is
(FR-2.3):

- The configured weight, if the backend is Up **and** the
  backend's pool is the active pool (see below).
- Zero in every other case.

The active pool is the first pool, in configuration order, that
contains at least one Up backend whose configured weight is
non-zero (FR-2.2). If no pool is active (e.g. all backends are
Down), every backend contributes zero weight and the frontend's
aggregate state is Down. A frontend with no backends at all, or
with every referenced backend still in Unknown, is itself Unknown.
A frontend with at least one non-zero effective weight is Up
(FR-2.4).

Whether effective weight zero also flushes existing flows depends
on the cause (FR-4.3):

- Up in a non-active pool: weight zero, **no** flush (standby
  pool).
- Down while `flush-on-down` is true: weight zero, flush.
- Disabled: weight zero, flush, always.
- Paused or Unknown: weight zero, no flush.

#### VPP Reconciliation

`maglevd` treats VPP's LB configuration as a desired-state
reconciliation target. The desired state is a pure function of
`(current config, current backend state)`; the observed state is
read back from VPP through the `lb` plugin's binary API. A sync
operation diffs the two and issues the minimal set of
`lb_vip_add_del`, `lb_as_add_del`, and `lb_as_set_weight` messages
to make them match.

Two triggers drive a sync:

1. **Event-driven, single VIP** (FR-4.1). When the health checker
   emits a backend transition, the reconciler recomputes desired
   state for every frontend that references that backend and
   syncs those VIPs. This is the primary path for convergence
   during incidents.
2. **Periodic, full** (FR-4.2). A background loop runs a full
   sync on a configurable interval (default thirty seconds).
   This is the safety net that closes gaps left by missed events,
   VPP restarts, or bugs in the event path.

For determinism (NFR-2.1), whenever a sync operation iterates
over ASes it does so in a total order defined by the numeric
representation of the AS address, with IPv4 addresses ordered
before IPv6. Two `maglevd` instances given the same input MUST
therefore issue the same `lb_as_add_del` sequence, which in turn
means VPP produces the same bucket-to-AS assignment regardless of
which instance is driving.

Operator mutations, event-driven syncs, and periodic full syncs
are serialized through a single mutex at the VPP-call boundary
(NFR-2.4); they never interleave.

#### Startup Warmup and Restart Neutrality

A naive sync loop would, on restart, immediately synthesize a
desired state in which every backend is Unknown, map every
backend through the effective-weight rules to zero, and push
"zero weight everywhere" into VPP before a single probe had
completed. The result would be a multi-second black hole on
every `maglevd` restart. NFR-1.2 forbids this, and the warmup
state machine is how it is enforced.

The warmup has three phases, keyed off two configurable delays
`startup-min-delay` (default five seconds) and `startup-max-delay`
(default thirty seconds):

1. **Hands-off.** From process start to `startup-min-delay`, the
   reconciler MUST NOT write anything to VPP at all. Event-driven
   syncs are suppressed; the periodic full sync is suppressed.
2. **Per-VIP release.** From `startup-min-delay` to
   `startup-max-delay`, a VIP becomes eligible for sync the moment
   every backend it references has produced at least one probe
   result (i.e. none are Unknown). Eligible VIPs are released
   individually so that healthy VIPs converge as fast as their
   slowest backend, without being held back by unrelated slow
   VIPs.
3. **Watchdog.** At `startup-max-delay`, any VIPs still held are
   released unconditionally by a final full sync. This bounds the
   worst-case blackout to `startup-max-delay` rather than "as long
   as the slowest backend takes".

The warmup clock is tied to process start, not to VPP reconnect
or configuration reload (NFR-1.3). Reconnecting to a flapping VPP
does not re-enter warmup, and `SIGHUP` does not re-enter warmup.

Setting both delays to zero disables the warmup entirely, which
is useful for tests but SHOULD NOT be done in production.

#### Configuration and Reload

Configuration lives in a single YAML file (FR-5.1), typically
`/etc/vpp-maglev/maglev.yaml`. It is validated in two distinct
phases (FR-5.2): a **parse** phase that catches YAML errors, and
a **semantic** phase that enforces structural invariants such as:

- Every frontend whose VIPs share an address MUST use backends of
  the same address family (IPv4 or IPv6), because VPP picks an
  encap type per VIP and mixing families on one VIP is not
  supported.
- Every backend referenced by a frontend MUST exist.
- Every referenced health check MUST exist.
- Every pool referenced by a frontend MUST contain at least one
  backend (FR-2.1).
- VPP LB knobs MUST satisfy plugin constraints: `flow-timeout`
  in `[1s, 120s]`, `sticky-buckets-per-core` a power of two,
  `sync-interval` strictly positive, `startup-max-delay` not less
  than `startup-min-delay`.
- `transition-history` MUST be at least one.

`maglevd --check` runs both phases and exits with code 0 on
success, 1 on parse errors, and 2 on semantic errors (NFR-5.2).
This exit code contract is what packaging scripts and systemd
`ExecStartPre` rely on.

On `SIGHUP` the same two-phase validation runs against the file
on disk. If either phase fails, `maglevd` MUST log the error and
leave the running configuration untouched (FR-5.4, NFR-1.4). On
success, the delta is applied atomically (NFR-2.2): new backends
spawn workers, removed backends have their workers stopped and
emit a terminal `Removed` event, changed backends restart their
workers, and metadata-only changes (address, weight, enable flag)
are updated in place without restarting anything. Operator
overrides (Pause, Disable) survive reloads (FR-3.4) but — to
repeat the point from FR-3.5 — do **not** survive a daemon
restart.

#### Lifecycle, Signals, and Security

`maglevd` handles three signals:

- **`SIGHUP`** triggers a configuration reload as described
  above.
- **`SIGTERM`** and **`SIGINT`** initiate a graceful shutdown:
  the gRPC server drains, stream subscribers are released, probe
  workers are cancelled, and the VPP connection is closed. VPP's
  last-programmed state is not torn down; traffic continues to
  flow (NFR-1.1).

`maglevd` requires two Linux capabilities, each tied to a
specific feature (NFR-4.1):

- **`CAP_NET_RAW`** is required if and only if any configured
  health check is of type ICMP. Without it, raw-socket creation
  will fail and all ICMP probes will error out.
- **`CAP_SYS_ADMIN`** is required if and only if
  `healthchecker.netns` is set. The kernel's `setns(CLONE_NEWNET)`
  call requires it; without it, every probe will fail on
  namespace entry.

The shipped Debian unit grants both capabilities through
`AmbientCapabilities` and `CapabilityBoundingSet`, which is why
the package "just works" out of the box. Hand-run invocations
SHOULD set capabilities explicitly (e.g. via `setcap`) rather
than running as root.

`maglevd` does not secure its own gRPC listener (NFR-4.2).
Operators SHOULD bind the listener to loopback, to a
control-plane VRF, or behind a firewall, depending on their
threat model. The design deliberately pushes transport security
out of the binary on the theory that every deployment already
has an answer for it.

#### Interfaces

**Presents.**

- **A gRPC service on a TCP listener** (default `:9090`). This
  is the *only* programmatic interface to `maglevd`. Every other
  component talks to `maglevd` through this interface and no
  other. The service has read-only methods (`List*`, `Get*`,
  `WatchEvents`, `CheckConfig`), mutating methods
  (`PauseBackend`, `ResumeBackend`, `EnableBackend`,
  `DisableBackend`, `SetFrontendPoolBackendWeight`,
  `ReloadConfig`, `SyncVPPLBState`), and a single streaming
  method (`WatchEvents`) that multiplexes log entries and state
  transitions to any number of subscribers with per-subscriber
  filters (FR-6.3). gRPC reflection is enabled by default so
  that ad-hoc tooling can introspect the service.
- **A Prometheus `/metrics` HTTP endpoint** on a separate
  listener (default `:9091`) (FR-6.2). Counters are updated
  inline as probes run and VPP calls complete; gauges are
  computed on each scrape from the current checker and VPP
  state, so there is no sampling lag.
- **Structured JSON logs on stdout**, via `log/slog`, at a
  configurable level (FR-6.1). Key events — daemon start, config
  load, VPP connect/disconnect, backend transitions, LB sync
  mutations, warmup milestones — are logged at `info` or higher
  so that a default-level deployment has enough to post-mortem
  an incident.
- **Process exit codes** from `--check`: 0, 1, or 2 as described
  above (NFR-5.2). These form a small but load-bearing interface
  to packaging and systemd.

**Consumes.**

- **A YAML configuration file** on disk, passed via `--config`
  or `MAGLEV_CONFIG`. This is the declarative source of truth
  for intent; everything the operator mutates at runtime is a
  delta on top of it, and every runtime delta is lost on a
  daemon restart (FR-3.5).
- **VPP's binary API socket** (default `/run/vpp/api.sock`).
  The connection auto-reconnects on drop (FR-4.4), and while
  disconnected, the reconciler silently queues no work — the
  next periodic sync closes any gap.
- **VPP's stats segment socket** (default `/run/vpp/stats.sock`).
  Read periodically (five-second cadence) for per-VIP packet
  and byte counters (FR-6.4). Readers are non-blocking
  (NFR-3.3); a stale snapshot is always available.
- **The Linux kernel's namespace subsystem**, when
  `healthchecker.netns` is set. Requires `CAP_SYS_ADMIN`.
- **Raw sockets**, for ICMP probes. Requires `CAP_NET_RAW`.

### VPP Dataplane

The VPP dataplane is not part of the `vpp-maglev` codebase, but
it is the component every other piece revolves around, and its
contract with `maglevd` defines what `maglevd` is allowed to do.

#### Responsibilities

VPP's `lb` plugin implements Maglev consistent hashing in the
forwarding fast path. It owns:

- **Global configuration** — an IPv4 source address and an IPv6
  source address used as the outer header for GRE-encapsulated
  traffic to ASes, the number of sticky buckets per worker core,
  and a per-flow idle timeout.
- **A set of VIPs**, each identified by an address prefix, an IP
  protocol, and a port. A VIP carries an encap type (GRE4 or
  GRE6, picked by the family of the AS addresses) and a flag
  for source-IP sticky hashing.
- **A set of ASes per VIP**, each identified by address, with an
  integer weight in `[0, 100]`, a `used`/`flushed` state, and a
  bucket count derived from the Maglev ring.

It does **not** own: health, configuration intent, operator
overrides, transition history, or metrics. Those belong to
`maglevd`.

#### Interfaces

**Presents.**

- **A binary API** (GoVPP-style message exchange) for reading
  and mutating VIP and AS state. `maglevd` is the sole user.
- **A stats segment** with per-VIP counters from the LB plugin
  (existing-flow, first-flow, untracked, no-server) and
  per-prefix FIB counters. The LB plugin bypasses the FIB for
  forwarded packets, so per-backend traffic counters are not
  available; this is a known limitation that operators consuming
  metrics need to understand.
- **The forwarded-traffic fast path itself**, which is the whole
  reason this project exists.

**Consumes.**

- `maglevd`'s binary-API writes — nothing else. There is no
  third party programming `lb` state in a working deployment.

### maglevc

`maglevc` is the interactive and scripting CLI. It is a
short-lived client with no persistent state and no background
work (NFR-5.4).

#### Responsibilities

- Provide a human-readable tab-completing shell for `maglevd`
  (FR-7.1).
- Dispatch one-shot commands for scripts and automation.
- Render state snapshots (frontends, backends, health checks,
  VPP LB state, VPP counters) with optional ANSI color.
- Stream events in real time (`watch events`) with filters.

#### Interaction Model

With no positional arguments, `maglevc` starts a readline-based
REPL with a nested command tree: `show`, `set`, `watch`,
`config`, plus the usual `help`, `exit`, `quit`. Tab completion
is built from the same command tree the dispatcher uses, so
completion can never drift from the actual command set. With
positional arguments, `maglevc` executes one command against the
server and exits — in this mode color is off by default so that
pipes and logs stay clean, but `--color=true` can be set
explicitly.

#### Interfaces

**Presents.**

- **An interactive TTY shell** and a **one-shot command mode**.
  Humans and scripts are the only consumers; there is no API,
  no socket, no file output.

**Consumes.**

- **`maglevd`'s gRPC service**, over insecure credentials by
  default. `maglevc` MUST NOT talk to VPP directly, MUST NOT
  read the config file directly, and MUST NOT maintain any
  state of its own across invocations (NFR-5.4). Everything it
  shows and everything it mutates goes through the gRPC API.

### maglevd-frontend

`maglevd-frontend` is an optional web dashboard (FR-7.2). Unlike
`maglevc`, it is a long-running process: it holds open gRPC
streams, caches snapshots, and serves HTTP.

#### Responsibilities

- Connect to one or more `maglevd` servers simultaneously.
- Maintain a cached view of each server's state: frontends,
  backends, health checks, VPP LB state, and VPP counters.
- Serve a SolidJS single-page application and a JSON API to
  browsers.
- Stream live updates to browsers so that dashboards update
  without polling (NFR-5.3).
- Expose an optional authenticated mutation surface (FR-7.3).

#### Multi-Server Multiplexing

A single `maglevd-frontend` process accepts a comma-separated
list of gRPC server addresses. For each one, it runs an
independent pool of goroutines: one to stream events, one to
refresh list-oriented data on a roughly one-second cadence, one
to refresh per-health-check detail, and one (debounced on
incoming events) to refresh VPP LB state and counters. Failures
on one server MUST NOT block the others, and the served JSON
state always reports per-server connection status so that the
SPA can mark partially-available views.

All per-server event streams publish into a single shared event
broker with a bounded replay buffer (capped both in time and in
event count, satisfying NFR-3.2). The broker assigns each event
a monotonic `epoch-seq` identifier so that browsers reconnecting
a dropped Server-Sent-Events stream can resume from where they
left off without a full refresh — and so that a broker restart,
which reshuffles the epoch, forces a full refresh rather than
silently handing out ambiguous IDs.

#### Read-Only and Admin Surfaces

The HTTP surface is partitioned into two paths (FR-7.3):

- **`/view/`** serves the SPA and a read-only JSON API. It is
  always publicly accessible: there is no auth, and there are
  no mutation endpoints under it at all. The design intent is
  that `/view/` can be exposed to a broader audience (NOC,
  management UIs, screens on walls) without risk.
- **`/admin/`** serves the SPA entry point and the mutating
  JSON API behind HTTP basic auth. Credentials come from
  `MAGLEV_FRONTEND_USER` and `MAGLEV_FRONTEND_PASSWORD`. If
  either is unset or empty, the `/admin/` path MUST return 404
  (NFR-4.3) — the admin surface is not merely locked, it is
  not advertised. This makes accidental exposure self-limiting:
  forgetting to set the env vars disables admin rather than
  leaving it open.

Both surfaces talk to the same underlying cache; the difference
is only what endpoints exist.

#### Interfaces

**Presents.**

- **An HTTP listener** (default `:8080`) serving:
  - `/view/` — the SolidJS SPA (embedded in the binary).
  - `/view/api/*` — read-only JSON endpoints for version,
    server list, aggregated state, and per-server state.
  - `/view/api/events` — an SSE stream bridged from the
    internal event broker, with `Last-Event-ID` replay.
  - `/admin/` — the SPA entry point, gated on basic auth.
  - `/admin/api/*` — mutating JSON endpoints that translate
    to gRPC mutations against the appropriate `maglevd`.
  - `/healthz` — a liveness probe.

**Consumes.**

- **One or more `maglevd` gRPC services.** As with `maglevc`,
  this is the *only* way `maglevd-frontend` reaches into the
  system. It MUST NOT read the YAML config file and MUST NOT
  talk to VPP directly (NFR-5.4).
- **Two environment variables**, `MAGLEV_FRONTEND_USER` and
  `MAGLEV_FRONTEND_PASSWORD`, for the optional admin surface.

### maglevt

`maglevt` is a small out-of-band probe TUI (FR-7.4). It is not
part of the control loop at all; it is a validation tool that
an operator runs on a laptop, a jump host, or a monitoring box
to see VIPs the way a client sees them.

#### Responsibilities

- Read one or more `maglev.yaml` files and enumerate TCP-style
  VIPs from the `frontends` section.
- Probe each VIP at a configurable interval with a real HTTP or
  HTTPS request against a configurable path.
- Measure latency (min/max/average and a handful of
  percentiles) and success rate over a rolling window.
- Tally the value of a configurable response header (by
  default, `X-IPng-Frontend`) so that operators can see which
  backend actually served each request. Because keep-alives are
  disabled by default, this tally reflects fresh Maglev hashing
  decisions rather than a pinned connection.

#### Scope Boundary

`maglevt` is intentionally decoupled from `maglevd`. It does
not talk gRPC, it does not read the VPP stats segment, and it
does not know or care whether the target VIPs are actually
served by the `vpp-maglev` control plane at all — it simply
probes addresses. This makes it useful in at least three
scenarios: validating a `maglevd` restart end-to-end from a
client perspective, debugging pool failover by watching the
header tally reshuffle, and sanity-checking that a given VIP is
reachable across deployments when the gRPC control plane is
unavailable or out of reach.

#### Interfaces

**Presents.**

- **A full-screen TUI** built on Bubble Tea, with a
  deterministic grid layout and a few interactive toggles (e.g.
  reverse-DNS lookup). There is no machine-readable output; if
  you need metrics, use Prometheus on `maglevd`.

**Consumes.**

- **One or more YAML configuration files**, which it parses
  with the same library `maglevd` uses. Only the subset of the
  schema describing frontends is actually consumed; unknown
  fields are ignored. Duplicate VIPs discovered across files
  are de-duplicated by `(scheme, address, port)` so that
  multi-file deployments don't double-probe.
- **The outbound network**, directly. No special capabilities
  are required — `maglevt` is a plain HTTP client.

## Operational Concerns

### Configuration Reload Semantics

Reload is triggered by `SIGHUP` to `maglevd`, or by the
`ReloadConfig` gRPC method. Both paths run the same validation
as `--check`. A reload MUST NOT partially apply (NFR-2.2):
either every change in the new file takes effect, or none of
them do. A reload MUST NOT restart unchanged probe workers; the
probe state machine is preserved precisely because operators
use reloads as a routine operation and expect backends whose
health-check definitions did not change to simply keep running.

Operator overrides (Pause, Disable) survive a reload as long as
the backend still exists in the new config (FR-3.4). A backend
that disappears from the new config transitions to `Removed`
and its worker is stopped; if it reappears in a later reload it
starts again in `Unknown` with a fresh counter.

A daemon **restart** is different from a reload. On restart,
the YAML configuration is the sole source of truth: every
runtime override is gone, every runtime weight mutation is gone
(FR-3.5). Operators who need an override to persist across
restarts must commit the intended state to the config file.

### Failure Modes

- **VPP restart.** `maglevd` detects the disconnect, enters a
  reconnect loop, and on reconnect reads VPP's version and
  current state (FR-4.4). The warmup clock is not reset by VPP
  reconnects (NFR-1.3) — a flapping VPP does not cause
  `maglevd` to go hands-off every time. The next periodic full
  sync pushes the current desired state into the freshly
  restarted plugin.
- **`maglevd` restart with VPP up.** Handled by the warmup
  state machine (NFR-1.2): new flows see the last-programmed
  weights until probes catch up, not zeros.
- **`maglevd` restart with VPP also down.** VPP comes back
  first, `maglevd` comes back second, warmup gates pushing
  anything until probes converge. This is the worst-case path,
  bounded by `startup-max-delay`.
- **Configuration reload with a broken file.** The reload is
  rejected; the running configuration is retained; an error
  is logged (FR-5.4). No probes are interrupted (NFR-1.4).
- **Probe namespace disappears.** Entering the namespace fails,
  the probe is counted as a failure, and the backend
  eventually transitions Down under normal rise/fall rules.
  There is no special-case handling; this is by design, because
  an operator removing the netns while `maglevd` is running is
  an operational error that SHOULD manifest as a visible Down,
  not as silent success.
- **gRPC subscriber too slow.** Per-subscriber event queues
  are bounded (NFR-3.2). A subscriber that cannot keep up MUST
  be dropped rather than backing up the central fan-out
  (NFR-3.4).
- **Mid-flight weight mutation during sync.** Operator weight
  changes and reconciler sync both route through the same
  state-protected code path, so mutations are serialized rather
  than interleaved with VPP writes (NFR-2.4).

### Observability

**Structured logging** (FR-6.1). All logs are slog-formatted
JSON written to stdout. The default level is `info`, which is
sized to produce one or two lines per incident rather than per
probe. The `debug` level dumps every probe attempt and every
VPP binary-API message, and is intended for post-mortem
investigation.

**Prometheus metrics** (FR-6.2, FR-6.4). `maglevd` exposes four
classes of metric: inline counters for probe outcomes,
probe-latency histograms, backend state-transition counters,
and VPP API and LB sync counters; and on-demand gauges for
current backend state, rise/fall counter values, configured
weights, VPP connection status, VPP uptime, VPP info labels,
and per-VIP LB plugin counters. Gauges are sampled from live
state on every scrape, so there is no sampling staleness.

**Streaming events** (FR-6.3). The gRPC `WatchEvents` method
multiplexes three event families into one stream: log events
(the same structured logs the daemon writes to stdout), backend
transitions (one per affected frontend, since a single backend
may participate in multiple frontends), and frontend aggregate
transitions (Up/Down/Unknown flips at the frontend level).
Clients MAY filter by event family and by minimum log level.
The web frontend consumes this stream and re-publishes it to
browsers over SSE, with an epoch-seq replay buffer layered on
top.

### Security and Capabilities

`maglevd` needs `CAP_NET_RAW` for ICMP probes and
`CAP_SYS_ADMIN` for netns entry (NFR-4.1). Neither is optional
for the feature that needs it, and neither is required
otherwise; operators who use neither feature MAY run `maglevd`
as an unprivileged user with no capabilities at all.

`maglevd-frontend` needs no special capabilities — it is a
plain HTTP client of `maglevd` and a plain HTTP server for
browsers. It does handle user credentials (basic auth), which
are read from the environment and held in process memory;
operators SHOULD terminate the frontend behind a TLS reverse
proxy if it is exposed beyond a trusted network.

`maglevc` and `maglevt` need no special capabilities.

All gRPC traffic runs insecure by default (NFR-4.2). Securing
transport is an operational decision, not a build-time one;
deployments that require mTLS SHOULD terminate gRPC at a
sidecar or colocate control and data plane on a trusted
segment.

### Concurrency Model

The concurrency model inside `maglevd` is deliberately simple
and local:

- Each backend owns exactly one probe worker goroutine
  (NFR-3.1). Workers do not share state with each other.
- All events — transitions and log records — travel through a
  single central channel which is then fanned out to bounded
  per-subscriber queues (NFR-3.2). The fan-out is the only
  place where multiple subscribers can observe the same event.
- The configuration pointer is swapped atomically on reload
  (NFR-2.2); readers take a read lock for the duration of a
  single access, so the live config is always internally
  consistent even mid-reload.
- The VPP stats snapshot is published as an atomic pointer
  (NFR-3.3), so Prometheus scrapes and gRPC reads of counters
  are wait-free.
- Reconciliation holds a mutex around VPP calls, which
  serializes operator mutations, event-driven syncs, and
  periodic full syncs against each other (NFR-2.4). This is
  intentional: the order in which VPP sees mutations matters
  for determinism, and serializing them is cheap at the scale
  of control-plane events.

Deadlock avoidance is structural rather than audited:
dependencies between subsystems are one-way. The checker does
not call into VPP; the reconciler reads checker state and calls
VPP; VPP never calls back. `maglevd-frontend` and `maglevc`
only read from `maglevd` over gRPC. There is no cycle in the
wait-for graph.

## Alternatives Considered

This is a retrofit of a shipped system, so the alternatives
here are the ones the code actively rejects, not speculative
designs.

- **Several probe schedulers sharing one goroutine pool.**
  Rejected in favor of one goroutine per backend. The
  per-backend model is trivially correct, has no shared state,
  and scales linearly with backend count at a cost of a few
  kilobytes per backend.
- **`maglevd-frontend` as a sidecar per `maglevd`.** Rejected
  in favor of one frontend speaking to many daemons. A single
  dashboard pane across a fleet is the common operator
  request; pushing multi-server logic into the frontend keeps
  the daemon simple.
- **Operator actions expressed as config edits plus SIGHUP.**
  Rejected in favor of direct gRPC mutations. Pausing a
  backend during an incident should not require editing a
  file, and the effect should survive subsequent reloads
  (FR-3.4) — though, by deliberate design, not a daemon
  restart (FR-3.5).
- **Persisting operator overrides across daemon restarts.**
  Rejected in favor of making the YAML config file the sole
  source of truth on startup (FR-3.5). Persisting runtime
  overrides would require an on-disk side store and a clear
  policy for what happens when the side store and the config
  file disagree; keeping the daemon stateless on startup is
  simpler and harder to get wrong.
- **Synchronous full sync after every transition.** Rejected
  in favor of event-driven single-VIP syncs with a periodic
  full sync as a safety net (FR-4.1, FR-4.2). Full syncs are
  cheap but not free, and the blast radius of a transient bug
  in the desired-state computation is smaller when
  per-transition work only touches one VIP.
- **Letting `maglevt` read `maglevd`'s gRPC.** Rejected in
  favor of probing the YAML file directly so that `maglevt`
  remains useful when `maglevd` itself is the thing being
  investigated.

## Open Questions

- **Mutual TLS for gRPC.** Currently insecure by default. A
  future version may wire in standard mTLS support once a
  credential-management story is picked.
- **Per-AS traffic counters.** The VPP `lb` plugin bypasses
  the FIB and therefore does not produce per-AS traffic
  counters. Surfacing real per-backend byte/packet counts
  would require a VPP-side change.
- **High-availability of the control plane.** Two `maglevd`
  instances on the same VPP would interleave writes harmlessly
  thanks to determinism (NFR-2.1), but there is no leader
  election and no formal story about which instance owns which
  VIPs. Today, operators run a single `maglevd` per VPP host.