vpp-maglev/docs/design.md

vpp-maglev Design Document

Metadata

Status	Retrofit — describes shipped behavior as of v1.0.0
Author	Pim van Pelt <pim@ipng.ch>
Last updated	2026-04-15
Audience	Operators and contributors who will read the source tree next

The key words MUST, MUST NOT, SHOULD, SHOULD NOT, and MAY are used as described in RFC 2119, and are reserved in this document for requirements that are actually enforced in code or by an external dependency. Plain-language descriptions of what the system or an operator can do are written in lowercase — "can", "will", "does" — and should not be read as normative.

Summary

vpp-maglev is a control plane for the VPP lb (Maglev load balancer) plugin. A single daemon — maglevd — probes a fleet of backends, maintains an authoritative view of their health, and programs the VPP dataplane so that traffic hashed to a given VIP lands only on healthy backends. Operators drive the system through maglevc (an interactive CLI) or maglevd-frontend (a read-only web dashboard with an optional authenticated admin surface). A small companion binary, maglevt, validates VIPs from outside the control plane by sending live HTTP probes and reporting failover behavior.

Background

VPP's lb plugin implements Maglev consistent hashing inside the dataplane: a VIP is backed by a pool of Application Servers (ASes), each with an integer weight in [0, 100], and incoming flows are hashed onto a bucket ring so that weight changes disturb as few existing flows as possible. The plugin knows nothing about backend health; if an AS dies while it holds buckets, traffic to those buckets is black-holed until something external tells lb to remove or re-weight the AS.

vpp-maglev is that external thing. Before vpp-maglev, operators maintained VIP configurations by hand and reacted to incidents with vppctl. The project replaces that loop with a daemon that owns the health story, reconciles it with the dataplane, and exposes the result through a uniform gRPC API so that CLIs, dashboards, and scripts all read the same source of truth.

Goals and Non-Goals

Product Goals

1. Accurate backend health. Detect that a backend is up, degraded, or down quickly enough to keep user-visible error rates low, and avoid flapping under transient faults.
2. Correct VPP state. The set of VIPs and per-AS weights in VPP converges to the configured intent, filtered by current health, for every supported failure mode.
3. Restart neutrality. Restarting maglevd with VPP already up MUST NOT cause traffic to be black-holed while health probes warm up.
4. Operator control. A human can pause, drain, or weight-shift a backend in seconds without editing config files.
5. Uniform observability. Every state transition, VPP API call, and probe result is emitted as a structured log, a Prometheus metric, or a streaming event — ideally all three.
6. One source of truth. Every other component (CLI, web frontend, scripts) reads maglevd through one typed interface. There is no secondary control plane.

Non-Goals

• vpp-maglev is not a VPP installer or packaging layer. It assumes VPP is already running with the lb plugin loaded.
• It does not implement its own dataplane fast path. All forwarding stays in VPP; maglevd only programs the plugin.
• It is not a generic service mesh. There is no L7 routing, cert issuance, service discovery, or east-west policy — only VIPs, pools, and backends.
• It is not a config store. Configuration is a YAML file on disk; the gRPC API can check and reload it but cannot author it.
• It does not secure its own transport. gRPC runs insecure by default; TLS, mTLS, or firewalls are the operator's responsibility.

Requirements

Each requirement carries a unique identifier (FR-X.Y or NFR-X.Y) so that later sections can cite it.

Functional Requirements

FR-1 Health checking

• FR-1.1 The system supports ICMP, TCP, HTTP, and HTTPS health checks, each with its own protocol-specific success criteria.
• FR-1.2 Each health check MUST apply HAProxy rise/fall semantics with operator-configurable thresholds.
• FR-1.3 A health check MAY declare distinct interval, fast-interval, and down-interval values so that recovery from a degraded or down state is faster than steady-state polling.
• FR-1.4 Each probe attempt is bounded by a configurable per-probe timeout, independent of the scheduling interval.
• FR-1.5 If the configuration sets healthchecker.netns, every probe MUST execute inside the named Linux network namespace.
• FR-1.6 The first probe result against a newly-created backend forces an immediate transition out of Unknown, without waiting for rise or fall consecutive results.
• FR-1.7 A backend MAY omit its healthcheck reference to declare itself static. A static backend is not probed and is treated as permanently Up; it still participates in pool failover and still honors operator Pause and Disable overrides.

FR-2 Aggregation and pool failover

• FR-2.1 A frontend MAY reference one or more named pools. Each referenced pool MUST contain at least one (backend, configured-weight) tuple; an empty pool is a configuration error and is rejected at load time.
• FR-2.2 At any time, exactly one pool — the first, in configuration order, that contains a healthy backend with non-zero configured weight — is active; backends in other pools contribute zero effective weight.
• FR-2.3 The effective weight of a (frontend, pool, backend) tuple is the configured weight when the backend is Up and the pool is active, and zero in every other case.
• FR-2.4 A frontend's aggregate state is Up when at least one backend has non-zero effective weight, Unknown when every referenced backend is still Unknown (or the frontend references no backends), and Down otherwise.

FR-3 Operator control

• FR-3.1 Operators can pause and resume individual backends at runtime. Pausing stops the probe worker, freezes the rise/fall counter, and drives effective weight to zero in every pool and every frontend that references the backend. Existing flows are not torn down; this is a soft drain.
• FR-3.2 Operators can disable and re-enable individual backends at runtime. Disabling drives effective weight to zero in every pool and every frontend that references the backend, and MUST cause existing flows to be torn down on the next VPP sync.
• FR-3.3 Operators can set the configured weight of a specific (frontend, pool, backend) tuple at runtime.
• FR-3.4 Operator overrides (Pause, Disable) and operator weight mutations survive a configuration reload (SIGHUP) as long as the underlying backend and tuple still exist in the new configuration.
• FR-3.5 Operator overrides and operator weight mutations do not survive a maglevd restart. After a restart, the YAML configuration file is authoritative for every backend and every tuple: paused backends come back unpaused, disabled backends come back enabled, mutated weights revert to the configured value. Operators who need persistent changes must edit the config file.

FR-4 VPP reconciliation

• FR-4.1 For every backend state transition that changes an effective weight, maglevd pushes the resulting AS state into VPP for every affected VIP.
• FR-4.2 maglevd runs a periodic full reconciliation on a configurable cadence (default thirty seconds) as a safety net against missed events and VPP restarts.
• FR-4.3 Weight-to-zero is communicated to VPP as a graceful drain by default; transitions to Disabled and transitions to Down while flush-on-down is true MUST tear existing flows down on the next sync.
• FR-4.4 maglevd tolerates VPP disconnects by auto-reconnecting and resuming reconciliation once the connection is re-established.

FR-5 Configuration

• FR-5.1 Configuration is loaded from a single YAML file specified at startup and referenced by all later operations.
• FR-5.2 Configuration validation distinguishes parse errors (malformed YAML) from semantic errors (structural invariants) and MUST report each with its own exit code from --check: 0 (OK), 1 (parse), 2 (semantic).
• FR-5.3 maglevd reloads its configuration on SIGHUP without restarting the process, without restarting unchanged probe workers, and without losing operator overrides (see FR-3.4).
• FR-5.4 A parse or semantic error encountered during reload MUST leave the running configuration in place.
• FR-5.5 The same validation and reload semantics are also reachable through gRPC (CheckConfig, ReloadConfig).

FR-6 Observability

• FR-6.1 All logs are emitted as structured JSON on stdout at a configurable level.
• FR-6.2 maglevd exposes Prometheus metrics for probe outcomes, probe latency, backend state transitions, VPP API traffic, and VPP LB sync mutations.
• FR-6.3 A streaming gRPC API multiplexes log entries, backend transitions, and frontend aggregate transitions to any number of subscribers with per-subscriber filters.
• FR-6.4 Per-VIP packet counters from VPP's stats segment are surfaced through both the gRPC API and the Prometheus surface.

FR-7 Clients and peripheral tools

• FR-7.1 An interactive CLI (maglevc) provides a tab-completing shell and a one-shot command mode, both backed by the same command tree.
• FR-7.2 A web frontend (maglevd-frontend) can multiplex more than one maglevd in a single process and present their combined state.
• FR-7.3 The web frontend partitions its HTTP surface into a public read-only path (/view/) and an authenticated mutating path (/admin/). If credentials are not configured, /admin/ MUST NOT be advertised (the path returns 404).
• FR-7.4 An out-of-band tester (maglevt) probes configured VIPs from outside the control plane, measures latency, and tallies a configurable response header.

Non-Functional Requirements

NFR-1 Availability and reliability

• NFR-1.1 A maglevd outage MUST NOT stop the dataplane. While maglevd is absent, VPP continues to forward traffic with its last-programmed state.
• NFR-1.2 Restarting maglevd with VPP up MUST NOT black-hole new flows during the probe warm-up window; this is enforced by the startup warmup state machine described under maglevd.
• NFR-1.3 The warmup clock is tied to process start and MUST NOT be reset by VPP reconnects or configuration reloads.
• NFR-1.4 A maglevd-side reload with a broken file MUST NOT interrupt any running probe.

NFR-2 Determinism and correctness

• NFR-2.1 Two maglevd instances given the same configuration and the same backend state MUST issue the same sequence of lb_as_add_del calls to VPP, so that VPP's bucket assignment is stable across process swaps. This is the job of the deterministic AS ordering rule.
• NFR-2.2 Configuration reload MUST be atomic: either every change in the new file takes effect, or none of them do.
• NFR-2.3 Probe scheduling SHOULD apply bounded jitter so that, after a daemon restart or a configuration reload, probes do not phase-lock to the wall clock.
• NFR-2.4 Operator mutations, event-driven syncs, and periodic full syncs against VPP MUST be serialized with respect to one another; they MUST NOT interleave.

NFR-3 Performance and scalability

• NFR-3.1 Probing N backends costs roughly N goroutines doing mostly idle waits; there is no central probe scheduler.
• NFR-3.2 Event fan-out, transition history, and per-subscriber event queues MUST all be bounded; no structure grows without limit under sustained load.
• NFR-3.3 VPP stats snapshots are published as an atomic pointer so that Prometheus scrapes and gRPC counter reads are wait-free.
• NFR-3.4 A gRPC subscriber that cannot keep up MUST be dropped rather than blocking the central fan-out.

NFR-4 Security

• NFR-4.1 maglevd runs with only the Linux capabilities it actually needs: CAP_NET_RAW only when ICMP probes are in use, CAP_SYS_ADMIN only when healthchecker.netns is set.
• NFR-4.2 gRPC transport security is explicitly out of scope; the daemon runs insecure by default and deployments SHOULD front it with a firewall, a trusted network, or a TLS-terminating sidecar.
• NFR-4.3 The web frontend's mutating surface MUST be hidden entirely (HTTP 404) when either of its basic-auth environment variables is unset.

NFR-5 Operability

• NFR-5.1 Every CLI flag on every binary SHOULD have an environment-variable equivalent so that the binaries can be driven purely through env in container deployments.
• NFR-5.2 maglevd --check MUST provide a stable exit-code contract (0 / 1 / 2) for use by packaging scripts and ExecStartPre handlers.
• NFR-5.3 Dashboards can track state in real time through the streaming event interface rather than by tight polling.
• NFR-5.4 maglevc and maglevd-frontend MUST NOT maintain any authoritative state of their own; all truth lives in maglevd.

Architecture Overview

Process Model

The system ships as three independent executables plus one optional companion tester:

• maglevd — the long-running daemon. Hosts both the health checker and the VPP control plane.
• maglevc — short-lived CLI client.
• maglevd-frontend — long-running web dashboard (optional).
• maglevt — short-lived out-of-band probe TUI (optional).

VPP itself is a fourth moving part, but it is an external dependency, not part of the vpp-maglev codebase.

Data Flow

Configuration flows in from a YAML file on disk (read by maglevd) and from runtime mutations issued over gRPC by maglevc or maglevd-frontend. Health state flows out of maglevd in three directions: into VPP (as AS weight changes), into Prometheus (as metrics), and into gRPC clients (as streaming events and snapshot reads). Traffic counters flow back in from VPP's stats segment and are surfaced through the same gRPC and Prometheus channels. No component writes to VPP except maglevd. No component serves maglevd's state except maglevd itself.

Components

maglevd

maglevd is the entire control plane. It is a single Go process that bundles three internal concerns — a fleet of probe workers, a VPP reconciler, and a gRPC server — around one shared, versioned view of (config, backend state, frontend state).

Responsibilities

• Load and validate configuration; accept reloads on SIGHUP (FR-5.3, FR-5.4).
• Run one health-check worker per backend defined in config (NFR-3.1).
• Maintain each backend's rise/fall counter and derive its state (FR-1.2, FR-1.6).
• Aggregate backend state into per-frontend state, honoring pool-based failover and per-backend operator overrides (FR-2.x, FR-3.x).
• Connect to VPP's binary API and stats socket, reconnecting automatically on disconnect (FR-4.4).
• Compute a desired VPP lb state from current configuration and health, and drive VPP to match it (FR-4.1, FR-4.2).
• Expose the whole picture through a gRPC service and a Prometheus /metrics endpoint (FR-6.x).

Probe Types and Intervals

Four probe types are supported (FR-1.1):

• ICMP — sends an echo request, expects a matching reply. This probe type MUST have access to a raw socket, which requires CAP_NET_RAW (NFR-4.1).
• TCP — establishes a TCP connection and immediately closes it. No payload is exchanged.
• HTTP — issues a request against a configured path, matches the response code against a configured numeric range, and optionally matches the response body against a regular expression.
• HTTPS — HTTP over TLS with configurable SNI and an option to skip certificate verification.

Each health check configures three candidate intervals (FR-1.3): the nominal interval, an optional faster fast-interval used while the counter is in its degraded zone, and an optional slower down-interval used while the backend is fully down. If an optional interval is not set, the nominal interval is used. Every scheduled sleep receives bounded random jitter; this is the mechanism that satisfies NFR-2.3.

Each probe also has a timeout (FR-1.4). The probe-level timeout bounds a single attempt; the interval bounds the time between the start of consecutive attempts, with the actual probe latency deducted from the next sleep so that slow probes do not push the schedule later and later.

If the configuration sets healthchecker.netns, every probe of every type MUST run inside that Linux network namespace (FR-1.5). Entering a netns requires CAP_SYS_ADMIN; without it, probes will fail and the backend will go down. This is a deliberate deployment choice, not a bug — see the security subsection below.

Rise/Fall State Machine

Each backend carries a single integer counter in the closed range [0, rise + fall − 1]. A backend is considered Up when the counter is at or above rise, and Down otherwise. A successful probe increments the counter, saturating at the maximum; a failing probe decrements it, saturating at zero. This is the HAProxy hysteresis model adapted to a single scalar (FR-1.2).

Four additional states overlay the rise/fall logic:

• Unknown — the backend has not yet produced any probe result since maglevd started (or since it was re-added by a reload). An Unknown backend contributes zero effective weight and the transition to Up or Down is taken on the first result rather than after rise or fall consecutive results (FR-1.6). This asymmetric rule lets fresh daemons discover the world quickly while still requiring hysteresis for steady-state flaps.
• Paused — operator override (FR-3.1). The probe worker is stopped and the counter is frozen. Effective weight is zero in every pool and every frontend that references the backend, but existing flows are not torn down; this is a soft drain.
• Disabled — operator override (FR-3.2). The probe worker is stopped and effective weight is zero in every pool and every frontend that references the backend. Unlike Paused, Disabled causes existing flows to be torn down on the next VPP sync (FR-4.3).
• Removed — the backend was deleted by a configuration reload. Its final transition is emitted on the event stream and then all references are dropped.

Backends declared static (no healthcheck reference in config, FR-1.7) bypass the rise/fall machinery entirely. They are not probed, their counter is not maintained, and they enter Up on startup via a single synthetic pass. They still participate in pool-failover weight computation like any other backend and still honor operator Pause and Disable overrides.

Operator overrides and operator weight mutations are held in process memory only. They survive a SIGHUP reload (FR-3.4) but do not survive a daemon restart (FR-3.5): when maglevd starts, the YAML file is the sole source of truth, and any earlier runtime mutation is gone. Operators who need durable changes must commit them to the configuration file.

Aggregation to Frontend State

A frontend references one or more named pools. Each referenced pool contains one or more backends with a per-reference configured weight in [0, 100] (FR-2.1). The effective weight that maglevd computes for a given (frontend, pool, backend) tuple is (FR-2.3):

• The configured weight, if the backend is Up and the backend's pool is the active pool (see below).
• Zero in every other case.

The active pool is the first pool, in configuration order, that contains at least one Up backend whose configured weight is non-zero (FR-2.2). If no pool is active (e.g. all backends are Down), every backend contributes zero weight and the frontend's aggregate state is Down. A frontend with no backends at all, or with every referenced backend still in Unknown, is itself Unknown. A frontend with at least one non-zero effective weight is Up (FR-2.4).

Whether effective weight zero also flushes existing flows depends on the cause (FR-4.3):

• Up in a non-active pool: weight zero, no flush (standby pool).
• Down while flush-on-down is true: weight zero, flush.
• Disabled: weight zero, flush, always.
• Paused or Unknown: weight zero, no flush.

VPP Reconciliation

maglevd treats VPP's LB configuration as a desired-state reconciliation target. The desired state is a pure function of (current config, current backend state); the observed state is read back from VPP through the lb plugin's binary API. A sync operation diffs the two and issues the minimal set of lb_vip_add_del, lb_as_add_del, and lb_as_set_weight messages to make them match.

Two triggers drive a sync:

1. Event-driven, single VIP (FR-4.1). When the health checker emits a backend transition, the reconciler recomputes desired state for every frontend that references that backend and syncs those VIPs. This is the primary path for convergence during incidents.
2. Periodic, full (FR-4.2). A background loop runs a full sync on a configurable interval (default thirty seconds). This is the safety net that closes gaps left by missed events, VPP restarts, or bugs in the event path.

For determinism (NFR-2.1), whenever a sync operation iterates over ASes it does so in a total order defined by the numeric representation of the AS address, with IPv4 addresses ordered before IPv6. Two maglevd instances given the same input MUST therefore issue the same lb_as_add_del sequence, which in turn means VPP produces the same bucket-to-AS assignment regardless of which instance is driving.

Operator mutations, event-driven syncs, and periodic full syncs are serialized through a single mutex at the VPP-call boundary (NFR-2.4); they never interleave.

Startup Warmup and Restart Neutrality

A naive sync loop would, on restart, immediately synthesize a desired state in which every backend is Unknown, map every backend through the effective-weight rules to zero, and push "zero weight everywhere" into VPP before a single probe had completed. The result would be a multi-second black hole on every maglevd restart. NFR-1.2 forbids this, and the warmup state machine is how it is enforced.

The warmup has three phases, keyed off two configurable delays startup-min-delay (default five seconds) and startup-max-delay (default thirty seconds):

1. Hands-off. From process start to startup-min-delay, the reconciler MUST NOT write anything to VPP at all. Event-driven syncs are suppressed; the periodic full sync is suppressed.
2. Per-VIP release. From startup-min-delay to startup-max-delay, a VIP becomes eligible for sync the moment every backend it references has produced at least one probe result (i.e. none are Unknown). Eligible VIPs are released individually so that healthy VIPs converge as fast as their slowest backend, without being held back by unrelated slow VIPs.
3. Watchdog. At startup-max-delay, any VIPs still held are released unconditionally by a final full sync. This bounds the worst-case blackout to startup-max-delay rather than "as long as the slowest backend takes".

The warmup clock is tied to process start, not to VPP reconnect or configuration reload (NFR-1.3). Reconnecting to a flapping VPP does not re-enter warmup, and SIGHUP does not re-enter warmup.

Setting both delays to zero disables the warmup entirely, which is useful for tests but SHOULD NOT be done in production.

Configuration and Reload

Configuration lives in a single YAML file (FR-5.1), typically /etc/vpp-maglev/maglev.yaml. It is validated in two distinct phases (FR-5.2): a parse phase that catches YAML errors, and a semantic phase that enforces structural invariants such as:

• Every frontend whose VIPs share an address MUST use backends of the same address family (IPv4 or IPv6), because VPP picks an encap type per VIP and mixing families on one VIP is not supported.
• Every backend referenced by a frontend MUST exist.
• Every referenced health check MUST exist.
• Every pool referenced by a frontend MUST contain at least one backend (FR-2.1).
• VPP LB knobs MUST satisfy plugin constraints: flow-timeout in [1s, 120s], sticky-buckets-per-core a power of two, sync-interval strictly positive, startup-max-delay not less than startup-min-delay.
• transition-history MUST be at least one.

maglevd --check runs both phases and exits with code 0 on success, 1 on parse errors, and 2 on semantic errors (NFR-5.2). This exit code contract is what packaging scripts and systemd ExecStartPre rely on.

On SIGHUP the same two-phase validation runs against the file on disk. If either phase fails, maglevd MUST log the error and leave the running configuration untouched (FR-5.4, NFR-1.4). On success, the delta is applied atomically (NFR-2.2): new backends spawn workers, removed backends have their workers stopped and emit a terminal Removed event, changed backends restart their workers, and metadata-only changes (address, weight, enable flag) are updated in place without restarting anything. Operator overrides (Pause, Disable) survive reloads (FR-3.4) but — to repeat the point from FR-3.5 — do not survive a daemon restart.

Lifecycle, Signals, and Security

maglevd handles three signals:

• SIGHUP triggers a configuration reload as described above.
• SIGTERM and SIGINT initiate a graceful shutdown: the gRPC server drains, stream subscribers are released, probe workers are cancelled, and the VPP connection is closed. VPP's last-programmed state is not torn down; traffic continues to flow (NFR-1.1).

maglevd requires two Linux capabilities, each tied to a specific feature (NFR-4.1):

• CAP_NET_RAW is required if and only if any configured health check is of type ICMP. Without it, raw-socket creation will fail and all ICMP probes will error out.
• CAP_SYS_ADMIN is required if and only if healthchecker.netns is set. The kernel's setns(CLONE_NEWNET) call requires it; without it, every probe will fail on namespace entry.

The shipped Debian unit grants both capabilities through AmbientCapabilities and CapabilityBoundingSet, which is why the package "just works" out of the box. Hand-run invocations SHOULD set capabilities explicitly (e.g. via setcap) rather than running as root.

maglevd does not secure its own gRPC listener (NFR-4.2). Operators SHOULD bind the listener to loopback, to a control-plane VRF, or behind a firewall, depending on their threat model. The design deliberately pushes transport security out of the binary on the theory that every deployment already has an answer for it.

Interfaces

Presents.

• A gRPC service on a TCP listener (default :9090). This is the only programmatic interface to maglevd. Every other component talks to maglevd through this interface and no other. The service has read-only methods (List*, Get*, WatchEvents, CheckConfig), mutating methods (PauseBackend, ResumeBackend, EnableBackend, DisableBackend, SetFrontendPoolBackendWeight, ReloadConfig, SyncVPPLBState), and a single streaming method (WatchEvents) that multiplexes log entries and state transitions to any number of subscribers with per-subscriber filters (FR-6.3). gRPC reflection is enabled by default so that ad-hoc tooling can introspect the service.
• A Prometheus /metrics HTTP endpoint on a separate listener (default :9091) (FR-6.2). Counters are updated inline as probes run and VPP calls complete; gauges are computed on each scrape from the current checker and VPP state, so there is no sampling lag.
• Structured JSON logs on stdout, via log/slog, at a configurable level (FR-6.1). Key events — daemon start, config load, VPP connect/disconnect, backend transitions, LB sync mutations, warmup milestones — are logged at info or higher so that a default-level deployment has enough to post-mortem an incident.
• Process exit codes from --check: 0, 1, or 2 as described above (NFR-5.2). These form a small but load-bearing interface to packaging and systemd.

Consumes.

• A YAML configuration file on disk, passed via --config or MAGLEV_CONFIG. This is the declarative source of truth for intent; everything the operator mutates at runtime is a delta on top of it, and every runtime delta is lost on a daemon restart (FR-3.5).
• VPP's binary API socket (default /run/vpp/api.sock). The connection auto-reconnects on drop (FR-4.4), and while disconnected, the reconciler silently queues no work — the next periodic sync closes any gap.
• VPP's stats segment socket (default /run/vpp/stats.sock). Read periodically (five-second cadence) for per-VIP packet and byte counters (FR-6.4). Readers are non-blocking (NFR-3.3); a stale snapshot is always available.
• The Linux kernel's namespace subsystem, when healthchecker.netns is set. Requires CAP_SYS_ADMIN.
• Raw sockets, for ICMP probes. Requires CAP_NET_RAW.

VPP Dataplane

The VPP dataplane is not part of the vpp-maglev codebase, but it is the component every other piece revolves around, and its contract with maglevd defines what maglevd is allowed to do.

Responsibilities

VPP's lb plugin implements Maglev consistent hashing in the forwarding fast path. It owns:

• Global configuration — an IPv4 source address and an IPv6 source address used as the outer header for GRE-encapsulated traffic to ASes, the number of sticky buckets per worker core, and a per-flow idle timeout.
• A set of VIPs, each identified by an address prefix, an IP protocol, and a port. A VIP carries an encap type (GRE4 or GRE6, picked by the family of the AS addresses) and a flag for source-IP sticky hashing.
• A set of ASes per VIP, each identified by address, with an integer weight in [0, 100], a used/flushed state, and a bucket count derived from the Maglev ring.

It does not own: health, configuration intent, operator overrides, transition history, or metrics. Those belong to maglevd.

Interfaces

Presents.

• A binary API (GoVPP-style message exchange) for reading and mutating VIP and AS state. maglevd is the sole user.
• A stats segment with per-VIP counters from the LB plugin (existing-flow, first-flow, untracked, no-server) and per-prefix FIB counters. The LB plugin bypasses the FIB for forwarded packets, so per-backend traffic counters are not available; this is a known limitation that operators consuming metrics need to understand.
• The forwarded-traffic fast path itself, which is the whole reason this project exists.

Consumes.

• maglevd's binary-API writes — nothing else. There is no third party programming lb state in a working deployment.

maglevc

maglevc is the interactive and scripting CLI. It is a short-lived client with no persistent state and no background work (NFR-5.4).

Responsibilities

• Provide a human-readable tab-completing shell for maglevd (FR-7.1).
• Dispatch one-shot commands for scripts and automation.
• Render state snapshots (frontends, backends, health checks, VPP LB state, VPP counters) with optional ANSI color.
• Stream events in real time (watch events) with filters.

Interaction Model

With no positional arguments, maglevc starts a readline-based REPL with a nested command tree: show, set, watch, config, plus the usual help, exit, quit. Tab completion is built from the same command tree the dispatcher uses, so completion can never drift from the actual command set. With positional arguments, maglevc executes one command against the server and exits — in this mode color is off by default so that pipes and logs stay clean, but --color=true can be set explicitly.

Interfaces

Presents.

• An interactive TTY shell and a one-shot command mode. Humans and scripts are the only consumers; there is no API, no socket, no file output.

Consumes.

• maglevd's gRPC service, over insecure credentials by default. maglevc MUST NOT talk to VPP directly, MUST NOT read the config file directly, and MUST NOT maintain any state of its own across invocations (NFR-5.4). Everything it shows and everything it mutates goes through the gRPC API.

maglevd-frontend

maglevd-frontend is an optional web dashboard (FR-7.2). Unlike maglevc, it is a long-running process: it holds open gRPC streams, caches snapshots, and serves HTTP.

Responsibilities

• Connect to one or more maglevd servers simultaneously.
• Maintain a cached view of each server's state: frontends, backends, health checks, VPP LB state, and VPP counters.
• Serve a SolidJS single-page application and a JSON API to browsers.
• Stream live updates to browsers so that dashboards update without polling (NFR-5.3).
• Expose an optional authenticated mutation surface (FR-7.3).

Multi-Server Multiplexing

A single maglevd-frontend process accepts a comma-separated list of gRPC server addresses. For each one, it runs an independent pool of goroutines: one to stream events, one to refresh list-oriented data on a roughly one-second cadence, one to refresh per-health-check detail, and one (debounced on incoming events) to refresh VPP LB state and counters. Failures on one server MUST NOT block the others, and the served JSON state always reports per-server connection status so that the SPA can mark partially-available views.

All per-server event streams publish into a single shared event broker with a bounded replay buffer (capped both in time and in event count, satisfying NFR-3.2). The broker assigns each event a monotonic epoch-seq identifier so that browsers reconnecting a dropped Server-Sent-Events stream can resume from where they left off without a full refresh — and so that a broker restart, which reshuffles the epoch, forces a full refresh rather than silently handing out ambiguous IDs.

Read-Only and Admin Surfaces

The HTTP surface is partitioned into two paths (FR-7.3):

• /view/ serves the SPA and a read-only JSON API. It is always publicly accessible: there is no auth, and there are no mutation endpoints under it at all. The design intent is that /view/ can be exposed to a broader audience (NOC, management UIs, screens on walls) without risk.
• /admin/ serves the SPA entry point and the mutating JSON API behind HTTP basic auth. Credentials come from MAGLEV_FRONTEND_USER and MAGLEV_FRONTEND_PASSWORD. If either is unset or empty, the /admin/ path MUST return 404 (NFR-4.3) — the admin surface is not merely locked, it is not advertised. This makes accidental exposure self-limiting: forgetting to set the env vars disables admin rather than leaving it open.

Both surfaces talk to the same underlying cache; the difference is only what endpoints exist.

Interfaces

Presents.

• An HTTP listener (default :8080) serving:
  • /view/ — the SolidJS SPA (embedded in the binary).
  • /view/api/* — read-only JSON endpoints for version, server list, aggregated state, and per-server state.
  • /view/api/events — an SSE stream bridged from the internal event broker, with Last-Event-ID replay.
  • /admin/ — the SPA entry point, gated on basic auth.
  • /admin/api/* — mutating JSON endpoints that translate to gRPC mutations against the appropriate maglevd.
  • /healthz — a liveness probe.

Consumes.

• One or more maglevd gRPC services. As with maglevc, this is the only way maglevd-frontend reaches into the system. It MUST NOT read the YAML config file and MUST NOT talk to VPP directly (NFR-5.4).
• Two environment variables, MAGLEV_FRONTEND_USER and MAGLEV_FRONTEND_PASSWORD, for the optional admin surface.

maglevt

maglevt is a small out-of-band probe TUI (FR-7.4). It is not part of the control loop at all; it is a validation tool that an operator runs on a laptop, a jump host, or a monitoring box to see VIPs the way a client sees them.

Responsibilities

• Read one or more maglev.yaml files and enumerate TCP-style VIPs from the frontends section.
• Probe each VIP at a configurable interval with a real HTTP or HTTPS request against a configurable path.
• Measure latency (min/max/average and a handful of percentiles) and success rate over a rolling window.
• Tally the value of a configurable response header (by default, X-IPng-Frontend) so that operators can see which backend actually served each request. Because keep-alives are disabled by default, this tally reflects fresh Maglev hashing decisions rather than a pinned connection.

Scope Boundary

maglevt is intentionally decoupled from maglevd. It does not talk gRPC, it does not read the VPP stats segment, and it does not know or care whether the target VIPs are actually served by the vpp-maglev control plane at all — it simply probes addresses. This makes it useful in at least three scenarios: validating a maglevd restart end-to-end from a client perspective, debugging pool failover by watching the header tally reshuffle, and sanity-checking that a given VIP is reachable across deployments when the gRPC control plane is unavailable or out of reach.

Interfaces

Presents.

• A full-screen TUI built on Bubble Tea, with a deterministic grid layout and a few interactive toggles (e.g. reverse-DNS lookup). There is no machine-readable output; if you need metrics, use Prometheus on maglevd.

Consumes.

• One or more YAML configuration files, which it parses with the same library maglevd uses. Only the subset of the schema describing frontends is actually consumed; unknown fields are ignored. Duplicate VIPs discovered across files are de-duplicated by (scheme, address, port) so that multi-file deployments don't double-probe.
• The outbound network, directly. No special capabilities are required — maglevt is a plain HTTP client.

Operational Concerns

Configuration Reload Semantics

Reload is triggered by SIGHUP to maglevd, or by the ReloadConfig gRPC method. Both paths run the same validation as --check. A reload MUST NOT partially apply (NFR-2.2): either every change in the new file takes effect, or none of them do. A reload MUST NOT restart unchanged probe workers; the probe state machine is preserved precisely because operators use reloads as a routine operation and expect backends whose health-check definitions did not change to simply keep running.

Operator overrides (Pause, Disable) survive a reload as long as the backend still exists in the new config (FR-3.4). A backend that disappears from the new config transitions to Removed and its worker is stopped; if it reappears in a later reload it starts again in Unknown with a fresh counter.

A daemon restart is different from a reload. On restart, the YAML configuration is the sole source of truth: every runtime override is gone, every runtime weight mutation is gone (FR-3.5). Operators who need an override to persist across restarts must commit the intended state to the config file.

Failure Modes

• VPP restart. maglevd detects the disconnect, enters a reconnect loop, and on reconnect reads VPP's version and current state (FR-4.4). The warmup clock is not reset by VPP reconnects (NFR-1.3) — a flapping VPP does not cause maglevd to go hands-off every time. The next periodic full sync pushes the current desired state into the freshly restarted plugin.
• maglevd restart with VPP up. Handled by the warmup state machine (NFR-1.2): new flows see the last-programmed weights until probes catch up, not zeros.
• maglevd restart with VPP also down. VPP comes back first, maglevd comes back second, warmup gates pushing anything until probes converge. This is the worst-case path, bounded by startup-max-delay.
• Configuration reload with a broken file. The reload is rejected; the running configuration is retained; an error is logged (FR-5.4). No probes are interrupted (NFR-1.4).
• Probe namespace disappears. Entering the namespace fails, the probe is counted as a failure, and the backend eventually transitions Down under normal rise/fall rules. There is no special-case handling; this is by design, because an operator removing the netns while maglevd is running is an operational error that SHOULD manifest as a visible Down, not as silent success.
• gRPC subscriber too slow. Per-subscriber event queues are bounded (NFR-3.2). A subscriber that cannot keep up MUST be dropped rather than backing up the central fan-out (NFR-3.4).
• Mid-flight weight mutation during sync. Operator weight changes and reconciler sync both route through the same state-protected code path, so mutations are serialized rather than interleaved with VPP writes (NFR-2.4).

Observability

Structured logging (FR-6.1). All logs are slog-formatted JSON written to stdout. The default level is info, which is sized to produce one or two lines per incident rather than per probe. The debug level dumps every probe attempt and every VPP binary-API message, and is intended for post-mortem investigation.

Prometheus metrics (FR-6.2, FR-6.4). maglevd exposes four classes of metric: inline counters for probe outcomes, probe-latency histograms, backend state-transition counters, and VPP API and LB sync counters; and on-demand gauges for current backend state, rise/fall counter values, configured weights, VPP connection status, VPP uptime, VPP info labels, and per-VIP LB plugin counters. Gauges are sampled from live state on every scrape, so there is no sampling staleness.

Streaming events (FR-6.3). The gRPC WatchEvents method multiplexes three event families into one stream: log events (the same structured logs the daemon writes to stdout), backend transitions (one per affected frontend, since a single backend may participate in multiple frontends), and frontend aggregate transitions (Up/Down/Unknown flips at the frontend level). Clients MAY filter by event family and by minimum log level. The web frontend consumes this stream and re-publishes it to browsers over SSE, with an epoch-seq replay buffer layered on top.

Security and Capabilities

maglevd needs CAP_NET_RAW for ICMP probes and CAP_SYS_ADMIN for netns entry (NFR-4.1). Neither is optional for the feature that needs it, and neither is required otherwise; operators who use neither feature MAY run maglevd as an unprivileged user with no capabilities at all.

maglevd-frontend needs no special capabilities — it is a plain HTTP client of maglevd and a plain HTTP server for browsers. It does handle user credentials (basic auth), which are read from the environment and held in process memory; operators SHOULD terminate the frontend behind a TLS reverse proxy if it is exposed beyond a trusted network.

maglevc and maglevt need no special capabilities.

All gRPC traffic runs insecure by default (NFR-4.2). Securing transport is an operational decision, not a build-time one; deployments that require mTLS SHOULD terminate gRPC at a sidecar or colocate control and data plane on a trusted segment.

Concurrency Model

The concurrency model inside maglevd is deliberately simple and local:

• Each backend owns exactly one probe worker goroutine (NFR-3.1). Workers do not share state with each other.
• All events — transitions and log records — travel through a single central channel which is then fanned out to bounded per-subscriber queues (NFR-3.2). The fan-out is the only place where multiple subscribers can observe the same event.
• The configuration pointer is swapped atomically on reload (NFR-2.2); readers take a read lock for the duration of a single access, so the live config is always internally consistent even mid-reload.
• The VPP stats snapshot is published as an atomic pointer (NFR-3.3), so Prometheus scrapes and gRPC reads of counters are wait-free.
• Reconciliation holds a mutex around VPP calls, which serializes operator mutations, event-driven syncs, and periodic full syncs against each other (NFR-2.4). This is intentional: the order in which VPP sees mutations matters for determinism, and serializing them is cheap at the scale of control-plane events.

Deadlock avoidance is structural rather than audited: dependencies between subsystems are one-way. The checker does not call into VPP; the reconciler reads checker state and calls VPP; VPP never calls back. maglevd-frontend and maglevc only read from maglevd over gRPC. There is no cycle in the wait-for graph.

Alternatives Considered

This is a retrofit of a shipped system, so the alternatives here are the ones the code actively rejects, not speculative designs.

• Several probe schedulers sharing one goroutine pool. Rejected in favor of one goroutine per backend. The per-backend model is trivially correct, has no shared state, and scales linearly with backend count at a cost of a few kilobytes per backend.
• maglevd-frontend as a sidecar per maglevd. Rejected in favor of one frontend speaking to many daemons. A single dashboard pane across a fleet is the common operator request; pushing multi-server logic into the frontend keeps the daemon simple.
• Operator actions expressed as config edits plus SIGHUP. Rejected in favor of direct gRPC mutations. Pausing a backend during an incident should not require editing a file, and the effect should survive subsequent reloads (FR-3.4) — though, by deliberate design, not a daemon restart (FR-3.5).
• Persisting operator overrides across daemon restarts. Rejected in favor of making the YAML config file the sole source of truth on startup (FR-3.5). Persisting runtime overrides would require an on-disk side store and a clear policy for what happens when the side store and the config file disagree; keeping the daemon stateless on startup is simpler and harder to get wrong.
• Synchronous full sync after every transition. Rejected in favor of event-driven single-VIP syncs with a periodic full sync as a safety net (FR-4.1, FR-4.2). Full syncs are cheap but not free, and the blast radius of a transient bug in the desired-state computation is smaller when per-transition work only touches one VIP.
• Letting maglevt read maglevd's gRPC. Rejected in favor of probing the YAML file directly so that maglevt remains useful when maglevd itself is the thing being investigated.

Open Questions

• Mutual TLS for gRPC. Currently insecure by default. A future version may wire in standard mTLS support once a credential-management story is picked.
• Per-AS traffic counters. The VPP lb plugin bypasses the FIB and therefore does not produce per-AS traffic counters. Surfacing real per-backend byte/packet counts would require a VPP-side change.
• High-availability of the control plane. Two maglevd instances on the same VPP would interleave writes harmlessly thanks to determinism (NFR-2.1), but there is no leader election and no formal story about which instance owns which VIPs. Today, operators run a single maglevd per VPP host.