Pim van Pelt
6d78921edd
Three reliability fixes bundled with docs updates. Restart-neutral VPP LB sync via a startup warmup window (internal/vpp/warmup.go). Before this, a maglevd restart would immediately issue SyncLBStateAll with every backend still in StateUnknown — mapped through BackendEffectiveWeight to weight 0 — and VPP would black-hole all new flows until the checker's rise counters caught up, several seconds later. The new warmup tracker owns a process-wide state machine gated by two config knobs: vpp.lb.startup-min-delay (default 5s) is an absolute hands-off window during which neither the periodic sync loop nor the per-transition reconciler touches VPP; vpp.lb. startup-max-delay (default 30s) is the watchdog for a per-VIP release phase that runs between the two, releasing each frontend as soon as every backend it references reaches a non-Unknown state. At max-delay a final SyncLBStateAll runs for any stragglers still in Unknown. Config reload does not reset the clock. Both delays can be set to 0 to disable the warmup entirely. The reconciler's suppressed-during-warmup events log at DEBUG so operators can still see them with --log-level debug. Unit tests cover the tracker state machine, allBackendsKnown precondition, and the zero-delay escape hatch. Deterministic AS iteration in VPP LB sync. reconcileVIP and recreateVIP now issue their lb_as_add_del / lb_as_set_weight calls in numeric IP order (IPv4 before IPv6, ascending within each family) via a new sortedIPKeys helper, instead of Go map iteration order. VPP's LB plugin breaks per-bucket ties in the Maglev lookup table by insertion position in its internal AS vec, so without a stable call order two maglevd instances on the same config could push identical AS sets into VPP in different orders and produce divergent new-flow tables. Numeric sort is used in preference to lexicographic so the sync log stays human-readable: string order would place 10.0.0.10 before 10.0.0.2, and the same problem in v6. Unit tests cover empty, single, v4/v6 numeric vs lexicographic, v4-before-v6 grouping, a 1000-iteration stability loop against Go's randomised map iteration, insertion-order invariance, and the desiredAS call-site type. maglevt interval fix. runProbeLoop used to sleep the full jittered interval after every probe, so a 100ms --interval with a 30ms probe actually produced a 130ms period. The sleep now subtracts result.Duration so cadence matches the flag. Probes that overrun clamp sleep to zero and fire the next probe immediately without trying to catch up on missed cycles — a slow backend doesn't get flooded with back-to-back probes at the moment it's already struggling. Docs. config-guide now documents flush-on-down and the new startup-min-delay / startup-max-delay knobs; user-guide's maglevd section explains the restart-neutrality property, the three warmup phases, and the relevant slog lines operators should watch for during a bounce. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
vpp-maglev
Health checker, gRPC control plane, CLI, and web dashboard for the VPP
lb (load-balancer) plugin. Runs as a set of three binaries under one
Debian package:
maglevd— the long-running health-checker daemon. Probes backends (HTTP, TCP, ICMP), tracks their aggregate state, programs the VPP dataplane via thelbplugin binary API, and exposes everything over a gRPC API + Prometheus/metricsendpoint.maglevc— the interactive CLI client. Tab-completing shell with inline help; also runs one-shot commands for scripting.maglevd-frontend— optional web dashboard. One binary with a SolidJS Single-Page-App; connects to one or more maglevds over gRPC and serves a live HTTP view (read-only/view/and optional basic-auth/admin/with mutating commands).
Build and install
make install-deps # installs all build-time dependencies
make # builds build/<arch>/ binaries
make test # runs all tests
make pkg-deb # creates a Debian package for amd64 and arm64
Requires Go 1.25+ and (for make proto) protoc with protoc-gen-go
and protoc-gen-go-grpc. The SolidJS bundle under
cmd/frontend/web/ is built automatically via make through the
maglevd-frontend-web target, which needs npm.
Produces vpp-maglev_<version>_amd64.deb and
vpp-maglev_<version>_arm64.deb in the build/ directory by
cross-compiling with GOOS=linux GOARCH=<arch>. Requires dpkg-deb
(available on any Debian/Ubuntu host). The installed binaries report
the exact git commit via maglevd --version (and similarly for
maglevc / maglevd-frontend).
Running
After installing, maglevd is enabled automatically but
maglevd-frontend is not — it's opt-in, so the web dashboard
doesn't surprise anyone who just wanted the daemon:
# edit /etc/vpp-maglev/maglev.yaml, then:
systemctl enable --now vpp-maglev
# optional: web dashboard. Edit /etc/default/vpp-maglev to set
# MAGLEV_FRONTEND_ARGS and (optionally) MAGLEV_FRONTEND_USER /
# MAGLEV_FRONTEND_PASSWORD for /admin/ access, then:
systemctl enable --now vpp-maglev-frontend
Or run the components by hand:
maglevd --config /etc/vpp-maglev/maglev.yaml --grpc-addr :9090
maglevd --version # print version and exit
maglevc --server localhost:9090 # interactive shell
maglevc show frontends # one-shot
maglevc -color=false show backends # one-shot, no ANSI color
maglevc set backend nginx0-ams pause
maglevd-frontend -server localhost:9090 -listen :8080
Send SIGHUP to maglevd to reload config without restarting.
maglevd requires:
CAP_NET_RAWfor ICMP health checks (raw sockets).CAP_SYS_ADMINwhenhealthchecker.netnsis set so probes cansetns(CLONE_NEWNET)into the dataplane namespace. Without it, every probe errors out withenter netns "<name>": operation not permitted.
The Debian systemd unit grants both via AmbientCapabilities /
CapabilityBoundingSet, so systemctl start vpp-maglev works out
of the box. When running by hand under a non-root user, grant them
via setcap cap_net_raw,cap_sys_admin=eip /usr/sbin/maglevd or
equivalent.
maglevd-frontend also ignores SIGHUP so a controlling-terminal
disconnect (e.g. closing the SSH session it was started from)
doesn't kill the daemon; SIGTERM / SIGINT remain the clean
shutdown signals.
Every flag on every binary also has an environment-variable
equivalent (e.g. MAGLEV_CONFIG, MAGLEV_GRPC_ADDR,
MAGLEV_SERVERS, MAGLEV_LISTEN, MAGLEV_LOG_LEVEL) so all three
programs can be driven entirely via env in containerized
deployments.
Documentation
- A minimal configuration file in debian/maglev.yaml shows every knob.
- docs/user-guide.md — flags, signals, and
maglevccommand reference. - docs/config-guide.md — full YAML reference.
- docs/healthchecks.md — health state machine, probe scheduling, rise/fall semantics.
- Manpages:
maglevd(8),maglevc(1),maglevd-frontend(8).
Docker
docker build -t maglevd .
docker run --cap-add NET_RAW \
-v /etc/vpp-maglev:/etc/vpp-maglev maglevd
# With netns-scoped health checks (maglev.yaml sets healthchecker.netns):
docker run --cap-add NET_RAW --cap-add SYS_ADMIN \
-v /etc/vpp-maglev:/etc/vpp-maglev \
-v /var/run/netns:/var/run/netns maglevd