This commit wires maglevd through to VPP's LB plugin end-to-end, using
locally-generated GoVPP bindings for the newer v2 API messages.
VPP binapi (vendored)
- New package internal/vpp/binapi/ containing lb, lb_types, ip_types, and
interface_types, generated from a local VPP build (~/src/vpp) via a new
'make vpp-binapi' target. GoVPP v0.12.0 upstream lacks the v2 messages we
need (lb_conf_get, lb_add_del_vip_v2, lb_add_del_as_v2, lb_as_v2_dump,
lb_as_set_weight), so we commit the generated output in-tree.
- All generated files go through our loggedChannel wrapper; every VPP API
send/receive is recorded at DEBUG via slog (vpp-api-send / vpp-api-recv /
vpp-api-send-multi / vpp-api-recv-multi) so the full wire-level trail is
auditable. NewAPIChannel is unexported — callers must use c.apiChannel().
Read path: GetLBState{All,VIP}
- GetLBStateAll returns a full snapshot (global conf + every VIP with its
attached application servers).
- GetLBStateVIP looks up a single VIP by (prefix, protocol, port) and
returns (nil, nil) when the VIP doesn't exist in VPP. This is the
efficient path for targeted updates on a busy LB.
- Helpers factored out: getLBConf, dumpAllVIPs, dumpASesForVIP, lookupVIP,
vipFromDetails.
Write path: SyncLBState{All,VIP}
- SyncLBStateAll reconciles every configured frontend with VPP: creates
missing VIPs, removes stale ones (with AS flush), and reconciles AS
membership and weights within VIPs that exist on both sides.
- SyncLBStateVIP targets a single frontend by name. Never removes VIPs.
Returns ErrFrontendNotFound (wrapped with the name) when the frontend
isn't in config, so callers can use errors.Is.
- Shared reconcileVIP helper does the per-VIP AS diff; removeVIP is used
only by the full-sync pass.
- LbAddDelVipV2 requests always set NewFlowsTableLength=1024. The .api
default=1024 annotation is only applied by VAT/CLI parsers, not wire-
level marshalling — sending 0 caused VPP to vec_validate with mask
0xFFFFFFFF and OOM-panic.
- Pool semantics: backends in the primary (first) pool of a frontend get
their configured weight; backends in secondary pools get weight 0. All
backends are installed so higher layers can flip weights on failover
without add/remove churn.
- Every individual change emits a DEBUG slog (vpp-lbsync-vip-add/del,
vpp-lbsync-as-add/del, vpp-lbsync-as-weight). Start/done INFO logs
carry a scope=all|vip label plus aggregate counts.
Global conf push: SetLBConf
- New SetLBConf(cfg) sends lb_conf with ipv4-src, ipv6-src, sticky-buckets,
and flow-timeout. Called automatically on VPP (re)connect and after
every config reload (via doReloadConfig). Results are cached on the
Client so redundant pushes are silently skipped — only actual changes
produce a vpp-lb-conf-set INFO log line.
Periodic drift reconciliation
- vpp.Client.lbSyncLoop runs in a goroutine tied to each VPP connection's
lifetime. Its first tick is immediate (startup and post-reconnect
sync quickly); subsequent ticks fire every vpp.lb.sync-interval from
config (default 30s). Purpose: catch drift if something/someone
modifies VPP state by hand. The loop uses a ConfigSource interface
(satisfied by checker.Checker via its new Config() accessor) to avoid
an import cycle with the checker package.
Config schema additions (maglev.vpp.lb)
- sync-interval: positive Go duration, default 30s.
- ipv4-src-address: REQUIRED. Used as the outer source for GRE4 encap
to application servers. Missing this is a hard semantic error —
maglevd --check exits 2 and the daemon refuses to start. VPP GRE
needs a source address and every VIP we program uses GRE, so there
is no meaningful config without it.
- ipv6-src-address: REQUIRED. Same treatment as ipv4-src-address.
- sticky-buckets-per-core: default 65536, must be a power of 2.
- flow-timeout: default 40s, must be a whole number of seconds in [1s, 120s].
- VPP validation runs at the end of convert() so structural errors in
healthchecks/backends/frontends surface first — operators fix those,
then get the VPP-specific requirements.
gRPC API
- New GetVPPLBState RPC returning VPPLBState: global conf + VIPs with
ASes. Mirrors the read-path but strips fields irrelevant to our
GRE-only deployment (srv_type, dscp, target_port).
- New SyncVPPLBState RPC with optional frontend_name. Unset → full sync
(may remove stale VIPs). Set → single-VIP sync (never removes).
Returns codes.NotFound for unknown frontends, codes.Unavailable when
VPP integration is disabled or disconnected.
maglevc (CLI)
- New 'show vpp lbstate' command displaying the LB plugin state. VPP-only
fields the dataplane irrelevant to GRE are suppressed. Per-AS lines use
a key-value format ("address X weight Y flow-table-buckets Z")
instead of a tabwriter column, which avoids the ANSI-color alignment
issue we hit with mixed label/data rows.
- New 'sync vpp lbstate [<name>]' command. Without a name, triggers a
full reconciliation; with a name, targets one frontend.
- Previous 'show vpp lb' renamed to 'show vpp lbstate' for consistency
with the new sync command.
Test fixtures
- validConfig and all ad-hoc config_test.go fixtures that reach the end
of convert() now include the two required vpp.lb src addresses.
- tests/01-maglevd/maglevd-lab/maglev.yaml gains a vpp.lb section so the
robot integration tests can still load the config.
- cmd/maglevc/tree_test.go gains expected paths for the new commands.
Docs
- config-guide.md: new 'vpp' section in the basic structure, detailed
vpp.lb field reference, noting ipv4/ipv6 src addresses as REQUIRED
(hard error) with no defaults; example config updated.
- user-guide.md: documented 'show vpp info', 'show vpp lbstate',
'sync vpp lbstate [<name>]', new --vpp-api-addr and --vpp-stats-addr
flags, the vpp-lb-conf-set log line, and corrected the pause/resume
description to reflect that pause cancels the probe goroutine.
- debian/maglev.yaml: example config gains a vpp.lb block with src
addresses and commented optional overrides.
167 lines
5.3 KiB
Go
167 lines
5.3 KiB
Go
// Copyright (c) 2026, Pim van Pelt <pim@ipng.ch>
|
|
|
|
package main
|
|
|
|
import (
|
|
"context"
|
|
"flag"
|
|
"fmt"
|
|
"log/slog"
|
|
"net"
|
|
"net/http"
|
|
"os"
|
|
"os/signal"
|
|
"syscall"
|
|
|
|
"github.com/prometheus/client_golang/prometheus"
|
|
"github.com/prometheus/client_golang/prometheus/promhttp"
|
|
"google.golang.org/grpc"
|
|
"google.golang.org/grpc/reflection"
|
|
|
|
buildinfo "git.ipng.ch/ipng/vpp-maglev/cmd"
|
|
"git.ipng.ch/ipng/vpp-maglev/internal/checker"
|
|
"git.ipng.ch/ipng/vpp-maglev/internal/config"
|
|
"git.ipng.ch/ipng/vpp-maglev/internal/grpcapi"
|
|
"git.ipng.ch/ipng/vpp-maglev/internal/metrics"
|
|
"git.ipng.ch/ipng/vpp-maglev/internal/vpp"
|
|
)
|
|
|
|
func main() {
|
|
if err := run(); err != nil {
|
|
slog.Error("startup-fatal", "err", err)
|
|
os.Exit(1)
|
|
}
|
|
}
|
|
|
|
func run() error {
|
|
// ---- flags / env --------------------------------------------------------
|
|
printVersion := flag.Bool("version", false, "print version and exit")
|
|
checkOnly := flag.Bool("check", false, "check config file and exit (0=ok, 1=parse error, 2=semantic error)")
|
|
enableReflection := flag.Bool("reflection", true, "enable gRPC server reflection (for grpcurl)")
|
|
configPath := stringFlag("config", "/etc/vpp-maglev/maglev.yaml", "MAGLEV_CONFIG", "path to maglev.yaml")
|
|
grpcAddr := stringFlag("grpc-addr", ":9090", "MAGLEV_GRPC_ADDR", "gRPC listen address")
|
|
metricsAddr := stringFlag("metrics-addr", ":9091", "MAGLEV_METRICS_ADDR", "Prometheus /metrics listen address (empty to disable)")
|
|
vppAPIAddr := stringFlag("vpp-api-addr", "/run/vpp/api.sock", "MAGLEV_VPP_API_ADDR", "VPP binary API socket path (empty to disable)")
|
|
vppStatsAddr := stringFlag("vpp-stats-addr", "/run/vpp/stats.sock", "MAGLEV_VPP_STATS_ADDR", "VPP stats socket path")
|
|
logLevel := stringFlag("log-level", "info", "MAGLEV_LOG_LEVEL", "log level (debug|info|warn|error)")
|
|
flag.Parse()
|
|
|
|
if *printVersion {
|
|
fmt.Printf("maglevd %s (commit %s, built %s)\n",
|
|
buildinfo.Version(), buildinfo.Commit(), buildinfo.Date())
|
|
return nil
|
|
}
|
|
|
|
if *checkOnly {
|
|
_, result := config.Check(*configPath)
|
|
if result.OK() {
|
|
fmt.Printf("config ok: %s\n", *configPath)
|
|
return nil
|
|
}
|
|
if result.ParseError != "" {
|
|
fmt.Fprintf(os.Stderr, "parse error: %s\n", result.ParseError)
|
|
os.Exit(1)
|
|
}
|
|
fmt.Fprintf(os.Stderr, "semantic error: %s\n", result.SemanticError)
|
|
os.Exit(2)
|
|
}
|
|
|
|
// ---- logging ------------------------------------------------------------
|
|
var level slog.Level
|
|
if err := level.UnmarshalText([]byte(*logLevel)); err != nil {
|
|
return fmt.Errorf("invalid log level %q: %w", *logLevel, err)
|
|
}
|
|
jsonHandler := slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: level})
|
|
logBroadcaster := grpcapi.NewLogBroadcaster(jsonHandler)
|
|
slog.SetDefault(slog.New(logBroadcaster))
|
|
slog.Info("starting", "version", buildinfo.Version(), "commit", buildinfo.Commit(), "date", buildinfo.Date())
|
|
|
|
// ---- config -------------------------------------------------------------
|
|
cfg, err := config.Load(*configPath)
|
|
if err != nil {
|
|
return fmt.Errorf("load config: %w", err)
|
|
}
|
|
slog.Info("config-loaded", "path", *configPath, "frontends", len(cfg.Frontends))
|
|
|
|
// ---- checker ------------------------------------------------------------
|
|
chkr := checker.New(cfg)
|
|
|
|
ctx, rootCancel := context.WithCancel(context.Background())
|
|
defer rootCancel()
|
|
|
|
go func() {
|
|
if err := chkr.Run(ctx); err != nil {
|
|
slog.Error("checker-exited", "err", err)
|
|
}
|
|
}()
|
|
|
|
// ---- VPP connection -----------------------------------------------------
|
|
var vppClient *vpp.Client
|
|
if *vppAPIAddr != "" {
|
|
vppClient = vpp.New(*vppAPIAddr, *vppStatsAddr)
|
|
vppClient.SetConfigSource(chkr)
|
|
go vppClient.Run(ctx)
|
|
}
|
|
|
|
// ---- gRPC server --------------------------------------------------------
|
|
lis, err := net.Listen("tcp", *grpcAddr)
|
|
if err != nil {
|
|
return fmt.Errorf("listen %s: %w", *grpcAddr, err)
|
|
}
|
|
srv := grpc.NewServer()
|
|
maglevServer := grpcapi.NewServer(ctx, chkr, logBroadcaster, *configPath, vppClient)
|
|
grpcapi.RegisterMaglevServer(srv, maglevServer)
|
|
if *enableReflection {
|
|
reflection.Register(srv)
|
|
}
|
|
slog.Info("grpc-listening", "addr", *grpcAddr, "reflection", *enableReflection)
|
|
|
|
go func() {
|
|
if err := srv.Serve(lis); err != nil {
|
|
slog.Error("grpc-serve-error", "err", err)
|
|
}
|
|
}()
|
|
|
|
// ---- Prometheus metrics -------------------------------------------------
|
|
if *metricsAddr != "" {
|
|
reg := prometheus.DefaultRegisterer
|
|
metrics.Register(reg, chkr)
|
|
|
|
mux := http.NewServeMux()
|
|
mux.Handle("/metrics", promhttp.Handler())
|
|
slog.Info("metrics-listening", "addr", *metricsAddr)
|
|
go func() {
|
|
if err := http.ListenAndServe(*metricsAddr, mux); err != nil && err != http.ErrServerClosed {
|
|
slog.Error("metrics-serve-error", "err", err)
|
|
}
|
|
}()
|
|
}
|
|
|
|
// ---- signal handling ----------------------------------------------------
|
|
sigCh := make(chan os.Signal, 1)
|
|
signal.Notify(sigCh, syscall.SIGHUP, syscall.SIGTERM, syscall.SIGINT)
|
|
|
|
for sig := range sigCh {
|
|
switch sig {
|
|
case syscall.SIGHUP:
|
|
maglevServer.TriggerReload()
|
|
|
|
case syscall.SIGTERM, syscall.SIGINT:
|
|
slog.Info("shutdown", "signal", sig)
|
|
rootCancel()
|
|
srv.GracefulStop()
|
|
return nil
|
|
}
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// stringFlag declares a flag that falls back to an environment variable.
|
|
func stringFlag(name, defaultVal, envKey, usage string) *string {
|
|
val := defaultVal
|
|
if v := os.Getenv(envKey); v != "" {
|
|
val = v
|
|
}
|
|
return flag.String(name, val, fmt.Sprintf("%s (env: %s)", usage, envKey))
|
|
}
|