VPP load-balancer dataplane integration: state, sync, and global conf
This commit wires maglevd through to VPP's LB plugin end-to-end, using
locally-generated GoVPP bindings for the newer v2 API messages.
VPP binapi (vendored)
- New package internal/vpp/binapi/ containing lb, lb_types, ip_types, and
interface_types, generated from a local VPP build (~/src/vpp) via a new
'make vpp-binapi' target. GoVPP v0.12.0 upstream lacks the v2 messages we
need (lb_conf_get, lb_add_del_vip_v2, lb_add_del_as_v2, lb_as_v2_dump,
lb_as_set_weight), so we commit the generated output in-tree.
- All generated files go through our loggedChannel wrapper; every VPP API
send/receive is recorded at DEBUG via slog (vpp-api-send / vpp-api-recv /
vpp-api-send-multi / vpp-api-recv-multi) so the full wire-level trail is
auditable. NewAPIChannel is unexported — callers must use c.apiChannel().
Read path: GetLBState{All,VIP}
- GetLBStateAll returns a full snapshot (global conf + every VIP with its
attached application servers).
- GetLBStateVIP looks up a single VIP by (prefix, protocol, port) and
returns (nil, nil) when the VIP doesn't exist in VPP. This is the
efficient path for targeted updates on a busy LB.
- Helpers factored out: getLBConf, dumpAllVIPs, dumpASesForVIP, lookupVIP,
vipFromDetails.
Write path: SyncLBState{All,VIP}
- SyncLBStateAll reconciles every configured frontend with VPP: creates
missing VIPs, removes stale ones (with AS flush), and reconciles AS
membership and weights within VIPs that exist on both sides.
- SyncLBStateVIP targets a single frontend by name. Never removes VIPs.
Returns ErrFrontendNotFound (wrapped with the name) when the frontend
isn't in config, so callers can use errors.Is.
- Shared reconcileVIP helper does the per-VIP AS diff; removeVIP is used
only by the full-sync pass.
- LbAddDelVipV2 requests always set NewFlowsTableLength=1024. The .api
default=1024 annotation is only applied by VAT/CLI parsers, not wire-
level marshalling — sending 0 caused VPP to vec_validate with mask
0xFFFFFFFF and OOM-panic.
- Pool semantics: backends in the primary (first) pool of a frontend get
their configured weight; backends in secondary pools get weight 0. All
backends are installed so higher layers can flip weights on failover
without add/remove churn.
- Every individual change emits a DEBUG slog (vpp-lbsync-vip-add/del,
vpp-lbsync-as-add/del, vpp-lbsync-as-weight). Start/done INFO logs
carry a scope=all|vip label plus aggregate counts.
Global conf push: SetLBConf
- New SetLBConf(cfg) sends lb_conf with ipv4-src, ipv6-src, sticky-buckets,
and flow-timeout. Called automatically on VPP (re)connect and after
every config reload (via doReloadConfig). Results are cached on the
Client so redundant pushes are silently skipped — only actual changes
produce a vpp-lb-conf-set INFO log line.
Periodic drift reconciliation
- vpp.Client.lbSyncLoop runs in a goroutine tied to each VPP connection's
lifetime. Its first tick is immediate (startup and post-reconnect
sync quickly); subsequent ticks fire every vpp.lb.sync-interval from
config (default 30s). Purpose: catch drift if something/someone
modifies VPP state by hand. The loop uses a ConfigSource interface
(satisfied by checker.Checker via its new Config() accessor) to avoid
an import cycle with the checker package.
Config schema additions (maglev.vpp.lb)
- sync-interval: positive Go duration, default 30s.
- ipv4-src-address: REQUIRED. Used as the outer source for GRE4 encap
to application servers. Missing this is a hard semantic error —
maglevd --check exits 2 and the daemon refuses to start. VPP GRE
needs a source address and every VIP we program uses GRE, so there
is no meaningful config without it.
- ipv6-src-address: REQUIRED. Same treatment as ipv4-src-address.
- sticky-buckets-per-core: default 65536, must be a power of 2.
- flow-timeout: default 40s, must be a whole number of seconds in [1s, 120s].
- VPP validation runs at the end of convert() so structural errors in
healthchecks/backends/frontends surface first — operators fix those,
then get the VPP-specific requirements.
gRPC API
- New GetVPPLBState RPC returning VPPLBState: global conf + VIPs with
ASes. Mirrors the read-path but strips fields irrelevant to our
GRE-only deployment (srv_type, dscp, target_port).
- New SyncVPPLBState RPC with optional frontend_name. Unset → full sync
(may remove stale VIPs). Set → single-VIP sync (never removes).
Returns codes.NotFound for unknown frontends, codes.Unavailable when
VPP integration is disabled or disconnected.
maglevc (CLI)
- New 'show vpp lbstate' command displaying the LB plugin state. VPP-only
fields the dataplane irrelevant to GRE are suppressed. Per-AS lines use
a key-value format ("address X weight Y flow-table-buckets Z")
instead of a tabwriter column, which avoids the ANSI-color alignment
issue we hit with mixed label/data rows.
- New 'sync vpp lbstate [<name>]' command. Without a name, triggers a
full reconciliation; with a name, targets one frontend.
- Previous 'show vpp lb' renamed to 'show vpp lbstate' for consistency
with the new sync command.
Test fixtures
- validConfig and all ad-hoc config_test.go fixtures that reach the end
of convert() now include the two required vpp.lb src addresses.
- tests/01-maglevd/maglevd-lab/maglev.yaml gains a vpp.lb section so the
robot integration tests can still load the config.
- cmd/maglevc/tree_test.go gains expected paths for the new commands.
Docs
- config-guide.md: new 'vpp' section in the basic structure, detailed
vpp.lb field reference, noting ipv4/ipv6 src addresses as REQUIRED
(hard error) with no defaults; example config updated.
- user-guide.md: documented 'show vpp info', 'show vpp lbstate',
'sync vpp lbstate [<name>]', new --vpp-api-addr and --vpp-stats-addr
flags, the vpp-lb-conf-set log line, and corrected the pause/resume
description to reflect that pause cancels the probe goroutine.
- debian/maglev.yaml: example config gains a vpp.lb block with src
addresses and commented optional overrides.
This commit is contained in:
@@ -17,6 +17,7 @@ import (
|
||||
// Config is the top-level parsed and validated configuration.
|
||||
type Config struct {
|
||||
HealthChecker HealthCheckerConfig
|
||||
VPP VPPConfig
|
||||
HealthChecks map[string]HealthCheck
|
||||
Backends map[string]Backend
|
||||
Frontends map[string]Frontend
|
||||
@@ -28,6 +29,37 @@ type HealthCheckerConfig struct {
|
||||
Netns string // network namespace for probes; "" = current netns
|
||||
}
|
||||
|
||||
// VPPConfig holds VPP-related configuration.
|
||||
type VPPConfig struct {
|
||||
LB VPPLBConfig
|
||||
}
|
||||
|
||||
// VPPLBConfig holds load-balancer integration settings.
|
||||
type VPPLBConfig struct {
|
||||
// SyncInterval is how often the full dataplane reconciliation runs,
|
||||
// catching drift (e.g. manual changes to VPP). Defaults to 30s.
|
||||
SyncInterval time.Duration
|
||||
|
||||
// IPv4SrcAddress is the source address VPP uses when encapsulating
|
||||
// IPv4 traffic into GRE4 tunnels to application servers. Required
|
||||
// when any frontend uses an IPv4 VIP; VPP GRE encap will fail if unset.
|
||||
IPv4SrcAddress net.IP
|
||||
|
||||
// IPv6SrcAddress is the source address VPP uses when encapsulating
|
||||
// IPv6 traffic into GRE6 tunnels. Required when any frontend uses an
|
||||
// IPv6 VIP; VPP GRE encap will fail if unset.
|
||||
IPv6SrcAddress net.IP
|
||||
|
||||
// StickyBucketsPerCore is the number of buckets (per worker thread) in
|
||||
// the established-flow table. Must be a power of 2. Defaults to 65536.
|
||||
StickyBucketsPerCore uint32
|
||||
|
||||
// FlowTimeout is the idle time after which an established flow is
|
||||
// removed from the table. Must be between 1 and 120 seconds inclusive.
|
||||
// Defaults to 40s.
|
||||
FlowTimeout time.Duration
|
||||
}
|
||||
|
||||
// HealthCheck describes how to probe a backend.
|
||||
type HealthCheck struct {
|
||||
Type string
|
||||
@@ -97,6 +129,7 @@ type rawConfig struct {
|
||||
|
||||
type rawMaglev struct {
|
||||
HealthChecker rawHealthCheckerCfg `yaml:"healthchecker"`
|
||||
VPP rawVPPCfg `yaml:"vpp"`
|
||||
HealthChecks map[string]rawHealthCheck `yaml:"healthchecks"`
|
||||
Backends map[string]rawBackend `yaml:"backends"`
|
||||
Frontends map[string]rawFrontend `yaml:"frontends"`
|
||||
@@ -107,6 +140,18 @@ type rawHealthCheckerCfg struct {
|
||||
Netns string `yaml:"netns"`
|
||||
}
|
||||
|
||||
type rawVPPCfg struct {
|
||||
LB rawVPPLBCfg `yaml:"lb"`
|
||||
}
|
||||
|
||||
type rawVPPLBCfg struct {
|
||||
SyncInterval string `yaml:"sync-interval"` // Go duration; default 30s
|
||||
IPv4SrcAddress string `yaml:"ipv4-src-address"`
|
||||
IPv6SrcAddress string `yaml:"ipv6-src-address"`
|
||||
StickyBucketsPerCore *uint32 `yaml:"sticky-buckets-per-core"` // default 65536
|
||||
FlowTimeout string `yaml:"flow-timeout"` // Go duration; default 40s, [1-120]s
|
||||
}
|
||||
|
||||
type rawHealthCheck struct {
|
||||
Type string `yaml:"type"`
|
||||
Port uint16 `yaml:"port"`
|
||||
@@ -255,9 +300,93 @@ func convert(r *rawMaglev) (*Config, error) {
|
||||
cfg.Frontends[name] = fe
|
||||
}
|
||||
|
||||
// ---- vpp ------------------------------------------------------------------
|
||||
// Runs last so structural errors in healthchecks/backends/frontends are
|
||||
// reported first; operators fix those, then we tell them about the VPP
|
||||
// src-address requirements.
|
||||
if err := convertVPP(&r.VPP, &cfg.VPP); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
// convertVPP parses and validates the maglev.vpp section. Missing src-address
|
||||
// fields are tolerated but logged at ERROR level so operators notice that VPP
|
||||
// GRE encap will fail without them.
|
||||
func convertVPP(r *rawVPPCfg, cfg *VPPConfig) error {
|
||||
// sync-interval: default 30s, must be > 0.
|
||||
if s := r.LB.SyncInterval; s != "" {
|
||||
d, err := time.ParseDuration(s)
|
||||
if err != nil {
|
||||
return fmt.Errorf("vpp.lb.sync-interval: %w", err)
|
||||
}
|
||||
if d <= 0 {
|
||||
return fmt.Errorf("vpp.lb.sync-interval must be > 0")
|
||||
}
|
||||
cfg.LB.SyncInterval = d
|
||||
} else {
|
||||
cfg.LB.SyncInterval = 30 * time.Second
|
||||
}
|
||||
|
||||
// ipv4-src-address: optional here, but warned below if missing.
|
||||
if s := r.LB.IPv4SrcAddress; s != "" {
|
||||
ip := net.ParseIP(s)
|
||||
if ip == nil || ip.To4() == nil {
|
||||
return fmt.Errorf("vpp.lb.ipv4-src-address: %q is not a valid IPv4 address", s)
|
||||
}
|
||||
cfg.LB.IPv4SrcAddress = ip.To4()
|
||||
}
|
||||
|
||||
// ipv6-src-address: optional here, but warned below if missing.
|
||||
if s := r.LB.IPv6SrcAddress; s != "" {
|
||||
ip := net.ParseIP(s)
|
||||
if ip == nil || ip.To4() != nil {
|
||||
return fmt.Errorf("vpp.lb.ipv6-src-address: %q is not a valid IPv6 address", s)
|
||||
}
|
||||
cfg.LB.IPv6SrcAddress = ip.To16()
|
||||
}
|
||||
|
||||
// sticky-buckets-per-core: default 65536, must be power of 2.
|
||||
if p := r.LB.StickyBucketsPerCore; p != nil {
|
||||
n := *p
|
||||
if n == 0 || n&(n-1) != 0 {
|
||||
return fmt.Errorf("vpp.lb.sticky-buckets-per-core: %d must be a power of 2", n)
|
||||
}
|
||||
cfg.LB.StickyBucketsPerCore = n
|
||||
} else {
|
||||
cfg.LB.StickyBucketsPerCore = 65536
|
||||
}
|
||||
|
||||
// flow-timeout: default 40s, must be 1-120s inclusive and a whole number of seconds.
|
||||
if s := r.LB.FlowTimeout; s != "" {
|
||||
d, err := time.ParseDuration(s)
|
||||
if err != nil {
|
||||
return fmt.Errorf("vpp.lb.flow-timeout: %w", err)
|
||||
}
|
||||
if d%time.Second != 0 {
|
||||
return fmt.Errorf("vpp.lb.flow-timeout: %s must be a whole number of seconds", d)
|
||||
}
|
||||
if d < time.Second || d > 120*time.Second {
|
||||
return fmt.Errorf("vpp.lb.flow-timeout: %s out of range [1s, 120s]", d)
|
||||
}
|
||||
cfg.LB.FlowTimeout = d
|
||||
} else {
|
||||
cfg.LB.FlowTimeout = 40 * time.Second
|
||||
}
|
||||
|
||||
// A missing src address is a hard error: VPP's GRE encap needs a source,
|
||||
// and every VIP we program uses GRE. Fail the config check so the
|
||||
// operator cannot start maglevd with a broken setup.
|
||||
if cfg.LB.IPv4SrcAddress == nil {
|
||||
return fmt.Errorf("vpp.lb.ipv4-src-address must be set; VPP GRE4 encap will fail for IPv4 VIPs")
|
||||
}
|
||||
if cfg.LB.IPv6SrcAddress == nil {
|
||||
return fmt.Errorf("vpp.lb.ipv6-src-address must be set; VPP GRE6 encap will fail for IPv6 VIPs")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func convertHealthCheck(r *rawHealthCheck) (HealthCheck, error) {
|
||||
h := HealthCheck{Type: r.Type, Port: r.Port}
|
||||
|
||||
|
||||
@@ -12,6 +12,10 @@ maglev:
|
||||
healthchecker:
|
||||
transition-history: 5
|
||||
netns: dataplane
|
||||
vpp:
|
||||
lb:
|
||||
ipv4-src-address: 10.0.0.1
|
||||
ipv6-src-address: 2001:db8::1
|
||||
healthchecks:
|
||||
http-check:
|
||||
type: http
|
||||
@@ -150,6 +154,10 @@ func TestValidConfig(t *testing.T) {
|
||||
func TestDefaults(t *testing.T) {
|
||||
raw := `
|
||||
maglev:
|
||||
vpp:
|
||||
lb:
|
||||
ipv4-src-address: 10.0.0.1
|
||||
ipv6-src-address: 2001:db8::1
|
||||
healthchecks:
|
||||
icmp:
|
||||
type: icmp
|
||||
@@ -196,6 +204,10 @@ func TestBackendNoHealthcheck(t *testing.T) {
|
||||
// A backend with no healthcheck reference is valid; probe is skipped.
|
||||
raw := `
|
||||
maglev:
|
||||
vpp:
|
||||
lb:
|
||||
ipv4-src-address: 10.0.0.1
|
||||
ipv6-src-address: 2001:db8::1
|
||||
healthchecks: {}
|
||||
backends:
|
||||
be:
|
||||
@@ -220,6 +232,10 @@ maglev:
|
||||
func TestOptionalIntervals(t *testing.T) {
|
||||
raw := `
|
||||
maglev:
|
||||
vpp:
|
||||
lb:
|
||||
ipv4-src-address: 10.0.0.1
|
||||
ipv6-src-address: 2001:db8::1
|
||||
healthchecks:
|
||||
icmp:
|
||||
type: icmp
|
||||
@@ -259,6 +275,10 @@ func TestValidationErrors(t *testing.T) {
|
||||
base := func(hcExtra, beExtra, feExtra string) string {
|
||||
return `
|
||||
maglev:
|
||||
vpp:
|
||||
lb:
|
||||
ipv4-src-address: 10.0.0.1
|
||||
ipv6-src-address: 2001:db8::1
|
||||
healthchecks:
|
||||
c:
|
||||
type: icmp
|
||||
@@ -294,6 +314,10 @@ maglev:
|
||||
name: "mixed backend address families in pool",
|
||||
yaml: `
|
||||
maglev:
|
||||
vpp:
|
||||
lb:
|
||||
ipv4-src-address: 10.0.0.1
|
||||
ipv6-src-address: 2001:db8::1
|
||||
healthchecks:
|
||||
c:
|
||||
type: icmp
|
||||
|
||||
Reference in New Issue
Block a user