VPP load-balancer dataplane integration: state, sync, and global conf
This commit wires maglevd through to VPP's LB plugin end-to-end, using
locally-generated GoVPP bindings for the newer v2 API messages.
VPP binapi (vendored)
- New package internal/vpp/binapi/ containing lb, lb_types, ip_types, and
interface_types, generated from a local VPP build (~/src/vpp) via a new
'make vpp-binapi' target. GoVPP v0.12.0 upstream lacks the v2 messages we
need (lb_conf_get, lb_add_del_vip_v2, lb_add_del_as_v2, lb_as_v2_dump,
lb_as_set_weight), so we commit the generated output in-tree.
- All generated files go through our loggedChannel wrapper; every VPP API
send/receive is recorded at DEBUG via slog (vpp-api-send / vpp-api-recv /
vpp-api-send-multi / vpp-api-recv-multi) so the full wire-level trail is
auditable. NewAPIChannel is unexported — callers must use c.apiChannel().
Read path: GetLBState{All,VIP}
- GetLBStateAll returns a full snapshot (global conf + every VIP with its
attached application servers).
- GetLBStateVIP looks up a single VIP by (prefix, protocol, port) and
returns (nil, nil) when the VIP doesn't exist in VPP. This is the
efficient path for targeted updates on a busy LB.
- Helpers factored out: getLBConf, dumpAllVIPs, dumpASesForVIP, lookupVIP,
vipFromDetails.
Write path: SyncLBState{All,VIP}
- SyncLBStateAll reconciles every configured frontend with VPP: creates
missing VIPs, removes stale ones (with AS flush), and reconciles AS
membership and weights within VIPs that exist on both sides.
- SyncLBStateVIP targets a single frontend by name. Never removes VIPs.
Returns ErrFrontendNotFound (wrapped with the name) when the frontend
isn't in config, so callers can use errors.Is.
- Shared reconcileVIP helper does the per-VIP AS diff; removeVIP is used
only by the full-sync pass.
- LbAddDelVipV2 requests always set NewFlowsTableLength=1024. The .api
default=1024 annotation is only applied by VAT/CLI parsers, not wire-
level marshalling — sending 0 caused VPP to vec_validate with mask
0xFFFFFFFF and OOM-panic.
- Pool semantics: backends in the primary (first) pool of a frontend get
their configured weight; backends in secondary pools get weight 0. All
backends are installed so higher layers can flip weights on failover
without add/remove churn.
- Every individual change emits a DEBUG slog (vpp-lbsync-vip-add/del,
vpp-lbsync-as-add/del, vpp-lbsync-as-weight). Start/done INFO logs
carry a scope=all|vip label plus aggregate counts.
Global conf push: SetLBConf
- New SetLBConf(cfg) sends lb_conf with ipv4-src, ipv6-src, sticky-buckets,
and flow-timeout. Called automatically on VPP (re)connect and after
every config reload (via doReloadConfig). Results are cached on the
Client so redundant pushes are silently skipped — only actual changes
produce a vpp-lb-conf-set INFO log line.
Periodic drift reconciliation
- vpp.Client.lbSyncLoop runs in a goroutine tied to each VPP connection's
lifetime. Its first tick is immediate (startup and post-reconnect
sync quickly); subsequent ticks fire every vpp.lb.sync-interval from
config (default 30s). Purpose: catch drift if something/someone
modifies VPP state by hand. The loop uses a ConfigSource interface
(satisfied by checker.Checker via its new Config() accessor) to avoid
an import cycle with the checker package.
Config schema additions (maglev.vpp.lb)
- sync-interval: positive Go duration, default 30s.
- ipv4-src-address: REQUIRED. Used as the outer source for GRE4 encap
to application servers. Missing this is a hard semantic error —
maglevd --check exits 2 and the daemon refuses to start. VPP GRE
needs a source address and every VIP we program uses GRE, so there
is no meaningful config without it.
- ipv6-src-address: REQUIRED. Same treatment as ipv4-src-address.
- sticky-buckets-per-core: default 65536, must be a power of 2.
- flow-timeout: default 40s, must be a whole number of seconds in [1s, 120s].
- VPP validation runs at the end of convert() so structural errors in
healthchecks/backends/frontends surface first — operators fix those,
then get the VPP-specific requirements.
gRPC API
- New GetVPPLBState RPC returning VPPLBState: global conf + VIPs with
ASes. Mirrors the read-path but strips fields irrelevant to our
GRE-only deployment (srv_type, dscp, target_port).
- New SyncVPPLBState RPC with optional frontend_name. Unset → full sync
(may remove stale VIPs). Set → single-VIP sync (never removes).
Returns codes.NotFound for unknown frontends, codes.Unavailable when
VPP integration is disabled or disconnected.
maglevc (CLI)
- New 'show vpp lbstate' command displaying the LB plugin state. VPP-only
fields the dataplane irrelevant to GRE are suppressed. Per-AS lines use
a key-value format ("address X weight Y flow-table-buckets Z")
instead of a tabwriter column, which avoids the ANSI-color alignment
issue we hit with mixed label/data rows.
- New 'sync vpp lbstate [<name>]' command. Without a name, triggers a
full reconciliation; with a name, targets one frontend.
- Previous 'show vpp lb' renamed to 'show vpp lbstate' for consistency
with the new sync command.
Test fixtures
- validConfig and all ad-hoc config_test.go fixtures that reach the end
of convert() now include the two required vpp.lb src addresses.
- tests/01-maglevd/maglevd-lab/maglev.yaml gains a vpp.lb section so the
robot integration tests can still load the config.
- cmd/maglevc/tree_test.go gains expected paths for the new commands.
Docs
- config-guide.md: new 'vpp' section in the basic structure, detailed
vpp.lb field reference, noting ipv4/ipv6 src addresses as REQUIRED
(hard error) with no defaults; example config updated.
- user-guide.md: documented 'show vpp info', 'show vpp lbstate',
'sync vpp lbstate [<name>]', new --vpp-api-addr and --vpp-stats-addr
flags, the vpp-lb-conf-set log line, and corrected the pause/resume
description to reflect that pause cancels the probe goroutine.
- debian/maglev.yaml: example config gains a vpp.lb block with src
addresses and commented optional overrides.
This commit is contained in:
@@ -71,12 +71,13 @@ func buildTree() *Node {
|
||||
Children: []*Node{showHealthCheckName},
|
||||
}
|
||||
|
||||
// show vpp info
|
||||
// show vpp info / lbstate
|
||||
showVPPInfo := &Node{Word: "info", Help: "Show VPP version, uptime, and connection status", Run: runShowVPPInfo}
|
||||
showVPPLBState := &Node{Word: "lbstate", Help: "Show VPP load-balancer state (VIPs and application servers)", Run: runShowVPPLBState}
|
||||
showVPP := &Node{
|
||||
Word: "vpp",
|
||||
Help: "VPP dataplane information",
|
||||
Children: []*Node{showVPPInfo},
|
||||
Children: []*Node{showVPPInfo, showVPPLBState},
|
||||
}
|
||||
|
||||
show.Children = []*Node{
|
||||
@@ -174,7 +175,34 @@ func buildTree() *Node {
|
||||
Children: []*Node{configCheck, configReload},
|
||||
}
|
||||
|
||||
root.Children = []*Node{show, set, watch, configNode, quit, exit}
|
||||
// sync vpp lbstate [<name>]
|
||||
//
|
||||
// Without a name: run SyncLBStateAll (may remove stale VIPs).
|
||||
// With a name: run SyncLBStateVIP(name) for just that frontend (no removals).
|
||||
syncVPPLBStateName := &Node{
|
||||
Word: "<name>",
|
||||
Help: "Sync a single frontend's VIP to VPP",
|
||||
Dynamic: dynFrontends,
|
||||
Run: runSyncVPPLBState,
|
||||
}
|
||||
syncVPPLBState := &Node{
|
||||
Word: "lbstate",
|
||||
Help: "Sync the VPP load-balancer dataplane from the running config",
|
||||
Run: runSyncVPPLBState,
|
||||
Children: []*Node{syncVPPLBStateName},
|
||||
}
|
||||
syncVPP := &Node{
|
||||
Word: "vpp",
|
||||
Help: "VPP dataplane sync commands",
|
||||
Children: []*Node{syncVPPLBState},
|
||||
}
|
||||
syncNode := &Node{
|
||||
Word: "sync",
|
||||
Help: "Reconcile dataplane state from the running config",
|
||||
Children: []*Node{syncVPP},
|
||||
}
|
||||
|
||||
root.Children = []*Node{show, set, watch, configNode, syncNode, quit, exit}
|
||||
return root
|
||||
}
|
||||
|
||||
@@ -235,6 +263,89 @@ func runShowVPPInfo(ctx context.Context, client grpcapi.MaglevClient, _ []string
|
||||
return w.Flush()
|
||||
}
|
||||
|
||||
func runShowVPPLBState(ctx context.Context, client grpcapi.MaglevClient, _ []string) error {
|
||||
ctx, cancel := context.WithTimeout(ctx, callTimeout)
|
||||
defer cancel()
|
||||
state, err := client.GetVPPLBState(ctx, &grpcapi.GetVPPLBStateRequest{})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// ---- global config ----
|
||||
w := tabwriter.NewWriter(os.Stdout, 0, 0, 2, ' ', 0)
|
||||
fmt.Fprintf(w, "%s\n", label("global"))
|
||||
if state.Conf.Ip4SrcAddress != "" {
|
||||
fmt.Fprintf(w, " %s\t%s\n", label("ip4-src"), state.Conf.Ip4SrcAddress)
|
||||
}
|
||||
if state.Conf.Ip6SrcAddress != "" {
|
||||
fmt.Fprintf(w, " %s\t%s\n", label("ip6-src"), state.Conf.Ip6SrcAddress)
|
||||
}
|
||||
fmt.Fprintf(w, " %s\t%d\n", label("sticky-buckets-per-core"), state.Conf.StickyBucketsPerCore)
|
||||
fmt.Fprintf(w, " %s\t%ds\n", label("flow-timeout"), state.Conf.FlowTimeout)
|
||||
if err := w.Flush(); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if len(state.Vips) == 0 {
|
||||
fmt.Println(label("vips") + " (none)")
|
||||
return nil
|
||||
}
|
||||
|
||||
// ---- per-VIP details ----
|
||||
for _, v := range state.Vips {
|
||||
fmt.Println()
|
||||
w := tabwriter.NewWriter(os.Stdout, 0, 0, 2, ' ', 0)
|
||||
fmt.Fprintf(w, "%s\t%s\n", label("vip"), v.Prefix)
|
||||
fmt.Fprintf(w, " %s\t%s\n", label("protocol"), protoString(v.Protocol))
|
||||
fmt.Fprintf(w, " %s\t%d\n", label("port"), v.Port)
|
||||
fmt.Fprintf(w, " %s\t%s\n", label("encap"), v.Encap)
|
||||
fmt.Fprintf(w, " %s\t%d\n", label("flow-table-length"), v.FlowTableLength)
|
||||
fmt.Fprintf(w, " %s\t%d\n", label("application-servers"), len(v.ApplicationServers))
|
||||
if err := w.Flush(); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, a := range v.ApplicationServers {
|
||||
fmt.Printf(" %s %s %s %d %s %d\n",
|
||||
label("address"), a.Address,
|
||||
label("weight"), a.Weight,
|
||||
label("flow-table-buckets"), a.NumBuckets)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// protoString renders an IP protocol number as a name (tcp, udp, any, or numeric).
|
||||
func protoString(p uint32) string {
|
||||
switch p {
|
||||
case 6:
|
||||
return "tcp"
|
||||
case 17:
|
||||
return "udp"
|
||||
case 255:
|
||||
return "any"
|
||||
}
|
||||
return fmt.Sprintf("%d", p)
|
||||
}
|
||||
|
||||
func runSyncVPPLBState(ctx context.Context, client grpcapi.MaglevClient, args []string) error {
|
||||
ctx, cancel := context.WithTimeout(ctx, callTimeout)
|
||||
defer cancel()
|
||||
req := &grpcapi.SyncVPPLBStateRequest{}
|
||||
if len(args) > 0 && args[0] != "" {
|
||||
name := args[0]
|
||||
req.FrontendName = &name
|
||||
}
|
||||
if _, err := client.SyncVPPLBState(ctx, req); err != nil {
|
||||
return err
|
||||
}
|
||||
if req.FrontendName != nil {
|
||||
fmt.Printf("synced frontend %q to VPP\n", *req.FrontendName)
|
||||
} else {
|
||||
fmt.Println("synced full LB state to VPP")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func runShowVersion(_ context.Context, _ grpcapi.MaglevClient, _ []string) error {
|
||||
fmt.Printf("maglevc %s (commit %s, built %s)\n",
|
||||
buildinfo.Version(), buildinfo.Commit(), buildinfo.Date())
|
||||
|
||||
Reference in New Issue
Block a user