VPP load-balancer dataplane integration: state, sync, and global conf
This commit wires maglevd through to VPP's LB plugin end-to-end, using
locally-generated GoVPP bindings for the newer v2 API messages.
VPP binapi (vendored)
- New package internal/vpp/binapi/ containing lb, lb_types, ip_types, and
interface_types, generated from a local VPP build (~/src/vpp) via a new
'make vpp-binapi' target. GoVPP v0.12.0 upstream lacks the v2 messages we
need (lb_conf_get, lb_add_del_vip_v2, lb_add_del_as_v2, lb_as_v2_dump,
lb_as_set_weight), so we commit the generated output in-tree.
- All generated files go through our loggedChannel wrapper; every VPP API
send/receive is recorded at DEBUG via slog (vpp-api-send / vpp-api-recv /
vpp-api-send-multi / vpp-api-recv-multi) so the full wire-level trail is
auditable. NewAPIChannel is unexported — callers must use c.apiChannel().
Read path: GetLBState{All,VIP}
- GetLBStateAll returns a full snapshot (global conf + every VIP with its
attached application servers).
- GetLBStateVIP looks up a single VIP by (prefix, protocol, port) and
returns (nil, nil) when the VIP doesn't exist in VPP. This is the
efficient path for targeted updates on a busy LB.
- Helpers factored out: getLBConf, dumpAllVIPs, dumpASesForVIP, lookupVIP,
vipFromDetails.
Write path: SyncLBState{All,VIP}
- SyncLBStateAll reconciles every configured frontend with VPP: creates
missing VIPs, removes stale ones (with AS flush), and reconciles AS
membership and weights within VIPs that exist on both sides.
- SyncLBStateVIP targets a single frontend by name. Never removes VIPs.
Returns ErrFrontendNotFound (wrapped with the name) when the frontend
isn't in config, so callers can use errors.Is.
- Shared reconcileVIP helper does the per-VIP AS diff; removeVIP is used
only by the full-sync pass.
- LbAddDelVipV2 requests always set NewFlowsTableLength=1024. The .api
default=1024 annotation is only applied by VAT/CLI parsers, not wire-
level marshalling — sending 0 caused VPP to vec_validate with mask
0xFFFFFFFF and OOM-panic.
- Pool semantics: backends in the primary (first) pool of a frontend get
their configured weight; backends in secondary pools get weight 0. All
backends are installed so higher layers can flip weights on failover
without add/remove churn.
- Every individual change emits a DEBUG slog (vpp-lbsync-vip-add/del,
vpp-lbsync-as-add/del, vpp-lbsync-as-weight). Start/done INFO logs
carry a scope=all|vip label plus aggregate counts.
Global conf push: SetLBConf
- New SetLBConf(cfg) sends lb_conf with ipv4-src, ipv6-src, sticky-buckets,
and flow-timeout. Called automatically on VPP (re)connect and after
every config reload (via doReloadConfig). Results are cached on the
Client so redundant pushes are silently skipped — only actual changes
produce a vpp-lb-conf-set INFO log line.
Periodic drift reconciliation
- vpp.Client.lbSyncLoop runs in a goroutine tied to each VPP connection's
lifetime. Its first tick is immediate (startup and post-reconnect
sync quickly); subsequent ticks fire every vpp.lb.sync-interval from
config (default 30s). Purpose: catch drift if something/someone
modifies VPP state by hand. The loop uses a ConfigSource interface
(satisfied by checker.Checker via its new Config() accessor) to avoid
an import cycle with the checker package.
Config schema additions (maglev.vpp.lb)
- sync-interval: positive Go duration, default 30s.
- ipv4-src-address: REQUIRED. Used as the outer source for GRE4 encap
to application servers. Missing this is a hard semantic error —
maglevd --check exits 2 and the daemon refuses to start. VPP GRE
needs a source address and every VIP we program uses GRE, so there
is no meaningful config without it.
- ipv6-src-address: REQUIRED. Same treatment as ipv4-src-address.
- sticky-buckets-per-core: default 65536, must be a power of 2.
- flow-timeout: default 40s, must be a whole number of seconds in [1s, 120s].
- VPP validation runs at the end of convert() so structural errors in
healthchecks/backends/frontends surface first — operators fix those,
then get the VPP-specific requirements.
gRPC API
- New GetVPPLBState RPC returning VPPLBState: global conf + VIPs with
ASes. Mirrors the read-path but strips fields irrelevant to our
GRE-only deployment (srv_type, dscp, target_port).
- New SyncVPPLBState RPC with optional frontend_name. Unset → full sync
(may remove stale VIPs). Set → single-VIP sync (never removes).
Returns codes.NotFound for unknown frontends, codes.Unavailable when
VPP integration is disabled or disconnected.
maglevc (CLI)
- New 'show vpp lbstate' command displaying the LB plugin state. VPP-only
fields the dataplane irrelevant to GRE are suppressed. Per-AS lines use
a key-value format ("address X weight Y flow-table-buckets Z")
instead of a tabwriter column, which avoids the ANSI-color alignment
issue we hit with mixed label/data rows.
- New 'sync vpp lbstate [<name>]' command. Without a name, triggers a
full reconciliation; with a name, targets one frontend.
- Previous 'show vpp lb' renamed to 'show vpp lbstate' for consistency
with the new sync command.
Test fixtures
- validConfig and all ad-hoc config_test.go fixtures that reach the end
of convert() now include the two required vpp.lb src addresses.
- tests/01-maglevd/maglevd-lab/maglev.yaml gains a vpp.lb section so the
robot integration tests can still load the config.
- cmd/maglevc/tree_test.go gains expected paths for the new commands.
Docs
- config-guide.md: new 'vpp' section in the basic structure, detailed
vpp.lb field reference, noting ipv4/ipv6 src addresses as REQUIRED
(hard error) with no defaults; example config updated.
- user-guide.md: documented 'show vpp info', 'show vpp lbstate',
'sync vpp lbstate [<name>]', new --vpp-api-addr and --vpp-stats-addr
flags, the vpp-lb-conf-set log line, and corrected the pause/resume
description to reflect that pause cancels the probe goroutine.
- debian/maglev.yaml: example config gains a vpp.lb block with src
addresses and commented optional overrides.
This commit is contained in:
File diff suppressed because it is too large
Load Diff
@@ -34,6 +34,8 @@ const (
|
||||
Maglev_CheckConfig_FullMethodName = "/maglev.Maglev/CheckConfig"
|
||||
Maglev_ReloadConfig_FullMethodName = "/maglev.Maglev/ReloadConfig"
|
||||
Maglev_GetVPPInfo_FullMethodName = "/maglev.Maglev/GetVPPInfo"
|
||||
Maglev_GetVPPLBState_FullMethodName = "/maglev.Maglev/GetVPPLBState"
|
||||
Maglev_SyncVPPLBState_FullMethodName = "/maglev.Maglev/SyncVPPLBState"
|
||||
)
|
||||
|
||||
// MaglevClient is the client API for Maglev service.
|
||||
@@ -57,6 +59,8 @@ type MaglevClient interface {
|
||||
CheckConfig(ctx context.Context, in *CheckConfigRequest, opts ...grpc.CallOption) (*CheckConfigResponse, error)
|
||||
ReloadConfig(ctx context.Context, in *ReloadConfigRequest, opts ...grpc.CallOption) (*ReloadConfigResponse, error)
|
||||
GetVPPInfo(ctx context.Context, in *GetVPPInfoRequest, opts ...grpc.CallOption) (*VPPInfo, error)
|
||||
GetVPPLBState(ctx context.Context, in *GetVPPLBStateRequest, opts ...grpc.CallOption) (*VPPLBState, error)
|
||||
SyncVPPLBState(ctx context.Context, in *SyncVPPLBStateRequest, opts ...grpc.CallOption) (*SyncVPPLBStateResponse, error)
|
||||
}
|
||||
|
||||
type maglevClient struct {
|
||||
@@ -226,6 +230,26 @@ func (c *maglevClient) GetVPPInfo(ctx context.Context, in *GetVPPInfoRequest, op
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (c *maglevClient) GetVPPLBState(ctx context.Context, in *GetVPPLBStateRequest, opts ...grpc.CallOption) (*VPPLBState, error) {
|
||||
cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
|
||||
out := new(VPPLBState)
|
||||
err := c.cc.Invoke(ctx, Maglev_GetVPPLBState_FullMethodName, in, out, cOpts...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (c *maglevClient) SyncVPPLBState(ctx context.Context, in *SyncVPPLBStateRequest, opts ...grpc.CallOption) (*SyncVPPLBStateResponse, error) {
|
||||
cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
|
||||
out := new(SyncVPPLBStateResponse)
|
||||
err := c.cc.Invoke(ctx, Maglev_SyncVPPLBState_FullMethodName, in, out, cOpts...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// MaglevServer is the server API for Maglev service.
|
||||
// All implementations must embed UnimplementedMaglevServer
|
||||
// for forward compatibility.
|
||||
@@ -247,6 +271,8 @@ type MaglevServer interface {
|
||||
CheckConfig(context.Context, *CheckConfigRequest) (*CheckConfigResponse, error)
|
||||
ReloadConfig(context.Context, *ReloadConfigRequest) (*ReloadConfigResponse, error)
|
||||
GetVPPInfo(context.Context, *GetVPPInfoRequest) (*VPPInfo, error)
|
||||
GetVPPLBState(context.Context, *GetVPPLBStateRequest) (*VPPLBState, error)
|
||||
SyncVPPLBState(context.Context, *SyncVPPLBStateRequest) (*SyncVPPLBStateResponse, error)
|
||||
mustEmbedUnimplementedMaglevServer()
|
||||
}
|
||||
|
||||
@@ -302,6 +328,12 @@ func (UnimplementedMaglevServer) ReloadConfig(context.Context, *ReloadConfigRequ
|
||||
func (UnimplementedMaglevServer) GetVPPInfo(context.Context, *GetVPPInfoRequest) (*VPPInfo, error) {
|
||||
return nil, status.Error(codes.Unimplemented, "method GetVPPInfo not implemented")
|
||||
}
|
||||
func (UnimplementedMaglevServer) GetVPPLBState(context.Context, *GetVPPLBStateRequest) (*VPPLBState, error) {
|
||||
return nil, status.Error(codes.Unimplemented, "method GetVPPLBState not implemented")
|
||||
}
|
||||
func (UnimplementedMaglevServer) SyncVPPLBState(context.Context, *SyncVPPLBStateRequest) (*SyncVPPLBStateResponse, error) {
|
||||
return nil, status.Error(codes.Unimplemented, "method SyncVPPLBState not implemented")
|
||||
}
|
||||
func (UnimplementedMaglevServer) mustEmbedUnimplementedMaglevServer() {}
|
||||
func (UnimplementedMaglevServer) testEmbeddedByValue() {}
|
||||
|
||||
@@ -586,6 +618,42 @@ func _Maglev_GetVPPInfo_Handler(srv interface{}, ctx context.Context, dec func(i
|
||||
return interceptor(ctx, in, info, handler)
|
||||
}
|
||||
|
||||
func _Maglev_GetVPPLBState_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
|
||||
in := new(GetVPPLBStateRequest)
|
||||
if err := dec(in); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if interceptor == nil {
|
||||
return srv.(MaglevServer).GetVPPLBState(ctx, in)
|
||||
}
|
||||
info := &grpc.UnaryServerInfo{
|
||||
Server: srv,
|
||||
FullMethod: Maglev_GetVPPLBState_FullMethodName,
|
||||
}
|
||||
handler := func(ctx context.Context, req interface{}) (interface{}, error) {
|
||||
return srv.(MaglevServer).GetVPPLBState(ctx, req.(*GetVPPLBStateRequest))
|
||||
}
|
||||
return interceptor(ctx, in, info, handler)
|
||||
}
|
||||
|
||||
func _Maglev_SyncVPPLBState_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
|
||||
in := new(SyncVPPLBStateRequest)
|
||||
if err := dec(in); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if interceptor == nil {
|
||||
return srv.(MaglevServer).SyncVPPLBState(ctx, in)
|
||||
}
|
||||
info := &grpc.UnaryServerInfo{
|
||||
Server: srv,
|
||||
FullMethod: Maglev_SyncVPPLBState_FullMethodName,
|
||||
}
|
||||
handler := func(ctx context.Context, req interface{}) (interface{}, error) {
|
||||
return srv.(MaglevServer).SyncVPPLBState(ctx, req.(*SyncVPPLBStateRequest))
|
||||
}
|
||||
return interceptor(ctx, in, info, handler)
|
||||
}
|
||||
|
||||
// Maglev_ServiceDesc is the grpc.ServiceDesc for Maglev service.
|
||||
// It's only intended for direct use with grpc.RegisterService,
|
||||
// and not to be introspected or modified (even as a copy)
|
||||
@@ -649,6 +717,14 @@ var Maglev_ServiceDesc = grpc.ServiceDesc{
|
||||
MethodName: "GetVPPInfo",
|
||||
Handler: _Maglev_GetVPPInfo_Handler,
|
||||
},
|
||||
{
|
||||
MethodName: "GetVPPLBState",
|
||||
Handler: _Maglev_GetVPPLBState_Handler,
|
||||
},
|
||||
{
|
||||
MethodName: "SyncVPPLBState",
|
||||
Handler: _Maglev_SyncVPPLBState_Handler,
|
||||
},
|
||||
},
|
||||
Streams: []grpc.StreamDesc{
|
||||
{
|
||||
|
||||
@@ -4,6 +4,7 @@ package grpcapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"net"
|
||||
|
||||
@@ -255,6 +256,13 @@ func (s *Server) doReloadConfig() *ReloadConfigResponse {
|
||||
ReloadError: err.Error(),
|
||||
}
|
||||
}
|
||||
// Push new global LB conf to VPP if anything changed. SetLBConf is a
|
||||
// no-op when VPP isn't connected or when the values are unchanged.
|
||||
if s.vppClient != nil {
|
||||
if err := s.vppClient.SetLBConf(newCfg); err != nil {
|
||||
slog.Warn("vpp-lb-conf-set-failed", "err", err)
|
||||
}
|
||||
}
|
||||
slog.Info("config-reload-done", "frontends", len(newCfg.Frontends))
|
||||
return &ReloadConfigResponse{Ok: true}
|
||||
}
|
||||
@@ -282,6 +290,84 @@ func (s *Server) GetVPPInfo(_ context.Context, _ *GetVPPInfoRequest) (*VPPInfo,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// GetVPPLBState returns a snapshot of the VPP load-balancer plugin state.
|
||||
func (s *Server) GetVPPLBState(_ context.Context, _ *GetVPPLBStateRequest) (*VPPLBState, error) {
|
||||
if s.vppClient == nil {
|
||||
return nil, status.Error(codes.Unavailable, "VPP integration is disabled")
|
||||
}
|
||||
state, err := s.vppClient.GetLBStateAll()
|
||||
if err != nil {
|
||||
return nil, status.Errorf(codes.Unavailable, "%v", err)
|
||||
}
|
||||
return lbStateToProto(state), nil
|
||||
}
|
||||
|
||||
// SyncVPPLBState runs the LB reconciler. With frontend_name unset it does a
|
||||
// full sync (SyncLBStateAll), which may remove stale VIPs. With frontend_name
|
||||
// set it does a single-VIP sync (SyncLBStateVIP) that only adds/updates.
|
||||
func (s *Server) SyncVPPLBState(_ context.Context, req *SyncVPPLBStateRequest) (*SyncVPPLBStateResponse, error) {
|
||||
if s.vppClient == nil {
|
||||
return nil, status.Error(codes.Unavailable, "VPP integration is disabled")
|
||||
}
|
||||
cfg := s.checker.Config()
|
||||
if req.FrontendName != nil && *req.FrontendName != "" {
|
||||
if err := s.vppClient.SyncLBStateVIP(cfg, *req.FrontendName); err != nil {
|
||||
if errors.Is(err, vpp.ErrFrontendNotFound) {
|
||||
return nil, status.Errorf(codes.NotFound, "%v", err)
|
||||
}
|
||||
return nil, status.Errorf(codes.Unavailable, "%v", err)
|
||||
}
|
||||
return &SyncVPPLBStateResponse{}, nil
|
||||
}
|
||||
if err := s.vppClient.SyncLBStateAll(cfg); err != nil {
|
||||
return nil, status.Errorf(codes.Unavailable, "%v", err)
|
||||
}
|
||||
return &SyncVPPLBStateResponse{}, nil
|
||||
}
|
||||
|
||||
// lbStateToProto converts the vpp package's LBState into the proto message.
|
||||
func lbStateToProto(s *vpp.LBState) *VPPLBState {
|
||||
out := &VPPLBState{
|
||||
Conf: &VPPLBConf{
|
||||
Ip4SrcAddress: ipStringOrEmpty(s.Conf.IP4SrcAddress),
|
||||
Ip6SrcAddress: ipStringOrEmpty(s.Conf.IP6SrcAddress),
|
||||
StickyBucketsPerCore: s.Conf.StickyBucketsPerCore,
|
||||
FlowTimeout: s.Conf.FlowTimeout,
|
||||
},
|
||||
}
|
||||
for _, v := range s.VIPs {
|
||||
pv := &VPPLBVIP{
|
||||
Prefix: v.Prefix.String(),
|
||||
Protocol: uint32(v.Protocol),
|
||||
Port: uint32(v.Port),
|
||||
Encap: v.Encap,
|
||||
FlowTableLength: uint32(v.FlowTableLength),
|
||||
}
|
||||
for _, a := range v.ASes {
|
||||
var ts int64
|
||||
if !a.InUseSince.IsZero() {
|
||||
ts = a.InUseSince.UnixNano()
|
||||
}
|
||||
pv.ApplicationServers = append(pv.ApplicationServers, &VPPLBAS{
|
||||
Address: a.Address.String(),
|
||||
Weight: uint32(a.Weight),
|
||||
Flags: uint32(a.Flags),
|
||||
NumBuckets: a.NumBuckets,
|
||||
InUseSinceNs: ts,
|
||||
})
|
||||
}
|
||||
out.Vips = append(out.Vips, pv)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func ipStringOrEmpty(ip net.IP) string {
|
||||
if len(ip) == 0 || ip.IsUnspecified() {
|
||||
return ""
|
||||
}
|
||||
return ip.String()
|
||||
}
|
||||
|
||||
// ---- conversion helpers ----------------------------------------------------
|
||||
|
||||
func frontendToProto(name string, fe config.Frontend) *FrontendInfo {
|
||||
|
||||
Reference in New Issue
Block a user