This commit wires maglevd through to VPP's LB plugin end-to-end, using
locally-generated GoVPP bindings for the newer v2 API messages.
VPP binapi (vendored)
- New package internal/vpp/binapi/ containing lb, lb_types, ip_types, and
interface_types, generated from a local VPP build (~/src/vpp) via a new
'make vpp-binapi' target. GoVPP v0.12.0 upstream lacks the v2 messages we
need (lb_conf_get, lb_add_del_vip_v2, lb_add_del_as_v2, lb_as_v2_dump,
lb_as_set_weight), so we commit the generated output in-tree.
- All generated files go through our loggedChannel wrapper; every VPP API
send/receive is recorded at DEBUG via slog (vpp-api-send / vpp-api-recv /
vpp-api-send-multi / vpp-api-recv-multi) so the full wire-level trail is
auditable. NewAPIChannel is unexported — callers must use c.apiChannel().
Read path: GetLBState{All,VIP}
- GetLBStateAll returns a full snapshot (global conf + every VIP with its
attached application servers).
- GetLBStateVIP looks up a single VIP by (prefix, protocol, port) and
returns (nil, nil) when the VIP doesn't exist in VPP. This is the
efficient path for targeted updates on a busy LB.
- Helpers factored out: getLBConf, dumpAllVIPs, dumpASesForVIP, lookupVIP,
vipFromDetails.
Write path: SyncLBState{All,VIP}
- SyncLBStateAll reconciles every configured frontend with VPP: creates
missing VIPs, removes stale ones (with AS flush), and reconciles AS
membership and weights within VIPs that exist on both sides.
- SyncLBStateVIP targets a single frontend by name. Never removes VIPs.
Returns ErrFrontendNotFound (wrapped with the name) when the frontend
isn't in config, so callers can use errors.Is.
- Shared reconcileVIP helper does the per-VIP AS diff; removeVIP is used
only by the full-sync pass.
- LbAddDelVipV2 requests always set NewFlowsTableLength=1024. The .api
default=1024 annotation is only applied by VAT/CLI parsers, not wire-
level marshalling — sending 0 caused VPP to vec_validate with mask
0xFFFFFFFF and OOM-panic.
- Pool semantics: backends in the primary (first) pool of a frontend get
their configured weight; backends in secondary pools get weight 0. All
backends are installed so higher layers can flip weights on failover
without add/remove churn.
- Every individual change emits a DEBUG slog (vpp-lbsync-vip-add/del,
vpp-lbsync-as-add/del, vpp-lbsync-as-weight). Start/done INFO logs
carry a scope=all|vip label plus aggregate counts.
Global conf push: SetLBConf
- New SetLBConf(cfg) sends lb_conf with ipv4-src, ipv6-src, sticky-buckets,
and flow-timeout. Called automatically on VPP (re)connect and after
every config reload (via doReloadConfig). Results are cached on the
Client so redundant pushes are silently skipped — only actual changes
produce a vpp-lb-conf-set INFO log line.
Periodic drift reconciliation
- vpp.Client.lbSyncLoop runs in a goroutine tied to each VPP connection's
lifetime. Its first tick is immediate (startup and post-reconnect
sync quickly); subsequent ticks fire every vpp.lb.sync-interval from
config (default 30s). Purpose: catch drift if something/someone
modifies VPP state by hand. The loop uses a ConfigSource interface
(satisfied by checker.Checker via its new Config() accessor) to avoid
an import cycle with the checker package.
Config schema additions (maglev.vpp.lb)
- sync-interval: positive Go duration, default 30s.
- ipv4-src-address: REQUIRED. Used as the outer source for GRE4 encap
to application servers. Missing this is a hard semantic error —
maglevd --check exits 2 and the daemon refuses to start. VPP GRE
needs a source address and every VIP we program uses GRE, so there
is no meaningful config without it.
- ipv6-src-address: REQUIRED. Same treatment as ipv4-src-address.
- sticky-buckets-per-core: default 65536, must be a power of 2.
- flow-timeout: default 40s, must be a whole number of seconds in [1s, 120s].
- VPP validation runs at the end of convert() so structural errors in
healthchecks/backends/frontends surface first — operators fix those,
then get the VPP-specific requirements.
gRPC API
- New GetVPPLBState RPC returning VPPLBState: global conf + VIPs with
ASes. Mirrors the read-path but strips fields irrelevant to our
GRE-only deployment (srv_type, dscp, target_port).
- New SyncVPPLBState RPC with optional frontend_name. Unset → full sync
(may remove stale VIPs). Set → single-VIP sync (never removes).
Returns codes.NotFound for unknown frontends, codes.Unavailable when
VPP integration is disabled or disconnected.
maglevc (CLI)
- New 'show vpp lbstate' command displaying the LB plugin state. VPP-only
fields the dataplane irrelevant to GRE are suppressed. Per-AS lines use
a key-value format ("address X weight Y flow-table-buckets Z")
instead of a tabwriter column, which avoids the ANSI-color alignment
issue we hit with mixed label/data rows.
- New 'sync vpp lbstate [<name>]' command. Without a name, triggers a
full reconciliation; with a name, targets one frontend.
- Previous 'show vpp lb' renamed to 'show vpp lbstate' for consistency
with the new sync command.
Test fixtures
- validConfig and all ad-hoc config_test.go fixtures that reach the end
of convert() now include the two required vpp.lb src addresses.
- tests/01-maglevd/maglevd-lab/maglev.yaml gains a vpp.lb section so the
robot integration tests can still load the config.
- cmd/maglevc/tree_test.go gains expected paths for the new commands.
Docs
- config-guide.md: new 'vpp' section in the basic structure, detailed
vpp.lb field reference, noting ipv4/ipv6 src addresses as REQUIRED
(hard error) with no defaults; example config updated.
- user-guide.md: documented 'show vpp info', 'show vpp lbstate',
'sync vpp lbstate [<name>]', new --vpp-api-addr and --vpp-stats-addr
flags, the vpp-lb-conf-set log line, and corrected the pause/resume
description to reflect that pause cancels the probe goroutine.
- debian/maglev.yaml: example config gains a vpp.lb block with src
addresses and commented optional overrides.
258 lines
7.1 KiB
Protocol Buffer
258 lines
7.1 KiB
Protocol Buffer
syntax = "proto3";
|
|
|
|
package maglev;
|
|
|
|
option go_package = "git.ipng.ch/ipng/vpp-maglev/internal/grpcapi";
|
|
|
|
// Maglev exposes the state of backend health for all frontends.
|
|
service Maglev {
|
|
rpc ListFrontends(ListFrontendsRequest) returns (ListFrontendsResponse);
|
|
rpc GetFrontend(GetFrontendRequest) returns (FrontendInfo);
|
|
rpc ListBackends(ListBackendsRequest) returns (ListBackendsResponse);
|
|
rpc GetBackend(GetBackendRequest) returns (BackendInfo);
|
|
rpc PauseBackend(BackendRequest) returns (BackendInfo);
|
|
rpc ResumeBackend(BackendRequest) returns (BackendInfo);
|
|
rpc EnableBackend(BackendRequest) returns (BackendInfo);
|
|
rpc DisableBackend(BackendRequest) returns (BackendInfo);
|
|
rpc ListHealthChecks(ListHealthChecksRequest) returns (ListHealthChecksResponse);
|
|
rpc GetHealthCheck(GetHealthCheckRequest) returns (HealthCheckInfo);
|
|
rpc SetFrontendPoolBackendWeight(SetWeightRequest) returns (FrontendInfo);
|
|
rpc WatchEvents(WatchRequest) returns (stream Event);
|
|
rpc CheckConfig(CheckConfigRequest) returns (CheckConfigResponse);
|
|
rpc ReloadConfig(ReloadConfigRequest) returns (ReloadConfigResponse);
|
|
rpc GetVPPInfo(GetVPPInfoRequest) returns (VPPInfo);
|
|
rpc GetVPPLBState(GetVPPLBStateRequest) returns (VPPLBState);
|
|
rpc SyncVPPLBState(SyncVPPLBStateRequest) returns (SyncVPPLBStateResponse);
|
|
}
|
|
|
|
// ---- requests ---------------------------------------------------------------
|
|
|
|
message ListFrontendsRequest {}
|
|
|
|
message GetFrontendRequest {
|
|
string name = 1;
|
|
}
|
|
|
|
message ListBackendsRequest {}
|
|
|
|
message GetBackendRequest {
|
|
string name = 1;
|
|
}
|
|
|
|
message BackendRequest {
|
|
string name = 1;
|
|
}
|
|
|
|
message ListHealthChecksRequest {}
|
|
|
|
message GetHealthCheckRequest {
|
|
string name = 1;
|
|
}
|
|
|
|
message CheckConfigRequest {}
|
|
|
|
message CheckConfigResponse {
|
|
bool ok = 1;
|
|
string parse_error = 2; // set when YAML cannot be read or parsed
|
|
string semantic_error = 3; // set when YAML is valid but semantically incorrect
|
|
}
|
|
|
|
message ReloadConfigRequest {}
|
|
|
|
message ReloadConfigResponse {
|
|
bool ok = 1;
|
|
string parse_error = 2; // set when YAML cannot be read or parsed
|
|
string semantic_error = 3; // set when YAML is valid but semantically incorrect
|
|
string reload_error = 4; // set when config is valid but the reload itself failed
|
|
}
|
|
|
|
message GetVPPInfoRequest {}
|
|
|
|
message VPPInfo {
|
|
string version = 1;
|
|
string build_date = 2;
|
|
string build_directory = 3;
|
|
uint32 pid = 4;
|
|
int64 boottime_ns = 5; // unix timestamp (ns) when VPP started (from /sys/boottime)
|
|
int64 connecttime_ns = 6; // unix timestamp (ns) when maglevd connected to VPP
|
|
}
|
|
|
|
// ---- VPP load-balancer state ------------------------------------------------
|
|
|
|
message GetVPPLBStateRequest {}
|
|
|
|
// VPPLBConf mirrors VPP's lb_conf_get_reply: global LB plugin settings.
|
|
message VPPLBConf {
|
|
string ip4_src_address = 1;
|
|
string ip6_src_address = 2;
|
|
uint32 sticky_buckets_per_core = 3;
|
|
uint32 flow_timeout = 4;
|
|
}
|
|
|
|
// VPPLBAS is one application server attached to a VIP.
|
|
message VPPLBAS {
|
|
string address = 1;
|
|
uint32 weight = 2; // 0-100
|
|
uint32 flags = 3; // VPP AS flags (bit 0 = used, bit 1 = flushed)
|
|
uint32 num_buckets = 4;
|
|
int64 in_use_since_ns = 5; // unix timestamp (ns), 0 if never used
|
|
}
|
|
|
|
// VPPLBVIP mirrors VPP's lb_vip_details plus the attached application servers.
|
|
// Note: srv_type, dscp, and target_port are intentionally omitted — maglevd
|
|
// only supports GRE encap, so NAT/L3DSR-specific fields don't apply.
|
|
message VPPLBVIP {
|
|
string prefix = 1; // CIDR, e.g. 192.0.2.1/32
|
|
uint32 protocol = 2; // 6=TCP, 17=UDP, 255=any
|
|
uint32 port = 3; // 0 = all-port VIP
|
|
string encap = 4; // gre4|gre6|l3dsr|nat4|nat6
|
|
uint32 flow_table_length = 5;
|
|
repeated VPPLBAS application_servers = 6;
|
|
}
|
|
|
|
message VPPLBState {
|
|
VPPLBConf conf = 1;
|
|
repeated VPPLBVIP vips = 2;
|
|
}
|
|
|
|
// SyncVPPLBStateRequest triggers a reconciliation between the maglev config
|
|
// and the VPP load-balancer dataplane. When frontend_name is set, only that
|
|
// frontend's VIP is synced (SyncLBStateVIP) and no VIPs are removed. When
|
|
// unset, a full reconciliation runs (SyncLBStateAll), which will also remove
|
|
// stale VIPs from VPP.
|
|
message SyncVPPLBStateRequest {
|
|
optional string frontend_name = 1;
|
|
}
|
|
|
|
message SyncVPPLBStateResponse {}
|
|
|
|
message SetWeightRequest {
|
|
string frontend = 1;
|
|
string pool = 2;
|
|
string backend = 3;
|
|
int32 weight = 4; // 0-100
|
|
}
|
|
|
|
// WatchRequest controls which event types are streamed. All fields default to
|
|
// true (i.e. an empty request subscribes to everything at info level).
|
|
message WatchRequest {
|
|
optional bool log = 1; // include log events (default: true)
|
|
string log_level = 2; // minimum log level: debug|info|warn|error (default: info)
|
|
optional bool backend = 3; // include backend transition events (default: true)
|
|
optional bool frontend = 4; // include frontend events (default: true)
|
|
}
|
|
|
|
// ---- responses --------------------------------------------------------------
|
|
|
|
message ListFrontendsResponse {
|
|
repeated string frontend_names = 1;
|
|
}
|
|
|
|
message PoolBackendInfo {
|
|
string name = 1;
|
|
int32 weight = 2;
|
|
}
|
|
|
|
message PoolInfo {
|
|
string name = 1;
|
|
repeated PoolBackendInfo backends = 2;
|
|
}
|
|
|
|
message FrontendInfo {
|
|
string name = 1;
|
|
string address = 2;
|
|
string protocol = 3;
|
|
uint32 port = 4;
|
|
repeated PoolInfo pools = 5;
|
|
string description = 6;
|
|
}
|
|
|
|
message ListBackendsResponse {
|
|
repeated string backend_names = 1;
|
|
}
|
|
|
|
message ListHealthChecksResponse {
|
|
repeated string names = 1;
|
|
}
|
|
|
|
message HTTPCheckParams {
|
|
string path = 1;
|
|
string host = 2;
|
|
int32 response_code_min = 3;
|
|
int32 response_code_max = 4;
|
|
string response_regexp = 5;
|
|
string server_name = 6;
|
|
bool insecure_skip_verify = 7;
|
|
}
|
|
|
|
message TCPCheckParams {
|
|
bool ssl = 1;
|
|
string server_name = 2;
|
|
bool insecure_skip_verify = 3;
|
|
}
|
|
|
|
message HealthCheckInfo {
|
|
string name = 1;
|
|
string type = 2;
|
|
uint32 port = 3;
|
|
string probe_ipv4_src = 4;
|
|
string probe_ipv6_src = 5;
|
|
int64 interval_ns = 6;
|
|
int64 fast_interval_ns = 7;
|
|
int64 down_interval_ns = 8;
|
|
int64 timeout_ns = 9;
|
|
int32 rise = 10;
|
|
int32 fall = 11;
|
|
HTTPCheckParams http = 12;
|
|
TCPCheckParams tcp = 13;
|
|
}
|
|
|
|
message BackendInfo {
|
|
string name = 1;
|
|
string address = 2;
|
|
string state = 3;
|
|
repeated TransitionRecord transitions = 4;
|
|
bool enabled = 5;
|
|
string healthcheck = 6;
|
|
}
|
|
|
|
message TransitionRecord {
|
|
string from = 1;
|
|
string to = 2;
|
|
int64 at_unix_ns = 3;
|
|
}
|
|
|
|
// ---- event stream -----------------------------------------------------------
|
|
|
|
// LogAttr is a single key/value attribute from a structured log record.
|
|
message LogAttr {
|
|
string key = 1;
|
|
string value = 2;
|
|
}
|
|
|
|
// LogEvent carries a single structured log record.
|
|
message LogEvent {
|
|
int64 at_unix_ns = 1;
|
|
string level = 2;
|
|
string msg = 3;
|
|
repeated LogAttr attrs = 4;
|
|
}
|
|
|
|
// BackendEvent is emitted on every backend state transition.
|
|
message BackendEvent {
|
|
string backend_name = 1;
|
|
TransitionRecord transition = 2;
|
|
}
|
|
|
|
// FrontendEvent is reserved for future frontend-level events.
|
|
message FrontendEvent {}
|
|
|
|
// Event is the envelope returned by WatchEvents.
|
|
message Event {
|
|
oneof event {
|
|
LogEvent log = 1;
|
|
BackendEvent backend = 2;
|
|
FrontendEvent frontend = 3;
|
|
}
|
|
}
|