Add a few hints based on previous issues filed in this repo.
Clarify that linux-cp must be used, the API/Stats socket must be accessible, and that no backwards compatibility is given.
Previous logging was very noisy when the agent connection to snmpd
drops:
[ERROR ] agentx.network - run : Empty PDU, connection closed!
[INFO ] agentx.network - disconnect : Disconnecting from localhost:705
[ERROR ] agentx.agent - run : An exception occurred: Empty PDU, disconnecting
[ERROR ] agentx.agent - run : Reconnecting
[INFO ] agentx.agent - run : Opening AgentX connection
[INFO ] agentx.network - connect : Connecting to localhost:705
[ERROR ] agentx.network - connect : Failed to connect to localhost:705
[ERROR ] agentx.agent - run : An exception occurred: Not connected
[ERROR ] agentx.agent - run : Reconnecting
[INFO ] agentx.agent - run : Opening AgentX connection
[INFO ] agentx.network - connect : Connecting to localhost:705
[ERROR ] agentx.network - connect : Failed to connect to localhost:705
[ERROR ] agentx.agent - run : An exception occurred: Not connected
[ERROR ] agentx.agent - run : Reconnecting
Also, reconnects were attempted every 0.1s, but field research shows
that snmpd, if it restarts, takes ~3-5 seconds to come back (note: this
is also due to a systemd delay in restarting it upon failures).
Hammering the connection is not useful.
This change refactors the logging, to avoid redundant messages:
- sleep 1s between attempts (reducing the loop by 10x)
- Either print 'Connected to' or 'Failed to connect to', not both.
- Remove the 'reconnecting' superfluous message
agentx/network.py always turned on debugging. It can be useful to have
debugging logs of the main application without the agentx debug logs, as
they are quite noisy.
Now, ./vpp-snmp-agent.py -d will turn on application debugging but NOT
agentx debugging. ./vpp-snmp-agent.py -d -dd will turn on both.
NOTE: ./vpp-snmp-agent.py -dd will do nothing, because the '-d' flag
determines the global logging level.
If VPP were to disconnect either the Stats Segment or the API endpoint,
for example if it crashes and restarts, vpp-snmp-agent will not detect
this. In such a situation, it will hold on to the stale stats and no
longer receive interface updates.
Before each run, send a control_ping() API request, and if that were to
fail (for example with Broken Pipe, or Connection Refused), disconnect
both API and Stats (in the vpp.disconnect() call, also invalidate the interface
and LCP cache), and then fail the update. The Agent runner will then retry
once per second until the connection (and control_ping()) succeeds.
TESTED:
- Start vpp-snmp-agent, it connects and starts up per normal.
- Exit / Kill vpp
- Upon the next update(), the control_ping() call will fail, causing the
agent to disconnect
- The agent will now loop:
[ERROR ] agentx.agent - update : VPP API: [Errno 1] Sendall error: BrokenPipeError(32, 'Broken pipe'), retrying
[WARNING ] agentx.agent - run : Update failed, last successful update was 1673345631.7658572
[INFO ] agentx.vppapi - connect : Connecting to VPP
[ERROR ] agentx.agent - update : VPP API: Not connected, api definitions not available, retrying
- Start VPP again, when its API endpoint is ready:
[INFO ] agentx.vppapi - connect : Connecting to VPP
[INFO ] agentx.vppapi - connect : VPP version is 23.02-rc0~199-gcfaf44020
[INFO ] agentx.vppapi - connect : Enabling VPP API interface events
[DEBUG ] agentx.agent - update : VPP API: control_ping_reply(_0=24, context=12, retval=0, client_index=0, vpe_pid=705326)
[INFO ] agentx.vppapi - get_ifaces : Requesting interfaces from VPP API
[INFO ] agentx.vppapi - get_lcp : Requesting LCPs from VPP API
- The agent resumes where it left off
- Set an initial vppapi.iface_dict and lcp_dict to None.
- Set an event watcher API call, with a callback
- When events happen, flush the iface/lcp cache (by setting them to None).
- When get_ifaces / get_lcp sees an empty cache, fetch the data from VPP
API and put into the cache for subsequent calls.
This way, the VPP API is only used upon startup (when the caches are
empty), and on interface add/del/changes (note: the events fire for
link, and admin up/down, but not for MTU changes).
One small race condition exists: if a new LCP is created, this does not
trigger an interface event. Adding a want_lcp_events() makes sense, but
until then, a few options remain:
0) race exists only if inerface was created; THEN the cache was
refreshed; and THEN the LCP was created.
1) create the lcp and then force a change to any interface (this will
create an sw_interface event and flush the cache)
2) restart vpp-snmp-agent
A simple convenience configfile can provide a mapping between VPP
interface names, Linux Control Plane interface names, and descriptions.
An example:
```
interfaces:
"TenGigabitEthernet6/0/0":
description: "Infra: xsw0.chrma0:2"
lcp: "xe1-0"
"TenGigabitEthernet6/0/0.3102":
description: "Infra: QinQ to Solnet for Daedalean"
lcp: "xe1-0.3102"
"TenGigabitEthernet6/0/0.310211":
description: "Cust: Daedalean IP Transit"
lcp: "xe1-0.3102.11"
```
This configuration file is completely optional. If the `-c` flag is
empty, or it's set but the file does not exist, the Agent will simply
enumerate all interfaces, and set the `ifAlias` OID to the same value
as the `ifName`. However, if the config file is read, it will change
the behavior as follows:
* Any `tapNN` interface names from VPP will be matched to their PHY by
looking up their Linux Control Plane interface. The `ifName` field
will be rewritten to the _LIP_ `host-if`. For example, `tap3` above
will become `xe1-0` while `tap3.310211` will become `xe1-0.3102.11`.
* The `ifAlias` OID for a PHY will be set to the `description` field.
* The `ifAlias` OID for a TAP will be set to the string `LCP: `
followed by its PHY `ifName`. For example, `xe1-0.3102.11` will
become `LCP TenGigabitEthernet6/0/0.310211 (tap9)`
When using SNMP BULK GET requests (from Zabbix in our case), the default value of 1024 truncates the request, resulting in malformed requests reaching the agent. Using an 8K buffer fixes this. A better approach perhaps would be to process the buffer using a loop.
Now that we're explicitly connecting via TCP to localhost:705 (which
can be overriden by the -a flag), we no longer need to run as root.
Therefore, update vpp-snmp-agent.service to run as user Debian-snmp
group vpp, so that /run/vpp/{api,stats}.sock are writable.
Be explicit on the commandline arguments in the service definition.
It now is tolerant to VPP restarts. Upon initialization, we connect(),
blocking all but the first thread from trying. The rest will see
self.connected=True and move on.
Then, on each/any error, call vpp.disconect() and set connected=False
which will make any subsequent AgentX updater run force a reconnect.
