Pim van Pelt
fdb77838b8
All checks were successful
continuous-integration/drone/push Build is passing
412 lines
18 KiB
Markdown
412 lines
18 KiB
Markdown
---
|
|
date: "2021-09-10T13:21:14Z"
|
|
title: VPP Linux CP - Part6
|
|
aliases:
|
|
- /s/articles/2021/09/11/vpp-6.html
|
|
---
|
|
|
|
|
|
{{< image width="200px" float="right" src="/assets/vpp/fdio-color.svg" alt="VPP" >}}
|
|
|
|
# About this series
|
|
|
|
Ever since I first saw VPP - the Vector Packet Processor - I have been deeply impressed with its
|
|
performance and versatility. For those of us who have used Cisco IOS/XR devices, like the classic
|
|
_ASR_ (aggregation services router), VPP will look and feel quite familiar as many of the approaches
|
|
are shared between the two. One thing notably missing, is the higher level control plane, that is
|
|
to say: there is no OSPF or ISIS, BGP, LDP and the like. This series of posts details my work on a
|
|
VPP _plugin_ which is called the **Linux Control Plane**, or LCP for short, which creates Linux network
|
|
devices that mirror their VPP dataplane counterpart. IPv4 and IPv6 traffic, and associated protocols
|
|
like ARP and IPv6 Neighbor Discovery can now be handled by Linux, while the heavy lifting of packet
|
|
forwarding is done by the VPP dataplane. Or, said another way: this plugin will allow Linux to use
|
|
VPP as a software ASIC for fast forwarding, filtering, NAT, and so on, while keeping control of the
|
|
interface state (links, addresses and routes) itself. When the plugin is completed, running software
|
|
like [FRR](https://frrouting.org/) or [Bird](https://bird.network.cz/) on top of VPP and achieving
|
|
>100Mpps and >100Gbps forwarding rates will be well in reach!
|
|
|
|
## SNMP in VPP
|
|
|
|
Now that the **Interface Mirror** and **Netlink Listener** plugins are in good shape, this post
|
|
shows a few finishing touches. First off, although the native habitat of VPP is [Prometheus](https://prometheus.io/),
|
|
many folks still run classic network monitoring systems like the popular [Obvervium](https://observium.org/)
|
|
or its sibling [LibreNMS](https://librenms.org/). Although the metrics-based approach is modern,
|
|
we really ought to have an old-skool [SNMP](https://datatracker.ietf.org/doc/html/rfc1157) interface
|
|
so that we can swear it _by the Old Gods and the New_.
|
|
|
|
### VPP's Stats Segment
|
|
|
|
VPP maintains lots of interesting statistics at runtime - for example for nodes and their activity,
|
|
but also, importantly, for each interface known to the system. So I take a look at the stats segment,
|
|
configured in `startup.conf`, and I notice that VPP will create socket in `/run/vpp/stats.sock` which
|
|
can be connected to. There's also a few introspection tools, notably `vpp_get_stats`, which can either
|
|
list, dump once, or continuously dump the data:
|
|
|
|
```
|
|
pim@hippo:~$ vpp_get_stats socket-name /run/vpp/stats.sock ls | wc -l
|
|
3800
|
|
pim@hippo:~$ vpp_get_stats socket-name /run/vpp/stats.sock dump /if/names
|
|
[0]: local0 /if/names
|
|
[1]: TenGigabitEthernet3/0/0 /if/names
|
|
[2]: TenGigabitEthernet3/0/1 /if/names
|
|
[3]: TenGigabitEthernet3/0/2 /if/names
|
|
[4]: TenGigabitEthernet3/0/3 /if/names
|
|
[5]: GigabitEthernet5/0/0 /if/names
|
|
[6]: GigabitEthernet5/0/1 /if/names
|
|
[7]: GigabitEthernet5/0/2 /if/names
|
|
[8]: GigabitEthernet5/0/3 /if/names
|
|
[9]: TwentyFiveGigabitEthernet11/0/0 /if/names
|
|
[10]: TwentyFiveGigabitEthernet11/0/1 /if/names
|
|
[11]: tap2 /if/names
|
|
[12]: TenGigabitEthernet3/0/1.1 /if/names
|
|
[13]: tap2.1 /if/names
|
|
[14]: TenGigabitEthernet3/0/1.2 /if/names
|
|
[15]: tap2.2 /if/names
|
|
[16]: TenGigabitEthernet3/0/1.3 /if/names
|
|
[17]: tap2.3 /if/names
|
|
[18]: tap3 /if/names
|
|
[19]: tap4 /if/names
|
|
```
|
|
|
|
Alright! Clearly, the `/if/` prefix is the one I'm looking for. I find a Python library that allows
|
|
for this data to be MMAPd and directly read as a dictionary, including some neat aggregation
|
|
functions (see `src/vpp-api/python/vpp_papi/vpp_stats.py`):
|
|
|
|
```
|
|
Counters can be accessed in either dimension.
|
|
stat['/if/rx'] - returns 2D lists
|
|
stat['/if/rx'][0] - returns counters for all interfaces for thread 0
|
|
stat['/if/rx'][0][1] - returns counter for interface 1 on thread 0
|
|
stat['/if/rx'][0][1]['packets'] - returns the packet counter
|
|
for interface 1 on thread 0
|
|
stat['/if/rx'][:, 1] - returns the counters for interface 1 on all threads
|
|
stat['/if/rx'][:, 1].packets() - returns the packet counters for
|
|
interface 1 on all threads
|
|
stat['/if/rx'][:, 1].sum_packets() - returns the sum of packet counters for
|
|
interface 1 on all threads
|
|
stat['/if/rx-miss'][:, 1].sum() - returns the sum of packet counters for
|
|
interface 1 on all threads for simple counters
|
|
```
|
|
|
|
Alright, so let's grab that file and refactor it into a small library for me to use, I do
|
|
this in [[this commit](https://git.ipng.ch/ipng/vpp-snmp-agent/commit/51eee915bf0f6267911da596b41a4475feaf212e)].
|
|
|
|
### VPP's API
|
|
|
|
In a previous project, I already got a little bit of exposure to the Python API (`vpp_papi`),
|
|
and it's pretty straight forward to use. Each API is published in a JSON file in
|
|
`/usr/share/vpp/api/{core,plugins}/` and those can be read by the Python library and
|
|
exposed to callers. This gives me full programmatic read/write access to the VPP runtime
|
|
configuration, which is super cool.
|
|
|
|
There are dozens of APIs to call (the Linux CP plugin even added one!), and in the case
|
|
of enumerating interfaces, we can see the definition in `core/interface.api.json` where
|
|
there is an element called `services.sw_interface_dump` which shows its reply is
|
|
`sw_interface_details`, and in that message we can see all the fields that will be
|
|
set in the request and all that will be present in the response. Nice! Here's a quick
|
|
demonstration:
|
|
|
|
```python
|
|
from vpp_papi import VPPApiClient
|
|
import os
|
|
import fnmatch
|
|
import sys
|
|
|
|
vpp_json_dir = '/usr/share/vpp/api/'
|
|
|
|
# construct a list of all the json api files
|
|
jsonfiles = []
|
|
for root, dirnames, filenames in os.walk(vpp_json_dir):
|
|
for filename in fnmatch.filter(filenames, '*.api.json'):
|
|
jsonfiles.append(os.path.join(root, filename))
|
|
|
|
vpp = VPPApiClient(apifiles=jsonfiles, server_address='/run/vpp/api.sock')
|
|
vpp.connect("test-client")
|
|
|
|
v = vpp.api.show_version()
|
|
print('VPP version is %s' % v.version)
|
|
|
|
iface_list = vpp.api.sw_interface_dump()
|
|
for iface in iface_list:
|
|
print("idx=%d name=%s mac=%s mtu=%d flags=%d" % (iface.sw_if_index,
|
|
iface.interface_name, iface.l2_address, iface.mtu[0], iface.flags))
|
|
```
|
|
|
|
The output:
|
|
```
|
|
$ python3 vppapi-test.py
|
|
VPP version is 21.10-rc0~325-g4976c3b72
|
|
idx=0 name=local0 mac=00:00:00:00:00:00 mtu=0 flags=0
|
|
idx=1 name=TenGigabitEthernet3/0/0 mac=68:05:ca:32:46:14 mtu=9000 flags=0
|
|
idx=2 name=TenGigabitEthernet3/0/1 mac=68:05:ca:32:46:15 mtu=1500 flags=3
|
|
idx=3 name=TenGigabitEthernet3/0/2 mac=68:05:ca:32:46:16 mtu=9000 flags=1
|
|
idx=4 name=TenGigabitEthernet3/0/3 mac=68:05:ca:32:46:17 mtu=9000 flags=1
|
|
idx=5 name=GigabitEthernet5/0/0 mac=a0:36:9f:c8:a0:54 mtu=9000 flags=0
|
|
idx=6 name=GigabitEthernet5/0/1 mac=a0:36:9f:c8:a0:55 mtu=9000 flags=0
|
|
idx=7 name=GigabitEthernet5/0/2 mac=a0:36:9f:c8:a0:56 mtu=9000 flags=0
|
|
idx=8 name=GigabitEthernet5/0/3 mac=a0:36:9f:c8:a0:57 mtu=9000 flags=0
|
|
idx=9 name=TwentyFiveGigabitEthernet11/0/0 mac=6c:b3:11:20:e0:c4 mtu=9000 flags=0
|
|
idx=10 name=TwentyFiveGigabitEthernet11/0/1 mac=6c:b3:11:20:e0:c6 mtu=9000 flags=0
|
|
idx=11 name=tap2 mac=02:fe:07:ae:31:c3 mtu=1500 flags=3
|
|
idx=12 name=TenGigabitEthernet3/0/1.1 mac=00:00:00:00:00:00 mtu=1500 flags=3
|
|
idx=13 name=tap2.1 mac=00:00:00:00:00:00 mtu=1500 flags=3
|
|
idx=14 name=TenGigabitEthernet3/0/1.2 mac=00:00:00:00:00:00 mtu=1500 flags=3
|
|
idx=15 name=tap2.2 mac=00:00:00:00:00:00 mtu=1500 flags=3
|
|
idx=16 name=TenGigabitEthernet3/0/1.3 mac=00:00:00:00:00:00 mtu=1500 flags=3
|
|
idx=17 name=tap2.3 mac=00:00:00:00:00:00 mtu=1500 flags=3
|
|
idx=18 name=tap3 mac=02:fe:95:db:3f:c4 mtu=9000 flags=3
|
|
idx=19 name=tap4 mac=02:fe:17:06:fc:af mtu=9000 flags=3
|
|
```
|
|
|
|
So I added a little abstration with some error handling and one main function
|
|
to return interfaces as a Python dictionary of those `sw_interface_details`
|
|
tuples in [[this commit](https://git.ipng.ch/ipng/vpp-snmp-agent/commit/51eee915bf0f6267911da596b41a4475feaf212e)].
|
|
|
|
### AgentX
|
|
|
|
Now that we are able to enumerate the interfaces and their metadata (like admin/oper
|
|
status, link speed, name, index, MAC address, and what have you), and as well
|
|
the highly sought after interface statistics as 64bit counters (with a wealth of
|
|
extra information like broadcast/multicast/unicast packets, octets received and
|
|
transmitted, errors and drops). I am ready to tie things together.
|
|
|
|
It took a bit of sleuthing, but I eventually found a library on sourceforge (!)
|
|
that has a rudimentary implementation of [RFC 2741](https://datatracker.ietf.org/doc/html/rfc2741)
|
|
which is the SNMP Agent Extensibility (AgentX) Protocol. In a nutshell, this allows
|
|
an external program to connect to the main SNMP daemon, register an interest in
|
|
certain OIDs, and get called whenever the SNMPd is being queried for them.
|
|
|
|
The flow is pretty simple (see section 6.2 of the RFC), the Agent (client):
|
|
1. opens a TCP or Unix domain socket to the SNMPd
|
|
1. sends an Open PDU, which the server will respond or reject.
|
|
1. (optionally) can send a Ping PDU, the server will respond.
|
|
1. registers an interest with Register PDU
|
|
|
|
It then waits and gets called by the SNMPd with Get PDUs (to retrieve one
|
|
single value), GetNext PDU (to enable snmpwalk), GetBulk PDU (to retrieve a whole
|
|
subsection of the MIB), all of which are answered by a Response PDU.
|
|
|
|
If the Agent is to support writing, it will also have to implement TestSet, CommitSet,
|
|
CommitUndoSet and CommitCleanupSet PDUs. For this agent, we don't need to implement
|
|
those, so I'll just ignore those requests and implement the read-only stuff. Sounds easy :)
|
|
|
|
The first order of business is to create the values for two main MIBs of interest:
|
|
|
|
1. `.iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.` - This table is an older variant
|
|
and it contains a bunch of relevant fields, one per interface, notably `ifIndex`,
|
|
`ifName`, `ifType`, `ifMtu`, `ifSpeed`, `ifPhysAddress`, `ifOperStatus`, `ifAdminStatus`
|
|
and a bunch of 32bit counters for octets/packets in and out of the interfaces.
|
|
1. `.iso.org.dod.internet.mgmt.mib-2.ifMIB.ifMIBObjects.ifXTable.` - This table is a makeover
|
|
of the other one (the **X** here stands for eXtra), and adds a few 64 bit counters
|
|
for the interface stats, and as well an `ifHighSpeed` which is in megabits instead of
|
|
kilobits in the previous MIB.
|
|
|
|
Populating these MIBs can be done periodically by retrieving the interfaces from VPP and
|
|
then simply walking the dictionary with Stats Segment data. I then register these two
|
|
main MIB entrypoints with SNMPd as I connect to it, and spit out the correct values
|
|
once asked with `GetPDU` or `GetNextPDU` requests, by issuing a corresponding `ResponsePDU`
|
|
to the SNMP server -- it takes care of all the rest!
|
|
|
|
The resulting code is in [[this
|
|
commit](https://git.ipng.ch/ipng/vpp-snmp-agent/commit/8c9c1e2b4aa1d40a981f17581f92bba133dd2c29)]
|
|
but you can also check out the whole thing on
|
|
[[Github](https://git.ipng.ch/ipng/vpp-snmp-agent)].
|
|
|
|
### Building
|
|
|
|
Shipping a bunch of Python files around is not ideal, so I decide to build this stuff
|
|
together in a binary that I can easily distribute to my machines: I just simply install
|
|
`pyinstaller` with _PIP_ and run it:
|
|
|
|
```
|
|
sudo pip install pyinstaller
|
|
pyinstaller vpp-snmp-agent.py --onefile
|
|
|
|
## Run it on console
|
|
dist/vpp-snmp-agent -h
|
|
usage: vpp-snmp-agent [-h] [-a ADDRESS] [-p PERIOD] [-d]
|
|
|
|
optional arguments:
|
|
-h, --help show this help message and exit
|
|
-a ADDRESS Location of the SNMPd agent (unix-path or host:port), default localhost:705
|
|
-p PERIOD Period to poll VPP, default 30 (seconds)
|
|
-d Enable debug, default False
|
|
|
|
|
|
## Install
|
|
sudo cp dist/vpp-snmp-agent /usr/sbin/
|
|
```
|
|
|
|
### Running
|
|
|
|
After installing `Net-SNMP`, the default in Ubuntu, I do have to ensure that it runs in
|
|
the correct namespace. So what I do is disable the systemd unit that ships with the Ubuntu
|
|
package, and instead create these:
|
|
|
|
```
|
|
pim@hippo:~/src/vpp-snmp-agentx$ cat < EOF | sudo tee /usr/lib/systemd/system/netns-dataplane.service
|
|
[Unit]
|
|
Description=Dataplane network namespace
|
|
After=systemd-sysctl.service network-pre.target
|
|
Before=network.target network-online.target
|
|
|
|
[Service]
|
|
Type=oneshot
|
|
RemainAfterExit=yes
|
|
|
|
# PrivateNetwork will create network namespace which can be
|
|
# used in JoinsNamespaceOf=.
|
|
PrivateNetwork=yes
|
|
|
|
# To set `ip netns` name for this namespace, we create a second namespace
|
|
# with required name, unmount it, and then bind our PrivateNetwork
|
|
# namespace to it. After this we can use our PrivateNetwork as a named
|
|
# namespace in `ip netns` commands.
|
|
ExecStartPre=-/usr/bin/echo "Creating dataplane network namespace"
|
|
ExecStart=-/usr/sbin/ip netns delete dataplane
|
|
ExecStart=-/usr/bin/mkdir -p /etc/netns/dataplane
|
|
ExecStart=-/usr/bin/touch /etc/netns/dataplane/resolv.conf
|
|
ExecStart=-/usr/sbin/ip netns add dataplane
|
|
ExecStart=-/usr/bin/umount /var/run/netns/dataplane
|
|
ExecStart=-/usr/bin/mount --bind /proc/self/ns/net /var/run/netns/dataplane
|
|
# Apply default sysctl for dataplane namespace
|
|
ExecStart=-/usr/sbin/ip netns exec dataplane /usr/lib/systemd/systemd-sysctl
|
|
ExecStop=-/usr/sbin/ip netns delete dataplane
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|
|
WantedBy=network-online.target
|
|
EOF
|
|
|
|
pim@hippo:~/src/vpp-snmp-agentx$ cat < EOF | sudo tee /usr/lib/systemd/system/snmpd-dataplane.service
|
|
[Unit]
|
|
Description=Simple Network Management Protocol (SNMP) Daemon.
|
|
After=network.target
|
|
ConditionPathExists=/etc/snmp/snmpd.conf
|
|
|
|
[Service]
|
|
Type=simple
|
|
ExecStartPre=/bin/mkdir -p /var/run/agentx-dataplane/
|
|
NetworkNamespacePath=/var/run/netns/dataplane
|
|
ExecStart=/usr/sbin/snmpd -LOw -u Debian-snmp -g vpp -I -smux,mteTrigger,mteTriggerConf -f -p /run/snmpd-dataplane.pid
|
|
ExecReload=/bin/kill -HUP \$MAINPID
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|
|
EOF
|
|
|
|
pim@hippo:~/src/vpp-snmp-agentx$ cat < EOF | sudo tee /usr/lib/systemd/system/vpp-snmp-agent.service
|
|
[Unit]
|
|
Description=SNMP AgentX Daemon for VPP dataplane statistics
|
|
After=network.target
|
|
ConditionPathExists=/etc/snmp/snmpd.conf
|
|
|
|
[Service]
|
|
Type=simple
|
|
NetworkNamespacePath=/var/run/netns/dataplane
|
|
ExecStart=/usr/sbin/vpp-snmp-agent
|
|
Group=vpp
|
|
ExecReload=/bin/kill -HUP \$MAINPID
|
|
Restart=on-failure
|
|
RestartSec=5s
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|
|
EOF
|
|
```
|
|
|
|
Note the use of `NetworkNamespacePath` here -- this ensures that the snmpd and its agent both
|
|
run in the `dataplane` namespace which was created by `netns-dataplane.service`.
|
|
|
|
## Results
|
|
|
|
I now install the binary and, using the `snmpd.conf` configuration file (see Appendix):
|
|
|
|
```
|
|
pim@hippo:~/src/vpp-snmp-agentx$ sudo systemctl stop snmpd
|
|
pim@hippo:~/src/vpp-snmp-agentx$ sudo systemctl disable snmpd
|
|
pim@hippo:~/src/vpp-snmp-agentx$ sudo systemctl daemon-reload
|
|
pim@hippo:~/src/vpp-snmp-agentx$ sudo systemctl enable netns-dataplane
|
|
pim@hippo:~/src/vpp-snmp-agentx$ sudo systemctl start netns-dataplane
|
|
pim@hippo:~/src/vpp-snmp-agentx$ sudo systemctl enable snmpd-dataplane
|
|
pim@hippo:~/src/vpp-snmp-agentx$ sudo systemctl start snmpd-dataplane
|
|
pim@hippo:~/src/vpp-snmp-agentx$ sudo systemctl enable vpp-snmp-agent
|
|
pim@hippo:~/src/vpp-snmp-agentx$ sudo systemctl start vpp-snmp-agent
|
|
|
|
pim@hippo:~/src/vpp-snmp-agentx$ sudo journalctl -u vpp-snmp-agent
|
|
[INFO ] agentx.agent - run : Calling setup
|
|
[INFO ] agentx.agent - setup : Connecting to VPP Stats...
|
|
[INFO ] agentx.vppapi - connect : Connecting to VPP
|
|
[INFO ] agentx.vppapi - connect : VPP version is 21.10-rc0~325-g4976c3b72
|
|
[INFO ] agentx.agent - run : Initial update
|
|
[INFO ] agentx.network - update : Setting initial serving dataset (740 OIDs)
|
|
[INFO ] agentx.agent - run : Opening AgentX connection
|
|
[INFO ] agentx.network - connect : Connecting to localhost:705
|
|
[INFO ] agentx.network - start : Registering: 1.3.6.1.2.1.2.2.1
|
|
[INFO ] agentx.network - start : Registering: 1.3.6.1.2.1.31.1.1.1
|
|
[INFO ] agentx.network - update : Replacing serving dataset (740 OIDs)
|
|
[INFO ] agentx.network - update : Replacing serving dataset (740 OIDs)
|
|
[INFO ] agentx.network - update : Replacing serving dataset (740 OIDs)
|
|
[INFO ] agentx.network - update : Replacing serving dataset (740 OIDs)
|
|
```
|
|
|
|
{{< image width="800px" src="/assets/vpp/librenms.png" alt="LibreNMS" >}}
|
|
|
|
## Appendix
|
|
|
|
#### SNMPd Config
|
|
|
|
```
|
|
$ cat << EOF | sudo tee /etc/snmp/snmpd.conf
|
|
com2sec readonly default <<some-string>>
|
|
|
|
group MyROGroup v2c readonly
|
|
view all included .1 80
|
|
access MyROGroup "" any noauth exact all none none
|
|
|
|
sysLocation Ruemlang, Zurich, Switzerland
|
|
sysContact noc@ipng.ch
|
|
|
|
master agentx
|
|
agentXSocket tcp:localhost:705,unix:/var/agentx/master,unix:/run/vpp/agentx.sock
|
|
|
|
agentaddress udp:161,udp6:161
|
|
|
|
#OS Distribution Detection
|
|
extend distro /usr/bin/distro
|
|
|
|
#Hardware Detection
|
|
extend manufacturer '/bin/cat /sys/devices/virtual/dmi/id/sys_vendor'
|
|
extend hardware '/bin/cat /sys/devices/virtual/dmi/id/product_name'
|
|
extend serial '/bin/cat /var/run/snmpd.serial'
|
|
EOF
|
|
```
|
|
|
|
Note the use of a few helpers here - `/usr/bin/distro` comes from LibreNMS [ref](https://docs.librenms.org/Support/SNMP-Configuration-Examples/)
|
|
and tries to figure out what distribution is used. The very last line of that file
|
|
echo's the found distribtion, to which I prepend the string, like `echo "VPP ${OSSTR}"`.
|
|
The other file of interest `/var/run/snmpd.serial` is computed at boot-time, by running
|
|
the following in `/etc/rc.local`:
|
|
|
|
```
|
|
# Assemble serial number for snmpd
|
|
BS=$(cat /sys/devices/virtual/dmi/id/board_serial)
|
|
PS=$(cat /sys/devices/virtual/dmi/id/product_serial)
|
|
echo $BS.$PS > /var/run/snmpd.serial
|
|
```
|
|
|
|
I have to do this, because SNMPd runs as non-privileged user, yet those DMI elements are
|
|
root-readable only (for reasons that are beyond me). Seeing as they will not change at
|
|
runtime anyway, I just create that file and cat it into the `serial` field. It then shows
|
|
up nicely in LibreNMS alongside the others.
|
|
|
|
{{< image width="200px" float="left" src="/assets/vpp/vpp.png" alt="VPP Hound" >}}
|
|
|
|
Oh, and one last thing. The VPP Hound logo!
|
|
|
|
In LibreNMS, the icons in the _devices_ view use a function that leveages this `distro`
|
|
field, by looking at the first word (in our case "VPP") with an extension of either .svg
|
|
or .png in an icons directory, usually `html/images/os/`. I dropped the hound of the
|
|
[fd.io](https://fd.io/) homepage in there, and will add the icon upstream for future use,
|
|
in this [[librenms PR](https://github.com/librenms/librenms/pull/13230)] and its companion
|
|
change to [[librenms-agent PR](https://github.com/librenms/librenms-agent/pull/374).
|