Pim van Pelt
fdb77838b8
All checks were successful
continuous-integration/drone/push Build is passing
385 lines
22 KiB
Markdown
385 lines
22 KiB
Markdown
---
|
|
date: "2023-04-09T11:01:14Z"
|
|
title: VPP - Monitoring
|
|
aliases:
|
|
- /s/articles/2023/04/09/vpp-stats.html
|
|
---
|
|
|
|
{{< image width="200px" float="right" src="/assets/vpp/fdio-color.svg" alt="VPP" >}}
|
|
|
|
# About this series
|
|
|
|
Ever since I first saw VPP - the Vector Packet Processor - I have been deeply impressed with its
|
|
performance and versatility. For those of us who have used Cisco IOS/XR devices, like the classic
|
|
_ASR_ (aggregation service router), VPP will look and feel quite familiar as many of the approaches
|
|
are shared between the two.
|
|
|
|
I've been working on the Linux Control Plane [[ref](https://git.ipng.ch/ipng/lcpng)], which you
|
|
can read all about in my series on VPP back in 2021:
|
|
|
|
[{: style="width:300px; float: right; margin-left: 1em;"}](https://video.ipng.ch/w/erc9sAofrSZ22qjPwmv6H4)
|
|
|
|
* [[Part 1]({{< ref "2021-08-12-vpp-1" >}})]: Punting traffic through TUN/TAP interfaces into Linux
|
|
* [[Part 2]({{< ref "2021-08-13-vpp-2" >}})]: Mirroring VPP interface configuration into Linux
|
|
* [[Part 3]({{< ref "2021-08-15-vpp-3" >}})]: Automatically creating sub-interfaces in Linux
|
|
* [[Part 4]({{< ref "2021-08-25-vpp-4" >}})]: Synchronize link state, MTU and addresses to Linux
|
|
* [[Part 5]({{< ref "2021-09-02-vpp-5" >}})]: Netlink Listener, synchronizing state from Linux to VPP
|
|
* [[Part 6]({{< ref "2021-09-10-vpp-6" >}})]: Observability with LibreNMS and VPP SNMP Agent
|
|
* [[Part 7]({{< ref "2021-09-21-vpp-7" >}})]: Productionizing and reference Supermicro fleet at IPng
|
|
|
|
With this, I can make a regular server running Linux use VPP as kind of a software ASIC for super
|
|
fast forwarding, filtering, NAT, and so on, while keeping control of the interface state (links,
|
|
addresses and routes) itself. With Linux CP, running software like [FRR](https://frrouting.org/) or
|
|
[Bird](https://bird.network.cz/) on top of VPP and achieving >150Mpps and >180Gbps forwarding
|
|
rates are easily in reach. If you find that hard to believe, check out [[my DENOG14
|
|
talk](https://video.ipng.ch/w/erc9sAofrSZ22qjPwmv6H4)] or click the thumbnail above. I am
|
|
continuously surprised at the performance per watt, and the performance per Swiss Franc spent.
|
|
|
|
## Monitoring VPP
|
|
|
|
Of course, it's important to be able to see what routers are _doing_ in production. For the longest
|
|
time, the _de facto_ standard for monitoring in the networking industry has been Simple Network
|
|
Management Protocol (SNMP), described in [[RFC 1157](https://www.rfc-editor.org/rfc/rfc1157)]. But
|
|
there's another way, using a metrics and time series system called _Borgmon_, originally designed by
|
|
Google [[ref](https://sre.google/sre-book/practical-alerting/)] but popularized by Soundcloud in an
|
|
open source interpretation called **Prometheus** [[ref](https://prometheus.io/)]. IPng Networks ♥ Prometheus.
|
|
|
|
I'm a really huge fan of Prometheus and its graphical frontend Grafana, as you can see with my work on
|
|
Mastodon in [[this article]({{< ref "2022-11-27-mastodon-3" >}})]. Join me on
|
|
[[ublog.tech](https://ublog.tech)] if you haven't joined the Fediverse yet. It's well monitored!
|
|
|
|
### SNMP
|
|
|
|
SNMP defines an extensible model by which parts of the OID (object identifier) tree can be delegated
|
|
to another process, and the main SNMP daemon will call out to it using an _AgentX_ protocol,
|
|
described in [[RFC 2741](https://datatracker.ietf.org/doc/html/rfc2741)]. In a nutshell, this
|
|
allows an external program to connect to the main SNMP daemon, register an interest in certain OIDs,
|
|
and get called whenever the SNMPd is being queried for them.
|
|
|
|
{{< image width="400px" float="right" src="/assets/vpp-stats/librenms.png" alt="LibreNMS" >}}
|
|
|
|
The flow is pretty simple (see section 6.2 of the RFC), the Agent (client):
|
|
1. opens a TCP or Unix domain socket to the SNMPd
|
|
1. sends an Open PDU, which the server will respond or reject.
|
|
1. (optionally) can send a Ping PDU, the server will respond.
|
|
1. registers an interest with Register PDU
|
|
|
|
It then waits and gets called by the SNMPd with Get PDUs (to retrieve one single value), GetNext PDU
|
|
(to enable snmpwalk), GetBulk PDU (to retrieve a whole subsection of the MIB), all of which are
|
|
answered by a Response PDU.
|
|
|
|
Using parts of a Python Agentx library written by GitHub user hosthvo
|
|
[[ref](https://github.com/hosthvo/pyagentx)], I tried my hands at writing one of these AgentX's.
|
|
The resulting source code is on [[GitHub](https://git.ipng.ch/ipng/vpp-snmp-agent)]. That's the
|
|
one that's running in production ever since I started running VPP routers at IPng Networks AS8298.
|
|
After the _AgentX_ exposes the dataplane interfaces and their statistics into _SNMP_, an open source
|
|
monitoring tool such as LibreNMS [[ref](https://librenms.org/)] can discover the routers and draw
|
|
pretty graphs, as well as detect when interfaces go down, or are overloaded, and so on. That's
|
|
pretty slick.
|
|
|
|
### VPP Stats Segment in Go
|
|
|
|
But if I may offer some critique on my own approach, SNMP monitoring is _very_ 1990s. I'm
|
|
continously surpsied that our industry is still clinging on to this archaic approach. VPP offers
|
|
_a lot_ of observability, its statistics segment is chock full of interesting counters and gauges
|
|
that can be really helpful to understand how the dataplane performs. If there are errors or a
|
|
bottleneck develops in the router, going over `show runtime` or `show errors` can be a life saver.
|
|
Let's take another look at that Stats Segment (the one that the SNMP AgentX connects to in order to
|
|
query it for packets/byte counters and interface names).
|
|
|
|
You can think of the Stats Segment as a directory hierarchy where each file represents a type of
|
|
counter. VPP comes with a small helper tool called VPP Stats FS, which uses a FUSE based read-only
|
|
filesystem to expose those counters in an intuitive way, so let's take a look
|
|
|
|
```
|
|
pim@hippo:~/src/vpp/extras/vpp_stats_fs$ sudo systemctl start vpp
|
|
pim@hippo:~/src/vpp/extras/vpp_stats_fs$ sudo make start
|
|
pim@hippo:~/src/vpp/extras/vpp_stats_fs$ mount | grep stats
|
|
rawBridge on /run/vpp/stats_fs_dir type fuse.rawBridge (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
|
|
|
|
pim@hippo:/run/vpp/stats_fs_dir$ ls -la
|
|
drwxr-xr-x 0 root root 0 Apr 9 14:07 bfd
|
|
drwxr-xr-x 0 root root 0 Apr 9 14:07 buffer-pools
|
|
drwxr-xr-x 0 root root 0 Apr 9 14:07 err
|
|
drwxr-xr-x 0 root root 0 Apr 9 14:07 if
|
|
drwxr-xr-x 0 root root 0 Apr 9 14:07 interfaces
|
|
drwxr-xr-x 0 root root 0 Apr 9 14:07 mem
|
|
drwxr-xr-x 0 root root 0 Apr 9 14:07 net
|
|
drwxr-xr-x 0 root root 0 Apr 9 14:07 node
|
|
drwxr-xr-x 0 root root 0 Apr 9 14:07 nodes
|
|
drwxr-xr-x 0 root root 0 Apr 9 14:07 sys
|
|
|
|
pim@hippo:/run/vpp/stats_fs_dir$ cat sys/boottime
|
|
1681042046.00
|
|
pim@hippo:/run/vpp/stats_fs_dir$ date +%s
|
|
1681042058
|
|
pim@hippo:~/src/vpp/extras/vpp_stats_fs$ sudo make stop
|
|
```
|
|
|
|
There's lots of really interesting stuff in here - for example in the `/sys` hierarchy we can see a
|
|
`boottime` file, and from there I can determine the uptime of the process. Further, the `/mem`
|
|
hierarchy shows the current memory usage for each of the _main_, _api_ and _stats_ segment heaps.
|
|
And of course, in the `/interfaces` hierarchy we can see all the usual packets and bytes counters
|
|
for any interface created in the dataplane.
|
|
|
|
### VPP Stats Segment in C
|
|
|
|
I wish I were good at Go, but I never really took to the language. I'm pretty good at Python, but
|
|
sorting through the stats segment isn't super quick as I've already noticed in the Python3 based
|
|
[[VPP SNMP Agent](https://git.ipng.ch/ipng/vpp-snmp-agent)]. I'm probably the world's least
|
|
terrible C programmer, so maybe I can take a look at the VPP Stats Client and make sense of it. Luckily,
|
|
there's an example already in `src/vpp/app/vpp_get_stats.c` and it reveals the following pattern:
|
|
|
|
1. assemble a vector of regular expression patterns in the hierarchy, or just `^/` to start
|
|
1. get a handle to the stats segment with `stats_segment_ls()` using the pattern(s)
|
|
1. use the handle to dump the stats segment into a vector with `stat_segment_dump()`.
|
|
1. iterate over the returned stats structure, each element has a type and a given name:
|
|
* ***STAT_DIR_TYPE_SCALAR_INDEX***: these are floating point doubles
|
|
* ***STAT_DIR_TYPE_COUNTER_VECTOR_SIMPLE***: single uint32 counter
|
|
* ***STAT_DIR_TYPE_COUNTER_VECTOR_COMBINED***: two uint32 counters
|
|
1. freeing the used stats structure with `stat_segment_data_free()`
|
|
|
|
The simple and combined stats turn out to be associative arrays, the outer of which notes the
|
|
_thread_ and the inner of which refers to the _index_. As such, a statistic of type
|
|
***VECTOR_SIMPLE*** can be decoded like so:
|
|
|
|
```
|
|
if (res[i].type == STAT_DIR_TYPE_COUNTER_VECTOR_SIMPLE)
|
|
for (k = 0; k < vec_len (res[i].simple_counter_vec); k++)
|
|
for (j = 0; j < vec_len (res[i].simple_counter_vec[k]); j++)
|
|
printf ("[%d @ %d]: %llu packets %s\n", j, k, res[i].simple_counter_vec[k][j], res[i].name);
|
|
```
|
|
|
|
The statistic of type ***VECTOR_COMBINED*** is very similar, except the union type there is a
|
|
`combined_counter_vec[k][j]` which has a member `.packets` and a member called `.bytes`. The
|
|
simplest form, ***SCALAR_INDEX***, is just a single floating point number attached to the name.
|
|
|
|
In principle, this should be really easy to sift through and decode. Now that I've figured that
|
|
out, let me dump a bunch of stats with the `vpp_get_stats` tool that comes with vanilla VPP:
|
|
|
|
```
|
|
pim@chrma0:~$ vpp_get_stats dump /interfaces/TenGig.*40121 | grep -v ': 0'
|
|
[0 @ 2]: 67057 packets /interfaces/TenGigabitEthernet81_0_0.40121/drops
|
|
[0 @ 2]: 76125287 packets /interfaces/TenGigabitEthernet81_0_0.40121/ip4
|
|
[0 @ 2]: 1793946 packets /interfaces/TenGigabitEthernet81_0_0.40121/ip6
|
|
[0 @ 2]: 77919629 packets, 66184628769 bytes /interfaces/TenGigabitEthernet81_0_0.40121/rx
|
|
[0 @ 0]: 7 packets, 610 bytes /interfaces/TenGigabitEthernet81_0_0.40121/tx
|
|
[0 @ 1]: 26687 packets, 18771919 bytes /interfaces/TenGigabitEthernet81_0_0.40121/tx
|
|
[0 @ 2]: 6448944 packets, 3663975508 bytes /interfaces/TenGigabitEthernet81_0_0.40121/tx
|
|
[0 @ 3]: 138924 packets, 20599785 bytes /interfaces/TenGigabitEthernet81_0_0.40121/tx
|
|
[0 @ 4]: 130720342 packets, 57436383614 bytes /interfaces/TenGigabitEthernet81_0_0.40121/tx
|
|
```
|
|
|
|
I can see both types of counter at play here, let me explain the first line: it is saying that the
|
|
counter of name `/interfaces/TenGigabitEthernet81_0_0.40121/drops`, at counter index 0, CPU thread
|
|
2, has a simple counter with value 67057. Taking the last line, this is a combined counter type with
|
|
name `/interfaces/TenGigabitEthernet81_0_0.40121/tx` at index 0, all five CPU threads (the main
|
|
thread and four worker threads) have all sent traffic into this interface, and the counters for each
|
|
in packets and bytes is given.
|
|
|
|
For readability's sake, my `grep -v` above doesn't print any counter that is 0. For example,
|
|
interface `Te81/0/0` has only one receive queue, and it's bound to thread 2. The other threads will
|
|
not receive any packets for it, consequently their `rx` counters stay zero:
|
|
|
|
```
|
|
pim@chrma0:~/src/vpp$ vpp_get_stats dump /interfaces/TenGig.*40121 | grep rx$
|
|
[0 @ 0]: 0 packets, 0 bytes /interfaces/TenGigabitEthernet81_0_0.40121/rx
|
|
[0 @ 1]: 0 packets, 0 bytes /interfaces/TenGigabitEthernet81_0_0.40121/rx
|
|
[0 @ 2]: 80720186 packets, 68458816253 bytes /interfaces/TenGigabitEthernet81_0_0.40121/rx
|
|
[0 @ 3]: 0 packets, 0 bytes /interfaces/TenGigabitEthernet81_0_0.40121/rx
|
|
[0 @ 4]: 0 packets, 0 bytes /interfaces/TenGigabitEthernet81_0_0.40121/rx
|
|
```
|
|
|
|
### Hierarchy: Pattern Matching
|
|
|
|
I quickly discover a pattern in most of these names: they start with a scope, say `/interfaces`,
|
|
then have a path entry for the interface name, and finally a specific counter (`/rx` or `/mpls`).
|
|
This is true also for the `/nodes` hiearchy, in which all VPP's graph nodes have a set of counters:
|
|
|
|
```
|
|
pim@chrma0:~$ vpp_get_stats dump /nodes/ip4-lookup | grep -v ': 0'
|
|
[0 @ 1]: 11365675493301 packets /nodes/ip4-lookup/clocks
|
|
[0 @ 2]: 3256664129799 packets /nodes/ip4-lookup/clocks
|
|
[0 @ 3]: 28364098623954 packets /nodes/ip4-lookup/clocks
|
|
[0 @ 4]: 30198798628761 packets /nodes/ip4-lookup/clocks
|
|
[0 @ 1]: 80870763789 packets /nodes/ip4-lookup/vectors
|
|
[0 @ 2]: 17392446654 packets /nodes/ip4-lookup/vectors
|
|
[0 @ 3]: 259363625369 packets /nodes/ip4-lookup/vectors
|
|
[0 @ 4]: 298176625181 packets /nodes/ip4-lookup/vectors
|
|
[0 @ 1]: 49730112811 packets /nodes/ip4-lookup/calls
|
|
[0 @ 2]: 13035172295 packets /nodes/ip4-lookup/calls
|
|
[0 @ 3]: 109088424231 packets /nodes/ip4-lookup/calls
|
|
[0 @ 4]: 119789874274 packets /nodes/ip4-lookup/calls
|
|
```
|
|
|
|
If you're ever seen the output of `show runtime`, it looks like this:
|
|
|
|
```
|
|
vpp# show runtime
|
|
Thread 1 vpp_wk_0 (lcore 28)
|
|
Time 3377500.2, 10 sec internal node vector rate 1.46 loops/sec 3301017.05
|
|
vector rates in 2.7440e6, out 2.7210e6, drop 3.6025e1, punt 7.2243e-5
|
|
Name State Calls Vectors Suspends Clocks Vectors/Call
|
|
...
|
|
ip4-lookup active 49732141978 80873724903 0 1.41e2 1.63
|
|
```
|
|
|
|
Hey look! On thread 1, which is called `vpp_wk_0` and is running on logical CPU core #28, there are
|
|
a bunch of VPP graph nodes that are all keeping stats of what they've been doing, and you can see
|
|
here that the following numbers line up between `show runtime` and the VPP Stats dumper:
|
|
|
|
* ***Name***: This is the name of the VPP graph node, in this case `ip4-lookup`, which is performing an
|
|
IPv4 FIB lookup to figure out what the L3 nexthop is of a given IPv4 packet we're trying to route.
|
|
* ***Calls***: How often did we invoke this graph node, 49.7 billion times so far.
|
|
* ***Vectors***: How many packets did we push through, 80.87 billion, humble brag.
|
|
* ***Clocks***: This one is a bit different -- you can see the cumulative clock cycles spent by
|
|
this CPU thread in the stats dump: 11365675493301 divided by 80870763789 packets is 140.54 CPU
|
|
cycles per packet. It's a cool interview question "How many CPU cycles does it take to do an
|
|
IPv4 routing table lookup". You now know the answer :-)
|
|
* ***Vectors/Call***: This is a measure of how busy the node is (did it run for only one packet,
|
|
or for many packets?). On average when the worker thread gave the `ip4-lookup` node some work to
|
|
do, there have been a total of 80873724903 packets handled in 49732141978 calls, so 1.626
|
|
packets per call. If ever you're handling 256 packets per call (the most VPP will allow per call),
|
|
your router will be sobbing.
|
|
|
|
### Prometheus Metrics
|
|
|
|
Prometheus has metrics which carry a name, and zero or more labels. The prometheus query language
|
|
can then use these labels to do aggregation, division, averages, and so on. As a practical example,
|
|
above I looked at interface stats and saw that the Rx/Tx numbers were counted one per thread. If
|
|
we'd like the total on the interface, it would be great if we could `sum without (thread,index)`,
|
|
which will have the effect of adding all of these numbers together.
|
|
For the monotonically increasing counter numbers (like the total vectors/calls/clocks per node), we
|
|
can take the running _rate_ of change, showing the time spent over the last minute, or so. This way,
|
|
spikes in traffic will clearly correlate both with a spike in packets/sec or bytes/sec on the
|
|
interface, but also a higher number of _vectors/call_, and correspondingly typically a lower number
|
|
of _clocks/vector_, as VPP gets more efficient when it can re-use the CPU's instruction and data
|
|
cache to do repeat work on multiple packets.
|
|
|
|
I decide to massage the statistic names a little bit, by transforming them of the basic format:
|
|
`prefix_suffix{label="X",index="A",thread="B"} value`
|
|
|
|
A few examples:
|
|
* The single counter that looks like `[6 @ 0]: 994403888 packets /mem/main heap` becomes:
|
|
* `mem{heap="main heap",index="6",thread="0"}`
|
|
* The combined counter `[0 @ 1]: 79582338270 packets, 16265349667188 bytes /interfaces/Te1_0_2/rx`
|
|
becomes:
|
|
* `interfaces_rx_packets{interface="Te1_0_2",index="0",thread="1"} 79582338270`
|
|
* `interfaces_rx_bytes{interface="Te1_0_2",index="0",thread="1"} 16265349667188`
|
|
* The node information running on, say thread 4, becomes:
|
|
* `nodes_clocks{node="ip4-lookup",index="0",thread="4"} 30198798628761`
|
|
* `nodes_vectors{node="ip4-lookup",index="0",thread="4"} 298176625181`
|
|
* `nodes_calls{node="ip4-lookup",index="0",thread="4"} 119789874274`
|
|
* `nodes_suspends{node="ip4-lookup",index="0",thread="4"} 0`
|
|
|
|
### VPP Exporter
|
|
|
|
I wish I had things like `split()` and `re.match()` but in C (well, I guess I do have POSIX regular
|
|
expressions...), but it's all a little bit more low level. Based on my basic loop that opens the
|
|
stats segment, registers its desired patterns, and then retrieves a vector of {name, type,
|
|
counter}-tuples, I decide to do a little bit of non-intrusive string tokenization first:
|
|
|
|
```
|
|
static int tokenize (const char *str, char delimiter, char **tokens, int *lengths) {
|
|
char *p = (char *) str;
|
|
char *savep = p;
|
|
int i = 0;
|
|
|
|
while (*p) if (*p == delimiter) {
|
|
tokens[i] = (char *) savep;
|
|
lengths[i] = (int) (p - savep);
|
|
i++; p++; savep = p;
|
|
} else p++;
|
|
tokens[i] = (char *) savep;
|
|
lengths[i] = (int) (p - savep);
|
|
return i++;
|
|
}
|
|
|
|
/* The call site */
|
|
char *tokens[10];
|
|
int lengths[10];
|
|
int num_tokens = tokenize (res[i].name, '/', tokens, lengths);
|
|
```
|
|
|
|
The tokenizer takes an array of N pointers to the resulting tokens, and their lengths. This sets it
|
|
aside from `strtok()` and friends, because those will overwrite the occurences of the delimiter in
|
|
the input string with `\0`, and as such cannot take a `const char *str` as input. This one leaves
|
|
the string alone though, and will return the tokens as {ptr, len}-tuples, including how many
|
|
tokens it found.
|
|
|
|
One thing I'll probably regret is that there's no bounds checking on the number of tokens -- if I
|
|
have more than 10 of these, I'll come to regret it. But for now, the depth of the hierarchy is only
|
|
3, so I should be fine. Besides, I got into a fight with ChatGPT after it declared a romantic
|
|
interest in my cat, so it won't write code for me anymore :-(
|
|
|
|
But using this simple tokenizer, and knowledge of the structure of well known hierarchy paths, the
|
|
rest of the exporter is quickly in hand. Some variables don't have a label (for example
|
|
`/sys/boottime`), but those that do will see that field transposed from the directory path
|
|
`/mem/main heap/free` into the label as I showed above.
|
|
|
|
### Results
|
|
|
|
{{< image width="400px" float="right" src="/assets/vpp-stats/grafana1.png" alt="Grafana 1" >}}
|
|
|
|
With this VPP Prometheus Exporter, I can now hook the VPP routers up to Prometheus and Grafana.
|
|
Aggregations in Grafana are super easy and scalable, due to the conversion of the static paths into
|
|
dynamically created labels on the prometheus metric names.
|
|
|
|
Drawing a graph of the running time spent by each individual VPP graph node might look something
|
|
like this:
|
|
|
|
```
|
|
sum without (node)(rate(nodes_clocks[60s]))
|
|
/
|
|
sum without (node)(rate(nodes_vectors[60s]))
|
|
```
|
|
|
|
The plot to the right shows a system under a loadtest that ramps up from 0% to 100% of line rate,
|
|
and the traces are the cummulative time spent in each node (on a logarithmic scale). The top purple
|
|
line represents `dpdk-input`. When a VPP dataplane is idle, the worker threads will be repeatedly
|
|
polling DPDK to ask it if it has something to do, spending 100% of their time being told "there is
|
|
nothing for you to do". But, once load starts appearing, the other nodes start spending CPU time,
|
|
for example the chain of IPv4 forwarding is `ethernet-input`, `ip4-input`, `ip4-lookup`, followed by
|
|
`ip4-rewrite` and ultimately the packet is transmitted on some other interface. When the system is
|
|
lightly loaded, the `ethernet-input` node for example will spend 1100 or so CPU cycles per packet,
|
|
but when the machine is under higher load, the time spent will decrease to as low as 22 CPU cycles
|
|
per packet. This is true for almost all of the nodes - VPP gets relatively _more efficient_ under
|
|
load.
|
|
|
|
{{< image width="400px" float="right" src="/assets/vpp-stats/grafana2.png" alt="Grafana 2" >}}
|
|
|
|
Another cool graph that I won't be able to see when using only LibreNMS and SNMP polling, is how
|
|
busy the router is. In VPP, each dispatch of the worker loop will poll DPDK and dispatch the packets
|
|
through the directed graph of nodes that I showed above. But how many packets can be moved through
|
|
the graph per CPU? The largest number of packets that VPP will ever offer into a call of the nodes
|
|
is 256. Typically an unloaded machine will have an average number of Vectors/Call of around 1.00.
|
|
When the worker thread is loaded, it may sit at around 130-150 Vectors/Call. If it's saturated, it
|
|
will quickly shoot up to 256.
|
|
|
|
As a good approximation, Vectors/Call normalized to 100% will be an indication of how busy the
|
|
dataplane is. In the picture above, between 10:30 and 11:00 my test router was pushing about 180Gbps
|
|
of traffic, but with large packets so its total vectors/call was modest (roughly 35-40), which you
|
|
can see as all threads there are running in the ~25% load range. Then at 11:00 a few threads got
|
|
hotter, and one of them completely saturated, and the traffic being forwarded by the CPU thread was
|
|
suffering _packetlo_, even though the others were absolutely fine... forwarding 150Mpps on a 10 year
|
|
old Dell R720!
|
|
|
|
### What's Next
|
|
|
|
Together with the graph above, I can also see how many CPU cycles are spent in which
|
|
type of operation. For example, encapsulation of GENEVE or VxLAN is not _free_, although it's also
|
|
not every expensive. If I know how many CPU cycles are available (roughly the clock speed of the CPU
|
|
threads, in our case Xeon X1518 (2.2GHz) or Xeon E5-2683 v4 CPUs (3GHz), I can pretty accurately
|
|
calculate what a given mix of traffic and features is going to cost, and how many packets/sec our
|
|
routers at IPng will be able to forward. Spoiler alert: it's way more than currently needed. Our
|
|
supermicros can handle roughly 35Mpps each, and considering a regular mixture of internet traffic
|
|
(called _imix_) is about 3Mpps per 10G, I will have room to spare for the time being
|
|
|
|
This is super valuable information for folks running VPP in production.
|
|
I haven't put the finishing touches on the VPP Prometheus Exporter, for example there are no
|
|
commandline flags yet, it doesn't listen on any port other than 9482 (the same one that the toy
|
|
exporter in `src/vpp/app/vpp_prometheus_export.c` ships with
|
|
[[ref](https://github.com/prometheus/prometheus/wiki/Default-port-allocations)]). My grafana
|
|
dashboard is also not fully completed yet. I hope to get that done in April, and publish both the
|
|
exporter and the dashboard on GitHub. Stay tuned!
|