---
date: "2024-02-17T12:17:54Z"
title: VPP on FreeBSD - Part 2
aliases:
- /s/articles/2024/02/17/vpp-freebsd-2.html
---
# About this series
{{< image width="300px" float="right" src="/assets/freebsd-vpp/freebsd-logo.png" alt="FreeBSD" >}}
Ever since I first saw VPP - the Vector Packet Processor - I have been deeply impressed with its
performance and versatility. For those of us who have used Cisco IOS/XR devices, like the classic
_ASR_ (aggregation services router), VPP will look and feel quite familiar as many of the approaches
are shared between the two. Over the years, folks have asked me regularly "What about BSD?" and to
my surprise, late last year I read an announcement from the _FreeBSD Foundation_
[[ref](https://freebsdfoundation.org/blog/2023-in-review-software-development/)] as they looked back
over 2023 and forward to 2024:
> ***Porting the Vector Packet Processor to FreeBSD***
>
> Vector Packet Processing (VPP) is an open-source, high-performance user space networking stack
> that provides fast packet processing suitable for software-defined networking and network function
> virtualization applications. VPP aims to optimize packet processing through vectorized operations
> and parallelism, making it well-suited for high-speed networking applications. In November of this
> year, the Foundation began a contract with Tom Jones, a FreeBSD developer specializing in network
> performance, to port VPP to FreeBSD. Under the contract, Tom will also allocate time for other
> tasks such as testing FreeBSD on common virtualization platforms to improve the desktop
> experience, improving hardware support on arm64 platforms, and adding support for low power idle
> on Intel and arm64 hardware.
In my first [[article]({{< ref "2024-02-10-vpp-freebsd-1" >}})], I wrote a sort of a _hello world_
by installing FreeBSD 14.0-RELEASE on both a VM and a bare metal Supermicro, and showed that Tom's
VPP branch compiles, runs and pings. In this article, I'll take a look at some comparative
performance numbers.
## Comparing implementations
FreeBSD has an extensive network stack, including regular _kernel_ based functionality such as
routing, filtering and bridging, a faster _netmap_ based datapath, including some userspace
utilities like a _netmap_ bridge, and of course completely userspace based dataplanes, such as the
VPP project that I'm working on here. Last week, I learned that VPP has a _netmap_ driver, and from
previous travels I am already quite familiar with its _DPDK_ based forwarding. I decide to do a
baseline loadtest for each of these on the Supermicro Xeon-D1518 that I installed last week. See the
[[article]({{< ref "2024-02-10-vpp-freebsd-1" >}})] for details on the setup.
The loadtests will use a common set of different configurations, using Cisco T-Rex's default
benchmark profile called `bench.py`:
1. **var2-1514b**: Large Packets, multiple flows with modulating source and destination IPv4
addresses, often called an 'iperf test', with packets of 1514 bytes.
1. **var2-imix**: Mixed Packets, multiple flows, often called an 'imix test', which includes a
bunch of 64b, 390b and 1514b packets.
1. **var2-64b**: Small Packets, still multiple flows, 64 bytes, which allows for multiple receive
queues and kernel or application threads.
1. **64b**: Small Packets, but now single flow, often called 'linerate test', with a packet size
of 64 bytes, limiting to one receive queue.
Each of these four loadtests might occur in only undirectionally (port0 -> port1) or bidirectionally
(port0 <-> port1). This yields eight different loadtests, each taking about 8 minutes. I put the kettle
on and get underway.
### FreeBSD 14: Kernel Bridge
The machine I'm testing has a quad-port Intel i350 (1Gbps copper, using the FreeBSD `igb(4)` driver),
a dual-port Intel X522 (10Gbps SFP+, using the `ix(4)` driver), and a dual-port Intel i710-XXV
(25Gbps SFP28, using the `ixl(4)` driver). I decide to live it up a little, and choose the 25G ports
for my loadtests today, even if I think this machine with its relatively low-end Xeon-D1518 CPU
will struggle a little bit at very high packet rates. No pain, no gain, _amirite_?
I take my fresh FreeBSD 14.0-RELEASE install, without any tinkering other than compiling a GENERIC
kernel that has support for the DPDK modules I'll need later. For my first loadtest, I create a
kernel based bridge as follows, just tying the two 25G interfaces together:
```
[pim@france /usr/obj]$ uname -a
FreeBSD france 14.0-RELEASE FreeBSD 14.0-RELEASE #0: Sat Feb 10 22:18:51 CET 2024 root@france:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64
[pim@france ~]$ dmesg | grep ixl
ixl0: <Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 - 2.3.3-k> mem 0xf8000000-0xf8ffffff,0xf9008000-0xf900ffff irq 16 at device 0.0 on pci7
ixl1: <Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 - 2.3.3-k> mem 0xf7000000-0xf7ffffff,0xf9000000-0xf9007fff irq 16 at device 0.1 on pci7
[pim@france ~]$ sudo ifconfig bridge0 create
[pim@france ~]$ sudo ifconfig bridge0 addm ixl0 addm ixl1 up
[pim@france ~]$ sudo ifconfig ixl0 up
[pim@france ~]$ sudo ifconfig ixl1 up
[pim@france ~]$ ifconfig bridge0
bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=0
ether 58:9c:fc:10:6c:2e
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: ixl1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 4 priority 128 path cost 800
member: ixl0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 3 priority 128 path cost 800
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>
```
One thing that I quickly realize, is that FreeBSD, when using hyperthreading, does have 8 threads
available, but only 4 of them participate in forwarding. When I put the machine under load, I see a
curious 399% spent in _kernel_ while I see 402% in _idle_:
{{< image src="/assets/freebsd-vpp/top-kernel-bridge.png" alt="FreeBSD top" >}}
When I then do a single-flow unidirectional loadtest, the expected outcome is that only one CPU
participates (100% in _kernel_ and 700% in _idle_) and if I perform a single-flow bidirectional
loadtest, my expectations are confirmed again, seeing two CPU threads do the work (200% in _kernel_
and 600% in _idle_).
While the math checks out, the performance is a little bit less impressive:
| Type | Uni/BiDir | Packets/Sec | L2 Bits/Sec | Line Rate |
| ---- | --------- | ----------- | -------- | --------- |
| vm=var2,size=1514 | Unidirectional | 2.02Mpps | 24.77Gbps | 99% |
| vm=var2,size=imix | Unidirectional | 3.48Mpps | 10.23Gbps | 43% |
| vm=var2,size=64 | Unidirectional | 3.61Mpps | 2.43Gbps | 9.7% |
| size=64 | Unidirectional | 1.22Mpps | 0.82Gbps | 3.2% |
| vm=var2,size=1514 | Bidirectional | 3.77Mpps | 46.31Gbps | 93% |
| vm=var2,size=imix | Bidirectional | 3.81Mpps | 11.22Gbps | 24% |
| vm=var2,size=64 | Bidirectional | 4.02Mpps | 2.69Gbps | 5.4% |
| size=64 | Bidirectional | 2.29Mpps | 1.54Gbps | 3.1% |
***Conclusion***: FreeBSD's kernel on this Xeon-D1518 processor can handle about 1.2Mpps per CPU
thread, and I can use only four of them. FreeBSD is happy to forward big packets, and I can
reasonably reach 2x25Gbps but once I start ramping up the packets/sec by lowering the packet size,
things very quickly deteriorate.
### FreeBSD 14: netmap Bridge
Tom pointed out a tool in the source tree, called the _netmap bridge_ originally written by Luigi
Rizzo and Matteo Landi. FreeBSD ships the source code, but you can also take a look at their GitHub
repository [[ref](https://github.com/luigirizzo/netmap/blob/master/apps/bridge/bridge.c)].
What is _netmap_ anyway? It's a framework for extremely fast and efficient packet I/O for userspace
and kernel clients, and for Virtual Machines. It runs on FreeBSD, Linux and some versions of
Windows. As an aside, my buddy Pavel from FastNetMon pointed out a blogpost from 2015 in which
Cloudflare folks described a way to do DDoS mitigation on Linux using traffic classification to
program the network cards to move certain offensive traffic to a dedicated hardware queue, and
service that queue from a _netmap_ client. If you're curious (I certainly was!), you might take a
look at that cool write-up
[[here](https://blog.cloudflare.com/single-rx-queue-kernel-bypass-with-netmap)].
I compile the code and put it to work, and the man-page tells me that I need to fiddle with the
interfaces a bit. They need to be:
* set to _promiscuous_, which makes sense as they have to receive ethernet frames sent to MAC
addresses other than their own
* turn off any hardware offloading, notably `-rxcsum -txcsum -tso4 -tso6 -lro`
* my user needs write permission to `/dev/netmap` to bind the interfaces from userspace.
```
[pim@france /usr/src/tools/tools/netmap]$ make
[pim@france /usr/src/tools/tools/netmap]$ cd /usr/obj/usr/src/amd64.amd64/tools/tools/netmap
[pim@france .../tools/netmap]$ sudo ifconfig ixl0 -rxcsum -txcsum -tso4 -tso6 -lro promisc
[pim@france .../tools/netmap]$ sudo ifconfig ixl1 -rxcsum -txcsum -tso4 -tso6 -lro promisc
[pim@france .../tools/netmap]$ sudo chmod 660 /dev/netmap
[pim@france .../tools/netmap]$ ./bridge -i netmap:ixl0 -i netmap:ixl1
065.804686 main [290] ------- zerocopy supported
065.804708 main [297] Wait 4 secs for link to come up...
075.810547 main [301] Ready to go, ixl0 0x0/4 <-> ixl1 0x0/4.
```
{{< image width="80px" float="left" src="/assets/shared/warning.png" alt="Warning" >}}
I start my first loadtest, which pretty immediately fails. It's an interesting behavior pattern which
I've not seen before. After staring at the problem, and reading the code of `bridge.c`, which is a
remarkably straight forward program, I restart the bridge utility, and traffic passes again but only
for a little while. Whoops!
I took a [[screencast](/assets/freebsd-vpp/netmap_bridge.cast)] in case any kind soul on freebsd-net
wants to take a closer look at this:
{{< image src="/assets/freebsd-vpp/netmap_bridge.gif" alt="FreeBSD netmap Bridge" >}}
I start a bit of trial and error in which I conclude that if I send **a lot** of traffic (like 10Mpps),
forwarding is fine; but if I send **a little** traffic (like 1kpps), at some point forwarding stops
alltogether. So while it's not great, this does allow me to measure the total throughput just by
sending a lot of traffic, say 30Mpps, and seeing what amount comes out the other side.
Here I go, and I'm having fun:
| Type | Uni/BiDir | Packets/Sec | L2 Bits/Sec | Line Rate |
| ---- | --------- | ----------- | -------- | --------- |
| vm=var2,size=1514 | Unidirectional | 2.04Mpps | 24.72Gbps | 100% |
| vm=var2,size=imix | Unidirectional | 8.16Mpps | 23.76Gbps | 100% |
| vm=var2,size=64 | Unidirectional | 10.83Mpps | 5.55Gbps | 29% |
| size=64 | Unidirectional | 11.42Mpps | 5.83Gbps | 31% |
| vm=var2,size=1514 | Bidirectional | 3.91Mpps | 47.27Gbps | 96% |
| vm=var2,size=imix | Bidirectional | 11.31Mpps | 32.74Gbps | 77% |
| vm=var2,size=64 | Bidirectional | 11.39Mpps | 5.83Gbps | 15% |
| size=64 | Bidirectional | 11.57Mpps | 5.93Gbps | 16% |
***Conclusion***: FreeBSD's _netmap_ implementation is also bound by packets/sec, and in this
setup, the Xeon-D1518 machine is capable of forwarding roughly 11.2Mpps. What I find cool is that
single flow or multiple flows doesn't seem to matter that much, in fact bidirectional 64b single
flow loadtest was most favorable at 11.57Mpps, which is _an order of magnitude_ better than using just
the kernel (which clocked in at 1.2Mpps).
### FreeBSD 14: VPP with netmap
It's good to have a baseline on this machine on how the FreeBSD kernel itself performs. But of
course this series is about Vector Packet Processing, so I now turn my attention to the VPP branch
that Tom shared with me. I wrote a bunch of details about the VM and bare metal install in my
[[first article]({{< ref "2024-02-10-vpp-freebsd-1" >}})] so I'll just go straight to the
configuration parts:
```
DBGvpp# create netmap name ixl0
DBGvpp# create netmap name ixl1
DBGvpp# set int state netmap-ixl0 up
DBGvpp# set int state netmap-ixl1 up
DBGvpp# set int l2 xconnect netmap-ixl0 netmap-ixl1
DBGvpp# set int l2 xconnect netmap-ixl1 netmap-ixl0
DBGvpp# show int
Name Idx State MTU (L3/IP4/IP6/MPLS) Counter Count
local0 0 down 0/0/0/0
netmap-ixl0 1 up 9000/0/0/0 rx packets 25622
rx bytes 1537320
tx packets 25437
tx bytes 1526220
netmap-ixl1 2 up 9000/0/0/0 rx packets 25437
rx bytes 1526220
tx packets 25622
tx bytes 1537320
```
At this point I can pretty much rule out that the _netmap_ `bridge.c` is the issue, because a
few seconds after introducing 10Kpps of traffic and seeing it successfully pass, the loadtester
receives no more packets, even though T-Rex is still sending it. However, about a minute later
I can _also_ see the RX **and** TX counters continue to increase in the VPP dataplane:
```
DBGvpp# show int
Name Idx State MTU (L3/IP4/IP6/MPLS) Counter Count
local0 0 down 0/0/0/0
netmap-ixl0 1 up 9000/0/0/0 rx packets 515843
rx bytes 30950580
tx packets 515657
tx bytes 30939420
netmap-ixl1 2 up 9000/0/0/0 rx packets 515657
rx bytes 30939420
tx packets 515843
tx bytes 30950580
```
.. and I can see that every packet that VPP received is accounted for: interface `ixl0` has received
515843 packets, and `ixl1` claims to have transmitted _exactly_ that amount of packets. So I think
perhaps they are getting lost somewhere on egress between the kernel and the Intel i710-XXV network
card.
However, counter to the previous case, I cannot sustain any reasonable amount of traffic, be it
1Kpps, 10Kpps or 10Mpps, the system pretty consistently comes to a halt mere seconds after
introducing the load. Restarting VPP makes it forward traffic again for a few seconds, just to end
up in the same upset state. I don't learn much.
***Conclusion***: This setup with VPP using _netmap_ does not yield results, for the moment. I have a
suspicion that whatever the cause is of the _netmap_ bridge in the previous test, is likely also the
culprit for this test.
### FreeBSD 14: VPP with DPDK
But not all is lost - I have one test left, and judging by what I learned last week when bringing up
the first test environment, this one is going to be a fair bit better. In my previous loadtests, the
network interfaces were on their usual kernel driver (`ixl(4)` in the case of the Intel i710-XXV
interfaces), but now I'm going to mix it up a little, and rebind these interfaces to a specific DPDK
driver called `nic_uio(4)` which stands for _Network Interface Card Userspace Input/Output_:
```
[pim@france ~]$ cat < EOF | sudo tee -a /boot/loader.conf
nic_uio_load="YES"
hw.nic_uio.bdfs="6:0:0,6:0:1"
EOF
```
After I reboot, the network interfaces are gone from the output of `ifconfig(8)`, which is good. I
start up VPP with a minimal config file [[ref](/assets/freebsd-vpp/startup.conf)], which defines
three worker threads and starts DPDK with 3 RX queues and 4 TX queues. It's a common question why
there would be one more TX queue. The explanation is that in VPP, there is one (1) _main_ thread and
zero or more _worker_ threads. If the _main_ thread wants to send traffic (for example, in a plugin
like _LLDP_ which sends periodic announcements), it would be most efficient to use a transmit queue
specific to that _main_ thread. Any return traffic will be picked up by the _DPDK Process_ on worker
threads (as _main_ does not have one of these). That's why the general rule num(TX) = num(RX)+1.
```
[pim@france ~/src/vpp]$ export STARTUP_CONF=/home/pim/src/startup.conf
[pim@france ~/src/vpp]$ gmake run-release
vpp# set int l2 xconnect TwentyFiveGigabitEthernet6/0/0 TwentyFiveGigabitEthernet6/0/1
vpp# set int l2 xconnect TwentyFiveGigabitEthernet6/0/1 TwentyFiveGigabitEthernet6/0/0
vpp# set int state TwentyFiveGigabitEthernet6/0/0 up
vpp# set int state TwentyFiveGigabitEthernet6/0/1 up
vpp# show int
Name Idx State MTU (L3/IP4/IP6/MPLS) Counter Count
TwentyFiveGigabitEthernet6/0/0 1 up 9000/0/0/0 rx packets 11615035382
rx bytes 1785998048960
tx packets 700076496
tx bytes 161043604594
TwentyFiveGigabitEthernet6/0/1 2 up 9000/0/0/0 rx packets 700076542
rx bytes 161043674054
tx packets 11615035440
tx bytes 1785998136540
local0 0 down 0/0/0/0
```
And with that, the dataplane shoots to life and starts forwarding (lots of) packets. To my great
relief, sending either 1kpps or 1Mpps "just works". I can run my loadtest as per normal, first with
1514 byte packets, then imix, then 64 byte packets, and finally single-flow 64 byte packets. And of
course, both unidirectionally and bidirectionally.
I take a look at the system load while the loadtests are running:
{{< image src="/assets/freebsd-vpp/top-vpp-dpdk.png" alt="FreeBSD top" >}}
It is fully expected that the VPP process is spinning 300% +epsilon of CPU time. This is because it
has started three _worker_ threads, and these are execuing the DPDK _Poll Mode Driver_ which is
essentially a tight loop that asks the network cards for work, and if there are any packets
arriving, executes on that work. As such, each _worker_ thread is always burning 100% of its
assigned CPU.
That said, I can take a look at finer grained statistics in the dataplane itself:
```
vpp# show run
Thread 0 vpp_main (lcore 0)
Time .9, 10 sec internal node vector rate 0.00 loops/sec 297041.19
vector rates in 0.0000e0, out 0.0000e0, drop 0.0000e0, punt 0.0000e0
Name State Calls Vectors Suspends Clocks Vectors/Call
ip4-full-reassembly-expire-wal any wait 0 0 18 2.39e3 0.00
ip6-full-reassembly-expire-wal any wait 0 0 18 3.08e3 0.00
unix-cli-process-0 active 0 0 9 7.62e4 0.00
unix-epoll-input polling 13066 0 0 1.50e5 0.00
---------------
Thread 1 vpp_wk_0 (lcore 1)
Time .9, 10 sec internal node vector rate 12.38 loops/sec 1467742.01
vector rates in 5.6294e6, out 5.6294e6, drop 0.0000e0, punt 0.0000e0
Name State Calls Vectors Suspends Clocks Vectors/Call
TwentyFiveGigabitEthernet6/0/1 active 399663 5047800 0 2.20e1 12.63
TwentyFiveGigabitEthernet6/0/1 active 399663 5047800 0 9.54e1 12.63
dpdk-input polling 1531252 5047800 0 1.45e2 3.29
ethernet-input active 399663 5047800 0 3.97e1 12.63
l2-input active 399663 5047800 0 2.93e1 12.63
l2-output active 399663 5047800 0 2.53e1 12.63
unix-epoll-input polling 1494 0 0 3.09e2 0.00
(et cetera)
```
I showed only one _worker_ thread's output, but there are actually three _worker_ threads, and they are
all doing similar work, because they are picking up 33% of the traffic each assigned to the three RX
queues in the network card.
While the overall CPU load is 300%, here I can see a different picture. Thread 0 (the _main_ thread)
is doing essentially ~nothing. It is polling a set of unix sockets in the node called
`unix-epoll-input`, but other than that, _main_ doesn't have much on its plate. Thread 1 however is
a _worker_ thread, and I can see that it is busy doing work:
* `dpdk-input`: it's polling the NIC for work, it has been called 1.53M times, and in total it has
handled just over 5.04M _vectors_ (which are packets). So I can derive, that each time the _Poll
Mode Driver_ gives work, on average there are 3.29 _vectors_ (packets), and each packet is
taking about 145 CPU clocks.
* `ethernet-input`: The DPDK vectors are all ethernet frames coming from the loadtester. Seeing as
I have cross connected all traffic from Tf6/0/0 to Tf6/0/1 and vice-versa, VPP knows that it
should handle the packets in the L2 forwarding path.
* `l2-input` is called with the (list of N) ethernet frames, which all get cross connected to the
output interface, in this case Tf6/0/1.
* `l2-output` prepares the ethernet frames for output into their egress interface.
* `TwentyFiveGigabitEthernet6/0/1-output` (**Note**: the name is truncated) If this were to have
been L3 traffic, this would be the place where the destination MAC address is inserted into the
ethernet frame, but since this is an L2 cross connect, the node simply passes the ethernet frames
through to the final egress node in DPDK.
* `TwentyFiveGigabitEthernet6/0/1-tx` (**Note**: the name is truncated) hands them to the DPDK
driver for marshalling on the wire.
Halfway through, I see that there's an issue with the distribution of ingress traffic over the
three workers, maybe you can spot it too:
```
---------------
Thread 1 vpp_wk_0 (lcore 1)
Time 56.7, 10 sec internal node vector rate 38.59 loops/sec 106879.84
vector rates in 7.2982e6, out 7.2982e6, drop 0.0000e0, punt 0.0000e0
Name State Calls Vectors Suspends Clocks Vectors/Call
TwentyFiveGigabitEthernet6/0/0 active 6689553 206899956 0 1.34e1 30.93
TwentyFiveGigabitEthernet6/0/0 active 6689553 206899956 0 1.37e2 30.93
TwentyFiveGigabitEthernet6/0/1 active 6688572 206902836 0 1.45e1 30.93
TwentyFiveGigabitEthernet6/0/1 active 6688572 206902836 0 1.34e2 30.93
dpdk-input polling 7128012 413802792 0 8.77e1 58.05
ethernet-input active 13378125 413802792 0 2.77e1 30.93
l2-input active 6809002 413802792 0 1.81e1 60.77
l2-output active 6809002 413802792 0 1.68e1 60.77
unix-epoll-input polling 6954 0 0 6.61e2 0.00
---------------
Thread 2 vpp_wk_1 (lcore 2)
Time 56.7, 10 sec internal node vector rate 256.00 loops/sec 7702.68
vector rates in 4.1188e6, out 4.1188e6, drop 0.0000e0, punt 0.0000e0
Name State Calls Vectors Suspends Clocks Vectors/Call
TwentyFiveGigabitEthernet6/0/0 active 456112 116764672 0 1.27e1 256.00
TwentyFiveGigabitEthernet6/0/0 active 456112 116764672 0 2.64e2 256.00
TwentyFiveGigabitEthernet6/0/1 active 456112 116764672 0 1.39e1 256.00
TwentyFiveGigabitEthernet6/0/1 active 456112 116764672 0 2.74e2 256.00
dpdk-input polling 456112 233529344 0 1.41e2 512.00
ethernet-input active 912224 233529344 0 5.71e1 256.00
l2-input active 912224 233529344 0 3.66e1 256.00
l2-output active 912224 233529344 0 1.70e1 256.00
unix-epoll-input polling 445 0 0 9.59e2 0.00
---------------
Thread 3 vpp_wk_2 (lcore 3)
Time 56.7, 10 sec internal node vector rate 256.00 loops/sec 7742.43
vector rates in 4.1188e6, out 4.1188e6, drop 0.0000e0, punt 0.0000e0
Name State Calls Vectors Suspends Clocks Vectors/Call
TwentyFiveGigabitEthernet6/0/0 active 456113 116764928 0 8.94e0 256.00
TwentyFiveGigabitEthernet6/0/0 active 456113 116764928 0 2.81e2 256.00
TwentyFiveGigabitEthernet6/0/1 active 456113 116764928 0 9.54e0 256.00
TwentyFiveGigabitEthernet6/0/1 active 456113 116764928 0 2.72e2 256.00
dpdk-input polling 456113 233529856 0 1.61e2 512.00
ethernet-input active 912226 233529856 0 4.50e1 256.00
l2-input active 912226 233529856 0 2.93e1 256.00
l2-output active 912226 233529856 0 1.23e1 256.00
unix-epoll-input polling 445 0 0 1.03e3 0.00
```
Thread 1 (`vpp_wk_0`) is handling 7.29Mpps and moderately loaded, while Thread 2 and 3 are handling
each 4.11Mpps and are completely pegged. That said, the relative amount of CPU clocks they are
spending per packet is reasonably similar, but they don't quite add up:
* Thread 1 is doing 7.29Mpps and is spending on average 449 CPU cycles per packet. I get this
number by adding up all of the values in the _Clocks_ column, except for the `unix-epoll-input`
node. But that's somewhat strange, because this Xeon D1518 clocks at 2.2GHz -- and yet 7.29M *
449 is 3.27GHz. My experience (in Linux) is that these numbers actually line up quite well.
* Thread 2 is doing 4.12Mpps and is spending on average 816 CPU cycles per packet. This kind of
makes sense as the cycles/packet is roughly double that of thread 1, and the packet/sec is
roughly half ... and the total of 4.12M * 816 is 3.36GHz.
* I can see similarly values for thread 3: 4.12Mpps and also 819 CPU cycles per packet which
amounts to VPP self-reporting using 3.37GHz worth of cycles on this thread.
When I look at the thread to CPU placement, I get another surprise:
```
vpp# show threads
ID Name Type LWP Sched Policy (Priority) lcore Core Socket State
0 vpp_main 100346 (nil) (n/a) 0 42949674294967
1 vpp_wk_0 workers 100473 (nil) (n/a) 1 42949674294967
2 vpp_wk_1 workers 100474 (nil) (n/a) 2 42949674294967
3 vpp_wk_2 workers 100475 (nil) (n/a) 3 42949674294967
vpp# show cpu
Model name: Intel(R) Xeon(R) CPU D-1518 @ 2.20GHz
Microarch model (family): [0x6] Broadwell ([0x56] Broadwell DE) stepping 0x3
Flags: sse3 pclmulqdq ssse3 sse41 sse42 avx rdrand avx2 bmi2 rtm pqm pqe
rdseed aes invariant_tsc
Base frequency: 2.19 GHz
```
The numbers in `show threads` are all messed up, and I don't quite know what to make of it yet. I
think the perhaps overly specific Linux implementation of the thread pool management is throwing off
FreeBSD a bit. Perhaps some profiling could be useful, so I make a note to discuss this with Tom or
the freebsd-net mailing list, who will know a fair bit more about this type of stuff on FreeBSD than
I do.
Anyway, functionally: this works. Performance wise: I have some questions :-) I let all eight
loadtests complete and without further ado, here's the results:
| Type | Uni/BiDir | Packets/Sec | L2 Bits/Sec | Line Rate |
| ---- | --------- | ----------- | -------- | --------- |
| vm=var2,size=1514 | Unidirectional | 2.01Mpps | 24.45Gbps | 99% |
| vm=var2,size=imix | Unidirectional | 8.07Mpps | 23.42Gbps | 99% |
| vm=var2,size=64 | Unidirectional | 23.93Mpps | 12.25Gbps | 64% |
| size=64 | Unidirectional | 12.80Mpps | 6.56Gbps | 34% |
| vm=var2,size=1514 | Bidirectional | 3.91Mpps | 47.35Gbps | 86% |
| vm=var2,size=imix | Bidirectional | 13.38Mpps | 38.81Gbps | 82% |
| vm=var2,size=64 | Bidirectional | 15.56Mpps | 7.97Gbps | 21% |
| size=64 | Bidirectional | 20.96Mpps | 10.73Gbps | 28% |
***Conclusion***: I have to say: 12.8Mpps on a unidirectional 64b single-flow loadtest (thereby only
being able to make use of one DPDK worker), and 20.96Mpps on a bidirectional 64b single-flow
loadtest, is not too shabby. But seeing as one CPU thread can do 12.8Mpps, I would imagine that
three CPU threads would perform at 38.4Mpps or there-abouts, but I'm seeing only 23.9Mpps and some
unexplained variance in per-thread performance.
## Results
I learned a lot! Some hilights:
1. The _netmap_ implementation is not playing ball for the moment, as forwarding stops consistently, in
both the `bridge.c` as well as the VPP plugin.
1. It is clear though, that _netmap_ is a fair bit faster (11.4Mpps) than _kernel forwarding_ which came in at
roughly 1.2Mpps per CPU thread. What's a bit troubling is that _netmap_ doesn't seem to work
very well in VPP -- traffic forwarding also stops here.
1. DPDK performs quite well on FreeBSD, I manage to see a throughput of 20.96Mpps which is almost
twice the throughput of _netmap_, which is cool but I can't quite explain the stark variance
in throughput between the worker threads. Perhaps VPP is placing the workers on hyperthreads?
Perhaps an equivalent of `isolcpus` in the Linux kernel would help?
For the curious, I've bundled up a few files that describe the machine and its setup:
[[dmesg](/assets/freebsd-vpp/france-dmesg.txt)]
[[pciconf](/assets/freebsd-vpp/france-pciconf.txt)]
[[loader.conf](/assets/freebsd-vpp/france-loader.conf)]
[[VPP startup.conf](/assets/freebsd-vpp/france-startup.conf)]