ipng/ipng.ch
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
fdb77838b8b04c9de622fad01d2fec97e091c1b6
ipng.ch/content/articles/2021-08-15-vpp-3.md
Pim van Pelt fdb77838b8
All checks were successful
continuous-integration/drone/push Build is passing
Details
Rewrite github.com to git.ipng.ch for popular repos
2025-05-04 21:54:16 +02:00

417 lines
19 KiB
Markdown
Raw Blame History

---
date: "2021-08-15T11:13:14Z"
title: VPP Linux CP - Part3
aliases:
- /s/articles/2021/08/15/vpp-3.html
---
{{< image width="200px" float="right" src="/assets/vpp/fdio-color.svg" alt="VPP" >}}
# About this series
Ever since I first saw VPP - the Vector Packet Processor - I have been deeply impressed with its
performance and versatility. For those of us who have used Cisco IOS/XR devices, like the classic
_ASR_ (aggregation services router), VPP will look and feel quite familiar as many of the approaches
are shared between the two. One thing notably missing, is the higher level control plane, that is
to say: there is no OSPF or ISIS, BGP, LDP and the like. This series of posts details my work on a
VPP _plugin_ which is called the **Linux Control Plane**, or LCP for short, which creates Linux network
devices that mirror their VPP dataplane counterpart. IPv4 and IPv6 traffic, and associated protocols
like ARP and IPv6 Neighbor Discovery can now be handled by Linux, while the heavy lifting of packet
forwarding is done by the VPP dataplane. Or, said another way: this plugin will allow Linux to use
VPP as a software ASIC for fast forwarding, filtering, NAT, and so on, while keeping control of the
interface state (links, addresses and routes) itself. When the plugin is completed, running software
like [FRR](https://frrouting.org/) or [Bird](https://bird.network.cz/) on top of VPP and achieving
&gt;100Mpps and &gt;100Gbps forwarding rates will be well in reach!
In this third post, I'll be adding a convenience feature that I think will be popular: the plugin
will now automatically create or delete _LIPs_ for sub-interfaces where-ever the parent has a _LIP_
configured.
## My test setup
I've extended the setup from the [first post]({{< ref "2021-08-12-vpp-1" >}}). The base
configuration for the `enp66s0f0` interface remains exactly the same, but I've also added
an LACP `bond0` interface, which also has the whole kitten kaboodle of sub-interfaces defined, see
below in the Appendix for details, but here's the table again for reference:
| Name | type | Addresses
|-----------------|------|----------
| enp66s0f0 | untagged | 10.0.1.2/30 2001:db8:0:1::2/64
| enp66s0f0.q | dot1q 1234 | 10.0.2.2/30 2001:db8:0:2::2/64
| enp66s0f0.qinq | outer dot1q 1234, inner dot1q 1000 | 10.0.3.2/30 2001:db8:0:3::2/64
| enp66s0f0.ad | dot1ad 2345 | 10.0.4.2/30 2001:db8:0:4::2/64
| enp66s0f0.qinad | outer dot1ad 2345, inner dot1q 1000 | 10.0.5.2/30 2001:db8:0:5::2/64
| bond0 | untagged | 10.1.1.2/30 2001:db8:1:1::2/64
| bond0.q | dot1q 1234 | 10.1.2.2/30 2001:db8:1:2::2/64
| bond0.qinq | outer dot1q 1234, inner dot1q 1000 | 10.1.3.2/30 2001:db8:1:3::2/64
| bond0.ad | dot1ad 2345 | 10.1.4.2/30 2001:db8:1:4::2/64
| bond0.qinad | outer dot1ad 2345, inner dot1q 1000 | 10.1.5.2/30 2001:db8:1:5::2/64
The goal of this post is to show what code needed to be written and which changes needed to be
made to the plugin, in order to automatically create and delete sub-interfaces.
### Startingpoint
Based on the state of the plugin after the [second post]({{< ref "2021-08-13-vpp-2" >}}),
operators must create _LIP_ instances for interfaces as well as each sub-interface
explicitly:
```
DBGvpp# lcp create TenGigabitEthernet3/0/0 host-if e0
DBGvpp# create sub TenGigabitEthernet3/0/0 1234
DBGvpp# lcp create TenGigabitEthernet3/0/0.1234 host-if e0.1234
DBGvpp# create sub TenGigabitEthernet3/0/0 1235 dot1q 1234 inner-dot1q 1000 exact-match
DBGvpp# lcp create TenGigabitEthernet3/0/0.1235 host-if e0.1235
DBGvpp# create sub TenGigabitEthernet3/0/0 1236 dot1ad 2345 exact-match
DBGvpp# lcp create TenGigabitEthernet3/0/0.1236 host-if e0.1236
DBGvpp# create sub TenGigabitEthernet3/0/0 1237 dot1ad 2345 inner-dot1q 1000 exact-match
DBGvpp# lcp create TenGigabitEthernet3/0/0.1237 host-if e0.1237
```
But one might ask -- is it really useful to have L3 interfaces in VPP without a companion interface
in an appropriate Linux namespace? I think the answer might be 'yes' for individual interfaces
(for example, in a mgmt VRF that has no need to run routing protocols), but I also think the answer
is probably 'no' for sub-interfaces, once their parent has a _LIP_ defined.
### Configuration
The original plugin (the one that ships with VPP 21.06) has a configuration flag that seems
promising by defining a flag `interface-auto-create`, but its implementation was never finished.
I've removed that flag and replaced it with a new one. The main reason for this decision is
that there are actually two kinds of auto configuration: the first one is detailed in this post,
but in the future, I will also make it possible to create VPP interfaces by creating their Linux
counterpart (eg. `ip link add link e0 name e0.1234 type vlan id 1234` with a configuration statement
that might be called `netlink-auto-subint`), and I'd like for the plugin to individually
enable/disable both types. Also, I find the name unfortunate, as the feature should create
_and delete LIPs_ on sub-interfaces, not just create them. So out with the old, in with the new :)
I have to acknowledge that not everybody will want automagically created interfaces, similar to
the original configuration, so I define a new configuration flag called `lcp-auto-subint` which goes
into the `linux-cp` module configuration stanza in VPP's `startup.conf`, which might look a little
like this:
```
linux-cp {
default netns dataplane
lcp-auto-subint
}
```
Based on this config, I set the startup default in `lcp_set_lcp_auto_subint()`, but I realize that
an administrator may want to turn it on/off at runtime, too, so I add a CLI getter/setter that
interacts with the flag in this [[commit](https://git.ipng.ch/ipng/lcpng/commit/d23aab2d95aabcf24efb9f7aecaf15b513633ab7)]:
```
DBGvpp# show lcp
lcp default netns dataplane
lcp lcp-auto-subint on
lcp lcp-sync off
DBGvpp# lcp lcp-auto-subint off
DBGvpp# show lcp
lcp default netns dataplane
lcp lcp-auto-subint off
lcp lcp-sync off
```
The prep work for the rest of the interface syncer starts with this
[[commit](https://git.ipng.ch/ipng/lcpng/commit/2d00de080bd26d80ce69441b1043de37e0326e0a)], and
for the rest of this blog post, the behavior will be in the 'on' position.
The code for the configuration toggle is in this
[[commit](https://git.ipng.ch/ipng/lcpng/commit/934446dcd97f51c82ddf133ad45b61b3aae14b2d)].
### Auto create/delete sub-interfaces
The original plugin code (that ships with VPP 21.06) made a start by defining a function called
`lcp_itf_phy_add()` and registering an intent with `VNET_SW_INTERFACE_ADD_DEL_FUNCTION()`. I've
moved the function to the source file I created in [Part 2]({{< ref "2021-08-13-vpp-2" >}})
(called `lcp_if_sync.c`), specifically to handle interface syncing, and gave it a name that
matches the VPP callback, so `lcp_itf_interface_add_del()`.
The logic of that function is pretty straight forward. I want to only continue if `lcp-auto-subint`
is set, and I only want to create or delete sub-interfaces, not parents. This way, the operator
can decide on a per-interface basis if they want it to participate in Linux (eg, issuing
`lcp create BondEthernet0 host-if be0`). After I've established that (a) the caller wants
auto-creation/auto-deletion, and (b) we're fielding a callback for a sub-int, all I must do is:
* On creation: does the parent interface `sw->sup_sw_if_index` have a `LIP`? If yes, let's
create a `LIP` for this sub-interface, too. We determine that Linux interface name by taking
the parent name (say, `be0`), and sticking the sub-int number after it, like `be0.1234`.
* On deletion: does this sub-interface we're fielding the callback for have a `LIP`? If yes,
then delete it.
I noticed that interface deletion had a bug (one that I fell victim to as well: it does not
remove the netlink device in the correct network namespace), which I fixed.
The code for the auto create/delete and the bugfix is in this
[[commit](https://git.ipng.ch/ipng/lcpng/commit/934446dcd97f51c82ddf133ad45b61b3aae14b2d)].
### Further Work
One other thing I noticed (and this is actually a bug!) is that on `BondEthernet`
interfaces, upon creation a temporary MAC is assigned, which is subsequently
overwritten by the first physical interface that is added to the bond, which means
that when a _LIP_ is created _before_ the first interface is added, its MAC
will be the temporary MAC. Compare:
```
vppctl create bond mode lacp load-balance l2
vppctl lcp create BondEthernet0 host-if be0
## MAC of be0 is now a temp/ephemeral MAC
vppctl bond add BondEthernet0 TenGigabitEthernet3/0/2
vppctl bond add BondEthernet0 TenGigabitEthernet3/0/3
## MAC of the BondEthernet0 device is now that of TenGigabitEthernet3/0/2
## MAC of TenGigabitEthernet3/0/3 is that of BondEthernet0 (ie. Te3/0/2)
```
In such a situation, `be0` will not be reachable unless it's manually set to the correct MAC.
I looked around but found no callback of event handler for MAC address changes in VPP -- so I
should add one probably, but in the mean time, I'll just add interfaces to the bond before
creating the _LIP_, like so:
```
vppctl create bond mode lacp load-balance l2
vppctl bond add BondEthernet0 TenGigabitEthernet3/0/2
## MAC of the BondEthernet0 device is now that of TenGigabitEthernet3/0/2
vppctl lcp create BondEthernet0 host-if be0
## MAC of be0 is now that of BondEthernet0
vppctl bond add BondEthernet0 TenGigabitEthernet3/0/3
## MAC of TenGigabitEthernet3/0/3 is that of BondEthernet0 (ie. Te3/0/2)
```
.. which is an adequate workaround for now.
## Results
After this code is in, the operator will only have to create a LIP for the main interfaces, and
the plugin will take care of the rest!
```
pim@hippo:~/src/lcpng$ grep 'create' config3.sh
vppctl lcp lcp-auto-subint on
vppctl lcp create TenGigabitEthernet3/0/0 host-if e0
vppctl create sub TenGigabitEthernet3/0/0 1234
vppctl create sub TenGigabitEthernet3/0/0 1235 dot1q 1234 inner-dot1q 1000 exact-match
vppctl create sub TenGigabitEthernet3/0/0 1236 dot1ad 2345 exact-match
vppctl create sub TenGigabitEthernet3/0/0 1237 dot1ad 2345 inner-dot1q 1000 exact-match
vppctl create bond mode lacp load-balance l2
vppctl lcp create BondEthernet0 host-if be0
vppctl create sub BondEthernet0 1234
vppctl create sub BondEthernet0 1235 dot1q 1234 inner-dot1q 1000 exact-match
vppctl create sub BondEthernet0 1236 dot1ad 2345 exact-match
vppctl create sub BondEthernet0 1237 dot1ad 2345 inner-dot1q 1000 exact-match
```
And as an end-to-end functional validation, now extended as well to ping the Ubuntu machine over
the LACP interface and all of its subinterfaces, works like a charm:
```
pim@hippo:~/src/lcpng$ sudo ip netns exec dataplane ip link | grep e0
1063: e0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UNKNOWN mode DEFAULT group default qlen 1000
1064: be0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UNKNOWN mode DEFAULT group default qlen 1000
209: e0.1234@e0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
210: e0.1235@e0.1234: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
211: e0.1236@e0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
212: e0.1237@e0.1236: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
213: be0.1234@be0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
214: be0.1235@be0.1234: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
215: be0.1236@be0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
216: be0.1237@be0.1236: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
# The TenGigabitEthernet3/0/0 (e0) interfaces
pim@hippo:~/src/lcpng$ fping 10.0.1.2 10.0.2.2 10.0.3.2 10.0.4.2 10.0.5.2
10.0.1.2 is alive
10.0.2.2 is alive
10.0.3.2 is alive
10.0.4.2 is alive
10.0.5.2 is alive
pim@hippo:~/src/lcpng$ fping6 2001:db8:0:1::2 2001:db8:0:2::2 \
2001:db8:0:3::2 2001:db8:0:4::2 2001:db8:0:5::2
2001:db8:0:1::2 is alive
2001:db8:0:2::2 is alive
2001:db8:0:3::2 is alive
2001:db8:0:4::2 is alive
2001:db8:0:5::2 is alive
## The BondEthernet0 (be0) interfaces
pim@hippo:~/src/lcpng$ fping 10.1.1.2 10.1.2.2 10.1.3.2 10.1.4.2 10.1.5.2
10.1.1.2 is alive
10.1.2.2 is alive
10.1.3.2 is alive
10.1.4.2 is alive
10.1.5.2 is alive
pim@hippo:~/src/lcpng$ fping6 2001:db8:1:1::2 2001:db8:1:2::2 \
2001:db8:1:3::2 2001:db8:1:4::2 2001:db8:1:5::2
2001:db8:1:1::2 is alive
2001:db8:1:2::2 is alive
2001:db8:1:3::2 is alive
2001:db8:1:4::2 is alive
2001:db8:1:5::2 is alive
```
## Credits
I'd like to make clear that the Linux CP plugin is a great collaboration between several great folks
and that my work stands on their shoulders. I've had a little bit of help along the way from Neale
Ranns, Matthew Smith and Jon Loeliger, and I'd like to thank them for their work!
## Appendix
#### Ubuntu config
```
# Untagged interface
ip addr add 10.0.1.2/30 dev enp66s0f0
ip addr add 2001:db8:0:1::2/64 dev enp66s0f0
ip link set enp66s0f0 up mtu 9000
# Single 802.1q tag 1234
ip link add link enp66s0f0 name enp66s0f0.q type vlan id 1234
ip link set enp66s0f0.q up mtu 9000
ip addr add 10.0.2.2/30 dev enp66s0f0.q
ip addr add 2001:db8:0:2::2/64 dev enp66s0f0.q
# Double 802.1q tag 1234 inner-tag 1000
ip link add link enp66s0f0.q name enp66s0f0.qinq type vlan id 1000
ip link set enp66s0f0.qinq up mtu 9000
ip addr add 10.0.3.2/30 dev enp66s0f0.qinq
ip addr add 2001:db8:0:3::2/64 dev enp66s0f0.qinq
# Single 802.1ad tag 2345
ip link add link enp66s0f0 name enp66s0f0.ad type vlan id 2345 proto 802.1ad
ip link set enp66s0f0.ad up mtu 9000
ip addr add 10.0.4.2/30 dev enp66s0f0.ad
ip addr add 2001:db8:0:4::2/64 dev enp66s0f0.ad
# Double 802.1ad tag 2345 inner-tag 1000
ip link add link enp66s0f0.ad name enp66s0f0.qinad type vlan id 1000 proto 802.1q
ip link set enp66s0f0.qinad up mtu 9000
ip addr add 10.0.5.2/30 dev enp66s0f0.qinad
ip addr add 2001:db8:0:5::2/64 dev enp66s0f0.qinad
## Bond interface
ip link add bond0 type bond mode 802.3ad
ip link set enp66s0f2 down
ip link set enp66s0f3 down
ip link set enp66s0f2 master bond0
ip link set enp66s0f3 master bond0
ip link set enp66s0f2 up
ip link set enp66s0f3 up
ip link set bond0 up
ip addr add 10.1.1.2/30 dev bond0
ip addr add 2001:db8:1:1::2/64 dev bond0
ip link set bond0 up mtu 9000
# Single 802.1q tag 1234
ip link add link bond0 name bond0.q type vlan id 1234
ip link set bond0.q up mtu 9000
ip addr add 10.1.2.2/30 dev bond0.q
ip addr add 2001:db8:1:2::2/64 dev bond0.q
# Double 802.1q tag 1234 inner-tag 1000
ip link add link bond0.q name bond0.qinq type vlan id 1000
ip link set bond0.qinq up mtu 9000
ip addr add 10.1.3.2/30 dev bond0.qinq
ip addr add 2001:db8:1:3::2/64 dev bond0.qinq
# Single 802.1ad tag 2345
ip link add link bond0 name bond0.ad type vlan id 2345 proto 802.1ad
ip link set bond0.ad up mtu 9000
ip addr add 10.1.4.2/30 dev bond0.ad
ip addr add 2001:db8:1:4::2/64 dev bond0.ad
# Double 802.1ad tag 2345 inner-tag 1000
ip link add link bond0.ad name bond0.qinad type vlan id 1000 proto 802.1q
ip link set bond0.qinad up mtu 9000
ip addr add 10.1.5.2/30 dev bond0.qinad
ip addr add 2001:db8:1:5::2/64 dev bond0.qinad
```
#### VPP config
```
## No more `lcp create` commands for sub-interfaces.
vppctl lcp default netns dataplane
vppctl lcp lcp-auto-subint on
vppctl lcp create TenGigabitEthernet3/0/0 host-if e0
vppctl set interface state TenGigabitEthernet3/0/0 up
vppctl set interface mtu packet 9000 TenGigabitEthernet3/0/0
vppctl set interface ip address TenGigabitEthernet3/0/0 10.0.1.1/30
vppctl set interface ip address TenGigabitEthernet3/0/0 2001:db8:0:1::1/64
vppctl create sub TenGigabitEthernet3/0/0 1234
vppctl set interface mtu packet 9000 TenGigabitEthernet3/0/0.1234
vppctl set interface state TenGigabitEthernet3/0/0.1234 up
vppctl set interface ip address TenGigabitEthernet3/0/0.1234 10.0.2.1/30
vppctl set interface ip address TenGigabitEthernet3/0/0.1234 2001:db8:0:2::1/64
vppctl create sub TenGigabitEthernet3/0/0 1235 dot1q 1234 inner-dot1q 1000 exact-match
vppctl set interface state TenGigabitEthernet3/0/0.1235 up
vppctl set interface mtu packet 9000 TenGigabitEthernet3/0/0.1235
vppctl set interface ip address TenGigabitEthernet3/0/0.1235 10.0.3.1/30
vppctl set interface ip address TenGigabitEthernet3/0/0.1235 2001:db8:0:3::1/64
vppctl create sub TenGigabitEthernet3/0/0 1236 dot1ad 2345 exact-match
vppctl set interface state TenGigabitEthernet3/0/0.1236 up
vppctl set interface mtu packet 9000 TenGigabitEthernet3/0/0.1236
vppctl set interface ip address TenGigabitEthernet3/0/0.1236 10.0.4.1/30
vppctl set interface ip address TenGigabitEthernet3/0/0.1236 2001:db8:0:4::1/64
vppctl create sub TenGigabitEthernet3/0/0 1237 dot1ad 2345 inner-dot1q 1000 exact-match
vppctl set interface state TenGigabitEthernet3/0/0.1237 up
vppctl set interface mtu packet 9000 TenGigabitEthernet3/0/0.1237
vppctl set interface ip address TenGigabitEthernet3/0/0.1237 10.0.5.1/30
vppctl set interface ip address TenGigabitEthernet3/0/0.1237 2001:db8:0:5::1/64
## The LACP bond
vppctl create bond mode lacp load-balance l2
vppctl bond add BondEthernet0 TenGigabitEthernet3/0/2
vppctl bond add BondEthernet0 TenGigabitEthernet3/0/3
vppctl lcp create BondEthernet0 host-if be0
vppctl set interface state TenGigabitEthernet3/0/2 up
vppctl set interface state TenGigabitEthernet3/0/3 up
vppctl set interface state BondEthernet0 up
vppctl set interface mtu packet 9000 BondEthernet0
vppctl set interface ip address BondEthernet0 10.1.1.1/30
vppctl set interface ip address BondEthernet0 2001:db8:1:1::1/64
vppctl create sub BondEthernet0 1234
vppctl set interface mtu packet 9000 BondEthernet0.1234
vppctl set interface state BondEthernet0.1234 up
vppctl set interface ip address BondEthernet0.1234 10.1.2.1/30
vppctl set interface ip address BondEthernet0.1234 2001:db8:1:2::1/64
vppctl create sub BondEthernet0 1235 dot1q 1234 inner-dot1q 1000 exact-match
vppctl set interface state BondEthernet0.1235 up
vppctl set interface mtu packet 9000 BondEthernet0.1235
vppctl set interface ip address BondEthernet0.1235 10.1.3.1/30
vppctl set interface ip address BondEthernet0.1235 2001:db8:1:3::1/64
vppctl create sub BondEthernet0 1236 dot1ad 2345 exact-match
vppctl set interface state BondEthernet0.1236 up
vppctl set interface mtu packet 9000 BondEthernet0.1236
vppctl set interface ip address BondEthernet0.1236 10.1.4.1/30
vppctl set interface ip address BondEthernet0.1236 2001:db8:1:4::1/64
vppctl create sub BondEthernet0 1237 dot1ad 2345 inner-dot1q 1000 exact-match
vppctl set interface state BondEthernet0.1237 up
vppctl set interface mtu packet 9000 BondEthernet0.1237
vppctl set interface ip address BondEthernet0.1237 10.1.5.1/30
vppctl set interface ip address BondEthernet0.1237 2001:db8:1:5::1/64
```
#### Final note
You may have noticed that the [commit] links are all git commits in my private working copy.
I want to wait until my [previous work](https://gerrit.fd.io/r/c/vpp/+/33481) is reviewed and
submitted before piling on more changes. Feel free to contact vpp-dev@ for more information in the
mean time :-)
Reference in New Issue View Git Blame Copy Permalink