--- date: "2021-08-15T11:13:14Z" title: VPP Linux CP - Part3 aliases: - /s/articles/2021/08/15/vpp-3.html --- {{< image width="200px" float="right" src="/assets/vpp/fdio-color.svg" alt="VPP" >}} # About this series Ever since I first saw VPP - the Vector Packet Processor - I have been deeply impressed with its performance and versatility. For those of us who have used Cisco IOS/XR devices, like the classic _ASR_ (aggregation services router), VPP will look and feel quite familiar as many of the approaches are shared between the two. One thing notably missing, is the higher level control plane, that is to say: there is no OSPF or ISIS, BGP, LDP and the like. This series of posts details my work on a VPP _plugin_ which is called the **Linux Control Plane**, or LCP for short, which creates Linux network devices that mirror their VPP dataplane counterpart. IPv4 and IPv6 traffic, and associated protocols like ARP and IPv6 Neighbor Discovery can now be handled by Linux, while the heavy lifting of packet forwarding is done by the VPP dataplane. Or, said another way: this plugin will allow Linux to use VPP as a software ASIC for fast forwarding, filtering, NAT, and so on, while keeping control of the interface state (links, addresses and routes) itself. When the plugin is completed, running software like [FRR](https://frrouting.org/) or [Bird](https://bird.network.cz/) on top of VPP and achieving >100Mpps and >100Gbps forwarding rates will be well in reach! In this third post, I'll be adding a convenience feature that I think will be popular: the plugin will now automatically create or delete _LIPs_ for sub-interfaces where-ever the parent has a _LIP_ configured. ## My test setup I've extended the setup from the [first post]({{< ref "2021-08-12-vpp-1" >}}). The base configuration for the `enp66s0f0` interface remains exactly the same, but I've also added an LACP `bond0` interface, which also has the whole kitten kaboodle of sub-interfaces defined, see below in the Appendix for details, but here's the table again for reference: | Name | type | Addresses |-----------------|------|---------- | enp66s0f0 | untagged | 10.0.1.2/30 2001:db8:0:1::2/64 | enp66s0f0.q | dot1q 1234 | 10.0.2.2/30 2001:db8:0:2::2/64 | enp66s0f0.qinq | outer dot1q 1234, inner dot1q 1000 | 10.0.3.2/30 2001:db8:0:3::2/64 | enp66s0f0.ad | dot1ad 2345 | 10.0.4.2/30 2001:db8:0:4::2/64 | enp66s0f0.qinad | outer dot1ad 2345, inner dot1q 1000 | 10.0.5.2/30 2001:db8:0:5::2/64 | bond0 | untagged | 10.1.1.2/30 2001:db8:1:1::2/64 | bond0.q | dot1q 1234 | 10.1.2.2/30 2001:db8:1:2::2/64 | bond0.qinq | outer dot1q 1234, inner dot1q 1000 | 10.1.3.2/30 2001:db8:1:3::2/64 | bond0.ad | dot1ad 2345 | 10.1.4.2/30 2001:db8:1:4::2/64 | bond0.qinad | outer dot1ad 2345, inner dot1q 1000 | 10.1.5.2/30 2001:db8:1:5::2/64 The goal of this post is to show what code needed to be written and which changes needed to be made to the plugin, in order to automatically create and delete sub-interfaces. ### Startingpoint Based on the state of the plugin after the [second post]({{< ref "2021-08-13-vpp-2" >}}), operators must create _LIP_ instances for interfaces as well as each sub-interface explicitly: ``` DBGvpp# lcp create TenGigabitEthernet3/0/0 host-if e0 DBGvpp# create sub TenGigabitEthernet3/0/0 1234 DBGvpp# lcp create TenGigabitEthernet3/0/0.1234 host-if e0.1234 DBGvpp# create sub TenGigabitEthernet3/0/0 1235 dot1q 1234 inner-dot1q 1000 exact-match DBGvpp# lcp create TenGigabitEthernet3/0/0.1235 host-if e0.1235 DBGvpp# create sub TenGigabitEthernet3/0/0 1236 dot1ad 2345 exact-match DBGvpp# lcp create TenGigabitEthernet3/0/0.1236 host-if e0.1236 DBGvpp# create sub TenGigabitEthernet3/0/0 1237 dot1ad 2345 inner-dot1q 1000 exact-match DBGvpp# lcp create TenGigabitEthernet3/0/0.1237 host-if e0.1237 ``` But one might ask -- is it really useful to have L3 interfaces in VPP without a companion interface in an appropriate Linux namespace? I think the answer might be 'yes' for individual interfaces (for example, in a mgmt VRF that has no need to run routing protocols), but I also think the answer is probably 'no' for sub-interfaces, once their parent has a _LIP_ defined. ### Configuration The original plugin (the one that ships with VPP 21.06) has a configuration flag that seems promising by defining a flag `interface-auto-create`, but its implementation was never finished. I've removed that flag and replaced it with a new one. The main reason for this decision is that there are actually two kinds of auto configuration: the first one is detailed in this post, but in the future, I will also make it possible to create VPP interfaces by creating their Linux counterpart (eg. `ip link add link e0 name e0.1234 type vlan id 1234` with a configuration statement that might be called `netlink-auto-subint`), and I'd like for the plugin to individually enable/disable both types. Also, I find the name unfortunate, as the feature should create _and delete LIPs_ on sub-interfaces, not just create them. So out with the old, in with the new :) I have to acknowledge that not everybody will want automagically created interfaces, similar to the original configuration, so I define a new configuration flag called `lcp-auto-subint` which goes into the `linux-cp` module configuration stanza in VPP's `startup.conf`, which might look a little like this: ``` linux-cp { default netns dataplane lcp-auto-subint } ``` Based on this config, I set the startup default in `lcp_set_lcp_auto_subint()`, but I realize that an administrator may want to turn it on/off at runtime, too, so I add a CLI getter/setter that interacts with the flag in this [[commit](https://git.ipng.ch/ipng/lcpng/commit/d23aab2d95aabcf24efb9f7aecaf15b513633ab7)]: ``` DBGvpp# show lcp lcp default netns dataplane lcp lcp-auto-subint on lcp lcp-sync off DBGvpp# lcp lcp-auto-subint off DBGvpp# show lcp lcp default netns dataplane lcp lcp-auto-subint off lcp lcp-sync off ``` The prep work for the rest of the interface syncer starts with this [[commit](https://git.ipng.ch/ipng/lcpng/commit/2d00de080bd26d80ce69441b1043de37e0326e0a)], and for the rest of this blog post, the behavior will be in the 'on' position. The code for the configuration toggle is in this [[commit](https://git.ipng.ch/ipng/lcpng/commit/934446dcd97f51c82ddf133ad45b61b3aae14b2d)]. ### Auto create/delete sub-interfaces The original plugin code (that ships with VPP 21.06) made a start by defining a function called `lcp_itf_phy_add()` and registering an intent with `VNET_SW_INTERFACE_ADD_DEL_FUNCTION()`. I've moved the function to the source file I created in [Part 2]({{< ref "2021-08-13-vpp-2" >}}) (called `lcp_if_sync.c`), specifically to handle interface syncing, and gave it a name that matches the VPP callback, so `lcp_itf_interface_add_del()`. The logic of that function is pretty straight forward. I want to only continue if `lcp-auto-subint` is set, and I only want to create or delete sub-interfaces, not parents. This way, the operator can decide on a per-interface basis if they want it to participate in Linux (eg, issuing `lcp create BondEthernet0 host-if be0`). After I've established that (a) the caller wants auto-creation/auto-deletion, and (b) we're fielding a callback for a sub-int, all I must do is: * On creation: does the parent interface `sw->sup_sw_if_index` have a `LIP`? If yes, let's create a `LIP` for this sub-interface, too. We determine that Linux interface name by taking the parent name (say, `be0`), and sticking the sub-int number after it, like `be0.1234`. * On deletion: does this sub-interface we're fielding the callback for have a `LIP`? If yes, then delete it. I noticed that interface deletion had a bug (one that I fell victim to as well: it does not remove the netlink device in the correct network namespace), which I fixed. The code for the auto create/delete and the bugfix is in this [[commit](https://git.ipng.ch/ipng/lcpng/commit/934446dcd97f51c82ddf133ad45b61b3aae14b2d)]. ### Further Work One other thing I noticed (and this is actually a bug!) is that on `BondEthernet` interfaces, upon creation a temporary MAC is assigned, which is subsequently overwritten by the first physical interface that is added to the bond, which means that when a _LIP_ is created _before_ the first interface is added, its MAC will be the temporary MAC. Compare: ``` vppctl create bond mode lacp load-balance l2 vppctl lcp create BondEthernet0 host-if be0 ## MAC of be0 is now a temp/ephemeral MAC vppctl bond add BondEthernet0 TenGigabitEthernet3/0/2 vppctl bond add BondEthernet0 TenGigabitEthernet3/0/3 ## MAC of the BondEthernet0 device is now that of TenGigabitEthernet3/0/2 ## MAC of TenGigabitEthernet3/0/3 is that of BondEthernet0 (ie. Te3/0/2) ``` In such a situation, `be0` will not be reachable unless it's manually set to the correct MAC. I looked around but found no callback of event handler for MAC address changes in VPP -- so I should add one probably, but in the mean time, I'll just add interfaces to the bond before creating the _LIP_, like so: ``` vppctl create bond mode lacp load-balance l2 vppctl bond add BondEthernet0 TenGigabitEthernet3/0/2 ## MAC of the BondEthernet0 device is now that of TenGigabitEthernet3/0/2 vppctl lcp create BondEthernet0 host-if be0 ## MAC of be0 is now that of BondEthernet0 vppctl bond add BondEthernet0 TenGigabitEthernet3/0/3 ## MAC of TenGigabitEthernet3/0/3 is that of BondEthernet0 (ie. Te3/0/2) ``` .. which is an adequate workaround for now. ## Results After this code is in, the operator will only have to create a LIP for the main interfaces, and the plugin will take care of the rest! ``` pim@hippo:~/src/lcpng$ grep 'create' config3.sh vppctl lcp lcp-auto-subint on vppctl lcp create TenGigabitEthernet3/0/0 host-if e0 vppctl create sub TenGigabitEthernet3/0/0 1234 vppctl create sub TenGigabitEthernet3/0/0 1235 dot1q 1234 inner-dot1q 1000 exact-match vppctl create sub TenGigabitEthernet3/0/0 1236 dot1ad 2345 exact-match vppctl create sub TenGigabitEthernet3/0/0 1237 dot1ad 2345 inner-dot1q 1000 exact-match vppctl create bond mode lacp load-balance l2 vppctl lcp create BondEthernet0 host-if be0 vppctl create sub BondEthernet0 1234 vppctl create sub BondEthernet0 1235 dot1q 1234 inner-dot1q 1000 exact-match vppctl create sub BondEthernet0 1236 dot1ad 2345 exact-match vppctl create sub BondEthernet0 1237 dot1ad 2345 inner-dot1q 1000 exact-match ``` And as an end-to-end functional validation, now extended as well to ping the Ubuntu machine over the LACP interface and all of its subinterfaces, works like a charm: ``` pim@hippo:~/src/lcpng$ sudo ip netns exec dataplane ip link | grep e0 1063: e0: mtu 9000 qdisc mq state UNKNOWN mode DEFAULT group default qlen 1000 1064: be0: mtu 9000 qdisc mq state UNKNOWN mode DEFAULT group default qlen 1000 209: e0.1234@e0: mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000 210: e0.1235@e0.1234: mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000 211: e0.1236@e0: mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000 212: e0.1237@e0.1236: mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000 213: be0.1234@be0: mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000 214: be0.1235@be0.1234: mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000 215: be0.1236@be0: mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000 216: be0.1237@be0.1236: mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000 # The TenGigabitEthernet3/0/0 (e0) interfaces pim@hippo:~/src/lcpng$ fping 10.0.1.2 10.0.2.2 10.0.3.2 10.0.4.2 10.0.5.2 10.0.1.2 is alive 10.0.2.2 is alive 10.0.3.2 is alive 10.0.4.2 is alive 10.0.5.2 is alive pim@hippo:~/src/lcpng$ fping6 2001:db8:0:1::2 2001:db8:0:2::2 \ 2001:db8:0:3::2 2001:db8:0:4::2 2001:db8:0:5::2 2001:db8:0:1::2 is alive 2001:db8:0:2::2 is alive 2001:db8:0:3::2 is alive 2001:db8:0:4::2 is alive 2001:db8:0:5::2 is alive ## The BondEthernet0 (be0) interfaces pim@hippo:~/src/lcpng$ fping 10.1.1.2 10.1.2.2 10.1.3.2 10.1.4.2 10.1.5.2 10.1.1.2 is alive 10.1.2.2 is alive 10.1.3.2 is alive 10.1.4.2 is alive 10.1.5.2 is alive pim@hippo:~/src/lcpng$ fping6 2001:db8:1:1::2 2001:db8:1:2::2 \ 2001:db8:1:3::2 2001:db8:1:4::2 2001:db8:1:5::2 2001:db8:1:1::2 is alive 2001:db8:1:2::2 is alive 2001:db8:1:3::2 is alive 2001:db8:1:4::2 is alive 2001:db8:1:5::2 is alive ``` ## Credits I'd like to make clear that the Linux CP plugin is a great collaboration between several great folks and that my work stands on their shoulders. I've had a little bit of help along the way from Neale Ranns, Matthew Smith and Jon Loeliger, and I'd like to thank them for their work! ## Appendix #### Ubuntu config ``` # Untagged interface ip addr add 10.0.1.2/30 dev enp66s0f0 ip addr add 2001:db8:0:1::2/64 dev enp66s0f0 ip link set enp66s0f0 up mtu 9000 # Single 802.1q tag 1234 ip link add link enp66s0f0 name enp66s0f0.q type vlan id 1234 ip link set enp66s0f0.q up mtu 9000 ip addr add 10.0.2.2/30 dev enp66s0f0.q ip addr add 2001:db8:0:2::2/64 dev enp66s0f0.q # Double 802.1q tag 1234 inner-tag 1000 ip link add link enp66s0f0.q name enp66s0f0.qinq type vlan id 1000 ip link set enp66s0f0.qinq up mtu 9000 ip addr add 10.0.3.2/30 dev enp66s0f0.qinq ip addr add 2001:db8:0:3::2/64 dev enp66s0f0.qinq # Single 802.1ad tag 2345 ip link add link enp66s0f0 name enp66s0f0.ad type vlan id 2345 proto 802.1ad ip link set enp66s0f0.ad up mtu 9000 ip addr add 10.0.4.2/30 dev enp66s0f0.ad ip addr add 2001:db8:0:4::2/64 dev enp66s0f0.ad # Double 802.1ad tag 2345 inner-tag 1000 ip link add link enp66s0f0.ad name enp66s0f0.qinad type vlan id 1000 proto 802.1q ip link set enp66s0f0.qinad up mtu 9000 ip addr add 10.0.5.2/30 dev enp66s0f0.qinad ip addr add 2001:db8:0:5::2/64 dev enp66s0f0.qinad ## Bond interface ip link add bond0 type bond mode 802.3ad ip link set enp66s0f2 down ip link set enp66s0f3 down ip link set enp66s0f2 master bond0 ip link set enp66s0f3 master bond0 ip link set enp66s0f2 up ip link set enp66s0f3 up ip link set bond0 up ip addr add 10.1.1.2/30 dev bond0 ip addr add 2001:db8:1:1::2/64 dev bond0 ip link set bond0 up mtu 9000 # Single 802.1q tag 1234 ip link add link bond0 name bond0.q type vlan id 1234 ip link set bond0.q up mtu 9000 ip addr add 10.1.2.2/30 dev bond0.q ip addr add 2001:db8:1:2::2/64 dev bond0.q # Double 802.1q tag 1234 inner-tag 1000 ip link add link bond0.q name bond0.qinq type vlan id 1000 ip link set bond0.qinq up mtu 9000 ip addr add 10.1.3.2/30 dev bond0.qinq ip addr add 2001:db8:1:3::2/64 dev bond0.qinq # Single 802.1ad tag 2345 ip link add link bond0 name bond0.ad type vlan id 2345 proto 802.1ad ip link set bond0.ad up mtu 9000 ip addr add 10.1.4.2/30 dev bond0.ad ip addr add 2001:db8:1:4::2/64 dev bond0.ad # Double 802.1ad tag 2345 inner-tag 1000 ip link add link bond0.ad name bond0.qinad type vlan id 1000 proto 802.1q ip link set bond0.qinad up mtu 9000 ip addr add 10.1.5.2/30 dev bond0.qinad ip addr add 2001:db8:1:5::2/64 dev bond0.qinad ``` #### VPP config ``` ## No more `lcp create` commands for sub-interfaces. vppctl lcp default netns dataplane vppctl lcp lcp-auto-subint on vppctl lcp create TenGigabitEthernet3/0/0 host-if e0 vppctl set interface state TenGigabitEthernet3/0/0 up vppctl set interface mtu packet 9000 TenGigabitEthernet3/0/0 vppctl set interface ip address TenGigabitEthernet3/0/0 10.0.1.1/30 vppctl set interface ip address TenGigabitEthernet3/0/0 2001:db8:0:1::1/64 vppctl create sub TenGigabitEthernet3/0/0 1234 vppctl set interface mtu packet 9000 TenGigabitEthernet3/0/0.1234 vppctl set interface state TenGigabitEthernet3/0/0.1234 up vppctl set interface ip address TenGigabitEthernet3/0/0.1234 10.0.2.1/30 vppctl set interface ip address TenGigabitEthernet3/0/0.1234 2001:db8:0:2::1/64 vppctl create sub TenGigabitEthernet3/0/0 1235 dot1q 1234 inner-dot1q 1000 exact-match vppctl set interface state TenGigabitEthernet3/0/0.1235 up vppctl set interface mtu packet 9000 TenGigabitEthernet3/0/0.1235 vppctl set interface ip address TenGigabitEthernet3/0/0.1235 10.0.3.1/30 vppctl set interface ip address TenGigabitEthernet3/0/0.1235 2001:db8:0:3::1/64 vppctl create sub TenGigabitEthernet3/0/0 1236 dot1ad 2345 exact-match vppctl set interface state TenGigabitEthernet3/0/0.1236 up vppctl set interface mtu packet 9000 TenGigabitEthernet3/0/0.1236 vppctl set interface ip address TenGigabitEthernet3/0/0.1236 10.0.4.1/30 vppctl set interface ip address TenGigabitEthernet3/0/0.1236 2001:db8:0:4::1/64 vppctl create sub TenGigabitEthernet3/0/0 1237 dot1ad 2345 inner-dot1q 1000 exact-match vppctl set interface state TenGigabitEthernet3/0/0.1237 up vppctl set interface mtu packet 9000 TenGigabitEthernet3/0/0.1237 vppctl set interface ip address TenGigabitEthernet3/0/0.1237 10.0.5.1/30 vppctl set interface ip address TenGigabitEthernet3/0/0.1237 2001:db8:0:5::1/64 ## The LACP bond vppctl create bond mode lacp load-balance l2 vppctl bond add BondEthernet0 TenGigabitEthernet3/0/2 vppctl bond add BondEthernet0 TenGigabitEthernet3/0/3 vppctl lcp create BondEthernet0 host-if be0 vppctl set interface state TenGigabitEthernet3/0/2 up vppctl set interface state TenGigabitEthernet3/0/3 up vppctl set interface state BondEthernet0 up vppctl set interface mtu packet 9000 BondEthernet0 vppctl set interface ip address BondEthernet0 10.1.1.1/30 vppctl set interface ip address BondEthernet0 2001:db8:1:1::1/64 vppctl create sub BondEthernet0 1234 vppctl set interface mtu packet 9000 BondEthernet0.1234 vppctl set interface state BondEthernet0.1234 up vppctl set interface ip address BondEthernet0.1234 10.1.2.1/30 vppctl set interface ip address BondEthernet0.1234 2001:db8:1:2::1/64 vppctl create sub BondEthernet0 1235 dot1q 1234 inner-dot1q 1000 exact-match vppctl set interface state BondEthernet0.1235 up vppctl set interface mtu packet 9000 BondEthernet0.1235 vppctl set interface ip address BondEthernet0.1235 10.1.3.1/30 vppctl set interface ip address BondEthernet0.1235 2001:db8:1:3::1/64 vppctl create sub BondEthernet0 1236 dot1ad 2345 exact-match vppctl set interface state BondEthernet0.1236 up vppctl set interface mtu packet 9000 BondEthernet0.1236 vppctl set interface ip address BondEthernet0.1236 10.1.4.1/30 vppctl set interface ip address BondEthernet0.1236 2001:db8:1:4::1/64 vppctl create sub BondEthernet0 1237 dot1ad 2345 inner-dot1q 1000 exact-match vppctl set interface state BondEthernet0.1237 up vppctl set interface mtu packet 9000 BondEthernet0.1237 vppctl set interface ip address BondEthernet0.1237 10.1.5.1/30 vppctl set interface ip address BondEthernet0.1237 2001:db8:1:5::1/64 ``` #### Final note You may have noticed that the [commit] links are all git commits in my private working copy. I want to wait until my [previous work](https://gerrit.fd.io/r/c/vpp/+/33481) is reviewed and submitted before piling on more changes. Feel free to contact vpp-dev@ for more information in the mean time :-)