Rewrite all images to Hugo format

2024-08-05 01:11:52 +02:00
parent 4230fd9acc
commit c1f1775c91
67 changed files with 29916 additions and 23 deletions
--- a/content/articles/2021-08-13-vpp-2.md
+++ b/content/articles/2021-08-13-vpp-2.md
@ -0,0 +1,345 @@
+---
+custom_css:
+- /assets/vpp/asciinema-player.css
+date: "2021-08-13T15:33:14Z"
+title: VPP Linux CP - Part2
+---
+
+
+{{< image width="200px" float="right" src="/assets/vpp/fdio-color.svg" alt="VPP" >}}
+
+# About this series
+
+Ever since I first saw VPP - the Vector Packet Processor - I have been deeply impressed with its
+performance and versatility. For those of us who have used Cisco IOS/XR devices, like the classic
+_ASR_ (aggregation services router), VPP will look and feel quite familiar as many of the approaches
+are shared between the two. One thing notably missing, is the higher level control plane, that is
+to say: there is no OSPF or ISIS, BGP, LDP and the like. This series of posts details my work on a
+VPP _plugin_ which is called the **Linux Control Plane**, or LCP for short, which creates Linux network
+devices that mirror their VPP dataplane counterpart. IPv4 and IPv6 traffic, and associated protocols
+like ARP and IPv6 Neighbor Discovery can now be handled by Linux, while the heavy lifting of packet
+forwarding is done by the VPP dataplane. Or, said another way: this plugin will allow Linux to use
+VPP as a software ASIC for fast forwarding, filtering, NAT, and so on, while keeping control of the
+interface state (links, addresses and routes) itself. When the plugin is completed, running software
+like [FRR](https://frrouting.org/) or [Bird](https://bird.network.cz/) on top of VPP and achieving
+&gt;100Mpps and &gt;100Gbps forwarding rates will be well in reach!
+
+In this second post, let's make the plugin a bit more useful by making it copy forward state changes
+to interfaces in VPP, into their Linux CP counterparts.
+
+## My test setup
+
+I'm using the same setup from the [previous post]({% post_url 2021-08-12-vpp-1 %}). The goal of this
+post is to show what code needed to be written and which changes needed to be made to the plugin, in
+order to propagate changes to VPP interfaces to the Linux TAP devices.
+
+### Startingpoint
+
+The `linux-cp` plugin that ships with VPP 21.06, even with my [changes](https://gerrit.fd.io/r/c/vpp/+/33481)
+is still _only_ able to create _LIP_ devices. It's not very user friendly to have to
+apply state changes meticulously on both sides, but it can be done:
+
+```
+vppctl lcp create TenGigabitEthernet3/0/0 host-if e0
+vppctl set interface state TenGigabitEthernet3/0/0 up
+vppctl set interface mtu packet 9000 TenGigabitEthernet3/0/0
+vppctl set interface ip address TenGigabitEthernet3/0/0 10.0.1.1/30
+vppctl set interface ip address TenGigabitEthernet3/0/0 2001:db8:0:1::1/64
+ip link set e0 up
+ip link set e0 mtu 9000
+ip addr add 10.0.1.1/30 dev e0
+ip addr add 2001:db8:0:1::1/64 dev e0
+```
+
+In this snippet, we can see that after creating the _LIP_, thus conjuring up the unconfigured
+`e0` interface in Linux, I changed the VPP interface in three ways:
+1.   I set the state of the VPP interface to 'up'
+1.   I set the MTU of the VPP interface to 9000
+1.   I add an IPv4 and IPv6 address to the interface
+
+Because state does not (yet) propagate, I have to make those changes as well on the Linux side
+with the subsequent `ip` commands.
+
+### Configuration
+
+I can imagine that operators want to have more control and facilitate the Linux and VPP changes
+themselves. This is why I'll start off by adding a variable called `lcp_sync`, along with a
+startup configuration keyword and a CLI setter. This allows me to turn the whole sync behavior on
+and off, for example in `startup.conf`:
+
+```
+linux-cp {
+  default netns dataplane
+  lcp-sync
+}
+```
+
+And in the CLI:
+```
+DBGvpp# show lcp
+lcp default netns dataplane
+lcp lcp-sync on
+
+DBGvpp# lcp lcp-sync off
+DBGvpp# show lcp
+lcp default netns dataplane
+lcp lcp-sync off
+```
+
+The prep work for the rest of the interface syncer starts with this
+[[commit](https://github.com/pimvanpelt/lcpng/commit/2d00de080bd26d80ce69441b1043de37e0326e0a)], and
+for the rest of this blog post, the behavior will be in the 'on' position.
+
+### Change interface: state
+
+Immediately, I find a dissonance between VPP and Linux: When Linux sets a parent interface down,
+all children go to state `M-DOWN`.  When Linux sets a parent interface up, all of its children
+automatically go to state `UP` and `LOWER_UP`. To illustrate:
+
+```
+ip link set enp66s0f1 down
+ip link add link enp66s0f1 name foo type vlan id 1234
+ip link set foo down
+## Both interfaces are down, which makes sense because I set them both down
+ip link | grep enp66s0f1
+9: enp66s0f1: <BROADCAST,MULTICAST> mtu 9000 qdisc mq state DOWN mode DEFAULT group default qlen 1000
+61: foo@enp66s0f1: <BROADCAST,MULTICAST,M-DOWN> mtu 9000 qdisc noop state DOWN mode DEFAULT group default qlen 1000
+
+ip link set enp66s0f1 up
+ip link | grep enp66s0f1
+## Both interfaces are up, which doesn't make sense because I only changed one of them!
+9: enp66s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000
+61: foo@enp66s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
+```
+
+VPP does not work this way. In VPP, the admin state of each interface is individually
+controllable, so it's possible to bring up the parent while leaving the sub-interface in
+the state it was. I did notice that you can't bring up a sub-interface if its parent
+is down, which I found counterintuitive, but that's neither here nor there.
+
+All of this is to say that we have to be careful when copying state forward, because as
+this [[commit](https://github.com/pimvanpelt/lcpng/commit/7c15c84f6c4739860a85c599779c199cb9efef03)]
+shows, issuing `set int state ... up` on an interface, won't touch its sub-interfaces in VPP, but
+the subsequent netlink message to bring the _LIP_ for that interface up, **will** update the
+children, thus desynchronising Linux and VPP: Linux will have interface **and all its
+sub-interfaces** up unconditionally; VPP will have the interface up and its sub-interfaces in
+whatever state they were before.
+
+To address this, a second
+[[commit](https://github.com/pimvanpelt/lcpng/commit/a3dc56c01461bdffcac8193ead654ae79225220f)] was
+needed. I'm not too sure I want to keep this behavior, but for now, it results in an intuitive
+end-state, which is that all interfaces states are exactly the same between Linux and VPP.
+
+```
+DBGvpp# create sub TenGigabitEthernet3/0/0 10
+DBGvpp# lcp create TenGigabitEthernet3/0/0 host-if e0
+DBGvpp# lcp create TenGigabitEthernet3/0/0.10 host-if e0.10
+DBGvpp# set int state TenGigabitEthernet3/0/0 up
+## Correct: parent is up, sub-int is not
+694: e0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UNKNOWN mode DEFAULT group default qlen 1000
+695: e0.10@e0: <BROADCAST,MULTICAST> mtu 9000 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
+
+DBGvpp# set int state TenGigabitEthernet3/0/0.10 up
+## Correct: both interfaces up
+694: e0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UNKNOWN mode DEFAULT group default qlen 1000
+695: e0.10@e0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
+
+DBGvpp# set int state TenGigabitEthernet3/0/0 down
+DBGvpp# set int state TenGigabitEthernet3/0/0.10 down
+DBGvpp# set int state TenGigabitEthernet3/0/0 up
+## Correct: only the parent is up
+694: e0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UNKNOWN mode DEFAULT group default qlen 1000
+695: e0.10@e0: <BROADCAST,MULTICAST> mtu 9000 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
+```
+
+### Change interface: MTU
+
+Finally, a straight forward
+[[commit](https://github.com/pimvanpelt/lcpng/commit/39bfa1615fd1cafe5df6d8fc9d34528e8d3906e2)], or
+so I thought. When the MTU changes in VPP (with `set interface mtu packet N <int>`), there is
+callback that can be registered which copies this into the _LIP_. I did notice a specific corner
+case: In VPP, a sub-interface can have a larger MTU than its parent. In Linux, this cannot happen,
+so the following remains problematic:
+
+```
+DBGvpp# create sub TenGigabitEthernet3/0/0 10
+DBGvpp# set int mtu packet 1500 TenGigabitEthernet3/0/0
+DBGvpp# set int mtu packet 9000 TenGigabitEthernet3/0/0.10
+## Incorrect: sub-int has larger MTU than parent, valid in VPP, not in Linux
+694: e0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UNKNOWN mode DEFAULT group default qlen 1000
+695: e0.10@e0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
+```
+
+I think the best way to ensure this works is to _clamp_ the sub-int to a maximum MTU of
+that of its parent, and revert the user's request to change the VPP sub-int to anything
+higher than that, perhaps logging an error explaining why. This means two things:
+1.   Any change in VPP of a child MTU to larger than its parent, must be reverted.
+1.   Any change in VPP of a parent MTU should ensure all children are clamped to at most that.
+
+I addressed the issue in this
+[[commit](https://github.com/pimvanpelt/lcpng/commit/79a395b3c9f0dae9a23e6fbf10c5f284b1facb85)].
+
+### Change interface: IP Addresses
+
+There are three scenarios in which IP addresses will need to be copied from
+VPP into the companion Linux devices:
+
+1.  `set interface ip address` adds an IPv4 or IPv6 address. This is handled by
+    `lcp_itf_ip[46]_add_del_interface_addr()` which is a callback installed in
+    `lcp_itf_pair_init()` at plugin initialization time.
+1.  `set interface ip address del` removes addresses. This is also handled by
+    `lcp_itf_ip[46]_add_del_interface_addr()` but curiously there is no
+    upstream `vnet_netlink_del_ip[46]_addr()` so I had to write them inline here.
+    I will try to get them upstreamed, as they appear to be obvious companions
+    in `vnet/device/netlink.h`.
+1.  This one is easy to overlook, but upon _LIP_ creation, it could be that there
+    are already L3 addresses present on the VPP interface. If so, set them in the
+    _LIP_ with `lcp_itf_set_interface_addr()`.
+
+This means with this
+[[commit](https://github.com/pimvanpelt/lcpng/commit/f7e1bb951d648a63dfa27d04ded0b6261b9e39fe)], at
+any time a new _LIP_ is created, the IPv4 and IPv6 address on the VPP interface are fully copied
+over by the third change, while at runtime, new addresses can be set/removed as well by the first
+and second change.
+
+### Further work
+
+I noticed that [Bird](https://bird.network.cz/) periodically scans the Linux
+interface list and (re)learns information from them. I have a suspicion that
+such a feature might be useful in the VPP plugin as well: I can imagine a
+periodical process that walks over the _LIP_ interface list, and compares
+what it finds in Linux with what is configured in VPP. What's not entirely
+clear to me is which direction should 'trump', that is, should the Linux
+state be forced into VPP, or should the VPP state be forced into Linux? I
+don't yet have a good feeling of the answer, so I'll punt on that for now.
+
+## Results
+
+After applying the configuration to VPP (in Appendix), here's the results:
+
+```
+pim@hippo:~/src/lcpng$ ip ro
+default via 194.1.163.65 dev enp6s0 proto static
+10.0.1.0/30 dev e0 proto kernel scope link src 10.0.1.1
+10.0.2.0/30 dev e0.1234 proto kernel scope link src 10.0.2.1
+10.0.3.0/30 dev e0.1235 proto kernel scope link src 10.0.3.1
+10.0.4.0/30 dev e0.1236 proto kernel scope link src 10.0.4.1
+10.0.5.0/30 dev e0.1237 proto kernel scope link src 10.0.5.1
+194.1.163.64/27 dev enp6s0 proto kernel scope link src 194.1.163.88
+
+pim@hippo:~/src/lcpng$ fping 10.0.1.2 10.0.2.2 10.0.3.2 10.0.4.2 10.0.5.2
+10.0.1.2 is alive
+10.0.2.2 is alive
+10.0.3.2 is alive
+10.0.4.2 is alive
+10.0.5.2 is alive
+
+pim@hippo:~/src/lcpng$ fping6 2001:db8:0:1::2 2001:db8:0:2::2 \
+  2001:db8:0:3::2 2001:db8:0:4::2 2001:db8:0:5::2
+2001:db8:0:1::2 is alive
+2001:db8:0:2::2 is alive
+2001:db8:0:3::2 is alive
+2001:db8:0:4::2 is alive
+2001:db8:0:5::2 is alive
+
+```
+
+In case you were wondering: my previous post ended in the same huzzah moment.  It did.
+
+The difference is that now the VPP configuration is _much shorter_! Comparing
+the Appendix from this post with my [first post]({% post_url 2021-08-12-vpp-1 %}), after
+all of this work I no longer have to manually copy the configuration (like link states,
+MTU changes, IP addresses) from VPP into Linux, instead the plugin does all of this work
+for me, and I can configure both sides entirely with `vppctl` commands!
+
+### Bonus screencast!
+
+Humor me as I [take the code out](https://asciinema.org/a/430411) for a 5 minute spin :-)
+
+<script id="asciicast-430411" src="https://asciinema.org/a/430411.js" async></script>
+
+
+## Credits
+
+I'd like to make clear that the Linux CP plugin is a great collaboration between several great folks
+and that my work stands on their shoulders. I've had a little bit of help along the way from Neale
+Ranns, Matthew Smith and Jon Loeliger, and I'd like to thank them for their work!
+
+## Appendix
+
+#### Ubuntu config
+```
+# Untagged interface
+ip addr add 10.0.1.2/30 dev enp66s0f0
+ip addr add 2001:db8:0:1::2/64 dev enp66s0f0
+ip link set enp66s0f0 up mtu 9000
+
+# Single 802.1q tag 1234
+ip link add link enp66s0f0 name enp66s0f0.q type vlan id 1234
+ip link set enp66s0f0.q up mtu 9000
+ip addr add 10.0.2.2/30 dev enp66s0f0.q
+ip addr add 2001:db8:0:2::2/64 dev enp66s0f0.q
+
+# Double 802.1q tag 1234 inner-tag 1000
+ip link add link enp66s0f0.q name enp66s0f0.qinq type vlan id 1000
+ip link set enp66s0f0.qinq up mtu 9000
+ip addr add 10.0.3.3/30 dev enp66s0f0.qinq
+ip addr add 2001:db8:0:3::2/64 dev enp66s0f0.qinq
+
+# Single 802.1ad tag 2345
+ip link add link enp66s0f0 name enp66s0f0.ad type vlan id 2345 proto 802.1ad
+ip link set enp66s0f0.ad up mtu 9000
+ip addr add 10.0.4.2/30 dev enp66s0f0.ad
+ip addr add 2001:db8:0:4::2/64 dev enp66s0f0.ad
+
+# Double 802.1ad tag 2345 inner-tag 1000
+ip link add link enp66s0f0.ad name enp66s0f0.qinad type vlan id 1000 proto 802.1q
+ip link set enp66s0f0.qinad up mtu 9000
+ip addr add 10.0.5.2/30 dev enp66s0f0.qinad
+ip addr add 2001:db8:0:5::2/64 dev enp66s0f0.qinad
+```
+
+#### VPP config
+```
+## Look mom, no `ip` commands!! :-)
+vppctl set interface state TenGigabitEthernet3/0/0 up
+vppctl lcp create TenGigabitEthernet3/0/0 host-if e0
+vppctl set interface mtu packet 9000 TenGigabitEthernet3/0/0
+vppctl set interface ip address TenGigabitEthernet3/0/0 10.0.1.1/30
+vppctl set interface ip address TenGigabitEthernet3/0/0 2001:db8:0:1::1/64
+
+vppctl create sub TenGigabitEthernet3/0/0 1234
+vppctl set interface mtu packet 9000 TenGigabitEthernet3/0/0.1234
+vppctl lcp create TenGigabitEthernet3/0/0.1234 host-if e0.1234
+vppctl set interface state TenGigabitEthernet3/0/0.1234 up
+vppctl set interface ip address TenGigabitEthernet3/0/0.1234 10.0.2.1/30
+vppctl set interface ip address TenGigabitEthernet3/0/0.1234 2001:db8:0:2::1/64
+
+vppctl create sub TenGigabitEthernet3/0/0 1235 dot1q 1234 inner-dot1q 1000 exact-match
+vppctl set interface state TenGigabitEthernet3/0/0.1235 up
+vppctl set interface mtu packet 9000 TenGigabitEthernet3/0/0.1235
+vppctl lcp create TenGigabitEthernet3/0/0.1235 host-if e0.1235
+vppctl set interface ip address TenGigabitEthernet3/0/0.1235 10.0.3.1/30
+vppctl set interface ip address TenGigabitEthernet3/0/0.1235 2001:db8:0:3::1/64
+
+vppctl create sub TenGigabitEthernet3/0/0 1236 dot1ad 2345 exact-match
+vppctl set interface state TenGigabitEthernet3/0/0.1236 up
+vppctl lcp create TenGigabitEthernet3/0/0.1236 host-if e0.1236
+vppctl set interface mtu packet 9000 TenGigabitEthernet3/0/0.1236
+vppctl set interface ip address TenGigabitEthernet3/0/0.1236 10.0.4.1/30
+vppctl set interface ip address TenGigabitEthernet3/0/0.1236 2001:db8:0:4::1/64
+
+vppctl create sub TenGigabitEthernet3/0/0 1237 dot1ad 2345 inner-dot1q 1000 exact-match
+vppctl set interface state TenGigabitEthernet3/0/0.1237 up
+vppctl set interface mtu packet 9000 TenGigabitEthernet3/0/0.1237
+vppctl set interface ip address TenGigabitEthernet3/0/0.1237 10.0.5.1/30
+vppctl set interface ip address TenGigabitEthernet3/0/0.1237 2001:db8:0:5::1/64
+vppctl lcp create TenGigabitEthernet3/0/0.1237 host-if e0.1237
+```
+
+#### Final note
+
+You may have noticed that the [commit] links are all git commits in my private working copy. I want
+to wait until my [previous work](https://gerrit.fd.io/r/c/vpp/+/33481) is reviewed and submitted
+before piling on more changes. Feel free to contact vpp-dev@ for more information in the mean time
+:-)