Add lcpng_nl_sync.c that will house these functions. Their purpose is to
take state learned from netlink messages, and apply that state to VPP.
Some rearranging/plumbing was necessary to get logging to be visible in
this new source file.
Then, we add lcp_nl_neigh_add() and _del() which look up the LIP, convert
the lladdr and ip address from Netlink into VPP variants, and then add or
remove the ip4/ip6 neighbor adjacency.
Register lcp_nl_init() which adds interface pair add/del callbacks.
lcb_nl_pair_add_cb: Initiate netlink listener for first interface in its
netns. If subsequent adds are in other netns, issue a warning. Keep
refcount.
lcb_nl_pair_del_cb: Remove listener when the last interface pair is
removed.
Socket is opened, file is added to VPP's epoll, with lcp_nl_read_cb()
and lcp_nl_error_cb() callbacks installed.
- lcp_nl_read_cb() calls lcp_nl_callback() which pushes netlink messages
onto a queue and issues NL_EVENT_READ event, any socket read error
issues NL_EVENT_READ_ERR event.
- lcp_nl_error_cb() simply issues NL_EVENT_READ_ERR event.
Then, initialize a process node called lcp_nl_process(), which handles:
- NL_EVENT_READ and call lcp_nl_process_msgs()
- if messages are left in the queue, reschedule consumption after M
msecs. This allows new netlink messages to continuously be read from
the kernel, even if we have lots of messages to consume.
- NL_EVENT_READ_ERR and close/reopens the netlink socket.
lcp_nl_process_msgs() processes up to N messages and/or for up to M
msecs, whichever comes first. For each, calling lcp_nl_dispatch().
lcp_nl_dispatch() ultimately just throws the message away after
logging it with format_nl_object()
Introduce lcp_main.lcp_sync, which determines if state changes made
to interfaces in VPP do or don't propagate into Linux.
- Add a startup.conf directive 'lcp-sync' to enable at startup time.
- Add CLI.short_help = "lcp lcp-sync [on|enable|off|disable]",
- Show the current value in "show lcp".
Gate changes in mtu, state and address on lcp_lcp_sync().
When the operator issues 'lcp lcp-sync on', it is prudent to do a
one-off sync of all interface attributes from VPP into Linux.
For this, add a lcp_itf_pair_sync_state_all() function.
In preparation of another feature 'netlink-auto-subint', rename
lcp_main's field to "lcp_auto_subint".
Add CLI .short_help = "lcp lcp-auto-subint [on|enable|off|disable]"
Show status of the field on "lcp show" output.
I was looking at the hw interface list, which makes sense for ethernet
devices but not for other devices, notably BondEthernets.
In addition, creating two separate interfaces with the same outer
(for example Gi3/0/0 dot1ad 2345 inner-dot1q 1000 AND the same on
Gi3/0/1) would yield an erratic match and a crash.
Switch to walking the sw interface list instead, and search for
the sup_sw_if_index that has the desired outer. Result:
BondEthernet0.{1234,1235,1236,1237} can be created and are functional.
Update the pair_config() parser to follow suite.
When the configuration 'lcp-auto-subint' is set, and the interface at
hand is a subinterface, in lcp_itf_interface_add_del():
- if it's a deletion and we're a sub-int, and we have a LIP: delete it.
- if it's a creation and we're a sub-int, and our parent has a LIP, create one.
Fix a few logging consistency issues (pair_del), and in
pair_delete_by_index() ensure that the right namespace is selected.
Due to this quirk with lip->lip_namespace not wanting to be a vec_dup()
string, rewrite them all to be strdup/free instead.
This first preparation moves lcp_itf_phy_add() to lcpng_if_sync.c
and renames it lcp_itf_interface_add_del().
It does all the pre-flight checks to validate that a new device, given
by sw_if_index, can have a LIP created:
- must be a sub-int
- must have a sw_sup_if_index, which itself has a LIP
However, I realize that I cannot create an interface from within an
interface add callback, so I'll have to schedule the child LIP to be
created by a process, after the callback returns.
I'll do that in the next commit.
I've made a few cosmetic adjustments:
- introduce debug, info, notice, warn and err loggers
- remove redundant logging, and set correct (conservative) log levels
- turn the sync-state error into a warning
And a little debt paydown:
- refactor sync-state into its own function, to be called instead of
all the spot fixes elsewhere. It's going to be the case that
sync-state is "the reconsiliation function".
- Fix a bug in lip->lip_namespace copy: vec_dup() there doesn't seem
to work for me, use strdup() instead and make a mental note to
reviist.
The plugin now works with 'lcpng default netns dataplane' in its
startup.conf; and with 'lcp default netns dataplane' as its first
command. A few of these fixes should go upstream, too, which I'll
do next.
I have been very careless in using the correct network namespace when
manipulating LCP host devices. Around any/every netlink write operation,
we must first clib_setns() into the correct namespace. So, wrap every
call of vnet_netlink_*() in all places.
For consistency, use the convention 'curr_ns_fd' (for the one we are
coming from) and 'vif_ns_fd' (to signal the one that the netlink VIF
is in).
Be careful as well to enter and exit everywhere without losing file
descriptors.
In VPP, a sub-interface can have a larger MTU than its parent.
In Linux, this cannot happen, so the following is problematic:
```
DBGvpp# create sub TenGigabitEthernet3/0/0 10
DBGvpp# set int mtu packet 1500 TenGigabitEthernet3/0/0
DBGvpp# set int mtu packet 9000 TenGigabitEthernet3/0/0.10
694: e0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UNKNOWN mode DEFAULT group default qlen 1000
695: e0.10@e0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
```
The best way to ensure this works is to clamp the sub-int to a maximum MTU
of that of its parent, and override the user request to change the VPP
sub-int to anything higher than that, perhaps logging an error explaining
why. This means two things:
1. Any change in VPP of a child MTU to larger than its parent, should be
reverted.
1 Any change in VPP of a parent MTU should ensure all children are clamped
to at most that.
There are three ways in which IP addresses will want to be copied
from VPP into the companion Linux devices:
1) set interface ip address ... adds an IPv4 or IPv6 address
- this is handled by lcp_itf_ip[46]_add_del_interface_addr() which
is a callback installed in lcp_itf_pair_init()
2) set interface ip address del ... removes them
- also handled by lcp_itf_ip[46]_add_del_interface_addr() but
curiously there is no upstream vnet_netlink_del_ip[46]_addr() so
I wrote them inline here - I will try to get them upstreamed, as
they appear to be obvious companions in vnet/device/netlink.h
3) Upon LIP creation, it could be that there are L3 addresses already
on the VPP interface. If so, set them with lcp_itf_set_interface_addr()
This means that now, at any time a new LIP is created, its state from
VPP is fully copied over (MTU, Link state, IPv4/IPv6 addresses)!
At runtime, new addresses can be set/removed as well.
This is a straight foward copy of the VPP L3 (packet) MTU into the Linux
host interface, for any change on an interface/sub-int for which a LIP is
defined.
As with Linux itself, no care is taken to ensure that the parent interface
(e0) has a higher MTU than the child interface (e0.10).
DBGvpp# create sub TenGigabitEthernet3/0/0 10
DBGvpp# lcp create TenGigabitEthernet3/0/0 host-if e0
DBGvpp# lcp create TenGigabitEthernet3/0/0.10 host-if e0.10
602: e0: <BROADCAST,MULTICAST> mtu 9000 qdisc mq state DOWN mode DEFAULT group default qlen 1000
603: e0.10@e0: <BROADCAST,MULTICAST,M-DOWN> mtu 9000 qdisc noop state DOWN mode DEFAULT group default qlen 1000
DBGvpp# set interface mtu packet 9216 TenGigabitEthernet3/0/0
602: e0: <BROADCAST,MULTICAST> mtu 9216 qdisc mq state DOWN mode DEFAULT group default qlen 1000
603: e0.10@e0: <BROADCAST,MULTICAST,M-DOWN> mtu 9000 qdisc noop state DOWN mode DEFAULT group default qlen 1000
DBGvpp# set interface mtu packet 9050 TenGigabitEthernet3/0/0
602: e0: <BROADCAST,MULTICAST> mtu 9050 qdisc mq state DOWN mode DEFAULT group default qlen 1000
603: e0.10@e0: <BROADCAST,MULTICAST,M-DOWN> mtu 9000 qdisc noop state DOWN mode DEFAULT group default qlen 1000
DBGvpp# set interface mtu packet 9050 TenGigabitEthernet3/0/0.10
602: e0: <BROADCAST,MULTICAST> mtu 9050 qdisc mq state DOWN mode DEFAULT group default qlen 1000
603: e0.10@e0: <BROADCAST,MULTICAST,M-DOWN> mtu 9050 qdisc noop state DOWN mode DEFAULT group default qlen 1000
DBGvpp# set interface mtu packet 1500 TenGigabitEthernet3/0/0.10
602: e0: <BROADCAST,MULTICAST> mtu 9050 qdisc mq state DOWN mode DEFAULT group default qlen 1000
603: e0.10@e0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
DBGvpp# set interface mtu packet 1500 TenGigabitEthernet3/0/0
602: e0: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
603: e0.10@e0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
When Linux changes link on a master interface, all of its children also change.
This is not true in VPP, where bringing down a phy while its sub-ints are up
will not change link on the sub-ints.
We are forced to undo that change by walking the sub-interfaces of a phy and
syncing their state back into linux.
For simplicity, just walk all interfaces, as others will be a no-op.
The approach may have to be revisited once netlink messages are
consumed, to ensure there is not an oscillation where netlink sets a
link, which forces all links to be reset, generating more netlink
messages, etc. Care should be taken when netlink consumption comes into
play!
DBGvpp# create sub TenGigabitEthernet3/0/0 1
DBGvpp# lcp create TenGigabitEthernet3/0/0 host-if e0
DBGvpp# lcp create TenGigabitEthernet3/0/0.1 host-if e0.1
593: e0: <BROADCAST,MULTICAST> mtu 9000 qdisc mq state DOWN mode DEFAULT group default qlen 1000
594: e0.1@e0: <BROADCAST,MULTICAST,M-DOWN> mtu 9000 qdisc noop state DOWN mode DEFAULT group default qlen 1000
DBGvpp# set int state TenGigabitEthernet3/0/0 up
593: e0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UNKNOWN mode DEFAULT group default qlen 1000
594: e0.1@e0: <BROADCAST,MULTICAST> mtu 9000 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
DBGvpp# set int state TenGigabitEthernet3/0/0.1 up
593: e0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UNKNOWN mode DEFAULT group default qlen 1000
594: e0.1@e0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
DBGvpp# set int state TenGigabitEthernet3/0/0 down
593: e0: <BROADCAST,MULTICAST> mtu 9000 qdisc mq state DOWN mode DEFAULT group default qlen 1000
594: e0.1@e0: <BROADCAST,MULTICAST,M-DOWN> mtu 9000 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
DBGvpp# set int state TenGigabitEthernet3/0/0 up
DBGvpp# set int state TenGigabitEthernet3/0/0.1 down
593: e0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000
594: e0.1@e0: <BROADCAST,MULTICAST> mtu 9000 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
DBGvpp# set int state TenGigabitEthernet3/0/0 down
593: e0: <BROADCAST,MULTICAST> mtu 9000 qdisc mq state DOWN mode DEFAULT group default qlen 1000
594: e0.1@e0: <BROADCAST,MULTICAST,M-DOWN> mtu 9000 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
This is the first in a series of functions that aims to copy forward
interface changes in the VPP dataplane into the linux interfaces.
Capture link state changes (set interface state ...) and apply them
to Linux.
There's an important dissonance here:
- When Linux sets a parent interface up, all children also go up.
ip link set enp66s0f1 down
ip link add link enp66s0f1 name foo type vlan id 1234
ip link set foo down
ip link | grep enp66s0f1
9: enp66s0f1: <BROADCAST,MULTICAST> mtu 9000 qdisc mq state DOWN mode DEFAULT group default qlen 1000
61: foo@enp66s0f1: <BROADCAST,MULTICAST,M-DOWN> mtu 9000 qdisc noop state DOWN mode DEFAULT group default qlen 1000
ip link set enp66s0f1 up
ip link | grep s0f1
9: enp66s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000
61: foo@enp66s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
While in VPP this is not so, there each individual interface and
sub-interface stands for itself. I think the proper fix here is to walk
all sub-interfaces when a phy changes, and force a sync of those from
VPP to LCP as well. I'll do that in a followup commit so it's easier to
roll back.
When I started in my copy, I removed a bunch of code and options that I
felt were distracting. I also renamed lots of elements like 'linux-cp'
and 'Linux CP' and 'Linux-CP' to just be 'lcpng'.
Now, rename all of this back, and make it ready for upstreaming.
The only diffs between my repo and upstream now are the includes and the
lcpng_interface.[ch] code changes, which is good.
In this situation, Linux and VPP really diverge. In VPP, any
sub-interface can carry arbitrary configuration, they can be dot1q,
dot1ad, with or without an inner dot1q. So the following is valid in
VPP:
vppctl create sub TenGigabitEthernet3/0/0 10 dot1ad 100 inner-dot1q 200 exact-match
In Linux however, double tagged interfaces have to be created as a chain
of two interfaces, first with the outer and then with the inner tag. So
there is no equivalent of the above command in Linux, where we must:
ip link add link e0 name e0.100 type vlan id 100 proto 802.1ad
ip link add link e0.100 name e0.100.200 type vlan id 200 proto 802.1q
So in order to create Q-in-Q sub-interfaces, for Linux their intermediary
parent must exist, while in VPP this is not true.
I have to make a compromise, so I'll be a bit more explicit and allow
this type of LCP to be created under these conditions:
* A sub-int exists with the intermediary (in this case, `dot1ad 100
exact-match`)
* That sub-int itself has an LCP, with a Linux interface device that
we can spawn the inner-dot1q 200 interface off of
Creation of qinq and qinad interfaces becomes thus:
vppctl create sub TenGigabitEthernet3/0/0 10 dot1ad 100 exact-match
vppctl create sub TenGigabitEthernet3/0/0 11 dot1ad 100 inner-dot1q 200 exact-match
vppctl lcpng create TenGigabitEthernet3/0/0 host-if e0
vppctl lcpng create TenGigabitEthernet3/0/0.10 host-if e0.10
vppctl lcpng create TenGigabitEthernet3/0/0.11 host-if e0.11
And the resulting situation in Linux:
pim@hippo:~/src/lcpng$ ip link | grep e0
397: e0: <BROADCAST,MULTICAST> mtu 9000 qdisc mq state DOWN mode DEFAULT group default qlen 1000
398: e0.10@e0: <BROADCAST,MULTICAST,M-DOWN> mtu 9000 qdisc noop state DOWN mode DEFAULT group default qlen 1000
399: e0.11@e0.10: <BROADCAST,MULTICAST,M-DOWN> mtu 9000 qdisc noop state DOWN mode DEFAULT group default qlen 1000
If there are no LCPs defined yet, and I try to create an LCP for a sub-int,
there is a NULL deref in lcp_itf_pair_get() because the pool hasn't been
initiated yet. Fix this by returning NULL if the pool isn't initialized,
and catching the invalid `lip` returning an error.
While I was here, I took a look at a few possible crashes:
- if the pair_create() is called with an invalid phy_sw_if_index, catch
this and return an error.
- if a pair_add_sub() is called but the parent cannot be found, catch
this and return an error.
DBGvpp# set interface state TenGigabitEthernet3/0/0 down
DBGvpp# lcpng create TenGigabitEthernet3/0/0 host-if e0
DBGvpp# set interface state TenGigabitEthernet3/0/1 up
DBGvpp# lcpng create TenGigabitEthernet3/0/1 host-if e1
Yields:
pim@hippo:~/src/lcpng$ ip link show e0
304: e0: <BROADCAST,MULTICAST> mtu 9000 qdisc mq state DOWN mode DEFAULT group default qlen 1000
link/ether 68:05:ca:32:46:14 brd ff:ff:ff:ff:ff:ff
pim@hippo:~/src/lcpng$ ip link show e1
305: e1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UNKNOWN mode DEFAULT group default qlen 1000
link/ether 68:05:ca:32:46:15 brd ff:ff:ff:ff:ff:ff
This is to debug why creating QinQ subinterfaces is failing. Consider
the following:
vppctl create sub TenGigabitEthernet3/0/0 1234
vppctl create sub TenGigabitEthernet3/0/0 1235 dot1q 1234 inner-dot1q 1000
vppctl lcpng create TenGigabitEthernet3/0/0 host-if e0
vppctl lcpng create TenGigabitEthernet3/0/0.1234 host-if e0.1234
vppctl lcpng create TenGigabitEthernet3/0/0.1235 host-if e0.1235
Creating the sub-interfaces works, the first is a normal .1q with
tag 1234. Creating its LCP works well, the parent is looked up,
a netlink VLAN link is created as a child, and it gets tag 1234.
Now the second one: it's also operating on parent Te3/0/0 which is
looked up, but now a netlink VLAN link is created as a child, again
with dot1q 1234: this interface already exists, so that's a no-go
and an error is thrown.
-- Thoughts on a fix for this:
I think the fix is probably retrieving the correct lip, ie not
the lip of the phy parent interface (e0) but the lip of the pair that
has the outer vlan 1234 already (e0.1234), and then asking netlink to
create a child interface with vlan 1235 on that e0.1234, rather than
the phy e0.
Although creating a dot1ad.dot1q or a dot1q.dot1q interface in VPP
is strictly valid, we will not be able to succeed without the
intermediate interface in the Linux model, so we return an error
in that case.
Remove the functionality that allows for a configuration of pairs in the
startup.conf -- I am intending on creating interfaces for each/any phy
and sub and tunnel interfaceb that is created.
Instead of injecting events and having a process listening for them,
simply use the callback to create LCP interfaces. This change removes
the old code and sets a helpful logging entry on line 870, in which
future automatic creations and removals of LCP taps will occur.
This way, three desirable usability properties are obtained:
1) on startup, all physical interfaces will be copied into LCP
2) on sub-interface or tunnel creation (or phy insertion), new
interfaces will be created.
3) Deletions and removals will allow for auto-cleanup of the LCP.
Still use a sensible default of 9216, but if the L3 packet size is set
on the VPP interface, copy it forward (just as we do in the 'host'
interface of the TAP itself, ie the interface created in the linux
namespace). Now they will all line up initially.
The TAP interface does copy forward the host MTU based on the VPP
interface's L3 MTU, but it should also ensure that the VPP tap
interface has an MTU that is greater-or-equal to those. Considering
users can set the interfaces at runtime (set interface mtu packet ...)
ensure that the tap MTU is large enough.
Ideally, a callback updates the MTU to the same value as the L3 MTU of
the VPP interface, or the L3 MTU of the host interface, if either of
those change. For now, it's a safe bet to take jumbo 9216.
This gives a lot of operational problems later. It's definitely reasonable
to be able to create tap interfaces in other namespaces, and this is
still possible (see below for syntax).
However, changing the runtime netns makes the netlink listener much more
complicated because it will have to listen on not just one netns, but all
of them, for netlink updates.
So, for now, let's remove the ability to set the namespace in the API.
Still possible:
- set at startup.conf in lcpng { netns <x> }
- force creating in 'lcpng create ... netns <x>'
This will nudge folks to create one singular namespace (say,
'dataplane', in the startup.conf), and then handle all netlink messages
in that namespace only.
