---
date: "2023-05-28T10:08:14Z"
title: VPP MPLS - Part 4
aliases:
- /s/articles/2023/05/28/vpp-mpls-4.html
---

{{< image width="200px" float="right" src="/assets/vpp/fdio-color.svg" alt="VPP" >}}

# About this series

**Special Thanks**: Adrian _vifino_ Pistol for writing this code and for the wonderful collaboration!

Ever since I first saw VPP - the Vector Packet Processor - I have been deeply impressed with its
performance and versatility. For those of us who have used Cisco IOS/XR devices, like the classic
_ASR_ (aggregation service router), VPP will look and feel quite familiar as many of the approaches
are shared between the two.

In the last three articles, I thought I had described "all we need to know" to perform MPLS using
the Linux Controlplane in VPP:

1. In the [[first article]({{< ref "2023-05-07-vpp-mpls-1" >}})] of this series, I took a look at MPLS
   in general.
2. In the [[second article]({{< ref "2023-05-17-vpp-mpls-2" >}})] of the series, I demonstrated a few
   special case labels (such as _Explicit Null_ and _Implicit Null_ which enables the fabled
   _Penultimate Hop Popping_ behavior of MPLS.
3. Then, in the [[third article]({{< ref "2023-05-21-vpp-mpls-3" >}})], I worked with
   [@vifino](https://chaos.social/@vifino) to implement the plumbing for MPLS in the Linux Control Plane
   plugin for VPP. He did most of the work, I just watched :)

As if in a state of premonition, I mentioned:
> Caveat empor, outside of a modest functional and load-test, this MPLS functionality
> hasn't seen a lot of mileage as it's only a few weeks old at this point, so it could definitely
> contain some rough edges. Use at your own risk, but if you did want to discuss issues, the
> [[vpp-dev@](mailto:vpp-dev@lists.fd.io)] mailinglist is a good first stop.


## Introduction

{{< image src="/assets/vpp-mpls/LAB v2.svg" alt="Lab Setup" >}}

As a reminder, the LAB we built is running VPP with a feature added to Linux Control Plane Plugin,
which lets it consume MPLS routes and program the IPv4/IPv6 routing table as well as the MPLS
forwarding table in VPP. At this point, we are running [[Gerrit 38702, PatchSet
10](https://gerrit.fd.io/r/c/vpp/+/38702/10)].

First, let me specify the problem statement: @vifino and I both noticed that sometimes, pinging from
one VPP node to another worked fine, while SSHing did not. This article describes an issue I
diagnosed, and provided a fix for, in the Linux Controlplane plugin implementation.

### Clue 1: Intermittent ping

My first finding is that our LAB machines run _all_ the VPP plugins, notably the `ping` plugin,
which means that VPP was responding to ping/ping6, and the Linux controlplane plugin sometimes did
not receive any traffic, while other times it did receive the traffic, say a TCP/syn for port 22,
and dutifully responded to that, but that syn/ack was never seen back.

If I were to disable the `ping` plugin, indeed pinging from seemingly random pairs of `vpp0-[0123]`
no longer works, while pinging direct neighbors (eg.  `vpp0-0.e1` to `vpp0-1.e0`) consistently works
well.

### Clue 2: Corrupted MPLS packets

Using the `tap0-0` virtual machine, which sees a copy of all packets on the Open vSwitch underlay in
our lab, I started tcpdumping and noticed two curious packets from time to time:

```
09:22:55.349977 52:54:00:03:10:00 > 52:54:00:02:10:01, ethertype 802.1Q (0x8100), length 140: vlan 22, p 0,
  ethertype MPLS unicast (0x8847), MPLS (label 2 (IPv6 explicit NULL), tc 0, [S], ttl 63)
    version error: 4 != 6

09:23:00.357583 52:54:00:01:10:00 > 52:54:00:00:10:01, ethertype 802.1Q (0x8100), length 160: vlan 20, p 0,
  ethertype MPLS unicast (0x8847), MPLS (label 0 (IPv4 explicit NULL), tc 0, [S], ttl 61)
    IP6, wrong link-layer encapsulation (invalid)
```

Looking at the payload of these broken packets, they are DNS packets coming from the `vpp0-3` Linux
Control Plane there, and they are being sent to either the IPv4 address of `192.168.10.4` or the
IPv6 address of `2001:678:d78:201::ffff`. Interestingly, these are the lab's resolvers, so I think
`vpp0-3` is just trying to resolve something.

### Clue 3: Vanishing MPLS packets

As I mentioned, some source/destination pairs in the lab do not seem to pass traffic, while others
are fine. One such case of _packetlo_ is any traffic from `vpp0-3` to the IPv4 address of
`vpp0-1.e0`. The path from `vpp0-3` to that IPv4 address should go out on `vpp0-3.e0` and into
`vpp0-2.e1`, but using tcpdump shows absolutely no such traffic at between `vpp0-3` and `vpp0-2`,
while I'd expect to see it on VLAN 22!

## Diagnosis

Well, based on **Clue 3**, I take a look at what is happening on `vpp0-3`. I start by looking at the Linux
controlplane view, where the route to `lab` looks like this:

```
root@vpp0-3:~$ ip route get 192.168.10.4
192.168.10.4/31 nhid 154  encap mpls  36 via 192.168.10.10 dev e0 proto ospf src 192.168.10.3 metric 20

root@vpp0-3:~$ tcpdump -evni e0 mpls 36
15:07:50.864605 52:54:00:03:10:00 > 52:54:00:02:10:01, ethertype MPLS unicast (0x8847), length 136:
  MPLS (label 36, tc 0, [S], ttl 64)
    (tos 0x0, ttl 64, id 15752, offset 0, flags [DF], proto UDP (17), length 118)
    192.168.10.3.36954 > 192.168.10.4.53: 20950+ PTR?
      1.9.0.0.0.0.0.0.0.0.0.0.0.0.0.0.3.0.0.0.8.7.d.0.8.7.6.0.1.0.0.2.ip6.arpa. (90)
```

Yes indeed, Linux is sending an IPv4 DNS packet out on `e0`, so what am I seeing on the switch fabric?
In the LAB diagram above, I can look up that traffic from `vpp0-3` destined to `vpp0-2` should show up
on VLAN 22:

```
root@tap0-0:~$ tcpdump -evni enp16s0f0 -s 1500 -X vlan 22 and mpls
15:19:56.453521 52:54:00:03:10:00 > 52:54:00:02:10:01, ethertype 802.1Q (0x8100), length 140: vlan 22, p 0,
  ethertype MPLS unicast (0x8847), MPLS (label 2 (IPv6 explicit NULL), tc 0, [S], ttl 63)
        version error: 4 != 6
        0x0000:  0000 213f 4500 0076 d17e 4000 4011 d3a0  ..!?E..v.~@.@...
        0x0010:  c0a8 0a03 c0a8 0a04 e139 0035 0062 0dde  .........9.5.b..
        0x0020:  079e 0100 0001 0000 0000 0000 0131 0139  .............1.9
        0x0030:  0130 0130 0130 0130 0130 0130 0130 0130  .0.0.0.0.0.0.0.0
        0x0040:  0130 0130 0130 0130 0130 0130 0133 0130  .0.0.0.0.0.0.3.0
        0x0050:  0130 0130 0138 0137 0164 0130 0138 0137  .0.0.8.7.d.0.8.7
        0x0060:  0136 0130 0131 0130 0130 0132 0369 7036  .6.0.1.0.0.2.ip6
        0x0070:  0461 7270 6100 000c 0001                 .arpa.....
```

### MPLS Corruption

Ouch, that hurts my eyes!  Linux sent an IPv4 packet into the TAP device carrying label value 36, so
why is it being observed as an _IPv6 Explicit Null_ with label value 2? That can't be right.  In an
attempt to learn more, I ask VPP to give me a packet trace. I happen to remember that on the way
from Linux to VPP, the `virtio-input` driver is used (while, on the way from the wire to VPP, I see
`dpdk-input` is used).

The trace teaches me something really valuable:

```
vpp0-3# trace add virtio-input 100
vpp0-3# show trace

00:03:27:192490: virtio-input
  virtio: hw_if_index 7 next-index 4 vring 0 len 136
    hdr: flags 0x00 gso_type 0x00 hdr_len 0 gso_size 0 csum_start 0 csum_offset 0 num_buffers 1
00:03:27:192500: ethernet-input
  MPLS: 52:54:00:03:10:00 -> 52:54:00:02:10:01
00:03:27:192504: mpls-input
  MPLS: next mpls-lookup[1]  label 36 ttl 64 exp 0
00:03:27:192506: mpls-lookup
  MPLS: next [6], lookup fib index 0, LB index 92 hash 0 label 36 eos 1
00:03:27:192510: mpls-label-imposition-pipe
    mpls-header:[ipv6-explicit-null:63:0:eos]
00:03:27:192512: mpls-output
  adj-idx 21 : mpls via fe80::5054:ff:fe02:1001 GigabitEthernet10/0/0: mtu:9000 next:2 flags:[] 5254000210015254000310008847 flow hash: 0x00000000
00:03:27:192515: GigabitEthernet10/0/0-output
  GigabitEthernet10/0/0 flags 0x00180005
  MPLS: 52:54:00:03:10:00 -> 52:54:00:02:10:01
  label 2 exp 0, s 1, ttl 63
00:03:27:192517: GigabitEthernet10/0/0-tx
  GigabitEthernet10/0/0 tx queue 0
  buffer 0x4c2ea1: current data 0, length 136, buffer-pool 0, ref-count 1, trace handle 0x7
                   l2-hdr-offset 0 l3-hdr-offset 14
  PKT MBUF: port 65535, nb_segs 1, pkt_len 136
    buf_len 2176, data_len 136, ol_flags 0x0, data_off 128, phys_addr 0x730ba8c0
    packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
    rss 0x0 fdir.hi 0x0 fdir.lo 0x0
  MPLS: 52:54:00:03:10:00 -> 52:54:00:02:10:01
  label 2 exp 0, s 1, ttl 63
```

At this point, I think I've figured it out. I can see clearly that the MPLS packet is seen coming
from Linux, and it has label value 36. But, it is then offered to graph node `mpls-input`, which does
what it is designed to do, namely look up the label in the FIB:

```
vpp0-3# show mpls fib 36
MPLS-VRF:0, fib_index:0 locks:[interface:4, CLI:1, lcp-rt:1, ]
36:neos/21 fib:0 index:88 locks:2
  lcp-rt-dynamic refs:1 src-flags:added,contributing,active,
    path-list:[50] locks:24 flags:shared, uPRF-list:38 len:1 itfs:[1, ]
      path:[66] pl-index:50 ip6 weight=1 pref=0 attached-nexthop:  oper-flags:resolved,
        fe80::5054:ff:fe02:1001 GigabitEthernet10/0/0
      [@0]: ipv6 via fe80::5054:ff:fe02:1001 GigabitEthernet10/0/0: mtu:9000 next:4 flags:[] 52540002100152540003100086dd
    Extensions:
     path:66  labels:[[ipv6-explicit-null pipe ttl:0 exp:0]]
 forwarding:   mpls-neos-chain
  [@0]: dpo-load-balance: [proto:mpls index:91 buckets:1 uRPF:38 to:[0:0]]
    [0] [@6]: mpls-label[@34]:[ipv6-explicit-null:64:0:neos]
        [@1]: mpls via fe80::5054:ff:fe02:1001 GigabitEthernet10/0/0: mtu:9000 next:2 flags:[] 5254000210015254000310008847
```

{{< image width="80px" float="left" src="/assets/shared/lightbulb.svg" alt="Lightbulb" >}}

Haha, I love it when the brain-ligutbulb goes to the _on_ position. What's happening is that when we
turned on the MPLS feature on the VPP `tap` that is connected to `e0`, and VPP saw an MPLS packet,
that it looked up in the MPLS FIB what to do with label 36, learning that it must _SWAP_ it for
_IPv6 Explicit NULL_ (which is label value 2), and send it out on Gi10/0/0 to an IPv6 nexthop. Yeah,
**that'll break all right**.

### MPLS Drops

OK, that explains the garbled packets, but what about the ones that I never even saw on the wire
(**Clue 3**)? Well, now that I've enjoyed my lightbulb moment, I know exactly where to look.
Consider the following route in Linux, which is sending out encapsulated with MPLS label value 37;
and consider also what happens if `mpls-input` receives an MPLS frame with that value:

```
root@vpp0-3:~# ip ro get 192.168.10.6
192.168.10.6  encap mpls  37 via 192.168.10.10 dev e0 src 192.168.10.3 uid 0

vpp0-3# show mpls fib 37
MPLS-VRF:0, fib_index:0 locks:[interface:4, CLI:1, lcp-rt:1, ]
```

.. that's right, there ***IS*** no entry. As such, I would expect VPP to not know what to do with
such a mislabeled packet, and drop it. Unsurprisingly at this point, here's a nice proof:

```
00:10:31:107882: virtio-input
  virtio: hw_if_index 7 next-index 4 vring 0 len 102
    hdr: flags 0x00 gso_type 0x00 hdr_len 0 gso_size 0 csum_start 0 csum_offset 0 num_buffers 1
00:10:31:107891: ethernet-input
  MPLS: 52:54:00:03:10:00 -> 52:54:00:02:10:01
00:10:31:107897: mpls-input
  MPLS: next mpls-lookup[1]  label 37 ttl 64 exp 0
00:10:31:107898: mpls-lookup
  MPLS: next [0], lookup fib index 0, LB index 22 hash 0 label 37 eos 1
00:10:31:107901: mpls-drop
  drop
00:10:31:107902: error-drop
  rx:tap1
00:10:31:107905: drop
  mpls-input: MPLS DROP DPO
```

Conclusion: ***tadaa.wav***. When VPP receives the MPLS packet from Linux, it has already been
routed (encapsulated and put in an MPLS packet that's meant to be sent to the next router), so it
should be left alone. Instead, VPP is forcing the packet through the MPLS FIB, where if I'm lucky
(and I'm not, clearly ...) the right thing happens. But, sometimes, the MPLS FIB has instructions
that are different to what Linux had intended, bad things happen, and kittens get hurt. I can't
allow that to happen. I like kittens!

## Fixing Linux CP + MPLS

Now that I know what's actually going on, the fix comes quickly into focus. Of course, when Linux
sends an MPLS packet, VPP **must not** do a FIB lookup. Instead, it should emit the packet on the
correct interface as-is. It sounds a little bit like re-arranging the directed graph that VPP uses
internally. I've never done this before, but why not give it a go .. you know, for science :)

VPP has a concept called _feature arcs_. These are codepoints where features can be inserted and
turned on/off. There's a feature arc for MPLS called `mpls-input`. I can create a graph node that
does anything I'd like to the packets at this point, and what I want to do is take the packet and
instead of offering it to the `mpls-input` node, just emit it on its egress interface using
`interface-output`.

First, I call `VLIB_NODE_FN` which defines a new node in VPP, and I call it `lcp_xc_mpls()`. I
register this node with `VLIB_REGISTER_NODE` giving it the symbolic name `linux-cp-xc-mpls` which
extends the existing code in this plugin for ARP and IPv4/IPv6 forwarding. Once the packet enters my
new node, there are two possible places for it to go, defined by the `next_nodes` field:

1.  ***LCP_XC_MPLS_NEXT_DROP***: If I can't figure out where this packet is headed (there should be
    an existing adjacency for it), I will send it to `error-drop` where it will be discarded.
1.  ***LCP_XC_MPLS_NEXT_IO***: If I do know, however, I ask VPP to send this packet simply to
    `interface-output`, where it will be marshalled on the wire, unmodified.

Taking this short cut for MPLS packets avoids them being looked up in the FIB, and in hindsight this
is no different to how IPv4 and IPv6 packets are also short circuited: for those, `ip4-lookup` and
`ip6-lookup` are also not called, but instead `lcp_xc_inline()` does the business.

I can inform VPP that my new node should be attached as a feature on the `mpls-input` arc,
by calling `VNET_FEATURE_INIT` with it.

Implementing the VPP node is a bit of fiddling - but I take inspiration from the existing function
`lc_xc_inline()` which does this for IPv4 and IPv6. Really all I must do, is two things:

1.  Using the _Linux Interface Pair (LIP)_  entry, figure out which physical interface corresponds
    to the TAP interface I just received the packet on, and then set the TX interface to that.
1.  Retrieve the ethernet adjacency based on the destination MAC address, use it to set the correct
    L2 nexthop. If I don't know what adjacency to use, set `LCP_XC_MPLS_NEXT_DROP` as the next node,
    otherwise set `LCP_XC_MPLS_NEXT_IO`.

The finishing touch on the graph node is to make sure that it's trace-aware. I use packet tracing _a
lot_, as can be seen as well in this article, so I'll detect if tracing for a given packet is turned
on, and if so, tack on a `lcp_xc_trace_t` object, so traces will reveal my new node in use.

Once the node is ready, I have one final step. When constructing the _Linux Interface Pair_ in
`lcp_itf_pair_add()`,  I will enable the newly created feature called `linux-cp-xc-mpls` on the
`mpls-input` feature arc for the TAP interface, by calling `vnet_feature_enable_disable()`.
Conversely, I'll disable the feature when removing the _LIP_ in `lcp_itf_pair_del()`.

## Results

After rebasing @vifino's change, I add my code in [[Gerrit 38702, PatchSet
11-14](https://gerrit.fd.io/r/c/vpp/+/38702/11..14)]. I think the simplest thing to show the effect
of the change is by taking a look at these MPLS packets that come in from Linux Controlplane, and
how they now get moved into `linux-cp-xc-mpls` instead of `mpls-input` before:

```
00:04:12:846748: virtio-input
  virtio: hw_if_index 7 next-index 4 vring 0 len 102
    hdr: flags 0x00 gso_type 0x00 hdr_len 0 gso_size 0 csum_start 0 csum_offset 0 num_buffers 1
00:04:12:846804: ethernet-input
  MPLS: 52:54:00:03:10:00 -> 52:54:00:02:10:01
00:04:12:846811: mpls-input
  MPLS: next BUG![3]  label 37 ttl 64 exp 0
00:04:12:846812: linux-cp-xc-mpls
  lcp-xc: itf:1 adj:21
00:04:12:846844: GigabitEthernet10/0/0-output
  GigabitEthernet10/0/0 flags 0x00180005
  MPLS: 52:54:00:03:10:00 -> 52:54:00:02:10:01
  label 37 exp 0, s 1, ttl 64
00:04:12:846846: GigabitEthernet10/0/0-tx
  GigabitEthernet10/0/0 tx queue 0
  buffer 0x4be948: current data 0, length 102, buffer-pool 0, ref-count 1, trace handle 0x0
                   l2-hdr-offset 0 l3-hdr-offset 14
  PKT MBUF: port 65535, nb_segs 1, pkt_len 102
    buf_len 2176, data_len 102, ol_flags 0x0, data_off 128, phys_addr 0x1f9a5280
    packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
    rss 0x0 fdir.hi 0x0 fdir.lo 0x0
  MPLS: 52:54:00:03:10:00 -> 52:54:00:02:10:01
  label 37 exp 0, s 1, ttl 64
```

The same is true for the original DNS packet with MPLS label 36 -- it just transmits out on
`Gi10/0/0` with the same label, which is dope! Indeed, no more garbled MPLS packets are seen, and
the following simple acceptance test shows that all machines can reach all other machines on the LAB
cluster with both IPv4 and IPv6:

```
ipng@vpp0-3:~$ fping -g 192.168.10.0 192.168.10.3
192.168.10.0 is alive
192.168.10.1 is alive
192.168.10.2 is alive
192.168.10.3 is alive

ipng@vpp0-3:~$ fping6 2001:678:d78:200:: 2001:678:d78:200::1 2001:678:d78:200::2 2001:678:d78:200::3
2001:678:d78:200:: is alive
2001:678:d78:200::1 is alive
2001:678:d78:200::2 is alive
2001:678:d78:200::3 is alive
```

My ping test here from `vpp0-3` tries to ping (via the Linux controlplane) each of the other
routers, including itself. It first does this with IPv4, and then with IPv6, showing that all
_eight_ possible destinations are alive. Progress, sweet sweet progress.

I then expand that with this nice oneliner:

```
pim@lab:~$ for af in 4 6; do \
  for node in $(seq 0 3); do \
    ssh -$af ipng@vpp0-$node "fping -g 192.168.10.0 192.168.10.3; \
      fping6 2001:678:d78:200:: 2001:678:d78:200::1 2001:678:d78:200::2 2001:678:d78:200::3"; \
    done \
  done | grep -c alive

64
```

Explanation: Taking both IPv4 and iPv6, I log in to all four nodes (so in total I invoke SSH 8
times), and then perform both `fping` operations, and receive each time _eight_ respondes,
_sixty-four_ in total.  This checks out. I am very pleased with my work.

## What's next

I joined forces with [@vifino](https://chaos.social/@vifino) who has effectively added MPLS handling
to the Linux Control Plane, so VPP can start to function as an MPLS router using FRR's label
distribution protocol implementation. Gosh, I wish Bird3 would have LDP :)

Our work is mostly complete, there's two pending Gerrit's which should be ready to review and
certainly ready to play with:

1.   [[Gerrit 38826](https://gerrit.fd.io/r/c/vpp/+/38826)]: This adds the ability to listen to internal
     state changes of an interface, so that the Linux Control Plane plugin can enable MPLS on the
     _LIP_ interfaces and Linux sysctl for MPLS input.
1.   [[Gerrit 38702/10](https://gerrit.fd.io/r/c/vpp/+/38702/10)]: This adds the ability to listen to Netlink
     messages in the Linux Control Plane plugin, and sensibly apply these routes to the IPv4, IPv6
     and MPLS FIB in the VPP dataplane.
1.   [[Gerrit 38702/14](https://gerrit.fd.io/r/c/vpp/+/38702/11..14)]: This Gerrit now also adds the
     ability to directly output MPLS packets from Linux out on the correct interface, without
     pulling it through the MPLS fib.

Finally, a note from your friendly neighborhood developers: this code is brand-new and has had _very
limited_ peer-review from the VPP developer community. It adds a significant feature to the Linux
Controlplane plugin, so make sure you both understand the semantics, the differences between Linux
and VPP, and the overall implementation before attempting to use in production. We're pretty sure
we got at least some of this right, but testing and runtime experience will tell.

I will be silently porting the change into my own copy of the Linux Controlplane called lcpng on
[[GitHub](https://github.com/pimvanpelt/lcpng.git)]. If you'd like to test this - reach out to the VPP
Developer [[mailinglist](mailto:vpp-dev@lists.fd.io)] any time!
