sabbatical.ipng.nl/content/blog/wk8day5.md

title, date

title	date
Week 8, Friday: Nothing works!	2024-09-20T21:55:00+02:00

{{< image frame="true" width="17em" float="right" src="/img/headline/testbeeld.png" alt="Credit: Nederlands Testbeeld 1994 " >}}

I wake up at 09:00 and assemble my plans for the day while drinking a cup of Joe. I really only have four things to do, but all of them may take a while. Marina helpfully adds two things to the list, but they are simple (and one of them, very rewarding).

1. This week Neil has been iterating on the sflow plugin and he has some questions about performance. I should take the current v4 of the repo out for a spin and report back. This should take about ninety minutes.

2. I am flying to Oslo tomorrow and my presentation for NONOG is only partly done. I still have to write about half of it. Luckily, the storyboard is in my mind, I just have to put it on paper and create a bit of artwork for it. Should take about two hours.

3. I also need to pack my stuff for tomorrow, and I am not planning on taking much more than a few set of clothes, my wallet, passport and laptop. Travel light! That way you can just say "I did not forget X, I just decided not to take X with me!" and make light of planning failures.

4. Yesterday I hung up the IPng switch, router, and hypervisor. But they are only connected to the OOB for the moment, and the connectivity from Luzern to Zurich and Lausanne needs to be prepared. That should take about an hour, including the VPP router installation and enrolling all the machines into Ansible.

Marina adds two more tasks:

1. I need to look up our booking to see how skimpy and cheap I was with luggage. Recently it's been a bit of a thing with airlines that give you no carryon and you're allowed only one personal item of a few kilograms and very limited (sub-backpack) size. I know we're on a small plane, so I need to figure out what's what, maybe do an online checkin and get us going. Easy peasy: fifteen minutes.

2. Marina requires a hug, one before lunch and one after lunch. Easy peasy: one minute of hug time per slot, two minutes total. And yaay for the chemical endorfin release, priceless!

All up, the day will be about five hours of 'stuff to do'. At 10:00 I start with the loadtests. Neil has outdone himself. There was an issue with v3 of the plugin, it was corrupting messages on the shared FIFO queue, but Neil replaced it with a small shellscript (of 25 lines or so) of a multi-writer single-reader FIFO implementation. I loadtest it and its performance is wonderful. I am seeing up to 130K samples/sec being sent out on the PSAMPLE netlink channel now, and their sequence numbers and contents look alright. The time spent in the dataplane i still minimal, and with this performance, we can scale up to 1Gpps (yes, one billion packets/sec!) through the dataplane with really small overhead of ~20 CPU cycles per packet on average. Nice work, Neil! I write up my findings on the ongoing developer mail thread, and leave the LAB router + loadtester powered up.

I eat lunch and I try to be a bit quick about it. Unfortunately, a light headache is plagueing me yet again, and for some reason I lost my appetite. I fried two beautiful dippy eggs on bread, but I only ate one of them. Marine dippy'd the second one (she only really enjoys the yolk, the eggwhite is, let us say, too healthy for her taste). One egg later, I'm downstairs again, starting on finishing my presentation for next week.

My presentation will consist of four sections: Babel+VPP, OSPFv3+VPP, the rollout at AS8298, and finally some performance stats and figures of a practical VPP instance. I've written the first two, and need to continue with #3 and then #4. I decide to give myself a reward: after writing the third section on the rollout, I will divert my attention to the IPng footprint in Stollen, Luzern. This reward system works, as I take about fourty minutes to complete this section, and it looks pretty good to me.

IPng Site Local VLLs

I think it'll be fun to make a live video recording of me configuring the IPng connectivity. There's currently a leased line from Zurich to Geneva, which I'll break open in Luzern, and terminate the L2VPN from Zurich newly in Luzern, and then create a new L2VPN from Luzern to Geneva. The recording goes absolutely terribly!! The camera fails halfway through, and after the L2VPN is up, it takes me a full frustating fourty-five minutes to debug what was going wrong. For a second there, I started to doubt Centec switches, but luckily my persistence paid off: it was a config issue on IP-Max's ASR9k.

You may not care for a 30min technobabble, if so, just skip ahead. Considering chances are you are a nerd, and because I kind of had fun with the frustration making it, here it is, cursing and all:

{{< video src="/media/vdo/2024-09-20 VLLs_12.11.53.mp4" >}}

IPng VPP Router

Alright, with that small (ahem) task out of the way, I get to put my mind to the second half of turning up IPng: the VPP router called chlzn0.ipng.ch, a Dell R630 with 2x25G, 4x10G and 2x1G. I've already precabled it and installed Debian, so all I need to do is install and configure VPP, and connect this router to chrma0.ipng.ch in Zurich, and chplo0.ipng.ch in Geneva. Easy peasy, right? Wrong. The turnup is mostly Ansible, but the configuration is a YAML config file that is maintained in Kees.

I configure a link on the first 25G port towards Zurich, and see that it can ping6 the router on the other side. Similarly, the second 25G port goes towards Geneva, and pings just fine. OSPFv3 for IPv4 and IPv6 quickly come up, and BFD starts running, and I'm only 15 minutes into the task. Yaay! But, it doesn't ping with IPv4, even though it learned the adjacencies and the whole FIB is programmed. Once again, I spend a good 45 minutes convincing myself it's a skill issue. Maybe it's the Mellanox NICs? Maybe it's the VPP install or an issue with Bird? And then I start doing DPDK traces and find the famous ip4-not-enabled which I came across in my article on [Babel with VPP]. Then it dawns on me -- I forgot to make the interfaces unnumbered, so they indeed did not have an IPv4 address at all. Jeez, Rick! One simple statement of unnumbered: loop0 in the YAML file, and the whole thing shoots into action.

All of the sudden, it's 18:30, crap! What was supposed to be a classic Rick-and-Morty twenty minute in-and-out, turned into a three hour affair. Hahah, I can be such a dweeb sometimes. But, Adriana is here and she's sleeping over tonight because she has a course at the local flower exchange in Wangen. Marina made a delicious broccoli and rice with minced meat, and it's very tasty indeed.

After dinner I once again go downstairs, because I still have to finish the fourth part of my presentation. Luckily, in this case I had already done all of the loadtests, so I manage to write it pretty quickly, until at 22:30 Marina informs me that the bed is full of my clothes still. Whoops, I forgot one of the tasks: packing for tomorrow! But my backpack is full of Stollen-leftovers, so the best I can do is vacate the clothes from the bed and put them downstairs.

I finish the presentation at 23:00 and I'm pretty happy with the results! I hope NONOG folks like what I have to say, and that I can fit it all in a fourty minute session. Finally, I hold my backpack upside down and shake it. A few thousand cables, fiber caps, and a few optics come gushing out. Great! Backpack is empty - so I put my clothes in instead, grab wallet + passport, and I'm ready for the trip tomorrow.

And, although I only had five simple tasks today, I ended up being super slow while literally nothing I set out to do went very well, except for giving Marina hugs :) I didn't even take any pictures today!