Friday, pechdag

content/blog/wk8day5.md

@@ -1,7 +1,114 @@
 ---
-title: "Week 8, Friday:"
+title: "Week 8, Friday: Nothing works!"
 date: 2024-09-20T21:55:00+02:00
-draft: true
 ---
 
-{{< image frame="true" width="17em" float="right" src="/img/headline/todo.png" alt="Credit: " >}}
+{{< image frame="true" width="17em" float="right" src="/img/headline/testbeeld.png" alt="Credit: Nederlands Testbeeld 1994 " >}}
+
+I wake up at 09:00 and assemble my plans for the day while drinking a cup of _Joe_. I really only
+have four things to do, but all of them may take a while. Marina helpfully adds two things to the
+list, but they are simple (and one of them, very rewarding).
+
+1.   This week Neil has been iterating on the `sflow` plugin and he has some questions about
+performance. I should take the current v4 of the repo out for a spin and report back. This should
+take about ninety minutes.
+1.   I am flying to Oslo tomorrow and my presentation for NONOG is only partly done. I still have to
+write about half of it. Luckily, the storyboard is in my mind, I just have to put it on paper and
+create a bit of artwork for it. Should take about two hours.
+1.   I also need to pack my stuff for tomorrow, and I am not planning on taking much more than a few
+set of clothes, my wallet, passport and laptop. Travel light! That way you can just say "I did not
+forget X, I just decided not to take X with me!" and make light of planning failures.
+1.   Yesterday I hung up the IPng switch, router, and hypervisor. But they are only connected to the
+OOB for the moment, and the connectivity from Luzern to Zurich and Lausanne needs to be prepared.
+That should take about an hour, including the VPP router installation and enrolling all the machines
+into Ansible.
+
+Marina adds two more tasks:
+1.   I need to look up our booking to see how skimpy and cheap I was with luggage. Recently it's
+been a bit of a thing with airlines that give you _no carryon_ and you're allowed only one personal
+item of a few kilograms and very limited (sub-backpack) size. I know we're on a small plane, so I
+need to figure out what's what, maybe do an online checkin and get us going. Easy peasy: fifteen
+minutes.
+1.   Marina requires a hug, one before lunch and one after lunch. Easy peasy: one minute of hug time
+per slot, two minutes total. And yaay for the chemical endorfin release, priceless!
+
+All up, the day will be about five hours of 'stuff to do'. At 10:00 I start with the loadtests. Neil
+has outdone himself. There was an issue with v3 of the plugin, it was corrupting messages on the
+shared FIFO queue, but Neil replaced it _with a small shellscript_ (of 25 lines or so) of a
+multi-writer single-reader FIFO implementation. I loadtest it and its performance is wonderful. I am
+seeing up to 130K samples/sec being sent out on the _PSAMPLE_ netlink channel now, and their
+sequence numbers and contents look alright. The time spent in the dataplane i still minimal, and
+with this performance, we can scale up to 1Gpps (yes, one _billion_ packets/sec!) through the
+dataplane with really small overhead of ~20 CPU cycles per packet on average. Nice work, Neil!
+I write up my findings on the ongoing developer mail thread, and leave the LAB router + loadtester
+powered up.
+
+I eat lunch and I try to be a bit quick about it. Unfortunately, a light headache is plagueing me
+yet again, and for some reason I lost my appetite. I fried two beautiful dippy eggs on bread, but I
+only ate one of them. Marine dippy'd the second one (she only really enjoys the yolk, the eggwhite
+is, let us say, too healthy for her taste). One egg later, I'm downstairs again, starting on
+finishing my presentation for next week.
+
+My presentation will consist of four sections: Babel+VPP, OSPFv3+VPP, the rollout at AS8298, and
+finally some performance stats and figures of a practical VPP instance. I've written the first two,
+and need to continue with #3 and then #4. I decide to give myself a reward: after writing the third
+section on the rollout, I will divert my attention to the IPng footprint in Stollen, Luzern. This
+reward system works, as I take about fourty minutes to complete this section, and it looks pretty
+good to me.
+
+## IPng Site Local VLLs
+
+I think it'll be fun to make a live video recording of me configuring the IPng connectivity. There's
+currently a leased line from Zurich to Geneva, which I'll break open in Luzern, and terminate the
+L2VPN from Zurich newly in Luzern, and then create a new L2VPN from Luzern to Geneva. The recording
+goes absolutely terribly!! The camera fails halfway through, and after the L2VPN is up, it takes me
+a full frustating fourty-five minutes to debug what was going wrong. For a second there, I staretd
+to doubt Centec switches, but luckily my persistence paid off: it was a config issue on IP-Max's
+ASR9k.
+
+You may not care for a 30min technobabble, if so, just skip ahead. Considering chances are you are a
+nerd, and because I kind of had fun with the frustration making it, here it is, cursing and all:
+
+{{< video src="/media/vdo/2024-09-20 VLLs_12.11.53.mp4" >}}
+
+## IPng VPP Router
+
+Alright, with that small (ahem) task out of the way, I get to put my mind to the second half of
+turning up IPng: the VPP router called `chlzn0.ipng.ch`, a Dell R630 with 2x25G, 4x10G and 2x1G.
+I've already precabled it and installed Debian, so all I need to do is install and configure VPP,
+and connect this router to `chrma0.ipng.ch` in Zurich, and `chplo0.ipng.ch` in Geneva. Easy peasy,
+right? Wrong. The turnup is mostly Ansible, but the configuration is a YAML config file that is
+maintained in Kees.
+
+I configure a link on the first 25G port towards Zurich, and see that it can ping6 the router on the
+other side. Similarly, the second 25G port goes towards Geneva, and pings just fine. OSPFv3 for IPv4
+and IPv6 quickly come up, and BFD starts running, and I'm only 15 minutes into the task. Yaay! But,
+it doesn't ping with IPv4, even though it learned the adjacencies and the whole FIB is programmed.
+Once again, I spend a good 45 minutes convincing myself it's a skill issue. Maybe it's the Mellanox
+NICs? Maybe it's the VPP install or an issue with Bird? And then I start doing DPDK traces and find
+the famous `ip4-not-enabled` which I came across in my article on [[Babel with
+VPP](https://ipng.ch/s/articles/2024/03/06/vpp-with-babel-part-1/)]. Then it dawns on me -- I forgot
+to make the interfaces `unnumbered`, so they indeed did not have an IPv4 address at all. Jeez, Rick!
+One simple statement of `unnumbered: loop0` in the YAML file, and the whole thing shoots into
+action.
+
+All of the sudden, it's 18:30, crap! What was supposed to be a classic Rick-and-Morty twenty minute
+in-and-out, turned into a three hour affair. Hahah, I can be such a dweeb sometimes. But, Adriana is
+here and she's sleeping over tonight because she has a course at the local flower exchange in
+Wangen. Marina made a delicious broccoli and rice with minced meat, and it's very tasty indeed. 
+
+After dinner I once again go downstairs, because I still have to finish the fourth part of my
+presentation. Luckily, in this case I had already done all of the loadtests, so I manage to write it
+pretty quickly, until at 22:30 Marina informs me that the bed is full of my clothes still. Whoops, I
+forgot one of the tasks: packing for tomorrow! But my backpack is full of Stollen-leftovers, so the
+best I can do is vacate the clothes from the bed and put them downstairs.
+
+I finish the presentation at 23:00 and I'm pretty happy with the results! I hope NONOG folks like
+what I have to say, and that I can fit it all in a fourty minute session. Finally, I hold my
+backpack upside down and shake it. A few thousand cables, fiber caps, and a few optics come gushing
+out. Great! Backpack is empty - so I put my clothes in instead, grab wallet + passport, and I'm
+ready for the trip tomorrow.
+
+And, although I only had five simple tasks today, I ended up being super slow while literally
+nothing I set out to do went very well, except for giving Marina hugs :) I didn't even take any
+pictures today!