Add Stollen day 1

content/blog/wk8day3.md

@@ -1,7 +1,134 @@
 ---
-title: "Week 8, Wednesday:"
+title: "Week 8, Wednesday: Stollen - Day 1"
 date: 2024-09-18T21:55:00+02:00
-draft: true
 ---
 
-{{< image frame="true" width="17em" float="right" src="/img/headline/todo.png" alt="Credit: " >}}
+{{< image frame="true" width="17em" float="right" src="/img/headline/yeahyouradio.png" alt="Credit: Yeah You Radio, YouTube" >}}
+
+This is the start of one of the things I have really been looking forward to for this sabbatical.
+It's a bit of a backstory though - a few years ago, a large customer of IP-Max wanted to have a
+second location, and after a few tours, one of which I helped organize, a few months ago they signed
+with a local Swiss datacenter owned by the cantonal utility company in Lucerne:
+[[EWL](https://ewl.ch/)].
+
+I've been in this facility before and as datacenters go, this one is very slick. It pumps up water
+from the _Vierwaltst&auml;dtersee_ lake, uses it for cooling the rooms, and then resells the heater
+water to the city for industrial and residential heating purposes. It's a really nice facility, and
+one of the things that immediately struck me as _an asset_ is that it's owned by the Canton, so it's
+near impossible that it gets sold to a large datacenter operator from the United States. And I like
+that.
+
+So, the customer signed for a modestly sized cage with about 14 racks and room for a large tape
+robot. IP-Max will be delivering the telecom links - 20Gbit to Zurich and 20Gbit to Geneva (via
+Lausanne). It's a perfect match, **and I get to build it**!! Fred and I have planned three days for
+the deploy - and about half the equipment is at my place (taken from service out of the Equinexit
+project), while the other half is in Geneva. So we'll meet in the middle, and while Fred is underway
+from Geneva, I'll leave a bit later from Br&uuml;ttisellen to meet him there. 
+
+In the morning, I finish grabbing all the stuff that I need. Yesterday I took care of the IP-Max
+equipment, and this morning I finish the install of the IPng footprint: a Centec MPLS switch, a VPP
+router and a hypervisor. A surprising amount of optics and DACs go into my bag, and I try not to
+forget anything; although I can always bring some more stuff tomorrow if we forget, as this deploy
+will take more than one day to complete.
+
+I drive over to Luzern after lunch and the weather is nice, the drive easy. I beat Fred there and
+get to the datacenter at around 14:00 or so, while Fred's ETA is 14:30. Since IP-Max already has a
+few customers in this place, there's already a microscopic footprint (just two switches at the
+moment), but that also means that I already have autonomous access with biometrics. I'm pleasantly
+surprised that my access has already been prepared / extended to the new room. Yes, you read that
+right: this is a private suite :-)
+
+I open the door to the room and see the racks completely empty and the airconditioner still powered
+off. It's *so nice* to walk into such a room when it's not yet in use. I don't know what it is,
+maybe it's that new-cage-smell, but I am incredibly happy and grateful to be able to play a part in
+this wonderful story.
+
+Building colocations itself is not super exciting, but perhaps as a juxtaposition with my day-job,
+being able to go into a room like this, completely build it up with power, links, routers, switches,
+fiber trays, PDUs, and so on, gives me this inmense feeling of accomplishment. I put my
+trolly-o-stuff in the room and hang out a little bit in the microkitchen until Fred arrives. I do
+bump into Marco, the facility director, and we have a chat about the goings-on, the plans, and the
+3x10G wavelengths that we'll take into service to add to the fourth one that's already live.
+
+Not to bore you with details too much (as chances are, you're in this line of business if you're
+reading my journal!), but here's a few things that Fred and I do on this day:
+
+1.   Two racks will be the _telcorack_ which carry the wavelengths to Zurich and Lausanne
+respectively. These racks will each have an ASR9k router, a Nexus customer switch, a Cisco
+management switch, and a Cisco terminalserver.
+1.   The _telcorack_ will have a few customer connections and a few supplier connections. These
+cross connects have to be picked up from the datacenter staff, tested, and together with a long-haul
+fiber provider, this brings us to Zurich and Lausanne.
+1.   The _telcorack_ will be interconnected with 24xSMF, and each rack down the line with have a
+24xSMF patch panel at the top that connects to the telcoracks. That means we have to pull *a lot* of
+fibers.
+1.   IPng's stuff gets to live in the first shared rack, with power and 4x10G to IP-Max for the
+backbone, and 1x 1G to IP-Max for an out-of-band [[PC Engines](https://pcengines.ch/)] APU6 with
+WiFi. I'll get to that tomorrow.
+
+All the cabling has to be cleaned, carefully administrated, Cisco / Nexus configs created, LACP
+bundles created, OSPF/OSPFv3 and LDP/MPLS enabled, and once we're ready, we can reroute two active
+wavelengths that used to go ZRH-LSN, to become ZRH-LUC and LUC-LSN instead. Of course, customer
+downtime must be minimized, but the mission is to finish this infratructure using the new three
+waves, move the customer load onto those, and then reconfigure the first wave to become a part of
+the LAG. Sounds easy, right?
+
+We spent the whole day and most of the evening doing the physical work. Racking the machines,
+guiding the fibers through yellow plastic trays overhead the racks, and taking notes of what fiber
+got plugged into which port. At the end of the afternoon, we've used north of fourty optics and
+fibers, and except for one wavelength (to Zurich), we're in working shape. There's also an issue
+with copper SFPs, god I hate those things, and we took a few of "the wrong ones" which do not
+provide link, so we bin them after Fred finds a few more in his bag.
+
+Several times during the afternoon, Marco drops by to take a look and ask if we need something. It's
+good that he did, because the patch panels were advertised to us as LC/PC (which is quite normal for
+intra-building cabling), but they turned out to be LC/APC (which is actually better with less
+reflection on the coupling), except we do not have LC/PC - LC/APC cables with us. Luckily, Marco has
+a bunch in his stash, and he's thinking of opening a _Hofladen_ so that other customers can
+pick-and-place some optics, fiber, UTP and power cables, velcro, and so on. Smart!
+
+At 20:30, our blood sugar starts to become dangerously low, and the oxygen level in this facility is
+quite low at 17%. The extinguisher in this facility is a so-called _oxyreduct_ which pushes out the
+oxygen making it impossible for fire to sustain. It also makes people light headed if they've been
+doing physical labor for six hours straight. Since we still don't have the network up yet, we use a
+4G router, paired with my APU6 consoleserver, which we connect via WireGuard to the IP-Max
+out-of-band router in Paris. This allows us to SSH to the APU6 remotely, and from there join the
+internal managment network to further configure things.
+
+We know just the remedy! In Fred's hotel there is a nice bistro, and they serve one of my favorites:
+_Fleisch auf heissem Stein_ (English: Meat on a hotstone). Fred takes a candid camera picture of me
+liberally using the pepper mill to season my sirloin of beef. He goes for something a bit more
+classy: Kanguru. To each their own, but I inform him that my choice was the superior one.
+
+We finish dinner at 22:00 and I drive home from Luzern to Br&uuml;ttisellen in completely clear
+driving conditions. Not many cars on the road at this hour, so it's only a fifty minute commute. As
+I pull into our driveway, Fred reports that the waves to Lausanne are up and running, but he has not
+managed to get the wave to Zurich up, yet. However, all we need is one leg, so the first router
+(`er01.luc01.ip-max.net`) has joined the IGP, iBGP and LDP. The management network can now be
+announced using its own VRF and MPLS/L3VPN into AS25091, so we no longer need the APU-4G-Wireguard
+construction. By the way, if you followed that last sentence, you're a nerd and I love you.
+
+Tomorrow we will complete the job. The link(s) to Zurich needs to be configured, the customers need
+to be moved to the new wavelength, and then the old wavelength needs to be added to the 20Gig LAG to
+Zurich. Also, and I'm quite excited about that, IPng's AS8298 gets to make a landing tomorrow. The
+priority of course is IP-Max's network. With that up, I can simply hook the APU6 into an OOB network
+provided by IP-Max, and configure my whole kit via IPMI.
+
+For now, I'm bushed and really excited. I hope I have a good night's sleep...
+
+## Pictures of the Day
+
+{{< gallery-category >}}
+  {{< gallery-photo fn="2024-09-18/IMG_1360.JPG" caption="Driving from Zurich to Luzern in the afternoon, it's gorgeous weather, and the drive is about 50 minutes.." >}}
+  {{< gallery-photo fn="2024-09-18/IMG_1362.JPG" caption="I'm the first in the room! It's completely switched off, with only empty racks and no IT equipment, not even the airconditioning has been turned on yet. I love it." >}}
+  {{< gallery-photo fn="2024-09-18/IMG_1364.JPG" caption="A selfie of me in the new cage with 14 racks that will be taken into service over the next few weeks. First order of business: get the IT and telecom equipment installed and configured." >}}
+  {{< gallery-photo fn="2024-09-18/IMG_1368.JPG" caption="The rear of the racks is against a curved wall. In German, Stollen are underground straight tunnels. Yes, this facility is built into a hillside along the Vierwaltst&auml;dtersee" >}}
+  {{< gallery-photo fn="2024-09-18/IMG_1377.JPG" caption="IP-Max is a diamond partner of this facility, and I can see great service for the new and prospect customers in this datacenter." >}}
+  {{< gallery-photo fn="2024-09-18/IMG_1378.JPG" caption="A view of the top of the racks with the cold aisle roof in front of us, and a bright yellow fibertray that will carry the fiber trunks between the racks." >}}
+  {{< gallery-photo fn="2024-09-18/IMG_1382.JPG" caption="I order a Farmer's Size (350g) beef filet, and it smells godly. Also: R&ouml;stikroketten, just sayin'" >}}
+  {{< gallery-photo fn="2024-09-18/IMG_1383.JPG" caption="Fred took a candid camera shot of me seasoning my steak. I'm in a happy place, clearly" >}}
+{{< /gallery-category >}}
+
+{{< gallery-modal >}}
+{{< gallery-script >}}
+