---
title: "Week 8, Wednesday: Stollen - Day 1"
date: 2024-09-18T21:55:00+02:00
---

{{< image frame="true" width="17em" float="right" src="/img/headline/yeahyouradio.png" alt="Credit: Yeah You Radio, YouTube" >}}

This is the start of one of the things I have really been looking forward to for this sabbatical.
It's a bit of a backstory though - a few years ago, a large customer of IP-Max wanted to have a
second location, and after a few tours, one of which I helped organize, a few months ago they signed
with a local Swiss datacenter owned by the cantonal utility company in Lucerne:
[[EWL](https://ewl.ch/)].

I've been in this facility before and as datacenters go, this one is very slick. It pumps up water
from the _Vierwaltst&auml;dtersee_ lake, uses it for cooling the rooms, and then resells the heater
water to the city for industrial and residential heating purposes. It's a really nice facility, and
one of the things that immediately struck me as _an asset_ is that it's owned by the Canton, so it's
near impossible that it gets sold to a large datacenter operator from the United States. And I like
that.

So, the customer signed for a modestly sized cage with about 14 racks and room for a large tape
robot. IP-Max will be delivering the telecom links - 20Gbit to Zurich and 20Gbit to Geneva (via
Lausanne). It's a perfect match, **and I get to build it**!! Fred and I have planned three days for
the deploy - and about half the equipment is at my place (taken from service out of the Equinexit
project), while the other half is in Geneva. So we'll meet in the middle, and while Fred is underway
from Geneva, I'll leave a bit later from Br&uuml;ttisellen to meet him there. 

In the morning, I finish grabbing all the stuff that I need. Yesterday I took care of the IP-Max
equipment, and this morning I finish the install of the IPng footprint: a Centec MPLS switch, a VPP
router and a hypervisor. A surprising amount of optics and DACs go into my bag, and I try not to
forget anything; although I can always bring some more stuff tomorrow if we forget, as this deploy
will take more than one day to complete.

I drive over to Luzern after lunch and the weather is nice, the drive easy. I beat Fred there and
get to the datacenter at around 14:00 or so, while Fred's ETA is 14:30. Since IP-Max already has a
few customers in this place, there's already a microscopic footprint (just two switches at the
moment), but that also means that I already have autonomous access with biometrics. I'm pleasantly
surprised that my access has already been prepared / extended to the new room. Yes, you read that
right: this is a private suite :-)

I open the door to the room and see the racks completely empty and the airconditioner still powered
off. It's *so nice* to walk into such a room when it's not yet in use. I don't know what it is,
maybe it's that new-cage-smell, but I am incredibly happy and grateful to be able to play a part in
this wonderful story.

Building colocations itself is not super exciting, but perhaps as a juxtaposition with my day-job,
being able to go into a room like this, completely build it up with power, links, routers, switches,
fiber trays, PDUs, and so on, gives me this inmense feeling of accomplishment. I put my
trolly-o-stuff in the room and hang out a little bit in the microkitchen until Fred arrives. I do
bump into Marco, the facility director, and we have a chat about the goings-on, the plans, and the
3x10G wavelengths that we'll take into service to add to the fourth one that's already live.

Not to bore you with details too much (as chances are, you're in this line of business if you're
reading my journal!), but here's a few things that Fred and I do on this day:

1.   Two racks will be the _telcorack_ which carry the wavelengths to Zurich and Lausanne
respectively. These racks will each have an ASR9k router, a Nexus customer switch, a Cisco
management switch, and a Cisco terminalserver.
1.   The _telcorack_ will have a few customer connections and a few supplier connections. These
cross connects have to be picked up from the datacenter staff, tested, and together with a long-haul
fiber provider, this brings us to Zurich and Lausanne.
1.   The _telcorack_ will be interconnected with 24xSMF, and each rack down the line with have a
24xSMF patch panel at the top that connects to the telcoracks. That means we have to pull *a lot* of
fibers.
1.   IPng's stuff gets to live in the first shared rack, with power and 4x10G to IP-Max for the
backbone, and 1x 1G to IP-Max for an out-of-band [[PC Engines](https://pcengines.ch/)] APU6 with
WiFi. I'll get to that tomorrow.

All the cabling has to be cleaned, carefully administrated, Cisco / Nexus configs created, LACP
bundles created, OSPF/OSPFv3 and LDP/MPLS enabled, and once we're ready, we can reroute two active
wavelengths that used to go ZRH-LSN, to become ZRH-LUC and LUC-LSN instead. Of course, customer
downtime must be minimized, but the mission is to finish this infratructure using the new three
waves, move the customer load onto those, and then reconfigure the first wave to become a part of
the LAG. Sounds easy, right?

We spent the whole day and most of the evening doing the physical work. Racking the machines,
guiding the fibers through yellow plastic trays overhead the racks, and taking notes of what fiber
got plugged into which port. At the end of the afternoon, we've used north of fourty optics and
fibers, and except for one wavelength (to Zurich), we're in working shape. There's also an issue
with copper SFPs, god I hate those things, and we took a few of "the wrong ones" which do not
provide link, so we bin them after Fred finds a few more in his bag.

Several times during the afternoon, Marco drops by to take a look and ask if we need something. It's
good that he did, because the patch panels were advertised to us as LC/PC (which is quite normal for
intra-building cabling), but they turned out to be LC/APC (which is actually better with less
reflection on the coupling), except we do not have LC/PC - LC/APC cables with us. Luckily, Marco has
a bunch in his stash, and he's thinking of opening a _Hofladen_ so that other customers can
pick-and-place some optics, fiber, UTP and power cables, velcro, and so on. Smart!

At 20:30, our blood sugar starts to become dangerously low, and the oxygen level in this facility is
quite low at 17%. The extinguisher in this facility is a so-called _oxyreduct_ which pushes out the
oxygen making it impossible for fire to sustain. It also makes people light headed if they've been
doing physical labor for six hours straight. Since we still don't have the network up yet, we use a
4G router, paired with my APU6 consoleserver, which we connect via WireGuard to the IP-Max
out-of-band router in Paris. This allows us to SSH to the APU6 remotely, and from there join the
internal managment network to further configure things.

We know just the remedy! In Fred's hotel there is a nice bistro, and they serve one of my favorites:
_Fleisch auf heissem Stein_ (English: Meat on a hotstone). Fred takes a candid camera picture of me
liberally using the pepper mill to season my sirloin of beef. He goes for something a bit more
classy: Kanguru. To each their own, but I inform him that my choice was the superior one.

We finish dinner at 22:00 and I drive home from Luzern to Br&uuml;ttisellen in completely clear
driving conditions. Not many cars on the road at this hour, so it's only a fifty minute commute. As
I pull into our driveway, Fred reports that the waves to Lausanne are up and running, but he has not
managed to get the wave to Zurich up, yet. However, all we need is one leg, so the first router
(`er01.luc01.ip-max.net`) has joined the IGP, iBGP and LDP. The management network can now be
announced using its own VRF and MPLS/L3VPN into AS25091, so we no longer need the APU-4G-Wireguard
construction. By the way, if you followed that last sentence, you're a nerd and I love you.

Tomorrow we will complete the job. The link(s) to Zurich needs to be configured, the customers need
to be moved to the new wavelength, and then the old wavelength needs to be added to the 20Gig LAG to
Zurich. Also, and I'm quite excited about that, IPng's AS8298 gets to make a landing tomorrow. The
priority of course is IP-Max's network. With that up, I can simply hook the APU6 into an OOB network
provided by IP-Max, and configure my whole kit via IPMI.

For now, I'm bushed and really excited. I hope I have a good night's sleep...

## Pictures of the Day

{{< gallery-category >}}
  {{< gallery-photo fn="2024-09-18/IMG_1360.JPG" caption="Driving from Zurich to Luzern in the afternoon, it's gorgeous weather, and the drive is about 50 minutes.." >}}
  {{< gallery-photo fn="2024-09-18/IMG_1362.JPG" caption="I'm the first in the room! It's completely switched off, with only empty racks and no IT equipment, not even the airconditioning has been turned on yet. I love it." >}}
  {{< gallery-photo fn="2024-09-18/IMG_1364.JPG" caption="A selfie of me in the new cage with 14 racks that will be taken into service over the next few weeks. First order of business: get the IT and telecom equipment installed and configured." >}}
  {{< gallery-photo fn="2024-09-18/IMG_1368.JPG" caption="The rear of the racks is against a curved wall. In German, Stollen are underground straight tunnels. Yes, this facility is built into a hillside along the Vierwaltst&auml;dtersee" >}}
  {{< gallery-photo fn="2024-09-18/IMG_1377.JPG" caption="IP-Max is a diamond partner of this facility, and I can see great service for the new and prospect customers in this datacenter." >}}
  {{< gallery-photo fn="2024-09-18/IMG_1378.JPG" caption="A view of the top of the racks with the cold aisle roof in front of us, and a bright yellow fibertray that will carry the fiber trunks between the racks." >}}
  {{< gallery-photo fn="2024-09-18/IMG_1382.JPG" caption="I order a Farmer's Size (350g) beef filet, and it smells godly. Also: R&ouml;stikroketten, just sayin'" >}}
  {{< gallery-photo fn="2024-09-18/IMG_1383.JPG" caption="Fred took a candid camera shot of me seasoning my steak. I'm in a happy place, clearly" >}}
{{< /gallery-category >}}

{{< gallery-modal >}}
{{< gallery-script >}}

