10 KiB
title, date
| title | date |
|---|---|
| Week 8, Wednesday: Stollen - Day 1 | 2024-09-18T21:55:00+02:00 |
{{< image frame="true" width="17em" float="right" src="/img/headline/yeahyouradio.png" alt="Credit: Yeah You Radio, YouTube" >}}
This is the start of one of the things I have really been looking forward to for this sabbatical. It's a bit of a backstory though - a few years ago, a large customer of IP-Max wanted to have a second location, and after a few tours, one of which I helped organize, a few months ago they signed with a local Swiss datacenter owned by the cantonal utility company in Lucerne: [EWL].
I've been in this facility before and as datacenters go, this one is very slick. It pumps up water from the Vierwaltstädtersee lake, uses it for cooling the rooms, and then resells the heater water to the city for industrial and residential heating purposes. It's a really nice facility, and one of the things that immediately struck me as an asset is that it's owned by the Canton, so it's near impossible that it gets sold to a large datacenter operator from the United States. And I like that.
So, the customer signed for a modestly sized cage with about 14 racks and room for a large tape robot. IP-Max will be delivering the telecom links - 20Gbit to Zurich and 20Gbit to Geneva (via Lausanne). It's a perfect match, and I get to build it!! Fred and I have planned three days for the deploy - and about half the equipment is at my place (taken from service out of the Equinexit project), while the other half is in Geneva. So we'll meet in the middle, and while Fred is underway from Geneva, I'll leave a bit later from Brüttisellen to meet him there.
In the morning, I finish grabbing all the stuff that I need. Yesterday I took care of the IP-Max equipment, and this morning I finish the install of the IPng footprint: a Centec MPLS switch, a VPP router and a hypervisor. A surprising amount of optics and DACs go into my bag, and I try not to forget anything; although I can always bring some more stuff tomorrow if we forget, as this deploy will take more than one day to complete.
I drive over to Luzern after lunch and the weather is nice, the drive easy. I beat Fred there and get to the datacenter at around 14:00 or so, while Fred's ETA is 14:30. Since IP-Max already has a few customers in this place, there's already a microscopic footprint (just two switches at the moment), but that also means that I already have autonomous access with biometrics. I'm pleasantly surprised that my access has already been prepared / extended to the new room. Yes, you read that right: this is a private suite :-)
I open the door to the room and see the racks completely empty and the airconditioner still powered off. It's so nice to walk into such a room when it's not yet in use. I don't know what it is, maybe it's that new-cage-smell, but I am incredibly happy and grateful to be able to play a part in this wonderful story.
Building colocations itself is not super exciting, but perhaps as a juxtaposition with my day-job, being able to go into a room like this, completely build it up with power, links, routers, switches, fiber trays, PDUs, and so on, gives me this inmense feeling of accomplishment. I put my trolly-o-stuff in the room and hang out a little bit in the microkitchen until Fred arrives. I do bump into Marco, the facility director, and we have a chat about the goings-on, the plans, and the 3x10G wavelengths that we'll take into service to add to the fourth one that's already live.
Not to bore you with details too much (as chances are, you're in this line of business if you're reading my journal!), but here's a few things that Fred and I do on this day:
- Two racks will be the telcorack which carry the wavelengths to Zurich and Lausanne respectively. These racks will each have an ASR9k router, a Nexus customer switch, a Cisco management switch, and a Cisco terminalserver.
- The telcorack will have a few customer connections and a few supplier connections. These cross connects have to be picked up from the datacenter staff, tested, and together with a long-haul fiber provider, this brings us to Zurich and Lausanne.
- The telcorack will be interconnected with 24xSMF, and each rack down the line with have a 24xSMF patch panel at the top that connects to the telcoracks. That means we have to pull a lot of fibers.
- IPng's stuff gets to live in the first shared rack, with power and 4x10G to IP-Max for the backbone, and 1x 1G to IP-Max for an out-of-band [PC Engines] APU6 with WiFi. I'll get to that tomorrow.
All the cabling has to be cleaned, carefully administrated, Cisco / Nexus configs created, LACP bundles created, OSPF/OSPFv3 and LDP/MPLS enabled, and once we're ready, we can reroute two active wavelengths that used to go ZRH-LSN, to become ZRH-LUC and LUC-LSN instead. Of course, customer downtime must be minimized, but the mission is to finish this infratructure using the new three waves, move the customer load onto those, and then reconfigure the first wave to become a part of the LAG. Sounds easy, right?
We spent the whole day and most of the evening doing the physical work. Racking the machines, guiding the fibers through yellow plastic trays overhead the racks, and taking notes of what fiber got plugged into which port. At the end of the afternoon, we've used north of fourty optics and fibers, and except for one wavelength (to Zurich), we're in working shape. There's also an issue with copper SFPs, god I hate those things, and we took a few of "the wrong ones" which do not provide link, so we bin them after Fred finds a few more in his bag.
Several times during the afternoon, Marco drops by to take a look and ask if we need something. It's good that he did, because the patch panels were advertised to us as LC/PC (which is quite normal for intra-building cabling), but they turned out to be LC/APC (which is actually better with less reflection on the coupling), except we do not have LC/PC - LC/APC cables with us. Luckily, Marco has a bunch in his stash, and he's thinking of opening a Hofladen so that other customers can pick-and-place some optics, fiber, UTP and power cables, velcro, and so on. Smart!
At 20:30, our blood sugar starts to become dangerously low, and the oxygen level in this facility is quite low at 17%. The extinguisher in this facility is a so-called oxyreduct which pushes out the oxygen making it impossible for fire to sustain. It also makes people light headed if they've been doing physical labor for six hours straight. Since we still don't have the network up yet, we use a 4G router, paired with my APU6 consoleserver, which we connect via WireGuard to the IP-Max out-of-band router in Paris. This allows us to SSH to the APU6 remotely, and from there join the internal managment network to further configure things.
We know just the remedy! In Fred's hotel there is a nice bistro, and they serve one of my favorites: Fleisch auf heissem Stein (English: Meat on a hotstone). Fred takes a candid camera picture of me liberally using the pepper mill to season my sirloin of beef. He goes for something a bit more classy: Kanguru. To each their own, but I inform him that my choice was the superior one.
We finish dinner at 22:00 and I drive home from Luzern to Brüttisellen in completely clear
driving conditions. Not many cars on the road at this hour, so it's only a fifty minute commute. As
I pull into our driveway, Fred reports that the waves to Lausanne are up and running, but he has not
managed to get the wave to Zurich up, yet. However, all we need is one leg, so the first router
(er01.luc01.ip-max.net) has joined the IGP, iBGP and LDP. The management network can now be
announced using its own VRF and MPLS/L3VPN into AS25091, so we no longer need the APU-4G-Wireguard
construction. By the way, if you followed that last sentence, you're a nerd and I love you.
Tomorrow we will complete the job. The link(s) to Zurich needs to be configured, the customers need to be moved to the new wavelength, and then the old wavelength needs to be added to the 20Gig LAG to Zurich. Also, and I'm quite excited about that, IPng's AS8298 gets to make a landing tomorrow. The priority of course is IP-Max's network. With that up, I can simply hook the APU6 into an OOB network provided by IP-Max, and configure my whole kit via IPMI.
For now, I'm bushed and really excited. I hope I have a good night's sleep...
Pictures of the Day
{{< gallery-category >}} {{< gallery-photo fn="2024-09-18/IMG_1360.JPG" caption="Driving from Zurich to Luzern in the afternoon, it's gorgeous weather, and the drive is about 50 minutes.." >}} {{< gallery-photo fn="2024-09-18/IMG_1362.JPG" caption="I'm the first in the room! It's completely switched off, with only empty racks and no IT equipment, not even the airconditioning has been turned on yet. I love it." >}} {{< gallery-photo fn="2024-09-18/IMG_1364.JPG" caption="A selfie of me in the new cage with 14 racks that will be taken into service over the next few weeks. First order of business: get the IT and telecom equipment installed and configured." >}} {{< gallery-photo fn="2024-09-18/IMG_1368.JPG" caption="The rear of the racks is against a curved wall. In German, Stollen are underground straight tunnels. Yes, this facility is built into a hillside along the Vierwaltstädtersee" >}} {{< gallery-photo fn="2024-09-18/IMG_1377.JPG" caption="IP-Max is a diamond partner of this facility, and I can see great service for the new and prospect customers in this datacenter." >}} {{< gallery-photo fn="2024-09-18/IMG_1378.JPG" caption="A view of the top of the racks with the cold aisle roof in front of us, and a bright yellow fibertray that will carry the fiber trunks between the racks." >}} {{< gallery-photo fn="2024-09-18/IMG_1382.JPG" caption="I order a Farmer's Size (350g) beef filet, and it smells godly. Also: Röstikroketten, just sayin'" >}} {{< gallery-photo fn="2024-09-18/IMG_1383.JPG" caption="Fred took a candid camera shot of me seasoning my steak. I'm in a happy place, clearly" >}} {{< /gallery-category >}}
{{< gallery-modal >}} {{< gallery-script >}}