Rewrite all images to Hugo format
This commit is contained in:
144
content/articles/2021-05-17-frankfurt.md
Normal file
144
content/articles/2021-05-17-frankfurt.md
Normal file
@ -0,0 +1,144 @@
|
||||
---
|
||||
date: "2021-05-17T22:27:34Z"
|
||||
title: IPng arrives in Frankfurt
|
||||
---
|
||||
|
||||
I've been planning a network expansion for a while now. For the next few weeks,
|
||||
I will be in total geek-mode as I travel to several European cities to deploy
|
||||
AS50869 on a european ring. At the same time, my buddy Fred from
|
||||
[IP-Max](https://ip-max.net/) has been wanting to go to Amsterdam. IP-Max's
|
||||
[network](https://as25091.peeringdb.com/) is considerably larger than mine, but
|
||||
it just never clicked with the right set of circumstances for them to deploy
|
||||
in the Netherlands, until the stars aligned ...
|
||||
|
||||
## Leadup to the Roadtrip
|
||||
|
||||
Usually, IP-Max deploys their routers by having them shipped into the destination
|
||||
location, but this time was special. We decided to make a roadtrip out of it,
|
||||
so Fred made his way from Geneva to Brüttisellen, stayed the night, and early
|
||||
on Monday May 17th, we packed up the car and started our trek.
|
||||
|
||||
It turns out we had estimated our risk profile completely wrong - we thought it
|
||||
would be hard to cross the border into Germany due to the ongoing pandemic, but
|
||||
actually that part was fine. The Germans had opened their borders for transit
|
||||
traffic and stays of up to 24hrs just a few days ago, and we both got a
|
||||
(negative) PCR test so we felt we had our bases covered.
|
||||
|
||||
## The Border
|
||||
|
||||
Then when we arrived at the border, perhaps because we had Geneva license
|
||||
plates, we were asked about our trip, business or pleasure, and we shared that
|
||||
we had some equipment with us. Thus begun the four-and-a-half hour customs
|
||||
exercise that was necessary for us to safely send our equipment off to
|
||||
the European Union. One would think it should be easy, but it actually wasn't
|
||||
quite that easy, considering we arrived at the border at 9am on a Monday, and
|
||||
the traffic into Switzerland was queueing up all expeditor and logistics
|
||||
companies, so nobody really was willing to help us out. But we made it and left
|
||||
again shortly after 1:30pm.
|
||||
|
||||
## Frankfurt
|
||||
|
||||
{{< image width="300px" float="right" src="/assets/network/defra0-rack.png" alt="IP-Max at Frankfurt" >}}
|
||||
|
||||
We arrived at Frankfurt Equinix FR5 at the Kleyerstrasse at around 5pm. The
|
||||
IP-Max rack was quickly found, and while Fred was installing their corporate
|
||||
Xen host to run remote VMs for the Frankfurt area, I deployed the first router
|
||||
of the trip: **defra0.ipng.ch**.
|
||||
|
||||
IP-Max at this location has a respectable 30G of DWDM capacity from three
|
||||
different vendors into Zurich, 30G of LAG capacity towards DE-CIX, and a
|
||||
10G DWDM wave into Anzin (France), which will be broken up for us in Amsterdam
|
||||
for a future blogpost - stay tuned :)
|
||||
|
||||
Making use of line card and route processor redundancy, we decided to use
|
||||
three line cards, reserving one TenGig ethernet port on each:
|
||||
|
||||
* Te0/0/0/4 -- EoMPLS to NTT/eShelter Rumlang (**chrma0.ipng.ch**)
|
||||
* Te0/1/0/4 -- EoMPLS to Interxion ZUR1 (**chgtg0.ipng.ch**)
|
||||
* Te0/2/0/4 -- EoMPLS to Amsterdam NIKHEF (**nlams0.ipng.ch**)
|
||||
|
||||
At each site, specifically those that are a bit further away, I deploy a
|
||||
standard issue [PCEngines APU](https://pcengines.ch/) with 802.11ac WiFi,
|
||||
serial, and IPMI access to any machine that may be there. If you ever visit
|
||||
a datacenter floor where I'm present, look for SSID _AS50869 FRA_ in the
|
||||
case of Kleyerstrasse. The password is _IPngGuest_, you're welcome to some
|
||||
bits of bandwidth in a pinch :)
|
||||
|
||||
You can see my router dangling off what looks like a fiber optic umbellical
|
||||
cord under **er01.fra05.ip-max.net**, right at the heart of the Frankfurt
|
||||
internet.
|
||||
|
||||
### Logical Configuration
|
||||
|
||||
{{< image width="300px" float="left" src="/assets/network/console-fra.png" alt="console-fra.ipng.nl" >}}
|
||||
|
||||
**console.fra.ipng.nl** At the top of the rack you can also see the blue APU3
|
||||
with its WiFi antennas. It takes an IPv4 /29 and IPv6 /64 from IP-Max AS25091
|
||||
which gives me access to my equipment even if bad things happen (and they will,
|
||||
it's just a matter of time!). It also exposes a WireGuard so that I can access
|
||||
it even without the need for SSH which can come in useful if a KVM console is
|
||||
required. Note the logo :-)
|
||||
|
||||
On the inside of the APU, it configures one RFC1918 wifi segment and another
|
||||
RFC1918 wired segment. In this case, the wired segment is connected to the
|
||||
IPMI port of the Supermicro router. I have really gotten used to this style
|
||||
of deployment -- I **start** with the OOB. Once the APU has power (and it does
|
||||
not need to have an uplink yet), I can already SSH to it from the wireless
|
||||
segment, and further configure it. Once it's done, I make a habit of rebooting
|
||||
it to ensure it comes up. Then, I can easily configure (and even entirely
|
||||
install!!) the server behind it using IPMI serial-over-lan and HTML5 KVM
|
||||
if need be. It's delicious. And, it has saved my ass several times over the
|
||||
years!
|
||||
|
||||
{{< image width="300px" float="left" src="/assets/network/defra0.png" alt="defra0.ipng.ch" >}}
|
||||
|
||||
**defra0.ipng.ch** Making use of the line card redundancy, there is now 3x
|
||||
10Gig connected to my router, which immediately makes it one of the better
|
||||
connected hosts in this facility. Logging in via IPMI, the [DANOS](https://danosproject.org)
|
||||
image is quickly configured. There's one link to Interxion ZUR1 in Glattbrugg,
|
||||
one link to eShelter in Rümlang, and one link up to Amsterdam. The
|
||||
interface towards Interxion ZUR1 doubles up as an egresspoint for now. There
|
||||
will be an IPv4/IPv6 transit session with AS25091, a [DE-CIX](https://de-cix.net)
|
||||
connection and possibly but probably not a [Kleyrex](https://kleyrex.net)
|
||||
connection, were it not for the murderous cross connect costs at this facility.
|
||||
|
||||
## The results
|
||||
|
||||
{{< image width="100px" float="right" src="/assets/network/iperf-chgtg0-defra0.png" alt="iperf" >}}
|
||||
|
||||
After the OSPF and OSPFv3 adjacencies came up, iBGP was next. For now, the
|
||||
machine is single-homed off of **chrma0.ipng.ch** but soon there will be as
|
||||
well a leg towards Amsterdam. So for now, all that we can do is test basic
|
||||
connectivity. So after finishing our trip to Amsterdam, and checking into
|
||||
our AirBnB ready to go through our quarantine song-and-dance, we spent a
|
||||
little time celebrating - we arrived at 1:30am, and turned in for the night
|
||||
at 3am. The next day, our groceries arrived, somehow unfortunately I had to
|
||||
be "well prepared" and ordered them to be delivered between 7-8am on Tuesday.
|
||||
|
||||
After a full day of _regular work_, we spent the evening taking a look at
|
||||
how my kit performs, and we are happy to report it's absolutely great:
|
||||
```
|
||||
pim@defra0:~$ iperf3 -c chgtg0.ipng.ch -P 10
|
||||
...
|
||||
[SUM] 0.00-10.00 sec 11.2 GBytes 9.63 Gbits/sec 281 sender
|
||||
[SUM] 0.00-10.02 sec 11.2 GBytes 9.56 Gbits/sec receiver
|
||||
|
||||
pim@defra0:~$ iperf3 -c chgtg0.ipng.ch -P 10 -R
|
||||
...
|
||||
[SUM] 0.00-10.01 sec 10.2 GBytes 8.73 Gbits/sec 550 sender
|
||||
[SUM] 0.00-10.00 sec 10.1 GBytes 8.70 Gbits/sec receiver
|
||||
|
||||
pim@defra0:~$ ping4 chrma0.ipng.ch
|
||||
PING chrma0.ipng.ch (194.1.163.0) 56(84) bytes of data.
|
||||
...
|
||||
--- chrma0.ipng.ch ping statistics ---
|
||||
9 packets transmitted, 9 received, 0% packet loss, time 20ms
|
||||
rtt min/avg/max/mdev = 5.864/6.022/6.173/0.072 ms
|
||||
```
|
||||
|
||||
The roundtrip latency to Zurich is about 6.0ms, and the performance is north of
|
||||
9Gbit in both directions for my router. Soon, we will go to Amsterdam, and
|
||||
deploy router number two (of four!) on this epic roadtrip: **nlams0.ipng.ch**
|
||||
which is a bucket list item of mine -- to peer at Amsterdam Science Park.
|
||||
|
||||
More on that later!
|
||||
Reference in New Issue
Block a user