147 lines
7.3 KiB
Markdown
147 lines
7.3 KiB
Markdown
---
|
|
date: "2021-05-17T22:27:34Z"
|
|
title: IPng arrives in Frankfurt
|
|
aliases:
|
|
- /s/articles/2021/05/17/frankfurt.html
|
|
---
|
|
|
|
I've been planning a network expansion for a while now. For the next few weeks,
|
|
I will be in total geek-mode as I travel to several European cities to deploy
|
|
AS50869 on a european ring. At the same time, my buddy Fred from
|
|
[IP-Max](https://ip-max.net/) has been wanting to go to Amsterdam. IP-Max's
|
|
[network](https://as25091.peeringdb.com/) is considerably larger than mine, but
|
|
it just never clicked with the right set of circumstances for them to deploy
|
|
in the Netherlands, until the stars aligned ...
|
|
|
|
## Leadup to the Roadtrip
|
|
|
|
Usually, IP-Max deploys their routers by having them shipped into the destination
|
|
location, but this time was special. We decided to make a roadtrip out of it,
|
|
so Fred made his way from Geneva to Brüttisellen, stayed the night, and early
|
|
on Monday May 17th, we packed up the car and started our trek.
|
|
|
|
It turns out we had estimated our risk profile completely wrong - we thought it
|
|
would be hard to cross the border into Germany due to the ongoing pandemic, but
|
|
actually that part was fine. The Germans had opened their borders for transit
|
|
traffic and stays of up to 24hrs just a few days ago, and we both got a
|
|
(negative) PCR test so we felt we had our bases covered.
|
|
|
|
## The Border
|
|
|
|
Then when we arrived at the border, perhaps because we had Geneva license
|
|
plates, we were asked about our trip, business or pleasure, and we shared that
|
|
we had some equipment with us. Thus begun the four-and-a-half hour customs
|
|
exercise that was necessary for us to safely send our equipment off to
|
|
the European Union. One would think it should be easy, but it actually wasn't
|
|
quite that easy, considering we arrived at the border at 9am on a Monday, and
|
|
the traffic into Switzerland was queueing up all expeditor and logistics
|
|
companies, so nobody really was willing to help us out. But we made it and left
|
|
again shortly after 1:30pm.
|
|
|
|
## Frankfurt
|
|
|
|
{{< image width="300px" float="right" src="/assets/network/defra0-rack.png" alt="IP-Max at Frankfurt" >}}
|
|
|
|
We arrived at Frankfurt Equinix FR5 at the Kleyerstrasse at around 5pm. The
|
|
IP-Max rack was quickly found, and while Fred was installing their corporate
|
|
Xen host to run remote VMs for the Frankfurt area, I deployed the first router
|
|
of the trip: **defra0.ipng.ch**.
|
|
|
|
IP-Max at this location has a respectable 30G of DWDM capacity from three
|
|
different vendors into Zurich, 30G of LAG capacity towards DE-CIX, and a
|
|
10G DWDM wave into Anzin (France), which will be broken up for us in Amsterdam
|
|
for a future blogpost - stay tuned :)
|
|
|
|
Making use of line card and route processor redundancy, we decided to use
|
|
three line cards, reserving one TenGig ethernet port on each:
|
|
|
|
* Te0/0/0/4 -- EoMPLS to NTT/eShelter Rumlang (**chrma0.ipng.ch**)
|
|
* Te0/1/0/4 -- EoMPLS to Interxion ZUR1 (**chgtg0.ipng.ch**)
|
|
* Te0/2/0/4 -- EoMPLS to Amsterdam NIKHEF (**nlams0.ipng.ch**)
|
|
|
|
At each site, specifically those that are a bit further away, I deploy a
|
|
standard issue [PCEngines APU](https://pcengines.ch/) with 802.11ac WiFi,
|
|
serial, and IPMI access to any machine that may be there. If you ever visit
|
|
a datacenter floor where I'm present, look for SSID _AS50869 FRA_ in the
|
|
case of Kleyerstrasse. The password is _IPngGuest_, you're welcome to some
|
|
bits of bandwidth in a pinch :)
|
|
|
|
You can see my router dangling off what looks like a fiber optic umbellical
|
|
cord under **er01.fra05.ip-max.net**, right at the heart of the Frankfurt
|
|
internet.
|
|
|
|
### Logical Configuration
|
|
|
|
{{< image width="300px" float="left" src="/assets/network/console-fra.png" alt="console-fra.ipng.nl" >}}
|
|
|
|
**console.fra.ipng.nl** At the top of the rack you can also see the blue APU3
|
|
with its WiFi antennas. It takes an IPv4 /29 and IPv6 /64 from IP-Max AS25091
|
|
which gives me access to my equipment even if bad things happen (and they will,
|
|
it's just a matter of time!). It also exposes a WireGuard so that I can access
|
|
it even without the need for SSH which can come in useful if a KVM console is
|
|
required. Note the logo :-)
|
|
|
|
On the inside of the APU, it configures one RFC1918 wifi segment and another
|
|
RFC1918 wired segment. In this case, the wired segment is connected to the
|
|
IPMI port of the Supermicro router. I have really gotten used to this style
|
|
of deployment -- I **start** with the OOB. Once the APU has power (and it does
|
|
not need to have an uplink yet), I can already SSH to it from the wireless
|
|
segment, and further configure it. Once it's done, I make a habit of rebooting
|
|
it to ensure it comes up. Then, I can easily configure (and even entirely
|
|
install!!) the server behind it using IPMI serial-over-lan and HTML5 KVM
|
|
if need be. It's delicious. And, it has saved my ass several times over the
|
|
years!
|
|
|
|
{{< image width="300px" float="left" src="/assets/network/defra0.png" alt="defra0.ipng.ch" >}}
|
|
|
|
**defra0.ipng.ch** Making use of the line card redundancy, there is now 3x
|
|
10Gig connected to my router, which immediately makes it one of the better
|
|
connected hosts in this facility. Logging in via IPMI, the [DANOS](https://danosproject.org)
|
|
image is quickly configured. There's one link to Interxion ZUR1 in Glattbrugg,
|
|
one link to eShelter in Rümlang, and one link up to Amsterdam. The
|
|
interface towards Interxion ZUR1 doubles up as an egresspoint for now. There
|
|
will be an IPv4/IPv6 transit session with AS25091, a [DE-CIX](https://de-cix.net)
|
|
connection and possibly but probably not a [Kleyrex](https://kleyrex.net)
|
|
connection, were it not for the murderous cross connect costs at this facility.
|
|
|
|
## The results
|
|
|
|
{{< image width="100px" float="right" src="/assets/network/iperf-chgtg0-defra0.png" alt="iperf" >}}
|
|
|
|
After the OSPF and OSPFv3 adjacencies came up, iBGP was next. For now, the
|
|
machine is single-homed off of **chrma0.ipng.ch** but soon there will be as
|
|
well a leg towards Amsterdam. So for now, all that we can do is test basic
|
|
connectivity. So after finishing our trip to Amsterdam, and checking into
|
|
our AirBnB ready to go through our quarantine song-and-dance, we spent a
|
|
little time celebrating - we arrived at 1:30am, and turned in for the night
|
|
at 3am. The next day, our groceries arrived, somehow unfortunately I had to
|
|
be "well prepared" and ordered them to be delivered between 7-8am on Tuesday.
|
|
|
|
After a full day of _regular work_, we spent the evening taking a look at
|
|
how my kit performs, and we are happy to report it's absolutely great:
|
|
```
|
|
pim@defra0:~$ iperf3 -c chgtg0.ipng.ch -P 10
|
|
...
|
|
[SUM] 0.00-10.00 sec 11.2 GBytes 9.63 Gbits/sec 281 sender
|
|
[SUM] 0.00-10.02 sec 11.2 GBytes 9.56 Gbits/sec receiver
|
|
|
|
pim@defra0:~$ iperf3 -c chgtg0.ipng.ch -P 10 -R
|
|
...
|
|
[SUM] 0.00-10.01 sec 10.2 GBytes 8.73 Gbits/sec 550 sender
|
|
[SUM] 0.00-10.00 sec 10.1 GBytes 8.70 Gbits/sec receiver
|
|
|
|
pim@defra0:~$ ping4 chrma0.ipng.ch
|
|
PING chrma0.ipng.ch (194.1.163.0) 56(84) bytes of data.
|
|
...
|
|
--- chrma0.ipng.ch ping statistics ---
|
|
9 packets transmitted, 9 received, 0% packet loss, time 20ms
|
|
rtt min/avg/max/mdev = 5.864/6.022/6.173/0.072 ms
|
|
```
|
|
|
|
The roundtrip latency to Zurich is about 6.0ms, and the performance is north of
|
|
9Gbit in both directions for my router. Soon, we will go to Amsterdam, and
|
|
deploy router number two (of four!) on this epic roadtrip: **nlams0.ipng.ch**
|
|
which is a bucket list item of mine -- to peer at Amsterdam Science Park.
|
|
|
|
More on that later!
|