---
date: "2021-05-17T22:27:34Z"
title: IPng arrives in Frankfurt
aliases:
- /s/articles/2021/05/17/frankfurt.html
---

I've been planning a network expansion for a while now. For the next few weeks,
I will be in total geek-mode as I travel to several European cities to deploy
AS50869 on a european ring. At the same time, my buddy Fred from
[IP-Max](https://ip-max.net/) has been wanting to go to Amsterdam. IP-Max's
[network](https://as25091.peeringdb.com/) is considerably larger than mine, but
it just never clicked with the right set of circumstances for them to deploy
in the Netherlands, until the stars aligned ...

## Leadup to the Roadtrip

Usually, IP-Max deploys their routers by having them shipped into the destination
location, but this time was special. We decided to make a roadtrip out of it,
so Fred made his way from Geneva to Br&uuml;ttisellen, stayed the night, and early
on Monday May 17th, we packed up the car and started our trek.

It turns out we had estimated our risk profile completely wrong - we thought it
would be hard to cross the border into Germany due to the ongoing pandemic, but
actually that part was fine. The Germans had opened their borders for transit
traffic and stays of up to 24hrs just a few days ago, and we both got a
(negative) PCR test so we felt we had our bases covered.

## The Border

Then when we arrived at the border, perhaps because we had Geneva license
plates, we were asked about our trip, business or pleasure, and we shared that
we had some equipment with us. Thus begun the four-and-a-half hour customs
exercise that was necessary for us to safely send our equipment off to
the European Union. One would think it should be easy, but it actually wasn't
quite that easy, considering we arrived at the border at 9am on a Monday, and
the traffic into Switzerland was queueing up all expeditor and logistics
companies, so nobody really was willing to help us out. But we made it and left
again shortly after 1:30pm.

## Frankfurt

{{< image width="300px" float="right" src="/assets/network/defra0-rack.png" alt="IP-Max at Frankfurt" >}}

We arrived at Frankfurt Equinix FR5 at the Kleyerstrasse at around 5pm. The
IP-Max rack was quickly found, and while Fred was installing their corporate
Xen host to run remote VMs for the Frankfurt area, I deployed the first router
of the trip: **defra0.ipng.ch**.

IP-Max at this location has a respectable 30G of DWDM capacity from three
different vendors into Zurich, 30G of LAG capacity towards DE-CIX, and a
10G DWDM wave into Anzin (France), which will be broken up for us in Amsterdam
for a future blogpost - stay tuned :)

Making use of line card and route processor redundancy, we decided to use
three line cards, reserving one TenGig ethernet port on each:

*   Te0/0/0/4 -- EoMPLS to NTT/eShelter Rumlang (**chrma0.ipng.ch**)
*   Te0/1/0/4 -- EoMPLS to Interxion ZUR1 (**chgtg0.ipng.ch**)
*   Te0/2/0/4 -- EoMPLS to Amsterdam NIKHEF (**nlams0.ipng.ch**)

At each site, specifically those that are a bit further away, I deploy a
standard issue [PCEngines APU](https://pcengines.ch/) with 802.11ac WiFi,
serial, and IPMI access to any machine that may be there. If you ever visit
a datacenter floor where I'm present, look for SSID _AS50869 FRA_ in the
case of Kleyerstrasse. The password is _IPngGuest_, you're welcome to some
bits of bandwidth in a pinch :)

You can see my router dangling off what looks like a fiber optic umbellical
cord under **er01.fra05.ip-max.net**, right at the heart of the Frankfurt
internet.

### Logical Configuration

{{< image width="300px" float="left" src="/assets/network/console-fra.png" alt="console-fra.ipng.nl" >}}

**console.fra.ipng.nl** At the top of the rack you can also see the blue APU3
with its WiFi antennas. It takes an IPv4 /29 and IPv6 /64 from IP-Max AS25091
which gives me access to my equipment even if bad things happen (and they will,
it's just a matter of time!). It also exposes a WireGuard so that I can access
it even without the need for SSH which can come in useful if a KVM console is
required. Note the logo :-)

On the inside of the APU, it configures one RFC1918 wifi segment and another
RFC1918 wired segment. In this case, the wired segment is connected to the
IPMI port of the Supermicro router. I have really gotten used to this style
of deployment -- I **start** with the OOB. Once the APU has power (and it does
not need to have an uplink yet), I can already SSH to it from the wireless
segment, and further configure it. Once it's done, I make a habit of rebooting
it to ensure it comes up. Then, I can easily configure (and even entirely
install!!) the server behind it using IPMI serial-over-lan and HTML5 KVM
if need be. It's delicious. And, it has saved my ass several times over the
years!

{{< image width="300px" float="left" src="/assets/network/defra0.png" alt="defra0.ipng.ch" >}}

**defra0.ipng.ch** Making use of the line card redundancy, there is now 3x
10Gig connected to my router, which immediately makes it one of the better
connected hosts in this facility. Logging in via IPMI, the [DANOS](https://danosproject.org)
image is quickly configured. There's one link to Interxion ZUR1 in Glattbrugg,
one link to eShelter in R&uuml;mlang, and one link up to Amsterdam. The
interface towards Interxion ZUR1 doubles up as an egresspoint for now. There
will be an IPv4/IPv6 transit session with AS25091, a [DE-CIX](https://de-cix.net)
connection and possibly but probably not a [Kleyrex](https://kleyrex.net)
connection, were it not for the murderous cross connect costs at this facility.

## The results

{{< image width="100px" float="right" src="/assets/network/iperf-chgtg0-defra0.png" alt="iperf" >}}

After the OSPF and OSPFv3 adjacencies came up, iBGP was next. For now, the
machine is single-homed off of **chrma0.ipng.ch** but soon there will be as
well a leg towards Amsterdam. So for now, all that we can do is test basic
connectivity. So after finishing our trip to Amsterdam, and checking into
our AirBnB ready to go through our quarantine song-and-dance, we spent a
little time celebrating - we arrived at 1:30am, and turned in for the night
at 3am. The next day, our groceries arrived, somehow unfortunately I had to
be "well prepared" and ordered them to be delivered between 7-8am on Tuesday.

After a full day of _regular work_, we spent the evening taking a look at
how my kit performs, and we are happy to report it's absolutely great:
```
pim@defra0:~$ iperf3 -c chgtg0.ipng.ch -P 10
...
[SUM]   0.00-10.00  sec  11.2 GBytes  9.63 Gbits/sec  281             sender
[SUM]   0.00-10.02  sec  11.2 GBytes  9.56 Gbits/sec                  receiver

pim@defra0:~$ iperf3 -c chgtg0.ipng.ch -P 10 -R
...
[SUM]   0.00-10.01  sec  10.2 GBytes  8.73 Gbits/sec  550             sender
[SUM]   0.00-10.00  sec  10.1 GBytes  8.70 Gbits/sec                  receiver

pim@defra0:~$ ping4 chrma0.ipng.ch
PING chrma0.ipng.ch (194.1.163.0) 56(84) bytes of data.
...
--- chrma0.ipng.ch ping statistics ---
9 packets transmitted, 9 received, 0% packet loss, time 20ms
rtt min/avg/max/mdev = 5.864/6.022/6.173/0.072 ms
```

The roundtrip latency to Zurich is about 6.0ms, and the performance is north of
9Gbit in both directions for my router. Soon, we will go to Amsterdam, and
deploy router number two (of four!) on this epic roadtrip: **nlams0.ipng.ch**
which is a bucket list item of mine -- to peer at Amsterdam Science Park.

More on that later!
