---
date: "2021-05-26T21:19:34Z"
title: IPng arrives in Amsterdam
aliases:
- /s/articles/2021/05/25/amsterdam.html
---

I've been planning a network expansion for a while now. For the next few weeks,
I will be in total geek-mode as I travel to several European cities to deploy
AS50869 on a european ring. At the same time, my buddy Fred from
[IP-Max](https://ip-max.net/) has been wanting to go to Amsterdam. IP-Max's
[network](https://as25091.peeringdb.com/) is considerably larger than mine, but
it just never clicked with the right set of circumstances for them to deploy
in the Netherlands, until the stars aligned ...

## Leadup for IP-Max

Usually, if I were to go deploy somewhere with IP-Max, I settle down on top of
(or underneath, or in some way physically close to) their router at a point of
presence of theirs. In Amsterdam though, it was different ... because IP-Max
had not yet _built_ a PoP here.

But I ask: why would that stop us? Fred told me last year that he had always
wanted to build out a PoP in Amsterdam, but somehow he never really found the
time. I offered to do the work to organize the local supplier chain, get a
good spot in a well connected place, long haul to France and Germany, and
otherwise exercise my (social) network to get it done.

In March 2021, I stumbled across rackspace at [NIKHEF](https://barm.nikhef.nl/housing/)
by working with the folks from [ERITAP](https://eritap.com/) who got their hands
on something that is less of a commodity: a full rack + power (the facility is
always chronically oversubscribed).

A few chores on the tasklist:
1.  Sign for rackspace. Check.
1.  Order the IP-Max standard-issue _small pop_ kit, which consists of:
    * One Cisco [ASR9001](https://www.cisco.com/c/en/us/products/collateral/routers/asr-9001-router/data_sheet_c78-685687.html)
    * One Nexus [3064PQ](https://www.cisco.com/c/en/us/support/switches/nexus-3064-switch/model.html)
    * One PCEngines [APU4](https://www.pcengines.ch/apu4c4.htm) for out-of-band
    * And all the power/copper/fiber cables, optics, serial dongles we might need
1.  Procure an out-of-band provider for our APU4, easily found at NIKHEF (thanks, [Arend](https://eritap.com/))!
1.  Get connectivity in and out of Amsterdam!

### Connectivity

The most important piece of planning is around the long haul connectivity.
Considering IP-Max already operates a circuit from Frankfurt (Germany) to
Anzin (France), I arranged for that link to be rerouted through Amsterdam
and broken into two segments: Frankfurt-Amsterdam and Amsterdam-Anzin. I
was like a kid in a candy store being able to meticulously choose the route
that the fiber takes -- over D&uuml;sseldorf, entering the Netherlands at
Emmerik, over Arnhem and Ede, and to Amsterdam. A very direct route, using
a 10Gig DWDM wave.

The other span goes from Amsterdam through Antwerp (Belgium) and Brussels
and finally landing in Anzin (near Lille, France), which was the previous
10Gig DWDM wave, so there is no increased latency even though the link is
broken up in Amsterdam. Yaay!

Delivery of the DWDM waves was ordered on March 30th, and although it should
normally take 25 working days to deliver, for some awkward reason with the
supplier it was going to take way longer than what we could afford, so a
spot of VP style escalation took place, and oh look! Now it would take four
weeks to turn up, was completed last Friday, which was just in time
for our trip. Double yaay!

## Staging Amsterdam

{{< image width="300px" float="right" src="/assets/network/nlams0-staging.png" alt="Staging Amsterdam" >}}

Because this is a completely new site for [IP-Max](https://ip-max.net) as well
as [IPng](https://ipng.ch/), we'll have to do a bit more work. And this suites
us just fine, because after driving through Frankfurt (see my [previous post]({{< ref "2021-05-17-frankfurt" >}})),
to the Netherlands, we have to stay in quarantine for five days (or, ten if we
happen to fail our PCR test after five days!), which gives us plenty of time
to stage and configure what will be our Cisco **er01.ams01.ip-max.net** and
our Nexus **as01.ams01.ip-max.net**.

Of course, figuring out how all of this fits together is a nice exercise, and
we planned to just _plug and play_ the ASR9k, which worked out rather
successfully by the way, so it had to be completely configured ahead of time.
We created the interfaces, DNS, routing protocols like OSPF, OSPFv3, MPLS/LDP,
BGP and all of the good stuff like ACLs, accounts and et cetera.

We staged the stuff in the laundry room of our AirBnB, being actually quite
grateful once the staging was complete and we could turn the machines off.

For IPng, staging **nlams0.ipng.ch** was already done ahead of time. So all
I really needed for it, was to ensure that the EoMPLS circuits were created
ahead of time. I was really looking forward to seeing if we could beat 14ms
to Amsterdam on the [IP-Max](https://ip-max.net/) network.

## Extracurriculars

{{< image width="400px" float="right" src="/assets/network/airbnb-staging.png" alt="Dell AirBNB" >}}

Besides the staging, we also ate some pretty delicious food:
*    Mushroom risotto
*    HotPot with Arend and Esther
*    Chicken vegetable soup
*    Tacos w/ Tapas
*    Steak w/ broccoli and potatoes
*    Red tuna w/ beans and herbs

But we also took the time to explore a little bit, for example on Kaz's new
boat through the canals and over the river Amstel. But mostly: we sat home and
enjoyed our quarantine the best we could :-)

## Deployment (day 1)
{{< image width="400px" float="right" src="/assets/network/nlams0-staging-day1.png" alt="IP-Max Staging" >}}

First before the day started, I drained the Frankfurt-Anzin link by raising
OSPF cost on **er01.fra01.ip-max.net** and **er01.lil02.ip-max.net** while Fred
notified customers and the IP-Max team of the impending update to the network.

We met up with [ERITAP](https://eritap.com/) on Monday 24th, or target deploy
date. We had labeled and packed up all of our gear, grabbed the car, and made
our way to the Watergraafsmeer to the place where the Internet landed in Europe
in 1982. Almost 40 years later, here we are: IP-Max is moving in!

The physical work was not very exciting. The Nexus, ASR, two APUs and my own
Supermicro were racked in only a few minutes. But then the interesting bits
begin -- how do we connect all of this without making a _Kabelsalat_ that you
so often see in people's racks.

{{< image width="400px" float="right" src="/assets/network/nlams0.png" alt="nlams0" >}}
But yet at the same time, both Fred and I were enthusiastic and couldn't wait
to see the ping time to Anzin and Frankfurt from here. I left Fred the honors
to connect his own brand new **er01.ams01.ip-max.net** by opening the patched
through loop from our supplier, and he was beaming once he saw OSPF and OSPFv3
adjacencies and a latency of just short of 6ms. But he was very kind to let
me do the second honors to connect the router to Anzin, at just over 5ms. That
is a really fantastic performance and very short path indeed. This will be fun
for my next adventure, I'm sure. We'll see the Dell pictured above appear as
**frlil0.ipng.ch** but I get ahead of myself ..

After we connected the whole thing up and did extensive ping tests, we
undrained the spans and saw a respectable 600Mbit of traffic traverse the
new router. Because there were a few other folks tinkering in the rack (for
example our friends from [Coloclue](https://coloclue.net/) we decided to
adjourn for the day and visit Paul and Henrieke up in Almere for a fabulous
homecooked meal (thanks again for the Pica&ntilde;a!) and we enjoyed being
followed by the cops when driving back out of Almere -- but we were not
bothered/hassled by them.

## Deployment (day 2)

{{< image width="400px" float="right" src="/assets/network/nlams0-pim-sad.png" alt="Pim Cries" >}}
But then (and this is technically day 2 because it was, let's just say, well
after midnight), as the IP-Max network calmed down for the night I did my
stress test and came to a horrible surprise, interface errors! They were
Frame Checksum Errors and while the performance from **defra0.ipng.ch** to
**nlams0.ipng.ch** was impeccable (9.2Gbit, yaay), the transfer speeds on
the reversed direction did stall out at about 35Mbit. That is **NOT** what
the Doctor ordered!

So luckily we had already decided to go back for a day2 to complete the
rack install, mostly for things like the fiber patch panel for IP-Max
customers in the ERITAP rack, and to ensure that our power, serial and
network cables would not come loose, because packets don't like loose
cables. Certainly we should avoid the electrons or photons falling onto
the floor...

But the weird thing about my link errors (as seen by the ASR9k) was that
usually the problem is either a duplex error (which was OK), or a dirty
fiber or transciever (which was unlikely considering this link was a
SFP+ DAC!). So that leaves either a faulty Cisco or a faulty Supermicro,
neither of which are appealing.

On day two, after breakfast, we had to do a few chores first (like the claim
for the VAT for imports, see our [previous post]({{< ref "2021-05-17-frankfurt" >}}),
and as well get a corona PCR test for the way to France (which was absolutely
horrible, by the way, I *still* feel my nose which was violated). So we hit
NIKHEF at around 4pm to finish the job and take care of a few small favors for
Coloclue, ERITAP and Byteworks, who are also in the same rack as IPng and
IP-Max.

## The results

After I replaced the DAC (ironically with an SFP+ optic), once OSPF and iBGP
came back to life, this is what it looked like:

```
pim@chumbucket:~$ traceroute nlams0.ipng.ch
traceroute to nlams0.ipng.ch (194.1.163.32), 30 hops max, 60 byte packets
 1  chbtl1.ipng.ch (194.1.163.67)  0.292 ms  0.216 ms  0.179 ms
 2  chgtg0.ipng.ch (194.1.163.19)  0.599 ms  0.565 ms  0.531 ms
 3  chrma0.ipng.ch (194.1.163.8)  0.873 ms  0.840 ms  0.806 ms
 4  defra0.ipng.ch (194.1.163.25)  6.783 ms  6.751 ms  6.718 ms
 5  nlams0.ipng.ch (194.1.163.32)  12.864 ms  12.831 ms  12.798 ms

pim@nlams0:~$ iperf3 -P 10 -c chgtg0.ipng.ch
...
[SUM]   0.00-10.00  sec  11.0 GBytes  9.49 Gbits/sec   95             sender
[SUM]   0.00-10.01  sec  11.0 GBytes  9.41 Gbits/sec                  receiver

pim@nlams0:~$ iperf3 -P 10 -c chgtg0.ipng.ch -R
...
[SUM]   0.00-10.01  sec  10.0 GBytes  8.62 Gbits/sec  339             sender
[SUM]   0.00-10.00  sec  9.98 GBytes  8.57 Gbits/sec                  receiver
```

{{< image width="400px" float="right" src="/assets/network/nlams0-pim-happy.png" alt="Pim Laughs" >}}
That will do, thanks. I cannot believe that the latency from my basement
workstation in Br&uuml;ttisellen, Switzerland, to the local internet
exchange is 0.8ms, then through to Frankfurt at 6.2ms and then all the way
to Amsterdam the end to end round trip latency is 12.2ms. I can stare at
the smokeping for hours!!

So I spent the reminder of the night hanging out with Fred while pumping
9Gbit in both directions for 2 hours while traffic was low. It's one thing
to do an `iperf` in your basement rack, but it's an entirely different feeling
to do an `iperf` spanning three countries in Europe (CH, DE and NL). I will
note that the spans from Zurich to Frankfurt didn't even get warm, although
the one from Frankfurt to Amsterdam kind of broke a sweat for a little while
there ...

And the coolest thing yet? We're not done with this trip.
