219 lines
11 KiB
Markdown
219 lines
11 KiB
Markdown
---
|
|
date: "2021-05-26T21:19:34Z"
|
|
title: IPng arrives in Amsterdam
|
|
aliases:
|
|
- /s/articles/2021/05/25/amsterdam.html
|
|
---
|
|
|
|
I've been planning a network expansion for a while now. For the next few weeks,
|
|
I will be in total geek-mode as I travel to several European cities to deploy
|
|
AS50869 on a european ring. At the same time, my buddy Fred from
|
|
[IP-Max](https://ip-max.net/) has been wanting to go to Amsterdam. IP-Max's
|
|
[network](https://as25091.peeringdb.com/) is considerably larger than mine, but
|
|
it just never clicked with the right set of circumstances for them to deploy
|
|
in the Netherlands, until the stars aligned ...
|
|
|
|
## Leadup for IP-Max
|
|
|
|
Usually, if I were to go deploy somewhere with IP-Max, I settle down on top of
|
|
(or underneath, or in some way physically close to) their router at a point of
|
|
presence of theirs. In Amsterdam though, it was different ... because IP-Max
|
|
had not yet _built_ a PoP here.
|
|
|
|
But I ask: why would that stop us? Fred told me last year that he had always
|
|
wanted to build out a PoP in Amsterdam, but somehow he never really found the
|
|
time. I offered to do the work to organize the local supplier chain, get a
|
|
good spot in a well connected place, long haul to France and Germany, and
|
|
otherwise exercise my (social) network to get it done.
|
|
|
|
In March 2021, I stumbled across rackspace at [NIKHEF](https://barm.nikhef.nl/housing/)
|
|
by working with the folks from [ERITAP](https://eritap.com/) who got their hands
|
|
on something that is less of a commodity: a full rack + power (the facility is
|
|
always chronically oversubscribed).
|
|
|
|
A few chores on the tasklist:
|
|
1. Sign for rackspace. Check.
|
|
1. Order the IP-Max standard-issue _small pop_ kit, which consists of:
|
|
* One Cisco [ASR9001](https://www.cisco.com/c/en/us/products/collateral/routers/asr-9001-router/data_sheet_c78-685687.html)
|
|
* One Nexus [3064PQ](https://www.cisco.com/c/en/us/support/switches/nexus-3064-switch/model.html)
|
|
* One PCEngines [APU4](https://www.pcengines.ch/apu4c4.htm) for out-of-band
|
|
* And all the power/copper/fiber cables, optics, serial dongles we might need
|
|
1. Procure an out-of-band provider for our APU4, easily found at NIKHEF (thanks, [Arend](https://eritap.com/))!
|
|
1. Get connectivity in and out of Amsterdam!
|
|
|
|
### Connectivity
|
|
|
|
The most important piece of planning is around the long haul connectivity.
|
|
Considering IP-Max already operates a circuit from Frankfurt (Germany) to
|
|
Anzin (France), I arranged for that link to be rerouted through Amsterdam
|
|
and broken into two segments: Frankfurt-Amsterdam and Amsterdam-Anzin. I
|
|
was like a kid in a candy store being able to meticulously choose the route
|
|
that the fiber takes -- over Düsseldorf, entering the Netherlands at
|
|
Emmerik, over Arnhem and Ede, and to Amsterdam. A very direct route, using
|
|
a 10Gig DWDM wave.
|
|
|
|
The other span goes from Amsterdam through Antwerp (Belgium) and Brussels
|
|
and finally landing in Anzin (near Lille, France), which was the previous
|
|
10Gig DWDM wave, so there is no increased latency even though the link is
|
|
broken up in Amsterdam. Yaay!
|
|
|
|
Delivery of the DWDM waves was ordered on March 30th, and although it should
|
|
normally take 25 working days to deliver, for some awkward reason with the
|
|
supplier it was going to take way longer than what we could afford, so a
|
|
spot of VP style escalation took place, and oh look! Now it would take four
|
|
weeks to turn up, was completed last Friday, which was just in time
|
|
for our trip. Double yaay!
|
|
|
|
## Staging Amsterdam
|
|
|
|
{{< image width="300px" float="right" src="/assets/network/nlams0-staging.png" alt="Staging Amsterdam" >}}
|
|
|
|
Because this is a completely new site for [IP-Max](https://ip-max.net) as well
|
|
as [IPng](https://ipng.ch/), we'll have to do a bit more work. And this suites
|
|
us just fine, because after driving through Frankfurt (see my [previous post]({{< ref "2021-05-17-frankfurt" >}})),
|
|
to the Netherlands, we have to stay in quarantine for five days (or, ten if we
|
|
happen to fail our PCR test after five days!), which gives us plenty of time
|
|
to stage and configure what will be our Cisco **er01.ams01.ip-max.net** and
|
|
our Nexus **as01.ams01.ip-max.net**.
|
|
|
|
Of course, figuring out how all of this fits together is a nice exercise, and
|
|
we planned to just _plug and play_ the ASR9k, which worked out rather
|
|
successfully by the way, so it had to be completely configured ahead of time.
|
|
We created the interfaces, DNS, routing protocols like OSPF, OSPFv3, MPLS/LDP,
|
|
BGP and all of the good stuff like ACLs, accounts and et cetera.
|
|
|
|
We staged the stuff in the laundry room of our AirBnB, being actually quite
|
|
grateful once the staging was complete and we could turn the machines off.
|
|
|
|
For IPng, staging **nlams0.ipng.ch** was already done ahead of time. So all
|
|
I really needed for it, was to ensure that the EoMPLS circuits were created
|
|
ahead of time. I was really looking forward to seeing if we could beat 14ms
|
|
to Amsterdam on the [IP-Max](https://ip-max.net/) network.
|
|
|
|
## Extracurriculars
|
|
|
|
{{< image width="400px" float="right" src="/assets/network/airbnb-staging.png" alt="Dell AirBNB" >}}
|
|
|
|
Besides the staging, we also ate some pretty delicious food:
|
|
* Mushroom risotto
|
|
* HotPot with Arend and Esther
|
|
* Chicken vegetable soup
|
|
* Tacos w/ Tapas
|
|
* Steak w/ broccoli and potatoes
|
|
* Red tuna w/ beans and herbs
|
|
|
|
But we also took the time to explore a little bit, for example on Kaz's new
|
|
boat through the canals and over the river Amstel. But mostly: we sat home and
|
|
enjoyed our quarantine the best we could :-)
|
|
|
|
## Deployment (day 1)
|
|
{{< image width="400px" float="right" src="/assets/network/nlams0-staging-day1.png" alt="IP-Max Staging" >}}
|
|
|
|
First before the day started, I drained the Frankfurt-Anzin link by raising
|
|
OSPF cost on **er01.fra01.ip-max.net** and **er01.lil02.ip-max.net** while Fred
|
|
notified customers and the IP-Max team of the impending update to the network.
|
|
|
|
We met up with [ERITAP](https://eritap.com/) on Monday 24th, or target deploy
|
|
date. We had labeled and packed up all of our gear, grabbed the car, and made
|
|
our way to the Watergraafsmeer to the place where the Internet landed in Europe
|
|
in 1982. Almost 40 years later, here we are: IP-Max is moving in!
|
|
|
|
The physical work was not very exciting. The Nexus, ASR, two APUs and my own
|
|
Supermicro were racked in only a few minutes. But then the interesting bits
|
|
begin -- how do we connect all of this without making a _Kabelsalat_ that you
|
|
so often see in people's racks.
|
|
|
|
{{< image width="400px" float="right" src="/assets/network/nlams0.png" alt="nlams0" >}}
|
|
But yet at the same time, both Fred and I were enthusiastic and couldn't wait
|
|
to see the ping time to Anzin and Frankfurt from here. I left Fred the honors
|
|
to connect his own brand new **er01.ams01.ip-max.net** by opening the patched
|
|
through loop from our supplier, and he was beaming once he saw OSPF and OSPFv3
|
|
adjacencies and a latency of just short of 6ms. But he was very kind to let
|
|
me do the second honors to connect the router to Anzin, at just over 5ms. That
|
|
is a really fantastic performance and very short path indeed. This will be fun
|
|
for my next adventure, I'm sure. We'll see the Dell pictured above appear as
|
|
**frlil0.ipng.ch** but I get ahead of myself ..
|
|
|
|
After we connected the whole thing up and did extensive ping tests, we
|
|
undrained the spans and saw a respectable 600Mbit of traffic traverse the
|
|
new router. Because there were a few other folks tinkering in the rack (for
|
|
example our friends from [Coloclue](https://coloclue.net/) we decided to
|
|
adjourn for the day and visit Paul and Henrieke up in Almere for a fabulous
|
|
homecooked meal (thanks again for the Picaña!) and we enjoyed being
|
|
followed by the cops when driving back out of Almere -- but we were not
|
|
bothered/hassled by them.
|
|
|
|
## Deployment (day 2)
|
|
|
|
{{< image width="400px" float="right" src="/assets/network/nlams0-pim-sad.png" alt="Pim Cries" >}}
|
|
But then (and this is technically day 2 because it was, let's just say, well
|
|
after midnight), as the IP-Max network calmed down for the night I did my
|
|
stress test and came to a horrible surprise, interface errors! They were
|
|
Frame Checksum Errors and while the performance from **defra0.ipng.ch** to
|
|
**nlams0.ipng.ch** was impeccable (9.2Gbit, yaay), the transfer speeds on
|
|
the reversed direction did stall out at about 35Mbit. That is **NOT** what
|
|
the Doctor ordered!
|
|
|
|
So luckily we had already decided to go back for a day2 to complete the
|
|
rack install, mostly for things like the fiber patch panel for IP-Max
|
|
customers in the ERITAP rack, and to ensure that our power, serial and
|
|
network cables would not come loose, because packets don't like loose
|
|
cables. Certainly we should avoid the electrons or photons falling onto
|
|
the floor...
|
|
|
|
But the weird thing about my link errors (as seen by the ASR9k) was that
|
|
usually the problem is either a duplex error (which was OK), or a dirty
|
|
fiber or transciever (which was unlikely considering this link was a
|
|
SFP+ DAC!). So that leaves either a faulty Cisco or a faulty Supermicro,
|
|
neither of which are appealing.
|
|
|
|
On day two, after breakfast, we had to do a few chores first (like the claim
|
|
for the VAT for imports, see our [previous post]({{< ref "2021-05-17-frankfurt" >}}),
|
|
and as well get a corona PCR test for the way to France (which was absolutely
|
|
horrible, by the way, I *still* feel my nose which was violated). So we hit
|
|
NIKHEF at around 4pm to finish the job and take care of a few small favors for
|
|
Coloclue, ERITAP and Byteworks, who are also in the same rack as IPng and
|
|
IP-Max.
|
|
|
|
## The results
|
|
|
|
After I replaced the DAC (ironically with an SFP+ optic), once OSPF and iBGP
|
|
came back to life, this is what it looked like:
|
|
|
|
```
|
|
pim@chumbucket:~$ traceroute nlams0.ipng.ch
|
|
traceroute to nlams0.ipng.ch (194.1.163.32), 30 hops max, 60 byte packets
|
|
1 chbtl1.ipng.ch (194.1.163.67) 0.292 ms 0.216 ms 0.179 ms
|
|
2 chgtg0.ipng.ch (194.1.163.19) 0.599 ms 0.565 ms 0.531 ms
|
|
3 chrma0.ipng.ch (194.1.163.8) 0.873 ms 0.840 ms 0.806 ms
|
|
4 defra0.ipng.ch (194.1.163.25) 6.783 ms 6.751 ms 6.718 ms
|
|
5 nlams0.ipng.ch (194.1.163.32) 12.864 ms 12.831 ms 12.798 ms
|
|
|
|
pim@nlams0:~$ iperf3 -P 10 -c chgtg0.ipng.ch
|
|
...
|
|
[SUM] 0.00-10.00 sec 11.0 GBytes 9.49 Gbits/sec 95 sender
|
|
[SUM] 0.00-10.01 sec 11.0 GBytes 9.41 Gbits/sec receiver
|
|
|
|
pim@nlams0:~$ iperf3 -P 10 -c chgtg0.ipng.ch -R
|
|
...
|
|
[SUM] 0.00-10.01 sec 10.0 GBytes 8.62 Gbits/sec 339 sender
|
|
[SUM] 0.00-10.00 sec 9.98 GBytes 8.57 Gbits/sec receiver
|
|
```
|
|
|
|
{{< image width="400px" float="right" src="/assets/network/nlams0-pim-happy.png" alt="Pim Laughs" >}}
|
|
That will do, thanks. I cannot believe that the latency from my basement
|
|
workstation in Brüttisellen, Switzerland, to the local internet
|
|
exchange is 0.8ms, then through to Frankfurt at 6.2ms and then all the way
|
|
to Amsterdam the end to end round trip latency is 12.2ms. I can stare at
|
|
the smokeping for hours!!
|
|
|
|
So I spent the reminder of the night hanging out with Fred while pumping
|
|
9Gbit in both directions for 2 hours while traffic was low. It's one thing
|
|
to do an `iperf` in your basement rack, but it's an entirely different feeling
|
|
to do an `iperf` spanning three countries in Europe (CH, DE and NL). I will
|
|
note that the spans from Zurich to Frankfurt didn't even get warm, although
|
|
the one from Frankfurt to Amsterdam kind of broke a sweat for a little while
|
|
there ...
|
|
|
|
And the coolest thing yet? We're not done with this trip.
|