188 lines
9.8 KiB
Markdown
188 lines
9.8 KiB
Markdown
---
|
|
date: "2021-06-01T18:16:42Z"
|
|
title: IPng arrives in Paris
|
|
aliases:
|
|
- /s/articles/2021/06/01/paris.html
|
|
---
|
|
|
|
I've been planning a network expansion for a while now. For the next few weeks,
|
|
I will be in total geek-mode as I travel to several European cities to deploy
|
|
AS50869 on a european ring. At the same time, my buddy Fred from
|
|
[IP-Max](https://ip-max.net/) has been wanting to go to Amsterdam. IP-Max's
|
|
[network](https://as25091.peeringdb.com/) is considerably larger than mine, but
|
|
it just never clicked with the right set of circumstances for them to deploy
|
|
in the Netherlands, until the stars aligned ...
|
|
|
|
## First a word
|
|
|
|
I just wanted to start with a note on how special it is to partner with [IP-Max](https://www.ip-max.net/)
|
|
and having known the founder Fred for so long affords me this specific trip.
|
|
It must be known that building a 10G pan-european ring is not an easy thing to
|
|
do, and I do appreciate very much the kindness Fred has shown IPng even though
|
|
we are under contract -- the ability to travel to every point of presence and
|
|
get the _founder's tour_ in each place, get to know the local DC ops, sales
|
|
directors, field technicians and local customers, is simply golden. Thank you.
|
|
|
|
## Deployment
|
|
|
|
{{< image width="300px" float="right" src="/assets/network/frpar0-rack.png" alt="Paris KDDI" >}}
|
|
|
|
The city of love, corny as it sounds, also happens to have quite a bit of _fibre
|
|
noire_, happily lit by hundreds of local and international carriers. There are
|
|
two places in the dead-center of the city, a _cabaret voltaire_ called Telehouse
|
|
TH2, and a new facility from [KDDI](https://fra.kddi.com/) which is in the same
|
|
physical block, aptly addressed as 65 Rue Léon Frot, 75011 Paris and that
|
|
is where my beloved router **frpar0.ipng.ch** will be. Note that in Lille,
|
|
actually I had to make do with **frggh0** which was in the town of
|
|
Sainghin-en-Mélantois. This one lives in Paris, none of that suburban
|
|
bullshit. This is the real deal. There's probably more connectivity in this
|
|
one block than in all of the Paris metro combined, maybe even more than
|
|
all of France combined -- peut être :)
|
|
|
|
After having visited the older location, we took the router, APU, cables and
|
|
optics to Léon Frot. The rack was quickly found, and it is obvious that
|
|
this is the location: A _fridge_ was awaiting us, and I reserved two Tengig
|
|
ports on the ASR9010, one towards Lille and one towards Zürich (which will be
|
|
Geneva later on).
|
|
|
|
I'm getting the hang of this VLL stuff after our adventures, previously in
|
|
[Lille]({{< ref "2021-05-28-lille" >}}) at CIV, which is a lazy 4.8ms away
|
|
from this place (Fred already speaks of making that more like 3.7ms with a
|
|
_small call_ to his buddy Laurent). So I went about my business, racking first
|
|
the WiFi enabled **console.par.ipng.nl**, connecting to WiFi with it, and
|
|
finding my router on **frpar0.ipng.ch** over IPMI serial-over-lan. Configuring
|
|
one Tengig port on the Intel X710 towards **er01.lil01.ip-max.net** and another
|
|
Tengig port towards **er01.zrh56.ip-max.net**.
|
|
|
|
I have two Supermicros on backorder, one of which will go to Lille to replace
|
|
the Dell R610 that I placed temporarily in that site, and the other will go to
|
|
Geneva. At that point, I will break up this VLL to become one from here to
|
|
Geneva and another from Geneva back home to Zürich. Seriously though I will
|
|
have to stop Fred's enthusiasm because he also mentioned somthing about
|
|
[LyonIX](https://www.rezopole.net/fr/ixp/lyonix) and another small stopover of
|
|
1HE and 35W over there... this is addictive, save me from myself!!1
|
|
|
|
## Connectivity on the FLAP
|
|
|
|
All week, Fred has been talking about _the FLAP_, which I know to be a term
|
|
but I really never bothered to ask about it -- it turns out, he explained,
|
|
that it stands for **F**rankfurt, **L**ondon, **A**msterdam, and **P**aris.
|
|
I'm not in London (yet, please don't dare me ...), however I can legitimately
|
|
claim I am on _the FLAP_ because I have a router in **L**ille. So there's that :)
|
|
|
|
{{< image float="left" src="/assets/network/frpar0.png" alt="frpar0" >}}
|
|
|
|
Fred has ordered my FranceIX connection this afternoon, delivered from a
|
|
20Gig LAG on **er02.par02.ip-max.net** and directly into my router there. In
|
|
the mean time, I will be busy configuring my DE-CIX port from [a previous post]({{< ref "2021-05-17-frankfurt" >}}).
|
|
|
|
The console server here (a standard issue APU3 with 802.11ac WiFi broadcasting
|
|
_AS50869 PAR_ with password _IPngGuest_, you're welcome), connects to the
|
|
router with IPMI, while the router itself connects via USB serial back to the
|
|
APU for maximum resilience.
|
|
|
|
## A hard knock life
|
|
|
|
{{< image width="300px" float="right" src="/assets/network/happy-fred.png" alt="happy-fred" >}}
|
|
|
|
It was not without troubles today. When configuring my VLL to Zurich, I had
|
|
misconfigured the **er01.zrh56.ip-max.net** side (a Cisco 7600, which is on its
|
|
way out), and the VLL would not come up. I could see traffic going in one
|
|
direction but not in the other ... which typically does not make OSPF adjacencies
|
|
happen. After about an hour of messing around, I puppydog-eyed to Fred who
|
|
proceeded to find my bug within 30 seconds: I needed to do some VLAN gymnastics
|
|
by adding `rewrite ingress tag pop 1 symmetric` and as well adding 4 bytes
|
|
to the MTU (so `mtu 9018` total, cuz packets gotta be sourced directly from the
|
|
[jumbo-jumbo club](https://www.rheinfelderbierhalle.com/)!).
|
|
|
|
But, Fred was miserable as well because he had updated a Xen hypervisor which
|
|
ended up not being able to boot because of a broken LVM configuration. So we
|
|
literally swapped laptops and while he fixed my VLL, I fixed his volume group
|
|
by running `update-initramfs -u -k all` from a recovery Debian USB stick. For
|
|
an extra bonus, here's a picture of Happy Fred at the local Cafe Leopard, where
|
|
pretty much every day you can find a host of locals and international nerds who
|
|
have emerged from the server floor.
|
|
|
|
And Mael also helped with the serial port - I had put it into `115200` baud
|
|
but not pinned it at `8N1` for the APU serial console. It shot into life
|
|
as soon as he gave that tip and I committed the config, bravo!
|
|
|
|
I tend to believe it will not be necessary for me to physically visit the
|
|
facility that often -- simple hardware, no spinning disks, an APU connecting to
|
|
IPMI for full HTML5 based KVM control and serial-over-lan, and the router
|
|
exposing a console back to the APU, which has aan OOB network connection from
|
|
[AS25091](https://as25091.peeringdb.com/). Yeah, I think I'll be good.
|
|
|
|
## The results
|
|
|
|
But this was a special one indeed, because up until now, my traceroutes kept
|
|
on getting longer and longer as I deployed in Frankfurt, Amsterdam, and Lille.
|
|
Deploying in Paris therefore looked like this initially, as the packets took,
|
|
let us say, the scenic route to my basement:
|
|
|
|
```
|
|
pim@frpar0:~$ traceroute chumbucket.ipng.nl
|
|
traceroute to chumbucket.ipng.nl (194.1.163.93), 30 hops max, 60 byte packets
|
|
1 frggh0.ipng.ch (194.1.163.30) 4.915 ms 4.885 ms 4.866 ms
|
|
2 nlams0.ipng.ch (194.1.163.28) 12.396 ms 12.398 ms 12.382 ms
|
|
3 defra0.ipng.ch (194.1.163.26) 18.536 ms 18.520 ms 18.541 ms
|
|
4 chrma0.ipng.ch (194.1.163.24) 24.572 ms 24.557 ms 24.542 ms
|
|
5 chgtg0.ipng.ch (194.1.163.9) 24.549 ms 24.510 ms 24.517 ms
|
|
6 chbtl1.ipng.ch (194.1.163.18) 24.707 ms 25.114 ms 25.038 ms
|
|
7 chumbucket.ipng.nl (194.1.163.93) 25.320 ms 25.564 ms 25.452 ms
|
|
```
|
|
|
|
That's quite the scenic route indeed. But! On this glorious day, at exactly
|
|
16:34 UTC, the Tengig european IPv4 and IPv6 ring was closed, with one final set
|
|
of OSPF adjacencies:
|
|
|
|
```
|
|
pim@frpar0:~$ show protocols ospfv3 neighbor
|
|
Neighbor ID Pri DeadTime State/IfState Duration I/F[State]
|
|
194.1.163.34 1 00:00:37 Full/PointToPoint 01:40:51 dp0p6s0f0.100[PointToPoint]
|
|
194.1.163.1 1 00:00:33 Full/PointToPoint 00:01:28 dp0p6s0f1.100[PointToPoint]
|
|
```
|
|
|
|
Which allowed the ring to hone in on shortest path at its best - East bound to
|
|
Frankfurt and Amsterdam, and West bound to Paris and Lille. Link and equipment
|
|
failures will not bother me that much, OSPF and OSPFv3 will take care of
|
|
rerouting me around network problems, which considering the ASR9k at IP-Max,
|
|
I do think will be the exception:
|
|
```
|
|
pim@chumbucket:~$ traceroute frggh0.ipng.ch
|
|
traceroute to frggh0.ipng.ch (194.1.163.34), 30 hops max, 60 byte packets
|
|
1 chbtl1.ipng.ch (194.1.163.67) 0.317 ms 0.238 ms 0.190 ms
|
|
2 chgtg0.ipng.ch (194.1.163.19) 0.619 ms 0.574 ms 0.531 ms
|
|
3 frpar0.ipng.ch (194.1.163.40) 15.271 ms 15.226 ms 15.174 ms
|
|
4 frggh0.ipng.ch (194.1.163.34) 20.059 ms 20.020 ms 19.977 ms
|
|
|
|
pim@chumbucket:~$ traceroute nlams0.ipng.ch
|
|
traceroute to nlams0.ipng.ch (194.1.163.32), 30 hops max, 60 byte packets
|
|
1 chbtl1.ipng.ch (194.1.163.67) 0.345 ms 0.198 ms 0.284 ms
|
|
2 chgtg0.ipng.ch (194.1.163.19) 0.610 ms 0.518 ms 0.538 ms
|
|
3 chrma0.ipng.ch (194.1.163.8) 0.732 ms 0.750 ms 0.716 ms
|
|
4 defra0.ipng.ch (194.1.163.25) 6.835 ms 6.802 ms 6.767 ms
|
|
5 nlams0.ipng.ch (194.1.163.32) 12.799 ms 12.765 ms 12.731 ms
|
|
```
|
|
|
|
Of course with impeccable throughput, bien sûr:
|
|
|
|
<iframe width="300" height="400" src="https://www.youtube.com/embed/9dLzGvXkqMI?autoplay=1&mute=0" style="width:300px; float: right; margin-left: 1em; margin-bottom: 1em;">
|
|
</iframe>
|
|
```
|
|
pim@frpar0:~$ iperf3 -c frggh0.ipng.ch -R
|
|
...
|
|
[ 5] 0.00-10.00 sec 10.7 GBytes 9.22 Gbits/sec 1 sender
|
|
[ 5] 0.00-10.00 sec 10.7 GBytes 9.22 Gbits/sec receiver
|
|
|
|
pim@frpar0:~$ iperf3 -c chgtg0.ipng.ch -R
|
|
...
|
|
[ 5] 0.00-10.01 sec 11.2 GBytes 9.42 Gbits/sec 1 sender
|
|
[ 5] 0.00-10.00 sec 11.2 GBytes 9.42 Gbits/sec receiver
|
|
```
|
|
|
|
I'm tired, but ultimately satisfied on taking my private AS50869 across
|
|
_the FLAP_, with a physical presence in each, and an IXP connection and
|
|
TenGig bidirectional, and TenGig IP Transit at each location. I think this
|
|
network is good to go for the next few years at least.
|