---
date: "2021-06-01T18:16:42Z"
title: IPng arrives in Paris
aliases:
- /s/articles/2021/06/01/paris.html
---

I've been planning a network expansion for a while now. For the next few weeks,
I will be in total geek-mode as I travel to several European cities to deploy
AS50869 on a european ring. At the same time, my buddy Fred from
[IP-Max](https://ip-max.net/) has been wanting to go to Amsterdam. IP-Max's
[network](https://as25091.peeringdb.com/) is considerably larger than mine, but
it just never clicked with the right set of circumstances for them to deploy
in the Netherlands, until the stars aligned ...

## First a word

I just wanted to start with a note on how special it is to partner with [IP-Max](https://www.ip-max.net/)
and having known the founder Fred for so long affords me this specific trip.
It must be known that building a 10G pan-european ring is not an easy thing to
do, and I do appreciate very much the kindness Fred has shown IPng even though
we are under contract -- the ability to travel to every point of presence and
get the _founder's tour_ in each place, get to know the local DC ops, sales
directors, field technicians and local customers, is simply golden. Thank you.

## Deployment

{{< image width="300px" float="right" src="/assets/network/frpar0-rack.png" alt="Paris KDDI" >}}

The city of love, corny as it sounds, also happens to have quite a bit of _fibre
noire_, happily lit by hundreds of local and international carriers. There are
two places in the dead-center of the city, a _cabaret voltaire_ called Telehouse
TH2, and a new facility from [KDDI](https://fra.kddi.com/) which is in the same
physical block, aptly addressed as 65 Rue L&eacute;on Frot, 75011 Paris and that
is where my beloved router **frpar0.ipng.ch** will be. Note that in Lille,
actually I had to make do with **frggh0** which was in the town of
Sainghin-en-M&eacute;lantois. This one lives in Paris, none of that suburban
bullshit. This is the real deal. There's probably more connectivity in this
one block than in all of the Paris metro combined, maybe even more than
all of France combined -- peut &ecirc;tre :)

After having visited the older location, we took the router, APU, cables and
optics to L&eacute;on Frot. The rack was quickly found, and it is obvious that
this is the location: A _fridge_ was awaiting us, and I reserved two Tengig
ports on the ASR9010, one towards Lille and one towards Z&uuml;rich (which will be
Geneva later on).

I'm getting the hang of this VLL stuff after our adventures, previously in
[Lille]({{< ref "2021-05-28-lille" >}}) at CIV, which is a lazy 4.8ms away
from this place (Fred already speaks of making that more like 3.7ms with a
_small call_ to his buddy Laurent). So I went about my business, racking first
the WiFi enabled **console.par.ipng.nl**, connecting to WiFi with it, and
finding my router on **frpar0.ipng.ch** over IPMI serial-over-lan. Configuring
one Tengig port on the Intel X710 towards **er01.lil01.ip-max.net** and another
Tengig port towards **er01.zrh56.ip-max.net**.

I have two Supermicros on backorder, one of which will go to Lille to replace
the Dell R610 that I placed temporarily in that site, and the other will go to
Geneva. At that point, I will break up this VLL to become one from here to
Geneva and another from Geneva back home to Z&uuml;rich. Seriously though I will
have to stop Fred's enthusiasm because he also mentioned somthing about
[LyonIX](https://www.rezopole.net/fr/ixp/lyonix) and another small stopover of
1HE and 35W over there... this is addictive, save me from myself!!1

## Connectivity on the FLAP

All week, Fred has been talking about _the FLAP_, which I know to be a term
but I really never bothered to ask about it -- it turns out, he explained,
that it stands for **F**rankfurt, **L**ondon, **A**msterdam, and **P**aris.
I'm not in London (yet, please don't dare me ...), however I can legitimately
claim I am on _the FLAP_ because I have a router in **L**ille. So there's that :)

{{< image float="left" src="/assets/network/frpar0.png" alt="frpar0" >}}

Fred has ordered my FranceIX connection this afternoon, delivered from a
20Gig LAG on **er02.par02.ip-max.net** and directly into my router there. In
the mean time, I will be busy configuring my DE-CIX port from [a previous post]({{< ref "2021-05-17-frankfurt" >}}).

The console server here (a standard issue APU3 with 802.11ac WiFi broadcasting
_AS50869 PAR_ with password _IPngGuest_, you're welcome), connects to the
router with IPMI, while the router itself connects via USB serial back to the
APU for maximum resilience.

## A hard knock life

{{< image width="300px" float="right" src="/assets/network/happy-fred.png" alt="happy-fred" >}}

It was not without troubles today. When configuring my VLL to Zurich, I had
misconfigured the **er01.zrh56.ip-max.net** side (a Cisco 7600, which is on its
way out), and the VLL would not come up. I could see traffic going in one
direction but not in the other ... which typically does not make OSPF adjacencies
happen. After about an hour of messing around, I puppydog-eyed to Fred who
proceeded to find my bug within 30 seconds: I needed to do some VLAN gymnastics
by adding `rewrite ingress tag pop 1 symmetric` and as well adding 4 bytes
to the MTU (so `mtu 9018` total, cuz packets gotta be sourced directly from the
[jumbo-jumbo club](https://www.rheinfelderbierhalle.com/)!).

But, Fred was miserable as well because he had updated a Xen hypervisor which
ended up not being able to boot because of a broken LVM configuration. So we
literally swapped laptops and while he fixed my VLL, I fixed his volume group
by running `update-initramfs -u -k all` from a recovery Debian USB stick. For
an extra bonus, here's a picture of Happy Fred at the local Cafe Leopard, where
pretty much every day you can find a host of locals and international nerds who
have emerged from the server floor.

And Mael also helped with the serial port - I had put it into `115200` baud
but not pinned it at `8N1` for the APU serial console. It shot into life
as soon as he gave that tip and I committed the config, bravo!

I tend to believe it will not be necessary for me to physically visit the
facility that often -- simple hardware, no spinning disks, an APU connecting to
IPMI for full HTML5 based KVM control and serial-over-lan, and the router
exposing a console back to the APU, which has aan OOB network connection from
[AS25091](https://as25091.peeringdb.com/). Yeah, I think I'll be good.

## The results

But this was a special one indeed, because up until now, my traceroutes kept
on getting longer and longer as I deployed in Frankfurt, Amsterdam, and Lille.
Deploying in Paris therefore looked like this initially, as the packets took,
let us say, the scenic route to my basement:

```
pim@frpar0:~$ traceroute chumbucket.ipng.nl
traceroute to chumbucket.ipng.nl (194.1.163.93), 30 hops max, 60 byte packets
 1  frggh0.ipng.ch (194.1.163.30)  4.915 ms  4.885 ms  4.866 ms
 2  nlams0.ipng.ch (194.1.163.28)  12.396 ms  12.398 ms  12.382 ms
 3  defra0.ipng.ch (194.1.163.26)  18.536 ms  18.520 ms  18.541 ms
 4  chrma0.ipng.ch (194.1.163.24)  24.572 ms  24.557 ms  24.542 ms
 5  chgtg0.ipng.ch (194.1.163.9)  24.549 ms  24.510 ms  24.517 ms
 6  chbtl1.ipng.ch (194.1.163.18)  24.707 ms  25.114 ms  25.038 ms
 7  chumbucket.ipng.nl (194.1.163.93)  25.320 ms  25.564 ms  25.452 ms
```

That's quite the scenic route indeed. But! On this glorious day, at exactly
16:34 UTC, the Tengig european IPv4 and IPv6 ring was closed, with one final set
of OSPF adjacencies:

```
pim@frpar0:~$ show protocols ospfv3 neighbor
Neighbor ID     Pri    DeadTime    State/IfState         Duration I/F[State]
194.1.163.34      1    00:00:37     Full/PointToPoint    01:40:51 dp0p6s0f0.100[PointToPoint]
194.1.163.1       1    00:00:33     Full/PointToPoint    00:01:28 dp0p6s0f1.100[PointToPoint]
```

Which allowed the ring to hone in on shortest path at its best - East bound to
Frankfurt and Amsterdam, and West bound to Paris and Lille. Link and equipment
failures will not bother me that much, OSPF and OSPFv3 will take care of
rerouting me around network problems, which considering the ASR9k at IP-Max,
I do think will be the exception:
```
pim@chumbucket:~$ traceroute frggh0.ipng.ch
traceroute to frggh0.ipng.ch (194.1.163.34), 30 hops max, 60 byte packets
 1  chbtl1.ipng.ch (194.1.163.67)  0.317 ms  0.238 ms  0.190 ms
 2  chgtg0.ipng.ch (194.1.163.19)  0.619 ms  0.574 ms  0.531 ms
 3  frpar0.ipng.ch (194.1.163.40)  15.271 ms  15.226 ms  15.174 ms
 4  frggh0.ipng.ch (194.1.163.34)  20.059 ms  20.020 ms  19.977 ms

pim@chumbucket:~$ traceroute nlams0.ipng.ch
traceroute to nlams0.ipng.ch (194.1.163.32), 30 hops max, 60 byte packets
 1  chbtl1.ipng.ch (194.1.163.67)  0.345 ms  0.198 ms  0.284 ms
 2  chgtg0.ipng.ch (194.1.163.19)  0.610 ms  0.518 ms  0.538 ms
 3  chrma0.ipng.ch (194.1.163.8)  0.732 ms  0.750 ms  0.716 ms
 4  defra0.ipng.ch (194.1.163.25)  6.835 ms  6.802 ms  6.767 ms
 5  nlams0.ipng.ch (194.1.163.32)  12.799 ms  12.765 ms  12.731 ms
```

Of course with impeccable throughput, bien s&ucirc;r:

<iframe width="300" height="400" src="https://www.youtube.com/embed/9dLzGvXkqMI?autoplay=1&mute=0" style="width:300px; float: right; margin-left: 1em; margin-bottom: 1em;">
</iframe>
```
pim@frpar0:~$ iperf3 -c frggh0.ipng.ch -R
...
[  5]   0.00-10.00  sec  10.7 GBytes  9.22 Gbits/sec    1             sender
[  5]   0.00-10.00  sec  10.7 GBytes  9.22 Gbits/sec                  receiver

pim@frpar0:~$ iperf3 -c chgtg0.ipng.ch -R
...
[  5]   0.00-10.01  sec  11.2 GBytes  9.42 Gbits/sec    1             sender
[  5]   0.00-10.00  sec  11.2 GBytes  9.42 Gbits/sec                  receiver
```

I'm tired, but ultimately satisfied on taking my private AS50869 across
_the FLAP_, with a physical presence in each, and an IXP connection and
TenGig bidirectional, and TenGig IP Transit at each location. I think this
network is good to go for the next few years at least.
