ipng.ch/content/articles/2021-07-03-geneva.md

---
date: "2021-07-03T22:16:44Z"
title: IPng arrives in Geneva
aliases:
- /s/articles/2021/07/03/geneva.html
---

I've been planning a network expansion for a while now. For the next few weeks,
I will be in total geek-mode as I travel to several European cities to deploy
AS50869 on a european ring. At the same time, my buddy Fred from
[IP-Max](https://ip-max.net/) has been wanting to go to Amsterdam. IP-Max's
[network](https://as25091.peeringdb.com/) is considerably larger than mine, but
it just never clicked with the right set of circumstances for them to deploy
in the Netherlands, until the stars aligned ...

## Deployment

{{< image width="400px" float="left" src="/assets/network/qdr.png" alt="Quai du Rh&ocirc;ne" >}}

After our adventure in [Frankfurt]({{< ref "2021-05-17-frankfurt" >}}),
[Amsterdam]({{< ref "2021-05-26-amsterdam" >}}), [Lille]({{< ref "2021-05-28-lille" >}}),
and [Paris]({{< ref "2021-06-01-paris" >}}) came to an end, I still had a few
loose ends to tie up. In particular, in Lille I had dropped an old Dell R610
while waiting for new Supermicros to be delivered. There is benefit to having
one standard footprint setup, in my case an PCEngines `APU2`, Supermicro
`5018D-FN8T` and Intel `X710-DA4` expansion NIC. They run fantastic with
[DANOS](https://danosproject.org/) and [VPP](https://fd.io/) applications.

Of course, we mustn't forget _home base_, Geneva, where IP-Max has its
headquarters in a beautiful mansion pictured here. At the same time, my
family likes to take one trip per month to a city we don't usually go, sort
of to keep up with real life as we are now more and more able to travel.
Marina has a niece in Geneva, who has lived and worked there for 20+ years,
so we figured we'd combine these things and stay the weekend at her place.

After making our way from Zurich to Geneva, a trip that took us just short
of six hours (!) by car, we arrived at the second half of the Belgium:Italy
eurocup soccer match. It was perhaps due to our tardiness and lack of physical
supportering, that the belgians lost the match that day. Sorry!

### Connectivity

{{< image width="400px" float="right" src="/assets/network/chplo0-rack.png" alt="Geneva Rack" >}}

My current circuit runs from Paris (Leon Frot), `frpar0.ipng.ch` over a direct
DWDM wave to Zurich where I pick it up on `chgtg0.ipng.ch` at Interxion
Glattbrugg. So what we'll do is break open this VLL at the IP-Max side,
insert the new router `chplo0.ipng.ch`, and reconfigure the Paris side
to go to the new router, and the new router to create another VLL back
to Zurich, which due to the toplogy of IP-Max's underlying DWDM network
will traverse Paris - Lyon - Geneva instead (shaving off ~1.5ms of latency
at the same time).

I hung up the `APU2` OOB server and the `5018D-FN8T` router, and another Dell
R610 to run virtual machines at Safehost SH1 in Plan-les-Ouates, a southern
suburb of Geneva. I connected one 10G port to `er01.gva20.ip-max.net` and
another 10G port to `er02.gva20.ip-max.net` to obtain maximum availability
benefits. As an example of what the configuration on the ASR9k platform looks
like for this type of operation, here's what I committed on `er01.gva20`.

Of course, first things first: let's ensure that the OOB machine has
connectivity, by allocating a /64 IPv6 and /29 IPv4. I usually configure
myself a BGP transit session in the same subnet, which means we'll want to
bridge the 1G UTP connection of the APU with the 10G fiber connection of
the Supermicro router, like so:

```
interface BVI911
 description Cust: IPng OOB and Transit
 ipv4 address 46.20.250.105 255.255.255.248
 ipv4 unreachables disable
 ipv6 nd suppress-ra
 ipv6 address 2a02:2528:ff05::1/64
 ipv6 enable
 load-interval 30
!
interface GigabitEthernet0/7/0/38
 description Cust: IPng APU (OOB)
 mtu 9064
 load-interval 30
 l2transport
 !
!
interface TenGigE0/1/0/3
 description Cust: IPng (VLL and Transit)
 mtu 9014
!
interface TenGigE0/1/0/3.911 l2transport
 encapsulation dot1q 911 exact
 rewrite ingress tag pop 1 symmetric
 mtu 9018
!

l2vpn
 bridge group BG_IPng
  bridge-domain BD_IPng911
   interface Te0/1/0/3.911
   !
   interface GigabitEthernet0/7/0/38
   !
   routed interface BVI911
  !
 !
!
```

After this, we pulled UTP cable and configured the `APU2`, which then has an
internal network towards the IPMI port of the Supermicro, and from there on,
the configuration becomes much easier. Of course, all config can be done
wirelessly, because the APU `console.plo.ipng.nl` acts as a WiFi access
point, so I connect to it and commit the network configs.

Once that's online and happy, the router `chplo0.ipng.ch` is next. For this,
on `er02.par02.ip-max.net`, I reconfigure the current VLL to point to the
loopback of this router `er01.gva20.ip-max.net` using the same `pw-id`. Then,
I can configure this router as follows:

```
interface TenGigE0/1/0/3.100 l2transport
 description Cust: IPng VLL to par02
 encapsulation dot1q 100
 rewrite ingress tag pop 1 symmetric
 mtu 9018
!

l2vpn
 pw-class EOMPLS-PW-CLASS
  encapsulation mpls
   transport-mode ethernet
  !
 !
 xconnect group IPng
  p2p IPng_to_par02
   interface TenGigE0/1/0/3.100
   neighbor ipv4 46.20.255.33 pw-id 210535705
    pw-class EOMPLS-PW-CLASS
   !
  !
 !
```

## The results

And with that, the pseudowire is constructed, and the original interface on
`frpar0.ipng.ch` directly sees the interface here on `chplo0.ipng.ch` using
jumboframes of 9000 bytes (+14 bytes of ethernet overhead and +4 bytes of VLAN
tag on the ingress interface). It is as if the routers are directly connected
by a very long ethernet cable, a _pseudo-wire_ if you wish. Super low pingtimes
are observed between this new router in Geneva and the existing two in Paris
and Zurich:

```
pim@chplo0:~$ /bin/ping -4 -c5 frpar0
PING frpar0.ipng.ch (194.1.163.33) 56(84) bytes of data.
64 bytes from frpar0.ipng.ch (194.1.163.33): icmp_seq=1 ttl=64 time=8.78 ms
64 bytes from frpar0.ipng.ch (194.1.163.33): icmp_seq=2 ttl=64 time=8.80 ms
64 bytes from frpar0.ipng.ch (194.1.163.33): icmp_seq=3 ttl=64 time=8.81 ms
64 bytes from frpar0.ipng.ch (194.1.163.33): icmp_seq=4 ttl=64 time=8.82 ms
64 bytes from frpar0.ipng.ch (194.1.163.33): icmp_seq=5 ttl=64 time=8.85 ms

--- frpar0.ipng.ch ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 10ms
rtt min/avg/max/mdev = 8.783/8.810/8.846/0.104 ms
pim@chplo0:~$ /bin/ping -6 -c5 chgtg0
PING chgtg0(chgtg0.ipng.ch (2001:678:d78::1)) 56 data bytes
64 bytes from chgtg0.ipng.ch (2001:678:d78::1): icmp_seq=1 ttl=64 time=4.51 ms
64 bytes from chgtg0.ipng.ch (2001:678:d78::1): icmp_seq=2 ttl=64 time=4.44 ms
64 bytes from chgtg0.ipng.ch (2001:678:d78::1): icmp_seq=3 ttl=64 time=4.36 ms
64 bytes from chgtg0.ipng.ch (2001:678:d78::1): icmp_seq=4 ttl=64 time=4.47 ms
64 bytes from chgtg0.ipng.ch (2001:678:d78::1): icmp_seq=5 ttl=64 time=4.41 ms

--- chgtg0 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 10ms
rtt min/avg/max/mdev = 4.362/4.436/4.506/0.077 ms
```

For good measure I've also connected to FreeIX, a new internet exchange project
I'm working on, that will span the Geneva, Zurich and Lugano areas. More on that
in a future post!

```
pim@chplo0:~$ iperf3 -4 -c 185.1.205.1 ## chgtg0.ipng.ch
Connecting to host 185.1.205.1, port 5201
[  5] local 185.1.205.2 port 46872 connected to 185.1.205.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   809 MBytes  6.78 Gbits/sec    4   11.4 MBytes
[  5]   1.00-2.00   sec   869 MBytes  7.29 Gbits/sec    0   11.4 MBytes
[  5]   2.00-3.00   sec   865 MBytes  7.25 Gbits/sec    0   11.4 MBytes
[  5]   3.00-4.00   sec   868 MBytes  7.28 Gbits/sec    0   11.4 MBytes
[  5]   4.00-5.00   sec   836 MBytes  7.01 Gbits/sec    0   11.4 MBytes
[  5]   5.00-6.00   sec   852 MBytes  7.15 Gbits/sec    0   11.4 MBytes
[  5]   6.00-7.00   sec   865 MBytes  7.26 Gbits/sec    0   11.4 MBytes
[  5]   7.00-8.00   sec   865 MBytes  7.26 Gbits/sec    0   11.4 MBytes
[  5]   8.00-9.00   sec   861 MBytes  7.22 Gbits/sec    0   11.4 MBytes
[  5]   9.00-10.00  sec   860 MBytes  7.22 Gbits/sec    0   11.4 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  8.35 GBytes  7.17 Gbits/sec    4             sender
[  5]   0.00-10.01  sec  8.35 GBytes  7.16 Gbits/sec                  receiver

iperf Done.
```

You kind of get used to performance stats like this, but that said, it's nice
to see that performance over FreeIX is slightly *lower* than performance over
the IPng backbone, and this is because on my VLLs, I can make use of jumbo
frames, which gives me 20% or so better performance (currently 9.62 Gbits/sec).

Currently I'm busy at work in the background completing the configuration, the
management environment and physical infrastructure for the internet exchange.
I'm planning to make a more complete post about the FreeIX project in a few
weeks once it's ready for launch. Stay tuned!