--- date: "2021-07-03T22:16:44Z" title: IPng arrives in Geneva aliases: - /s/articles/2021/07/03/geneva.html --- I've been planning a network expansion for a while now. For the next few weeks, I will be in total geek-mode as I travel to several European cities to deploy AS50869 on a european ring. At the same time, my buddy Fred from [IP-Max](https://ip-max.net/) has been wanting to go to Amsterdam. IP-Max's [network](https://as25091.peeringdb.com/) is considerably larger than mine, but it just never clicked with the right set of circumstances for them to deploy in the Netherlands, until the stars aligned ... ## Deployment {{< image width="400px" float="left" src="/assets/network/qdr.png" alt="Quai du Rhône" >}} After our adventure in [Frankfurt]({{< ref "2021-05-17-frankfurt" >}}), [Amsterdam]({{< ref "2021-05-26-amsterdam" >}}), [Lille]({{< ref "2021-05-28-lille" >}}), and [Paris]({{< ref "2021-06-01-paris" >}}) came to an end, I still had a few loose ends to tie up. In particular, in Lille I had dropped an old Dell R610 while waiting for new Supermicros to be delivered. There is benefit to having one standard footprint setup, in my case an PCEngines `APU2`, Supermicro `5018D-FN8T` and Intel `X710-DA4` expansion NIC. They run fantastic with [DANOS](https://danosproject.org/) and [VPP](https://fd.io/) applications. Of course, we mustn't forget _home base_, Geneva, where IP-Max has its headquarters in a beautiful mansion pictured here. At the same time, my family likes to take one trip per month to a city we don't usually go, sort of to keep up with real life as we are now more and more able to travel. Marina has a niece in Geneva, who has lived and worked there for 20+ years, so we figured we'd combine these things and stay the weekend at her place. After making our way from Zurich to Geneva, a trip that took us just short of six hours (!) by car, we arrived at the second half of the Belgium:Italy eurocup soccer match. It was perhaps due to our tardiness and lack of physical supportering, that the belgians lost the match that day. Sorry! ### Connectivity {{< image width="400px" float="right" src="/assets/network/chplo0-rack.png" alt="Geneva Rack" >}} My current circuit runs from Paris (Leon Frot), `frpar0.ipng.ch` over a direct DWDM wave to Zurich where I pick it up on `chgtg0.ipng.ch` at Interxion Glattbrugg. So what we'll do is break open this VLL at the IP-Max side, insert the new router `chplo0.ipng.ch`, and reconfigure the Paris side to go to the new router, and the new router to create another VLL back to Zurich, which due to the toplogy of IP-Max's underlying DWDM network will traverse Paris - Lyon - Geneva instead (shaving off ~1.5ms of latency at the same time). I hung up the `APU2` OOB server and the `5018D-FN8T` router, and another Dell R610 to run virtual machines at Safehost SH1 in Plan-les-Ouates, a southern suburb of Geneva. I connected one 10G port to `er01.gva20.ip-max.net` and another 10G port to `er02.gva20.ip-max.net` to obtain maximum availability benefits. As an example of what the configuration on the ASR9k platform looks like for this type of operation, here's what I committed on `er01.gva20`. Of course, first things first: let's ensure that the OOB machine has connectivity, by allocating a /64 IPv6 and /29 IPv4. I usually configure myself a BGP transit session in the same subnet, which means we'll want to bridge the 1G UTP connection of the APU with the 10G fiber connection of the Supermicro router, like so: ``` interface BVI911 description Cust: IPng OOB and Transit ipv4 address 46.20.250.105 255.255.255.248 ipv4 unreachables disable ipv6 nd suppress-ra ipv6 address 2a02:2528:ff05::1/64 ipv6 enable load-interval 30 ! interface GigabitEthernet0/7/0/38 description Cust: IPng APU (OOB) mtu 9064 load-interval 30 l2transport ! ! interface TenGigE0/1/0/3 description Cust: IPng (VLL and Transit) mtu 9014 ! interface TenGigE0/1/0/3.911 l2transport encapsulation dot1q 911 exact rewrite ingress tag pop 1 symmetric mtu 9018 ! l2vpn bridge group BG_IPng bridge-domain BD_IPng911 interface Te0/1/0/3.911 ! interface GigabitEthernet0/7/0/38 ! routed interface BVI911 ! ! ! ``` After this, we pulled UTP cable and configured the `APU2`, which then has an internal network towards the IPMI port of the Supermicro, and from there on, the configuration becomes much easier. Of course, all config can be done wirelessly, because the APU `console.plo.ipng.nl` acts as a WiFi access point, so I connect to it and commit the network configs. Once that's online and happy, the router `chplo0.ipng.ch` is next. For this, on `er02.par02.ip-max.net`, I reconfigure the current VLL to point to the loopback of this router `er01.gva20.ip-max.net` using the same `pw-id`. Then, I can configure this router as follows: ``` interface TenGigE0/1/0/3.100 l2transport description Cust: IPng VLL to par02 encapsulation dot1q 100 rewrite ingress tag pop 1 symmetric mtu 9018 ! l2vpn pw-class EOMPLS-PW-CLASS encapsulation mpls transport-mode ethernet ! ! xconnect group IPng p2p IPng_to_par02 interface TenGigE0/1/0/3.100 neighbor ipv4 46.20.255.33 pw-id 210535705 pw-class EOMPLS-PW-CLASS ! ! ! ``` ## The results And with that, the pseudowire is constructed, and the original interface on `frpar0.ipng.ch` directly sees the interface here on `chplo0.ipng.ch` using jumboframes of 9000 bytes (+14 bytes of ethernet overhead and +4 bytes of VLAN tag on the ingress interface). It is as if the routers are directly connected by a very long ethernet cable, a _pseudo-wire_ if you wish. Super low pingtimes are observed between this new router in Geneva and the existing two in Paris and Zurich: ``` pim@chplo0:~$ /bin/ping -4 -c5 frpar0 PING frpar0.ipng.ch (194.1.163.33) 56(84) bytes of data. 64 bytes from frpar0.ipng.ch (194.1.163.33): icmp_seq=1 ttl=64 time=8.78 ms 64 bytes from frpar0.ipng.ch (194.1.163.33): icmp_seq=2 ttl=64 time=8.80 ms 64 bytes from frpar0.ipng.ch (194.1.163.33): icmp_seq=3 ttl=64 time=8.81 ms 64 bytes from frpar0.ipng.ch (194.1.163.33): icmp_seq=4 ttl=64 time=8.82 ms 64 bytes from frpar0.ipng.ch (194.1.163.33): icmp_seq=5 ttl=64 time=8.85 ms --- frpar0.ipng.ch ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 10ms rtt min/avg/max/mdev = 8.783/8.810/8.846/0.104 ms pim@chplo0:~$ /bin/ping -6 -c5 chgtg0 PING chgtg0(chgtg0.ipng.ch (2001:678:d78::1)) 56 data bytes 64 bytes from chgtg0.ipng.ch (2001:678:d78::1): icmp_seq=1 ttl=64 time=4.51 ms 64 bytes from chgtg0.ipng.ch (2001:678:d78::1): icmp_seq=2 ttl=64 time=4.44 ms 64 bytes from chgtg0.ipng.ch (2001:678:d78::1): icmp_seq=3 ttl=64 time=4.36 ms 64 bytes from chgtg0.ipng.ch (2001:678:d78::1): icmp_seq=4 ttl=64 time=4.47 ms 64 bytes from chgtg0.ipng.ch (2001:678:d78::1): icmp_seq=5 ttl=64 time=4.41 ms --- chgtg0 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 10ms rtt min/avg/max/mdev = 4.362/4.436/4.506/0.077 ms ``` For good measure I've also connected to FreeIX, a new internet exchange project I'm working on, that will span the Geneva, Zurich and Lugano areas. More on that in a future post! ``` pim@chplo0:~$ iperf3 -4 -c 185.1.205.1 ## chgtg0.ipng.ch Connecting to host 185.1.205.1, port 5201 [ 5] local 185.1.205.2 port 46872 connected to 185.1.205.1 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 809 MBytes 6.78 Gbits/sec 4 11.4 MBytes [ 5] 1.00-2.00 sec 869 MBytes 7.29 Gbits/sec 0 11.4 MBytes [ 5] 2.00-3.00 sec 865 MBytes 7.25 Gbits/sec 0 11.4 MBytes [ 5] 3.00-4.00 sec 868 MBytes 7.28 Gbits/sec 0 11.4 MBytes [ 5] 4.00-5.00 sec 836 MBytes 7.01 Gbits/sec 0 11.4 MBytes [ 5] 5.00-6.00 sec 852 MBytes 7.15 Gbits/sec 0 11.4 MBytes [ 5] 6.00-7.00 sec 865 MBytes 7.26 Gbits/sec 0 11.4 MBytes [ 5] 7.00-8.00 sec 865 MBytes 7.26 Gbits/sec 0 11.4 MBytes [ 5] 8.00-9.00 sec 861 MBytes 7.22 Gbits/sec 0 11.4 MBytes [ 5] 9.00-10.00 sec 860 MBytes 7.22 Gbits/sec 0 11.4 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 8.35 GBytes 7.17 Gbits/sec 4 sender [ 5] 0.00-10.01 sec 8.35 GBytes 7.16 Gbits/sec receiver iperf Done. ``` You kind of get used to performance stats like this, but that said, it's nice to see that performance over FreeIX is slightly *lower* than performance over the IPng backbone, and this is because on my VLLs, I can make use of jumbo frames, which gives me 20% or so better performance (currently 9.62 Gbits/sec). Currently I'm busy at work in the background completing the configuration, the management environment and physical infrastructure for the internet exchange. I'm planning to make a more complete post about the FreeIX project in a few weeks once it's ready for launch. Stay tuned!