213 lines
8.8 KiB
Markdown
213 lines
8.8 KiB
Markdown
---
|
|
date: "2021-07-03T22:16:44Z"
|
|
title: IPng arrives in Geneva
|
|
aliases:
|
|
- /s/articles/2021/07/03/geneva.html
|
|
---
|
|
|
|
I've been planning a network expansion for a while now. For the next few weeks,
|
|
I will be in total geek-mode as I travel to several European cities to deploy
|
|
AS50869 on a european ring. At the same time, my buddy Fred from
|
|
[IP-Max](https://ip-max.net/) has been wanting to go to Amsterdam. IP-Max's
|
|
[network](https://as25091.peeringdb.com/) is considerably larger than mine, but
|
|
it just never clicked with the right set of circumstances for them to deploy
|
|
in the Netherlands, until the stars aligned ...
|
|
|
|
## Deployment
|
|
|
|
{{< image width="400px" float="left" src="/assets/network/qdr.png" alt="Quai du Rhône" >}}
|
|
|
|
After our adventure in [Frankfurt]({{< ref "2021-05-17-frankfurt" >}}),
|
|
[Amsterdam]({{< ref "2021-05-26-amsterdam" >}}), [Lille]({{< ref "2021-05-28-lille" >}}),
|
|
and [Paris]({{< ref "2021-06-01-paris" >}}) came to an end, I still had a few
|
|
loose ends to tie up. In particular, in Lille I had dropped an old Dell R610
|
|
while waiting for new Supermicros to be delivered. There is benefit to having
|
|
one standard footprint setup, in my case an PCEngines `APU2`, Supermicro
|
|
`5018D-FN8T` and Intel `X710-DA4` expansion NIC. They run fantastic with
|
|
[DANOS](https://danosproject.org/) and [VPP](https://fd.io/) applications.
|
|
|
|
Of course, we mustn't forget _home base_, Geneva, where IP-Max has its
|
|
headquarters in a beautiful mansion pictured here. At the same time, my
|
|
family likes to take one trip per month to a city we don't usually go, sort
|
|
of to keep up with real life as we are now more and more able to travel.
|
|
Marina has a niece in Geneva, who has lived and worked there for 20+ years,
|
|
so we figured we'd combine these things and stay the weekend at her place.
|
|
|
|
After making our way from Zurich to Geneva, a trip that took us just short
|
|
of six hours (!) by car, we arrived at the second half of the Belgium:Italy
|
|
eurocup soccer match. It was perhaps due to our tardiness and lack of physical
|
|
supportering, that the belgians lost the match that day. Sorry!
|
|
|
|
### Connectivity
|
|
|
|
{{< image width="400px" float="right" src="/assets/network/chplo0-rack.png" alt="Geneva Rack" >}}
|
|
|
|
My current circuit runs from Paris (Leon Frot), `frpar0.ipng.ch` over a direct
|
|
DWDM wave to Zurich where I pick it up on `chgtg0.ipng.ch` at Interxion
|
|
Glattbrugg. So what we'll do is break open this VLL at the IP-Max side,
|
|
insert the new router `chplo0.ipng.ch`, and reconfigure the Paris side
|
|
to go to the new router, and the new router to create another VLL back
|
|
to Zurich, which due to the toplogy of IP-Max's underlying DWDM network
|
|
will traverse Paris - Lyon - Geneva instead (shaving off ~1.5ms of latency
|
|
at the same time).
|
|
|
|
I hung up the `APU2` OOB server and the `5018D-FN8T` router, and another Dell
|
|
R610 to run virtual machines at Safehost SH1 in Plan-les-Ouates, a southern
|
|
suburb of Geneva. I connected one 10G port to `er01.gva20.ip-max.net` and
|
|
another 10G port to `er02.gva20.ip-max.net` to obtain maximum availability
|
|
benefits. As an example of what the configuration on the ASR9k platform looks
|
|
like for this type of operation, here's what I committed on `er01.gva20`.
|
|
|
|
Of course, first things first: let's ensure that the OOB machine has
|
|
connectivity, by allocating a /64 IPv6 and /29 IPv4. I usually configure
|
|
myself a BGP transit session in the same subnet, which means we'll want to
|
|
bridge the 1G UTP connection of the APU with the 10G fiber connection of
|
|
the Supermicro router, like so:
|
|
|
|
```
|
|
interface BVI911
|
|
description Cust: IPng OOB and Transit
|
|
ipv4 address 46.20.250.105 255.255.255.248
|
|
ipv4 unreachables disable
|
|
ipv6 nd suppress-ra
|
|
ipv6 address 2a02:2528:ff05::1/64
|
|
ipv6 enable
|
|
load-interval 30
|
|
!
|
|
interface GigabitEthernet0/7/0/38
|
|
description Cust: IPng APU (OOB)
|
|
mtu 9064
|
|
load-interval 30
|
|
l2transport
|
|
!
|
|
!
|
|
interface TenGigE0/1/0/3
|
|
description Cust: IPng (VLL and Transit)
|
|
mtu 9014
|
|
!
|
|
interface TenGigE0/1/0/3.911 l2transport
|
|
encapsulation dot1q 911 exact
|
|
rewrite ingress tag pop 1 symmetric
|
|
mtu 9018
|
|
!
|
|
|
|
l2vpn
|
|
bridge group BG_IPng
|
|
bridge-domain BD_IPng911
|
|
interface Te0/1/0/3.911
|
|
!
|
|
interface GigabitEthernet0/7/0/38
|
|
!
|
|
routed interface BVI911
|
|
!
|
|
!
|
|
!
|
|
```
|
|
|
|
After this, we pulled UTP cable and configured the `APU2`, which then has an
|
|
internal network towards the IPMI port of the Supermicro, and from there on,
|
|
the configuration becomes much easier. Of course, all config can be done
|
|
wirelessly, because the APU `console.plo.ipng.nl` acts as a WiFi access
|
|
point, so I connect to it and commit the network configs.
|
|
|
|
Once that's online and happy, the router `chplo0.ipng.ch` is next. For this,
|
|
on `er02.par02.ip-max.net`, I reconfigure the current VLL to point to the
|
|
loopback of this router `er01.gva20.ip-max.net` using the same `pw-id`. Then,
|
|
I can configure this router as follows:
|
|
|
|
```
|
|
interface TenGigE0/1/0/3.100 l2transport
|
|
description Cust: IPng VLL to par02
|
|
encapsulation dot1q 100
|
|
rewrite ingress tag pop 1 symmetric
|
|
mtu 9018
|
|
!
|
|
|
|
l2vpn
|
|
pw-class EOMPLS-PW-CLASS
|
|
encapsulation mpls
|
|
transport-mode ethernet
|
|
!
|
|
!
|
|
xconnect group IPng
|
|
p2p IPng_to_par02
|
|
interface TenGigE0/1/0/3.100
|
|
neighbor ipv4 46.20.255.33 pw-id 210535705
|
|
pw-class EOMPLS-PW-CLASS
|
|
!
|
|
!
|
|
!
|
|
```
|
|
|
|
## The results
|
|
|
|
And with that, the pseudowire is constructed, and the original interface on
|
|
`frpar0.ipng.ch` directly sees the interface here on `chplo0.ipng.ch` using
|
|
jumboframes of 9000 bytes (+14 bytes of ethernet overhead and +4 bytes of VLAN
|
|
tag on the ingress interface). It is as if the routers are directly connected
|
|
by a very long ethernet cable, a _pseudo-wire_ if you wish. Super low pingtimes
|
|
are observed between this new router in Geneva and the existing two in Paris
|
|
and Zurich:
|
|
|
|
```
|
|
pim@chplo0:~$ /bin/ping -4 -c5 frpar0
|
|
PING frpar0.ipng.ch (194.1.163.33) 56(84) bytes of data.
|
|
64 bytes from frpar0.ipng.ch (194.1.163.33): icmp_seq=1 ttl=64 time=8.78 ms
|
|
64 bytes from frpar0.ipng.ch (194.1.163.33): icmp_seq=2 ttl=64 time=8.80 ms
|
|
64 bytes from frpar0.ipng.ch (194.1.163.33): icmp_seq=3 ttl=64 time=8.81 ms
|
|
64 bytes from frpar0.ipng.ch (194.1.163.33): icmp_seq=4 ttl=64 time=8.82 ms
|
|
64 bytes from frpar0.ipng.ch (194.1.163.33): icmp_seq=5 ttl=64 time=8.85 ms
|
|
|
|
--- frpar0.ipng.ch ping statistics ---
|
|
5 packets transmitted, 5 received, 0% packet loss, time 10ms
|
|
rtt min/avg/max/mdev = 8.783/8.810/8.846/0.104 ms
|
|
pim@chplo0:~$ /bin/ping -6 -c5 chgtg0
|
|
PING chgtg0(chgtg0.ipng.ch (2001:678:d78::1)) 56 data bytes
|
|
64 bytes from chgtg0.ipng.ch (2001:678:d78::1): icmp_seq=1 ttl=64 time=4.51 ms
|
|
64 bytes from chgtg0.ipng.ch (2001:678:d78::1): icmp_seq=2 ttl=64 time=4.44 ms
|
|
64 bytes from chgtg0.ipng.ch (2001:678:d78::1): icmp_seq=3 ttl=64 time=4.36 ms
|
|
64 bytes from chgtg0.ipng.ch (2001:678:d78::1): icmp_seq=4 ttl=64 time=4.47 ms
|
|
64 bytes from chgtg0.ipng.ch (2001:678:d78::1): icmp_seq=5 ttl=64 time=4.41 ms
|
|
|
|
--- chgtg0 ping statistics ---
|
|
5 packets transmitted, 5 received, 0% packet loss, time 10ms
|
|
rtt min/avg/max/mdev = 4.362/4.436/4.506/0.077 ms
|
|
```
|
|
|
|
For good measure I've also connected to FreeIX, a new internet exchange project
|
|
I'm working on, that will span the Geneva, Zurich and Lugano areas. More on that
|
|
in a future post!
|
|
|
|
```
|
|
pim@chplo0:~$ iperf3 -4 -c 185.1.205.1 ## chgtg0.ipng.ch
|
|
Connecting to host 185.1.205.1, port 5201
|
|
[ 5] local 185.1.205.2 port 46872 connected to 185.1.205.1 port 5201
|
|
[ ID] Interval Transfer Bitrate Retr Cwnd
|
|
[ 5] 0.00-1.00 sec 809 MBytes 6.78 Gbits/sec 4 11.4 MBytes
|
|
[ 5] 1.00-2.00 sec 869 MBytes 7.29 Gbits/sec 0 11.4 MBytes
|
|
[ 5] 2.00-3.00 sec 865 MBytes 7.25 Gbits/sec 0 11.4 MBytes
|
|
[ 5] 3.00-4.00 sec 868 MBytes 7.28 Gbits/sec 0 11.4 MBytes
|
|
[ 5] 4.00-5.00 sec 836 MBytes 7.01 Gbits/sec 0 11.4 MBytes
|
|
[ 5] 5.00-6.00 sec 852 MBytes 7.15 Gbits/sec 0 11.4 MBytes
|
|
[ 5] 6.00-7.00 sec 865 MBytes 7.26 Gbits/sec 0 11.4 MBytes
|
|
[ 5] 7.00-8.00 sec 865 MBytes 7.26 Gbits/sec 0 11.4 MBytes
|
|
[ 5] 8.00-9.00 sec 861 MBytes 7.22 Gbits/sec 0 11.4 MBytes
|
|
[ 5] 9.00-10.00 sec 860 MBytes 7.22 Gbits/sec 0 11.4 MBytes
|
|
- - - - - - - - - - - - - - - - - - - - - - - - -
|
|
[ ID] Interval Transfer Bitrate Retr
|
|
[ 5] 0.00-10.00 sec 8.35 GBytes 7.17 Gbits/sec 4 sender
|
|
[ 5] 0.00-10.01 sec 8.35 GBytes 7.16 Gbits/sec receiver
|
|
|
|
iperf Done.
|
|
```
|
|
|
|
You kind of get used to performance stats like this, but that said, it's nice
|
|
to see that performance over FreeIX is slightly *lower* than performance over
|
|
the IPng backbone, and this is because on my VLLs, I can make use of jumbo
|
|
frames, which gives me 20% or so better performance (currently 9.62 Gbits/sec).
|
|
|
|
Currently I'm busy at work in the background completing the configuration, the
|
|
management environment and physical infrastructure for the internet exchange.
|
|
I'm planning to make a more complete post about the FreeIX project in a few
|
|
weeks once it's ready for launch. Stay tuned!
|