ipng.ch/content/articles/2021-02-27-network.md

---
date: "2021-02-27T23:46:12Z"
title: IPng Network
aliases:
- /s/articles/2021/02/27/network.html
---

# Introduction to IPng Networks

At IPng Networks, we run a modest network with European reach. With our home base
in Zurich, Switzerland, we are pretty well connected into the Swiss internet scene.
We operate four sites in Zurich, and an additional set of sites in European cities,
each of which are described on this post. If you're curious as to how the network
runs, you can find two main pieces here: Firstly, the physical parts, where exactly
are IPng's routers and switches, what types of kit does the ISP use, and so on.
Secondly, the logical parts, what operating systems and configurations are in use.

## Physical

### Zurich Metropolitan Area

The Canton of Zurich, Switzerland is our home-base, and it's where IPng
Networks GmbH is registered. The local commercial datacenter scene is dominated
by Interxion, NTT and Equinix. The small town of Brüttisellen (zipcode
CH-8306), is where our founder lives and, due to the ongoing Corona pandemic,
where he works from home.

{{< image width="400px" float="left" src="/assets/network/zurich-ring.png" alt="Zurich Metro" >}}

In Br&uuml;ttisellen, marked with **C**, we have our first two routers,
`chbtl0.ipng.ch` and `chbtl1.ipng.ch`, racked in our office. There are only two
fiber operators in this town - UPC and Swisscom. The orange trace (**C** to **D**)
is a leased line from UPC, which we rent from [Openfactory](https://openfactory.net/)
and it gets terminated at Interxion Glattbrugg, where our first router
called `chgtg0.ipng.ch` is located. From there, Openfactory rents darkfiber
to multiple locations - but notably the dark purple trace (**D** to **E**)
that connects from Interxion Glattbrugg to NTT R&uuml;mlang, where our second
router called `chrma0.ipng.ch` is located.

We rent a 10G CWDM wave between these two datacenters, directly connecting these
two routers. Now, Equinix also has a sizable footprint in Z&uuml;rich, and
operating ZH04 (**B** where we only have passive optical presence) in the
Industriekwartier (our local internet exchange [SwissIX](https://swissix.net/)
 was born in the now defunct Equinix ZH01 office building). From the neighboring
building Equinix ZH04, our partner [IP-Max](https://ip-max.net/) rents dark fiber
to Equinix ZH05 in the Zurich Allmend area (the light purple trace **B** to **F**),
and from there, IP-Max rents dark fiber to NTT R&uuml;mlang again (**F** to **E**),
completing the ring. We rent a 10G circuit on that path, to redundantly connect our
routers `chgtg0` and `chrma0`.  If at any time we'd need to connect partners
or customers, we can do so at a moment's notice, as rackspace is available in
all Equinix sites for IPng Networks.

The green link (**D** to **B**) is a 10G carrier ethernet circuit between Interxion,
over the light purple path (**B** to **A**) on its last mile to Albisrieden, where
we built a very small colocation site, which you can read about in more detail in our
[informational post]({{< ref "2022-02-24-colo" >}}) - the colo is open for private
individuals and small businesses ([contact](/s/contact/) us for details!).

### European Ring

At IPng, we are strong believers in a free and open Internet. Having seen
the shakeout of internet backbone providers over the last two decades, it
seems to be a race to the bottom, with mergers, acquisitions and takeovers
of datacenters and network carriers. Prices are going lower, and small fish
traffic (let's be honest, IPng Networks is definitely a small provider), to
the point that purchasing IP transit is cheaper than connecting to local
Internet exchange points. We've decided specifically to go the extra mile,
quite literally, and plot a path to several continental european internet
hubs.

{{< image width="400px" float="left" src="/assets/network/european-ring.png" alt="European Ring" >}}

***Frankfurt*** - Connected from NTT's datacenter at R&uuml;mlang (Zurich) with
a first 10G circuit, and from Interxion's datacenter at Glattbrugg (Zurich)
with a second 10G circuit, this is our first hop into the world. Here, we
connect to [DE-CIX](https://de-cix.net/) from Equinix FR5 at the Kleyerstrasse.
More details in our post [IPng Arrives in Frankfurt]({{< ref "2021-05-17-frankfurt" >}}).

***Amsterdam*** - The Amsterdam Science Park is where European Internet was born.
[NIKHEF](https://www.nikhef.nl/) is where we rent rackspace that connects with a 10G
circuit to Frankfurt, and a 10G circuit onwards towards Lille. We connect to
[Speed-IX](https://speed-ix.net/), [LSIX](https://lsix.net/), [NL-IX](https://nl-ix.net),
and an exchange point we help run called [FrysIX](https://www.frys-ix.net/).
More details in our post [IPng Arrives in Amsterdam]({{< ref "2021-05-26-amsterdam" >}}).

***Lille*** - [IP-Max](https://ip-max.net/) does lots of business in this
region, with presence in both local datacenters here, one in Lille and one in
Anzin. IPng has a point of presence here too, at the [CIV1](https://www.civ.fr/)
facility, with a northbound 10G circuit to Amsterdam, and a southbound 10G
circuit to Paris. Here, we connect to [LillIX](https://lillix.fr/).
More details in our post [IPng Arrives in Lille]({{< ref "2021-05-28-lille" >}}).

***Paris*** - Where two large facilities are placed back-to-back in the middle
of the city, originally Telehouse TH2, with a new facility at L&eacute;on Frot,
where we pick up a 10G circuit from Lille and further on the ring with a 10G
circuit to Geneva. Here, we connect to [FranceIX](https://franceix.net).
More details in our post [IPng Arrives in Paris]({{< ref "2021-06-01-paris" >}}).

***Geneva*** - The home-base of [IP-Max](https://ip-max.net) is where we close
our ring. From Paris, IP-Max has two redundant paths back to Switzerland, the first
being a DWDM link from to Zurich, and the second being a DWDM link to Lyon and
then into Geneva. Here, at [SafeHost](https://safehost.com/) in Plan les Ouates,
is where we have our fourth Swiss point of presence, with a connection to our very
own [Free-IX](https://free-ix.net/) and a 10G circuit to Interxion at Glattbrugg
(Zurich), and of course to Paris.
More details in our post [IPng Arrives in Geneva]({{< ref "2021-07-03-geneva" >}}).

## Logical

As a small operator, we'd love to be able to boast the newest Juniper [PTX10016](https://www.juniper.net/us/en/products/routers/ptx-series.html)
routers but we neither have the rack space, the power budget, and to be
perfectly honest, the monetary budget to run these at IPng Networks. But it
turns out, we know a fair bit about hardware silicon, architecture and the
controlplane software running on commercial routers.

We've decided to go a different route. In our opinion, at speeds under 100Gbit,
it's perfectly viable to use software routers on off-the-shelf hardware, notably
Intel network cards and CPUs, notably those that have support for the
[Dataplane Development Kit](https://dpdk.org/) (aka DPDK), which offers libraries
to accelerate packet processing workloads, which turn ordinary servers into very
performant routers. Two notable applications are [VPP](https://fd.io/) and
[Danos](https://danosproject.org).

### VPP

VPP originally comes from the house of Cisco [[ref](https://www.cisco.com/c/dam/m/en_us/service-provider/ciscoknowledgenetwork/files/592_05_25-16-fdio_is_the_future_of_software_dataplanes-v2.pdf)] and looks quite a bit like
the commercial ASR9k platform. In development since 2002, VPP is production
code currently running in shipping products. It runs in user space on multiple
architectures including x86, ARM, and Power architectures on both x86 servers
and embedded devices. The design of VPP is hardware, kernel, and deployment
(bare metal, VM, container) agnostic. It runs completely in userspace.

We've contributed a little bit to the Control Plane abstraction [[ref](https://docs.fd.io/vpp/21.06/dc/d2e/clicmd_src_plugins_linux-cp.html)],
which allows users to combine the throughput of a dataplane with usual routing
software like [Bird](https://bird.network.cz/) or [FRR](https://frrouting.org/).
We've been running it in production since December 2020 on `chbtl1.ipng.ch`.
It's our ultimate goal to run VPP and Linux Control Plane on the entire network,
as the design and architecture really resonates with us as software and systems
engineers.

### DANOS

The Disaggregated Network Operating System (DANOS) project originally comes
from AT&T’s “dNOS” software framework and provides an open, cost-effective and
flexible alternative to traditional networking equipment. As part of The Linux
Foundation, it now incorporates contributions from complementary open source
communities in building a standardized distributed Network Operating System (NOS)
to speed the adoption and use of white boxes in a service provider’s
infrastructure.

We've been using DANOS since its first release in August 2019, and it's
currently our routing platform of choice -- it combines the sheer speed of
DPDK with a [Vyatta](https://en.wikipedia.org/wiki/Vyatta) command line
interface. As an appliance, care was taken to complete the _whole package_,
with SNMP, YANG interface, image and upgrade management, interface monitoring
with wireshark semantics, et cetera. Performing easily at wire speed 10G
workloads (including 64byte ethernet frames), and being completely open source,
it fits very well with our philosophy of an open and free internet.