---
date: "2022-02-24T13:46:12Z"
title: IPng Networks - Colocation
aliases:
- /s/articles/2022/02/24/colo.html
---

## Introduction

As with most companies, it started with an opportunity. I got my hands on a location which has a
raised floor at 60m2 and a significant power connection of 3x200A, and a metro fiber connection at
10Gbps. I asked my buddy Luuk 'what would it take to turn this into a colo?' and the rest is history.
Thanks to Daedalean AG who benefit from this infrastructure as well, making this first small colocation
site was not only interesting, but also very rewarding.

The colocation business is murder in Zurich - there are several very large datacenters (Equinix, NTT,
Coloz&uuml;ri, Interxion) all directly in or around the city, and I'm known to dwell in most of these. The
networking and service provider industry is quite small and well organized into _Network Operator Groups_,
so I work under the assumption that everybody knows everybody. I definitely like to pitch in and share
what I have built, both the physical bits but also the narrative.

This article describes the small serverroom I built at a partner's premises in Zurich Albisrieden. The
colo is _open for business_, that is to say: Please feel free to [reach out](/s/contact) if you're interested.

## Physical

{{< image width="180px" float="right" src="/assets/colo/power.png" alt="Power" >}}

It starts with a competent power distribution. Pictured to the right is a 200Amp 3-phase distribution panel
at Daedalean AG in Zurich. There's another similar panel on the other side of the floor, and both are
directly connected to EWZ and have plenty of smaller and larger breakers available (the room it's in used
to be a serverroom of the previous tenant, the City of Zurich).

{{< image width="180px" float="left" src="/assets/colo/eastron-sdm630.png" alt="Eastron SDM630" >}}

I start with installing a set of Eastron SDM630 power meters, so that I know what is being used
by IPng Networks, and can pay my dues, as well as remotely read the state and power consumption using
MODBUS, yielding two 3-phase supplies with 32A breakers on each.

{{< image width="180px" float="left" src="/assets/colo/pdus.png" alt="PDUs" >}}

Then, I go scouring on the Internet, to find a few second hand 19" racks. I actually find two 800x1000mm racks
but they are all the way across Switzerland. However, they're very affordable, but what's better, they each come
with two APC power distribution and remotely switchable zero-u power distribution strips. Score!

<hr />

{{< image width="180px" float="right" src="/assets/colo/racks-installed1.png" alt="Racks Installed" >}}

Laura and I rented a little (with which I mean: huge) minivan and went to pick up the racks. The folks at
Daedalean kindly helped us schlepp them up the stairs to the serverroom, and we installed the racks in the
serverroom, connecting them redundantly to power using the four PDUs. I have to be honest: there is no battery
or diesel backup in this room, as it's in the middle of the city and it'd be weird to have generators on site
for such a small room. It's a compromise we have to make.

{{< image width="180px" float="left" src="/assets/colo/racks-installed2.png" alt="Racks Installed w/ doors" >}}

Of course, I have to supply some form of eye-candy, so I decide to make a few decals for the racks, so that they
sport the _IPng @ DDLN_ designation. There are a few other racks and infrastructure in the same room, of course,
and it's cool to be able to identify IPng's kit upon entering the room. They even have doors, look!

The floor space here is about 60m2 of usable serverroom, so there is plenty of room to grow, and if the network
ever grows larger than 2x10G uplinks, it is definitely possible to rent dark fiber from this location thanks to
the liberal Swiss telco situation. But for now, we start small with 1x 10G layer2 backhaul to Interxion in
Glattbrugg. In 2022, I expect to expand with a second 10G layer2 backhaul to NTT in R&uuml;mlang to make the site
fully redundant.

<!-- ![Racks Pre-Installed](/assets/colo/racks-preinstall.png){: style="width:180px; float: right; margin-left: 2em; margin-bottom: 1em;"} -->

<!-- ![Storage](/assets/colo/storage.png){: style="width:180px; float: right; margin-left: 2em; margin-bottom: 1em;"} -->

## Logical

The physical situation is sorted, we have cooling, power, 19" racks with PDUs, and uplink connectivity. It's time
to think about a simple yet redundant colocation setup:

{{< image width="800px" src="/assets/colo/DDLN Logical Sketch.png" alt="Design" >}}

In this design, I'm keeping it relatively straight forward. The 10G ethernet leased line from Solnet plugs into one
switch, and the 10G leased line from Init7 plugs into the other. Everything is then built in pairs.
I bring:
* Two switches (Mikrotik CRS354, with 48x1G, 4x10G and 2x40G), two power supplies, connect them with 40G together.
* Two Dell R630 routers running VPP (of course), two power supplies, with 3x10G each:
  * One leg goes back-to-back for OSPF/OSPFv3 between the two routers
  * One leg goes to each switch; the "local" leg will be in a VLAN into the uplink VLL, and expose the router on the
    colocation VLAN and any L2 backhaul services. The "remote" leg will be in a VLAN to the other uplink VLL.
* Two Supermicro hypervisors, each connected with 10G to their own switch
* Two PCEngines APU4 machines, each connected to Daedalean's corporate network for OOB
  * These have serial connection to the PDUs and Mikrotik switches
  * They also have mgmt network connection to the Dell VPP routers and Mikrotik switches
  * They also run a Wireguard access service which exposes an IPMI VLAN for colo clusters

The result is that each of these can fail without disturbing traffic to/from the servers in the colocation. Each
server in the colo gets two power connections (one on each feed), two 1Gbps ports (one for IPMI and one for Internet).

The logical colocation network has VRRP configured for direct/live failover of IPv4 and IPv6 gateways, but the VPP
routers can offer full redundant IPv4 and IPv6 transit, as well as L2 backhaul to any other location where IPng
Networks has a presence (which is [quite a few](https://as8298.peeringdb.com/)).

## Conclusion

The colocation that I built, together with Daedalean, is very special. It's not carrier grade, it doesn't have
a building/room wide UPS or diesel generators, but it does have competent power, cooling, physical and logical
deployment. But most of all: it redundantly connects to AS8298 and offers full N+1 redundancy on the logical
level.

If you're interested in hosting a server in this colocation, [contact us](/s/contact/)!
