---
date: "2021-10-24T08:49:09Z"
title: IPng acquires AS8298
aliases:
- /s/articles/2021/10/24/as8298.html
---

# A Brief History

In January of 2003, my buddy Jeroen announced a project called the [Ghost Route Hunters](/assets/as8298/RIPE44-IPv6-GRH.pdf), after the industry had been plagued for a few years with anomalies in the DFZ - routes would show up with phantom BGP paths, unable to be traced down to a source or faulty implementation. Jeroen presented his [findings](/assets/as8298/RIPE46-IPv6-Routing-Table-Anomalies.pdf) at RIPE-46 and for years after this, the industry used the [SixXS GRH](https://www.sixxs.net/tools/grh/how/) as a distributed looking glass. At the time, one of SixXS's point of presence providers kindly lent the project AS8298 to build this looking glass and underlying infrastructure.

After running SixXS for 16 years, Jeroen and I decided to [Sunset]({{< ref "2017-03-01-sixxs-sunset" >}}) it, which meant that in June of 2017, the Ghost Route Hunter project came to an end as well, and as we tore down the infrastructure, AS8298 became dormant.

Then in August of 2021, I was doing a little bit of cleaning on the IPng Networks serving infrastructure, and came across some old mail from RIPE NCC about that AS number. And while IPng Networks is running [just fine]({{< ref "2021-02-27-network" >}}) on AS50869 today, it would be just that little bit cooler if it were to run on AS8298. So, I embarked on a journey to move a running ISP into a new AS number, which sounds like fun! This post describes the situation going in to this renumbering project, and there will be another post, likely in January 2022, that describes the retrospective (this future post may be either celebratory, or a huge postmortem, to be determined).

## The Plan

#### Step 0. Acquire the AS

First off, I had to actually acquire the AS number. Back in the days (I'm speaking of 2002), RIPE NCC was a little bit less formal than it is today. As such, our loaned AS number was simply _registered_ to SixXS, which is not a legal entity. Later, to do things properly, it was placed in Jeroen's custody (by creating ORG-SIXX1-RIPE). But precisely because Jeroen nor SixXS was an LIR at that time, the previous holder (Easynet, with LIR `de.easynet`, later acquired by Sky, also called British Sky Broadcasting, trading under LIR `uk.bskyb`) became the sponsoring LIR. So I had to arrange two things:
1.  A Transfer Agreement between Jeroen and myself; that signaled his willingness to transfer the AS number to me. This is [boilerplate](https://www.ripe.net/manage-ips-and-asns/resource-transfers-and-mergers/transfer-of-ip-addresses-and-as-numbers/transfer-agreement-template) stuff, and example contracts can be downloaded on the RIPE website.
1.  An agreement between Sky and IPng Networks; that signaled the transfer from sponsoring LIR to our LIR `ch.ipng`. This is rather non-bureaucratic and a well traveled path; sponsoring LIRs and movements of resource between holders happen all the time.

With the permission of the previous holder, and with the help of the previous sponsoring LIR, the transfer itself was a matter of filing the correct paperwork at RIPE NCC, quoting the transfer agreement, and providing identification for the offering party (Jeroen) and the receiving party (Pim). And within a matter of a few days, the AS number was transfered to `ORG-PVP9-RIPE`, the `ch.ipng` LIR.

**NOTE** - In case you're wondering, I registered the `ch.ipng` LIR a few months *before* I incorporated IPng Networks as a swiss limited liability company (called a `GmbH`, or *Gesellschaft mit beschr&auml;nkter Haftung* in German), so for now I'm still trading as my natural person. RIPE has a cooldown period of two years before new LIRs can acquire/merge/rename. I do expect that some time in 2023 the PeeringDB page and bgp.he.net and friends to drop my personal name and take my company name :) For now, trading as Pim will have to do, slightly more business risk, but just as much fun!

#### Step 1. Split AS50869 into two networks


{{< image width="300px" float="right" src="/assets/network/zurich-ring.png" alt="Zurich Metro" >}}

The autonomous system of IPng Networks spans two main parts. Firstly, in Zurich IPng Networks operates four sites and six routers:
*   Two in a private colocation site at Daedalean (**A**) in Albisrieden called `ddln0` and `ddln1`, they are running [DANOS](https://danosproject.org/)
*   Two at our offices in Br&uuml;ttisellen (**C**), called `chbtl0` and `chbtl1`, they are running [Debian](https://debian.org/)
*   One at Interxion ZUR1 datacenter in Glattbrugg (**D**), called `chgtg0`, running [VPP]({{< ref "2021-09-21-vpp-7" >}}), connecting to a public internet exchange CHIX-CH and taking transit from IP-Max and Openfactory.
*   One at NTT's datacenter in R&uuml;mlang (**E**), called `chrma0`, also running [VPP]({{< ref "2021-09-21-vpp-7" >}}), connecting to a public internet exchange SwissIX and taking transit from IP-Max and Meerfarbig.

NOTE: You can read a lot about my work on VPP in a series of [VPP articles](/s/articles/), please take a look!

There's a few downstream IP Transit networks and lots of local connected networks, such as in the DDLN colo. Then, from `chrma0`, we connect to our european ring northwards (towards Frankfurt), and from `chgtg0` we connect to our european ring south-westwards (towards Geneva).

{{< image width="300px" float="right" src="/assets/network/european-ring.png" alt="European Ring" >}}

That ring, then, consists of five additional sites and five routers, all running [VPP]({{< ref "2021-09-21-vpp-7" >}}):
*   Frankfurt: `defra0`, connecting to four DE-CIX exchangepoints in Frankfurt itself directly, and remotely to Munich, D&uuml;sseldorf and Hamburg
*   Amsterdam: `nlams0`, connecting to NL-IX, SpeedIX, FrysIX (our favorite!), and LSIX; we also pick up two transit providers (A2B and Coloclue).
*   Lille: `frggh0`, connecting to the northern france exchange called LillIX
*   Paris: `frpar0`, connecting to two FranceIX exchange points, directly in Paris, and remotely to Marseille
*   Geneva: `chplo0`, connecting to our very own Free IX

Every one of these sites has an upstream session with AS25091 (IP-Max). Considering these folks are organizationally very close to me, it is easy for me to rejigger any one of those sessions between AS50869 (current) and AS8298 (new). And considering AS8298 has been a member of our as-set `AS-IPNG` for a while, it'll also be a natural propagation to rely on IP-Max, even if some peering sessions might be down.

So before I start, IPng Networks' view from AS25091 in Amsterdam looks like this:
```
   Network            Next Hop            Metric LocPrf Weight Path
*  92.119.38.0/24     46.20.242.210                          0 50869 i
*  176.119.215.0/24   46.20.242.210                          0 50869 60557 i
*  185.36.229.0/24    46.20.242.210                          0 50869 212855 i
*  185.173.128.0/24   46.20.242.210                          0 50869 57777 i
*  185.209.12.0/24    46.20.242.210                          0 50869 212323 i
*  192.31.196.0/24    46.20.242.210                          0 50869 112 i
*  192.175.48.0/24    46.20.242.210                          0 50869 112 i
*  194.1.163.0/24     46.20.242.210                          0 50869 i
*  194.126.235.0/24   46.20.242.210                          0 50869 i

   Network            Next Hop            Metric LocPrf Weight Path
*  2001:4:112::/48    2a02:2528:1902::210                    0 50869 112 i
*  2001:678:3d4::/48  2a02:2528:1902::210                    0 50869 201723 i
*  2001:678:ce4::/48  2a02:2528:1902::210                    0 50869 60557 i
*  2001:678:ce8::/48  2a02:2528:1902::210                    0 50869 60557 i
*  2001:678:cec::/48  2a02:2528:1902::210                    0 50869 60557 i
*  2001:678:cf0::/48  2a02:2528:1902::210                    0 50869 60557 i
*  2001:678:d78::/48  2a02:2528:1902::210                    0 50869 i
*  2001:67c:6bc::/48  2a02:2528:1902::210                    0 50869 201723 i
*  2620:4f:8000::/48  2a02:2528:1902::210                    0 50869 112 i
*  2a07:cd40::/29     2a02:2528:1902::210                    0 50869 212855 i
*  2a0b:dd80::/29     2a02:2528:1902::210                    0 50869 i
*  2a0b:dd80::/32     2a02:2528:1902::210                    0 50869 i
*  2a0d:8d06::/32     2a02:2528:1902::210                    0 50869 60557 i
*  2a0e:fd40:200::/48 2a02:2528:1902::210                    0 50869 60557 i
*  2a0e:fd45:da0::/48 2a02:2528:1902::210                    0 50869 60557 i
*  2a10:d200::/29     2a02:2528:1902::210                    0 50869 212323 i
*  2a10:fc40::/29     2a02:2528:1902::210                    0 50869 57777 i
```

#### Step 2. Restrict the routers that originate our prefixes

As a preparation to actually starting to use AS8298, I'll create RPKI records of authorization for all of our prefixes in **both** AS50869 and AS8298, and I'll add `route:` objects for all of them in both as well.

Now, I'm ready to make my first networking topology change: instead of originating our prefixes in _all_ routers, I will originate our prefixes in AS50869 only from _two_ routers: `chbtl0` and `chbtl1`. Nobody on the internet will notice this change, the as-path will remain `^50869$` for all prefixes.

I'll also prepare the two routers to speak to eachother with an iBGP session (rather than to IPng Networks' route-reflectors, which are still in AS50869).

#### Step 3. Convert Br&uuml;ttisellen to AS8298

Now, I'll switch these routers `chbtl0` and `chbtl1` out of AS50869 and into AS8298. The only damage at this point might be that my personal Spotify and Netflix stop working, and my family yells at me (but they do that all the time anyway, so it's a wash...). If things go poorly, the backout plan is to switch back to AS50869 and return things to normal. But if things go well, from this point onwards everybody will see our own IPng Networks prefixes via as-path `^50869_8298$` and effectively, AS50869 will become a transit provider for AS8298, which will be singlehomed for the moment.

My buddy Max runs a small /29 and /64 exchangepoint in Br&uuml;ttisellen with only three members on it - IPng Networks, Stucchinet and Openfactory. I will ask both of them to be my canary, and change their peering session from AS50869 to AS8298. If things go bad, that's no worries, I can drop/disable these peering sessions as I have sessions with both as well in other places. But it'd be a good place to see and test if things work as expected.

#### Step 4. Convert DDLN to AS8298

Now that our IPv4 and IPv6 prefixes have moved and AS50869 does not originate prefixes anymore, you may wonder 'what about the colo?'. Indeed, the colo runs at Daedalean behind `ddln0` and `ddln1`, both of which are still in AS50869. However, the only way to ever be able to reach those prefixes, would be to find an entrypoint in AS50869 (as it is the only uplink of AS8298). All routers in AS50869 and AS8298 share an underlying OSPF and OSPFv3 interior gateway protocol, which means that if anything is destined to `194.1.163.0/24` or `2001:678:d78::/48`, those packets will find their way to the correct location using the IGP. That's neat, because it means that even though `ddln0` is speaking BGP in AS50869, it will happily forward traffic to its more specific prefixes from AS8298.

Considering there's only one IP transit network in DDLN, those two routers will be the first for me to convert. After converting them to AS8298, they will receive transit from AS50869 just like the ones in Br&uuml;ttisellen. I'll rejigger Jeroen's AS57777 to receive transit directly from AS8298, and it will then be the first to transition. Jeroen's prefixes will briefly become `^50869_8298_57777$`, which will be the only change, but this will validate that, indeed, AS8298 can provide transit. Apart from the longer as-path, the physical path that IP packets take will remain the same because Jeroen’s network is currently *cough* singlehomed at DDLN.

Now that I have four routers in AS8298, I'll add a iBGP sessions directly only amongst the pairs, rather than in a full mesh on these routers. I've now created _two islands_ of AS8298, interconnected by AS50869. You'd think this is bad, but it's actually a fine situation to be in and there will be no loss of service, because:
*   we've already established that AS50869 has reachability to all more-specifics in AS8298 (Step 3)
*   AS50869 has reachability to AS57777 via its downstream AS8298
*   AS50869 is the only way in and out of the DDLN or Br&uuml;ttisellen routers

Nevertheless, I'd rather swiftly continue with the next step, and reconnect these two islands. It's is a good time for me to give a headsup to the larger internet exchanges (notably SwissIX) so folks can prepare for what's coming. I think many NOC teams know how to establish/remove a peering session, but I do expect the ITIL-automated teams to not have a playbook for "peer on IP 192.0.2.1 has changed from AS666 to AS42". I'll observe their performance on this task and take notes, as there's quite a few public IXPs to go...

#### Step 5. Convert R&uuml;mlang and Glattbrugg to AS8298

After four of our routers (two redundant pairs) have been operating in AS8298 for a few days, I'm ready to renumber our first machine connected to a public internet exchange: `chgtg0` is connected to [CHIX-CH](https://chix.ch/). I'll contact the members with whom IPng Networks has a direct peering, and of course the internet exchange folks, to ask them to renumber our AS50869 to AS8298. After restarting our router into the new AS, one by one I'll establish sessions with our peers there - this is an important exercise because I'll be doing this later in Step 5 on every peering/border router in the european ring, and there will be in total ~1'850 BGP adjacencies that have to be renumbered.

At CHIX-CH, IPng has in total four BGP sessions with the two routeservers in AS212100, and in total 38 direct BGP sessions; two of which are somewhat important: AS13030 (Init7) and AS15169 (Google), to which a large fraction of our traffic flows. While upgrading this router, I'll also switch my one downstream network (Daedalean itself, operating AS212323) to receive transit from AS8298. Because I've canaried this with Jeroen's AS57777 in the DDLN colo previously, I'll be reasonably certain at this point that it'll work well. If not, they have alternative uplinks (notably AS174), so they should be fine without me.

At SwissIX, IPng has as well four BGP sessions with the two routeservers in AS42476, and in total 132 direct BGP sessions. I think that, once these two peering routers are complete, I'll checkpoint and let things run like this for a while. Let's take a few weeks off, giving me a while to hunt down peers at SwissIX and CH-IX to catch up and re-establish their sessions with the new AS8298 :)

After this step, phase one of the transition is complete, and AS8298 (and its networks AS57777 and AS212323) will be directly visible in Switzerland, and still tucked away behind `^50869_8298_212323$` for the international traffic. I will however have 2 transit sessions from IP-Max (AS25091), 2 transit sessions from Openfactory (AS58299) and 1 transit session from Meerfarbig (AS34549); so it is expected to be a quite stable network at this point, which is good.

#### Step 6. Convert European Ring to AS8298

For this task, I'll swap my iBGP sessions to all use the new AS8298. I do this by first dismantling `rr0.chbtl0.ipng.ch` and bring it back up as an iBGP speaker in the new AS; then one by one push all routers to speak to that route reflector (in addition to the existing route reflectors in AS50869). After that stabilizes, rinse and repeat with `rr0.chplo0.ipng.ch`; and finally finish the job with `rr0.frggh0.ipng.ch`. The two pairs of routers who were by themselves in AS8298 (`chbtl0`/`chbtl1` in Br&uuml;ttisellen; and `ddln0`/`ddln1` at Daedalean) can now be reattached to the iBGP route-reflectors as well.

It will be a bit of a schlepp, but now comes the notification of all international peers (there are an additional ~250 direct peerings) and downstreams (there are three left: Raymon, Jelle and Krik), and upstreams (there are two additional ones: Coloclue, and A2B). While normally this is a matter of merely swapping the AS number, it has to be done on both sides - on my side, I can do it with a one-line change to the git repository, and it'll be pushed by Kees (the network build and push automation that was inspired by Coloclue's [Kees](https://github.com/coloclue/kees/)), on the remote side it will be a matter of patience. One by one, folks will (or.. won't) update their peering session. The only folks I'll actively chase is the DE-CIX, FranceIX, NL-IX, LSIX and FrysIX routeserver operators, as the vast majority of adjacencies are learned via those. By means of a headsup one week in advance, and a few reminders on the day of, and the day after maintenance, I should minimize downtime. But, in this case, because I already have two transit providers in Switzerland (AS25091 IP-Max, and AS58299 Openfactory) who provide me full tables in AS8298, it should be operationally smooth sailing.
At the end of this exercise, the as-path will be `^8298$` for my own prefixes and `^8298_.*$` for my downstream networks, and AS50869 will no longer be in any as-path in the DFZ. Rolling back is tricky, although Bird can do individual peering sessions with differing AS numbers, I don't think this is a good idea; as it'll mean many (many) changes into the repository; so in the interest of simplicity and don't break things that work, I will do them router-by-router rather than session-by-session; and send a few reminders to folks to update their records to match my peeringdb entries.

#### Step 7. Repurpose/retire AS50869

There's really not that much to do -- delete the `route:` and `route6:` objects and remove RPKI ROAs for the old announcments; but mostly it'll be a matter of hunting down peering partners who have not (yet) updated their records and sessions. I imagine lots of folks will hesitate and be unfamiliar with this type of operation (even though it literally is an `s/50869/8298/g` for them). I'll take most of December to remind folks a few times, and ultimately just clean up broken peering sessions in January 2022.

And of course, then lick my wounds and count pokemon - on October 24th, the day this post was published, Hurricane Electric showed 1'845 adjacencies in total for AS50869, of which 1'653 IPv4 and 1'430 IPv6. I will consider it a success if I lose less than 200 adjacencies. I'll keep AS50869 around, as a test AS number to do a few experiments.

## The Timeline

Most all intrusive maintenances will be done in maintenance windows between 22:00 - 03:00 UTC from Thursday to Friday. The tentative planning for the project, which starts on October 22nd and lasts through end of year (10 weeks):

1.   2021-10-22 - RIPE updates for `route:`, `route6:` and RPKI ROAs
1.   2021-10-24 - Originate prefixes from chbtl0/chbtl1 (no-op for the DFZ)
1.   2021-10-28 - Move chbtl0/chbtl1 at Br&uuml;ttisellen to AS8298
1.   2021-10-29 - Update Br&uuml;ttisellen IXP (AS58299 and AS58280), canary upstream AS58299
1.   2021-10-29 - Headsup to CHIX-CH and SwissIX peers and Transit partners, announcing move
1.   2021-11-04 - Move ddln0/ddln1 at Daedalean to AS8298, canary downstream AS57777
1.   2021-11-11 - Move chgtg0 Glattbrugg to AS8298, add downstream AS212323
1.   2021-11-12 - Move CHIX-CH peers to AS8298
1.   2021-11-15 - Move chrma0 R&uuml;mlang to AS8298
1.   2021-11-16 - Move SwissIX peers to AS8298
1.   Two week cooldown period, start to move IXPs to AS8298
1.   2021-11-29 - Headsup to all european IXPs and IP Transit partners, announcing move
1.   2021-12-02 - Move defra0, nlams0, frggh0, frpar0, chplo0 to AS8298
1.   2021-12    - Move european IXPs to AS8298
1.   2021-12-06 - First reminder for peerings that did not re-establish
1.   2021-12-13 - Second reminder for peerings that did not re-establish
1.   2021-12-20 - Third (final) reminder for peerings that did not re-establish
1.   2022-01-10 - Remove peerings that did not re-establish

## Appendix

This blogpost turned into a talk at [RIPE #83](/assets/as8298/RIPE83-IPngNetworks.pdf), in case you wanted to take a look at [the recording](https://ripe83.ripe.net/archives/video/649/) and [a few questions](https://ripe83.ripe.net/archives/video/650/).
