217 lines
13 KiB
Markdown
217 lines
13 KiB
Markdown
---
|
|
date: "2021-08-26T12:55:44Z"
|
|
title: Fiber7-X in 1790BRE
|
|
aliases:
|
|
- /s/articles/2021/08/28/fiber7-x.html
|
|
---
|
|
|
|
## Introduction
|
|
|
|
I've been a very happy Init7 customer since 2016, when the fiber to the home
|
|
ISP I was a subscriber at back then, a small company called Easyzone, got
|
|
acquired by Init7. The technical situation in Wangen-Brüttisellen was
|
|
a bit different back in 2016. There was a switch provided by Litecom in which
|
|
ports were resold OEM to upstream ISPs, and Litecom would provide the L2
|
|
backhaul to a central place to hand off the customers to the ISPs, in my case
|
|
Easyzone. In Oct'16, Fredy asked me if I could do a test of
|
|
Fiber7-on-Litecom, which I did and reported on in a [blog post]({{< ref "2016-10-07-fiber7-litexchange" >}}).
|
|
|
|
Some time early 2017, Init7 deployed a POP in Dietlikon (790BRE) and then
|
|
magically another one in Brüttisellen (1790BRE). It's a funny story
|
|
why the Dietlikon point of presence is called 790BRE, but I'll leave that
|
|
for the bar, not this post :-)
|
|
|
|
## Fiber7's Next Gen
|
|
|
|
Some of us read a rather curious tweet in back in May:
|
|
|
|
{{< image width="400px" float="center" src="/assets/fiber7-x/tweet.png" alt="Tweet-X2" >}}
|
|
|
|
Translated -- _'7 years ago our Gigabit-Internet was born. To celebrate this day,
|
|
here's a riddle for #Nerds: Gordon Moore's law says dictates doubling every 18 months.
|
|
What does that mean for our 7 year old Fiber7?'_ Well, 7 years is 84 months, and
|
|
doubling every 18 months means 84/18 = 4.6667 doublings and 1Gbpbs * 2^4.6667 =
|
|
25.4Gbps. Holy shitballs, Init7 just announced that their new platform will offer 25G
|
|
symmetric ethernet?!
|
|
|
|
"I wonder what that will cost?", I remember myself thinking. "**The same price**",
|
|
was the answer. I can see why -- monitoring my own family's use, we're doing a good
|
|
60Mbit or so when we stream Netflix and/or Spotify (which we all do daily). And some
|
|
IPTV maybe at 4k will go for a few hundred megs, but the only time we actually use
|
|
the gigabit, is when we do a speedtest of an iperf :-) Moreover, offering 25G fits
|
|
the company's marketing strategy well, because our larger Swiss national telco and
|
|
cable providers are all muddying the waters with their DOCSIS and GPON offering,
|
|
both of which _can_ do 10Gbit, but it's a TDM (time division multiplexing) offering
|
|
which makes any number of subscribers share that bandwidth to a central office. And
|
|
when I say any number, it's easy to imagine 128 and 256 subscribers on one XGSPON,
|
|
and many of those transponders in a telco line terminator, each with redundant uplinks
|
|
of 2x10G or sometimes 2x40G. But that's an oversubscription of easily 2000x, taking
|
|
128 (subscribers per PON) x16 (PONs per linecard) x8 (linecards), is 16K subscribers
|
|
of 10G using 80G (or only 20G) of uplink bandwidth. That's massively inferior from
|
|
a technical perspective. And, as we'll see below, it doesn't really allow for
|
|
advanced services, like L2 backhaul from the subscriber to a central office.
|
|
|
|
Now to be fair, the 1790BRE pop that I am personally connected to has 2x 10G uplinks
|
|
and ~200 or so 1G downlinks, which is also a local overbooking of 10:1, or 20:1 if only
|
|
one of the uplinks is used at any given time. Worth noting, sometimes several cities
|
|
are daisy chained, which makes for larger overbooking if you're deep in the Fiber7
|
|
access network. I am pretty close (790BRE-790SCW-790OER-Core; and an alternate path of
|
|
780EFF-Core; only one of which is used because the Fiber7 edge switches use OSPF and
|
|
a limited TCAM space means only few if any public routes are there; I assume a default
|
|
is injected into OSPF at every core site and limited traffic engineering is done).
|
|
The longer the traceroute, the cooler it looks, but the more customers are ahead of
|
|
you, causing more overbooking. YMMV ;-)
|
|
|
|
## Upgrading 1790BRE
|
|
|
|
{{< image width="300px" float="right" src="/assets/fiber7-x/before.png" alt="Before" >}}
|
|
|
|
Wouldn't it be cool if Init7 upgraded to 100G intra-pop? Well, this is the story
|
|
of their Access'21 project! My buddy Pascal (who is now the CTO at Init7, good
|
|
choice!), explained it to me in a business call back in June, but also shared it
|
|
in a [presentation](https://indico.uknof.org.uk/event/57/timetable/?view=standard) which I definitely encourage
|
|
you to browse through. If you thought I was jaded on GPON, check out their assessment,
|
|
it's totally next level!
|
|
|
|
Anyway, the new POPs are based on Cisco's C9500 switches, which come in two variants:
|
|
Access switches are C9500-48Y4C which take 48x SPF28 (1/10/25Gbit) and 4x QSFP+ (40/100Gbit)
|
|
and aggregation switches are C9500-32C which take 32x QSFP+ (40/100Gbit).
|
|
|
|
As a subscriber, we all got a courtesy headsup on the date of 1790BRE's upgrade.
|
|
It was [scheduled](https://as13030.net/status/?ticket=4238550) for Thursday Aug 26th
|
|
starting at midnight. As I've written about before (for example at the bottom of my
|
|
[Bucketlist post]({{< ref "2021-07-26-bucketlist" >}})), I really enjoy the immediate
|
|
gratification of physical labor in a datacenter. Most of my projects at work are on the
|
|
quarters-to-years timeframe, and being able to do a thing and see the result of that
|
|
thing ~immmediately, is a huge boost for me.
|
|
|
|
So I offered to let one of the two Init7 people take the night off and help perform the
|
|
upgrade myself. The picture on the right is how the switch looked like until now, with
|
|
four linecards of 48x1G trunked into 2x10G uplinks, one towards Effretikon and one
|
|
towards Dietlikon. It's an aging Cisco 4510 switch (they were released around 2010),
|
|
but it has served us well here in Brüttisellen for many years, thank you, little
|
|
chassis!
|
|
|
|
## The Upgrade
|
|
|
|
{{< image width="300px" float="right" src="/assets/fiber7-x/during.png" alt="During" >}}
|
|
|
|
I met the Init7 engineer in front of the Werke Wangen-Brüttisellen, which is about
|
|
170m from my house, as the photons fly, at around 23:30. We chatted for a little while,
|
|
I had already gotten to know him due to mutual hosting at NTT in Rümlang, so of
|
|
course our basement ISPs peer over [CommunityIX](https://communityix.ch/) and so on,
|
|
but it's cool to put a face to the name.
|
|
|
|
The new switches were already racked by Pascal previously, and DWDM multiplexers have
|
|
appeared, and that what used to be a simplex fiber, is now two pairs of duplex fibers.
|
|
Maybe DWDM services are in reach for me at some point? I should look in to that ... but
|
|
for now let's focus on the task at hand.
|
|
|
|
In the picture on the right, you can see from top to bottom: DWDM mux to ZH11/790ZHB
|
|
which immediately struck my eye as clever - it's a 8 channel DWDM mux with channels C31-C38
|
|
and two wideband passthroughs, one is 1310W which means "a wideband 1310nm" which is where
|
|
the 100G optics are sending; and the other is UPG which is an upgrade port, allowing to add
|
|
more DWDM channels in a separate mux into the fiber at a later date, at the expense of
|
|
2dB or so of insertion loss. Nice. The second is an identical unit, a DWDM mux to 780EFF
|
|
which has again one 100G 1310nm wideband channel towards Effretikon and then on to
|
|
Winterthur, and CH31 in use with what is the original C4510 switch (that link used to
|
|
be a dark fiber with vanilla 10G optics connecting 1790BRE with 780EFF).
|
|
|
|
Then there are two redundant aggregation switches (the 32x100G kind), which have each
|
|
four access switches connected to them, with the pink cables. Those are interesting:
|
|
to make 100G very cheap, optics can make use of 4x25G lasers that each take one fiber,
|
|
so 8 fibers in total, and those pink cables are 12-fiber multimode trunks with an
|
|
[MPO](https://vitextech.com/mpo-mtp-connectors-difference/) connector. The optics for this
|
|
type of connection are super cheap, for example this [Flexoptix](https://www.flexoptix.net/en/qsfp28-sr4-transceiver-100-gigabit-mm-850nm-100m-1db-ddm-dom.html?co8502=76636) one. I have the 40G variant at home, also running multimode
|
|
4x10G MPO cables, at a fraction of the price of singlemode single-laser variants. So
|
|
when people say "multimode is useless, always use singlemode", point them at this post
|
|
please!
|
|
|
|
{{< image width="300px" float="left" src="/assets/fiber7-x/after.png" alt="After" >}}
|
|
|
|
There were 11 subscribers who upgraded their service, ten of them to 10Gbps (myself
|
|
included) and one of them to 25Gbps, lucky bastard. So in a first pass we shut down all
|
|
the ports on the C4510 and moved over optics and fibers one by one into the new C9500
|
|
switches, of which there were four.
|
|
|
|
Werke Wangen-Brüttisellen (the local telcoroom owners in my town) historically did do
|
|
a great job at labeling every fiber with little numbered clips, so it's easy to ensure
|
|
that what used to be fiber #33, is now still in port #33. I worked from the right,
|
|
taking two optics from the old switch, moving them into the new switch, and reinserting
|
|
the fibers. The Init7 engineer worked from the left, doing the same. We managed to
|
|
complete this swap-over in record time, according to Pascal who was monitoring from
|
|
remote, and reconfiguring the switches to put the subscribers back into service. We
|
|
started at 00:05 and completed the physical reconfiguration at 01:21am. Go, us!
|
|
|
|
After the physical work, we conducted an Init7 post-maintenance ritual which was eating
|
|
a cheeseburger to replenish our body's salt and fat contents. We did that at my place
|
|
and luckily I have access to a microwave oven and also some Blairs Mega Death hotsauce
|
|
(with liquid rage) which my buddy enthusiastically drizzled onto the burger, but it did
|
|
make him burp just a little bit as sweat poured out of his face. That was fun! I took
|
|
some more pictures, published with permission, in [this album](https://photos.app.goo.gl/VozxYvnuXSQPBePG7).
|
|
|
|
<hr />
|
|
### 10G VLL
|
|
|
|
One more thing! I had waited to order this until the time was right, and the upgrade of
|
|
1790BRE was it -- since I operate AS50869, a little basement ISP, I had always hoped to
|
|
change my 1500 byte MTU L3 service into a Jumboframe capable L2 service. After some
|
|
negotiation on the contractuals, I signed an order ahead of this maintenance to upgrade
|
|
to a 10G virtual leased line (VLL) from this place to the NTT datacenter in Rümlang.
|
|
|
|
In the afternoon, I had already patched my side of the link in the datacenter, and I
|
|
noticed that the Init7 side of the patch was dangling in their rack without an optic. So
|
|
we went to the datacenter (at 2am, the drive from my house to NTT is 9 minutes, without
|
|
speeding!), and plugged in an optic to let my lonely photons hit a friendly receiver.
|
|
|
|
I then got to configure the VLL together with my buddy, which was a hilight of the night
|
|
for me. I now have access to a spiffy new 10 gigabit VLL operating at 9190 MTU, from
|
|
1790BRE directly to my router `chrma0.ipng.ch` at NTT Rümlang, while previously I
|
|
had secured a 1G carrier ethernet operating at 9000 MTU directly to my router
|
|
`chgtg0.ipng.ch` at Interxion Glattbrugg. Between the two sites, I have a CWDM wave
|
|
which currently runs 10G optics but I have the 25G CWDM optics and switches ready for
|
|
deployment. It's somewhat (ok, utterly) over the top, but I like (ok, love) it.
|
|
|
|
```
|
|
pim@chbtl0:~$ show protocols ospfv3 neighbor
|
|
Neighbor ID Pri DeadTime State/IfState Duration I/F[State]
|
|
194.1.163.4 1 00:00:38 Full/PointToPoint 87d05:37:45 dp0p6s0f3[PointToPoint]
|
|
194.1.163.86 1 00:00:31 Full/DROther 16:18:39 dp0p6s0f2.101[BDR]
|
|
194.1.163.87 1 00:00:30 Full/DR 7d15:48:41 dp0p6s0f2.101[BDR]
|
|
194.1.163.0 1 00:00:38 Full/PointToPoint 2d12:02:19 dp0p6s0f0[PointToPoint]
|
|
```
|
|
|
|
The latency from my workstation on which I'm writing this blogpost to, say, my Bucketlist
|
|
location of NIKHEF in the Amsterdam Watergraafsmeer, is pretty much as fast as light
|
|
goes (I've seen 12.2ms, but considering it's ~820km, this is not bad at all):
|
|
|
|
```
|
|
pim@chumbucket:~$ traceroute gripe
|
|
traceroute to gripe (94.142.241.186), 30 hops max, 60 byte packets
|
|
1 chbtl0.ipng.ch (194.1.163.66) 0.211 ms 0.186 ms 0.189 ms
|
|
2 chrma0.ipng.ch (194.1.163.17) 1.463 ms 1.416 ms 1.432 ms
|
|
3 defra0.ipng.ch (194.1.163.25) 7.376 ms 7.344 ms 7.330 ms
|
|
4 nlams0.ipng.ch (194.1.163.27) 12.952 ms 13.115 ms 12.925 ms
|
|
5 gripe.ipng.nl (94.142.241.186) 13.250 ms 13.337 ms 13.223 ms
|
|
```
|
|
|
|
And, due to the work we did above, now the bandwidth is up to par as well, with
|
|
comparable down- and upload speeds of 9.2Gbit from NL>CH and 8.9Gbit from
|
|
CH>NL, and, while I'm not going to prove it here, this would work equally
|
|
well with 9000 byte, 1500 byte or 64 byte frames due to my use of DPDK based
|
|
routers who just don't G.A.F. :
|
|
```
|
|
pim@chumbucket:~$ iperf3 -c nlams0.ipng.ch -R -P 10 ## Richtung Schweiz!
|
|
Connecting to host nlams0, port 5201
|
|
Reverse mode, remote host nlams0 is sending
|
|
...
|
|
[SUM] 0.00-10.01 sec 10.8 GBytes 9.26 Gbits/sec 53 sender
|
|
[SUM] 0.00-10.00 sec 10.7 GBytes 9.19 Gbits/sec receiver
|
|
|
|
pim@chumbucket:~$ iperf3 -c nlams0.ipng.ch -P 10 ## Naar Nederland toe!
|
|
Connecting to host nlams0, port 5201
|
|
...
|
|
[SUM] 0.00-10.00 sec 9.93 GBytes 8.87 Gbits/sec 405 sender
|
|
[SUM] 0.00-10.02 sec 9.91 GBytes 8.84 Gbits/sec receiver
|
|
```
|