---
date: "2021-07-19T16:12:54Z"
title: 'Review: PCEngines APU6 (with SFP)'
aliases:
- /s/articles/2021/07/19/pcengines-apu6.html
---

*    Author: Pim van Pelt <[pim@ipng.nl](mailto:pim@ipng.nl)>
*    Reviewed: Pascal Dornier <[pdornier@pcengines.ch](mailto:pdornier@pcengines.ch)>
*    Status: Draft - Review - **Approved**

I did this test back in February, but can now finally publish the results! This little SBC is
definitely going to be a hit in the ISP industry. See more information about it [here](https://www.pcengines.ch/apu6b4.htm).

[PC Engines](https://pcengines.ch/) develops and sells small single board computers for networking
to a worldwide customer base. This article discusses a new/unreleased product which PC Engines has
developed, which has specific significance in the network operator community: an SBC which comes
with three RJ45/UTP based network ports, and one SFP optical port.

# Executive Summary

Due to the use of Intel i210-IS on the SFP port and i211-AT on the three copper ports, and due to
it having no moving parts (fans, hard disks, etc), this SBC is an excellent choice for network
appliances such as out-of-band or serial consoles in a datacenter, or routers in a small business
or home office.

## Detailed findings

{{< image width="300px" float="right" src="/assets/pcengines-apu6/apu6.png" alt="APU6" >}}


The [APU](https://www.pcengines.ch/apu2.htm) series boards typically ship with 2GB or 4GB of DRAM,
2, 3 or 4 Intel i211-AT network interfaces, and a four core AMD GX-412TC (running at 1GHz). This
review is about the following **APU6** unit, which comes with 4GB of DRAM (this preproduction unit
came with 2GB, but that will be fixed in the production version), 3x i211-AT for the RJ45
network interfaces, and one i210-IS with an SFP cage.

One other significant difference is visible -- the trusty rusty DB9 connector that exposes the first
serial RS232 port is replaced with a modern CP2104 (USB vendor `10c4:ea60`) from Silicon Labs which
exposes the serial port as TTL/serial on a micro USB connector rather than RS232, neat!

## Transceiver Compatibility

{{< image float="right" src="/assets/pcengines-apu6/optics.png" alt="Optics" >}}

The small form-factor pluggable (SFP) is a compact, hot-pluggable network interface module used for
both telecommunication and data communications applications. An SFP interface on networking hardware
is a modular slot for a media-specific transceiver in order to connect a fiber-optic cable or
sometimes a copper cable. Such a slot is typically called a _cage_.

The SFP port accepts most/any optics brand and configuration (Copper, regular 850nm/1310nm/1550nm
based, BiDi as commonly used in FTTH deployments, CWDM for use behind an OADM). I tried 6 different
vendors and types, see below for results. All modules worked, regardless of vendor or brand.

I tried 6 different SFP modules, all successfully. See the links in the list for an output of an
optical diagnostics tool (using the SFF-8472 standard for SFP/SFP+ management).

Each module provided link and passed traffic. The loadtest below was done with the BiDi optics
in one interface and a boring RJ45 copper cable in another. It's going to be fantastic to be able
to use these APU6's in a datacenter setting as remote / out-of-band serial devices, specifically
nowadays where UTP is becoming a scarcity and everybody has fiber infrastructure in their racks.

Vendor  | Type              | Description                     | Details
------- | ----------------- | ------------------------------- | ------------------
Finisar | FTLF8519P2BNL-RB  | 850nm duplex                    | [sfp0.txt](/assets/pcengines-apu6/sfp0.txt)
Generic | Unknown(no DOM)   | 850nm duplex                    | [sfp1.txt](/assets/pcengines-apu6/sfp1.txt)
Cisco   | GLC-LH-SMD        | 1310nm duplex                   | [sfp2.txt](/assets/pcengines-apu6/sfp2.txt)
Cisco   | SFP-GE-BX-D       | 1490nm Bidirectional (FTTH CPE) | [sfp3.txt](/assets/pcengines-apu6/sfp3.txt)
Cisco   | SFP-GE-BX-U       | 1310nm Bidirectional (FTTH COR) | [sfp3.txt](/assets/pcengines-apu6/sfp3.txt)
Cisco   | BT-OC24-20A       | 1550nm OC24 SDH                 | [sfp4.txt](/assets/pcengines-apu6/sfp4.txt)
Finisar | FTRJ1319P1BTL-C7  | 1310nm 20km (w/ 6dB attenuator) | [sfp5.txt](/assets/pcengines-apu6/sfp5.txt)


## Network Loadtest

The choice of Intel i210/i211 network controller on this board allows operators to use Intel's
DPDK with relatively high performance, compared to regular (kernel) based routing. I loadtested
Linux (Ubuntu 20.04), OpenBSD (6.8), and two lesser known but way cooler DPDK open source
appliances called Danos ([ref](https://www.danosproject.org/)) and VPP ([ref](https://fd.io/))
respectively.

Specifically worth calling out that while Linux and OpenBSD struggled, both DPDK appliances had
absolutely no problems filling a bidirectional gigabit stream of "regular internet traffic"
(referred to as `imix`), and came close to _line rate_ with "64b UDP packets". The line rate of
a gigabit ethernet is 1.48Mpps in one direction, and my loadtests stressed both directions
simultaneously.

### Methodology

For the loadtests, I used Cisco's T-Rex ([ref](https://trex-tgn.cisco.com/)) in stateless mode,
with a custom Python controller that ramps up and down traffic from the loadtester to the device
under test (DUT) by sending traffic out `port0` to the DUT, and expecting that traffic to be
presented back out from the DUT to its `port1`, and vice versa (out from `port1` -> DUT -> back
in on `port0`). The loadtester first sends a few seconds of warmup, this is to ensure the DUT is
passing traffic and offers the ability to inspect the traffic before the actual rampup. Then
the loadteser ramps up linearly from zero to 100% of line rate (in our case, line rate is
one gigabit in both directions), finally it holds the traffic at full line rate for a certain
duration. If at any time the loadtester fails to see the traffic it's emitting return on its
second port, it flags the DUT as saturated; and this is noted as the maximum bits/second and/or
packets/second.

```
usage: trex-loadtest.bin [-h] [-s SERVER] [-p PROFILE_FILE] [-o OUTPUT_FILE] [-wm WARMUP_MULT]
                         [-wd WARMUP_DURATION] [-rt RAMPUP_TARGET]
                         [-rd RAMPUP_DURATION] [-hd HOLD_DURATION]

T-Rex Stateless Loadtester -- pim@ipng.nl

optional arguments:
  -h, --help            show this help message and exit
  -s SERVER, --server SERVER
                        Remote trex address (default: 127.0.0.1)
  -p PROFILE_FILE, --profile PROFILE_FILE
                        STL profile file to replay (default: imix.py)
  -o OUTPUT_FILE, --output OUTPUT_FILE
                        File to write results into, use "-" for stdout (default: -)
  -wm WARMUP_MULT, --warmup_mult WARMUP_MULT
                        During warmup, send this "mult" (default: 1kpps)
  -wd WARMUP_DURATION, --warmup_duration WARMUP_DURATION
                        Duration of warmup, in seconds (default: 30)
  -rt RAMPUP_TARGET, --rampup_target RAMPUP_TARGET
                        Target percentage of line rate to ramp up to (default: 100)
  -rd RAMPUP_DURATION, --rampup_duration RAMPUP_DURATION
                        Time to take to ramp up to target percentage of line rate, in seconds (default: 600)
  -hd HOLD_DURATION, --hold_duration HOLD_DURATION
                        Time to hold the loadtest at target percentage, in seconds (default: 30)
```

It's worth pointing out that almost all systems are _pps-bound_ not _bps-bound_. A typical rant
I have is that network vendors are imprecise when they specify their throughput "up to 40Gbit"
they more often than not mean "under carefully crafted conditions" such as utilizing jumboframes
(9216 bytes rather than "usual" 1500 byte MTU found on ethernet, which is easier on the router
than a typical internet mixture (closer to 1100 bytes), and much easier yet than if the router
is asked to forward 64 byte packets, for instance in a DDoS attack); and only in one direction;
and only using exactly one source/destination IP address/port, which is a little bit easier to
do than to look up a destination in a forwarding table containing 1M destinations -- for context
a current internet backbone router carries ~845K IPv4 destinations and ~105K IPv6 destinations.

### Results

Product | Loadtest | Throughput (pps) | Throughput (bps)  | % of linerate | Details
------- | -------- | ---------------- | ------------------| ------------- | ----------
Linux   | imix     | 150.21 Kpps      | 452.81 Mbps       | 45.28%        | [apu6-linux-imix.json](/assets/pcengines-apu6/apu6-linux-imix.json)
OpenBSD | imix     | 145.52 Kpps      | 444.51 Mbps       | 44.45%        | [apu6-openbsd-imix.json](/assets/pcengines-apu6/apu6-openbsd-imix.json)
**VPP**     | **imix**     | **654.40 Kpps**      | **2.00 Gbps**         | **199.90%**       | [apu6-vpp-imix.json](/assets/pcengines-apu6/apu6-vpp-imix.json)
**Danos**   | **imix**     | **655.53 Kpps**      | **2.00 Gbps**         | **200.24%**       | [apu6-danos-imix.json](/assets/pcengines-apu6/apu6-danos-imix.json)
Linux   | 64b      | 96.93 Kpps       | 65.14 Mbps        | 6.51%         | [apu6-linux-64b.json](/assets/pcengines-apu6/apu6-linux-64b.json)
OpenBSD | 64b      | 152.09 Kpps      | 102.20 Mbps       | 10.22%        | [apu6-openbsd-64b.json](/assets/pcengines-apu6/apu6-openbsd-64b.json)
VPP     | 64b      | 1.78 Mpps        | 1.19 Gbps         | 119.49%       | [apu6-vpp-64b.json](/assets/pcengines-apu6/apu6-vpp-64b.json)
**Danos**   | **64b**      | **2.30 Mpps**        | **1.55 Gbps**         | **154.62%**       | [apu6-danos-64b.json](/assets/pcengines-apu6/apu6-danos-64b.json)

{{< image width="800px" src="/assets/pcengines-apu6/results.png" alt="Results" >}}

For more information on the methodology and the scripts that drew these graphs, take a look
at my buddy Michal's [GitHub Page](https://github.com/wejn/trex-loadtest-viz), which, given
time, will probably turn into its own subsection of this website (I can only imagine the value
of a corpus of loadtests of popular equipment in the consumer arena).

## Caveats

The unit was shipped to me free of charge by PC Engines for the purposes of load- and systems
integration testing. Other than that, this is not a paid endorsement and views of this review
are my own.

## Open Questions

### SFP I2C

Considering the target audience, I wonder if there is a possibility to break out the I2C pins from
the SFP cage into a header on the board, so that users can connect them through to the CPU's I2C
controller (or bitbang directly on GPIO pins), and use the APU6 as an SFP flasher. I think that
would come in incredibly handy in a datacenter setting.

### CPU bound

The DPDK based router implementations are CPU bound, and could benefit from a little bit more power.
I am duly impressed by the throughput seen in terms of packets/sec/watt, but considering a typical
router has a (forwarding) dataplane and needs as well a (configuration) controlplane, we are short
about 30% CPU cycles. If a controlplane (like Bird or FRR ([ref](https://frrouting.org)) is dedicated
one core, that leaves us three cores for forwarding, with which we obtain roughly 154% of linerate,
we'll need that `200/154 == 1.298` to obtain line rate in both directions. That said, the APU6 has
absolutely no problems saturating a gigabit **in both directions** under normal (==imix)
circumstances.

# Appendix

## Appendix 1 - Terminology

**Term** | **Description**
-------- | ---------------
OADM     | **optical add drop multiplexer** -- a device used in wavelength-division multiplexing systems for multiplexing and routing different channels of light into or out of a single mode fiber (SMF)
ONT      | **optical network terminal** - The ONT converts fiber-optic light signals to copper based electric signals, usually Ethernet.
OTO      | **optical telecommunication outlet** - The OTO is a fiber optic outlet that allows easy termination of cables in an office and home environment.  Installed OTOs are referred to by their OTO-ID.
CARP     | **common address redundancy protocol** - Its purpose is to allow multiple hosts on the same network segment to share an IP address. CARP is a secure, free alternative to the Virtual Router Redundancy Protocol (VRRP) and the Hot Standby Router Protocol (HSRP).
SIT      | **simple internet transition** - Its purpose is to interconnect isolated IPv6 networks, located in global IPv4 Internet via tunnels.
STB      | **set top box** - a device that enables a television set to become a user interface to the Internet and also enables a television set to receive and decode digital television (DTV) broadcasts.
GRE      | **generic routing encapsulation** - a tunneling protocol developed by Cisco Systems that can encapsulate a wide variety of network layer protocols inside virtual point-to-point links over an Internet Protocol network.
L2VPN    | **layer2 virtual private network** - a service that emulates a switched Ethernet (V)LAN across a pseudo-wire (typically an IP tunnel)
DHCP     | **dynamic host configuration protocol** - an IPv4 network protocol that enables a server to automatically assign an IP address to a computer from a defined range of numbers.
DHCP6-PD | **Dynamic host configuration protocol: prefix delegation** - an IPv6 network protocol that enables a server to automatically assign network prefixes to a customer from a defined range of numbers.
NDP NS/NA | **neighbor discovery protocol: neighbor solicitation / advertisement** - an ipv6 specific protocol to discover and judge reachability of other nodes on a shared link.
NDP RS/RA | **neighbor discovery protocol: router solicitation / advertisement** - an ipv6 specific protocol to discover and install local address and gateway information.
SBC       | **single board computer** - a compute computer with all peripherals and components directly attached to the board.
