Add multi-arch build instructions, on Summer (linux/amd64) and Jessica-Orb (linux/arm64)

Set client<->vpp MTU to 1500

README.md

@@ -91,13 +91,137 @@ bind-mount `/etc/vpp/bootstrap.vpp`.
 
 ### Building
 
+To build, this container uses Docker's `buildx`, for which on Debian Bookworm it's required to use
+the upstream (docker.com) packages described [[here](https://docs.docker.com/engine/install/debian/)].
+To allow the buildx to build for multi-arch, it's also required to install the Qemu `binfmt`
+emulators, with:
+
+```bash
+docker run --privileged --rm tonistiigi/binfmt --install all
+```
+
+Then, ongoing builds can be cross-platform and take about 1500 seconds on an AMD64 i7-12700T
+The buildx invocation will build 'latest' and then tag it with the current VPP package release,
+which you can get from `vppcfg show version`, like so:
+
 ```bash
 IMG=git.ipng.ch/ipng/vpp-containerlab
+ARCH=linux/$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/')
 TAG=latest
-docker build --no-cache --build-arg REPO=release -f docker/Dockerfile.bookworm -t $IMG docker/
-docker image tag $IMG $IMG:$TAG
-docker push $IMG
-docker push $IMG:$TAG
+docker buildx build --load --platform $ARCH \
+  --tag $IMG:$TAG -f docker/Dockerfile docker/
+
+TAG=v25.10-release
+docker buildx build --load --build-arg REPO=2510 --platform $ARCH \
+  --tag $IMG:$TAG -f docker/Dockerfile docker/
+```
+
+#### Sideloading locally built VPP packages
+
+Instead of pulling VPP from packagecloud, you can sideload locally built `.deb` packages using
+Docker buildx's `--build-context` flag. This is useful for testing unreleased VPP builds or
+working around version-specific issues (for example, VPP 25.10 fails to start on kernels that
+do not expose NUMA topology via sysfs, such as OrbStack on Apple Silicon; VPP 26.06+ fixes this).
+
+Point `--build-context vppdebs=<path>` at a directory containing `libvppinfra_*.deb`,
+`vpp_*.deb`, and `vpp-plugin-core_*.deb`. If the context is not provided, the build falls back
+to packagecloud as normal. The `.deb` files are bind-mounted during the build and never stored
+in an image layer. **Note:** the directory must contain `.deb` files for exactly one VPP version;
+if multiple versions are present the glob patterns will match ambiguously and the build will fail.
+
+```bash
+# Build from locally compiled VPP packages (e.g. from ~/src/vpp after make pkg-deb):
+IMG=git.ipng.ch/ipng/vpp-containerlab
+ARCH=linux/$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/')
+VPPDEBS=~/src/vpp/build-root
+docker buildx build --load --platform $ARCH \
+  --build-context vppdebs=$VPPDEBS \
+  --tag $IMG:latest -f docker/Dockerfile docker/
+
+# Build from packagecloud as normal (no --build-context needed):
+docker buildx build --load --platform $ARCH \
+  --tag $IMG:latest -f docker/Dockerfile docker/
+```
+
+### Multiarch
+
+Building a combined `linux/amd64` + `linux/arm64` manifest requires two machines building natively
+— one per architecture. The setup below uses `summer` (amd64, Linux) and `jessica` (arm64, macOS
+running OrbStack). **VPP must be compiled on each machine before building the Docker image**, because
+the sideloader mounts locally built `.deb` files that are architecture-specific.
+
+#### Setup
+
+On `jessica`, the Docker daemon runs inside OrbStack's Linux VM. Expose its SSH port so `summer`
+can reach it. OrbStack listens on `127.0.0.1:32222`; add a jump-host entry to `~/.ssh/config` on
+`summer`:
+
+```
+Host jessica-orb
+    HostName 127.0.0.1
+    Port 32222
+    User pim
+    ProxyCommand ssh jessica -W 127.0.0.1:32222
+    IdentityFile ~/.ssh/jessica-orb-key
+    IdentitiesOnly yes
+    UserKnownHostsFile /dev/null
+    StrictHostKeyChecking no
+```
+
+Copy OrbStack's SSH key from `jessica` to `summer`:
+
+```bash
+scp jessica:~/.orbstack/ssh/id_ed25519 ~/.ssh/jessica-orb-key
+chmod 600 ~/.ssh/jessica-orb-key
+```
+
+Verify the full chain works:
+
+```bash
+ssh jessica-orb 'uname -m && docker info | head -3'
+# expected: aarch64
+```
+
+Create the multiarch builder (run once on `summer`):
+
+```bash
+docker buildx create --name multiarch --driver docker-container --platform linux/amd64 --node summer-amd64
+docker buildx create --append --name multiarch --driver docker-container --platform linux/arm64 --node jessica-arm64 ssh://jessica-orb
+docker buildx inspect multiarch --bootstrap
+```
+
+#### Build
+
+Build VPP on both machines first (`make pkg-deb` in your VPP source tree on both `summer` and the
+OrbStack VM on `jessica`). When sideloading `.deb` files, Docker sends the build context from the
+client to every builder node — meaning `summer`'s amd64 debs would be sent to `jessica-orb` for
+the arm64 build (wrong arch). The solution is to build each platform separately on its native
+machine and combine them into a manifest.
+
+```bash
+IMG=git.ipng.ch/ipng/vpp-containerlab
+VPPDEBS=~/src/vpp/build-root
+
+# Step 1: build amd64 on summer, push with platform tag
+docker buildx build --platform linux/amd64 \
+  --build-context vppdebs=$VPPDEBS \
+  --push --tag $IMG:latest-amd64 \
+  -f docker/Dockerfile docker/
+
+# Step 2: build arm64 natively on jessica-orb, push with platform tag
+#   (repo and VPP debs must be present on jessica-orb at the same paths)
+ssh jessica-orb "cd ~/src/vpp-containerlab && \
+  docker buildx build --platform linux/arm64 \
+    --build-context vppdebs=$VPPDEBS \
+    --push --tag $IMG:latest-arm64 \
+    -f docker/Dockerfile docker/"
+
+# Step 3: combine into a single multi-arch manifest and push in one step
+# (docker buildx build --push produces manifest lists, so use imagetools, not docker manifest)
+docker buildx imagetools create \
+  --tag $IMG:latest \
+  $IMG:latest-amd64 \
+  $IMG:latest-arm64
 ```
 
 ### Testing standalone container
@@ -110,7 +234,7 @@ docker run --cap-add=NET_ADMIN --cap-add=SYS_NICE --cap-add=SYS_PTRACE \
            --device=/dev/net/tun:/dev/net/tun \
            --device=/dev/vhost-net:/dev/vhost-net \
            --privileged --name clab-pim \
-           docker.io/pimvanpelt/vpp-containerlab:latest
+           git.ipng.ch/ipng/vpp-containerlab:latest
 docker network connect clab-network clab-pim
 ```
 
@@ -172,7 +296,3 @@ Containerlab will attach these `veth` pairs to the container, and replace our Do
 that waits for all of these interfaces to be added (typically called `if-wait.sh`). In our own CMD,
 we then generate a config file called `/etc/vpp/clab.vpp` which contains the necessary VPP commands
 to take control over these `veth` pairs.
-
-In addition, you can add more commands that'll execute on startup by copying in
-`/etc/vpp/manual-pre.vpp` (to be executed _before_ the containerlab stuff) or
-`/etc/vpp/manual-post.vpp` (to be executed _after_ the containerlab stuff).

docker/Dockerfile

@@ -0,0 +1,50 @@
+# Default empty stage for local VPP debs. Override at build time with:
+#   --build-context vppdebs=/path/to/debs  (e.g. ~/src/vpp/build-root/)
+# If not overridden, falls back to installing VPP from packagecloud (ARG REPO).
+FROM scratch AS vppdebs
+
+FROM ubuntu:noble
+ARG DEBIAN_FRONTEND=noninteractive
+ARG VPP_INSTALL_SKIP_SYSCTL=true
+ARG REPO=release
+EXPOSE 22/tcp
+RUN apt-get update && apt-get -y install curl procps tcpdump iproute2 iptables \
+  iputils-ping net-tools git python3 python3-pip vim-tiny openssh-server \
+  mtr-tiny traceroute && apt-get clean
+
+# Install VPP - sideload from local debs if --build-context vppdebs=<path> is provided,
+# otherwise install from packagecloud. Debs are bind-mounted and never stored in a layer.
+RUN --mount=type=bind,from=vppdebs,target=/tmp/vpp-debs \
+    mkdir -p /var/log/vpp /root/.ssh/ && \
+    if ls /tmp/vpp-debs/vpp_*.deb 1>/dev/null 2>&1; then \
+      apt-get -y install /tmp/vpp-debs/libvppinfra_*.deb \
+                         /tmp/vpp-debs/vpp_*.deb \
+                         /tmp/vpp-debs/vpp-plugin-core_*.deb; \
+    else \
+      curl -s https://packagecloud.io/install/repositories/fdio/${REPO}/script.deb.sh | bash && \
+      apt-get -y install vpp vpp-plugin-core; \
+    fi && \
+    apt-get clean
+
+# Build vppcfg
+RUN pip install --break-system-packages build netaddr yamale argparse pyyaml ipaddress && \
+    git clone https://git.ipng.ch/ipng/vppcfg.git && cd vppcfg && python3 -m build && \
+    pip install --break-system-packages dist/vppcfg-*-py3-none-any.whl
+
+# Install FRR
+RUN curl -s -o /usr/share/keyrings/frrouting.gpg https://deb.frrouting.org/frr/keys.gpg && \
+    echo deb '[signed-by=/usr/share/keyrings/frrouting.gpg]' https://deb.frrouting.org/frr noble frr-stable \
+         > /etc/apt/sources.list.d/frr.list && \
+    apt -y update && apt -y install frr frr-pythontools && apt clean
+
+# Install Bird2
+RUN curl -s -o /usr/share/keyrings/cznic-labs-pkg.gpg https://pkg.labs.nic.cz/gpg && \
+    echo "deb [signed-by=/usr/share/keyrings/cznic-labs-pkg.gpg] https://pkg.labs.nic.cz/bird2 noble main" \
+         > /etc/apt/sources.list.d/cznic-labs-bird2.list  && \
+    apt -y update && apt -y install bird2 && apt clean
+
+# Config files
+COPY files/etc/ /etc/
+COPY files/init-container.sh /sbin/
+RUN chmod 755 /sbin/init-container.sh
+CMD ["/sbin/init-container.sh"]

docker/Dockerfile.bookworm

@@ -1,32 +0,0 @@
-FROM debian:bookworm
-ARG DEBIAN_FRONTEND=noninteractive
-ARG VPP_INSTALL_SKIP_SYSCTL=true
-ARG REPO=release
-EXPOSE 22/tcp
-RUN apt-get update && apt-get -y install curl procps tcpdump iproute2 iptables \
-  iputils-ping net-tools git python3 python3-pip vim-tiny openssh-server bird2 \
-  mtr-tiny traceroute && apt-get clean
-
-# Install VPP
-RUN mkdir -p /var/log/vpp /root/.ssh/
-RUN curl -s https://packagecloud.io/install/repositories/fdio/${REPO}/script.deb.sh |  bash
-RUN apt-get update && apt-get -y install vpp vpp-plugin-core && apt-get clean
-
-# Build vppcfg
-RUN pip install --break-system-packages build netaddr yamale argparse pyyaml ipaddress
-RUN git clone https://git.ipng.ch/ipng/vppcfg.git && cd vppcfg && python3 -m build && \
-    pip install --break-system-packages dist/vppcfg-*-py3-none-any.whl
-
-# Install FRR
-RUN curl -s -o /usr/share/keyrings/frrouting.gpg https://deb.frrouting.org/frr/keys.gpg && \
-   echo deb '[signed-by=/usr/share/keyrings/frrouting.gpg]' https://deb.frrouting.org/frr bookworm frr-stable \
-        > /etc/apt/sources.list.d/frr.list && \
-   apt -y update && apt -y install frr frr-pythontools && apt clean
-
-# Config files
-COPY files/etc/vpp/* /etc/vpp/
-COPY files/etc/bird/* /etc/bird/
-COPY files/etc/frr/* /etc/frr/
-COPY files/init-container.sh /sbin/
-RUN chmod 755 /sbin/init-container.sh
-CMD ["/sbin/init-container.sh"]

docker/files/etc/vpp/startup.conf

@@ -34,11 +34,16 @@ statseg {
 }
 
 plugins {
-  plugin default { enable }
-  plugin dpdk_plugin.so { disable }
+  plugin default { disable }
+  plugin acl_plugin.so { enable }
+  plugin geneve_plugin.so { enable }
+  plugin gre_plugin.so { enable }
+  plugin ipip_plugin.so { enable }
   plugin linux_cp_plugin.so { enable }
   plugin linux_nl_plugin.so { enable }
   plugin sflow_plugin.so { enable }
+  plugin tap_plugin.so { enable }
+  plugin vxlan_plugin.so { enable }
 }
 
 linux-cp {

vpp-bird.clab.yml

@@ -19,7 +19,7 @@ topology:
     client1:
       kind: linux
       exec:
-        - ip link set address 00:c1:ab:00:00:01 dev eth1
+        - ip link set address 00:c1:ab:00:00:01 mtu 1500 dev eth1
         - ip addr add 10.82.98.66/28 dev eth1
         - ip route add 10.82.98.0/24 via 10.82.98.65
         - ip addr add 2001:db8:8298:101::2/64 dev eth1
@@ -27,7 +27,7 @@ topology:
     client2:
       kind: linux
       exec:
-        - ip link set address 00:c1:ab:00:00:02 dev eth1
+        - ip link set address 00:c1:ab:00:00:02 mtu 1500 dev eth1
         - ip addr add 10.82.98.82/28 dev eth1
         - ip route add 10.82.98.0/24 via 10.82.98.81
         - ip addr add 2001:db8:8298:102::2/64 dev eth1

vpp-frr.clab.yml

@@ -21,7 +21,7 @@ topology:
     client1:
       kind: linux
       exec:
-        - ip link set address 00:c1:ab:00:00:01 dev eth1
+        - ip link set address 00:c1:ab:00:00:01 mtu 1500 dev eth1
         - ip addr add 10.82.98.66/28 dev eth1
         - ip route add 10.82.98.0/24 via 10.82.98.65
         - ip addr add 2001:db8:8298:101::2/64 dev eth1
@@ -29,7 +29,7 @@ topology:
     client2:
       kind: linux
       exec:
-        - ip link set address 00:c1:ab:00:00:02 dev eth1
+        - ip link set address 00:c1:ab:00:00:02 mtu 1500 dev eth1
         - ip addr add 10.82.98.82/28 dev eth1
         - ip route add 10.82.98.0/24 via 10.82.98.81
         - ip addr add 2001:db8:8298:102::2/64 dev eth1