diff --git a/BUILDING.md b/BUILDING.md new file mode 100644 index 0000000..876d349 --- /dev/null +++ b/BUILDING.md @@ -0,0 +1,224 @@ +# Building vpp-containerlab + +This docker container creates a VPP instance based on the latest VPP release. It starts up as per +normal, using /etc/vpp/startup.conf (which Containerlab might replace when it starts its +containers). Once started, it'll execute `/etc/vpp/bootstrap.vpp` within the dataplane. There are +two relevant files: + +1. `clab.vpp` -- generated by `files/init-container.sh`. Its purpose is to bind the `veth` + interfaces that containerlab has added to the container into the VPP dataplane (see below). +1. `vppcfg.vpp` -- generated by `files/init-container.sh`. Its purpose is to read the user + specified `vppcfg.yaml` file and convert it into VPP CLI commands. If no YAML file is + specified, or if it is not syntactically valid, an empty file is generated instead. + +For Containerlab users who wish to have more control over their VPP bootstrap, it's possible to +bind-mount `/etc/vpp/bootstrap.vpp`. + +## Building + +To build, this container uses Docker's `buildx`, for which on Debian Bookworm it's required to use +the upstream (docker.com) packages described [[here](https://docs.docker.com/engine/install/debian/)]. +To allow the buildx to build for multi-arch, it's also required to install the Qemu `binfmt` +emulators, with: + +```bash +docker run --privileged --rm tonistiigi/binfmt --install all +``` + +Then, ongoing builds can be cross-platform and take about 1500 seconds on an AMD64 i7-12700T +The buildx invocation will build 'latest' and then tag it with the current VPP package release, +which you can get from `vppcfg show version`, like so: + +```bash +IMG=git.ipng.ch/ipng/vpp-containerlab +ARCH=linux/$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/') +TAG=latest +docker buildx build --load --platform $ARCH \ + --tag $IMG:$TAG -f docker/Dockerfile docker/ + +TAG=v25.10-release +docker buildx build --load --build-arg REPO=2510 --platform $ARCH \ + --tag $IMG:$TAG -f docker/Dockerfile docker/ +``` + +### Sideloading locally built VPP packages + +Instead of pulling VPP from packagecloud, you can sideload locally built `.deb` packages using +Docker buildx's `--build-context` flag. This is useful for testing unreleased VPP builds or +working around version-specific issues (for example, VPP 25.10 fails to start on kernels that +do not expose NUMA topology via sysfs, such as OrbStack on Apple Silicon; VPP 26.06+ fixes this). + +Point `--build-context vppdebs=` at a directory containing `libvppinfra_*.deb`, +`vpp_*.deb`, and `vpp-plugin-core_*.deb`. If the context is not provided, the build falls back +to packagecloud as normal. The `.deb` files are bind-mounted during the build and never stored +in an image layer. **Note:** the directory must contain `.deb` files for exactly one VPP version; +if multiple versions are present the glob patterns will match ambiguously and the build will fail. + +```bash +# Build from locally compiled VPP packages (e.g. from ~/src/vpp after make pkg-deb): +IMG=git.ipng.ch/ipng/vpp-containerlab +ARCH=linux/$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/') +VPPDEBS=~/src/vpp/build-root +docker buildx build --load --platform $ARCH \ + --build-context vppdebs=$VPPDEBS \ + --tag $IMG:latest -f docker/Dockerfile docker/ + +# Build from packagecloud as normal (no --build-context needed): +docker buildx build --load --platform $ARCH \ + --tag $IMG:latest -f docker/Dockerfile docker/ +``` + +### Multiarch + +Building a combined `linux/amd64` + `linux/arm64` manifest requires two machines building natively +— one per architecture. The setup below uses `summer` (amd64, Linux) and `jessica` (arm64, macOS +running OrbStack). **VPP must be compiled on each machine before building the Docker image**, because +the sideloader mounts locally built `.deb` files that are architecture-specific. + +#### Setup + +On `jessica`, the Docker daemon runs inside OrbStack's Linux VM. Expose its SSH port so `summer` +can reach it. OrbStack listens on `127.0.0.1:32222`; add a jump-host entry to `~/.ssh/config` on +`summer`: + +``` +Host jessica-orb + HostName 127.0.0.1 + Port 32222 + User pim + ProxyCommand ssh jessica -W 127.0.0.1:32222 + IdentityFile ~/.ssh/jessica-orb-key + IdentitiesOnly yes + UserKnownHostsFile /dev/null + StrictHostKeyChecking no +``` + +Copy OrbStack's SSH key from `jessica` to `summer`: + +```bash +scp jessica:~/.orbstack/ssh/id_ed25519 ~/.ssh/jessica-orb-key +chmod 600 ~/.ssh/jessica-orb-key +``` + +Verify the full chain works: + +```bash +ssh jessica-orb 'uname -m && docker info | head -3' +# expected: aarch64 +``` + +Create the multiarch builder (run once on `summer`): + +```bash +docker buildx create --name multiarch --driver docker-container --platform linux/amd64 --node summer-amd64 +docker buildx create --append --name multiarch --driver docker-container --platform linux/arm64 --node jessica-arm64 ssh://jessica-orb +docker buildx inspect multiarch --bootstrap +``` + +#### Build + +Build VPP on both machines first (`make pkg-deb` in your VPP source tree on both `summer` and the +OrbStack VM on `jessica`). When sideloading `.deb` files, Docker sends the build context from the +client to every builder node — meaning `summer`'s amd64 debs would be sent to `jessica-orb` for +the arm64 build (wrong arch). The solution is to build each platform separately on its native +machine and combine them into a manifest. + +```bash +IMG=git.ipng.ch/ipng/vpp-containerlab +VPPDEBS=~/src/vpp/build-root + +# Step 1: build amd64 on summer, push with platform tag +docker buildx build --platform linux/amd64 \ + --build-context vppdebs=$VPPDEBS \ + --push --tag $IMG:latest-amd64 \ + -f docker/Dockerfile docker/ + +# Step 2: build arm64 natively on jessica-orb, push with platform tag +# (repo and VPP debs must be present on jessica-orb at the same paths) +# Note: $IMG and $VPPDEBS expand on summer before being sent over SSH -- set them first. +ssh jessica-orb "cd ~/src/vpp-containerlab && \ + docker buildx build --platform linux/arm64 \ + --build-context vppdebs=$VPPDEBS \ + --push --tag $IMG:latest-arm64 \ + -f docker/Dockerfile docker/" + +# Step 3: combine into a single multi-arch manifest and push in one step +# (docker buildx build --push produces manifest lists, so use imagetools, not docker manifest) +docker buildx imagetools create \ + --tag $IMG:latest \ + $IMG:latest-amd64 \ + $IMG:latest-arm64 +``` + +## Testing standalone container + +```bash +docker network create --driver=bridge clab-network --subnet=192.0.2.0/24 \ + --ipv6 --subnet=2001:db8::/64 +docker rm clab-pim +docker run --cap-add=NET_ADMIN --cap-add=SYS_NICE --cap-add=SYS_PTRACE \ + --device=/dev/net/tun:/dev/net/tun \ + --device=/dev/vhost-net:/dev/vhost-net \ + --privileged --name clab-pim \ + git.ipng.ch/ipng/vpp-containerlab:latest +docker network connect clab-network clab-pim +``` + +### A note on DPDK + +DPDK will be disabled by default as it requires hugepages and VFIO and/or UIO to use physical +network cards. If DPDK at some future point is desired, mapping VFIO can be done by adding this: +``` + --device=/dev/vfio/vfio:/dev/vfio/vfio +``` + +or in Containerlab, using the `devices` feature: + +```yaml +my-node: + image: git.ipng.ch/ipng/vpp-containerlab:latest + kind: fdio_vpp + devices: + - /dev/vfio/vfio + - /dev/net/tun + - /dev/vhost-net +``` + +If using DPDK in a container, one of the userspace IO kernel drivers must be loaded in the host +kernel. Options are `igb_uio`, `vfio_pci`, or `uio_pci_generic`: + +```bash +$ sudo modprobe igb_uio +$ sudo modprobe vfio_pci +$ sudo modprobe uio_pci_generic +``` + +Particularly the VFIO driver needs to be present before one can attempt to bindmount +`/dev/vfio/vfio` into the container! + +## Configuring VPP + +When Containerlab starts the docker containers, it'll offer one or more `veth` point to point +network links, which will show up as `eth1` and further. `eth0` is the default NIC that belongs to +the management plane in Containerlab (the one which you'll see with `containerlab inspect`). Before +VPP can use these `veth` interfaces, it needs to bind them, like so: + +```bash +docker exec -it clab-pim vppctl +``` + +and then within the VPP control shell: + +``` +create host-interface v2 name eth1 +set interface name host-eth1 eth1 +set interface mtu 1500 eth1 +set interface ip address eth1 192.0.2.2/24 +set interface ip address eth1 2001:db8::2/64 +set interface state eth1 up +``` + +Containerlab will attach these `veth` pairs to the container, and replace our Docker CMD with one +that waits for all of these interfaces to be added (typically called `if-wait.sh`). In our own CMD, +we then generate a config file called `/etc/vpp/clab.vpp` which contains the necessary VPP commands +to take control over these `veth` pairs. diff --git a/README.md b/README.md index 63d4da6..785edee 100644 --- a/README.md +++ b/README.md @@ -75,224 +75,5 @@ From the vantage point of `client1`, the first hop represents the `vpp1` node, w ## Developer Documentation -This docker container creates a VPP instance based on the latest VPP release. It starts up as per -normal, using /etc/vpp/startup.conf (which Containerlab might replace when it starts its -containers). Once started, it'll execute `/etc/vpp/bootstrap.vpp` within the dataplane. There are -two relevant files: - -1. `clab.vpp` -- generated by `files/init-container.sh`. Its purpose is to bind the `veth` - interfaces that containerlab has added to the container into the VPP dataplane (see below). -1. `vppcfg.vpp` -- generated by `files/init-container.sh`. Its purpose is to read the user - specified `vppcfg.yaml` file and convert it into VPP CLI commands. If no YAML file is - specified, or if it is not syntactically valid, an empty file is generated instead. - -For Containerlab users who wish to have more control over their VPP bootstrap, it's possible to -bind-mount `/etc/vpp/bootstrap.vpp`. - -### Building - -To build, this container uses Docker's `buildx`, for which on Debian Bookworm it's required to use -the upstream (docker.com) packages described [[here](https://docs.docker.com/engine/install/debian/)]. -To allow the buildx to build for multi-arch, it's also required to install the Qemu `binfmt` -emulators, with: - -```bash -docker run --privileged --rm tonistiigi/binfmt --install all -``` - -Then, ongoing builds can be cross-platform and take about 1500 seconds on an AMD64 i7-12700T -The buildx invocation will build 'latest' and then tag it with the current VPP package release, -which you can get from `vppcfg show version`, like so: - -```bash -IMG=git.ipng.ch/ipng/vpp-containerlab -ARCH=linux/$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/') -TAG=latest -docker buildx build --load --platform $ARCH \ - --tag $IMG:$TAG -f docker/Dockerfile docker/ - -TAG=v25.10-release -docker buildx build --load --build-arg REPO=2510 --platform $ARCH \ - --tag $IMG:$TAG -f docker/Dockerfile docker/ -``` - -#### Sideloading locally built VPP packages - -Instead of pulling VPP from packagecloud, you can sideload locally built `.deb` packages using -Docker buildx's `--build-context` flag. This is useful for testing unreleased VPP builds or -working around version-specific issues (for example, VPP 25.10 fails to start on kernels that -do not expose NUMA topology via sysfs, such as OrbStack on Apple Silicon; VPP 26.06+ fixes this). - -Point `--build-context vppdebs=` at a directory containing `libvppinfra_*.deb`, -`vpp_*.deb`, and `vpp-plugin-core_*.deb`. If the context is not provided, the build falls back -to packagecloud as normal. The `.deb` files are bind-mounted during the build and never stored -in an image layer. **Note:** the directory must contain `.deb` files for exactly one VPP version; -if multiple versions are present the glob patterns will match ambiguously and the build will fail. - -```bash -# Build from locally compiled VPP packages (e.g. from ~/src/vpp after make pkg-deb): -IMG=git.ipng.ch/ipng/vpp-containerlab -ARCH=linux/$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/') -VPPDEBS=~/src/vpp/build-root -docker buildx build --load --platform $ARCH \ - --build-context vppdebs=$VPPDEBS \ - --tag $IMG:latest -f docker/Dockerfile docker/ - -# Build from packagecloud as normal (no --build-context needed): -docker buildx build --load --platform $ARCH \ - --tag $IMG:latest -f docker/Dockerfile docker/ -``` - -### Multiarch - -Building a combined `linux/amd64` + `linux/arm64` manifest requires two machines building natively -— one per architecture. The setup below uses `summer` (amd64, Linux) and `jessica` (arm64, macOS -running OrbStack). **VPP must be compiled on each machine before building the Docker image**, because -the sideloader mounts locally built `.deb` files that are architecture-specific. - -#### Setup - -On `jessica`, the Docker daemon runs inside OrbStack's Linux VM. Expose its SSH port so `summer` -can reach it. OrbStack listens on `127.0.0.1:32222`; add a jump-host entry to `~/.ssh/config` on -`summer`: - -``` -Host jessica-orb - HostName 127.0.0.1 - Port 32222 - User pim - ProxyCommand ssh jessica -W 127.0.0.1:32222 - IdentityFile ~/.ssh/jessica-orb-key - IdentitiesOnly yes - UserKnownHostsFile /dev/null - StrictHostKeyChecking no -``` - -Copy OrbStack's SSH key from `jessica` to `summer`: - -```bash -scp jessica:~/.orbstack/ssh/id_ed25519 ~/.ssh/jessica-orb-key -chmod 600 ~/.ssh/jessica-orb-key -``` - -Verify the full chain works: - -```bash -ssh jessica-orb 'uname -m && docker info | head -3' -# expected: aarch64 -``` - -Create the multiarch builder (run once on `summer`): - -```bash -docker buildx create --name multiarch --driver docker-container --platform linux/amd64 --node summer-amd64 -docker buildx create --append --name multiarch --driver docker-container --platform linux/arm64 --node jessica-arm64 ssh://jessica-orb -docker buildx inspect multiarch --bootstrap -``` - -#### Build - -Build VPP on both machines first (`make pkg-deb` in your VPP source tree on both `summer` and the -OrbStack VM on `jessica`). When sideloading `.deb` files, Docker sends the build context from the -client to every builder node — meaning `summer`'s amd64 debs would be sent to `jessica-orb` for -the arm64 build (wrong arch). The solution is to build each platform separately on its native -machine and combine them into a manifest. - -```bash -IMG=git.ipng.ch/ipng/vpp-containerlab -VPPDEBS=~/src/vpp/build-root - -# Step 1: build amd64 on summer, push with platform tag -docker buildx build --platform linux/amd64 \ - --build-context vppdebs=$VPPDEBS \ - --push --tag $IMG:latest-amd64 \ - -f docker/Dockerfile docker/ - -# Step 2: build arm64 natively on jessica-orb, push with platform tag -# (repo and VPP debs must be present on jessica-orb at the same paths) -ssh jessica-orb "cd ~/src/vpp-containerlab && \ - docker buildx build --platform linux/arm64 \ - --build-context vppdebs=$VPPDEBS \ - --push --tag $IMG:latest-arm64 \ - -f docker/Dockerfile docker/" - -# Step 3: combine into a single multi-arch manifest and push in one step -# (docker buildx build --push produces manifest lists, so use imagetools, not docker manifest) -docker buildx imagetools create \ - --tag $IMG:latest \ - $IMG:latest-amd64 \ - $IMG:latest-arm64 -``` - -### Testing standalone container - -```bash -docker network create --driver=bridge clab-network --subnet=192.0.2.0/24 \ - --ipv6 --subnet=2001:db8::/64 -docker rm clab-pim -docker run --cap-add=NET_ADMIN --cap-add=SYS_NICE --cap-add=SYS_PTRACE \ - --device=/dev/net/tun:/dev/net/tun \ - --device=/dev/vhost-net:/dev/vhost-net \ - --privileged --name clab-pim \ - git.ipng.ch/ipng/vpp-containerlab:latest -docker network connect clab-network clab-pim -``` - -#### A note on DPDK - -DPDK will be disabled by default as it requires hugepages and VFIO and/or UIO to use physical -network cards. If DPDK at some future point is desired, mapping VFIO can be done by adding this: -``` - --device=/dev/vfio/vfio:/dev/vfio/vfio -``` - -or in Containerlab, using the `devices` feature: - -```yaml -my-node: - image: git.ipng.ch/ipng/vpp-containerlab:latest - kind: fdio_vpp - devices: - - /dev/vfio/vfio - - /dev/net/tun - - /dev/vhost-net -``` - -If using DPDK in a container, one of the userspace IO kernel drivers must be loaded in the host -kernel. Options are `igb_uio`, `vfio_pci`, or `uio_pci_generic`: - -```bash -$ sudo modprobe igb_uio -$ sudo modprobe vfio_pci -$ sudo modprobe uio_pci_generic -``` - -Particularly the VFIO driver needs to be present before one can attempt to bindmount -`/dev/vfio/vfio` into the container! - -### Configuring VPP - -When Containerlab starts the docker containers, it'll offer one or more `veth` point to point -network links, which will show up as `eth1` and further. `eth0` is the default NIC that belongs to -the management plane in Containerlab (the one which you'll see with `containerlab inspect`). Before -VPP can use these `veth` interfaces, it needs to bind them, like so: - -```bash -docker exec -it clab-pim vppctl -``` - -and then within the VPP control shell: - -``` -create host-interface v2 name eth1 -set interface name host-eth1 eth1 -set interface mtu 1500 eth1 -set interface ip address eth1 192.0.2.2/24 -set interface ip address eth1 2001:db8::2/64 -set interface state eth1 up -``` - -Containerlab will attach these `veth` pairs to the container, and replace our Docker CMD with one -that waits for all of these interfaces to be added (typically called `if-wait.sh`). In our own CMD, -we then generate a config file called `/etc/vpp/clab.vpp` which contains the necessary VPP commands -to take control over these `veth` pairs. +See [BUILDING.md](BUILDING.md) for instructions on building the image, sideloading locally built +VPP packages, multiarch builds, testing, and configuring VPP.