ipng/nginx-ipng-stats-plugin
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Support multiple device-pinned listens sharing a single port

Browse Source 
Nginx's config-level duplicate-listen check rejected the
documented pattern of `listen 80 device=X ipng_source_tag=A;
listen 80 device=Y ipng_source_tag=B;` with "a duplicate listen
0.0.0.0:80", and even when the dedup was bypassed the kernel
refused the second bind() because the first socket was already
holding the port without SO_BINDTODEVICE.

The listen wrapper now detects same-sockaddr duplicates before
the core handler sees them and records them with `needs_clone=1`.
In init_module, phase 1 clones an ngx_listening_t for each such
duplicate, phase 3 closes every inherited naked fd, and phase 4
rebinds every target with SO_REUSEADDR + SO_REUSEPORT +
SO_BINDTODEVICE set before bind(). SO_REUSEPORT keeps
`nginx -s reload` from colliding with the still-bound sockets
held by old workers during graceful drain; IPV6_V6ONLY matches
nginx's default so the IPv6 listen doesn't claim the IPv4
wildcard and collide with sibling IPv4-specific listens.

Restructure 01-module to cover the pattern end-to-end: four
device-pinned listens on port 8080 (eth1 shares tag `tag1`
across v4 and v6; eth2 splits into `tag2-v4` / `tag2-v6`),
clients and server both get IPv6 addresses, and a new
"Per-(device, family) request count accuracy" case proves that
10 requests on each of the four combinations yields tag1=20,
tag2-v4=10, tag2-v6=10. Mgmt/direct traffic moves to port 9180
so it no longer clashes with the shared-port wildcards.

Document the constraint in docs/user-guide.md: all listens on
a given port must carry `device=`, and direct traffic belongs
on a separate port.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Pim van Pelt
2026-04-18 11:45:40 +02:00
parent fdef2a552b
commit df05bae8a3
7 changed files with 400 additions and 153 deletions
Download Patch File Download Diff File Expand all files Collapse all files

121
tests/01-module/01-e2e.robot
View File

@@ -18,7 +18,7 @@ ${SERVER} clab-${lab-name}-server
${CLIENT1} clab-${lab-name}-client1
${CLIENT2} clab-${lab-name}-client2
${SCRAPE_URL} http://172.20.40.2:9113/.well-known/ipng/statsz
${SERVER_MGMT} http://172.20.40.2:8080
${SERVER_MGMT} http://172.20.40.2:9180
*** Test Cases ***
@@ -44,22 +44,42 @@ JSON scrape
# --- Per-device attribution ---
Attribute cl1 via eth1
[Documentation] Traffic on server:eth1 carries source_tag=cl1, vip=10.0.1.1.
Attribute tag1 via eth1 (v4)
[Documentation] IPv4 traffic on server:eth1 carries source_tag=tag1.
Send Fast Requests ${CLIENT1} 10.0.1.1 5
Wait For Flush
${output} = Scrape Prometheus
Should Contain ${output} source_tag="cl1"
Should Contain ${output} source_tag="tag1"
Should Contain ${output} vip="10.0.1.1"
Attribute cl2 via eth2
[Documentation] Traffic on server:eth2 carries source_tag=cl2, vip=10.0.2.1.
Attribute tag2-v4 via eth2 (v4)
[Documentation] IPv4 traffic on server:eth2 carries source_tag=tag2-v4.
Send Fast Requests ${CLIENT2} 10.0.2.1 5
Wait For Flush
${output} = Scrape Prometheus
Should Contain ${output} source_tag="cl2"
Should Contain ${output} source_tag="tag2-v4"
Should Contain ${output} vip="10.0.2.1"
Attribute tag1 via eth1 (v6)
[Documentation] IPv6 traffic on server:eth1 carries source_tag=tag1
... — same tag as v4, demonstrating that tag= can be
... shared across address families for one device.
Send Fast Requests v6 ${CLIENT1} 2001:db8:1::1 5
Wait For Flush
${output} = Scrape With Filter source_tag=tag1
Should Contain ${output} source_tag="tag1"
Should Contain ${output} vip="2001:db8:1::1"
Attribute tag2-v6 via eth2 (v6)
[Documentation] IPv6 traffic on server:eth2 carries source_tag=tag2-v6
... — distinct from the eth2 v4 tag, demonstrating
... per-(device, family) attribution.
Send Fast Requests v6 ${CLIENT2} 2001:db8:2::1 5
Wait For Flush
${output} = Scrape Prometheus
Should Contain ${output} source_tag="tag2-v6"
Should Contain ${output} vip="2001:db8:2::1"
Direct traffic tagged
[Documentation] Mgmt-interface traffic carries source_tag=direct.
${rc} ${output} = Run And Return Rc And Output
@@ -76,7 +96,7 @@ Per-class code counters
Docker Exec Ignore Rc ${CLIENT1} curl -s http://10.0.1.1:8080/notfound
Docker Exec Ignore Rc ${CLIENT1} curl -s http://10.0.1.1:8080/notfound
Wait For Flush
${output} = Scrape With Filter source_tag=cl1
${output} = Scrape With Filter source_tag=tag1
Should Contain ${output} code="4xx"
Should Contain ${output} code="2xx"
@@ -86,11 +106,11 @@ Duration histogram
[Documentation] proxy_pass to a 50 ms backend populates sum and buckets.
Send Slow Requests ${CLIENT1} 10.0.1.1 3
Wait For Flush
${prom} = Scrape With Filter source_tag=cl1
${prom} = Scrape With Filter source_tag=tag1
Should Match Regexp ${prom} request_duration_seconds_sum\\{[^}]*\\}\\s+\\d+\\.\\d*[1-9]
${rc} ${json} = Run And Return Rc And Output
... curl -sf -H 'Accept: application/json' '${SCRAPE_URL}?source_tag=cl1' | python3 -m json.tool
... curl -sf -H 'Accept: application/json' '${SCRAPE_URL}?source_tag=tag1' | python3 -m json.tool
Should Be Equal As Integers ${rc} 0
Should Contain ${json} request_duration_ms
Should Contain ${json} buckets
@@ -98,14 +118,14 @@ Duration histogram
# --- Scrape filters ---
Filter by source_tag
[Documentation] ?source_tag=cl1 returns cl1 only; cl2 only.
${output} = Scrape With Filter source_tag=cl1
Should Contain ${output} source_tag="cl1"
Should Not Contain ${output} source_tag="cl2"
[Documentation] ?source_tag=tag1 returns tag1 only; tag2-v4 only.
${output} = Scrape With Filter source_tag=tag1
Should Contain ${output} source_tag="tag1"
Should Not Contain ${output} source_tag="tag2-v4"
${output} = Scrape With Filter source_tag=cl2
Should Contain ${output} source_tag="cl2"
Should Not Contain ${output} source_tag="cl1"
${output} = Scrape With Filter source_tag=tag2-v4
Should Contain ${output} source_tag="tag2-v4"
Should Not Contain ${output} source_tag="tag1"
Filter by VIP
[Documentation] ?vip=10.0.1.1 excludes 10.0.2.1.
@@ -115,10 +135,10 @@ Filter by VIP
Filter combined
[Documentation] source_tag + vip intersection.
${output} = Scrape With Filter source_tag=cl1&vip=10.0.1.1
Should Contain ${output} source_tag="cl1"
${output} = Scrape With Filter source_tag=tag1&vip=10.0.1.1
Should Contain ${output} source_tag="tag1"
Should Contain ${output} vip="10.0.1.1"
Should Not Contain ${output} source_tag="cl2"
Should Not Contain ${output} source_tag="tag2-v4"
Filter unknown tag
[Documentation] Unknown source_tag returns empty data set.
@@ -128,18 +148,18 @@ Filter unknown tag
# --- nginx variable ---
Variable in access log
[Documentation] $ipng_source_tag appears as cl1, cl2, direct in log.
[Documentation] $ipng_source_tag appears as tag1, tag2-v4, direct in log.
${output} = Docker Exec ${SERVER} cat /var/log/nginx/access.log
Should Match Regexp ${output} src=cl1
Should Match Regexp ${output} src=cl2
Should Match Regexp ${output} src=tag1
Should Match Regexp ${output} src=tag2-v4
Should Match Regexp ${output} src=direct
UDP logtail
[Documentation] ipng_stats_logtail udp:// sends log lines to a local
... nc listener; captured file has all sources and VIPs.
${output} = Docker Exec ${SERVER} cat /var/log/nginx/logtail-udp.log
Should Match Regexp ${output} cl1
Should Match Regexp ${output} cl2
Should Match Regexp ${output} tag1
Should Match Regexp ${output} tag2-v4
Should Match Regexp ${output} direct
Should Match Regexp ${output} 10\\.0\\.1\\.1
Should Match Regexp ${output} 10\\.0\\.2\\.1
@@ -166,10 +186,10 @@ VIP in access log
Counters survive reload
[Documentation] Shared-memory zone persists across nginx -s reload.
${before} = Get Request Count cl1
${before} = Get Request Count tag1
Docker Exec ${SERVER} nginx -s reload
Sleep 2s Wait for new workers
${after} = Get Request Count cl1
${after} = Get Request Count tag1
Should Be True ${after} >= ${before}
... Counters dropped after reload: before=${before} after=${after}
@@ -177,24 +197,37 @@ Traffic after reload
[Documentation] New requests are counted after reload.
Send Fast Requests ${CLIENT1} 10.0.1.1 3
Wait For Flush
${output} = Scrape With Filter source_tag=cl1
Should Contain ${output} source_tag="cl1"
${output} = Scrape With Filter source_tag=tag1
Should Contain ${output} source_tag="tag1"
# --- Counter correctness ---
Request count accuracy
[Documentation] 10 requests per client yields exactly 10 delta.
${before_cl1} = Get Request Count cl1
${before_cl2} = Get Request Count cl2
Send Fast Requests ${CLIENT1} 10.0.1.1 10
Send Fast Requests ${CLIENT2} 10.0.2.1 10
Per-(device, family) request count accuracy
[Documentation] 10 requests on each of the four (device, family)
... combinations yields tag1=20, tag2-v4=10, tag2-v6=10.
... Demonstrates that one device can combine v4+v6 under
... a single tag while another device can split them.
${before_tag1} = Get Request Count tag1
${before_tag2v4} = Get Request Count tag2-v4
${before_tag2v6} = Get Request Count tag2-v6
Send Fast Requests ${CLIENT1} 10.0.1.1 10
Send Fast Requests v6 ${CLIENT1} 2001:db8:1::1 10
Send Fast Requests ${CLIENT2} 10.0.2.1 10
Send Fast Requests v6 ${CLIENT2} 2001:db8:2::1 10
Wait For Flush
${after_cl1} = Get Request Count cl1
${after_cl2} = Get Request Count cl2
${delta_cl1} = Evaluate ${after_cl1} - ${before_cl1}
${delta_cl2} = Evaluate ${after_cl2} - ${before_cl2}
Should Be Equal As Integers ${delta_cl1} 10
Should Be Equal As Integers ${delta_cl2} 10
${after_tag1} = Get Request Count tag1
${after_tag2v4} = Get Request Count tag2-v4
${after_tag2v6} = Get Request Count tag2-v6
${delta_tag1} = Evaluate ${after_tag1} - ${before_tag1}
${delta_tag2v4} = Evaluate ${after_tag2v4} - ${before_tag2v4}
${delta_tag2v6} = Evaluate ${after_tag2v6} - ${before_tag2v6}
Should Be Equal As Integers ${delta_tag1} 20
Should Be Equal As Integers ${delta_tag2v4} 10
Should Be Equal As Integers ${delta_tag2v6} 10
*** Keywords ***
@@ -249,6 +282,12 @@ Send Fast Requests
Docker Exec ${client} curl -sf http://${server_ip}:8080/
END
Send Fast Requests v6
[Arguments] ${client} ${server_ip} ${count}
FOR ${i} IN RANGE ${count}
Docker Exec ${client} curl -sf http://[${server_ip}]:8080/
END
Send Slow Requests
[Arguments] ${client} ${server_ip} ${count}
FOR ${i} IN RANGE ${count}

3
tests/01-module/lab/client/start.sh
View File

@@ -15,6 +15,9 @@ while ! ip link show eth1 > /dev/null 2>&1; do
done
ip link set eth1 up
ip addr add ${MY_IP} dev eth1
if [ -n "${MY_IP6}" ]; then
ip -6 addr add ${MY_IP6} dev eth1 nodad
fi
# Remove the default route so packets to 10.0.x.0/24 go out eth1
# (the connected route) instead of through the mgmt bridge.

13
tests/01-module/lab/ipng-stats.clab.yml
View File

@@ -3,12 +3,13 @@
#
# Three nodes:
# server — nginx with the module, a slow Python backend, two data-plane interfaces
# client1 — sends traffic via eth1 (attributed to source_tag=cl1)
# client2 — sends traffic via eth2 (attributed to source_tag=cl2)
# client1 — sends traffic via eth1 (attributed to source_tag=tag1 for both v4 and v6)
# client2 — sends traffic via eth2 (attributed to source_tag=tag2-v4 for v4,
# tag2-v6 for v6 — demonstrates per-(device, family) attribution)
#
# Links:
# server:eth1 ←→ client1:eth1 (10.0.1.0/24)
# server:eth2 ←→ client2:eth1 (10.0.2.0/24)
# Links (each carries a /24 and a /64):
# server:eth1 ←→ client1:eth1 (10.0.1.0/24 + 2001:db8:1::/64)
# server:eth2 ←→ client2:eth1 (10.0.2.0/24 + 2001:db8:2::/64)
name: ipng-stats-test
@@ -38,6 +39,7 @@ topology:
cmd: bash /start.sh
env:
MY_IP: 10.0.1.2/24
MY_IP6: 2001:db8:1::2/64
client2:
kind: linux
@@ -48,6 +50,7 @@ topology:
cmd: bash /start.sh
env:
MY_IP: 10.0.2.2/24
MY_IP6: 2001:db8:2::2/64
links:
- endpoints: ["server:eth1", "client1:eth1"]

38
tests/01-module/lab/server/nginx.conf
View File

@@ -1,5 +1,17 @@
# SPDX-License-Identifier: Apache-2.0
# Test nginx configuration for the ipng_stats module.
#
# Data plane (port 8080) uses four wildcard listens — two address
# families × two devices — to exercise per-(device, family)
# attribution. eth1 uses the same tag (`tag1`) for IPv4 and IPv6,
# while eth2 splits them (`tag2-v4` / `tag2-v6`) so the e2e suite
# can verify that the module can either combine or distinguish
# families per device.
#
# Mgmt/direct traffic hits a separate server block on port 9180.
# Mixing a naked `listen 8080;` or a specific-address `listen
# 172.20.40.2:8080;` with device-tagged wildcards on the same port
# is not supported — see docs/user-guide.md.
load_module /usr/lib/nginx/modules/ngx_http_ipng_stats_module.so;
@@ -30,12 +42,13 @@ http {
ipng_stats_logtail ipng_stats_logtail udp://127.0.0.1:9514 buffer=4k flush=500ms if=$logtail_enabled;
server {
# Mgmt-only listener for direct traffic (tagged "direct").
listen 172.20.40.2:8080;
# Per-interface listeners for attributed traffic.
listen 10.0.1.1:8080 device=eth1 ipng_source_tag=cl1;
listen 10.0.2.1:8080 device=eth2 ipng_source_tag=cl2;
# Per-device wildcard listens. All four share port 8080; the
# kernel's SO_BINDTODEVICE filtering routes each incoming packet
# to the socket pinned to the interface it arrived on.
listen 8080 device=eth1 ipng_source_tag=tag1;
listen [::]:8080 device=eth1 ipng_source_tag=tag1;
listen 8080 device=eth2 ipng_source_tag=tag2-v4;
listen [::]:8080 device=eth2 ipng_source_tag=tag2-v6;
server_name _;
@@ -52,6 +65,19 @@ http {
}
}
server {
# Direct (mgmt) traffic: no device binding on the listen,
# `ipng_stats_default_source direct;` therefore tags it "direct".
# Separate port so it doesn't collide with the device-tagged
# wildcards above.
listen 172.20.40.2:9180;
server_name _;
location / {
return 200 "ok direct\n";
}
}
server {
listen 172.20.40.2:9113;

2
tests/01-module/lab/server/start.sh
View File

@@ -32,6 +32,8 @@ done
ip addr add 10.0.1.1/24 dev eth1
ip addr add 10.0.2.1/24 dev eth2
ip -6 addr add 2001:db8:1::1/64 dev eth1 nodad
ip -6 addr add 2001:db8:2::1/64 dev eth2 nodad
# Slow backend: 50 ms sleep per request.
python3 /opt/config/slow-backend.py &