438 lines
19 KiB
Plaintext
438 lines
19 KiB
Plaintext
# SPDX-License-Identifier: Apache-2.0
|
||
*** Settings ***
|
||
Documentation End-to-end tests for ngx_http_ipng_stats_module.
|
||
... Deploys a 3-node containerlab topology and validates
|
||
... attribution, counters, histograms, filters, variables,
|
||
... and reload semantics.
|
||
Library OperatingSystem
|
||
Library String
|
||
Suite Setup Deploy Lab
|
||
Suite Teardown Cleanup Lab
|
||
|
||
*** Variables ***
|
||
${lab-name} ipng-stats-test
|
||
${lab-file} lab/ipng-stats.clab.yml
|
||
${runtime} docker
|
||
${CLAB_BIN} sudo containerlab
|
||
${SERVER} clab-${lab-name}-server
|
||
${CLIENT1} clab-${lab-name}-client1
|
||
${CLIENT2} clab-${lab-name}-client2
|
||
${SCRAPE_URL} http://172.20.40.2:9113/.well-known/ipng/statsz
|
||
${SERVER_MGMT} http://172.20.40.2:9180
|
||
|
||
*** Test Cases ***
|
||
|
||
# --- Basic functionality ---
|
||
|
||
Module loads
|
||
[Documentation] nginx -t passes with the module loaded.
|
||
${output} = Docker Exec ${SERVER} nginx -t 2>&1
|
||
Should Contain ${output} syntax is ok
|
||
|
||
Shared-listen-include across multiple server blocks
|
||
[Documentation] Three server blocks all pull in the same
|
||
... ipng-listens.inc (see docs/user-guide.md). The
|
||
... include also carries `reuseport` on every listen
|
||
... — nginx core would normally reject the second
|
||
... server block with "duplicate listen options", but
|
||
... the wrapper strips socket-level options on a
|
||
... repeat (cross-cscf) sockaddr so the first cscf
|
||
... owns the reuseport-cloned socket and the rest
|
||
... merge cleanly. With worker_processes unset
|
||
... (default 1), reuseport produces one socket per
|
||
... (worker × family), i.e. 2 on :8080 here.
|
||
${output} = Docker Exec ${SERVER} nginx -t 2>&1
|
||
Should Not Contain ${output} conflicting server name
|
||
Should Not Contain ${output} duplicate listen
|
||
${listens} = Docker Exec ${SERVER} ss -tlnH
|
||
${count} = Get Regexp Matches ${listens} :8080\\s
|
||
Length Should Be ${count} 2
|
||
... Expected 2 listening sockets on port 8080 (v4+v6 wildcards); got ${count}
|
||
# Proves the cross-cscf option-stripping path actually fired for
|
||
# the 2nd and 3rd server blocks at daemon startup — the wrapper
|
||
# logs a NOTICE per stripped listen via ngx_conf_log_error. We
|
||
# read the container's startup log rather than `nginx -t`'s
|
||
# output because nginx keeps the config-parse log level at ERR
|
||
# until the error_log directive has been fully applied, so
|
||
# `nginx -t` suppresses the NOTICE even though the wrapper is
|
||
# invoked.
|
||
${rc} ${startup_log} = Run And Return Rc And Output
|
||
... docker logs ${SERVER} 2>&1
|
||
Should Contain ${startup_log} stripped socket options from duplicate listen
|
||
|
||
Prometheus scrape
|
||
[Documentation] Scrape returns HELP/TYPE preamble.
|
||
${output} = Scrape Prometheus
|
||
Should Contain ${output} nginx-ipng-stats-plugin
|
||
Should Contain ${output} nginx_ipng_requests_total
|
||
|
||
JSON scrape
|
||
[Documentation] Accept: application/json returns valid JSON with schema.
|
||
${rc} ${output} = Run And Return Rc And Output
|
||
... curl -sf -H 'Accept: application/json' ${SCRAPE_URL} | python3 -m json.tool
|
||
Should Be Equal As Integers ${rc} 0
|
||
Should Contain ${output} "schema": 2
|
||
|
||
# --- Per-device attribution ---
|
||
|
||
Attribute tag1 via eth1 (v4)
|
||
[Documentation] IPv4 traffic on server:eth1 carries source_tag=tag1.
|
||
Send Fast Requests ${CLIENT1} 10.0.1.1 5
|
||
Wait For Flush
|
||
${output} = Scrape Prometheus
|
||
Should Contain ${output} source_tag="tag1"
|
||
Should Contain ${output} vip="10.0.1.1"
|
||
|
||
Attribute tag2-v4 via eth2 (v4)
|
||
[Documentation] IPv4 traffic on server:eth2 carries source_tag=tag2-v4.
|
||
Send Fast Requests ${CLIENT2} 10.0.2.1 5
|
||
Wait For Flush
|
||
${output} = Scrape Prometheus
|
||
Should Contain ${output} source_tag="tag2-v4"
|
||
Should Contain ${output} vip="10.0.2.1"
|
||
|
||
Attribute tag1 via eth1 (v6)
|
||
[Documentation] IPv6 traffic on server:eth1 carries source_tag=tag1
|
||
... — same tag as v4, demonstrating that tag= can be
|
||
... shared across address families for one device.
|
||
Send Fast Requests v6 ${CLIENT1} 2001:db8:1::1 5
|
||
Wait For Flush
|
||
${output} = Scrape With Filter source_tag=tag1
|
||
Should Contain ${output} source_tag="tag1"
|
||
Should Contain ${output} vip="2001:db8:1::1"
|
||
|
||
Attribute tag2-v6 via eth2 (v6)
|
||
[Documentation] IPv6 traffic on server:eth2 carries source_tag=tag2-v6
|
||
... — distinct from the eth2 v4 tag, demonstrating
|
||
... per-(device, family) attribution.
|
||
Send Fast Requests v6 ${CLIENT2} 2001:db8:2::1 5
|
||
Wait For Flush
|
||
${output} = Scrape Prometheus
|
||
Should Contain ${output} source_tag="tag2-v6"
|
||
Should Contain ${output} vip="2001:db8:2::1"
|
||
|
||
Direct traffic tagged
|
||
[Documentation] Mgmt-interface traffic carries source_tag=direct.
|
||
${rc} ${output} = Run And Return Rc And Output
|
||
... curl -sf ${SERVER_MGMT}/
|
||
Should Be Equal As Integers ${rc} 0
|
||
Wait For Flush
|
||
${output} = Scrape Prometheus
|
||
Should Contain ${output} source_tag="direct"
|
||
|
||
# --- Status code tracking ---
|
||
|
||
Per-class code counters
|
||
[Documentation] 4xx and 2xx appear as class-bucketed code= labels.
|
||
Docker Exec Ignore Rc ${CLIENT1} curl -s http://10.0.1.1:8080/notfound
|
||
Docker Exec Ignore Rc ${CLIENT1} curl -s http://10.0.1.1:8080/notfound
|
||
Wait For Flush
|
||
${output} = Scrape With Filter source_tag=tag1
|
||
Should Contain ${output} code="4xx"
|
||
Should Contain ${output} code="2xx"
|
||
|
||
# --- Duration histogram ---
|
||
|
||
Duration histogram
|
||
[Documentation] proxy_pass to a 50 ms backend populates sum and buckets.
|
||
Send Slow Requests ${CLIENT1} 10.0.1.1 3
|
||
Wait For Flush
|
||
${prom} = Scrape With Filter source_tag=tag1
|
||
Should Match Regexp ${prom} request_duration_seconds_sum\\{[^}]*\\}\\s+\\d+\\.\\d*[1-9]
|
||
|
||
${rc} ${json} = Run And Return Rc And Output
|
||
... curl -sf -H 'Accept: application/json' '${SCRAPE_URL}?source_tag=tag1' | python3 -m json.tool
|
||
Should Be Equal As Integers ${rc} 0
|
||
Should Contain ${json} request_duration_ms
|
||
Should Contain ${json} buckets
|
||
|
||
# --- In-flight gauges ---
|
||
|
||
Gauge preamble in Prometheus output
|
||
[Documentation] The scrape preamble advertises the three gauge metric
|
||
... names and types. Present whether or not any traffic
|
||
... has ever been observed.
|
||
${output} = Scrape Prometheus
|
||
Should Match Regexp ${output} (?m)^# TYPE nginx_ipng_active gauge$
|
||
Should Match Regexp ${output} (?m)^# TYPE nginx_ipng_reading gauge$
|
||
Should Match Regexp ${output} (?m)^# TYPE nginx_ipng_writing gauge$
|
||
|
||
Gauges zero at rest
|
||
[Documentation] After the earlier traffic tests have drained and the
|
||
... flush tick has run, the in-flight gauges for (tag1,
|
||
... 10.0.1.1) must all report zero — no request is still
|
||
... resident in the plugin's per-request ctx.
|
||
Wait For Flush
|
||
${output} = Scrape With Filter source_tag=tag1&vip=10.0.1.1
|
||
Should Match Regexp ${output} nginx_ipng_active\\{[^}]*\\}\\s+0\\b
|
||
Should Match Regexp ${output} nginx_ipng_reading\\{[^}]*\\}\\s+0\\b
|
||
Should Match Regexp ${output} nginx_ipng_writing\\{[^}]*\\}\\s+0\\b
|
||
|
||
Gauges appear in JSON output
|
||
[Documentation] Each record in the JSON output carries a gauges object
|
||
... with active/reading/writing keys.
|
||
${rc} ${output} = Run And Return Rc And Output
|
||
... curl -sf -H 'Accept: application/json' '${SCRAPE_URL}?source_tag=tag1&vip=10.0.1.1' | python3 -m json.tool
|
||
Should Be Equal As Integers ${rc} 0
|
||
Should Contain ${output} "gauges"
|
||
Should Contain ${output} "active"
|
||
Should Contain ${output} "reading"
|
||
Should Contain ${output} "writing"
|
||
|
||
Gauges observed non-zero under concurrent load
|
||
[Documentation] Fire a burst of concurrent requests against the
|
||
... /slow backend (50 ms each, single-threaded) and
|
||
... observe at least one of active/reading/writing
|
||
... going non-zero for (tag1, 10.0.1.1) while the
|
||
... burst is in flight. After it drains the gauges
|
||
... return to zero. Uses 20 requests (drain <= ~2s
|
||
... given the single-threaded backend) to keep the
|
||
... test fast but still wide enough for the scrape
|
||
... to catch overlap.
|
||
Run docker exec -d ${CLIENT1} sh -c 'for i in $(seq 20); do curl -sf http://10.0.1.1:8080/slow >/dev/null & done; wait'
|
||
Wait Until Keyword Succeeds 5s 50ms Nonzero Gauge Observed tag1 10.0.1.1
|
||
Wait Until Keyword Succeeds 15s 500ms Gauges Drained tag1 10.0.1.1
|
||
|
||
# --- Scrape filters ---
|
||
|
||
Filter by source_tag
|
||
[Documentation] ?source_tag=tag1 returns tag1 only; tag2-v4 only.
|
||
${output} = Scrape With Filter source_tag=tag1
|
||
Should Contain ${output} source_tag="tag1"
|
||
Should Not Contain ${output} source_tag="tag2-v4"
|
||
|
||
${output} = Scrape With Filter source_tag=tag2-v4
|
||
Should Contain ${output} source_tag="tag2-v4"
|
||
Should Not Contain ${output} source_tag="tag1"
|
||
|
||
Filter by VIP
|
||
[Documentation] ?vip=10.0.1.1 excludes 10.0.2.1.
|
||
${output} = Scrape With Filter vip=10.0.1.1
|
||
Should Contain ${output} vip="10.0.1.1"
|
||
Should Not Contain ${output} vip="10.0.2.1"
|
||
|
||
Filter combined
|
||
[Documentation] source_tag + vip intersection.
|
||
${output} = Scrape With Filter source_tag=tag1&vip=10.0.1.1
|
||
Should Contain ${output} source_tag="tag1"
|
||
Should Contain ${output} vip="10.0.1.1"
|
||
Should Not Contain ${output} source_tag="tag2-v4"
|
||
|
||
Filter unknown tag
|
||
[Documentation] Unknown source_tag returns empty data set.
|
||
${output} = Scrape With Filter source_tag=nonexistent
|
||
Should Not Contain ${output} nginx_ipng_requests_total{
|
||
|
||
# --- nginx variable ---
|
||
|
||
Variable in access log
|
||
[Documentation] $ipng_source_tag appears as tag1, tag2-v4, direct in log.
|
||
${output} = Docker Exec ${SERVER} cat /var/log/nginx/access.log
|
||
Should Match Regexp ${output} src=tag1
|
||
Should Match Regexp ${output} src=tag2-v4
|
||
Should Match Regexp ${output} src=direct
|
||
|
||
UDP logtail
|
||
[Documentation] ipng_stats_logtail udp:// sends log lines to a local
|
||
... nc listener; captured file has all sources and VIPs.
|
||
${output} = Docker Exec ${SERVER} cat /var/log/nginx/logtail-udp.log
|
||
Should Match Regexp ${output} tag1
|
||
Should Match Regexp ${output} tag2-v4
|
||
Should Match Regexp ${output} direct
|
||
Should Match Regexp ${output} 10\\.0\\.1\\.1
|
||
Should Match Regexp ${output} 10\\.0\\.2\\.1
|
||
# Tab-separated format
|
||
Should Match Regexp ${output} \\t
|
||
|
||
Logtail if= filter
|
||
[Documentation] Requests to /notfound are suppressed from logtail by
|
||
... the if=$logtail_enabled condition, but still counted.
|
||
${output} = Docker Exec ${SERVER} cat /var/log/nginx/logtail-udp.log
|
||
Should Not Contain ${output} /notfound
|
||
# But /notfound IS in the regular access log (not filtered there).
|
||
${access} = Docker Exec ${SERVER} cat /var/log/nginx/access.log
|
||
Should Contain ${access} /notfound
|
||
|
||
VIP in access log
|
||
[Documentation] $server_addr resolves to real IPs, not 0.0.0.0.
|
||
${output} = Docker Exec ${SERVER} cat /var/log/nginx/access.log
|
||
Should Contain ${output} vip=10.0.1.1
|
||
Should Contain ${output} vip=10.0.2.1
|
||
Should Not Contain ${output} vip=0.0.0.0
|
||
|
||
# --- Reload resilience ---
|
||
|
||
Counters survive reload
|
||
[Documentation] Shared-memory zone persists across nginx -s reload.
|
||
${before} = Get Request Count tag1
|
||
Docker Exec ${SERVER} nginx -s reload
|
||
Sleep 2s Wait for new workers
|
||
${after} = Get Request Count tag1
|
||
Should Be True ${after} >= ${before}
|
||
... Counters dropped after reload: before=${before} after=${after}
|
||
|
||
Traffic after reload
|
||
[Documentation] New requests are counted after reload.
|
||
Send Fast Requests ${CLIENT1} 10.0.1.1 3
|
||
Wait For Flush
|
||
${output} = Scrape With Filter source_tag=tag1
|
||
Should Contain ${output} source_tag="tag1"
|
||
|
||
# --- Counter correctness ---
|
||
|
||
Per-(device, family) request count accuracy
|
||
[Documentation] 10 requests on each of the four (device, family)
|
||
... combinations yields tag1=20, tag2-v4=10, tag2-v6=10.
|
||
... Demonstrates that one device can combine v4+v6 under
|
||
... a single tag while another device can split them.
|
||
${before_tag1} = Get Request Count tag1
|
||
${before_tag2v4} = Get Request Count tag2-v4
|
||
${before_tag2v6} = Get Request Count tag2-v6
|
||
|
||
Send Fast Requests ${CLIENT1} 10.0.1.1 10
|
||
Send Fast Requests v6 ${CLIENT1} 2001:db8:1::1 10
|
||
Send Fast Requests ${CLIENT2} 10.0.2.1 10
|
||
Send Fast Requests v6 ${CLIENT2} 2001:db8:2::1 10
|
||
Wait For Flush
|
||
|
||
${after_tag1} = Get Request Count tag1
|
||
${after_tag2v4} = Get Request Count tag2-v4
|
||
${after_tag2v6} = Get Request Count tag2-v6
|
||
|
||
${delta_tag1} = Evaluate ${after_tag1} - ${before_tag1}
|
||
${delta_tag2v4} = Evaluate ${after_tag2v4} - ${before_tag2v4}
|
||
${delta_tag2v6} = Evaluate ${after_tag2v6} - ${before_tag2v6}
|
||
|
||
Should Be Equal As Integers ${delta_tag1} 20
|
||
Should Be Equal As Integers ${delta_tag2v4} 10
|
||
Should Be Equal As Integers ${delta_tag2v6} 10
|
||
|
||
*** Keywords ***
|
||
|
||
# --- Lab lifecycle ---
|
||
|
||
Deploy Lab
|
||
Require Deb Build
|
||
Run ${CLAB_BIN} --runtime ${runtime} destroy -t ${CURDIR}/${lab-file} --cleanup 2>&1 || true
|
||
${rc} ${output} = Run And Return Rc And Output
|
||
... ${CLAB_BIN} --runtime ${runtime} deploy -t ${CURDIR}/${lab-file}
|
||
Log ${output}
|
||
Should Be Equal As Integers ${rc} 0
|
||
Wait Until Keyword Succeeds 90s 3s Server Is Ready
|
||
Wait Until Keyword Succeeds 60s 3s Client Can Reach Server ${CLIENT1} 10.0.1.1
|
||
Wait Until Keyword Succeeds 60s 3s Client Can Reach Server ${CLIENT2} 10.0.2.1
|
||
|
||
Require Deb Build
|
||
[Documentation] Fail fast with an actionable message if the user
|
||
... forgot to run `make pkg-deb` before invoking this
|
||
... suite. The server container dpkg-installs the
|
||
... built .deb via its bind-mount of build/.
|
||
${rc} ${output} = Run And Return Rc And Output
|
||
... bash -c 'ls ${EXECDIR}/build/libnginx-mod-http-ipng-stats_*.deb 2>/dev/null'
|
||
Run Keyword If ${rc} != 0
|
||
... Fail Module .deb not found — run `make pkg-deb` first.
|
||
|
||
Server Is Ready
|
||
${rc} ${output} = Run And Return Rc And Output
|
||
... curl -sf ${SCRAPE_URL}
|
||
Should Be Equal As Integers ${rc} 0
|
||
|
||
Client Can Reach Server
|
||
[Arguments] ${client} ${server_ip}
|
||
${rc} ${output} = Run And Return Rc And Output
|
||
... docker exec ${client} curl -sf http://${server_ip}:8080/
|
||
Should Be Equal As Integers ${rc} 0
|
||
|
||
Cleanup Lab
|
||
Run docker logs ${SERVER} > ${EXECDIR}/tests/out/server-docker.log 2>&1
|
||
Run docker exec ${SERVER} cat /var/log/nginx/access.log > ${EXECDIR}/tests/out/server-access.log 2>&1
|
||
Run docker exec ${SERVER} cat /var/log/nginx/error.log > ${EXECDIR}/tests/out/server-error.log 2>&1
|
||
Run docker exec ${SERVER} cat /var/log/nginx/logtail-udp.log > ${EXECDIR}/tests/out/server-logtail-udp.log 2>&1
|
||
Run docker exec ${SERVER} ip addr > ${EXECDIR}/tests/out/server-ip-addr.log 2>&1
|
||
Run docker exec ${SERVER} ip route > ${EXECDIR}/tests/out/server-ip-route.log 2>&1
|
||
Run ${CLAB_BIN} --runtime ${runtime} destroy -t ${CURDIR}/${lab-file} --cleanup
|
||
|
||
# --- Traffic generation ---
|
||
|
||
Send Fast Requests
|
||
[Arguments] ${client} ${server_ip} ${count}
|
||
FOR ${i} IN RANGE ${count}
|
||
Docker Exec ${client} curl -sf http://${server_ip}:8080/
|
||
END
|
||
|
||
Send Fast Requests v6
|
||
[Arguments] ${client} ${server_ip} ${count}
|
||
FOR ${i} IN RANGE ${count}
|
||
Docker Exec ${client} curl -sf http://[${server_ip}]:8080/
|
||
END
|
||
|
||
Send Slow Requests
|
||
[Arguments] ${client} ${server_ip} ${count}
|
||
FOR ${i} IN RANGE ${count}
|
||
Docker Exec ${client} curl -sf http://${server_ip}:8080/slow
|
||
END
|
||
|
||
Wait For Flush
|
||
Sleep 2s
|
||
|
||
# --- Scraping ---
|
||
|
||
Scrape Prometheus
|
||
${rc} ${output} = Run And Return Rc And Output
|
||
... curl -sf ${SCRAPE_URL}
|
||
Should Be Equal As Integers ${rc} 0
|
||
RETURN ${output}
|
||
|
||
Scrape With Filter
|
||
[Arguments] ${filter}
|
||
${rc} ${output} = Run And Return Rc And Output
|
||
... curl -sf '${SCRAPE_URL}?${filter}'
|
||
Should Be Equal As Integers ${rc} 0
|
||
RETURN ${output}
|
||
|
||
Get Request Count
|
||
[Arguments] ${source}
|
||
${output} = Scrape With Filter source_tag=${source}
|
||
${matches} = Get Regexp Matches ${output}
|
||
... nginx_ipng_requests_total\\{[^}]*\\}\\s+(\\d+) 1
|
||
${total} = Set Variable 0
|
||
FOR ${m} IN @{matches}
|
||
${total} = Evaluate ${total} + ${m}
|
||
END
|
||
RETURN ${total}
|
||
|
||
Nonzero Gauge Observed
|
||
[Documentation] Fails unless the scrape shows a non-zero value for at
|
||
... least one of {active, reading, writing} on the given
|
||
... (source_tag, vip) pair.
|
||
[Arguments] ${source} ${vip}
|
||
${output} = Scrape With Filter source_tag=${source}&vip=${vip}
|
||
Should Match Regexp ${output}
|
||
... nginx_ipng_(active|reading|writing)\\{[^}]*\\}\\s+[1-9]\\d*
|
||
|
||
Gauges Drained
|
||
[Documentation] Fails unless all three gauges are exactly zero for
|
||
... the given (source_tag, vip) pair. Use with Wait
|
||
... Until Keyword Succeeds to poll until a request burst
|
||
... has fully finalized.
|
||
[Arguments] ${source} ${vip}
|
||
${output} = Scrape With Filter source_tag=${source}&vip=${vip}
|
||
Should Match Regexp ${output} nginx_ipng_active\\{[^}]*\\}\\s+0\\b
|
||
Should Match Regexp ${output} nginx_ipng_reading\\{[^}]*\\}\\s+0\\b
|
||
Should Match Regexp ${output} nginx_ipng_writing\\{[^}]*\\}\\s+0\\b
|
||
|
||
# --- Container helpers ---
|
||
|
||
Docker Exec
|
||
[Arguments] ${container} ${cmd}
|
||
${rc} ${output} = Run And Return Rc And Output
|
||
... docker exec ${container} ${cmd}
|
||
Should Be Equal As Integers ${rc} 0
|
||
RETURN ${output}
|
||
|
||
Docker Exec Ignore Rc
|
||
[Arguments] ${container} ${cmd}
|
||
${rc} ${output} = Run And Return Rc And Output
|
||
... docker exec ${container} ${cmd}
|
||
RETURN ${output}
|