PRE-RELEASE v0.8.2

This commit is contained in:
2026-04-19 22:34:30 +02:00
parent 7ed77f5b22
commit 4b1b412fb7
7 changed files with 475 additions and 14 deletions

View File

@@ -23,7 +23,7 @@ BUILD_DIR := $(CURDIR)/build
# the package version from there directly. The C code picks up VERSION # the package version from there directly. The C code picks up VERSION
# via the generated src/version.h (written by the version-header target # via the generated src/version.h (written by the version-header target
# below and depended on by the module build). # below and depended on by the module build).
VERSION := 0.7.2 VERSION := 0.8.2
NGINX_SRC ?= NGINX_SRC ?=

View File

@@ -7,10 +7,10 @@ nginx on Debian Trixie.
The module attributes every HTTP request to the interface it arrived on, reading the ingress `ifindex` per connection from the The module attributes every HTTP request to the interface it arrived on, reading the ingress `ifindex` per connection from the
kernel's `IP_PKTINFO` / `IPV6_PKTINFO` cmsg. Listening sockets stay plain wildcards, so outgoing packets follow the normal kernel's `IP_PKTINFO` / `IPV6_PKTINFO` cmsg. Listening sockets stay plain wildcards, so outgoing packets follow the normal
routing table — which is what makes this safe for DSR / maglev deployments where the SYN arrives via a GRE tunnel and the routing table — which is what makes this safe for DSR / maglev deployments where the SYN arrives via a GRE tunnel and the
SYN-ACK must leave via the default route. Counters — requests, status codes, bytes, latency histograms — are exposed as SYN-ACK must leave via the default route. Counters — requests, status codes, bytes, latency histograms — plus point-in-time
Prometheus text or JSON from a single HTTP scrape endpoint, filtered per-source. This is useful for any deployment where gauges of requests currently in flight (`active`, `reading`, `writing`) are exposed as Prometheus text or JSON from a single
traffic arrives on distinct interfaces — GRE tunnels, VLANs, bonded links, or plain ethernet — and per-interface observability HTTP scrape endpoint, filtered per-source. This is useful for any deployment where traffic arrives on distinct interfaces —
is needed. GRE tunnels, VLANs, bonded links, or plain ethernet — and per-interface observability is needed.
Without any `device=`/`ipng_source_tag=` parameters, the module still counts and exposes per-VIP traffic under the configurable Without any `device=`/`ipng_source_tag=` parameters, the module still counts and exposes per-VIP traffic under the configurable
default source tag (`direct`), which makes it a useful plain observability module for any nginx host. default source tag (`direct`), which makes it a useful plain observability module for any nginx host.

19
debian/changelog vendored
View File

@@ -1,3 +1,22 @@
nginx-ipng-stats-plugin (0.8.2-1) unstable; urgency=medium
* Pre-release v0.8.2.
- New per (source_tag, vip) in-flight gauges:
nginx_ipng_active, nginx_ipng_reading, nginx_ipng_writing.
Lifecycle: POST_READ handler increments active+reading
on each main request, a header filter transitions
reading->writing when headers are sent, and a pool
cleanup decrements on request finalization. Gauges live
in a dedicated rbtree in the shared zone; the slab mutex
is taken only on first insert per (source, vip) pair —
subsequent transitions are lock-free atomic inc/dec on
the cached node. Subrequests and internal redirects do
not double-count. Gauges are emitted in both Prometheus
(`gauge` typed) and JSON (`"gauges":{...}` per record)
scrape outputs. Invariant: reading + writing = active.
-- Pim van Pelt <pim@ipng.ch> Thu, 23 Apr 2026 12:00:00 +0200
nginx-ipng-stats-plugin (0.7.2-1) unstable; urgency=medium nginx-ipng-stats-plugin (0.7.2-1) unstable; urgency=medium
* Pre-release v0.7.2. * Pre-release v0.7.2.

View File

@@ -128,6 +128,12 @@ Each requirement carries a unique identifier (`FR-X.Y` or `NFR-X.Y`) so that lat
codes are observed. Operators who need a full per-code breakdown SHOULD enable `ipng_stats_logtail` (FR-8) and derive the per-code codes are observed. Operators who need a full per-code breakdown SHOULD enable `ipng_stats_logtail` (FR-8) and derive the per-code
view from the access-log stream off the hot path; the stats zone intentionally trades that resolution for a much smaller scrape view from the access-log stream off the hot path; the stats zone intentionally trades that resolution for a much smaller scrape
response. response.
- **FR-2.7** The module MUST additionally maintain, per `(source, vip)` pair, three point-in-time gauges of requests currently in
flight: `active` (observed at `POST_READ` but not yet finalized), `reading` (in the pre-response phases — rewrite/access/content),
and `writing` (past the header-send transition). The invariant `reading + writing = active` MUST hold at any instant. Subrequests
and internal redirects MUST NOT double-count the parent request. Updates to the gauges are atomic increments/decrements on the
request lifecycle hooks — no slab lock after the first time a `(source, vip)` pair is seen — so the hot-path rule in FR-4.1 still
holds for ordinary counter updates while gauges are maintained lock-free.
**FR-3 Scrape endpoint** **FR-3 Scrape endpoint**
@@ -147,6 +153,8 @@ Each requirement carries a unique identifier (`FR-X.Y` or `NFR-X.Y`) so that lat
`source_tag` and `vip`. Counter metrics (`nginx_ipng_requests_total`, `nginx_ipng_bytes_{in,out}_total`, `nginx_ipng_latency_total`) `source_tag` and `vip`. Counter metrics (`nginx_ipng_requests_total`, `nginx_ipng_bytes_{in,out}_total`, `nginx_ipng_latency_total`)
additionally carry a `code` label with a class value (`1xx`..`5xx`/`unknown`). Histogram series (duration, upstream response, additionally carry a `code` label with a class value (`1xx`..`5xx`/`unknown`). Histogram series (duration, upstream response,
request/response byte size) MUST NOT carry a `code` label — they aggregate across all classes for a given `(source, vip)` pair. request/response byte size) MUST NOT carry a `code` label — they aggregate across all classes for a given `(source, vip)` pair.
Gauge series (`nginx_ipng_active`, `nginx_ipng_reading`, `nginx_ipng_writing`) MUST be labelled with `source_tag` and `vip` only
(no `code`) and MUST be typed as `gauge` in the exposition preamble.
**FR-4 Hot path and flush** **FR-4 Hot path and flush**
@@ -407,6 +415,12 @@ Histogram lanes are kept per `(source, vip, class)` in storage, then summed acro
A parallel table keyed by `(source_id, vip_id)` — one row per VIP — holds the EWMAs for instantaneous rate. EWMAs are floats but updated A parallel table keyed by `(source_id, vip_id)` — one row per VIP — holds the EWMAs for instantaneous rate. EWMAs are floats but updated
only from the flush tick, so there is no float contention on the request path. only from the flush tick, so there is no float contention on the request path.
A second, smaller rbtree lives alongside the counter tree — one node per `(source_id, vip_id)` pair — holding three atomic gauge
lanes (`active`, `reading`, `writing`; FR-2.7). Unlike the counter path, gauges are updated from request lifecycle hooks
(`POST_READ`, header filter, pool cleanup) with atomic inc/dec directly on the shared node. The slab mutex is taken only the first
time a `(source, vip)` pair is seen; subsequent transitions on that pair are lock-free. Gauge nodes are never evicted — their
cardinality equals the number of distinct `(source, vip)` pairs and is small in practice.
The module also keeps a small string interning table for source and VIP strings, keyed by the integer IDs above, so that the scrape The module also keeps a small string interning table for source and VIP strings, keyed by the integer IDs above, so that the scrape
endpoint can recover the original strings without re-parsing configuration. endpoint can recover the original strings without re-parsing configuration.

View File

@@ -249,6 +249,14 @@ nginx_ipng_bytes_in_total{source_tag="mg1",vip="192.0.2.10",code="2xx"} 9876543
# Histogram series (request_duration, upstream_response, bytes_in, bytes_out) # Histogram series (request_duration, upstream_response, bytes_in, bytes_out)
# do NOT carry a `code` label — they aggregate across classes per (source, vip). # do NOT carry a `code` label — they aggregate across classes per (source, vip).
nginx_ipng_request_duration_seconds_bucket{source_tag="mg1",vip="192.0.2.10",le="0.050"} 11200 nginx_ipng_request_duration_seconds_bucket{source_tag="mg1",vip="192.0.2.10",le="0.050"} 11200
# In-flight gauges per (source, vip). These are point-in-time request counts,
# not rates: `active` = requests observed at POST_READ that haven't finalized
# yet; `reading` = in pre-response phases (rewrite/access/content); `writing`
# = past header send. reading + writing = active at any instant.
nginx_ipng_active{source_tag="mg1",vip="192.0.2.10"} 3
nginx_ipng_reading{source_tag="mg1",vip="192.0.2.10"} 1
nginx_ipng_writing{source_tag="mg1",vip="192.0.2.10"} 2
``` ```
For JSON output instead, set the `Accept` header: For JSON output instead, set the `Accept` header:
@@ -300,6 +308,11 @@ sum by (vip) (rate(nginx_ipng_requests_total[5m]))
# p95 request duration per (source_tag, vip): # p95 request duration per (source_tag, vip):
histogram_quantile(0.95, histogram_quantile(0.95,
sum by (source_tag, vip, le) (rate(nginx_ipng_request_duration_seconds_bucket[5m]))) sum by (source_tag, vip, le) (rate(nginx_ipng_request_duration_seconds_bucket[5m])))
# In-flight concurrency per (source_tag, vip). Gauges are exported as-is;
# use max_over_time for load-shedding alerts or avg_over_time for capacity
# planning:
max_over_time(nginx_ipng_active[5m])
``` ```
## 6. Set up a global logtail access log ## 6. Set up a global logtail access log

View File

@@ -159,6 +159,34 @@ typedef struct ngx_http_ipng_stats_slot_s {
} ngx_http_ipng_stats_slot_t; } ngx_http_ipng_stats_slot_t;
/* In-flight gauge node: one per (source_id, vip_id) pair, tracking the
* count of requests currently in each lifecycle phase. Inserted under
* the slab mutex the first time a pair is observed, then cached in the
* per-request ctx so transitions touch only the atomic lanes — no
* locking on the hot path after the first hit for a given pair.
*
* Never evicted: one node per distinct (source, vip) pair is small in
* practice (tens) and the node is cheap. */
typedef struct {
ngx_rbtree_node_t rbnode; /* key = hash of (source_id, vip_id) */
ngx_queue_t lru; /* for iteration at render time */
ngx_uint_t source_id;
ngx_uint_t vip_id;
ngx_atomic_uint_t active;
ngx_atomic_uint_t reading;
ngx_atomic_uint_t writing;
} ngx_http_ipng_stats_gauge_t;
/* Per-request state carried through the lifecycle hooks. Allocated at
* POST_READ, transitioned by the header filter, decremented by the
* pool cleanup. `state` is 1 = reading, 2 = writing, 0 = not tracked. */
typedef struct {
ngx_http_ipng_stats_gauge_t *gauge;
ngx_uint_t state;
} ngx_http_ipng_stats_ctx_t;
/* String interning tables live at the head of the shared-memory zone. /* String interning tables live at the head of the shared-memory zone.
* They're flat arrays of ngx_str_t whose data pointers reference memory * They're flat arrays of ngx_str_t whose data pointers reference memory
* allocated from the zone's slab pool. Workers look up strings by * allocated from the zone's slab pool. Workers look up strings by
@@ -179,6 +207,13 @@ typedef struct {
ngx_http_ipng_stats_intern_t sources; ngx_http_ipng_stats_intern_t sources;
ngx_http_ipng_stats_intern_t vips; ngx_http_ipng_stats_intern_t vips;
/* In-flight gauges, keyed by (source_id, vip_id). Atomically
* updated from the request lifecycle hooks; snapshotted by the
* scrape handler under the slab mutex. */
ngx_rbtree_t gauge_rbtree;
ngx_rbtree_node_t gauge_sentinel;
ngx_queue_t gauge_lru;
/* Meta-counters for the plugin itself (FR-6 observability of /* Meta-counters for the plugin itself (FR-6 observability of
* the plugin in the design doc). */ * the plugin in the design doc). */
ngx_atomic_uint_t zone_full_events; ngx_atomic_uint_t zone_full_events;
@@ -328,6 +363,12 @@ static void ngx_http_ipng_stats_exit_worker(ngx_cycle_t *cycle);
static void ngx_http_ipng_stats_rescan_timer(ngx_event_t *ev); static void ngx_http_ipng_stats_rescan_timer(ngx_event_t *ev);
static ngx_int_t ngx_http_ipng_stats_log_handler(ngx_http_request_t *r); static ngx_int_t ngx_http_ipng_stats_log_handler(ngx_http_request_t *r);
static ngx_int_t ngx_http_ipng_stats_post_read_handler(ngx_http_request_t *r);
static ngx_int_t ngx_http_ipng_stats_header_filter(ngx_http_request_t *r);
static void ngx_http_ipng_stats_ctx_cleanup(void *data);
static ngx_http_ipng_stats_gauge_t *ngx_http_ipng_stats_gauge_get(
ngx_http_ipng_stats_shctx_t *sh, ngx_slab_pool_t *slab,
ngx_uint_t source_id, ngx_uint_t vip_id);
static ngx_int_t ngx_http_ipng_stats_content_handler(ngx_http_request_t *r); static ngx_int_t ngx_http_ipng_stats_content_handler(ngx_http_request_t *r);
static void ngx_http_ipng_stats_flush_timer(ngx_event_t *ev); static void ngx_http_ipng_stats_flush_timer(ngx_event_t *ev);
@@ -365,6 +406,8 @@ static char *(*ngx_http_core_listen_orig)(ngx_conf_t *cf,
static ngx_http_ipng_stats_worker_t ngx_http_ipng_stats_worker; static ngx_http_ipng_stats_worker_t ngx_http_ipng_stats_worker;
static ngx_http_output_header_filter_pt ngx_http_ipng_stats_next_header_filter;
extern ngx_module_t ngx_http_core_module; extern ngx_module_t ngx_http_core_module;
@@ -1200,7 +1243,7 @@ ngx_http_ipng_stats_logtail(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
/* ----------------------------------------------------------------- */ /* ----------------------------------------------------------------- */
/* Postconfig: install log-phase handler */ /* Postconfig: install phase handlers and header filter */
/* ----------------------------------------------------------------- */ /* ----------------------------------------------------------------- */
static ngx_int_t static ngx_int_t
@@ -1240,6 +1283,22 @@ ngx_http_ipng_stats_postconfig(ngx_conf_t *cf)
} }
*h = ngx_http_ipng_stats_log_handler; *h = ngx_http_ipng_stats_log_handler;
/* POST_READ is the earliest phase at which the request is parsed
* enough to resolve source and vip; we register the in-flight
* gauge there so `reading` covers rewrite/access/content. */
h = ngx_array_push(&cmcf->phases[NGX_HTTP_POST_READ_PHASE].handlers);
if (h == NULL) {
return NGX_ERROR;
}
*h = ngx_http_ipng_stats_post_read_handler;
/* Header filter: transitions reading -> writing when nginx starts
* sending the response. Inserted at the top of the chain so the
* transition is observed before any downstream filter mutates the
* response. */
ngx_http_ipng_stats_next_header_filter = ngx_http_top_header_filter;
ngx_http_top_header_filter = ngx_http_ipng_stats_header_filter;
return NGX_OK; return NGX_OK;
} }
@@ -1313,6 +1372,10 @@ ngx_http_ipng_stats_init_zone(ngx_shm_zone_t *shm_zone, void *data)
ngx_http_ipng_stats_rbtree_insert); ngx_http_ipng_stats_rbtree_insert);
ngx_queue_init(&sh->lru); ngx_queue_init(&sh->lru);
ngx_rbtree_init(&sh->gauge_rbtree, &sh->gauge_sentinel,
ngx_http_ipng_stats_rbtree_insert);
ngx_queue_init(&sh->gauge_lru);
sh->sources.nalloc = 16; sh->sources.nalloc = 16;
sh->sources.entries = ngx_slab_alloc(slab, sh->sources.entries = ngx_slab_alloc(slab,
sh->sources.nalloc * sizeof(ngx_str_t)); sh->sources.nalloc * sizeof(ngx_str_t));
@@ -2116,6 +2179,141 @@ ngx_http_ipng_stats_log_handler(ngx_http_request_t *r)
} }
/* ----------------------------------------------------------------- */
/* In-flight gauge lifecycle hooks */
/* ----------------------------------------------------------------- */
/* POST_READ phase handler: first point at which the request is fully
* parsed enough to resolve source and vip. Finds or creates the
* gauge node, increments active + reading, and registers a pool
* cleanup that decrements on request finalization. Skips subrequests
* and internal redirects (the gauge is already ticking on r->main). */
static ngx_int_t
ngx_http_ipng_stats_post_read_handler(ngx_http_request_t *r)
{
ngx_http_ipng_stats_main_conf_t *imcf;
ngx_http_ipng_stats_loc_conf_t *ilcf;
ngx_http_ipng_stats_shctx_t *sh;
ngx_slab_pool_t *slab;
ngx_http_ipng_stats_ctx_t *ctx;
ngx_http_ipng_stats_gauge_t *g;
ngx_pool_cleanup_t *cln;
ngx_str_t source, vip;
u_char vipbuf[NGX_SOCKADDR_STRLEN];
ngx_uint_t source_id, vip_id;
if (r != r->main || r->internal) {
return NGX_DECLINED;
}
imcf = ngx_http_get_module_main_conf(r, ngx_http_ipng_stats_module);
if (imcf == NULL || imcf->shm_zone == NULL || !imcf->enabled) {
return NGX_DECLINED;
}
ilcf = ngx_http_get_module_loc_conf(r, ngx_http_ipng_stats_module);
if (ilcf == NULL || !ilcf->enabled) {
return NGX_DECLINED;
}
if (ngx_http_ipng_stats_resolve_source(r, imcf, &source) != NGX_OK) {
return NGX_DECLINED;
}
if (ngx_connection_local_sockaddr(r->connection, NULL, 0) != NGX_OK) {
return NGX_DECLINED;
}
if (ngx_http_ipng_stats_canonical_vip(r, vipbuf, sizeof(vipbuf), &vip)
!= NGX_OK)
{
return NGX_DECLINED;
}
slab = (ngx_slab_pool_t *) imcf->shm_zone->shm.addr;
sh = imcf->shm_zone->data;
ngx_shmtx_lock(&slab->mutex);
if (ngx_http_ipng_stats_intern_shared(sh, slab, &sh->sources, &source,
&source_id) != NGX_OK
|| ngx_http_ipng_stats_intern_shared(sh, slab, &sh->vips, &vip,
&vip_id) != NGX_OK)
{
ngx_shmtx_unlock(&slab->mutex);
return NGX_DECLINED;
}
ngx_shmtx_unlock(&slab->mutex);
g = ngx_http_ipng_stats_gauge_get(sh, slab, source_id, vip_id);
if (g == NULL) {
return NGX_DECLINED;
}
ctx = ngx_pcalloc(r->pool, sizeof(*ctx));
cln = ngx_pool_cleanup_add(r->pool, 0);
if (ctx == NULL || cln == NULL) {
return NGX_DECLINED;
}
ctx->gauge = g;
ctx->state = 1; /* reading */
cln->handler = ngx_http_ipng_stats_ctx_cleanup;
cln->data = ctx;
ngx_http_set_ctx(r, ctx, ngx_http_ipng_stats_module);
(void) ngx_atomic_fetch_add(&g->active, 1);
(void) ngx_atomic_fetch_add(&g->reading, 1);
return NGX_DECLINED;
}
/* Header filter: first call for the main request transitions reading
* -> writing. Subrequests have their own header-filter invocations;
* we ignore those so a subrequest doesn't prematurely flip the main
* request's gauge state. Internal redirects re-enter the filter
* chain; the state check prevents double transitions. */
static ngx_int_t
ngx_http_ipng_stats_header_filter(ngx_http_request_t *r)
{
ngx_http_ipng_stats_ctx_t *ctx;
if (r == r->main) {
ctx = ngx_http_get_module_ctx(r, ngx_http_ipng_stats_module);
if (ctx != NULL && ctx->gauge != NULL && ctx->state == 1) {
(void) ngx_atomic_fetch_add(&ctx->gauge->reading,
(ngx_atomic_uint_t) -1);
(void) ngx_atomic_fetch_add(&ctx->gauge->writing, 1);
ctx->state = 2;
}
}
return ngx_http_ipng_stats_next_header_filter(r);
}
static void
ngx_http_ipng_stats_ctx_cleanup(void *data)
{
ngx_http_ipng_stats_ctx_t *ctx = data;
if (ctx->gauge == NULL) {
return;
}
if (ctx->state == 1) {
(void) ngx_atomic_fetch_add(&ctx->gauge->reading,
(ngx_atomic_uint_t) -1);
} else if (ctx->state == 2) {
(void) ngx_atomic_fetch_add(&ctx->gauge->writing,
(ngx_atomic_uint_t) -1);
}
(void) ngx_atomic_fetch_add(&ctx->gauge->active,
(ngx_atomic_uint_t) -1);
ctx->gauge = NULL;
ctx->state = 0;
}
/* ----------------------------------------------------------------- */ /* ----------------------------------------------------------------- */
/* String interning (called under slab mutex) */ /* String interning (called under slab mutex) */
/* ----------------------------------------------------------------- */ /* ----------------------------------------------------------------- */
@@ -2170,6 +2368,66 @@ ngx_http_ipng_stats_intern_shared(ngx_http_ipng_stats_shctx_t *sh,
} }
/* ----------------------------------------------------------------- */
/* In-flight gauges */
/* ----------------------------------------------------------------- */
/* Hash of (source_id, vip_id). Separate constants from the counter
* rbtree's hash so collisions don't line up between the two trees. */
static ngx_inline ngx_uint_t
ngx_http_ipng_stats_gauge_hash(ngx_uint_t source_id, ngx_uint_t vip_id)
{
return (source_id * 2246822519u) ^ (vip_id * 3266489917u);
}
/* Find or create the gauge node for (source_id, vip_id). Takes the
* slab mutex only when a new node must be inserted; the caller holds
* the returned pointer for the rest of the request and does lock-free
* atomic inc/dec on the lanes. Returns NULL on slab exhaustion. */
static ngx_http_ipng_stats_gauge_t *
ngx_http_ipng_stats_gauge_get(ngx_http_ipng_stats_shctx_t *sh,
ngx_slab_pool_t *slab, ngx_uint_t source_id, ngx_uint_t vip_id)
{
ngx_uint_t hash;
ngx_rbtree_node_t *rb;
ngx_http_ipng_stats_gauge_t *g = NULL;
hash = ngx_http_ipng_stats_gauge_hash(source_id, vip_id);
ngx_shmtx_lock(&slab->mutex);
rb = sh->gauge_rbtree.root;
while (rb != &sh->gauge_sentinel) {
if (hash < rb->key) { rb = rb->left; continue; }
if (hash > rb->key) { rb = rb->right; continue; }
g = (ngx_http_ipng_stats_gauge_t *) rb;
if (g->source_id == source_id && g->vip_id == vip_id) {
break;
}
rb = rb->right;
g = NULL;
}
if (g == NULL) {
g = ngx_slab_calloc_locked(slab, sizeof(*g));
if (g == NULL) {
(void) ngx_atomic_fetch_add(&sh->zone_full_events, 1);
ngx_shmtx_unlock(&slab->mutex);
return NULL;
}
g->rbnode.key = hash;
g->source_id = source_id;
g->vip_id = vip_id;
ngx_rbtree_insert(&sh->gauge_rbtree, &g->rbnode);
ngx_queue_insert_tail(&sh->gauge_lru, &g->lru);
}
ngx_shmtx_unlock(&slab->mutex);
return g;
}
/* ----------------------------------------------------------------- */ /* ----------------------------------------------------------------- */
/* Global logtail: write + flush */ /* Global logtail: write + flush */
/* ----------------------------------------------------------------- */ /* ----------------------------------------------------------------- */
@@ -2461,6 +2719,9 @@ typedef struct {
uint64_t bytes_out_sum; uint64_t bytes_out_sum;
uint64_t req_total; /* total requests for this (source, vip) */ uint64_t req_total; /* total requests for this (source, vip) */
uint64_t up_total; /* upstream observations */ uint64_t up_total; /* upstream observations */
uint64_t active; /* in-flight gauges (request lifecycle) */
uint64_t reading;
uint64_t writing;
uint64_t *dhist; /* nbuckets+1 */ uint64_t *dhist; /* nbuckets+1 */
uint64_t *uhist; uint64_t *uhist;
uint64_t *bin_hist; /* nbytebuckets+1 */ uint64_t *bin_hist; /* nbytebuckets+1 */
@@ -2737,6 +2998,60 @@ ngx_http_ipng_stats_snapshot_nodes(ngx_http_ipng_stats_shctx_t *sh,
} }
/* Walk the gauge rbtree under the slab mutex and fold in-flight gauge
* values into the aggregation table. Creates new agg entries for
* (source, vip) pairs that have in-flight requests but no completed
* ones yet, so scrapes reflect load even before the first log-phase
* update for a pair. Respects the same filter semantics as the
* counter walk. */
static void
ngx_http_ipng_stats_snapshot_gauges(ngx_http_ipng_stats_shctx_t *sh,
ngx_str_t *filter_src, ngx_str_t *filter_vip,
ngx_str_t *src_tbl, ngx_uint_t n_src,
ngx_str_t *vip_tbl, ngx_uint_t n_vip,
ngx_http_ipng_stats_agg_t *aggs, ngx_uint_t naggs_alloc,
ngx_uint_t *naggs_io)
{
ngx_queue_t *q;
ngx_http_ipng_stats_gauge_t *g;
ngx_str_t *src_entry, *vip_entry;
ngx_http_ipng_stats_agg_t *a;
for (q = ngx_queue_head(&sh->gauge_lru);
q != ngx_queue_sentinel(&sh->gauge_lru);
q = ngx_queue_next(q))
{
g = ngx_queue_data(q, ngx_http_ipng_stats_gauge_t, lru);
if (g->source_id >= n_src || g->vip_id >= n_vip) continue;
src_entry = &src_tbl[g->source_id];
vip_entry = &vip_tbl[g->vip_id];
if (filter_src->len > 0
&& (src_entry->len != filter_src->len
|| ngx_memcmp(src_entry->data, filter_src->data,
filter_src->len) != 0))
{
continue;
}
if (filter_vip->len > 0
&& (vip_entry->len != filter_vip->len
|| ngx_memcmp(vip_entry->data, filter_vip->data,
filter_vip->len) != 0))
{
continue;
}
a = ngx_http_ipng_stats_agg_get(aggs, naggs_io, naggs_alloc,
g->source_id, g->vip_id);
if (a == NULL) continue;
a->active = g->active;
a->reading = g->reading;
a->writing = g->writing;
}
}
/* -- Prometheus ---------------------------------------------------- */ /* -- Prometheus ---------------------------------------------------- */
@@ -2817,6 +3132,13 @@ ngx_http_ipng_stats_render_prom(ngx_http_request_t *r,
"# TYPE nginx_ipng_bytes_in histogram\n" "# TYPE nginx_ipng_bytes_in histogram\n"
"# HELP nginx_ipng_bytes_out Request size histogram in bytes.\n" "# HELP nginx_ipng_bytes_out Request size histogram in bytes.\n"
"# TYPE nginx_ipng_bytes_out histogram\n" "# TYPE nginx_ipng_bytes_out histogram\n"
"# HELP nginx_ipng_active Requests currently in flight.\n"
"# TYPE nginx_ipng_active gauge\n"
"# HELP nginx_ipng_reading In-flight requests in the pre-response "
"phases (rewrite/access/content).\n"
"# TYPE nginx_ipng_reading gauge\n"
"# HELP nginx_ipng_writing In-flight requests past header send.\n"
"# TYPE nginx_ipng_writing gauge\n"
"# HELP nginx_ipng_ifindex_misses_total Connections whose ingress " "# HELP nginx_ipng_ifindex_misses_total Connections whose ingress "
"ifindex did not match any configured device= binding.\n" "ifindex did not match any configured device= binding.\n"
"# TYPE nginx_ipng_ifindex_misses_total counter\n" "# TYPE nginx_ipng_ifindex_misses_total counter\n"
@@ -2891,6 +3213,10 @@ ngx_http_ipng_stats_render_prom(ngx_http_request_t *r,
snaps, nsnaps_alloc, &nsnaps, snaps, nsnaps_alloc, &nsnaps,
aggs, naggs_alloc, &naggs); aggs, naggs_alloc, &naggs);
ngx_http_ipng_stats_snapshot_gauges(sh, filter_source, filter_vip,
src_tbl, n_src, vip_tbl, n_vip,
aggs, naggs_alloc, &naggs);
ngx_shmtx_unlock(&slab->mutex); ngx_shmtx_unlock(&slab->mutex);
/* Per-node counters. */ /* Per-node counters. */
@@ -2915,9 +3241,11 @@ ngx_http_ipng_stats_render_prom(ngx_http_request_t *r,
} }
} }
/* One chain link per (source, vip) for the four aggregated histograms. /* One chain link per (source, vip) for the four aggregated
* Size: per-bucket line ~96B, + sum/count/+Inf per metric ~96B each. */ * histograms plus the three in-flight gauges. Size: per-bucket
hist_sz = 256 + 96 * (2 * (nb + 1) + 2 * (nbb + 1)) + 4 * 200; * line ~96B, sum/count/+Inf per metric ~96B each, three gauge
* lines ~80B each. */
hist_sz = 512 + 96 * (2 * (nb + 1) + 2 * (nbb + 1)) + 4 * 200;
for (i = 0; i < naggs; i++) { for (i = 0; i < naggs; i++) {
ngx_http_ipng_stats_agg_t *a = &aggs[i]; ngx_http_ipng_stats_agg_t *a = &aggs[i];
@@ -2944,6 +3272,13 @@ ngx_http_ipng_stats_render_prom(ngx_http_request_t *r,
"nginx_ipng_bytes_out", src, vip, "nginx_ipng_bytes_out", src, vip,
imcf->byte_bucket_bounds, nbb, a->bout_hist, imcf->byte_bucket_bounds, nbb, a->bout_hist,
(double) a->bytes_out_sum, 0); (double) a->bytes_out_sum, 0);
p = ngx_sprintf(p,
"nginx_ipng_active{source_tag=\"%V\",vip=\"%V\"} %uL\n"
"nginx_ipng_reading{source_tag=\"%V\",vip=\"%V\"} %uL\n"
"nginx_ipng_writing{source_tag=\"%V\",vip=\"%V\"} %uL\n",
src, vip, a->active,
src, vip, a->reading,
src, vip, a->writing);
cl->buf->last = p; cl->buf->last = p;
if (ngx_http_ipng_stats_append(&last, cl) != NGX_OK) { if (ngx_http_ipng_stats_append(&last, cl) != NGX_OK) {
return NGX_HTTP_INTERNAL_SERVER_ERROR; return NGX_HTTP_INTERNAL_SERVER_ERROR;
@@ -3045,11 +3380,15 @@ ngx_http_ipng_stats_render_json(ngx_http_request_t *r,
snaps, nsnaps_alloc, &nsnaps, snaps, nsnaps_alloc, &nsnaps,
aggs, naggs_alloc, &naggs); aggs, naggs_alloc, &naggs);
ngx_http_ipng_stats_snapshot_gauges(sh, filter_source, filter_vip,
src_tbl, n_src, vip_tbl, n_vip,
aggs, naggs_alloc, &naggs);
ngx_shmtx_unlock(&slab->mutex); ngx_shmtx_unlock(&slab->mutex);
/* One JSON record per aggregated (source, vip). Size upper-bound /* One JSON record per aggregated (source, vip). Size upper-bound
* accounts for: fixed overhead, up to NCLASSES class entries, 4 * accounts for: fixed overhead, up to NCLASSES class entries, 4
* histograms. */ * histograms, 3 gauges. */
rec_sz = 512 rec_sz = 512
+ 160 * NGX_HTTP_IPNG_STATS_NCLASSES + 160 * NGX_HTTP_IPNG_STATS_NCLASSES
+ 48 * (2 * (nb + 1) + 2 * (nbb + 1)) + 48 * (2 * (nb + 1) + 2 * (nbb + 1))
@@ -3131,7 +3470,9 @@ ngx_http_ipng_stats_render_json(ngx_http_request_t *r,
imcf->byte_bucket_bounds[j], a->bout_hist[j]); imcf->byte_bucket_bounds[j], a->bout_hist[j]);
} }
} }
p = ngx_sprintf(p, "}}}"); p = ngx_sprintf(p,
"}},\"gauges\":{\"active\":%uL,\"reading\":%uL,\"writing\":%uL}}",
a->active, a->reading, a->writing);
cl->buf->last = p; cl->buf->last = p;
if (ngx_http_ipng_stats_append(&last, cl) != NGX_OK) { if (ngx_http_ipng_stats_append(&last, cl) != NGX_OK) {

View File

@@ -49,9 +49,16 @@ Shared-listen-include across multiple server blocks
Length Should Be ${count} 2 Length Should Be ${count} 2
... Expected 2 listening sockets on port 8080 (v4+v6 wildcards); got ${count} ... Expected 2 listening sockets on port 8080 (v4+v6 wildcards); got ${count}
# Proves the cross-cscf option-stripping path actually fired for # Proves the cross-cscf option-stripping path actually fired for
# the 2nd and 3rd server blocks. `nginx -t` replays the whole # the 2nd and 3rd server blocks at daemon startup — the wrapper
# config and emits the wrapper's NOTICE each time it strips. # logs a NOTICE per stripped listen via ngx_conf_log_error. We
Should Contain ${output} stripped socket options from duplicate listen # read the container's startup log rather than `nginx -t`'s
# output because nginx keeps the config-parse log level at ERR
# until the error_log directive has been fully applied, so
# `nginx -t` suppresses the NOTICE even though the wrapper is
# invoked.
${rc} ${startup_log} = Run And Return Rc And Output
... docker logs ${SERVER} 2>&1
Should Contain ${startup_log} stripped socket options from duplicate listen
Prometheus scrape Prometheus scrape
[Documentation] Scrape returns HELP/TYPE preamble. [Documentation] Scrape returns HELP/TYPE preamble.
@@ -139,6 +146,53 @@ Duration histogram
Should Contain ${json} request_duration_ms Should Contain ${json} request_duration_ms
Should Contain ${json} buckets Should Contain ${json} buckets
# --- In-flight gauges ---
Gauge preamble in Prometheus output
[Documentation] The scrape preamble advertises the three gauge metric
... names and types. Present whether or not any traffic
... has ever been observed.
${output} = Scrape Prometheus
Should Match Regexp ${output} (?m)^# TYPE nginx_ipng_active gauge$
Should Match Regexp ${output} (?m)^# TYPE nginx_ipng_reading gauge$
Should Match Regexp ${output} (?m)^# TYPE nginx_ipng_writing gauge$
Gauges zero at rest
[Documentation] After the earlier traffic tests have drained and the
... flush tick has run, the in-flight gauges for (tag1,
... 10.0.1.1) must all report zero — no request is still
... resident in the plugin's per-request ctx.
Wait For Flush
${output} = Scrape With Filter source_tag=tag1&vip=10.0.1.1
Should Match Regexp ${output} nginx_ipng_active\\{[^}]*\\}\\s+0\\b
Should Match Regexp ${output} nginx_ipng_reading\\{[^}]*\\}\\s+0\\b
Should Match Regexp ${output} nginx_ipng_writing\\{[^}]*\\}\\s+0\\b
Gauges appear in JSON output
[Documentation] Each record in the JSON output carries a gauges object
... with active/reading/writing keys.
${rc} ${output} = Run And Return Rc And Output
... curl -sf -H 'Accept: application/json' '${SCRAPE_URL}?source_tag=tag1&vip=10.0.1.1' | python3 -m json.tool
Should Be Equal As Integers ${rc} 0
Should Contain ${output} "gauges"
Should Contain ${output} "active"
Should Contain ${output} "reading"
Should Contain ${output} "writing"
Gauges observed non-zero under concurrent load
[Documentation] Fire a burst of concurrent requests against the
... /slow backend (50 ms each, single-threaded) and
... observe at least one of active/reading/writing
... going non-zero for (tag1, 10.0.1.1) while the
... burst is in flight. After it drains the gauges
... return to zero. Uses 20 requests (drain <= ~2s
... given the single-threaded backend) to keep the
... test fast but still wide enough for the scrape
... to catch overlap.
Run docker exec -d ${CLIENT1} sh -c 'for i in $(seq 20); do curl -sf http://10.0.1.1:8080/slow >/dev/null & done; wait'
Wait Until Keyword Succeeds 5s 50ms Nonzero Gauge Observed tag1 10.0.1.1
Wait Until Keyword Succeeds 15s 500ms Gauges Drained tag1 10.0.1.1
# --- Scrape filters --- # --- Scrape filters ---
Filter by source_tag Filter by source_tag
@@ -347,6 +401,26 @@ Get Request Count
END END
RETURN ${total} RETURN ${total}
Nonzero Gauge Observed
[Documentation] Fails unless the scrape shows a non-zero value for at
... least one of {active, reading, writing} on the given
... (source_tag, vip) pair.
[Arguments] ${source} ${vip}
${output} = Scrape With Filter source_tag=${source}&vip=${vip}
Should Match Regexp ${output}
... nginx_ipng_(active|reading|writing)\\{[^}]*\\}\\s+[1-9]\\d*
Gauges Drained
[Documentation] Fails unless all three gauges are exactly zero for
... the given (source_tag, vip) pair. Use with Wait
... Until Keyword Succeeds to poll until a request burst
... has fully finalized.
[Arguments] ${source} ${vip}
${output} = Scrape With Filter source_tag=${source}&vip=${vip}
Should Match Regexp ${output} nginx_ipng_active\\{[^}]*\\}\\s+0\\b
Should Match Regexp ${output} nginx_ipng_reading\\{[^}]*\\}\\s+0\\b
Should Match Regexp ${output} nginx_ipng_writing\\{[^}]*\\}\\s+0\\b
# --- Container helpers --- # --- Container helpers ---
Docker Exec Docker Exec