PRE-RELEASE v0.8.2

This commit is contained in:
2026-04-19 22:34:30 +02:00
parent 7ed77f5b22
commit 4b1b412fb7
7 changed files with 475 additions and 14 deletions

View File

@@ -23,7 +23,7 @@ BUILD_DIR := $(CURDIR)/build
# the package version from there directly. The C code picks up VERSION
# via the generated src/version.h (written by the version-header target
# below and depended on by the module build).
VERSION := 0.7.2
VERSION := 0.8.2
NGINX_SRC ?=

View File

@@ -7,10 +7,10 @@ nginx on Debian Trixie.
The module attributes every HTTP request to the interface it arrived on, reading the ingress `ifindex` per connection from the
kernel's `IP_PKTINFO` / `IPV6_PKTINFO` cmsg. Listening sockets stay plain wildcards, so outgoing packets follow the normal
routing table — which is what makes this safe for DSR / maglev deployments where the SYN arrives via a GRE tunnel and the
SYN-ACK must leave via the default route. Counters — requests, status codes, bytes, latency histograms — are exposed as
Prometheus text or JSON from a single HTTP scrape endpoint, filtered per-source. This is useful for any deployment where
traffic arrives on distinct interfaces — GRE tunnels, VLANs, bonded links, or plain ethernet — and per-interface observability
is needed.
SYN-ACK must leave via the default route. Counters — requests, status codes, bytes, latency histograms — plus point-in-time
gauges of requests currently in flight (`active`, `reading`, `writing`) are exposed as Prometheus text or JSON from a single
HTTP scrape endpoint, filtered per-source. This is useful for any deployment where traffic arrives on distinct interfaces —
GRE tunnels, VLANs, bonded links, or plain ethernet — and per-interface observability is needed.
Without any `device=`/`ipng_source_tag=` parameters, the module still counts and exposes per-VIP traffic under the configurable
default source tag (`direct`), which makes it a useful plain observability module for any nginx host.

19
debian/changelog vendored
View File

@@ -1,3 +1,22 @@
nginx-ipng-stats-plugin (0.8.2-1) unstable; urgency=medium
* Pre-release v0.8.2.
- New per (source_tag, vip) in-flight gauges:
nginx_ipng_active, nginx_ipng_reading, nginx_ipng_writing.
Lifecycle: POST_READ handler increments active+reading
on each main request, a header filter transitions
reading->writing when headers are sent, and a pool
cleanup decrements on request finalization. Gauges live
in a dedicated rbtree in the shared zone; the slab mutex
is taken only on first insert per (source, vip) pair —
subsequent transitions are lock-free atomic inc/dec on
the cached node. Subrequests and internal redirects do
not double-count. Gauges are emitted in both Prometheus
(`gauge` typed) and JSON (`"gauges":{...}` per record)
scrape outputs. Invariant: reading + writing = active.
-- Pim van Pelt <pim@ipng.ch> Thu, 23 Apr 2026 12:00:00 +0200
nginx-ipng-stats-plugin (0.7.2-1) unstable; urgency=medium
* Pre-release v0.7.2.

View File

@@ -128,6 +128,12 @@ Each requirement carries a unique identifier (`FR-X.Y` or `NFR-X.Y`) so that lat
codes are observed. Operators who need a full per-code breakdown SHOULD enable `ipng_stats_logtail` (FR-8) and derive the per-code
view from the access-log stream off the hot path; the stats zone intentionally trades that resolution for a much smaller scrape
response.
- **FR-2.7** The module MUST additionally maintain, per `(source, vip)` pair, three point-in-time gauges of requests currently in
flight: `active` (observed at `POST_READ` but not yet finalized), `reading` (in the pre-response phases — rewrite/access/content),
and `writing` (past the header-send transition). The invariant `reading + writing = active` MUST hold at any instant. Subrequests
and internal redirects MUST NOT double-count the parent request. Updates to the gauges are atomic increments/decrements on the
request lifecycle hooks — no slab lock after the first time a `(source, vip)` pair is seen — so the hot-path rule in FR-4.1 still
holds for ordinary counter updates while gauges are maintained lock-free.
**FR-3 Scrape endpoint**
@@ -147,6 +153,8 @@ Each requirement carries a unique identifier (`FR-X.Y` or `NFR-X.Y`) so that lat
`source_tag` and `vip`. Counter metrics (`nginx_ipng_requests_total`, `nginx_ipng_bytes_{in,out}_total`, `nginx_ipng_latency_total`)
additionally carry a `code` label with a class value (`1xx`..`5xx`/`unknown`). Histogram series (duration, upstream response,
request/response byte size) MUST NOT carry a `code` label — they aggregate across all classes for a given `(source, vip)` pair.
Gauge series (`nginx_ipng_active`, `nginx_ipng_reading`, `nginx_ipng_writing`) MUST be labelled with `source_tag` and `vip` only
(no `code`) and MUST be typed as `gauge` in the exposition preamble.
**FR-4 Hot path and flush**
@@ -407,6 +415,12 @@ Histogram lanes are kept per `(source, vip, class)` in storage, then summed acro
A parallel table keyed by `(source_id, vip_id)` — one row per VIP — holds the EWMAs for instantaneous rate. EWMAs are floats but updated
only from the flush tick, so there is no float contention on the request path.
A second, smaller rbtree lives alongside the counter tree — one node per `(source_id, vip_id)` pair — holding three atomic gauge
lanes (`active`, `reading`, `writing`; FR-2.7). Unlike the counter path, gauges are updated from request lifecycle hooks
(`POST_READ`, header filter, pool cleanup) with atomic inc/dec directly on the shared node. The slab mutex is taken only the first
time a `(source, vip)` pair is seen; subsequent transitions on that pair are lock-free. Gauge nodes are never evicted — their
cardinality equals the number of distinct `(source, vip)` pairs and is small in practice.
The module also keeps a small string interning table for source and VIP strings, keyed by the integer IDs above, so that the scrape
endpoint can recover the original strings without re-parsing configuration.

View File

@@ -249,6 +249,14 @@ nginx_ipng_bytes_in_total{source_tag="mg1",vip="192.0.2.10",code="2xx"} 9876543
# Histogram series (request_duration, upstream_response, bytes_in, bytes_out)
# do NOT carry a `code` label — they aggregate across classes per (source, vip).
nginx_ipng_request_duration_seconds_bucket{source_tag="mg1",vip="192.0.2.10",le="0.050"} 11200
# In-flight gauges per (source, vip). These are point-in-time request counts,
# not rates: `active` = requests observed at POST_READ that haven't finalized
# yet; `reading` = in pre-response phases (rewrite/access/content); `writing`
# = past header send. reading + writing = active at any instant.
nginx_ipng_active{source_tag="mg1",vip="192.0.2.10"} 3
nginx_ipng_reading{source_tag="mg1",vip="192.0.2.10"} 1
nginx_ipng_writing{source_tag="mg1",vip="192.0.2.10"} 2
```
For JSON output instead, set the `Accept` header:
@@ -300,6 +308,11 @@ sum by (vip) (rate(nginx_ipng_requests_total[5m]))
# p95 request duration per (source_tag, vip):
histogram_quantile(0.95,
sum by (source_tag, vip, le) (rate(nginx_ipng_request_duration_seconds_bucket[5m])))
# In-flight concurrency per (source_tag, vip). Gauges are exported as-is;
# use max_over_time for load-shedding alerts or avg_over_time for capacity
# planning:
max_over_time(nginx_ipng_active[5m])
```
## 6. Set up a global logtail access log

View File

@@ -159,6 +159,34 @@ typedef struct ngx_http_ipng_stats_slot_s {
} ngx_http_ipng_stats_slot_t;
/* In-flight gauge node: one per (source_id, vip_id) pair, tracking the
* count of requests currently in each lifecycle phase. Inserted under
* the slab mutex the first time a pair is observed, then cached in the
* per-request ctx so transitions touch only the atomic lanes — no
* locking on the hot path after the first hit for a given pair.
*
* Never evicted: one node per distinct (source, vip) pair is small in
* practice (tens) and the node is cheap. */
typedef struct {
ngx_rbtree_node_t rbnode; /* key = hash of (source_id, vip_id) */
ngx_queue_t lru; /* for iteration at render time */
ngx_uint_t source_id;
ngx_uint_t vip_id;
ngx_atomic_uint_t active;
ngx_atomic_uint_t reading;
ngx_atomic_uint_t writing;
} ngx_http_ipng_stats_gauge_t;
/* Per-request state carried through the lifecycle hooks. Allocated at
* POST_READ, transitioned by the header filter, decremented by the
* pool cleanup. `state` is 1 = reading, 2 = writing, 0 = not tracked. */
typedef struct {
ngx_http_ipng_stats_gauge_t *gauge;
ngx_uint_t state;
} ngx_http_ipng_stats_ctx_t;
/* String interning tables live at the head of the shared-memory zone.
* They're flat arrays of ngx_str_t whose data pointers reference memory
* allocated from the zone's slab pool. Workers look up strings by
@@ -179,6 +207,13 @@ typedef struct {
ngx_http_ipng_stats_intern_t sources;
ngx_http_ipng_stats_intern_t vips;
/* In-flight gauges, keyed by (source_id, vip_id). Atomically
* updated from the request lifecycle hooks; snapshotted by the
* scrape handler under the slab mutex. */
ngx_rbtree_t gauge_rbtree;
ngx_rbtree_node_t gauge_sentinel;
ngx_queue_t gauge_lru;
/* Meta-counters for the plugin itself (FR-6 observability of
* the plugin in the design doc). */
ngx_atomic_uint_t zone_full_events;
@@ -328,6 +363,12 @@ static void ngx_http_ipng_stats_exit_worker(ngx_cycle_t *cycle);
static void ngx_http_ipng_stats_rescan_timer(ngx_event_t *ev);
static ngx_int_t ngx_http_ipng_stats_log_handler(ngx_http_request_t *r);
static ngx_int_t ngx_http_ipng_stats_post_read_handler(ngx_http_request_t *r);
static ngx_int_t ngx_http_ipng_stats_header_filter(ngx_http_request_t *r);
static void ngx_http_ipng_stats_ctx_cleanup(void *data);
static ngx_http_ipng_stats_gauge_t *ngx_http_ipng_stats_gauge_get(
ngx_http_ipng_stats_shctx_t *sh, ngx_slab_pool_t *slab,
ngx_uint_t source_id, ngx_uint_t vip_id);
static ngx_int_t ngx_http_ipng_stats_content_handler(ngx_http_request_t *r);
static void ngx_http_ipng_stats_flush_timer(ngx_event_t *ev);
@@ -365,6 +406,8 @@ static char *(*ngx_http_core_listen_orig)(ngx_conf_t *cf,
static ngx_http_ipng_stats_worker_t ngx_http_ipng_stats_worker;
static ngx_http_output_header_filter_pt ngx_http_ipng_stats_next_header_filter;
extern ngx_module_t ngx_http_core_module;
@@ -1200,7 +1243,7 @@ ngx_http_ipng_stats_logtail(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
/* ----------------------------------------------------------------- */
/* Postconfig: install log-phase handler */
/* Postconfig: install phase handlers and header filter */
/* ----------------------------------------------------------------- */
static ngx_int_t
@@ -1240,6 +1283,22 @@ ngx_http_ipng_stats_postconfig(ngx_conf_t *cf)
}
*h = ngx_http_ipng_stats_log_handler;
/* POST_READ is the earliest phase at which the request is parsed
* enough to resolve source and vip; we register the in-flight
* gauge there so `reading` covers rewrite/access/content. */
h = ngx_array_push(&cmcf->phases[NGX_HTTP_POST_READ_PHASE].handlers);
if (h == NULL) {
return NGX_ERROR;
}
*h = ngx_http_ipng_stats_post_read_handler;
/* Header filter: transitions reading -> writing when nginx starts
* sending the response. Inserted at the top of the chain so the
* transition is observed before any downstream filter mutates the
* response. */
ngx_http_ipng_stats_next_header_filter = ngx_http_top_header_filter;
ngx_http_top_header_filter = ngx_http_ipng_stats_header_filter;
return NGX_OK;
}
@@ -1313,6 +1372,10 @@ ngx_http_ipng_stats_init_zone(ngx_shm_zone_t *shm_zone, void *data)
ngx_http_ipng_stats_rbtree_insert);
ngx_queue_init(&sh->lru);
ngx_rbtree_init(&sh->gauge_rbtree, &sh->gauge_sentinel,
ngx_http_ipng_stats_rbtree_insert);
ngx_queue_init(&sh->gauge_lru);
sh->sources.nalloc = 16;
sh->sources.entries = ngx_slab_alloc(slab,
sh->sources.nalloc * sizeof(ngx_str_t));
@@ -2116,6 +2179,141 @@ ngx_http_ipng_stats_log_handler(ngx_http_request_t *r)
}
/* ----------------------------------------------------------------- */
/* In-flight gauge lifecycle hooks */
/* ----------------------------------------------------------------- */
/* POST_READ phase handler: first point at which the request is fully
* parsed enough to resolve source and vip. Finds or creates the
* gauge node, increments active + reading, and registers a pool
* cleanup that decrements on request finalization. Skips subrequests
* and internal redirects (the gauge is already ticking on r->main). */
static ngx_int_t
ngx_http_ipng_stats_post_read_handler(ngx_http_request_t *r)
{
ngx_http_ipng_stats_main_conf_t *imcf;
ngx_http_ipng_stats_loc_conf_t *ilcf;
ngx_http_ipng_stats_shctx_t *sh;
ngx_slab_pool_t *slab;
ngx_http_ipng_stats_ctx_t *ctx;
ngx_http_ipng_stats_gauge_t *g;
ngx_pool_cleanup_t *cln;
ngx_str_t source, vip;
u_char vipbuf[NGX_SOCKADDR_STRLEN];
ngx_uint_t source_id, vip_id;
if (r != r->main || r->internal) {
return NGX_DECLINED;
}
imcf = ngx_http_get_module_main_conf(r, ngx_http_ipng_stats_module);
if (imcf == NULL || imcf->shm_zone == NULL || !imcf->enabled) {
return NGX_DECLINED;
}
ilcf = ngx_http_get_module_loc_conf(r, ngx_http_ipng_stats_module);
if (ilcf == NULL || !ilcf->enabled) {
return NGX_DECLINED;
}
if (ngx_http_ipng_stats_resolve_source(r, imcf, &source) != NGX_OK) {
return NGX_DECLINED;
}
if (ngx_connection_local_sockaddr(r->connection, NULL, 0) != NGX_OK) {
return NGX_DECLINED;
}
if (ngx_http_ipng_stats_canonical_vip(r, vipbuf, sizeof(vipbuf), &vip)
!= NGX_OK)
{
return NGX_DECLINED;
}
slab = (ngx_slab_pool_t *) imcf->shm_zone->shm.addr;
sh = imcf->shm_zone->data;
ngx_shmtx_lock(&slab->mutex);
if (ngx_http_ipng_stats_intern_shared(sh, slab, &sh->sources, &source,
&source_id) != NGX_OK
|| ngx_http_ipng_stats_intern_shared(sh, slab, &sh->vips, &vip,
&vip_id) != NGX_OK)
{
ngx_shmtx_unlock(&slab->mutex);
return NGX_DECLINED;
}
ngx_shmtx_unlock(&slab->mutex);
g = ngx_http_ipng_stats_gauge_get(sh, slab, source_id, vip_id);
if (g == NULL) {
return NGX_DECLINED;
}
ctx = ngx_pcalloc(r->pool, sizeof(*ctx));
cln = ngx_pool_cleanup_add(r->pool, 0);
if (ctx == NULL || cln == NULL) {
return NGX_DECLINED;
}
ctx->gauge = g;
ctx->state = 1; /* reading */
cln->handler = ngx_http_ipng_stats_ctx_cleanup;
cln->data = ctx;
ngx_http_set_ctx(r, ctx, ngx_http_ipng_stats_module);
(void) ngx_atomic_fetch_add(&g->active, 1);
(void) ngx_atomic_fetch_add(&g->reading, 1);
return NGX_DECLINED;
}
/* Header filter: first call for the main request transitions reading
* -> writing. Subrequests have their own header-filter invocations;
* we ignore those so a subrequest doesn't prematurely flip the main
* request's gauge state. Internal redirects re-enter the filter
* chain; the state check prevents double transitions. */
static ngx_int_t
ngx_http_ipng_stats_header_filter(ngx_http_request_t *r)
{
ngx_http_ipng_stats_ctx_t *ctx;
if (r == r->main) {
ctx = ngx_http_get_module_ctx(r, ngx_http_ipng_stats_module);
if (ctx != NULL && ctx->gauge != NULL && ctx->state == 1) {
(void) ngx_atomic_fetch_add(&ctx->gauge->reading,
(ngx_atomic_uint_t) -1);
(void) ngx_atomic_fetch_add(&ctx->gauge->writing, 1);
ctx->state = 2;
}
}
return ngx_http_ipng_stats_next_header_filter(r);
}
static void
ngx_http_ipng_stats_ctx_cleanup(void *data)
{
ngx_http_ipng_stats_ctx_t *ctx = data;
if (ctx->gauge == NULL) {
return;
}
if (ctx->state == 1) {
(void) ngx_atomic_fetch_add(&ctx->gauge->reading,
(ngx_atomic_uint_t) -1);
} else if (ctx->state == 2) {
(void) ngx_atomic_fetch_add(&ctx->gauge->writing,
(ngx_atomic_uint_t) -1);
}
(void) ngx_atomic_fetch_add(&ctx->gauge->active,
(ngx_atomic_uint_t) -1);
ctx->gauge = NULL;
ctx->state = 0;
}
/* ----------------------------------------------------------------- */
/* String interning (called under slab mutex) */
/* ----------------------------------------------------------------- */
@@ -2170,6 +2368,66 @@ ngx_http_ipng_stats_intern_shared(ngx_http_ipng_stats_shctx_t *sh,
}
/* ----------------------------------------------------------------- */
/* In-flight gauges */
/* ----------------------------------------------------------------- */
/* Hash of (source_id, vip_id). Separate constants from the counter
* rbtree's hash so collisions don't line up between the two trees. */
static ngx_inline ngx_uint_t
ngx_http_ipng_stats_gauge_hash(ngx_uint_t source_id, ngx_uint_t vip_id)
{
return (source_id * 2246822519u) ^ (vip_id * 3266489917u);
}
/* Find or create the gauge node for (source_id, vip_id). Takes the
* slab mutex only when a new node must be inserted; the caller holds
* the returned pointer for the rest of the request and does lock-free
* atomic inc/dec on the lanes. Returns NULL on slab exhaustion. */
static ngx_http_ipng_stats_gauge_t *
ngx_http_ipng_stats_gauge_get(ngx_http_ipng_stats_shctx_t *sh,
ngx_slab_pool_t *slab, ngx_uint_t source_id, ngx_uint_t vip_id)
{
ngx_uint_t hash;
ngx_rbtree_node_t *rb;
ngx_http_ipng_stats_gauge_t *g = NULL;
hash = ngx_http_ipng_stats_gauge_hash(source_id, vip_id);
ngx_shmtx_lock(&slab->mutex);
rb = sh->gauge_rbtree.root;
while (rb != &sh->gauge_sentinel) {
if (hash < rb->key) { rb = rb->left; continue; }
if (hash > rb->key) { rb = rb->right; continue; }
g = (ngx_http_ipng_stats_gauge_t *) rb;
if (g->source_id == source_id && g->vip_id == vip_id) {
break;
}
rb = rb->right;
g = NULL;
}
if (g == NULL) {
g = ngx_slab_calloc_locked(slab, sizeof(*g));
if (g == NULL) {
(void) ngx_atomic_fetch_add(&sh->zone_full_events, 1);
ngx_shmtx_unlock(&slab->mutex);
return NULL;
}
g->rbnode.key = hash;
g->source_id = source_id;
g->vip_id = vip_id;
ngx_rbtree_insert(&sh->gauge_rbtree, &g->rbnode);
ngx_queue_insert_tail(&sh->gauge_lru, &g->lru);
}
ngx_shmtx_unlock(&slab->mutex);
return g;
}
/* ----------------------------------------------------------------- */
/* Global logtail: write + flush */
/* ----------------------------------------------------------------- */
@@ -2461,6 +2719,9 @@ typedef struct {
uint64_t bytes_out_sum;
uint64_t req_total; /* total requests for this (source, vip) */
uint64_t up_total; /* upstream observations */
uint64_t active; /* in-flight gauges (request lifecycle) */
uint64_t reading;
uint64_t writing;
uint64_t *dhist; /* nbuckets+1 */
uint64_t *uhist;
uint64_t *bin_hist; /* nbytebuckets+1 */
@@ -2737,6 +2998,60 @@ ngx_http_ipng_stats_snapshot_nodes(ngx_http_ipng_stats_shctx_t *sh,
}
/* Walk the gauge rbtree under the slab mutex and fold in-flight gauge
* values into the aggregation table. Creates new agg entries for
* (source, vip) pairs that have in-flight requests but no completed
* ones yet, so scrapes reflect load even before the first log-phase
* update for a pair. Respects the same filter semantics as the
* counter walk. */
static void
ngx_http_ipng_stats_snapshot_gauges(ngx_http_ipng_stats_shctx_t *sh,
ngx_str_t *filter_src, ngx_str_t *filter_vip,
ngx_str_t *src_tbl, ngx_uint_t n_src,
ngx_str_t *vip_tbl, ngx_uint_t n_vip,
ngx_http_ipng_stats_agg_t *aggs, ngx_uint_t naggs_alloc,
ngx_uint_t *naggs_io)
{
ngx_queue_t *q;
ngx_http_ipng_stats_gauge_t *g;
ngx_str_t *src_entry, *vip_entry;
ngx_http_ipng_stats_agg_t *a;
for (q = ngx_queue_head(&sh->gauge_lru);
q != ngx_queue_sentinel(&sh->gauge_lru);
q = ngx_queue_next(q))
{
g = ngx_queue_data(q, ngx_http_ipng_stats_gauge_t, lru);
if (g->source_id >= n_src || g->vip_id >= n_vip) continue;
src_entry = &src_tbl[g->source_id];
vip_entry = &vip_tbl[g->vip_id];
if (filter_src->len > 0
&& (src_entry->len != filter_src->len
|| ngx_memcmp(src_entry->data, filter_src->data,
filter_src->len) != 0))
{
continue;
}
if (filter_vip->len > 0
&& (vip_entry->len != filter_vip->len
|| ngx_memcmp(vip_entry->data, filter_vip->data,
filter_vip->len) != 0))
{
continue;
}
a = ngx_http_ipng_stats_agg_get(aggs, naggs_io, naggs_alloc,
g->source_id, g->vip_id);
if (a == NULL) continue;
a->active = g->active;
a->reading = g->reading;
a->writing = g->writing;
}
}
/* -- Prometheus ---------------------------------------------------- */
@@ -2817,6 +3132,13 @@ ngx_http_ipng_stats_render_prom(ngx_http_request_t *r,
"# TYPE nginx_ipng_bytes_in histogram\n"
"# HELP nginx_ipng_bytes_out Request size histogram in bytes.\n"
"# TYPE nginx_ipng_bytes_out histogram\n"
"# HELP nginx_ipng_active Requests currently in flight.\n"
"# TYPE nginx_ipng_active gauge\n"
"# HELP nginx_ipng_reading In-flight requests in the pre-response "
"phases (rewrite/access/content).\n"
"# TYPE nginx_ipng_reading gauge\n"
"# HELP nginx_ipng_writing In-flight requests past header send.\n"
"# TYPE nginx_ipng_writing gauge\n"
"# HELP nginx_ipng_ifindex_misses_total Connections whose ingress "
"ifindex did not match any configured device= binding.\n"
"# TYPE nginx_ipng_ifindex_misses_total counter\n"
@@ -2891,6 +3213,10 @@ ngx_http_ipng_stats_render_prom(ngx_http_request_t *r,
snaps, nsnaps_alloc, &nsnaps,
aggs, naggs_alloc, &naggs);
ngx_http_ipng_stats_snapshot_gauges(sh, filter_source, filter_vip,
src_tbl, n_src, vip_tbl, n_vip,
aggs, naggs_alloc, &naggs);
ngx_shmtx_unlock(&slab->mutex);
/* Per-node counters. */
@@ -2915,9 +3241,11 @@ ngx_http_ipng_stats_render_prom(ngx_http_request_t *r,
}
}
/* One chain link per (source, vip) for the four aggregated histograms.
* Size: per-bucket line ~96B, + sum/count/+Inf per metric ~96B each. */
hist_sz = 256 + 96 * (2 * (nb + 1) + 2 * (nbb + 1)) + 4 * 200;
/* One chain link per (source, vip) for the four aggregated
* histograms plus the three in-flight gauges. Size: per-bucket
* line ~96B, sum/count/+Inf per metric ~96B each, three gauge
* lines ~80B each. */
hist_sz = 512 + 96 * (2 * (nb + 1) + 2 * (nbb + 1)) + 4 * 200;
for (i = 0; i < naggs; i++) {
ngx_http_ipng_stats_agg_t *a = &aggs[i];
@@ -2944,6 +3272,13 @@ ngx_http_ipng_stats_render_prom(ngx_http_request_t *r,
"nginx_ipng_bytes_out", src, vip,
imcf->byte_bucket_bounds, nbb, a->bout_hist,
(double) a->bytes_out_sum, 0);
p = ngx_sprintf(p,
"nginx_ipng_active{source_tag=\"%V\",vip=\"%V\"} %uL\n"
"nginx_ipng_reading{source_tag=\"%V\",vip=\"%V\"} %uL\n"
"nginx_ipng_writing{source_tag=\"%V\",vip=\"%V\"} %uL\n",
src, vip, a->active,
src, vip, a->reading,
src, vip, a->writing);
cl->buf->last = p;
if (ngx_http_ipng_stats_append(&last, cl) != NGX_OK) {
return NGX_HTTP_INTERNAL_SERVER_ERROR;
@@ -3045,11 +3380,15 @@ ngx_http_ipng_stats_render_json(ngx_http_request_t *r,
snaps, nsnaps_alloc, &nsnaps,
aggs, naggs_alloc, &naggs);
ngx_http_ipng_stats_snapshot_gauges(sh, filter_source, filter_vip,
src_tbl, n_src, vip_tbl, n_vip,
aggs, naggs_alloc, &naggs);
ngx_shmtx_unlock(&slab->mutex);
/* One JSON record per aggregated (source, vip). Size upper-bound
* accounts for: fixed overhead, up to NCLASSES class entries, 4
* histograms. */
* histograms, 3 gauges. */
rec_sz = 512
+ 160 * NGX_HTTP_IPNG_STATS_NCLASSES
+ 48 * (2 * (nb + 1) + 2 * (nbb + 1))
@@ -3131,7 +3470,9 @@ ngx_http_ipng_stats_render_json(ngx_http_request_t *r,
imcf->byte_bucket_bounds[j], a->bout_hist[j]);
}
}
p = ngx_sprintf(p, "}}}");
p = ngx_sprintf(p,
"}},\"gauges\":{\"active\":%uL,\"reading\":%uL,\"writing\":%uL}}",
a->active, a->reading, a->writing);
cl->buf->last = p;
if (ngx_http_ipng_stats_append(&last, cl) != NGX_OK) {

View File

@@ -49,9 +49,16 @@ Shared-listen-include across multiple server blocks
Length Should Be ${count} 2
... Expected 2 listening sockets on port 8080 (v4+v6 wildcards); got ${count}
# Proves the cross-cscf option-stripping path actually fired for
# the 2nd and 3rd server blocks. `nginx -t` replays the whole
# config and emits the wrapper's NOTICE each time it strips.
Should Contain ${output} stripped socket options from duplicate listen
# the 2nd and 3rd server blocks at daemon startup — the wrapper
# logs a NOTICE per stripped listen via ngx_conf_log_error. We
# read the container's startup log rather than `nginx -t`'s
# output because nginx keeps the config-parse log level at ERR
# until the error_log directive has been fully applied, so
# `nginx -t` suppresses the NOTICE even though the wrapper is
# invoked.
${rc} ${startup_log} = Run And Return Rc And Output
... docker logs ${SERVER} 2>&1
Should Contain ${startup_log} stripped socket options from duplicate listen
Prometheus scrape
[Documentation] Scrape returns HELP/TYPE preamble.
@@ -139,6 +146,53 @@ Duration histogram
Should Contain ${json} request_duration_ms
Should Contain ${json} buckets
# --- In-flight gauges ---
Gauge preamble in Prometheus output
[Documentation] The scrape preamble advertises the three gauge metric
... names and types. Present whether or not any traffic
... has ever been observed.
${output} = Scrape Prometheus
Should Match Regexp ${output} (?m)^# TYPE nginx_ipng_active gauge$
Should Match Regexp ${output} (?m)^# TYPE nginx_ipng_reading gauge$
Should Match Regexp ${output} (?m)^# TYPE nginx_ipng_writing gauge$
Gauges zero at rest
[Documentation] After the earlier traffic tests have drained and the
... flush tick has run, the in-flight gauges for (tag1,
... 10.0.1.1) must all report zero — no request is still
... resident in the plugin's per-request ctx.
Wait For Flush
${output} = Scrape With Filter source_tag=tag1&vip=10.0.1.1
Should Match Regexp ${output} nginx_ipng_active\\{[^}]*\\}\\s+0\\b
Should Match Regexp ${output} nginx_ipng_reading\\{[^}]*\\}\\s+0\\b
Should Match Regexp ${output} nginx_ipng_writing\\{[^}]*\\}\\s+0\\b
Gauges appear in JSON output
[Documentation] Each record in the JSON output carries a gauges object
... with active/reading/writing keys.
${rc} ${output} = Run And Return Rc And Output
... curl -sf -H 'Accept: application/json' '${SCRAPE_URL}?source_tag=tag1&vip=10.0.1.1' | python3 -m json.tool
Should Be Equal As Integers ${rc} 0
Should Contain ${output} "gauges"
Should Contain ${output} "active"
Should Contain ${output} "reading"
Should Contain ${output} "writing"
Gauges observed non-zero under concurrent load
[Documentation] Fire a burst of concurrent requests against the
... /slow backend (50 ms each, single-threaded) and
... observe at least one of active/reading/writing
... going non-zero for (tag1, 10.0.1.1) while the
... burst is in flight. After it drains the gauges
... return to zero. Uses 20 requests (drain <= ~2s
... given the single-threaded backend) to keep the
... test fast but still wide enough for the scrape
... to catch overlap.
Run docker exec -d ${CLIENT1} sh -c 'for i in $(seq 20); do curl -sf http://10.0.1.1:8080/slow >/dev/null & done; wait'
Wait Until Keyword Succeeds 5s 50ms Nonzero Gauge Observed tag1 10.0.1.1
Wait Until Keyword Succeeds 15s 500ms Gauges Drained tag1 10.0.1.1
# --- Scrape filters ---
Filter by source_tag
@@ -347,6 +401,26 @@ Get Request Count
END
RETURN ${total}
Nonzero Gauge Observed
[Documentation] Fails unless the scrape shows a non-zero value for at
... least one of {active, reading, writing} on the given
... (source_tag, vip) pair.
[Arguments] ${source} ${vip}
${output} = Scrape With Filter source_tag=${source}&vip=${vip}
Should Match Regexp ${output}
... nginx_ipng_(active|reading|writing)\\{[^}]*\\}\\s+[1-9]\\d*
Gauges Drained
[Documentation] Fails unless all three gauges are exactly zero for
... the given (source_tag, vip) pair. Use with Wait
... Until Keyword Succeeds to poll until a request burst
... has fully finalized.
[Arguments] ${source} ${vip}
${output} = Scrape With Filter source_tag=${source}&vip=${vip}
Should Match Regexp ${output} nginx_ipng_active\\{[^}]*\\}\\s+0\\b
Should Match Regexp ${output} nginx_ipng_reading\\{[^}]*\\}\\s+0\\b
Should Match Regexp ${output} nginx_ipng_writing\\{[^}]*\\}\\s+0\\b
# --- Container helpers ---
Docker Exec