diff --git a/Makefile b/Makefile index 8848eb3..3fa0beb 100644 --- a/Makefile +++ b/Makefile @@ -23,7 +23,7 @@ BUILD_DIR := $(CURDIR)/build # the package version from there directly. The C code picks up VERSION # via the generated src/version.h (written by the version-header target # below and depended on by the module build). -VERSION := 0.7.2 +VERSION := 0.8.2 NGINX_SRC ?= diff --git a/README.md b/README.md index c937428..a693b81 100644 --- a/README.md +++ b/README.md @@ -7,10 +7,10 @@ nginx on Debian Trixie. The module attributes every HTTP request to the interface it arrived on, reading the ingress `ifindex` per connection from the kernel's `IP_PKTINFO` / `IPV6_PKTINFO` cmsg. Listening sockets stay plain wildcards, so outgoing packets follow the normal routing table — which is what makes this safe for DSR / maglev deployments where the SYN arrives via a GRE tunnel and the -SYN-ACK must leave via the default route. Counters — requests, status codes, bytes, latency histograms — are exposed as -Prometheus text or JSON from a single HTTP scrape endpoint, filtered per-source. This is useful for any deployment where -traffic arrives on distinct interfaces — GRE tunnels, VLANs, bonded links, or plain ethernet — and per-interface observability -is needed. +SYN-ACK must leave via the default route. Counters — requests, status codes, bytes, latency histograms — plus point-in-time +gauges of requests currently in flight (`active`, `reading`, `writing`) are exposed as Prometheus text or JSON from a single +HTTP scrape endpoint, filtered per-source. This is useful for any deployment where traffic arrives on distinct interfaces — +GRE tunnels, VLANs, bonded links, or plain ethernet — and per-interface observability is needed. Without any `device=`/`ipng_source_tag=` parameters, the module still counts and exposes per-VIP traffic under the configurable default source tag (`direct`), which makes it a useful plain observability module for any nginx host. diff --git a/debian/changelog b/debian/changelog index e5db033..baa3144 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,22 @@ +nginx-ipng-stats-plugin (0.8.2-1) unstable; urgency=medium + + * Pre-release v0.8.2. + - New per (source_tag, vip) in-flight gauges: + nginx_ipng_active, nginx_ipng_reading, nginx_ipng_writing. + Lifecycle: POST_READ handler increments active+reading + on each main request, a header filter transitions + reading->writing when headers are sent, and a pool + cleanup decrements on request finalization. Gauges live + in a dedicated rbtree in the shared zone; the slab mutex + is taken only on first insert per (source, vip) pair — + subsequent transitions are lock-free atomic inc/dec on + the cached node. Subrequests and internal redirects do + not double-count. Gauges are emitted in both Prometheus + (`gauge` typed) and JSON (`"gauges":{...}` per record) + scrape outputs. Invariant: reading + writing = active. + + -- Pim van Pelt Thu, 23 Apr 2026 12:00:00 +0200 + nginx-ipng-stats-plugin (0.7.2-1) unstable; urgency=medium * Pre-release v0.7.2. diff --git a/docs/design.md b/docs/design.md index ccf1e9d..161a024 100644 --- a/docs/design.md +++ b/docs/design.md @@ -128,6 +128,12 @@ Each requirement carries a unique identifier (`FR-X.Y` or `NFR-X.Y`) so that lat codes are observed. Operators who need a full per-code breakdown SHOULD enable `ipng_stats_logtail` (FR-8) and derive the per-code view from the access-log stream off the hot path; the stats zone intentionally trades that resolution for a much smaller scrape response. +- **FR-2.7** The module MUST additionally maintain, per `(source, vip)` pair, three point-in-time gauges of requests currently in + flight: `active` (observed at `POST_READ` but not yet finalized), `reading` (in the pre-response phases — rewrite/access/content), + and `writing` (past the header-send transition). The invariant `reading + writing = active` MUST hold at any instant. Subrequests + and internal redirects MUST NOT double-count the parent request. Updates to the gauges are atomic increments/decrements on the + request lifecycle hooks — no slab lock after the first time a `(source, vip)` pair is seen — so the hot-path rule in FR-4.1 still + holds for ordinary counter updates while gauges are maintained lock-free. **FR-3 Scrape endpoint** @@ -147,6 +153,8 @@ Each requirement carries a unique identifier (`FR-X.Y` or `NFR-X.Y`) so that lat `source_tag` and `vip`. Counter metrics (`nginx_ipng_requests_total`, `nginx_ipng_bytes_{in,out}_total`, `nginx_ipng_latency_total`) additionally carry a `code` label with a class value (`1xx`..`5xx`/`unknown`). Histogram series (duration, upstream response, request/response byte size) MUST NOT carry a `code` label — they aggregate across all classes for a given `(source, vip)` pair. + Gauge series (`nginx_ipng_active`, `nginx_ipng_reading`, `nginx_ipng_writing`) MUST be labelled with `source_tag` and `vip` only + (no `code`) and MUST be typed as `gauge` in the exposition preamble. **FR-4 Hot path and flush** @@ -407,6 +415,12 @@ Histogram lanes are kept per `(source, vip, class)` in storage, then summed acro A parallel table keyed by `(source_id, vip_id)` — one row per VIP — holds the EWMAs for instantaneous rate. EWMAs are floats but updated only from the flush tick, so there is no float contention on the request path. +A second, smaller rbtree lives alongside the counter tree — one node per `(source_id, vip_id)` pair — holding three atomic gauge +lanes (`active`, `reading`, `writing`; FR-2.7). Unlike the counter path, gauges are updated from request lifecycle hooks +(`POST_READ`, header filter, pool cleanup) with atomic inc/dec directly on the shared node. The slab mutex is taken only the first +time a `(source, vip)` pair is seen; subsequent transitions on that pair are lock-free. Gauge nodes are never evicted — their +cardinality equals the number of distinct `(source, vip)` pairs and is small in practice. + The module also keeps a small string interning table for source and VIP strings, keyed by the integer IDs above, so that the scrape endpoint can recover the original strings without re-parsing configuration. diff --git a/docs/user-guide.md b/docs/user-guide.md index 937ecc5..f9c0f7b 100644 --- a/docs/user-guide.md +++ b/docs/user-guide.md @@ -249,6 +249,14 @@ nginx_ipng_bytes_in_total{source_tag="mg1",vip="192.0.2.10",code="2xx"} 9876543 # Histogram series (request_duration, upstream_response, bytes_in, bytes_out) # do NOT carry a `code` label — they aggregate across classes per (source, vip). nginx_ipng_request_duration_seconds_bucket{source_tag="mg1",vip="192.0.2.10",le="0.050"} 11200 + +# In-flight gauges per (source, vip). These are point-in-time request counts, +# not rates: `active` = requests observed at POST_READ that haven't finalized +# yet; `reading` = in pre-response phases (rewrite/access/content); `writing` +# = past header send. reading + writing = active at any instant. +nginx_ipng_active{source_tag="mg1",vip="192.0.2.10"} 3 +nginx_ipng_reading{source_tag="mg1",vip="192.0.2.10"} 1 +nginx_ipng_writing{source_tag="mg1",vip="192.0.2.10"} 2 ``` For JSON output instead, set the `Accept` header: @@ -300,6 +308,11 @@ sum by (vip) (rate(nginx_ipng_requests_total[5m])) # p95 request duration per (source_tag, vip): histogram_quantile(0.95, sum by (source_tag, vip, le) (rate(nginx_ipng_request_duration_seconds_bucket[5m]))) + +# In-flight concurrency per (source_tag, vip). Gauges are exported as-is; +# use max_over_time for load-shedding alerts or avg_over_time for capacity +# planning: +max_over_time(nginx_ipng_active[5m]) ``` ## 6. Set up a global logtail access log diff --git a/src/ngx_http_ipng_stats_module.c b/src/ngx_http_ipng_stats_module.c index c575b84..ee54f6d 100644 --- a/src/ngx_http_ipng_stats_module.c +++ b/src/ngx_http_ipng_stats_module.c @@ -159,6 +159,34 @@ typedef struct ngx_http_ipng_stats_slot_s { } ngx_http_ipng_stats_slot_t; +/* In-flight gauge node: one per (source_id, vip_id) pair, tracking the + * count of requests currently in each lifecycle phase. Inserted under + * the slab mutex the first time a pair is observed, then cached in the + * per-request ctx so transitions touch only the atomic lanes — no + * locking on the hot path after the first hit for a given pair. + * + * Never evicted: one node per distinct (source, vip) pair is small in + * practice (tens) and the node is cheap. */ +typedef struct { + ngx_rbtree_node_t rbnode; /* key = hash of (source_id, vip_id) */ + ngx_queue_t lru; /* for iteration at render time */ + ngx_uint_t source_id; + ngx_uint_t vip_id; + ngx_atomic_uint_t active; + ngx_atomic_uint_t reading; + ngx_atomic_uint_t writing; +} ngx_http_ipng_stats_gauge_t; + + +/* Per-request state carried through the lifecycle hooks. Allocated at + * POST_READ, transitioned by the header filter, decremented by the + * pool cleanup. `state` is 1 = reading, 2 = writing, 0 = not tracked. */ +typedef struct { + ngx_http_ipng_stats_gauge_t *gauge; + ngx_uint_t state; +} ngx_http_ipng_stats_ctx_t; + + /* String interning tables live at the head of the shared-memory zone. * They're flat arrays of ngx_str_t whose data pointers reference memory * allocated from the zone's slab pool. Workers look up strings by @@ -179,6 +207,13 @@ typedef struct { ngx_http_ipng_stats_intern_t sources; ngx_http_ipng_stats_intern_t vips; + /* In-flight gauges, keyed by (source_id, vip_id). Atomically + * updated from the request lifecycle hooks; snapshotted by the + * scrape handler under the slab mutex. */ + ngx_rbtree_t gauge_rbtree; + ngx_rbtree_node_t gauge_sentinel; + ngx_queue_t gauge_lru; + /* Meta-counters for the plugin itself (FR-6 observability of * the plugin in the design doc). */ ngx_atomic_uint_t zone_full_events; @@ -328,6 +363,12 @@ static void ngx_http_ipng_stats_exit_worker(ngx_cycle_t *cycle); static void ngx_http_ipng_stats_rescan_timer(ngx_event_t *ev); static ngx_int_t ngx_http_ipng_stats_log_handler(ngx_http_request_t *r); +static ngx_int_t ngx_http_ipng_stats_post_read_handler(ngx_http_request_t *r); +static ngx_int_t ngx_http_ipng_stats_header_filter(ngx_http_request_t *r); +static void ngx_http_ipng_stats_ctx_cleanup(void *data); +static ngx_http_ipng_stats_gauge_t *ngx_http_ipng_stats_gauge_get( + ngx_http_ipng_stats_shctx_t *sh, ngx_slab_pool_t *slab, + ngx_uint_t source_id, ngx_uint_t vip_id); static ngx_int_t ngx_http_ipng_stats_content_handler(ngx_http_request_t *r); static void ngx_http_ipng_stats_flush_timer(ngx_event_t *ev); @@ -365,6 +406,8 @@ static char *(*ngx_http_core_listen_orig)(ngx_conf_t *cf, static ngx_http_ipng_stats_worker_t ngx_http_ipng_stats_worker; +static ngx_http_output_header_filter_pt ngx_http_ipng_stats_next_header_filter; + extern ngx_module_t ngx_http_core_module; @@ -1200,7 +1243,7 @@ ngx_http_ipng_stats_logtail(ngx_conf_t *cf, ngx_command_t *cmd, void *conf) /* ----------------------------------------------------------------- */ -/* Postconfig: install log-phase handler */ +/* Postconfig: install phase handlers and header filter */ /* ----------------------------------------------------------------- */ static ngx_int_t @@ -1240,6 +1283,22 @@ ngx_http_ipng_stats_postconfig(ngx_conf_t *cf) } *h = ngx_http_ipng_stats_log_handler; + /* POST_READ is the earliest phase at which the request is parsed + * enough to resolve source and vip; we register the in-flight + * gauge there so `reading` covers rewrite/access/content. */ + h = ngx_array_push(&cmcf->phases[NGX_HTTP_POST_READ_PHASE].handlers); + if (h == NULL) { + return NGX_ERROR; + } + *h = ngx_http_ipng_stats_post_read_handler; + + /* Header filter: transitions reading -> writing when nginx starts + * sending the response. Inserted at the top of the chain so the + * transition is observed before any downstream filter mutates the + * response. */ + ngx_http_ipng_stats_next_header_filter = ngx_http_top_header_filter; + ngx_http_top_header_filter = ngx_http_ipng_stats_header_filter; + return NGX_OK; } @@ -1313,6 +1372,10 @@ ngx_http_ipng_stats_init_zone(ngx_shm_zone_t *shm_zone, void *data) ngx_http_ipng_stats_rbtree_insert); ngx_queue_init(&sh->lru); + ngx_rbtree_init(&sh->gauge_rbtree, &sh->gauge_sentinel, + ngx_http_ipng_stats_rbtree_insert); + ngx_queue_init(&sh->gauge_lru); + sh->sources.nalloc = 16; sh->sources.entries = ngx_slab_alloc(slab, sh->sources.nalloc * sizeof(ngx_str_t)); @@ -2116,6 +2179,141 @@ ngx_http_ipng_stats_log_handler(ngx_http_request_t *r) } +/* ----------------------------------------------------------------- */ +/* In-flight gauge lifecycle hooks */ +/* ----------------------------------------------------------------- */ + +/* POST_READ phase handler: first point at which the request is fully + * parsed enough to resolve source and vip. Finds or creates the + * gauge node, increments active + reading, and registers a pool + * cleanup that decrements on request finalization. Skips subrequests + * and internal redirects (the gauge is already ticking on r->main). */ +static ngx_int_t +ngx_http_ipng_stats_post_read_handler(ngx_http_request_t *r) +{ + ngx_http_ipng_stats_main_conf_t *imcf; + ngx_http_ipng_stats_loc_conf_t *ilcf; + ngx_http_ipng_stats_shctx_t *sh; + ngx_slab_pool_t *slab; + ngx_http_ipng_stats_ctx_t *ctx; + ngx_http_ipng_stats_gauge_t *g; + ngx_pool_cleanup_t *cln; + ngx_str_t source, vip; + u_char vipbuf[NGX_SOCKADDR_STRLEN]; + ngx_uint_t source_id, vip_id; + + if (r != r->main || r->internal) { + return NGX_DECLINED; + } + + imcf = ngx_http_get_module_main_conf(r, ngx_http_ipng_stats_module); + if (imcf == NULL || imcf->shm_zone == NULL || !imcf->enabled) { + return NGX_DECLINED; + } + + ilcf = ngx_http_get_module_loc_conf(r, ngx_http_ipng_stats_module); + if (ilcf == NULL || !ilcf->enabled) { + return NGX_DECLINED; + } + + if (ngx_http_ipng_stats_resolve_source(r, imcf, &source) != NGX_OK) { + return NGX_DECLINED; + } + + if (ngx_connection_local_sockaddr(r->connection, NULL, 0) != NGX_OK) { + return NGX_DECLINED; + } + + if (ngx_http_ipng_stats_canonical_vip(r, vipbuf, sizeof(vipbuf), &vip) + != NGX_OK) + { + return NGX_DECLINED; + } + + slab = (ngx_slab_pool_t *) imcf->shm_zone->shm.addr; + sh = imcf->shm_zone->data; + + ngx_shmtx_lock(&slab->mutex); + if (ngx_http_ipng_stats_intern_shared(sh, slab, &sh->sources, &source, + &source_id) != NGX_OK + || ngx_http_ipng_stats_intern_shared(sh, slab, &sh->vips, &vip, + &vip_id) != NGX_OK) + { + ngx_shmtx_unlock(&slab->mutex); + return NGX_DECLINED; + } + ngx_shmtx_unlock(&slab->mutex); + + g = ngx_http_ipng_stats_gauge_get(sh, slab, source_id, vip_id); + if (g == NULL) { + return NGX_DECLINED; + } + + ctx = ngx_pcalloc(r->pool, sizeof(*ctx)); + cln = ngx_pool_cleanup_add(r->pool, 0); + if (ctx == NULL || cln == NULL) { + return NGX_DECLINED; + } + ctx->gauge = g; + ctx->state = 1; /* reading */ + cln->handler = ngx_http_ipng_stats_ctx_cleanup; + cln->data = ctx; + + ngx_http_set_ctx(r, ctx, ngx_http_ipng_stats_module); + + (void) ngx_atomic_fetch_add(&g->active, 1); + (void) ngx_atomic_fetch_add(&g->reading, 1); + + return NGX_DECLINED; +} + + +/* Header filter: first call for the main request transitions reading + * -> writing. Subrequests have their own header-filter invocations; + * we ignore those so a subrequest doesn't prematurely flip the main + * request's gauge state. Internal redirects re-enter the filter + * chain; the state check prevents double transitions. */ +static ngx_int_t +ngx_http_ipng_stats_header_filter(ngx_http_request_t *r) +{ + ngx_http_ipng_stats_ctx_t *ctx; + + if (r == r->main) { + ctx = ngx_http_get_module_ctx(r, ngx_http_ipng_stats_module); + if (ctx != NULL && ctx->gauge != NULL && ctx->state == 1) { + (void) ngx_atomic_fetch_add(&ctx->gauge->reading, + (ngx_atomic_uint_t) -1); + (void) ngx_atomic_fetch_add(&ctx->gauge->writing, 1); + ctx->state = 2; + } + } + + return ngx_http_ipng_stats_next_header_filter(r); +} + + +static void +ngx_http_ipng_stats_ctx_cleanup(void *data) +{ + ngx_http_ipng_stats_ctx_t *ctx = data; + + if (ctx->gauge == NULL) { + return; + } + if (ctx->state == 1) { + (void) ngx_atomic_fetch_add(&ctx->gauge->reading, + (ngx_atomic_uint_t) -1); + } else if (ctx->state == 2) { + (void) ngx_atomic_fetch_add(&ctx->gauge->writing, + (ngx_atomic_uint_t) -1); + } + (void) ngx_atomic_fetch_add(&ctx->gauge->active, + (ngx_atomic_uint_t) -1); + ctx->gauge = NULL; + ctx->state = 0; +} + + /* ----------------------------------------------------------------- */ /* String interning (called under slab mutex) */ /* ----------------------------------------------------------------- */ @@ -2170,6 +2368,66 @@ ngx_http_ipng_stats_intern_shared(ngx_http_ipng_stats_shctx_t *sh, } +/* ----------------------------------------------------------------- */ +/* In-flight gauges */ +/* ----------------------------------------------------------------- */ + +/* Hash of (source_id, vip_id). Separate constants from the counter + * rbtree's hash so collisions don't line up between the two trees. */ +static ngx_inline ngx_uint_t +ngx_http_ipng_stats_gauge_hash(ngx_uint_t source_id, ngx_uint_t vip_id) +{ + return (source_id * 2246822519u) ^ (vip_id * 3266489917u); +} + + +/* Find or create the gauge node for (source_id, vip_id). Takes the + * slab mutex only when a new node must be inserted; the caller holds + * the returned pointer for the rest of the request and does lock-free + * atomic inc/dec on the lanes. Returns NULL on slab exhaustion. */ +static ngx_http_ipng_stats_gauge_t * +ngx_http_ipng_stats_gauge_get(ngx_http_ipng_stats_shctx_t *sh, + ngx_slab_pool_t *slab, ngx_uint_t source_id, ngx_uint_t vip_id) +{ + ngx_uint_t hash; + ngx_rbtree_node_t *rb; + ngx_http_ipng_stats_gauge_t *g = NULL; + + hash = ngx_http_ipng_stats_gauge_hash(source_id, vip_id); + + ngx_shmtx_lock(&slab->mutex); + + rb = sh->gauge_rbtree.root; + while (rb != &sh->gauge_sentinel) { + if (hash < rb->key) { rb = rb->left; continue; } + if (hash > rb->key) { rb = rb->right; continue; } + g = (ngx_http_ipng_stats_gauge_t *) rb; + if (g->source_id == source_id && g->vip_id == vip_id) { + break; + } + rb = rb->right; + g = NULL; + } + + if (g == NULL) { + g = ngx_slab_calloc_locked(slab, sizeof(*g)); + if (g == NULL) { + (void) ngx_atomic_fetch_add(&sh->zone_full_events, 1); + ngx_shmtx_unlock(&slab->mutex); + return NULL; + } + g->rbnode.key = hash; + g->source_id = source_id; + g->vip_id = vip_id; + ngx_rbtree_insert(&sh->gauge_rbtree, &g->rbnode); + ngx_queue_insert_tail(&sh->gauge_lru, &g->lru); + } + + ngx_shmtx_unlock(&slab->mutex); + return g; +} + + /* ----------------------------------------------------------------- */ /* Global logtail: write + flush */ /* ----------------------------------------------------------------- */ @@ -2461,6 +2719,9 @@ typedef struct { uint64_t bytes_out_sum; uint64_t req_total; /* total requests for this (source, vip) */ uint64_t up_total; /* upstream observations */ + uint64_t active; /* in-flight gauges (request lifecycle) */ + uint64_t reading; + uint64_t writing; uint64_t *dhist; /* nbuckets+1 */ uint64_t *uhist; uint64_t *bin_hist; /* nbytebuckets+1 */ @@ -2737,6 +2998,60 @@ ngx_http_ipng_stats_snapshot_nodes(ngx_http_ipng_stats_shctx_t *sh, } +/* Walk the gauge rbtree under the slab mutex and fold in-flight gauge + * values into the aggregation table. Creates new agg entries for + * (source, vip) pairs that have in-flight requests but no completed + * ones yet, so scrapes reflect load even before the first log-phase + * update for a pair. Respects the same filter semantics as the + * counter walk. */ +static void +ngx_http_ipng_stats_snapshot_gauges(ngx_http_ipng_stats_shctx_t *sh, + ngx_str_t *filter_src, ngx_str_t *filter_vip, + ngx_str_t *src_tbl, ngx_uint_t n_src, + ngx_str_t *vip_tbl, ngx_uint_t n_vip, + ngx_http_ipng_stats_agg_t *aggs, ngx_uint_t naggs_alloc, + ngx_uint_t *naggs_io) +{ + ngx_queue_t *q; + ngx_http_ipng_stats_gauge_t *g; + ngx_str_t *src_entry, *vip_entry; + ngx_http_ipng_stats_agg_t *a; + + for (q = ngx_queue_head(&sh->gauge_lru); + q != ngx_queue_sentinel(&sh->gauge_lru); + q = ngx_queue_next(q)) + { + g = ngx_queue_data(q, ngx_http_ipng_stats_gauge_t, lru); + if (g->source_id >= n_src || g->vip_id >= n_vip) continue; + + src_entry = &src_tbl[g->source_id]; + vip_entry = &vip_tbl[g->vip_id]; + + if (filter_src->len > 0 + && (src_entry->len != filter_src->len + || ngx_memcmp(src_entry->data, filter_src->data, + filter_src->len) != 0)) + { + continue; + } + if (filter_vip->len > 0 + && (vip_entry->len != filter_vip->len + || ngx_memcmp(vip_entry->data, filter_vip->data, + filter_vip->len) != 0)) + { + continue; + } + + a = ngx_http_ipng_stats_agg_get(aggs, naggs_io, naggs_alloc, + g->source_id, g->vip_id); + if (a == NULL) continue; + a->active = g->active; + a->reading = g->reading; + a->writing = g->writing; + } +} + + /* -- Prometheus ---------------------------------------------------- */ @@ -2817,6 +3132,13 @@ ngx_http_ipng_stats_render_prom(ngx_http_request_t *r, "# TYPE nginx_ipng_bytes_in histogram\n" "# HELP nginx_ipng_bytes_out Request size histogram in bytes.\n" "# TYPE nginx_ipng_bytes_out histogram\n" + "# HELP nginx_ipng_active Requests currently in flight.\n" + "# TYPE nginx_ipng_active gauge\n" + "# HELP nginx_ipng_reading In-flight requests in the pre-response " + "phases (rewrite/access/content).\n" + "# TYPE nginx_ipng_reading gauge\n" + "# HELP nginx_ipng_writing In-flight requests past header send.\n" + "# TYPE nginx_ipng_writing gauge\n" "# HELP nginx_ipng_ifindex_misses_total Connections whose ingress " "ifindex did not match any configured device= binding.\n" "# TYPE nginx_ipng_ifindex_misses_total counter\n" @@ -2891,6 +3213,10 @@ ngx_http_ipng_stats_render_prom(ngx_http_request_t *r, snaps, nsnaps_alloc, &nsnaps, aggs, naggs_alloc, &naggs); + ngx_http_ipng_stats_snapshot_gauges(sh, filter_source, filter_vip, + src_tbl, n_src, vip_tbl, n_vip, + aggs, naggs_alloc, &naggs); + ngx_shmtx_unlock(&slab->mutex); /* Per-node counters. */ @@ -2915,9 +3241,11 @@ ngx_http_ipng_stats_render_prom(ngx_http_request_t *r, } } - /* One chain link per (source, vip) for the four aggregated histograms. - * Size: per-bucket line ~96B, + sum/count/+Inf per metric ~96B each. */ - hist_sz = 256 + 96 * (2 * (nb + 1) + 2 * (nbb + 1)) + 4 * 200; + /* One chain link per (source, vip) for the four aggregated + * histograms plus the three in-flight gauges. Size: per-bucket + * line ~96B, sum/count/+Inf per metric ~96B each, three gauge + * lines ~80B each. */ + hist_sz = 512 + 96 * (2 * (nb + 1) + 2 * (nbb + 1)) + 4 * 200; for (i = 0; i < naggs; i++) { ngx_http_ipng_stats_agg_t *a = &aggs[i]; @@ -2944,6 +3272,13 @@ ngx_http_ipng_stats_render_prom(ngx_http_request_t *r, "nginx_ipng_bytes_out", src, vip, imcf->byte_bucket_bounds, nbb, a->bout_hist, (double) a->bytes_out_sum, 0); + p = ngx_sprintf(p, + "nginx_ipng_active{source_tag=\"%V\",vip=\"%V\"} %uL\n" + "nginx_ipng_reading{source_tag=\"%V\",vip=\"%V\"} %uL\n" + "nginx_ipng_writing{source_tag=\"%V\",vip=\"%V\"} %uL\n", + src, vip, a->active, + src, vip, a->reading, + src, vip, a->writing); cl->buf->last = p; if (ngx_http_ipng_stats_append(&last, cl) != NGX_OK) { return NGX_HTTP_INTERNAL_SERVER_ERROR; @@ -3045,11 +3380,15 @@ ngx_http_ipng_stats_render_json(ngx_http_request_t *r, snaps, nsnaps_alloc, &nsnaps, aggs, naggs_alloc, &naggs); + ngx_http_ipng_stats_snapshot_gauges(sh, filter_source, filter_vip, + src_tbl, n_src, vip_tbl, n_vip, + aggs, naggs_alloc, &naggs); + ngx_shmtx_unlock(&slab->mutex); /* One JSON record per aggregated (source, vip). Size upper-bound * accounts for: fixed overhead, up to NCLASSES class entries, 4 - * histograms. */ + * histograms, 3 gauges. */ rec_sz = 512 + 160 * NGX_HTTP_IPNG_STATS_NCLASSES + 48 * (2 * (nb + 1) + 2 * (nbb + 1)) @@ -3131,7 +3470,9 @@ ngx_http_ipng_stats_render_json(ngx_http_request_t *r, imcf->byte_bucket_bounds[j], a->bout_hist[j]); } } - p = ngx_sprintf(p, "}}}"); + p = ngx_sprintf(p, + "}},\"gauges\":{\"active\":%uL,\"reading\":%uL,\"writing\":%uL}}", + a->active, a->reading, a->writing); cl->buf->last = p; if (ngx_http_ipng_stats_append(&last, cl) != NGX_OK) { diff --git a/tests/01-module/01-e2e.robot b/tests/01-module/01-e2e.robot index e8da7c0..b06eca6 100644 --- a/tests/01-module/01-e2e.robot +++ b/tests/01-module/01-e2e.robot @@ -49,9 +49,16 @@ Shared-listen-include across multiple server blocks Length Should Be ${count} 2 ... Expected 2 listening sockets on port 8080 (v4+v6 wildcards); got ${count} # Proves the cross-cscf option-stripping path actually fired for - # the 2nd and 3rd server blocks. `nginx -t` replays the whole - # config and emits the wrapper's NOTICE each time it strips. - Should Contain ${output} stripped socket options from duplicate listen + # the 2nd and 3rd server blocks at daemon startup — the wrapper + # logs a NOTICE per stripped listen via ngx_conf_log_error. We + # read the container's startup log rather than `nginx -t`'s + # output because nginx keeps the config-parse log level at ERR + # until the error_log directive has been fully applied, so + # `nginx -t` suppresses the NOTICE even though the wrapper is + # invoked. + ${rc} ${startup_log} = Run And Return Rc And Output + ... docker logs ${SERVER} 2>&1 + Should Contain ${startup_log} stripped socket options from duplicate listen Prometheus scrape [Documentation] Scrape returns HELP/TYPE preamble. @@ -139,6 +146,53 @@ Duration histogram Should Contain ${json} request_duration_ms Should Contain ${json} buckets +# --- In-flight gauges --- + +Gauge preamble in Prometheus output + [Documentation] The scrape preamble advertises the three gauge metric + ... names and types. Present whether or not any traffic + ... has ever been observed. + ${output} = Scrape Prometheus + Should Match Regexp ${output} (?m)^# TYPE nginx_ipng_active gauge$ + Should Match Regexp ${output} (?m)^# TYPE nginx_ipng_reading gauge$ + Should Match Regexp ${output} (?m)^# TYPE nginx_ipng_writing gauge$ + +Gauges zero at rest + [Documentation] After the earlier traffic tests have drained and the + ... flush tick has run, the in-flight gauges for (tag1, + ... 10.0.1.1) must all report zero — no request is still + ... resident in the plugin's per-request ctx. + Wait For Flush + ${output} = Scrape With Filter source_tag=tag1&vip=10.0.1.1 + Should Match Regexp ${output} nginx_ipng_active\\{[^}]*\\}\\s+0\\b + Should Match Regexp ${output} nginx_ipng_reading\\{[^}]*\\}\\s+0\\b + Should Match Regexp ${output} nginx_ipng_writing\\{[^}]*\\}\\s+0\\b + +Gauges appear in JSON output + [Documentation] Each record in the JSON output carries a gauges object + ... with active/reading/writing keys. + ${rc} ${output} = Run And Return Rc And Output + ... curl -sf -H 'Accept: application/json' '${SCRAPE_URL}?source_tag=tag1&vip=10.0.1.1' | python3 -m json.tool + Should Be Equal As Integers ${rc} 0 + Should Contain ${output} "gauges" + Should Contain ${output} "active" + Should Contain ${output} "reading" + Should Contain ${output} "writing" + +Gauges observed non-zero under concurrent load + [Documentation] Fire a burst of concurrent requests against the + ... /slow backend (50 ms each, single-threaded) and + ... observe at least one of active/reading/writing + ... going non-zero for (tag1, 10.0.1.1) while the + ... burst is in flight. After it drains the gauges + ... return to zero. Uses 20 requests (drain <= ~2s + ... given the single-threaded backend) to keep the + ... test fast but still wide enough for the scrape + ... to catch overlap. + Run docker exec -d ${CLIENT1} sh -c 'for i in $(seq 20); do curl -sf http://10.0.1.1:8080/slow >/dev/null & done; wait' + Wait Until Keyword Succeeds 5s 50ms Nonzero Gauge Observed tag1 10.0.1.1 + Wait Until Keyword Succeeds 15s 500ms Gauges Drained tag1 10.0.1.1 + # --- Scrape filters --- Filter by source_tag @@ -347,6 +401,26 @@ Get Request Count END RETURN ${total} +Nonzero Gauge Observed + [Documentation] Fails unless the scrape shows a non-zero value for at + ... least one of {active, reading, writing} on the given + ... (source_tag, vip) pair. + [Arguments] ${source} ${vip} + ${output} = Scrape With Filter source_tag=${source}&vip=${vip} + Should Match Regexp ${output} + ... nginx_ipng_(active|reading|writing)\\{[^}]*\\}\\s+[1-9]\\d* + +Gauges Drained + [Documentation] Fails unless all three gauges are exactly zero for + ... the given (source_tag, vip) pair. Use with Wait + ... Until Keyword Succeeds to poll until a request burst + ... has fully finalized. + [Arguments] ${source} ${vip} + ${output} = Scrape With Filter source_tag=${source}&vip=${vip} + Should Match Regexp ${output} nginx_ipng_active\\{[^}]*\\}\\s+0\\b + Should Match Regexp ${output} nginx_ipng_reading\\{[^}]*\\}\\s+0\\b + Should Match Regexp ${output} nginx_ipng_writing\\{[^}]*\\}\\s+0\\b + # --- Container helpers --- Docker Exec