nginx-ipng-stats-plugin/docs/nginx-issues.md

Upstream nginx quirks that affect this plugin

This file catalogs behaviors of nginx itself (not bugs in this plugin) that have a visible effect on the counters the plugin exposes. It exists so a future reader can page the context back in quickly instead of re-deriving it from the nginx source.

All path:line citations below are into the nginx source tree (checked against tip as of 2026-04), not into this repository.

TL;DR

• r->connection->sent is per-request (reset between HTTP/1.1 keepalive requests, per-stream on HTTP/2). Safe to += in the log phase.
• r->request_length is not a faithful count of bytes received from the client. It is only updated on specific code paths inside nginx, and the discard-body path — which is taken on every request that nginx rejects before a handler reads the body — bypasses it entirely. On a workload with lots of early rejections (rate-limit 403, auth_request denial, 413 with fixed Content-Length, return 4xx; on POST), $request_length and therefore nginx_ipng_bytes_in_total under-count the wire traffic substantially.

r->request_length — where it is, and is not, updated

Path	Updates request_length?	Source
Buffered body (ngx_http_do_read_client_request_body)	yes	ngx_http_request_body.c:397 (after every recv), preread at :120
Unbuffered body (ngx_http_read_unbuffered_request_body)	yes (HTTP/1 — delegates to the buffered path)	ngx_http_request_body.c:265
Discard body (ngx_http_discard_request_body → ngx_http_read_discarded_request_body)	NO	ngx_http_request_body.c:782-841 — reads into a stack buffer, never touches request_length
Chunked HTTP/1	yes (indirect)	Wire bytes counted at :397 before the chunked parser runs
HTTP/2 DATA payload	yes (payload only)	ngx_http_v2_filter_request_body at v2/ngx_http_v2.c:4201
HTTP/2 DATA via stream->skip_data	NO	v2/ngx_http_v2.c:1077-1082 — skipped stream data is silently dropped
HTTP/2 frame headers, SETTINGS, WINDOW_UPDATE, PING, PRIORITY, RST_STREAM, GOAWAY, padding	NO	Not counted anywhere
HTTP/2 HEADERS / CONTINUATION	yes (decoded size only)	v2/ngx_http_v2.c:1333, 1968
HTTP/3 DATA	yes (payload only)	v3/ngx_http_v3_request.c:1559, 1614, 1660, 1668
Pipelined-request carry-over in header_in	subtracted back out	ngx_http_request_body.c:541 (r->request_length -= n)
TLS handshake / record framing	never visible to nginx	Below c->recv

413 client_max_body_size specifically

• Fixed Content-Length: ngx_http_core_module.c:1006-1018 detects the overrun right after headers are parsed and calls ngx_http_discard_request_body() before finalizing 413. That path does not bump request_length, so $request_length ends up reflecting only the request line + headers. Body bytes that actually streamed in before nginx closed are invisible.
• Chunked HTTP/1: ngx_http_request_body.c:1142-1154 trips 413 inside the body filter after the wire bytes have already been counted at :397, so they are included — but only up to what nginx had recved before returning.
• HTTP/2 chunked: v2/ngx_http_v2.c:4216-4223 returns 413 after the r->request_length += at :4201, so partial-body bytes do count.

The common "rate-limit 403" pattern

limit_req rejects in PREACCESS. On rejection, nginx finalizes with 403, which calls ngx_http_discard_request_body(). On HTTP/1 this sends the body into the stack-buffer discard reader (not counted). On HTTP/2 it sets the stream's skip_data flag at v2/ngx_http_v2.c:1077-1082, so the client's DATA frames are parsed and their payload silently dropped (also not counted). Either way, $request_length reflects headers only.

This is the dominant source of under-counting on rate-limited abusive workloads. Observed example: a Certificate Transparency client hammering POST /ct/v1/add-chain at ~460 req/s, each carrying a ~10 KB cert chain in the body, all hitting 403. 460 × 10 KB = ~4.6 MB/s of inbound bytes completely absent from $request_length.

r->connection->sent — this one is fine

• HTTP/1.1 keepalive: reset at ngx_http_request.c:3493 (in ngx_http_keepalive_handler, which begins at :3356) when a new request arrives on an idle keepalive connection, and at ngx_http_request.c:3226 (in ngx_http_set_keepalive, which begins at :3128) for the pipelined-next-request case.
• HTTP/2: each request has a fake per-stream connection fc initialized with fc->sent = 0 at v2/ngx_http_v2.c:3039-3044, so r->connection->sent is per-stream. Framing overhead on the real TCP connection (HEADERS, SETTINGS, WINDOW_UPDATE, PING, GOAWAY) is invisible to r->connection->sent — that is consistent with how $bytes_sent behaves and is fine for per-request accounting.
• ngx_http_log_module.c:880 (ngx_http_log_bytes_sent) just prints r->connection->sent, confirming this is the same value $bytes_sent exposes.

The plugin's slot->bytes_out += r->connection->sent in the log handler is therefore correct.

What the plugin does about it

Option A (implemented): Content-Length fallback on discard

In the log handler, when r->discard_body is set and r->headers_in.content_length_n > 0, the plugin adds content_length_n to bin_sz instead of trusting r->request_length.

• Covers the rate-limit-403 / auth-denial / 413-fixed-CL / early return 4xx; cases on any request where the client advertised a Content-Length (HTTP/1 or HTTP/2).
• Does not cover chunked requests with no Content-Length header (those already count whatever bytes nginx recved before closing; the post-close bytes are gone in any accounting scheme).
• Over-counts when a client announces a Content-Length then hangs up partway through (nginx never received the remainder). In practice, abusive clients like the CT log hammer case send the full body before nginx's RST propagates, so the over-count is small.
• Under-counts HTTP/2 framing overhead (frame headers, control frames). That overhead is small (~1% at typical DATA frame sizes) and can be ignored for a first cut.

See src/ngx_http_ipng_stats_module.c — search for discard_body.

Option B (not implemented): per-connection recv wrapping

Replace c->recv on each accepted HTTP connection with a wrapper that increments a per-connection byte counter, then delegates to the original. At log phase, use (counter_at_log - counter_at_request_start) as the authoritative bytes_in. Captures discarded bodies, HTTP/2 framing, HTTP/2 skip_data bodies, partial uploads, and chunked — everything that crossed nginx's c->recv. TLS/TCP overhead remains invisible (below the recv abstraction). Revisit if the A-residual gap turns out to matter.