Wire-format and metric overhaul. Both file and UDP ingest now share one
versioned ParseLine that dispatches on the v<N>\t prefix; v1 stays
unchanged, v2 adds $bytes_sent (replacing $body_bytes_sent),
$request_length, $upstream_response_time, and $upstream_status. File
ingest gains the same versioning, and the legacy positional file format
is removed (no live deployments).
Prometheus exposition is rewritten:
- nginx_http_bytes_sent and nginx_http_request_duration_seconds gain
a source_tag label.
- nginx_http_requests_by_source_total gains status_class.
- New v2-only metrics: nginx_http_request_bytes,
nginx_http_upstream_duration_seconds,
nginx_http_upstream_requests_total{status_class}.
- Dropped nginx_http_response_body_bytes_by_source (subsumed by the
dual-labeled bytes_sent metric).
Adds 'make fixstyle' (gofmt -w) and clears all golangci-lint findings
across the repo (errcheck, S1001, ST1005, unused).
Docs in design.md FR-2/FR-8 and user-guide.md are rewritten to present
v2 as the recommended log format.
nginx-ipng-stats-plugin's ipng_stats_logtail directive buffers many log
lines into a single UDP datagram (default buffer=64k flush=1s). The
listener was treating each datagram as exactly one log line, so any
datagram with N>1 lines failed the v1 field-count check and dropped
silently. In production this showed up as logtail_udp_packets_received_total
roughly 4x logtail_udp_loglines_success_total — matching typical
burst-coalesced 4-lines-per-batch ratios.
Fix: strip trailing CRLF, split the payload on '\n', parse each
non-empty line independently. Counter semantics now match the names:
packets_received — datagrams off the socket (one per recvfrom)
loglines_success — log lines parsed OK (may be many per datagram)
loglines_consumed — log lines forwarded to the store (not dropped)
After the fix, loglines_success ≈ packets_received × avg_lines_per_batch.
Regression test TestUDPListenerBatchedDatagram sends one datagram with
three '\n'-separated v1 lines and asserts all three LogRecords arrive,
plus loglines_success >= 3 * packets_received.
Docs (user-guide.md, design.md) now explain the datagram-vs-line unit
distinction so operators don't misread the ratio.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
No runtime change — the listener already uses net.ListenUDP +
ReadFromUDP, which is the unconnected-socket pattern that accepts
datagrams from any source. nginx reloads (new workers with fresh
ephemeral source ports) are handled transparently.
- udp.go: expanded comment on Run() explaining the design choice and
contrasting with the `nc -k -u -l` latching quirk (which is an nc
bug, not a kernel behaviour).
- udp_test.go: new TestUDPListenerMultipleSources regresses against
the multi-worker scenario by sending from three independent
ListenPacket sockets (three different ephemeral source ports).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pim van Pelt