fix: UDP listener parses batched datagrams
nginx-ipng-stats-plugin's ipng_stats_logtail directive buffers many log lines into a single UDP datagram (default buffer=64k flush=1s). The listener was treating each datagram as exactly one log line, so any datagram with N>1 lines failed the v1 field-count check and dropped silently. In production this showed up as logtail_udp_packets_received_total roughly 4x logtail_udp_loglines_success_total — matching typical burst-coalesced 4-lines-per-batch ratios. Fix: strip trailing CRLF, split the payload on '\n', parse each non-empty line independently. Counter semantics now match the names: packets_received — datagrams off the socket (one per recvfrom) loglines_success — log lines parsed OK (may be many per datagram) loglines_consumed — log lines forwarded to the store (not dropped) After the fix, loglines_success ≈ packets_received × avg_lines_per_batch. Regression test TestUDPListenerBatchedDatagram sends one datagram with three '\n'-separated v1 lines and asserts all three LogRecords arrive, plus loglines_success >= 3 * packets_received. Docs (user-guide.md, design.md) now explain the datagram-vs-line unit distinction so operators don't misread the ratio. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -251,9 +251,10 @@ Each requirement carries a unique identifier (`FR-X.Y` or `NFR-X.Y`) so that lat
|
||||
`nginx_http_response_body_bytes_by_source{source_tag, le}`. These are separate metric names to avoid inconsistent label sets
|
||||
under a single name.
|
||||
- **FR-8.5** The collector MUST expose three counters that let operators distinguish UDP parse failures from back-pressure drops:
|
||||
`logtail_udp_packets_received_total` (datagrams off the socket),
|
||||
`logtail_udp_loglines_success_total` (parsed OK), and
|
||||
`logtail_udp_loglines_consumed_total` (forwarded to the store — i.e. not dropped).
|
||||
`logtail_udp_packets_received_total` (datagrams off the socket, one increment per `recvfrom`),
|
||||
`logtail_udp_loglines_success_total` (log lines that parsed OK, incremented once per log line — a single batched datagram from
|
||||
the nginx plugin may contribute many), and
|
||||
`logtail_udp_loglines_consumed_total` (log lines forwarded to the store channel — i.e. not dropped by back-pressure).
|
||||
|
||||
### Non-Functional Requirements
|
||||
|
||||
@@ -539,8 +540,8 @@ Primary channel is the collector's Prometheus endpoint (FR-8). Beyond the per-ho
|
||||
three UDP counters give direct visibility into the UDP ingest path:
|
||||
|
||||
- `logtail_udp_packets_received_total` — what arrived.
|
||||
- `logtail_udp_loglines_success_total` — what parsed cleanly.
|
||||
- `logtail_udp_loglines_consumed_total` — what made it to the store (i.e. was not dropped by a full channel).
|
||||
- `logtail_udp_loglines_success_total` — log lines that parsed cleanly (one datagram may contribute many).
|
||||
- `logtail_udp_loglines_consumed_total` — log lines that made it to the store (i.e. were not dropped by a full channel).
|
||||
|
||||
`received - success` is the parse-failure rate; `success - consumed` is the back-pressure drop rate. Operators should alert on both
|
||||
being non-zero.
|
||||
|
||||
Reference in New Issue
Block a user