Pim van Pelt
143aad9063
Build and release tooling:
- Makefile with help as default; targets: build/build-amd64/build-arm64,
test, lint, proto, pkg-deb, docker, docker-push, clean, plus
install-deps (+ three sub-targets for apt / Go toolchain / Go tools).
- internal/version package; -ldflags -X injects Version/Commit/Date into
every binary. -version flag on all four binaries (nginx-logtail version
for the CLI).
- Dockerfile takes VERSION/COMMIT/DATE build-args and forwards them.
- .deb output lands in build/; .gitignore ignores /build/.
Debian package:
- debian/build-deb.sh packages all four static binaries into a single
nginx-logtail_<ver>_<arch>.deb using dpkg-deb.
- Binary layout: /usr/sbin/nginx-logtail-{collector,aggregator,frontend}
and /usr/bin/nginx-logtail.
- nginx-logtail(8) manpage.
- Three systemd units (collector, aggregator, frontend) shipped under
/lib/systemd/system/. Installed but never enabled or started — the
operator opts in per host.
- Collector runs as _logtail:www-data (log access); aggregator and
frontend as _logtail:_logtail. postinst creates the system user/group
idempotently.
- Single shared env file /etc/default/nginx-logtail rendered from a
template at first install with %HOSTNAME% substituted. Sensible
defaults for every COLLECTOR_*, AGGREGATOR_*, FRONTEND_* variable;
plus COLLECTOR_ARGS / AGGREGATOR_ARGS / FRONTEND_ARGS escape hatches
appended to ExecStart. Not a dpkg conffile: operator edits survive
upgrades and dpkg --purge removes it.
Versioned UDP wire format:
- ParseUDPLine dispatches on a leading "v<N>\t" tag; v1 routes to the
existing 12-field parser. Unknown/missing versions fail closed so
future v2 parsers can land before emitters are upgraded.
- Tests updated; design.md FR-2.2 rewritten to make the version tag
normative.
Docs:
- README.md gains a Quick Start (Debian / Docker Compose / from source).
- user-guide.md rewritten around Installation and Configuration: full
env-var table, UDP-only default explained, precise file/UDP log_format
layouts, note that operators can emit "0" for unknown \$is_tor / \$asn.
- Drilldown cycle, frontend filter table, and CLI --group-by list all
include source_tag. UDP counters documented in the Prometheus section.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
241 lines
5.7 KiB
Groff
241 lines
5.7 KiB
Groff
.TH NGINX-LOGTAIL 8 "April 2026" "nginx-logtail 0.9.1" "System Manager's Manual"
|
|
.SH NAME
|
|
nginx-logtail \- real-time top-K traffic analysis for nginx clusters
|
|
.SH SYNOPSIS
|
|
.B nginx-logtail-collector
|
|
.RI [ options ]
|
|
.br
|
|
.B nginx-logtail-aggregator
|
|
.RI [ options ]
|
|
.br
|
|
.B nginx-logtail-frontend
|
|
.RI [ options ]
|
|
.br
|
|
.B nginx-logtail
|
|
.IR subcommand
|
|
.RI [ options ]
|
|
.SH DESCRIPTION
|
|
.PP
|
|
.B nginx-logtail
|
|
is a four-binary Go system for real-time analysis of nginx traffic across a
|
|
fleet of hosts. Each nginx host runs a
|
|
.B collector
|
|
that ingests logs from files (via
|
|
.BR fsnotify ),
|
|
from a UDP socket (fed by the
|
|
.B nginx-ipng-stats-plugin
|
|
\fBipng_stats_logtail\fR directive), or both. The collector maintains
|
|
in-memory ranked top-K counters over 1m/5m/15m/60m/6h/24h windows and
|
|
exposes them via gRPC on
|
|
.IR :9090 .
|
|
A central
|
|
.B aggregator
|
|
subscribes to every collector, merges their snapshot streams, and serves
|
|
the same gRPC contract on
|
|
.IR :9091 .
|
|
The
|
|
.B frontend
|
|
renders a server-side HTML dashboard (no JavaScript) on
|
|
.I :8080
|
|
against any
|
|
.I LogtailService
|
|
endpoint. The CLI,
|
|
.BR nginx-logtail ,
|
|
offers the same queries as a shell companion.
|
|
.PP
|
|
Operators typically run the collector on every nginx host as a systemd
|
|
unit, the aggregator and frontend on a central host (either as systemd
|
|
units or via the shipped
|
|
.B docker-compose.yml
|
|
), and invoke
|
|
.B nginx-logtail
|
|
from an operator laptop.
|
|
.PP
|
|
The Debian package installs three systemd units —
|
|
.BR nginx-logtail-collector.service ,
|
|
.BR nginx-logtail-aggregator.service ,
|
|
.BR nginx-logtail-frontend.service
|
|
— under
|
|
.IR /lib/systemd/system/ .
|
|
None are enabled or started on install: the operator opts into each
|
|
service per-host with
|
|
.BR "systemctl enable --now" .
|
|
Services run as the system user
|
|
.B _logtail
|
|
(created by
|
|
.BR postinst ).
|
|
The collector uses
|
|
.B _logtail:www-data
|
|
so it can read nginx access logs that are group-readable by
|
|
.BR www-data ;
|
|
the aggregator and frontend use
|
|
.BR _logtail:_logtail .
|
|
All three units read a single environment file,
|
|
.IR /etc/default/nginx-logtail ,
|
|
generated by the package's postinst on first install (with the short
|
|
hostname substituted for
|
|
.B COLLECTOR_SOURCE
|
|
and
|
|
.BR AGGREGATOR_SOURCE ).
|
|
The file is not a dpkg conffile: the template lives at
|
|
.IR /usr/share/nginx-logtail/default.template ,
|
|
the operator's edits to
|
|
.I /etc/default/nginx-logtail
|
|
survive upgrades, and
|
|
.B dpkg --purge
|
|
removes it.
|
|
Every flag of every binary has a matching
|
|
.BR COLLECTOR_* ,
|
|
.BR AGGREGATOR_* ,
|
|
or
|
|
.BR FRONTEND_ *
|
|
env var; set them in the defaults file. For flags without an env-var form,
|
|
or temporary overrides, append to
|
|
.BR COLLECTOR_ARGS ,
|
|
.BR AGGREGATOR_ARGS ,
|
|
or
|
|
.BR FRONTEND_ARGS .
|
|
.SH COMPONENTS
|
|
.TP
|
|
.B nginx-logtail-collector
|
|
Installed in
|
|
.IR /usr/sbin .
|
|
Tails nginx access logs and/or receives UDP datagrams on
|
|
.B --logtail-port
|
|
(default disabled). Exposes
|
|
.I LogtailService
|
|
gRPC on
|
|
.B --listen
|
|
(default
|
|
.IR :9090 )
|
|
and Prometheus metrics on
|
|
.B --prom-listen
|
|
(default
|
|
.IR :9100 ).
|
|
Pass
|
|
.B --version
|
|
to print build metadata.
|
|
.TP
|
|
.B nginx-logtail-aggregator
|
|
Installed in
|
|
.IR /usr/sbin .
|
|
Subscribes to each address in
|
|
.B --collectors
|
|
and merges their streams. Serves
|
|
.I LogtailService
|
|
on
|
|
.B --listen
|
|
(default
|
|
.IR :9091 ).
|
|
On restart, backfills its ring buffers from every collector via
|
|
.IR DumpSnapshots .
|
|
.TP
|
|
.B nginx-logtail-frontend
|
|
Installed in
|
|
.IR /usr/sbin .
|
|
HTTP dashboard on
|
|
.B --listen
|
|
(default
|
|
.IR :8080 )
|
|
against
|
|
.B --target
|
|
(default
|
|
.IR localhost:9091 ,
|
|
the aggregator). URL-driven filter state; append
|
|
.I &raw=1
|
|
to any dashboard URL for JSON output.
|
|
.TP
|
|
.B nginx-logtail
|
|
Installed in
|
|
.IR /usr/bin .
|
|
CLI for
|
|
.BR topn ,
|
|
.BR trend ,
|
|
.BR stream ,
|
|
and
|
|
.B targets
|
|
queries. Accepts
|
|
.BI \-\-target " host:port[,host:port...]"
|
|
for concurrent fan-out.
|
|
.SH FILES
|
|
.TP
|
|
.I /usr/sbin/nginx-logtail-collector
|
|
Collector daemon binary.
|
|
.TP
|
|
.I /usr/sbin/nginx-logtail-aggregator
|
|
Aggregator daemon binary.
|
|
.TP
|
|
.I /usr/sbin/nginx-logtail-frontend
|
|
Frontend HTTP server binary.
|
|
.TP
|
|
.I /usr/bin/nginx-logtail
|
|
CLI binary.
|
|
.TP
|
|
.I /usr/share/doc/nginx-logtail/
|
|
README, copyright, and pointer to the design and user-guide documents.
|
|
.SH EXAMPLES
|
|
.PP
|
|
Run a collector reading one log file and listening on UDP 9514:
|
|
.PP
|
|
.RS
|
|
.nf
|
|
nginx-logtail-collector \\
|
|
--logs /var/log/nginx/access.log \\
|
|
--logtail-port 9514 \\
|
|
--source $(hostname)
|
|
.fi
|
|
.RE
|
|
.PP
|
|
Query the top 10 websites over the last 5 minutes:
|
|
.PP
|
|
.RS
|
|
.nf
|
|
nginx-logtail topn --target agg:9091 --window 5m --n 10
|
|
.fi
|
|
.RE
|
|
.PP
|
|
Show all HTTP 429s by client prefix over the last minute:
|
|
.PP
|
|
.RS
|
|
.nf
|
|
nginx-logtail topn --target agg:9091 --window 1m \\
|
|
--group-by prefix --status 429
|
|
.fi
|
|
.RE
|
|
.SH ENVIRONMENT
|
|
All three daemons read
|
|
.IR /etc/default/nginx-logtail .
|
|
The file is self-documenting — every env var each binary recognises is
|
|
listed with a short description and its default value. Representative
|
|
variables:
|
|
.IP \fBCOLLECTOR_LOGS\fR
|
|
Comma-separated log file paths or globs.
|
|
.IP \fBCOLLECTOR_LOGTAIL_PORT\fR
|
|
UDP port for
|
|
.I ipng_stats_logtail
|
|
input; 0 disables the listener.
|
|
.IP \fBAGGREGATOR_COLLECTORS\fR
|
|
Comma-separated collector addresses. Mandatory.
|
|
.IP \fBFRONTEND_TARGET\fR
|
|
gRPC endpoint the frontend queries (aggregator or collector).
|
|
.IP \fBCOLLECTOR_ARGS\fR, \fBAGGREGATOR_ARGS\fR, \fBFRONTEND_ARGS\fR
|
|
Raw argv appended after the env-var-derived flags; use for flags
|
|
without an env-var form or for temporary overrides.
|
|
.SH SECURITY
|
|
gRPC endpoints are cleartext HTTP/2 by default. The UDP listener binds to
|
|
.I 127.0.0.1
|
|
unless
|
|
.B --logtail-bind
|
|
is set explicitly. Expose beyond a trusted network only behind a TLS
|
|
terminator.
|
|
.SH SEE ALSO
|
|
.BR nginx (8),
|
|
.BR systemd (1).
|
|
.PP
|
|
Full design and operator guide:
|
|
.IR /usr/share/doc/nginx-logtail/README.md .
|
|
.SH AUTHORS
|
|
Pim van Pelt <pim@ipng.ch>, with Claude Code.
|
|
.SH BUGS
|
|
Report issues at https://git.ipng.ch/ipng/nginx-logtail.
|