PRE-RELEASE 0.9.1: Makefile, Debian packaging, versioned UDP
Build and release tooling:
- Makefile with help as default; targets: build/build-amd64/build-arm64,
test, lint, proto, pkg-deb, docker, docker-push, clean, plus
install-deps (+ three sub-targets for apt / Go toolchain / Go tools).
- internal/version package; -ldflags -X injects Version/Commit/Date into
every binary. -version flag on all four binaries (nginx-logtail version
for the CLI).
- Dockerfile takes VERSION/COMMIT/DATE build-args and forwards them.
- .deb output lands in build/; .gitignore ignores /build/.
Debian package:
- debian/build-deb.sh packages all four static binaries into a single
nginx-logtail_<ver>_<arch>.deb using dpkg-deb.
- Binary layout: /usr/sbin/nginx-logtail-{collector,aggregator,frontend}
and /usr/bin/nginx-logtail.
- nginx-logtail(8) manpage.
- Three systemd units (collector, aggregator, frontend) shipped under
/lib/systemd/system/. Installed but never enabled or started — the
operator opts in per host.
- Collector runs as _logtail:www-data (log access); aggregator and
frontend as _logtail:_logtail. postinst creates the system user/group
idempotently.
- Single shared env file /etc/default/nginx-logtail rendered from a
template at first install with %HOSTNAME% substituted. Sensible
defaults for every COLLECTOR_*, AGGREGATOR_*, FRONTEND_* variable;
plus COLLECTOR_ARGS / AGGREGATOR_ARGS / FRONTEND_ARGS escape hatches
appended to ExecStart. Not a dpkg conffile: operator edits survive
upgrades and dpkg --purge removes it.
Versioned UDP wire format:
- ParseUDPLine dispatches on a leading "v<N>\t" tag; v1 routes to the
existing 12-field parser. Unknown/missing versions fail closed so
future v2 parsers can land before emitters are upgraded.
- Tests updated; design.md FR-2.2 rewritten to make the version tag
normative.
Docs:
- README.md gains a Quick Start (Debian / Docker Compose / from source).
- user-guide.md rewritten around Installation and Configuration: full
env-var table, UDP-only default explained, precise file/UDP log_format
layouts, note that operators can emit "0" for unknown \$is_tor / \$asn.
- Drilldown cycle, frontend filter table, and CLI --group-by list all
include source_tag. UDP counters documented in the Prometheus section.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
240
debian/nginx-logtail.8
vendored
Normal file
240
debian/nginx-logtail.8
vendored
Normal file
@@ -0,0 +1,240 @@
|
||||
.TH NGINX-LOGTAIL 8 "April 2026" "nginx-logtail 0.9.1" "System Manager's Manual"
|
||||
.SH NAME
|
||||
nginx-logtail \- real-time top-K traffic analysis for nginx clusters
|
||||
.SH SYNOPSIS
|
||||
.B nginx-logtail-collector
|
||||
.RI [ options ]
|
||||
.br
|
||||
.B nginx-logtail-aggregator
|
||||
.RI [ options ]
|
||||
.br
|
||||
.B nginx-logtail-frontend
|
||||
.RI [ options ]
|
||||
.br
|
||||
.B nginx-logtail
|
||||
.IR subcommand
|
||||
.RI [ options ]
|
||||
.SH DESCRIPTION
|
||||
.PP
|
||||
.B nginx-logtail
|
||||
is a four-binary Go system for real-time analysis of nginx traffic across a
|
||||
fleet of hosts. Each nginx host runs a
|
||||
.B collector
|
||||
that ingests logs from files (via
|
||||
.BR fsnotify ),
|
||||
from a UDP socket (fed by the
|
||||
.B nginx-ipng-stats-plugin
|
||||
\fBipng_stats_logtail\fR directive), or both. The collector maintains
|
||||
in-memory ranked top-K counters over 1m/5m/15m/60m/6h/24h windows and
|
||||
exposes them via gRPC on
|
||||
.IR :9090 .
|
||||
A central
|
||||
.B aggregator
|
||||
subscribes to every collector, merges their snapshot streams, and serves
|
||||
the same gRPC contract on
|
||||
.IR :9091 .
|
||||
The
|
||||
.B frontend
|
||||
renders a server-side HTML dashboard (no JavaScript) on
|
||||
.I :8080
|
||||
against any
|
||||
.I LogtailService
|
||||
endpoint. The CLI,
|
||||
.BR nginx-logtail ,
|
||||
offers the same queries as a shell companion.
|
||||
.PP
|
||||
Operators typically run the collector on every nginx host as a systemd
|
||||
unit, the aggregator and frontend on a central host (either as systemd
|
||||
units or via the shipped
|
||||
.B docker-compose.yml
|
||||
), and invoke
|
||||
.B nginx-logtail
|
||||
from an operator laptop.
|
||||
.PP
|
||||
The Debian package installs three systemd units —
|
||||
.BR nginx-logtail-collector.service ,
|
||||
.BR nginx-logtail-aggregator.service ,
|
||||
.BR nginx-logtail-frontend.service
|
||||
— under
|
||||
.IR /lib/systemd/system/ .
|
||||
None are enabled or started on install: the operator opts into each
|
||||
service per-host with
|
||||
.BR "systemctl enable --now" .
|
||||
Services run as the system user
|
||||
.B _logtail
|
||||
(created by
|
||||
.BR postinst ).
|
||||
The collector uses
|
||||
.B _logtail:www-data
|
||||
so it can read nginx access logs that are group-readable by
|
||||
.BR www-data ;
|
||||
the aggregator and frontend use
|
||||
.BR _logtail:_logtail .
|
||||
All three units read a single environment file,
|
||||
.IR /etc/default/nginx-logtail ,
|
||||
generated by the package's postinst on first install (with the short
|
||||
hostname substituted for
|
||||
.B COLLECTOR_SOURCE
|
||||
and
|
||||
.BR AGGREGATOR_SOURCE ).
|
||||
The file is not a dpkg conffile: the template lives at
|
||||
.IR /usr/share/nginx-logtail/default.template ,
|
||||
the operator's edits to
|
||||
.I /etc/default/nginx-logtail
|
||||
survive upgrades, and
|
||||
.B dpkg --purge
|
||||
removes it.
|
||||
Every flag of every binary has a matching
|
||||
.BR COLLECTOR_* ,
|
||||
.BR AGGREGATOR_* ,
|
||||
or
|
||||
.BR FRONTEND_ *
|
||||
env var; set them in the defaults file. For flags without an env-var form,
|
||||
or temporary overrides, append to
|
||||
.BR COLLECTOR_ARGS ,
|
||||
.BR AGGREGATOR_ARGS ,
|
||||
or
|
||||
.BR FRONTEND_ARGS .
|
||||
.SH COMPONENTS
|
||||
.TP
|
||||
.B nginx-logtail-collector
|
||||
Installed in
|
||||
.IR /usr/sbin .
|
||||
Tails nginx access logs and/or receives UDP datagrams on
|
||||
.B --logtail-port
|
||||
(default disabled). Exposes
|
||||
.I LogtailService
|
||||
gRPC on
|
||||
.B --listen
|
||||
(default
|
||||
.IR :9090 )
|
||||
and Prometheus metrics on
|
||||
.B --prom-listen
|
||||
(default
|
||||
.IR :9100 ).
|
||||
Pass
|
||||
.B --version
|
||||
to print build metadata.
|
||||
.TP
|
||||
.B nginx-logtail-aggregator
|
||||
Installed in
|
||||
.IR /usr/sbin .
|
||||
Subscribes to each address in
|
||||
.B --collectors
|
||||
and merges their streams. Serves
|
||||
.I LogtailService
|
||||
on
|
||||
.B --listen
|
||||
(default
|
||||
.IR :9091 ).
|
||||
On restart, backfills its ring buffers from every collector via
|
||||
.IR DumpSnapshots .
|
||||
.TP
|
||||
.B nginx-logtail-frontend
|
||||
Installed in
|
||||
.IR /usr/sbin .
|
||||
HTTP dashboard on
|
||||
.B --listen
|
||||
(default
|
||||
.IR :8080 )
|
||||
against
|
||||
.B --target
|
||||
(default
|
||||
.IR localhost:9091 ,
|
||||
the aggregator). URL-driven filter state; append
|
||||
.I &raw=1
|
||||
to any dashboard URL for JSON output.
|
||||
.TP
|
||||
.B nginx-logtail
|
||||
Installed in
|
||||
.IR /usr/bin .
|
||||
CLI for
|
||||
.BR topn ,
|
||||
.BR trend ,
|
||||
.BR stream ,
|
||||
and
|
||||
.B targets
|
||||
queries. Accepts
|
||||
.BI \-\-target " host:port[,host:port...]"
|
||||
for concurrent fan-out.
|
||||
.SH FILES
|
||||
.TP
|
||||
.I /usr/sbin/nginx-logtail-collector
|
||||
Collector daemon binary.
|
||||
.TP
|
||||
.I /usr/sbin/nginx-logtail-aggregator
|
||||
Aggregator daemon binary.
|
||||
.TP
|
||||
.I /usr/sbin/nginx-logtail-frontend
|
||||
Frontend HTTP server binary.
|
||||
.TP
|
||||
.I /usr/bin/nginx-logtail
|
||||
CLI binary.
|
||||
.TP
|
||||
.I /usr/share/doc/nginx-logtail/
|
||||
README, copyright, and pointer to the design and user-guide documents.
|
||||
.SH EXAMPLES
|
||||
.PP
|
||||
Run a collector reading one log file and listening on UDP 9514:
|
||||
.PP
|
||||
.RS
|
||||
.nf
|
||||
nginx-logtail-collector \\
|
||||
--logs /var/log/nginx/access.log \\
|
||||
--logtail-port 9514 \\
|
||||
--source $(hostname)
|
||||
.fi
|
||||
.RE
|
||||
.PP
|
||||
Query the top 10 websites over the last 5 minutes:
|
||||
.PP
|
||||
.RS
|
||||
.nf
|
||||
nginx-logtail topn --target agg:9091 --window 5m --n 10
|
||||
.fi
|
||||
.RE
|
||||
.PP
|
||||
Show all HTTP 429s by client prefix over the last minute:
|
||||
.PP
|
||||
.RS
|
||||
.nf
|
||||
nginx-logtail topn --target agg:9091 --window 1m \\
|
||||
--group-by prefix --status 429
|
||||
.fi
|
||||
.RE
|
||||
.SH ENVIRONMENT
|
||||
All three daemons read
|
||||
.IR /etc/default/nginx-logtail .
|
||||
The file is self-documenting — every env var each binary recognises is
|
||||
listed with a short description and its default value. Representative
|
||||
variables:
|
||||
.IP \fBCOLLECTOR_LOGS\fR
|
||||
Comma-separated log file paths or globs.
|
||||
.IP \fBCOLLECTOR_LOGTAIL_PORT\fR
|
||||
UDP port for
|
||||
.I ipng_stats_logtail
|
||||
input; 0 disables the listener.
|
||||
.IP \fBAGGREGATOR_COLLECTORS\fR
|
||||
Comma-separated collector addresses. Mandatory.
|
||||
.IP \fBFRONTEND_TARGET\fR
|
||||
gRPC endpoint the frontend queries (aggregator or collector).
|
||||
.IP \fBCOLLECTOR_ARGS\fR, \fBAGGREGATOR_ARGS\fR, \fBFRONTEND_ARGS\fR
|
||||
Raw argv appended after the env-var-derived flags; use for flags
|
||||
without an env-var form or for temporary overrides.
|
||||
.SH SECURITY
|
||||
gRPC endpoints are cleartext HTTP/2 by default. The UDP listener binds to
|
||||
.I 127.0.0.1
|
||||
unless
|
||||
.B --logtail-bind
|
||||
is set explicitly. Expose beyond a trusted network only behind a TLS
|
||||
terminator.
|
||||
.SH SEE ALSO
|
||||
.BR nginx (8),
|
||||
.BR systemd (1).
|
||||
.PP
|
||||
Full design and operator guide:
|
||||
.IR /usr/share/doc/nginx-logtail/README.md .
|
||||
.SH AUTHORS
|
||||
Pim van Pelt <pim@ipng.ch>, with Claude Code.
|
||||
.SH BUGS
|
||||
Report issues at https://git.ipng.ch/ipng/nginx-logtail.
|
||||
Reference in New Issue
Block a user