ipng/nginx-logtail
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
589030cb00b1bd0e230f981d078223b425d24455
nginx-logtail/debian/nginx-logtail.8
Pim van Pelt 143aad9063 PRE-RELEASE 0.9.1: Makefile, Debian packaging, versioned UDP 
Build and release tooling:
- Makefile with help as default; targets: build/build-amd64/build-arm64,
  test, lint, proto, pkg-deb, docker, docker-push, clean, plus
  install-deps (+ three sub-targets for apt / Go toolchain / Go tools).
- internal/version package; -ldflags -X injects Version/Commit/Date into
  every binary. -version flag on all four binaries (nginx-logtail version
  for the CLI).
- Dockerfile takes VERSION/COMMIT/DATE build-args and forwards them.
- .deb output lands in build/; .gitignore ignores /build/.

Debian package:
- debian/build-deb.sh packages all four static binaries into a single
  nginx-logtail_<ver>_<arch>.deb using dpkg-deb.
- Binary layout: /usr/sbin/nginx-logtail-{collector,aggregator,frontend}
  and /usr/bin/nginx-logtail.
- nginx-logtail(8) manpage.
- Three systemd units (collector, aggregator, frontend) shipped under
  /lib/systemd/system/. Installed but never enabled or started — the
  operator opts in per host.
- Collector runs as _logtail:www-data (log access); aggregator and
  frontend as _logtail:_logtail. postinst creates the system user/group
  idempotently.
- Single shared env file /etc/default/nginx-logtail rendered from a
  template at first install with %HOSTNAME% substituted. Sensible
  defaults for every COLLECTOR_*, AGGREGATOR_*, FRONTEND_* variable;
  plus COLLECTOR_ARGS / AGGREGATOR_ARGS / FRONTEND_ARGS escape hatches
  appended to ExecStart. Not a dpkg conffile: operator edits survive
  upgrades and dpkg --purge removes it.

Versioned UDP wire format:
- ParseUDPLine dispatches on a leading "v<N>\t" tag; v1 routes to the
  existing 12-field parser. Unknown/missing versions fail closed so
  future v2 parsers can land before emitters are upgraded.
- Tests updated; design.md FR-2.2 rewritten to make the version tag
  normative.

Docs:
- README.md gains a Quick Start (Debian / Docker Compose / from source).
- user-guide.md rewritten around Installation and Configuration: full
  env-var table, UDP-only default explained, precise file/UDP log_format
  layouts, note that operators can emit "0" for unknown \$is_tor / \$asn.
- Drilldown cycle, frontend filter table, and CLI --group-by list all
  include source_tag. UDP counters documented in the Prometheus section.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-17 10:35:08 +02:00

241 lines
5.7 KiB
Groff
Raw Blame History

.TH NGINX-LOGTAIL 8 "April 2026" "nginx-logtail 0.9.1" "System Manager's Manual"
.SH NAME
nginx-logtail \- real-time top-K traffic analysis for nginx clusters
.SH SYNOPSIS
.B nginx-logtail-collector
.RI [ options ]
.br
.B nginx-logtail-aggregator
.RI [ options ]
.br
.B nginx-logtail-frontend
.RI [ options ]
.br
.B nginx-logtail
.IR subcommand
.RI [ options ]
.SH DESCRIPTION
.PP
.B nginx-logtail
is a four-binary Go system for real-time analysis of nginx traffic across a
fleet of hosts. Each nginx host runs a
.B collector
that ingests logs from files (via
.BR fsnotify ),
from a UDP socket (fed by the
.B nginx-ipng-stats-plugin
\fBipng_stats_logtail\fR directive), or both. The collector maintains
in-memory ranked top-K counters over 1m/5m/15m/60m/6h/24h windows and
exposes them via gRPC on
.IR :9090 .
A central
.B aggregator
subscribes to every collector, merges their snapshot streams, and serves
the same gRPC contract on
.IR :9091 .
The
.B frontend
renders a server-side HTML dashboard (no JavaScript) on
.I :8080
against any
.I LogtailService
endpoint. The CLI,
.BR nginx-logtail ,
offers the same queries as a shell companion.
.PP
Operators typically run the collector on every nginx host as a systemd
unit, the aggregator and frontend on a central host (either as systemd
units or via the shipped
.B docker-compose.yml
), and invoke
.B nginx-logtail
from an operator laptop.
.PP
The Debian package installs three systemd units —
.BR nginx-logtail-collector.service ,
.BR nginx-logtail-aggregator.service ,
.BR nginx-logtail-frontend.service
— under
.IR /lib/systemd/system/ .
None are enabled or started on install: the operator opts into each
service per-host with
.BR "systemctl enable --now" .
Services run as the system user
.B _logtail
(created by
.BR postinst ).
The collector uses
.B _logtail:www-data
so it can read nginx access logs that are group-readable by
.BR www-data ;
the aggregator and frontend use
.BR _logtail:_logtail .
All three units read a single environment file,
.IR /etc/default/nginx-logtail ,
generated by the package's postinst on first install (with the short
hostname substituted for
.B COLLECTOR_SOURCE
and
.BR AGGREGATOR_SOURCE ).
The file is not a dpkg conffile: the template lives at
.IR /usr/share/nginx-logtail/default.template ,
the operator's edits to
.I /etc/default/nginx-logtail
survive upgrades, and
.B dpkg --purge
removes it.
Every flag of every binary has a matching
.BR COLLECTOR_* ,
.BR AGGREGATOR_* ,
or
.BR FRONTEND_ *
env var; set them in the defaults file. For flags without an env-var form,
or temporary overrides, append to
.BR COLLECTOR_ARGS ,
.BR AGGREGATOR_ARGS ,
or
.BR FRONTEND_ARGS .
.SH COMPONENTS
.TP
.B nginx-logtail-collector
Installed in
.IR /usr/sbin .
Tails nginx access logs and/or receives UDP datagrams on
.B --logtail-port
(default disabled). Exposes
.I LogtailService
gRPC on
.B --listen
(default
.IR :9090 )
and Prometheus metrics on
.B --prom-listen
(default
.IR :9100 ).
Pass
.B --version
to print build metadata.
.TP
.B nginx-logtail-aggregator
Installed in
.IR /usr/sbin .
Subscribes to each address in
.B --collectors
and merges their streams. Serves
.I LogtailService
on
.B --listen
(default
.IR :9091 ).
On restart, backfills its ring buffers from every collector via
.IR DumpSnapshots .
.TP
.B nginx-logtail-frontend
Installed in
.IR /usr/sbin .
HTTP dashboard on
.B --listen
(default
.IR :8080 )
against
.B --target
(default
.IR localhost:9091 ,
the aggregator). URL-driven filter state; append
.I &raw=1
to any dashboard URL for JSON output.
.TP
.B nginx-logtail
Installed in
.IR /usr/bin .
CLI for
.BR topn ,
.BR trend ,
.BR stream ,
and
.B targets
queries. Accepts
.BI \-\-target " host:port[,host:port...]"
for concurrent fan-out.
.SH FILES
.TP
.I /usr/sbin/nginx-logtail-collector
Collector daemon binary.
.TP
.I /usr/sbin/nginx-logtail-aggregator
Aggregator daemon binary.
.TP
.I /usr/sbin/nginx-logtail-frontend
Frontend HTTP server binary.
.TP
.I /usr/bin/nginx-logtail
CLI binary.
.TP
.I /usr/share/doc/nginx-logtail/
README, copyright, and pointer to the design and user-guide documents.
.SH EXAMPLES
.PP
Run a collector reading one log file and listening on UDP 9514:
.PP
.RS
.nf
nginx-logtail-collector \\
--logs /var/log/nginx/access.log \\
--logtail-port 9514 \\
--source $(hostname)
.fi
.RE
.PP
Query the top 10 websites over the last 5 minutes:
.PP
.RS
.nf
nginx-logtail topn --target agg:9091 --window 5m --n 10
.fi
.RE
.PP
Show all HTTP 429s by client prefix over the last minute:
.PP
.RS
.nf
nginx-logtail topn --target agg:9091 --window 1m \\
--group-by prefix --status 429
.fi
.RE
.SH ENVIRONMENT
All three daemons read
.IR /etc/default/nginx-logtail .
The file is self-documenting — every env var each binary recognises is
listed with a short description and its default value. Representative
variables:
.IP \fBCOLLECTOR_LOGS\fR
Comma-separated log file paths or globs.
.IP \fBCOLLECTOR_LOGTAIL_PORT\fR
UDP port for
.I ipng_stats_logtail
input; 0 disables the listener.
.IP \fBAGGREGATOR_COLLECTORS\fR
Comma-separated collector addresses. Mandatory.
.IP \fBFRONTEND_TARGET\fR
gRPC endpoint the frontend queries (aggregator or collector).
.IP \fBCOLLECTOR_ARGS\fR, \fBAGGREGATOR_ARGS\fR, \fBFRONTEND_ARGS\fR
Raw argv appended after the env-var-derived flags; use for flags
without an env-var form or for temporary overrides.
.SH SECURITY
gRPC endpoints are cleartext HTTP/2 by default. The UDP listener binds to
.I 127.0.0.1
unless
.B --logtail-bind
is set explicitly. Expose beyond a trusted network only behind a TLS
terminator.
.SH SEE ALSO
.BR nginx (8),
.BR systemd (1).
.PP
Full design and operator guide:
.IR /usr/share/doc/nginx-logtail/README.md .
.SH AUTHORS
Pim van Pelt <pim@ipng.ch>, with Claude Code.
.SH BUGS
Report issues at https://git.ipng.ch/ipng/nginx-logtail.
Reference in New Issue View Git Blame Copy Permalink