PRE-RELEASE 0.9.1: Makefile, Debian packaging, versioned UDP

Build and release tooling:
- Makefile with help as default; targets: build/build-amd64/build-arm64,
  test, lint, proto, pkg-deb, docker, docker-push, clean, plus
  install-deps (+ three sub-targets for apt / Go toolchain / Go tools).
- internal/version package; -ldflags -X injects Version/Commit/Date into
  every binary. -version flag on all four binaries (nginx-logtail version
  for the CLI).
- Dockerfile takes VERSION/COMMIT/DATE build-args and forwards them.
- .deb output lands in build/; .gitignore ignores /build/.

Debian package:
- debian/build-deb.sh packages all four static binaries into a single
  nginx-logtail_<ver>_<arch>.deb using dpkg-deb.
- Binary layout: /usr/sbin/nginx-logtail-{collector,aggregator,frontend}
  and /usr/bin/nginx-logtail.
- nginx-logtail(8) manpage.
- Three systemd units (collector, aggregator, frontend) shipped under
  /lib/systemd/system/. Installed but never enabled or started — the
  operator opts in per host.
- Collector runs as _logtail:www-data (log access); aggregator and
  frontend as _logtail:_logtail. postinst creates the system user/group
  idempotently.
- Single shared env file /etc/default/nginx-logtail rendered from a
  template at first install with %HOSTNAME% substituted. Sensible
  defaults for every COLLECTOR_*, AGGREGATOR_*, FRONTEND_* variable;
  plus COLLECTOR_ARGS / AGGREGATOR_ARGS / FRONTEND_ARGS escape hatches
  appended to ExecStart. Not a dpkg conffile: operator edits survive
  upgrades and dpkg --purge removes it.

Versioned UDP wire format:
- ParseUDPLine dispatches on a leading "v<N>\t" tag; v1 routes to the
  existing 12-field parser. Unknown/missing versions fail closed so
  future v2 parsers can land before emitters are upgraded.
- Tests updated; design.md FR-2.2 rewritten to make the version tag
  normative.

Docs:
- README.md gains a Quick Start (Debian / Docker Compose / from source).
- user-guide.md rewritten around Installation and Configuration: full
  env-var table, UDP-only default explained, precise file/UDP log_format
  layouts, note that operators can emit "0" for unknown \$is_tor / \$asn.
- Drilldown cycle, frontend filter table, and CLI --group-by list all
  include source_tag. UDP counters documented in the Prometheus section.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-04-17 10:35:08 +02:00
parent 577ed3dad5
commit 143aad9063
23 changed files with 1214 additions and 114 deletions

91
debian/build-deb.sh vendored Executable file
View File

@@ -0,0 +1,91 @@
#!/usr/bin/env bash
# Build a minimal .deb for nginx-logtail containing the four static binaries.
# Expects `make build-<arch>` to have already populated build/<arch>/.
#
# Usage: debian/build-deb.sh <arch> <version>
# arch: amd64 | arm64
# version: e.g. 0.9.1
set -euo pipefail
if [ "$#" -ne 2 ]; then
echo "usage: $0 <arch> <version>" >&2
exit 1
fi
ARCH="$1"
VERSION="$2"
PKG="nginx-logtail"
STAGE="$(mktemp -d)"
chmod 0755 "$STAGE"
# Output into build/ alongside the per-arch binary trees so `make clean`
# wipes everything in one rm and .gitignore only needs to ignore build/.
OUT_DIR="build"
mkdir -p "${OUT_DIR}"
OUT="${OUT_DIR}/${PKG}_${VERSION}_${ARCH}.deb"
trap 'rm -rf "$STAGE"' EXIT
BUILD_DIR="build/${ARCH}"
for b in collector aggregator frontend cli; do
if [ ! -x "${BUILD_DIR}/${b}" ]; then
echo "error: ${BUILD_DIR}/${b} not found — run 'make build-${ARCH}' first" >&2
exit 1
fi
done
install -d -m 0755 \
"${STAGE}/DEBIAN" \
"${STAGE}/usr/sbin" \
"${STAGE}/usr/bin" \
"${STAGE}/usr/share/doc/${PKG}" \
"${STAGE}/usr/share/man/man8" \
"${STAGE}/usr/share/${PKG}" \
"${STAGE}/lib/systemd/system"
install -m 0755 "${BUILD_DIR}/collector" "${STAGE}/usr/sbin/nginx-logtail-collector"
install -m 0755 "${BUILD_DIR}/aggregator" "${STAGE}/usr/sbin/nginx-logtail-aggregator"
install -m 0755 "${BUILD_DIR}/frontend" "${STAGE}/usr/sbin/nginx-logtail-frontend"
install -m 0755 "${BUILD_DIR}/cli" "${STAGE}/usr/bin/nginx-logtail"
install -m 0644 LICENSE "${STAGE}/usr/share/doc/${PKG}/copyright"
install -m 0644 README.md "${STAGE}/usr/share/doc/${PKG}/README.md"
# Manpage: gzip per Debian policy (lintian only checks for .gz).
gzip -n -9 -c debian/nginx-logtail.8 > "${STAGE}/usr/share/man/man8/nginx-logtail.8.gz"
chmod 0644 "${STAGE}/usr/share/man/man8/nginx-logtail.8.gz"
# systemd units. Installed, not enabled or started — operator opts in.
install -m 0644 debian/nginx-logtail-collector.service "${STAGE}/lib/systemd/system/"
install -m 0644 debian/nginx-logtail-aggregator.service "${STAGE}/lib/systemd/system/"
install -m 0644 debian/nginx-logtail-frontend.service "${STAGE}/lib/systemd/system/"
# Defaults template. postinst renders this to /etc/default/nginx-logtail on
# first install with %HOSTNAME% substituted. Not a dpkg conffile — operator
# edits survive upgrades because postinst only writes when the file is absent.
install -m 0644 debian/default.template "${STAGE}/usr/share/${PKG}/default.template"
# Maintainer scripts: postinst creates _logtail user and renders defaults;
# prerm stops running services; postrm reloads systemd and removes the
# generated defaults file on purge.
install -m 0755 debian/postinst "${STAGE}/DEBIAN/postinst"
install -m 0755 debian/postrm "${STAGE}/DEBIAN/postrm"
install -m 0755 debian/prerm "${STAGE}/DEBIAN/prerm"
cat > "${STAGE}/DEBIAN/control" <<EOF
Package: ${PKG}
Version: ${VERSION}
Section: net
Priority: optional
Architecture: ${ARCH}
Maintainer: Pim van Pelt <pim@ipng.ch>
Homepage: https://git.ipng.ch/ipng/nginx-logtail
Description: Real-time top-K traffic analysis for nginx clusters
nginx-logtail is a four-binary Go system that ingests nginx access
logs (from files or UDP) and answers ranked top-K queries over
configurable time windows. See /usr/share/doc/nginx-logtail/README.md.
EOF
dpkg-deb --build --root-owner-group "${STAGE}" "${OUT}"
echo "built ${OUT}"

97
debian/default.template vendored Normal file
View File

@@ -0,0 +1,97 @@
# /etc/default/nginx-logtail
#
# Shared configuration for the nginx-logtail collector, aggregator, and
# frontend systemd units. Every flag that every binary accepts has a
# matching environment variable (COLLECTOR_*, AGGREGATOR_*, FRONTEND_*);
# the units start their binary with no explicit arguments beyond the
# optional *_ARGS escape hatch, so everything is driven from here.
#
# This file is generated by nginx-logtail's postinst on first install
# (hostname substituted) and is NOT a dpkg conffile. Operator edits are
# preserved across upgrades. `dpkg --purge nginx-logtail` removes it.
# ==========================================================================
# Collector (nginx-logtail-collector.service)
# ==========================================================================
# gRPC listen address for TopN/Trend queries and the aggregator's
# StreamSnapshots subscription.
COLLECTOR_LISTEN=:9090
# Prometheus /metrics listen address. Set to "" to disable the endpoint.
COLLECTOR_PROM_LISTEN=:9100
# Comma-separated log file paths or glob patterns to tail. At least one of
# COLLECTOR_LOGS, COLLECTOR_LOGS_FILE, or COLLECTOR_LOGTAIL_PORT must be set,
# otherwise the collector refuses to start. Leave empty to run UDP-only (no
# file tailer goroutine is started when no patterns are supplied).
COLLECTOR_LOGS=
# Alternative to COLLECTOR_LOGS: a file listing one path/glob per line.
# Lines starting with # are ignored.
COLLECTOR_LOGS_FILE=
# Name for this collector in query responses, ListTargets, and snapshot
# streams. Defaults to the short hostname at install time.
COLLECTOR_SOURCE=%HOSTNAME%
# IPv4 prefix length for client address bucketing (CIDR). /24 groups a
# class-C worth of clients into one key.
COLLECTOR_V4PREFIX=24
# IPv6 prefix length. /48 matches the typical residential allocation.
COLLECTOR_V6PREFIX=48
# How often to rescan COLLECTOR_LOGS globs for new/removed files.
COLLECTOR_SCAN_INTERVAL=10s
# UDP port that receives ipng_stats_logtail datagrams from the companion
# nginx-ipng-stats-plugin. Set to 0 to disable the UDP listener entirely.
COLLECTOR_LOGTAIL_PORT=9514
# UDP bind address. Keep as 127.0.0.1 unless the plugin emits from a
# different host; the listener has no authentication.
COLLECTOR_LOGTAIL_BIND=127.0.0.1
# Extra arguments appended to the collector argv after the env-var-derived
# flags. Useful for flags without an env-var form, or temporary overrides.
COLLECTOR_ARGS=
# ==========================================================================
# Aggregator (nginx-logtail-aggregator.service)
# ==========================================================================
# gRPC listen address. Frontend and CLI point their --target at this.
AGGREGATOR_LISTEN=:9091
# Comma-separated host:port addresses of every collector this aggregator
# should subscribe to. Mandatory — aggregator refuses to start empty.
AGGREGATOR_COLLECTORS=localhost:9090
# Display name for this aggregator in query responses.
AGGREGATOR_SOURCE=%HOSTNAME%
# Extra arguments appended to the aggregator argv.
AGGREGATOR_ARGS=
# ==========================================================================
# Frontend (nginx-logtail-frontend.service)
# ==========================================================================
# HTTP listen address for the dashboard.
FRONTEND_LISTEN=:8080
# Default gRPC endpoint the dashboard queries. The aggregator by default;
# override with ?target=host:port per request, or change here to point
# directly at a collector.
FRONTEND_TARGET=localhost:9091
# Default number of table rows shown per view. Dashboard users can
# override with ?n=N on individual URLs.
FRONTEND_N=25
# Meta-refresh interval (seconds). Set 0 to disable auto-refresh.
FRONTEND_REFRESH=30
# Extra arguments appended to the frontend argv.
FRONTEND_ARGS=

23
debian/nginx-logtail-aggregator.service vendored Normal file
View File

@@ -0,0 +1,23 @@
[Unit]
Description=nginx-logtail aggregator
Documentation=man:nginx-logtail(8)
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=_logtail
Group=_logtail
EnvironmentFile=-/etc/default/nginx-logtail
ExecStart=/usr/sbin/nginx-logtail-aggregator $AGGREGATOR_ARGS
Restart=on-failure
RestartSec=5
# Aggregator needs no filesystem access beyond its binary.
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
NoNewPrivileges=yes
[Install]
WantedBy=multi-user.target

26
debian/nginx-logtail-collector.service vendored Normal file
View File

@@ -0,0 +1,26 @@
[Unit]
Description=nginx-logtail collector
Documentation=man:nginx-logtail(8)
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
# Group=www-data lets the collector read nginx access logs that are group-readable
# by www-data. Override with a drop-in if your nginx uses a different group.
User=_logtail
Group=www-data
EnvironmentFile=-/etc/default/nginx-logtail
ExecStart=/usr/sbin/nginx-logtail-collector $COLLECTOR_ARGS
Restart=on-failure
RestartSec=5
# Basic hardening — override with a drop-in if your deployment needs more.
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
NoNewPrivileges=yes
ReadOnlyPaths=/var/log
[Install]
WantedBy=multi-user.target

22
debian/nginx-logtail-frontend.service vendored Normal file
View File

@@ -0,0 +1,22 @@
[Unit]
Description=nginx-logtail frontend
Documentation=man:nginx-logtail(8)
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=_logtail
Group=_logtail
EnvironmentFile=-/etc/default/nginx-logtail
ExecStart=/usr/sbin/nginx-logtail-frontend $FRONTEND_ARGS
Restart=on-failure
RestartSec=5
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
NoNewPrivileges=yes
[Install]
WantedBy=multi-user.target

240
debian/nginx-logtail.8 vendored Normal file
View File

@@ -0,0 +1,240 @@
.TH NGINX-LOGTAIL 8 "April 2026" "nginx-logtail 0.9.1" "System Manager's Manual"
.SH NAME
nginx-logtail \- real-time top-K traffic analysis for nginx clusters
.SH SYNOPSIS
.B nginx-logtail-collector
.RI [ options ]
.br
.B nginx-logtail-aggregator
.RI [ options ]
.br
.B nginx-logtail-frontend
.RI [ options ]
.br
.B nginx-logtail
.IR subcommand
.RI [ options ]
.SH DESCRIPTION
.PP
.B nginx-logtail
is a four-binary Go system for real-time analysis of nginx traffic across a
fleet of hosts. Each nginx host runs a
.B collector
that ingests logs from files (via
.BR fsnotify ),
from a UDP socket (fed by the
.B nginx-ipng-stats-plugin
\fBipng_stats_logtail\fR directive), or both. The collector maintains
in-memory ranked top-K counters over 1m/5m/15m/60m/6h/24h windows and
exposes them via gRPC on
.IR :9090 .
A central
.B aggregator
subscribes to every collector, merges their snapshot streams, and serves
the same gRPC contract on
.IR :9091 .
The
.B frontend
renders a server-side HTML dashboard (no JavaScript) on
.I :8080
against any
.I LogtailService
endpoint. The CLI,
.BR nginx-logtail ,
offers the same queries as a shell companion.
.PP
Operators typically run the collector on every nginx host as a systemd
unit, the aggregator and frontend on a central host (either as systemd
units or via the shipped
.B docker-compose.yml
), and invoke
.B nginx-logtail
from an operator laptop.
.PP
The Debian package installs three systemd units —
.BR nginx-logtail-collector.service ,
.BR nginx-logtail-aggregator.service ,
.BR nginx-logtail-frontend.service
— under
.IR /lib/systemd/system/ .
None are enabled or started on install: the operator opts into each
service per-host with
.BR "systemctl enable --now" .
Services run as the system user
.B _logtail
(created by
.BR postinst ).
The collector uses
.B _logtail:www-data
so it can read nginx access logs that are group-readable by
.BR www-data ;
the aggregator and frontend use
.BR _logtail:_logtail .
All three units read a single environment file,
.IR /etc/default/nginx-logtail ,
generated by the package's postinst on first install (with the short
hostname substituted for
.B COLLECTOR_SOURCE
and
.BR AGGREGATOR_SOURCE ).
The file is not a dpkg conffile: the template lives at
.IR /usr/share/nginx-logtail/default.template ,
the operator's edits to
.I /etc/default/nginx-logtail
survive upgrades, and
.B dpkg --purge
removes it.
Every flag of every binary has a matching
.BR COLLECTOR_* ,
.BR AGGREGATOR_* ,
or
.BR FRONTEND_ *
env var; set them in the defaults file. For flags without an env-var form,
or temporary overrides, append to
.BR COLLECTOR_ARGS ,
.BR AGGREGATOR_ARGS ,
or
.BR FRONTEND_ARGS .
.SH COMPONENTS
.TP
.B nginx-logtail-collector
Installed in
.IR /usr/sbin .
Tails nginx access logs and/or receives UDP datagrams on
.B --logtail-port
(default disabled). Exposes
.I LogtailService
gRPC on
.B --listen
(default
.IR :9090 )
and Prometheus metrics on
.B --prom-listen
(default
.IR :9100 ).
Pass
.B --version
to print build metadata.
.TP
.B nginx-logtail-aggregator
Installed in
.IR /usr/sbin .
Subscribes to each address in
.B --collectors
and merges their streams. Serves
.I LogtailService
on
.B --listen
(default
.IR :9091 ).
On restart, backfills its ring buffers from every collector via
.IR DumpSnapshots .
.TP
.B nginx-logtail-frontend
Installed in
.IR /usr/sbin .
HTTP dashboard on
.B --listen
(default
.IR :8080 )
against
.B --target
(default
.IR localhost:9091 ,
the aggregator). URL-driven filter state; append
.I &raw=1
to any dashboard URL for JSON output.
.TP
.B nginx-logtail
Installed in
.IR /usr/bin .
CLI for
.BR topn ,
.BR trend ,
.BR stream ,
and
.B targets
queries. Accepts
.BI \-\-target " host:port[,host:port...]"
for concurrent fan-out.
.SH FILES
.TP
.I /usr/sbin/nginx-logtail-collector
Collector daemon binary.
.TP
.I /usr/sbin/nginx-logtail-aggregator
Aggregator daemon binary.
.TP
.I /usr/sbin/nginx-logtail-frontend
Frontend HTTP server binary.
.TP
.I /usr/bin/nginx-logtail
CLI binary.
.TP
.I /usr/share/doc/nginx-logtail/
README, copyright, and pointer to the design and user-guide documents.
.SH EXAMPLES
.PP
Run a collector reading one log file and listening on UDP 9514:
.PP
.RS
.nf
nginx-logtail-collector \\
--logs /var/log/nginx/access.log \\
--logtail-port 9514 \\
--source $(hostname)
.fi
.RE
.PP
Query the top 10 websites over the last 5 minutes:
.PP
.RS
.nf
nginx-logtail topn --target agg:9091 --window 5m --n 10
.fi
.RE
.PP
Show all HTTP 429s by client prefix over the last minute:
.PP
.RS
.nf
nginx-logtail topn --target agg:9091 --window 1m \\
--group-by prefix --status 429
.fi
.RE
.SH ENVIRONMENT
All three daemons read
.IR /etc/default/nginx-logtail .
The file is self-documenting — every env var each binary recognises is
listed with a short description and its default value. Representative
variables:
.IP \fBCOLLECTOR_LOGS\fR
Comma-separated log file paths or globs.
.IP \fBCOLLECTOR_LOGTAIL_PORT\fR
UDP port for
.I ipng_stats_logtail
input; 0 disables the listener.
.IP \fBAGGREGATOR_COLLECTORS\fR
Comma-separated collector addresses. Mandatory.
.IP \fBFRONTEND_TARGET\fR
gRPC endpoint the frontend queries (aggregator or collector).
.IP \fBCOLLECTOR_ARGS\fR, \fBAGGREGATOR_ARGS\fR, \fBFRONTEND_ARGS\fR
Raw argv appended after the env-var-derived flags; use for flags
without an env-var form or for temporary overrides.
.SH SECURITY
gRPC endpoints are cleartext HTTP/2 by default. The UDP listener binds to
.I 127.0.0.1
unless
.B --logtail-bind
is set explicitly. Expose beyond a trusted network only behind a TLS
terminator.
.SH SEE ALSO
.BR nginx (8),
.BR systemd (1).
.PP
Full design and operator guide:
.IR /usr/share/doc/nginx-logtail/README.md .
.SH AUTHORS
Pim van Pelt <pim@ipng.ch>, with Claude Code.
.SH BUGS
Report issues at https://git.ipng.ch/ipng/nginx-logtail.

50
debian/postinst vendored Executable file
View File

@@ -0,0 +1,50 @@
#!/bin/sh
# Runs after the package is unpacked. We:
# 1. create the system user/group _logtail (idempotent);
# 2. on first install, render /etc/default/nginx-logtail from the template;
# 3. reload systemd.
#
# We deliberately do NOT enable or start the units — some hosts run only the
# collector, some only the aggregator, some run both with the frontend, some
# run neither. The operator is expected to run:
#
# systemctl enable --now nginx-logtail-collector.service
# systemctl enable --now nginx-logtail-aggregator.service
# systemctl enable --now nginx-logtail-frontend.service
#
# on the hosts that should run each service.
set -e
TEMPLATE=/usr/share/nginx-logtail/default.template
TARGET=/etc/default/nginx-logtail
if [ "$1" = configure ]; then
if ! getent group _logtail >/dev/null; then
addgroup --system _logtail
fi
if ! getent passwd _logtail >/dev/null; then
adduser --system --ingroup _logtail \
--no-create-home --home /nonexistent \
--shell /usr/sbin/nologin \
--gecos "nginx-logtail" \
_logtail
fi
# First install: $2 is empty. Render the template with the current
# short hostname, but never clobber an existing file (in case the
# operator dropped one in manually before installing).
if [ -z "$2" ] && [ ! -e "$TARGET" ]; then
HOSTNAME_SHORT="$(hostname -s 2>/dev/null || hostname 2>/dev/null || echo localhost)"
# Use a delimiter unlikely to appear in hostnames.
sed "s|%HOSTNAME%|${HOSTNAME_SHORT}|g" "$TEMPLATE" > "$TARGET"
chmod 0644 "$TARGET"
chown root:root "$TARGET"
fi
if [ -d /run/systemd/system ]; then
systemctl daemon-reload || true
fi
fi
#DEBHELPER#
exit 0

23
debian/postrm vendored Executable file
View File

@@ -0,0 +1,23 @@
#!/bin/sh
# Runs after the package is removed or purged. Drop systemd's view of the
# units so they disappear from `systemctl list-unit-files`. On purge, also
# remove the generated /etc/default/nginx-logtail (we don't ship it as a
# conffile; postinst renders it from a template on first install).
set -e
case "$1" in
purge)
rm -f /etc/default/nginx-logtail
if [ -d /run/systemd/system ]; then
systemctl daemon-reload || true
fi
;;
remove)
if [ -d /run/systemd/system ]; then
systemctl daemon-reload || true
fi
;;
esac
#DEBHELPER#
exit 0

17
debian/prerm vendored Executable file
View File

@@ -0,0 +1,17 @@
#!/bin/sh
# Runs before the package is removed. Stop any running instances cleanly so
# the files we're about to delete aren't held open.
set -e
case "$1" in
remove|upgrade|deconfigure)
if [ -d /run/systemd/system ]; then
for unit in nginx-logtail-collector.service nginx-logtail-aggregator.service nginx-logtail-frontend.service; do
systemctl stop "$unit" 2>/dev/null || true
done
fi
;;
esac
#DEBHELPER#
exit 0