Pim van Pelt
143aad9063
Build and release tooling:
- Makefile with help as default; targets: build/build-amd64/build-arm64,
test, lint, proto, pkg-deb, docker, docker-push, clean, plus
install-deps (+ three sub-targets for apt / Go toolchain / Go tools).
- internal/version package; -ldflags -X injects Version/Commit/Date into
every binary. -version flag on all four binaries (nginx-logtail version
for the CLI).
- Dockerfile takes VERSION/COMMIT/DATE build-args and forwards them.
- .deb output lands in build/; .gitignore ignores /build/.
Debian package:
- debian/build-deb.sh packages all four static binaries into a single
nginx-logtail_<ver>_<arch>.deb using dpkg-deb.
- Binary layout: /usr/sbin/nginx-logtail-{collector,aggregator,frontend}
and /usr/bin/nginx-logtail.
- nginx-logtail(8) manpage.
- Three systemd units (collector, aggregator, frontend) shipped under
/lib/systemd/system/. Installed but never enabled or started — the
operator opts in per host.
- Collector runs as _logtail:www-data (log access); aggregator and
frontend as _logtail:_logtail. postinst creates the system user/group
idempotently.
- Single shared env file /etc/default/nginx-logtail rendered from a
template at first install with %HOSTNAME% substituted. Sensible
defaults for every COLLECTOR_*, AGGREGATOR_*, FRONTEND_* variable;
plus COLLECTOR_ARGS / AGGREGATOR_ARGS / FRONTEND_ARGS escape hatches
appended to ExecStart. Not a dpkg conffile: operator edits survive
upgrades and dpkg --purge removes it.
Versioned UDP wire format:
- ParseUDPLine dispatches on a leading "v<N>\t" tag; v1 routes to the
existing 12-field parser. Unknown/missing versions fail closed so
future v2 parsers can land before emitters are upgraded.
- Tests updated; design.md FR-2.2 rewritten to make the version tag
normative.
Docs:
- README.md gains a Quick Start (Debian / Docker Compose / from source).
- user-guide.md rewritten around Installation and Configuration: full
env-var table, UDP-only default explained, precise file/UDP log_format
layouts, note that operators can emit "0" for unknown \$is_tor / \$asn.
- Drilldown cycle, frontend filter table, and CLI --group-by list all
include source_tag. UDP counters documented in the Prometheus section.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3.0 KiB
PREAMBLE
Although this computer program has a permissive license (AP2.0), if you came here looking to ask questions, you're better off just moving on :) This program is shared AS-IS and really without any intent for anybody but IPng Networks to use it. Also, in case the structure of the repo and the style of this README wasn't already clear, this program is 100% written and maintained by Claude Code.
You have been warned :)
What is this?
This project consists of four components:
- A log collector that tails NGINX (or Apache) logs and/or receives logs over UDP from
nginx-ipng-stats-plugin, aggregating counts per website, client address, URI, status, ASN, and source tag. It buckets these into windows of 1m, 5m, 15m, 60m, 6h, and 24h and exposes them over gRPC. - An aggregator that subscribes to any number of collectors and serves a merged view on the same gRPC surface.
- An HTTP frontend that renders a drilldown dashboard (zero JavaScript, server-side SVG sparklines) against any collector or the aggregator.
- A CLI for shell queries, returning tables or JSON.
Written in Go, released under [APACHE]. Runs as systemd units, in Docker, or any
combination.
Quick Start
Three deployment flavors. Pick whichever suits the host.
Debian package. Build once, install the .deb on every nginx host (for the collector) and
on one central host (for the aggregator + frontend):
make install-deps # one-time: apt deps, Go toolchain, go tools
make pkg-deb # produces nginx-logtail_<ver>_{amd64,arm64}.deb
# on each nginx host:
sudo dpkg -i nginx-logtail_*_amd64.deb
sudo $EDITOR /etc/default/nginx-logtail # defaults to UDP-only on :9514; set COLLECTOR_LOGS=... to also tail files
sudo systemctl enable --now nginx-logtail-collector.service
# on the central host:
sudo dpkg -i nginx-logtail_*_amd64.deb
sudo systemctl enable --now nginx-logtail-aggregator.service nginx-logtail-frontend.service
# dashboard now at http://<central>:8080
Binaries land at /usr/sbin/nginx-logtail-{collector,aggregator,frontend} and the CLI at
/usr/bin/nginx-logtail. All three services run as the _logtail system user (collector uses
Group=www-data for log access). None are auto-enabled, so installing the package is safe on
any host.
Docker Compose. Runs the aggregator and frontend in one stack; point collectors (on each nginx host) at the aggregator:
AGGREGATOR_COLLECTORS=nginx1:9090,nginx2:9090 docker compose up -d
# frontend on :8080, aggregator gRPC on :9091
From source (make).
make build # build/<arch>/{collector,aggregator,frontend,cli}
make test
./build/*/nginx-logtail -version
make help lists every target.
See [User Guide] for operator-facing documentation, or [Design] for the normative requirements and architectural rationale.