pim
6647f95be4
Wire-format and metric overhaul. Both file and UDP ingest now share one
versioned ParseLine that dispatches on the v<N>\t prefix; v1 stays
unchanged, v2 adds $bytes_sent (replacing $body_bytes_sent),
$request_length, $upstream_response_time, and $upstream_status. File
ingest gains the same versioning, and the legacy positional file format
is removed (no live deployments).
Prometheus exposition is rewritten:
- nginx_http_bytes_sent and nginx_http_request_duration_seconds gain
a source_tag label.
- nginx_http_requests_by_source_total gains status_class.
- New v2-only metrics: nginx_http_request_bytes,
nginx_http_upstream_duration_seconds,
nginx_http_upstream_requests_total{status_class}.
- Dropped nginx_http_response_body_bytes_by_source (subsumed by the
dual-labeled bytes_sent metric).
Adds 'make fixstyle' (gofmt -w) and clears all golangci-lint findings
across the repo (errcheck, S1001, ST1005, unused).
Docs in design.md FR-2/FR-8 and user-guide.md are rewritten to present
v2 as the recommended log format.
PREAMBLE
Although this computer program has a permissive license (AP2.0), if you came here looking to ask questions, you're better off just moving on :) This program is shared AS-IS and really without any intent for anybody but IPng Networks to use it. Also, in case the structure of the repo and the style of this README wasn't already clear, this program is 100% written and maintained by Claude Code.
You have been warned :)
What is this?
This project consists of four components:
- A log collector that tails NGINX (or Apache) logs and/or receives logs over UDP from
nginx-ipng-stats-plugin, aggregating counts per website, client address, URI, status, ASN, and source tag. It buckets these into windows of 1m, 5m, 15m, 60m, 6h, and 24h and exposes them over gRPC. - An aggregator that subscribes to any number of collectors and serves a merged view on the same gRPC surface.
- An HTTP frontend that renders a drilldown dashboard (zero JavaScript, server-side SVG sparklines) against any collector or the aggregator.
- A CLI for shell queries, returning tables or JSON.
Written in Go, released under [APACHE]. Runs as systemd units, in Docker, or any
combination.
Quick Start
Three deployment flavors. Pick whichever suits the host.
Debian package. Build once, install the .deb on every nginx host (for the collector) and
on one central host (for the aggregator + frontend):
make install-deps # one-time: apt deps, Go toolchain, go tools
make pkg-deb # produces nginx-logtail_<ver>_{amd64,arm64}.deb
# on each nginx host:
sudo dpkg -i nginx-logtail_*_amd64.deb
sudo $EDITOR /etc/default/nginx-logtail # defaults to UDP-only on :9514; set COLLECTOR_LOGS=... to also tail files
sudo systemctl enable --now nginx-logtail-collector.service
# on the central host:
sudo dpkg -i nginx-logtail_*_amd64.deb
sudo systemctl enable --now nginx-logtail-aggregator.service nginx-logtail-frontend.service
# dashboard now at http://<central>:8080
Binaries land at /usr/sbin/nginx-logtail-{collector,aggregator,frontend} and the CLI at
/usr/bin/nginx-logtail. All three services run as the _logtail system user (collector uses
Group=www-data for log access). None are auto-enabled, so installing the package is safe on
any host.
Docker Compose. Runs the aggregator and frontend in one stack; point collectors (on each nginx host) at the aggregator:
AGGREGATOR_COLLECTORS=nginx1:9090,nginx2:9090 docker compose up -d
# frontend on :8080, aggregator gRPC on :9091
From source (make).
make build # build/<arch>/{collector,aggregator,frontend,cli}
make test
./build/*/nginx-logtail -version
make help lists every target.
See [User Guide] for operator-facing documentation, or [Design] for the normative requirements and architectural rationale.