# Collector v0 — Implementation Plan ✓ COMPLETE

Module path: `git.ipng.ch/ipng/nginx-logtail`

**Scope:** A working collector that tails files, aggregates into memory, and serves `TopN`,
`Trend`, and `StreamSnapshots` over gRPC. Full vertical slice, no optimisation passes yet.

---

## Step 1 — Repo scaffolding
- `go mod init git.ipng.ch/ipng/nginx-logtail`
- `.gitignore`
- Install deps: `google.golang.org/grpc`, `google.golang.org/protobuf`, `github.com/fsnotify/fsnotify`

## Step 2 — Proto (`proto/logtail.proto`)
Write the full proto file as specified in README.md DESIGN § Protobuf API. Generate Go stubs with
`protoc`. Commit generated files. This defines the contract everything else builds on.

## Step 3 — Parser (`cmd/collector/parser.go`)
- `LogRecord` struct: `Website`, `ClientPrefix`, `URI`, `Status string`
- `ParseLine(line string) (LogRecord, bool)` — `SplitN` on tab, discard query string at `?`,
  return `false` for lines with fewer than 8 fields
- `TruncateIP(addr string, v4bits, v6bits int) string` — handle IPv4 and IPv6
- Unit-tested with table-driven tests: normal line, short line, IPv6, query string stripping,
  /24 and /48 truncation

## Step 4 — Store (`cmd/collector/store.go`)
Implement in order, each piece testable independently:

1. **`Tuple4` and live map** — `map[Tuple4]int64`, cap enforcement at 100K, `Ingest(r LogRecord)`
2. **Fine ring buffer** — `[60]Snapshot` circular array, `rotate()` heap-selects top-50K from
   live map, appends to ring, resets live map
3. **Coarse ring buffer** — `[288]Snapshot`, populated every 5 fine rotations by merging
   the last 5 fine snapshots into a top-5K snapshot
4. **`QueryTopN(filter, groupBy, n, window)`** — RLock, sum bucket range, group by dimension,
   apply filter, heap-select top N
5. **`QueryTrend(filter, window)`** — per-bucket count sum, returns one point per bucket
6. **`Store.Run(ch <-chan LogRecord)`** — single goroutine: read channel → `Ingest`, minute
   ticker → `rotate()`
7. **Snapshot broadcast** — per-subscriber buffered channel fan-out;
   `Subscribe() <-chan Snapshot` / `Unsubscribe(ch)`

## Step 5 — Tailer (`cmd/collector/tailer.go`)
- `Tailer` struct: path, fsnotify watcher, output channel
- On start: open file, seek to EOF, register fsnotify watch
- On `fsnotify.Write`: `bufio.Scanner` reads all new lines, sends `LogRecord` to channel
- On `fsnotify.Rename` / `Remove`: drain to EOF, close fd, retry open with 100 ms backoff
  (up to 5 s), resume from position 0 — no lines lost between drain and reopen
- `Tailer.Run(ctx context.Context)` — blocks until context cancelled

## Step 6 — gRPC server (`cmd/collector/server.go`)
- `Server` wraps `*Store`, implements `LogtailServiceServer`
- `TopN`: `store.QueryTopN` → marshal to proto response
- `Trend`: `store.QueryTrend` → marshal to proto response
- `StreamSnapshots`: `store.Subscribe()`, loop sending snapshots until client disconnects
  or context done, then `store.Unsubscribe(ch)`

## Step 7 — Main (`cmd/collector/main.go`)
Flags:
- `--listen`    default `:9090`
- `--logs`      comma-separated log file paths
- `--source`    name for this collector instance (default: hostname)
- `--v4prefix`  default `24`
- `--v6prefix`  default `48`

Wire-up: create channel → start `store.Run` goroutine → start one `Tailer` goroutine per log
path → start gRPC server → `signal.NotifyContext` for clean shutdown on SIGINT/SIGTERM.

## Step 8 — Smoke test
- Generate fake log lines at 10K/s (small Go script or shell one-liner)
- Run collector against them
- Use `grpcurl` to call `TopN` and verify results
- Check `runtime.MemStats` to confirm memory stays well under 1 GB

---

## Deferred (not in v0)
- `cmd/cli`, `cmd/aggregator`, `cmd/frontend`
- ClickHouse export
- TLS / auth
- Prometheus metrics endpoint

---

## Implementation notes

### Deviation from plan: MultiTailer

Step 5 planned one `Tailer` struct per file. During implementation this was changed to a single
`MultiTailer` with one shared `fsnotify.Watcher`. Reason: one watcher per file creates one inotify
instance per file; the kernel default limit is 128 instances per user, which would be hit with
100s of log files. The `MultiTailer` uses a single instance and routes events by path via a
`map[string]*fileState`.

### Deviation from plan: IPv6 /48 semantics

The design doc said "truncate to /48". `/48` keeps the first three full 16-bit groups intact
(e.g. `2001:db8:cafe::1` → `2001:db8:cafe::/48`). An early test expected `2001:db8:ca00::/48`
(truncating mid-group), which was wrong. The code is correct; the test was fixed.

---

## Test results

Run with: `go test ./cmd/collector/ -v -count=1 -timeout 120s`

| Test                        | What it covers                                     |
|-----------------------------|----------------------------------------------------|
| `TestParseLine` (7 cases)   | Tab parsing, query string stripping, bad lines     |
| `TestTruncateIP`            | IPv4 /24 and IPv6 /48 masking                      |
| `TestIngestAndRotate`       | Live map → fine ring rotation                      |
| `TestLiveMapCap`            | Hard cap at 100 K entries, no panic beyond cap     |
| `TestQueryTopN`             | Ranked results from ring buffer                    |
| `TestQueryTopNWithFilter`   | Filter by HTTP status code                         |
| `TestQueryTrend`            | Per-bucket counts, oldest-first ordering           |
| `TestCoarseRingPopulated`   | 5 fine ticks → 1 coarse bucket, count aggregation  |
| `TestSubscribeBroadcast`    | Fan-out channel delivery after rotation            |
| `TestTopKOrdering`          | Heap select returns correct top-K descending       |
| `TestMultiTailerReadsLines` | Live file write → LogRecord received on channel    |
| `TestMultiTailerMultipleFiles` | 5 files, one watcher, all lines received        |
| `TestMultiTailerLogRotation`| RENAME → drain → retry → new file tailed correctly |
| `TestExpandGlobs`           | Glob pattern expands to matching files only        |
| `TestExpandGlobsDeduplication` | Same file via path + glob deduplicated to one   |
| `TestMemoryBudget`          | Full ring fill stays within 1 GB heap              |
| `TestGRPCEndToEnd`          | Real gRPC server: TopN, filtered TopN, Trend, StreamSnapshots |

**Total: 17 tests, all passing.**

---

## Benchmark results

Run with: `go test ./cmd/collector/ -bench=. -benchtime=3s`

Hardware: 12th Gen Intel Core i7-12700T

| Benchmark          | ns/op | throughput     | headroom vs 10K/s |
|--------------------|-------|----------------|-------------------|
| `BenchmarkParseLine` | 418 | ~2.4M lines/s  | 240×              |
| `BenchmarkIngest`    | 152 | ~6.5M records/s| 650×              |

Both the parser and the store ingestion goroutine have several hundred times more capacity than
the 10 000 lines/second peak requirement. The bottleneck at scale will be fsnotify event delivery
and kernel I/O, not the Go code.