# ctool Design Document ## Metadata | | | | --- | --- | | **Status** | Retrofit — describes shipped behavior as of `v0.1.0` | | **Author** | Pim van Pelt `` | | **Last updated** | 2026-04-21 | | **Audience** | Operators and contributors who will read the source tree next | The key words **MUST**, **MUST NOT**, **SHOULD**, **SHOULD NOT**, and **MAY** are used as described in [RFC 2119](https://datatracker.ietf.org/doc/html/rfc2119), and are reserved in this document for requirements that are actually enforced in code or by an external dependency. Plain-language descriptions of what the system or an operator can do are written in lowercase — "can", "will", "does" — and should not be read as normative. ## Summary `ctool` is a small collection of command-line tools for working with [Static CT API](https://c2sp.org/static-ct-api) logs. It ships as a single Go binary that dispatches to one of several subcommands. Today there are two: `fetch`, which decodes one or more entries from a log tile as structured JSON, and `tail`, which follows a log's checkpoint and prints a one-liner per new certificate. The binary is also exported as two busybox-style symlinks, `ctfetch` and `ctail`, so that scripts can call the subcommand directly without the outer `ctool` prefix. ## Background The Static CT API defines a stateless on-disk layout for Certificate Transparency logs: each leaf lives in a 256-entry gzip-compressed *data tile* fetched by path, and a Merkle tree over those leaves is published as a parallel set of *hash tiles*. Because the layout is static, a log is just a tree of files on a web server — anyone with an HTTP client and a decoder can read one. The CT ecosystem is shifting from the old RFC 6962 "logs are a database" model to this static-tile model; operators running or watching such logs need ad-hoc tools to poke at tiles by hand and to watch a log grow in real time. `ctool` is those tools, gathered under one roof so that packaging, versioning, and future shared utilities live in a single place. The project deliberately stays small: each subcommand is a thin wrapper over [`filippo.io/sunlight`](https://pkg.go.dev/filippo.io/sunlight) and Go's standard `crypto/x509`, with a small shared helper package for the fiddly parts (issuer fetching, SCT decoding, CT log list enrichment). ## Goals and Non-Goals ### Product Goals 1. **Operator utility first.** The tools exist to make Static CT logs inspectable by a human with a shell. Scriptable output (JSON for `fetch`, fixed-column lines for `tail`) is a first-class concern. 2. **No state.** Every invocation is independent; there are no databases, no lock files, no background processes. 3. **Portable packaging.** A single statically-linked binary that works on any Linux amd64/arm64 host without touching system libraries. 4. **Extensible dispatcher.** `ctool` will grow new subcommands over time. Adding one MUST NOT require a new binary, a new Debian package, or any changes outside the `cmd/ctool` tree. ### Non-Goals - `ctool` is not a CT log itself. It does not host, sign, or distribute tiles. - It is not a CT monitor in the Google-Chrome-policy sense. Detecting misissuance is left to whatever the operator feeds `ctool tail`'s output into. - It is not a general x509 pretty-printer. The decoded fields are the ones CT operators typically care about (SCTs, issuer chain, validity range); a general "dump every extension" mode is out of scope. - It does not secure its own outbound traffic. Plain HTTP is accepted when the operator passes an `http://` URL; it is the operator's responsibility to prefer HTTPS for production logs. - It is not a long-running service. `tail` runs in a loop, but it is a foreground process — there is no daemon, no systemd unit, no PID file. ## Requirements Each requirement carries a unique identifier (`FR-X.Y` or `NFR-X.Y`) so that later sections can cite it. ### Functional Requirements **FR-1 Dispatcher (`ctool`)** - **FR-1.1** A single binary MUST dispatch its first positional argument to a named subcommand. Unknown subcommands MUST exit non-zero with a usage banner on stderr. - **FR-1.2** The binary MUST recognize busybox-style invocation: when `os.Args[0]`'s basename is `ctfetch` or `ctail` (with or without a platform-native extension such as `.exe`), the corresponding subcommand is invoked directly and the remaining argv is passed through unchanged. - **FR-1.3** The dispatcher MUST expose a `version` subcommand that prints the build version, git commit hash, and build date on stdout and exits zero. The same information MUST also be reachable as `ctool -version`, `ctool --version`, `ctool fetch version`, and `ctool tail version`, so that "find out what you have" works regardless of which form the operator guesses first. - **FR-1.4** Usage banners emitted by a subcommand MUST refer to the tool by its effective invocation name — `ctfetch ...` when invoked through the symlink, `ctool fetch ...` when invoked through the outer dispatcher — so that copy/paste from an error message yields a command that works. - **FR-1.5** Adding a new subcommand MUST require only: a new `cmd_.go` file in `cmd/ctool/`, a new `case` in the dispatcher's switch, and (optionally) a new busybox symlink in the package layout. No changes to the Makefile's build graph and no new binaries are required. **FR-2 `fetch` subcommand** - **FR-2.1** `ctool fetch` MUST operate in two modes, distinguished by whether the second positional argument parses as a signed decimal integer: - **Leaf-index mode**: ` [modifiers]` fetches the data tile containing `leaf-index` and decodes the single entry at that position. - **Tile-dump mode**: ` [modifiers]` fetches or reads one tile and decodes every entry in it. - **FR-2.2** Tile-dump mode MUST auto-detect hash tiles versus data tiles from the tile contents. For hash tiles, the output is the list of 32-byte SHA-256 node hashes and the `+sct`, `+issuer`, `+ctlog`, and `+all` modifiers MUST be rejected as an error. - **FR-2.3** The output format MUST be pretty-printed JSON on stdout with a trailing newline. - **FR-2.4** The following positional modifiers MUST be accepted, and any other `+`-prefixed token MUST be rejected as an unknown argument: - `+sct` — decode embedded SCTs on final certificates. - `+issuer` — fetch and parse the issuer certificate from the log's `/issuer/` endpoint. - `+ctlog` — enrich each SCT with operator and state data from the CT log list. - `+all` — shorthand for all three of the above. - **FR-2.5** When `+issuer` is used with tile-dump mode on a local file, the operator MUST provide `--monitoring-url`; the subcommand MUST otherwise derive the log root from the tile URL by stripping the `/tile/...` path component. - **FR-2.6** Partial tile suffixes (`.p/N`) MUST be tried first when the path advertises one; on HTTP 404 the full tile MUST be fetched by stripping the suffix. **FR-3 `tail` subcommand** - **FR-3.1** `ctool tail` MUST poll the log's `/checkpoint` endpoint at a configurable interval (default 15s, minimum 1s), and for each completed data tile that appears after the current cursor, MUST decode every entry and print a one-line summary to stdout. - **FR-3.2** By default, the cursor starts at the current tree tip so that only *new* entries are printed. `--from-leaf N` MUST override this so that operators can replay from a known index (including `--from-leaf 0` for a full backfill). - **FR-3.3** The one-liner format MUST be fixed-column and space-separated: leaf index (right-aligned, nine columns), type (`cert` or `pre `), validity range (`YYYY-MM-DD..YYYY-MM-DD`), issuer label (up to 40 chars, truncated with `...`), subject name. - **FR-3.4** The subject name MUST be the first DNS SAN; falling back to the certificate's subject common name; falling back to the literal `(unknown)` if neither is present. - **FR-3.5** The issuer label MUST prepend the issuer organisation to the issuer CN when the CN's first word is not already contained in the organisation name (e.g. `R13` is shown as `Let's Encrypt R13`, but `Let's Encrypt Authority X3` is shown verbatim). - **FR-3.6** Status and error messages MUST go to stderr; one-liners MUST go to stdout, so that the output stream is safe to pipe into `grep`, `awk`, or a log shipper without interleaving diagnostic noise. - **FR-3.7** A tile MUST NOT be fetched until the checkpoint confirms it is complete (256 entries), to avoid unnecessary 404s at the tree tip. - **FR-3.8** A configurable minimum delay between outgoing HTTP requests MUST be enforced (default 2s, minimum 100ms), to stay a polite distance below any per-IP rate limit a log operator might enforce. **FR-4 Packaging and versioning** - **FR-4.1** The build MUST produce statically linked binaries for `linux/amd64` and `linux/arm64`. `CGO_ENABLED=0` MUST be the default so that the resulting binary has no libc dependency and runs on any Linux host of the matching architecture. - **FR-4.2** The version, commit hash, and build date MUST be injected at link time via `-ldflags -X`, so that `ctool version` on a shipped binary identifies the exact build without requiring access to the source tree. The defaults baked into the source MUST keep `go run` / `go build` without ldflags usable. - **FR-4.3** The Debian package MUST install `ctool` under `/usr/bin`, MUST provide `/usr/bin/ctfetch` and `/usr/bin/ctail` as symlinks to it, and MUST install the three section-1 manpages (`ctool.1`, `ctfetch.1`, `ctail.1`) gzipped under `/usr/share/man/man1`. ### Non-Functional Requirements **NFR-1 Availability and reliability** - **NFR-1.1** A failing HTTP request MUST NOT crash `tail`. The error MUST be logged to stderr and the next poll MUST happen on schedule. - **NFR-1.2** A malformed tile MUST NOT be silently skipped. If a tile cannot be read, the subcommand MUST return an error that names the offending leaf index. **NFR-2 Determinism and correctness** - **NFR-2.1** Given the same tile input, `fetch` MUST produce byte-identical JSON output. No timestamps, random IDs, or map iteration order MAY leak into the output. - **NFR-2.2** `tail`'s polling interval timer MUST start when the checkpoint is fetched, not when the previous loop finished, so that time spent fetching data tiles counts against the interval and the next poll stays on schedule. **NFR-3 Performance and scalability** - **NFR-3.1** `fetch` MUST fetch at most one tile per invocation. Leaf-index mode derives the one tile that contains the requested index; tile-dump mode operates on exactly the tile the operator named. - **NFR-3.2** Caches (CT log list, issuer certificates) MUST be scoped to a single invocation. No on-disk cache, no cross-invocation state. **NFR-4 Security** - **NFR-4.1** Tiles MUST be decompressed with a bounded expansion ratio (currently 100×) to prevent a malicious log from driving the client out of memory via a zip bomb. - **NFR-4.2** The tools MUST NOT require any Linux capabilities or root privileges. They are plain HTTP clients and run unprivileged. - **NFR-4.3** `ctool` MUST NOT write anywhere on the filesystem except when the operator explicitly asks it to (e.g. by redirecting stdout). There is no cache directory, no log file, no state directory. **NFR-5 Operability** - **NFR-5.1** Every CLI flag SHOULD have a sensible default so that the most common invocation is a single URL plus maybe a leaf index. - **NFR-5.2** The JSON output of `fetch` SHOULD be stable across patch releases within a minor version. Field additions are allowed; renames and removals are not. - **NFR-5.3** Subcommand usage banners SHOULD include at least one concrete example against `mon.ct.ipng.ch`, which is the reference log we test against. ## Architecture Overview ### Process Model `ctool` is a short-lived foreground process. There are no daemons, no sockets, no persistent state. Each invocation: 1. Parses argv and dispatches to a subcommand. 2. Makes some number of HTTP requests (or reads a local file). 3. Writes JSON or one-liners to stdout. 4. Exits. `tail` is the one exception to step 3 being a single write: it loops, making one checkpoint request per interval and zero or more tile requests per loop iteration. It still exits on Ctrl-C or on an unrecoverable error; there is no restart policy because there is no supervisor. ### Data Flow Configuration flows **in** as command-line flags and positional arguments (there is no config file, no environment variables, no `/etc/` anything). Data flows **in** from the Static CT log's HTTP surface: the `/checkpoint` endpoint, the `/tile/...` tree, and the `/issuer/` endpoint. The CT log list JSON (default: `gstatic.com/ct/log_list/v3/all_logs_list.json`) is consumed optionally when `+ctlog` is requested. Data flows **out** as JSON or fixed-column lines on stdout, and as status/error messages on stderr. ## Components ### `ctool` (dispatcher) `ctool` is the outer binary: a Go `main` package under `cmd/ctool/` whose job is to decide which subcommand to run and hand argv off to it. It is deliberately tiny — today about forty lines of code — because every new subcommand adds a `case` to its `switch` and nothing else. #### Responsibilities - Examine `os.Args[0]` and, if it matches the basename of a known busybox symlink (`ctfetch`, `ctail`), route directly to that subcommand (FR-1.2). - Otherwise, examine `os.Args[1]` and route to the named subcommand, print usage and exit non-zero on unknown names (FR-1.1). - Handle the `version` / `-version` / `--version` top-level alias by printing the build version and exiting zero (FR-1.3). - Provide a single `cmdName()` helper that subcommands use when building usage banners, so that every banner refers to the effective invocation name (FR-1.4). #### Extension Model Adding a new subcommand `foo` is a three-line change: 1. Add `cmd/ctool/cmd_foo.go` with an exported `runFoo(args []string)` function. 2. Add `case "foo": runFoo(os.Args[2:])` to `main()`'s switch and a matching line to `usage()`. 3. (Optional, if busybox-style invocation is wanted) Add `case "ctfoo": runFoo(os.Args[1:])` to the symlink switch, add the symlink to `debian/build-deb.sh`, and add a `ctfoo(1)` manpage. No Makefile change is required (FR-1.5) because the build target globs over `./cmd/ctool/`. No new Debian package is required; the existing `ctool` package gets the new subcommand for free, and optionally a new symlink. `version` is specifically *not* a subcommand file — it lives in the dispatcher because every subcommand reuses the same `printVersion()` helper for its own `version` alias (`ctool fetch version`, `ctool tail version`). #### Interfaces **Presents.** - **A command-line interface** driven by `os.Args`. The exit-code contract is 0 on success, 1 on unknown subcommand or subcommand-internal fatal error, and whatever the subcommand chooses otherwise. - **`stdout` and `stderr` streams**. Structured output (JSON, one-liners) goes to stdout; banners, progress messages, and errors go to stderr (FR-3.6, NFR-5.3). **Consumes.** - **`os.Args`** — the sole input to the dispatcher. ### `fetch` `fetch` (reachable as `ctool fetch` or `ctfetch`) decodes one or more entries from a Static CT log tile. #### Responsibilities - Distinguish leaf-index mode from tile-dump mode by whether `args[1]` parses as a decimal integer (FR-2.1). - Fetch the relevant tile (or read it from disk), decompress it with a bounded ratio (NFR-4.1), and decode each leaf. - Optionally enrich each entry with decoded SCTs, the issuer certificate, and CT-log-list metadata per the `+sct`, `+issuer`, `+ctlog`, and `+all` modifiers (FR-2.4). - In tile-dump mode, auto-detect hash tiles versus data tiles and refuse the cert-oriented modifiers on hash tiles (FR-2.2). - Emit pretty-printed JSON on stdout (FR-2.3). #### Tile Fetching The tile URL is derived by [`filippo.io/sunlight`]' `TilePath` function from the requested leaf index; `fetch` tries the partial tile suffix first (`.p/N`) and falls back to the full tile on a 404 (FR-2.6). Both paths return gzipped bytes, which are then decompressed with `io.LimitReader` capping expansion at 100× the input size (NFR-4.1). Tile-dump mode on a local file skips the URL derivation and simply reads the file. #### Enrichment The optional modifiers trigger network calls outside the main tile fetch: - `+sct` parses the SCT list extension from the DER cert in-memory; no extra network calls. - `+issuer` fetches `/issuer/` from the log root for each referenced issuer fingerprint. Results are cached for the lifetime of the invocation (NFR-3.2). - `+ctlog` fetches the CT log list JSON once per invocation and looks up each SCT's log ID against it. `+all` is syntactic sugar for all three. #### Interfaces **Presents.** - **Pretty-printed JSON** on stdout, with a trailing newline. The top-level object is either a single `Entry` (leaf-index mode), a `DumpResult` with an `entries` array (data tile), or a `DumpResult` with a `hash_tile` field (hash tile). **Consumes.** - **Positional arguments** (log URL, leaf index, modifiers) and two flags (`--logs-list-url`, `--monitoring-url`) from argv. - **The Static CT log's HTTP surface** — checkpoint is not read by `fetch`, but the tile tree and the issuer endpoint are. - **The outbound network** directly. No proxy handling, no capability requirements (NFR-4.2). ### `tail` `tail` (reachable as `ctool tail` or `ctail`) follows a Static CT log's checkpoint and prints a one-line summary per new entry. #### Responsibilities - Poll the log's `/checkpoint` on a configurable interval and track the current tree size (FR-3.1). - For each completed data tile whose entries have not yet been printed, fetch it, decompress it, decode each leaf, and print a fixed-column one-liner to stdout (FR-3.1, FR-3.3). - Throttle outbound HTTP requests with a configurable minimum inter-request delay (FR-3.8). - Respect a configurable starting leaf index so that operators can replay from any point in the log (FR-3.2). #### Poll Loop Each iteration fetches the checkpoint first and records the timestamp; then walks forward from the cursor tile by tile, stopping at the first tile that the checkpoint does not yet confirm complete (FR-3.7). The next poll sleeps until `checkpoint_time + interval`, so that the time spent fetching data tiles counts against the interval and consecutive polls stay on a stable cadence (NFR-2.2). A tile fetch failure is logged to stderr and the iteration is retried on the next poll (NFR-1.1); the cursor does not advance past a leaf that was never printed, so no entries are lost even across transient log outages. #### One-Liner Format Columns are separated by a single space, each of fixed width. The format string is documented in `docs/ctail.md`, reproduced here: ``` %9d %-4s %-21s %-40s %s ``` Fields: leaf index (decimal), type (`cert` or `pre `), validity range (`YYYY-MM-DD..YYYY-MM-DD`), issuer label (truncated to 40 with `...`), subject name. The truncation rule and the issuer-with-org prepending rule are both tested (FR-3.3, FR-3.4, FR-3.5). #### Interfaces **Presents.** - **Fixed-column one-liners** on stdout, one per log entry. - **Status and error messages** on stderr (FR-3.6). **Consumes.** - **Positional ``** plus flags `--interval`, `--from-leaf`, `--rate-limit`, `--user-agent`. - **The log's HTTP surface** (`/checkpoint` and the tile tree). - **The outbound network** directly. ### Shared Helpers (`internal/utils`) `internal/utils` holds the code that both `fetch` and `tail` call into but neither one should own: tile fetching with partial fallback, bounded gzip decompression, SCT decoding, issuer fetching with per-invocation caching, and CT-log-list enrichment. The package is deliberately import-only from `cmd/ctool/`; nothing outside the binary imports it. The helpers are not tested in isolation today beyond what the binary-level tests exercise; if the package grows substantially, per-package unit tests SHOULD be added before it becomes difficult to reason about. ## Operational Concerns ### Packaging The repository ships a single Debian package, `ctool`, built for `amd64` and `arm64`. The package contains `/usr/bin/ctool` plus two symlinks (`ctfetch`, `ctail`) and three manpages under `/usr/share/man/man1`. There are no conffiles, no maintainer scripts, and no systemd units; the package is a pure binary-plus-docs bundle (FR-4.3). Binaries are produced with `CGO_ENABLED=0` (FR-4.1). The effect is that the binary has no libc dependency and runs on any Linux amd64/arm64 host regardless of glibc version, at the cost of Go's `net` package using the pure-Go resolver instead of `getaddrinfo` (i.e. `/etc/nsswitch.conf` and NSS modules are ignored). `ctool` only does DNS-over-UDP lookups for log hostnames, so this is fine. ### Versioning Versioning follows [Semantic Versioning](https://semver.org/). The authoritative version string lives in `Makefile`'s `VERSION` variable; on each build, the Makefile stamps `main.version`, `main.commit`, and `main.date` with `-ldflags -X` (FR-4.2). The commit and date are derived from `git rev-parse --short HEAD` and an ISO-8601 UTC timestamp at build time. `ctool version` prints all three; the Debian package's `Version:` field carries only the release version. ### Failure Modes - **Log returns 404 on a tile we expected.** Treat as a transient error; `tail` logs to stderr and retries on the next poll (NFR-1.1). `fetch` exits non-zero with the HTTP status in the error message. - **Log returns gzip bomb.** The decompress helper's `LimitReader` caps expansion at 100× the input size (NFR-4.1); an attempt to exceed it surfaces as a decode error, not an out-of-memory crash. - **Checkpoint is malformed.** `tail` logs a "checkpoint error" on stderr and sleeps for one interval before retrying. - **CT log list is unreachable.** `+ctlog` degrades: a warning is printed to stderr and entries go out without the `ctlog` enrichment rather than failing the whole invocation. - **Operator passes a hash tile URL with `+sct` / `+issuer` / `+ctlog`.** The subcommand detects the tile type and exits with an error explaining that cert-oriented modifiers do not apply to hash tiles (FR-2.2). ### Observability `ctool` has no metrics and no structured logging surface of its own. Observability in practice is whatever the operator builds around the output streams: - Pipe `ctool tail` into a log shipper or `awk` to watch for specific issuers or subjects. - Pipe `ctool fetch ... +all` into `jq` for ad-hoc inspection. - Exit codes (0 / non-zero) are the only machine-readable signal of success or failure. If a future operational need calls for structured logs or metrics (e.g. a `ctool watch` subcommand that is meant to be run under systemd), it can be added without disturbing `fetch` or `tail`. ### Security The tools run unprivileged (NFR-4.2) and write nothing to disk outside stdout (NFR-4.3). They do not handle credentials, do not parse user-supplied certificates for signature verification, and do not act as a TLS server. TLS verification on outbound requests uses Go's default roots; there is no flag to disable it. Operators probing a log over plain HTTP accept the risk of in-flight modification. ## Alternatives Considered - **Separate `ctfetch` and `ctail` binaries, no dispatcher.** Rejected in favor of one binary with symlinks. A single binary is simpler to package, ship, and version, and scales better as new subcommands land (FR-1.5). Operators who prefer the separate-binary UX still get it through the `ctfetch` / `ctail` symlinks. - **A full CLI framework (cobra, urfave/cli).** Rejected because the dispatcher is under fifty lines and the subcommand parsers use the standard `flag` package. A framework would add a dependency and a build surface without improving the operator experience at this scale. If the subcommand count grows past ~five, this decision should be revisited. - **Embedding a local cache for issuer certificates across invocations.** Rejected for now (NFR-3.2). Caching introduces a state directory, a cache-invalidation policy, and a new failure mode (stale issuer); since `fetch` is typically run ad-hoc rather than in a tight loop, the per-invocation cache is sufficient. - **Making `tail` a daemon with a systemd unit.** Rejected in favor of a plain foreground loop. Operators who want a daemon wrap it in `systemd-run`, `tmux`, or their orchestration tool of choice; the tool itself stays stateless and exit-code honest. ## Open Questions - **Output schema stability.** `fetch`'s JSON output is currently whatever `internal/utils` emits. NFR-5.2 commits to additive compatibility across patch releases, but a minor-version bump may still reshape fields. A more explicit schema document (or a `--schema-version` flag) would help downstream consumers. - **More subcommands.** The dispatcher is built for growth — candidate ideas include `ctool verify` (checkpoint signature + consistency proof) and `ctool stats` (issuer/validity distribution over a time window). These are out of scope for v0.1 but the FR-1.x requirements exist precisely so the additions stay cheap. - **Proxy and retry policy.** There is no HTTP retry today. A log that 502s intermittently will surface those as visible errors on stderr (NFR-1.1) but the cursor still stalls until the log recovers. A configurable retry/backoff policy may be worth adding once we see a deployment hit this in practice.