diff --git a/content/articles/2025-08-24-ctlog-3.md b/content/articles/2025-08-24-ctlog-3.md new file mode 100644 index 0000000..7495a2a --- /dev/null +++ b/content/articles/2025-08-24-ctlog-3.md @@ -0,0 +1,508 @@ +--- +date: "2025-08-24T12:07:23Z" +title: 'Certificate Transparency - Part 3 - Operations' +--- + +{{< image width="10em" float="right" src="/assets/ctlog/ctlog-logo-ipng.png" alt="ctlog logo" >}} + +# Introduction + +There once was a Dutch company called [[DigiNotar](https://en.wikipedia.org/wiki/DigiNotar)], as the +name suggests it was a form of _digital notary_, and they were in the business of issuing security +certificates. Unfortunately, in June of 2011, their IT infrastructure was compromised and +subsequently it issued hundreds of fraudulent SSL certificates, some of which were used for +man-in-the-middle attacks on Iranian Gmail users. Not cool. + +Google launched a project called **Certificate Transparency**, because it was becoming more common +that the root of trust given to _Certification Authorities_ could no longer be unilateraly trusted. +These attacks showed that the lack of transparency in the way CAs operated was a significant risk to +the Web Public Key Infrastructure. It led to the creation of this ambitious +[[project](https://certificate.transparency.dev/)] to improve security online by bringing +accountability to the system that protects our online services with _SSL_ (Secure Socket Layer) +and _TLS_ (Transport Layer Security). + +In 2013, [[RFC 6962](https://datatracker.ietf.org/doc/html/rfc6962)] was published by the IETF. It +describes an experimental protocol for publicly logging the existence of Transport Layer Security +(TLS) certificates as they are issued or observed, in a manner that allows anyone to audit +certificate authority (CA) activity and notice the issuance of suspect certificates as well as to +audit the certificate logs themselves. The intent is that eventually clients would refuse to honor +certificates that do not appear in a log, effectively forcing CAs to add all issued certificates to +the logs. + +In the first two articles of this series, I explored [[Sunlight]({{< ref 2025-07-26-ctlog-1 >}})] +and [[TesseraCT]({{< ref 2025-08-10-ctlog-2 >}})], two open source implementations of the Static CT +protocol. In this final article, I'll share the details on how I created the environment and +production instances for four logs that IPng will be providing: Rennet and Lipase are two +ingredients to make cheese and will serve as our staging/testing logs. Gouda and Halloumi are two +delicious cheeses that pay hommage to our heritage, Jeroen and I being Dutch and Antonis being +Greek. + +## Hardware + +At IPng Networks, all hypervisors are from the same brand: Dell's Poweredge line. In this project, +Jeroen is also contributing a server, and it so happens that he also has a Dell Poweredge. We're +both running Debian on our hypervisor, so we install a fresh VM with Debian 13.0, codenamed +_Trixie_, and give the machine 16GB of memory, 8 vCPU and a 16GB boot disk. Boot disks are placed on +the hypervisor's ZFS pool, and a blockdevice snapshot is taken every 6hrs. This allows the boot disk +to be rolled back to a last known good point in case an upgrade goes south. If you haven't seen it +yet, take a look at [[zrepl](https://zrepl.github.io/)], a one-stop, integrated solution for ZFS +replication. This tool is incredibly powerful, and can do snapshot management, sourcing / sinking +to remote hosts, of course using incremental snapshots as they are native to ZFS. + +Once the machine is up, we pass three four enterprise-class storage, in our case 3.84TB Kioxia NVMe +drives, model _KXD51RUE3T84_ which are PCIe 3.1 x4 lanes, and NVMe 1.2.1 specification with a good +durability and reasonable (albeit not stellar) read throughput of ~2700MB/s, write throughput of +~800MB/s with 240 kIOPS random read and 21 kIOPS random write. My attention is also drawn to a +specific specification point: these drives allow for 1.0 DWPD, which stands for _Drive Writes Per +Day_, in other words they are not going to run themselves off a cliff after a few petabytes of +writes, and I am reminded that a CT Log wants to write to disk a lot during normal operation. + +The point of these logs is to **keep them safe**, and the most important aspects of the compute +environment are the use of ECC memory to detect single bit errors, and dependable storage. Toshiba +makes a great product. + +``` +ctlog1:~$ sudo zpool create -f -o ashift=12 -o autotrim=on -O atime=off -O xattr=sa \ + ssd-vol0 raidz2 /dev/disk/by-id/nvme-KXD51RUE3T84_TOSHIBA_*M +ctlog1:~$ sudo zfs create -o encryption=on -o keyformat=passphrase ssd-vol0/enc +ctlog1:~$ sudo zfs create ssd-vol0/logs +ctlog1:~$ for log in lipase; do \ + for shard in 2025h2 2026h1 2026h2 2027h1 2027h2; do \ + sudo zfs create ssd-vol0/logs/${log}${shard} \ + done \ + done +``` + +The hypervisor will use PCI passthrough for the NVMe drives, and we'll handle ZFS directly on the +VM. The first command creates a ZFS raidz2 pool using 4kB blocks, turns of _atime_ (which avoids one +metadata write for each read!), and turns on SSD trimming in ZFS, a very useful feature. + +Then I'll create an encrypted volume for the configuration and key material. This way, if the +machine is ever physically transported, the keys will be safe in transit. Finally, I'll create the +temporal log shards starting at 2025h2, all the way through to 2027h2 for our testing log called +_Lipase_ and our production log called _Halloumi_ on Jeroen's machine. On my own machine, it'll be +_Rennet_ for the testing log and _Gouda_ for the production log. + +## Sunlight + +{{< image width="10em" float="right" src="/assets/ctlog/sunlight-logo.png" alt="Sunlight logo" >}} + +I set up Sunlight first. as its authors have extensive operational notes both in terms of the +[[config](https://config.sunlight.geomys.org/)] of Geomys' _Tuscolo_ log, as well as on the +[[Sunlight](https://sunlight.dev)] homepage. I really appreciate that Filippo added some +[[Gists](https://gist.github.com/FiloSottile/989338e6ba8e03f2c699590ce83f537b)] and +[[Doc](https://docs.google.com/document/d/1ID8dX5VuvvrgJrM0Re-jt6Wjhx1eZp-trbpSIYtOhRE/edit?tab=t.0#heading=h.y3yghdo4mdij)] +with pretty much all I need to know to run one too. Our Rennet and Gouda logs use very similar +approach for their configuration, with one notable exception: the VMs do not have a public IP +address, and are tucked away in a private network called IPng Site Local. I'll get back to that +later. + +``` +ctlog@ctlog0:/ssd-vol0/enc/sunlight$ cat << EOF | tee sunlight-staging.yaml +listen: + - "[::]:16420" +checkpoints: /ssd-vol0/shared/checkpoints.db +logs: + - shortname: rennet2025h2 + inception: 2025-07-28 + period: 200 + poolsize: 750 + submissionprefix: https://rennet2025h2.log.ct.ipng.ch + monitoringprefix: https://rennet2025h2.mon.ct.ipng.ch + ccadbroots: testing + extraroots: /ssd-vol0/enc/sunlight/extra-roots-staging.pem + secret: /ssd-vol0/enc/sunlight/keys/rennet2025h2.seed.bin + cache: /ssd-vol0/logs/rennet2025h2/cache.db + localdirectory: /ssd-vol0/logs/rennet2025h2/data + notafterstart: 2025-07-01T00:00:00Z + notafterlimit: 2026-01-01T00:00:00Z +... +EOF +ctlog@ctlog0:/ssd-vol0/enc/sunlight$ cat << EOF | tee skylight-staging.yaml +listen: + - "[::]:16421" +homeredirect: https://ipng.ch/s/ct/ +logs: + - shortname: rennet2025h2 + monitoringprefix: https://rennet2025h2.mon.ct.ipng.ch + localdirectory: /ssd-vol0/logs/rennet2025h2/data + staging: true +... +``` + +In the first configuration file, I'll tell _Sunlight_ (the write path component) to listen on port +`16420` and I'll tell _Skylight_ (the read path component) to listen on port `16421`. I've disabled +the automatic certificate renewals, and will handle SSL upstream: +1. Most importantly, I will be using a common frontend pool with a wildcard certificate for +`*.ct.ipng.ch`. I wrote about [[DNS-01]({{< ref 2023-03-24-lego-dns01 >}})] before, it's a very +convenient way for IPng to do certificate pool management. I will be sharing certificate for all log +types under this certificate. +1. ACME/HTTP-01 could be made to work with a bit of effort; plumbing through the `/.well-known/` +URIs on the frontend and pointing them to these instances. But then the cert would have to be copied +from Sunlight back to the frontends. + +I've noticed that when the log doesn't exist yet, I can start Sunlight and it'll create the bits and +pieces on the local filesystem and start writing checkpoints. But if the log already exists, I am +required to have the _monitoringprefix_ active, otherwise Sunlight won't start up. It's a small +thing, as I will have the read path operational in a few simple steps. Anyway, all five logshards +for Rennet, and a few days later, for Gouda, are operational this way. + +Skylight provides all the things I need to serve the data back, which is a huge help. The [[Static +Log Spec](https://github.com/C2SP/C2SP/blob/main/static-ct-api.md)] is very clear on things like +compression, content-type, cache-control and other headers. Skylight makes this a breeze, as it read +a configuration file very similar to the Sunlight write-path one, and takes care of it all for me. + +## TesseraCT + +{{< image width="10em" float="right" src="/assets/ctlog/tesseract-logo.png" alt="TesseraCT logo" >}} + +Good news came to our community on August 14th, when Google's TrustFabric team announced their Alpha +milestone of [[TesseraCT](https://blog.transparency.dev/introducing-tesseract)]. And the release +also moved the POSIX variant from experimental alongside the already further along GCP and AWS +personalities. After playing around with it with Al and the team, I think I've learned enough to get +us going in a public instance. + +One thing I liked about Sunlight is its compact YAML file that described the pertinent bits of the +system, and that I can serve any number of logs with the same process. On the other hand, TesseraCT +can serve only one log per process. Both have pro's and con's, notably if any poisonous submission +would be offered, Sunlight might take down all logs, while TesseraCT would only take down the log +receiving the offensive submission. On the other hand, maintaining separate processes is cumbersome. + + +### TesseraCT genconf + +I decide to automate this by vibing a little tool called `tesseract-genconf`, which I've published on +[[Gitea](https://git.ipng.ch/certificate-transparency/cheese)]. What it does is take a YAML file +describing the logs, and outputs the bits and pieces needed to operate multiple separate processes +that together form the sharded static log. I've attempted to stay mostly compatible with the +Sunlight YAML configuration, and came up with a variant like this one: + +``` +ctlog@ctlog1:/ssd-vol0/enc/tesseract$ cat << EOF | tee tesseract-staging.yaml +roots: /ssd-vol0/enc/tesseract/roots.pem +logs: + - shortname: lipase2025h2 + listen: "[::]:16900" + submissionprefix: https://lipase2025h2.log.ct.ipng.ch + monitoringprefix: https://lipase2025h2.mon.ct.ipng.ch + extraroots: /ssd-vol0/enc/tesseract/extra-roots-staging.pem + secret: /ssd-vol0/enc/tesseract/keys/lipase2025h2.pem + localdirectory: /ssd-vol0/logs/lipase2025h2/data + notafterstart: 2025-07-01T00:00:00Z + notafterlimit: 2026-01-01T00:00:00Z +... +EOF +``` + +With this snippet, I have all the information I need. Here's the steps I take to construct the log +itself: + +***1. Generate keys*** + +The keys are `prime256v1` and the format that TesseraCT accepts did change since I wrote up my first +[[deep dive]({{< ref 2025-07-26-ctlog-1 >}})] a few weeks ago. Now, the tool accepts a `PEM` format +private key, from which the _Log ID_ and _Public Key_ can be derived. So off I go: + +``` +ctlog@ctlog1:/ssd-vol0/enc/tesseract$ tesseract-genconf -c tesseract-staging.yaml gen-key +Generated /ssd-vol0/enc/tesseract/keys/lipase2025h2.pem +Generated /ssd-vol0/enc/tesseract/keys/lipase2026h1.pem +Generated /ssd-vol0/enc/tesseract/keys/lipase2026h2.pem +Generated /ssd-vol0/enc/tesseract/keys/lipase2027h1.pem +Generated /ssd-vol0/enc/tesseract/keys/lipase2027h2.pem +``` + +Of course, if a file already exists at that location, it'll just print a warning like: +``` +Key already exists: /ssd-vol0/enc/tesseract/keys/lipase2025h2.pem (skipped) +``` + +***2. Generate JSON/HTML*** + +I will be operating the read-path with NGINX. Log operators have started speaking about their log +metadata in terms of a small JSON file called `log.v3.json`, and Skylight does a good job of +exposing that one, alongside all the other pertinent metadata. So I'll generate these files for each +of the logs: + +``` +ctlog@ctlog1:/ssd-vol0/enc/tesseract$ tesseract-genconf -c tesseract-staging.yaml gen-html +Generated /ssd-vol0/logs/lipase2025h2/data/index.html +Generated /ssd-vol0/logs/lipase2025h2/data/log.v3.json +Generated /ssd-vol0/logs/lipase2026h1/data/index.html +Generated /ssd-vol0/logs/lipase2026h1/data/log.v3.json +Generated /ssd-vol0/logs/lipase2026h2/data/index.html +Generated /ssd-vol0/logs/lipase2026h2/data/log.v3.json +Generated /ssd-vol0/logs/lipase2027h1/data/index.html +Generated /ssd-vol0/logs/lipase2027h1/data/log.v3.json +Generated /ssd-vol0/logs/lipase2027h2/data/index.html +Generated /ssd-vol0/logs/lipase2027h2/data/log.v3.json +``` + +{{< image width="60%" src="/assets/ctlog/lipase.png" alt="TesseraCT Lipase Log" >}} + +It's nice to see a familiar look-and-feel for these logs appear in those `index.html` (which all +cross-link to each other within the logs specificied in `tesseract-staging.yaml`, which is dope. + +***3. Generate Roots*** + +Antonis had seen this before (thanks for the explanation!) but TesseraCT does not natively implement +fetching of the [[CCADB](https://www.ccadb.org/)] roots. But, he points out, you can just get them +from any other running log instance, so I'll implement a `gen-roots` command: + +``` +ctlog@ctlog1:/ssd-vol0/enc/tesseract$ tesseract-genconf gen-roots \ + --source https://tuscolo2027h1.sunlight.geomys.org --output production-roots.pem +Fetching roots from: https://tuscolo2027h1.sunlight.geomys.org/ct/v1/get-roots +2025/08/25 08:24:58 Warning: Failed to parse certificate, skipping: x509: negative serial number +Successfully wrote 248 certificates to tusc.pem (out of 249 total) + +ctlog@ctlog1:/ssd-vol0/enc/tesseract$ tesseract-genconf gen-roots \ + --source https://navigli2027h1.sunlight.geomys.org --output testing-roots.pem +Fetching roots from: https://navigli2027h1.sunlight.geomys.org/ct/v1/get-roots +Successfully wrote 82 certificates to tusc.pem (out of 82 total) +``` + +I can do this regularly, say daily, in a cronjob and if the files were to change, restart the +TesseraCT processes. It's not ideal (because the restart might be briefly disruptive), but it's a +reasonable option for the time being. + +***4. Generate TesseraCT cmdline*** + +I will be running TesseraCT as a _templated unit_ in systemd. These are system unit files that have +an argument, they will have an @ in their name, like so: + +``` +ctlog@ctlog1:/ssd-vol0/enc/tesseract$ cat << EOF | sudo tee /lib/systemd/system/tesseract@.service +[Unit] +Description=Tesseract CT Log service for %i +ConditionFileExists=/ssd-vol0/logs/%i/data/.env +After=network.target + +[Service] +# The %i here refers to the instance name, e.g., "lipase2025h2" +# This path should point to where your instance-specific .env files are located +EnvironmentFile=/ssd-vol0/logs/%i/data/.env +ExecStart=/home/ctlog/bin/tesseract-posix $TESSERACT_ARGS +User=ctlog +Group=ctlog +Restart=on-failure +RestartSec=5 + +[Install] +WantedBy=multi-user.target +EOF +``` + +I can now implement a `gen-env` command for my tool: + +``` +ctlog@ctlog1:/ssd-vol0/enc/tesseract$ tesseract-genconf -c tesseract-staging.yaml gen-env +Generated /ssd-vol0/logs/lipase2025h2/data/roots.pem +Generated /ssd-vol0/logs/lipase2025h2/data/.env +Generated /ssd-vol0/logs/lipase2026h1/data/roots.pem +Generated /ssd-vol0/logs/lipase2026h1/data/.env +Generated /ssd-vol0/logs/lipase2026h2/data/roots.pem +Generated /ssd-vol0/logs/lipase2026h2/data/.env +Generated /ssd-vol0/logs/lipase2027h1/data/roots.pem +Generated /ssd-vol0/logs/lipase2027h1/data/.env +Generated /ssd-vol0/logs/lipase2027h2/data/roots.pem +Generated /ssd-vol0/logs/lipase2027h2/data/.env +``` + +Looking at one of those .env files, I can show the exact commandline I'll be feeding to the +`tesseract-posix` binary: + +``` +ctlog@ctlog1:/ssd-vol0/enc/tesseract$ cat /ssd-vol0/logs/lipase2025h2/data/.env +TESSERACT_ARGS="--private_key=/ssd-vol0/enc/tesseract/keys/lipase2025h2.pem + --origin=lipase2025h2.log.ct.ipng.ch --storage_dir=/ssd-vol0/logs/lipase2025h2/data + --roots_pem_file=/ssd-vol0/logs/lipase2025h2/data/roots.pem --http_endpoint=[::]:16900" +OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 +``` + +{{< image width="7em" float="left" src="/assets/shared/warning.png" alt="Warning" >}} +A quick operational note on OpenTelemetry (also often referred to as Otel): Al and the TrustFabric +team added open telemetry to the TesseraCT personalities, as it was mostly already implemented in +the underlying Tessera library. By default, it'll try to send its telemetry to localhost using +`https`, which makes sense in those cases where the collector is on a different machine. In my case, +I'll keep `otelcol` (the collector) on the same machine. Its job is to consume the Otel telemetry +stream, and turn those back into Prometheus `/metrics` endpoint on port `:9464`. + +The `gen-env` command also assembles the per-instance `roots.pem` file. For staging logs, it'll take +the file pointed to by the `roots:` key, and append any per-log `extraroots:` files. For me, these +extraroots are empty and the main roots file points at either the testing roots that came from +_Rennet_ (our Sunlight staging log), or the production roots that came from _Gouda_. A job well done! + +***5. Generate NGINX*** + +When I first ran my tests, I noticed that the log check tool called `ct-fsck` threw errors on my +read path. Filippo explained that the HTTP headers matter in the Static CT specification. Tiles, +Issuers, and Checkpoint must all have specific caching and content type headers set. This is what +makes Skylight such a gem - I get to read it (and the spec!) to see what I'm supposed to be serving. + +And thus, `gen-nginx` command is born, and listens on port `:8080` for requests: + +``` +ctlog@ctlog1:/ssd-vol0/enc/tesseract$ tesseract-genconf -c tesseract-staging.yaml gen-nginx +Generated nginx config: /ssd-vol0/logs/lipase2025h2/data/lipase2025h2.mon.ct.ipng.ch.conf +Generated nginx config: /ssd-vol0/logs/lipase2026h1/data/lipase2026h1.mon.ct.ipng.ch.conf +Generated nginx config: /ssd-vol0/logs/lipase2026h2/data/lipase2026h2.mon.ct.ipng.ch.conf +Generated nginx config: /ssd-vol0/logs/lipase2027h1/data/lipase2027h1.mon.ct.ipng.ch.conf +Generated nginx config: /ssd-vol0/logs/lipase2027h2/data/lipase2027h2.mon.ct.ipng.ch.conf +``` + +All that's left for me to do is symlink these from `/etc/nginx-sites-enabled/` and the read-path is +off to the races. With these commands in the `tesseract-genconf` tool, I am hoping that future +travelers have an easy time setting up their static log. Please let me know if you'd like to use, or +contribute, to the tool. You can find me in the Transparency Dev Slack, in #ct and also #cheese. + + +## IPng Frontends + +{{< image width="18em" float="right" src="/assets/ctlog/MPLS Backbone - CTLog.svg" alt="ctlog at ipng" >}} + +IPng Networks has a private internal network called [[IPng Site Local]({{< ref 2023-03-11-mpls-core +>}})], which is not routed on the internet. Our [[Frontends]({{< ref 2023-03-17-ipng-frontends >}})] +are the only things that have public IPv4 and IPv6 addresses. It allows for things like anycasted +webservers and loadbalancing with +[[Maglev](https://research.google/pubs/maglev-a-fast-and-reliable-software-network-load-balancer/)]. + +The IPng Site Local network kind of looks like the picture to the right. The hypervisors running the +Sunlight and TesseraCT logs are at NTT Zurich1 in Rümlang, Switzerland. The IPng frontends are +in green, and the sweet thing is, some of them run in IPng's own ISP network (AS8298), while others +run in partner networks (like IP-Max AS25091, and Coloclue AS8283). This means that I will benefit +from some pretty solid connectivity redundancy. + +The frontends are provisioned with Ansible. There are two aspects to them - firstly, a _certbot_ +instance maintains the Let's Encrypt wildcard certificates for `*.ct.ipng.ch`. There's a machine +tucked away somewhere called `lego.net.ipng.ch` -- again, not exposed on the internet -- and its job +is to renew certificates and copy them to the machines that need them. Next, a cluster of NGINX +servers uses these certificates to expose IPng and customer services to the Internet. + +I can tie it all together with a snippet like so, for which I apologize in advance - it's quite a +wall of text: + +``` +map $http_user_agent $no_cache_ctlog_lipase { + "~*TesseraCT fsck" 1; + default 0; +} + +server { + listen [::]:443 ssl http2; + listen 0.0.0.0:443 ssl http2; + ssl_certificate /etc/certs/ct.ipng.ch/fullchain.pem; + ssl_certificate_key /etc/certs/ct.ipng.ch/privkey.pem; + include /etc/nginx/conf.d/options-ssl-nginx.inc; + ssl_dhparam /etc/nginx/conf.d/ssl-dhparams.inc; + + server_name lipase2025h2.log.ct.ipng.ch; + access_log /nginx/logs/lipase2025h2.log.ct.ipng.ch-access.log upstream buffer=512k flush=5s; + include /etc/nginx/conf.d/ipng-headers.inc; + + location = / { + proxy_http_version 1.1; + proxy_set_header Host lipase2025h2.mon.ct.ipng.ch; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_pass http://ctlog1.net.ipng.ch:8080/index.html; + } + + location = /metrics { + proxy_http_version 1.1; + proxy_set_header Host $host; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_pass http://ctlog1.net.ipng.ch:9464; + } + + location / { + proxy_http_version 1.1; + proxy_set_header Host $host; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_pass http://ctlog1.net.ipng.ch:16900; + } +} + +server { + listen [::]:443 ssl http2; + listen 0.0.0.0:443 ssl http2; + ssl_certificate /etc/certs/ct.ipng.ch/fullchain.pem; + ssl_certificate_key /etc/certs/ct.ipng.ch/privkey.pem; + include /etc/nginx/conf.d/options-ssl-nginx.inc; + ssl_dhparam /etc/nginx/conf.d/ssl-dhparams.inc; + + server_name lipase2025h2.mon.ct.ipng.ch; + access_log /nginx/logs/lipase2025h2.mon.ct.ipng.ch-access.log upstream buffer=512k flush=5s; + include /etc/nginx/conf.d/ipng-headers.inc; + + location = /checkpoint { + proxy_http_version 1.1; + proxy_set_header Host $host; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + + proxy_pass http://ctlog1.net.ipng.ch:8080; + } + + location / { + proxy_http_version 1.1; + proxy_set_header Host $host; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + + include /etc/nginx/conf.d/ipng-upstream-headers.inc; + proxy_cache ipng_cache; + proxy_cache_key "$scheme://$host$request_uri"; + proxy_cache_valid 200 24h; + proxy_cache_revalidate off; + proxy_cache_bypass $no_cache_ctlog_lipase; + proxy_no_cache $no_cache_ctlog_lipase; + + proxy_pass http://ctlog1.net.ipng.ch:8080; + } +} +``` + +Taking _Lipase_ shard 2025h2 as an example, The submission path (on `*.log.ct.ipng.ch`) will show +the same `index.html` as the monitoring path (on `*.mon.ct.ipng.ch`), to provide some consistency +with Sunlight logs. Otherwise, the `/metrics` endpoint is forwarded to the `otelcol` running on port +`:9464`, and the rest (the `/ct/v1/` and so on) are sent to the first port `:16900` of the +TesseraCT. + +Then the read-path makes a special-case of the `/checkpoint` endpoint, which it does not cache. That +request (as all others) are forwarded to port `:8080` which is where NGINX is running. Other +requests (notably `/tile` and `/issuer`) are cacheable, so I'll cache these on the upstream NGINX +servers, both for resilience as well as for performance. Having four of these NGINX upstream will +allow the Static CT logs (regardless of being Sunlight or TesseraCT) to serve very high read-rates. + +## What's Next + +I need to spend a little bit of time thinking about rate limites, specifically write-ratelimits. I +think I'll use a request limiter in upstream NGINX, to allow for each IP or /24 or /48 subnet to +only send a fixed number of requests/sec. I'll probably keep that part private though, as it's a +good rule of thumb to never offer information to attackers. + +Together with Antonis Chariton and Jeroen Massar, IPng Networks will be offering both TesseraCT and +Sunlight logs on the public internet. One final step is to productionize both logs, and file the +paperwork for them in the community. At this point our Sunlight log has been running for a month or +so, and we've filed the paperwork for it to be included at Apple and Google. + +I'm going to have folks poke at _Lipase_ as well, after which I'll try to run a few `ct-fsck` to +make sure the logs are sane, before offering them into the inclusion program as well. Wish us luck! diff --git a/content/ctlog.md b/content/ctlog.md index 2621f24..0afa062 100644 --- a/content/ctlog.md +++ b/content/ctlog.md @@ -56,14 +56,14 @@ Our TesseraCT logs: * A staging log called [[Lipase](https://lipase2025h2.log.ct.ipng.ch/)], incepted 2025-08-22, starting from temporal shared `lipase2025h2`. * A production log called [[Halloumi](https://halloumi2025h2.log.ct.ipng.ch/)], incepted 2025-08-24, - starting from temporal shared `lipase2025h2`. + starting from temporal shared `halloumi2025h2`. ## Operational Details You can read more details about our infrastructure on: * **[[TesseraCT]({{< ref 2025-07-26-ctlog-1 >}})]** - published on 2025-07-26. * **[[Sunlight]({{< ref 2025-08-10-ctlog-2 >}})]** - published on 2025-08-10. -* **Operational Notes** - (todo) +* **[[Operations]({{< ref 2025-08-24-ctlog-3 >}})]** - published on 2025-08-24. The operators of this infrastructure are **Antonis Chariton**, **Jeroen Massar** and **Pim van Pelt**. \ You can reach us via e-mail at [[](mailto:ct-ops@ipng.ch)]. diff --git a/static/assets/ctlog/lipase.png b/static/assets/ctlog/lipase.png new file mode 100644 index 0000000..ee0b918 --- /dev/null +++ b/static/assets/ctlog/lipase.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b1e732558ebc9de3622e7ed4ef4ecf575e7f95708791270aca8f28fcd7453855 +size 762648