--- date: "2025-08-24T12:07:23Z" title: 'Certificate Transparency - Part 3 - Operations' --- {{< image width="10em" float="right" src="/assets/ctlog/ctlog-logo-ipng.png" alt="ctlog logo" >}} # Introduction There once was a Dutch company called [[DigiNotar](https://en.wikipedia.org/wiki/DigiNotar)], as the name suggests it was a form of _digital notary_, and they were in the business of issuing security certificates. Unfortunately, in June of 2011, their IT infrastructure was compromised and subsequently it issued hundreds of fraudulent SSL certificates, some of which were used for man-in-the-middle attacks on Iranian Gmail users. Not cool. Google launched a project called **Certificate Transparency**, because it was becoming more common that the root of trust given to _Certification Authorities_ could no longer be unilateraly trusted. These attacks showed that the lack of transparency in the way CAs operated was a significant risk to the Web Public Key Infrastructure. It led to the creation of this ambitious [[project](https://certificate.transparency.dev/)] to improve security online by bringing accountability to the system that protects our online services with _SSL_ (Secure Socket Layer) and _TLS_ (Transport Layer Security). In 2013, [[RFC 6962](https://datatracker.ietf.org/doc/html/rfc6962)] was published by the IETF. It describes an experimental protocol for publicly logging the existence of Transport Layer Security (TLS) certificates as they are issued or observed, in a manner that allows anyone to audit certificate authority (CA) activity and notice the issuance of suspect certificates as well as to audit the certificate logs themselves. The intent is that eventually clients would refuse to honor certificates that do not appear in a log, effectively forcing CAs to add all issued certificates to the logs. In the first two articles of this series, I explored [[Sunlight]({{< ref 2025-07-26-ctlog-1 >}})] and [[TesseraCT]({{< ref 2025-08-10-ctlog-2 >}})], two open source implementations of the Static CT protocol. In this final article, I'll share the details on how I created the environment and production instances for four logs that IPng will be providing: Rennet and Lipase are two ingredients to make cheese and will serve as our staging/testing logs. Gouda and Halloumi are two delicious cheeses that pay hommage to our heritage, Jeroen and I being Dutch and Antonis being Greek. ## Hardware At IPng Networks, all hypervisors are from the same brand: Dell's Poweredge line. In this project, Jeroen is also contributing a server, and it so happens that he also has a Dell Poweredge. We're both running Debian on our hypervisor, so we install a fresh VM with Debian 13.0, codenamed _Trixie_, and give the machine 16GB of memory, 8 vCPU and a 16GB boot disk. Boot disks are placed on the hypervisor's ZFS pool, and a blockdevice snapshot is taken every 6hrs. This allows the boot disk to be rolled back to a last known good point in case an upgrade goes south. If you haven't seen it yet, take a look at [[zrepl](https://zrepl.github.io/)], a one-stop, integrated solution for ZFS replication. This tool is incredibly powerful, and can do snapshot management, sourcing / sinking to remote hosts, of course using incremental snapshots as they are native to ZFS. Once the machine is up, we pass three four enterprise-class storage, in our case 3.84TB Kioxia NVMe drives, model _KXD51RUE3T84_ which are PCIe 3.1 x4 lanes, and NVMe 1.2.1 specification with a good durability and reasonable (albeit not stellar) read throughput of ~2700MB/s, write throughput of ~800MB/s with 240 kIOPS random read and 21 kIOPS random write. My attention is also drawn to a specific specification point: these drives allow for 1.0 DWPD, which stands for _Drive Writes Per Day_, in other words they are not going to run themselves off a cliff after a few petabytes of writes, and I am reminded that a CT Log wants to write to disk a lot during normal operation. The point of these logs is to **keep them safe**, and the most important aspects of the compute environment are the use of ECC memory to detect single bit errors, and dependable storage. Toshiba makes a great product. ``` ctlog1:~$ sudo zpool create -f -o ashift=12 -o autotrim=on -O atime=off -O xattr=sa \ ssd-vol0 raidz2 /dev/disk/by-id/nvme-KXD51RUE3T84_TOSHIBA_*M ctlog1:~$ sudo zfs create -o encryption=on -o keyformat=passphrase ssd-vol0/enc ctlog1:~$ sudo zfs create ssd-vol0/logs ctlog1:~$ for log in lipase; do \ for shard in 2025h2 2026h1 2026h2 2027h1 2027h2; do \ sudo zfs create ssd-vol0/logs/${log}${shard} \ done \ done ``` The hypervisor will use PCI passthrough for the NVMe drives, and we'll handle ZFS directly on the VM. The first command creates a ZFS raidz2 pool using 4kB blocks, turns of _atime_ (which avoids one metadata write for each read!), and turns on SSD trimming in ZFS, a very useful feature. Then I'll create an encrypted volume for the configuration and key material. This way, if the machine is ever physically transported, the keys will be safe in transit. Finally, I'll create the temporal log shards starting at 2025h2, all the way through to 2027h2 for our testing log called _Lipase_ and our production log called _Halloumi_ on Jeroen's machine. On my own machine, it'll be _Rennet_ for the testing log and _Gouda_ for the production log. ## Sunlight {{< image width="10em" float="right" src="/assets/ctlog/sunlight-logo.png" alt="Sunlight logo" >}} I set up Sunlight first. as its authors have extensive operational notes both in terms of the [[config](https://config.sunlight.geomys.org/)] of Geomys' _Tuscolo_ log, as well as on the [[Sunlight](https://sunlight.dev)] homepage. I really appreciate that Filippo added some [[Gists](https://gist.github.com/FiloSottile/989338e6ba8e03f2c699590ce83f537b)] and [[Doc](https://docs.google.com/document/d/1ID8dX5VuvvrgJrM0Re-jt6Wjhx1eZp-trbpSIYtOhRE/edit?tab=t.0#heading=h.y3yghdo4mdij)] with pretty much all I need to know to run one too. Our Rennet and Gouda logs use very similar approach for their configuration, with one notable exception: the VMs do not have a public IP address, and are tucked away in a private network called IPng Site Local. I'll get back to that later. ``` ctlog@ctlog0:/ssd-vol0/enc/sunlight$ cat << EOF | tee sunlight-staging.yaml listen: - "[::]:16420" checkpoints: /ssd-vol0/shared/checkpoints.db logs: - shortname: rennet2025h2 inception: 2025-07-28 period: 200 poolsize: 750 submissionprefix: https://rennet2025h2.log.ct.ipng.ch monitoringprefix: https://rennet2025h2.mon.ct.ipng.ch ccadbroots: testing extraroots: /ssd-vol0/enc/sunlight/extra-roots-staging.pem secret: /ssd-vol0/enc/sunlight/keys/rennet2025h2.seed.bin cache: /ssd-vol0/logs/rennet2025h2/cache.db localdirectory: /ssd-vol0/logs/rennet2025h2/data notafterstart: 2025-07-01T00:00:00Z notafterlimit: 2026-01-01T00:00:00Z ... EOF ctlog@ctlog0:/ssd-vol0/enc/sunlight$ cat << EOF | tee skylight-staging.yaml listen: - "[::]:16421" homeredirect: https://ipng.ch/s/ct/ logs: - shortname: rennet2025h2 monitoringprefix: https://rennet2025h2.mon.ct.ipng.ch localdirectory: /ssd-vol0/logs/rennet2025h2/data staging: true ... ``` In the first configuration file, I'll tell _Sunlight_ (the write path component) to listen on port `16420` and I'll tell _Skylight_ (the read path component) to listen on port `16421`. I've disabled the automatic certificate renewals, and will handle SSL upstream: 1. Most importantly, I will be using a common frontend pool with a wildcard certificate for `*.ct.ipng.ch`. I wrote about [[DNS-01]({{< ref 2023-03-24-lego-dns01 >}})] before, it's a very convenient way for IPng to do certificate pool management. I will be sharing certificate for all log types under this certificate. 1. ACME/HTTP-01 could be made to work with a bit of effort; plumbing through the `/.well-known/` URIs on the frontend and pointing them to these instances. But then the cert would have to be copied from Sunlight back to the frontends. I've noticed that when the log doesn't exist yet, I can start Sunlight and it'll create the bits and pieces on the local filesystem and start writing checkpoints. But if the log already exists, I am required to have the _monitoringprefix_ active, otherwise Sunlight won't start up. It's a small thing, as I will have the read path operational in a few simple steps. Anyway, all five logshards for Rennet, and a few days later, for Gouda, are operational this way. Skylight provides all the things I need to serve the data back, which is a huge help. The [[Static Log Spec](https://github.com/C2SP/C2SP/blob/main/static-ct-api.md)] is very clear on things like compression, content-type, cache-control and other headers. Skylight makes this a breeze, as it read a configuration file very similar to the Sunlight write-path one, and takes care of it all for me. ## TesseraCT {{< image width="10em" float="right" src="/assets/ctlog/tesseract-logo.png" alt="TesseraCT logo" >}} Good news came to our community on August 14th, when Google's TrustFabric team announced their Alpha milestone of [[TesseraCT](https://blog.transparency.dev/introducing-tesseract)]. And the release also moved the POSIX variant from experimental alongside the already further along GCP and AWS personalities. After playing around with it with Al and the team, I think I've learned enough to get us going in a public instance. One thing I liked about Sunlight is its compact YAML file that described the pertinent bits of the system, and that I can serve any number of logs with the same process. On the other hand, TesseraCT can serve only one log per process. Both have pro's and con's, notably if any poisonous submission would be offered, Sunlight might take down all logs, while TesseraCT would only take down the log receiving the offensive submission. On the other hand, maintaining separate processes is cumbersome. ### TesseraCT genconf I decide to automate this by vibing a little tool called `tesseract-genconf`, which I've published on [[Gitea](https://git.ipng.ch/certificate-transparency/cheese)]. What it does is take a YAML file describing the logs, and outputs the bits and pieces needed to operate multiple separate processes that together form the sharded static log. I've attempted to stay mostly compatible with the Sunlight YAML configuration, and came up with a variant like this one: ``` ctlog@ctlog1:/ssd-vol0/enc/tesseract$ cat << EOF | tee tesseract-staging.yaml roots: /ssd-vol0/enc/tesseract/roots.pem logs: - shortname: lipase2025h2 listen: "[::]:16900" submissionprefix: https://lipase2025h2.log.ct.ipng.ch monitoringprefix: https://lipase2025h2.mon.ct.ipng.ch extraroots: /ssd-vol0/enc/tesseract/extra-roots-staging.pem secret: /ssd-vol0/enc/tesseract/keys/lipase2025h2.pem localdirectory: /ssd-vol0/logs/lipase2025h2/data notafterstart: 2025-07-01T00:00:00Z notafterlimit: 2026-01-01T00:00:00Z ... EOF ``` With this snippet, I have all the information I need. Here's the steps I take to construct the log itself: ***1. Generate keys*** The keys are `prime256v1` and the format that TesseraCT accepts did change since I wrote up my first [[deep dive]({{< ref 2025-07-26-ctlog-1 >}})] a few weeks ago. Now, the tool accepts a `PEM` format private key, from which the _Log ID_ and _Public Key_ can be derived. So off I go: ``` ctlog@ctlog1:/ssd-vol0/enc/tesseract$ tesseract-genconf -c tesseract-staging.yaml gen-key Generated /ssd-vol0/enc/tesseract/keys/lipase2025h2.pem Generated /ssd-vol0/enc/tesseract/keys/lipase2026h1.pem Generated /ssd-vol0/enc/tesseract/keys/lipase2026h2.pem Generated /ssd-vol0/enc/tesseract/keys/lipase2027h1.pem Generated /ssd-vol0/enc/tesseract/keys/lipase2027h2.pem ``` Of course, if a file already exists at that location, it'll just print a warning like: ``` Key already exists: /ssd-vol0/enc/tesseract/keys/lipase2025h2.pem (skipped) ``` ***2. Generate JSON/HTML*** I will be operating the read-path with NGINX. Log operators have started speaking about their log metadata in terms of a small JSON file called `log.v3.json`, and Skylight does a good job of exposing that one, alongside all the other pertinent metadata. So I'll generate these files for each of the logs: ``` ctlog@ctlog1:/ssd-vol0/enc/tesseract$ tesseract-genconf -c tesseract-staging.yaml gen-html Generated /ssd-vol0/logs/lipase2025h2/data/index.html Generated /ssd-vol0/logs/lipase2025h2/data/log.v3.json Generated /ssd-vol0/logs/lipase2026h1/data/index.html Generated /ssd-vol0/logs/lipase2026h1/data/log.v3.json Generated /ssd-vol0/logs/lipase2026h2/data/index.html Generated /ssd-vol0/logs/lipase2026h2/data/log.v3.json Generated /ssd-vol0/logs/lipase2027h1/data/index.html Generated /ssd-vol0/logs/lipase2027h1/data/log.v3.json Generated /ssd-vol0/logs/lipase2027h2/data/index.html Generated /ssd-vol0/logs/lipase2027h2/data/log.v3.json ``` {{< image width="60%" src="/assets/ctlog/lipase.png" alt="TesseraCT Lipase Log" >}} It's nice to see a familiar look-and-feel for these logs appear in those `index.html` (which all cross-link to each other within the logs specificied in `tesseract-staging.yaml`, which is dope. ***3. Generate Roots*** Antonis had seen this before (thanks for the explanation!) but TesseraCT does not natively implement fetching of the [[CCADB](https://www.ccadb.org/)] roots. But, he points out, you can just get them from any other running log instance, so I'll implement a `gen-roots` command: ``` ctlog@ctlog1:/ssd-vol0/enc/tesseract$ tesseract-genconf gen-roots \ --source https://tuscolo2027h1.sunlight.geomys.org --output production-roots.pem Fetching roots from: https://tuscolo2027h1.sunlight.geomys.org/ct/v1/get-roots 2025/08/25 08:24:58 Warning: Failed to parse certificate, skipping: x509: negative serial number Successfully wrote 248 certificates to tusc.pem (out of 249 total) ctlog@ctlog1:/ssd-vol0/enc/tesseract$ tesseract-genconf gen-roots \ --source https://navigli2027h1.sunlight.geomys.org --output testing-roots.pem Fetching roots from: https://navigli2027h1.sunlight.geomys.org/ct/v1/get-roots Successfully wrote 82 certificates to tusc.pem (out of 82 total) ``` I can do this regularly, say daily, in a cronjob and if the files were to change, restart the TesseraCT processes. It's not ideal (because the restart might be briefly disruptive), but it's a reasonable option for the time being. ***4. Generate TesseraCT cmdline*** I will be running TesseraCT as a _templated unit_ in systemd. These are system unit files that have an argument, they will have an @ in their name, like so: ``` ctlog@ctlog1:/ssd-vol0/enc/tesseract$ cat << EOF | sudo tee /lib/systemd/system/tesseract@.service [Unit] Description=Tesseract CT Log service for %i ConditionFileExists=/ssd-vol0/logs/%i/data/.env After=network.target [Service] # The %i here refers to the instance name, e.g., "lipase2025h2" # This path should point to where your instance-specific .env files are located EnvironmentFile=/ssd-vol0/logs/%i/data/.env ExecStart=/home/ctlog/bin/tesseract-posix $TESSERACT_ARGS User=ctlog Group=ctlog Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target EOF ``` I can now implement a `gen-env` command for my tool: ``` ctlog@ctlog1:/ssd-vol0/enc/tesseract$ tesseract-genconf -c tesseract-staging.yaml gen-env Generated /ssd-vol0/logs/lipase2025h2/data/roots.pem Generated /ssd-vol0/logs/lipase2025h2/data/.env Generated /ssd-vol0/logs/lipase2026h1/data/roots.pem Generated /ssd-vol0/logs/lipase2026h1/data/.env Generated /ssd-vol0/logs/lipase2026h2/data/roots.pem Generated /ssd-vol0/logs/lipase2026h2/data/.env Generated /ssd-vol0/logs/lipase2027h1/data/roots.pem Generated /ssd-vol0/logs/lipase2027h1/data/.env Generated /ssd-vol0/logs/lipase2027h2/data/roots.pem Generated /ssd-vol0/logs/lipase2027h2/data/.env ``` Looking at one of those .env files, I can show the exact commandline I'll be feeding to the `tesseract-posix` binary: ``` ctlog@ctlog1:/ssd-vol0/enc/tesseract$ cat /ssd-vol0/logs/lipase2025h2/data/.env TESSERACT_ARGS="--private_key=/ssd-vol0/enc/tesseract/keys/lipase2025h2.pem --origin=lipase2025h2.log.ct.ipng.ch --storage_dir=/ssd-vol0/logs/lipase2025h2/data --roots_pem_file=/ssd-vol0/logs/lipase2025h2/data/roots.pem --http_endpoint=[::]:16900" OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 ``` {{< image width="7em" float="left" src="/assets/shared/warning.png" alt="Warning" >}} A quick operational note on OpenTelemetry (also often referred to as Otel): Al and the TrustFabric team added open telemetry to the TesseraCT personalities, as it was mostly already implemented in the underlying Tessera library. By default, it'll try to send its telemetry to localhost using `https`, which makes sense in those cases where the collector is on a different machine. In my case, I'll keep `otelcol` (the collector) on the same machine. Its job is to consume the Otel telemetry stream, and turn those back into Prometheus `/metrics` endpoint on port `:9464`. The `gen-env` command also assembles the per-instance `roots.pem` file. For staging logs, it'll take the file pointed to by the `roots:` key, and append any per-log `extraroots:` files. For me, these extraroots are empty and the main roots file points at either the testing roots that came from _Rennet_ (our Sunlight staging log), or the production roots that came from _Gouda_. A job well done! ***5. Generate NGINX*** When I first ran my tests, I noticed that the log check tool called `ct-fsck` threw errors on my read path. Filippo explained that the HTTP headers matter in the Static CT specification. Tiles, Issuers, and Checkpoint must all have specific caching and content type headers set. This is what makes Skylight such a gem - I get to read it (and the spec!) to see what I'm supposed to be serving. And thus, `gen-nginx` command is born, and listens on port `:8080` for requests: ``` ctlog@ctlog1:/ssd-vol0/enc/tesseract$ tesseract-genconf -c tesseract-staging.yaml gen-nginx Generated nginx config: /ssd-vol0/logs/lipase2025h2/data/lipase2025h2.mon.ct.ipng.ch.conf Generated nginx config: /ssd-vol0/logs/lipase2026h1/data/lipase2026h1.mon.ct.ipng.ch.conf Generated nginx config: /ssd-vol0/logs/lipase2026h2/data/lipase2026h2.mon.ct.ipng.ch.conf Generated nginx config: /ssd-vol0/logs/lipase2027h1/data/lipase2027h1.mon.ct.ipng.ch.conf Generated nginx config: /ssd-vol0/logs/lipase2027h2/data/lipase2027h2.mon.ct.ipng.ch.conf ``` All that's left for me to do is symlink these from `/etc/nginx-sites-enabled/` and the read-path is off to the races. With these commands in the `tesseract-genconf` tool, I am hoping that future travelers have an easy time setting up their static log. Please let me know if you'd like to use, or contribute, to the tool. You can find me in the Transparency Dev Slack, in #ct and also #cheese. ## IPng Frontends {{< image width="18em" float="right" src="/assets/ctlog/MPLS Backbone - CTLog.svg" alt="ctlog at ipng" >}} IPng Networks has a private internal network called [[IPng Site Local]({{< ref 2023-03-11-mpls-core >}})], which is not routed on the internet. Our [[Frontends]({{< ref 2023-03-17-ipng-frontends >}})] are the only things that have public IPv4 and IPv6 addresses. It allows for things like anycasted webservers and loadbalancing with [[Maglev](https://research.google/pubs/maglev-a-fast-and-reliable-software-network-load-balancer/)]. The IPng Site Local network kind of looks like the picture to the right. The hypervisors running the Sunlight and TesseraCT logs are at NTT Zurich1 in Rümlang, Switzerland. The IPng frontends are in green, and the sweet thing is, some of them run in IPng's own ISP network (AS8298), while others run in partner networks (like IP-Max AS25091, and Coloclue AS8283). This means that I will benefit from some pretty solid connectivity redundancy. The frontends are provisioned with Ansible. There are two aspects to them - firstly, a _certbot_ instance maintains the Let's Encrypt wildcard certificates for `*.ct.ipng.ch`. There's a machine tucked away somewhere called `lego.net.ipng.ch` -- again, not exposed on the internet -- and its job is to renew certificates and copy them to the machines that need them. Next, a cluster of NGINX servers uses these certificates to expose IPng and customer services to the Internet. I can tie it all together with a snippet like so, for which I apologize in advance - it's quite a wall of text: ``` map $http_user_agent $no_cache_ctlog_lipase { "~*TesseraCT fsck" 1; default 0; } server { listen [::]:443 ssl http2; listen 0.0.0.0:443 ssl http2; ssl_certificate /etc/certs/ct.ipng.ch/fullchain.pem; ssl_certificate_key /etc/certs/ct.ipng.ch/privkey.pem; include /etc/nginx/conf.d/options-ssl-nginx.inc; ssl_dhparam /etc/nginx/conf.d/ssl-dhparams.inc; server_name lipase2025h2.log.ct.ipng.ch; access_log /nginx/logs/lipase2025h2.log.ct.ipng.ch-access.log upstream buffer=512k flush=5s; include /etc/nginx/conf.d/ipng-headers.inc; location = / { proxy_http_version 1.1; proxy_set_header Host lipase2025h2.mon.ct.ipng.ch; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_pass http://ctlog1.net.ipng.ch:8080/index.html; } location = /metrics { proxy_http_version 1.1; proxy_set_header Host $host; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_pass http://ctlog1.net.ipng.ch:9464; } location / { proxy_http_version 1.1; proxy_set_header Host $host; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_pass http://ctlog1.net.ipng.ch:16900; } } server { listen [::]:443 ssl http2; listen 0.0.0.0:443 ssl http2; ssl_certificate /etc/certs/ct.ipng.ch/fullchain.pem; ssl_certificate_key /etc/certs/ct.ipng.ch/privkey.pem; include /etc/nginx/conf.d/options-ssl-nginx.inc; ssl_dhparam /etc/nginx/conf.d/ssl-dhparams.inc; server_name lipase2025h2.mon.ct.ipng.ch; access_log /nginx/logs/lipase2025h2.mon.ct.ipng.ch-access.log upstream buffer=512k flush=5s; include /etc/nginx/conf.d/ipng-headers.inc; location = /checkpoint { proxy_http_version 1.1; proxy_set_header Host $host; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_pass http://ctlog1.net.ipng.ch:8080; } location / { proxy_http_version 1.1; proxy_set_header Host $host; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; include /etc/nginx/conf.d/ipng-upstream-headers.inc; proxy_cache ipng_cache; proxy_cache_key "$scheme://$host$request_uri"; proxy_cache_valid 200 24h; proxy_cache_revalidate off; proxy_cache_bypass $no_cache_ctlog_lipase; proxy_no_cache $no_cache_ctlog_lipase; proxy_pass http://ctlog1.net.ipng.ch:8080; } } ``` Taking _Lipase_ shard 2025h2 as an example, The submission path (on `*.log.ct.ipng.ch`) will show the same `index.html` as the monitoring path (on `*.mon.ct.ipng.ch`), to provide some consistency with Sunlight logs. Otherwise, the `/metrics` endpoint is forwarded to the `otelcol` running on port `:9464`, and the rest (the `/ct/v1/` and so on) are sent to the first port `:16900` of the TesseraCT. Then the read-path makes a special-case of the `/checkpoint` endpoint, which it does not cache. That request (as all others) are forwarded to port `:8080` which is where NGINX is running. Other requests (notably `/tile` and `/issuer`) are cacheable, so I'll cache these on the upstream NGINX servers, both for resilience as well as for performance. Having four of these NGINX upstream will allow the Static CT logs (regardless of being Sunlight or TesseraCT) to serve very high read-rates. ## What's Next I need to spend a little bit of time thinking about rate limites, specifically write-ratelimits. I think I'll use a request limiter in upstream NGINX, to allow for each IP or /24 or /48 subnet to only send a fixed number of requests/sec. I'll probably keep that part private though, as it's a good rule of thumb to never offer information to attackers. Together with Antonis Chariton and Jeroen Massar, IPng Networks will be offering both TesseraCT and Sunlight logs on the public internet. One final step is to productionize both logs, and file the paperwork for them in the community. At this point our Sunlight log has been running for a month or so, and we've filed the paperwork for it to be included at Apple and Google. I'm going to have folks poke at _Lipase_ as well, after which I'll try to run a few `ct-fsck` to make sure the logs are sane, before offering them into the inclusion program as well. Wish us luck!