diff --git a/content/articles/2025-07-26-ctlog-1.md b/content/articles/2025-07-26-ctlog-1.md index 31ed75f..be1b189 100644 --- a/content/articles/2025-07-26-ctlog-1.md +++ b/content/articles/2025-07-26-ctlog-1.md @@ -14,7 +14,7 @@ subsequently it issued hundreds of fraudulent SSL certificates, some of which we man-in-the-middle attacks on Iranian Gmail users. Not cool. Google launched a project called **Certificate Transparency**, because it was becoming more common -that the root of trust given to _Certificate Authorities_ could no longer be unilateraly trusted. +that the root of trust given to _Certification Authorities_ could no longer be unilateraly trusted. These attacks showed that the lack of transparency in the way CAs operated was a significant risk to the Web Public Key Infrastructure. It led to the creation of this ambitious [[project](https://certificate.transparency.dev/)] to improve security online by bringing @@ -36,12 +36,12 @@ different implementations. One will be [[Sunlight](https://sunlight.dev/)], and ## Static Certificate Transparency In this context, _Logs_ are network services that implement the protocol operations for submissions -and queries that are defined in this RFC. A few years ago, my buddy Antonis asked me if I would be -willing to run a log, but operationally they were very complex and expensive to run. However, over -the years, the concept of _Static Logs_ put running on in reach. The [[Static CT -API](https://github.com/C2SP/C2SP/blob/main/static-ct-api.md)] defines a read-path HTTP static asset -hierarchy (for monitoring) to be implemented alongside the write-path RFC 6962 endpoints (for -submission). +and queries that are defined in a specification that builds on the previous RFC. A few years ago, +my buddy Antonis asked me if I would be willing to run a log, but operationally they were very +complex and expensive to run. However, over the years, the concept of _Static Logs_ put running one +in reach. This [[Static CT API](https://github.com/C2SP/C2SP/blob/main/static-ct-api.md)] defines a +read-path HTTP static asset hierarchy (for monitoring) to be implemented alongside the write-path +RFC 6962 endpoints (for submission). Aside from the different read endpoints, a log that implements the Static API is a regular CT log that can work alongside RFC 6962 logs and that fulfills the same purpose. In particular, it requires @@ -58,8 +58,8 @@ with no merge delay. {{< image width="18em" float="right" src="/assets/ctlog/MPLS Backbone - CTLog.svg" alt="ctlog at ipng" >}} In the diagram, I've drawn an overview of IPng's network. In {{< boldcolor color="red" >}}red{{< -/boldcolor >}} a european backbone network is provided by an [[BGP Free Core -network](2022-12-09-oem-switch-2.md)]. It operates a private IPv4, IPv6 and MPLS network, called +/boldcolor >}} a european backbone network is provided by a [[BGP Free Core +network]({{< ref 2022-12-09-oem-switch-2 >}})]. It operates a private IPv4, IPv6, and MPLS network, called _IPng Site Local_, which is not connected to the internet. On top of that, IPng offers L2 and L3 services, for example using [[VPP]({{< ref 2021-02-27-network >}})]. @@ -68,7 +68,7 @@ NGINX frontends. They connect into _IPng Site Local_ and can reach all hyperviso systems. They also connect to the Internet with a single IPv4 and IPv6 address. One might say that SSL is _added and removed here :-)_ [[ref](/assets/ctlog/nsa_slide.jpg)]. -Then in {{< boldcolor color="orange" >}}orange{{< /boldcolor >}} I built a set of [[Minio]({{< ref +Then in {{< boldcolor color="orange" >}}orange{{< /boldcolor >}} I built a set of [[MinIO]({{< ref 2025-05-28-minio-1 >}})] S3 storage pools. Amongst others, I serve the static content from the IPng website from these pools, providing fancy redundancy and caching. I wrote about its design in [[this article]({{< ref 2025-06-01-minio-2 >}})]. @@ -84,23 +84,26 @@ fact that there are many independent _Log_ operators, and folks wanting to valid simply monitor many. If there is a gap in coverage, say due to any given _Log_'s downtime, this will not necessarily be problematic. It does mean that I may have to suppress the SRE in me... -## Minio +## MinIO My first instinct is to leverage the distributed storage IPng has, but as I'll show in the rest of this article, maybe a simpler, more elegant design could be superior, precisely because individual log reliability is not _as important_ as having many available log _instances_ to choose from. From operators in the field I understand that the world-wide generation of certificates is roughly -17M/day, which amounts of some 200-250qps of writes. My first thought is to see how fast my open -source S3 machines can go, really. I'm curious also as to the difference between SSD and spinning -disks. +17M/day, which amounts of some 200-250qps of writes. Antonis explains that certs with a validity +if 180 days or less will need two CT log entries, while certs with a validity more than 180d will +need three CT log entries. So the write rate is roughly 2.2x that, as an upper bound. + +My first thought is to see how fast my open source S3 machines can go, really. I'm curious also as +to the difference between SSD and spinning disks. I boot two Dell R630s in the Lab. These machines have two Xeon E5-2640 v4 CPUs for a total of 20 cores and 40 threads, and 512GB of DDR4 memory. They also sport a SAS controller. In one machine I place 6pcs 1.2TB SAS3 disks (HPE part number EG1200JEHMC), and in the second machine I place 6pcs of 1.92TB enterprise storage (Samsung part number P1633N19). -I spin up a 6-device Minio cluster on both and take them out for a spin using [[S3 +I spin up a 6-device MinIO cluster on both and take them out for a spin using [[S3 Benchmark](https://github.com/wasabi-tech/s3-benchmark.git)] from Wasabi Tech. ``` @@ -118,7 +121,7 @@ The loadtest above does a bunch of runs with varying parameters. First it tries object sizes of 4MB, 1MB, 8kB and 4kB respectively. Then it tries to do this with either 1 thread, 8 threads or 32 threads. Finally it tests both the disk-based variant as well as the SSD based one. The loadtest runs from a third machine, so that the Dell R630 disk tanks can stay completely -dedicated to their task of running Minio. +dedicated to their task of running MinIO. {{< image width="100%" src="/assets/ctlog/minio_8kb_performance.png" alt="MinIO 8kb disk vs SSD" >}} @@ -165,16 +168,16 @@ large objects: This makes me draw an interesting conclusion: seeing as CT Logs are read/write heavy (every couple of seconds, the Merkle tree is recomputed which is reasonably disk-intensive), SeaweedFS might be a -slight better choice. IPng Networks has three Minio deployments, but no SeaweedFS deployments. Yet. +slight better choice. IPng Networks has three MinIO deployments, but no SeaweedFS deployments. Yet. # Tessera -Tessera is a Go library for building tile-based transparency logs (tlogs) -[[ref](https://github.com/C2SP/C2SP/blob/main/tlog-tiles.md)]. It is the logical successor to the -approach that Google took when building and operating _Logs_ using its predecessor called -[[Trillian](https://github.com/google/trillian)]. The implementation and its APIs bake-in current -best-practices based on the lessons learned over the past decade of building and operating -transparency logs in production environments and at scale. +[[Tessera](https://github.com/transparency-dev/tessera.git)] is a Go library for building tile-based +transparency logs (tlogs) [[ref](https://github.com/C2SP/C2SP/blob/main/tlog-tiles.md)]. It is the +logical successor to the approach that Google took when building and operating _Logs_ using its +predecessor called [[Trillian](https://github.com/google/trillian)]. The implementation and its APIs +bake-in current best-practices based on the lessons learned over the past decade of building and +operating transparency logs in production environments and at scale. Tessera was introduced at the Transparency.Dev summit in October 2024. I first watch Al and Martin [[introduce](https://www.youtube.com/watch?v=9j_8FbQ9qSc)] it at last year's summit. At a high @@ -277,7 +280,7 @@ GRANT ALL PRIVILEGES ON tesseract.* TO 'tesseract'@'localhost'; GRANT ALL PRIVILEGES ON tesseract_antispam.* TO 'tesseract'@'localhost'; ``` -Finally, I use the SSD Minio lab-machine that I just loadtested to create an S3 bucket. +Finally, I use the SSD MinIO lab-machine that I just loadtested to create an S3 bucket. ``` pim@ctlog-test:~$ mc mb minio-ssd/tesseract-test @@ -373,9 +376,9 @@ same `--origin` that I configured the write-path to accept. In my case this is in `/tmp/public_key.pem`. The text there is the _DER_ (Distinguished Encoding Rules), stored as a base64 encoded string. What follows next was the most difficult for me to understand, as I was thinking the hammer would read some log from the internet somewhere and replay it locally. Al -explains that actually, the `hammer` tool synthetically creates all of these logs itself, and it +explains that actually, the `hammer` tool synthetically creates all of these entries itself, and it regularly reads the `checkpoint` from the `--log_url` place, while it writes its certificates to -`--write_log_url`. The last few flags just inform the `hammer` how mny read and write ops/sec it +`--write_log_url`. The last few flags just inform the `hammer` how many read and write ops/sec it should generate, and with that explanation my brain plays _tadaa.wav_ and I am ready to go. ``` @@ -411,7 +414,7 @@ W0727 15:55:02.727962 348475 aws.go:345] GarbageCollect failed: failed to delet E0727 15:55:10.448973 348475 append_lifecycle.go:293] followerStats: follower "AWS antispam" EntriesProcessed(): failed to read follow coordination info: Error 1040: Too many connections ``` -I see on the Minio instance that it's doing about 150/s of GETs and 15/s of PUTs, which is totally +I see on the MinIO instance that it's doing about 150/s of GETs and 15/s of PUTs, which is totally reasonable: ``` @@ -612,7 +615,7 @@ The throughput is highly variable though, seemingly between 3700/sec and 5100/se find out that the `hammer` is completely saturating the CPU on the machine, leaving very little room for the `posix` TesseraCT to serve. I'm going to need more machines! -So I start a `hammer` loadtester on the two now-idle Minio servers, and run them at about 6000qps +So I start a `hammer` loadtester on the two now-idle MinIO servers, and run them at about 6000qps **each**, for a total of 12000 certs/sec. And my little `posix` binary is keeping up like a champ: ```