ipng/ipng.ch
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Typo fixes h/t alcutter@ jeroen@
All checks were successful
continuous-integration/drone/push Build is passing
Details

Browse Source
This commit is contained in:
Pim van Pelt
2025-07-27 17:30:24 +00:00
parent bc2a9bb352
commit 6a59b7d7e6
1 changed files with 31 additions and 28 deletions
Download Patch File Download Diff File Expand all files Collapse all files

59
content/articles/2025-07-26-ctlog-1.md
View File

@ -14,7 +14,7 @@ subsequently it issued hundreds of fraudulent SSL certificates, some of which we
man-in-the-middle attacks on Iranian Gmail users. Not cool. man-in-the-middle attacks on Iranian Gmail users. Not cool.
Google launched a project called **Certificate Transparency**, because it was becoming more common Google launched a project called **Certificate Transparency**, because it was becoming more common
that the root of trust given to _Certificate Authorities_ could no longer be unilateraly trusted. that the root of trust given to _Certification Authorities_ could no longer be unilateraly trusted.
These attacks showed that the lack of transparency in the way CAs operated was a significant risk to These attacks showed that the lack of transparency in the way CAs operated was a significant risk to
the Web Public Key Infrastructure. It led to the creation of this ambitious the Web Public Key Infrastructure. It led to the creation of this ambitious
[[project](https://certificate.transparency.dev/)] to improve security online by bringing [[project](https://certificate.transparency.dev/)] to improve security online by bringing
@ -36,12 +36,12 @@ different implementations. One will be [[Sunlight](https://sunlight.dev/)], and
## Static Certificate Transparency ## Static Certificate Transparency
In this context, _Logs_ are network services that implement the protocol operations for submissions In this context, _Logs_ are network services that implement the protocol operations for submissions
and queries that are defined in this RFC. A few years ago, my buddy Antonis asked me if I would be and queries that are defined in a specification that builds on the previous RFC. A few years ago,
willing to run a log, but operationally they were very complex and expensive to run. However, over my buddy Antonis asked me if I would be willing to run a log, but operationally they were very
the years, the concept of _Static Logs_ put running on in reach. The [[Static CT complex and expensive to run. However, over the years, the concept of _Static Logs_ put running one
API](https://github.com/C2SP/C2SP/blob/main/static-ct-api.md)] defines a read-path HTTP static asset in reach. This [[Static CT API](https://github.com/C2SP/C2SP/blob/main/static-ct-api.md)] defines a
hierarchy (for monitoring) to be implemented alongside the write-path RFC 6962 endpoints (for read-path HTTP static asset hierarchy (for monitoring) to be implemented alongside the write-path
submission). RFC 6962 endpoints (for submission).
Aside from the different read endpoints, a log that implements the Static API is a regular CT log Aside from the different read endpoints, a log that implements the Static API is a regular CT log
that can work alongside RFC 6962 logs and that fulfills the same purpose. In particular, it requires that can work alongside RFC 6962 logs and that fulfills the same purpose. In particular, it requires
@ -58,8 +58,8 @@ with no merge delay.
{{< image width="18em" float="right" src="/assets/ctlog/MPLS Backbone - CTLog.svg" alt="ctlog at ipng" >}} {{< image width="18em" float="right" src="/assets/ctlog/MPLS Backbone - CTLog.svg" alt="ctlog at ipng" >}}
In the diagram, I've drawn an overview of IPng's network. In {{< boldcolor color="red" >}}red{{< In the diagram, I've drawn an overview of IPng's network. In {{< boldcolor color="red" >}}red{{<
/boldcolor >}} a european backbone network is provided by an [[BGP Free Core /boldcolor >}} a european backbone network is provided by a [[BGP Free Core
network](2022-12-09-oem-switch-2.md)]. It operates a private IPv4, IPv6 and MPLS network, called network]({{< ref 2022-12-09-oem-switch-2 >}})]. It operates a private IPv4, IPv6, and MPLS network, called
_IPng Site Local_, which is not connected to the internet. On top of that, IPng offers L2 and L3 _IPng Site Local_, which is not connected to the internet. On top of that, IPng offers L2 and L3
services, for example using [[VPP]({{< ref 2021-02-27-network >}})]. services, for example using [[VPP]({{< ref 2021-02-27-network >}})].
@ -68,7 +68,7 @@ NGINX frontends. They connect into _IPng Site Local_ and can reach all hyperviso
systems. They also connect to the Internet with a single IPv4 and IPv6 address. One might say that systems. They also connect to the Internet with a single IPv4 and IPv6 address. One might say that
SSL is _added and removed here :-)_ [[ref](/assets/ctlog/nsa_slide.jpg)]. SSL is _added and removed here :-)_ [[ref](/assets/ctlog/nsa_slide.jpg)].
Then in {{< boldcolor color="orange" >}}orange{{< /boldcolor >}} I built a set of [[Minio]({{< ref Then in {{< boldcolor color="orange" >}}orange{{< /boldcolor >}} I built a set of [[MinIO]({{< ref
2025-05-28-minio-1 >}})] S3 storage pools. Amongst others, I serve the static content from the IPng 2025-05-28-minio-1 >}})] S3 storage pools. Amongst others, I serve the static content from the IPng
website from these pools, providing fancy redundancy and caching. I wrote about its design in [[this website from these pools, providing fancy redundancy and caching. I wrote about its design in [[this
article]({{< ref 2025-06-01-minio-2 >}})]. article]({{< ref 2025-06-01-minio-2 >}})].
@ -84,23 +84,26 @@ fact that there are many independent _Log_ operators, and folks wanting to valid
simply monitor many. If there is a gap in coverage, say due to any given _Log_'s downtime, this will simply monitor many. If there is a gap in coverage, say due to any given _Log_'s downtime, this will
not necessarily be problematic. It does mean that I may have to suppress the SRE in me... not necessarily be problematic. It does mean that I may have to suppress the SRE in me...
## Minio ## MinIO
My first instinct is to leverage the distributed storage IPng has, but as I'll show in the rest of My first instinct is to leverage the distributed storage IPng has, but as I'll show in the rest of
this article, maybe a simpler, more elegant design could be superior, precisely because individual this article, maybe a simpler, more elegant design could be superior, precisely because individual
log reliability is not _as important_ as having many available log _instances_ to choose from. log reliability is not _as important_ as having many available log _instances_ to choose from.
From operators in the field I understand that the world-wide generation of certificates is roughly From operators in the field I understand that the world-wide generation of certificates is roughly
17M/day, which amounts of some 200-250qps of writes. My first thought is to see how fast my open 17M/day, which amounts of some 200-250qps of writes. Antonis explains that certs with a validity
source S3 machines can go, really. I'm curious also as to the difference between SSD and spinning if 180 days or less will need two CT log entries, while certs with a validity more than 180d will
disks. need three CT log entries. So the write rate is roughly 2.2x that, as an upper bound.
My first thought is to see how fast my open source S3 machines can go, really. I'm curious also as
to the difference between SSD and spinning disks.
I boot two Dell R630s in the Lab. These machines have two Xeon E5-2640 v4 CPUs for a total of 20 I boot two Dell R630s in the Lab. These machines have two Xeon E5-2640 v4 CPUs for a total of 20
cores and 40 threads, and 512GB of DDR4 memory. They also sport a SAS controller. In one machine I cores and 40 threads, and 512GB of DDR4 memory. They also sport a SAS controller. In one machine I
place 6pcs 1.2TB SAS3 disks (HPE part number EG1200JEHMC), and in the second machine I place 6pcs place 6pcs 1.2TB SAS3 disks (HPE part number EG1200JEHMC), and in the second machine I place 6pcs
of 1.92TB enterprise storage (Samsung part number P1633N19). of 1.92TB enterprise storage (Samsung part number P1633N19).
I spin up a 6-device Minio cluster on both and take them out for a spin using [[S3 I spin up a 6-device MinIO cluster on both and take them out for a spin using [[S3
Benchmark](https://github.com/wasabi-tech/s3-benchmark.git)] from Wasabi Tech. Benchmark](https://github.com/wasabi-tech/s3-benchmark.git)] from Wasabi Tech.
``` ```
@ -118,7 +121,7 @@ The loadtest above does a bunch of runs with varying parameters. First it tries
object sizes of 4MB, 1MB, 8kB and 4kB respectively. Then it tries to do this with either 1 thread, 8 object sizes of 4MB, 1MB, 8kB and 4kB respectively. Then it tries to do this with either 1 thread, 8
threads or 32 threads. Finally it tests both the disk-based variant as well as the SSD based one. threads or 32 threads. Finally it tests both the disk-based variant as well as the SSD based one.
The loadtest runs from a third machine, so that the Dell R630 disk tanks can stay completely The loadtest runs from a third machine, so that the Dell R630 disk tanks can stay completely
dedicated to their task of running Minio. dedicated to their task of running MinIO.
{{< image width="100%" src="/assets/ctlog/minio_8kb_performance.png" alt="MinIO 8kb disk vs SSD" >}} {{< image width="100%" src="/assets/ctlog/minio_8kb_performance.png" alt="MinIO 8kb disk vs SSD" >}}
@ -165,16 +168,16 @@ large objects:
This makes me draw an interesting conclusion: seeing as CT Logs are read/write heavy (every couple This makes me draw an interesting conclusion: seeing as CT Logs are read/write heavy (every couple
of seconds, the Merkle tree is recomputed which is reasonably disk-intensive), SeaweedFS might be a of seconds, the Merkle tree is recomputed which is reasonably disk-intensive), SeaweedFS might be a
slight better choice. IPng Networks has three Minio deployments, but no SeaweedFS deployments. Yet. slight better choice. IPng Networks has three MinIO deployments, but no SeaweedFS deployments. Yet.
# Tessera # Tessera
Tessera is a Go library for building tile-based transparency logs (tlogs) [[Tessera](https://github.com/transparency-dev/tessera.git)] is a Go library for building tile-based
[[ref](https://github.com/C2SP/C2SP/blob/main/tlog-tiles.md)]. It is the logical successor to the transparency logs (tlogs) [[ref](https://github.com/C2SP/C2SP/blob/main/tlog-tiles.md)]. It is the
approach that Google took when building and operating _Logs_ using its predecessor called logical successor to the approach that Google took when building and operating _Logs_ using its
[[Trillian](https://github.com/google/trillian)]. The implementation and its APIs bake-in current predecessor called [[Trillian](https://github.com/google/trillian)]. The implementation and its APIs
best-practices based on the lessons learned over the past decade of building and operating bake-in current best-practices based on the lessons learned over the past decade of building and
transparency logs in production environments and at scale. operating transparency logs in production environments and at scale.
Tessera was introduced at the Transparency.Dev summit in October 2024. I first watch Al and Martin Tessera was introduced at the Transparency.Dev summit in October 2024. I first watch Al and Martin
[[introduce](https://www.youtube.com/watch?v=9j_8FbQ9qSc)] it at last year's summit. At a high [[introduce](https://www.youtube.com/watch?v=9j_8FbQ9qSc)] it at last year's summit. At a high
@ -277,7 +280,7 @@ GRANT ALL PRIVILEGES ON tesseract.* TO 'tesseract'@'localhost';
GRANT ALL PRIVILEGES ON tesseract_antispam.* TO 'tesseract'@'localhost'; GRANT ALL PRIVILEGES ON tesseract_antispam.* TO 'tesseract'@'localhost';
``` ```
Finally, I use the SSD Minio lab-machine that I just loadtested to create an S3 bucket. Finally, I use the SSD MinIO lab-machine that I just loadtested to create an S3 bucket.
``` ```
pim@ctlog-test:~$ mc mb minio-ssd/tesseract-test pim@ctlog-test:~$ mc mb minio-ssd/tesseract-test
@ -373,9 +376,9 @@ same `--origin` that I configured the write-path to accept. In my case this is
in `/tmp/public_key.pem`. The text there is the _DER_ (Distinguished Encoding Rules), stored as a in `/tmp/public_key.pem`. The text there is the _DER_ (Distinguished Encoding Rules), stored as a
base64 encoded string. What follows next was the most difficult for me to understand, as I was base64 encoded string. What follows next was the most difficult for me to understand, as I was
thinking the hammer would read some log from the internet somewhere and replay it locally. Al thinking the hammer would read some log from the internet somewhere and replay it locally. Al
explains that actually, the `hammer` tool synthetically creates all of these logs itself, and it explains that actually, the `hammer` tool synthetically creates all of these entries itself, and it
regularly reads the `checkpoint` from the `--log_url` place, while it writes its certificates to regularly reads the `checkpoint` from the `--log_url` place, while it writes its certificates to
`--write_log_url`. The last few flags just inform the `hammer` how mny read and write ops/sec it `--write_log_url`. The last few flags just inform the `hammer` how many read and write ops/sec it
should generate, and with that explanation my brain plays _tadaa.wav_ and I am ready to go. should generate, and with that explanation my brain plays _tadaa.wav_ and I am ready to go.
``` ```
@ -411,7 +414,7 @@ W0727 15:55:02.727962 348475 aws.go:345] GarbageCollect failed: failed to delet
E0727 15:55:10.448973 348475 append_lifecycle.go:293] followerStats: follower "AWS antispam" EntriesProcessed(): failed to read follow coordination info: Error 1040: Too many connections E0727 15:55:10.448973 348475 append_lifecycle.go:293] followerStats: follower "AWS antispam" EntriesProcessed(): failed to read follow coordination info: Error 1040: Too many connections
``` ```
I see on the Minio instance that it's doing about 150/s of GETs and 15/s of PUTs, which is totally I see on the MinIO instance that it's doing about 150/s of GETs and 15/s of PUTs, which is totally
reasonable: reasonable:
``` ```
@ -612,7 +615,7 @@ The throughput is highly variable though, seemingly between 3700/sec and 5100/se
find out that the `hammer` is completely saturating the CPU on the machine, leaving very little room find out that the `hammer` is completely saturating the CPU on the machine, leaving very little room
for the `posix` TesseraCT to serve. I'm going to need more machines! for the `posix` TesseraCT to serve. I'm going to need more machines!
So I start a `hammer` loadtester on the two now-idle Minio servers, and run them at about 6000qps So I start a `hammer` loadtester on the two now-idle MinIO servers, and run them at about 6000qps
**each**, for a total of 12000 certs/sec. And my little `posix` binary is keeping up like a champ: **each**, for a total of 12000 certs/sec. And my little `posix` binary is keeping up like a champ:
``` ```