diff --git a/content/articles/2025-07-26-ctlog-1.md b/content/articles/2025-07-26-ctlog-1.md new file mode 100644 index 0000000..31ed75f --- /dev/null +++ b/content/articles/2025-07-26-ctlog-1.md @@ -0,0 +1,696 @@ +--- +date: "2025-07-26T22:07:23Z" +title: 'Certificate Transparency - Part 1' +--- + +{{< image width="10em" float="right" src="/assets/ctlog/ctlog-logo-ipng.png" alt="ctlog logo" >}} + +# Introduction + +There once was a Dutch company called [[DigiNotar](https://en.wikipedia.org/wiki/DigiNotar)], as the +name suggests it was a form of _digital notary_, and they were in the business of issuing security +certificates. Unfortunately, in June of 2011, their IT infrastructure was compromised and +subsequently it issued hundreds of fraudulent SSL certificates, some of which were used for +man-in-the-middle attacks on Iranian Gmail users. Not cool. + +Google launched a project called **Certificate Transparency**, because it was becoming more common +that the root of trust given to _Certificate Authorities_ could no longer be unilateraly trusted. +These attacks showed that the lack of transparency in the way CAs operated was a significant risk to +the Web Public Key Infrastructure. It led to the creation of this ambitious +[[project](https://certificate.transparency.dev/)] to improve security online by bringing +accountability to the system that protects our online services with _SSL_ (Secure Socket Layer) +and _TLS_ (Transport Layer Security). + +In 2013, [[RFC 6962](https://datatracker.ietf.org/doc/html/rfc6962)] was published by the IETF. It +describes an experimental protocol for publicly logging the existence of Transport Layer Security +(TLS) certificates as they are issued or observed, in a manner that allows anyone to audit +certificate authority (CA) activity and notice the issuance of suspect certificates as well as to +audit the certificate logs themselves. The intent is that eventually clients would refuse to honor +certificates that do not appear in a log, effectively forcing CAs to add all issued certificates to +the logs. + +This series explores and documents how IPng Networks will be running two Static CT _Logs_ with two +different implementations. One will be [[Sunlight](https://sunlight.dev/)], and the other will be +[[TesseraCT](https://github.com/transparency-dev/tesseract)]. + +## Static Certificate Transparency + +In this context, _Logs_ are network services that implement the protocol operations for submissions +and queries that are defined in this RFC. A few years ago, my buddy Antonis asked me if I would be +willing to run a log, but operationally they were very complex and expensive to run. However, over +the years, the concept of _Static Logs_ put running on in reach. The [[Static CT +API](https://github.com/C2SP/C2SP/blob/main/static-ct-api.md)] defines a read-path HTTP static asset +hierarchy (for monitoring) to be implemented alongside the write-path RFC 6962 endpoints (for +submission). + +Aside from the different read endpoints, a log that implements the Static API is a regular CT log +that can work alongside RFC 6962 logs and that fulfills the same purpose. In particular, it requires +no modification to submitters and TLS clients. + +If you only read one document about Static CT, read Filippo Valsorda's excellent +[[paper](https://filippo.io/a-different-CT-log)]. It describes a radically cheaper and easier to +operate [[Certificate Transparency](https://certificate.transparency.dev/)] log that is backed by a +consistent object storage, and can scale to 30x the current issuance rate for 2-10% of the costs +with no merge delay. + +## Scalable, Cheap, Reliable: choose two + +{{< image width="18em" float="right" src="/assets/ctlog/MPLS Backbone - CTLog.svg" alt="ctlog at ipng" >}} + +In the diagram, I've drawn an overview of IPng's network. In {{< boldcolor color="red" >}}red{{< +/boldcolor >}} a european backbone network is provided by an [[BGP Free Core +network](2022-12-09-oem-switch-2.md)]. It operates a private IPv4, IPv6 and MPLS network, called +_IPng Site Local_, which is not connected to the internet. On top of that, IPng offers L2 and L3 +services, for example using [[VPP]({{< ref 2021-02-27-network >}})]. + +In {{< boldcolor color="lightgreen" >}}green{{< /boldcolor >}} I built a cluster of replicated +NGINX frontends. They connect into _IPng Site Local_ and can reach all hypervisors, VMs, and storage +systems. They also connect to the Internet with a single IPv4 and IPv6 address. One might say that +SSL is _added and removed here :-)_ [[ref](/assets/ctlog/nsa_slide.jpg)]. + +Then in {{< boldcolor color="orange" >}}orange{{< /boldcolor >}} I built a set of [[Minio]({{< ref +2025-05-28-minio-1 >}})] S3 storage pools. Amongst others, I serve the static content from the IPng +website from these pools, providing fancy redundancy and caching. I wrote about its design in [[this +article]({{< ref 2025-06-01-minio-2 >}})]. + +Finally, I turn my attention to the {{< boldcolor color="blue" >}}blue{{< /boldcolor >}} which is +two hypervisors, one run by [[IPng](https://ipng.ch/)] and the other by [[Massar](https://massars.net/)]. Each +of them will be running one of the _Log_ implementations. IPng provides two large ZFS storage tanks +for offsite backup, in case a hypervisor decides to check out, and daily backups to an S3 bucket +using Restic. + +Having explained all of this, I am well aware that end to end reliability will be coming from the +fact that there are many independent _Log_ operators, and folks wanting to validate certificates can +simply monitor many. If there is a gap in coverage, say due to any given _Log_'s downtime, this will +not necessarily be problematic. It does mean that I may have to suppress the SRE in me... + +## Minio + +My first instinct is to leverage the distributed storage IPng has, but as I'll show in the rest of +this article, maybe a simpler, more elegant design could be superior, precisely because individual +log reliability is not _as important_ as having many available log _instances_ to choose from. + +From operators in the field I understand that the world-wide generation of certificates is roughly +17M/day, which amounts of some 200-250qps of writes. My first thought is to see how fast my open +source S3 machines can go, really. I'm curious also as to the difference between SSD and spinning +disks. + +I boot two Dell R630s in the Lab. These machines have two Xeon E5-2640 v4 CPUs for a total of 20 +cores and 40 threads, and 512GB of DDR4 memory. They also sport a SAS controller. In one machine I +place 6pcs 1.2TB SAS3 disks (HPE part number EG1200JEHMC), and in the second machine I place 6pcs +of 1.92TB enterprise storage (Samsung part number P1633N19). + +I spin up a 6-device Minio cluster on both and take them out for a spin using [[S3 +Benchmark](https://github.com/wasabi-tech/s3-benchmark.git)] from Wasabi Tech. + +``` +pim@ctlog-test:~/src/s3-benchmark$ for dev in disk ssd; do \ + for t in 1 8 32; do \ + for z in 4M 1M 8k 4k; do \ + ./s3-benchmark -a $KEY -s $SECRET -u http://minio-$dev:9000 -t $t -z $z \ + | tee -a minio-results.txt; \ + done; \ + done; \ +done +``` + +The loadtest above does a bunch of runs with varying parameters. First it tries to read and write +object sizes of 4MB, 1MB, 8kB and 4kB respectively. Then it tries to do this with either 1 thread, 8 +threads or 32 threads. Finally it tests both the disk-based variant as well as the SSD based one. +The loadtest runs from a third machine, so that the Dell R630 disk tanks can stay completely +dedicated to their task of running Minio. + +{{< image width="100%" src="/assets/ctlog/minio_8kb_performance.png" alt="MinIO 8kb disk vs SSD" >}} + +The left-hand side graph feels pretty natural to me. With one thread, uploading 8kB objects will +quickly hit the IOPS rate of the disks, each of which have to participate in the write due to EC:3 +encoding when using six disks, and it tops out at ~56 PUT/s. The single thread hitting SSDs will not +hit that limit, and has ~371 PUT/s which I found a bit underwhelming. But, when performing the +loadtest with either 8 or 32 write threads, the hard disks become only marginally faster (topping +out at 240 PUT/s), while the SSDs really start to shine, with 3850 PUT/s. Pretty good performance. + +On the read-side, I am pleasantly surprised that there's not really that much of a difference +between disks and SSDs. This is likely because the host filesystem cache is playing a large role, so +the 1-thread performance is equivalent (765 GET/s for disks, 677 GET/s for SSDs), and the 32-thread +performance is also equivalent (at 7624 GET/s for disks with 7261 GET/s for SSDs). I do wonder why +the hard disks consistently outperform the SSDs with all the other variables (OS, MinIO version, +hardware) the same. + +## Sidequest: SeaweedFS + +Something that has long caught my attention is the way in which +[[SeaweedFS](https://github.com/seaweedfs/seaweedfs)] approaches blob storage. Many operators have +great success with many small file writes in SeaweedFS compared to MinIO and even AWS S3 storage. +This is because writes with WeedFS are not broken into erasure-sets, which would require every disk +to write a small part or checksum of the data, but rather files are replicated within the cluster in +their entirety on different disks, racks or datacenters. I won't bore you with the details of +SeaweedFS but I'll tack on a docker [[compose file](/assets/ctlog/seaweedfs.docker-compose.yml)] +that I used at the end of this article, if you're curious. + +{{< image width="100%" src="/assets/ctlog/size_comparison_8t.png" alt="MinIO vs SeaWeedFS" >}} + +In the write-path, SeaweedFS dominates in all cases, due to its different way of achieving durable +storage (per-file replication in SeaweedFS versus all-disk erasure-sets in MinIO): +* 4k: 3,384 ops/sec vs MinIO's 111 ops/sec (30x faster!) +* 8k: 3,332 ops/sec vs MinIO's 111 ops/sec (30x faster!) +* 1M: 383 ops/sec vs MinIO's 44 ops/sec (9x faster) +* 4M: 104 ops/sec vs MinIO's 32 ops/sec (4x faster) + +For the read-path, in GET operations MinIO is better at small objects, and really dominates the +large objects: +* 4k: 7,411 ops/sec vs SeaweedFS 5,014 ops/sec +* 8k: 7,666 ops/sec vs SeaweedFS 5,165 ops/sec +* 1M: 5,466 ops/sec vs SeaweedFS 2,212 ops/sec +* 4M: 3,084 ops/sec vs SeaweedFS 646 ops/sec + +This makes me draw an interesting conclusion: seeing as CT Logs are read/write heavy (every couple +of seconds, the Merkle tree is recomputed which is reasonably disk-intensive), SeaweedFS might be a +slight better choice. IPng Networks has three Minio deployments, but no SeaweedFS deployments. Yet. + +# Tessera + +Tessera is a Go library for building tile-based transparency logs (tlogs) +[[ref](https://github.com/C2SP/C2SP/blob/main/tlog-tiles.md)]. It is the logical successor to the +approach that Google took when building and operating _Logs_ using its predecessor called +[[Trillian](https://github.com/google/trillian)]. The implementation and its APIs bake-in current +best-practices based on the lessons learned over the past decade of building and operating +transparency logs in production environments and at scale. + +Tessera was introduced at the Transparency.Dev summit in October 2024. I first watch Al and Martin +[[introduce](https://www.youtube.com/watch?v=9j_8FbQ9qSc)] it at last year's summit. At a high +level, it wraps what used to be a whole kubernetes cluster full of components, into a single library +that can be used with Cloud based services, either like AWS S3 and RDS database, or like GCP's GCS +storage and Spanner database. However, Google also made is easy to use a regular POSIX filesystem +implementation. + +## TesseraCT + +{{< image width="10em" float="right" src="/assets/ctlog/tesseract-logo.png" alt="tesseract logo" >}} + +While Tessera is a library, a CT log implementation comes from its sibling GitHub repository called +[[TesseraCT](https://github.com/transparency-dev/tesseract)]. Because it leverages Tessera under the +hood, TesseraCT can run on GCP, AWS, POSIX-compliant, or on S3-compatible systems alongside a MySQL +database. In order to provide ecosystem agility and to control the growth of CT Log sizes, new CT +Logs must be temporally sharded, defining a certificate expiry range denoted in the form of two +dates: `[rangeBegin, rangeEnd)`. The certificate expiry range allows a Log to reject otherwise valid +logging submissions for certificates that expire before or after this defined range, thus +partitioning the set of publicly-trusted certificates that each Log will accept. I will be expected +to keep logs for an extended period of time, say 3-5 years. + +It's time for me to figure out what this TesseraCT thing can do .. are you ready? Let's go! + +### TesseraCT: S3 and SQL + +TesseraCT comes with a few so-called _personalities_. Those are an implementation of the underlying +storage infrastructure in an opinionated way. The first personality I look at is the `aws` one in +`cmd/tesseract/aws`. I notice that this personality does make hard assumptions about the use of AWS +which is unfortunate as the documentation says '.. or self-hosted S3 and MySQL database'. However, +the `aws` personality assumes the AWS SecretManager in order to fetch its signing key. Before I +can be successful, I need to detangle that. + +#### TesseraCT: AWS and Local Signer + +First, I change `cmd/tesseract/aws/main.go` to add two new flags: + +* ***-signer_public_key_file***: a path to the public key for checkpoints and SCT signer +* ***-signer_private_key_file***: a path to the private key for checkpoints and SCT signer + +I then change the program to assume if these flags are both set, the user will want a +_NewLocalSigner_ instead of a _NewSecretsManagerSigner_. Now all I have to do is implement the +signer interface in a package `local_signer.go`. There, function _NewLocalSigner()_ will read the +public and private PEM from file, decode them, and create an _ECDSAWithSHA256Signer_ with them, a +simple example to show what I mean: + +``` +// NewLocalSigner creates a new signer that uses the ECDSA P-256 key pair from +// local disk files for signing digests. +func NewLocalSigner(publicKeyFile, privateKeyFile string) (*ECDSAWithSHA256Signer, error) { + // Read public key + publicKeyPEM, err := os.ReadFile(publicKeyFile) + publicPemBlock, rest := pem.Decode(publicKeyPEM) + + var publicKey crypto.PublicKey + publicKey, err = x509.ParsePKIXPublicKey(publicPemBlock.Bytes) + ecdsaPublicKey, ok := publicKey.(*ecdsa.PublicKey) + + // Read private key + privateKeyPEM, err := os.ReadFile(privateKeyFile) + privatePemBlock, rest := pem.Decode(privateKeyPEM) + + var ecdsaPrivateKey *ecdsa.PrivateKey + ecdsaPrivateKey, err = x509.ParseECPrivateKey(privatePemBlock.Bytes) + + // Verify the correctness of the signer key pair + if !ecdsaPrivateKey.PublicKey.Equal(ecdsaPublicKey) { + return nil, errors.New("signer key pair doesn't match") + } + + return &ECDSAWithSHA256Signer{ + publicKey: ecdsaPublicKey, + privateKey: ecdsaPrivateKey, + }, nil +} +``` + +In the snippet above I omitted all of the error handling, but the local signer logic itself is +hopefully clear. And with that, I am liberated from Amazon's Cloud offering and can run this thing +all by myself! + +#### TesseraCT: Running with S3, MySQL, and Local Signer + +First, I need to create a suitable ECDSA key: +``` +pim@ctlog-test:~$ openssl ecparam -name prime256v1 -genkey -noout -out /tmp/private_key.pem +pim@ctlog-test:~$ openssl ec -in /tmp/private_key.pem -pubout -out /tmp/public_key.pem +``` + +Then, I'll install the MySQL server and create the databases: + +``` +pim@ctlog-test:~$ sudo apt install default-mysql-server +pim@ctlog-test:~$ sudo mysql -u root + +CREATE USER 'tesseract'@'localhost' IDENTIFIED BY ''; +CREATE DATABASE tesseract; +CREATE DATABASE tesseract_antispam; +GRANT ALL PRIVILEGES ON tesseract.* TO 'tesseract'@'localhost'; +GRANT ALL PRIVILEGES ON tesseract_antispam.* TO 'tesseract'@'localhost'; +``` + +Finally, I use the SSD Minio lab-machine that I just loadtested to create an S3 bucket. + +``` +pim@ctlog-test:~$ mc mb minio-ssd/tesseract-test +pim@ctlog-test:~$ cat << EOF > /tmp/minio-access.json +{ "Version": "2012-10-17", "Statement": [ { + "Effect": "Allow", + "Action": [ "s3:ListBucket", "s3:PutObject", "s3:GetObject", "s3:DeleteObject" ], + "Resource": [ "arn:aws:s3:::tesseract-test/*", "arn:aws:s3:::tesseract-test" ] + } ] +} +EOF +pim@ctlog-test:~$ mc admin user add minio-ssd +pim@ctlog-test:~$ mc admin policy create minio-ssd tesseract-test-access /tmp/minio-access.json +pim@ctlog-test:~$ mc admin policy attach minio-ssd tesseract-test-access --user +pim@ctlog-test:~$ mc anonymous set public minio-ssd/tesseract-test +``` + +{{< image width="6em" float="left" src="/assets/shared/brain.png" alt="brain" >}} + +After some fiddling, I understand that the AWS software development kit makes some assumptions that +you'll be using .. _quelle surprise_ .. AWS services. But you can also use local S3 services by +setting a few key environment variables. I had heard of the S3 access and secret key environment +variables before, but I now need to also use a different S3 endpoint. That little detour into the +codebase only took me .. several hours. + +Armed with that knowledge, I can build and finally start my TesseraCT instance: +``` +pim@ctlog-test:~/src/tesseract/cmd/tesseract/aws$ go build -o ~/aws . +pim@ctlog-test:~$ export AWS_DEFAULT_REGION="us-east-1" +pim@ctlog-test:~$ export AWS_ACCESS_KEY_ID="" +pim@ctlog-test:~$ export AWS_SECRET_ACCESS_KEY="" +pim@ctlog-test:~$ export AWS_ENDPOINT_URL_S3="http://minio-ssd.lab.ipng.ch:9000/" +pim@ctlog-test:~$ ./aws --http_endpoint='[::]:6962' \ + --origin=ctlog-test.lab.ipng.ch/test-ecdsa \ + --bucket=tesseract-test \ + --db_host=ctlog-test.lab.ipng.ch \ + --db_user=tesseract \ + --db_password= \ + --db_name=tesseract \ + --antispam_db_name=tesseract_antispam \ + --signer_public_key_file=/tmp/public_key.pem \ + --signer_private_key_file=/tmp/private_key.pem \ + --roots_pem_file=internal/hammer/testdata/test_root_ca_cert.pem + +I0727 15:13:04.666056 337461 main.go:128] **** CT HTTP Server Starting **** +``` + +Hah! I think most of the command line flags and environment variables should make sense, but I was +struggling for a while with the `--roots_pem_file` and the `--origin` flags, so I phoned a friend +(Al Cutter, Googler extraordinaire and an expert in Tessera/CT). He explained to me that the Log is +actually an open endpoint to which anybody might POST data. However, to avoid folks abusing the log +infrastructure, each POST is expected to come from one of the certificate authorities listed in the +`--roots_pem_file`. OK, that makes sense. + +Then, the `--origin` flag designates how my log calls itself. In the resulting `checkpoint` file it +will enumerate a hash of the latest merged and published Merkle tree. In case a server serves +multiple logs, it uses the `--origin` flag to make the destinction which checksum belongs to which. + +``` +pim@ctlog-test:~/src/tesseract$ curl http://tesseract-test.minio-ssd.lab.ipng.ch:9000/checkpoint +ctlog-test.lab.ipng.ch/test-ecdsa +0 +JGPitKWWI0aGuCfC2k1n/p9xdWAYPm5RZPNDXkCEVUU= + +— ctlog-test.lab.ipng.ch/test-ecdsa L+IHdQAAAZhMCONUBAMARjBEAiA/nc9dig6U//vPg7SoTHjt9bxP5K+x3w4MYKpIRn4ULQIgUY5zijRK8qyuJGvZaItDEmP1gohCt+wI+sESBnhkuqo= +``` + +When creating the bucket above, I used `mc anonymous set public`, which made the S3 bucket +world-readable. I can now execute the whole read-path simply by hitting the S3 service. Check. + +#### TesseraCT: Loadtesting S3/MySQL + +{{< image width="12em" float="right" src="/assets/ctlog/stop-hammer-time.jpg" alt="Stop, hammer time" >}} + +The write path is a server on `[::]:6962`. I should be able to write a log to it, but how? Here's +where I am grateful to find a tool in the TesseraCT GitHub repository called `hammer`. This hammer +sets up read and write traffic to a Static CT API log to test correctness and performance under +load. The traffic is sent according to the [[Static CT API](https://c2sp.org/static-ct-api)] spec. +Slick! + +The tool start a text-based UI (my favorite! also when using Cisco T-Rex loadtester) in the terminal +that shows the current status, logs, and supports increasing/decreasing read and write traffic. This +TUI allows for a level of interactivity when probing a new configuration of a log in order to find +any cliffs where performance degrades. For real load-testing applications, especially headless runs +as part of a CI pipeline, it is recommended to run the tool with `-show_ui=false` in order to disable +the UI. + +I'm a bit lost in the somewhat terse +[[README.md](https://github.com/transparency-dev/tesseract/tree/main/internal/hammer)], but my buddy +Al comes to my rescue and explains the flags to me. First of all, the loadtester wants to hit the +same `--origin` that I configured the write-path to accept. In my case this is +`ctlog-test.lab.ipng.ch/test-ecdsa`. Then, it needs the public key for that _Log_, which I can find +in `/tmp/public_key.pem`. The text there is the _DER_ (Distinguished Encoding Rules), stored as a +base64 encoded string. What follows next was the most difficult for me to understand, as I was +thinking the hammer would read some log from the internet somewhere and replay it locally. Al +explains that actually, the `hammer` tool synthetically creates all of these logs itself, and it +regularly reads the `checkpoint` from the `--log_url` place, while it writes its certificates to +`--write_log_url`. The last few flags just inform the `hammer` how mny read and write ops/sec it +should generate, and with that explanation my brain plays _tadaa.wav_ and I am ready to go. + +``` +pim@ctlog-test:~/src/tesseract$ go run ./internal/hammer \ + --origin=ctlog-test.lab.ipng.ch/test-ecdsa \ + --log_public_key=MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEucHtDWe9GYNicPnuGWbEX8rJg/VnDcXs8z40KdoNidBKy6/ZXw2u+NW1XAUnGpXcZozxufsgOMhijsWb25r7jw== \ + --log_url=http://tesseract-test.minio-ssd.lab.ipng.ch:9000/ \ + --write_log_url=http://localhost:6962/ctlog-test.lab.ipng.ch/test-ecdsa/ \ + --max_read_ops=0 \ + --num_writers=5000 \ + --max_write_ops=100 +``` + +{{< image width="30em" float="right" src="/assets/ctlog/ctlog-loadtest1.png" alt="S3/MySQL Loadtest 100qps" >}} + +Cool! It seems that the loadtest is happily chugging along at 100qps. The log is consuming them in +the HTTP write-path by accepting POST requests to +`/ctlog-test.lab.ipng.ch/test-ecdsa/ct/v1/add-chain`, where hammer is offering them at a rate of +100qps, with a configured probability of duplicates set at 10%. What that means is that every now +and again, it'll repeat a previous request. The purpose of this is to stress test the so-called +`antispam` implementation. When `hammer` sends its requests, it signs them with a certificate that +was issued by the CA described in `internal/hammer/testdata/test_root_ca_cert.pem`, which is why +TesseraCT accepts them. + +I raise the write load by using the '>' key a few times. I notice things are great at 500qps, which +is nice because that's double what we are to expect. But I start seeing a bit more noise at 600qps. +When I raise the write-rate to 1000qps, all hell breaks loose on the logs of the server (and similar +logs in the `hammer` loadtester: + +``` +W0727 15:54:33.419881 348475 handlers.go:168] ctlog-test.lab.ipng.ch/test-ecdsa: AddChain handler error: couldn't store the leaf: failed to fetch entry bundle at index 0: failed to fetch resource: getObject: failed to create reader for object "tile/data/000" in bucket "tesseract-test": operation error S3: GetObject, context deadline exceeded +W0727 15:55:02.727962 348475 aws.go:345] GarbageCollect failed: failed to delete one or more objects: failed to delete objects: operation error S3: DeleteObjects, https response error StatusCode: 400, RequestID: 1856202CA3C4B83F, HostID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8, api error MalformedXML: The XML you provided was not well-formed or did not validate against our published schema. +E0727 15:55:10.448973 348475 append_lifecycle.go:293] followerStats: follower "AWS antispam" EntriesProcessed(): failed to read follow coordination info: Error 1040: Too many connections +``` + +I see on the Minio instance that it's doing about 150/s of GETs and 15/s of PUTs, which is totally +reasonable: + +``` +pim@ctlog-test:~/src/tesseract$ mc admin trace --stats ssd +Duration: 6m9s ▰▱▱ +RX Rate:↑ 34 MiB/m +TX Rate:↓ 2.3 GiB/m +RPM : 10588.1 +------------- +Call Count RPM Avg Time Min Time Max Time Avg TTFB Max TTFB Avg Size Rate /min +s3.GetObject 60558 (92.9%) 9837.2 4.3ms 708µs 48.1ms 3.9ms 47.8ms ↑144B ↓246K ↑1.4M ↓2.3G +s3.PutObject 2199 (3.4%) 357.2 5.3ms 2.4ms 32.7ms 5.3ms 32.7ms ↑92K ↑32M +s3.DeleteMultipleObjects 1212 (1.9%) 196.9 877µs 290µs 41.1ms 850µs 41.1ms ↑230B ↓369B ↑44K ↓71K +s3.ListObjectsV2 1212 (1.9%) 196.9 18.4ms 999µs 52.8ms 18.3ms 52.7ms ↑131B ↓261B ↑25K ↓50K +``` + +Another nice way to see what makes it through is this oneliner, which reads the `checkpoint` every +second, and once it changes, shows the delta in seconds and how many certs were written: + +``` +pim@ctlog-test:~/src/tesseract$ T=0; O=0; while :; do \ + N=$(curl -sS http://tesseract-test.minio-ssd.lab.ipng.ch:9000/checkpoint | grep -E '^[0-9]+$'); \ + if [ "$N" -eq "$O" ]; then \ + echo -n .; \ + else \ + echo " $T seconds $((N-O)) certs"; O=$N; T=0; echo -n $N\ ; + fi; \ + T=$((T+1)); sleep 1; done +1012905 .... 5 seconds 2081 certs +1014986 .... 5 seconds 2126 certs +1017112 .... 5 seconds 1913 certs +1019025 .... 5 seconds 2588 certs +1021613 .... 5 seconds 2591 certs +1024204 .... 5 seconds 2197 certs +``` + +So I can see that the checkpoint is refreshed every 5 seconds and between 1913 and 2591 certs are +written each time. And indeed, at 400/s there are no errors or warnings at all. At this write rate, +TesseraCT is using about 2.9 CPUs/s, with MariaDB using 0.3 CPUs/s, but the hammer is using 6.0 +CPUs/s. Overall, the machine is perfectly happily serving for a few hours under this load test. + +***Conclusion: a write-rate of 400/s should be safe with S3+MySQL*** + +### TesseraCT: POSIX + +I have been playing with this idea of having a reliable read-path by having the S3 cluster be +redundant, or by replicating the S3 bucket. But Al asks: why not use our experimental POSIX? +We discuss two very important benefits, but also two drawbacks: + +* On the plus side: + 1. There is no need for S3 storage, read/writing to a local ZFS raidz2 pool instead. + 1. There is no need for MySQL, as the POSIX implementation can use a local badger instance + also on the local filesystem. +* On the drawbacks: + 1. There is a SPOF in the read-path, as the single VM must handle both. The write-path always + has a SPOF on the TesseraCT VM. + 1. Local storage is more expensive than S3 storage, and can be used only for the purposes of + one application (and at best, shared with other VMs on the same hypervisor). + +Come to think of it, this is maybe not such a bad tradeoff. I do kind of like having a single-VM +with a single-binary and no other moving parts. It greatly simplifies the architecture, and for the +read-path I can (and will) still use multiple upstream NGINX machines in IPng's network. + +I consider myself nerd-sniped, and take a look at the POSIX variant. I have a few SAS3 +solid state storage (NetAPP part number X447_S1633800AMD), which I plug into the `ctlog-test` +machine. + +``` +pim@ctlog-test:~$ sudo zpool create -o ashift=12 -o autotrim=on -o ssd-vol0 mirror \ + /dev/disk/by-id/wwn-0x5002538a0??????? +pim@ctlog-test:~$ sudo zfs create ssd-vol0/tesseract-test +pim@ctlog-test:~$ sudo chown pim:pim /ssd-vol0/tesseract-test +pim@ctlog-test:~/src/tesseract$ go run ./cmd/experimental/posix --http_endpoint='[::]:6962' \ + --origin=ctlog-test.lab.ipng.ch/test-ecdsa \ + --private_key=/tmp/private_key.pem \ + --storage_dir=/ssd-vol0/tesseract-test \ + --roots_pem_file=internal/hammer/testdata/test_root_ca_cert.pem +badger 2025/07/27 16:29:15 INFO: All 0 tables opened in 0s +badger 2025/07/27 16:29:15 INFO: Discard stats nextEmptySlot: 0 +badger 2025/07/27 16:29:15 INFO: Set nextTxnTs to 0 +I0727 16:29:15.032845 363156 files.go:502] Initializing directory for POSIX log at "/ssd-vol0/tesseract-test" (this should only happen ONCE per log!) +I0727 16:29:15.034101 363156 main.go:97] **** CT HTTP Server Starting **** + +pim@ctlog-test:~/src/tesseract$ cat /ssd-vol0/tesseract-test/checkpoint +ctlog-test.lab.ipng.ch/test-ecdsa +0 +47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU= + +— ctlog-test.lab.ipng.ch/test-ecdsa L+IHdQAAAZhMSgC8BAMARzBFAiBjT5zdkniKlryqlUlx/gLHOtVK26zuWwrc4BlyTVzCWgIhAJ0GIrlrP7YGzRaHjzdB5tnS5rpP3LeOsPbpLateaiFc +``` + +Alright, I can see the log started and created an empty checkpoint file. Nice! + +Before I can loadtest it, I will need to get the read-path to become visible. The `hammer` can read +a checkpoint from local `file:///` prefixes, but I'll have to serve them over the network eventually +anyway, so I create the following NGINX config for it: + +``` +server { + listen 80 default_server backlog=4096; + listen [::]:80 default_server backlog=4096; + root /ssd-vol0/tesseract-test/; + index index.html index.htm index.nginx-debian.html; + + server_name _; + + access_log /var/log/nginx/access.log combined buffer=512k flush=5s; + + location / { + try_files $uri $uri/ =404; + tcp_nopush on; + sendfile on; + tcp_nodelay on; + keepalive_timeout 65; + keepalive_requests 1000; + } +} +``` + +Just a couple of small thoughts on this configuration. I'm using buffered access logs, to avoid +excessive disk writes in the read-path. Then, I'm using kernel `sendfile()` which will instruct the +kernel to serve the static objects directly, so that NGINX can move on. Further, I'll allow for a +long keepalive in HTTP 1.1, so that future requests can use the same TCP connection, and I'll set +the flag `tcp_nodelay` and `tcp_nopush` to just blast the data out without waiting. + +Without much ado: + +``` +pim@ctlog-test:~/src/tesseract$ curl -sS ctlog-test.lab.ipng.ch/checkpoint +ctlog-test.lab.ipng.ch/test-ecdsa +0 +47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU= + +— ctlog-test.lab.ipng.ch/test-ecdsa L+IHdQAAAZhMTfksBAMASDBGAiEAqADLH0P/SRVloF6G1ezlWG3Exf+sTzPIY5u6VjAKLqACIQCkJO2N0dZQuDHvkbnzL8Hd91oyU41bVqfD3vs5EwUouA== +``` + +#### TesseraCT: Loadtesting POSIX + +The loadtesting is roughly the same. I start the `hammer` with the same 500qps of write rate, which +was roughly where the S3+MySQL variant topped. My checkpoint tracker shows the following: + +``` +pim@ctlog-test:~/src/tesseract$ T=0; O=0; while :; do \ + N=$(curl -sS http://localhost/checkpoint | grep -E '^[0-9]+$'); \ + if [ "$N" -eq "$O" ]; then \ + echo -n .; \ + else \ + echo " $T seconds $((N-O)) certs"; O=$N; T=0; echo -n $N\ ; + fi; \ + T=$((T+1)); sleep 1; done +59250 ......... 10 seconds 5244 certs +64494 ......... 10 seconds 5000 certs +69494 ......... 10 seconds 5000 certs +74494 ......... 10 seconds 5000 certs +79494 ......... 10 seconds 5256 certs +79494 ......... 10 seconds 5256 certs +84750 ......... 10 seconds 5244 certs +89994 ......... 10 seconds 5256 certs +95250 ......... 10 seconds 5000 certs +100250 ......... 10 seconds 5000 certs +105250 ......... 10 seconds 5000 certs +``` + +I learn two things. First, the checkpoint interval in this `posix` variant is 10 seconds, compared +to the 5 seconds of the `aws` variant I tested before. I dive into the code, because there doesn't +seem to be a `--checkpoint_interval` flag. In the `tessera` library, I find +`DefaultCheckpointInterval` which is set to 10 seconds. I change it to be 2 seconds instead, and +restart the `posix` binary: + +``` +238250 . 2 seconds 1000 certs +239250 . 2 seconds 1000 certs +240250 . 2 seconds 1000 certs +241250 . 2 seconds 1000 certs +242250 . 2 seconds 1000 certs +243250 . 2 seconds 1000 certs +244250 . 2 seconds 1000 certs +``` + +{{< image width="30em" float="right" src="/assets/ctlog/ctlog-loadtest2.png" alt="Posix Loadtest 5000qps" >}} + +Very nice! Maybe I can write a few more certs? I restart the `hammer` with 5000/s, which somewhat to my +surprise, ends up serving! + +``` +642608 . 2 seconds 6155 certs +648763 . 2 seconds 10256 certs +659019 . 2 seconds 9237 certs +668256 . 2 seconds 8800 certs +677056 . 2 seconds 8729 certs +685785 . 2 seconds 8237 certs +694022 . 2 seconds 7487 certs +701509 . 2 seconds 8572 certs +710081 . 2 seconds 7413 certs +``` + +The throughput is highly variable though, seemingly between 3700/sec and 5100/sec, and I quickly +find out that the `hammer` is completely saturating the CPU on the machine, leaving very little room +for the `posix` TesseraCT to serve. I'm going to need more machines! + +So I start a `hammer` loadtester on the two now-idle Minio servers, and run them at about 6000qps +**each**, for a total of 12000 certs/sec. And my little `posix` binary is keeping up like a champ: + +``` +2987169 . 2 seconds 23040 certs +3010209 . 2 seconds 23040 certs +3033249 . 2 seconds 21760 certs +3055009 . 2 seconds 21504 certs +3076513 . 2 seconds 23808 certs +3100321 . 2 seconds 22528 certs +``` + +One thing is reasonably clear, the `posix` TesseraCT is CPU bound, not disk bound. The CPU is now +running at about 18.5 CPUs/s (with 20 cores), which is pretty much all this Dell has to offer. The +NetAPP enterprise solid state drives are not impressed: + +``` +pim@ctlog-test:~/src/tesseract$ zpool iostat -v ssd-vol0 10 100 + capacity operations bandwidth +pool alloc free read write read write +-------------------------- ----- ----- ----- ----- ----- ----- +ssd-vol0 11.4G 733G 0 3.13K 0 117M + mirror-0 11.4G 733G 0 3.13K 0 117M + wwn-0x5002538a05302930 - - 0 1.04K 0 39.1M + wwn-0x5002538a053069f0 - - 0 1.06K 0 39.1M + wwn-0x5002538a06313ed0 - - 0 1.02K 0 39.1M +-------------------------- ----- ----- ----- ----- ----- ----- + +pim@ctlog-test:~/src/tesseract$ zpool iostat -l ssd-vol0 10 + capacity operations bandwidth total_wait disk_wait syncq_wait asyncq_wait scrub trim +pool alloc free read write read write read write read write read write read write wait wait +---------- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- +ssd-vol0 14.0G 730G 0 1.48K 0 35.4M - 2ms - 535us - 1us - 3ms - 50ms +ssd-vol0 14.0G 730G 0 1.12K 0 23.0M - 1ms - 733us - 2us - 1ms - 44ms +ssd-vol0 14.1G 730G 0 1.42K 0 45.3M - 508us - 122us - 914ns - 2ms - 41ms +ssd-vol0 14.2G 730G 0 678 0 21.0M - 863us - 144us - 2us - 2ms - - +``` + +## Results + +OK, that kind of seals the deal for me. The write path needs about 250 certs/sec and I'm hammering +now with 12'000 certs/sec, with room to spare. But what about the read path? The cool thing about +the static log is that reads are all entirely done by NGINX. The only file that isn't cacheable is +the `checkpoint` file which gets updated every two seconds (or ten seconds in the default `tessera` +settings). + +So I start yet another `hammer` whose job it is to read back from the static filesystem: + +``` +pim@ctlog-test:~/src/tesseract$ curl localhost/nginx_status; sleep 60; curl localhost/nginx_status +Active connections: 10556 +server accepts handled requests + 25302 25302 1492918 +Reading: 0 Writing: 1 Waiting: 10555 +Active connections: 7791 +server accepts handled requests + 25764 25764 1727631 +Reading: 0 Writing: 1 Waiting: 7790 +``` + +And I can see that it's keeping up quite nicely. In one minute, it handled (1727631-1492918) or +234713 requests, which is a cool 3911 requests/sec. All these read/write hammers are kind of +saturating the `ctlog-test` machine though: + +{{< image width="100%" src="/assets/ctlog/ctlog-loadtest3.png" alt="Posix Loadtest 8000qps write, 4000qps read" >}} + +But after a little bit of fiddling, I can assert my conclusion: + +***Conclusion: a write-rate of 8'000/s alongside a read-rate of 4'000/s should be safe with POSIX*** + +## What's Next + +I am going to offer such a machine in production together with Antonis Chariton, and Jeroen Massar. +I plan to do a few additional things: + +* Test Sunlight as well on the same hardware. It would be nice to see a comparison between write + rates of the two implementations. +* Work with Al Cutter and the Transparency Dev team to close a few small gaps (like the + `local_signer.go` and some Prometheus monitoring of the `posix` binary. +* Install and launch both under `*.ct.ipng.ch`, which in itself deserves its own report, showing + how I intend to do log cycling and care/feeding, as well as report on the real production + experience running these CT Logs. diff --git a/static/assets/ctlog/MPLS Backbone - CTLog.svg b/static/assets/ctlog/MPLS Backbone - CTLog.svg new file mode 100644 index 0000000..0e80d9d --- /dev/null +++ b/static/assets/ctlog/MPLS Backbone - CTLog.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/static/assets/ctlog/ctlog-loadtest1.png b/static/assets/ctlog/ctlog-loadtest1.png new file mode 100644 index 0000000..ef69d54 --- /dev/null +++ b/static/assets/ctlog/ctlog-loadtest1.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9388b651ab3c7889bb1deea8248d8514dfb63ef533459f98a485ddd346acf9fd +size 510308 diff --git a/static/assets/ctlog/ctlog-loadtest2.png b/static/assets/ctlog/ctlog-loadtest2.png new file mode 100644 index 0000000..19057cf --- /dev/null +++ b/static/assets/ctlog/ctlog-loadtest2.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:0a319682432c614a35a873498beed058db36440107c67d50b03b78a9fc01322a +size 1445234 diff --git a/static/assets/ctlog/ctlog-loadtest3.png b/static/assets/ctlog/ctlog-loadtest3.png new file mode 100644 index 0000000..231c3fc --- /dev/null +++ b/static/assets/ctlog/ctlog-loadtest3.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:a534ff58704035dc75e2056f5dd5e6c0d77a3baacb26a42685c32baed93e3b4f +size 1537933 diff --git a/static/assets/ctlog/ctlog-logo-ipng.png b/static/assets/ctlog/ctlog-logo-ipng.png new file mode 100644 index 0000000..e4b689a --- /dev/null +++ b/static/assets/ctlog/ctlog-logo-ipng.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:1e0d9653f6ad5d3c602157e1eda5408c57300692c2894bbf61c887735a2424c8 +size 653973 diff --git a/static/assets/ctlog/minio-results.txt b/static/assets/ctlog/minio-results.txt new file mode 100644 index 0000000..1ebf334 --- /dev/null +++ b/static/assets/ctlog/minio-results.txt @@ -0,0 +1,164 @@ +Wasabi benchmark program v2.0 +Parameters: url=http://minio-disk:9000, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=1, loops=1, size=4M +Loop 1: PUT time 60.0 secs, objects = 813, speed = 54.2MB/sec, 13.5 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 23168, speed = 1.5GB/sec, 386.1 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 2.2 secs, 371.2 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-disk:9000, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=1, loops=1, size=1M +2025/07/20 16:07:25 WARNING: createBucket wasabi-benchmark-bucket error, ignoring BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it. + status code: 409, request id: 1853FACEBAC4D052, host id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8 +Loop 1: PUT time 60.0 secs, objects = 1221, speed = 20.3MB/sec, 20.3 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 31000, speed = 516.7MB/sec, 516.7 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 3.2 secs, 376.5 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-disk:9000, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=1, loops=1, size=8k +2025/07/20 16:09:29 WARNING: createBucket wasabi-benchmark-bucket error, ignoring BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it. + status code: 409, request id: 1853FAEB70060604, host id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8 +Loop 1: PUT time 60.0 secs, objects = 3353, speed = 447KB/sec, 55.9 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 45913, speed = 6MB/sec, 765.2 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 9.3 secs, 361.6 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-disk:9000, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=1, loops=1, size=4k +2025/07/20 16:11:38 WARNING: createBucket wasabi-benchmark-bucket error, ignoring BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it. + status code: 409, request id: 1853FB098B162788, host id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8 +Loop 1: PUT time 60.0 secs, objects = 3404, speed = 226.9KB/sec, 56.7 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 45230, speed = 2.9MB/sec, 753.8 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 9.4 secs, 362.6 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-disk:9000, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=8, loops=1, size=4M +2025/07/20 16:13:47 WARNING: createBucket wasabi-benchmark-bucket error, ignoring BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it. + status code: 409, request id: 1853FB27AE890E75, host id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8 +Loop 1: PUT time 60.1 secs, objects = 1898, speed = 126.4MB/sec, 31.6 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 185034, speed = 12GB/sec, 3083.9 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 0.4 secs, 4267.8 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-disk:9000, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=8, loops=1, size=1M +2025/07/20 16:15:48 WARNING: createBucket wasabi-benchmark-bucket error, ignoring BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it. + status code: 409, request id: 1853FB43C0386015, host id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8 +Loop 1: PUT time 60.2 secs, objects = 2627, speed = 43.7MB/sec, 43.7 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 327959, speed = 5.3GB/sec, 5465.9 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 0.6 secs, 4045.6 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-disk:9000, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=8, loops=1, size=8k +2025/07/20 16:17:49 WARNING: createBucket wasabi-benchmark-bucket error, ignoring BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it. + status code: 409, request id: 1853FB5FE2012590, host id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8 +Loop 1: PUT time 60.0 secs, objects = 6663, speed = 887.7KB/sec, 111.0 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 459962, speed = 59.9MB/sec, 7666.0 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 1.7 secs, 3890.9 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-disk:9000, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=8, loops=1, size=4k +2025/07/20 16:19:50 WARNING: createBucket wasabi-benchmark-bucket error, ignoring BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it. + status code: 409, request id: 1853FB7C3CF0FFCA, host id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8 +Loop 1: PUT time 60.1 secs, objects = 6673, speed = 444.4KB/sec, 111.1 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 444637, speed = 28.9MB/sec, 7410.5 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 1.5 secs, 4411.8 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-disk:9000, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=32, loops=1, size=4M +2025/07/20 16:21:52 WARNING: createBucket wasabi-benchmark-bucket error, ignoring BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it. + status code: 409, request id: 1853FB988DB60881, host id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8 +Loop 1: PUT time 60.2 secs, objects = 3093, speed = 205.5MB/sec, 51.4 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 168750, speed = 11GB/sec, 2811.4 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 0.3 secs, 9112.2 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-disk:9000, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=32, loops=1, size=1M +2025/07/20 16:23:53 WARNING: createBucket wasabi-benchmark-bucket error, ignoring BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it. + status code: 409, request id: 1853FBB4A1E534DE, host id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8 +Loop 1: PUT time 60.2 secs, objects = 4652, speed = 77.2MB/sec, 77.2 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 351187, speed = 5.7GB/sec, 5852.8 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 0.6 secs, 8141.6 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-disk:9000, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=32, loops=1, size=8k +2025/07/20 16:25:54 WARNING: createBucket wasabi-benchmark-bucket error, ignoring BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it. + status code: 409, request id: 1853FBD0C4764C64, host id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8 +Loop 1: PUT time 60.1 secs, objects = 14497, speed = 1.9MB/sec, 241.4 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 457437, speed = 59.6MB/sec, 7623.7 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 1.7 secs, 8353.6 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-disk:9000, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=32, loops=1, size=4k +2025/07/20 16:27:55 WARNING: createBucket wasabi-benchmark-bucket error, ignoring BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it. + status code: 409, request id: 1853FBED210B0792, host id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8 +Loop 1: PUT time 60.1 secs, objects = 14459, speed = 962.6KB/sec, 240.7 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 466680, speed = 30.4MB/sec, 7777.7 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 1.7 secs, 8605.3 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-ssd:9000, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=1, loops=1, size=4M +Loop 1: PUT time 60.0 secs, objects = 1866, speed = 124.4MB/sec, 31.1 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 16400, speed = 1.1GB/sec, 273.3 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 5.1 secs, 369.3 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-ssd:9000, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=1, loops=1, size=1M +2025/07/20 16:32:02 WARNING: createBucket wasabi-benchmark-bucket error, ignoring BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it. + status code: 409, request id: 1853FC25AE815718, host id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8 +Loop 1: PUT time 60.0 secs, objects = 5459, speed = 91MB/sec, 91.0 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 25090, speed = 418.2MB/sec, 418.2 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 14.8 secs, 369.8 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-ssd:9000, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=1, loops=1, size=8k +2025/07/20 16:34:17 WARNING: createBucket wasabi-benchmark-bucket error, ignoring BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it. + status code: 409, request id: 1853FC4514A78873, host id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8 +Loop 1: PUT time 60.0 secs, objects = 22278, speed = 2.9MB/sec, 371.3 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 40626, speed = 5.3MB/sec, 677.1 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 61.6 secs, 361.8 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-ssd:9000, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=1, loops=1, size=4k +2025/07/20 16:37:19 WARNING: createBucket wasabi-benchmark-bucket error, ignoring BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it. + status code: 409, request id: 1853FC6F629ACFAC, host id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8 +Loop 1: PUT time 60.0 secs, objects = 23394, speed = 1.5MB/sec, 389.9 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 39249, speed = 2.6MB/sec, 654.1 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 64.5 secs, 363.0 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-ssd:9000, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=8, loops=1, size=4M +2025/07/20 16:40:23 WARNING: createBucket wasabi-benchmark-bucket error, ignoring BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it. + status code: 409, request id: 1853FC9A5D101971, host id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8 +Loop 1: PUT time 60.0 secs, objects = 10564, speed = 704.1MB/sec, 176.0 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 20682, speed = 1.3GB/sec, 344.6 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 2.5 secs, 4178.8 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-ssd:9000, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=8, loops=1, size=1M +2025/07/20 16:42:26 WARNING: createBucket wasabi-benchmark-bucket error, ignoring BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it. + status code: 409, request id: 1853FCB6EB0A45D9, host id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8 +Loop 1: PUT time 60.0 secs, objects = 26550, speed = 442.4MB/sec, 442.4 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 124810, speed = 2GB/sec, 2080.1 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 6.6 secs, 4049.2 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-ssd:9000, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=8, loops=1, size=8k +2025/07/20 16:44:32 WARNING: createBucket wasabi-benchmark-bucket error, ignoring BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it. + status code: 409, request id: 1853FCD4684A110E, host id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8 +Loop 1: PUT time 60.0 secs, objects = 129363, speed = 16.8MB/sec, 2155.9 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 423956, speed = 55.2MB/sec, 7065.8 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 32.4 secs, 3992.0 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-ssd:9000, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=8, loops=1, size=4k +2025/07/20 16:47:05 WARNING: createBucket wasabi-benchmark-bucket error, ignoring BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it. + status code: 409, request id: 1853FCF7EA4857CF, host id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8 +Loop 1: PUT time 60.0 secs, objects = 123067, speed = 8MB/sec, 2051.0 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 357694, speed = 23.3MB/sec, 5961.4 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 30.9 secs, 3986.0 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-ssd:9000, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=32, loops=1, size=4M +2025/07/20 16:49:36 WARNING: createBucket wasabi-benchmark-bucket error, ignoring BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it. + status code: 409, request id: 1853FD1B12EFDEBC, host id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8 +Loop 1: PUT time 60.1 secs, objects = 13131, speed = 873.3MB/sec, 218.3 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.1 secs, objects = 18630, speed = 1.2GB/sec, 310.2 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 1.7 secs, 7787.5 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-ssd:9000, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=32, loops=1, size=1M +2025/07/20 16:51:38 WARNING: createBucket wasabi-benchmark-bucket error, ignoring BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it. + status code: 409, request id: 1853FD3779E97644, host id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8 +Loop 1: PUT time 60.1 secs, objects = 40226, speed = 669.8MB/sec, 669.8 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 85692, speed = 1.4GB/sec, 1427.8 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 4.7 secs, 8610.2 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-ssd:9000, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=32, loops=1, size=8k +2025/07/20 16:53:42 WARNING: createBucket wasabi-benchmark-bucket error, ignoring BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it. + status code: 409, request id: 1853FD5489FB2F1F, host id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8 +Loop 1: PUT time 60.0 secs, objects = 230985, speed = 30.1MB/sec, 3849.3 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 435703, speed = 56.7MB/sec, 7261.1 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 25.8 secs, 8945.8 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-ssd:9000, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=32, loops=1, size=4k +2025/07/20 16:56:08 WARNING: createBucket wasabi-benchmark-bucket error, ignoring BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it. + status code: 409, request id: 1853FD7683B9BB96, host id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8 +Loop 1: PUT time 60.0 secs, objects = 228647, speed = 14.9MB/sec, 3810.4 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 452412, speed = 29.5MB/sec, 7539.9 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 27.2 secs, 8418.0 deletes/sec. Slowdowns = 0 diff --git a/static/assets/ctlog/minio_8kb_performance.png b/static/assets/ctlog/minio_8kb_performance.png new file mode 100644 index 0000000..adba580 --- /dev/null +++ b/static/assets/ctlog/minio_8kb_performance.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:af88f0b31bd2556a9f960cd358941c434b84bfcbc4a5589946a1b8bb4cab7562 +size 27406 diff --git a/static/assets/ctlog/nsa_slide.jpg b/static/assets/ctlog/nsa_slide.jpg new file mode 100644 index 0000000..92cd134 --- /dev/null +++ b/static/assets/ctlog/nsa_slide.jpg @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7387d06c4a455a352ee720c6c3ac242d8a48e5bff494a6ab6d5cbb15d60e888b +size 48725 diff --git a/static/assets/ctlog/seaweedfs-results.txt b/static/assets/ctlog/seaweedfs-results.txt new file mode 100644 index 0000000..f0b7f8a --- /dev/null +++ b/static/assets/ctlog/seaweedfs-results.txt @@ -0,0 +1,80 @@ +Wasabi benchmark program v2.0 +Parameters: url=http://minio-disk:8333, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=1, loops=1, size=1M +Loop 1: PUT time 60.0 secs, objects = 1994, speed = 33.2MB/sec, 33.2 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 29243, speed = 487.4MB/sec, 487.4 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 2.8 secs, 701.4 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-disk:8333, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=1, loops=1, size=8k +Loop 1: PUT time 60.0 secs, objects = 13634, speed = 1.8MB/sec, 227.2 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 32284, speed = 4.2MB/sec, 538.1 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 18.7 secs, 727.8 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-disk:8333, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=8, loops=1, size=1M +Loop 1: PUT time 62.0 secs, objects = 23733, speed = 382.8MB/sec, 382.8 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 132708, speed = 2.2GB/sec, 2211.7 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 3.7 secs, 6490.1 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-disk:8333, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=8, loops=1, size=8k +Loop 1: PUT time 60.0 secs, objects = 199925, speed = 26MB/sec, 3331.9 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 309937, speed = 40.4MB/sec, 5165.3 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 31.2 secs, 6406.0 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-ssd:8333, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=1, loops=1, size=1M +Loop 1: PUT time 60.0 secs, objects = 1975, speed = 32.9MB/sec, 32.9 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 29898, speed = 498.3MB/sec, 498.3 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 2.7 secs, 726.6 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-ssd:8333, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=1, loops=1, size=8k +Loop 1: PUT time 60.0 secs, objects = 13662, speed = 1.8MB/sec, 227.7 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 31865, speed = 4.1MB/sec, 531.1 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 18.8 secs, 726.9 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-ssd:8333, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=8, loops=1, size=1M +Loop 1: PUT time 60.0 secs, objects = 26622, speed = 443.6MB/sec, 443.6 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 117688, speed = 1.9GB/sec, 1961.3 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 4.1 secs, 6499.5 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-ssd:8333, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=8, loops=1, size=8k +Loop 1: PUT time 60.0 secs, objects = 198238, speed = 25.8MB/sec, 3303.9 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 312868, speed = 40.7MB/sec, 5214.3 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 30.8 secs, 6432.7 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-disk:8333, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=8, loops=1, size=4M +Loop 1: PUT time 60.1 secs, objects = 6220, speed = 414.2MB/sec, 103.6 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 38773, speed = 2.5GB/sec, 646.1 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 0.9 secs, 6693.3 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-disk:8333, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=8, loops=1, size=4k +Loop 1: PUT time 60.0 secs, objects = 203033, speed = 13.2MB/sec, 3383.8 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 300824, speed = 19.6MB/sec, 5013.6 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 31.1 secs, 6528.6 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-disk:8333, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=32, loops=1, size=4M +Loop 1: PUT time 60.3 secs, objects = 13181, speed = 874.2MB/sec, 218.6 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.1 secs, objects = 18575, speed = 1.2GB/sec, 309.3 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 0.8 secs, 17547.2 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-disk:8333, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=32, loops=1, size=4k +Loop 1: PUT time 60.0 secs, objects = 495006, speed = 32.2MB/sec, 8249.5 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 465947, speed = 30.3MB/sec, 7765.4 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 41.4 secs, 11961.3 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-ssd:8333, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=8, loops=1, size=4M +Loop 1: PUT time 60.1 secs, objects = 7073, speed = 471MB/sec, 117.8 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 31248, speed = 2GB/sec, 520.7 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 1.1 secs, 6576.1 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-ssd:8333, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=8, loops=1, size=4k +Loop 1: PUT time 60.0 secs, objects = 214387, speed = 14MB/sec, 3573.0 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 297586, speed = 19.4MB/sec, 4959.7 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 32.9 secs, 6519.8 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-ssd:8333, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=32, loops=1, size=4M +Loop 1: PUT time 60.1 secs, objects = 14365, speed = 956MB/sec, 239.0 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.1 secs, objects = 18113, speed = 1.2GB/sec, 301.6 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 0.8 secs, 18655.8 deletes/sec. Slowdowns = 0 +Wasabi benchmark program v2.0 +Parameters: url=http://minio-ssd:8333, bucket=wasabi-benchmark-bucket, region=us-east-1, duration=60, threads=32, loops=1, size=4k +Loop 1: PUT time 60.0 secs, objects = 489736, speed = 31.9MB/sec, 8161.8 operations/sec. Slowdowns = 0 +Loop 1: GET time 60.0 secs, objects = 460296, speed = 30MB/sec, 7671.2 operations/sec. Slowdowns = 0 +Loop 1: DELETE time 41.0 secs, 11957.6 deletes/sec. Slowdowns = 0 diff --git a/static/assets/ctlog/seaweedfs.docker-compose.yml b/static/assets/ctlog/seaweedfs.docker-compose.yml new file mode 100644 index 0000000..1e4f767 --- /dev/null +++ b/static/assets/ctlog/seaweedfs.docker-compose.yml @@ -0,0 +1,116 @@ +# Test Setup for SeaweedFS with 6 disks, a Filer an an S3 API +# +# Use with the following .env file +# root@minio-ssd:~# cat /opt/seaweedfs/.env +# AWS_ACCESS_KEY_ID="hottentotten" +# AWS_SECRET_ACCESS_KEY="tentententoonstelling" + +services: +# Master + master0: + image: chrislusf/seaweedfs + ports: + - 9333:9333 + - 19333:19333 + command: "-v=1 master -volumeSizeLimitMB 100 -resumeState=false -ip=master0 -ip.bind=0.0.0.0 -port=9333 -mdir=/var/lib/seaweedfs/master" + volumes: + - ./data/master0:/var/lib/seaweedfs/master + restart: unless-stopped + +# Volume Server 1 + volume1: + image: chrislusf/seaweedfs + command: 'volume -dataCenter=dc1 -rack=r1 -mserver="master0:9333" -port=8081 -preStopSeconds=1 -dir=/var/lib/seaweedfs/volume1' + volumes: + - /data/disk1:/var/lib/seaweedfs/volume1 + depends_on: + - master0 + restart: unless-stopped + +# Volume Server 2 + volume2: + image: chrislusf/seaweedfs + command: 'volume -dataCenter=dc1 -rack=r1 -mserver="master0:9333" -port=8082 -preStopSeconds=1 -dir=/var/lib/seaweedfs/volume2' + volumes: + - /data/disk2:/var/lib/seaweedfs/volume2 + depends_on: + - master0 + restart: unless-stopped + +# Volume Server 3 + volume3: + image: chrislusf/seaweedfs + command: 'volume -dataCenter=dc1 -rack=r1 -mserver="master0:9333" -port=8083 -preStopSeconds=1 -dir=/var/lib/seaweedfs/volume3' + volumes: + - /data/disk3:/var/lib/seaweedfs/volume3 + depends_on: + - master0 + restart: unless-stopped + +# Volume Server 4 + volume4: + image: chrislusf/seaweedfs + command: 'volume -dataCenter=dc1 -rack=r1 -mserver="master0:9333" -port=8084 -preStopSeconds=1 -dir=/var/lib/seaweedfs/volume4' + volumes: + - /data/disk4:/var/lib/seaweedfs/volume4 + depends_on: + - master0 + restart: unless-stopped + +# Volume Server 5 + volume5: + image: chrislusf/seaweedfs + command: 'volume -dataCenter=dc1 -rack=r1 -mserver="master0:9333" -port=8085 -preStopSeconds=1 -dir=/var/lib/seaweedfs/volume5' + volumes: + - /data/disk5:/var/lib/seaweedfs/volume5 + depends_on: + - master0 + restart: unless-stopped + +# Volume Server 6 + volume6: + image: chrislusf/seaweedfs + command: 'volume -dataCenter=dc1 -rack=r1 -mserver="master0:9333" -port=8086 -preStopSeconds=1 -dir=/var/lib/seaweedfs/volume6' + volumes: + - /data/disk6:/var/lib/seaweedfs/volume6 + depends_on: + - master0 + restart: unless-stopped + +# Filer + filer: + image: chrislusf/seaweedfs + ports: + - 8888:8888 + - 18888:18888 + command: 'filer -defaultReplicaPlacement=002 -iam -master="master0:9333"' + volumes: + - ./data/filer:/data + depends_on: + - master0 + - volume1 + - volume2 + - volume3 + - volume4 + - volume5 + - volume6 + restart: unless-stopped + +# S3 API + s3: + image: chrislusf/seaweedfs + ports: + - 8333:8333 + command: 's3 -filer="filer:8888" -ip.bind=0.0.0.0' + env_file: + - .env + depends_on: + - master0 + - volume1 + - volume2 + - volume3 + - volume4 + - volume5 + - volume6 + - filer + restart: unless-stopped diff --git a/static/assets/ctlog/size_comparison_8t.png b/static/assets/ctlog/size_comparison_8t.png new file mode 100644 index 0000000..aa82fe4 --- /dev/null +++ b/static/assets/ctlog/size_comparison_8t.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c0af594f02a318de99bede3ce4870efd5d66cbb12ff685932925f7fe1ab0af65 +size 32257 diff --git a/static/assets/ctlog/stop-hammer-time.jpg b/static/assets/ctlog/stop-hammer-time.jpg new file mode 100644 index 0000000..9f54ca8 --- /dev/null +++ b/static/assets/ctlog/stop-hammer-time.jpg @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b0c7c78ab9d7eae9cc0052a6afffd938288acab6e4c8c63f85b0fba7c64a7fe3 +size 5758 diff --git a/static/assets/ctlog/tesseract-logo.png b/static/assets/ctlog/tesseract-logo.png new file mode 100644 index 0000000..7ee24b2 --- /dev/null +++ b/static/assets/ctlog/tesseract-logo.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:97b58c6e54465a2e581f0ab8ddf417e019ed83edea43c31e1235a20b5e065cc2 +size 2952259 diff --git a/themes/hugo-theme-ipng/layouts/shortcodes/boldcolor.html b/themes/hugo-theme-ipng/layouts/shortcodes/boldcolor.html new file mode 100644 index 0000000..75a9bd2 --- /dev/null +++ b/themes/hugo-theme-ipng/layouts/shortcodes/boldcolor.html @@ -0,0 +1 @@ +{{ .Inner }}