This commit is contained in:
473
content/articles/2025-06-01-minio-2.md
Normal file
473
content/articles/2025-06-01-minio-2.md
Normal file
@ -0,0 +1,473 @@
|
||||
---
|
||||
date: "2025-06-01T10:07:23Z"
|
||||
title: 'Case Study: Minio S3 - Part 2'
|
||||
---
|
||||
|
||||
{{< image float="right" src="/assets/minio/minio-logo.png" alt="MinIO Logo" width="6em" >}}
|
||||
|
||||
# Introduction
|
||||
|
||||
Amazon Simple Storage Service (Amazon S3) is an object storage service offering industry-leading
|
||||
scalability, data availability, security, and performance. Millions of customers of all sizes and
|
||||
industries store, manage, analyze, and protect any amount of data for virtually any use case, such
|
||||
as data lakes, cloud-native applications, and mobile apps. With cost-effective storage classes and
|
||||
easy-to-use management features, you can optimize costs, organize and analyze data, and configure
|
||||
fine-tuned access controls to meet specific business and compliance requirements.
|
||||
|
||||
Amazon's S3 became the _de facto_ standard object storage system, and there exist several fully open
|
||||
source implementations of the protocol. One of them is MinIO: designed to allow enterprises to
|
||||
consolidate all of their data on a single, private cloud namespace. Architected using the same
|
||||
principles as the hyperscalers, AIStor delivers performance at scale at a fraction of the cost
|
||||
compared to the public cloud.
|
||||
|
||||
IPng Networks is an Internet Service Provider, but I also dabble in self-hosting things, for
|
||||
example [[PeerTube](https://video.ipng.ch/)], [[Mastodon](https://ublog.tech/)],
|
||||
[[Immich](https://photos.ipng.ch/)], [[Pixelfed](https://pix.ublog.tech/)] and of course
|
||||
[[Hugo](https://ipng/ch/)]. These services all have one thing in common: they tend to use lots of
|
||||
storage when they grow. At IPng Networks, all hypervisors ship with enterprise SAS flash drives,
|
||||
mostly 1.92TB and 3.84TB. Scaling up each of these services, and backing them up safely, can be
|
||||
quite the headache.
|
||||
|
||||
In a [[previous article]({{< ref 2025-05-28-minio-1 >}})], I talked through the install of a
|
||||
redundant set of three Minio machines. In this article, I'll start putting them to good use.
|
||||
|
||||
## Use Case: Restic
|
||||
|
||||
{{< image float="right" src="/assets/minio/restic-logo.png" alt="Restic Logo" width="12em" >}}
|
||||
|
||||
[[Restic](https://restic.org/)] is a modern backup program that can back up your files from multiple
|
||||
host OS, to many different storage types, easily, effectively, securely, verifiably and freely. With
|
||||
a sales pitch like that, what's not to love? Actually, I am a long-time
|
||||
[[BorgBackup](https://www.borgbackup.org/)] user, and I think I'll keep that running. However, for
|
||||
resilience, and because I've heard only good things about Restic, I'll make a second backup of the
|
||||
routers, hypervisors, and virtual machines using Restic.
|
||||
|
||||
Restic can use S3 buckets out of the box (incidentally, so can BorgBackup). To configure it, I use
|
||||
a mixture of environment variables and flags. But first, let me create a bucket for the backups.
|
||||
|
||||
```
|
||||
pim@glootie:~$ mc mb chbtl0/ipng-restic
|
||||
pim@glootie:~$ mc admin user add chbtl0/ <key> <secret>
|
||||
pim@glootie:~$ cat << EOF | tee ipng-restic-access.json
|
||||
{
|
||||
"PolicyName": "ipng-restic-access",
|
||||
"Policy": {
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": [ "s3:DeleteObject", "s3:GetObject", "s3:ListBucket", "s3:PutObject" ],
|
||||
"Resource": [ "arn:aws:s3:::ipng-restic", "arn:aws:s3:::ipng-restic/*" ]
|
||||
}
|
||||
]
|
||||
},
|
||||
}
|
||||
EOF
|
||||
pim@glootie:~$ mc admin policy create chbtl0/ ipng-restic-access.json
|
||||
pim@glootie:~$ mc admin policy attach chbtl0/ ipng-restic-access --user <key>
|
||||
```
|
||||
|
||||
First, I'll create a bucket called `ipng-restic`. Then, I'll create a _user_ with a given secret
|
||||
_key_. To protect the innocent, and my backups, I'll not disclose them. Next, I'll create an
|
||||
IAM policy that allows for Get/List/Put/Delete to be performed on the bucket and its contents, and
|
||||
finally I'll attach this policy to the user I just created.
|
||||
|
||||
To run a Restic backup, I'll first have to create a so-called _repository_. The repository has a
|
||||
location and a password, which Restic uses to encrypt the data. Because I'm using S3, I'll also need
|
||||
to specify the key and secret:
|
||||
|
||||
```
|
||||
root@glootie:~# RESTIC_PASSWORD="changeme"
|
||||
root@glootie:~# RESTIC_REPOSITORY="s3:https://s3.chbtl0.ipng.ch/ipng-restic/$(hostname)/"
|
||||
root@glootie:~# AWS_ACCESS_KEY_ID="<key>"
|
||||
root@glootie:~# AWS_SECRET_ACCESS_KEY:="<secret>"
|
||||
root@glootie:~# export RESTIC_PASSWORD RESTIC_REPOSITORY AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY
|
||||
root@glootie:~# restic init
|
||||
created restic repository 807cf25e85 at s3:https://s3.chbtl0.ipng.ch/ipng-restic/glootie.ipng.ch/
|
||||
```
|
||||
|
||||
Restic prints out some repository finterprint of the latest 'snapshot' it just created. Taking a
|
||||
look on the MinIO install:
|
||||
|
||||
```
|
||||
pim@glootie:~$ mc stat chbtl0/ipng-restic/glootie.ipng.ch/
|
||||
Name : config
|
||||
Date : 2025-06-01 12:01:43 UTC
|
||||
Size : 155 B
|
||||
ETag : 661a43f72c43080649712e45da14da3a
|
||||
Type : file
|
||||
Metadata :
|
||||
Content-Type: application/octet-stream
|
||||
|
||||
Name : keys/
|
||||
Date : 2025-06-01 12:03:33 UTC
|
||||
Type : folder
|
||||
```
|
||||
|
||||
Cool. Now I'm ready to make my first full backup:
|
||||
|
||||
```
|
||||
root@glootie:~# ARGS="--exclude /proc --exclude /sys --exclude /dev --exclude /run"
|
||||
root@glootie:~# ARGS="$ARGS --exclude-if-present .nobackup"
|
||||
root@glootie:~# restic backup $ARGS /
|
||||
...
|
||||
processed 1141426 files, 131.111 GiB in 15:12
|
||||
snapshot 34476c74 saved
|
||||
```
|
||||
|
||||
Once the backup completes, the Restic authors advise me to also do a check of the repository, and to
|
||||
prune it so that it keeps a finite amount of daily, weekly and monthly backups. My further journey
|
||||
for Restic looks a bit like this:
|
||||
|
||||
```
|
||||
root@glootie:~# restic check
|
||||
using temporary cache in /tmp/restic-check-cache-2712250731
|
||||
create exclusive lock for repository
|
||||
load indexes
|
||||
check all packs
|
||||
check snapshots, trees and blobs
|
||||
[0:04] 100.00% 1 / 1 snapshots
|
||||
|
||||
no errors were found
|
||||
|
||||
root@glootie:~# restic forget --prune --keep-daily 8 --keep-weekly 5 --keep-monthly 6
|
||||
repository 34476c74 opened (version 2, compression level auto)
|
||||
Applying Policy: keep 8 daily, 5 weekly, 6 monthly snapshots
|
||||
keep 1 snapshots:
|
||||
ID Time Host Tags Reasons Paths
|
||||
---------------------------------------------------------------------------------
|
||||
34476c74 2025-06-01 12:18:54 glootie.ipng.ch daily snapshot /
|
||||
weekly snapshot
|
||||
monthly snapshot
|
||||
----------------------------------------------------------------------------------
|
||||
1 snapshots
|
||||
```
|
||||
|
||||
Right on! I proceed to update the Ansible configs at IPng to roll this out against the entire fleet
|
||||
of 152 hosts at IPng Networks. I do this in a little tool called `bitcron`, which I wrote for a
|
||||
previous company I worked at: [[BIT](https://bit.nl)] in the Netherlands. Bitcron allows me to
|
||||
create relatively elegant cronjobs that can raise warnings, errors and fatal issues. If no issues
|
||||
are found, an e-mail can be sent to a bitbucket address, but if warnings or errors are found, a
|
||||
different _monitored_ address will be used. Bitcron is kind of cool, and I wrote it in 2001. Maybe
|
||||
I'll write about it, for old time's sake. I wonder if the folks at BIT still use it?
|
||||
|
||||
## Use Case: NGINX
|
||||
|
||||
{{< image float="right" src="/assets/minio/nginx-logo.png" alt="NGINX Logo" width="11em" >}}
|
||||
|
||||
OK, with the first use case out of the way, I turn my attention to a second - in my opinion more
|
||||
interesting - use case. In the [[previous article]({{< ref 2025-05-28-minio-1 >}})], I created a
|
||||
public bucket called `ipng-web-assets` in which I stored 6.50GB of website data belonging to the
|
||||
IPng website, and some material I posted when I was on my
|
||||
[[Sabbatical](https://sabbatical.ipng.nl/)] last year.
|
||||
|
||||
### MinIO: Bucket Replication
|
||||
|
||||
First things first: redundancy. These web assets are currently pushed to all four nginx machines,
|
||||
and statically served. If I were to replace them with a single S3 bucket, I would create a single
|
||||
point of failure, and that's _no bueno_!
|
||||
|
||||
Off I go, creating a replicated bucket using two MinIO instances (`chbtl0` and `ddln0`):
|
||||
|
||||
```
|
||||
pim@glootie:~$ mc mb ddln0/ipng-web-assets
|
||||
pim@glootie:~$ mc anonymous set download ddln0/ipng-web-assets
|
||||
pim@glootie:~$ mc admin user add ddln0/ <replkey> <replsecret>
|
||||
pim@glootie:~$ cat << EOF | tee ipng-web-assets-access.json
|
||||
{
|
||||
"PolicyName": "ipng-web-assets-access",
|
||||
"Policy": {
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": [ "s3:DeleteObject", "s3:GetObject", "s3:ListBucket", "s3:PutObject" ],
|
||||
"Resource": [ "arn:aws:s3:::ipng-web-assets", "arn:aws:s3:::ipng-web-assets/*" ]
|
||||
}
|
||||
]
|
||||
},
|
||||
}
|
||||
EOF
|
||||
pim@glootie:~$ mc admin policy create ddln0/ ipng-web-assets-access.json
|
||||
pim@glootie:~$ mc admin policy attach ddln0/ ipng-web-assets-access --user <replkey>
|
||||
pim@glootie:~$ mc replicate add chbtl0/ipng-web-assets \
|
||||
--remote-bucket https://<key>:<secret>@s3.ddln0.ipng.ch/ipng-web-assets
|
||||
```
|
||||
|
||||
What happens next is pure magic. I've told `chbtl0` that I want it to replicate all existing and
|
||||
future changes to that bucket to its neighbor `ddln0`. Only minutes later, I check the replication
|
||||
status, just to see that it's _already done_:
|
||||
|
||||
```
|
||||
pim@glootie:~$ mc replicate status chbtl0/ipng-web-assets
|
||||
Replication status since 1 hour
|
||||
s3.ddln0.ipng.ch
|
||||
Replicated: 142 objects (6.5 GiB)
|
||||
Queued: ● 0 objects, 0 B (avg: 4 objects, 915 MiB ; max: 0 objects, 0 B)
|
||||
Workers: 0 (avg: 0; max: 0)
|
||||
Transfer Rate: 15 kB/s (avg: 88 MB/s; max: 719 MB/s
|
||||
Latency: 3ms (avg: 3ms; max: 7ms)
|
||||
Link: ● online (total downtime: 0 milliseconds)
|
||||
Errors: 0 in last 1 minute; 0 in last 1hr; 0 since uptime
|
||||
Configured Max Bandwidth (Bps): 644 GB/s Current Bandwidth (Bps): 975 B/s
|
||||
pim@summer:~/src/ipng-web-assets$ mc ls ddln0/ipng-web-assets/
|
||||
[2025-06-01 12:42:22 CEST] 0B ipng.ch/
|
||||
[2025-06-01 12:42:22 CEST] 0B sabbatical.ipng.nl/
|
||||
```
|
||||
|
||||
MinIO has pumped the data from bucket `ipng-web-assets` to the other machine at an average of 88MB/s
|
||||
with a peak throughput of 719MB/s (probably for the larger VM images). And indeed, looking at the
|
||||
|
||||
### MinIO: Missing directory index
|
||||
|
||||
I take a look at what I just built, on the following URL:
|
||||
* [https://ipng-web-assets.s3.ddln0.ipng.ch/sabbatical.ipng.nl/media/vdo/IMG_0406_0.mp4](https://ipng-web-assets.s3.ddln0.ipng.ch/sabbatical.ipng.nl/media/vdo/IMG_0406_0.mp4)
|
||||
|
||||
That checks out, and I can see the mess that was my room when I first went on sabbatical. By the
|
||||
way, I totally cleaned it up, see
|
||||
[[here](https://sabbatical.ipng.nl/blog/2024/08/01/thursday-basement-done/)] for proof. I can't,
|
||||
however, see the directory listing:
|
||||
|
||||
```
|
||||
pim@glootie:~$ curl https://ipng-web-assets.s3.ddln0.ipng.ch/sabbatical.ipng.nl/media/vdo/
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<Error>
|
||||
<Code>NoSuchKey</Code>
|
||||
<Message>The specified key does not exist.</Message>
|
||||
<Key>sabbatical.ipng.nl/media/vdo/</Key>
|
||||
<BucketName>ipng-web-assets</BucketName>
|
||||
<Resource>/sabbatical.ipng.nl/media/vdo/</Resource>
|
||||
<RequestId>1844EC0CFEBF3C5F</RequestId>
|
||||
<HostId>dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8</HostId>
|
||||
</Error>
|
||||
```
|
||||
|
||||
That's unfortunate, because some of the IPng articles link to a directory full of files, which I'd
|
||||
like to be shown so that my readers can navigate through the directories. Surely I'm not the first
|
||||
to encouter this? And sure enough, I'm not
|
||||
[[ref](https://github.com/glowinthedark/index-html-generator)] by user `glowinthedark` who wrote a
|
||||
little python script that generates `index.html` files for their Caddy file server. I'll take me
|
||||
some of that Python, thank you!
|
||||
|
||||
With the following little script, my setup is complete:
|
||||
|
||||
```
|
||||
pim@glootie:~/src/ipng-web-assets$ cat push.sh
|
||||
#!/usr/bin/env bash
|
||||
|
||||
echo "Generating index.html files ..."
|
||||
for D in */media; do
|
||||
echo "* Directory $D"
|
||||
./genindex.py -r $D
|
||||
done
|
||||
echo "Done (genindex)"
|
||||
echo ""
|
||||
|
||||
echo "Mirroring directoro to S3 Bucket"
|
||||
mc mirror --remove --overwrite . chbtl0/ipng-web-assets/
|
||||
echo "Done (mc mirror)"
|
||||
echo ""
|
||||
pim@glootie:~/src/ipng-web-assets$ ./push.sh
|
||||
```
|
||||
|
||||
Only a few seconds after I run `./push.sh`, the replication is complete and I have two identical
|
||||
copies of my media:
|
||||
|
||||
1. [https://ipng-web-assets.s3.chbtl0.ipng.ch/ipng.ch/media/](https://ipng-web-assets.s3.chbtl0.ipng.ch/ipng.ch/media/index.html)
|
||||
1. [https://ipng-web-assets.s3.ddln0.ipng.ch/ipng.ch/media/](https://ipng-web-assets.s3.ddln0.ipng.ch/ipng.ch/media/index.html)
|
||||
|
||||
|
||||
### NGINX: Proxy to Minio
|
||||
|
||||
Before moving to S3 storage, my NGINX frontends all kept a copy of the IPng media on local NVME
|
||||
disk. That's great for reliability, as each NGINX instance is completely hermetic and standalone.
|
||||
However, it's not great for scaling: the current NGINX instances only have 16GB of local storage,
|
||||
and I'd rather not have my static web asset data outgrow that filesystem. From before, I already had
|
||||
an NGINX config that served the Hugo static data from `/var/www/ipng.ch/ and the `/media'
|
||||
subdirectory from a different directory in `/var/www/ipng-web-assets/ipng.ch/media`.
|
||||
|
||||
Moving to redundant S3 storage backenda is straight forward:
|
||||
|
||||
```
|
||||
upstream minio_ipng {
|
||||
least_conn;
|
||||
server minio0.chbtl0.net.ipng.ch:9000;
|
||||
server minio0.ddln0.net.ipng.ch:9000;
|
||||
}
|
||||
|
||||
server {
|
||||
...
|
||||
location / {
|
||||
root /var/www/ipng.ch/;
|
||||
}
|
||||
|
||||
location /media {
|
||||
proxy_set_header Host $http_host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
|
||||
proxy_connect_timeout 300;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Connection "";
|
||||
chunked_transfer_encoding off;
|
||||
|
||||
rewrite (.*)/$ $1/index.html;
|
||||
|
||||
proxy_pass http://minio_ipng/ipng-web-assets/ipng.ch/media;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
I want to make note of a few things:
|
||||
1. The `upstream` definition here uses IPng Site Local entrypoints, considering the NGINX servers
|
||||
all have direct MTU=9000 access to the MinIO instances. I'll put both in there, in a
|
||||
round-robin configuration favoring the replica with _least connections_.
|
||||
1. Deeplinking to directory names without the trailing `/index.html` would serve a 404 from the
|
||||
backend, so I'll intercept these and rewrite directory to always include the `/index.html'.
|
||||
1. The used upstream endpoint is _path-based_, that is to say has the bucketname and website name
|
||||
included. This whole location used to be simply `root /var/www/ipng-web-assets/ipng.ch/media/`
|
||||
so the mental change is quite small.
|
||||
|
||||
### NGINX: Caching
|
||||
|
||||
|
||||
After deploying the S3 upstream on all IPng websites, I can delete the old
|
||||
`/var/www/ipng-web-assets/` directory and reclaim about 7GB of diskspace. This gives me an idea ...
|
||||
|
||||
{{< image width="8em" float="left" src="/assets/shared/brain.png" alt="brain" >}}
|
||||
|
||||
On the one hand it's great that I will pull these assets from Minio and all, but at the same time,
|
||||
it's a tad inefficient to retrieve them from, say, Zurich to Amsterdam just to serve them onto the
|
||||
internet again. If at any time something on the IPng website goes viral, it'd be nice to be able to
|
||||
serve them directly from the edge, right?
|
||||
|
||||
A webcache. What could _possibly_ go wrong :)
|
||||
|
||||
NGINX is really really good at caching content. It has a powerful engine to store, scan, revalidate
|
||||
and match any content and upstream headers. It's also very well documented, so I take a look at the
|
||||
proxy module's documentation [[here](https://nginx.org/en/docs/http/ngx_http_proxy_module.html)] and
|
||||
in particular a useful [[blog](https://blog.nginx.org/blog/nginx-caching-guide)] on their website.
|
||||
|
||||
The first thing I need to do is create what is called a _key zone_, which is a region of memory in
|
||||
which URL keys are stored with some metadata. Having a copy of the keys in memory enables NGINX to
|
||||
quickly determine if a request is a HIT or a MISS without having to go to disk, greatly speeding up
|
||||
the check.
|
||||
|
||||
In `/etc/nginx/conf.d/ipng-cache.conf` I add the following NGINX cache:
|
||||
|
||||
```
|
||||
proxy_cache_path /var/www/nginx-cache levels=1:2 keys_zone=ipng_cache:10m max_size=8g
|
||||
inactive=24h use_temp_path=off;
|
||||
```
|
||||
|
||||
With this statement, I'll create a 2-level subdirectory, and allocate 10MB of space, which should
|
||||
hold on the order of 100K entries. The maximum size I'll allow the cache to grow to is 8GB, and I'll
|
||||
mark any object inactive if it's not been referenced for 24 hours. I learn that inactive is
|
||||
different to expired content. If a cache element has expired, but NGINX can't reach the upstream
|
||||
for a new copy, it can be configured to serve a inactive (stale) copy from the cache. That's dope,
|
||||
as it serves as an extra layer of defence in case the network or all available S3 replicas take the
|
||||
day off. I'll ask NGINX to avoid writing objects first to a tmp directory and them moving them into
|
||||
the `/var/www/nginx-cache` directory. These are recommendations I grab from the manual.
|
||||
|
||||
Within the `location` block I configured above, I'm now ready to enable this cache. I'll do that by
|
||||
adding two include files, which I'll reference in all sites that I want to have make use of this
|
||||
cache:
|
||||
|
||||
First, to enable the cache, I write the following snippet:
|
||||
```
|
||||
pim@nginx0-nlams1:~$ cat /etc/nginx/conf.d/ipng-cache.inc
|
||||
proxy_cache ipng_cache;
|
||||
proxy_ignore_headers Cache-Control;
|
||||
proxy_cache_valid any 1h;
|
||||
proxy_cache_revalidate on;
|
||||
proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
|
||||
proxy_cache_background_update on;
|
||||
```
|
||||
|
||||
Then, I find it useful to emit a few debugging HTTP headers, and at the same time I see that Minio
|
||||
emits a bunch of HTTP headers that may not be safe for me to propagate, so I pen two more snippets:
|
||||
|
||||
```
|
||||
pim@nginx0-nlams1:~$ cat /etc/nginx/conf.d/ipng-strip-minio-headers.inc
|
||||
proxy_hide_header x-minio-deployment-id;
|
||||
proxy_hide_header x-amz-request-id;
|
||||
proxy_hide_header x-amz-id-2;
|
||||
proxy_hide_header x-amz-replication-status;
|
||||
proxy_hide_header x-amz-version-id;
|
||||
|
||||
pim@nginx0-nlams1:~$ cat /etc/nginx/conf.d/ipng-add-upstream-headers.inc
|
||||
add_header X-IPng-Frontend $hostname always;
|
||||
add_header X-IPng-Upstream $upstream_addr always;
|
||||
add_header X-IPng-Upstream-Status $upstream_status always;
|
||||
add_header X-IPng-Cache-Status $upstream_cache_status;
|
||||
```
|
||||
|
||||
With that, I am ready to enable caching of the IPng `/media` location:
|
||||
|
||||
```
|
||||
location /media {
|
||||
...
|
||||
include /etc/nginx/conf.d/ipng-strip-minio-headers.inc;
|
||||
include /etc/nginx/conf.d/ipng-add-upstream-headers.inc;
|
||||
include /etc/nginx/conf.d/ipng-cache.inc;
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
## Results
|
||||
|
||||
I run the Ansible playbook for the NGINX cluster and take a look at the replica at Coloclue in
|
||||
Amsterdam, called `nginx0.nlams1.ipng.ch`. Notably, it'll have to retrieve the file from a MinIO
|
||||
replica in Zurich (12ms away), so it's expected to take a little while.
|
||||
|
||||
The first attempt:
|
||||
|
||||
```
|
||||
pim@nginx0-nlams1:~$ curl -v -o /dev/null --connect-to ipng.ch:443:localhost:443 \
|
||||
https://ipng.ch/media/vpp-proto/vpp-proto-bookworm.qcow2.lrz
|
||||
...
|
||||
< last-modified: Sun, 01 Jun 2025 12:37:52 GMT
|
||||
< x-ipng-frontend: nginx0-nlams1
|
||||
< x-ipng-cache-status: MISS
|
||||
< x-ipng-upstream: [2001:678:d78:503::b]:9000
|
||||
< x-ipng-upstream-status: 200
|
||||
|
||||
100 711M 100 711M 0 0 26.2M 0 0:00:27 0:00:27 --:--:-- 26.6M
|
||||
```
|
||||
|
||||
|
||||
OK, that's respectable, I've read the file at 26MB/s. Of course I just turned on the cache, so the
|
||||
NGINX fetches the file from Zurich while handing it over to my `curl` here. It notifies me by means
|
||||
of a HTTP header that the cache was a `MISS`, and then which upstream server it contacted to
|
||||
retrieve the object.
|
||||
|
||||
But look at what happens the _second_ time I run the same command:
|
||||
|
||||
```
|
||||
pim@nginx0-nlams1:~$ curl -v -o /dev/null --connect-to ipng.ch:443:localhost:443 \
|
||||
https://ipng.ch/media/vpp-proto/vpp-proto-bookworm.qcow2.lrz
|
||||
< last-modified: Sun, 01 Jun 2025 12:37:52 GMT
|
||||
< x-ipng-frontend: nginx0-nlams1
|
||||
< x-ipng-cache-status: HIT
|
||||
|
||||
100 711M 100 711M 0 0 436M 0 0:00:01 0:00:01 --:--:-- 437M
|
||||
```
|
||||
|
||||
|
||||
Holy moly! First I see the object has the same _Last-Modified_ header, but I now also see that the
|
||||
_Cache-Status_ was a `HIT`, and there is no mention of any upstream server. I do however see the
|
||||
file come in at a whopping 437MB/s which is 16x faster than over the network!! Nice work, NGINX!
|
||||
|
||||
{{< image float="right" src="/assets/minio/rack-2.png" alt="Rack-o-Minio" width="12em" >}}
|
||||
|
||||
# What's Next
|
||||
|
||||
I'm going to deploy the third MinIO replica in Rümlang once the disks arrive. I'll release the
|
||||
~4TB of disk used currently in Restic backups for the fleet, and put that ZFS capacity to other use.
|
||||
Now, creating services like PeerTube, Mastodon, Pixelfed, Loops, NextCloud and what-have-you, will
|
||||
become much easier for me. And with the per-bucket replication between MinIO deployments, I also
|
||||
think this is a great way to auto-backup important data. First off, it'll be RS8.4 on the MinIO node
|
||||
itself, and secondly, user data will be copied automatically to a neighboring facility.
|
||||
|
||||
I've convinced myself that S3 storage is a great service to operate, and that MinIO is awesome.
|
BIN
static/assets/minio/nginx-logo.png
(Stored with Git LFS)
Normal file
BIN
static/assets/minio/nginx-logo.png
(Stored with Git LFS)
Normal file
Binary file not shown.
BIN
static/assets/minio/rack-2.png
(Stored with Git LFS)
Normal file
BIN
static/assets/minio/rack-2.png
(Stored with Git LFS)
Normal file
Binary file not shown.
BIN
static/assets/minio/restic-logo.png
(Stored with Git LFS)
Normal file
BIN
static/assets/minio/restic-logo.png
(Stored with Git LFS)
Normal file
Binary file not shown.
Reference in New Issue
Block a user