---
date: "2025-06-01T10:07:23Z"
title: 'Case Study: Minio S3 - Part 2'
---

{{< image float="right" src="/assets/minio/minio-logo.png" alt="MinIO Logo" width="6em" >}}

# Introduction

Amazon Simple Storage Service (Amazon S3) is an object storage service offering industry-leading
scalability, data availability, security, and performance. Millions of customers of all sizes and
industries store, manage, analyze, and protect any amount of data for virtually any use case, such
as data lakes, cloud-native applications, and mobile apps. With cost-effective storage classes and
easy-to-use management features, you can optimize costs, organize and analyze data, and configure
fine-tuned access controls to meet specific business and compliance requirements. 

Amazon's S3 became the _de facto_ standard object storage system, and there exist several fully open
source implementations of the protocol. One of them is MinIO: designed to allow enterprises to
consolidate all of their data on a single, private cloud namespace. Architected using the same
principles as the hyperscalers, AIStor delivers performance at scale at a fraction of the cost
compared to the public cloud.

IPng Networks is an Internet Service Provider, but I also dabble in self-hosting things, for
example [[PeerTube](https://video.ipng.ch/)], [[Mastodon](https://ublog.tech/)],
[[Immich](https://photos.ipng.ch/)], [[Pixelfed](https://pix.ublog.tech/)] and of course
[[Hugo](https://ipng.ch/)]. These services all have one thing in common: they tend to use lots of
storage when they grow. At IPng Networks, all hypervisors ship with enterprise SAS flash drives,
mostly 1.92TB and 3.84TB. Scaling up each of these services, and backing them up safely, can be
quite the headache.

In a [[previous article]({{< ref 2025-05-28-minio-1 >}})], I talked through the install of a
redundant set of three Minio machines. In this article, I'll start putting them to good use.

## Use Case: Restic

{{< image float="right" src="/assets/minio/restic-logo.png" alt="Restic Logo" width="12em" >}}

[[Restic](https://restic.org/)] is a modern backup program that can back up your files from multiple
host OS, to many different storage types, easily, effectively, securely, verifiably and freely. With
a sales pitch like that, what's not to love? Actually, I am a long-time
[[BorgBackup](https://www.borgbackup.org/)] user, and I think I'll keep that running. However, for
resilience, and because I've heard only good things about Restic, I'll make a second backup of the
routers, hypervisors, and virtual machines using Restic.

Restic can use S3 buckets out of the box (incidentally, so can BorgBackup). To configure it, I use
a mixture of environment variables and flags. But first, let me create a bucket for the backups.

```
pim@glootie:~$ mc mb chbtl0/ipng-restic
pim@glootie:~$ mc admin user add chbtl0/ <key> <secret>
pim@glootie:~$ cat << EOF | tee ipng-restic-access.json
{
 "PolicyName": "ipng-restic-access",
 "Policy": {
  "Version": "2012-10-17",
  "Statement": [
   {
    "Effect": "Allow",
    "Action": [ "s3:DeleteObject", "s3:GetObject", "s3:ListBucket", "s3:PutObject" ],
    "Resource": [ "arn:aws:s3:::ipng-restic", "arn:aws:s3:::ipng-restic/*" ]
   }
  ]
 },
}
EOF
pim@glootie:~$ mc admin policy create chbtl0/ ipng-restic-access.json
pim@glootie:~$ mc admin policy attach chbtl0/ ipng-restic-access --user <key>
```

First, I'll create a bucket called `ipng-restic`. Then, I'll create a _user_ with a given secret
_key_. To protect the innocent, and my backups, I'll not disclose them. Next, I'll create an
IAM policy that allows for Get/List/Put/Delete to be performed on the bucket and its contents, and
finally I'll attach this policy to the user I just created.

To run a Restic backup, I'll first have to create a so-called _repository_. The repository has a
location and a password, which Restic uses to encrypt the data. Because I'm using S3, I'll also need
to specify the key and secret:

```
root@glootie:~# RESTIC_PASSWORD="changeme"
root@glootie:~# RESTIC_REPOSITORY="s3:https://s3.chbtl0.ipng.ch/ipng-restic/$(hostname)/"
root@glootie:~# AWS_ACCESS_KEY_ID="<key>"
root@glootie:~# AWS_SECRET_ACCESS_KEY:="<secret>"
root@glootie:~# export RESTIC_PASSWORD RESTIC_REPOSITORY AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY
root@glootie:~# restic init
created restic repository 807cf25e85 at s3:https://s3.chbtl0.ipng.ch/ipng-restic/glootie.ipng.ch/
```

Restic prints out some repository finterprint of the latest 'snapshot' it just created. Taking a
look on the MinIO install:

```
pim@glootie:~$ mc stat chbtl0/ipng-restic/glootie.ipng.ch/
Name      : config
Date      : 2025-06-01 12:01:43 UTC 
Size      : 155 B  
ETag      : 661a43f72c43080649712e45da14da3a 
Type      : file 
Metadata  :
  Content-Type: application/octet-stream 

Name      : keys/
Date      : 2025-06-01 12:03:33 UTC 
Type      : folder 
```

Cool. Now I'm ready to make my first full backup:

```
root@glootie:~# ARGS="--exclude /proc --exclude /sys --exclude /dev --exclude /run"
root@glootie:~# ARGS="$ARGS --exclude-if-present .nobackup"
root@glootie:~# restic backup $ARGS /
...
processed 1141426 files, 131.111 GiB in 15:12
snapshot 34476c74 saved
```

Once the backup completes, the Restic authors advise me to also do a check of the repository, and to
prune it so that it keeps a finite amount of daily, weekly and monthly backups. My further journey
for Restic looks a bit like this:

```
root@glootie:~# restic check
using temporary cache in /tmp/restic-check-cache-2712250731
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:04] 100.00%  1 / 1 snapshots

no errors were found

root@glootie:~# restic forget --prune --keep-daily 8 --keep-weekly 5 --keep-monthly 6
repository 34476c74 opened (version 2, compression level auto)
Applying Policy: keep 8 daily, 5 weekly, 6 monthly snapshots
keep 1 snapshots:
ID        Time                 Host           Tags        Reasons           Paths
---------------------------------------------------------------------------------
34476c74  2025-06-01 12:18:54  glootie.ipng.ch            daily snapshot    /
                                                          weekly snapshot
                                                          monthly snapshot
----------------------------------------------------------------------------------
1 snapshots
```

Right on! I proceed to update the Ansible configs at IPng to roll this out against the entire fleet
of 152 hosts at IPng Networks. I do this in a little tool called `bitcron`, which I wrote for a
previous company I worked at: [[BIT](https://bit.nl)] in the Netherlands. Bitcron allows me to
create relatively elegant cronjobs that can raise warnings, errors and fatal issues. If no issues
are found, an e-mail can be sent to a bitbucket address, but if warnings or errors are found, a
different _monitored_ address will be used. Bitcron is kind of cool, and I wrote it in 2001.  Maybe
I'll write about it, for old time's sake. I wonder if the folks at BIT still use it?

## Use Case: NGINX

{{< image float="right" src="/assets/minio/nginx-logo.png" alt="NGINX Logo" width="11em" >}}

OK, with the first use case out of the way, I turn my attention to a second - in my opinion more
interesting - use case. In the [[previous article]({{< ref 2025-05-28-minio-1 >}})], I created a
public bucket called `ipng-web-assets` in which I stored 6.50GB of website data belonging to the
IPng website, and some material I posted when I was on my
[[Sabbatical](https://sabbatical.ipng.nl/)] last year.

### MinIO: Bucket Replication

First things first: redundancy. These web assets are currently pushed to all four nginx machines,
and statically served. If I were to replace them with a single S3 bucket, I would create a single
point of failure, and that's _no bueno_!

Off I go, creating a replicated bucket using two MinIO instances (`chbtl0` and `ddln0`):

```
pim@glootie:~$ mc mb ddln0/ipng-web-assets
pim@glootie:~$ mc anonymous set download ddln0/ipng-web-assets
pim@glootie:~$ mc admin user add ddln0/ <replkey> <replsecret>
pim@glootie:~$ cat << EOF | tee ipng-web-assets-access.json
{
 "PolicyName": "ipng-web-assets-access",
 "Policy": {
  "Version": "2012-10-17",
  "Statement": [
   {
    "Effect": "Allow",
    "Action": [ "s3:DeleteObject", "s3:GetObject", "s3:ListBucket", "s3:PutObject" ],
    "Resource": [ "arn:aws:s3:::ipng-web-assets", "arn:aws:s3:::ipng-web-assets/*" ]
   }
  ]
 },
}
EOF
pim@glootie:~$ mc admin policy create ddln0/ ipng-web-assets-access.json
pim@glootie:~$ mc admin policy attach ddln0/ ipng-web-assets-access --user <replkey>
pim@glootie:~$ mc replicate add chbtl0/ipng-web-assets \
                  --remote-bucket https://<key>:<secret>@s3.ddln0.ipng.ch/ipng-web-assets
```

What happens next is pure magic. I've told `chbtl0` that I want it to replicate all existing and
future changes to that bucket to its neighbor `ddln0`. Only minutes later, I check the replication
status, just to see that it's _already done_:

```
pim@glootie:~$ mc replicate status chbtl0/ipng-web-assets
  Replication status since 1 hour                                                                     
  s3.ddln0.ipng.ch
  Replicated:                   142 objects (6.5 GiB)                                             
  Queued:                       ● 0 objects, 0 B (avg: 4 objects, 915 MiB ; max: 0 objects, 0 B)  
  Workers:                      0 (avg: 0; max: 0)                                                
  Transfer Rate:                15 kB/s (avg: 88 MB/s; max: 719 MB/s                              
  Latency:                      3ms (avg: 3ms; max: 7ms)                                          
  Link:                         ● online (total downtime: 0 milliseconds)                         
  Errors:                       0 in last 1 minute; 0 in last 1hr; 0 since uptime                 
  Configured Max Bandwidth (Bps): 644 GB/s   Current Bandwidth (Bps): 975 B/s                     
pim@summer:~/src/ipng-web-assets$ mc ls ddln0/ipng-web-assets/
[2025-06-01 12:42:22 CEST]     0B ipng.ch/
[2025-06-01 12:42:22 CEST]     0B sabbatical.ipng.nl/
```

MinIO has pumped the data from bucket `ipng-web-assets` to the other machine at an average of 88MB/s
with a peak throughput of 719MB/s (probably for the larger VM images). And indeed, looking at the
remote machine, it is fully caught up after the push, within only a minute or so with a completely
fresh copy. Nice!

### MinIO: Missing directory index

I take a look at what I just built, on the following URL:
*   [https://ipng-web-assets.s3.ddln0.ipng.ch/sabbatical.ipng.nl/media/vdo/IMG_0406_0.mp4](https://ipng-web-assets.s3.ddln0.ipng.ch/sabbatical.ipng.nl/media/vdo/IMG_0406_0.mp4)

That checks out, and I can see the mess that was my room when I first went on sabbatical. By the
way, I totally cleaned it up, see
[[here](https://sabbatical.ipng.nl/blog/2024/08/01/thursday-basement-done/)] for proof. I can't,
however, see the directory listing:

```
pim@glootie:~$ curl https://ipng-web-assets.s3.ddln0.ipng.ch/sabbatical.ipng.nl/media/vdo/
<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>NoSuchKey</Code>
  <Message>The specified key does not exist.</Message>
  <Key>sabbatical.ipng.nl/media/vdo/</Key>
  <BucketName>ipng-web-assets</BucketName>
  <Resource>/sabbatical.ipng.nl/media/vdo/</Resource>
  <RequestId>1844EC0CFEBF3C5F</RequestId>
  <HostId>dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8</HostId>
</Error>
```

That's unfortunate, because some of the IPng articles link to a directory full of files, which I'd
like to be shown so that my readers can navigate through the directories. Surely I'm not the first
to encounter this? And sure enough, I'm not
[[ref](https://github.com/glowinthedark/index-html-generator)] by user `glowinthedark` who wrote a
little python script that generates `index.html` files for their Caddy file server. I'll take me
some of that Python, thank you!

With the following little script, my setup is complete:

```
pim@glootie:~/src/ipng-web-assets$ cat push.sh 
#!/usr/bin/env bash

echo "Generating index.html files ..."
for D in */media; do
  echo "* Directory $D"
  ./genindex.py -r $D
done
echo "Done (genindex)"
echo ""

echo "Mirroring directoro to S3 Bucket"
mc mirror --remove --overwrite . chbtl0/ipng-web-assets/
echo "Done (mc mirror)"
echo ""
pim@glootie:~/src/ipng-web-assets$ ./push.sh 
```

Only a few seconds after I run `./push.sh`, the replication is complete and I have two identical
copies of my media:

1.   [https://ipng-web-assets.s3.chbtl0.ipng.ch/ipng.ch/media/](https://ipng-web-assets.s3.chbtl0.ipng.ch/ipng.ch/media/index.html)
1.   [https://ipng-web-assets.s3.ddln0.ipng.ch/ipng.ch/media/](https://ipng-web-assets.s3.ddln0.ipng.ch/ipng.ch/media/index.html)


### NGINX: Proxy to Minio

Before moving to S3 storage, my NGINX frontends all kept a copy of the IPng media on local NVME
disk. That's great for reliability, as each NGINX instance is completely hermetic and standalone.
However, it's not great for scaling: the current NGINX instances only have 16GB of local storage,
and I'd rather not have my static web asset data outgrow that filesystem. From before, I already had
an NGINX config that served the Hugo static data from `/var/www/ipng.ch/ and the `/media'
subdirectory from a different directory in `/var/www/ipng-web-assets/ipng.ch/media`.

Moving to redundant S3 storage backenda is straight forward:

```
upstream minio_ipng {
  least_conn;
  server minio0.chbtl0.net.ipng.ch:9000;
  server minio0.ddln0.net.ipng.ch:9000;
}

server {
  ...
  location / {
    root /var/www/ipng.ch/;
  }

  location /media {
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_connect_timeout 300;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    chunked_transfer_encoding off;

    rewrite (.*)/$ $1/index.html;

    proxy_pass http://minio_ipng/ipng-web-assets/ipng.ch/media;
  }
}
```

I want to make note of a few things:
1.   The `upstream` definition here uses IPng Site Local entrypoints, considering the NGINX servers
     all have direct MTU=9000 access to the MinIO instances. I'll put both in there, in a
     round-robin configuration favoring the replica with _least connections_.
1.   Deeplinking to directory names without the trailing `/index.html` would serve a 404 from the
     backend, so I'll intercept these and rewrite directory to always include the `/index.html'.
1.   The used upstream endpoint is _path-based_, that is to say has the bucketname and website name
     included. This whole location used to be simply `root /var/www/ipng-web-assets/ipng.ch/media/`
     so the mental change is quite small.

### NGINX: Caching


After deploying the S3 upstream on all IPng websites, I can delete the old
`/var/www/ipng-web-assets/` directory and reclaim about 7GB of diskspace. This gives me an idea ...

{{< image width="8em" float="left" src="/assets/shared/brain.png" alt="brain" >}}

On the one hand it's great that I will pull these assets from Minio and all, but at the same time,
it's a tad inefficient to retrieve them from, say, Zurich to Amsterdam just to serve them onto the
internet again. If at any time something on the IPng website goes viral, it'd be nice to be able to
serve them directly from the edge, right?

A webcache. What could _possibly_ go wrong :)

NGINX is really really good at caching content. It has a powerful engine to store, scan, revalidate
and match any content and upstream headers. It's also very well documented, so I take a look at the
proxy module's documentation [[here](https://nginx.org/en/docs/http/ngx_http_proxy_module.html)] and
in particular a useful [[blog](https://blog.nginx.org/blog/nginx-caching-guide)] on their website.

The first thing I need to do is create what is called a _key zone_, which is a region of memory in
which URL keys are stored with some metadata. Having a copy of the keys in memory enables NGINX to
quickly determine if a request is a HIT or a MISS without having to go to disk, greatly speeding up
the check.

In `/etc/nginx/conf.d/ipng-cache.conf` I add the following NGINX cache:

```
proxy_cache_path /var/www/nginx-cache levels=1:2 keys_zone=ipng_cache:10m max_size=8g
                 inactive=24h use_temp_path=off;
```

With this statement, I'll create a 2-level subdirectory, and allocate 10MB of space, which should
hold on the order of 100K entries. The maximum size I'll allow the cache to grow to is 8GB, and I'll
mark any object inactive if it's not been referenced for 24 hours. I learn that inactive is
different to expired content. If a cache element has expired, but NGINX can't reach the upstream
for a new copy, it can be configured to serve a inactive (stale) copy from the cache. That's dope,
as it serves as an extra layer of defence in case the network or all available S3 replicas take the
day off. I'll ask NGINX to avoid writing objects first to a tmp directory and them moving them into
the `/var/www/nginx-cache` directory.  These are recommendations I grab from the manual.

Within the `location` block I configured above, I'm now ready to enable this cache. I'll do that by
adding two include files, which I'll reference in all sites that I want to have make use of this
cache:

First, to enable the cache, I write the following snippet:
```
pim@nginx0-nlams1:~$ cat /etc/nginx/conf.d/ipng-cache.inc
proxy_cache ipng_cache;
proxy_ignore_headers Cache-Control;
proxy_cache_valid any 1h;
proxy_cache_revalidate on;
proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
proxy_cache_background_update on;
```

Then, I find it useful to emit a few debugging HTTP headers, and at the same time I see that Minio
emits a bunch of HTTP headers that may not be safe for me to propagate, so I pen two more snippets:

```
pim@nginx0-nlams1:~$ cat /etc/nginx/conf.d/ipng-strip-minio-headers.inc 
proxy_hide_header x-minio-deployment-id;
proxy_hide_header x-amz-request-id;
proxy_hide_header x-amz-id-2;
proxy_hide_header x-amz-replication-status;
proxy_hide_header x-amz-version-id;

pim@nginx0-nlams1:~$ cat /etc/nginx/conf.d/ipng-add-upstream-headers.inc 
add_header X-IPng-Frontend $hostname always;
add_header X-IPng-Upstream $upstream_addr always;
add_header X-IPng-Upstream-Status $upstream_status always;
add_header X-IPng-Cache-Status $upstream_cache_status;
```

With that, I am ready to enable caching of the IPng `/media` location:

```
  location /media {
    ...
    include /etc/nginx/conf.d/ipng-strip-minio-headers.inc;
    include /etc/nginx/conf.d/ipng-add-upstream-headers.inc;
    include /etc/nginx/conf.d/ipng-cache.inc;
    ...
}
```

## Results

I run the Ansible playbook for the NGINX cluster and take a look at the replica at Coloclue in
Amsterdam, called `nginx0.nlams1.ipng.ch`. Notably, it'll have to retrieve the file from a MinIO
replica in Zurich (12ms away), so it's expected to take a little while.

The first attempt:

```
pim@nginx0-nlams1:~$ curl -v -o /dev/null --connect-to ipng.ch:443:localhost:443 \
                     https://ipng.ch/media/vpp-proto/vpp-proto-bookworm.qcow2.lrz
...
< last-modified: Sun, 01 Jun 2025 12:37:52 GMT
< x-ipng-frontend: nginx0-nlams1
< x-ipng-cache-status: MISS
< x-ipng-upstream: [2001:678:d78:503::b]:9000
< x-ipng-upstream-status: 200

100  711M  100  711M    0     0  26.2M      0  0:00:27  0:00:27 --:--:-- 26.6M
```


OK, that's respectable, I've read the file at 26MB/s. Of course I just turned on the cache, so the
NGINX fetches the file from Zurich while handing it over to my `curl` here. It notifies me by means
of a HTTP header that the cache was a `MISS`, and then which upstream server it contacted to
retrieve the object. 

But look at what happens the _second_ time I run the same command:

```
pim@nginx0-nlams1:~$ curl -v -o /dev/null --connect-to ipng.ch:443:localhost:443 \
                     https://ipng.ch/media/vpp-proto/vpp-proto-bookworm.qcow2.lrz
< last-modified: Sun, 01 Jun 2025 12:37:52 GMT
< x-ipng-frontend: nginx0-nlams1
< x-ipng-cache-status: HIT
 
100  711M  100  711M    0     0   436M      0  0:00:01  0:00:01 --:--:--  437M
```


Holy moly! First I see the object has the same _Last-Modified_ header, but I now also see that the
_Cache-Status_ was a `HIT`, and there is no mention of any upstream server. I do however see the
file come in at a whopping 437MB/s which is 16x faster than over the network!! Nice work, NGINX!

{{< image float="right" src="/assets/minio/rack-2.png" alt="Rack-o-Minio" width="12em" >}}

# What's Next

I'm going to deploy the third MinIO replica in R&uuml;mlang once the disks arrive. I'll release the
~4TB of disk used currently in Restic backups for the fleet, and put that ZFS capacity to other use.
Now, creating services like PeerTube, Mastodon, Pixelfed, Loops, NextCloud and what-have-you, will
become much easier for me. And with the per-bucket replication between MinIO deployments, I also
think this is a great way to auto-backup important data. First off, it'll be RS8.4 on the MinIO node
itself, and secondly, user data will be copied automatically to a neighboring facility.

I've convinced myself that S3 storage is a great service to operate, and that MinIO is awesome.
