From 5042f822ef7b077efa6d8283b17b534bca2fbca8 Mon Sep 17 00:00:00 2001 From: Pim van Pelt Date: Sun, 1 Jun 2025 12:53:16 +0200 Subject: [PATCH] Minio Article #1 --- content/articles/2025-05-28-minio-1.md | 713 +++++++++++ static/assets/minio/console-1.png | 3 + static/assets/minio/console-2.png | 3 + static/assets/minio/disks.png | 3 + static/assets/minio/minio-ec.svg | 1633 ++++++++++++++++++++++++ static/assets/minio/minio-logo.png | 3 + static/assets/minio/nagios.png | 3 + static/assets/minio/rack.png | 3 + 8 files changed, 2364 insertions(+) create mode 100644 content/articles/2025-05-28-minio-1.md create mode 100644 static/assets/minio/console-1.png create mode 100644 static/assets/minio/console-2.png create mode 100644 static/assets/minio/disks.png create mode 100644 static/assets/minio/minio-ec.svg create mode 100644 static/assets/minio/minio-logo.png create mode 100644 static/assets/minio/nagios.png create mode 100644 static/assets/minio/rack.png diff --git a/content/articles/2025-05-28-minio-1.md b/content/articles/2025-05-28-minio-1.md new file mode 100644 index 0000000..96df283 --- /dev/null +++ b/content/articles/2025-05-28-minio-1.md @@ -0,0 +1,713 @@ +--- +date: "2025-05-28T22:07:23Z" +title: 'Case Study: Minio S3 - Part 1' +--- + +{{< image float="right" src="/assets/minio/minio-logo.png" alt="MinIO Logo" width="6em" >}} + +# Introduction + +Amazon Simple Storage Service (Amazon S3) is an object storage service offering industry-leading +scalability, data availability, security, and performance. Millions of customers of all sizes and +industries store, manage, analyze, and protect any amount of data for virtually any use case, such +as data lakes, cloud-native applications, and mobile apps. With cost-effective storage classes and +easy-to-use management features, you can optimize costs, organize and analyze data, and configure +fine-tuned access controls to meet specific business and compliance requirements. + +Amazon's S3 became the _de facto_ standard object storage system, and there exist several fully open +source implementations of the protocol. One of them is MinIO: designed to allow enterprises to +consolidate all of their data on a single, private cloud namespace. Architected using the same +principles as the hyperscalers, AIStor delivers performance at scale at a fraction of the cost +compared to the public cloud. + +IPng Networks is an Internet Service Provider, but I also dabble in self-hosting things, for +example [[PeerTube](https://video.ipng.ch/)], [[Mastodon](https://ublog.tech/)], +[[Immich](https://photos.ipng.ch/)], [[Pixelfed](https://pix.ublog.tech/)] and of course +[[Hugo](https://ipng/ch/)]. These services all have one thing in common: they tend to use lots of +storage when they grow. At IPng Networks, all hypervisors ship with enterprise SAS flash drives, +mostly 1.92TB and 3.84TB. Scaling up each of these services, and backing them up safely, can be +quite the headache. + +This article is for the storage-buffs. I'll set up a set of distributed MinIO nodes from scatch. + +## Physical + +{{< image float="right" src="/assets/minio/disks.png" alt="MinIO Disks" width="16em" >}} + +I'll start with the basics. I still have a few Dell R720 servers laying around, they are getting a +bit older but still have 24 cores and 64GB of memory. First I need to get me some disks. I order +36pcs of 16TB SATA enterprise disk, a mixture of Seagate EXOS and Toshiba MG series disks. I've once +learned (the hard way), that buying a big stack of disks from one production run is a risk - so I'll +mix and match the drives. + +Three trays of caddies and a melted credit card later, I have 576TB of SATA disks safely in hand. +Each machine will carry 192TB of raw storage. The nice thing about this chassis is that Dell can +ship them with 12x 3.5" SAS slots in the front, and 2x 2.5" SAS slots in the rear of the chassis. + +So I'll install Debian Bookworm on one small 480G SSD in software RAID1. + +### Cloning an install + +I have three identical machines so in total I'll want six of these SSDs. I temporarily screw the +other five in 3.5" drive caddies and plug them into the first installed Dell, which I've called +`minio-proto`: + + +``` +pim@minio-proto:~$ for i in b c d e f; do + sudo dd if=/dev/sda of=/dev/sd${i} bs=512 count=1; + sudo mdadm --manage /dev/md0 --add /dev/md${i}1 +done +pim@minio-proto:~$ sudo mdadm --manage /dev/md0 --grow 6 +pim@minio-proto:~$ watch cat /proc/mdstat +pim@minio-proto:~$ for i in a b c d e f; do + sudo grub-install /dev/sd$i +done +``` + +{{< image float="right" src="/assets/minio/rack.png" alt="MinIO Rack" width="16em" >}} + +The first command takes my installed disk, `/dev/sda`, and copies the first sector over to the other +five. This will give them the same partition table. Next, I'll add the first partition of each disk +to the raidset. Then, I'll expand the raidset to have six members, after which the kernel starts a +recovery process that syncs the newly added paritions to `/dev/md0` (by copying from `/dev/sda` to +all other disks at once). Finally, I'll watch this exciting movie and grab a cup of tea. + + +Once the disks are fully copied, I'll shut down the machine and distribute the disks to their +respective Dell R720, two each. Once they boot they will all be identical. I'll need to make sure +their hostnames, and machine/host-id are unique, otherwise things like bridges will have overlapping +MAC addresses - ask me how I know: + +``` +pim@minio-proto:~$ sudo mdadm --manage /dev/md0 --grow -n 2 +pim@minio-proto:~$ sudo rm /etc/ssh/ssh_host* +pim@minio-proto:~$ sudo hostname minio0-chbtl0 +pim@minio-proto:~$ sudo dpkg-reconfigure openssh-server +pim@minio-proto:~$ sudo dd if=/dev/random of=/etc/hostid bs=4 count=1 +pim@minio-proto:~$ sudo /usr/bin/dbus-uuidgen > /etc/machine-id +pim@minio-proto:~$ sudo reboot +``` + +After which I have three beautiful and unique machines: +* `minio0.chbtl0.net.ipng.ch`: which will go into my server rack at the IPng office. +* `minio0.ddln0.net.ipng.ch`: which will go to [[Daedalean]({{< ref + 2022-02-24-colo >}})], doing AI since before it was all about vibe coding. +* `minio0.chrma0.net.ipng.ch`: which will go to [[IP-Max](https://ip-max.net/)], one of the best + ISPs on the planet. 🥰 + + +## Deploying Minio + +The user guide that MinIO provides +[[ref](https://min.io/docs/minio/linux/operations/installation.html)] is super good, arguably one of +the best documented open source projects I've ever seen. it shows me that I can do three types of +install. A 'Standalone' with one disk, a 'Standalone Multi-Drive', and a 'Distributed' deployment. +I decide to make three independent standalone multi-drive installs. This way, I have less shared +fate, and will be immune to network partitions (as these are going to be in three different +physical locations). I've also read about per-bucket _replication_, which will be an excellent way +to get geographical distribution and active/active instances to work together. + +I feel good about the single-machine multi-drive decision. I follow the install guide +[[ref](https://min.io/docs/minio/linux/operations/install-deploy-manage/deploy-minio-single-node-multi-drive.html#minio-snmd)] +for this deployment type. + +### IPng Frontends + +At IPng I use a private IPv4/IPv6/MPLS network that is not connected to the internet. I call this +network [[IPng Site Local]({{< ref 2023-03-11-mpls-core.md >}})]. But how will users reach my Minio +install? I have four redundantly and geographically deployed frontends, two in the Netherlands and +two in Switzerland. I've described the frontend setup in a [[previous article]({{< ref +2023-03-17-ipng-frontends >}})] and the certificate management in [[this article]({{< ref +2023-03-24-lego-dns01 >}})]. + +I've decided to run the service on these three regionalized endpoints: +1. `s3.chbtl0.ipng.ch` which will back into `minio0.chbtl0.net.ipng.ch` +1. `s3.ddln0.ipng.ch` which will back into `minio0.ddln0.net.ipng.ch` +1. `s3.chrma0.ipng.ch` which will back into `minio0.chrma0.net.ipng.ch` + +The first thing I take note of is that S3 buckets can be either addressed _by path_, in other words +something like `s3.chbtl0.ipng.ch/my-bucket/README.md`, but they can also be addressed by virtual +host, like so: `my-bucket.s3.chbtl0.ipng.ch/README.md`. A subtle difference, but from the docs I +understand that Minio needs to have control of the whole space under its main domain. + +There's a small implication to this requirement -- the Web Console that ships with MinIO (eh, well, +maybe that's going to change, more on that later), will want to have its own domain-name, so I +choose something simple: `cons0-s3.chbtl0.ipng.ch` and so on. This way, somebody might still be able +to have a bucket name called `cons0` :) + +#### Let's Encrypt Certificates + +Alright, so I will be neading nine domains into this new certificate which I'll simply call +`s3.ipng.ch`. I configure it in Ansible: + +``` +certbot: + certs: +... + s3.ipng.ch: + groups: [ 'nginx', 'minio' ] + altnames: + - 's3.chbtl0.ipng.ch' + - 'cons0-s3.chbtl0.ipng.ch' + - '*.s3.chbtl0.ipng.ch' + - 's3.ddln0.ipng.ch' + - 'cons0-s3.ddln0.ipng.ch' + - '*.s3.ddln0.ipng.ch' + - 's3.chrma0.ipng.ch' + - 'cons0-s3.chrma0.ipng.ch' + - '*.s3.chrma0.ipng.ch' +``` + +I run the `certbot` playbook and it does two things: +1. On the machines from group `nginx` and `minio`, it will ensure there exists a user `lego` with + an SSH key and write permissions to `/etc/lego/`; this is where the automation will write (and + update) the certificate keys. +1. On the `lego` machine, it'll create two files. One is the certificate requestor, and the other + is a certificate distribution script that will copy the cert to the right machine(s) when it + renews. + +On the `lego` machine, I'll run the cert request for the first time: + +``` +lego@lego:~$ bin/certbot:s3.ipng.ch +lego@lego:~$ RENEWED_LINEAGE=/home/lego/acme-dns/live/s3.ipng.ch bin/certbot-distribute +``` + +The first script asks me to add the _acme-challenge DNS entries, which I'll do, for example on the +`s3.chbtl0.ipng.ch` instance (and similar for the `ddln0` and `chrma0` ones: + +``` +$ORIGIN chbtl0.ipng.ch. +_acme-challenge.s3 CNAME 51f16fd0-8eb6-455c-b5cd-96fad12ef8fd.auth.ipng.ch. +_acme-challenge.cons0-s3 CNAME 450477b8-74c9-4b9e-bbeb-de49c3f95379.auth.ipng.ch. +s3 CNAME nginx0.ipng.ch. +*.s3 CNAME nginx0.ipng.ch. +cons0-s3 CNAME nginx0.ipng.ch. +``` + +I push and reload the `ipng.ch` zonefile with these changes after which the certificate gets +requested and a cronjob added to check for renewals. The second script will copy the newly created +cert to all three `minio` machines, and all four `nginx` machines. From now on, every 90 days, a new +cert will be automatically generated and distributed. Slick! + +#### NGINX Configs + +With the LE wildcard certs in hand, I can create an NGINX frontend for these minio deployments. + +First, a simple redirector service that punts people on port 80 to port 443: + +``` +server { + listen [::]:80; + listen 0.0.0.0:80; + + server_name cons0-s3.chbtl0.ipng.ch s3.chbtl0.ipng.ch *.s3.chbtl0.ipng.ch; + access_log /var/log/nginx/s3.chbtl0.ipng.ch-access.log; + include /etc/nginx/conf.d/ipng-headers.inc; + + location / { + return 301 https://$server_name$request_uri; + } +} +``` + +Next, the Minio API service itself which runs on port 9000, with a configuration snippet inspired by +the MinIO [[docs](https://min.io/docs/minio/linux/integrations/setup-nginx-proxy-with-minio.html)]: + +``` +server { + listen [::]:443 ssl http2; + listen 0.0.0.0:443 ssl http2; + ssl_certificate /etc/certs/s3.ipng.ch/fullchain.pem; + ssl_certificate_key /etc/certs/s3.ipng.ch/privkey.pem; + include /etc/nginx/conf.d/options-ssl-nginx.inc; + ssl_dhparam /etc/nginx/conf.d/ssl-dhparams.inc; + + server_name s3.chbtl0.ipng.ch *.s3.chbtl0.ipng.ch; + access_log /var/log/nginx/s3.chbtl0.ipng.ch-access.log upstream; + include /etc/nginx/conf.d/ipng-headers.inc; + + add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always; + + ignore_invalid_headers off; + client_max_body_size 0; + # Disable buffering + proxy_buffering off; + proxy_request_buffering off; + + location / { + proxy_set_header Host $http_host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + + proxy_connect_timeout 300; + proxy_http_version 1.1; + proxy_set_header Connection ""; + chunked_transfer_encoding off; + + proxy_pass http://minio0.chbtl0.net.ipng.ch:9000; + } +} +``` + +Finally, the Minio Console service which runs on port 9090: + +``` +include /etc/nginx/conf.d/geo-ipng-trusted.inc; + +server { + listen [::]:443 ssl http2; + listen 0.0.0.0:443 ssl http2; + ssl_certificate /etc/certs/s3.ipng.ch/fullchain.pem; + ssl_certificate_key /etc/certs/s3.ipng.ch/privkey.pem; + include /etc/nginx/conf.d/options-ssl-nginx.inc; + ssl_dhparam /etc/nginx/conf.d/ssl-dhparams.inc; + + server_name cons0-s3.chbtl0.ipng.ch; + access_log /var/log/nginx/cons0-s3.chbtl0.ipng.ch-access.log upstream; + include /etc/nginx/conf.d/ipng-headers.inc; + + add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always; + + ignore_invalid_headers off; + client_max_body_size 0; + # Disable buffering + proxy_buffering off; + proxy_request_buffering off; + + location / { + if ($geo_ipng_trusted = 0) { rewrite ^ https://ipng.ch/ break; } + proxy_set_header Host $http_host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-NginX-Proxy true; + + real_ip_header X-Real-IP; + proxy_connect_timeout 300; + chunked_transfer_encoding off; + + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + + proxy_pass http://minio0.chbtl0.net.ipng.ch:9090; + } +} +``` + +This last one has an NGINX trick. It will only allow users in if they are in the map called +`geo_ipng_trusted`, which contains a set of IPv4 and IPv6 prefixes. Visitors who are not in this map +will receive an HTTP redirect back to the [[IPng.ch](https://ipng.ch/)] homepage instead. + +I run the Ansible Playbook which contains the NGINX changes to all frontends, but of course nothing +runs yet, because I haven't yet started MinIO backends. + +### MinIO Backends + +The first thing I need to do is get those disks mounted. MinIO likes using XFS, so I'll install that +and prepare the disks as follows: + +``` +pim@minio0-chbtl0:~$ sudo apt install xfsprogs +pim@minio0-chbtl0:~$ sudo modprobe xfs +pim@minio0-chbtl0:~$ echo xfs | sudo tee -a /etc/modules +pim@minio0-chbtl0:~$ sudo update-initramfs -k all -u +pim@minio0-chbtl0:~$ for i in a b c d e f g h i j k l; do sudo mkfs.xfs /dev/sd$i; done +pim@minio0-chbtl0:~$ blkid | awk 'BEGIN {i=1} /TYPE="xfs"/ { + printf "%s /minio/disk%d xfs defaults 0 2\n",$2,i; i++; + }' | sudo tee -a /etc/fstab +pim@minio0-chbtl0:~$ for i in `seq 1 12`; do sudo mkdir -p /minio/disk$i; done +pim@minio0-chbtl0:~$ sudo mount -t xfs -a +pim@minio0-chbtl0:~$ sudo chown -R minio-user: /minio/ +``` + +From the top: I'll install `xfsprogs` which contains the things I need to manipulate XFS filesystems +in Debian. Then I'll install the `xfs` kernel module, and make sure it gets inserted upon subsequent +startup by adding it to `/etc/modules` and regenerating the initrd for the installed kernels. + +Next, I'll format all twelve 16TB disks (which are `/dev/sda` - `/dev/sdl` on these machines), and +add their resulting blockdevice id's to `/etc/fstab` so they get persistently mounted on reboot. + +Finally, I'll create their mountpoints, mount all XFS filesystems, and chown them to the user that +MinIO is running as. End result: + +``` +pim@minio0-chbtl0:~$ df -T +Filesystem Type 1K-blocks Used Available Use% Mounted on +udev devtmpfs 32950856 0 32950856 0% /dev +tmpfs tmpfs 6595340 1508 6593832 1% /run +/dev/md0 ext4 114695308 5423976 103398948 5% / +tmpfs tmpfs 32976680 0 32976680 0% /dev/shm +tmpfs tmpfs 5120 4 5116 1% /run/lock +/dev/sda xfs 15623792640 121505936 15502286704 1% /minio/disk1 +/dev/sde xfs 15623792640 121505968 15502286672 1% /minio/disk12 +/dev/sdi xfs 15623792640 121505968 15502286672 1% /minio/disk11 +/dev/sdl xfs 15623792640 121505904 15502286736 1% /minio/disk10 +/dev/sdd xfs 15623792640 121505936 15502286704 1% /minio/disk4 +/dev/sdb xfs 15623792640 121505968 15502286672 1% /minio/disk3 +/dev/sdk xfs 15623792640 121505936 15502286704 1% /minio/disk5 +/dev/sdc xfs 15623792640 121505936 15502286704 1% /minio/disk9 +/dev/sdf xfs 15623792640 121506000 15502286640 1% /minio/disk2 +/dev/sdj xfs 15623792640 121505968 15502286672 1% /minio/disk7 +/dev/sdg xfs 15623792640 121506000 15502286640 1% /minio/disk8 +/dev/sdh xfs 15623792640 121505968 15502286672 1% /minio/disk6 +tmpfs tmpfs 6595336 0 6595336 0% /run/user/0 +``` + +MinIO likes to be configured using environment variables - and this is likely because it's a popupar +thing to run in a containerized environment like Kubernetes. The maintainers ship it also as a +Debian package, which will read its environment from `/etc/default/minio`, and I'll prepare that +file as follows: + +``` +pim@minio0-chbtl0:~$ cat << EOF | sudo tee /etc/default/minio +MINIO_DOMAIN="s3.chbtl0.ipng.ch,minio0.chbtl0.net.ipng.ch" +MINIO_ROOT_USER="XXX" +MINIO_ROOT_PASSWORD="YYY" +MINIO_VOLUMES="/minio/disk{1...12}" +MINIO_OPTS="--console-address :9001" +EOF +pim@minio0-chbtl0:~$ sudo systemctl enable --now minio +pim@minio0-chbtl0:~$ sudo journalctl -u minio +May 31 10:44:11 minio0-chbtl0 minio[690420]: MinIO Object Storage Server +May 31 10:44:11 minio0-chbtl0 minio[690420]: Copyright: 2015-2025 MinIO, Inc. +May 31 10:44:11 minio0-chbtl0 minio[690420]: License: GNU AGPLv3 - https://www.gnu.org/licenses/agpl-3.0.html +May 31 10:44:11 minio0-chbtl0 minio[690420]: Version: RELEASE.2025-05-24T17-08-30Z (go1.24.3 linux/amd64) +May 31 10:44:11 minio0-chbtl0 minio[690420]: API: http://198.19.4.11:9000 http://127.0.0.1:9000 +May 31 10:44:11 minio0-chbtl0 minio[690420]: WebUI: https://cons0-s3.chbtl0.ipng.ch/ +May 31 10:44:11 minio0-chbtl0 minio[690420]: Docs: https://docs.min.io + +pim@minio0-chbtl0:~$ sudo ipmitool sensor | grep Watts +Pwr Consumption | 154.000 | Watts +``` + +Incidentally - I am pretty pleased with this 192TB disk tank, sporting 24 cores, 64GB memory and +2x10G network, casually hanging out at 154 Watts of power all up. Slick! + +{{< image float="right" src="/assets/minio/minio-ec.svg" alt="MinIO Erasure Coding" width="22em" >}} + +MinIO implements _erasure coding_ as a core component in providing availability and resiliency +during drive or node-level failure events. MinIO partitions each object into data and parity shards +and distributes those shards across a single so-called _erasure set_. Under the hood, it uses +[[Reed-Solomon](https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_correction)] erasure coding +implementation and partitions the object for distribution. From the MinIO website, I'll borrow a +diagram to show how it looks like on a single node like mine to the right. + +Anyway, MinIO detects 12 disks and installs an erasure set with 8 data disks and 4 parity disks, +which it calls `EC:4` encoding, also known in the industry as `RS8.4`. +Just like that, the thing shoots to life. Awesome! + +### MinIO Client + +On Summer, I'll install the MinIO Client called `mc`. This is easy because the maintainers ship a +Linux binary which I can just download. On OpenBSD, they don't do that. Not a problem though, on +Squanchy, Pencilvester and Glootie, I will just `go install` the client. Using the `mc` commandline, +I can all any of the S3 APIs on my new MinIO instance: + +``` +pim@summer:~$ set +o history +pim@summer:~$ mc alias set chbtl0 https://s3.chbtl0.ipng.ch/ +pim@summer:~$ set -o history +pim@summer:~$ mc admin info chbtl0/ +● s3.chbtl0.ipng.ch + Uptime: 22 hours + Version: 2025-05-24T17:08:30Z + Network: 1/1 OK + Drives: 12/12 OK + Pool: 1 + +┌──────┬───────────────────────┬─────────────────────┬──────────────┐ +│ Pool │ Drives Usage │ Erasure stripe size │ Erasure sets │ +│ 1st │ 0.8% (total: 116 TiB) │ 12 │ 1 │ +└──────┴───────────────────────┴─────────────────────┴──────────────┘ + +95 GiB Used, 5 Buckets, 5,859 Objects, 318 Versions, 1 Delete Marker +12 drives online, 0 drives offline, EC:4 + +``` + +Cool beans. I think I should get rid of this root account though, I've installed those credentials +into the `/etc/default/minio` environment file, but I don't want to keep them out in the open. So +I'll make an account for myself and assign me reasonable privileges, called `consoleAdmin` in the +default install: + +``` +pim@summer:~$ set +o history +pim@summer:~$ mc admin user add chbtl0/ +pim@summer:~$ mc admin policy info chbtl0 consoleAdmin +pim@summer:~$ mc admin policy attach chbtl0 consoleAdmin --user= +pim@summer:~$ mc alias set chbtl0 https://s3.chbtl0.ipng.ch/ +pim@summer:~$ set -o history +``` + +OK, I feel less gross now that I'm not operating as root on the MinIO deployment. Using my new +user-powers, let me set some metadata on my new minio server: + +``` +pim@summer:~$ mc admin config set chbtl0/ site name=chbtl0 region=switzerland +Successfully applied new settings. +Please restart your server 'mc admin service restart chbtl0/'. +pim@summer:~$ mc admin service restart chbtl0/ +Service status: ▰▰▱ [DONE] +Summary: + ┌───────────────┬─────────────────────────────┐ + │ Servers: │ 1 online, 0 offline, 0 hung │ + │ Restart Time: │ 61.322886ms │ + └───────────────┴─────────────────────────────┘ +pim@summer:~$ mc admin config get chbtl0/ site +site name=chbtl0 region=switzerland +``` + +By the way, what's really cool about these open standards is that both the Amazon `aws` client works +with MinIO, but `mc` also works with AWS! +### MinIO Console + +Although I'm pretty good with APIs and command line tools, there's some benefit also in using a +Graphical User Interface. MinIO ships with one, but there as a bit of a kerfuffle in the MinIO +community. Unfortunately, these are pretty common -- Redis (an open source key/value storage system) +changed their offering abruptly. Terraform (an open source infrastructure-as-code tool) changed +their licensing at some point. Ansible (an open source machine management tool) changed their +offering also. MinIO developers decided to strip their console of ~all features recently. The gnarly +bits are discussed on +[[reddit](https://www.reddit.com/r/selfhosted/comments/1kva3pw/avoid_minio_developers_introduce_trojan_horse/)]. +but suffice to say: the same thing that happened in literally 100% of the other cases, also happened +here. Somebody decided to simply fork the code from before it was changed. + +Enter OpenMaxIO. A cringe worthy name, but it gets the job done. Reading up on the +[[GitHub](https://github.com/OpenMaxIO/openmaxio-object-browser/issues/5)], reviving the fully +working console is pretty straight forward -- that is, once somebody spent a few days figuring it +out. Thank you `icesvz` for this excellent pointer. With this, I can create a systemd service for +the console and start it: + +``` +pim@minio0-chbtl0:~$ cat << EOF | sudo tee -a /etc/default/minio +## NOTE(pim): For openmaxio console service +CONSOLE_MINIO_SERVER="http://localhost:9000" +MINIO_BROWSER_REDIRECT_URL="https://cons0-s3.chbtl0.ipng.ch/" +EOF +pim@minio0-chbtl0:~$ cat << EOF | sudo tee /lib/systemd/system/minio-console.service +[Unit] +Description=OpenMaxIO Console Service +Wants=network-online.target +After=network-online.target +AssertFileIsExecutable=/usr/local/bin/minio-console + +[Service] +Type=simple + +WorkingDirectory=/usr/local + +User=minio-user +Group=minio-user +ProtectProc=invisible + +EnvironmentFile=-/etc/default/minio +ExecStart=/usr/local/bin/minio-console server +Restart=always +LimitNOFILE=1048576 +MemoryAccounting=no +TasksMax=infinity +TimeoutSec=infinity +OOMScoreAdjust=-1000 +SendSIGKILL=no + +[Install] +WantedBy=multi-user.target +EOF +pim@minio0-chbtl0:~$ sudo systemctl enable --now minio-console +pim@minio0-chbtl0:~$ sudo systemctl restart minio +``` + +The first snippet is an update to the MinIO configuration that instructs it to redirect users who +are not trying to use the API to the console endpoint on `cons0-s3.chbtl0.ipng.ch`, and then the +console-server needs to know where to find the API, which from its vantage point is running on +`localhost:9000`. Hello, beautiful fully featured console: + +{{< image src="/assets/minio/console-1.png" alt="MinIO Console" >}} + +### MinIO Prometheus + +MinIO ships with a prometheus metrics endpoint, and I notice on its console that it has a nice +metrics tab, which is fully greyed out. This is most likely because, well, I don't have a Prometheus +install here yet. I decide to keep the storage nodes self-contained and start a Prometheus server on +the local machine. I can always plumb that to IPng's Grafana instance later. + +For now, I'll install Prometheus as follows: + +``` +pim@minio0-chbtl0:~$ cat << EOF | sudo tee -a /etc/default/minio +## NOTE(pim): Metrics for minio-console +MINIO_PROMETHEUS_AUTH_TYPE="public" +CONSOLE_PROMETHEUS_URL="http://localhost:19090/" +CONSOLE_PROMETHEUS_JOB_ID="minio-job" +EOF + +pim@minio0-chbtl0:~$ sudo apt install prometheus +pim@minio0-chbtl0:~$ cat << EOF | sudo tee /etc/default/prometheus +ARGS="--web.listen-address='[::]:19090' --storage.tsdb.retention.size=16GB" +EOF +pim@minio0-chbtl0:~$ cat << EOF | sudo tee /etc/prometheus/prometheus.yml +global: + scrape_interval: 60s + +scrape_configs: + - job_name: minio-job + metrics_path: /minio/v2/metrics/cluster + static_configs: + - targets: ['localhost:9000'] + labels: + cluster: minio0-chbtl0 + + - job_name: minio-job-node + metrics_path: /minio/v2/metrics/node + static_configs: + - targets: ['localhost:9000'] + labels: + cluster: minio0-chbtl0 + + - job_name: minio-job-bucket + metrics_path: /minio/v2/metrics/bucket + static_configs: + - targets: ['localhost:9000'] + labels: + cluster: minio0-chbtl0 + + - job_name: minio-job-resource + metrics_path: /minio/v2/metrics/resource + static_configs: + - targets: ['localhost:9000'] + labels: + cluster: minio0-chbtl0 + + - job_name: node + static_configs: + - targets: ['localhost:9100'] + labels: + cluster: minio0-chbtl0 +pim@minio0-chbtl0:~$ sudo systemctl restart minio prometheus +``` + +In the first snippet, I'll tell MinIO where it should find its Prometheus instance. Since the MinIO +console service is running on port 9090, and this is also the default port for Prometheus, I will +run Promtheus on port 19090 instead. From reading the MinIO docs, I can see that normally MinIO will +want prometheus to authenticate to it before it'll allow the endpoints to be scraped. I'll turn that +off by making these public. On the IPng Frontends, I can always remove access to /minio/v2 and +simply use the IPng Site Local access for local Prometheus scrapers instead. + +After telling Prometheus its runtime arguments (in `/etc/default/prometheus`) and its scraping +endpoints (in `/etc/prometheus/prometheus.yml`), I can restart minio and prometheus. A few minutes +later, I can see the _Metrics_ tab in the console come to life. + +But now that I have this prometheus running on the MinIO node, I can also add it to IPng's Grafana +configuration, by adding a new data source on `minio0.chbtl0.net.ipng.ch:19090` and pointing the +default Grafana [[Dashboard](https://grafana.com/grafana/dashboards/13502-minio-dashboard/)] at it: + +{{< image src="/assets/minio/console-2.png" alt="Grafana Dashboard" >}} + +A two-for-one: I will both be able to see metrics directly in the console, but also I will be able +to hook up these per-node prometheus instances into IPng's alertmanager also, and I've read some +[[docs](https://min.io/docs/minio/linux/operations/monitoring/collect-minio-metrics-using-prometheus.html)] +on the concepts. I'm really liking the experience so far! + +### MinIO Nagios + +Prometheus is fancy and all, but at IPng Networks, I've been doing monitoring for a while now. As a +dinosaur, I still have an active [[Nagios](https://www.nagios.org/)] install, which autogenerates +all of its configuration using the Ansible repository I have. So for the new Ansible group called +`minio`, I will autogenerate the following snippet: + +``` +define command { + command_name ipng_check_minio + command_line $USER1$/check_http -E -H $HOSTALIAS$ -I $ARG1$ -p $ARG2$ -u $ARG3$ -r '$ARG4$' +} + +define service { + hostgroup_name ipng:minio:ipv6 + service_description minio6:api + check_command ipng_check_minio!$_HOSTADDRESS6$!9000!/minio/health/cluster! + use ipng-service-fast + notification_interval 0 ; set > 0 if you want to be renotified +} + +define service { + hostgroup_name ipng:minio:ipv6 + service_description minio6:prom + check_command ipng_check_minio!$_HOSTADDRESS6$!19090!/classic/targets!minio-job + use ipng-service-fast + notification_interval 0 ; set > 0 if you want to be renotified +} + +define service { + hostgroup_name ipng:minio:ipv6 + service_description minio6:console + check_command ipng_check_minio!$_HOSTADDRESS6$!9090!/!MinIO Console + use ipng-service-fast + notification_interval 0 ; set > 0 if you want to be renotified +} +``` + +I've shown the snippet for IPv6 but I also have three services defined for legacy IP in the +hostgroup `ipng:minio:ipv4`. The check command here uses `-I` which has the IPv4 or IPv6 address to +talk to, `-p` for the port to consule, `-u` for the URI to hit and an option `-r` for a regular +expression to expect in the output. For the Nagios afficianados out there: my Ansible `groups` +correspond one to one with autogenerated Nagios `hostgroups`. This allows me to add arbitrary checks +by group-type, like above in the `ipng:minio` group for IPv4 and IPv6. + +In the MinIO [[docs](https://min.io/docs/minio/linux/operations/monitoring/healthcheck-probe.html)] +I read up on the Healthcheck API. I choose to monitor the _Cluster Write Quorum_ on my minio +deployments. For Prometheus, I decide to hit the `targets` endpoint and expect the `minio-job` to be +among them. Finally, for the MinIO Console, I expect to see a login screen with the words `MinIO +Console` in the returned page. I guessed right, because Nagios is all green: + +{{< image src="/assets/minio/nagios.png" alt="Nagios Dashboard" >}} + +## My First Bucket + +The IPng website is a statically generated Hugo site, and when-ever I submit a change to my Git +repo, a CI/CD runner (called [[Drone](https://www.drone.io/)]), picks up the change. It re-builds +the static website, and copies it to four redundant NGINX servers. + +But IPng's website has amassed quite a bit of extra files (like VM images and VPP packages that I +publish), which are copied separately using a simple push script I have in my home directory. This +avoids all those big media files from cluttering the Git repository. I decide to move this stuff +into S3: + +``` +pim@summer:~/src/ipng-web-assets$ echo 'Gruezi World.' > ipng.ch/media/README.md +pim@summer:~/src/ipng-web-assets$ mc mb chbtl0/ipng-web-assets +pim@summer:~/src/ipng-web-assets$ mc mirror . chbtl0/ipng-web-assets/ +...ch/media/README.md: 6.50 GiB / 6.50 GiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 236.38 MiB/s 28s +pim@summer:~/src/ipng-web-assets$ mc anonymous set download chbtl0/ipng-web-assets/ +``` + +OK, two things that immediately jump out at me. This stuff is **fast**: Summer is connected with a +2.5GbE network card, and she's running hard, copying the 6.5GB of data that are in these web assets +essentially at line rate. It doesn't really surprise me because Summer is running off of Gen4 NVME, +while MinIO has 12 spinning disks which each can write about 160MB/s or so sustained +[[ref](https://www.seagate.com/www-content/datasheets/pdfs/exos-x16-DS2011-1-1904US-en_US.pdf)], +with 24 CPUs to tend to the NIC (2x10G) and disks (2x SSD, 12x LFF). Should be plenty! + +The second is that MinIO allows for buckets to be publicly shared in three ways: 1) read-only by +setting `download`; 2) write-only by setting `upload`, and 3) read-write by setting `public`. +I set `download` here, which means I should be able to fetch an asset now publicly: + +``` +pim@summer:~$ curl https://s3.chbtl0.ipng.ch/ipng-web-assets/ipng.ch/media/README.md +Gruezi World. +pim@summer:~$ curl https://ipng-web-assets.s3.chbtl0.ipng.ch/ipng.ch/media/README.md +Gruezi World. +``` + +The first `curl` here shows the path-based access, while the second one shows an equivalent +virtual-host based access. Both retrieve the file I just pushed via the public Internet. Whoot! + +# What's Next + +I'm going to be moving [[Restic](https://restic.net/)] backups from IPng's ZFS storage pool to this +S3 service over the next few days. I'll also migrate PeerTube and possibly Mastodon from NVME based +storage to replicated S3 buckets as well. Finally, the IPng website media that I mentinoed above, +should make for a nice followup article. Stay tuned! diff --git a/static/assets/minio/console-1.png b/static/assets/minio/console-1.png new file mode 100644 index 0000000..128185f --- /dev/null +++ b/static/assets/minio/console-1.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:2fd52d2c08bab7bfbd016c544af65a536fae4ddd0f6d522d8cf3ca769f85c970 +size 79417 diff --git a/static/assets/minio/console-2.png b/static/assets/minio/console-2.png new file mode 100644 index 0000000..2e3982a --- /dev/null +++ b/static/assets/minio/console-2.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:80f7d18b234ccb6c7c6dab03e06bf0bb7f4fded0605d912e9f10b13f9816ef5c +size 290425 diff --git a/static/assets/minio/disks.png b/static/assets/minio/disks.png new file mode 100644 index 0000000..485eef7 --- /dev/null +++ b/static/assets/minio/disks.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e061a18ea0de22fe1dc73becc514928a0972e895463254ede1a9a5124f5414c2 +size 821126 diff --git a/static/assets/minio/minio-ec.svg b/static/assets/minio/minio-ec.svg new file mode 100644 index 0000000..9675b82 --- /dev/null +++ b/static/assets/minio/minio-ec.svg @@ -0,0 +1,1633 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/static/assets/minio/minio-logo.png b/static/assets/minio/minio-logo.png new file mode 100644 index 0000000..0d11780 --- /dev/null +++ b/static/assets/minio/minio-logo.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:4e2ef5ed79a9f4195e8e5201d39ff9e715caa8661521cb3f09b7b15c84c5f725 +size 8114 diff --git a/static/assets/minio/nagios.png b/static/assets/minio/nagios.png new file mode 100644 index 0000000..defcae1 --- /dev/null +++ b/static/assets/minio/nagios.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:680090b80eed7c4794287ff2979dd1e3383786369104155aa322ebe2c8cd6369 +size 211666 diff --git a/static/assets/minio/rack.png b/static/assets/minio/rack.png new file mode 100644 index 0000000..5a70de0 --- /dev/null +++ b/static/assets/minio/rack.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:087de953d52304b8d490fca9d9b9d33bc455ddf21e574a66e7813d9b21c6f2fe +size 765752