--- date: "2023-08-06T05:35:14Z" title: Pixelfed - Part 1 - Installing aliases: - /s/articles/2023/08/06/pixelfed-1.html --- # About this series {{< image width="200px" float="right" src="/assets/pixelfed/pixelfed-logo.svg" alt="Pixelfed" >}} I have seen companies achieve great successes in the space of consumer internet and entertainment industry. I've been feeling less enthusiastic about the stronghold that these corporations have over my digital presence. I am the first to admit that using "free" services is convenient, but these companies are sometimes taking away my autonomy and exerting control over society. To each their own of course, but for me it's time to take back a little bit of responsibility for my online social presence, away from centrally hosted services and to privately operated ones. After having written a fair bit about my Mastodon [[install]({{< ref "2022-11-20-mastodon-1" >}})] and [[monitoring]({{< ref "2022-11-27-mastodon-3" >}})], I've been using it every day. This morning, my buddy Ramón asked if he could make a second account on **ublog.tech** for his _Campervan Adventures_, and notably to post pics of where he and his family went. But if pics is your jam, why not ... [[Pixelfed](https://pixelfed.org/)]! ## Introduction Similar to how blogging is the act of publishing updates to a website, microblogging is the act of publishing small updates to a stream of updates on your profile. Very similar to the relationship between _Facebook_ and _Instagram_, _Mastodon_ and _Pixelfed_ give the ability to post and share, cross-link, discuss, comment and like, across the entire _Fediverse_. Except, Pixelfed doesn't do this in a centralized way, and I get to be a steward of my own data. As is common in the _Fediverse_, groups of people congregate on a given server, of which they become a user by creating an account on that server. Then, they interact with one another on that server, but users can also interact with folks on _other_ servers. Instead of following **@IPngNetworks**, they might follow a user on a given server domain, like **@IPngNetworks@pix.ublog.tech**. This way, all these servers can be run _independently_ but interact with each other using a common protocol (called ActivityPub). I've heard this concept be compared to choosing an e-mail provider: I might choose Google's gmail.com, and you might use Microsoft's live.com. However we can send e-mails back and forth due to this common protocol (called SMTP). ### pix.uBlog.tech I thought I would give it a go, mostly out of engineering curiosity but also because I more strongly feel today that we (the users) ought to take a bit more ownership back. I've been a regular blogging and micro-blogging user since approximately for ever, and I think it may be a good investment of my time to learn a bit more about the architecture of Pixelfed. So, I've decided to build and _productionize_ a server instance. Previously, I registered [uBlog.tech](https://ublog.tech) and have been running that for about a year as a _Mastodon_ instance. Incidentally, if you're reading this and would like to participate, the server welcomes users in the network-, systems- and software engineering disciplines. But, before I can get to the fun parts though, I have to do a bunch of work to get this server in a shape in which it can be trusted with user generated content. ### The IPng environment #### Pixelfed: Virtual Machine I provision a VM with 8vCPUs (dedicated on the underlying hypervisor), including 16GB of memory and one virtio network card. For disks, I will use two block devices, one small one of 16GB (vda) that is created on the hypervisor's `ssd-vol1/libvirt/pixelfed-disk0`, to be used only for boot, logs and OS. Then, a second one (vdb) is created at 2TB on `vol0/pixelfed-disk1` and it will be used for Pixelfed itself. I simply install Debian into **vda** using `virt-install`. At IPng Networks we have some ansible-style automation that takes over the machine, and further installs all sorts of Debian packages that we use (like a Prometheus node exporter, more on that later), and sets up a firewall that allows SSH access for our trusted networks, and otherwise only allows port 80 because this is to be a (backend) webserver behind the NGINX cluster. After installing Debian Bullseye, I'll create the following ZFS filesystems on **vdb**: ``` pim@pixelfed:~$ sudo zpool create data /dev/vdb pim@pixelfed:~$ sudo zfs create -o data/pixelfed -V10G pim@pixelfed:~$ sudo zfs create -o mountpoint=/data/pixelfed/pixelfed/storage data/pixelfed-storage pim@pixelfed:~$ sudo zfs create -o mountpoint=/var/lib/mysql data/mysql -V20G pim@pixelfed:~$ sudo zfs create -o mountpoint=/var/lib/redis data/redis -V2G ``` As a sidenote, I realize that this ZFS filesystem pool consists only of **vdb**, but its underlying blockdevice is protected in a raidz, and it is copied incrementally daily off-site by the hypervisor. I'm pretty confident on safety here, but I prefer to use ZFS for the virtual machine guests as well, because now I can do local snapshotting, of say `data/pixelfed`, and I can more easily grow/shrink the datasets for the supporting services, as well as isolate them individually against sibling wildgrowth. The VM gets one virtual NIC, which will connect to the [[IPng Site Local]({{< ref "2023-03-17-ipng-frontends" >}})] network using jumboframes. This way, the machine itself is disconnected from the internet, saving a few IPv4 addresses and allowing for the IPng NGINX frontends to expose it. I give it the name `pixelfed.net.ipng.ch` with addresses 198.19.4.141 and 2001:678:d78:507::d, which will be firewalled and NATed via the IPng SL gateways. #### IPng Frontend: Wildcard SSL I run most websites behind a cluster of NGINX webservers, which are carrying an SSL certificate which support wildcards. The system is using [[DNS-01]({{< ref "2023-03-24-lego-dns01" >}})] challenges, so the first order of business is to expand the certificate from serving only [[ublog.tech](https://ublog.tech)] (which is in use by the companion Mastodon instance), to include as well _*.ublog.tech_ so that I can add the new Pixelfed instance as [[pix.ublog.tech](https://pix.ublog.tech)]: ``` lego@lego:~$ certbot certonly --config-dir /home/lego/acme-dns --logs-dir /home/lego/logs \ --work-dir /home/lego/workdir --manual \ --manual-auth-hook /home/lego/acme-dns/acme-dns-auth.py \ --preferred-challenges dns --debug-challenges \ -d ipng.ch -d *.ipng.ch -d *.net.ipng.ch \ -d ipng.nl -d *.ipng.nl \ -d ipng.eu -d *.ipng.eu \ -d ipng.li -d *.ipng.li \ -d ublog.tech -d *.ublog.tech \ -d as8298.net \ -d as50869.net CERTFILE=/home/lego/acme-dns/live/ipng.ch/fullchain.pem KEYFILE=/home/lego/acme-dns/live/ipng.ch/privkey.pem MACHS="nginx0.chrma0.ipng.ch nginx0.chplo0.ipng.ch nginx0.nlams1.ipng.ch nginx0.nlams2.ipng.ch" for MACH in $MACHS; do fping -q $MACH 2>/dev/null || { echo "$MACH: Skipping (unreachable)" continue } echo $MACH: Copying $CERT scp -q $CERTFILE $MACH:/etc/nginx/certs/$CERT.crt scp -q $KEYFILE $MACH:/etc/nginx/certs/$CERT.key echo $MACH: Reloading nginx ssh $MACH 'sudo systemctl reload nginx' done ``` The first command here requests a certificate with `certbot`, and note the addition of the flag `-d *.ublog.tech`. It'll correctly say that there are 11 existing domains in this certificate, and ask me if I'd like to request a new cert with the 12th one added. I answer yes, and a few seconds later, `acme-dns` has answered all of Let's Encrypt's challenges, and issues a certificate. The second command then distributes that certificate to the four NGINX frontends, and reloads the cert. Now, I can use the hostname `pix.ublog.tech`, as far as the SSL certs are concerned. Of course, the regular certbot cronjob renews the cert regularly, so I tucked away the second part here into a script called `bin/certbot-distribute`, using the `RENEWED_LINEAGE` variable that certbot(1) sets when using the flag `--deploy-hook`: ``` lego@lego:~$ cat /etc/cron.d/certbot SHELL=/bin/sh PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin 0 */12 * * * lego perl -e 'sleep int(rand(43200))' && \ certbot -q renew --config-dir /home/lego/acme-dns \ --logs-dir /home/lego/logs --work-dir /home/lego/workdir \ --deploy-hook "/home/lego/bin/certbot-distribute" ``` #### IPng Frontend: NGINX The previous `certbot-distribute` shell script has copied the certificate to four separate NGINX instances, two in Amsterdam hosted at AS8283 (Coloclue), one in Zurich hosted at AS25091 (IP-Max), and one in Geneva hosted at AS8298 (IPng Networks). Each of these NGINX servers has a frontend IPv4 and IPv6 address, and a backend jumboframe enabled interface in IPng Site Local (198.19.0.0/16). Because updating the configuration on four production machines is cumbersome, I previously created an Ansible playbook, which I now add this new site to: ``` pim@squanchy:~/src/ipng-ansible$ cat roles/nginx/files/sites-available/pix.ublog.tech.conf server { listen [::]:80; listen 0.0.0.0:80; server_name pix.ublog.tech; access_log /var/log/nginx/pix.ublog.tech-access.log; include /etc/nginx/conf.d/ipng-headers.inc; include "conf.d/lego.inc"; location / { return 301 https://$host$request_uri; } } server { listen [::]:443 ssl http2; listen 0.0.0.0:443 ssl http2; ssl_certificate /etc/nginx/certs/ipng.ch.crt; ssl_certificate_key /etc/nginx/certs/ipng.ch.key; include /etc/nginx/conf.d/options-ssl-nginx.inc; ssl_dhparam /etc/nginx/conf.d/ssl-dhparams.inc; server_name pix.ublog.tech; access_log /var/log/nginx/pix.ublog.tech-access.log upstream; include /etc/nginx/conf.d/ipng-headers.inc; keepalive_timeout 70; sendfile on; client_max_body_size 80m; location / { proxy_pass http://pixelfed.net.ipng.ch:80; proxy_set_header Host $host; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } } ``` The configuration is very straight forward. The first server block bounces all traffic destined to port 80 towards its port 443 equivalent. The second server block (listening on port 443) contains the certificate I just renewed serve for `*.ublog.tech` which allows the cluster to offload SSL and forward the traffic on the internal private network on to the VM I created earlier. One quick Ansible playbook run later, and the reversed proxies are ready to rock and roll: {{< image src="/assets/pixelfed/ansible.png" alt="Ansible" >}} Of course, this website will just timeout for the time being, because there's nothing listening (yet) on `pixelfed.net.ipng.ch:80`. #### Installing Pixelfed So off I go, installing Pixelfed on the new Debian VM. First, I'll install the set of Debian packages this instance will need, including PHP 8.1 (which is the minimum supported, according to the Pixelfed docs): ``` pim@pixelfed:~$ sudo apt install apt-transport-https lsb-release ca-certificates git wget curl \ build-essential apache2 mariadb-server pngquant optipng jpegoptim gifsicle ffmpeg redis pim@pixelfed:~$ sudo wget -O /etc/apt/trusted.gpg.d/php.gpg https://packages.sury.org/php/apt.gpg pim@pixelfed:~$ echo "deb https://packages.sury.org/php/ $(lsb_release -sc) main" \ | sudo tee -a /etc/apt/sources.list.d/php.list pim@pixelfed:~$ apt update pim@pixelfed:~$ apt-get install php8.1-fpm php8.1 php8.1-common php8.1-cli php8.1-gd \ php8.1-mbstring php8.1-xml php8.1-bcmath php8.1-pgsql php8.1-curl php8.1-xml php8.1-xmlrpc \ php8.1-imagick php8.1-gd php8.1-mysql php8.1-cli php8.1-intl php8.1-zip php8.1-redis ``` After all those bits and bytes settle on the filesystem, I simply follow the regular [[install guide](https://docs.pixelfed.org/running-pixelfed/installation/)] from the upstream documentation. I update the PHP config to allow larger uploads: ``` pim@pixelfed:~$ sudo vim /etc/php/8.1/fpm/php.ini upload_max_filesize = 100M post_max_size = 100M ``` I create a FastCGI pool for Pixelfed: ``` pim@pixelfed:~$ cat << EOF | sudo tee /etc/php/8.1/fpm/pool.d/pixelfed.conf [pixelfed] user = pixelfed group = pixelfed listen.owner = www-data listen.group = www-data listen.mode = 0660 listen = /var/run/php.pixelfed.sock pm = dynamic pm.max_children = 20 pm.start_servers = 5 pm.min_spare_servers = 5 pm.max_spare_servers = 20 chdir = /data/pixelfed php_flag[display_errors] = on php_admin_value[error_log] = /data/pixelfed/php.error.log php_admin_flag[log_errors] = on php_admin_value[open_basedir] = /data/pixelfed:/usr/share/:/tmp:/var/lib/php EOF ``` I reference this pool in a simple non-SSL Apache config, after enabling the modules that Pixelfed needs: ``` pim@pixelfed:~$ cat << EOF | sudo tee /etc/apache2/sites-available/pixelfed.conf ServerName pix.ublog.tech ServerAdmin pixelfed@ublog.tech DocumentRoot /data/pixelfed/pixelfed/public LogLevel debug Options Indexes FollowSymLinks AllowOverride All Require all granted ErrorLog ${APACHE_LOG_DIR}/pixelfed.error.log CustomLog ${APACHE_LOG_DIR}/pixelfed.access.log combined SetHandler "proxy:unix:/var/run/php.pixelfed.sock|fcgi://localhost" EOF ``` I create a user and database, and finally download the Pixelfed sourcecode and install the `composer` tool: ``` pim@pixelfed:~$ sudo useradd pixelfed -m -d /data/pixelfed -s /bin/bash -r -c "Pixelfed User" pim@pixelfed:~$ sudo mysql CREATE DATABASE pixelfed; GRANT ALL ON pixelfed.* TO pixelfed@localhost IDENTIFIED BY ''; exit pim@pixelfed:~$ wget -O composer-setup.php https://getcomposer.org/installer pim@pixelfed:~$ sudo php composer-setup.php pim@pixelfed:~$ sudo cp composer.phar /usr/local/bin/composer pim@pixelfed:~$ rm composer-setup.php pim@pixelfed:~$ sudo su pixelfed pixelfed@pixelfed:~$ git clone -b dev https://github.com/pixelfed/pixelfed.git pixelfed pixelfed@pixelfed:~$ cd pixelfed pixelfed@pixelfed:/data/pixelfed/pixelfed$ composer install --no-ansi --no-interaction --optimize-autoloader pixelfed@pixelfed:/data/pixelfed/pixelfed$ composer update ``` With the basic installation of pacakges and dependencies all squared away, I'm ready to configure the instance: ``` pixelfed@pixelfed:/data/pixelfed/pixelfed$ vim .env APP_NAME="uBlog Pixelfed" APP_URL="https://pix.ublog.tech" APP_DOMAIN="pix.ublog.tech" ADMIN_DOMAIN="pix.ublog.tech" SESSION_DOMAIN="pix.ublog.tech" TRUST_PROXIES="*" # Database Configuration DB_CONNECTION="mysql" DB_HOST="127.0.0.1" DB_PORT="3306" DB_DATABASE="pixelfed" DB_USERNAME="pixelfed" DB_PASSWORD="" MAIL_DRIVER=smtp MAIL_HOST=localhost MAIL_PORT=25 MAIL_FROM_ADDRESS="pixelfed@ublog.tech" MAIL_FROM_NAME="uBlog Pixelfed" pixelfed@pixelfed:/data/pixelfed/pixelfed$ php artisan key:generate pixelfed@pixelfed:/data/pixelfed/pixelfed$ php artisan storage:link pixelfed@pixelfed:/data/pixelfed/pixelfed$ php artisan migrate --force pixelfed@pixelfed:/data/pixelfed/pixelfed$ php artisan import:cities pixelfed@pixelfed:/data/pixelfed/pixelfed$ php artisan instance:actor pixelfed@pixelfed:/data/pixelfed/pixelfed$ php artisan passport:keys pixelfed@pixelfed:/data/pixelfed/pixelfed$ php artisan route:cache pixelfed@pixelfed:/data/pixelfed/pixelfed$ php artisan view:cache pixelfed@pixelfed:/data/pixelfed/pixelfed$ php artisan config:cache ``` Pixelfed is based on [[Laravel](https://laravel.com/)], a PHP framework for _Web Artisans_ (which I guess now that I run both LibreNMS, IXPManager and Pixelfed, makes me one too?). Laravel has two runner types commonly used. One is task queuing via a module called _Laravel Horizon_, which uses Redis to store work items to be consumed by task workers: ``` pim@pixelfed:~$ cat << EOF | sudo tee /lib/systemd/system/pixelfed.service [Unit] Description=Pixelfed task queueing via Laravel Horizon After=network.target Requires=mariadb Requires=php-fpm Requires=redis Requires=apache [Service] Type=simple ExecStart=/usr/bin/php /data/pixelfed/pixelfed/artisan horizon User=pixelfed Restart=on-failure [Install] WantedBy=multi-user.target pim@pixelfed:~$ sudo systemctl enable --now pixelfed ``` The other type of runner is periodic tasks, typically configured in a crontab, like so: ``` pim@pixelfed:~$ cat << EOF | sudo tee /etc/cron.d/pixelfed * * * * * pixelfed /usr/bin/php /data/pixelfed/pixelfed/artisan schedule:run >> /dev/null 2>&1 EOF ``` After running the _schedule:run_ module once by hand, it exits cleanly, so I think this is good to go even though I'm not a huge fan of redirecting output to `/dev/null` like that. I will create one `admin` user on the commandline first: ``` pixelfed@pixelfed:/data/pixelfed/pixelfed$ php artisan user:create ``` And now that everything is ready, I can put the icing on the cake by enabling and starting the Apache2 webserver: ``` pim@pixelfed:~$ sudo a2enmod rewrite proxy proxy_fcgi pim@pixelfed:~$ sudo a2ensite pixelfed pim@pixelfed:~$ sudo systemctl restart apache2 ``` {{< image src="/assets/pixelfed/fipo.png" alt="First Post" >}} ### Finishing Touches #### File permissions After signing up, logging in and uploading my first post (which is of a BLT sandwich and a bowl of noodles, of course), I noticed that the permissions are overly strict, and the pictures I just uploaded are not visible. I noticed that the PHP FastCGI is running as user `pixelfed` while the webserver is running as user `www-data`, and the former is writing files with permissions `rw-------` and directories with `rwx------`, which doesn't seem quite right to me, so I make a small edit in `config/filesystems.php`, changing the `0600` to `0644` and the `0700` to `0755`, after which my post is visible. #### uBlog's logo {{< image width="300px" float="right" src="/assets/pixelfed/ublog.png" alt="uBlog" >}} Although I do like the Pixelfed logo, I wanted to keep a **ublog.tech** branding, so I replaced the `public/storage/headers/default.jpg` with my own mountains-picture in roughly the same size. By the way, I took that picture in Grindelwald, Switzerland during a [[serene moment]({{< ref "2021-07-26-bucketlist" >}})] in which I discovered why tinkering with things like this is so important to my mental health. #### Backups Of course, since Ramón is a good friend, I would not want to lose his pictures. Data integrity and durability is important to me. It's the one thing that typically the commercial vendors do really well, and my pride prohibits me from losing data due to things like "disk failure" or "computer broken" or "datacenter on fire". To honor this promise, I handle backups in three main ways: zrepl(1), borg(1) and mysqldump(1). * **VM Block Devices** are running on the hypervisor's ZFS on either the SSD pool, or the disk pool, or both. Using a tool called **zrepl(1)** (which I described a little bit in a [[previous post]({{< ref "2022-10-14-lab-1" >}})]), I create a snapshot every 12hrs on the local blockdevice, and incrementally copy away those snapshots daily to the remote fileservers. ``` pim@hvn0.ddln0:~$ sudo cat /etc/zrepl/zrepl.yaml jobs: - name: snap-libvirt type: snap filesystems: { "ssd-vol0/libvirt<": true, "ssd-vol1/libvirt<": true } snapshotting: type: periodic prefix: zrepl_ interval: 12h pruning: keep: - type: grid grid: 4x12h(keep=all) | 7x1d regex: "^zrepl_.*" - type: push name: "push-st0-chplo0" filesystems: { "ssd-vol0/libvirt<": true, "ssd-vol1/libvirt<": true } connect: type: ssh+stdinserver host: st0.chplo0.net.ipng.ch user: root port: 22 identity_file: /etc/zrepl/ssh/identity snapshotting: type: manual send: encrypted: false pruning: keep_sender: - type: not_replicated - type: last_n count: 10 regex: ^zrepl_.*$ # optional keep_receiver: - type: grid grid: 8x12h(keep=all) | 7x1d | 6x7d regex: "^zrepl_.*" ``` * **Filesystem Backups** make a daily copy of their entire VM filesystem using **borgbackup(1)** to a set of two remote fileservers. This way, the important file metadata, configs for the virtual machines, and so on, are all safely stored remotely. ``` pim@pixelfed:~$ sudo mkdir -p /etc/borgmatic/ssh pim@pixelfed:~$ sudo ssh-keygen -t ecdsa -f /etc/borgmatic/ssh/identity -C root@pixelfed.net.ipng.ch pim@pixelfed:~$ cat << EOF | sudo tee /etc/borgmatic/config.yaml location: source_directories: - / repositories: - u022eaebe661@st0.chbtl0.ipng.ch:borg/{fqdn} - u022eaebe661@st0.chplo0.ipng.ch:borg/{fqdn} exclude_patterns: - /proc - /sys - /dev - /run - /swap.img exclude_if_present: - .nobackup - .borgskip storage: encryption_passphrase: ssh_command: "ssh -i /etc/borgmatic/identity -6" compression: lz4 umask: 0077 lock_wait: 5 retention: keep_daily: 7 keep_weekly: 4 keep_monthly: 6 consistency: checks: - repository - archives check_last: 3 output: color: false ``` * **MySQL** has a running binary log to recover from failures/restarts, but I also run a daily mysqldump(1) operation that dumps the database to the local filesystem, allowing for quick and painless recovery. As the dump is a regular file on the filesystem, it'll be picked up by the filesystem backup every night as well, for long term and off-site safety. ``` pim@pixelfed:~$ sudo zfs create data/mysql-backups pim@pixelfed:~$ cat << EOF | sudo tee /etc/cron.d/bitcron 25 5 * * * root /usr/local/bin/bitcron mysql-backup.cron EOF ``` For my friends at AS12859 [[bit.nl](https://www.bit.nl/)], I still use `bitcron(1)` :-) For the rest of you -- bitcron is a little wrapper written in Bash that defines a few primitives such as logging, iteration, info/warning/error/fatals etc, and then runs whatever you define in a function called `bitcron_main()`, sending e-mail to an operator only if there are warnings or errors, and otherwise logging to `/var/log/bitcron`. The gist of the mysql-backup bitcron is this: ``` echo "Rotating the $DESTDIR directory" rotate 10 echo "Done (rotate)" echo "" echo "Creating $DESTDIR/0/ to store today's backup" mkdir -p $DESTDIR/0 || fatal "Could not create $DESTDIR/0/" echo "Done (mkdir)" echo "" echo "Fetching databases" DBS=$(echo 'show databases' | mysql -u$MYSQLUSER -p$MYSQLPASS | egrep -v '^Database') echo "Done (fetching DBs)" echo "" echo "Backing up all databases" for DB in $DBS; do echo " * Database $DB" mysqldump --single-transaction -u$MYSQLUSER -p$MYSQLPASS -a $DB | gzip -9 -c \ > $DESTDIR/0/mysqldump_$DB.gz \ || warning "Could not dump database $DB" done echo "Done backing up all databases" echo "" ``` ## What's next Now that the server is up, and I have a small amount of users (mostly folks I know from the tech industry), I took some time to explore both the Fediverse, reach out to friends old and new, participate in a few random discussions possibly about food, datacenter pics and camping trips, as well as fiddle with the iOS and Android apps (for now, I've settled on Vernissage after switching my iPhone away from the horrible HEIC format which literally nobody supports). This is going to be fun :) Now, I think I'm ready to further productionize the experience. It's important to monitor these applications, so in an upcoming post I'll be looking at how to do _blackbox_ and _whitebox_ monitoring on this instance. If you're looking for a home, feel free to sign up at [https://pix.ublog.tech/](https://pix.ublog.tech/) as I'm sure that having a bit more load / traffic on this instance will allow me to learn (and in turn, to share with others)! Of course, my Mastodon instance at [https://ublog.tech/](https://ublog.tech) is also happy to serve.