--- date: "2024-05-17T10:56:54Z" title: 'Case Study: IPng''s mail servers' aliases: - /s/articles/2024/05/17/smtp.html --- ## Intro I have seen companies achieve great successes in the space of consumer internet and entertainment industry. I've been feeling less enthusiastic about the stronghold that these corporations have over my digital presence. I am the first to admit that using "free" services is convenient, but these companies are sometimes taking away my autonomy and exerting control over society. To each their own of course, but for the last few years, I've been more and more inclined to take back a little bit of responsibility for my online social presence, away from centrally hosted services and to privately operated ones. {{< image width="120px" float="right" src="/assets/smtp/gmail_logo.png" alt="GMail" >}} First off - I **love** Google's Workspace products. I started using GMail just after it launched, back in 2004. Its user interface is sleek, performant, and very intuitive. Its filtering, granted, could be a bit less ... robotic, but that's made up by labels and an incredibly comprehensive search function. I would dare say that between GMail and Photos, those are my absolute favorite products on the internet. That said, I have been running e-mail servers since well before Google existed as a company. I started off at M.C.G.V. Stack, the computer club of the University of Eindhoven, in 1995. We ran sendmail back then, and until about two months ago, I have continuously run sendmail in production using the PaPHosting platform [[ref](https://paphosting.net/)] that I wrote with my buddies Paul and Jeroen. However, two things happened, both of them somewhat nerdsnipe-esque: 1. Mrs IPngNetworks said "Well if you are going to use NextCloud and PeerTube and PixelFed and Mastodon, why would you not run your own mailserver?" 1. I added a forward for `event@frys-ix.net` on my Sendmail relays at PaPHosting, and was tipped by my buddy Jelle that his e-mail to it was bouncing due to SPF strictness. ### I tried to resist ... {{< image width="150px" float="left" src="/assets/smtp/pulling_hair.png" alt="Pulling Hair" >}} My main argument against running a mailserver has been the mailspool. Before I moved to GMail, I had the misfortune of having my mail and primary DNS fail, running on `bfib.ipng.nl` at the time, a server 700km away from me, without redundancy. Even the nameserver slaves went beyond their zone refresh. It was not a good month for me, even though it was twenty years or so ago :) Last year, during a roadtrip with Fred, he and I spent a few long hours restoring a backup after a catastrophic failure of a hypervisor at IP-Max on which his mailserver was running. Luckily, backups were awesome and saved the day, but having to go into a Red Alert mode and not being able to communicate, really can be stressful. I don't want to run mailservers!!!1 ### .. but resistance is futile After this nerdsnipe, I had a short conversation with Jeroen who mentioned that since I last had a look at this, Dovecot, a popular imap/pop3 server, had gained the ability to do mailbox synchronization across multiple machines. That's a really nifty feature - but also it meant that there will be no more single points of failure, if I do this properly. Oh crap, there's no longer an argument of resistance? Nerd-snipe accepted! Let me first introduce the mail^W main characters of my story: | ![Postfix](/assets/smtp/postfix_logo.png){: style="width:100px; margin: 1em;"} | ![Dovecot](/assets/smtp/dovecot_logo.png){: style="width:100px; margin: 1em;"} | ![NGINX](/assets/smtp/nginx_logo.png){: style="width:100px; margin: 1em;"} | ![rspamd](/assets/smtp/rspamd_logo.png){: style="width:100px; margin: 1em;"} | ![Unbound](/assets/smtp/unbound_logo.png){: style="width:100px; margin: 1em;"} | ![Postfix](/assets/smtp/roundcube_logo.png){: style="width:100px; margin: 1em;"} | * ***Postfix***: is Wietse Venema's mail server that started life at IBM research as an alternative to the widely-used Sendmail program. After eight years at Google, Wietse continues to maintain Postfix. * ***Dovecot***: an open source IMAP and POP3 email server for Linux/UNIX-like systems, written with security primarily in mind. Dovecot is an excellent choice for both small and large installations. * ***NGINX***: an HTTP and reverse proxy server, a mail proxy server, and a generic TCP/UDP proxy server, originally written by Igor Sysoev. * ***Rspamd***: an advanced spam filtering system and email processing framework that allows evaluation of messages by a number of rules including regular expressions, statistical analysis and custom services such as URL black lists. * ***OpenDKIM***: is a community effort to develop and maintain a C library for producing DKIM-aware applications and an open source milter for providing DKIM service. * ***Unbound***: a validating, recursive, caching DNS resolver. It is designed to be fast and lean and incorporates modern features based on open standards. * ***Roundcube***: a web-based IMAP email client. Roundcube's most prominent feature is the pervasive use of Ajax technology. In the rest of this article, I'll go over four main parts that I used to build a fully redundant and self-healing mail service at IPng Networks: 1. **Green**: `smtp-in.ipng.ch` which handles inbound e-mail 1. **Red**: `imap.ipng.ch` which serves mailboxes to users 1. **Blue**: `smtp-out.ipng.ch` which handles outbound e-mail 1. **Magenta**: `webmail.ipng.ch` which exposes the mail in a web browser Let me start with a functional diagram, using those colors: {{< image src="/assets/smtp/IPng Mail Cluster.svg" alt="IPng Mail Cluster" >}} As you can see in this diagram, I will be separating concerns and splitting the design into three discrete parts, which will also be in three sets of redundantly configured backend servers running on IPng's hypervisors in Zurich (CH), Lille (FR) and Amsterdam (NL). ### 1. Outbound: smtp-out I'm going to start with a relatively simple component first: outbound mail. This service will be listening on the _smtp submission_ port 587, require TLS and user authentication from clients, validate outbound e-mail using a spam detection agent, and finally provide DKIM signing on all outbound e-mails. It should spool and retry the delivery in case there is a temporary issue (like greylisting, or server failure) on the receiving side. Because the only way to send e-mail will be using TLS and user authentication, the smtp-out servers themselves will not need to do any DNSBL lookups, which is convenient because it means I can put them behind a loadbalancer and serve them entirely within IPng Site Local. If you're curious as to what this site local thing means, basically it's an internal network spanning all IPng's points of presence, with an IPv4, IPv6 and MPLS backbone that is disconnected from the internet. For more details on the design goals, take a look at the [[article]({{< ref "2023-03-11-mpls-core" >}})] I wrote about it last year. #### Debian VMs I'll take three identical virtual machines, hosted on three separate hypervisors each in their own country. ``` pim@summer:~$ dig ANY smtp-out.net.ipng.ch smtp-out.net.ipng.ch. 60 IN A 198.19.6.73 smtp-out.net.ipng.ch. 60 IN A 198.19.4.230 smtp-out.net.ipng.ch. 60 IN A 198.19.6.135 smtp-out.net.ipng.ch. 60 IN AAAA 2001:678:d78:50e::9 smtp-out.net.ipng.ch. 60 IN AAAA 2001:678:d78:50a::6 smtp-out.net.ipng.ch. 60 IN AAAA 2001:678:d78:510::7 ```` I will give them each 8GB of memory, 4 vCPUs, and 16GB of bootdisk. I'm pretty confident that the whole system will be running in only a fraction of that. I will install a standard issue Debian Bookworm (12.5), and while my VMs by default have 4 virtual NICs, I only need one, connected to the IPng Site Local: ``` pim@smtp-out-chrma0:~$ ip -br a lo UNKNOWN 127.0.0.1/8 ::1/128 enp1s0f0 UP 198.19.6.135/27 2001:678:d78:510::7/64 fe80::5054:ff:fe99:81b5/64 enp1s0f1 UP fe80::5054:ff:fe99:81b6/64 enp1s0f2 UP fe80::5054:ff:fe99:81b7/64 enp1s0f3 UP fe80::5054:ff:fe99:81b8/64 pim@smtp-out-chrma0:~$ mtr -6 -c5 -r dns.google Start: 2024-05-17T13:49:28+0200 HOST: smtp-out-chrma0 Loss% Snt Last Avg Best Wrst StDev 1.|-- msw1.chrma0.net.ipng.ch 0.0% 5 1.6 1.5 1.3 1.6 0.1 2.|-- msw0.chrma0.net.ipng.ch 0.0% 5 1.4 1.3 1.3 1.4 0.1 3.|-- msw0.chbtl0.net.ipng.ch 0.0% 5 3.2 3.1 2.8 3.2 0.2 4.|-- hvn0.chbtl0.net.ipng.ch 0.0% 5 1.5 1.5 1.4 1.5 0.0 5.|-- chbtl0.ipng.ch 0.0% 5 1.6 1.7 1.6 1.7 0.0 6.|-- chrma0.ipng.ch 0.0% 5 2.4 2.4 2.4 2.5 0.0 7.|-- as15169.lup.swissix.ch 0.0% 5 3.2 3.8 3.2 5.6 1.0 8.|-- 2001:4860:0:1::6083 0.0% 5 4.5 4.5 4.5 4.5 0.0 9.|-- 2001:4860:0:1::12f9 0.0% 5 3.4 3.5 3.4 3.5 0.0 10.|-- dns.google 0.0% 5 3.8 3.9 3.8 4.0 0.1 ``` One cool observation: these machines are not really connected to the internet - you'll note that their IPv4 address is from reserved space, and their IPv6 supernet (2001:678:d78:500::/56) is filtered at the border. I'll get to that later! #### Postfix I will install Postfix, and make a few adjustments to its config. First off, this mailserver will only be receiving `submission` mail, which is port 587. It will not participate or listen to the regular `smtp` port 25, nor `smtps` port 465, as such the `master.cf` file for Postfix becomes: ``` #smtp inet n - y - - smtpd # -o smtpd_sasl_auth_enable=no submission inet n - y - - smtpd -o syslog_name=postfix/submission -o smtpd_tls_security_level=encrypt -o smtpd_sasl_auth_enable=yes -o smtpd_reject_unlisted_recipient=no -o smtpd_client_restrictions=permit_sasl_authenticated,permit_mynetworks,reject -o milter_macro_daemon_name=ORIGINATING #smtps inet n - y - - smtpd # -o syslog_name=postfix/smtps # -o smtpd_tls_wrappermode=yes # -o smtpd_sasl_auth_enable=yes ``` The only thing I will make note of is that the `submission` service has a set of client restrictions. In other words, to be able to use this service, the client must either be SASL authenticated, or from a list of network prefixes that are allowed to relay. If neither of those two conditions are satisfied, relaying will be denied. Now, I understand that pasting in the entire postfix configuration is a bit verbose, but honestly I've spent many an hour trying to puzzle together an end-to-end valid configuration, so I'm just going to swim upstream and post the whole `main.cf`, which I'll try to annotate the broad strokes of, in case there's anybody out there trying to learn: ``` myhostname = smtp-out.ipng.ch myorigin = smtp-out.ipng.ch mydestination = $myhostname, smtp-out.chrma0.net.ipng.ch, localhost.net.ipng.ch, localhost mynetworks = 127.0.0.0/8, [::1]/128 recipient_delimiter = + inet_interfaces = all inet_protocols = all biff = no # appending .domain is the MUA's job. append_dot_mydomain = no readme_directory = no # See http://www.postfix.org/COMPATIBILITY_README.html -- default to 3.6 on fresh installs. compatibility_level = 3.6 # SMTP Server smtpd_banner = $myhostname ESMTP $mail_name (smtp-out.chrma0.net.ipng.ch) smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination smtpd_tls_cert_file = /etc/certs/ipng.ch/fullchain.pem smtpd_tls_key_file = /etc/certs/ipng.ch/privkey.pem smtpd_tls_CAfile = /etc/ssl/certs/ca-certificates.crt smtpd_tls_CApath = /etc/ssl/certs smtpd_use_tls = yes smtpd_tls_received_header = yes smtpd_tls_auth_only = yes smtpd_tls_session_cache_database = btree:$data_directory/smtpd_scache smtpd_client_connection_count_limit = 4 smtpd_client_connection_rate_limit = 10 smtpd_client_message_rate_limit = 60 smtpd_client_event_limit_exceptions = $mynetworks # Dovecot auth smtpd_sasl_type = dovecot smtpd_sasl_path = private/auth smtpd_sasl_authenticated_header = yes smtpd_sasl_auth_enable = yes smtpd_sasl_security_options = noanonymous, noplaintext smtpd_sasl_tls_security_options = noanonymous # SMTP Client smtp_use_tls = yes smtp_tls_note_starttls_offer = yes smtp_tls_cert_file = /etc/certs/ipng.ch/fullchain.pem smtp_tls_key_file = /etc/certs/ipng.ch/privkey.pem smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt smtp_tls_CApath = /etc/ssl/certs smtp_tls_mandatory_ciphers = medium smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache smtp_tls_security_level = encrypt header_size_limit = 4096000 message_size_limit = 52428800 mailbox_size_limit = 0 # OpenDKIM, Rspamd smtpd_milters = inet:localhost:8891,inet:rspamd.net.ipng.ch:11332 non_smtpd_milters = $smtpd_milters # Local aliases alias_maps = hash:/etc/postfix/aliases alias_database = hash:/etc/postfix/aliases ``` ***Hostnames***: The full (internal) hostname for the server is `smtp-out.$(site).net.ipng.ch`, in this case for `chrma0` in Rümlang, Switzerland. However, when clients connect to the public hostname `smtp-out.ipng.ch`, they will expect that the TLS certificate matches _that_ hostname. This is why I let the server present itself as simply `smtp-out.ipng.ch`, which will also be its public DNS name later, but put the internal FQDN for debugging purposes between parenthesis. See the `smtpd_banner` and `myhostname` for the destinction. I'll load up the `*.ipng.ch` wildcard certificate which I described in my Let's Encrypt [[DNS-01]({{< ref "2023-03-24-lego-dns01" >}})] article. ***Authorization***: I will make Postfix accept relaying for those users that are either in the `mynetworks` (which is only localhost) OR `sasl_authenticated` (ie. presenting a username and password). This password exchange will only be possible after encryption has been triggered using the `STARTTLS` SMTP feature. This way, user/pass combos will be safe on the network. ***Authentication***: Those username and password combos can come from a few places. One popular way to do this is via a `dovecot` authentication service. Via the `smtpd_sasl_path`, I tell Postfix to ask these authentication questions using the dovecot protocol on a certain file path. I'll let Dovecot listen in the `/var/spool/postfix/private/auth` directory. This is how Postfix will know which user to relay for, and which to deny. ***DKIM/SPF***: These days, most (large and small) mail providers will be suspicious of e-mail that is delivered to them without proper SPF and DKIM fields. ***DKIM*** is a mechanism to create a cryptographic signature over some of the E-Mail header fields (usually From/Subject/Date), which can be checked by the recipient for validity. ***SPF*** is a mechanism to use DNS to inform receiving mailservers of which are the valid IPv4/IPv6 addresses that should be used to deliver mail for a given sender domain. #### Dovecot (auth) The configuration for Dovecot is incredibly simple. The only thing I do is create a mostly empty `dovecot.conf` file which defines the `auth` service listening in the place where Postfix expects it. Then, I add a password file called `sasl-users` which will contain user:password tuples: ``` service auth { unix_listener /var/spool/postfix/private/auth { mode = 0660 # Assuming the default Postfix user and group user = postfix group = postfix } } passdb { driver = passwd-file args = username_format=%n /etc/dovecot/sasl-users } ``` I can use `doveadm pw` to generate such passwords. I do this in an upstream Ansible repository and then push out the same configuration to any number of `smtp-out` servers, so they are all configured identically to this one. #### OpenDKIM (signing) Now that I can authorize (via SASL) and authenticate (via Dovecot backend) a user, it will be entitled to use the `smtp-out` Postfix to send e-mail. However, there's a good chance that recipients will bounce the e-mail, unless it comes with a DKIM signature, and from the _correct_ IP addresses. To configure DKIM signing, I use OpenDKIM, which I give the following `/etc/opendkim.conf` file: ``` pim@smtp-out-chrma0:~$ cat /etc/opendkim.conf Syslog yes LogWhy yes UMask 007 Mode sv AlwaysAddARHeader yes SignatureAlgorithm rsa-sha256 X-Header no KeyTable refile:/etc/opendkim/keytable SigningTable refile:/etc/opendkim/signers RequireSafeKeys false Canonicalization relaxed TrustAnchorFile /usr/share/dns/root.key UserID opendkim PidFile /run/opendkim/opendkim.pid Socket inet6:8891 ``` It opens a socket at port 8891, which is where Postfix expects it, based on its `smtpd_milter` configuration option. It will look at the so-called **SigningTable** to determine which outbound e-mail addresses it can sign. This table looks up From addresses, including wildcards, and informs which symbolic keyname in the **KeyTable** to use for the signature, like so: ``` pim@smtp-out-chrma0:/etc/opendkim$ cat signers *@*.ipng.nl ipng-nl *@ipng.nl ipng-nl *@*.ipng.ch ipng-ch *@ipng.ch ipng-ch *@*.ublog.tech ublog *@ublog.tech ublog ... pim@smtp-out-chrma0:/etc/opendkim$ cat keytable ipng-nl ipng.nl:DKIM2022:/etc/opendkim/keys/DKIM2022-ipng.nl-private ipng-ch ipng.ch:DKIM2022:/etc/opendkim/keys/DKIM2022-ipng.ch-private ublog ublog.tech:DKIM2022:/etc/opendkim/keys/DKIM2022-ublog.tech-private ... ``` This allows OpenDKIM to sign messages for any number of domains, using the correct key. Slick! #### NGINX Now that I have three of these identical VMs, I am ready to hook them up to the internet. On the way in, I will point `smtp-out.ipng.ch` to our NGINX cluster. I wrote about that cluster in a [[previous article]({{< ref "2023-03-17-ipng-frontends" >}})]. I will add a snippet there, that exposes these VMs behind a TCP loadbalancer like so: ``` pim@squanchy:~/src/ipng-ansible/roles/nginx/files/streams-available$ cat smtp-out.ipng.ch.conf upstream smtp_out { server smtp-out.chrma0.net.ipng.ch:587 fail_timeout=10s max_fails=2; server smtp-out.frggh0.net.ipng.ch:587 fail_timeout=10s max_fails=2 backup; server smtp-out.nlams2.net.ipng.ch:587 fail_timeout=10s max_fails=2 backup; } server { listen [::]:587; listen 0.0.0.0:587; proxy_pass smtp_out; } ``` I make use of the `backup` keyword, which will make the loadbalancer choose, if it's available, the primary server in `chrma0`. If it were to go down, no problem, two connection failures within ten seconds will make NGINX choose the alternative ones in `frggh0` or `nlams2`. #### IPng Site Local gateway When the `smtp-out` server receives the e-mail from the customer/client, it'll spool it and start to deliver it to the remote MX record. To do this, it'll create an outbound connection from its cozy spot within IPng Site Local (which, you will remember, is not connected directly to the internet). There are three redundant gateways in IPng Site Local (in Geneva, Brüttisellen and Amsterdam). If any of these were to go down for maintenance or fail, the network will use OSPF E1 to find the next closest default gateway. I wrote about how this entire european network is connected via three gateways that are self-repairing in this [[article]({{< ref "2023-03-11-mpls-core" >}})], in case you're curious. But, for the purposes of SMTP, it means that each of the internal `smtp-out` VMs will be seen by remote mailservers as NATted via ***one of these egress points***. This allows me to determine the SPF records in DNS. With that, I'm ready to share the publicly visible details for this service: ``` _spf.ipng.ch. 3600 IN TXT "v=spf1 include:_spf4.ipng.ch include:_spf6.ipng.ch ~all" _spf4.ipng.ch. 3600 IN TXT "v=spf1 ip4:46.20.246.112/28 ip4:46.20.243.176/28 ip4:94.142.245.80/29" "ip4:94.142.241.184/29 ip4:194.1.163.0/24 ~all" _spf6.ipng.ch. 3600 IN TXT "v=spf1 ip6:2a02:2528:ff00::/40 ip6:2a02:898:146::/48" "ip6:2001:678:d78::/48 ~all" smtp-out.ipng.ch. 3600 IN CNAME nginx0.ipng.ch. nginx0.ipng.ch. 600 IN A 194.1.163.151 nginx0.ipng.ch. 600 IN A 46.20.246.124 nginx0.ipng.ch. 600 IN A 94.142.241.189 nginx0.ipng.ch. 600 IN AAAA 2001:678:d78:7::151 nginx0.ipng.ch. 600 IN AAAA 2a02:2528:ff00::124 nginx0.ipng.ch. 600 IN AAAA 2a02:898:146::5 ``` To re-iterate one point: the _inbound_ path of the mail is via the redundant cluster of `nginx0` entrypoints, while the _outbound_ path will be seen from `gw0.chbtl0.ipng.ch`, `gw0.chplo0.ipng.ch` or `gw0.nlams3.ipng.ch`, which are all covered by the SPF records for IPv4 and IPv6. #### Bonus: opensmtpd on clients By the way, every single server (VM, hypervisor, router) at IPng Neworks will all use smtp-out to send e-mail. I use `opensmtpd` for that, and it's incredibly simple: ``` pim@squanchy:~$ cat /etc/mail/smtpd.conf table aliases file:/etc/mail/aliases table secrets file:/etc/mail/secrets listen on localhost action "local_mail" mbox alias action "outbound" relay host "smtp+tls://ipng@smtp-out.ipng.ch:587" auth mail-from "@ipng.ch" match from local for local action "local_mail" match from local for any action "outbound" pim@squanchy:~$ sudo cat /etc/mail/secrets ipng bastion: ``` {{< image width="120px" float="left" src="/assets/shared/lightbulb.svg" alt="Lightbulb" >}} What happens here is, every time this server `squanchy` wants to send an e-mail, it will use an SMTP session with TLS, on port 587, of the machine called `smtp-out.ipng.ch`, and it'll authenticate using the opensmtpd realm called `ipng`, which maps to a username:password tuple in the _secrets_ file. It will also rewrite the envelope to be always from `@ipng.ch`. As a best practice I organize my SMTP users by Ansible group. Squanchy is in the group `bastion`, hence its username. By doing it this way, I can make use of the DKIM and SPF, which makes all mails properly formatted, routed, signed and delivered. I love it, so much! ### 2. Inbound: smtp-in The `smtp-out` service I described in the previous section is completely standalone. That is to say, its purpose is only to receive submitted mail from humans and servers, sign it, spool it if need be, and deliver it. But users also want to deliver e-mail to me and my customers. For this, I'll build a second cluster of redundant _inbound_ mailservers: `smtp-in`. Here, the base setup is not too different from above, so I won't repeat it. I'll take three identical VMs, in three different datacenters, and install them with Debian and Postfix as well. But, contrary to the outbound servers, here I will make them listen to the _smtp_ port 25 and the _smtps_ port 465, and I'll turn off the ability to authenticate with SASL (and thereby, refuse to forward any e-mail that I'm not the MX record for), making `master.cf` look like this: ``` smtp inet n - y - - smtpd -o smtpd_sasl_auth_enable=no #submission inet n - y - - smtpd # -o syslog_name=postfix/submission # -o smtpd_tls_security_level=encrypt # -o smtpd_sasl_auth_enable=yes # -o smtpd_reject_unlisted_recipient=no # -o smtpd_client_restrictions=permit_sasl_authenticated,permit_mynetworks,reject # -o milter_macro_daemon_name=ORIGINATING smtps inet n - y - - smtpd -o syslog_name=postfix/smtps -o smtpd_tls_wrappermode=yes -o smtpd_sasl_auth_enable=no ``` Many of the `main.cf` attributes are the same, unsurprisingly the `myhostname` configuration option is set to `smtp-in.ipng.ch`, which is going to be expected to match the wildcard SSL certificate from the `smtpd_tls_cert_file` config option. The banner is a bit more telling, as it shows also the FQDN hostname (eg. `smtp-in.frggh0.net.ipng.ch`), helpful when debugging. ``` # Impose DNSBL restrictions at SMTP time smtpd_recipient_restrictions = permit_mynetworks, reject_invalid_helo_hostname, reject_non_fqdn_recipient, reject_unknown_recipient_domain, reject_unauth_pipelining, reject_unauth_destination, reject_rbl_client zen.spamhaus.org=127.0.0.[2..11], reject_rhsbl_sender dbl.spamhaus.org=127.0.1.[2..99], reject_rhsbl_helo dbl.spamhaus.org=127.0.1.[2..99], reject_rhsbl_reverse_client dbl.spamhaus.org=127.0.1.[2..99], warn_if_reject reject_rbl_client zen.spamhaus.org=127.255.255.[1..255], reject_rbl_client dnsbl-1.uceprotect.net, reject_rbl_client bl.0spam.org=127.0.0.[7..9], permit # Milter for rspamd smtpd_milters = inet:rspamd.net.ipng.ch:11332 milter_default_action = accept # PostSRSd sender_canonical_maps = tcp:localhost:10001 sender_canonical_classes = envelope_sender recipient_canonical_maps = tcp:localhost:10002 recipient_canonical_classes= envelope_recipient,header_recipient # Virtual domains virtual_alias_domains = hash:/etc/postfix/virtual-domains virtual_alias_maps = hash:/etc/postfix/virtual ``` The config arguably is quite compact. but I will hilight four specific pieces. ***DNSBL***: When connecting and receiving the envelope (ie. `MAIL FROM` and `RCPT TO` in the SMTP transaction), I'll ask Postfix to do a bunch of DNS blocklist lookups. Many sender domains, and infected hosts/networks are mapped in public DNSBL zones, notably [[Spamhaus](https://www.spamhaus.org/)], [[UCEProtect](http://www.uceprotect.net/en/index.php)], and [[0Spam](https://0spam.org/)] do a great job at identifying malicious and spammy domain-names and networks. So I'll tell Postfix to reject folks attempting to connect from these low-reputation places. ***Rspamd***: Here's where I hook up a redundant cluster of rspamd servers. Each e-mail, once accepted, will be routed through this milter, and after thinking about it a little bit, the Rspamd server will either answer: * ***greylisted***: where Rspamd recommends a _tempfail_ so the remote mailserver comes back after a few minutes after connecting for the first time, many spammers will not do this. * ***blocked***: if Rspamd finds the e-mail is egregious, it'll simply recommend a _permfail_ so Postfix immediately rejects it. * ***tagged***: if Rspamd is iffy about the e-mail, it may insert an `X-Spam` header, so that downstream mail clients like Thunderbird or Mail.app can decide for themselves to consider the e-mail junk or not. ***PostSRS***: This is a really useful feature which allows Postfix to safely _forward_ an e-mail to another mailhost. Perhaps best explained with an example, notably the aforementioned nerdsnipe from my buddy Jelle: 1. Let's say `jelle@luteijn.email` sends an e-mail to `event@frys-ix.net` for which IPng is the mailhost. 1. Jelle configured his SPF records to allow mail to come from either `ip4:185.36.229.0/24` or `ip6:2a07:cd40::/29`, and if it comes from neither of those, to hard fail the SPF check `-all`. 1. My spiffy `smtp-in.ipng.ch` receives this e-mail and decides to forward it internally by rewriting it to `foo@eritap.com`. 1. The mailserver for `eritap.com` now sees an e-mail coming From: `jelle@luteijn.email` going to its `foo@eritap.com`. It does an SPF check and concludes: Yikes! That mailserver `smtp-in.ipng.ch` is NOT authorized to send e-mail on behalf of Jelle, so reject it! A kitten gets hurt, which is obviously unacceptable. To handle this, PostSRSd detects when such a forward is about to happen, and rewrites the envelope From: header to be something that `smtp-in.ipng.ch` might be allowed to deliver mail for: something in the `@ipng.ch` domain! Using a secret (shared between the replicas of IPng's `smtp-in` cluster), it can insert a little cryptographic signature as it does this rewrite. In the example above, the e-mail from `jelle@luteijn.email` will be rewritten to an envelope such as `SRS0=CCIM=MT=luteijn.email=jelle@ipng.ch` and while hideous, it **is** in the `@ipng.ch` domain. If a bounce for this e-mail were to be generated, PostSRSd can also rewrite in reverse, re-assembling the original envelope From when sending the bounce on to Jelle's mailserver. I configure Postfix to do this using the _sender_ and _recipient_ canonical maps. I read these from a server running on localhost port 10001 and 10002 respectively. This is where PostSRSd does its magic. Oh, what's that I hear? The telephone is ringing! 1982 called, and it wants to change the title of [[RFC821](https://datatracker.ietf.org/doc/html/rfc821)] from SMTP to CMTP (**Convoluted Mail Transfer Protocol**). ***Virtual***: With all of that out of the way, I can now receive and forward aliased e-mails. I won't be using local mail delivery (to unix users on the local machine), but rather I will forward the mails for my local users onwards to what is called a redundant `maildrop` server. So for the virtualized part of the Postfix config, I have things like this: ``` pim@smtp-in-chrma0:~$ cat /etc/postfix/virtual-domains ublog.tech ublog.tech frys-ix.net frys-ix.net ipng.nl ipng.nl ipng.ch ipng.ch ... pim@smtp-in-chrma0:~$ cat /etc/postfix/virtual ## Virtual domain: ipng.ch postmaster@ipng.ch pim+postmaster@maildrop.net.ipng.ch hostmaster@ipng.ch pim+hostmaster@maildrop.net.ipng.ch abuse@ipng.ch pim+abuse@maildrop.net.ipng.ch pim@ipng.ch pim+ipng@maildrop.net.ipng.ch noreply@ipng.ch /dev/null ... ## Virtual domain: ipng.nl @ipng.nl @ipng.ch ## Virtual domain: frys-ix.net postmaster@frys-ix.net pim+postmaster@maildrop.net.ipng.ch hostmaster@frys-ix.net pim+hostmaster@maildrop.net.ipng.ch abuse@frys-ix.net pim+abuse@maildrop.net.ipng.ch noc@frys-ix.net pim+frysix@maildrop.net.ipng.ch,noc@eritap.com pim@frys-ix.net pim+frysix@maildrop.net.ipng.ch arend@frys-ix.net arend+frysix@eritap.com event@frys-ix.net someplace@example.com ... ``` The first file here `virtual_alias_domains`, simply explains to Postfix which domains it is to accept e-mail for. This avoids users trying to use it as a relay. If the domain is not listed in the lefthand side of the table, it's not welcome here. But then once Postfix knows it's supposed to be accepting e-mail for this domain, it will consult the `virtual_alias_maps` configuration. Here, I showed three domains, and a few features: * I can simply forward along `pim@ipng.ch` to `pim+ipng@maildrop.net.ipng.ch`. Cool. * I can toss the email by passing it to `/dev/null` (useful for things like `noreply@` and `nobody@`) * I can forward it to multiple recipients as well, for example `noc@frys-ix.net` goes to me and Eritap (hoi, Arend!) When such a forward happens, PostSRSd kicks in, and for that e-mail, the envelope rewrite will happen such that `smtp-in` can safely deliver this to even the strictest of SPF users. #### Why no NGINX ? There's an important technical reason for me not to be able to use an inbound loadbalancer, even though I'd love to frontend port 25 and 465 on IPng's nginx cluster. I have enabled the use of DNSBL, which implies that Postfix needs to know the remotely connecting IPv4 and IPv6 addresses. While for domain-based blocklists this is not important, for IP based ones like _zen.spamhaus.org_ it is critical. Therefore, I will assign a public IPv4 and IPv6 address to each of the machines in the cluster. They will be used in a round-robin way, and if one of them is down for a while, remote mail servers will automatically and gracefully use another replica. With that, the public DNS entries: ``` ublog.tech. 86400 IN MX 10 smtp-in.ipng.ch. ipng.nl. 86400 IN MX 10 smtp-in.ipng.ch. ipng.ch. 86400 IN MX 10 smtp-in.ipng.ch. smtp-in.ipng.ch. 60 IN A 46.20.246.125 smtp-in.ipng.ch. 60 IN A 94.142.245.85 smtp-in.ipng.ch. 60 IN A 194.1.163.141 smtp-in.ipng.ch. 60 IN AAAA 2a02:2528:ff00::125 smtp-in.ipng.ch. 60 IN AAAA 2a02:898:146:1::5 smtp-in.ipng.ch. 60 IN AAAA 2001:678:d78:6::141 ``` ### 3. Dovecot: maildrop Remember when I said that mail to `pim@ipng.ch` is forwarded to `pim+ipng@maildrop.net.ipng.ch`? Doing this allows me to have replicated, fully redundant, IMAP servers! As it turns out, Dovecot, a very popular open source pop3/imap server, has the ability to do realtime synchronization between multiple machines serving the same user. On these servers, I'll start with enabling Postfix only using the `smtp` and `smtps` transport in `master.cf`. The maildrop servers will be entirely within IPng Site Local, and cannot be reached from the internet directly, just the same as the `smtp-out` server replicas. Postfix on the server receives mail from the `smtp-in` servers as the final destination for an e-mail. It does this very similar to the `smtp-in` server pool I described above, with two notable differences: 1. It does not need to do DNSBL lookups or spam analysis -- those have already happened upstream from these maildrop servers by the `smtp-in` servers. That's also why these can be safely tucked away in IPng Site Local. 1. Their virtual maps point to what is called an ***LMTP***: Local Mail Transport Protocol, where I'll ask Postfix to pump them into a redundalty replicated Dovecot pair. ``` # Completely virtual virtual_alias_maps = hash:/etc/postfix/virtual-maildrop virtual_mailbox_domains = maildrop.net.ipng.ch virtual_transport = lmtp:unix:private/dovecot-lmtp pim@maildrop0-chbtl0:$ cat /etc/postfix/virtual-maildrop pim@maildrop.net.ipng.ch pim ``` What I've done here is define only one `virtual_mailbox_domains` entry, for which I look up the users in the `virtual_alias_maps` and use a `virtual_transport` to deliver the enduser (`pim`) to a unix domain socket in `/var/spool/postfix/private/dovecot-lmtp`. Once again, mail servers are super simple after you've spent ten hours reading configuration manuals and RFCs and asked at least three other people how they did theirs.... Super... Simple! #### Dovecot By default, Dovecot ships with a very elaborate configuration file hierarchy. I decide to replace it with an autogenerated one from Ansible that has fewer includes (namely: none at all). Here's the features that I want to enable in Dovecot: * ***UserDB***: To define username, password and mail directory for users. * ***LMTP***: To be a local recepticle for the Postfix delivery * ***IMAP***: To serve SSL enabled IMAP to mail clients like Mail.app, Thunderbird, Roundcube, etc. * ***Replicator***: To replicate mailboxes between pairs of Dovecot servers. * ***Sieve***: To allow users to create mail filters using the Sieve protocol. Starting from the easier bits, here's how I configure the User Database in `dovecot.conf`: ``` passdb { driver = passwd-file args = username_format=%n /etc/dovecot/maildrop-users } userdb { driver = passwd-file args = username_format=%n /etc/dovecot/maildrop-users default_fields = uid=vmail gid=vmail home=/var/dovecot/users/%n } mail_plugins = $mail_plugins notify push_notification replication mail_location = mdbox:~/mdbox ``` I can add a user `pim` with an encrypted password from `doveadm pw` like so: ``` pim@maildrop0-chbtl0:/etc/dovecot$ sudo cat maildrop-users ... pim:{CRYPT}$2y$:::: ``` Due to the `passdb` option, this user can authenticate with username and password, and due to the `userdb` option, this user receives a mailbox homedir in the specified location. One important observation is that the _unix_ user is `vmail:vmail` for every mailbox. This is pretty cool as it allows the whole mail delivery system to be virtualized under Dovecot's guidance. Slick! There are two tried-and-tested mailbox formats: Maildir and mbox. mbox is one giant file per mail folder, and can be expensive to search and sort and delete mails out of. Maildir is cheaper to search and sort and delete, but is essentially one file per e-mail, which is bulky. Dovecot has its own high performance mailbox, which is the best of both worlds: an indexed append-only chunked mail format called mdbox. I learned more about the options and trade offs reading [[this doc](https://doc.dovecot.org/admin_manual/mailbox_formats/dbox/)]. #### Dovecot: LMTP The following `dovecot.conf` snippet ties Postfix into Dovecot: ``` protocols = $protocols lmtp protocol lmtp { mail_plugins = $mail_plugins sieve } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { mode = 0660 user = postfix group = postfix } } ``` Recall that in Postfix above, the `virtual_transport` field specified the same location. This is how user `pim` gets mail handed to Dovecot. One other tidbit here is that the LMTP protocol enables a plugin called `sieve`. What this does, is upon receipt of each e-mail, a list of filters is run through, on the server side! It is here that I can tell Dovecot that some mail goes to different folders and sub-folders, some might be forwarded, marked read or discarded entirely. I'll get to that in a minute. #### Dovecot: IMAP Then, I enable SSL enabled IMAP in `dovecot.conf`: ``` disable_plaintext_auth = yes auth_mechanisms = plain login protocols = $protocols imap protocol imap { mail_max_userip_connections = 50 mail_plugins = $mail_plugins imap_sieve } service imap-login { inet_listener imap { port = 0 ## Disabled } inet_listener imaps { port = 993 } } ``` With this snippet, I instruct Dovecot to disable any plain-text authentication, and use either `plain` or `login` challenges to authenticate users. I'll disable the un-encrypted IMAP listener by setting its port to 0, and I'll allow for an IMAP+SSL listener on the common port 993, which will be presenting a `*.ipng.ch` wildcard certificate that's shared between all sorts of services at IPng. #### Dovecot: Replication And now for something really magical. Dovecot can be instructed to replicate in multi-master (ie. read/write) mailboxes to remote machines also running Dovecot. This is called ***dsync*** and it's hella cool! In reading the [[docs](https://doc.dovecot.org/configuration_manual/replication/)], I take note that the same user should be directed to a stable replica in normal use, but changes do not get lost even if the same user modifies mails simultaneously on both replicas, some mails just might have to be redownloaded in that case. The replication is done by looking at Dovecot index files (not what exists in filesystem), so no mails get lost due to filesystem corruption or an accidental `rm -rf`, they will simply be replicated back. This is amazing!! The configuration for it is remarkably straight forward: ``` mail_plugins = $mail_plugins notify replication # Replication details replication_max_conns = 10 replication_full_sync_interval = 1h service aggregator { fifo_listener replication-notify-fifo { user = vmail group = vmail mode = 0660 } unix_listener replication-notify { user = vmail group = vmail mode = 0660 } } # Enable doveadm replicator commands service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0660 user = vmail group = vmail } } doveadm_port = 63301 doveadm_password = service doveadm { vsz_limit=512 MB inet_listener { port = 63301 } } plugin { mail_replica = tcp:maildrop0.ddln0.net.ipng.ch } ``` To try to explain this - The first service, the `aggregator` opens some notification FIFOs that will notify listeners of new replication events. Then, Dovecot will start a process called `replicator`, which gets these cues when there is work to be done. It will connect to a `mail_replica` on another host, on the `doveadm_port` (in my case 63301) which is protected by a shared password. And with that, all e-mail that is delivered via **LMTP** on this machine, is both retrievable via **IMAPS** but also gets copied to the remote machine `maildrop0.ddln0.net.ipng.ch` (and in _its_ configuration, it'll synchronize mail to `maildrop0.chbtl0.net.ipng.ch`). Nice! #### Dovecot: Sieve Having a flat mailbox is just no fun (unless you're using GMail, in which case: tolerable). Enter _Sieve_, described in [[RFC5228](https://datatracker.ietf.org/doc/html/rfc5228)]. Scripts written in Sieve are executed during final delivery, when the message is moved to the user-accessible mailbox. In systems where the Mail Transfer Agent (MTA) does final delivery, such as traditional Unix mail, it is reasonable to filter when the MTA deposits mail into the user's mailbox. ``` protocols = $protocols sieve plugin { sieve = ~/.dovecot.sieve sieve_global_path = /etc/dovecot/sieve/default.sieve sieve_dir = ~/sieve sieve_global_dir = /etc/dovecot/sieve/ sieve_extensions = +editheader sieve_before = /etc/dovecot/sieve/before.d sieve_after = /etc/dovecot/sieve/after.d } plugin { sieve_plugins = sieve_imapsieve sieve_extprograms # From elsewhere to Junk folder imapsieve_mailbox1_name = Junk imapsieve_mailbox1_causes = COPY imapsieve_mailbox1_before = file:/etc/dovecot/sieve/report-spam.sieve # From Junk folder to elsewhere imapsieve_mailbox2_name = * imapsieve_mailbox2_from = Junk imapsieve_mailbox2_causes = COPY imapsieve_mailbox2_before = file:/etc/dovecot/sieve/report-ham.sieve sieve_pipe_bin_dir = /etc/dovecot/sieve sieve_global_extensions = +vnd.dovecot.pipe } ``` This is a mouthful, but only because it's hella cool. By default, each mailbox will have a `.dovecot.sieve` file that is consulted at each delivery. If no file exists there, the default sieve will be used. But also, some sieve filters might happen either `sieve_before` the users' one is called, or `sieve_after`. Then, in the plugin I create two specific triggers: 1. if a file is copied to the Junk folder, I will run it through a script called `report-spam.sieve`. 1. similarly, if it is moved *out of* the Junk folder, I'll run the `report-ham.sieve` script. Using an `rspamc` client, I can wheel over the cluster of Rspamd servers one by one and offer them these two events (the train-spam and train-ham are similar, so I'll only show one): ``` pim@maildrop0-chbtl0:~$ cat /etc/dovecot/sieve/report-spam.sieve require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment", "variables"]; if environment :matches "imap.email" "*" { set "email" "${1}"; } pipe :copy "train-spam.sh" [ "${email}" ]; pim@maildrop0-chbtl0:~$ cat /etc/dovecot/sieve/train-spam.sh logger learning spam /usr/bin/rspamc -h rspamd.net.ipng.ch:11332 learn_spam ``` Users of Dovecot can now add their Sieve configs to their mailbox: ``` pim@maildrop0-chbtl0:~$ sudo ls -la /var/dovecot/users/pim/ lrwxrwxrwx 1 vmail vmail 19 Apr 2 17:27 .dovecot.sieve -> sieve/ipng_v1.sieve -rw------- 1 vmail vmail 2113 Mar 29 11:35 .dovecot.sieve.log -rw------- 1 vmail vmail 5001 May 14 14:34 .dovecot.svbin drwx------ 4 vmail vmail 4096 May 17 16:02 mdbox drwx------ 3 vmail vmail 4096 May 14 14:30 sieve ``` but seeing as (a) it's tedious to have to edit these files on multiple dovecot replicas, and (b) my users will not receive access to `vmail` user in order to actually do that, as it would be a security risk, I need one more thing. #### Dovecot: IMAP Sieve Dovecot has an implementation of a replication-aware Sieve filter editor called `managesieve`: ``` service managesieve-login { inet_listener sieve { port = 4190 } } service managesieve { process_limit = 256 } protocol sieve { } ``` It will use the IMAP credentials to allow users to edit their _Sieve_ filter online. For example, Thunderbird has a plugin for it, which does syntax checking and what-not. When the filter is edited, it is syntax checked, compiled and replicated to the other Dovecot instance. {{< image src="/assets/smtp/sieve_thunderbird.png" alt="Sieve Thunderbird" >}} #### NGINX I have an imap server and a mangesieve server, redundantly running on two Dovecot machines. I recall reading in the Dovecot manual that it is slightly preferable to have users go to a consistent replica and not bounce around between them. Luckily, I can do exactly that using the NGINX frontends: ``` upstream imap { server maildrop0.chbtl0.net.ipng.ch:993 fail_timeout=10s max_fails=2; server maildrop0.ddln0.net.ipng.ch:993 fail_timeout=10s max_fails=2 backup; } server { listen [::]:993; listen 0.0.0.0:993; proxy_pass imap; } upstream sieve { server maildrop0.chbtl0.net.ipng.ch:4190 fail_timeout=10s max_fails=2; server maildrop0.ddln0.net.ipng.ch:4190 fail_timeout=10s max_fails=2 backup; } server { listen [::]:4190; listen 0.0.0.0:4190; proxy_pass sieve; } ``` I keep port 993 for `maildrop` as well as port 587 for `smtp-out` unfiltered on the NGINX cluster. I'm a little bit more protective of the `managesieve` service, so port 4190 is allowed only when users are connected to the VPN or the internal (office/home) network. Now, you'll recall that in the `smtp-in` servers, I forward mail to `pim@maildrop.net.ipng.ch`, for which the redundant Dovecot servers are both accepting mail. On the way in, I can see to it that the primary replica is used , by giving it a slightly lower preference in DNS MX records: ``` maildrop.net.ipng.ch. 300 IN MX 10 maildrop0.chbtl0.net.ipng.ch. maildrop.net.ipng.ch. 300 IN MX 20 maildrop0.ddln0.net.ipng.ch. imap.ipng.ch. 60 IN CNAME nginx0.ipng.ch. nginx0.ipng.ch. 600 IN A 194.1.163.151 nginx0.ipng.ch. 600 IN A 46.20.246.124 nginx0.ipng.ch. 600 IN A 94.142.241.189 nginx0.ipng.ch. 600 IN AAAA 2001:678:d78:7::151 nginx0.ipng.ch. 600 IN AAAA 2a02:2528:ff00::124 nginx0.ipng.ch. 600 IN AAAA 2a02:898:146::5 ``` This will make the `smtp-in` hosts prefer to use the `chbtl0` maildrop replica when it's available. If ever it were to go down, they will automatically fail over and use `ddln0`, which will replicate back any changes while `chbtl0` is down for maintenance or hardware failure. On the way to out, the nginx cluster will prefer to use `chbtl0` as well, as it has marked the `ddln0` replica as `backup`. ### 4. Webmail: Roundcube Now that I have all of the infrastructure up and running, I thought I'd put some icing on the cake with Roundcube, a web-based IMAP email client. Roundcube's most prominent feature is the pervasive use of Ajax technology. It also comes with an online Sieve editor, and runs in Docker. What more can I ask for? Installing it is really really easy in my case. Since I have an nginx cluster to frontend it and do the SSL offloading, I choose the simplest version with the following `docker-compose.yaml`: ``` version: '2' services: roundcubemail: image: roundcube/roundcubemail:latest container_name: roundcubemail volumes: - ./www:/var/www/html - ./db/sqlite:/var/roundcube/db ports: - 9002:80 environment: - ROUNDCUBEMAIL_DB_TYPE=sqlite - ROUNDCUBEMAIL_SKIN=elastic - ROUNDCUBEMAIL_DEFAULT_HOST=ssl://maildrop0.net.ipng.ch - ROUNDCUBEMAIL_DEFAULT_PORT=993 - ROUNDCUBEMAIL_SMTP_SERVER=tls://smtp-out.net.ipng.ch - ROUNDCUBEMAIL_SMTP_PORT=587 ``` There's a small snag, in that by default the SMTP user and password are expected to be the same as for the IMAP server, which is not the case for my design. So, I create a user `roundcube` on the `smtp-out` cluster and give it a suitable password. I nose around a little bit, and decide my preference is to have _threaded_ view by default, and I also enable the `managesieve` plugin: ``` $config['log_driver'] = 'stdout'; $config['zipdownload_selection'] = true; $config['des_key'] = ''; $config['enable_spellcheck'] = true; $config['spellcheck_engine'] = 'pspell'; $config['smtp_user'] = 'roundcube'; $config['smtp_pass'] = ''; $config['plugins'] = array('managesieve'); $config['managesieve_host'] = 'tls://maildrop0.net.ipng.ch:4190'; $config['default_list_mode'] = 'threads'; ``` I start the docker containers, and very quickly after, Roundcube shoots to life. I can expose it behind the nginx cluster, while keeping it accessible only for VPN + office/home network users: ``` server { listen [::]:80; listen 0.0.0.0:80; server_name webmail.ipng.ch webmail.net.ipng.ch webmail; access_log /var/log/nginx/webmail.ipng.ch-access.log; include /etc/nginx/conf.d/ipng-headers.inc; location / { return 301 https://webmail.ipng.ch$request_uri; } } geo $allowed_user { default 0; include /etc/nginx/conf.d/geo-ipng.inc; } server { listen [::]:443 ssl http2; listen 0.0.0.0:443 ssl http2; ssl_certificate /etc/certs/ipng.ch/fullchain.pem; ssl_certificate_key /etc/certs/ipng.ch/privkey.pem; include /etc/nginx/conf.d/options-ssl-nginx.inc; ssl_dhparam /etc/nginx/conf.d/ssl-dhparams.inc; server_name webmail.ipng.ch; access_log /var/log/nginx/webmail.ipng.ch-access.log upstream; include /etc/nginx/conf.d/ipng-headers.inc; if ($allowed_user = 0) { rewrite ^ https://ipng.ch/ break; } location / { proxy_pass http://docker0.frggh0.net.ipng.ch:9002; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; } } ``` The configuration has one neat trick in it -- it uses the `geo` module in NGINX to assert that the client address is used to set the value of `allowed_user`. It will be 1 if the client connected from any network defined in the `geo-ipng.inc` file, and 0 otherwise. I then use it to bounce unwanted visitors back to the main [[website](https://ipng.ch/)], and expose Roundcube for those that are welcome. While the Roundcube instance is not replicated, it's also non-essential. I will be using Thunderbird, Mail.app and other clients more regularly than Roundcube. It may just be handy in a pinch to either check mail using a browser, but also to edit the _Sieve_ filters easily. In my defense, considering roundcube is pretty much stateless, I can actually just run multiple copies of it on a few docker hosts at IPng -- then in the nginx configs I might use a similar construct as for the `maildrop` and `smtp-out` services, with a primary and hot standby. But that will be for that one day that the docker host in Lille dies AND I decided I absolutely require Roundcube precisely on that day :)