# Private server setup

This is documentation of a setup used for uberspace.net from 2020 to 2024: a private multi-user system with commonly used network services: email, XMPP, web publishing, and a few others. See also: a simpler server setup, based on this one, but simplified.

My motivation for setting such a system is use of quality services with control over them, as well as fun and learning. The purpose of this documentation is to share some tips and experience with others, and to simplify potential future migration.

The guiding principles are use of libre, relatively well-maintained, reliable, secure, and lightweight software; open, standardised, federated, commonly supported network protocols. Generally that which works well, is useful, and will likely stay that way for a while.

## Preparation

### Prerequisites

- A domain name, costs about $15 per year.
- A server with an IPv4 address, preferably a /64 IPv6 subnet, reverse DNS (DNSSEC for those would be nice, but rarely deployed as of 2020). 5 GB of disk space and 512 MB of main memory should suffice, but 2--4 times as much would be more comfortable. As for bandwidth, here it is about 2 GB/month—mostly because of web crawlers—but may vary quite a bit. Costs perhaps $35 to $80 per year for a remote VM.
- Time: a day or two for the initial setup or migration, optionally a few more days for tweaking.
- Thousands of years of scientific and technological advancement, including more recent work by many individuals and some organizations on libre software and infrastructures used here directly. Available freely.

Additional maintainers, backup servers, and domain names may be useful, but their availability is not assumed.

### Operating system and software choices

Debian GNU/Linux is a good, well-maintained system; its stable branch is appropriate for low-maintenance systems.

The software used here is fairly common and should be easily available from system repositories on other common POSIX systems, though the versions, default configuration, paths, and other minor details may differ. Which is one of the reasons why this is a human-readable document, and not just a collection of ansible roles: it should be relatively easy to adapt for other systems and requirements.

The software is often listed along with alternatives. Security track records are available for some of it: knot DNS CVEs, nginx CVEs, cgit CVEs, Dovecot CVEs, Postfix CVEs, mlmmj CVEs, MHOnArc CVEs, Prosody CVEs, znc CVEs, fail2ban CVEs. Additionally, there are vulnerabilities in other common daemons (e.g., OpenSSH CVEs, Rsyslog CVEs), libraries (OpenSSL CVEs, plenty of others), kernel itself (Linux kernel CVEs).

### On migration

If migrating (and not just setting a new server), services can be moved and tested one by one, by changing corresponding DNS records. Some migration tips are included here, the described system changed hosters a couple of times.

Ideally there shouldn't be any sensitive data stored unencrypted on a server, and passwords should be changed after migration. Though it's not always practical, and data wiping on the old server may be needed. `shred(1)` might work, though not on file systems with journaling, snapshots, etc. One can try to overwrite disks with random data using either `rand(1ssl)` (`openssl rand -out random-file $(( 2**30 ))`) or `dd(1)` (`dd if=/dev/urandom of=random-file bs=1M count=1024`), though it won't overwrite bad sectors. And usually it's not practical to destroy the disks physically either. But a combination of all the approaches (and especially not storing or even having any sensitive data) reduces chances of a data breach.

## Initial setup

It is a good idea to follow RFC 1178 (Choosing a Name for Your Computer). The theme here is pastries: "muffin", "cupcake", and now "tart".

Usually root access is given initially, so the first thing to do is to add a user or few, and SSH keys for them:

    root@tart:~# adduser defanor
    root@tart:~# usermod -G sudo defanor
    root@tart:~# su defanor
    defanor@tart:/root$ mkdir ~/.ssh
    defanor@tart:/root$ vi ~/.ssh/authorized_keys

Then log out, log in as the user (to ensure that it's set properly), and edit `/etc/ssh/sshd_config` to prohibit password authentication and root login:

    --- sshd_config 2020-12-15 15:38:05.088873498 +0100
    +++ /etc/ssh/sshd_config        2020-12-16 01:50:40.567136824 +0100
    @@ -29,7 +29,7 @@
     # Authentication:

     #LoginGraceTime 2m
    -PermitRootLogin yes
    +PermitRootLogin no
     #StrictModes yes
     #MaxAuthTries 6
     #MaxSessions 10
    @@ -53,7 +53,7 @@
     #IgnoreRhosts yes

     # To disable tunneled clear text passwords, change to no here!
    -#PasswordAuthentication yes
    +PasswordAuthentication no
     #PermitEmptyPasswords no

     # Change to yes to enable challenge-response passwords (beware issues with

The `root` user's password should also be "locked", to avoid its unauthorized exploitation (for instance, dovecot with PAM authentication lets postfix to authenticate the "root" user with blank password, even though dovecot itself won't allow authentication with a blank password). That is done with `sudo passwd -l root`. Though on physical servers, it may be preferable to set a strong passphrase: on a failure to boot completely, the system may prompt for the root user's password specifically.

Update package lists and the system at once: `sudo apt update && sudo apt upgrade`.

Whether for `resolv.conf(5)` or for a local caching DNS server, the choice of name servers is challenging: it's good if a hoster (or an ISP) provides a decent DNS server, but it's useful to set a backup server anyway, especially if there will be outgoing connections or use of email blacklists. One should be careful and check that those are fast, don't cut out less common record types, support DNSSEC. Many are rather poor, but it isn't nice to use (and load) root servers directly. It isn't great for everyone to depend on a few major companies' servers either, but fortunately there's quite a few options, and they can be combined easily: e.g., hoster's DNS, followed by OpenDNS, Level3, Hurricane Electric, and/or Verisign. Generally larger networking-related companies tend to be relatively reliable, and I'd skip a couple of largest ones -- both to reduce centralisation, and because those are rather unpleasant companies (Google and Cloudflare) at the time of writing. `/etc/resolv.conf` may be updated by different means, but when relying on DHCP for network configuration, it's updated by `dhclient(1)`. Either static or additional DNS servers can be set in `/etc/dhcp/dhclient.conf`:

    --- dhclient.conf       2020-12-20 22:15:32.997683197 +0000
    +++ /etc/dhcp/dhclient.conf     2023-06-16 16:15:02.682698681 +0000
    @@ -19,6 +19,11 @@
            netbios-name-servers, netbios-scope, interface-mtu,
            rfc3442-classless-static-routes, ntp-servers;

    +# Try the local nameserver first.
    +prepend domain-name-servers 127.0.0.1;
    +# Use OpenDNS as a backup.
    +append domain-name-servers 208.67.222.222;
    +
     #send dhcp-client-identifier 1:0:a0:24:ab:fb:9c;
     #send dhcp-lease-time 3600;
     #supersede domain-name "fugue.com home.vix.com";

Set time zone to UTC (or anything else, but plain UTC appears to cause least amounts of headache) with `sudo dpkg-reconfigure tzdata`. By default rsyslog would write files in RFC 3339, but its default configuration on Debian sets the format to "traditional"; comment it out in `/etc/rsyslog.conf` to get RFC 3339 timestamps (which are particularly useful for log sharing, since they include time zone):

    --- rsyslog.conf        2021-04-20 13:23:42.240862482 +0000
    +++ /etc/rsyslog.conf   2021-04-20 13:24:18.826187613 +0000
    @@ -29,7 +29,7 @@
     # Use traditional timestamp format.
     # To enable high precision timestamps, comment out the following line.
     #
    -$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
    +#$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

     #
     # Set the default permissions for all log files.

Though that was removed in Debian 12 (bookworm), so now the default Debian configuration is good, unless you want to see priorities; for that, RSYSLOG_SyslogProtocol23Format should be set, matching RFC 5424. And rsyslogd is deprecated since Debian 12, with it defaulting to using just systemd's journald.

While I tend to edit remote files using local Emacs and TRAMP, Emacs is nice to have on the server as well: `apt install emacs`, and `echo 'export EDITOR=emacs' >> ~/.profile`.

## Authoritative name server

DNSSEC support is both useful by itself and needed for DANE (RFC 6693). RFC 2136 (dynamic DNS updates) support is preferable for automation of X.509 certificate acquisition with ACME (as covered in the next section). Some of the suitable options are BIND and knot DNS. Install knot DNS and generate a key we'll use for dynamic updates:

    $ sudo apt install knot knot-dnsutils
    $ sudo keymgr -t certbot hmac-sha512

Edit `/etc/knot/knot.conf` (here and further in the text, replace `SECRET_TSIG_KEY` with the generated key), here we are also setting `ns6.gandi.net` as a secondary (slave) nameserver, except for the ACME challenge subdomain (since we need to update that quickly, and gandi may lag by a day). Listening for connections to 127.0.0.2 and to public addresses, but leaving 127.0.0.1:53 and [::1]:53 for a local caching DNS server (alternatively, it could be set to listen on a port other than 53 for local connections).

    --- /etc/knot/knot.conf.dpkg-dist       2023-04-04 11:54:46.000000000 +0000
    +++ /etc/knot/knot.conf 2023-06-16 15:54:06.525946516 +0000
    @@ -4,8 +4,7 @@
     server:
         rundir: "/run/knot"
         user: knot:knot
    -    automatic-acl: on
    -#    listen: [ 127.0.0.1@53, ::1@53 ]
    +    listen: [ 127.0.0.2@53, 157.90.29.18@53, 2a01:4f8:1c0c:73c7::1@53 ]

     log:
       - target: syslog
    @@ -14,23 +13,71 @@
     database:
         storage: "/var/lib/knot"

    +
     remote:
    -#  - id: secondary
    +#  - id: slave
     #    address: 192.168.1.1@53
     #
    -#  - id: primary
    +#  - id: master
     #    address: 192.168.2.1@53

    +key:
    +  - id: certbot
    +    algorithm: hmac-sha512
    +    secret: SECRET_TSIG_KEY
    +
    +acl:
    +  - id: update_acl
    +    address: [127.0.0.1, ::1]
    +    action: update
    +    key: certbot
    +
    +#  - id: acl_slave
    +#    address: 192.168.1.1
    +#    action: transfer
    +
    +#  - id: acl_master
    +#    address: 192.168.2.1
    +#    action: notify
    +
     template:
       - id: default
         storage: "/var/lib/knot"
         file: "%s.zone"

    +# https://docs.gandi.net/en/domain_names/advanced_users/secondary_nameserver.html
    +remote:
    +  - id: gandi
    +    address: [217.70.177.40, 2001:4b98:d:1::40]
    +
    +acl:
    +  - id: gandi_slave_acl
    +    address: [217.70.177.40, 2001:4b98:d:1::40]
    +    action: transfer
    +
     zone:
    -#    # Primary zone
    +  - domain: uberspace.net
    +    file: uberspace.net.zone
    +    notify: gandi
    +    acl: [update_acl, gandi_slave_acl]
    +    zonefile-load: difference
    +    dnssec-signing: on
    +    semantic-checks: on
    +
    +    # ACME challenge is delegated to a separate zone in order to avoid
    +    # propagation delays.
    +  - domain: _acme-challenge.uberspace.net
    +    file: _acme-challenge.uberspace.net.zone
    +    acl: [update_acl]
    +    dnssec-signing: on
    +    semantic-checks: on
    +
    +#    # Master zone
     #  - domain: example.com
    -#    notify: secondary
    +#    notify: slave
    +#    acl: acl_slave

    -#    # Secondary zone
    +#    # Slave zone
     #  - domain: example.net
    -#    master: primary
    +#    master: master
    +#    acl: acl_master

Add records into the zone file, `/var/lib/knot/uberspace.net.zone`. Fingerprints for `SSHFP` records can be obtained with `ssh-keygen -r uberspace.net`, and a DKIM key will be in `/etc/dkimkeys/tart2020.txt` once we'll set OpenDKIM (see below; the key itself is not included here for being too long); `tlsa311._dane.uberspace.net.` is referenced in many `CNAME` records, but will be set in the next section, once we'll have an X.509 certificate.

    uberspace.net.      	10800	SOA	tart.uberspace.net. hostmaster.uberspace.net. 123 43200 7200 2419200 86400
    uberspace.net.      	10800	NS	ns6.gandi.net.
    uberspace.net.      	10800	NS	tart.uberspace.net.
    uberspace.net.      	10800	A	157.90.29.18
    uberspace.net.      	10800	AAAA	2a01:4f8:1c0c:73c7::1
    uberspace.net.      	10800	MX	10 tart.uberspace.net.
    uberspace.net.      	10800	TXT	"v=spf1 a mx ~all"
    uberspace.net.      	10800	CAA	0 issue "letsencrypt.org;validationmethods=dns-01"
    _acme-challenge.uberspace.net.	10800	NS	tart.uberspace.net.
    _acme-challenge.uberspace.net.	10800	DS	29089 13 2 DAE95C79F1ED7322BCBCE0795C039B7A908FC99DEF295FE94F1DC33D9861690D
    _acme-challenge.uberspace.net.	10800	DS	29089 13 4 5934D1B227023FEF257234FBCB2C4E13D6054E62B6A900E1DA1C3FAE81218826AAE022F1916DB3A3151ED0E5CB0B2ABF
    _dmarc.uberspace.net.	10800	TXT	"v=DMARC1; p=quarantine"
    _adsp._domainkey.uberspace.net.	10800	TXT	"dkim=all"
    tart2020._domainkey.uberspace.net. 10800	TXT	"v=DKIM1; h=sha256; k=rsa; p=LONG_KEY_HERE"
    _143._tcp.uberspace.net.	10800	CNAME	tlsa311._dane.uberspace.net.
    _25._tcp.uberspace.net.	10800	CNAME	tlsa311._dane.uberspace.net.
    _443._tcp.uberspace.net.	10800	CNAME	tlsa311._dane.uberspace.net.
    _5222._tcp.uberspace.net.	10800	CNAME	tlsa311._dane.uberspace.net.
    _5269._tcp.uberspace.net.	10800	CNAME	tlsa311._dane.uberspace.net.
    _587._tcp.uberspace.net.	10800	CNAME	tlsa311._dane.uberspace.net.
    _993._tcp.uberspace.net.	10800	CNAME	tlsa311._dane.uberspace.net.
    _imap._tcp.uberspace.net.	10800	SRV	10 5 143 tart.uberspace.net.
    _imaps._tcp.uberspace.net.	10800	SRV	10 5 993 tart.uberspace.net.
    _submission._tcp.uberspace.net.	10800	SRV	10 5 587 tart.uberspace.net.
    _xmpp-client._tcp.uberspace.net. 10800	SRV	10 5 5222 tart.uberspace.net.
    _xmpp-server._tcp.uberspace.net. 10800	SRV	10 5 5269 tart.uberspace.net.
    _xmppconnect.uberspace.net.	10800	TXT	"_xmpp-client-xbosh=https://xmpp.uberspace.net/http-bind"
    _xmppconnect.uberspace.net.	10800	TXT	"_xmpp-client-websocket=wss://xmpp.uberspace.net/xmpp-websocket"
    _5269._tcp.conference.uberspace.net. 10800	CNAME	tlsa311._dane.uberspace.net.
    _xmpp-server._tcp.conference.uberspace.net. 10800	SRV	10 5 5269 tart.uberspace.net.
    defanor.uberspace.net.	10800	CNAME	tart.uberspace.net.
    _443._tcp.defanor.uberspace.net. 10800	CNAME	tlsa311._dane.uberspace.net.
    git.uberspace.net.  	10800	CNAME	tart.uberspace.net.
    _443._tcp.git.uberspace.net.	10800	CNAME	tlsa311._dane.uberspace.net.
    lists.uberspace.net.	10800	A	157.90.29.18
    lists.uberspace.net.	10800	AAAA	2a01:4f8:1c0c:73c7::1
    lists.uberspace.net.	10800	MX	10 tart.uberspace.net.
    lists.uberspace.net.	10800	TXT	"v=spf1 redirect=uberspace.net"
    _443._tcp.lists.uberspace.net.	10800	CNAME	tlsa311._dane.uberspace.net.
    paranoia.uberspace.net.	10800	CNAME	tart.uberspace.net.
    _443._tcp.paranoia.uberspace.net. 10800	CNAME	tlsa311._dane.uberspace.net.
    paste.uberspace.net.	10800	CNAME	tart.uberspace.net.
    _443._tcp.paste.uberspace.net.	10800	CNAME	tlsa311._dane.uberspace.net.
    tart.uberspace.net. 	10800	SSHFP	1 2 210A803DE18A5652E77AE4871A31BD07D272725B145A709F031F867402BA49EB
    tart.uberspace.net. 	10800	SSHFP	2 2 15431F1E626734EB9B8490EDA624E557493920F1FAF22FC7845E3971F2973DEB
    tart.uberspace.net. 	10800	SSHFP	3 2 04D5812158B0B61F2808845370DE5C137F75224B833B85A0AE76127CBF642433
    tart.uberspace.net. 	10800	SSHFP	4 2 A05343607D0886DF55A8E85BA4ADD9D82C17488C5700FD6B35C96DB41B34B276
    tart.uberspace.net. 	10800	A	157.90.29.18
    tart.uberspace.net. 	10800	AAAA	2a01:4f8:1c0c:73c7::1
    _143._tcp.tart.uberspace.net.	10800	CNAME	tlsa311._dane.uberspace.net.
    _25._tcp.tart.uberspace.net.	10800	CNAME	tlsa311._dane.uberspace.net.
    _443._tcp.tart.uberspace.net.	10800	CNAME	tlsa311._dane.uberspace.net.
    _5222._tcp.tart.uberspace.net.	10800	CNAME	tlsa311._dane.uberspace.net.
    _5269._tcp.tart.uberspace.net.	10800	CNAME	tlsa311._dane.uberspace.net.
    _587._tcp.tart.uberspace.net.	10800	CNAME	tlsa311._dane.uberspace.net.
    _993._tcp.tart.uberspace.net.	10800	CNAME	tlsa311._dane.uberspace.net.
    www.uberspace.net.  	10800	CNAME	tart.uberspace.net.
    _443._tcp.www.uberspace.net.	10800	CNAME	tlsa311._dane.uberspace.net.
    xmpp.uberspace.net.  	10800	CNAME	tart.uberspace.net.
    _443._tcp.xmpp.uberspace.net.	10800	CNAME	tlsa311._dane.uberspace.net.

And `/var/lib/knot/_acme-challenge.uberspace.net.zone`, a separate zone for ACME DNS challenges using only the primary nameserver:

    _acme-challenge.uberspace.net.	10800	SOA	tart.uberspace.net. hostmaster.uberspace.net. 715 43200 7200 2419200 86400
    _acme-challenge.uberspace.net.	10800	NS	tart.uberspace.net.

Now `sudo knotc reload` should load the zone files and update them with signatures. `sudo keymgr _acme-challenge.uberspace.net ds` shows DS records, needed to delegate a DNSSEC-enabled zone (in addition to regular NS records). `sudo keymgr uberspace.net dnskey` shows a DNSKEY, from which DS records can be derived, and which may be asked for by a domain registrar (to be set in a control panel, so that they'll derive the DS records, and set them). Glue records for the name server, and name server's host itself also go into the registrar's control panel.

## X.509 certificates with an ACME client

Most of the target protocols can work over TLS; Let's Encrypt and a few other CAs provide X.509 certificates for it using the ACME protocol (RFC 8555). Both the protocol and the software are awkward, but it's still an improvement over older CAs. The DNS challenge allows to issue wildcard certificates, and helps to avoid ugly hacks needed with the HTTP challenge.

Certbot is used here, being Let's Encrypt's recommended client, and the alternatives (such as dehydrated) not being much better. Although there is uacme, which does look promising, and which I use locally. Certbot is intended to run as a superuser, but that's undesirable and unnecessary in this setup, so we'll set a new user and a group for it; the services using its keys and certificates would then be in its group, and the keys will be group-readable. And we'll set it to use the ACME DNS challenge (using certbot-dns-rfc2136). Actually it's still quite awkward (`certbot` doesn't need access to the keys, for one), but better than the defaults, and unfortunately even this involves a relatively complex setup.

Hopefully in the future it will be viable to use opportunistic IPsec (likely relying on DNS: IPSECKEY or IPSECA; both strongSwan and libreswan seem to support IPSECKEY somehow, but I haven't found how to make DNS interception for opportunistic IPsec work; also the Unbound caching DNS server has ipsecmod, but it's not included for Debian 11), getting rid of PKIX together with TLS and all the awkwardness around it. TLSA RR updates are configured below, as a small step away from PKIX and closer to how it may work with IPsec.

Be very careful with it: it's hard to check that everything is set properly, and can easily be the source of most of the issues with a system.

    $ sudo apt install certbot python3-certbot-dns-rfc2136
    $ sudo addgroup --system letsencrypt
    $ sudo adduser --system --no-create-home --home /var/lib/letsencrypt/ --ingroup letsencrypt letsencrypt
    $ sudo touch /etc/letsencrypt/rfc2136
    $ sudo chmod 600 /etc/letsencrypt/rfc2136
    $ sudo chown -R letsencrypt:letsencrypt /etc/letsencrypt/ /var/log/letsencrypt/ /var/lib/letsencrypt/

Edit `/etc/letsencrypt/rfc2136`:

    dns_rfc2136_server = 127.0.0.2
    dns_rfc2136_port = 53
    dns_rfc2136_name = certbot
    dns_rfc2136_secret = SECRET_TSIG_KEY
    dns_rfc2136_algorithm = HMAC-SHA512

Request a certificate, change file permissions:

    $ sudo -u letsencrypt certbot certonly --agree-tos --email 'info@uberspace.net' \
    > -n --dns-rfc2136 --dns-rfc2136-credentials=/etc/letsencrypt/rfc2136 \
    > -d 'uberspace.net' -d '*.uberspace.net'
    $ sudo chmod 0640 /etc/letsencrypt/live/uberspace.net/privkey.pem

Add a script to update TLSA records, `/usr/local/bin/cert-deploy.sh`:

    #!/bin/sh
    set -e

    fp=$(openssl x509 -pubkey -noout -in \
                 "/etc/letsencrypt/live/uberspace.net/cert.pem" |
             openssl rsa -pubin -outform der 2>/dev/null |
             openssl dgst -sha256 |
             cut -d ' ' -f 2)

    knsupdate <<EOF
    server 127.0.0.2
    key hmac-sha512:certbot SECRET_TSIG_KEY
    zone uberspace.net.
    origin uberspace.net.
    ttl 10800
    update del tlsa311._dane TLSA
    update add tlsa311._dane TLSA 3 1 1 $fp
    send
    quit
    EOF
    exit 0

Don't forget to adjust permissions for it, since it contains a secret key:

    $ sudo chown letsencrypt:letsencrypt /usr/local/bin/cert-deploy.sh
    $ sudo chmod 0500 /usr/local/bin/cert-deploy.sh

Add `/etc/cron.weekly/certificate-refresh`, to renew and reload a certificate at once:

    #!/bin/sh
    set -e
    sudo -u letsencrypt /usr/bin/certbot -q renew
    chmod 0640 /etc/letsencrypt/live/uberspace.net/privkey.pem
    systemctl reload postfix dovecot nginx prosody
    sudo -u letsencrypt /usr/local/bin/cert-deploy.sh
    exit 0

And make it executable with `sudo chmod +x /etc/cron.weekly/certificate-refresh`.

Finally, disable default certbot renewal tasks: remove (or comment out) `/etc/cron.d/certbot`, and disable its systemd timer with `sudo systemctl disable certbot.timer`.

Now adding a user into the `letsencrypt` group would grant them access to the keys, e.g.: `sudo usermod -G letsencrypt prosody`.

When we'll be configuring servers, there often will be a choice between direct TLS and STARTTLS, required and optional use of TLS, restriction of used ciphers. Being overly strict leads to failures, sometimes for no good reason. I tend to require TLS for SMTP, IMAP, and XMPP clients, but not for public documents served over HTTP. Clients should also be encouraged to not rely on it too much, and use OpenPGP or other end-to-end encryption methods when it matters.

## WWW

I'm usually installing nginx (though there are a few other httpd alternatives) with cgit at once, `sudo apt install nginx cgit fcgiwrap`. `/etc/nginx/nginx.conf` edits:

    --- nginx.conf  2020-12-15 20:17:38.367101592 +0000
    +++ /etc/nginx/nginx.conf       2022-09-06 10:06:15.349750001 +0000
    @@ -16,8 +16,6 @@

            sendfile on;
            tcp_nopush on;
    -       tcp_nodelay on;
    -       keepalive_timeout 65;
            types_hash_max_size 2048;
            # server_tokens off;

    @@ -27,12 +25,17 @@
            include /etc/nginx/mime.types;
            default_type application/octet-stream;

    +        # Corresponds to the HTTP file upload size limit in Prosody.
    +        client_max_body_size 20M;
    +
            ##
            # SSL Settings
            ##

    -       ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
    +       ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
            ssl_prefer_server_ciphers on;
    +        ssl_certificate     /etc/letsencrypt/live/uberspace.net/fullchain.pem;
    +        ssl_certificate_key /etc/letsencrypt/live/uberspace.net/privkey.pem;

            ##
            # Logging Settings
    @@ -52,7 +55,9 @@
            # gzip_comp_level 6;
            # gzip_buffers 16 8k;
            # gzip_http_version 1.1;
    -       # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
    +       gzip_types text/plain text/css application/json application/javascript
    +       text/xml application/xml application/xml+rss text/javascript
    +       application/xhtml+xml;

            ##
            # Virtual Host Configs

`/etc/cgitrc` (with just a couple of repositories):

    #
    # cgit config
    # see cgitrc(5) for details

    css=/cgit.css
    logo=/cgit.png

    virtual-root=/
    root-title=uberspace.net git repositories
    clone-prefix=https://git.uberspace.net/

    repo.url=pgxhtml
    repo.path=/srv/git/pgxhtml
    repo.desc=XSLT-based PostgreSQL web interface
    repo.owner=defanor

    repo.url=rexmpp
    repo.path=/srv/git/rexmpp
    repo.desc=A reusable XMPP library
    repo.owner=defanor

Attempt to reduce crawler log spam with `/usr/share/cgit/robots.txt`:

    --- robots.txt  2020-12-19 17:25:29.863643316 +0100
    +++ /usr/share/cgit/robots.txt  2020-12-19 17:26:29.062565176 +0100
    @@ -1,3 +1,4 @@
     User-agent: *
     Disallow: /*/snapshot/*
    +Disallow: /*?*
     Allow: /

`/etc/nginx/conf.d/cgit.conf`:

    server {
        listen [::]:80;
        listen 80;
        listen [::]:443 ssl;
        listen 443 ssl;

        server_name git.uberspace.net;

        root /usr/share/cgit;
        try_files $uri @cgit;

        location @cgit {
            include             fastcgi_params;
            fastcgi_param       SCRIPT_FILENAME /usr/lib/cgit/cgit.cgi;
            fastcgi_param       PATH_INFO       $uri;
            fastcgi_param       QUERY_STRING    $args;
            fastcgi_param       HTTP_HOST       $server_name;
            fastcgi_pass        unix:/run/fcgiwrap.socket;
        }
     }

A sample static website (paste service, serving files that were simply uploaded with rsync), `/etc/nginx/conf.d/paste.conf`:

    server {
        listen 80;
        listen [::]:80;
        listen 443 ssl;
        listen [::]:443 ssl;

        server_name paste.uberspace.net;

        location / {
            charset utf-8;
            root /srv/paste/;
        }
    }

Setting reverse proxies for XMPP file upload and XMPP-over-HTTPS (needed because the available client software on some systems--iOS, most notably--is highly unreliable; using converse.js, which goes into `/var/www/xmpp/`) in `/etc/nginx/conf.d/xmpp.conf`:

    server {
        listen 80;
        listen [::]:80;
        server_name xmpp.uberspace.net;
        return 301 https://$server_name$request_uri;
    }

    server {
        listen 443 ssl;
        listen [::]:443 ssl;

        server_name xmpp.uberspace.net;

        location /upload/ {
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-For $remote_addr;
            proxy_pass http://localhost:5280;
        }

        location /http-bind {
            proxy_pass http://localhost:5280/http-bind;
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-For $remote_addr;
            proxy_buffering off;
            tcp_nodelay on;
        }

        location /xmpp-websocket {
            proxy_pass http://localhost:5280/xmpp-websocket;
            proxy_http_version 1.1;
            proxy_set_header Connection "Upgrade";
            proxy_set_header Upgrade $http_upgrade;

            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-For $remote_addr;
            proxy_read_timeout 900s;
        }

        location / {
            charset utf-8;
            root /var/www/xmpp;
        }
    }

HTTPS is enabled for the default website, and the directive preventing "not found" errors from being logged is removed, so that `fail2ban` would notice and block spammy vulnerability scanners:

    --- default     2020-12-15 22:58:40.632894873 +0100
    +++ /etc/nginx/sites-available/default  2020-12-24 00:38:26.996563289 +0100
    @@ -24,8 +24,8 @@

            # SSL configuration
            #
    -       # listen 443 ssl default_server;
    -       # listen [::]:443 ssl default_server;
    +       listen 443 ssl default_server;
    +       listen [::]:443 ssl default_server;
            #
            # Note: You should disable gzip for SSL traffic.
            # See: https://bugs.debian.org/773332
    @@ -45,12 +45,6 @@

            server_name _;

    -       location / {
    -               # First attempt to serve request as file, then
    -               # as directory, then fall back to displaying a 404.
    -               try_files $uri $uri/ =404;
    -       }
    -
            # pass PHP scripts to FastCGI server
            #
            #location ~ \.php$ {

At this point we can also set the `.well-known` directory (RFC 5785) with a XEP-0156/RFC 7395 `host-meta` file for XMPP-over-HTTPS, and a Web Key Directory for OpenPGP key discovery (generate with `gpg --list-options show-only-fpr-mbox -k '@uberspace.net' | /usr/lib/gnupg/gpg-wks-client -v --install-key`).

## Caching name server

Since we will be setting a mail server, it is best to have a local caching DNS server: external ones tend to run into DNSBL limits. Simply `sudo apt install knot-resolver` and `sudo systemctl enable --now kresd@1.service`.

## Email

We'll use Dovecot and Postfix with OpenDKIM and SPF Engine. Additionally, mlmmj will be set for mailing list management, and MHonArc for mailing list archiving. Some of the viable alternatives are Courier Mail Server (providing SMTP, IMAP, and mailing lists at once), Exim for SMTP, Mailman for mailing list management (though Mailman 3 uses a few times more memory than this whole system together, and there are other bloat-related issues with it).

### Dovecot

This one is easy, and handy to setup first out of the email programs: then we can move all the user maildirs (maildir is a generally nice format, which is handy for migration by simply moving it with a home directory), check that they work (that is, messages are available via IMAP), and use Dovecot's SASL for Postfix too. Install it with `sudo apt install dovecot-imapd`, edit `/etc/dovecot/conf.d/10-mail.conf`:

    --- 10-mail.conf        2020-12-15 16:40:21.880949141 +0100
    +++ /etc/dovecot/conf.d/10-mail.conf    2020-12-15 16:41:31.704881933 +0100
    @@ -27,7 +27,7 @@
     #
     # <doc/wiki/MailLocation.txt>
     #
    -mail_location = mbox:~/mail:INBOX=/var/mail/%u
    +mail_location = maildir:~/Maildir:LAYOUT=fs

     # If you need to set multiple mailbox locations or want to change default
     # namespace settings, you can do it by defining namespace sections.

Edit `/etc/dovecot/conf.d/10-ssl.conf` too:

    --- 10-ssl.conf 2020-12-15 16:31:54.059688146 +0100
    +++ /etc/dovecot/conf.d/10-ssl.conf     2020-12-15 16:34:39.057359523 +0100
    @@ -3,14 +3,14 @@
     ##

     # SSL/TLS support: yes, no, required. <doc/wiki/SSL.txt>
    -ssl = yes
    +ssl = required

     # PEM encoded X.509 SSL/TLS certificate and private key. They're opened before
     # dropping root privileges, so keep the key file unreadable by anyone but
     # root. Included doc/mkcert.sh can be used to easily generate self-signed
     # certificate, just make sure to update the domains in dovecot-openssl.cnf
    -ssl_cert = </etc/dovecot/private/dovecot.pem
    -ssl_key = </etc/dovecot/private/dovecot.key
    +ssl_cert = </etc/letsencrypt/live/uberspace.net/fullchain.pem
    +ssl_key = </etc/letsencrypt/live/uberspace.net/privkey.pem

     # If key file is password protected, give the password here. Alternatively
     # give it when starting dovecot with -p parameter. Since this file is often

Then just reload (or restart) it, and it should be ready to serve messages. Though once we'll install Postfix, to use Dovecot SASL, we'll also need the following `/etc/dovecot/conf.d/10-master.conf` edits:

    --- 10-master.conf      2020-12-15 19:12:44.948427098 +0100
    +++ /etc/dovecot/conf.d/10-master.conf  2020-12-15 19:13:42.139594734 +0100
    @@ -104,9 +104,11 @@
       }

       # Postfix smtp-auth
    -  #unix_listener /var/spool/postfix/private/auth {
    -  #  mode = 0666
    -  #}
    +  unix_listener /var/spool/postfix/private/auth {
    +    mode = 0666
    +    user = postfix
    +    group = postfix
    +  }

       # Auth process is run as this user.
       #user = $default_internal_user

### OpenDKIM

Install and generate a key:

    $ sudo apt install opendkim opendkim-tools
    $ sudo -u opendkim opendkim-genkey -D /etc/dkimkeys -d uberspace.net -s tart2020

Then the key from `/etc/dkimkeys/tart2020.txt` should be added into the zone file, and its parameters should be set in `/etc/opendkim.conf`:

    --- opendkim.conf       2020-12-15 17:41:13.715731125 +0100
    +++ /etc/opendkim.conf  2020-12-15 17:52:34.965843653 +0100
    @@ -10,9 +10,9 @@

     # Sign for example.com with key in /etc/dkimkeys/dkim.key using
     # selector '2007' (e.g. 2007._domainkey.example.com)
    -#Domain                        example.com
    -#KeyFile               /etc/dkimkeys/dkim.key
    -#Selector              2007
    +Domain                 uberspace.net
    +KeyFile                        /etc/dkimkeys/tart2020.private
    +Selector               tart2020

     # Commonly-used options; the commented-out versions show the defaults.
     #Canonicalization      simple
    @@ -30,8 +30,8 @@
     # ##  inet:port                   to listen on all interfaces
     # ##  local:/path/to/socket       to listen on a UNIX domain socket
     #
    -#Socket                  inet:8892@localhost
    -Socket                 local:/var/run/opendkim/opendkim.sock
    +Socket                  inet:8892@localhost
    +#Socket                        local:/var/run/opendkim/opendkim.sock

     ##  PidFile filename
     ###      default (none)

The records can then be checked with `sudo -u opendkim opendkim-testkey -d uberspace.net -s tart2020 -vvv`.

### Postfix

Install Postfix and SPF Engine with `sudo apt install postfix postfix-policyd-spf-python`, edit `/etc/postfix/master.cf` to enable a dedicated submission port, postscreen, SPF, mailing lists:

    --- master.cf   2020-12-15 18:06:15.011289517 +0100
    +++ /etc/postfix/master.cf      2021-01-02 16:53:24.430963345 +0100
    @@ -9,23 +9,28 @@
     # service type  private unpriv  chroot  wakeup  maxproc command + args
     #               (yes)   (yes)   (no)    (never) (100)
     # ==========================================================================
    -smtp      inet  n       -       y       -       -       smtpd
    -#smtp      inet  n       -       y       -       1       postscreen
    -#smtpd     pass  -       -       y       -       -       smtpd
    -#dnsblog   unix  -       -       y       -       0       dnsblog
    -#tlsproxy  unix  -       -       y       -       0       tlsproxy
    -#submission inet n       -       y       -       -       smtpd
    -#  -o syslog_name=postfix/submission
    -#  -o smtpd_tls_security_level=encrypt
    -#  -o smtpd_sasl_auth_enable=yes
    -#  -o smtpd_tls_auth_only=yes
    -#  -o smtpd_reject_unlisted_recipient=no
    -#  -o smtpd_client_restrictions=$mua_client_restrictions
    -#  -o smtpd_helo_restrictions=$mua_helo_restrictions
    -#  -o smtpd_sender_restrictions=$mua_sender_restrictions
    -#  -o smtpd_recipient_restrictions=
    -#  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
    -#  -o milter_macro_daemon_name=ORIGINATING
    +#smtp      inet  n       -       y       -       -       smtpd
    +smtp      inet  n       -       y       -       1       postscreen
    +smtpd     pass  -       -       y       -       -       smtpd
    +dnsblog   unix  -       -       y       -       0       dnsblog
    +tlsproxy  unix  -       -       y       -       0       tlsproxy
    +submission inet n       -       y       -       -       smtpd
    +  -o syslog_name=postfix/submission
    +  -o smtpd_tls_security_level=encrypt
    +  -o smtpd_sasl_auth_enable=yes
    +  -o smtpd_sasl_type=dovecot
    +  -o smtpd_sasl_path=private/auth
    +  -o smtpd_sasl_security_options=noanonymous
    +  -o smtpd_sasl_local_domain=$myhostname
    +  -o smtpd_tls_auth_only=yes
    +  -o smtpd_reject_unlisted_recipient=no
    +  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
    +  -o { smtpd_recipient_restrictions = reject_non_fqdn_recipient,
    +                                      reject_unknown_recipient_domain,
    +                                      permit_sasl_authenticated,
    +                                      reject }
    +  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
    +  -o milter_macro_daemon_name=ORIGINATING
     #smtps     inet  n       -       y       -       -       smtpd
     #  -o syslog_name=postfix/smtps
     #  -o smtpd_tls_wrappermode=yes
    @@ -124,4 +129,9 @@
     mailman   unix  -       n       n       -       -       pipe
       flags=FR user=list argv=/usr/lib/mailman/bin/postfix-to-mailman.py
       ${nexthop} ${user}
    -
    +# SPF with postfix-policyd-spf-python
    +policyd-spf  unix  -       n       n       -       0       spawn
    +  user=policyd-spf argv=/usr/bin/policyd-spf
    +# mlmmj mailing lists
    +mlmmj   unix  -       n       n       -       -       pipe
    +  flags=ORhu user=mlmmj argv=/usr/bin/mlmmj-receive -F -L /var/spool/mlmmj/$nexthop

Also edit `/etc/postfix/main.cf` (partially based on Postfix Anti-UCE Cheat Sheet and rob0's postscreen(8) configuration; for reference, see postconf(5); Postfix Documentation is nice and extensive; DNSWL's 127.0.10.3 is excluded here), along with the DNSWL category for marketing:

    --- main.cf     2020-12-15 17:03:13.489245160 +0000
    +++ /etc/postfix/main.cf        2022-10-06 21:49:38.436893028 +0000
    @@ -24,24 +24,91 @@

     # TLS parameters
    -smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
    -smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
    +smtpd_tls_cert_file=/etc/letsencrypt/live/uberspace.net/fullchain.pem
    +smtpd_tls_key_file=/etc/letsencrypt/live/uberspace.net/privkey.pem
     smtpd_use_tls=yes
     smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
     smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
    +smtp_tls_security_level = may

     # See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for
     # information on enabling SSL in the smtp client.

     smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination
    -myhostname = tart
    +myhostname = tart.uberspace.net
     alias_maps = hash:/etc/aliases
     alias_database = hash:/etc/aliases
     myorigin = /etc/mailname
     mydestination = $myhostname, uberspace.net, tart, localhost.localdomain, localhost
     relayhost =
     mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
    -mailbox_size_limit = 0
    +# 20 MiB for a message
    +message_size_limit = 20971520
    +# 1 GiB for a mailbox
    +mailbox_size_limit = 1073741824
     recipient_delimiter = +
     inet_interfaces = all
     inet_protocols = all
    +
    +# Store messages in ~/Maildir/
    +home_mailbox = Maildir/
    +
    +# OpenDKIM
    +smtpd_milters = inet:localhost:8892
    +non_smtpd_milters = $smtpd_milters
    +milter_default_action = accept
    +internal_mail_filter_classes = bounce
    +
    +# Postscreen
    +postscreen_access_list = permit_mynetworks,
    +                         cidr:/etc/postfix/postscreen_access.cidr
    +postscreen_blacklist_action = drop
    +postscreen_greet_action = drop
    +postscreen_pipelining_enable = yes
    +postscreen_non_smtp_command_enable = yes
    +postscreen_bare_newline_enable = yes
    +postscreen_bare_newline_action = enforce
    +postscreen_dnsbl_action = enforce
    +postscreen_dnsbl_sites = zen.spamhaus.org*3
    +        b.barracudacentral.org*2
    +        bl.spameatingmonkey.net*2
    +        bl.spamcop.net
    +        dnsbl.sorbs.net
    +        psbl.surriel.com
    +        bl.mailspike.net
    +        list.dnswl.org=127.0.[0..14;16..255].0*-2
    +        list.dnswl.org=127.0.[0..14;16..255].1*-3
    +        list.dnswl.org=127.0.[0..9;11..14;16..255].[2..3]*-4
    +postscreen_dnsbl_threshold = 3
    +postscreen_dnsbl_whitelist_threshold = -1
    +
    +# Other anti-UCE
    +smtpd_helo_required = yes
    +disable_vrfy_command = yes
    +smtpd_recipient_restrictions =
    +        reject_invalid_hostname,
    +        reject_non_fqdn_hostname,
    +        reject_non_fqdn_sender,
    +        reject_non_fqdn_recipient,
    +        reject_unknown_sender_domain,
    +        reject_unknown_recipient_domain,
    +        permit_mynetworks,
    +        # reject_unknown_client_hostname,
    +        reject_unauth_destination,
    +        check_sender_access hash:/etc/postfix/sender_checks,
    +        check_client_access hash:/etc/postfix/client_checks,
    +        # reject_rbl_client sbl.spamhaus.org,
    +        # reject_rbl_client pbl.spamhaus.org
    +        reject_rbl_client bl.spamcop.net,
    +        reject_rbl_client cbl.abuseat.org,
    +        #reject_rbl_client noptr.spamrats.com,
    +        check_policy_service unix:private/policyd-spf,
    +        permit
    +smtpd_data_restrictions =
    +                        reject_unauth_pipelining,
    +                        permit
    +
    +# Mailing lists with mlmmj
    +mlmmj_destination_recipient_limit = 1
    +transport_maps = hash:/etc/postfix/transport
    +relay_domains = lists.uberspace.net

Create a few files for access control:

    $ sudo -e /etc/postfix/postscreen_access.cidr
    $ sudo -e /etc/postfix/client_checks
    $ sudo postmap /etc/postfix/client_checks
    $ sudo -e /etc/postfix/sender_checks
    $ sudo postmap /etc/postfix/sender_checks
    $ sudo -e /etc/aliases
    $ sudo postalias /etc/aliases

A few aliases should be set, as per RFC 2142 and XEP-0157:

    postmaster:    root
    hostmaster:    root
    webmaster:     root
    xmpp:          root
    abuse:         root
    info:          root
    support:       root
    security:      root
    root:          defanor

A `/etc/postfix/postscreen_access.cidr` example:

    # GitHub SPF, 2020-12-19
    192.30.252.0/22 permit

    # bendel.debian.org, 2020-12-19
    82.195.75.100 permit
    2001:41b8:202:deb:216:36ff:fe40:4002 permit

    # Spammy networks: passing all the checks, but sending spam and
    # ignoring (or not accepting) abuse reports
    103.66.104.0/22 reject
    31.192.232.0/21 reject

DNS records should be checked, reverse DNS records should be set too, and the addresses can be added into DNSWL. Additional spam filtering can be added too, but I'm hardly getting any spam with just these settings (rare spam that gets through I'm reporting and adding into access checks).

### Mailing lists

Postfix is already configured above to hook up mlmmj, here is an example `/etc/postfix/transport` file for a single list (called "general"):

    general@lists.uberspace.net mlmmj:general

Installation, configuration, and usage of mlmmj and mhonarc:

    $ # Initial setup
    $ sudo apt install mlmmj mhonarc
    $ sudo adduser --system --no-create-home --home /var/spool/mlmmj/ mlmmj
    $ sudo mkdir /var/www/lists/
    $ sudo chown mlmmj /var/spool/mlmmj/ /var/www/lists/

    $ # Create a list
    $ sudo -u mlmmj mlmmj-make-ml -L general
    $ echo tart.uberspace.net | sudo -u mlmmj tee /var/spool/mlmmj/general/control/smtphelo
    $ sudo -u mlmmj mkdir /var/www/lists/general

And the following in `/etc/cron.hourly/mlmmj-maintenance`:

    #!/bin/sh
    set -e
    /usr/bin/mlmmj-maintd -F -L /var/spool/mlmmj/general/
    sudo -u mlmmj mhonarc -quiet -spammode -add -umask 022 \
         -outdir /var/www/lists/general /var/spool/mlmmj/general/archive/
    exit 0

There's a room for tinkering and improvement, but that's a basic and lightweight setup that is quite usable.

## XMPP

Prosody is a popular XMPP server option; it's fairly feature-rich and easy to set. Among other usable XMPP server alternatives is ejabberd, though IIRC it required more resources, its documentation is rather awkward, and generally looks rather enterprise-y. coturn is a TURN/STUN server, helping clients to establish connections for file transfers and calls. Install Prosody, its modules, lua-unbound resolver library, and coturn with `sudo apt install prosody prosody-modules lua-unbound coturn`. Set a firewall/blocklist (since unfortunately there are spammy servers, as with email): `sudo mkdir /etc/prosody/firewall`, create `/etc/prosody/firewall/spammers.txt`:

    jabber.infos.ru
    jabber.gg
    isgeek.info

And `/etc/prosody/firewall/spammers.pfw`:

    %LIST spammers: file:/etc/prosody/firewall/spammers.txt

    ::deliver

    CHECK LIST: spammers contains $<@from|host>
    BOUNCE=policy-violation (Your server is blocked due to spam)

Edit `/etc/prosody/prosody.cfg.lua` (among other things, enable `smacks` for Prosody to at least attempt to deliver messages, set certificate paths, and enable HTTP file upload with the reverse proxy we've configured above):

    --- prosody.cfg.lua.dpkg-dist   2023-02-22 08:56:38.000000000 +0000
    +++ /etc/prosody/prosody.cfg.lua        2023-06-14 11:06:23.182713133 +0000
    @@ -21,7 +21,7 @@
     -- for the server. Note that you must create the accounts separately
     -- (see https://prosody.im/doc/creating_accounts for info)
     -- Example: admins = { "user1@example.com", "user2@example.net" }
    -admins = { }
    +admins = { "defanor@uberspace.net" }

     -- This option allows you to specify additional locations where Prosody
     -- will search first for modules. For additional modules you can install, see
    @@ -62,7 +62,7 @@
                    "time"; -- Let others know the time here on this server
                    "uptime"; -- Report how long server has been running
                    "version"; -- Replies to server version requests
    -               --"mam"; -- Store recent messages to allow multi-device synchronization
    +               "mam"; -- Store recent messages to allow multi-device synchronization
                    --"turn_external"; -- Provide external STUN/TURN service for e.g. audio/video calls

            -- Admin interfaces
    @@ -70,9 +70,9 @@
                    "admin_shell"; -- Allow secure administration via 'prosodyctl shell'

            -- HTTP modules
    -               --"bosh"; -- Enable BOSH clients, aka "Jabber over HTTP"
    +               "bosh"; -- Enable BOSH clients, aka "Jabber over HTTP"
                    --"http_openmetrics"; -- for exposing metrics to stats collectors
    -               --"websocket"; -- XMPP over WebSockets
    +               "websocket"; -- XMPP over WebSockets

            -- Other specific functionality
                    "posix"; -- POSIX functionality, sends server to background, enables syslog, etc.
    @@ -81,12 +81,15 @@
                    --"legacyauth"; -- Legacy authentication. Only used by some old clients and bots.
                    --"mimicking"; -- Prevent address spoofing
                    --"motd"; -- Send a message to users when they log in
    -               --"proxy65"; -- Enables a file transfer proxy service which clients behind NAT can use
    +               "proxy65"; -- Enables a file transfer proxy service which clients behind NAT can use
                    --"s2s_bidi"; -- Bi-directional server-to-server (XEP-0288)
                    --"server_contact_info"; -- Publish contact information for this service
    -               --"tombstones"; -- Prevent registration of deleted accounts
                    --"watchregistrations"; -- Alert admins of registrations
                    --"welcome"; -- Welcome users who register accounts
    +        -- Custom modules
    +                "cloud_notify";
    +                "firewall";
    +                "turncredentials";
     }

     -- These modules are auto-loaded, but should you want
    @@ -97,12 +100,52 @@
            -- "s2s"; -- Handle server-to-server connections
     }

    +smacks_enabled_s2s = true
    +consider_bosh_secure = true
    +consider_websocket_secure = true
    +cross_domain_bosh = {"https://xmpp.uberspace.net"}
    +cross_domain_websocket = {"https://xmpp.uberspace.net"}
    +
    +http_external_url = "https://xmpp.uberspace.net/"
    +trusted_proxies = { "127.0.0.1", "::1" }
    +http_ports = { 5280 }
    +http_interfaces = { "127.0.0.1", "::1" }
    +https_ports = { 5281 }
    +https_interfaces = { "127.0.0.1", "::1" }
    +
    +http_upload_file_size_limit = 20*1024*1024
    +http_max_content_size = 100*1024*1024
    +
    +turncredentials_host = "uberspace.net"
    +turncredentials_secret = "TURN_SECRET_STRING_HERE"
    +
    +
    +firewall_scripts = {
    +    "module:scripts/jabberspam-simple-blocklist.pfw";
    +    "/etc/prosody/firewall/spammers.pfw";
    +}
    +
    +-- Disable account creation by default, for security
    +-- For more information see https://prosody.im/doc/creating_accounts
    +allow_registration = false
    +
    +
     -- Debian:
     --   Please, don't change this option since /run/prosody/
     --   is one of the few directories Prosody is allowed to write to
     --
     pidfile = "/run/prosody/prosody.pid";

    +-- Force clients to use encrypted connections? This option will
    +-- prevent clients from authenticating unless they are using encryption.
    +
    +c2s_require_encryption = true
    +
    +-- Force servers to use encrypted connections? This option will
    +-- prevent servers from authenticating unless they are using encryption.
    +
    +s2s_require_encryption = true
    +
     -- Server-to-server authentication
     -- Require valid certificates for server-to-server connections?
     -- If false, other methods such as dialback (DNS) may be used instead.
    @@ -128,7 +171,7 @@

     limits = {
            c2s = {
    -               rate = "10kb/s";
    +               rate = "100kb/s";
            };
            s2sin = {
                    rate = "30kb/s";
    @@ -196,7 +239,7 @@
     --  Logs errors to syslog also
     log = {
            -- Log files (change 'info' to 'debug' for debug logs):
    -       info = "/var/log/prosody/prosody.log";
    +       debug = "/var/log/prosody/prosody.log";
            error = "/var/log/prosody/prosody.err";
            -- Syslog:
            { levels = { "error" }; to = "syslog";  };
    @@ -216,7 +259,12 @@
     -- (from e.g. Let's Encrypt) see https://prosody.im/doc/certificates

     -- Location of directory to find certificates in (relative to main config file):
    -certificates = "certs"
    +-- certificates = "certs"
    +ssl = {
    +   certificate = "/etc/letsencrypt/live/uberspace.net/fullchain.pem";
    +   key = "/etc/letsencrypt/live/uberspace.net/privkey.pem";
    +}
    +

     ----------- Virtual hosts -----------
     -- You need to add a VirtualHost entry for each domain you wish Prosody to serve.
    @@ -230,7 +278,7 @@
     -- Component definitions in their own config files. This line includes
     -- all config files in /etc/prosody/conf.d/

    -VirtualHost "localhost"
    +VirtualHost "uberspace.net"
     -- Prosody requires at least one enabled VirtualHost to function. You can
     -- safely remove or disable 'localhost' once you have added another.

    @@ -243,12 +291,13 @@
     -- For more information on components, see https://prosody.im/doc/components

     ---Set up a MUC (multi-user chat) room server on conference.example.com:
    ---Component "conference.example.com" "muc"
    +Component "conference.uberspace.net" "muc"
     --- Store MUC messages in an archive and allow users to access it
    ---modules_enabled = { "muc_mam" }
    +modules_enabled = { "muc_mam" }

     ---Set up a file sharing component
     --Component "share.example.com" "http_file_share"
    +Component "xmpp.uberspace.net" "http_upload"

     ---Set up an external component (default component port is 5347)
     --

And edit `/etc/turnserver.conf`:

    --- turnserver.conf     2021-10-09 12:21:45.220209649 +0000
    +++ /etc/turnserver.conf        2023-01-21 12:42:25.383638421 +0000
    @@ -233,7 +233,7 @@
     # Use either lt-cred-mech or use-auth-secret in the conf
     # to avoid any confusion.
     #
    -#use-auth-secret
    +use-auth-secret

     # 'Static' authentication secret value (a string) for TURN REST API only.
     # If not set, then the turn server
    @@ -241,7 +241,7 @@
     # in the user database (if present). The database-stored  value can be changed on-the-fly
     # by a separate program, so this is why that mode is considered 'dynamic'.
     #
    -#static-auth-secret=north
    +static-auth-secret=TURN_SECRET_STRING_HERE

     # Server name used for
     # the oAuth authentication purposes.
    @@ -355,7 +355,7 @@
     # Note: If the default realm is not specified, then realm falls back to the host domain name.
     #       If the domain name string is empty, or set to '(None)', then it is initialized as an empty string.
     #
    -#realm=mycompany.org
    +realm=uberspace.net

     # This flag sets the origin consistency
     # check. Across the session, all requests must have the same
    @@ -456,14 +456,14 @@
     # configuration file.
     # Use PEM file format.
     #
    -#cert=/usr/local/etc/turn_server_cert.pem
    +cert=/etc/letsencrypt/live/uberspace.net/fullchain.pem

     # Private key file.
     # Use an absolute path or path relative to the
     # configuration file.
     # Use PEM file format.
     #
    -#pkey=/usr/local/etc/turn_server_pkey.pem
    +pkey=/etc/letsencrypt/live/uberspace.net/privkey.pem

     # Private key file password, if it is in encoded format.
     # This option has no default value.

Then run `sudo usermod -G letsencrypt turnserver`, restart `coturn`.

When migrating, `/var/lib/prosody/` can be simply moved to a new server with this setup (while it would be a bit more complicated with non-file-based storage).

## Other tweaks and services

### znc

znc can be installed with `sudo apt install znc`, though it won't add a user or a service. To do that, `sudo adduser --system --home /var/lib/znc/ znc`, and add `/etc/systemd/system/znc.service`:

    [Unit]
    Description=ZNC, an advanced IRC bouncer
    After=network.target

    [Service]
    ExecStart=/usr/bin/znc -f
    User=znc

    [Install]
    WantedBy=multi-user.target

Then configure it or move its existing configuration (which is also just about moving its home directory, `/var/lib/znc/`, to a new server), let it access the X.509 key and certificate (`sudo usermod -G letsencrypt znc`) enable and start (`sudo systemctl enable --now znc`).

On upgrading to Debian 11 (bullseye), I had to comment out loading of the `partyline` module for it to start.

### Gophernicus

Gophernicus wasn't available from Debian repositories before bullseye, but had upstream packaging, so initially I've build it (but there's no need to in Debian 11):

    defanor@tart:~/src$ sudo apt install git build-essential fakeroot libwrap0-dev debhelper
    defanor@tart:~/src$ git clone 'https://github.com/gophernicus/gophernicus'
    defanor@tart:~/src$ cd gophernicus
    defanor@tart:~/src/gophernicus$ git checkout 3.0.1
    defanor@tart:~/src/gophernicus$ make deb
    defanor@tart:~/src/gophernicus$ sudo dpkg -i ../gophernicus_3.0.1-1_amd64.deb

After updating to Debian 11, I had to `systemctl enable gophernicus.socket`.

### nftables

A firewall is nice to set, at least to reduce garbage in the logs. nftables replaces iptables as a netfilter interface on Linux, so we'll use that: `sudo apt install nftables`, `sudo systemctl enable nftables`, and edit `/etc/nftables.conf`. For instance, based on the simple ruleset for a server:

    #!/usr/sbin/nft -f

    flush ruleset

    table inet filter {
      # Large networks from which registered users don't connect: APNIC,
      # AFRNIC, LACNIC. Also subnets of ISPs that seem to ignore abuse
      # reports: Centurylink, AS209605, AS133398 (serveroffer.lt),
      # AS133398 (tele-asia.net), AS34665 (pindc.ru), AS33915 (ziggo.nl),
      # AS200391 (fastvps.biz), AS212370/AS213371 (peenq.nl), AS51395
      # (privacyfirst.{sh,digital}), AS20978 (ttnet.com.tr), AS12876
      # (online.net), AS35807 (sknt.ru), AS44493 (profitserver.ru).
      set not-users {
        type ipv4_addr
        flags interval
        elements = {
          1.0.0.0/8, 14.0.0.0/8, 27.0.0.0/8, 36.0.0.0/8, 39.0.0.0/8,
          41.0.0.0-43.255.255.255, 49.0.0.0/8, 58.0.0.0-61.255.255.255,
          101.0.0.0-103.255.255.255, 105.0.0.0-106.255.255.255,
          110.0.0.0-126.255.255.255, 133.0.0.0/8, 150.0.0.0/8,
          153.0.0.0-154.255.255.255, 163.0.0.0/8, 171.0.0.0/8,
          175.0.0.0/8, 177.0.0.0/8, 179.0.0.0-183.255.255.255,
          186.0.0.0/7, 189.0.0.0-191.255.255.255, 196.0.0.0/7,
          200.0.0.0/6, 210.0.0.0/7, 218.0.0.0-223.255.255.255,
          63.152.0.0/13, 63.224.0.0/13, 64.91.0.0/17, 67.0.0.0/13,
          67.232.0.0/13, 69.29.0.0/16, 69.68.0.0/15, 69.179.0.0/16,
          70.56.0.0/14, 71.0.0.0/14, 71.32.0.0/13, 71.48.0.0/13,
          71.208.0.0/12, 75.160.0.0/12, 76.0.0.0/13, 97.112.0.0/12,
          98.125.0.0/16, 168.103.0.0/16, 173.202.0.0/16, 174.16.0.0/12,
          184.96.0.0/13, 184.156.0.0/14, 207.118.0.0/15,
          141.98.10.0/24, 185.36.81.0/24,
          45.125.64.0/22,
          31.184.192.0/20, 5.188.210.0/24, 37.139.53.0/24,
          82.74.0.0/15,
          5.188.206.0/24,
          77.247.110.0/24, 37.49.225.0/24,
          91.192.100.0/22,
          37.154.0.0/15,
          51.15.0.0/16,
          188.243.192.0/18, 88.201.204.0/22,
          31.192.232.0/21
        }
      }

      # Trusted hosts.
      set trusted {
        type ipv4_addr
        flags interval
      }

      # Known spammers. These will be updated dynamically.
      set spammers {
        type ipv4_addr
        flags interval
      }

      # Known persistent spammers from ISPs ignoring abuse reports, and
      # aiming services where even little server-to-server spam is
      # annoying (XMPP, email); a manually edited list. Currently in it:
      # sknt.ru.
      set s2s-spammers {
        type ipv4_addr
        flags interval
        elements = { 188.243.192.232 }
      }
      # Same, for IPv6. Contents: sknt.ru.
      set s2s-spammers-v6 {
        type ipv6_addr
        flags interval
        elements = { 2a05:3580:cd00::/40 }
      }

      chain input {
        type filter hook input priority 0; policy drop;

        # Allow traffic from established and related packets.
        ct state established,related accept

        # Drop invalid packets.
        ct state invalid drop

        # Allow loopback traffic.
        iifname lo accept

        # Allow traffic from trusted hosts.
        ip saddr @trusted accept

        # Allow all ICMP and IGMP traffic, but enforce a rate limit to
        # help prevent some types of flood attacks.
        ip protocol icmp limit rate 4/second accept
        ip6 nexthdr ipv6-icmp limit rate 4/second accept
        ip protocol igmp limit rate 4/second accept

        # Restrict connections from likely spammers.
        ip saddr @spammers limit rate over 4/hour drop

        # Block connections from known spammers.
        ip saddr @s2s-spammers drop
        ip6 saddr @s2s-spammers-v6 drop

        # Allow non-users to access public services, just limit the rate.
        tcp dport {25, 53, 70, 443, 5269} ip saddr @not-users limit rate 1/minute accept
        udp dport {53} ip saddr @not-users limit rate 1/minute accept

        # That's it for non-users.
        ip saddr @not-users drop

        # Allow SSH, SMTP, DNS, Gopher, HTTP, IMAP, HTTPS, SMTP
        # submission, IMAPS, ZNC, and XMPP-related ports: file transfer
        # proxy, client connections, server connections, TURN/STUN.
        tcp dport { 22, 25, 53, 70, 80, 143, 443, 587, 993, 1500, 5000,
                    5222, 5269, 3478, 3479, 5349, 5350, 49152-65535
                  } accept

        # Also allow UDP ports: 53 for DNS, 68 for DHCP, 3478 for
        # TURN/STUN (along with 5349 for TLS, port+1 as alternatives,
        # 49152-65535 for relaying).
        udp dport {53, 68, 3478, 3479, 5349, 5350, 49152-65535} accept
      }
      chain forward {
        type filter hook forward priority 0; policy drop;
      }
      chain output {
        type filter hook output priority 0; policy accept;
      }
    }

And `sudo nft -f /etc/nftables.conf` to reload it. One should be careful with blocking large subnets, as it is rather frustrating to be on the other side of blocking. Restricting access to private services (the ones requiring authentication) is fine if it's known that there's no legitimate users connecting from those subnets, and access to public services can be rate limited. It may also be nice to only allow SSH connections from your addresses, if all the possible addresses (or subnets) are static and known.

One may also consider using dynamically generated blocklists, based on spam/bruteforce spotted by others. Some of the seemingly nice lists are available at blacklists.co, Charles B. Haley, DataPlane.org, IPsum, attackers.ongoing.today, Threat Sourcing, Anti-Attacks, blocklist.net.ua, blocklist.de. An example `/etc/cron.daily/blacklist-refresh`:

    #!/bin/sh
    set -e
    nft flush set inet filter spammers
    curl --silent --compressed --max-filesize 1M \
         'https://raw.githubusercontent.com/stamparm/ipsum/master/levels/3.txt' |
         cut -f 1 |
         grep -P '^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$' |
         grep -E '^[0-9\.]{7,15}$' |   # a sanity check
         head -n 80000 |               # and another one
         xargs -L 4096 echo |
         tr ' ' ',' |
         xargs -i nft add element inet filter spammers '{{}}'
    exit 0

### fail2ban

To further reduce garbage in the logs, install fail2ban with `sudo apt install fail2ban`, and enable relevant jails in `/etc/fail2ban/jail.local`:

    [DEFAULT]
    bantime = 24h
    findtime = 1h
    banaction = nftables-multiport
    mode = aggressive

    [sshd]
    enabled = true

    [postfix]
    enabled = true

    [dovecot]
    enabled = true

    [nginx-botsearch]
    enabled = true

Fail2ban should be restarted after reloading nftables, to reapply its rules. There are some minor warts (failing to resolve "imap" service on Debian 10, not handling some of the common postscreen messages, likely more), which can be fixed in `jail.local`. Not including workarounds here, since they are temporary, but that's one more thing that happens commonly with software: one should be prepared to run into, report, and/or fix issues that arise.

## Continued maintenance

It's a good idea to follow developments in the used software, protocols, and infrastructures. There are relatively low-traffic mailing lists for announcements (e.g., debian-security-announce and debian-announce, among Debian mailing lists), helping to track when urgent updates are needed, and which are also handy for periodically confirming that one's mail server works.

There are plenty of system monitoring tools; I don't use any on this server, but `atop` (for CLI) and munin (for static HTML and PNG files or FastCGI, with all kinds of graphs) are among fairly nice, simple, and lightweight ones for load monitoring. Then there are collectd, Nagios, Zabbix, but they (including web frontends) tend to be not quite as lightweight.

For munin, below is a `/etc/systemd/system/munin-cgi@.service` that worked for me (on a different Debian 11 system) to run its FastCGI processes, to be served via nginx (with munin nginx configuration); apparently it was important to set the configuration file path explicitly:

    [Unit]
    Description=Munin FastCGI service

    [Service]
    Type=forking
    Environment="MUNIN_CONFIG=/etc/munin/munin.conf"
    ExecStart=spawn-fcgi -s /var/run/munin/fastcgi-%i.sock -U www-data \
                         -u munin -g munin /usr/lib/munin/cgi/munin-cgi-%i

    [Install]
    WantedBy=multi-user.target

Then `systemctl start munin-cgi@html munin-cgi@graph`, and enable them. In the `/etc/munin/munin.conf` file, I've set/uncommented `html_strategy cgi`, `graph_strategy cgi`, `cgiurl_graph /munin-cgi/munin-cgi-graph`.

Occasionally I'm making backups with `tar` and `gpg`, and downloading them with `rsync`, though the server isn't supposed to have anything stored that is valuable, hard to restore, and only available on it; perhaps its configuration was the closest thing to it, until this document was composed. An example: `sudo tar cz /etc/ /var/lib/{prosody,znc,knot} /var/www/html/ /srv/ /home/*/Maildir/ /usr/local/ | gpg -e -r defanor@uberspace.net -o "$(uname -n)-$(date --rfc-3339=date).tgz.gpg"`.

Sometimes I report particularly spammy addresses (bruteforce and spam attempts) to their hosters or ISPs. Not sure how well that works, but more people monitoring and reporting network abuse should lead to its reduction. See my notes on network abuse for a log of those.

Things may go wrong and require fixing sometimes. There are single points of failure in this setup, so such a system shouldn't be a single point of failure for one's well-being. In my experience, issues with such a system rarely arise and can be fixed quickly, with it being simpler and more reliable than desktop systems, but this may vary.