Backups, What a Concept

Sat, 27 Dec 2025 00:00:00 +0000

How it started

For about three years I ran a single-user Mastodon instance at strangeweb.page in AWS, where the non-IaC bits were mostly manually put together. It was also a big example of "Do as I say, don't do as I do" because I wasn't maintaining any backups.

How it's going

Recently I moved that instance from AWS EC2 to a Hetzner VM, which meant I had to actually go through the steps of taking the configuration and data from one host to another. As part of that exercise I cobbled together a small shell script that takes those vital bits and copies them to a safe place.

In my case that's object storage at a different provider, avoiding a single point of failure.

The script

  #!/bin/sh

  . "$0.env"

  echo "$(date): Creating a DB snapshot"

  # Create a backup of the mastodon tables:
  sudo -u mastodon \
     pg_dump \
     -Fc mastodon_production \
     -f /home/mastodon/backup.dump

  echo "$(date): Creating a Redis snapshot"

  # Create a snapshot of the redis data:
  redis-cli SAVE

  for src in \
      /home/mastodon/live/.env.production \
      /var/lib/redis/dump.rdb \
      /home/mastodon/backup.dump; do
      echo "$(date): Backing up $src"
      aws s3 \
          --endpoint "https://$ENDPOINT" \
          cp \
          --quiet \
          "$src" \
          "s3://$BUCKET/snapshot/mastodon/"
  done

  echo "$(date): Done"

The .env file contains two settings: ENDPOINT and BUCKET. The ENDPOINT is necessary because the slightly older awscli version on the host was ignoring the endpoint_url directive in ~/.aws/config.

It's a fairly simple file, pointing to an object storage bucket at Scaleway:

[default]
region = nl-ams
output = json
services = scw-nl-ams

The other missing piece is ~/.aws/credentials with credentials for that bucket/account.

Disclaimer

Going back to that "Do as I say, don't do as I do" bit, I have not yet performed the ultimate test of restoring a test-server from the backed up data. So if I'm missing something, maybe let me know so I can fix it before it's too late? 😅

I also know that this only maintains a single snapshot unless the object storage has version support and it's enabled.

Moving to Europe (part 2)

Sun, 13 Jul 2025 00:00:00 +0000

The saga continues

After I moved my (three) domains, most of the DNS, and my static websites to Europe I got side-tracked. An interaction with a former colleague nudged me to pick that back up.

Services

When I last wrote about it I thought I would land on UpCloud or Scaleway for a virtual server (or two), but now I'm actually working with a virtual server at Hetzner. They do things a little bit differently, so I'm still getting used to that. Both have projects, which feels like the mental model equivalent of AWS VPCs? Hetzner has performance/usage graphs integrated into their web UI, whereas Scaleway has a hosted Grafana service (for which you need to create a dedicated user). Hetzner's virtual servers seem more generous with their root volumes, but then it may not be as easy to change that out later, so choose wisely?

That said, for something that mostly needs to be compute, storage, and a web presence, Hetzner's offerings seem quite nice.

Currently I'm putting together a "services" instance that will host things like webmail (Roundcube), bookmarks (Readeck), passwords (Vaultwarden, or something along those lines), etc. This would be the same Services project hinted at in the Read Later project page. The goal here will be to make it relatively painless to blow it away and spin it up elsewhere, so a layer of Terraform, some cloud-init fun, and automatically generated and managed LetsEncrypt certificates are going to be part of that.

Object Storage

For this, Scaleway still seems to be the most reasonable price, in particular for buckets with not a lot of content. Hetzner starts at 4.99 Euro for even an empty bucket (which then gets you 1TB of storage and traffic), by comparison. That's a non-starter for a bucket where I expect to have just a few dozen GB of data. For completeness, UpCloud's buckets also start at 5 Euros which then includes 250GB of storage, and states there are no data transfer fees, period?

Open questions

For moving my mail server I still need to do some research to find out which of these 3 (or some alternative I've overlooked or discarded unfairly) is the better pick in terms of IP reputation, or spam abuse complaint handling. Anyone know?

Previously

Moving to Europe

Hetzner on East of the Sun, West of the Moon