VPS backups: rsync, restic and the 3-2-1 rule

Why keep your own backups if the server is reliable?

Reliable hardware and an SLA protect against a failed disk, not against mistakes and attacks: rm -rf in the wrong directory, a bad upgrade, a break-in, ransomware, a table dropped by accident or an unpaid server. In every one of these cases a copy on the same VPS disappears along with the data. That is why your own off-server copies are always needed - no matter where the server runs or what the provider promises.

What is the 3-2-1 rule?

The 3-2-1 rule is the classic backup scheme: 3 copies of the data (the live one and two backups), on 2 different media or systems, with 1 copy somewhere else. For a VPS it looks like this:

Step 1. Decide what to back up

• /etc - configs for nginx, systemd, SSH and the firewall: setting them up again takes hours;
• application data - /home, /opt, /srv, /var/www or wherever your projects live, including .env files;
• database dumps - PostgreSQL, MySQL, SQLite, not the files of a running database;
• Docker volumes (/var/lib/docker/volumes) if data lives in containers - again, preferably via a database dump;
• there is no need to copy the whole system: /usr, /var/cache, /proc, /tmp and node_modules come back by installing the OS and packages.

Step 2. Dump databases before the backup

Database files copied while the database runs are almost certainly inconsistent, and the database will not start from them. A dump is taken from the running database without stopping it and is always consistent. An example for PostgreSQL - more in the guide to PostgreSQL on a VPS:

sudo mkdir -p /var/backups/db
sudo -u postgres pg_dump -Fc myapp | sudo tee /var/backups/db/myapp.dump > /dev/null
sudo ls -lh /var/backups/db

Step 3. Set up SSH key login to the backup server

Scheduled backups run without a password, so they need a dedicated SSH key for root. ssh-copy-id asks for the backup user's password on the second server once, and the entry in /root/.ssh/config gives it the short name backup-server. More on keys in the guide to connecting over SSH:

sudo ssh-keygen -t ed25519 -N "" -f /root/.ssh/backup_key
sudo ssh-copy-id -i /root/.ssh/backup_key.pub backup@203.0.113.20
sudo tee -a /root/.ssh/config > /dev/null <<'EOF'
Host backup-server
    HostName 203.0.113.20
    User backup
    IdentityFile /root/.ssh/backup_key
EOF

Step 4. The simple option: rsync to another server

rsync copies only changed files, so a repeat run takes seconds. -a preserves permissions and timestamps, and --delete removes files from the copy that no longer exist on the server. rsync has to be installed on both servers:

sudo apt install -y rsync
sudo rsync -a --delete /etc /home /opt /var/backups backup-server:/srv/backup/myvps/

Step 5. restic: encryption, deduplication and history

restic stores backups as snapshots: each snapshot is the full state at the time of the run, but identical parts of files are stored once (deduplication), so 30 daily snapshots of slowly changing data take little more space than one. Everything is encrypted on your server, and the backup server sees only encrypted blobs. The SFTP repository uses the key from step 3:

sudo apt install -y restic
sudo sh -c 'openssl rand -base64 32 > /root/.restic-password && chmod 600 /root/.restic-password'
sudo -i
export RESTIC_REPOSITORY=sftp:backup-server:/srv/restic/myvps
export RESTIC_PASSWORD_FILE=/root/.restic-password
restic init

The first snapshot, the snapshot list, pruning by retention policy and a repository integrity check. forget --prune keeps the last 7 daily, 4 weekly and 6 monthly snapshots and deletes data no snapshot references any more:

restic backup /etc /home /opt /var/backups --exclude-caches
restic snapshots
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune
restic check

Besides SFTP, restic writes to S3-compatible storage - a convenient place for the second backup under the 3-2-1 rule. The list of supported backends is in the documentation at restic.net.

Step 6. Run backups on a schedule

Put the dump and restic into one script: set -eu stops it at the first error, so a broken dump is never recorded as a successful backup:

sudo tee /usr/local/sbin/backup.sh > /dev/null <<'EOF'
#!/bin/sh
set -eu
export RESTIC_REPOSITORY=sftp:backup-server:/srv/restic/myvps
export RESTIC_PASSWORD_FILE=/root/.restic-password

# database dump - if the server runs PostgreSQL
mkdir -p /var/backups/db
runuser -u postgres -- pg_dump -Fc myapp > /var/backups/db/myapp.dump

restic backup /etc /home /opt /var/backups --exclude-caches
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune
EOF
sudo chmod 700 /usr/local/sbin/backup.sh

The systemd timer runs the script every night at 03:30 with a random delay of up to 15 minutes, and Persistent=true catches up on a missed run if the server was off at the time. Each run's result is visible in journalctl -u backup:

sudo tee /etc/systemd/system/backup.service > /dev/null <<'EOF'
[Unit]
Description=Restic backup
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/backup.sh
EOF
sudo tee /etc/systemd/system/backup.timer > /dev/null <<'EOF'
[Unit]
Description=Daily restic backup

[Timer]
OnCalendar=*-*-* 03:30:00
RandomizedDelaySec=15m
Persistent=true

[Install]
WantedBy=timers.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now backup.timer
systemctl list-timers backup.timer

If you prefer cron, the same job is one line in sudo crontab -e:

30 3 * * * /usr/local/sbin/backup.sh >> /var/log/backup.log 2>&1

Step 7. Test the restore

A backup that has never been restored is a hope, not a backup. Once a month restore a few files into a temporary directory and compare them with the originals, and load the database dump into a separate database with pg_restore:

restic snapshots
restic restore latest --target /root/restore-test --include /etc/nginx
diff -r /etc/nginx /root/restore-test/etc/nginx && echo OK

Time the whole thing while you are at it: how long it takes to bring everything up on a clean server - install the OS and packages, restore /etc, the data and the database. Write that recovery plan down before an outage, not during one.

Where to store backups

Tihost has three locations - Germany, Finland and Poland - so the backup server can sit in a different country from the main one: an outage at one data center will not hit both copies. See the guide to choosing a VPS location. Starter configurations work well as backup storage - the largest of them has the biggest disk: