Sam Hooke

The Important Files: Part 5

All notes in this series:

In these notes I configure borg on my FreeNAS jail (named fnbbu) to automatically backup up every night to both a local repository, and a remote repository at rsync.net.

rsync.net vs Wasabi §

First, a quick comparison of rsync.net and Wasabi on price alone. Note that there are many aspects we could use to compare them (uptime, data center distribution, technical support, etc.) so this is not a well balanced comparison.

Wasabi is $4.99/TB/month with a 1TB minimum. So prices start at $4.99 for 1TB per month.

rsync.net is $0.02/GB/month (using the borg discount) with a 40GB minimum. So prices start at $0.80 for 40GB per month

This means up until the 250GB point, rsync.net is cheaper than Wasabi.

I’m starting at $9.60 (£7.37) for 40GB for a year with rsync.net.

Create borg repository on rsync.net §

NOTE: rsync.net does not permit interactive SSH sessions. We use the -t option and pass in the command to execute on the remote server, after which stdout is printed and then the session terminates.

Commands here are run from our FreeNAS jail.

First, let’s create a folder on rsync.net for our borg repository (this may or may not be necessary):

$ ssh -t 12154@ch-s012.rsync.net "mkdir borg-repo; chmod 700 borg-repo"

Next, we initialise the remote borg repository.

$ borg init --encryption=repokey 12154@ch-s012.rsync.net:borg-repo

NOTE: if your borg client is 1.x and borg server is 0.x, you may see an error similar to the following. In my case it says the error can safely be ignored.

Remote: Borg 0.29.0: exception in RPC call:
Remote: Traceback (most recent call last):
Remote:   File "/usr/home/kibab/borgbackup-0.29.0/borg/remote.py", line 96, in serve
Remote: TypeError: open() takes from 2 to 5 positional arguments but 7 were given
Remote: Platform: FreeBSD ch-s012.rsync.net 11.1-RELEASE-p6 FreeBSD 11.1-RELEASE-p6 #0: Tue Dec 19 13:52:29 PST 2017     user@11_1:/usr/src/sys/amd64/compile/kernel.11_1amd64 amd64
Remote: Python: CPython 3.4.3
Remote:
Please note:
If you see a TypeError complaining about the number of positional arguments
given to open(), you can ignore it if it comes from a borg version < 1.0.7.
This TypeError is a cosmetic side effect of the compatibility code borg
clients >= 1.0.7 have to support older borg servers.
This problem will go away as soon as the server has been upgraded to 1.0.7+.
Enter new passphrase:
Enter same passphrase again:
Do you want your passphrase to be displayed for verification? [yN]: n
Remote: Borg 0.29.0: exception in RPC call:
Remote: Traceback (most recent call last):
Remote:   File "/usr/home/kibab/borgbackup-0.29.0/borg/remote.py", line 91, in serve
Remote: borg.remote.InvalidRPCMethod: get_free_nonce
Remote: Platform: FreeBSD ch-s012.rsync.net 11.1-RELEASE-p6 FreeBSD 11.1-RELEASE-p6 #0: Tue Dec 19 13:52:29 PST 2017     user@11_1:/usr/src/sys/amd64/compile/kernel.11_1amd64 amd64
Remote: Python: CPython 3.4.3
Remote:
Please upgrade to borg version 1.1+ on the server for safer AES-CTR nonce handling.

By default repositories initialized with this version will produce security
errors if written to with an older version (up to and including Borg 1.0.8).

If you want to use these older versions, you can disable the check by running:
borg upgrade --disable-tam ssh://12154@ch-s012.rsync.net/./borg-repo

See https://borgbackup.readthedocs.io/en/stable/changes.html#pre-1-0-9-manifest-spoofing-vulnerability for details about the security implications.

The remote repository was created successfully.

Set up passwordless SSH login to Rsync.net §

rsync.net provide excellent documentation on how to set up SSH keys to allow passwordless login. Esssentialy, you first generate your key pair:

$ ssh-keygen -t rsa

Then you upload the public key to rsync.net, overwriting the authorized_keys file:

$ scp ~/.ssh/id_rsa.pub 123@tv-s009.rsync.net:.ssh/authorized_keys

That’s it! To verify, try running a command over SSH:

$ ssh 123@tv-s009.rsync.net ls -lrtha

Write backup script §

Next we write a script to perform the backup. Following is what I am using:

#!/bin/sh

REPO_LOCAL=/mnt/backups/borg-repo
REPO_REMOTE=12154@ch-s012.rsync.net:borg-repo

export BORG_PASSPHRASE='yoursecretpassphrase'

echo "Running backup job"
date "+%Y-%m-%d %H:%M:%S"

# Bail if borg is already running, maybe previous run didn't finish
ps cax | grep python3.6 > /dev/null
if [ $? -eq 0 ]; then
    echo "Backup already running"
    exit
fi

echo "Creating local backup"
date "+%Y-%m-%d %H:%M:%S"

borg create -v --stats                                     \
    $REPO_LOCAL::'{hostname}-{utcnow:%Y-%m-%dT%H:%M:%S}'   \
    /mnt/media/photos                                      \

echo "Creating remote backup"
date "+%Y-%m-%d %H:%M:%S"

borg create -v --stats                                     \
    $REPO_REMOTE::'{hostname}-{utcnow:%Y-%m-%dT%H:%M:%S}'  \
    /mnt/media/photos                                      \

# Use the `prune` subcommand to maintain 7 daily, 4 weekly and 6 monthly
# archives of THIS machine. The 'prefix is very important to  limit prune's
# operation, else it would prune other matching archives.

echo "Pruning local backups"
date "+%Y-%m-%d %H:%M:%S"

borg prune -v --list $REPO_LOCAL --prefix '{hostname}-' \
    --keep-daily=7 --keep-weekly=4 --keep-monthly=6

echo "Pruning remote backups"
date "+%Y-%m-%d %H:%M:%S"

borg prune -v --list $REPO_REMOTE --prefix '{hostname}-' \
    --keep-daily=7 --keep-weekly=4 --keep-monthly=6

echo "Finished backup job"
date "+%Y-%m-%d %H:%M:%S"

This script creates a backup and stores it locally, then creates a backup and stores it remotely. It then prunes the local and remote backups. At various points it logs with timestamps.

Copy script into jail §

To get this script inside the jail, I copy it from my laptop at 192.168.0.13 into the jail. I run the scp command from inside the jail in order to avoid the step of setting up an SSH server inside the jail, since one already runs on my laptop.

$ scp marbs@192.168.0.13:/home/marbs/borg/borg_backup_from_jail.sh .

Give it execute permissions:

$ chmod +x borg_backup_from_jail.sh

Set borg passphrase in shell §

This section is optional, but it is useful because it will allow you to run borg manually without being prompted for the passphrase each time, e.g. if you want to use borg list to inspect the repositories.

Let’s set the BORG_PASSPHRASE environment variable. The fnbbu jail uses the csh shell by default, so if we store the passphrase in .cshrc it will be loaded automatically when the shell opens when we log in to the jail.

$ vi .cshrc

I append the following line to the existing list of setenv, using tabs rather than spaces for the gaps, because that is what the others have done.

setenv  BORG_PASSPHRASE yoursecretpassphrase

Test that the script runs manually §

Run the command and verify it works as expected:

$ ./borg_backup_from_jail.sh

Set up cron to run script daily §

Let’s use cron to schedule this script to run daily. Open up crontab for editing:

crontab -e

I’m going to schedule the backup to run at 5 minutes past midnight, every day. Use -x to get sh to log debug info (e.g. each command that is executed), and use >> /root/backup.log to keep a log of the backups in case you want to debug them, or verify they are working as expected.

0 5 * * * sh -x /root/borg_backup_from_jail.sh >> /root/backup.log

Check over the next few days and weeks to verify that it is backing up and pruning as expected. Read up on pruning to better understand how it works.

Next up, the most important step of all: verify that you can restore from backup.

Generally helpful links:

Possibly useful in future:

https://torsion.org/borgmatic/

All notes in this series: