The Important Files: Part 4

29 July 2018

All notes in this series:

In these notes I create a FreeNAS jail (using iocage rather than warden), install borg inside the jail, and use borg to communicate with a remote borg server hosted by rsync.net. Initially I try using Wasabi rather than rsync.net, but decide against Wasabi.

As usual, the notes were gathered as I ran through the process, and tidied up a little afterwards, so this is not a slick guide. I go back on myself and try alternative routes.

Set up jail with borg installed §

Going to install borg in a FreeNAS jail and set it up to back up to Wasabi. Here are the official borg installation instructions.

Through the FreeNAS UI, I create jail named “borg-jail”.

(NOTE: I am using FreeNAS 11.1. Later in these notes I find out that FreeNAS is in the process of transitioning from one “jail manager” to another. From FreeNAS 11.2 onwards “iocage” will be the supported jail, but prior to 11.2 “warden” is the supported jail. FreeNAS 11.1 includes both iocage and warden, which allows you to prepare for the transition, however jails created in the UI of FreeNAS 11.1 are warden jails. At a later step I retrace my steps and recreate the jail in the CLI using iocage).

In the UI, open jail console.

Find the borg package:

# pkg search borg
py27-borg.localrole-3.0.2_1    PAS plugin which can manage local roles via an ad
apter lookup                                                                    
py36-borgbackup-1.1.6          Deduplicating backup program

The one we want is borgbackup. Install the package:

# pkg install py36-borgbackup-1.1.6
Updating FreeBSD repository catalogue...                                        
FreeBSD repository is up to date.                                               
All repositories are up to date.                                                
The following 9 package(s) will be affected (of 0 checked):                     
                                                                                
New packages to be INSTALLED:                                                   
        py36-borgbackup: 1.1.6                                                  
        python36: 3.6.6_1                                                       
        compat7x-amd64: 7.4.704000.201310.1                                     
        liblz4: 1.8.2,1                                                         
        py36-setuptools: 40.0.0                                                 
        py36-msgpack: 0.5.6                                                     
        py36-llfuse: 1.2_1                                                      
        fusefs-libs: 2.9.7                                                      
        zstd: 1.3.4                                                             
                                                                                
Number of packages to be installed: 9                                           
                                                                                
The process will require 132 MiB more space.                                    
22 MiB to be downloaded.                                                        
                                                                                
Proceed with this action? [y/N]:

Add storage to jail §

This is so that borg can read the storage to back it up.

Select the jail in the UI and click “Add storage” at the bottom.

Set source to /mnt/storage/media.

Set destination to /mnt/media.

Set as read-only. Borg only needs to read from this mount, and by not allowing writes we avoid the possibility of a bug in borg (or in our mis-configuration of borg) potentially deleting the very data we are trying to preserve.

Create borg-backups storage §

As well using a remote borg server to store backups remotely, I will be using another local mount to store backups locally.

Through the UI, create a new dataset at /mnt/storage/backups.

Add borg-backups storage to jail §

Set source to /mnt/storage/backups.

Set destination to /mnt/backups.

Do not set as read-only. Borg needs to write to this mount in order to store the backup repository.

Allow sshing into FreeNAS jail §

Through the UI, you can open up a console into the jail. If you want to be able to remotely SSH into the jail, follow these steps:

(NOTE: Again, this is not really necessary because further on I switch to using iocage rather than warden. I found the console in the FreeNAS UI clunky for interacting with the jail and wanted a real terminal, so decided to set up SSH. Further on when I switch to iocage this is no longer necessary, since iocage jails can only be accessed through the CLI in FreeNAS 11.1).

Edit rc.conf:

$ vi /etc/rc.conf

$ sshd_enable="YES"

Edit sshd_config:

$ vi /etc/ssh/sshd_config

Start the SSH service:

$ service ssh start

By default the root jail user has no password, so one needs to be set in order to log in via SSH:

$ passwd

Create the backup repository §

Now let’s use borg to create a local borg repository:

$ borg init --encryption=repokey /mnt/backups/borg-repo

Unfortunately, the above command gives the following error:

Undefined symbol fdatasync

It turns out I am not the first to encounter this issue. The advice is to use an iocage jail rather than a warden jail, since warden jails are EOL for FreeNAS 11.1.

Jails: warden (old) vs iocage (new) §

FreeNAS v11.2 was scheduled to come out 20 days ago on 2018-07-09. It should include iocage as the default jail manager with the UI.

Instead of waiting for that release, let’s see if we can use the new iocage jail manager with our current installation of FreeNAS 11.1.

Creating a jail through the UI creates a warden jail by default.

To create an iocage jail it must be done through the CLI.

Switch back to the FreeNAS CLI. No more UIs from here on!

Before you can make a jail with iocage, it needs to fetch a release. Since we are using FreeNAS 11.1, we must select the 11.1-RELEASE option. (At least, I tried using 11-2.RELEASE and although it downloaded, it was not possible to create a jail).

root@freenas:~ # iocage fetch
Setting up zpool [storage] for iocage usage
 If you wish to change please use "iocage activate"
Creating storage/iocage
Creating storage/iocage/download
Creating storage/iocage/images
Creating storage/iocage/jails
Creating storage/iocage/log
Creating storage/iocage/releases
Creating storage/iocage/templates
[0] 9.3-RELEASE (EOL)
[1] 10.1-RELEASE (EOL)
[2] 10.2-RELEASE (EOL)
[3] 10.3-RELEASE (EOL)
[4] 10.4-RELEASE
[5] 11.0-RELEASE (EOL)
[6] 11.1-RELEASE
[7] 11.2-RELEASE

Type the number of the desired RELEASE
Press [Enter] to fetch the default selection: (11.1-RELEASE)
Type EXIT to quit:
Fetching: 11.1-RELEASE

Downloading : MANIFEST [####################] 100%
Downloading : base.txz [####################] 100%
Downloading : lib32.txz [####################] 100%
Downloading : doc.txz [####################] 100%
Downloading : src.txz [####################] 100%
Extracting: base.txz...
Extracting: lib32.txz...
Extracting: doc.txz...
Extracting: src.txz...

* Updating 11.1-RELEASE to the latest patch level...
src component not installed, skipped
Looking up update.FreeBSD.org mirrors... 2 mirrors found.
Fetching public key from update4.freebsd.org... done.
Fetching metadata signature for 11.1-RELEASE from update4.freebsd.org... done.
Fetching metadata index... done.
Fetching 2 metadata files... done.
Inspecting system... done.
Preparing to download files... done.
Fetching 153 patches.....10....20....30....40....50....60....70....80....90....100....110....120....130....140....150. done.
Applying patches... done.

The following files will be updated as part of updating to 11.1-RELEASE-p11:
/bin/freebsd-version
/bin/pgrep
/bin/pkill
...[snip]...
/usr/share/zoneinfo/Pacific/Pago_Pago
/usr/share/zoneinfo/Pacific/Tongatapu
/usr/share/zoneinfo/zone.tab

WARNING: FreeBSD 11.1-RELEASE is approaching its End-of-Life date.
It is strongly recommended that you upgrade to a newer
release within the next 2 months.
src component not installed, skipped
Installing updates... done.
root@freenas:~ #

Now you can create a jail named borg-jail with iocage:

root@freenas:~ # iocage create -r 11.1-RELEASE -n borg-jail

Configure iocage jail networking §

Jails created through the UI seem to be automatically configured with an IP address, while iocage jails are not:

root@freenas:~ # iocage list
+-----+-----------+-------+--------------+-----+
| JID |   NAME    | STATE |   RELEASE    | IP4 |
+=====+===========+=======+==============+=====+
| 3   | borg-jail | up    | 11.1-RELEASE | -   |
+-----+-----------+-------+--------------+-----+

In order for it to be useful for our purposes, we need to give it an IP address. This can be done as follows. bge0 is the name of the network interface on the FreeNAS, 192.168.0.3/24 is the address and subnet that I am assigning to borg-jail.

root@freenas:~ # iocage set ip4_addr="bge0|192.168.0.3/24" borg-jail
Property: ip4_addr has been updated to bge0|192.168.0.3/24
root@freenas:~ # iocage list
+-----+-----------+-------+--------------+-------------+
| JID |   NAME    | STATE |   RELEASE    |     IP4     |
+=====+===========+=======+==============+=============+
| 3   | borg-jail | up    | 11.1-RELEASE | 192.168.0.3 |
+-----+-----------+-------+--------------+-------------+
root@freenas:~ #

Mount storage §

As with the warden jail, we need to configure this iocage jail to have storage. This can be done through the CLI as follows:

iocage fstab -a borg-jail "/mnt/storage/media  /mnt/media  nullfs  ro  0  0"
iocage fstab -a borg-jail "/mnt/storage/backups  /mnt/backups  nullfs  rw  0  0"

Now we can start the jail:

iocage start borg-jail

Unfortunately, upon starting I got this issue:

resource deadlock avoided

Again it turned out I am not the first person to encounter this.

Had this problem trying to start the jail after setting the fstab configuration:

Solving the resource deadlock issue §

Fortunately, I found that by simply deleting the jail and re-creating it the problem went away…!

This time however I named the jail fnbbu rather than borg-jail, short for FreeNAS Box Backer Upper.

Starting the jail §

This time when starting the jail I got errors about the mounted directories not existing. Fortunately simply creating the directories and then trying again seemed to fix things:

root@freenas:~ # iocage start fnbbu
* Starting fnbbu
  + Start FAILED
jail: mount.fstab: /mnt/iocage/jails/fnbbu/root/mnt/media: No such file or directory

root@freenas:~ # mkdir /mnt/iocage/jails/fnbbu/root/mnt/mediaroot@freenas:~ # iocage start fnbbu
* Starting fnbbu
  + Start FAILED
jail: mount.fstab: /mnt/iocage/jails/fnbbu/root/mnt/media: No such file or directory

root@freenas:~ # mkdir /mnt/iocage/jails/fnbbu/root/mnt/media
root@freenas:~ # iocage start fnbbu
* Starting fnbbu
  + Start FAILED
jail: mount.fstab: /mnt/iocage/jails/fnbbu/root/mnt/backups: No such file or directory

root@freenas:~ # mkdir /mnt/iocage/jails/fnbbu/root/mnt/backups
root@freenas:~ # iocage start fnbbu
* Starting fnbbu
  + Started OK
  + Starting services OK
root@freenas:~ #

Installing borg §

Now we can go inside our fnbbu jail and install borg:

root@freenas:~ # iocage console fnbbu

As before…

root@fnbbu:~ # pkg search borg
root@fnbbu:~ # pkg install py36-borgbackup-1.1.6

Tentative testing… §

Ran through some quick borg examples just to verify that things are working this time:

root@fnbbu:~ # borg init --encryption=repokey /mnt/backups/borg-repo

Creating the repository worked this time - hooray!

I won’t post the commands here, but at this point I did manage to create backups in the repository, and very importantly, managed to retrieve files out of the backup and restore them in place.

rclone §

(NOTE: At this point my plan was to use rclone to copy the borg repository to Wasabi. Later I change to instead simply using borg to create a remote backup inside a remote borg repository hosted by rsync.net, removing the need for rclone).

Install rclone in same jail as borg.

root@fnbbu:~ # pkg search rclone
bomberclone-0.11.9_3           Reimplementation of Atomic Bomber Man
rclone-1.38                    Sync files to and from various cloud services

The one we want is rclone, not bomberclone:

root@fnbbu:~ # pkg install rclone-1.38
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        rclone: 1.38

Number of packages to be installed: 1

The process will require 16 MiB more space.
4 MiB to be downloaded.

Proceed with this action? [y/N]: y
[fnbbu] [1/1] Fetching rclone-1.38.txz: 100%    4 MiB   3.9MB/s    00:01
Checking integrity... done (0 conflicting)
[fnbbu] [1/1] Installing rclone-1.38...
[fnbbu] [1/1] Extracting rclone-1.38: 100%
root@fnbbu:~ #

Set up rclone configuration for Wasabi:

rclone config

At this point I struggled to get rclone to talk to Wasabi. This prompted me to look into other options again.

rsync.net §

There is a special discount for rsync.net users who only want to use borg, and none of the other fancy features. More about that in the next installament of these notes.