The Important Files: Part 7
All notes in this series:
- (1) The Important Files: Part 1
- (2) The Important Files: Part 2
- (3) The Important Files: Part 3
- (4) The Important Files: Part 4
- (5) The Important Files: Part 5
- (6) The Important Files: Part 6
- (7) The Important Files: Part 7
- (8) The Important Files: Part 8
Three years ago, I first set up my NAS using FreeNAS (see part 1). A few months later it was powered off, placed in a shipping container along with the rest of my household contents, as I moved across the world from the UK to Seattle. Now that I’ve been in Seattle for almost two years, I’ve finally got around to setting up my NAS again(!). These notes cover the process of getting it up and running, recovering from a dying USB stick, and bringing everything up-to-date.
An overview of this post is:
- Powered on NAS
- Unable to boot due to error message
- Determined error was due to USB stick used for FreeNAS OS dying
- Discovered that FreeNAS has become TrueNAS
- Used ddrescue to clone dying USB stick
- Resurrected NAS using cloned USB stick
- Decided to use SSD instead of USB stick to avoid same issue happening in future
- Ordered parts to fit extra SSD in NAS enclosure
- Installed FreeNAS on SSD
- Exported NAS config from USB stick
- Imported NAS config onto SSD
- Upgraded OS on SSD from FreeNAS to TrueNAS
- Verified backups still work
Powering on the NAS after two or more years gathering dust §
Powered on the NAS with a monitor connected. Was greeted with the following error:
Aug 14 04:30:58 freenas kernel: <118>Middleware startup is idle for more than 240 seconds
Aug 14 04:30:58 freenas kernel: <118>##############################################################
Aug 14 04:30:58 freenas kernel: <118>MIDDLEWARED FAILED TO START, SYSTEM WILL NOT BEHAVE CORRECTLY!
Aug 14 04:30:58 freenas kernel: <118>##############################################################
Aug 14 04:30:58 freenas kernel: <118>Loading early kernel modules:
Aug 14 04:30:58 freenas kernel: <118>Syncing disks...
Aug 14 04:30:58 freenas kernel: <118>Failed to run middleware call. Daemon not running?
Aug 14 04:30:58 freenas kernel: <118>Failed to run middleware call. Daemon not running?
Aug 14 04:30:58 freenas kernel: <118>Failed to run middleware call. Daemon not running?
The NAS was booting from a USB stick with FreeNAS installed. Some searching quickly indicated that the USB stick was most likely dying, causing this middleware error.
Further, I discovered that FreeNAS has become TrueNAS CORE. On my USB stick was installed FreeNAS 11.1. I decided to first fix the booting issue, sticking with FreeNAS 11.1, and only once that was fixed to upgrade to TrueNAS CORE 12.0.
Rescuing the USB stick §
Installing ddrescue §
dd can be used to create a perfect copy of a disk. ddrescue is the same idea, but designed to improve your chances of successfully copying a dying disk.
Note that there exists a dd_rescue and ddrescue, which serve the same purpose but are otherwise completely different! ddrescue sounded preferable.
On my Debian laptop, the package for ddrescue is gddrescue (note the “g”):
sudo apt install gddrescue
Recovery overview §
I will use ddrescue to copy the dying USB stick (aka source) onto a brand new, identical USB stick (aka destination).
Identifying the devices §
Before running ddrescue, we must be certain which drive is source and which is destination. To do that, first we list all drives without either USB sticks connected:
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 119.2G 0 disk
├─sda1 8:1 0 512M 0 part /boot/efi
├─sda2 8:2 0 110.8G 0 part /
└─sda3 8:3 0 7.9G 0 part [SWAP]
Then we connect the destination USB drive:
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 119.2G 0 disk
├─sda1 8:1 0 512M 0 part /boot/efi
├─sda2 8:2 0 110.8G 0 part /
└─sda3 8:3 0 7.9G 0 part [SWAP]
sdb 8:16 1 28.9G 0 disk
└─sdb1 8:17 1 28.9G 0 part /media/sam/KINGSTON
The destination shows up as /dev/sdb mounted to /media/sam/KINGSTON.
We don’t want it to be mounted, because we want to ensure ddrescue is the only thing writing to it. Unmount it:
sam@cpe1704tks:~/Documents/sam-hooke$ umount /media/sam/KINGSTON
sam@cpe1704tks:~/Documents/sam-hooke$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 119.2G 0 disk
├─sda1 8:1 0 512M 0 part /boot/efi
├─sda2 8:2 0 110.8G 0 part /
└─sda3 8:3 0 7.9G 0 part [SWAP]
sdb 8:16 1 28.9G 0 disk
└─sdb1 8:17 1 28.9G 0 part
sam@cpe1704tks:~/Documents/sam-hooke$
The destination USB drive is now unmounted.
Then we connect the source USB drive:
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 119.2G 0 disk
├─sda1 8:1 0 512M 0 part /boot/efi
├─sda2 8:2 0 110.8G 0 part /
└─sda3 8:3 0 7.9G 0 part [SWAP]
sdb 8:16 1 28.9G 0 disk
└─sdb1 8:17 1 28.9G 0 part
sdc 8:32 1 28.8G 0 disk
├─sdc1 8:33 1 512K 0 part
└─sdc2 8:34 1 28.8G 0 part
The source shows up as /dev/sdc and does not automatically mount (presumably because my Debian laptop does not understand ZFS).
The ddrescue command §
The GNU ddrescue manual includes good examples of how to run the tool.
Following Section 10, Example 2 from the official documentation, we are going to run the following commands. Note that sudo is required:
ddrescue -f -n /dev/sdc /dev/sdb rescue.mapfile
ddrescue -d -f -r3 /dev/sdc /dev/sdb rescue.mapfile
The flags are as follows:
-f: Overwrite output device or partition (even if it exists).-n: Skip the scraping phase.-d: Use direct disk access for input file (skip any cache).-r3: Exit after 3 retry passes (defaults to 0).
Actually running the ddrescue command §
Here goes:
(Started at 19:07)
$ sudo ddrescue -f -n /dev/sdc /dev/sdb rescue.mapfile
[sudo] password for sam:
GNU ddrescue 1.23
Press Ctrl-C to interrupt
ipos: 698417 kB, non-trimmed: 0 B, current rate: 24510 kB/s
opos: 698417 kB, non-scraped: 0 B, average rate: 24083 kB/s
non-tried: 30268 MB, bad-sector: 0 B, error rate: 0 B/s
rescued: 698417 kB, bad areas: 0, run time: 29s
pct rescued: 2.25%, read errors: 0, remaining time: 20m
time since last successful read: 0s
Copying non-tried blocks... Pass 1 (forwards)
After 1 hour (at 20:11)
GNU ddrescue 1.23
Press Ctrl-C to interrupt
ipos: 17225 MB, non-trimmed: 0 B, current rate: 24248 kB/s
opos: 17225 MB, non-scraped: 0 B, average rate: 4572 kB/s
non-tried: 13740 MB, bad-sector: 0 B, error rate: 0 B/s
rescued: 17225 MB, bad areas: 0, run time: 1h 2m 47s
pct rescued: 55.62%, read errors: 0, remaining time: 9m
time since last successful read: 0s
Copying non-tried blocks... Pass 1 (forwards)
Finish (at 20:50):
GNU ddrescue 1.23
Press Ctrl-C to interrupt
ipos: 30966 MB, non-trimmed: 0 B, current rate: 6094 kB/s
opos: 30966 MB, non-scraped: 0 B, average rate: 5007 kB/s
non-tried: 0 B, bad-sector: 0 B, error rate: 0 B/s
rescued: 30966 MB, bad areas: 0, run time: 1h 43m 3s
pct rescued: 100.00%, read errors: 0, remaining time: n/a
time since last successful read: n/a
Finished
Second stage:
$ sudo ddrescue -d -f -r3 /dev/sdc /dev/sdb rescue.mapfile
[sudo] password for sam:
GNU ddrescue 1.23
Press Ctrl-C to interrupt
Initial status (read from mapfile)
rescued: 30966 MB, tried: 0 B, bad-sector: 0 B, bad areas: 0
Current status
ipos: 0 B, non-trimmed: 0 B, current rate: 0 B/s
opos: 0 B, non-scraped: 0 B, average rate: 0 B/s
non-tried: 0 B, bad-sector: 0 B, error rate: 0 B/s
rescued: 30966 MB, bad areas: 0, run time: 0s
pct rescued: 100.00%, read errors: 0, remaining time: n/a
time since last successful read: n/a
Finished
Instantaneous!
The destination USB stick is now an identical clone of the source, with no errors from the copying process.
Boot from cloned USB stick §
Inserted the cloned USB stick into the NAS and booted. No error this time! Was able to log in through the web UI.
Export NAS config §
Through the FreeNAS web UI, browsed to: System -> General, then clicked “Save Config” and ticked “Export Password Secret Seed” (to allow the config to be used with a new boot device).
Bought SSD (and related parts) §
Bought the following:
- 500GB SSD - Perhaps a little overkill, but I’d like this to last longer than the USB stick did!
- SATA cable - For connecting SSD to motherboard.
- 4-pin molex to SATA power cable - To power the SSD using the 4-pin molex. The server has no spare SATA power cables, but does have a spare 4-pin molex intended for an optical drive.
- 5.25" to 2.5" mounting kit - To mount the SSD in the optical drive bay.
Install SSD §
- Removed the cloned USB stick.
- Put mounting kit in optical drive bay. (I have no plans to install an optical drive).
- This mounting kit was not ideal for my optical drive. The kit assumes you can screw it in place from outside the enclosure, which I cannot. Ideally I could do with a kit that slides into the grooves intended for an optical drive. However, friction alone seems enough to hold it firmly in place!
- Attach SSD into mounting kit.
- Connect SSD SATA power to 4-pin molex using convertor.
- Connect SSD SATA to motherboard with SATA cable.
Install FreeNAS to SSD §
Follow official FreeNAS 11.1 installation guide to install FreeNAS 11.1 to SSD. (Disconnected the pool drives first just to be absolutely certain I install to the right drive, though the install prompt lets you choose the target drive).
Import NAS config into FreeNAS §
Through the FreeNAS web UI, browse to: System -> General, then “Upload Config”. Supply the config that was saved earlier.
Shut down FreeNAS. Reconnect pool drives. Power up FreeNAS.
At this point, the recovery procedure is complete: we have migrated the OS from a dying USB stick to an SSD!
Update from FreeNAS to TrueNAS §
In the FreeNAS web UI, went to: System -> Update. Switched train to TrueNAS and let the system automatically update.
Verify backups still work §
SSH into the NAS, then list all the jails:
root@freenas:~ # iocage list
+-----+-----------+-------+--------------+-------------+
| JID | NAME | STATE | RELEASE | IP4 |
+=====+===========+=======+==============+=============+
| - | borg-jail | down | 11.1-RELEASE | 192.168.0.3 |
+-----+-----------+-------+--------------+-------------+
| - | fnbbu | down | 11.1-RELEASE | 192.168.0.3 |
+-----+-----------+-------+--------------+-------------+
The borg-jail is not used. The fnbbu (FreeNAS Box Backer-Upper) is the jail that performs the backups to rsync.net. It has not powered up automatically. Let’s start it up:
root@freenas:~ # iocage start fnbbu
* Starting fnbbu
+ Started OK
+ Starting services OK
root@freenas:~ # iocage list
+-----+-----------+-------+--------------+-------------+
| JID | NAME | STATE | RELEASE | IP4 |
+=====+===========+=======+==============+=============+
| - | borg-jail | down | 11.1-RELEASE | 192.168.0.3 |
+-----+-----------+-------+--------------+-------------+
| 1 | fnbbu | up | 11.1-RELEASE | 192.168.0.3 |
+-----+-----------+-------+--------------+-------------+
Now let’s log in to fnbbu and check that the cron job for the back ups is still configured:
root@freenas:~ # iocage console fnbbu
Last login: Sun Aug 19 09:58:30 on pts/0
FreeBSD 11.1-STABLE (FreeNAS.amd64) #0 r321665+79f05c3dd3d(HEAD): Wed Jan 23 12:02:34 EST 2019
Welcome to FreeBSD!
Release Notes, Errata: https://www.FreeBSD.org/releases/
Security Advisories: https://www.FreeBSD.org/security/
FreeBSD Handbook: https://www.FreeBSD.org/handbook/
FreeBSD FAQ: https://www.FreeBSD.org/faq/
Questions List: https://lists.FreeBSD.org/mailman/listinfo/freebsd-questions/
FreeBSD Forums: https://forums.FreeBSD.org/
Documents installed with the system are in the /usr/local/share/doc/freebsd/
directory, or can be installed later with: pkg install en-freebsd-doc
For other languages, replace "en" with a language code like de or fr.
Show the version of FreeBSD installed: freebsd-version ; uname -a
Please include that output and any error messages when posting questions.
Introduction to manual pages: man man
FreeBSD directory layout: man hier
Edit /etc/motd to change this login announcement.
root@fnbbu:~ # crontab -e
0 5 * * * sh -x /root/borg_backup_from_jail.sh >> /root/backup.log
It’s been a while (a few years) since a backup actually took place, so let’s manually kick one off now:
root@fnbbu:~ # ./borg_backup_from_jail.sh
Running backup job
[...]
70 minutes later, the backup finishes successfully. Viewing in rsync.net verifies that the backup has taken place. That was painless!
All notes in this series:
- (1) The Important Files: Part 1
- (2) The Important Files: Part 2
- (3) The Important Files: Part 3
- (4) The Important Files: Part 4
- (5) The Important Files: Part 5
- (6) The Important Files: Part 6
- (7) The Important Files: Part 7
- (8) The Important Files: Part 8