Sam Hooke

The Important Files: Part 10

TL;DR: Tried to run backup, it failed because the rsync.net username and hostname have changed. Updated the script and re-ran the backup, and it completed successfully after 3.5 hours. It took so long because this is the first time it has been run (successfully) in a few months.

The backup job is failing.

Let’s try and run it manually.

ssh root@192.168.1.10
iocage list
iocage console fnbbu

When I run borg_backup_from_jail.sh it asks for a password.

I’m guessing this is because I was migrated to a new host with a new username?

Let’s take a look at borg_backup_from_jail.sh:

root@fnbbu:~ # cat borg_backup_from_jail.sh
borg_backup_from_jail.sh
#!/bin/sh

REPO_LOCAL=/mnt/backups/borg-repo
REPO_REMOTE=98765@ab-c123.rsync.net:borg-repo

echo "***START***"
echo "Running backup job"
date "+%Y-%m-%d %H:%M:%S"

# Bail if borg is already running, maybe previous run didn't finish
ps cax | grep python3.6 > /dev/null
if [ $? -eq 0 ]; then
    echo "Backup already running"
    echo "***END***"
    exit
fi

export BORG_PASSPHRASE='yoursecretpassphrase'

echo "Creating local backup"
date "+%Y-%m-%d %H:%M:%S"

borg create -v --stats                                     \
    $REPO_LOCAL::'{hostname}-{utcnow:%Y-%m-%dT%H:%M:%S}'   \
    /mnt/media/photos                                      \

echo "Creating remote backup"
date "+%Y-%m-%d %H:%M:%S"

borg create -v --stats                                     \
    $REPO_REMOTE::'{hostname}-{utcnow:%Y-%m-%dT%H:%M:%S}'  \
    /mnt/media/photos                                      \

# Use the `prune` subcommand to maintain 7 daily, 4 weekly and 6 monthly
# archives of THIS machine. The prefix is very important to limit prune's
# operation, else it would prune other matching archives.

echo "Pruning local backups"
date "+%Y-%m-%d %H:%M:%S"

borg prune -v --list $REPO_LOCAL --prefix '{hostname}-' \
    --keep-daily=7 --keep-weekly=4 --keep-monthly=6

echo "Pruning remote backups"
date "+%Y-%m-%d %H:%M:%S"

borg prune -v --list $REPO_REMOTE --prefix '{hostname}-' \
    --keep-daily=7 --keep-weekly=4 --keep-monthly=6

echo "Finished backup job"
date "+%Y-%m-%d %H:%M:%S"
echo "***END***"

Yes, that has my old host and username hard-coded.

Okay, let’s try changing this line:

- REPO_REMOTE=98765@ab-c123.rsync.net:borg-repo
+ REPO_REMOTE=zx4455@zx4455.rsync.net

And try again:

root@fnbbu:~ # ./borg_backup_from_jail.sh

***START***
Running backup job
2024-09-30 16:56:32
Creating local backup
2024-09-30 16:56:32
Killed stale lock localhost@255332415579751.3181-0.
------------------------------------------------------------------------------
Archive name: fnbbu-2024-09-30T16:56:33
Archive fingerprint: 037114dde2430cabf62b0095a565e41b5312ac8a0d4fa2cfb31ba8cae1a726d8
Time (start): Mon, 2024-09-30 16:56:34
Time (end):   Mon, 2024-09-30 16:57:17
Duration: 43.15 seconds
Number of files: 45968
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:              307.72 GB            306.48 GB              7.95 MB
All archives:                5.31 TB              5.28 TB            303.34 GB

                       Unique chunks         Total chunks
Chunk index:                  155534              2699363
------------------------------------------------------------------------------
Creating remote backup
2024-09-30 16:57:21
The authenticity of host 'zx4455.rsync.net (77.109.148.29)' can't be established.
ECDSA key fingerprint is SHA256:aDNHfL1BivW6XWQ5ni1JgHCT3nMGNzzXM2zpGbNxr9M.
No matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)? yes
Remote: Warning: Permanently added 'zx4455.rsync.net' (ECDSA) to the list of known hosts.
Warning: The repository at location ssh://zx4455@zx4455.rsync.net/./borg-repo was previously located at ssh://98765@ab-c123.rsync.net/./borg-repo
Do you want to continue? [yN] y

So far so good. Just have to accept the new host.

And wait. Lots is backing up.

Computer went to sleep. Let’s quickly disable sleep and try again. (Could use screen but as a one-off disabling sleep temporarily is easier).

$ ssh root@192.168.1.10
root@192.168.1.10's password:
Last login: Mon Sep 30 16:50:31 2024 from 192.168.1.204
FreeBSD 12.2-RELEASE-p12 ec84e0c52a1(HEAD) TRUENAS

        TrueNAS (c) 2009-2022, iXsystems, Inc.
        All rights reserved.
        TrueNAS code is released under the modified BSD license with some
        files copyrighted by (c) iXsystems, Inc.

        For more information, documentation, help or support, go here:
        http://truenas.com
Welcome to FreeNAS

Warning: settings changed through the CLI are not written to
the configuration database and will be reset on reboot.

root@freenas:~ # iocage console fnbbu
Last login: Mon Sep 30 16:50:45 on pts/1
FreeBSD 12.2-RELEASE-p12 ec84e0c52a1(HEAD) TRUENAS

Welcome to FreeBSD!

Release Notes, Errata: https://www.FreeBSD.org/releases/
Security Advisories:   https://www.FreeBSD.org/security/
FreeBSD Handbook:      https://www.FreeBSD.org/handbook/
FreeBSD FAQ:           https://www.FreeBSD.org/faq/
Questions List: https://lists.FreeBSD.org/mailman/listinfo/freebsd-questions/
FreeBSD Forums:        https://forums.FreeBSD.org/

Documents installed with the system are in the /usr/local/share/doc/freebsd/
directory, or can be installed later with:  pkg install en-freebsd-doc
For other languages, replace "en" with a language code like de or fr.

Show the version of FreeBSD installed:  freebsd-version ; uname -a
Please include that output and any error messages when posting questions.
Introduction to manual pages:  man man
FreeBSD directory layout:      man hier

Edit /etc/motd to change this login announcement.
root@fnbbu:~ #
root@fnbbu:~ # ./borg_backup_from_jail.sh
***START***
Running backup job
2024-09-30 17:55:11
Creating local backup
2024-09-30 17:55:11
------------------------------------------------------------------------------
Archive name: fnbbu-2024-09-30T17:55:12
Archive fingerprint: 4bc561a84e8a45b5d568d2145ed0a13e2de944d0b00b885c717f1d5c9e8cfcbc
Time (start): Mon, 2024-09-30 17:55:13
Time (end):   Mon, 2024-09-30 17:55:57
Duration: 44.30 seconds
Number of files: 45968
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:              307.72 GB            306.48 GB              7.96 MB
All archives:                5.62 TB              5.59 TB            303.35 GB

                       Unique chunks         Total chunks
Chunk index:                  155661              2854840
------------------------------------------------------------------------------
Creating remote backup
2024-09-30 17:56:01
Remote: compaction freed about 1.68 kB repository space.
Remote: compaction freed about 1.74 kB repository space.
Remote: compaction freed about 1.74 kB repository space.

An hour passes… still waiting…

Left it running overnight.

Next day:

Remote: compaction freed about 1.74 kB repository space.
Remote: compaction freed about 1.74 kB repository space.
Remote: compaction freed about 1.73 kB repository space.
Remote: compaction freed about 1.74 kB repository space.
Remote: compaction freed about 1.74 kB repository space.
Remote: compaction freed about 1.74 kB repository space.
Remote: compaction freed about 1.74 kB repository space.
Remote: compaction freed about 1.74 kB repository space.
Remote: compaction freed about 1.74 kB repository space.
Remote: compaction freed about 1.74 kB repository space.
------------------------------------------------------------------------------
Archive name: fnbbu-2024-09-30T17:56:02
Archive fingerprint: 7b0205db2ed7140a766f555ef586e9ef7e77a0e15aecc5eb67044575f1ea84ca
Time (start): Mon, 2024-09-30 17:56:05
Time (end):   Mon, 2024-09-30 21:30:20
Duration: 3 hours 34 minutes 14.69 seconds
Number of files: 45968
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:              307.72 GB            306.48 GB             27.98 GB
All archives:                4.97 TB              4.95 TB            301.49 GB

                       Unique chunks         Total chunks
Chunk index:                  154706              2534458
------------------------------------------------------------------------------
Pruning local backups
2024-09-30 21:30:25
Keeping archive: fnbbu-2024-09-30T17:55:12            Mon, 2024-09-30 17:55:13 [4bc561a84e8a45b5d568d2145ed0a13e2de944d0b00b885c717f1d5c9e8cfcbc]
Pruning archive: fnbbu-2024-09-30T16:56:33            Mon, 2024-09-30 16:56:34 [037114dde2430cabf62b0095a565e41b5312ac8a0d4fa2cfb31ba8cae1a726d8] (1/3)
Pruning archive: fnbbu-2024-09-30T16:54:14            Mon, 2024-09-30 16:54:15 [593d4023013def440aeea1404d72806041686483a93892e8c4b60b70a807e5ca] (2/3)
Pruning archive: fnbbu-2024-09-30T15:49:35            Mon, 2024-09-30 15:49:36 [339850d3530c42f36ba38424af3710d487bdb98191d14c206bd2993e0ca45e54] (3/3)
Keeping archive: fnbbu-2024-09-19T05:00:13            Thu, 2024-09-19 05:00:14 [23be4fd6a7b7126cc04975d062e2d0442a18f4b9dcca10caf87cfcd30ca2234c]
Keeping archive: fnbbu-2024-09-04T05:00:08            Wed, 2024-09-04 05:00:09 [88ec88bf32d55f5e3b26613582e23fcf27f4b768186ccf7eaa3e864cd2e21942]
Keeping archive: fnbbu-2024-09-03T05:00:11            Tue, 2024-09-03 05:00:12 [b60f356e736a891aa6aebd5b04cf9335d86aef2a32968c87cff0ff9e4ea17a08]
Keeping archive: fnbbu-2024-09-02T05:00:05            Mon, 2024-09-02 05:00:06 [0db220ed0b5b01001f5e014b203dffe6a74a0ac02f4a517dba1d017bec275980]
Keeping archive: fnbbu-2024-04-27T05:00:07            Sat, 2024-04-27 05:00:08 [fa960ede8f11f6b4a3c0c8fdabfc492f7db3c8d439b84b1bccd64d7dd56c513c]
Keeping archive: fnbbu-2024-04-26T05:00:12            Fri, 2024-04-26 05:00:13 [7265f7b06a74f25d0ea8572ddcd429e350dc8a1e52eb9f6c190baa159c7ba7c8]
Keeping archive: fnbbu-2024-04-21T05:00:02            Sun, 2024-04-21 05:00:03 [8b1144e6e241b3a557d9a71aea604593351b13e726acd300e2f353a740765907]
Keeping archive: fnbbu-2024-04-12T17:05:21            Fri, 2024-04-12 17:05:22 [c03fc2932f443e02e0be96fbda83492ef4bad30a328282b78fd170e50128c137]
Keeping archive: fnbbu-2024-04-07T05:00:12            Sun, 2024-04-07 05:00:13 [67f973744caef502eaf6dfb7ea6e36f3a2ec5b9326c68c8b2234668f6ee1aa6b]
Keeping archive: fnbbu-2024-01-05T05:00:09            Fri, 2024-01-05 05:00:10 [d899f30c3379e802499540d8088073bf7ec4ee7ef03c745b9eed60b3235d8253]
Keeping archive: fnbbu-2023-12-31T05:00:04            Sun, 2023-12-31 05:00:05 [5ad329200cc1d813ab8a3fe3af8aa4860ebf74c10f001cee1930e549d6200998]
Keeping archive: fnbbu-2023-10-14T05:00:08            Sat, 2023-10-14 05:00:09 [f20fd2dcd1649a4bfcd1d796f13e60e6e591c08f4f32ee4439d6c54f6c89d867]
Keeping archive: fnbbu-2023-05-23T05:00:06            Tue, 2023-05-23 05:00:07 [f88a568e32a714dd5478264188419b70cec0ee14e73a7feb30865c1d8618133e]
Keeping archive: fnbbu-2023-04-17T05:00:03            Mon, 2023-04-17 05:00:04 [5cf3beea47595b09963c80161c63cf595798d29d8fc2c69e01ed9e59d3afc5fe]
Keeping archive: fnbbu-2023-03-17T05:00:07            Fri, 2023-03-17 05:00:08 [d336298496187c5bddd32ad7bc3f6e5d01b51706e37d4db016b246497f778307]
Keeping archive: fnbbu-2023-02-28T05:00:07            Tue, 2023-02-28 05:00:08 [9108fd5b0a52e94e80a4444e029ba8c57ecc197007db42c59da2c1497e6cdc23]
Pruning remote backups
2024-09-30 21:30:41
Keeping archive: fnbbu-2024-09-30T17:56:02            Mon, 2024-09-30 17:56:05 [7b0205db2ed7140a766f555ef586e9ef7e77a0e15aecc5eb67044575f1ea84ca]
Pruning archive: fnbbu-2024-09-30T16:57:22.checkpoint Mon, 2024-09-30 16:57:32 [4d439c77a6b226fef9604bdf946ca63b528d75022ee002b3469f5b1ca86343d8] (1/2)
Keeping archive: fnbbu-2024-04-12T17:06:23            Fri, 2024-04-12 17:06:26 [7537cc5b56001aa788a288cda57f3da210da71bfb8816d3e24d3085c21cb468a]
Keeping archive: fnbbu-2024-04-09T05:00:59            Tue, 2024-04-09 05:01:02 [f8eb68ce5668d90c1e62d35cf9ef37ee5b77ecfbae48f6c2e6eb78327a4060a1]
Keeping archive: fnbbu-2024-04-08T05:01:06            Mon, 2024-04-08 05:01:09 [a42c470c93f1de8fe97d9447601e60d5e86b9916c3715429709c82f25e10c6fe]
Keeping archive: fnbbu-2024-04-07T05:01:13            Sun, 2024-04-07 05:01:16 [5c837da29dc332d7eb20ab643d2174d166b54db1c959608bf715f04137edd889]
Keeping archive: fnbbu-2024-01-05T05:01:03            Fri, 2024-01-05 05:01:06 [2f1c01cdf6826d691038f569d6932ad9417b4cb7a3877d95ccf6ba153d39e25b]
Keeping archive: fnbbu-2024-01-04T05:00:56            Thu, 2024-01-04 05:00:58 [2c42543f1a59a3336597ef0f50388567ce8d6236a8bca1121a7ded5c86e88be1]
Pruning archive: fnbbu-2024-01-03T05:00:50            Wed, 2024-01-03 05:00:54 [198006e9c13c19b6e5c8d8976358a6be1638b3f132ce3b5f7379d74b0dd07144] (2/2)
Keeping archive: fnbbu-2023-12-31T05:04:00            Sun, 2023-12-31 05:04:03 [e9f74b65f8f4690c4a694161db190673d4a2ac9d42d8f2d56d02d0f42f60ce8a]
Keeping archive: fnbbu-2023-12-22T05:00:56            Fri, 2023-12-22 05:01:00 [7d16c08e2b9baa4204026ded25ff8d675396627ed384dff83cebd46fe8cd0d66]
Keeping archive: fnbbu-2023-12-17T05:06:48            Sun, 2023-12-17 05:06:52 [0893b0ce5b750c76d7a249262ca40ebdb417f459a70f58783addc4bfee812f64]
Keeping archive: fnbbu-2023-10-14T05:00:57            Sat, 2023-10-14 05:01:00 [510f0ab0fe4a61212f13fee9d6a0b29f71b5e6bfd86a4d9e9f0f4643bd6aa297]
Keeping archive: fnbbu-2023-05-23T05:08:31            Tue, 2023-05-23 05:08:36 [ac6d428a5b4bf4361b695614a1c81a0c94067f9c5afc8a34da475dbd1ee227b8]
Keeping archive: fnbbu-2023-04-17T05:00:52            Mon, 2023-04-17 05:00:57 [5c4836675a431da320f5b0ce54ed727a484a2f0aa8119f4800b8726eca114c53]
Keeping archive: fnbbu-2023-03-17T05:00:54            Fri, 2023-03-17 05:00:59 [0baccfb4a66976075ce4337065d9a8ea7777621752c8bf6ef7dd96691494e178]
Keeping archive: fnbbu-2023-02-28T05:00:54            Tue, 2023-02-28 05:00:59 [dbffcba395442ea550f309a00641d526fd4cb125d5b99e2bc97231925711784a]
Keeping archive: fnbbu-2023-01-16T05:00:46            Mon, 2023-01-16 05:00:51 [b13699f3afe8c2eb85ddba0f06fb05c7c614b6e408d771daebe2dc2abca16f0d]
Keeping archive: fnbbu-2022-12-31T05:00:55            Sat, 2022-12-31 05:01:00 [9303f3a234b64c2dc1c6b992e68715f88107d749619e0e943f873c0db7ffdace]
Remote: compaction freed about 12.19 MB repository space.
Finished backup job
2024-09-30 21:30:55
***END***
root@fnbbu:~ #

The duration was:

3 hours 34 minutes 14.69 seconds

Backup successful. It was the first time in 268 days, so there was a lot to catch up on.

Re-enabled computer to sleep after 30 minutes.

All notes in this series: