Introduction
In my previous post, I showed how to use Avamar to protect S3/Bucket data. This time around, I’m going to look at the mechanics of it with NetWorker.
Like Avamar, this technique requires the configuration of a ‘proxy’ style system within your environment running Linux and the s3fs-fuse system. My test environment for this process involved:
- Data Domain Virtual Edition running DDOS 6.2.0.30
- NetWorker Server (CentOS 6.10/Final) running 19.2
- NetWorker Client (CentOS 7.7.1908) running 19.2
- s3fs-fuse 1.85-1.el7
- curl-7.29.0-54.el7
- libcurl-7.29.0-54.el7
I won’t go through the bucket setup and loading details in this post like I did with Avamar – please see the Avamar post for those details if you’re wanting more background there.
Now, using NetWorker to backup an s3fs-fuse mount requires a little bit more work. In normal backup operations, NetWorker has an approach to backup where it aborts/fails the backup if it sees that a sub-directory inode has changed between when it reads the content of the parent directory, and when it gets to backing up that subdirectory. Unfortunately, for some reason, subdirectories presented within the s3fs-fuse mount point trigger this behaviour.
Here’s an example of the output NetWorker gives:
32034:save: Warning inode number changed for /s3/bzip2-1.0.6/ may not be recoverable
In normal circumstances, when NetWorker hits that condition it doesn’t attempt to traverse into the subdirectory, which obviously presents a problem. However, there is a solution which I’ll take you through.
Just make NetWorker cross
In normal operations, NetWorker expects that every filesystem you want to be protected will have a different save set associated with it – filesystems of /, /usr, /usr/local and /data would each have a save operation triggered for them so you end up with savesets of /, /usr, /usr/local and /data. However, you can, in manual operations, use “save -x” to tell NetWorker to cross mount-points.
In such circumstances, NetWorker will still give the inode alert I mentioned earlier, but it will traverse into each subdirectory.
So once you’ve got an S3 or other S3-compatible bucket mounted on your filesystem, you can successfully protect its contents by running save -x against the path using the appropriate commands for pools and retention – e.g., in my lab if I wanted to save to the BoostBackup pool against the server ‘orilla’ with a retention of 31 days, I’d use the command:
# save -x -s orilla -b BoostBackup -e "+31 days"
I was pretty happy when I got to this point, but nothing beats a bit of scripting, right?
If you have to do something more than once, automate it
That was the motto of the Unix system administration group I joined in 1996. In fact, so strong was the motto of the group that I’ve spent the last 20+ years of my career watching the growth of automation as if it were something special and new with wry amusement.
So I wrote a simple script called saves3.pl, to help perform full and incremental manual saves (yes, you can still do a manual incremental save if you really want) of an s3fs mount point.
You can download saves3.pl in zip format here. Here’s the help output from it:
[root@abydos ~]# saves3.pl -h Usage: saves3.pl [-h] [-v] -d dir -b pool -s server [-l lvl] [-D] [-c client] [-e exp] Where: -h Print this help and exit. -v Print version number and exit. -d dir Path that S3 bucket is mounted at. -b pool NetWorker pool to write backup to. -s server NetWorker server to talk to (defaults to this host) -c client NetWorker client to associate the backup with (defaults to this host) -l lvl Backup level (full, or incr - defaults to full) -e exp Expiration date in valid nsr_date format. Defaults to "+31 days" -D Go into debug logging mode. (Writes to /nsr/applog). Performs a backup of an object store (e.g,. AWS S3) mounted via s3fs-fuse to the nominated NetWorker server/pool. While technically a manual backup, if 'incr' is specified as a level, we look for the most recent backup of the same path for this client and tag to that backup's savetime. This script requires the NetWorker extended client package to be installed. See comments at start of script for additional information.
(As the script notes in the help section, you will need the Extended Client package installed.)
There’s a bucket to backup, Dear Liza
So here’s a view of my bucket, mounted at /s3 on my client ‘abydos’:
[root@abydos ~]# ls /s3 bzip2-1.0.6 test.txt zip-3.0 [root@abydos ~]# cat /s3/test.txt This is a bucket object. Climate change is real and human made.
An example backup with logging enabled looks like the following in my lab:
[root@abydos ~]# saves3.pl -d /s3 -b BoostBackup -s orilla -D -e "+2 weeks"
With logging enabled, all execution data is written to /nsr/applogs using a filename of saves3_YYYY-MM-DD-HHMMSS.log. The output will be a mix of the output from the save command itself, and informational output from saves3.pl:
[root@abydos ~]# more /nsr/applogs/saves3_2019-12-01-170233.log Calling backup save -x -s orilla -c abydos.turbamentis.int -b BoostBackup -e "+2 weeks" /s3 181407:save: Step (1 of 7) for PID-5805: Save has been started on the client 'abydos.turbamentis.int'. 175313:save: Step (2 of 7) for PID-5805: Running the backup on the client 'abydos.turbamentis.int' for the selected save sets.
So you can see there’s nothing particularly special about the save command being executed.
So what about an incremental backup? Let’s start by copying some additional content into the mount point:
[root@abydos s3]# ls bzip2-1.0.6 test.txt zip-3.0 [root@abydos s3]# cp -R /usr/share/doc/tar-1.26 . [root@abydos s3]# ls bzip2-1.0.6 tar-1.26 test.txt zip-3.0
To run an incremental backup, it’s pretty much the same command, but we add a “-l incr“. This has saves3.pl check to see what the most recent backup for the mount point is, and use save’s “-t savetime” to chain the backup as an incremental.
[root@abydos /]# saves3.pl -d /s3 -b BoostBackup -s orilla -D -e "+2 weeks" -l incr
Now, I can use nsrinfo to check and confirm that only captured the incremental changes:
[root@abydos /]# mminfo -s orilla -q "client=abydos,name=/s3" -otR -r "savetime(26),nsavetime"
date time save time
01/12/19 17:09:26 1575180566
01/12/19 17:02:33 1575180153
(There’s no point asking for the level of the backup from mminfo because these will be flagged as manual backups.)
[root@abydos /]# nsrinfo -s orilla -t 1575180566 abydos scanning client `abydos' for savetime 1575180566(Sun 01 Dec 2019 17:09:26 AEDT) from the backup namespace on server orilla /s3/tar-1.26/AUTHORS /s3/tar-1.26/COPYING /s3/tar-1.26/ChangeLog /s3/tar-1.26/ChangeLog.1 /s3/tar-1.26/NEWS /s3/tar-1.26/README /s3/tar-1.26/THANKS /s3/tar-1.26/TODO /s3/tar-1.26/ /s3/ /s3// / 12 objects found
So you can see there it only saved the files that had changed since the previous backup.
Recovering
To recover data, you’ll simply use the standard NetWorker recover command – or NMC. Since I’m logged onto my client as I’m typing this, I’m using the CLI. For example, here’s a sample showing a relocated recovery:
[root@abydos /]# cd /s3
[root@abydos s3]# recover -s orilla
179040:recover: Step (1 of 11) for PID-5977: Recovery has been started on the client 'abydos.turbamentis.int'.
179733:recover: Step (2 of 11) for PID-5977: Checking if the set of exclusion files is given for the recovery of the selected files for the client 'abydos.turbamentis.int'.
179738:recover: Step (3 of 11) for PID-5977: Setting the server 'orilla' for the media database calls for the recovery of the selected files for the client 'abydos.turbamentis.int'.
179747:recover: Identified the interactive file level recovery with PID 5977 for the client 'abydos.turbamentis.int'. Updating the total number of steps from 11 to 5.
179748:recover: Step (4 of 5) for PID-5977: Initializing a recovery job with the nsrjobd for the recovery of the selected file(s) for the client 'abydos.turbamentis.int'.
Current working directory is /s3/
recover> ls
bzip2-1.0.6 tar-1.26 test.txt zip-3.0
recover> add -q *
23 file(s) marked for recovery
recover> relocate /tmp/nsr_recover
recover> verbose
6486:recover: verbose mode off
recover> recover
Volumes needed (all on-line):
BoostBackup.002 at neutronium.turbamentis.int_boostBackup
179728:recover: Initializing the recovery session with the NetWorker server 'orilla' to recover the selected files.
Recover start time: Sun 01 Dec 2019 17:15:51 AEDT
179718:recover: Initializing to recover multiple streams of data from the NetWorker server 'orilla'.
Requesting 2 recover session(s) from server.
179714:recover: Initializing the child threads to recover the given recovery stream from the NetWorker server 'orilla'.
179713:recover: Initializing to set up a connection to start reading the data from the recover stream.
Enabling compressed restore for save set ID '1390631538' with 'Data Domain' device path '/orilla/boostBackup'.
Successfully established the direct file retrieval session for save set ID '1390631538' with 'Data Domain' volume 'BoostBackup.002'.
179716:recover: Coordinating all the child recovery processes.
Enabling compressed restore for save set ID '1407408344' with 'Data Domain' device path '/orilla/boostBackup'.
Successfully established the direct file retrieval session for save set ID '1407408344' with 'Data Domain' volume 'BoostBackup.002'.
179717:recover: Releasing all the resources held by the recovery process.
179721:recover: All recovery streams from the NetWorker server 'orilla' are read successfully.
Recover completion time: Sun 01 Dec 2019 17:15:53 AEDT
To verify the recovery I checked the relocated ‘test.txt’ file, previously shown in the original location:
[root@abydos ~]# cat /tmp/nsr_recover/test.txt This is a bucket object. Climate change is real and human made.
And now, a recovery directly back into the bucket:
[root@abydos ~]# cd /s3
[root@abydos s3]# ls
bzip2-1.0.6 tar-1.26 test.txt zip-3.0
[root@abydos s3]# rm -rf *
[root@abydos s3]# recover -s orilla
179040:recover: Step (1 of 11) for PID-6026: Recovery has been started on the client 'abydos.turbamentis.int'.
179733:recover: Step (2 of 11) for PID-6026: Checking if the set of exclusion files is given for the recovery of the selected files for the client 'abydos.turbamentis.int'.
179738:recover: Step (3 of 11) for PID-6026: Setting the server 'orilla' for the media database calls for the recovery of the selected files for the client 'abydos.turbamentis.int'.
179747:recover: Identified the interactive file level recovery with PID 6026 for the client 'abydos.turbamentis.int'. Updating the total number of steps from 11 to 5.
179748:recover: Step (4 of 5) for PID-6026: Initializing a recovery job with the nsrjobd for the recovery of the selected file(s) for the client 'abydos.turbamentis.int'.
Current working directory is /s3/
recover> verbose
6486:recover: verbose mode off
recover> add *
23 file(s) marked for recovery
recover> recover
Volumes needed (all on-line):
BoostBackup.002 at rd=oa.turbamentis.int:neutronium.turbamentis.int_boostBackup
With the recovery done, I quit the command with CTRL+D and checked out the recovered text file:
recover> 179068:recover: Step (5 of 5) for PID-6026: The recovery completed successfully on the client 'abydos.turbamentis.int'. [root@abydos s3]# cat /s3/test.txt This is a bucket object. Climate change is real and human made.
Wrapping Up
So there you go – to perform a backup using NetWorker of s3fs-fuse mounted data, all you need to do is call ‘save’ with a ‘-x’ option, but that doesn’t prevent you from getting the job done. (You could configure an automated clone job in NetWorker for the data for automated replication once the backup has been ingested.)