Avamar Basics – Using Post-Scripts

The Post-Script Challenge

I had an interesting conundrum come up recently. Someone needed to have Avamar backup an Oracle database, but exclude certain tablespaces from the backup. Now, that’s straightforward: you can create and run a custom RMAN script from the Oracle server, of course.

Where it got interesting was that the need was for Avamar to automatically run this backup, with the RMAN job picking up the retention type (daily, monthly, etc.) and expiration time.

Avamar writes the RMAN script though when you configure an Oracle backup policy – so the challenge was getting Avamar to run a custom Oracle RMAN script and make sure the script pulled in those parameters.

Challenge accepted!

The Thought Process

Now, pre-scripts and post-scripts can be quite useful for extending the functionality of a backup product, and this was to be no exception. The approach is straightforward – start a filesystem backup of something – either a small, token amount of data (e.g., /etc) or the actual filesystem backup.

I did toy with pre-scripts for a while, but Avamar has a token-based security method with an expiration time. What does that mean? Well, if you start a backup with a specific dataset (e.g., /etc/hosts), and that dataset doesn’t start backing up within 5 minutes (the default), the token issued expires. I.e., if you run an Oracle RMAN backup via your pre-script, it has to finish in 5 minutes in the default configuration. (This is unrelated to the timeout associated with pre-scripts or post-scripts.) Of course, you can extend the token expiration time, but I didn’t want to go down that path – security is there for a reason, after all.

A post-script is another thing though. The security-token is issued for the intended dataset – the filesystem dataset – not the manual backup that you’ll run in the post-script. So you’re OK to use the sequence below:

1. Run filesystem client backup for some nominated content.
2. Use a post-script to work kick off the Oracle RMAN backup with the appropriate configuration.

Number 1, above is pretty simple. And if I didn’t have to do any customisation, #2 would have been easy, too. But remember the requirement – use the job’s expiration time and retention type for the RMAN backup.

So in terms of process flow, I’m looking at something like this:

In short:

1. Backup policy runs filesystem backup with post-processing script configured.
2. Filesystem backup activities take place.
3. Post-script kicks off for RMAN backup.
4. Oracle backup data is written out.
5. Post-script finishes.

Aside: In my testing, I discovered the glorious Linux utility, stress. If you need to do testing on a system where you need a slow something down (in my case, slowing backups down to make sure they go past default timeouts), stress is a beautiful utility. I was able to grind my Oracle host to a screeching halt without actually crashing it – extending a ~5-minute backup to over two hours.

The RMAN Bits

Now, any backup product that works with Oracle RMAN has a very defined path for injecting its parameters into an RMAN script. Let’s look at a basic RMAN script:

connect target /

run {
   allocate channel t1 type 'SBT_TAPE' PARMS="SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar64.so" send '"--bindir=/usr/local/avamar/bin" "--flagfile=/home/oracle/rman/avamar/flagfile.txt" "--taskfile=/home/oracle/rman/avamar/taskfile.txt"';
   allocate channel t2 type 'SBT_TAPE' PARMS="SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar64.so" send '"--bindir=/usr/local/avamar/bin" "--flagfile=/home/oracle/rman/avamar/flagfile.txt" "--taskfile=/home/oracle/rman/avamar/taskfile.txt"';
   backup full filesperset 2 format 'FULL_%d_%U' (database);
   backup filesperset 5 (archivelog all delete input format 'a1_%s_%p');

   release channel t1;
   release channel t2;
}

The key thing to note of course is the allocate channel commands. That’s the avenue for you to inject backup-software parameters into an RMAN script. To simplify the parameter injection, Avamar uses two parameter files, viz.:

• Flagfile: Includes all the Avamar parameters required to authorise the connection to the Avamar server, identify the client, and yes, amongst other parameters, the expiration time.
• Taskfile: This gives details to Avamar about the type of operation (e.g., backup) and the number of channels you’re allocating. This information lets the AUI track the operation and even cancel it if necessary.

For example, here’s the taskfile referenced by the above script:

--no_of_channels=2
--operation=backup

That’s pretty straightforward: the number of channels matches what I’m allocating in the RMAN script, and it’s a backup operation. Now, the flagfile referenced would typically look like similar to the following:

--pidname=Oracle
--pidnum=1002
--logfile=/usr/local/avamar/var/clientlogs/avtar.log
--vardir=/usr/local/avamar/var/clientlogs
--id=userName
--ap=password
--path=/clients/aal.turbamentis.int
--server=kronos.turbamentis.int
--ddr=true
--ddr-index=1
--expires=31

Don’t worry about the password being there – Avamar includes functionality for encrypting a password for storage just on that client – and only usable on that specific client. (That way, you never have a plaintext password sitting in a file on the system.)

The Oracle client documentation for Avamar covers the key parameters, including as pidname and pidnum. (The pidnum of 1002 for instance means “this is a Linux client”.) A key value of note though is ‘expires‘. That can either be days, or Unix epoch time – above it would mean 31 days.

OK, so they’re the basic pieces if you’re just going to hop onto a Linux Oracle server and run the RMAN script. But there was the other requirement: adding expiration time and retention type into RMAN backup. You’ll note the retention type isn’t included in the above flagfile. That’s because it’s not there by default for a manually executed RMAN backup. But you can inject it if you want, using –retention-type=string where string can be a comma-separated sequence of the retention types this backup will be assigned – e.g., “daily,weekly,monthly”.

Now, when Avamar runs a pre-script or a post-script, it doesn’t pass across any environment variables that define the expiration time or retention type. How could I get my hands on them?

Avamar loves logfiles though. And into the log files it writes those details. And Perl loves to read log files — and I love to use Perl.

The Perl Script

So here’s the process in a bit more detail:

1. Backup policy starts a job for nominated data on the client.
2. Filesystem backup runs.
3. Post-script runs, executes a Perl process that does the following:
   1. Performs a process listing to find the running avtar process.
   2. Extracts the named logfile from the avatar process.
   3. Reads the logfile and extracts the expiration time (in epoch format), and retention type.
   4. Reads a ‘base’/template flag file that includes all parameters other than retention type/expiration time.
   5. Writes a new flagfile containing those extracted details.
   6. Runs the RMAN backup.

So the Perl script I came up with to do just that is below:

#!/usr/bin/perl -w

# This utility can be invoked by Avamar as a post-process option to build up a flagfile that
# allows a local RMAN file to be run with appropriate retention type and expiration time.
#
# The utility is provided as-is.

use strict;

# First, find our log file.
my $binary = "avtar";
my $cmd = "ps -eaf | grep $binary";
my $logfile = "not found";
if (open(CMD,"$cmd 2>&1 |")) {
	while (<CMD>) {
		my $line = $_;
		chomp $line;
		my @words = split(/\s+/,$line);
		foreach my $word (@words) {
			if ($word =~ /^\-\-logfile=\"(.*)\"/) {
				$logfile = $1;
			}
		}
	}
	close(CMD);
}

# Print the logfile name.
print "Logfile = $logfile\n";

if ($logfile eq "not found") {
	die "Unable to interpret the log file name from the process listing. Unable to continue.\n";
}

# Now, open the log file and find the expires flag.
my $expiryTime = -1;
my $retentionType = "none";
if (open(LOGGY,"<", $logfile)) {
	while (<LOGGY>) {
		my $line = $_;
		chomp $line;
		if ($line =~ /value=\"(\d+)\".*name=\"expire\"/) {
			$expiryTime = $1;
		}
		if ($line =~ /value=\"([^\"]*)\".*name=\"retention-type\"/) {
			$retentionType = $1;
		}
	}
	close(LOGGY);
}

print "Expiry time = $expiryTime\n";
print "Retention type = $retentionType\n";

if ($expiryTime < 1) {
	# Either fail, or use a default number of days.
	# Currently we run with an expiry time of 31 days -- but this probably should change to a hard
	# fail.
	$expiryTime = 31;
}

# base Flags = where the standard flags (except for expiry) are stored
my $flagBase = "/home/oracle/rman/avamar/flagfile.base";

# Flag file we create
my $flagFile = "/home/oracle/rman/avamar/flagfile.txt";

if ($expiryTime > 0) {
	if (open(FFR,$flagBase)) {
		if (open(FFW,">$flagFile")) {
			while (<FFR>) {
				print FFW $_;
			}
			close(FFR);
			print FFW "--expires=$expiryTime\n";
			print FFW "--retention-type=$retentionType\n";
		}
		close(FFW);
	}

	if (-f $flagFile) {
		# We assume we've successfully written the flagfile.
		system("chown oracle:oinstall $flagFile");
		system("su - oracle -c \"rman \@/home/oracle/rman/avamar/full.rman\"");
	}
}

I’m not professing to my script being a finished product – it’s a bit of a mixed bag in terms of error checking, but it is a proof-of-concept, so I’m OK with that.

I did some testing of two different backup configuration options — one was just some small content for the filesystem backup (/etc). The other was to do a more comprehensive filesystem backup. Both worked without a hitch.

The Backup Policy

The key policy setup details for this sort of configuration are within the dataset pane:

Now, there’s one other thing you may have to do on the client depending on the size of the database: adjust the post-script timeout value. You do this by making an entry in the avtar.cmd file (or even creating the file with the singular entry) specifying an appropriate length timeout. On Linux systems, that’s /var/avamar/avtar.cmd, and I created the following:

--run-at-end-clauses=timeout-seconds=7200

(That’s where that stress utility came in handy. It was the first time I’d had to adjust post-script timeouts and I wanted to make sure I was doing it right.)

With everything configured, the Activity monitor in Avamar showed it was a successful approach:

You can see the two backups here. The first backup (not highlighted) is the filesystem backup, and once it finishes, it kicks off the on-demand Oracle backup with the appropriate expiration time and retention type in place – that’s the >32GB Oracle RMAN backup.

Wrapping Up

This was a fun challenge, and a good way for me to dive more into Avamar/Oracle backups. It’s also a pertinent reminder: backup products won’t always do exactly what you want. You can’t have a checkbox option for everything, because the team behind any backup product won’t be able to think of every possible contingency for every possible company in the world. Instead, extensibility is a key attribute for enterprise backup software. While these days people think of extensibility in terms of REST APIs and all those funky things, we shouldn’t be afraid to get our hands dirty with regular scripting from time to time, either.