A question I’ve been asked a few times is “how do we use mminfo to determine dependencies between PSS savesets?”
When we do a Parallel Save Stream (PSS) saveset, NetWorker breaks down the saveset into multiple streams and names them using the old method used back in NetWorker v5.x/etc. days for continuation savesets. Back in those days, NetWorker used a cut-off limit of 2GB when writing a saveset, so if you needed to backup a 5GB filesystem called ‘/home’, you’d get saveset names of:
/home
<1>/home
<2>/home
Now, we don’t need to do those continuation savesets any more, so the nomenclature gets used for identifying PSS savesets. If you go to recover from a PSS backup, NetWorker automatically uses the various PSS savesets that were generated, and it handles recycling them appropriately – i.e., it doesn’t think that /home and <1>/home are different savesets, as such.
But how?
My first thought when I was asked this was that given we were using continuation saveset nomenclature, it would be as easy as chaining via a “pssid” option in mminfo. However, that’s not the case, so I did some more digging and found out how we get this information out of mminfo. (Thanks, Matt, Kumar, Pavan!)
So, let’s start with an example of a PSS backup I generate in my lab of a saveset, “/Volumes/Documents/Alteran” on the host rama.
$ mminfo -q "client=rama.turbamentis.int,name=/Volumes/Alteran/Documents,level=full,savetime>=3 weeks ago"
volume client date size level name
Backup.03 rama.turbamentis.int 03/03/19 515 GB full /Volumes/Alteran/Documents
That’s our starting point. The next thing is to work out what the saveset ID is so we can do a bit more digging, so our query becomes:
$ mminfo -q "client=rama.turbamentis.int,name=/Volumes/Alteran/Documents,level=full,savetime>=3 weeks ago" -r ssid
3178921591
The saveset ID helps here because we’re going to use the “super verbose mode” option in mminfo, -S. Now you’ll note I’m deliberately querying based on the core saveset name. This is effectively the ‘parent’ for the saveset. So, my new query based on saveset ID and focusing on the super verbose output is:
$ mminfo -S -q "ssid=3178921591"
ssid=3178921591 savetime=03/03/19 00:00:35 (1551531635) rama.turbamentis.int:/Volumes/Alteran/Documents
level=full sflags=vF size=528098212924 files=10282 insert=03/03/19
create=03/03/19 complete=03/03/19 browse=03/04/19 23:59:59 retent=03/04/19 23:59:59
clientid=c7237fcb-00000004-58eb5ef3-58eb5ef2-00375000-9f778f56
*backup start time: 1551531632;
*mbs anchor saveset time: 1551531632;
*mbs dependents: \
4d303631-00000006-be7a7e75-5c7a7e75-01435000-9f778f56,
0142dba6-00000006-bf7a7e75-5c7a7e75-01425000-9f778f56,
5d1f8601-00000006-c07a7e75-5c7a7e75-01415000-9f778f56;
*policy action jobid: 1729123;
*policy action name: "backup: 1551531635";
*policy name: "Platinum: 1551531635";
*policy workflow name: "Documents: 1551531635";
group: Platinum_Documents;
saveset features: CLIENT_SAVETIME;
Clone #1: cloneid=1551531635 time=03/03/19 00:00:35 retent=03/04/19 flags=
frag@ 0 volid=3844106399 file/rec= 0/0 rn=0 last=03/03/19
I’ve italicised the output you want to look for in this scenario: you’ll see a section of the output called “mbs dependents” – these are the <x>Name savesets that are associated with the core saveset we queries from. If you’re not familiar with that lengthy hexadecimal string format, it’s what we refer to as the “long SSID”; in actual fact, you can get the long SSID for any saveset just by giving a length attribute to the ssid output field:
$ mminfo -q "client=rama.turbamentis.int,name=/Volumes/Alteran/Documents,level=full,savetime>=3 weeks ago" -r "ssid,ssid(53)"
ssid ssid
3178921591 79eeaeec-00000006-bd7a7e77-5c7a7e77-01445000-9f778f56
The long SSID is useful not only in this case for us to do the tracing of dependent saveset sections, but also if you’re using an advanced file type device, you’ll actually find the saveset on disk using that filename.
You can also query based on the long SSID as well, so in this case now I’ve got the details of the ‘mbs dependents’, I can query any one of those to find out their specific details as follows:
$ mminfo -S -q "ssid=4d303631-00000006-be7a7e75-5c7a7e75-01435000-9f778f56"
ssid=3195698805 savetime=03/03/19 00:00:34 (1551531634) rama.turbamentis.int:<3>/Volumes/Alteran/Documents
level=full sflags=vF size=554835612580 files=3896 insert=03/03/19
create=03/03/19 complete=03/03/19 browse=03/04/19 23:59:59 retent=03/04/19 23:59:59
clientid=c7237fcb-00000004-58eb5ef3-58eb5ef2-00375000-9f778f56
*backup start time: 1551531632;
*mbs dependents: /Volumes/Alteran/Documents;
*policy action jobid: 1729123;
*policy action name: "backup: 1551531632";
*policy name: "Platinum: 1551531632";
*policy workflow name: "Documents: 1551531632";
group: Platinum_Documents;
saveset features: CLIENT_SAVETIME;
Clone #1: cloneid=1551531632 time=03/03/19 00:00:32 retent=03/04/19 flags=
frag@ 0 volid=3844106399 file/rec= 0/0 rn=0 last=03/03/19
You’ll notice in this output, as a child PSS entry, the mbs dependents simply gives the parent saveset name.
So, how can you use this in scripting? Well below, you’ll find a very rough and ready Perl script that builds a view of the dependent details for a PSS saveset. I can guarantee the following is not in this script:
- Any form of error checking
- Handling Windows saveset names
- Handling different pools
The script has to be run on the NetWorker backup server at the moment, using the arguments “-c clientName” and “-n savesetName” where the saveset name is obviously the name of the base PSS saveset. So in this case, running for my client of “rama” and a saveset name of “/Volumes/Alteran/Documents“, I got output such as the following:
$ ./pss-dependents.pl -c rama -n /Volumes/Alteran/Documents
rama:/Volumes/Alteran/Documents @ 03/02/19
Level = full, Size = 508 GB, SSID = 3528823916
3545601131 <3>/Volumes/Alteran/Documents (533 GB)
3579155563 <2>/Volumes/Alteran/Documents (568 GB)
3562378347 <1>/Volumes/Alteran/Documents (548 GB)
rama:/Volumes/Alteran/Documents @ 17/02/19
Level = 1, Size = 4 B, SSID = 896010610
929565040 <3>/Volumes/Alteran/Documents (4 B)
946342256 <2>/Volumes/Alteran/Documents (4 B)
912787824 <1>/Volumes/Alteran/Documents (4 B)
rama:/Volumes/Alteran/Documents @ 18/02/19
Level = incr, Size = 105 MB, SSID = 3765000946
3781778159 <3>/Volumes/Alteran/Documents (119 MB)
3798555375 <2>/Volumes/Alteran/Documents (115 MB)
3815332591 <1>/Volumes/Alteran/Documents (106 MB)
rama:/Volumes/Alteran/Documents @ 19/02/19
Level = incr, Size = 4 B, SSID = 2339023986
2355801200 <3>/Volumes/Alteran/Documents (4 B)
2372578416 <2>/Volumes/Alteran/Documents (4 B)
2389355632 <1>/Volumes/Alteran/Documents (4 B)
rama:/Volumes/Alteran/Documents @ 20/02/19
Level = incr, Size = 4 B, SSID = 913047026
963378672 <3>/Volumes/Alteran/Documents (4 B)
946601456 <2>/Volumes/Alteran/Documents (4 B)
929824240 <1>/Volumes/Alteran/Documents (4 B)
(Abridged, there was a chunk more output.)
The output is sorted by nsavetime for the core PSS saveset name, so you’ll get a rough view of the chaining of the saveset by dependencies.
$ cat pss-dependents.pl | awk '{gsub("\t"," ");print}' -
#!/usr/bin/perl -w
use strict;
use Getopt::Std;
my %opts;
my $client = "none";
my $name = "none";
if (getopts('c:n:',\%opts)) {
if (defined($opts{c})) {
$client = $opts{c};
} else {
die "You must supply a client name (-c client)\n";
}
if (defined($opts{n})) {
$name = $opts{n};
} else {
die "You must supply a saveset name (-n name)\n";
}
}
my %ssids = ();
my $mminfo = "mminfo -q \"client=$client,name=$name\" -r ssid,nsavetime,savetime,level,sumsize -xc,";
if (open(MMI,"$mminfo 2>&1 |")) {
while (<MMI>) {
my $line = $_;
next if ($line =~ /ssid/);
chomp $line;
my ($ssid,$nsavetime,$savetime,$level,$sumsize) = (split(/\,/,$line))[0,1,2,3,4];
$ssids{$nsavetime}{ssid} = $ssid;
$ssids{$nsavetime}{level} = $level;
$ssids{$nsavetime}{savetime} = $savetime;
$ssids{$nsavetime}{sumsize} = $sumsize;
}
close(MMI);
}
foreach my $nsavetime (sort {$a <=> $b} keys %ssids) {
my $savetime = $ssids{$nsavetime}{savetime};
my $level = $ssids{$nsavetime}{level};
my $ssid = $ssids{$nsavetime}{ssid};
my $sumsize = $ssids{$nsavetime}{sumsize};
print "$client:$name @ $savetime\n";
print "\tLevel = $level, Size = $sumsize, SSID = $ssid\n";
my $mminfoVerbose = "mminfo -q \"ssid=$ssid\" -S";
$/ = ";\n";
my @dependents = ();
if (open(MMI,"$mminfoVerbose 2>&1 |")) {
while (<MMI>) {
my $line = $_;
chomp $line;
if ($line =~ /^\s+\*mbs dependents:\s\\\n(.*)$/s) {
my $deps = $1;
@dependents = split(/\,\s*/,$deps);
#foreach my $dep (@dependents) {
# print "\t$dep\n";
#}
}
}
close(MMI);
}
$/ = "\n";
foreach my $dep (@dependents) {
if (open(MMI,"mminfo -q \"ssid=$dep\" -r ssid,name,sumsize -xc, 2>&1 |")) {
while (<MMI>) {
my $line = $_;
chomp $line;
next if ($line =~ /ssid/);
my ($dSSID,$dName,$dSumsize) = (split(/\,/,$line))[0,1,2];
print "\t\t$dSSID $dName ($dSumsize)\n";
}
close(MMI);
}
}
print "\n";
}
So there you have it – pulling out details from NetWorker for the parallel savesets created in a PSS operation.
Hi Preston,
thank you so much for pointing out the tech details. This is good to know but will it help an admin?
IMHO there are 3 major issues:
– with mminfo already, one should be able to see at least a flag whether a save set is part of a PPS
backup. In this case, the admin would at least get an indication about this fact.
– So the naming convention was taken from the continuation save sets. It was already available.
Unfortunately , the naming convention for the CFI’s ‘continuation directories’ was also available
but it has not been taken under consideration. Here you ave the ‘index’ appened, like
D:\TEST
D:\TEST
D:\TEST
etc … which would be more logical. I am pretty sure, an admin would never get the idea to
search for a save set by the ‘index number’.
– Finally, all these ‘partial’ save sets are totally independent. This means one could
– use (scripted) cloning by ssid and easily forget dependent save sets
– delete the ‘leader’ leaving the others as ‘orphans’ behind
– delete an ‘index’ part making the rest becoming worthless
just to name a few possibilities.
In general, I love this feature. It would love to use it.
But for the moment we just see to many reasons why it could be dangerous.
Hi Carsten,
The flag is effectively available in mminfo – per the blog article, it’s accessible via the mminfo -S option. There are some details for savesets that are only revealed in mminfo -S, so this is in that sense no different from those other scenarios.
I actually believe the ssName,ssName nomenclature makes sense — perhaps more so than just lots of ssName names appearing.
In relation to scripting, I’d suggest that modern NetWorker’s cloning capabilities are far enough advanced that most if not all of the traditional reasons for scripting cloning rather than having it done automatically is no longer required. And I say that as someone who used to write commercial code to compliment NetWorker scripting: NetWorker 8 introduced a lot of additional cloning options, NetWorker 9 enhanced that further. While by the time NetWorker 9 was released I’d lost control of the code I previously wrote, there was nothing in the cloning functionality that I’d written previously that wasn’t available then as a cloning policy within NetWorker, accessible and controllable from within NMC.
All scripting carries hazards – I’ve seen more than a few mistakes over the years in scripts, and having wrote and maintained for years a script which simulated NetWorker’s dependency tracking on a per-group basis, I literally had nightmares that I’d make a mistake in a release causing data deletion. I’m reminded of the old set of jokes about how or if different programming languages will let you shoot yourself in the foot, and scripting is no different: as soon as you start doing scripting, particularly where you might say, auto-answer yes to a prompt, you introduce the risk of doing something that causes problems. (That’s regardless of product, vendor, etc.)
Rest assured, there are some very large environments making extensive use of PSS to significantly speed up their backup operations.
Cheers,
Preston.
Hi Preston,
I understand your arguments … although I have another opinion in certain areas.
But of course I only represent a single customer.
However, with respect to the ‘mminfo flag’ I still believe that there is the necessity for such already in a ‘simple report’.
It does not make sense to check through every single, potential ssid candidate with
‘mminfo -S -q ssid=#’
just to verify whether this save set has an ‘independent nature’ or whether it is a PSS save set.
A course selection/filtering of potential candidates must be possible from a standard
“mminfo -q -r” command.
Hi Preston,
We recently had an issue doing a save set restore of a PSS backup. The server had loads of files causing our NMC recover and winworkr to crash:freeze whenever we expended it for recovery. Is there a sequence we need to put in to perform a save set recovery and do I have to mention all the Ssid’s in a single recover command or should I run recover command per pss ssid ?
Thanks
It could be that winworkr might have memory problems if handling a large file selection.
Consider using recover (CLI) or NMC instead.
Hello Preston,
Your script works when checking for non windows saveset names, but when I try using a windows saveset name, it is failing with:
# ./pss-dependents.pl -c client-name -n D:\\
sh: -c: line 0: unexpected EOF while looking for matching `”‘
sh: -c: line 1: syntax error: unexpected end of file
I localized the problem to this line:
my $mminfo = “mminfo -q \”client=$client,name=$name\” -r ssid,nsavetime,savetime,level,sumsize -xc,”;
If i remove the saveset name query part, then there is no error:
my $mminfo = “mminfo -q \”client=$client” -r ssid,nsavetime,savetime,level,sumsize -xc,”;
# ./pss-dependents.pl -c client-name -n ‘D:\’
client-name:D:\ @ 07/29/2022
Level = full, Size = 71
(etc)
I am not that great with troubleshooting perl scripts, and would appreciate your help here.
update: looks like a problem with mminfo itself. (stupid mminfo). It works when I modified the line to use:
my $mminfo = “mminfo -q client=$client -r ssid,nsavetime,savetime,level,sumsize -xc, -N $name”;
# ./pss-dependents.pl -c client-name -n ‘D:\\’
client-name:D:\\ @ 08/12/2022
Level = full, Size = 45 GB, SSID = 3086387205
client-name:D:\\ @ 08/27/2022
Level = full, Size = 24 GB, SSID = 285880370