{"id":1122,"date":"2009-10-05T13:18:06","date_gmt":"2009-10-05T03:18:06","guid":{"rendered":"http:\/\/nsrd.wordpress.com\/?p=1122"},"modified":"2018-12-12T15:33:37","modified_gmt":"2018-12-12T05:33:37","slug":"how-much-arent-you-backing-up","status":"publish","type":"post","link":"https:\/\/nsrd.info\/blog\/2009\/10\/05\/how-much-arent-you-backing-up\/","title":{"rendered":"How much aren&#8217;t you backing up?"},"content":{"rendered":"<p>Do you have a clear picture of everything that you&#8217;re not backing up?&nbsp;For many sites, the answer is not as clear cut as they may think.<\/p>\n<p>It&#8217;s easy to quantify the simple stuff &#8211; QA or test servers\/environments that literally aren&#8217;t configured within the backup environment.<\/p>\n<p>It&#8217;s also relatively easy to quantify the more esoteric things within a datacentre &#8211; PABXs, switch configurations, etc. (Though in a well run backup environment, <a title=\"Expecting a more complete backup\" href=\"https:\/\/nsrd.info\/blog\/2009\/03\/30\/expecting-a-more-complete-backup\/\" target=\"_blank\">there&#8217;s no reason why you can&#8217;t configure scripts<\/a> that, as part of the backup process, logs onto such devices and retrieves the configuration, etc.)<\/p>\n<p>It should also be very, very easy to quantify what data on any individual system that you&#8217;re not backing up &#8211; e.g., knowing that for fileservers you may be backing up everything except for files that have a &#8220;.mp3&#8221; extension.<\/p>\n<p>What most sites find difficult to quantify is the quasi-backup situations \u2013 files and\/or data that they <em>are<\/em> backing up, but which is useless in a recovery scenario. Now, many readers of that last sentence will probably think of one of the more immediate examples: live database files that are being &#8220;accidentally&#8221; picked up in the filesystem backup (even if they&#8217;re being backed up elsewhere, by a module). Yes, such a backup does fall into this category, but there are other types of backups which are even less likely to be considered.<\/p>\n<p>I&#8217;m talking about information that you only need during a disaster recovery \u2013 or worse, a site disaster recovery. Let&#8217;s consider an average Unix (or Linux) system. (Windows is no different, I just want to give some command line details here.) If a physical server goes up in smoke, and a new one has to be built, there&#8217;s a couple of things that have to be considered <em>pre-recovery<\/em>:<\/p>\n<ul>\n<li>What was the partition layout?<\/li>\n<li>What disks were configured in what styles of RAID layout?<\/li>\n<\/ul>\n<p>In an average backup environment, this sort of information isn&#8217;t preserved. Sure, if you&#8217;ve got say, HomeBase licenses (taking the EMC approach), or using some other sort of bare metal recovery system, <em>and that system supports your exact environment<\/em>*, then you may find that such information <em>is<\/em> preserved and <em>is<\/em> available.<\/p>\n<p>But what about the high percentage of cases where it&#8217;s not?<\/p>\n<p>This is where the backup process needs to be configured\/extended to support generation of system or disaster recovery information. It&#8217;s all very good for instance, for a Linux machine to say that you can just recover &#8220;\/etc\/fstab&#8221;, but what if you can&#8217;t remember the size of the partitions referenced by that file system table? Or, what if <em>you aren&#8217;t there<\/em> to remember what the size of the partitions were? (Memory is a wonderful yet entirely fallible and human-dependent process. Disaster recovery situations shouldn&#8217;t be bound by what we can or can&#8217;t remember about the systems, and so we have to gather all the information required to support disaster recovery.)<\/p>\n<p>On a running system, there&#8217;s all sorts of tools available to gather this sort of information, but when the system isn&#8217;t running, we can&#8217;t run the tools, so we need to run them in advance, either as part of the backup process or as a scheduled, checked-upon function. (My preference is to incorporate it into the backup process.)<\/p>\n<p>For instance, consider that Linux scenario &#8211; we can quickly assemble the details of all partition sizes on a system with one simple command \u2013 e.g.:<\/p>\n<pre>[root@nox ~]# fdisk -l\n\nDisk \/dev\/sda: 1000.2 GB, 1000204886016 bytes\n255 heads, 63 sectors\/track, 121601 cylinders\nUnits = cylinders of 16065 * 512 = 8225280 bytes\n\n Device Boot&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Start&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; End&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Blocks&nbsp;&nbsp; Id&nbsp; System\n\/dev\/sda1&nbsp;&nbsp; *&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 2089&nbsp;&nbsp;&nbsp; 16779861&nbsp;&nbsp; fd&nbsp; Linux raid autodetect\n\/dev\/sda2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 2090&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 2220&nbsp;&nbsp;&nbsp;&nbsp; 1052257+&nbsp; 82&nbsp; Linux swap \/ Solaris\n\/dev\/sda3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 2221&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 19457&nbsp;&nbsp; 138456202+&nbsp; fd&nbsp; Linux raid autodetect\n\/dev\/sda4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 19458&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 121601&nbsp;&nbsp; 820471680&nbsp;&nbsp;&nbsp; 5&nbsp; Extended\n\/dev\/sda5&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 19458&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 19701&nbsp;&nbsp;&nbsp;&nbsp; 1959898+&nbsp; 82&nbsp; Linux swap \/ Solaris\n\/dev\/sda6&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 19702&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 121601&nbsp;&nbsp; 818511718+&nbsp; fd&nbsp; Linux raid autodetect\n\nDisk \/dev\/sdb: 1000.2 GB, 1000204886016 bytes\n255 heads, 63 sectors\/track, 121601 cylinders\nUnits = cylinders of 16065 * 512 = 8225280 bytes\n\n Device Boot&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Start&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; End&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Blocks&nbsp;&nbsp; Id&nbsp; System\n\/dev\/sdb1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 250&nbsp;&nbsp;&nbsp;&nbsp; 2008093+&nbsp; 82&nbsp; Linux swap \/ Solaris\n\/dev\/sdb2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 251&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 121601&nbsp;&nbsp; 974751907+&nbsp; 83&nbsp; Linux\n\nDisk \/dev\/sdc: 1000.2 GB, 1000204886016 bytes\n255 heads, 63 sectors\/track, 121601 cylinders\nUnits = cylinders of 16065 * 512 = 8225280 bytes\n\n Device Boot&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Start&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; End&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Blocks&nbsp;&nbsp; Id&nbsp; System\n\/dev\/sdc1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 121601&nbsp;&nbsp; 976760001&nbsp;&nbsp; 83&nbsp; Linux\n\nDisk \/dev\/sdd: 1000.2 GB, 1000204886016 bytes\n255 heads, 63 sectors\/track, 121601 cylinders\nUnits = cylinders of 16065 * 512 = 8225280 bytes\n\n Device Boot&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Start&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; End&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Blocks&nbsp;&nbsp; Id&nbsp; System\n\/dev\/sdd1&nbsp;&nbsp; *&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 2089&nbsp;&nbsp;&nbsp; 16779861&nbsp;&nbsp; fd&nbsp; Linux raid autodetect\n\/dev\/sdd2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 2090&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 2220&nbsp;&nbsp;&nbsp;&nbsp; 1052257+&nbsp; 82&nbsp; Linux swap \/ Solaris\n\/dev\/sdd3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 2221&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 19457&nbsp;&nbsp; 138456202+&nbsp; fd&nbsp; Linux raid autodetect\n\/dev\/sdd4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 19458&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 121601&nbsp;&nbsp; 820471680&nbsp;&nbsp;&nbsp; 5&nbsp; Extended\n\/dev\/sdd5&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 19458&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 19701&nbsp;&nbsp;&nbsp;&nbsp; 1959898+&nbsp; 82&nbsp; Linux swap \/ Solaris\n\/dev\/sdd6&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 19702&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 121601&nbsp;&nbsp; 818511718+&nbsp; fd&nbsp; Linux raid autodetect<\/pre>\n<p>That wasn&#8217;t entirely hard. Scripting that to occur at the start of the backup process isn&#8217;t difficult either. For systems that have RAID, there&#8217;s another, equally simple command to extract RAID layouts as well \u2013 again, for Linux:<\/p>\n<pre>[root@nox ~]# cat \/proc\/mdstat\nPersonalities : [raid1]\nmd1 : active raid1 sda3[0] sdd3[1]\n 138456128 blocks [2\/2] [UU]\n\nmd2 : active raid1 sda6[0] sdd6[1]\n 818511616 blocks [2\/2] [UU]\n\nmd0 : active raid1 sda1[0] sdd1[1]\n 16779776 blocks [2\/2] [UU]\n\nunused devices: &lt;none&gt;<\/pre>\n<p>I don&#8217;t want to consume realms of pages discussing what, for each operating system you should be gathering. The average system administrator for any individual platform should, with a cup of coffee (or other preferred beverage) in hand, should be able to sit down and in under 10 minutes jot down the sorts of information that would need to be gathered <em>in advance<\/em> of a disaster to assist in the total system rebuild of an operating system of a machine they administer.<\/p>\n<p>Once these information gathering steps have been determined, they can be inserted into the backup process as a pre-backup command. (In NetWorker parlance, this would be via a savepnpc &#8220;pre&#8221; script. Other backup products will equally feature such options.) Once the information is gathered, a copy should be kept on the backup server <em>as well as<\/em> in an offsite location. (I&#8217;ll give you a useful cloud backup function now: it&#8217;s called <em>Google Mail<\/em>. Great for offsiting bootstraps and system configuration details.)<\/p>\n<p>When it comes to disaster recovery, such information can take the guess work or reliance on memory out of the equation, allowing a system or backup administrator in any (potentially sleep-deprived) state, with any level of knowledge about the system in question, to conduct the recovery with a much higher degree of certainty.<\/p>\n<p>&#8212;<br \/>\n* Due to what they offer to do, bare metal recovery (BMR) products tend to be <em>highly<\/em> specific in which operating system variants, etc., they support. In my experience a significantly higher number of sites <em>don&#8217;t<\/em> use BMR than <em>do<\/em>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Do you have a clear picture of everything that you&#8217;re not backing up?&nbsp;For many sites, the answer is not as&hellip;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[3,5,15,16,17,20],"tags":[173,176,328,694,972],"class_list":["post-1122","post","type-post","status-publish","format-standard","hentry","category-architecture","category-backup-theory","category-linux","category-networker","category-policies","category-scripting","tag-bmr","tag-bootstrap","tag-disaster-recovery","tag-offsite","tag-system-configuration"],"aioseo_notices":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/pKpIN-i6","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/1122","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/comments?post=1122"}],"version-history":[{"count":1,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/1122\/revisions"}],"predecessor-version":[{"id":7607,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/1122\/revisions\/7607"}],"wp:attachment":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/media?parent=1122"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/categories?post=1122"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/tags?post=1122"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}