{"id":8254,"date":"2019-07-17T08:28:56","date_gmt":"2019-07-16T22:28:56","guid":{"rendered":"https:\/\/nsrd.info\/blog\/?p=8254"},"modified":"2019-07-17T08:33:10","modified_gmt":"2019-07-16T22:33:10","slug":"disaster-movies-a-model-for-data-protection","status":"publish","type":"post","link":"https:\/\/nsrd.info\/blog\/2019\/07\/17\/disaster-movies-a-model-for-data-protection\/","title":{"rendered":"Disaster Movies: A Model for Data Protection"},"content":{"rendered":"\n<p>So, the other night, my husband and I finally got around to watching <em>San Andreas<\/em>, the 2015 movie featuring Dwayne Johnson. I&#8217;d heard at best average reviews for it, but it had two things going for it: Dwayne Johnson, and it was there on Netflix when I couldn&#8217;t think of anything else to watch.<\/p>\n\n\n\n<p><em>San Andreas<\/em> is, without a doubt, a typical disaster movie in the same vein as <em>Armageddon<\/em> (1998), <em>2012<\/em> (2009), <em>Volcano<\/em> (1997), <em>Deep Impact<\/em> (1998), <em>Poseidon<\/em> (2006), <em>Sunshine<\/em> (2007), and a myriad of other disaster movies, new and old.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"900\" height=\"641\" src=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2019\/07\/bigstock-Apocalyptic-City.jpg\" alt=\"\" class=\"wp-image-8256\" srcset=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2019\/07\/bigstock-Apocalyptic-City.jpg 900w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2019\/07\/bigstock-Apocalyptic-City-300x214.jpg 300w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2019\/07\/bigstock-Apocalyptic-City-768x547.jpg 768w\" sizes=\"auto, (max-width: 900px) 100vw, 900px\" \/><figcaption>Disaster strikes!<\/figcaption><\/figure>\n\n\n\n<p>Disaster movies follow a classic convention where:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Something bad is going to happen<\/li><li>Something bad happens<\/li><li>People who are reeling from the disaster have something <em>else<\/em> bad happen to them<\/li><li>When they&#8217;re recovering from that they have something <em>else<\/em> bad happen to them<\/li><li>(Ad infinitum.)<\/li><li><em>Eventually<\/em> they escape the disaster. (Usually.)<\/li><\/ul>\n\n\n\n<p>It struck me when watching San Andreas the other night that disaster movies have a strong parallel to data protection. Why? Because it&#8217;s not the initial kick that does you in, it&#8217;s the cascading problems.<\/p>\n\n\n\n<p><em>Cascading problems<\/em>. If the universe was fair we&#8217;d only have to deal with one problem at a time, but the universe runs to its own schedule, and there&#8217;s a reason why <em>Murphy&#8217;s Law<\/em> resonates with so many people. (Or maybe if the universe was fair, we&#8217;d not have problems at all, but that&#8217;s another story for another time.)<\/p>\n\n\n\n<p>While it&#8217;s easy to forget, we&#8217;ve often recognised the risks of cascading problems. Take RAID-6 for example: Someone (or perhaps many someones), at some point, looked at RAID-5 and said, &#8220;You know what would really suck? Having a drive fail when we&#8217;re trying to rebuild from a drive failure.&#8221; Then, Shazam! RAID-6 was born. (Fast voiceover: <em>Actual real sequence of events may differ from description. Always consult your data protection professional before switching from RAID-6 to RAID-5.<\/em>)<\/p>\n\n\n\n<p>One of the fundamental aspects of data protection planning \u2014&nbsp;not just disaster recovery planning (although, it&#8217;s critical there too), is to seriously think about the impact of cascading failures. <\/p>\n\n\n\n<p>Effectively, by considering the risk of cascading failures, you&#8217;re opening your data protection process to contingency planning.  There&#8217;s a limit to the number of cascading failures you can plan for \u2014&nbsp;and by extension, the amount of contingency planning you can do. I&#8217;d suggest for instance that if you&#8217;re at the point of trying to plan for recovering your Sharepoint farm after an extinction level event, you&#8217;re probably over-thinking it.<\/p>\n\n\n\n<p>But there&#8217;s a level you have to think it through, and I&#8217;d suggest you&#8217;ll probably want to keep 2 numbers in mind as a starting point: 2, and 3.<\/p>\n\n\n\n<p>For non-crtical data \u2014&nbsp;for anything that the business can afford to lose, but would rather not, then you can probably focus on thinking about 2 cascaded failures. For example: a file needs to be recovered. That&#8217;s not a failure in itself \u2014&nbsp;we&#8217;d think of that as the trigger event.<\/p>\n\n\n\n<p>If we&#8217;re talking non-critical data, then you want your data protection to at least handle cascading failures such as:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>The primary backup target is inaccessible, <em>and<\/em><ul><li>Redundancy has been activated for the secondary backup target<\/li><\/ul><\/li><\/ul>\n\n\n\n<p>So if we&#8217;re talking say, NetWorker and cloned backups, this means that you want to be able to recover data even when:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Your original copy is offline (you might be doing a DDOS upgrade just at the time the recovery request comes through), <em>and<\/em><ul><li>A drive has failed in your DD at the remote site.<\/li><\/ul><\/li><\/ul>\n\n\n\n<p>If you&#8217;re still dealing with tape, it would equally apply:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>The tape library at your primary datacenter is experiencing a fault, <em>and<\/em><ul><li>There&#8217;s a failed tape <em>drive<\/em> in the secondary datacenter.<\/li><\/ul><\/li><\/ul>\n\n\n\n<p>But what about <em>3<\/em>? Well, that&#8217;s what you need to think of for your production, or critical data:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Your primary site is inaccessible because someone pressed the &#8220;computer room shutdown&#8221; button instead of the &#8220;computer room exit&#8221; button.<ul><li>So, you need to run the recovery from a failover backup server, <em>and<\/em> a failover backup\/clone target \u2014&nbsp;i.e., you have <em>two<\/em> components that have &#8216;failed&#8217; straight away<\/li><li>Your secondary site Data Domain experienced a drive failure overnight (your third cascaded failure) and there&#8217;s a Dell EMC technician signing themselves in at security to replace the drive.<\/li><\/ul><\/li><\/ul>\n\n\n\n<p>As I mention in my <a rel=\"noreferrer noopener\" aria-label=\"books (opens in a new tab)\" href=\"https:\/\/www.amazon.com\/Data-Protection-Ensuring-Availability\/dp\/1482244152\/ref=mt_paperback?_encoding=UTF8&amp;me=\" target=\"_blank\"><strong>books<\/strong><\/a>, this is all <em>risk vs cost<\/em>, in the same way that the x-9&#8217;s protection strategy is for business critical systems is. Sure, you might <em>want<\/em> to achieve 9-9&#8217;s availability (99.9999999% available), but beyond 5-9s, the costs to meet the next 9 availability vs the <em>conditions<\/em> it protects from could increase more than you&#8217;re willing to pay.<\/p>\n\n\n\n<p>2 and 3 are good starting points, but they&#8217;re not the end of it. Consider banks and other financial institutions \u2014&nbsp;there&#8217;s increasing focus on making sure they have a cyber-recovery solution \u2014 an airgapped (physical or electronic) tertiary copy of their critical data. For that sort of data, there&#8217;s a recognised additional layer of contingency planning, and in some instances <em>cascading failures<\/em> that&#8217;s catered for with cyber-recovery that you just won&#8217;t get in a classic 2-datacentre configuration.<\/p>\n\n\n\n<p>So here&#8217;s my tip: if you want to get your infrastructure team, your disaster recovery team, or your business continuity team to effectively think about contingency planning, plan for a long meeting, order some popcorn and snacks, and instead of doing a boring &#8220;This.Is.Our.Agenda.Today&#8221;, play a disaster movie to get everyone thinking about cascading failures.<\/p>\n\n\n\n<p>When the end-credits roll, you&#8217;re ready to talk about the contingency planning that&#8217;ll help you survive multiple failures at once.<\/p>\n\n\n\n<hr class=\"wp-block-separator\"\/>\n\n\n\n<p>Hey, if you&#8217;re reading this before 31 July 2019, there&#8217;s a book giveaway competition <a href=\"https:\/\/nsrd.info\/blog\/2019\/06\/16\/book-giveaway-competition-protecting-information-assets-and-it-infrastructure-in-the-cloud\/\"><strong>here<\/strong> \u2014\u00a0check it out!<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>So, the other night, my husband and I finally got around to watching San Andreas, the 2015 movie featuring Dwayne&hellip;<\/p>\n","protected":false},"author":1,"featured_media":8256,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[5,1442],"tags":[1525,252,328],"class_list":["post-8254","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-backup-theory","category-disaster-recovery","tag-cascading-failure","tag-contingency","tag-disaster-recovery"],"aioseo_notices":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2019\/07\/bigstock-Apocalyptic-City.jpg","jetpack_shortlink":"https:\/\/wp.me\/pKpIN-298","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/8254","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/comments?post=8254"}],"version-history":[{"count":5,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/8254\/revisions"}],"predecessor-version":[{"id":8261,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/8254\/revisions\/8261"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/media\/8256"}],"wp:attachment":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/media?parent=8254"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/categories?post=8254"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/tags?post=8254"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}