{"id":5454,"date":"2015-02-25T16:23:21","date_gmt":"2015-02-25T06:23:21","guid":{"rendered":"http:\/\/nsrd.info\/blog\/?p=5454"},"modified":"2018-12-11T13:01:45","modified_gmt":"2018-12-11T03:01:45","slug":"testing-and-debugging-an-emergency-restore","status":"publish","type":"post","link":"https:\/\/nsrd.info\/blog\/2015\/02\/25\/testing-and-debugging-an-emergency-restore\/","title":{"rendered":"Testing (and debugging) an emergency restore"},"content":{"rendered":"<p>A few days ago I had some spare time up my sleeve, and I decided to test out the Emergency Restore function in NetWorker VBA\/EBR. After all, you never want to test out emergency recovery procedures for the first time&nbsp;<em>in an emergency<\/em>, so I wanted to be prepared.<\/p>\n<p>If you&#8217;ve not seen it, the Emergency Restore panel is accessed from your EBR appliance (https:\/\/applianceName:8580\/ebr-configure) and&nbsp;looks like the following:<\/p>\n<p><a href=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2015\/02\/ebr-er-1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-5455 size-full\" src=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2015\/02\/ebr-er-1.png\" alt=\"EBR Emergency Restore Panel\" width=\"851\" height=\"384\" srcset=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2015\/02\/ebr-er-1.png 851w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2015\/02\/ebr-er-1-300x135.png 300w\" sizes=\"auto, (max-width: 851px) 100vw, 851px\" \/><\/a><\/p>\n<p>The goal of&nbsp;the Emergency Restore function is simple: you have a virtual machine you urgently need to restore,&nbsp;<em>but<\/em> the vCenter server is also down. Of course, in an ideal scenario, you should never need to use the Emergency Restore&nbsp;function, but&nbsp;<em>ideal<\/em> and&nbsp;<em>reality<\/em> don&#8217;t always converge with 100% overlap.<\/p>\n<p>In this scenario, to simulate my vCenter server being down,&nbsp;I went into vCenter, selected the ESX server I wanted to recover a virtual machine for (<strong>c64<\/strong>), and&nbsp;<em>disconnected<\/em> from it. To all intents and purposes to the ESX server, vCenter was down \u2013 at least, enough to satisfy&nbsp;VBA that I really needed to use the Emergency Restore function.<\/p>\n<p>Once you&#8217;ve selected the VM, and the backup of the VM you want to restore, you click the&nbsp;<em>Restore<\/em> button to get things underway. The first prompt looks like the following:<\/p>\n<p><a href=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2015\/02\/ebr-er-2.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-5456\" src=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2015\/02\/ebr-er-2.png\" alt=\"EBR ESX Connection Prompt\" width=\"421\" height=\"292\" srcset=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2015\/02\/ebr-er-2.png 421w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2015\/02\/ebr-er-2-300x208.png 300w\" sizes=\"auto, (max-width: 421px) 100vw, 421px\" \/><\/a>(Yes, my ESX server is named after the Commodore 64. For what it&#8217;s worth, my vCenter server is&nbsp;<em>c128<\/em> and a smaller ESX server I&#8217;ve got configured is&nbsp;<em>plus4<\/em>.)<\/p>\n<p>Entering the ESX server details and login credentials,&nbsp;you click OK to jump through to the recovery options (including the name&nbsp;of the new virtual machine):<\/p>\n<p><a href=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2015\/02\/ebr-er-3.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-5457\" src=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2015\/02\/ebr-er-3.png\" alt=\"EBR - Recovery Options\" width=\"425\" height=\"318\" srcset=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2015\/02\/ebr-er-3.png 425w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2015\/02\/ebr-er-3-300x224.png 300w\" sizes=\"auto, (max-width: 425px) 100vw, 425px\" \/><\/a>After you&nbsp;fill in the new virtual machine name and choose the datastore&nbsp;you want to recover from, it&#8217;s as simple as clicking&nbsp;<em>Restore<\/em> and the ball is rolling. Except&#8230;<\/p>\n<p><a href=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2015\/02\/ebr-er-4.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-5458\" src=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2015\/02\/ebr-er-4.png\" alt=\"EBR Emergency Restore Error\" width=\"424\" height=\"317\" srcset=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2015\/02\/ebr-er-4.png 424w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2015\/02\/ebr-er-4-300x224.png 300w\" sizes=\"auto, (max-width: 424px) 100vw, 424px\" \/><\/a><\/p>\n<p>After about 5 minutes, it failed, and the error I got was:<\/p>\n<blockquote><p>Restore failed.<\/p>\n<p>Server could not create a restore task at this time. Please ensure your ESX host is resolvable by your DNS server. In addition, as configuration changes may take a few minutes to become effective, please try again at a later time.<\/p><\/blockquote>\n<p>From a cursory inspection, I couldn&#8217;t find any reference&nbsp;to the error on the support website,&nbsp;so I initially thought I must have done something wrong. Having re-read the Emergency Restore section of the VMware Integration Guide a few times, I was&nbsp;confident I&nbsp;hadn&#8217;t missed anything, so I figured the ESX server might have been taking a few minutes to be sufficiently standalone after the disconnection, and gave it a good ten or fifteen minutes before reattempting,&nbsp;but got the same error.<\/p>\n<p>So I went through and did a bit of digging on the actual EBR server itself, diving into&nbsp;the logs there. I eventually re-ran&nbsp;the recovery while tailing the EBR logs, and noticed it attempting to connect to a&nbsp;Data Domain system I knew was down at the time &#8230; and had my&nbsp;<em>ahah!<\/em> moment.<\/p>\n<p>You see I&#8217;d previously backed up the virtual machine to one Data Domain, but when I needed to run some other tests, changed my configuration and started backing up the&nbsp;virtual infrastructure to another Data Domain. EBR needed&nbsp;<em>both<\/em> online to complete the recovery, of course!<\/p>\n<p>Once I had&nbsp;the original Data Domain powered up and running, the Emergency Restore went&nbsp;without a single hitch, and&nbsp;I was pleased to see this little message:<\/p>\n<p><a href=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2015\/02\/ebr-er-5.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-5459\" src=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2015\/02\/ebr-er-5.png\" alt=\"Successful submission of restore job\" width=\"247\" height=\"102\"><\/a><\/p>\n<p>Before too long I was seeing&nbsp;good progress on the restore:<\/p>\n<p><a href=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2015\/02\/ebr-er-6.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-5460\" src=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2015\/02\/ebr-er-6.png\" alt=\"Emergency Restore Progress\" width=\"851\" height=\"437\" srcset=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2015\/02\/ebr-er-6.png 851w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2015\/02\/ebr-er-6-300x154.png 300w\" sizes=\"auto, (max-width: 851px) 100vw, 851px\" \/><\/a><\/p>\n<p>And not long after that, I saw the sort of message you always want to see in an emergency recovery:<\/p>\n<p><a href=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2015\/02\/ebr-er-7.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-5461\" src=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2015\/02\/ebr-er-7.png\" alt=\"EBR Emergency Recovery Complete\" width=\"854\" height=\"438\" srcset=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2015\/02\/ebr-er-7.png 854w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2015\/02\/ebr-er-7-300x154.png 300w\" sizes=\"auto, (max-width: 854px) 100vw, 854px\" \/><\/a><\/p>\n<p>There you have it \u2013 the Emergency Restore function tested well away from any&nbsp;emergency&nbsp;situation, and a bit of debugging while I was at it.<\/p>\n<p>I&#8217;m sure you&#8217;ll hope you never need to use the Emergency Restore feature within your virtual environment, but knowing it&#8217;s there \u2013 and knowing&nbsp;how simple the process is \u2013 might help you avoid&nbsp;serious problems in an emergency.<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A few days ago I had some spare time up my sleeve, and I decided to test out the Emergency&hellip;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[1181,16,19,1213],"tags":[1214,372,1249,788,1252,1215,1163,1216,1084,1087],"class_list":["post-5454","post","type-post","status-publish","format-standard","hentry","category-data-domain-2","category-networker","category-recovery","category-vba","tag-emergency-restore","tag-esx","tag-networker","tag-recover","tag-recovery","tag-restore","tag-vba","tag-vcenter","tag-virtualisation","tag-vmware"],"aioseo_notices":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/pKpIN-1pY","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/5454","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/comments?post=5454"}],"version-history":[{"count":3,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/5454\/revisions"}],"predecessor-version":[{"id":7438,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/5454\/revisions\/7438"}],"wp:attachment":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/media?parent=5454"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/categories?post=5454"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/tags?post=5454"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}