{"id":7232,"date":"2018-11-06T16:08:35","date_gmt":"2018-11-06T06:08:35","guid":{"rendered":"https:\/\/nsrd.info\/blog\/?p=7232"},"modified":"2018-12-11T07:06:19","modified_gmt":"2018-12-10T21:06:19","slug":"a-simple-linux-kvm-backup-framework-using-networker","status":"publish","type":"post","link":"https:\/\/nsrd.info\/blog\/2018\/11\/06\/a-simple-linux-kvm-backup-framework-using-networker\/","title":{"rendered":"A Simple Linux KVM Backup Framework using NetWorker"},"content":{"rendered":"<p>Of late I&#8217;ve been playing around a little bit with KVM. I must admit, it&#8217;s not really my virtualisation system of choice, but I&#8217;ve been using VMware in some form or another since the late 90s, so that could as much as anything be a case of&nbsp;<em>what-I&#8217;m-used-to-syndrome<\/em>.<\/p>\n<p><a href=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/bigStock-Conceptual-System.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-7233\" src=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/bigStock-Conceptual-System.jpg\" alt=\"Blue glowing hardware fractal computer generated abstract background\" width=\"900\" height=\"675\" srcset=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/bigStock-Conceptual-System.jpg 900w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/bigStock-Conceptual-System-300x225.jpg 300w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/bigStock-Conceptual-System-768x576.jpg 768w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/bigStock-Conceptual-System-267x200.jpg 267w\" sizes=\"auto, (max-width: 900px) 100vw, 900px\" \/><\/a>Because I&#8217;m a backup geek, once I&#8217;ve been using something for a short while, my thoughts invariably turn to the question: how would I back this up? This leads to gratitude that NetWorker and Avamar are both&nbsp;<em>framework<\/em> based backup products: sure, they come with a lot of built-in functionality; but if you want something that&#8217;s not there by default, and you have a little bit of determination, you can always come up with a solution.<\/p>\n<p>The simplest solution for KVM would be to just shutdown the virtual machines when I needed to, then back them up. That&#8217;s fine for a lab environment, which is where I&#8217;m running them, but I don&#8217;t like doing things as if I&#8217;m in a lab environment; instead, I wanted to come up with something that could be run in a more life-like situation. So that meant being able to do backups around snapshots.<\/p>\n<p>With Linux KVM, that&#8217;s a bit of work. My test environment was a CentOS 7 server, which meant I had to go through the process of enabling the out-of-band QEMU options for CentOS &#8212; the default KVM install includes older API versions that don&#8217;t support live snapshot processes. On the CentOS 7 server, I deployed a simple CentOS 7 guest, too. Ideally, if you want to be able to quiesce KVM guest filesystems when you backup (i.e., commit writes to better capture the content of the filesystem during backup), you&#8217;ll need to ensure the virtual machine has the <em>qemu-guest-agent<\/em> package installed, too.<\/p>\n<p>The short view of the end to end process works like this:<\/p>\n<ul>\n<li>Identify the virtual machine(s) running on a KVM server<\/li>\n<li>For each virtual machine:\n<ul>\n<li>Identify the virtual disk(s)<\/li>\n<li>Take a snapshot of the virtual machine&#8217;s disks<\/li>\n<li>Backup the original virtual machine files (QCOW2 format)<\/li>\n<li>Release the snapshot and delete the snapshot file(s)<\/li>\n<li>Generate an XML configuration dump of the virtual machine<\/li>\n<li>Backup the XML configuration dump<\/li>\n<li>Delete the XML configuration dump<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>It was relatively easy to do the backup as an ad-hoc, manual process (and, of course, verify recovery!), but I wanted to go a little further, so I decided to write a bit of a proof of concept script. In fact, I was able to verify the backup and recovery process using both NetWorker and Avamar, but for the script I decided to roll my sleeves up and do a bit of working with NetWorker.<\/p>\n<p>The end result gives me a backup process that looks like the following:<\/p>\n<figure id=\"attachment_7234\" aria-describedby=\"caption-attachment-7234\" style=\"width: 833px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Backup-Command-and-mminfo.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-7234\" src=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Backup-Command-and-mminfo.png\" alt=\"KVM Backup Command and mminfo\" width=\"833\" height=\"595\" srcset=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Backup-Command-and-mminfo.png 833w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Backup-Command-and-mminfo-300x214.png 300w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Backup-Command-and-mminfo-768x549.png 768w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Backup-Command-and-mminfo-280x200.png 280w\" sizes=\"auto, (max-width: 833px) 100vw, 833px\" \/><\/a><figcaption id=\"caption-attachment-7234\" class=\"wp-caption-text\">KVM Backup Command and mminfo<\/figcaption><\/figure>\n<p>In the above, I&#8217;ve executed the script,&nbsp;<em>savekvm.pl<\/em> with a 7 day retention time (-r 7), for the BoostBackup pool (-b BoostBackup), to the backup server orilla (-s orilla), with the quiesce guest option (-Q) enabled. The backup writes reasonably verbose logging to the programmed log file (\/nsr\/applogs\/savekvm.log), but is designed to be reasonably quiet on the actual console itself. (After all, it&#8217;s just a proof of concept.)<\/p>\n<p>However, as you can see from the subsequent mminfo command, I&#8217;m making use of the NetWorker option to generate a&nbsp;<em>name<\/em> for the saveset to logically name the content that I&#8217;ve backed up &#8211; in this case, a naming format of:<\/p>\n<p style=\"padding-left: 30px;\"><em>hypervisor:KVM:vmGuest:{config|diskName}<\/em><\/p>\n<p>So in this example, I&#8217;ve got a virtual machine guest called &#8216;cent7&#8217; running on the hypervisor\/server &#8216;oa&#8217;; &#8216;cent7&#8217; has two virtual disks, vda and vdb (effectively the KVM equivalent of sda and sdb), and of course, a backup of the configuration file.<\/p>\n<p>The full command syntax available by the way for my&nbsp;<em>savekvm.pl<\/em> file is:<\/p>\n<figure id=\"attachment_7235\" aria-describedby=\"caption-attachment-7235\" style=\"width: 833px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Command-Syntax.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-7235\" src=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Command-Syntax.png\" alt=\"KVM Command Syntax\" width=\"833\" height=\"742\" srcset=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Command-Syntax.png 833w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Command-Syntax-300x267.png 300w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Command-Syntax-768x684.png 768w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Command-Syntax-225x200.png 225w\" sizes=\"auto, (max-width: 833px) 100vw, 833px\" \/><\/a><figcaption id=\"caption-attachment-7235\" class=\"wp-caption-text\">KVM Command Syntax<\/figcaption><\/figure>\n<p>As you can see, there&#8217;s some defaults built into the script for the server, the pool and the retention time. There&#8217;s also options such as logging the details around what commands would be executed, without actually executing them; useful if you&#8217;re wanting to see what would go on during a backup process. The log file you get when you execute looks a little like the following:<\/p>\n<figure id=\"attachment_7236\" aria-describedby=\"caption-attachment-7236\" style=\"width: 833px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-savekvm_log.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-7236\" src=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-savekvm_log.png\" alt=\"KVM savekvm_log\" width=\"833\" height=\"916\" srcset=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-savekvm_log.png 833w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-savekvm_log-273x300.png 273w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-savekvm_log-768x845.png 768w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-savekvm_log-182x200.png 182w\" sizes=\"auto, (max-width: 833px) 100vw, 833px\" \/><\/a><figcaption id=\"caption-attachment-7236\" class=\"wp-caption-text\">KVM savekvm.log<\/figcaption><\/figure>\n<p>To test this out in a live scenario, I setup a script on my virtual machine which would generate content to one of its filesystems, and started running it just before I started the backup process. So working across two terminal sessions, the combined view looked like:<\/p>\n<figure id=\"attachment_7237\" aria-describedby=\"caption-attachment-7237\" style=\"width: 1666px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Changing-VMs-while-backing-up.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-7237\" src=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Changing-VMs-while-backing-up.png\" alt=\"KVM Changing VMs while backing up\" width=\"1666\" height=\"916\" srcset=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Changing-VMs-while-backing-up.png 1666w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Changing-VMs-while-backing-up-300x165.png 300w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Changing-VMs-while-backing-up-768x422.png 768w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Changing-VMs-while-backing-up-1024x563.png 1024w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Changing-VMs-while-backing-up-364x200.png 364w\" sizes=\"auto, (max-width: 1666px) 100vw, 1666px\" \/><\/a><figcaption id=\"caption-attachment-7237\" class=\"wp-caption-text\">KVM Changing VMs while backing up<\/figcaption><\/figure>\n<p>So on the left hand terminal, there&#8217;s the ssh into the guest virtual machine to run a script called \/root\/fillerup.sh which just executes dd commands into \/d\/data. After starting that, I then flicked across to the second terminal to run the backup. Ergo, when we restore the virtual machine, we should get back some content into \/d\/data &#8212; at least, whatever had been generated and could be quiesced at the start of the snapshot process.<\/p>\n<p>After the backup was completed, in order to test, I needed to destroy the virtual machine so I could recover it. (Relax, I&#8217;ve tested this a few times.) The process therefore was:<\/p>\n<figure id=\"attachment_7238\" aria-describedby=\"caption-attachment-7238\" style=\"width: 833px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-delete-virtual-machine.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-7238\" src=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-delete-virtual-machine.png\" alt=\"KVM delete virtual machine\" width=\"833\" height=\"916\" srcset=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-delete-virtual-machine.png 833w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-delete-virtual-machine-273x300.png 273w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-delete-virtual-machine-768x845.png 768w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-delete-virtual-machine-182x200.png 182w\" sizes=\"auto, (max-width: 833px) 100vw, 833px\" \/><\/a><figcaption id=\"caption-attachment-7238\" class=\"wp-caption-text\">KVM delete virtual machine<\/figcaption><\/figure>\n<p>So in the above, the deletion of the virtual machine consisted of: shutting it down, undefining it, then deleting the virtual disks. After that was done, I could use the NetWorker&nbsp;<em>recover<\/em> command to get the data back. This is a two step recovery &#8211; recover the virtual disks, and then recover the configuration file. Here&#8217;s the virtual disk recovery process:<\/p>\n<figure id=\"attachment_7239\" aria-describedby=\"caption-attachment-7239\" style=\"width: 833px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Recover-Virtual-Disks.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-7239\" src=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Recover-Virtual-Disks.png\" alt=\"KVM Recover Virtual Disks\" width=\"833\" height=\"916\" srcset=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Recover-Virtual-Disks.png 833w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Recover-Virtual-Disks-273x300.png 273w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Recover-Virtual-Disks-768x845.png 768w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Recover-Virtual-Disks-182x200.png 182w\" sizes=\"auto, (max-width: 833px) 100vw, 833px\" \/><\/a><figcaption id=\"caption-attachment-7239\" class=\"wp-caption-text\">KVM Recover Virtual Disks<\/figcaption><\/figure>\n<p>After the virtual disks were recovered, I then recovered the configuration file. Now, I could have recovered them all at the same time, but I generate the configuration file to a different location, which means a relocated recovery into a single directory would result in sub-directories being created, and I wanted to keep it simple:<\/p>\n<figure id=\"attachment_7240\" aria-describedby=\"caption-attachment-7240\" style=\"width: 833px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Recover-Config-File.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-7240\" src=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Recover-Config-File.png\" alt=\"KVM Recover Config File\" width=\"833\" height=\"659\" srcset=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Recover-Config-File.png 833w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Recover-Config-File-300x237.png 300w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Recover-Config-File-768x608.png 768w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Recover-Config-File-253x200.png 253w\" sizes=\"auto, (max-width: 833px) 100vw, 833px\" \/><\/a><figcaption id=\"caption-attachment-7240\" class=\"wp-caption-text\">KVM Recover Config File<\/figcaption><\/figure>\n<p>So at that point, that&#8217;s left me with the configuration file and .qcow2 files into the \/home\/kvm\/cent7 directory. To complete the process, all I need to do is:<\/p>\n<ul>\n<li>Import the virtual machine from the restored configuration dump file<\/li>\n<li>Start the virtual machine<\/li>\n<li>Log onto the virtual machine and confirm there were some files created in \/d\/data<\/li>\n<\/ul>\n<p>Here&#8217;s what that looked like:<\/p>\n<figure id=\"attachment_7241\" aria-describedby=\"caption-attachment-7241\" style=\"width: 833px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Recreation-and-Verification.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-7241\" src=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Recreation-and-Verification.png\" alt=\"KVM Recreation and Verification\" width=\"833\" height=\"916\" srcset=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Recreation-and-Verification.png 833w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Recreation-and-Verification-273x300.png 273w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Recreation-and-Verification-768x845.png 768w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/KVM-Recreation-and-Verification-182x200.png 182w\" sizes=\"auto, (max-width: 833px) 100vw, 833px\" \/><\/a><figcaption id=\"caption-attachment-7241\" class=\"wp-caption-text\">KVM Recreation and Verification<\/figcaption><\/figure>\n<p>So there you have it: using NetWorker to backup and recover KVM.<\/p>\n<p>This is a classic example of what I&#8217;ve always said about enterprise backup products: it doesn&#8217;t matter what options they ship with so long as you&#8217;ve got the capability of extending their functionality to suit your needs. The script I&#8217;ve created, for instance, could in theory be called as part of a&nbsp;<em>pre command<\/em> when doing a hypervisor backup to first backup the guest virtual machines. (Or it could be run by the KVM administrator.) Because in this case we name the savesets based on the hypervisor, guest and disk\/config, it theoretically makes it easy to track virtual machines if they get moved between hypervisors (though I don&#8217;t have a full environment to test that under).<\/p>\n<p>If you&#8217;ve got KVM guests on Linux that you want to backup using NetWorker, feel free to check out version 1 of my savekvm script, <a href=\"https:\/\/nsrd.info\/scripts\/savekvm.zip\">which you can download from here<\/a>. (As always when you download a script from somewhere: test it out, review it carefully, and use it at your own risk!)<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Of late I&#8217;ve been playing around a little bit with KVM. I must admit, it&#8217;s not really my virtualisation system&hellip;<\/p>\n","protected":false},"author":1,"featured_media":7233,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[1154,16,19,20],"tags":[134,206,1488,1487,1248,1249,1274],"class_list":["post-7232","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-avamar-2","category-networker","category-recovery","category-scripting","tag-avamar","tag-centos","tag-image-backup","tag-kvm","tag-linux","tag-networker","tag-rhel"],"aioseo_notices":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2018\/11\/bigStock-Conceptual-System.jpg","jetpack_shortlink":"https:\/\/wp.me\/pKpIN-1SE","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/7232","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/comments?post=7232"}],"version-history":[{"count":2,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/7232\/revisions"}],"predecessor-version":[{"id":7332,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/7232\/revisions\/7332"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/media\/7233"}],"wp:attachment":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/media?parent=7232"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/categories?post=7232"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/tags?post=7232"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}