{"id":5808,"date":"2016-02-03T18:01:52","date_gmt":"2016-02-03T08:01:52","guid":{"rendered":"http:\/\/nsrd.info\/blog\/?p=5808"},"modified":"2018-12-11T10:58:31","modified_gmt":"2018-12-11T00:58:31","slug":"client-load-filesystem-and-database-backups","status":"publish","type":"post","link":"https:\/\/nsrd.info\/blog\/2016\/02\/03\/client-load-filesystem-and-database-backups\/","title":{"rendered":"Client Load: Filesystem and Database Backups"},"content":{"rendered":"<p>A question I get asked periodically is &#8220;can I backup my filesystem and database at the same time?&#8221;<\/p>\n<p>As is often the case, the answer is: &#8220;it depends&#8221;.<\/p>\n<p><a href=\"https:\/\/nsrd.info\/blog\/2016\/02\/03\/client-load-filesystem-and-database-backups\/istock-photo-server-fireman\/\" rel=\"attachment wp-att-5809\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-5809\" src=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2016\/02\/iStock-Photo-Server-Fireman.jpg\" alt=\"Server on Fire\" width=\"425\" height=\"282\" srcset=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2016\/02\/iStock-Photo-Server-Fireman.jpg 425w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2016\/02\/iStock-Photo-Server-Fireman-300x199.jpg 300w\" sizes=\"auto, (max-width: 425px) 100vw, 425px\" \/><\/a>Or, to put it another way: it depends on what&nbsp;the specific client can handle at the time.<\/p>\n<p>For the most part, backup products have a fairly basic design requirement: get the data from the source&nbsp;(let&#8217;s say &#8220;the client&#8221;, ignoring options like ProtectPoint for the moment) to the destination (protection storage) as quickly as possible. The faster the better, in fact.&nbsp;So if we want backups&nbsp;done as fast as possible, wouldn&#8217;t it make sense to backup the filesystem and any databases on the client at the same time? Well&nbsp;\u2013 the answer is &#8220;it depends&#8221;, and it comes down to the&nbsp;<em>impact<\/em> it has on the client and the&nbsp;<em>compatibility<\/em> of the client to the process.<\/p>\n<p>First, let&#8217;s consider&nbsp;compatibility \u2013 if both the&nbsp;filesystem and database backup process use the same snapshot&nbsp;mechanism for instance, and only one can have a snapshot operational at&nbsp;any given time, that immediately rules out doing both at once.&nbsp;That&#8217;s the most obvious scenario, but the more subtle one almost comes back to the age-old parallelism problem: how fast is&nbsp;<em>too<\/em> fast?<\/p>\n<p>If we&#8217;re simultaneously conducting a&nbsp;complete filesystem read (say, in the case of a full backup)&nbsp;<em>and<\/em> simultaneously&nbsp;reading an entire database&nbsp;<em>and<\/em> the database and filesystem we&#8217;re reading from both reside on the same physical LUN, there is the potential the&nbsp;two reads will be counter-productive:&nbsp;if the underlying physical LUN is in fact a single disk, you&#8217;re practically guaranteed that&#8217;s the case, for instance. We wouldn&#8217;t normally want RAID-less storage for&nbsp;pretty&nbsp;much anything in production, but&nbsp;just slipping RAID into the equation doesn&#8217;t guarantee we can achieve both reads simultaneously without impact to the client \u2013 particularly if the client is already doing&nbsp;<em>other things<\/em>.&nbsp;<em>Production things<\/em>.<\/p>\n<p>Virtualisation doesn&#8217;t write a blank cheque,&nbsp;either; image level backup with databases&nbsp;<em>in the image<\/em> are a bit of a holy grail in the backup&nbsp;industry but even in those situations where it may be supported, it&#8217;s not supported for&nbsp;<em>every database type<\/em>; so it&#8217;s still more common than not to see situations where you have&nbsp;virtual\/image level backups for the guest for crash consistency on the file and operating system components, and then&nbsp;an in-guest database agent running for that true guaranteed database recoverability. Do you want a database and image based backup happening at the same time? Your hypervisor is furiously reading&nbsp;the image file&nbsp;<em>while<\/em> the in-guest agent is furiously reading the database.<\/p>\n<p>In each case that&#8217;s just at a per client level. Zooming out for a bit in a datacentre with hundreds or thousands of hosts all accessing shared storage via shared networking, usually via shared compute resources as well,&nbsp;<em>how long is a piece of string<\/em>&nbsp;becomes a exponentially increasing question as the number of shared resources and items sharing those resources start to come into play.<\/p>\n<p>Unless you have an overflow of compute resources and&nbsp;SSD offering more IO than your systems can ever need,&nbsp;<em>can I backup my&nbsp;filesystem and databases at the same time<\/em> is very much a non-trivial question. In fact, it becomes a bit of an art, as does&nbsp;all performance tuning. So&nbsp;rather than directly answering the question, I&#8217;ll&nbsp;make a few suggestions to be considered along the way as&nbsp;<em>you<\/em> answer the question for your environment:<\/p>\n<ul>\n<li><strong>Recommendation<\/strong>:&nbsp;Particularly for&nbsp;traditional filesystem agent + traditional database agent backups,&nbsp;<em>never<\/em> start the two within&nbsp;five minutes, and&nbsp;<em>preferably<\/em> give half an hour gap between starts. I.e., overlap is OK, concurrency for starting should be avoided where possible.<\/li>\n<li><strong>Recommendation<\/strong>: Make sure the two functions&nbsp;<em>can<\/em> be concurrently executed. I.e., if&nbsp;one blocks the other from running at the same time, you have your answer.<\/li>\n<li><strong>Remember<\/strong>: It&#8217;s all parallelism. Rather than a former CEO leaping around stage shouting &#8220;developers, developers, developers!&#8221;&nbsp;imagine me leaping around shouting &#8220;parallelism, parallelism, parallelism!&#8221;* \u2013 at the&nbsp;end of the day each concurrent filesystem backup&nbsp;uses a unit of parallelism and each concurrent database backup uses a unit of parallelism, so if you exceed what the client can naturally do based on memory, CPU resources, network resources or disk resources, you have your answer.<\/li>\n<li><strong>Remember<\/strong>: Backup isn&#8217;t&nbsp;<strong>ABC<\/strong>, it&#8217;s <strong>CDE<\/strong>:&nbsp;<strong>C<\/strong>ompression, <strong>D<\/strong>eduplication, <strong>E<\/strong>ncryption: Each function will adjust the&nbsp;performance characteristics of the host you&#8217;re backing up \u2013 sometimes subtly, sometimes not so.&nbsp;Compression and encryption are easier to understand: if you&#8217;re doing&nbsp;either as a client-CPU function you&#8217;re likely going to be hammering the&nbsp;host. Deduplication gets trickier of course \u2013 you might be doing a bit more CPU processing on the host, but over a shorter period of time if the net result if a 50-99% reduction in the&nbsp;amount of data you&#8217;re sending.<\/li>\n<li><strong>Remember<\/strong>:&nbsp;You need the up-close&nbsp;<em>and<\/em>&nbsp;big picture view. It&#8217;s rare we have systems so isolated any more that you can consider this in the perspective of a single host. What&#8217;s the rest of the environment doing or likely to be doing?<\/li>\n<li><strong>Remember<\/strong>: &#8216;More magic&#8217; is better than &#8216;magic&#8217;. (OK,&nbsp;it&#8217;s unrelated, <a href=\"http:\/\/catb.org\/jargon\/html\/magic-story.html\" target=\"_blank\">but it&#8217;s always a good story to tell<\/a>.)<\/li>\n<li><strong>Most&nbsp;importantly<\/strong>: Test. Once you&#8217;ve looked at your environment, once you&#8217;ve worked out the parallelism, once&nbsp;you&#8217;re happy the combined impact of a filesystem and database backup won&#8217;t go&nbsp;beyond the operational allowances on&nbsp;the host \u2013 particularly on anything remotely approaching mission critical \u2013&nbsp;<em>test it<\/em>.<\/li>\n<\/ul>\n<p>If you were hoping there was an easy answer, the only one I can give you is&nbsp;<em>don&#8217;t<\/em>, but that&#8217;s just making a blanket assumption you can never or&nbsp;should never do it.&nbsp;It&#8217;s the glib\/easy answer \u2013 the real answer is: only you can answer&nbsp;the question.<\/p>\n<p>But trust me: when you do, it&#8217;s immensely satisfying.<\/p>\n<p>On another note: I&#8217;m pleased to say I made it into the EMC Elect programme for another year \u2013 that&#8217;s every year since it started! If you&#8217;re looking for some great technical people within the EMC community (partners, employees, customers) to keep an eye on, <a href=\"https:\/\/community.emc.com\/community\/connect\/emc_elect\" target=\"_blank\">make sure you check out the announcement page<\/a>.<\/p>\n<p>&#8212;<br \/>\n* Try saying &#8220;parallelism, parallelism, parallelism!&#8221; three times fast when you had a speech impediment as a kid. It doesn&#8217;t look good.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A question I get asked periodically is &#8220;can I backup my filesystem and database at the same time?&#8221; As is&hellip;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[5,1133,9],"tags":[1293,382,1263,732],"class_list":["post-5808","post","type-post","status-publish","format-standard","hentry","category-backup-theory","category-best-practice","category-databases","tag-database-backup","tag-filesystem","tag-image-level-backup","tag-performance"],"aioseo_notices":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/pKpIN-1vG","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/5808","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/comments?post=5808"}],"version-history":[{"count":7,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/5808\/revisions"}],"predecessor-version":[{"id":7420,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/5808\/revisions\/7420"}],"wp:attachment":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/media?parent=5808"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/categories?post=5808"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/tags?post=5808"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}