{"id":2864,"date":"2011-02-19T14:32:39","date_gmt":"2011-02-19T04:32:39","guid":{"rendered":"http:\/\/nsrd.info\/blog\/?p=2864"},"modified":"2018-12-11T18:19:26","modified_gmt":"2018-12-11T08:19:26","slug":"smoothing-fulls","status":"publish","type":"post","link":"https:\/\/nsrd.info\/blog\/2011\/02\/19\/smoothing-fulls\/","title":{"rendered":"Smoothing Fulls"},"content":{"rendered":"<p>A common convention in organisations is that full backups are run on the weekend \u2013 regardless of whether that&#8217;s every weekend, or just one weekend a month.<\/p>\n<p>While there are undoubtedly some businesses that <strong>must<\/strong> run backups like this, it is by and large done out of convention rather than consideration; i.e., it is frequently done because &#8220;it&#8217;s the done thing&#8221; rather than &#8220;it&#8217;s the necessary thing&#8221;.<\/p>\n<p>This, unsurprisingly, puts pressure on backup resources, hardware requirements and data growth management \u2013 pressure that may actually be totally unnecessary. In short: if you do your full backups on the weekend because &#8220;that&#8217;s the way it&#8217;s always been done&#8221; then you might want to stop and consider the alternative.<\/p>\n<p>Let&#8217;s take a sample small business, and map out their weekly backup cycle. This isn&#8217;t actually one of my customers, but it is an average of several of them:<\/p>\n<ul>\n<li>Monday \u2013 150 GB backup.<\/li>\n<li>Tuesday \u2013 203 GB backup.<\/li>\n<li>Wednesday \u2013 168 GB backup.<\/li>\n<li>Thursday \u2013 317 GB backup.<\/li>\n<li>Friday \u2013 2114 GB backup.<\/li>\n<li>Saturday \u2013 3619 GB backup.<\/li>\n<li>Sunday \u2013 744 GB backup.<\/li>\n<\/ul>\n<p>If you&#8217;re doing a basic weekly backup cycle in a small to medium company then this sort of minimum, maximum and peak loading probably looks awfully familiar.<\/p>\n<p>Graphing this out, we get an interesting view of the backup size peaks and troughs:<\/p>\n<p><a href=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2011\/02\/GB-backup-nonsmoothed.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-2865\" title=\"GB backup non-smoothed (bar)\" src=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2011\/02\/GB-backup-nonsmoothed.png\" alt=\"GB backup non-smoothed (bar)\" width=\"485\" height=\"374\" srcset=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2011\/02\/GB-backup-nonsmoothed.png 485w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2011\/02\/GB-backup-nonsmoothed-300x231.png 300w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2011\/02\/GB-backup-nonsmoothed-389x300.png 389w\" sizes=\"auto, (max-width: 485px) 100vw, 485px\" \/><\/a><\/p>\n<p>&#8220;Streuth!&#8221;, an ocker Australian would say, &#8220;That&#8217;s a bloody big difference in backup sizes!&#8221;<\/p>\n<p>In fact, if we look at those backup sizes on a comparative pie chart, we get the following:<\/p>\n<p><a href=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2011\/02\/GB-backup-nonsmoothed-pie.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-2866\" title=\"GB backup non-smoothed (pie)\" src=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2011\/02\/GB-backup-nonsmoothed-pie.png\" alt=\"GB backup non-smoothed (pie)\" width=\"356\" height=\"213\" srcset=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2011\/02\/GB-backup-nonsmoothed-pie.png 356w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2011\/02\/GB-backup-nonsmoothed-pie-300x179.png 300w\" sizes=\"auto, (max-width: 356px) 100vw, 356px\" \/><\/a>Viewed this way, the weekend backups take up a significant percentage of the overall backup activity \u2013 which means they become a dominating factor in determining an optimum backup environment size. In fact, it shows us that 88% of the total amount of data backed up in a week is backed up in just 43% of the week &#8211; 3 out of the 7 days. The remaining 12% of data backed up during the week places no pressure on the backup environment at all.<\/p>\n<p>If we come up with an average backup speed \u2013 let&#8217;s say 50MB\/s for a smaller environment \u2013 we can see how long, in average terms, each day&#8217;s backup takes:<\/p>\n<p><a href=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2011\/02\/hours-to-backup-nonsmoothed.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-2867\" title=\"Hours to backup (non-smoothed) at 50MB\/s\" src=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2011\/02\/hours-to-backup-nonsmoothed.png\" alt=\"Hours to backup (non-smoothed) at 50MB\/s\" width=\"352\" height=\"262\" srcset=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2011\/02\/hours-to-backup-nonsmoothed.png 352w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2011\/02\/hours-to-backup-nonsmoothed-300x223.png 300w\" sizes=\"auto, (max-width: 352px) 100vw, 352px\" \/><\/a>Ouch \u2013 our system barely needs to tick over Monday through to Thursday, but once the weekend hits, it&#8217;s really having to work hard to get everything backed up.<\/p>\n<p>The net result? Backup windows may be regularly overrun, and even a moderate amount of data growth may necessitate new capital investment.<\/p>\n<p>Or will it? Let&#8217;s instead consider the same amount of data backed up, but with full backups spread out over the entire week. Now, admittedly here I&#8217;m not averaging numbers, but spreading sizes pseudo-randomly out over the week to match the previous amount of data specified. So our numbers instead look like:<\/p>\n<ul>\n<li>Monday \u2013 983 GB<\/li>\n<li>Tuesday \u2013 733 GB<\/li>\n<li>Wednesday \u2013 842 GB<\/li>\n<li>Thursday \u2013 928 GB<\/li>\n<li>Friday \u2013 1357 GB<\/li>\n<li>Saturday \u2013 1536 GB<\/li>\n<li>Sunday \u2013 986 GB<\/li>\n<\/ul>\n<p><em>[Edit: Qualification &#8211; an anonymous reader here questioned whether I meant doing a full backup every day. I didn&#8217;t quite explain my thinking here, sorry. I mean spreading out the full backups so that instead of trying to do them all over a short period, they&#8217;re spread out over the week. E.g., instead of every server doing a full backup on the weekend, some would do full backups on Monday, some on Tuesday, some on Wednesday, etc.]<\/em><\/p>\n<p>If we graph that, using the same minimum\/maximum as before, the spreading of full backups has smoothed the daily backup sizes considerably:<\/p>\n<p style=\"text-align: center;\"><a href=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2011\/02\/GB-backup-smoothed.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-2868\" title=\"GB backup smoothed (bar)\" src=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2011\/02\/GB-backup-smoothed.png\" alt=\"GB backup smoothed (bar)\" width=\"486\" height=\"354\" srcset=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2011\/02\/GB-backup-smoothed.png 486w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2011\/02\/GB-backup-smoothed-300x218.png 300w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2011\/02\/GB-backup-smoothed-411x300.png 411w\" sizes=\"auto, (max-width: 486px) 100vw, 486px\" \/><\/a><\/p>\n<p>Moving on to a pie graph, we can see that no single day dominates like before:<\/p>\n<p><a href=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2011\/02\/GB-backup-smoothed-pie.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-2869\" title=\"GB backup smoothed (pie)\" src=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2011\/02\/GB-backup-smoothed-pie.png\" alt=\"GB backup smoothed (pie)\" width=\"355\" height=\"211\" srcset=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2011\/02\/GB-backup-smoothed-pie.png 355w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2011\/02\/GB-backup-smoothed-pie-300x178.png 300w\" sizes=\"auto, (max-width: 355px) 100vw, 355px\" \/><\/a>While Friday\/Saturday\/Sunday still create a reasonable hit in the backup sizing, it&#8217;s just 53% of the size. So the balancing has substantially reduced the strain of weekend backups \u2013 sure, each week day the system has to do a bit more, but the overall pressure is considerably less. This is strongly demonstrated by looking at the daily hours of operation, at 50MB\/s:<\/p>\n<p><a href=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2011\/02\/hours-to-backup-smoothed.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-2870\" title=\"Hours to backup at 50MB\/s (smoothed)\" src=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2011\/02\/hours-to-backup-smoothed.png\" alt=\"Hours to backup at 50MB\/s (smoothed)\" width=\"349\" height=\"281\" srcset=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2011\/02\/hours-to-backup-smoothed.png 349w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2011\/02\/hours-to-backup-smoothed-300x241.png 300w\" sizes=\"auto, (max-width: 349px) 100vw, 349px\" \/><\/a>Instead of minimum run times in the order of less than an hour, but maximum run times of over 20 hours, we can now see a much more manageable peak run time of 8.74 hours.<\/p>\n<p>Next time you notice that your full backups are overrunning, or causing stress on your backup windows, stop for a moment and ask yourself: can you smooth your backup load by spreading the fulls out across the week? You may be surprised by the answer.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A common convention in organisations is that full backups are run on the weekend \u2013 regardless of whether that&#8217;s every&hellip;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[3,5,16],"tags":[154,401,403,899],"class_list":["post-2864","post","type-post","status-publish","format-standard","hentry","category-architecture","category-backup-theory","category-networker","tag-backup-window","tag-full-backups","tag-fulls","tag-smoothing"],"aioseo_notices":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/pKpIN-Kc","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/2864","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/comments?post=2864"}],"version-history":[{"count":1,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/2864\/revisions"}],"predecessor-version":[{"id":7526,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/2864\/revisions\/7526"}],"wp:attachment":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/media?parent=2864"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/categories?post=2864"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/tags?post=2864"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}