{"id":6262,"date":"2017-05-05T07:29:07","date_gmt":"2017-05-04T21:29:07","guid":{"rendered":"http:\/\/nsrd.info\/blog\/?p=6262"},"modified":"2018-12-11T08:39:13","modified_gmt":"2018-12-10T22:39:13","slug":"architecture-matters-when-you-dedupe","status":"publish","type":"post","link":"https:\/\/nsrd.info\/blog\/2017\/05\/05\/architecture-matters-when-you-dedupe\/","title":{"rendered":"Architecture Matters: When you dedupe"},"content":{"rendered":"

There was a time, comparatively not that long ago, when the biggest governing factor in LAN capacity for a datacentre was not the primary production workloads, but the mechanics of getting a full backup from each host over to the backup media. If you’ve been around in the data protection industry long enough you’ll have had experience of that \u2013 for instance, the drive towards 1Gbit networks over Fast Ethernet started more often than not in datacentres I was involved in thanks to backup. Likewise, the first systems I saw being attached directly to 10Gbit backbones in datacentres were the backup infrastructure.<\/p>\n

Well architected deduplication can eliminate that consideration. That’s not to say you won’t eventually need 10Gbit, 40Gbit or even more in your datacentre, but if deduplication is architected correctly, you won’t need to deploy that next level up of network performance to meet your backup requirements.<\/p>\n

In this blog article I want to take you through an example of why deduplication architecture matters, and I’ll focus on something that amazingly still gets consideration from time to time: post-ingest deduplication.<\/p>\n

Before I get started \u2013 obviously, Data Domain doesn’t use post-ingest deduplication. Its pre-ingest deduplication ensures the only data written to the appliance is already deduplicated, and it further increases efficiency by pushing deduplication segmentation and processing out to the individual clients (in a NetWorker\/Avamar environment) to limit the amount of data flowing across the network.<\/p>\n

A post-deduplication architecture though has your protection appliance feature two distinct tiers of storage \u2013 the landing or staging tier, and the deduplication tier. So that means when it’s time to do a backup, all your clients send all their data across the network to sit, in original sized format, on the staging tier:<\/p>\n

\"Post<\/a><\/p>\n

In the example above we’ve already had backups run to the post-ingest deduplication appliance; so there’s a heap of deduplicated data sitting in the deduplication tier, but our staging tier has just landed all the backups from each of the clients in the environment. (If it were NetWorker writing to the appliance, each of those backups would be the full sized savesets.)<\/p>\n

Now, at some point after the backup completes (usually a preconfigured time), post-processing kicks in. This is effectively a data-migration window in a post-ingest appliance where all the data in the staging tier has to be read and processed for deduplication. For example, using the example above, we might start with inspecting ‘Backup01’ for commonality to data on the deduplication tier:<\/p>\n

\"Post<\/a><\/p>\n

So the post-ingest processing engine starts by reading through all the content of Backup01<\/strong> and constructs fingerprint analysis of the data that has landed.<\/p>\n

\"Post<\/a><\/p>\n

As fingerprints are assembled, data can be compared against the data already residing in the deduplication tier. This may result in signature matches or signature misses, indicating new data that needs to be copied into the deduplication tier.<\/p>\n

\"Post<\/a><\/p>\n

In this it’s similar to regular deduplication \u2013 signature matches result in pointers for existing data being updated and extended, and a signature miss results in needing to store new data on the deduplication tier.<\/p>\n

\"Post<\/a><\/p>\n

Once the first backup file written to the staging tier has been dealt with, we can delete that file from the staging area and move onto the second backup file to start the process all over again. And we keep doing that over and over and over on the staging tier until we’re left with an empty staging tier:<\/p>\n

\"Post<\/a><\/p>\n

Of course, that’s not the end of the process \u2013 then the deduplication tier will have to run its regular housekeeping operations to remove data that’s no longer referenced by anything.<\/p>\n

Architecturally, post-ingest deduplication is a kazoo to pre-ingest deduplication’s symphony orchestra. Sure, you might technically get to hear the 1812 Overture, but it’s not really going to be the same, right?<\/p>\n

Let’s go through where architecturally, post-ingest deduplication fails you:<\/p>\n

    \n
  1. The network becomes your bottleneck again. You have to send all your backup data to the appliance.<\/li>\n
  2. The staging tier has to have at least as much capacity available as the size of your biggest backup<\/em>, assuming it can execute its post-process deduplication within the window between when your previous backup finishes and your next backup starts.<\/li>\n
  3. The deduplication process becomes entirely<\/em> spindle bound. If you’re using spinning disk, that’s a nightmare. If you’re using SSD, that’s $$$.<\/li>\n
  4. There’s no way of telling how much space will be occupied on the deduplication tier after deduplication processing completes. This can lead you into very messy situations where say, the staging tier can’t empty because the deduplication tier has filled. (Yes, capacity maintenance is a requirement still on pre-ingest deduplication systems, but it’s half<\/strong><\/span> the effort.)<\/li>\n<\/ol>\n

    What this means is simple: post-ingest deduplication architectures are asking you to pay for their architectural inefficiencies<\/em>. That’s where:<\/p>\n

      \n
    1. You have to pay to increase your network bandwidth to get a complete copy of your data from client to protection storage within your backup window.<\/li>\n
    2. You have to pay for both the staging tier storage and the deduplication tier storage. (In fact, the staging tier is often a lot bigger<\/em> than the size of your biggest backups in a 24-hour window so the deduplication can be handled in time.)<\/li>\n
    3. You have to factor the additional housekeeping operations into blackout windows, outages, etc. Housekeeping almost invariably becomes a daily rather than a weekly task, too.<\/li>\n<\/ol>\n

      Compare all that to pre-ingest deduplication:<\/p>\n

      \"Pre-Ingest<\/a><\/p>\n

      Using pre-ingest deduplication, especially Boost based deduplication, the segmentation and hashing happen directly where the data is, and rather than sending the entire<\/em> data to be protected from the client to the Data Domain, we only send the unique<\/strong> data. Data that already resides on the Data Domain? All we’ll have sent is a tiny fingerprint so the Data Domain can confirm it’s already there (and update its pointers for existing data), then moved on. After your first backup, that potentially means that on a day to day basis your network requirements for backup are reduced by 95% or more.<\/p>\n

      That’s why architecture matters: you’re either doing it right, or you’re paying the price for someone else’s inefficiency.<\/p>\n

      \n

      If you want to see more about how a well architected backup environment looks \u2013 technology, people and processes, check out my book, Data Protection: Ensuring Data Availability<\/strong><\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"

      There was a time, comparatively not that long ago, when the biggest governing factor in LAN capacity for a datacentre was…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[3],"tags":[1241,301,1361,625,732],"class_list":["post-6262","post","type-post","status-publish","format-standard","hentry","category-architecture","tag-architecture","tag-deduplication","tag-efficiency","tag-network","tag-performance"],"aioseo_notices":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/pKpIN-1D0","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/6262","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/comments?post=6262"}],"version-history":[{"count":8,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/6262\/revisions"}],"predecessor-version":[{"id":7389,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/6262\/revisions\/7389"}],"wp:attachment":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/media?parent=6262"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/categories?post=6262"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/tags?post=6262"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}