{"id":5578,"date":"2015-05-20T20:21:17","date_gmt":"2015-05-20T10:21:17","guid":{"rendered":"http:\/\/nsrd.info\/blog\/?p=5578"},"modified":"2018-12-11T12:04:59","modified_gmt":"2018-12-11T02:04:59","slug":"pool-size-and-deduplication","status":"publish","type":"post","link":"https:\/\/nsrd.info\/blog\/2015\/05\/20\/pool-size-and-deduplication\/","title":{"rendered":"Pool size and deduplication"},"content":{"rendered":"

When you start looking into deduplication, one of the things that becomes immediately apparent is … size matters<\/em>. In particular, the size of your deduplication pool<\/em> matters.<\/p>\n

\"Deduplication<\/a><\/p>\n

In this respect, what I’m referring to is the analysis pool for comparison when performing deduplication. If we’re only talking target based deduplication, that’s simple \u2013 it’s the size of the bucket you’re writing your backup to. However, the problems with a purely target based deduplication approach to data protection are network congestion and time wasted \u2013 a full backup of a 1TB fileserver will still see 1TB of data transferred over the network to have most of its data dropped as being duplicate. That’s an awful lot of packets going to \/dev\/null, and an awful lot of bandwidth wasted.<\/p>\n

For example, consider the following diagram being of a solution using target only deduplication (e.g., VTL only or no Boost API on the hosts):<\/p>\n

\"Dedupe<\/a><\/p>\n

In this diagram, consider the outline arrow heads to indicate where<\/em> deduplication is being evaluated. Thus, if each server had 1TB of storage to be backed up, then each server would send 1TB of storage over to the Data Domain to be backed up, with deduplication performed only at the target end. That’s not how deduplication has to work now, but it’s a reminder of where we were only a relatively short period of time ago.<\/p>\n

That’s why source based deduplication (e.g., NetWorker Client Direct with a DDBoost enabled connection, or Data Domain Boost for Enterprise Applications) brings so many efficiencies to a data protection system. While there’ll be a touch more processing performed on the individual clients, that’ll be significantly outweighed by the ofttimes massive<\/em> reduction in data sent onto the network for ingestion into the deduplication appliance.<\/p>\n

So that might look more like:<\/p>\n

\"Source<\/a><\/p>\n

I.e., in this diagram with outline arrow heads indicating location of deduplication activities, we get an immediate change \u2013 each of those hosts will still have 1TB of backup to perform, but<\/em> they’ll evaluate via hashing mechanisms whether or not that data actually needs to be sent to the target appliance.<\/p>\n

There’s still efficiencies to be had even here though, which is where the original point about pool size<\/em> becomes critical. To understand why, let’s look at the diagram a slightly different way:<\/p>\n

\"Source<\/a><\/p>\n

In this case, we’ve still got source deduplication, but the merged lines represent something far more important … we’ve got global<\/em>, source deduplication.<\/p>\n

Or to put it a slightly different way:<\/p>\n