{"id":5344,"date":"2014-10-27T18:56:59","date_gmt":"2014-10-27T08:56:59","guid":{"rendered":"http:\/\/nsrd.info\/blog\/?p=5344"},"modified":"2018-12-11T13:39:20","modified_gmt":"2018-12-11T03:39:20","slug":"parallel-save-streams-for-hdfs-performance-gains","status":"publish","type":"post","link":"https:\/\/nsrd.info\/blog\/2014\/10\/27\/parallel-save-streams-for-hdfs-performance-gains\/","title":{"rendered":"Parallel Save Streams for HDFS Performance Gains"},"content":{"rendered":"

One of the great features in NetWorker 8.1 was Parallel Save Streams (PSS). This allows for a single High Density File System (HDFS) to be split into multiple concurrent savesets to speed up the backup walk process and therefore the overall backup.<\/p>\n

In NetWorker 8.2 this was expanded to also support Windows filesystems.<\/p>\n

Traditionally, of course, a single filesystem or single saveset, if left to NetWorker, will be backed up as a single save operation:<\/p>\n

\"Traditional<\/a><\/p>\n

With PSS enabled, what would otherwise be a single saveset is split automatically by NetWorker and ends up looking like:<\/p>\n

\"Parallel<\/a><\/p>\n

I’ve previously mentioned parallel save streams, but it occurred to me that periodic test backups I do in my home lab server against a Synology filesystem might be the perfect way of seeing the difference PSS can make.<\/p>\n

Now, we all know how fun Synology storage is, and I have a 1513+ with 5 x Hitachi 3TB HDS723030ALA640 drives in a RAID-5 configuration, which is my home NAS server*. It’s connected to my backbone gigabit network via a TP-Link SG2216 16 port managed switch, as is my main lab server, a HP Microserver N40L with Dual AMD 1.5 Turion processors and 4GB of RAM. Hardly a power-house server, and certainly not even a recommended NetWorker server configuration.<\/p>\n

Synology of course, curse them, don’t support NDMP, so the Synology filesystem is mounted on the backup server via read-only NFS and backed up via the mount point.<\/p>\n

In a previous backup attempt using a standard single save stream, the backup device was an AFTD consisting of RAID-0 SATA drives plugged into the server directly. Here was the backup results:<\/p>\n

 orilla.turbamentis.int: \/synology\/homeshare level=full, 310 GB 48:47:48  44179 files<\/pre>\n
48 hours, 47 minutes. With saveset compression turned on.<\/p>\n
It occurred to me recently to see whether I’d get a performance gain by switching such a backup to parallel save streams. Keeping saveset compression turned on, this was the result:<\/p>\n
orilla.turbamentis.int:\/synology\/homeshare parallel save streams summary orilla.turbamentis.int: \/synology\/homeshare level=full, 371 GB 04:00:14  40990 files<\/pre>\n
4,000 less files to be sure, but a drop in backup time from 48 hours 47 minutes down to 4 hours and 14 seconds.<\/p>\n
If you’re needing to do traditional backups with high density filesystems, you really should evaluate parallel save streams.<\/p>\n
—
\n* Yes, I gave in and bought a home NAS server.<\/p>\n","protected":false},"excerpt":{"rendered":"
One of the great features in NetWorker 8.1 was Parallel Save Streams (PSS). This allows for a single High Density File System (HDFS) to…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[5,16],"tags":[1195,1196,1197,1194,1193,732,1192],"class_list":["post-5344","post","type-post","status-publish","format-standard","hentry","category-backup-theory","category-networker","tag-hdfs","tag-high-density-file-system","tag-high-density-filesystem","tag-nfs","tag-parallel-save-streams","tag-performance","tag-pss"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t