{"id":5344,"date":"2014-10-27T18:56:59","date_gmt":"2014-10-27T08:56:59","guid":{"rendered":"http:\/\/nsrd.info\/blog\/?p=5344"},"modified":"2018-12-11T13:39:20","modified_gmt":"2018-12-11T03:39:20","slug":"parallel-save-streams-for-hdfs-performance-gains","status":"publish","type":"post","link":"https:\/\/nsrd.info\/blog\/2014\/10\/27\/parallel-save-streams-for-hdfs-performance-gains\/","title":{"rendered":"Parallel Save Streams for HDFS Performance Gains"},"content":{"rendered":"<p>One of the great features in&nbsp;NetWorker 8.1 was Parallel Save Streams (PSS). This allows&nbsp;for&nbsp;a single&nbsp;High Density File System (HDFS) to be split into multiple concurrent savesets to speed up the backup walk process and therefore the overall backup.<\/p>\n<p>In NetWorker 8.2 this was expanded to also support Windows filesystems.<\/p>\n<p>Traditionally, of course, a single&nbsp;filesystem or single saveset, if left to NetWorker, will be backed up as a single save operation:<\/p>\n<p><a href=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2014\/10\/Traditional-Saveset.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-5345\" src=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2014\/10\/Traditional-Saveset.png\" alt=\"Traditional Saveset Breakdown\" width=\"673\" height=\"246\" srcset=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2014\/10\/Traditional-Saveset.png 673w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2014\/10\/Traditional-Saveset-300x109.png 300w\" sizes=\"auto, (max-width: 673px) 100vw, 673px\" \/><\/a><\/p>\n<p>With PSS enabled,&nbsp;what would otherwise be a single saveset is split automatically by&nbsp;NetWorker and ends up looking like:<\/p>\n<p><a href=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2014\/10\/Parallel-Save-Stream.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-5346\" src=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2014\/10\/Parallel-Save-Stream.png\" alt=\"Parallel Save Streams\" width=\"673\" height=\"276\" srcset=\"https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2014\/10\/Parallel-Save-Stream.png 673w, https:\/\/nsrd.info\/blog\/wp-content\/uploads\/2014\/10\/Parallel-Save-Stream-300x123.png 300w\" sizes=\"auto, (max-width: 673px) 100vw, 673px\" \/><\/a><\/p>\n<p>I&#8217;ve&nbsp;previously mentioned parallel save streams, but it occurred to me that periodic test&nbsp;backups I do in my&nbsp;home lab server against a Synology filesystem might be the perfect&nbsp;way of seeing the difference PSS can make.<\/p>\n<p>Now, we all know how fun Synology storage is, and I have a 1513+ with 5 x Hitachi 3TB&nbsp;HDS723030ALA640 drives in a RAID-5 configuration, which is my home NAS server*. It&#8217;s connected to my backbone&nbsp;gigabit network via a TP-Link&nbsp;SG2216 16 port managed switch, as is my main lab&nbsp;server, a HP Microserver N40L with Dual AMD 1.5 Turion processors and 4GB of RAM. Hardly a power-house server, and certainly not even a recommended&nbsp;NetWorker server&nbsp;configuration.<\/p>\n<p>Synology of course, curse them, don&#8217;t support NDMP, so&nbsp;the&nbsp;Synology filesystem is mounted on the backup server via read-only NFS and backed up via the mount point.<\/p>\n<p>In a previous backup attempt using&nbsp;a standard single save&nbsp;stream, the backup device was an AFTD&nbsp;consisting of RAID-0 SATA drives plugged into the server directly. Here was the backup results:<\/p>\n<pre>&nbsp;orilla.turbamentis.int: \/synology\/homeshare level=full, 310 GB 48:47:48 &nbsp;44179 files<\/pre>\n<p>48 hours, 47 minutes. With&nbsp;saveset&nbsp;compression turned on.<\/p>\n<p>It occurred to me recently to see whether I&#8217;d get a performance gain by switching such a backup to parallel save streams. Keeping saveset compression turned&nbsp;on, this was the result:<\/p>\n<pre>orilla.turbamentis.int:\/synology\/homeshare parallel save streams summary orilla.turbamentis.int: \/synology\/homeshare level=full, 371 GB 04:00:14 &nbsp;40990 files<\/pre>\n<p>4,000 less files to be sure, but a drop in backup time from 48 hours 47 minutes down to 4 hours and 14 seconds.<\/p>\n<p>If you&#8217;re needing to do traditional backups with high density filesystems, you really should evaluate&nbsp;parallel save streams.<\/p>\n<p>&#8212;<br \/>\n* Yes, I gave in and bought a home NAS server.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>One of the great features in&nbsp;NetWorker 8.1 was Parallel Save Streams (PSS). This allows&nbsp;for&nbsp;a single&nbsp;High Density File System (HDFS) to&hellip;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[5,16],"tags":[1195,1196,1197,1194,1193,732,1192],"class_list":["post-5344","post","type-post","status-publish","format-standard","hentry","category-backup-theory","category-networker","tag-hdfs","tag-high-density-file-system","tag-high-density-filesystem","tag-nfs","tag-parallel-save-streams","tag-performance","tag-pss"],"aioseo_notices":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/pKpIN-1oc","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/5344","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/comments?post=5344"}],"version-history":[{"count":4,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/5344\/revisions"}],"predecessor-version":[{"id":7448,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/5344\/revisions\/7448"}],"wp:attachment":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/media?parent=5344"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/categories?post=5344"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/tags?post=5344"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}