Frequent visitors to this blog will be well aware of the various comments I’ve made about the impact of filesystems on the performance of backup. Figuring it was time to actually churn out some data, I’ve done some controlled testing to demonstrate how filesystem traversal impedes backup performance.<\/p>\n
The environment:<\/p>\n
Obviously this isn’t a production performance environment \u2013 but honestly, it doesn’t matter: it’s all about the percentages and MB\/s performance differences between having to walk a filesystem to backup a lot of files, and then backup a single file that is an archive of those files. Those sort of differences remain the same regardless of whether you’re in a production environment or a lab environment.<\/p>\n
The reason backup-to-disk was used was to two-fold, with the reasons being:<\/p>\n
The test filesystem generated was 34GB in size, with 68,725 files spread across 9000 directories. Such a filesystem would be relatively indicative of a small-scale, moderately disorganised fileserver being primarily used for automated and manual document storage, Windows profile directories, etc.<\/p>\n
In order to demonstrate how the performance varies depending on the number of files on disk, a series of tests were run, with the first test below reflecting a tar<\/em> of the entire directory structure, and the final test representing all files in place. The tests in-between represent various numbers of files in place, with others replaced by tarred subdirectories. I.e., the net result is that in every case<\/em> it was the (net) same<\/strong> data being backed up, but just different numbers of individual files vs tar files (of those same files).<\/p>\n Here are the results:<\/p>\n The “all files” scenario, with approximately 77,725* files and directories gave an averaged performance of 10.7 MB\/s<\/strong>, whereas the backup of the tar of the filesystem averaged at 28.3 MB\/s<\/strong>. Bear in mind in each instance the same setup, the same data<\/em> was used, with the only<\/em> difference being the impact of walking the filesystem and processing individual files rather than a single chunk of data.<\/p>\n As you can see, that’s a relatively big change in performance – a little over 10 MB\/s difference between the backup that requires an ongoing filesystem walk and the backup that requires practically no traversal of a filesystem at all.<\/p>\n In case you’re wondering:<\/p>\n Thus, it cannot be said that this issue is caused by any time-impact of NetWorker processing indices for the number of files being backed up.<\/p>\n This is why, when examining performance for filesystem backups, we need to consider various options such as:<\/p>\n Clearly one important thing comes from needing to backup filesystems with lots<\/em> of files \u2013 it’s not something you can just point at high speed tape and hope to immediately get a good backup out of; rather, you need to architect<\/em> a compatible solution for your environment. Charging in headlong and working on the assumption that (a) your tape is fast, therefore the backup will be fast, (b) your source disk is fast, therefore the backup will be fast, or (c) that large block transfers are quick therefore filesystem traversals will be quick \u2013 are all flawed approaches.<\/p>\n — So at 35GB, the filesystem backup had an averaged performance of 10.1 MB\/s<\/strong>, whereas the backup of the tar of the filesystem (weighing in at 37GB) had an averaged performance of 29.3 MB\/s<\/strong>.<\/p>\n ** A product which, based on recent postings on the NetWorker mailing list, appears to be going away, so maybe it’s not really an option any more.<\/p>\n","protected":false},"excerpt":{"rendered":" Frequent visitors to this blog will be well aware of the various comments I’ve made about the impact of filesystems…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[3,5,16],"tags":[310,944],"class_list":["post-534","post","type-post","status-publish","format-standard","hentry","category-architecture","category-backup-theory","category-networker","tag-dense-filesystems","tag-store-index-entries"],"aioseo_notices":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/pKpIN-8C","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/534","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/comments?post=534"}],"version-history":[{"count":0,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/534\/revisions"}],"wp:attachment":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/media?parent=534"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/categories?post=534"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/tags?post=534"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}\n\n
\n # Files<\/strong><\/td>\n Time (min\/sec)
\n<\/strong><\/td>\n<\/tr>\n\n 5<\/td>\n 20m 29s<\/td>\n<\/tr>\n \n 659<\/td>\n 21m 7s<\/td>\n<\/tr>\n \n 2,554<\/td>\n 24m 34s<\/td>\n<\/tr>\n \n 19,712<\/td>\n 29m 29s<\/td>\n<\/tr>\n \n 27,275<\/td>\n 33m 33s<\/td>\n<\/tr>\n \n 31,047<\/td>\n 33m 45s<\/td>\n<\/tr>\n \n 39,981<\/td>\n 38m 51s<\/td>\n<\/tr>\n \n 46,483<\/td>\n 38m 56s<\/td>\n<\/tr>\n \n 77,725<\/td>\n 54m 15s<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n \n
\n
\n* A filesystem with ~70,000 files may not be sufficiently dense to make my point, so moving on to another scenario, I tweaked some settings on my random filesystem generator, and ended up with a filesystem that comprised approximately 4,900,000 files and directories, occupying approximately 35GB. Again, same systems and network settings were used, and both a filesystem\/directory backup was performed, as well as a backup of a single, monolithic tar file of the data. (Due to overheads, the tar file ended up being 37 GB.) Here’s the results:<\/p>\n\n