{"id":755,"date":"2009-09-28T07:51:53","date_gmt":"2009-09-27T21:51:53","guid":{"rendered":"http:\/\/nsrd.wordpress.com\/?p=755"},"modified":"2009-09-28T07:51:53","modified_gmt":"2009-09-27T21:51:53","slug":"routine-filesystem-checks-on-disk-backup-units","status":"publish","type":"post","link":"https:\/\/nsrd.info\/blog\/2009\/09\/28\/routine-filesystem-checks-on-disk-backup-units\/","title":{"rendered":"Routine filesystem checks on disk backup units"},"content":{"rendered":"<p>On Linu, filesystems typically have two settings regarding getting complete checks on boot. These are:<\/p>\n<ul>\n<li>Maximum number of mounts before a check<\/li>\n<li>Interval between checks<\/li>\n<\/ul>\n<p>The default settings, while reasonably suitable for smaller partitions, are very unsuitable for large partitions, such as what you find in disk backup units. In fact, if you don&#8217;t pay particular attention to these settings, you may find after a routine reboot that your backup server (or storage node) can take hours to become available. For instance, it&#8217;s not unheard of to see even sub-20TB DBU environments (as say, 10 x 2TB filesystems) take several <em>hours<\/em> to complete mandatory checks on filesystems after what should have just been a routine reboot.<\/p>\n<p>There are two approaches that you can take to this:<\/p>\n<ul>\n<li>If you want to leave the checks enabled, it&#8217;s reasonably imperative to ensure that at most only one disk backup unit filesystem will be checked at one time after a reboot; this will at least reduce the size of any check-on-reboot. Thus, ensure you:\n<ul>\n<li>Configure each filesystem so that it will have a different number of maximum mounts before check than any other filesystem, <em>and<\/em>,<\/li>\n<li>Configure the interval (days) between checks for each filesystem to be a <em>significantly <\/em>different number.<\/li>\n<\/ul>\n<\/li>\n<li>If you don&#8217;t want periodic filesystem checks to ever interfere with the reboot process, you need to:\n<ul>\n<li>Ensure that following a non-graceful restart of the server the DBU filesystems are unmounted and checked <em>before<\/em> any new backup or recovery activities are done, <em>and<\/em>,<\/li>\n<li>Ensure that there are processes &#8211; planned maintenance windows if you will &#8211; for manual running of the filesystem checks that are being skipped.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>Neither option is particularly &#8220;attractive&#8221;. In the first case, you can still, if you cherish uptime or don&#8217;t need to reboot your backup server often, get into a situation where multiple filesystems need to be checked on reboot if they&#8217;ve all exceeded their days-between-checks parameter. In the second instance, you&#8217;re having to insert human driven processes into what should normally be a routine operating system function. In particular with the manual option, there must be a process in place to NetWorker shutdown + checking <em>even in the middle of the night if an OS crash occurs<\/em>.<\/p>\n<p>Actually, the above list is a little limited &#8211; there&#8217;s a couple of other options that you can consider as well &#8211; though they&#8217;re a little more left of field:<\/p>\n<ul>\n<li>Build into the change control process the timings for complete filesystem checks <em>in case they happen<\/em>, <em>or<\/em><\/li>\n<li>Build into the change control process or reboot procedure for the backup server\/storage nodes the requirement to temporarily disable filesystem checks (using say, tune2fs) so that you <em>know<\/em> the reboot to be done won&#8217;t be costly in terms of time.<\/li>\n<\/ul>\n<p>Personally, I&#8217;m looking forward to <a title=\"btrfs Wiki\" href=\"http:\/\/btrfs.wiki.kernel.org\/index.php\/Main_Page\" target=\"_blank\">btrfs<\/a> &#8211; in reality, a modern filesystem such as that should solve most, if not all, of the problems discussed above.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>On Linu, filesystems typically have two settings regarding getting complete checks on boot. These are: Maximum number of mounts before&hellip;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[15,16],"tags":[258,1248,1249,732,786],"class_list":["post-755","post","type-post","status-publish","format-standard","hentry","category-linux","category-networker","tag-crash","tag-linux","tag-networker","tag-performance","tag-reboot"],"aioseo_notices":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/pKpIN-cb","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/755","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/comments?post=755"}],"version-history":[{"count":0,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/755\/revisions"}],"wp:attachment":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/media?parent=755"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/categories?post=755"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/tags?post=755"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}