{"id":5352,"date":"2014-11-18T18:37:58","date_gmt":"2014-11-18T08:37:58","guid":{"rendered":"http:\/\/nsrd.info\/blog\/?p=5352"},"modified":"2018-12-11T13:38:45","modified_gmt":"2018-12-11T03:38:45","slug":"not-so-squeezy","status":"publish","type":"post","link":"https:\/\/nsrd.info\/blog\/2014\/11\/18\/not-so-squeezy\/","title":{"rendered":"Not so squeezy"},"content":{"rendered":"

It’s funny, the little tools you build up over the years as someone heavily involved in backup, particularly when it comes to testing.<\/p>\n

I have two tools that help me with filesystem and performance testing – one I call generate-filesystem<\/em>, and one called genbf<\/em> (generate big file).<\/p>\n

The genbf<\/em> tool came about when I wanted files that were highly resistant to being compressed \u2013 and indeed, to subsequently being deduplicated<\/em> as well. Sure, bigasm can produce good results, but it isn’t guaranteed to produce highly random data. That’s where genbf comes in. Best of all, it’s fast. For example, a 1GB file on my 12-core lab server gets created in under 10 seconds:<\/p>\n

[pmdg@orilla test]$ date; genbf.pl -s 1024 -f test.dat; date<\/strong>\nTue Nov 18 19:08:24 AEDT 2014\nProgress:\n     Pre-generating random data chunk. (This may take a while.)\n     0% of random data chunk generated.\n     10% of random data chunk generated.\n     20% of random data chunk generated.\n     30% of random data chunk generated.\n     40% of random data chunk generated.\n     50% of random data chunk generated.\n     60% of random data chunk generated.\n     70% of random data chunk generated.\n     80% of random data chunk generated.\n     90% of random data chunk generated.\n Creating 1024 MB file test.dat\nWrote data file in 5121 chunks.\nTue Nov 18 19:08:33 AEDT 2014<\/pre>\n
OK, OK, a 1GB file can be created quickly if you’re just pulling in from \/dev\/zero, but here’s the file size difference pre and post-compressed:<\/p>\n
[pmdg@orilla test]$ ls -al test.dat<\/strong> \n-rw-rw-r-- 1 pmdg pmdg 1073741824 Nov 18 19:08 test.dat\n[pmdg@orilla test]$ pbzip2 -r test.dat<\/strong>\n[pmdg@orilla test]$ ls -al test.dat.bz2<\/strong> \n-rw-rw-r-- 1 pmdg pmdg 1065615793 Nov 18 19:08 test.dat.bz2<\/pre>\n
(If you haven’t heard of pbzip2<\/a><\/em>, enlighten yourself and support the author. It’s brilliant.)<\/p>\n
When it comes to subsequently sending the generated data to Data Domain, the deduplication is extremely low – 20 x 1GB files using the standard setting above, for instance, yields an almost straight additional 20GB occupied space.<\/p>\n
If you want to try it out, you can download it from here<\/a>. (You’ll need Perl on your system.) Standard usage is below:<\/p>\n
\"genbf<\/a><\/p>\n
 <\/p>\n","protected":false},"excerpt":{"rendered":"
It’s funny, the little tools you build up over the years as someone heavily involved in backup, particularly when it…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[20,25],"tags":[242,301,1198],"class_list":["post-5352","post","type-post","status-publish","format-standard","hentry","category-scripting","category-tidbits","tag-compression","tag-deduplication","tag-random-file"],"aioseo_notices":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/pKpIN-1ok","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/5352","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/comments?post=5352"}],"version-history":[{"count":4,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/5352\/revisions"}],"predecessor-version":[{"id":7447,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/5352\/revisions\/7447"}],"wp:attachment":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/media?parent=5352"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/categories?post=5352"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/tags?post=5352"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}