{"id":5543,"date":"2015-05-01T15:36:04","date_gmt":"2015-05-01T05:36:04","guid":{"rendered":"http:\/\/nsrd.info\/blog\/?p=5543"},"modified":"2018-12-11T12:58:29","modified_gmt":"2018-12-11T02:58:29","slug":"files-and-files-and-files","status":"publish","type":"post","link":"https:\/\/nsrd.info\/blog\/2015\/05\/01\/files-and-files-and-files\/","title":{"rendered":"Files and files and files"},"content":{"rendered":"

A while ago, I gave away a utility I find quite handy in lab and testing situations called genbf<\/em>. If you’ll recall, it can be used to generate large files which are not susceptible to compression or deduplication. (You can find that utility here<\/a>.)<\/p>\n

At the time I mentioned another utility I use called generate-filesystem<\/em>. While genbf<\/em> is designed to produce potentially very large files that don’t yield to compression, generate-filesystem<\/em> (or genfs2<\/em> as I’m now calling it) is designed to create a random filesystem for you. It’s not the same of course as taking say, a snapshot copy of your production fileserver, but if you’re wanting a completely isolated lab and some random content to do performance testing against, it’ll do the trick nicely. In fact, I’ve used it (or predecessors of it) multiple times when I’ve blogged about block based backups, filesystem density and parallel save streams.<\/p>\n

\"genfs2\"<\/a><\/p>\n

Overall it produces files that don’t yield all that much to compression. A 26GB directory structure with 50,000 files created with it compressed down to just 25GB in a test I ran a short while ago. That’s where genfs2 comes in handy – you can create really<\/em> dense test filesystems with almost no effort on your part. (Yes, 50,000 files isn’t necessarily dense, but that was just a small run.)<\/p>\n

It is however random by default on how many files it creates, and unless you give it an actual filesystem count limit, it can easily fill a filesystem if you let it run wild. You see, rather than having fixed limits for files and directories at each directory level, it works with upper and lower bounds (which you can override) and chooses a random number at each time. It even randomly chooses how many directories it nests down based on upper\/lower limits that you can override as well.<\/p>\n

Here’s what the usage information for it looks like:<\/p>\n

$ .\/genfs2.pl -h<\/strong>\nSyntax: genfs2.pl [-d minDir] [-D maxDir] [-f minFile] [-F maxFile] [-r minRecurse] [-R maxRecurse] -t target [-s minSize] [-S maxSize] [-l minLength] [-L maxLength] [-C] [-P dCsize] [-T mfc] [-q] [-I]\n\nCreates a randomly populated directory structure for backup\/recovery \nand general performance testing. Files created are typically non-\ncompressible.\n\nAll options other than target are optional. Values in parantheses beside\nexplanations denote defaults that are used if not supplied.\n\nWhere:\n\n    -d minDir      Minimum number of directories per layer. (5)\n    -D maxDir      Maximum number of directories per layer. (10)\n    -f minFile     Minimum number of files per layer. (5)\n    -F maxFile     Maximum number of files per layer. (10)\n    -r minRecurse  Minimum recursion depth for base directories. (5)\n    -R maxRecurse  Maximum recursion depth for base directories. (10)\n    -t target      Target where directories are to start being created.\n                   Target must already exist. This option MUST be supplied.\n    -s minSize     Minimum file size (in bytes). (1 K)\n    -S maxSize     Maximum file size (in bytes). (1 MB)\n    -l minLength   Minimum filename\/dirname length. (5)\n    -L maxLength   Maximum filename\/dirname length. (15)\n    -P dCsize      Pre-generate random data-chunk at least dcSize bytes.\n                   Will default to 52428800 bytes.\n    -C             Try to provide compressible files.\n    -I             Use lorem ipsum filenames.\n    -T             mfc Specify maximum number of files that will be created.\n                   Does not include directories in count.\n    -q             Quiet mode. Only print updates to the file-count.\n\nE.g.:\n\n.\/genfs2.pl -r 2 -R 32 -s 512 -S 65536 -t \/d\/06\/test\n\nWould generate a random filesystem starting in \/d\/06\/test, with a minimum\nrecursion depth of 2 and a maximum recursion depth of 32, with a minimum\nfilesize of 512 bytes and a maximum filesize of 64K.\n\n<\/pre>\n
Areas where this utility can be useful include:<\/p>\n