Utilities
Do you remember CVS? While I’d previously been a fan of RCS, I switched to CVS very early after it came out and just kept on with it over the years. Since then, all the utilities I wrote got shoved into CVS.
Recently I figured it was time I updated my perspective on source control. In that spirit, I’ve configured a GitHub repository to host some utilities.
I’ve started the repository with two utilities I’ve used a lot over the years: genbf2.pl and genfs2.pl. (That’s “generate big file” and “generate filesystem” respectively.) The repository is here: https://github.com/backupbear/backup-test-utilities.
Assuming I keep reminding myself to go back and add things, I’ll hopefully end up adding some other backup-related utilities to the above repository.
genbf2.pl
The genbf2.pl utility is pretty simple: it’s designed to generate a large file (you get to choose the size, of course) that’s relatively incompressible. This is handy if you want to test performance without compression/deduplication getting in the way. (Obviously, if you send the same generated file multiple times to a deduplication target without any changes, you’ll get awful deduplication on the first write, and pretty damn good deduplication for the 2nd+ writes. It’s not magic, after all.)
I’ve used this utility quite a few times in the past for performance testing, and while you can compress the file a little, it’s typically well less than a 1% size reduction.
genfs2.pl
Dense and ultra-dense filesystems are the bane of a backup environment. There are some ways around the problem – block-based backup, image-based backup (if the filesystem is within a VM), and parallel save-streams.
But testing the impact of dense and ultra-dense filesystems on backup is usually a challenge because to do that testing, you need a dense filesystem. Getting something that’s not predictable in layout and sufficiently dense requires more work than just endlessly copying the text-based dumps of all the RFCs or RISKS digests.
That’s where my genfs2.pl utility comes into play. It’ll generate a filesystem full of random files and directories of varying sizes. You can run it multiple times against the same target to build up more and more files. (As an example, with a little patience I was able to create an ultra-dense filesystem of more than 60,000,000 files in a 100GB folder.)
In fact, not only can you run the utility multiple times to increase filesystem density, but if the underlying storage supports it, you can open up a few console sessions and get multiple runs happening simultaneously. (This is particularly useful if you’re writing to RAID, or SSD/NVMe-based storage.)
Tools
I’m wanting to get back to some of the activities I used to love doing – web-related and database-related programming. So to kick-start that process, I converted a couple of spreadsheet-based utilities into PHP-based tools I’m now hosting on this site.
Base Converter
I can’t say how irksome I find the “mebibyte/megabyte” stuff. And yes, I can type details for conversion into Google and usually get the results I want, but that’s not as fun as writing it myself.
Monthly Growth Tables
Working in infrastructure-related segments, it’s common for me to have to run calculations for monthly growth based on an annual growth rate over a period of time. And I like to have that automatically generated for me in table format to make it easy to copy and paste.
My monthly growth table not only generates a table, but it can also output the calculations as a CSV file for download as well.
