Let’s consider a theoretical scenario: your business has been hit by Ransomware, and a chunk of data on a fileserver has been crypto-locked. So the question is: what will get the data back faster – paying the ransom (assuming the criminals actually return a key that will allow decryption), or recovering the data?
It should be recovery, right? That’s what I thought too when I discussing it with some colleagues recently, but like anyone else I’m capable of making incorrect assumptions, so I wanted to run some tests to compare.
So in my home lab, I setup and patched a new Windows 2016 server install with 4GB of RAM and 2 x vCPU running in VMware, on SSD. I added to the virtual machine a 100GB thickly provisioned disk (eager zeroed so there’d be no distortion due to VMDK expansion) to the server, and copied to it about 42 GB of random documents: PDF files, plain text files, HTML files, a few graphics here and there, and just a few .OVA files. In total, 37,230 files.
Backups were done to a 0.5TB DDVE running DDOS 6.2.0.5 sitting on another SSD on the same VMware ESX server. (In fact, NetWorker and the vProxy also reside on the same ESX server. The NetWorker and vProxy instances run on another SSD.)
So, what was the test?
Assuming all of the data on that Windows 2016 server’s ‘file’ drive was encrypted, would it be faster to recover from the most recent backup, or (presumably after paying the ransom and getting an unlock key), faster to decrypt the data?
Being disinclined to chase down an actual Cryptoware virus, I hunted for some Windows command line encryption tools. On that front I figured the goal should be something that works quickly – sure, you could do some sort of high-end GPG encryption using 4K key sizes, or big one-time-pad ciphers, but surely a critical aspect of Cryptoware is to churn through as much data/as many files as possible in as short a period of time as possible? After a lot of futzing around, I settled on crypt. It fulfilled the purpose in that it didn’t try to boil the ocean and had a simple enough command line interface that it allowed for a single command to recursively encrypt or decrypt an entire directory tree. That utility uses the (now old) Windows Crypt API, but still gets the job done. Probably not the most secure encryption method these days but it satisfies the “do it quickly” rule.
- Test – Agent based backup and restore using PSS:
- (Backup: 8 minutes and 42 seconds for just the drive containing the files.)
- Recover: 14 minutes 29 seconds.
- Test – Virtual machine backup and restore:
- (First backup:19 minutes 52 seconds for a full image based backup).)
- (Subsequent virtual synthetic full backup: 1 minute 0 seconds (no changes to VM).)
- File level recovery from image level backup: 38 minutes and 33 seconds.
- Changed Block Tracking restore: 11 minutes and 56 seconds.
- Test – Decrypt:
- (Encryption took 1 hour 21 minutes, 12 seconds.)
- Decryption took 1 hour 4 minutes, 48 seconds.
That’s all pretty definitive results there: if we’re backing up that data via a conventional NetWorker agent and using parallel save-streams (PSS), I was able to recover the 42GB in less than 15 minutes. File level recovery from image level backup took a while longer, as you’d expect – there’s a bit more work going on in the background, but I’m also recovering a large number of files, there. A changed block tracking recovery (where we do an overwrite recovery of just the blocks that had changed – i.e., everything that had been encrypted) was the real winner: 11 minutes and 56 seconds for a 146 GB virtual machine.
Decrypting it all? Over an hour.
Which serves to prove my point: backup is your first line of defence from Cryptoware style attack; if you’ve designed your backup environment properly it’ll let you get your data back a hell of a lot faster than paying a ransom and decrypting the data. Of course, the great thing about Data Domain when it comes to this scenario is that it gives you layers of protection, viz.:
- Using the Boost protocol means that your backup devices aren’t actually ‘mounted’ (in terms of a visible filesystem) on your backup server/storage nodes. So any crypto tool that somehow hits your backup server isn’t going to corrupt your backups themselves: they won’t be visible to it.
- Taking it to the next level, NetWorker now fully supports interacting with Data Domain’s Retention Lock functionality. So if you’re feeling more concerned on that front (e.g., your business is concerned that someone might write a tool that issues delete-backup-commands for common backup products), you can enable Retention Lock and ensure that backups, once written, stay there.
- Taking it to the next level again – you can also integrate with a cyber-recovery environment if you have to be ultra cautious – of either insider actors, or to meet compliance regulations of course.
If you’re worried about ransomware in your environment, your first step should be to design your backup for recovery.