Bunkers vs Vaults

I want to spend a little while talking about bunkers vs vaults.

I’ve been thinking about what constitutes a vault vs a bunker quite a bit recently, and there are two reasons why this keeps percolating in my noggin’: public cloud, and physical tape. In this post, I want to concentrate on the latter. Now, tapes are something I’m highly familiar with. In my first system administration job, I was working with QIC tapes, DAT tapes, DLT tapes and AIT tapes. Working for systems integrators for years, I’ve dealt with SuperDLT, a large number of LTO generations and some of the StorageTek 9840 generation tapes, too.

In a recent update post on the LTO Roadmap, Chris Mellor of Blocks and Files wrote:

The Linear Tape Open (LTO) organisation has cut the capacity of its next LTO-9 generation from 24TB raw to 18TB, saying it is balancing capacity and cost for the current market.

LTO org changes tape roadmap strategy, cuts LTO-9 capacity to build air-gapped ransomware defence for storage, Chris Mellor, September 10 2020, Blocks and Files.

The title obviously grabbed my attention there: “build air-gapped ransomware defence for storage”. Mellor goes on to say:

Helpfully the LTO announcement includes a quote from Christophe Bertrand, senior analyst at ESG: “Tape’s role is shifting, which further enhances its value. Other than archiving massive data sets for active archives or longer retention at low cost, tape has the ability to create an isolated and ‘air-gapped’ layer. This emerging use case is perfect for keeping data out of reach of cyber-attackers.”

Ibid.

Look, there is no doubt that tapes, if removed from the protection environment, provide an air-gap. They’re literally off-line from the rest of your environment, but you have to ask yourself: is that enough?

Let’s start by having a look at Dell’s Cyber Vault solution. This is, without a doubt, the world’s leading Cyber Vault solution. (It is the only solution endorsed by the US Sheltered Harbor Program, for instance.) By comparison, most other vendors say they have a Cyber Vault solution, but it’ll either be:

• A tertiary copy (on-premises, or in the public cloud), or
• A tertiary copy that requires a substantial percentage of data landing in the storage to be analysed in the public cloud, which is hardly ideal.

So, Dell’s solution looks like this:

Now, the Vault does generate a tertiary copy of some of your data, namely the mission critical data. While there can be a lot of analysis in determining what that data is, you can sort of summarise the analysis with the answer to the following question:

What data do you have, that if that data disappeared, your company would also disappear?

Getting a secure copy of that data is critical, but you have to get it in such a way that it’s not visible to the backup application. Otherwise, if someone compromises access to your backup application, they have visibility over the vaulting process, which gives them a greater surface area to attack.

So, you’ve got a copy of the data? What good does that do you? Well, nothing if you don’t actually analyse the data. That’s where the Dell solution comes into its own. Data Domain retention lock and snapshots help secure the data, but they’re still only part of the solution.

Leveraging IndexEngines CyberSense software, the data is scanned inline on the Data Domain within the vault (i.e., without rehydration to a staging location), and comprehensively analysed to detect a large variety of threats. This requires no internet connectivity at all; it doesn’t depend on simple “virus signatures” like an anti-virus tool. What good would that be, after all? If anti-virus applications could solve eliminate the ransomware problem from ever happening, the world would be a happier place. No, CyberSense uses machine-learning information to perform deep analysis to look for patterns associated with cyber attacks.

And when it finds compromised data, it’s able to work with the Cyber Vault software to provide a report of the way the data has been compromised, and what the last known good copy of the data is.

So let’s look at a tape air-gap in comparison.

So here’s what a tape air-gap gives you. Once you’ve identified the data you want to air-gap, you clone it to tapes, and send it off-site. Lovely! When you have a Cyber attack, you then go through the following process (while potentially up to 100% of the business is offline):

• Recall the media with the good copies of the data.
• But hang-on, most Cyber attacks run over several days. In fact, the most wicked of Cyber attacks sometimes run for months, slowly preparing. So, you don’t know the media that has the good copies of the data.
• So you’d best recall the last month’s worth of media then.
• Now start recovering the data.
• Except, oops, that data you just recovered looks like it was already corrupted, so you’ll have to delete that recovery and start again.
• Except oops, when testing that recovered data, it turned out the cyber attack software was also recovered, subsequently activated, and caused new issues, requiring more work.

Data re-use is something I’m particularly passionate about: if you’re building an optimised data protection environment, finding ways to re-use the data allows you to derive greater value from your solution. (Ironically, data re-use is often an excellent justification to move away from tape.)

One of my university lecturers used to describe a colleague he’d worked with years prior, who always had to have a contingency plan for everything. To the point where it was clearly a psychological condition – he regaled us with the story once of how he’d been out for a bike-ride with his colleague down a winding country road when his colleague abruptly stopped and pulled out his disaster preparation journal because he’d just thought, “What if I’m riding along a road and a meteor strikes nearby. What do I do?”

Clearly you can go too far on disaster preparation. But, tape doesn’t really go far enough. Using tape for a cyber recovery solution is like waiting until your house is burning down before deciding to buy insurance.

The problem with using tape as an air-gap is simple: it’s stagnant data. Yes, it’s air-gapped, but it’s also so totally offline that you can’t analyse it to help yourself identify a cyber attack. If you could identify a cyber attack as it started with 100% certainty, that might be sufficient. But you can’t. And equally, because that data is stagnant, you have no way of knowing what data is a golden, safe copy to restore in the event of a cyber attack.

Cyber recovery isn’t disaster recovery. Having an extra copy with different physical access controls is merely the beginning of a cyber recovery strategy, and that’s why there is a world of difference between a vault and a bunker.

In fact, if you consider the key properties of true cyber recovery, a tape bunker is only going to give you one of the four foundations. Those foundations are:

• Isolate: Like a Data Domain air-gapped solution, tapes will give you an isolated copy of your data.
• Lock: Contrary to popular belief, tapes don’t really give you a locked data solution. The key reason for this is that they aren’t observable. If you remove a Data Domain from a vaulting solution, it’ll be noticed. If you remove a tape from a bunker, no-one will know unless you try to recall it. That’s not a lock.
• Validate: Here’s where tapes fail you entirely. You can’t use tapes for daily online, multi-streamed data analysis.
• Report: Because your bunker can’t validate data, you can’t report on its state or its usability.

Tape doesn’t provide a true cyber defence. It provides a data bunker only. When the proverbial stuff hits the spinning metal blades, you’ll be left with a few boxes of tapes and no idea what data on them can be used. (You may even not know what data is on them.) Your bunker could be entirely compromised and you wouldn’t even know it until you go to use it.