Myspace Data Loss – Some Questions

While it’s been murmuring in the background, and something affected users have been well aware of for a while, over the past few days the story has well and truly broken that Myspace, had lost during a server migration seemingly all files uploaded to it before 2016. Of the issue, TechCrunch stated:

It’s not as if the internet needed another cautionary tale about backing up data, but for many artists, this news is heartbreaking nonetheless. Myspace has issued a tersely worded message noting that a huge amount of user-uploaded music has been lost during a server migration.

The once-dominant social network posted a note on its site reading, “As a result of a server migration project, any photos, videos, and audio files you uploaded more than three years ago may no longer be available on or from Myspace. We apologize for the inconvenience.”

Myspace may have lost more than a decade’s worth of user music, Brian Heater, 18 March 2019, TechCrunch.

TechCrunch is right – it’s not as if we needed another cautionary tale about backing up data, but we certainly have one here. The various stories posted about the MySpace data loss all share a common timeline of events: somewhere, over a year ago, users started noticing that links they clicked on for older music files, etc., stopped working. Myspace initially assured users that the issues would be fixed, before eventually releasing the statement quoted by TechCrunch.

Now, ‘server migration’ can mean quite a lot, so rather than trying to do forensics from afar, I’m left after reading the various descriptions wanting to know the answers to a few questions.

Before I start, it is worth looping back to TechCrunch’s comment about it being a cautionary tale to backup data: and it should be a cautionary tale for the end user. After all, buried in the Myspace terms of service is the standard sort of all-caps/shouty liability waiver you’d expect to see just about anywhere, which says Myspace is never, ever liable in any situation that leads to, amongst other things:

ANY OTHER TECHNICAL OR OTHER MALFUNCTION, INCLUDING LOSSES OR DAMAGES IN THE FORM OF LOST PROFITS, LOSS OF GOODWILL, LOSS OF DATA, WORK STOPPAGE, ACCURACY OF RESULTS, OR EQUIPMENT FAILURE OR MALFUNCTION, EVEN IF MYSPACE HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Myspace Terms of Service, 15.1

Was there a backup?

The first obvious question when reading the MySpace terms of service of course is: was there actually a backup of the content that was lost? By backup, I literally mean an honest-to-goodness backup, where the affected files and data were being copied from the storage platform to another platform on a regular basis? Their terms of service make it abundantly clear that they’re not liable if there’s a loss of data, so maybe they just didn’t bother to actually backup that data at all?

The reason for asking of course is that presumably, had there been an actual backup, the backup could have been restored from. If there was a backup, that leads to some other questions.

Or was the backup compromised by the migration?

The next question that springs to mind is – if there was a backup, was it somehow compromised by the migration? Was there shared storage between the two platforms, or some service dependency that resulted in backups being lost when the data was lost? (There’s a few ways I can envisage this.)

Or was the cost of recovery deemed too high?

What if this was some sort of cold storage backed backup – tape, or something like Amazon Glacier. Did Myspace have a backup, but review the cost of performing a recovery (in time, or money, or both), and decide it would take too long, or be too costly?

Or did the backup miss the files/data that was supposed to be backed up?

Was it a configuration issue or other human error that resulted in the assumption that the data was being backed up, but it wasn’t? If so, who was meant to be in charge of data protection and what can be done to improve those checking processes?

Or do Myspace only keep short term/operational recovery copies of data?

The other scenario is that this data really was protected, but only for a short period of time. Only older data was lost. So maybe Myspace only keep backups for the last 30 days, for instance, and have no long term retention?

What sort of server migration was it?

There’s a lot of different meanings to “server migration”. If you’re a really small business, it can mean something as basic as buying a new server, installing it, then manually copying the data from the old server to the new server. But Myspace was once huge, so we can presume there was more effort involved than that.

Was it a server migration only, or a server and storage migration?

Was Myspace moving the hosting services (e.g., Web server application cluster) from one set of servers to another but the actual file content remained in the same location? If so, how does one lose the storage in this situation? Was it somehow a loss of connectivity to the storage? (E.g., stubbed archive files that for some reason or another can’t have the originals accessed if the stubs are lost? Which takes us back to the backup questions.)

Was it a full service migration?

Like, the sort of full service migration we see when moving from one cloud provider to another, or from a datacenter to the cloud, or the cloud to a datacenter.

…because there’s a fundamental question about planning and execution here…

Regardless of whether it’s a application service migration only, or a combined server and data migration, the lingering question is not a technical one, but a process one – what actual failure was there in process checks and/or planning that saw the data irrevocably lost?

After all, when performing a migration we’d normally expect to see a certain rigor applied, first in the planning stage to identify as many potential obstacles as possible, have a view of timeframes, understand the rollback process, and so on. Then secondly, during the migration phase itself, monitoring it and verifying that it’s going successfully, and rolling it back if there’s an issue.

If you’re doing a data migration, you’d expect there to be some sort of logging of the migration process, after all. I can imagine in a worst case scenario if you’re a smaller business you might literally have someone tasked with reviewing the log files at the end of the migration process and verify there were no errors. At that point, maybe (and it’s still a maybe) they might miss an error about a single file or chunk of data not copying. But by all accounts this is the equivalent of 50,000,000 MP3 files. If we assume an average of 6MB per file (and it’s higher these days with higher encoding rates, but these files are from an older time), that’s somewhere in the order of 300TB of data that went *kapow*.

300TB isn’t big in the grand scheme of things, but it’s big enough that it should have been noticed, right? So what happened in the planning and execution that resulted in that not being noticed in time for it to be recovered from?

(Or was it the equivalent of someone issuing a move of the data then accidentally deleting the destination at the end, thinking they’d done a copy?)

What are the lessons for the users?

I’m not anticipating getting answers to the above questions. I’m not someone who holds shares in Myspace, I’m not a director of any company related to them: I’m just someone who has made a career almost solely focused on data protection and has written books on it, in addition to blogging about it for 10+ years.

So instead we focus on the things we can answer, without any forensic analysis of the events: what are the lessons for the users – and for that matter, any users of any social media platform?

1. Social media isn’t a backup.
2. When you upload something to social media, make sure you keep your own copy of it.
3. Make sure you protect (i.e., backup) your own copy of what you place on social media.
4. All the social media companies have the same sorts of terms and services: they’re not responsible for losing your data.

For many, hearing this now will be like shutting the barn door after the horse has bolted, but if there’s anything I’ve learnt in data protection, there’s always a next time. If you we can pivot the data loss situation into a prime example prompting social media users to keep in mind that the data remains something they need to actively protect local copies over, at least something useful will come out of this situation.