{"id":10168,"date":"2021-03-22T10:36:16","date_gmt":"2021-03-22T00:36:16","guid":{"rendered":"https:\/\/nsrd.info\/blog\/?p=10168"},"modified":"2021-03-22T10:36:18","modified_gmt":"2021-03-22T00:36:18","slug":"of-cascading-failures","status":"publish","type":"post","link":"https:\/\/nsrd.info\/blog\/2021\/03\/22\/of-cascading-failures\/","title":{"rendered":"Of Cascading Failures"},"content":{"rendered":"\n

I got a real-world reminder in cascading failures this weekend, and that’s as good a time as any for me to pull up a chair, and chat about why we backup.<\/p>\n\n\n\n

Cascading Failures Averted<\/h2>\n\n\n\n

I have a few RAID systems in my environment at home. The latest is an OWC 8-bay Thunderbolt-3 enclosure attached to my Mac Mini. When I retired my 2013 Mac Pro last year, I kept it around to run vSphere within VMware Fusion, and attached to it is a 4-drive Thunderbolt-2 Promise system.<\/p>\n\n\n\n

In addition to those direct-attach units, I’ve got three Synology systems as well. A DS1513+ (5-drive) with an extension pack, a DS414 (4-drive) and a DS414j (4-drive). The DS1513+ is our “home storage” platform, and that’s where I likely averted<\/em> a cascading failure late last year.<\/p>\n\n\n\n

That system had been set up with 5 x 3TB drives initially, and around November last year, I realised the drive stats were showing that those drives had managed to log over 61,000 hours of operation \u2013\u00a0pretty much 7 years of run time. That made me a wee bit nervous, so I made use of the Synology Hybrid RAID function to not only replace the drives but also expand the home-share capacity using 6TB drives.<\/p>\n\n\n\n

What I didn’t think to do was to check the drives in the DS414. It turned out, they’d been running for almost as long.<\/p>\n\n\n\n

Cascading Failures in Play<\/h2>\n\n\n\n

So on Friday, I got an email from my DS414 warning that the number of bad sectors being remapped on one of its drives had increased quite a bit. So, I logged on and triggered a deep S.M.A.R.T. test, which a few hours prompted an alert that the drive was failing and should be replaced ASAP.<\/p>\n\n\n\n

Of course, all 4 drives in the Synology had been purchased at the same time, were the same age, and as you’d expect, had similar run hours. In fact, like the DS1513+ when I replaced its drives, these were all now coming up to the 61,000 hour mark.<\/p>\n\n\n\n

So with some trepidation, I pulled the failing drive, added a spare 3TB drive, and let it start rebuilding the RAID unit.<\/p>\n\n\n\n

Then Saturday morning, I woke to news of multiple IO failures during the rebuild. Thankfully, they weren’t total drive failures, but two different drives had suffered IO timeouts that eventually rendered the data filesystem corrupted during the rebuild. (Thankfully, the OS partitions had been spared \u2013 so I was at least able to get an email about the news.) I was offered the opportunity to reboot and run fsck \u2013 and 7 hours later from that, I was told the filesystem was unrecoverable.<\/p>\n\n\n\n

So it was time to pull out my wallet and fork out some $$$ for four new drives to replace what was rapidly turning into a house of horrors within the DS414.<\/p>\n\n\n\n

Lucky Failure<\/h2>\n\n\n\n

All things considered, I’m counting myself lucky with the failure. The DS414 was a backup storage volume. While I lost some backups, they weren’t long-term retention backups, so the data loss could be dealt with (in the scheme of home storage failures) by running new backups. (My long-term retention\/critical backups get cloud as well as local protection.)<\/p>\n\n\n\n

I can summarise a few quick notes from the overall process:<\/p>\n\n\n\n