On January 31, GitLab suffered a significant issue resulting in a data loss situation. In their own words, the replica of their production database was deleted, the production database was then accidentally deleted, then it turned out their backups hadn’t run. They got systems back with snapshots, but not without permanently losing some data. This in itself is an excellent example of the need for multiple data protection strategies; your data protection should not represent a single point of failure within the business, so having layered approaches to achieve a variety of retention times, RPOs, RTOs and the potential for cascading failures is always critical.
To their credit, they’ve published a comprehensive postmortem of the issue and Root Cause Analysis (RCA) of the entire issue (here), and must be applauded for being so open with everything that went wrong – as well as the steps they’re taking to avoid it happening again.
But I do think some of the statements in the postmortem and RCA require a little more analysis, as they’re indicative of some of the challenges that take place in data protection.
I’m not going to speak to the scenario that led to the production, rather than replica database, being deleted. This falls into the category of “ooh crap” system administration mistakes that sadly, many of us will make in our careers. As the saying goes: accidents happen. (I have literally been in the situation of accidentally deleting a production database rather than its replica, and I can well and truly sympathise with any system or application administrator making that mistake.)
Within GitLab’s RCA under “Problem 2: restoring GitLab.com took over 18 hours”, several statements were made that irk me as a long-term data protection specialist:
Why could we not use the standard backup procedure? – The standard backup procedure uses pg_dump to perform a logical backup of the database. This procedure failed silently because it was using PostgreSQL 9.2, while GitLab.com runs on PostgreSQL 9.6.
As evidenced by a later statement (see the next RCA statement below), the procedure did not fail silently; instead, GitLab chose to filter the output of the backup process in a way that they did not monitor. There is, quite simply, a significant difference between fail silently and silently ignored results. The latter is a far more accurate statement than the former. A command that fails silently is one that exits with no error condition or alert. Instead:
Why did the backup procedure fail silently? – Notifications were sent upon failure, but because of the Emails being rejected there was no indication of failure. The sender was an automated process with no other means to report any errors.
The pg_dump command didn’t fail silently, as previously asserted. It generated output which was silently ignored due to a system configuration error. Yes, a system failed to accept the emails, and a system therefore failed to send the emails, but at the end of the day, a human failed to see or otherwise check as to why the backup reports were not being received. This is actually a critical reason why we need zero error policies – in data protection, no error should be allowed to continue without investigation and rectification, and a change in or lack of reporting or monitoring data for data protection activities must be treated as an error for investigation.
Why were Azure disk snapshots not enabled? – We assumed our other backup procedures were sufficient. Furthermore, restoring these snapshots can take days.
Simple lesson: If you’re going to assume something in data protection, assume it’s not working, not that it is.
Why was the backup procedure not tested on a regular basis? – Because there was no ownership, as a result nobody was responsible for testing the procedure.
There are two sections of the answer that should serve as a dire warning: “there was no ownership”, “nobody was responsible”. This is a mistake many businesses make, but I don’t for a second believe there was no ownership. Instead, there was a failure to understand ownership. Looking at the “Team | GitLab” page, I see:
- Dmitriy Zaporozhets, “Co-founder, Chief Technical Officer (CTO)”
- From a technical perspective the buck stops with the CTO. The CTO does own the data protection status for the business from an IT perspective.
- Sid Sijbrandij, “Co-founder, Chief Executive Officer (CEO)”
- From a business perspective, the buck stops with the CEO. The CEO does own the data protection status for the business from an operational perspective, and from having the CTO reporting directly up.
- Bruce Armstrong and Villi Iltchev, “Board of Directors”
- The Board of Directors is responsible for ensuring the business is running legally, safely and financially securely. They indirectly own all procedures and processes within the business.
- Stan Hu, “VP of Engineering”
- Vice-President of Engineering, reporting to the CEO. If the CTO sets the technical direction of the company, an engineering or infrastructure leader is responsible for making sure the company’s IT works correctly. That includes data protection functions.
- Pablo Carranza, “Production Lead”
- Reporting to the Infrastructure Director (a position currently open). Data protection is a production function.
- Infrastructure Director:
- Currently assigned to Sid (see above), as an open position, the infrastructure director is another link in the chain of responsibility and ownership for data protection functions.
I’m not calling these people out to shame them, or rub salt into their wounds – mistakes happen. But I am suggesting GitLab has abnegated its collective responsibility by simply suggesting “there was no ownership”, when in fact, as evidenced by their “Team” page, there was. In fact, there was plenty of ownership, but it was clearly not appropriately understood along the technical lines of the business, and indeed right up into the senior operational lines of the business.
You don’t get to say that no-one owned the data protection functions. Only that no-one understood they owned the data protection functions. One day we might stop having these discussions. But clearly not today.
Preston,
Very well said. From a technical ownership perspective, I’m not sure what is more concerning: that they didn’t understand the tools, methods, and techniques available to them, or that according to the official story, they don’t understand that they didn’t understand those elements.
The failure of email alerts always worries me. When creating email alerts, many people set them up but don’t even do simple delivery tests.
Negative control: Change the database name to something bogus for a moment and make sure you get an alert.
Positive control: send an email that says ‘test alert’ and only alarm if it’s NOT received in a certain interval.
At the very least, run a negative control. The positive control is suggested, but sometimes difficult to implement in various systems without too many false positives, which defeat its purpose.
Rob,
Great points. There is an art to automatically monitoring something, and that always includes detecting a lack of detection, and confirming up-front and at regular intervals that normal conditions are still being detected, and also that abnormal states still trigger errors.
Maybe the problem was that too many people had a responsibility. There’s an old saying “when it’s everybody’s job it’s nobody’s job”. And the solution to missing alerts is to have the automatic process email the result of all critical operations, success as well as failure, then have someone check all those emails. Every . single . day.
It’s true that if multiple people own a responsibility it dilutes responsibility. That being said my point was more that in a chain of staff, multiple people should have an understanding and an appreciation of the data protection processes, and for GitLab to say that no-one “owned” the backup process is not correct. Functionally people did own the backup process, but did not understand their responsibility.
Maybe the problem was that too many people had a responsibility. There’s an old saying “when it’s everybody’s job it’s nobody’s job”. And the solution to missing alerts is to have the automatic process email the result of all critical operations, success as well as failure, then have someone check all those emails. Every . single . day.