30 May Datacenter Outage Incident Report
10 June 2015
Linode Incident Report
We've received a report from Linode regarding the incident which took almost all of our websites offline for a few hours last month. The following are excepts:
- "At approximately 01:30 UTC, on May 30, 2015, the power utility (PG&E) experienced an outage affecting our Fremont datacenter. Seven of the facility’s eight generators started correctly and provided uninterrupted power. Unfortunately, one generator experienced an electromechanical failure and failed to start. This caused an outage which affected our entire deployment in Fremont."
- "Linode apologizes for this power interruption and any inconvenience it has caused you."
- "Our colocation provider is in the process of reevaluating their maintenance procedures."
To access the full report see the link below.
Chirp Incident Report
We remain satisfied with Linode as a mature and responsible hosting provider. As soon as power was restored to the data centre we were back online. Their last serious outage was at the same location in 2012 and we've not had problems at other nodes in London and Tokyo.
In case of a more prolonged outage we are equipped to restore websites from various backup locations and can re-delegate most/all of our domains even when the hosting servers are already down.
That option was not used this time because the information we received was that services would be restored at Fremont in 1-2 hours (which then slipped another 1-2 hours) and it would take that long for the DNS updates to propagate.
Clients who wish to maintain an online presence in case of a repeat of this incident should contact us to discuss options.