What a day to make a person feel alive! We woke up to the insistent alerts that the power was down in a datacenter. What an adrenaline rush… quickly followed by a rush to said datacenter.
It is always the seatbelt that strangles you. In this case, it was the backup power generator that caused a cascading GFI incident all the way back to Silicon Valley Power.
Today we learned why our Ops team gave us all keychain flashlights—during a power outage, it’s dark in the dungeons (except for the sparks from the parts that are causing the darkness).
To our relief, when the power was back, only our raid storage backups had been corrupted. The only stuff we lost was the stuff we had backed up. As for our crawlfiles—where the Web is our backup—we didn’t lose any of that.
The theory for the datacenter outage is pasted below in its entirety…
The currently postulated scenario for what happened: a surge from SVP caused a surge arrestor to do its job and absorb the energy surge. However in the process of doing so it ruptured and spewed a cloud of thick black smoke onto a high voltage power bus. The smoke and debris was enough to allow the high voltage power to form an arc to a grounded metal cabinet, which would have been spectacular to see if anyone had been watching. The large current flowing into the ground circuit caused numerous safety cutoffs to trip, much like when you press the legendary Big Red Button (emergency power off), because abnormal ground current is ordinarily a potentially lethal situation. The equipment decides under the circumstances that it’s safer to kill power than operators. —Eric
Never a dull day!
