Jack, re backup snafus, here's a case in point which popped up on my NANOG Radar. It happened only yesterday:
everquest.station.sony.com
Since this message will very likely scroll, soon, I'll post it below.
Regards, Frank Coluccio
====
Snip Begins:
Network Status Sat Nov 27 16:15:57 EST 1999
Hi folks,
We sincerely apologize for the downtime today. Here's what happened:
1) At 6:15am PST San Diego Gas & Electric shut the power off in our building. This was a planned outage, except they weren't supposed to turn the power off in our Building. It came as a great surprise for a lot of people in our building.
2) Everything switched as it should to backup power, but it caused many network problems because only major things get backup juice, and some of the minor things became problems. The cutover to the generator isn't as quick as anyone would have hoped (it's supposed to be immediate),
3) As of right now everything is up and running on a backup Gasoline generator (you would have to see this thing to believe it). We are prepared for emergencies, but a few things we weren't ready for bit us where it hurt.
Strangely enough, the biggest problem we had was that our Security system which is supposed to have 12 hours worth of backup battery failed and we were stuck outside of our own building (we went outside temporarily to make sure the Gas Generator was ok). We just got back in (don't ask) and everything is ok now with the exception of Brell which is down for emergency hardware maintenece.
I realize this sounds pretty bizzare, and let me tell you it was.
On the bright side, even with power cut off to our building (actually the EQ servers are located in 2 different facilities both of which have backup juice) everything is back up now.
Again, we apologize
Sincerely.
<delete>
------End Snip
|
