A last post from the NANOG list on the Tororonto Explosion, followed by some comments of my own.
The NANOG poster stated:
I had customers from St. Catherines all the way to well out past oshawa affected. A lot of voice connections outside the downtown core were hit-and-miss, whereby after trying about 10 or so times, I could make the call. For the most part, things seemed to stable out at about 16:00 EDT.
My Comments:
During the mid-Seventies I was assigned the on-the-fly task of being the AT&T Long Lines Restoration Coordinator, on the spot, during the worst fire disaster that has ever hit the telecommunications industry during peace time.
It was the 13th Street (Second Avenue) Tandem Office incident in Manhattan which served not only as a local end office for parts of lower Manhattan, but also the transiting office for most of AT&T's interstate-bound traffic from the Long Lines HQ Building (NY1, NY2, NY4, Network Radio and TV, and Overseas Control at 32 6th Avenue), as well as much of NY Telephone Downstate traffic routes.
Over 700,000 residential and business phone lines were knocked out for several months, to boot.
While I wouldn't want to take anything away from the horrific service disruption that occurred in the Hinsdale, Illinois incident in '88, the latter was a mere BarBQue compared to the blaze that took place at 13th Street, which raged for two days. It eventually claimed fourteen fire fighters, and untold physical losses to telco plant and services.
The firefighters, May They Rest In Peace, in case you are wondering, had been directly exposed to two full days of pcb's resulting from burning cables and other carcinogens during the lengthy siege, and for weeks beyond that point during clearing and demising procedures.
Today's story concerning the Toronto Central Office Explosion highlights a clear lack of self-healing capability and a lackluster level of survivability which exists in all COs today, despite all that's been written to the contrary with respect to self-healing SONET rings (which are only effective if breaks occur in outside plant while the COs are intact), DACS switching, automatic protection switching, etc.
When disaster strikes the CO, there is usually little recourse but to perform manual extrication measures from the particular situation, because the potential scope of what can go wrong (and usually does when the shit hits the fan) is just too broad to economically prepared against, much less be able to predict.
With the levels of densification now being facilitated through DWDM, and the ensuing higher levels of customer data that they will support, these risks and potential liabilities can only increase over time in terms that are truly logarithmic, unless a more intelligent means is derived to distribute said risk. I have been following this sort of thing for quite some time, now, perhaps due to the personal level of involvement and near trauma that I experienced twenty some odd years ago.
More recently during the Seaport Fire at a Consolidated Edison substation that took out a good part of the downtown Manhattan Wall Street Area during the Early Nineties, my consultancy was commissioned by a Japanese consortium, whose brokerage and insurance members were hit extremely hard by the incident.
After receiving powers or agency allowing us to act on the consortium's behalf before the NY State PSC and in contacts with various utilities, we came to realize that a general lack of preparedness is not something that is unique to the Telecommunications industry. There's some scary stuff going on out there, folks. And with the increased dependence on ever increasing amounts of information and processing services, it gets scarier all the time.
One way that has been moderately successful in combating severe utility outages is for the industry to mobilize mutual assistance programs, such as the NY City MAP, which was moderately successful in restoring services during the World Trade Center bombing earlier this decade. Power grid alliances are also helpful, but neither of these do any good to the local residents or businesses who are fed power or telecomms directly from end offices of either utility type when they are neutralized by disasters.
Comments welcome, as always.
Regards, Frank Coluccio |
