More on Toronto meltdown--email backs up phone net--
**multiple sources**
Again, email became the backup for phone networks. :-)
Mr. Gutteridge, for example, had to
co-ordinate the city's emergency fire and
ambulance services from the radio room at
Fire Hall No. 1 on Adelaide Street when both
his office line and cellphone failed. When the
backup system conked out at Bell Canada,
ambulance services managed to notify Mr.
Gutteridge about the magnitude of the
problem by E-mail.
Phones go dead, Toronto
put on hold
Fire in Bell Canada switching station
knocks out more than 100,000 lines
PETER CHENEY
The Globe and Mail; With files from Krista
Foss, Dave Leeder, Oliver Bertin and Paul
Knox
Saturday, July 17, 1999
Toronto -- An accidentally dropped tool was
the beginning of a chain-reaction disaster that
led to a communications meltdown for
Canada's biggest city yesterday.
For Toronto, it was a day when the phones
didn't ring, credit cards didn't work, and
countless plans went out the window, from
ordering an airline ticket to closing a real
estate deal.
The breakdown, which lasted for most of the
business day, had repercussions across the
country. Credit-card transactions as far away
as Vancouver couldn't be processed, and
hundreds of bank machines went out of
service.
Toronto found itself having what amounted to
an electronic nervous breakdown. Some
brokerages couldn't process trades.
Customers found themselves unable to call an
ambulance, order a taxi, or pay with plastic.
At the Art Gallery of Ontario, extra guards
were required for Old Masters paintings after
security lines went down. Traffic-light
sequencing was knocked out. Travel agents
couldn't book flights -- or take calls from
customers.
And if you were hoping to get rich quick,
there was bad news: Ontario Lottery Corp.
terminals were out of service, making it
impossible to buy last-minute tickets for last
night's unusually high Super 7 draw.
"People are passing up $10-million," said
H.W. Chan, manager of the Sun Wa Book
Store on Spadina Avenue.
The cause of the problems was a telephone
system breakdown that began with an
early-morning fire at a Bell Canada switching
centre on Simcoe Street downtown. The fire
reportedly began after a repairman dropped a
tool.
The tool landed on electrical equipment and
the fire spread quickly. At its peak, more than
70 firefighters were on the scene.
What followed was a series of failures that
revealed the fragility of the complex
communication systems society takes for
granted. Although backup batteries were in
place to power the switching system, they
were designed to last only a few hours.
The backup plan called for the use of diesel
emergency generators after the batteries
failed, but officials decided that wasn't safe
because of the water left by emergency
sprinklers.
When the batteries began failing, at around
10.30 a.m., service to approximately 113,000
Bell phone lines was wiped out. Most of
those lines were in Toronto's downtown core,
the most communications-intensive patch in
Canada.
The breakdown left Barry Gutteridge,
Toronto's commissioner of works and
emergency services, shaken about the city's
vulnerability. Mr. Gutteridge said there will be
an investigation into the accident, with a view
to reducing the city's exposure in the future.
Other than a repairman injured at the site of
the explosion, Mr. Gutteridge said, the
disruption injured no one. Instead, it caused a
series of potential crises that were averted
only through luck and improvisation.
Mr. Gutteridge, for example, had to
co-ordinate the city's emergency fire and
ambulance services from the radio room at
Fire Hall No. 1 on Adelaide Street when both
his office line and cellphone failed. When the
backup system conked out at Bell Canada,
ambulance services managed to notify Mr.
Gutteridge about the magnitude of the
problem by E-mail.
Despite the frustrating communications
problems, Mr. Gutteridge said, fire and
ambulance services responded efficiently. The
host of complications included the failure of all
telephone lines to the Hospital for Sick
Children. A mobile radio unit was sent to the
hospital to handle emergency calls.
Bell Canada spokesman Don Hogarth said
the 911 emergency service was maintained,
although its capacity to handle calls was
impaired. Mr. Gutteridge said that if 911 had
gone down, mobile radio units would have
been sent to affected areas to give members
of the public a way of calling in emergencies.
Mr. Hogarth said the area most affected was
between College Street to the north, Queen
Street to the south, Bathurst Street to the
west and Bay Street to the east. The
shutdown "zigzagged" to areas outside that
core zone as well, he said, depending which
phone lines they relied on.
Several investigations are being conducted,
including a Labour Ministry investigation into
the industrial accident, the fire marshal's
investigation, and Bell Canada's investigation,
in which Mr. Gutteridge said the city will be
involved.
The effects of the outage were widespread,
ranging from the institutional to the personal.
Nancy Tarek of Oakville sat in the lobby of
Toronto General Hospital for several hours
yesterday, pumped full of painkillers, Valium
and other sedatives after a medical
procedure.
Because of her condition, Ms. Tarek wasn't
allowed to go home by herself. But because
the phones were out, she couldn't call her
family to come pick her up.
"I'm just sitting here half-medicated," she said.
Some Torontonians found virtually all their
communication options cut off: Telephones,
fax machines, pagers and cellphones routed
through the Adelaide Street Bell switching
system were all knocked out.
Many securities dealers had problems
communicating trades and relied on
cellphones until phone lines were back up, but
the Toronto Stock Exchange kept operating.
At University Avenue Funds, mutual-fund
sales people who couldn't make calls simply
went home.
A skeleton staff remained, processing
transactions made before the phones crashed.
"I'm trying to fax over trades to the bank and
they won't go," accountant Shelina Dossa
said.
Almost one-tenth of the cash machines
operated by the country's six big banks were
out of service for parts of the day, the
Canadian Bankers Association said. The
Toronto-Dominion Bank was hardest hit.
Hundreds of bank branches lost access to
their systems. Many simply shut their doors
and referred customers to other areas where
phone lines were still working.
The electronic failure created a short-lived
bonanza for couriers, who suddenly found
themselves in high demand. At the Printing
House copy centre on University Avenue,
manager Chris Gennings said the cost of a
courier had been driven up by the briefly
altered market conditions.
"You go out on the street and offer them $10
and they say the going rate's $20," he said.
"And if you argue, suddenly the going rate's
$25."
The breakdown created a nightmare for
retailers, who were unable to authorize debit
or credit card transactions. Some, including
Loblaws, accommodated customers -- and
created a bankers' nightmare -- by taking
customers' debit card numbers and phone
numbers so banks could call them back to
confirm the transaction.
Some businesses decided to do credit card
transactions even though they couldn't get
them approved.
"I hope and pray a lot of trustworthy people
are shopping today," one retail manager said.
Hospitals and other medical services were
seriously affected. Phone service was out at
the Hospital for Sick Children, Mount Sinai
Hospital, Toronto Western Hospital and
Toronto General Hospital. Hospitals were
also affected by the failure of pagers, which
they use to track down specialists, surgeons
and doctors on call.
Sick Children's poison-information and
medical-information lines were shut down.
Those two lines usually receive nearly 400
calls a day from all over the city and province.
The failure created chaos for many law
offices, which found they were unable to close
real-estate deals because the main
clearinghouse for title searches was
unavailable, putting millions of dollars worth of
potential transactions in jeopardy.
Travel agents were particularly hard hit. David
Gallie, manager of the Flight Centre on Queen
Street West, said the day was a wipeout for
his business.
"I've lost $30,000 worth of sales," he said.
"Clients can't order tickets. I can't call the
airlines, and I can't book a seat. And I can't
sell a ticket because the credit-card
authorization system is down."
For Toronto police, the failure meant the loss
of phone and computer systems, although
their radios and most cellphones still worked.
Constable Don Petrie, who works in the
Eaton Centre, said the phone failure gave
police "a bit of a taste" of what could happen
if the millennium bug wipes out computer
systems on January 1.
"It's a bucket of cold water," he said. "It
shakes you back to what it was like when we
didn't have these services." |
