More: AT&T blames switching equipment for outage
Mercury News - Posted at 11:09 p.m. PDT Tuesday, April 14, 1998
AT&T Corp. on Tuesday blamed problems with two pieces of
switching equipment -- made by San Jose-based Cisco Systems Inc. --
for the computer network problem that crippled automated teller
machines and credit card systems nationwide for almost a day.
Yet even after service was restored Tuesday afternoon, officials at
both companies said they weren't sure what the root cause of the
outage was.
AT&T is the nation's largest supplier of data network services to
companies. The collapse of the company's ''frame-relay'' network
affected only a portion of its customers -- and had no impact on
conventional or cellular phone service. Still, it was the worst such
failure ever, industry experts agreed.
''This sort of thing is going to happen infrequently, but more and more
in the future,'' said Howard Anderson, managing director of the Yankee
Group, a technology research firm. ''And it makes you realize how vital
to the lifeblood of the economy these complex computer networks have
become.''
Anderson had firsthand experience with the problem. He said he had
just emerged from a Tower Records store in San Jose where there had
been a long line of frustrated customers whose credit card transactions
could not be processed. ''I had cash, so it was great for me,'' Anderson
said. ''I went straight to the front of the line.''
Throughout California, probably a thousand of AT&T's business
customers were affected, a company spokeswoman estimated. Among
them was Wells Fargo, which connected about half of its branches and
automatic teller machines to the company's central computers in
Sacramento through AT&T's frame-relay network.
The branches stayed open after the outage but the bank machines shut
down, prompting Wells Fargo to scramble for another way to connect
the automatic tellers to the company's computers, said spokesman
Mark Marymee. Working with MCI Communications Corp. and 3Com
Corp. and the Roseville Telephone Company, a local phone company
near Sacramento, Wells Fargo had almost all of its connections
re-established by the time branches opened Tuesday, Marymee said.
Elsewhere, the impact of the network breakdown was uneven. Some
businesses and institutions noticed slowdowns. These included
Northwest Airlines, in its processing of reservations, and the American
Red Cross, in the tracking of its blood supply.
Many businesses suffered financial losses as a result, though only a
few offered estimated Tuesday. Tele-Communications Inc., the cable
television giant, pegged its losses at as much as $5 million -- resulting
from the inability of TCI representatives who field requests for
pay-per-view services, installation and repairs at 500 offices to log in to
the company's central data base to get access to billing information or
to process orders.
The breakdown was something of a black eye for Cisco, which has
been trying to sign up more phone companies as customers, breaking
into the markets of Lucent Technologies Inc. and Northern Telecom
Ltd.
''We view this interruption as unacceptable, and apologize to our joint
customers who have been affected,'' John Chambers, president and
CEO of Cisco, said in a press release Tuesday. ''The Cisco team,
together with AT&T, will continue to focus on this matter until we are
convinced that all problems associated with this outage have been
identified and resolved.''
In the aftermath of the outage, Cisco rushed two new pieces of its
equipment to AT&T's switching offices in Albany, N.Y., and
Cambridge, Mass., via a chartered jet, said Darryl Miller of FOB
America, the company that delivered the equipment.
The AT&T network that went down is called a frame relay network
because of the the method it uses to handle data at high speeds. The
data packets are sent almost instantaneously over fiber optic
telecommunications lines and complex switches across the country.
AT&T's frame relay network has 145 switching hubs, or nodes.
Company executives said on Tuesday that the exact cause of the
breakdown had not yet been determined. But one executive said that
the problem began with a data transmission between the Albany and
Cambridge hubs. The problem cascaded uncontrollably to the other
hubs in the network, for some as-yet undetermined reason.
Frank Ianna, AT&T's vice president for network and computer
services, said it would probably take two or three days to pinpoint the
precise cause of the breakdown. He said the cause was most likely
''some combination of hardware and software'' failure.
o~~~ O |
