4/15/98 NYT article. AT&T Computer Snag Renders Credit Cards Useless for a Day
[COMS rescues WFC. More details about CSCO failure]
By STEVE LOHR
EW YORK -- From Monday afternoon until Tuesday afternoon,
companies and consumers across the United States were given
an unsettling lesson in just how dependent on vast computer networks
the commerce of everyday life has become.
A few million people found that their credit cards were useless and
automated teller machines at banks were dead because a national
high-speed data network of the AT&T Corp. had crashed. The
network breakdown did not affect the conventional or cellular
telephone service of customers of AT&T, the nation's largest
telecommunications company. Nor did it interrupt Internet access. But
the flow of data for transactions involving credit cards, bank accounts,
travel reservations, and the like were seriously disrupted.
AT&T is the nation's largest supplier of data network services to
companies. The data network was restored to nearly normal operation
by Tuesday afternoon, but its collapse was the worst such failure ever,
industry experts agreed. They said it was the data network equivalent of
the breakdown of AT&T's long-distance switching system in January
1990, when about half of the nation's long-distance calls ended in busy
signals or recorded messages for nine hours.
"This sort of thing is going to happen infrequently, but more and more in
the future," said Howard Anderson, managing director of the Yankee
Group, a technology research firm. "And it makes you realize how vital
to the lifeblood of the economy these complex computer networks have
become."
Anderson had firsthand experience with the problem. In a telephone
interview yesterday afternoon, he said he had just emerged from a
Tower Records store in San Jose, Calif., where there had been a long
line of frustrated customers, whose credit card transactions could not
be processed. "I had cash, so it was great for me," Anderson said. "I
went straight to the front of the line."
The impact of the network breakdown was uneven. Some businesses
and institutions noticed slowdowns. These included Northwest Airlines,
in its processing of reservations, and the American Red Cross, in the
tracking of its blood supply. At the Wells Fargo Bank, based in San
Francisco, the automated teller machines at 1,023 branches in
California went dead for several hours, but another 520 branches in the
state were unaffected.
By late Tuesday afternoon, Wells Fargo had enlisted the help of MCI
Communications, Roseville Telephone, and 3Com to hastily patch
together a backup network.
"We've never had a failure like this before, but we were able to restore
service in about seven hours," said Sherry Nash, Wells Fargo's senior
vice president in charge of data networks.
There is no way to estimate just how much business was lost
nationwide because of the data network breakdown. But the losses are
likely to be big, since thousands of companies were affected and many
of those do business in hundreds or thousands of locations.
Tele-Communications Inc., a leading cable-television system based in
Denver, was one of the few companies that had estimated its losses by
last night. Rex Mammel, a network administrator, calculated them at as
much as $5 million. The losses, he said resulted from the inability of TCI
representatives who field requests for pay-per-view services,
installation and repairs at 500 offices to log in to the company's central
data base to get access to billing information or to process orders.
The AT&T network that went down is known as a frame relay network
-- named for the method it uses to handle data at high speeds. The data
packets are sent almost instantaneously over fiber optic
telecommunications lines and complex switches across the country.
AT&T's frame relay network has 145 switching hubs, or nodes.
Company executives said on Tuesday that the exact cause of the
breakdown had not yet been determined. But one executive said that
the problem began with a data transmission between two hubs on the
network -- one in Albany, N.Y., and another in Cambridge, Mass. The
problem cascaded uncontrollably to the other hubs in the network, for
some as-yet undetermined reason.
Frank Ianna, AT&T's vice president for network and computer
services, said it would probably take two or three days to pinpoint the
precise cause of the breakdown. He said the cause was most likely
"some combination of hardware and software" failure.
In a conference call on Tuesday, C. Michael Armstrong, the chairman
of AT&T, told reporters: "To say the least, this has been a very difficult
20 hours for our customers. Frankly, these outages have let our
customers down, and I want to apologize to each and every one of
them."
Until the cause of the network breakdown is found and solved,
Armstrong said, AT&T frame relay customers will not be charged. For
AT&T, frame relay services are a $1-billion-a-year business, growing
at more than 30 percent annually.
Indeed, throughout the telecommunications industry, data transmission
services -- far more than voice services -- are the high-growth business
for the future. AT&T's rivals, including MCI, Worldcom and Sprint, are
all competing aggressively in the data network field.
And as these networks grow and handle more traffic, industry experts
predict, all the suppliers will inevitably experience breakdowns.
Home | Sections | Contents | Search | Forums | Help
Copyright 1998 The New York Times Company |
