Dropped circuits, session timeouts hit Qwest frame relay net
By David Rohde
Network World, 02/04/00
Qwest has confirmed that its frame relay and ATM users suffered
numerous dropped circuits and other disruptions over an 11-day
period ending last weekend.
The problem was caused by a combination of an apparently
overloaded transport circuit in Qwest's intercity network and
processing speeds on Qwest's Lucent frame/ATM switches that were
too slow for the amount of user traffic coming in.
As a result, between Jan. 18 and 29, some users saw individual
permanent virtual circuits (PVC) go down for up to several hours at a
time when rerouting routines failed to kick in quickly enough. Others
saw time-sensitive applications, such as SNA traffic, time out even
though their PVCs were technically still up and running.
To fix the problem, Qwest confirms that it literally forklifted out its
Lucent frame/ATM switch in Los Angeles on Sunday, Jan. 22, and
replaced it with a newer model. It then changed out the processing
cards on the rest of its switches, and also lit three new OC-12
segments for use by frame and ATM customers.
No users have reported to Network World that they saw their entire
WANs go down. And it appears many Qwest customers got by
totally unscathed.
But the incident comes on the heels of complaints that Qwest has
fallen behind on installation intervals and customer service as its sales
force generates loads of new customer traffic. Some users say
Qwest's response again followed this pattern.
One user based in the Northeast who asked not to be identified says
that at one point he kept losing connectivity with three of his frame
relay sites for "five to 20 minutes out of every hour." Several days
into the incident, the user says, Qwest reps told him to stop reporting
trouble because the company had opened a master ticket for all the
complaints they were receiving. "They told us not to call anymore
when it happened, because there was nothing they could do for us,"
he says.
The incident began Jan. 18 when Qwest engineers isolated several
reported failures of frame PVCs to an OC-12 link between Atlanta
and Fort Worth, Qwest spokesman Tyler Gronbach says. Because
the traffic was not being rerouted properly, the carrier added the three
additional OC-12 links, but then it noticed an unusual number of
errors coming out the Los Angeles switch, again relating to attempts
to move traffic to alternate routes. It was not immediately clear
whether the problem in that switch was triggered by the network
slowdown in the East or was coincidental.
Gronbach concedes that Qwest had been due to upgrade to faster
processor units for all its Lucent switches later in the year. Analysts
took that cue to absolve Lucent of blame for the incident, instead
noting that Qwest will probably have to install many new switches to
meet its sizzling growth rate. Fourth-quarter voice and data service
sales reached $1.16 billion, a 73% increase over the previous year.
Many of Qwest's frame/ATM switches have been using older
versions of Lucent's hardware and software, and such problems can
happen to any carrier "whenever they are having to roll out the
switches so quickly," says Steven Taylor, president of Distributed
Networking Associates in Greensboro, N.C.
Other analysts had a harsher reaction, charging that Qwest continues
to sell more services than its network rollout and back-office support
can handle. "If there was ever an award for marketing, it would have
to go to Qwest," says Bob Morrison, president of the Morrison
Group, a telecom user consultancy in Thousand Oaks, Calif. "But
they should also get the biggest loser award for not looking ahead to
all the capacity they're going to need." Southern California remains a
hotbed of Qwest problems, he charges, citing one recent T-1 order
that took up to 120 days and another that has not even received an
installation date after 30 days.
In any case, the recent frame and ATM problems were not limited to
these areas. For example, one user cited dropped circuits for sites in
Oregon, South Dakota, West Virginia and Ohio. Qwest officials
decline to state how many frame relay switches they now have.
A Lucent spokeswoman declined comment other than to say that the
switches and processing units involved are "the same hardware that is
used in many of the world's largest service-provider networks."
Qwest's Gronbach says the company will honor the terms of its frame
relay service-level agreement, conceding that a number of users'
guarantees for network availability and latency were broken by the
incident.
networkworld.com |
