To: zbyslaw owczarczyk who wrote (16864 ) 2/4/2000 8:45:00 PM From: zbyslaw owczarczyk Read Replies (1) | Respond to of 18016
Dropped circuits, session timeouts hit Qwest frame relay net By David Rohde Network World, 02/04/00 Qwest has confirmed that its frame relay and ATM users suffered numerous dropped circuits and other disruptions over an 11-day period ending last weekend. The problem was caused by a combination of an apparently overloaded transport circuit in Qwest's intercity network and processing speeds on Qwest's Lucent frame/ATM switches that were too slow for the amount of user traffic coming in. As a result, between Jan. 18 and 29, some users saw individual permanent virtual circuits (PVC) go down for up to several hours at a time when rerouting routines failed to kick in quickly enough. Others saw time-sensitive applications, such as SNA traffic, time out even though their PVCs were technically still up and running. To fix the problem, Qwest confirms that it literally forklifted out its Lucent frame/ATM switch in Los Angeles on Sunday, Jan. 22, and replaced it with a newer model. It then changed out the processing cards on the rest of its switches, and also lit three new OC-12 segments for use by frame and ATM customers. No users have reported to Network World that they saw their entire WANs go down. And it appears many Qwest customers got by totally unscathed. But the incident comes on the heels of complaints that Qwest has fallen behind on installation intervals and customer service as its sales force generates loads of new customer traffic. Some users say Qwest's response again followed this pattern. One user based in the Northeast who asked not to be identified says that at one point he kept losing connectivity with three of his frame relay sites for "five to 20 minutes out of every hour." Several days into the incident, the user says, Qwest reps told him to stop reporting trouble because the company had opened a master ticket for all the complaints they were receiving. "They told us not to call anymore when it happened, because there was nothing they could do for us," he says. The incident began Jan. 18 when Qwest engineers isolated several reported failures of frame PVCs to an OC-12 link between Atlanta and Fort Worth, Qwest spokesman Tyler Gronbach says. Because the traffic was not being rerouted properly, the carrier added the three additional OC-12 links, but then it noticed an unusual number of errors coming out the Los Angeles switch, again relating to attempts to move traffic to alternate routes. It was not immediately clear whether the problem in that switch was triggered by the network slowdown in the East or was coincidental. Gronbach concedes that Qwest had been due to upgrade to faster processor units for all its Lucent switches later in the year. Analysts took that cue to absolve Lucent of blame for the incident, instead noting that Qwest will probably have to install many new switches to meet its sizzling growth rate. Fourth-quarter voice and data service sales reached $1.16 billion, a 73% increase over the previous year. Many of Qwest's frame/ATM switches have been using older versions of Lucent's hardware and software, and such problems can happen to any carrier "whenever they are having to roll out the switches so quickly," says Steven Taylor, president of Distributed Networking Associates in Greensboro, N.C. Other analysts had a harsher reaction, charging that Qwest continues to sell more services than its network rollout and back-office support can handle. "If there was ever an award for marketing, it would have to go to Qwest," says Bob Morrison, president of the Morrison Group, a telecom user consultancy in Thousand Oaks, Calif. "But they should also get the biggest loser award for not looking ahead to all the capacity they're going to need." Southern California remains a hotbed of Qwest problems, he charges, citing one recent T-1 order that took up to 120 days and another that has not even received an installation date after 30 days. In any case, the recent frame and ATM problems were not limited to these areas. For example, one user cited dropped circuits for sites in Oregon, South Dakota, West Virginia and Ohio. Qwest officials decline to state how many frame relay switches they now have. A Lucent spokeswoman declined comment other than to say that the switches and processing units involved are "the same hardware that is used in many of the world's largest service-provider networks." Qwest's Gronbach says the company will honor the terms of its frame relay service-level agreement, conceding that a number of users' guarantees for network availability and latency were broken by the incident.networkworld.com