Briand,
MCI could of been up and running in 24 hours, but they attempted to fix the problem for 10 days. AT&T had a complete system failure and had to back out.
....The original congestion in the network that began Aug. 5 was not due to excess customer traffic, but was the result of excessive CPU utilization in the Lucent frame relay switches triggered by something in a new software load. That generated too much administrative traffic, which caused each switch to toggle repeatedly between its live and back-up modes, Beaumont says.
The permanent virtual circuits in the network began going down after the network kicked in an automatic mechanism to deal with the administrative congestion: shutting down some DS-3 routes. The carrier network never actually destroyed the addresses of the frame relay permanent virtual circuits, but the effect was the same because the virtual circuits - which are not fixed routes, but rather virtual ones depending on network conditions - could not be successfully set up.
The switches that were affected by the problem were those now known as the Lucent 9000 frame relay switches and the Lucent CBX 500 frame/ATM switches. These were previously manufactured by Ascend Communications, which Lucent bought earlier this year. In some of the customer communications, MCI WorldCom technicians continued to call the first type of switch the "Cascade 9000," a reference to its original manufacturer, a company bought by Ascend.
Lucent says it accepts its share of responsibility for the problem, but appears to reject Ebbers' outright blaming of the vendor for the problem. A company spokeswoman has intimated that the technicians on hand to install the new software, which included both MCI WorldCom and Lucent personnel, might be at fault. "It was not a software problem, but a process problem," she says. She adds that the same version of the frame relay switch software has been successfully loaded into other carriers' networks, though she would not identify exactly which carriers.
Both Beaumont and Ebbers spent much of Monday deflecting criticism that they failed to explain, either publicly or to their own account representatives, exactly what was going on during the 10-day crisis. "Could we have been more effective in our PR? Well, we have to take it as a learning experience," Beaumont says. One problem was that because MCI WorldCom could not identify a root cause for the problem early on, Beaumont says: "We didn't have a lot we could say."
networkworld.com |
