Hagar, the problem now appears to have been much worse than reported.
As an investor in NN ( not an MCI customer <bg> ) I want to try and get to the bottom of who is at fault and,
without holding TM on a pedestal I want to know if NN has a better mousetrap than CSCO,LU or NT.
I am not an engineer but I can read police reports. Here it says:
Domestically, MCI uses BNX switches from Nortel.
WorldCom, prior to its merger with MCI, built its
frame-relay network with equipment from companies
that ultimately were acquired by Cisco and Lucent. The
carrier is upgrading the whole network with Lucent
switches.
So the issue here is simple:
Either MCI/WCOM failed because they have a patchwork type of incompatible switches
and/or the new LU switches are bad.
MCI/WCOM has 3 options.
a) find out what's at fault in above scenario and fix it . ( I doubt they will be able to seeing the complexity and recurrent failures to do so ).
b) Redo the whole system with LU switches.( will their angry customers give them time? )
c)Go with CSCO
c)Take a bold new approach and go with NN which is the only carrier class ATM switch that hasn't failed, and have NN set their network up.
As an angry customer, ready to ditch MCI/WCOM/LU in toto, I think I like c) best,
back later
TA
----------------------------------
MCI Frame Net Melts Down
techweb.com
(08/12/99, 7:00 p.m. ET)
By Terry Sweeny & Chuck Moozakis, InternetWeek
Irate IT executives are moving quickly to
exact compensation from MCI WorldCom for
a prolonged frame-relay network outage
that sporadically halted traffic.
The outage, originating from Lucent switches but
downplayed by MCI WorldCom as "congestion",
slowed frame-relay traffic to a crawl starting Aug. 5.
While the permanent virtual circuits that customers use
to access the Hyperstream network never completely
failed, congestion brought many users' transactions to a
halt for extended periods.
Early in the week, an MCI WorldCom official said 15
percent of the carrier's customers were affected. By
midweek, MCI WorldCom increased its estimate to 30
percent.
"The congestion is being eliminated, and we're slowly
bringing the network back, a spokeswoman said,
though the company strongly resisted characterizing the
event as an outage.
Yet in a conference call Monday with the Chicago
Board of Trade, which was hit hard by the outage, a
top MCI executive described the problem as a
"meltdown of the network," according to a statement
issued by CBOT president Thomas Donovan.
During the outage, more than 250 CBOT workstations
in Paris, Tokyo, London, and the United States were
knocked out. Those workstations support the board of
trade's Project A, a global electronic trading system for
soybeans, corn, and U.S. Treasury Bonds.
"The latest problem with MCI WorldCom comes
despite numerous past assurances from them for
improved Project A service," Donovan said.
He said the outage came just two days after he met with
MCI technicians to discuss the unsatisfactory quality of
service that they have been providing.
On Tuesday, CBOT resumed trading on Project A,
though the system went down again for a few hours
early Wednesday morning before being restored.
Other high-level IT managers said they were hit hard by
the outage and were dissatisfied with MCI's handling of
the situation.
Bill Bartkus, vice president of information systems at
national truck-stop operator TravelCenters of America,
said some TravelCenters locations have had no data
communications service since last week, while others
have experienced severe latency. "We're not satisfied at
all with this," Bartkus said. "There has been a serious
impact on our business."
Although TravelCenters doesn't have a service-level
agreement (SLA) with MCI WorldCom, Bartkus said
he will ask for some financial redress to offset business
losses, which he said he couldn't immediately quantify.
Another IT exec looked askance at MCI's claims that
the failure was actually congestion.
"In the real world, the measure of whether a network is
down is if traffic is actually flowing," said Nick
Colakovic, MIS manager at First Industrial Realty
Trust, a Chicago-based real estate company. "In the
current outage, this is not the case, so the network is
down. I would be happy if I had one half of my
network, but I'm not getting anything from them."
The failures were impacting major metropolitan areas,
including Chicago, Denver, Los Angeles, and New
York.
The network problem was also having a cascade effect
on other providers that interface to the MCI
WorldCom network as part of IP peering
agreements.
"We still have about 50 customers absolutely down and
probably another 10 to 15 that are experiencing severe
congestion and packet loss of up to 60 to 70 percent,"
said Dan Sprouse, network operations center
supervisor at ISP RMI.Net, on Wednesday morning.
"We are very upset because MCI WorldCom has only
been feeding us happy stuff instead of telling us what is
really going on," Sprouse said. "We have had customers
tell us that they may go out of business because of this."
A source at a tier 1 ISP said the carrier had gotten no
information or explanation from MCI WorldCom for
the problems, an unusual wall of silence. Under normal
operating conditions, service providers with peering
agreements alert each other when there are anomalies
or problems on the magnitude of what MCI Worldcom
experienced.
A source in MCI WorldCom's customer service center
said the carrier and its equipment vendors tried to
rebuild the network Monday morning, only to see it fail
again Monday afternoon. The MCI WorldCom
spokeswoman denied such work had been done or was
even necessary.
"They certainly are sugarcoating the thing," said Rick
Malone, principal at consultancy Vertical Systems
Group.
Vertical Systems Group estimates MCI WorldCom has
about 20 percent of the 800,000 domestic frame-relay
ports in service today, second only to AT&T's 34
percent.
Malone described the subnetworks that comprise MCI
WorldCom's frame relay backbone as "somewhat of a
patchwork," given all the equipment vendors involved.
Domestically, MCI uses BNX switches from Nortel.
WorldCom, prior to its merger with MCI, built its
frame-relay network with equipment from companies
that ultimately were acquired by Cisco and Lucent. The
carrier is upgrading the whole network with Lucent
switches.
"MCI always told everyone that by keeping the
networks in separate domains, they wouldn't be subject
to a networkwide, catastrophic event like AT&T's,"
Malone said.
AT&T's Interspan frame-relay network crashed in April
1998, affecting 6,600 customers. AT&T CEO Michael
Armstrong quickly apologized to customers, and AT&T
technicians began working with them to get their
networks back up and running.
MCI WorldCom did confirm that Lucent technicians
are working on site with line cards and software to
rectify the network problems.
With its Hyperstream frame-relay services, MCI
WorldCom offers a service-level guarantee of 99.99
percent core network availability, 99.9 percent
end-to-end network availability, four-hour end-to-end
mean time for repairs, no more than a 60-millisecond
one-way network transit delay, and 99.99 percent
frame delivery within the committed information rate.
"All [MCI WorldCom] SLAs are backed by a
comprehensive credit structure, which offers credits to
all frame-relay ports and PVCs, not just those below
the SLA threshold," the service provider said in a
statement. "Credits take effect in the first month of
noncompliance. Monetary consequence of failing to
meet SLA equals 5 percent deduction from [the] whole
bill in any month that the guarantee was not delivered."
It's uncertain whether MCI WorldCom will offer all
affected customers refunds or credits on their
frame-relay bills, as AT&T did, according to another
MCI official. While the service provider may be
studying some form of compensation, so far nothing had
been decided.
Kate Gerwig of tele.com and Margie Semilof of
Computer Reseller News contributed to this story.
Related Stories:
MCI WorldCom Outage Is Bigger Than Reported
techweb.com
you said
I agree, absolutely. I am familiar to varying degrees with a lot of the WAN products and the vendors. I know a lot about some. To hold NN products and TM on a pedestal
because of the other vendors publicized network downtime is funny.
What is clear to me is that MCI botched this technically and with their own customers. I suspect a price will be paid and it will be MCI, and the public will probably hear a
story. |
