October 19, 2001
Trade Center Attack Highlights Problem
In Telecom Sector's Legacy of Monopoly
By SHAWN YOUNG and DENNIS K. BERMAN
Staff Reporters of THE WALL STREET JOURNAL
NEW YORK -- George Famulare, a 28-year
veteran of New York's local phone company, was
doing his expenses in one of the world's largest
communications hubs on the morning of Sept. 11
when he heard a loud thud.
Initially, he was infuriated, thinking a renovator's
scaffold had fallen off the landmark 1926 building at
140 West St., just feet away from the World Trade
Center. Its 32 floors house the humming gray
equipment boxes that carry nearly 30% of Lower
Manhattan's phone and data traffic, capacity equal
to that of Cincinnati.
Mr. Famulare, in charge of Verizon Communications Inc.'s buildings south of Midtown, saw people
running away but implored employees to stay put. "We're safer in the building. The building is a tank."
The events of Sept. 11 would quickly prove otherwise. By the end of the
day, Verizon's hub sustained the worst damage ever to the nation's
communications network. Getting calls going again has proved to be the
most significant challenge Verizon, the largest local phone company, has
ever faced, and it is prompting a rethinking of the security of America's highly concentrated
telecommunications systems in an age of terrorism.
The disaster also has sparked a new round of debate about telecom deregulation itself. Verizon
executives have loudly criticized the rules that require them to rent parts of their networks -- and space
in their central offices -- to competitors. They argue that the arrangement has discouraged rivals from
building alternative networks by making it too cheap and easy to piggyback on the regional giants.
But competitors draw the opposite lesson, saying the catastrophe points to the danger of having one
company in control of so much. "Verizon is the incumbent, and the wiring has been in the ground for
forever and a day. They have not built a redundant network," says Tom Jones, director of Spectrotel
Inc., a competing carrier based in New Jersey.
All across the country, towns and smaller cities rely on only one hub, meaning that they could lose touch
completely if that hub were wiped out. In many larger cities, phone traffic is funneled into very
concentrated routes in and out of town. And yet, as of Sept. 11, an industry-led federal committee that
addresses phone reliability hadn't discussed terrorism contingency plans in at least a year and a half.
'Everything on the Table'
"This is now a whole new layer of preparedness that our
industry and our country needs to be thinking about," says
Ivan Seidenberg, Verizon's president and co-chief
executive. "Whether we need antiaircraft missiles on the tops
of buildings, I don't think so," he adds. "But I'm willing to put
everything on the table."
Already, changes are being made. The federal government
has asked the nation's wireless carriers to assign priority to
government agencies and emergency personnel in the event
of another disaster. On Sept. 11, many rescue workers
couldn't get calls through. AT&T Wireless Services Inc., which ran many of its switches through West
Street, will now spread them throughout the city. Verizon is adding more backup capacity and alternate
routes and has asked regulators to raise wholesale rates in New York to cover the expense.
Many businesses are taking the matter into their own hands. Merrill Lynch & Co. is conducting a global
review of its communications providers to ensure the brokerage firm has multiple sources in each
location. Phones in the firm's New York headquarters across from the Trade Center went out on Sept.
11, even though it split its local and long-distance service between Verizon and AT&T Corp., which
lost a major switching hub in the Trade Center. The firm is even using a set of lasers to beam data
across the Hudson River between locations in New York and New Jersey.
At the New York office of headhunting firm Spencer Stuart, where employees are still working from
cellphones, officials are ordering different lines from different carriers that reach their office by different
routes and hubs. "It's just something that wasn't worth the cost before the unthinkable happened," says
Rick Abel, chief technology officer of the Chicago-based firm.
E Commerce Group Inc., which had more than 40 high-speed data lines disabled on West Street, is
building a new facility in Charlotte, N.C., so the New York-based company can have "geographic
diversity" to spread out its risk beyond backup offices it already maintains in New Jersey.
By nearly all accounts, New York City's phone network -- and the
people who run it -- performed well under incredible stress. By the
Friday after the attack, Verizon had moved the equivalent of 2.1 million
voice and data lines around Lower Manhattan. Stringing fiber-optics
through open trenches and windows, the company was able to restore the
New York Stock Exchange less than a week after the assault. Verizon
workers wearing respirators had to climb 23 flights of stairs in the dark to
lug down servers to run the exchange's price-quotation system.
It was an impressive performance compared with what was once
considered the nation's worst telecommunications disaster, a relatively
limited fire inside a Chicago switching station on Mother's Day in 1988
that closed O'Hare Airport and wiped out service for 38,000 local
customers for as long as a month. Since then, the industry has used
technology and better planning to create far more reliable networks, focusing on adding backup circuits
to fiber lines that seemed vulnerable to an errant backhoe or natural disaster. Most networks can now
respond within milliseconds if a particular link is broken, rerouting traffic through hundreds of alternate
links.
Vulnerable 'Nodes'
What the World Trade Center attack showed, however, is the vulnerability of the final, local link to
phones and computers through the nation's telecom hubs, or "nodes," which act as collecting points for
traffic. More than one month after the attacks, thousands of residents and businesses are without basic
phone service.
"The weak point is the local exchange carrier," says Todd Tanner,
president of the Tanner Group, a Salt Lake City firm that helps
clients such as the Department of Defense and Walt Disney Co.
build toll-free call centers. "If you knocked out the central office in
downtown Salt Lake City, you'd not only take out the local office,
but the switch that connects all the long-distance carriers to the
people in Utah."
The New York City telecom system is itself a relic of historical
quirks, with different systems built on top of one another, forming
a kind of technology sediment. Manhole access to the telecom
network is still managed entirely by the Empire City Subway Co.,
a subsidiary of Verizon that dates back to 1890. While
maintaining hundreds of separate routes around the city, many
competitors' lines still travel over the same paths laid down more than 100 years ago. Under the Sept.
11 wreckage, Verizon officials uncovered defunct wooden cable sheaths dated 1926.
Foolproofing the physical infrastructure against future terrorist attacks could require hundreds of billions
of dollars of new investment. Some say it makes sense to spread out the nation's most densely
concentrated hubs. Others look to switching methods used by the Internet, which was designed to
withstand nuclear attacks, as a model.
But finding any consensus about what to do will be difficult. Verizon's Mr. Seidenberg says the terrorist
attacks prove that only the biggest phone companies are up to the task of securing the nation's phone
system, and says current regulations discourage the development of true, alternate networks. He
defends Verizon's performance, pointing out that the company has 70 hubs in New York City and four
besides West Street in the financial district, which allowed the company to restore service to the New
York Stock Exchange and crucial parts of the financial system in a matter of days. He remains
concerned about security, though, and is now making plans to tighten it around the company's hubs.
Among other things, Verizon is making plans to require background checks on employees of
competitors who use Verizon's facilities.
Aid From Competitors
Others say the extensive damage to Verizon's hub demonstrated that it is dangerous to rely on one
company. They point out that only with the help of smaller competitors did parts of New York get their
dial tones back.
"We have buildings that would have been waiting for Verizon to get the service up if there were not
alternatives," says Agostino Cangemi, the New York City commissioner in charge of telecom
franchising. Allegiance Telecom Inc. of Dallas provided service to some city agencies disconnected,
while Time Warner Telecom Inc. is now operating thousands of lines through fiber it activated after the
attack.
Such activities aside, Sept. 11 hammered home the fact that local phone business has essentially
remained a monopoly. Dozens of long-distance networks crisscross the nation, but there is only one
dominant local carrier in each city, leading to concentration in hubs such as West Street. Even
competing telecom companies that build their own infrastructure mostly rely on connections provided
by the regional phone companies in the same central offices. In a study for Amtrak by the Tanner
Group, less than 10% of competitive local carriers were found to have facilities truly separate from the
local Bell companies.
When it was built in 1926, 140 West St. sat in near-isolation
among the markets, bars and luncheonettes along the
Hudson River. Considered the first truly Art Deco
skyscraper, the building is far more glamorous than most
central offices that form the architecture of the phone system
because it once served as headquarters for the New York
Telephone Co. The French architect Le Corbusier so
admired the 486-foot-tall building that he featured it in his
famous work, "Toward a New Architecture."
At the time of its construction, phone companies were already replacing telephone operators with
banks of electromechanical switches. Originally developed by a frustrated mortician whose calls were
being diverted to a competitor by the rival's phone-operator wife, the switches of the time used
motor-driven magnets traveling on thin vertical bars to connect the calls. A full panel office would cover
about 6,000 square feet, and serve 10,000 lines.
Advances in electronics led to the development of mega-hubs such as West Street. By Sept. 11, racks
of gently humming electronics were serving 4.5 million data circuits, along with 300,000 phone lines.
Such concentration occurred across the nation: Between 1990 and 1999, the number of local Bell
central offices inched up less than 1% to 9,968, while the number of phone lines increased 34%,
according to the Federal Communications Commission.
Mr. Famulare and Verizon officials knew the concentration of West Street well. After crawling out of
the building through a tunnel left by a chair when the first tower collapsed, Mr. Famulare came back
twice to try and turn off all essential power to conserve energy. On his way back a third time, giant steel
girders from the collapse of 7 World Trade Center, a 47-story building adjacent to the towers, crashed
through the third-floor walls, falling all the way to the second level of the hub's basement. Water from
broken lines in the street and from fire hoses poured into the building, making it impossible to operate
the generators and batteries that were supposed to run the equipment in the event of a power failure.
When the power ran out, the calls stopped getting through. By the next
morning, "water was rushing down the stairs," recalls Paul Lacouture,
head of Verizon's network. "It was like it was raining inside the building in
a dust storm."
The damage was so great that Verizon has had to rebuild much of the
subterranean network connecting southern Manhattan by rerouting lines
through other hubs. Where its manholes weren't buried under the remains
of the World Trade Center, workers used special industrial vacuum
cleaners to suck out debris. The rerouting meant that thousands of data
paths had to be redrawn, a task complicated by the fact that many large
operations, such as major brokerages and the Big Board, have custom
setups that had to be redone virtually from scratch.
The tentacles of West Street reached across the city. Nearly five weeks after the attack,
municipal-bond brokerage Lebenthal & Co. is still without 210 of its 225 phone lines, which were
hooked into the West Street switching station two blocks from its office at 120 Broadway. "This is a
phone business," says President Alexandra Lebenthal. "I don't know how many of our clients are
frustrated because they can't get in touch with us."
For now, the company is coping with a makeshift messaging system: All calls are now forwarded to a
remote answering service, where operators send employees e-mails with the details of the call. Once a
Lebenthal employee receives the e-mail, he then rings the caller back on a cellular phone. "Because
we've been sending so many e-mails back and forth, once a week, that whole system crashes," adds
Ms. Lebenthal. To compound matters, the building's cellular reception is poor, which has forced
employees to cluster around the office windows, often leaning against the wall to help find a stronger
cellular signal.
Even businesses far away from the damage were affected. Officials at the Limited Inc. were shocked to
discover that the main circuit that delivered service to an office on 42nd Street came out of West
Street. The phones at E Commerce Group, directly across from the Trade Center, somehow kept
working because officials had ordered some copper links that had curiously been routed through a hub
on 37th Street.
Delicate Balance
The damage to the West Street hub showed the delicate balance of the current system, in which
changes in one location have unforeseen impacts on another. As large companies fled to new quarters
around the city and nearby suburbs, the normal patterns of phone traffic in the New York area were
disrupted, leading to congestion that left some callers hearing fast busy signals and recordings. After
Lehman Brothers Inc. took over a Sheraton hotel in Midtown, its bankers quickly discovered that the
phone system, engineered for tourist traffic, could only handle about 75 outgoing calls at a time.
Verizon still has 16,000 lines out of service, many of them in Chinatown, because the district's phone
lines passed right under the Trade Center, where they remain buried and waterlogged. At Mott and
Worth Streets, Verizon has opened gaping holes in the street, where crews working around the clock
are installing the plastic sheaths that will soon house the new phone lines. Instead of opening just the
tops of manholes, the company has dug what look like foxholes more than 10 feet long so it can fit
more workers in at one time.
Back at West Street, Mr. Famulare is still working nearly constantly. A 60-foot plywood wall blocks
off access to the rear of the building, where gigantic cranes are still excavating the remains of 7 World
Trade Center and trying not to, in their jargon, "rub" West Street. Much of the building is now back in
service, and Mr. Famulare speaks with evident pride as he looks at his old "tank" being restored to
service. "We have resurrected it," he says.
Write to Shawn Young at shawn.young@wsj.com and Dennis K. Berman at dennis.berman@wsj.com |
