Teledotcom
teledotcom.com
The Blame Game
Slow bandwidth provisioning is weighing down long-haul and local access network providers. Why are they pointing
fingers at each other?
by Amy Larsen DeCarlo and Brian Washburn
Related story:
The (Half) Million-Dollar Mile
The endless expansion of the Web is a bittersweet blessing for service providers. On one hand, the business case
couldn't be better. Everyone everywhere wants faster Internet access, and they want it yesterday. So long-haul
providers are working furiously to light fiber across the country and around the world. On the access side, metro area
providers are busy installing the latest optical equipment. And optic transmission vendors are busy hyping their
equipment's rapid provisioning functions. But despite efforts by the entire industry, it still takes weeks--if not
months--for customers to get service.
At the crux of the provisioning issue is the fact that demand far exceeds capacity. Carriers must ensure rapid
provisioning, but inherent problems within the network are slowing down the process. While optical networks can
handle all the bandwidth necessary for intense traffic, the laborious process of provisioning is hampering the potential
success for providers. In fact, until providers can forego the truck rolls and human intervention needed to provision
bandwidth and implement a way to automate the process, optical networks and service providers will not reach their
potential.
In the midst of provisioning problems in both long-haul and metropolitan-area networks (MANs), both sides have taken
to pointing fingers, which only makes the problems worse. MAN providers insist the long-haul carriers aren't providing
bandwidth quickly enough or delivering quality of service (QoS). But their long-distance counterparts say local carriers
aren't providing access and capacity as quickly as they say they will. The bottom line is that snail-paced provisioning
won't be solved until both carriers resolve their issues. For every group of providers that can't solve the problem, another
group is waiting to beat them out of customers and revenue.
Going the Distance
Long-haul and MAN providers have different needs to fulfill the demand for quicker, automated provisioning, but they
have one thing in common: Ultimately, they must rely on each other. To do that, they must resolve their own issues first.
There's truth to the accusation that long-haul providers can't provision bandwidth quickly enough. And the answer to
the problem is complex. Ultimately, it boils down to a complete change in the way new networks are built.
Older long-haul networks were built to handle voice--a completely different animal than data. When voice ruled the
network, not only was it easier to estimate capacity requirements but 80 percent of traffic stayed local. Providers had a
lot less to worry about when it came to provisioning. Voice traffic patterns and capacity requirements were easier to
predict for voice than data traffic because changes could be foreseen by customers' physical locations and population
growth. For data traffic, the backbone still bears a heavy traffic load despite local Web caching efforts. Couple that with
the unpredictable nature of Internet protocol (IP) traffic, and long-haul carriers face the daunting challenge of how to
respond to dynamic traffic changes in the network as they happen.
Optical networks are meant to be the answer to this problem. Fiber optics can not only promise light-speed transmissions
but also carry massive loads of voice, video and data traffic because of innovations like dense wavelength-division
multiplexing (DWDM), where multiple lasers simultaneously send multiple wavelengths of light over the same fiber.
The problem is being able to light the fiber quickly enough and then provision the capacity in the right places as
traffic needs grow and shift around the network. That's why intense customer demand is a double-edged sword.
Internet service providers (ISPs) and other long-haul customers are clamoring for more and cheaper bandwidth. "We are
starting to see some very savvy customers walking through the door and asking for managed wavelength services," says
Al Van Thit, senior director of transmission engineering for Qwest Communications International Inc.
Unfortunately, Internet traffic doubles every three to six months while the capacity ceiling on new equipment has been
doubling roughly every nine months.And unlike their networks, optic installation technicians don't move at the speed of
light. "It is almost impossible to believe that with the tens of thousands of miles of fiber carriers are laying, there isn't
enough capacity to support the demand," says Andrew McCormick, a senior analyst with Aberdeen Group Inc.
(Boston). "But then you have to realize that much of that fiber is still dark because the optical equipment to light it isn't
in the field yet." Right now it can take months to turn up service on a new OC-48 (2.4 Gbit/s) circuit reaching from coast
to coast, from installation to lighting the fiber. Carriers are in agreement that provisioning time needs to be cut from
months to minutes. "We would like to cut our circuit provisioning time by about 90 percent," says Jeff Young, vice
president of network technology for Cable & Wireless North America (C&W North America, Vienna, Va.).
The obvious answer is to automate provisioning and remove many of the manual steps that slow down the process of
turning up new service, but that is hard to accomplish in existing networks. Getting a nationwide optic network to a
months-to-minutes level of responsiveness, especially if long-haul providers are kept busy just trying to keep up with
customer demand, is another story.
"Our first challenge is to be able to provision, period. Then we have to take that up a level to be able to provision
services in a reasonable amount of time," says Chris Rothlis, vice president of engineering for Broadwing Inc.
(Cincinnati). Broadwing is currently constructing an 18,500-mile optical network with equipment from Alcatel N.V.,
Ciena Corp. (Linthicum, Md.), Corvis Corp. (Columbia, Md.) and Nortel Networks Corp.
Moving away from existing point-to-point ring topologies designed for telephony traffic and toward an optical
mesh infrastructure is probably the only surefire answer. Ring topology relies heavily on redundant equipment, and
at least half of the fiber capacity is used for restoration and protection in case of a fiber cut or some other service
interruption. A meshed architecture can be arranged into logical pathways that allow for fast routing changes in the case
of a fault. This cuts down on the need for redundant equipment and fiber, and it also opens up the possibility for
reprovisioning circuits to provide for fluctuations in capacity requirements.
"Once the infrastructure is actually built, it is much easier to add capacity to a mesh topology than a ring. It is just a
question of getting [long-haul carriers] to stop adding new rings and move to a mesh architecture," says Joe Bass, vice
president and general manager of the wavelength routing division for Cisco Systems Inc. (San Jose, Calif.).
That flexibility is key for service providers that are eager to add new business by increasing customer orders. For
providers like Broadwing that are building their infrastructure from scratch, a mesh architecture was a no-brainer. "Our
vision was to have the basic infrastructure in place so we could easily provision new service and then reprovision
services as capacity needs changed," Rothlis says. Provisioning speed is crucial to a market newcomer such as
Broadwing that, despite having competitive local service provider status in several states, competes against long-haul
providers like Qwest.
And Broadwing's marketing plan is a perfect example of how crucial provisioning time is to being competitive. In its
effort to woo and win new accounts, the provider guarantees it will deliver new circuits to a customer within 45 days of
the time its provisioning department accepts the order. If Broadwing fails to install the circuit by the guaranteed delivery
date, the company will issue a one-month credit for free service. If the provider misses that date by more than a month, it
will give the customer a two-month service credit. That could turn into a serious financial strain if provisioning time isn't
as quick as promised.
But Rothlis says provisioning is getting easier for Broadwing, largely because it is becoming a more automated process.
New, more sophisticated management software is lending an intelligence to the optical core that has been missing in the
past. The first point-and-click provisioning software tools are making it to market this quarter from vendors such as
Corvis, Lucent and Nortel. These tools make it possible for network operators to provision wavelengths across a circuit
without necessarily having to send someone out into the field. They save providers money in the long term, since
providers can activate circuits remotely rather than having to deploy technicians. Automating the process also has the
potential to eliminate human error. "We need to take human hands out of the equation," says C&W North America's
Young. The closer the technology gets to taking manual labor out of provisioning, he says, the fewer provisioning
mistakes.
Even as long-haul carriers find ways to solve their bandwidth provisioning issues, though, their problems don't end
there. Local carriers, they say, aren't pulling their weight when it comes to speeding service delivery in the metro area,
effectively retarding business growth and profits. Once the bandwidth is easily provisioned, a weak connection on the
local access side could make that all meaningless.
"We have major issues with the need for bandwidth in the metro area. We think a lot of the new players that are touting
their ability to deliver high-speed access aren't actually expending the capital to do that," says Fred Harris, vice president
of design, applications and services for Sprint Corp. City Lights But MAN providers say their concern isn't offering up
the capacity and access quickly enough. Rather, it's how to offer access that is scalable enough to handle increases or
decreases in need, as well as the popular concept of bandwidth on demand.
At first glance, MAN providers seem to be leading a charmed life. These companies build rings using the latest optical
equipment--and they're not bogged down by decades of legacy equipment like long-haul providers are. What's more,
their backbone capacity is measured in gigabits, which means these providers can drown the most demanding dot-coms
with local bandwidth.
And MAN providers say they have speed on their side. Executives from metro carriers such as IntelliSpace (New York),
Sphera Optical Networks Inc. (New York) and Davnet Ltd. (Melbourne, Australia) tout provisioning time in minutes.
They say they can literally pour on capacity faster than the customer can place the order and sign the paperwork.
So why are other metro carriers, such as Electric Lightwave Inc. (Vancouver, Wash.), Lightwave Communications Inc.
(Milford, Conn.) and Time Warner Telecom LLC (Greenwood Village, Colo.), deploying similarly advanced optic
equipment and offering provisioning in weeks instead? Despite accusations from long-haul service providers that MAN
providers aren't moving quickly enough, they say most customers consider bandwidth a part of their overall business
strategy and therefore place orders with long lead times.
Bandwidth on demand--the elusive concept that customers could order capacity as they need it--is full of caveats, and
not all providers can or should easily jump into it. For one thing, it is not for casual users: MAN providers won't hook in
customers for bandwidth on demand without a steady service contract. And service providers could lose money by
investing in infrastructure to offer bandwidth on demand that customers won't end up using.
This is not to say bandwidth shouldn't be provisioned in a timely manner. "Two years ago, it could've taken 30 to 50
days to order equipment and get the pieces in place," says Gary Liberman, vice president of service activation for West
Coast regional provider Electric Lightwave. "Currently for on-net service, it takes about 20 days. For private lines, it
takes 25 days to add carriers. We're shooting to reduce that to 10 days." But the reality, Liberman says, is that customers
are comfortable with these time frames. That's because companies know what they need, and budget and order capacity
well in advance. When the rare emergency comes up, Liberman maintains, the company has turned around same-day
orders.
The ultimate solution to the problem of timely bandwidth provisioning on the access side is to use intelligent optical
switches and routers that can sense fluctuations in traffic and make sure bandwidth becomes available wherever it's
needed. Standards bodies such as the Optical Internetworking Forum are working on signaling standards to handle that
kind of dynamic bandwidth provisioning to regional and metropolitan networks. Vendors such as Tellium Inc.
(Oceanport, N.J.) and Avici Systems Inc. (North Billerica, Mass.) are already demonstrating technology that performs
instant, automated provisioning.
That sophisticated automated provisioning will become even more crucial as new bandwidth-intensive voice, video and
data applications hit the marketplace. "You can draw a direct parallel to a 56k modem," says Don Smith, president of
Nortel's optical Internet unit. "Once you get a feel for high-speed access, you never want to go back."
Sphera, a startup that sells Synchronous Optical Network (Sonet) links to service providers connecting their metro
facilities, is wrestling with issues of automating provisioning. According to Sphera technology officer Richard van
Leeuwen, the company is looking at ways to throttle capacity up and down based on time of day, as well as examining
methods of distributing providers' traffic loads around the network, depending on time of day.
IntelliSpace has also taken on the bandwidth-on-demand challenge and is rolling out the service to its customers under
the eValve brand. IntelliSpace obviously doesn't see the need to make customers wait for bandwidth, but it does see the
need for providers to approach with care. A customer can call in today and have additional capacity turned on in five
minutes, says IntelliSpace chief technology officer Carlo Lalomia. By early December, he forecasts that the provider's
Web-based provisioning will be up and running. But, he cautions, automated provisioning is hazardous terrain, and
building such a system isn't straightforward. "It's based on a complicated algorithm of the amount of capacity available
to the buildings, plus subscribed capacity, plus current utilization. Then [the system] allocates what it thinks is available,"
Lalomia says.
In the meantime, service providers are split on the issue of how to address changing needs for capacity and bandwidth
on demand until these other systems are surefire. Liberman suggests a simple answer: Overbuild. A customer that
requests a 128-kbit/s circuit should receive at least a T1 (1.544 Mbit/s). That way there's always the option to offer
premium-priced bursts above the allocated rate, just in case.
But Tony Thakur, vice president of engineering and technology at Time Warner Telecom, has a different take, splitting
the provisioning issue into two fronts. When it comes to installing bandwidth on demand just in case it catches on with
users, his answer is no. "It's a huge investment. You don't want to install the optic gear in hopes that you might recoup
the investment," Thakur reasons. However, some service providers that connect with Time Warner Telecom are looking
at real-time bandwidth provisioning and are sure that's what they need; Thakur sees the model making financial sense in
these cases.
Once the basic issue of bandwidth on demand and scalable access is solved, MAN providers still have a bone to pick
with their long-haul counterparts, who they say are falling short. "We're trying to position our business for dynamic
bandwidth allocation, but the big guys do not want to do that," says Eugene Pisman, IntelliSpace chief network
architect. Pisman accuses long-haul carriers of failing to offer even a basic IP QoS feature such as Differentiated Services
(DiffServ). DiffServ places priority flags inside ordinary TCP/IP packets. Routers that support DiffServ can read these
embedded bits and use the information to decide which packets should have priority.
IntelliSpace has decided to take the problem on itself, intending to buy into long-haul capacity and offer its own
services. "We will extend QoS out into Tier 1," Pisman says. Lightwave Communications has similar designs to go
nationwide, according to Aubrohn King, the company's director of operations. And as an Australian export with MANs
dotting Asia, Davnet expects to haul both nationwide and international traffic under its own banner within the next
year.
In each case, it's not just a matter of installing advanced services. By taking over the end-to-end circuit, these MAN
providers expect to cut down drastically on overall provisioning delays. Time Warner Telecom, which already owns
end-to-end infrastructure, is focusing on this goal in the coming months: By this time next year, it intends to rein in
on-net provisioning times from 30 to 45 days down to three days, according to Thakur.
But most typical MAN providers don't intend to build their own long-haul networks. They'll buy capacity from their
national and global network counterparts, and plan to be dependent on them for a long time. So while the two sides are
busy hammering out their own infrastructure issues, they might want to eliminate the finger pointing and work together
for an end-to-end solution.
The (Half) Million-Dollar Mile
Local access providers might be quick when it comes to turning on capacity inside the metropolitan-area network
(MAN) or in the Internet cloud, but when it comes to hooking up customers located outside the MAN, the delays pile
up.
Customers located in buildings already served by a provider usually have to wait only a few days before they receive
service. But when an order is placed from a building that isn't currently served by the provider receiving the business, a
third-party provider must interconnect circuits or the carrier must add fiber drops to the network, which could take
weeks. Even worse, if the metro provider needs to lay fiber to the building first, get out the calendar: That can take three
to six months.
When it comes to delays, long-haul providers lambaste local carriers while the MAN providers don't mince words
concerning the speed of the incumbent local providers. But putting metro infrastructure in the ground is time-consuming
and can cost up to $500,000 per mile, according to Alex Benik of The Yankee Group (Boston). "They have to make the
business case on a building-by-building basis," he says.
Sky-high installation costs have led dozens of MAN providers to Metromedia Fiber Network Inc. (White Plains, N.Y.), a
dark fiber giant in metro areas. Metromedia builds out fiber and sells it to providers, which save money in installation
costs and manpower in many cases. The final connection from backbone to building is what makes a MAN expensive
and precious, and the metro operators know it. "There are thousands of miles of long-distance fiber," says Davnet Ltd.
(Melbourne, Australia) chief network architect Isaac Orr, referring to the ready availability of competitive pricing and
bandwidth exchanges. "There aren't a lot of companies that are dealing with the last mile to the customer."
Questions or comments on this story? Letters to the Editor |
