The High Capacity Core
The search for greater efficiency at the heart of the Internet will drive the next round of router RFPs
June 1, 2002
By: Joan Engebretson
America's Network
americasnetwork.com
Just how fast the Internet is growing continues to be hotly contested. One thing is certain, though. For the major Internet backbone providers, the traffic growth on key routes is beginning to outpace the capabilities of traditional routers.
In the past, carriers upgraded to higher Sonet speeds every two years or so. The number of ports on core routers remained the same, but the data rates that those ports could support went from OC-12 to OC-48, and then to OC-192. After that, things changed.
Service providers do not envision upgrading to OC-768 any time soon. That kind of transmission equipment hasn't been developed because dense wave division multiplexing (DWDM) largely negated that need. Instead, service providers are beginning to look for core routers with a larger number of OC-192 ports.
Traditional core routers don't have enough of the highest-speed ports, so the largest network operators have begun to connect multiple routers together in their core points of presence (POPs). The problem with that is that it wastes a lot of ports (Figure 1). Those ports cost $150,000 or so apiece - and when they're used to connect to another core router within the same POP, that investment doesn't generate any revenue.
A crop of new products - mostly from startups, but increasingly, from dominant vendors (Figure 2) - are poised to address this need. And despite the downturn, this is one category that continues to receive substantial development dollars.
"Three-hundred million dollars went into the startups just this year," says Richard Norman, chief technology officer and president of one such startup, Hyperchip. "That's in addition to what Juniper and Cisco are spending."
Most of the larger Internet backbone network operators say that they expect to issue RFPs for core routers late this year or early next.
When asked what their most important criteria will be, operators are virtually unanimous. Bill Leighton, vice president of development for AT&T Laboratories, seems to be speaking on behalf of an entire industry when he says, "The two biggest capabilities to us are scalability and reliability."
Although that may sound like a platitude, it's a far cry from what formerly was the primary driver behind core Internet purchase decisions. When keeping up with traffic growth was the key challenge, operators were in no position to sweat the details.
Today, though, operators are realizing that that attitude may be what has caused the Internet backbone business to be an under-performer in terms of profitability.
"The [return on investment] on these networks must be greater than for T-bills, and that isn't the case now," says Faizel Lakhani, vice president of product and strategic marketing for Caspian, a startup, still operating in stealth mode, that is developing a high-capacity core router.
Operators now acknowledge that they must improve the overall economics of their IP networks. Router density and scalability play into that by reducing non-revenue generating ports. Some players suggest that the new high-density routers could have an even greater impact by eliminating or reducing the need for aggregation routers.
Reliability also affects router economics by potentially reducing operating expenses. Ultimately, some developers say, a more reliable router also could generate capital savings by eliminating the need for redundant network elements.
Maxed-out routers
TeleChoice analyst Liza Henderson divides high-capacity core routers into two subcategories: terabit routers that have higher port densities on a single chassis, and terabit-plus routers that can expand ever further by connecting multiple chassis together.
Some question the need for multiple chassis. "What Juniper and Cisco do is to build a product to hit a broad market. A single chassis will be OK for most operators," says Mark Seery, RHK's program director for switching and routing. "A multi-chassis product can only hit a narrow market."
Seery points out that although Juniper has indicated that it plans to eventually support a multi-chassis design with its recently announced T640 model, company executives claim that customers have not yet asked them to commercialize the multi-chassis version. Seery adds that if silicon development continues to progress, that could minimize the need for a multi-chassis design.
Others say that those who downplay the need for a terabit-plus product are not looking at the issue in the right way.
"If you scale an individual POP beyond the capacity of a single router, it's very obvious," says Hyperchip's Norman. "Others have the problem, but instead of 20 routers in a room trying to be one big router, the problem is spread out so they don't realize it."
Norman argues that a high-capacity core router could pave the way for a new network architecture that would reduce the number of hops that data must make as it travels across the Internet. Currently, most networks have at least three tiers of routers. These include edge routers, to which customers connect; aggregation routers that combine multiple lower-speed connections from edge routers onto a smaller number of higher-speed connections; and core routers, which take traffic in from aggregation routers and send it to other core routers (and vice versa). Considering that traffic must pass through each type of router on its way in and out of the core, its easy to see how the hops can quickly add up.
Currently, Norman says, the average packet goes through 10 routers within the US. "That means you're tying up 10 router ports," he says. "That could drop by a factor of five or six. It won't happen overnight, but we can evolve toward that by putting in a regional super core with five to 10 fully meshed routers around the country. That reduces hops to three or four."
Avici advocates a similar approach. "You can get rid of the aggregation layer if you accommodate the feature set associated with it and provide OC-3 to OC-192 connections on a single platform," says Esmeralda Swartz, Avici director of marketing. Avici recently added a new feature set to its core router to support aggregation-layer functionality such as peering and security.
Whether they believe in the need for the highest-capacity core routers, industry observers do agree on one thing. The most challenging task in developing such a product is the software to enable multiple chassis to function as a single router. Indeed, some question how well such an approach will work.
"Can a router do packet forwarding at 19 terabits?" asks Fred Harris, vice president of research, architecture and design for Sprint. "I'd be skeptical."
Others question whether established vendors will have the wherewithal to undertake such a complex software development program - and how easy it will be to upgrade existing operating systems. Some question Juniper's use of a central hub for its planned multi-chassis offering. Others question whether Cisco's IOS operating system, originally developed to serve an enterprise environment, can be made to scale to required levels. Cisco has not yet announced plans in the high-capacity core router space, although several sources say the company is working on something.
If Cisco finds that it has to re-architect its product, that company, instead, may opt to buy one of the startups, speculates TeleChoice's Henderson. "Some of these startups could be positioning themselves to be bought by Cisco."
To each his own
Vendors show much less consensus in their thinking about network operators' other hot button - core router reliability. Virtually every developer claims to have something unique there, and virtually every one of those claims requires a laborious explanation.
Pluris, for example, touts transparent control plane failover, while Hyperchip talks about the ability to do in-service upgrades to route processors, and Avici sings the praises of hitless server failover.
One thing Avici has going for it, though, is that AT&T already has deployed its product and claims that it has displayed six-9s reliability, meaning that it functions without outages 99.9999% of the time. That's a cut above the five-9s reliability that makes a product carrier-class. In comparison, most sources agree that the vast majority of today's public IP networks operate at less than five-9s reliability.
Avici's Swartz says that AT&T obtained six-9s reliability with a redundant router architecture. Without redundancy, she says, the product can support five-9s reliability - a capability that has been verified by BT Exact, the testing firm that was spun out of British Telecom.
"Operators have set up redundant routers to get the network to stay up," Swartz says."The way they built the voice network was nothing like this." Ultimately, Avici hopes to eliminate the need for service providers to design their networks with redundant routers - which, in turn, should improve IP network margins. "We're getting to where gross margins on IP are the same as for 'cash cow' services," says Swartz.
Clearly, much is riding on the next round of core router RFPs for both startup and established vendors, and a key question on everyone's mind is how open service providers will be to using a non-traditional vendor.
"There's always an advantage to the incumbent, mostly in provisioning, billing and in the back office, but that's not insurmountable," says Dave Garbin, senior director of network strategy for Cable & Wireless. "We may qualify more than one vendor - maybe one for innovation and one for stability."
C&W expects to issue an RFI for a high-capacity core router later this year and to develop a "short list" of potential vendors by October, says Chris Liljenstolpe, C&W senior director of network technology. At that time, the company would buy a small number of routers and deploy them in the network for about three months before issuing an RFP.
"The RFP will be demand-driven," Garbin says. "We're waiting to see if there is a return to normalcy." Garbin adds that the most likely timeframe for issuing the RFP would be early in 2003, with the goal of putting some devices in the network within three months and a full rollout over a six month period.
Garbin and Liljenstolpe agree that the multi-chassis concept has appeal. But they will also strive to maintain a capability that's built into their current network architecture. C&W currently uses a two-level hierarchy at the core, explains Garbin. "In front of the core, we use a traffic engineering layer that performs Layer 2 switching." Ideally, C&W would like to see a multi-chassis system that could behave like multiple routers so that the network operator could handle its core routing and traffic engineering requirements in the same device. To support this, Garbin insists that the two different routing tables must be kept separate.
Garbin declines to offer specific feedback on any of the high-capacity core router vendors, but does provide some overall observations. "A number of folks made assumptions they shouldn't have made," he says. One such incorrect assumption was that carriers can easily support 23-inch equipment bays. "Those who are at 23 inches or wider will be at a disadvantage in comparison with those who have 19-inch products," says Garbin.
Another concern relates to how multiple chassis interconnect. "Some vendors selected early on to do copper only, rather than a copper/optical hybrid or all-optical - and that limits how far away you can put the different chassis," Garbin says.
Giving startups a shot
Another network operator that appears willing to consider new vendors is Genuity. That company recently looked at high-speed backbone routers but is holding off on making a purchase decision, says Steve Blumenthal, Genuity senior vice president and chief technology officer. "We want to link multiple chassis together without having to burn up expensive OC-192 optical ports, and several new routers allow for a flexible extension of the backplane. We ran an RFI last year and based on the responses, selected several to bring in for evaluation. Typically our RFI process is pretty thorough. We typically select a few to test and then do an RFP at the very end, mostly focused on terms, conditions and maintenance. I'm not sure when we would issue an RFP. We may extend the looking and testing phase and then make a purchase decision late this year or early next."
In making its decision to use Avici, AT&T also has demonstrated a willingness to use non-traditional vendors. "Some large vendors are large now because we bought from them," Leighton says.
AT&T will look again at its core router needs around the end of this year, Leighton adds.
In comparison with C&W, Genuity and AT&T; Sprint appears to be the least likely of the major Internet backbone operators to make a change from its existing router vendor. The company has a "go-to-market" alliance with Cisco, through which the two companies exchange product development information.
Tellingly, when asked what the company would look for in a core router, Sprint's Harris says, "There are a lot of things we would like to see done that we've discovered through our internal monitoring. We've been able to pinpoint phenomena that need to be corrected. In our alliance with Cisco, we're feeding that information back and we're looking for improvements in the next generation of routers, such as the ability to be more resilient. To support real-time applications, for example, we need to minimize the time it takes to converge the network if there is a link outage."
Sprint also expects to issue a core router RFP within six to 12 months, says Harris. "If the right product is available, we would deploy it in about 18 months," he adds.
The other of the largest Internet backbone operators, WorldCom, declined to be interviewed for this article. But RHK's Seery offers his take on that company's core router plans. "They definitely want a multi-chassis product," says Seery, "but haven't started trials yet."
Adds Nancee Ruzicka, program manager for The Yankee Group, "WorldCom uses Cisco and Juniper, and it was the first and biggest fan of Juniper. I don't have any evidence that says that won't continue."
Although operators of existing Internet backbone networks are the most obvious core router prospects, some developers are seeing interest from another corner. The RBOCs are exploring their options in that area in preparation for receiving widespread approvals - or legislative permission - to offer interLATA data services, says Norman of Hyperchip.
He claims that the RBOCs will acquire existing backbone networks and will look to make those networks operate more efficiently by using a network architecture that eliminates the aggregation layer. RBOCs likely will need a platform to support that vision in 2003, Norman says.
As the preceding story on RBOC metro networks reminds us (page 36), however, it's important to recognize that RBOCs may simply be exploring their options. History suggests that only a fraction of Bell company RFPs generate actual purchases. The established Internet backbone operators, on the other hand, say that the majority of their RFPs will result in purchases, usually within six months. |
