Well look here...the most extensive RAS technology evaluation ever, and ASND and COMS take turns kicking CSCO's porky bloated obesity all over the place!
Hmmm...I believe it's settled then. If I don't begin to establish my short position next time CSCO is 85+, I might not get another chance before doughboy-networker meets Jenny Craig! ;o)
Style Pts.
--------------------------------------------------------------------------------------------
RAS benchmark by data com magazine:ASND vs COMS,CSCO,BAY
By Deval Shah and Helen Holzbaur
Bigger, But How Much Better?
RASs have the brawn, but tests reveal that they don't all
scale the same
Modem prices are plummeting. Corporate road warriors and
telecommuters have got HQ looking like a ghost town. And
Internet traffic keeps climbing.
What's it all mean to net managers? They've got more remote
users than they can handle.
Vendors say they can tame these troubles with the latest
generation of remote access servers (RASs). Most of these
bulked-up units handle 100 calls or more per chassis. They're also
supposed to be pretty smooth operators, boasting dedicated
processors on each module so that performance scales linearly as
modems are added.
Impressive claims. To check out how real they are, Data
Communications and National Software Testing Laboratories Inc.
(NSTL, Conshohocken, Pa.) invited 20 vendors to take part in the
largest public evaluation of RAS technology ever conducted. Seven
vendors accepted the challenge, submitting gear that can field up to
200 concurrent dial-in calls from our gargantuan test bed (see
Table 1). Besides appraising scalability, we also assessed each
box's ability to deal with day-to-day management chores.
(MORE INFO: Lab Test Vendor Participants)
When we'd sorted through all the data, it became clear we had one
of those good news/bad news stories on our hands. The good news
is that all vendors have done an excellent job enhancing their
management tools (although setup is still an issue).
The bad news? Not all RASs scale identically, even under the
moderate traffic load we offered. Some couldn't complete calls from
even 100 simultaneous clients. Several came with faulty hardware.
That said, some products were standouts. The fastest was the
Total Control Hiper Access System from 3Com Corp. (Santa Clara,
Calif.), a beefed-up version of the Total Control hub that now
accommodates up to 450 calls per chassis. Also out front was the
box that pioneered the big RAS market: the MAX TNT from
Ascend Communications Inc. (Alameda, Calif.)-one of the easiest
units to set up. Both boxes earned Tester's Choice awards.
Getting A Grip
One thing that would help almost all these products is a more
intuitive setup routine. Granted, this is only used when a RAS is
installed, but too many boxes bury key WAN configuration
parameters deep inside various menus. For example, a RAS must
use the same calling parameters as the telco switch, yet even
vendor engineers sometimes found it difficult to line everything up.
One notable exception is Ascend's MAX TNT, which is as close to
a plug-and-play box as any we tested.
Once the RAS is set up, it requires simple yet comprehensive
management capabilities to look after hundreds of dial-up users.
Fortunately, vendors have done their homework here: All the boxes
we tested offer easy-to-grasp graphical tools.
Several vendors are starting to work the Web into their
management. For instance, the Commplete Communications Server
RAS from Multi-Tech Systems Inc. (Mounds View, Minn.) uses a
Web interface for most configuration and monitoring. And Shiva
Corp. (Bedford, Mass.) also has a Web-based monitoring utility.
We didn't just gawk at the glitzy interfaces, though. To see how
useful these management tools really are, we came up with several
scenarios based on real-world needs (as identified by managers of
large corporate networks). Each RAS was run through the range of
events and graded on a scale of 1 to 5, with 5 being best (see "Test
Methodology"). Although our ratings are admittedly subjective,
they are based on first-hand experience.
We broke the scenarios into three general categories-remote
management, troubleshooting, and fault tolerance-and created
specific problems in each. For a device to score a 5, it had to
diagnose and fix a problem quickly and easily. A score of 4 meant
we could spot and correct a problem, but only by spending a
significant amount of time with the management tools. A 3
indicates we could identify the problem but not correct it. A 2 meant
we could gather information about the problem but not diagnose it
with certainty. And a 1 meant we couldn't spot the problem.
Reaching the RAS
In our first remote management scenario, a hypothetical net
manager receives a spate of phone calls from irate end-users,
suggesting the RAS may be down. The net manager has to dial into
the RAS's management port, reboot the unit, and verify that it
returns to service. All devices breezed through this event with
perfect scores (see Table 2).
Devices fared nearly as well in the next event, which called for the
net manager to remotely add and delete user accounts on the RAS.
Here we found the Ascend and 3Com interfaces slightly more
challenging to navigate-a caveat that also holds true for the Cisco
AS5300 Universal Access Server from Cisco Systems Inc. (San
Jose, Calif.). All the other RASs earned perfect scores. Notably,
the RAServer 2900 from RAScom Inc. (Salem, N.H.) employs
on-board versions of Windows NT (one per module) for user
account maintenance.
In the last remote management event, an attacker is suspected of
using an authorized end-user's account to trespass on the
corporate network. To respond, the net manager had to do three
things: pinpoint the user, force a disconnect, and prohibit that user
from reconnecting. All the devices could pinpoint with no problems,
except Multi-Tech and RAScom, which each earned a 4. Cisco and
Multi-Tech scored 4 when it came to forcing a disconnect, and
Ascend, Multi-Tech, and 3Com rated 4 for blocking a reconnect. All
the other devices earned a 5 in these scenarios.
Looking for Trouble
The next scenario involved two basic troubleshooting
tasks-identifying faulty modems and taking them out of service. We
evaluated ease of completion both for net managers on-site with
the RAS and for those at remote locations. Almost all products
aced the on-site test. RAScom's device scored a 4. Even though
the unit uses Windows NT for many functions, we had to switch to
its command-line interface to manage some hardware parameters.
We would have preferred handling everything from the same
interface. When it came to tackling the same tasks remotely, all
devices earned top ratings except for the Bay Networks Inc. (Santa
Clara, Calif.) and RAScom boxes, which both scored 4.
The final scenario involved a local power brownout, which we
simulated by reducing power to the RAS. We did this in one of two
ways: Either cut power to one of several power supplies in the
same chassis or cut power to one chassis among several configured
to appear as one logical entity. The goal in both cases was to
determine what effect cutting power would have on the rest of the
modems and chassis. We also evaluated how easily power could
be restored and whether turning the power supply back on would
have any effect on active connections. All these scenarios assumed
our hypothetical network manager was on site; we also evaluated
what a net manager at a remote site could do to diagnose and fix
the problem.
Power Play
All products were able to take a power hit with no effect on other
chassis (or on other modules within the same chassis). Bay's RAS
was the only device for which cutting power affected incoming calls.
Its setup associates phone numbers with specific modules, so calls
to numbers on a failed module aren't shifted to other modules.
When it came to restoring power, all devices except Bay's and
Shiva's garnered perfect scores. The hitch with those two
products? Neither offered redundant power supplies in the units we
tested (Bay's chassis does support this feature). Configurations
with multiple chassis handled active connections flawlessly-while
calls would drop on the chassis with the power hit, calls to other
chassis weren't affected and incoming calls were rerouted to these
chassis.
We also evaluated remote management of power supply problems.
All RASs except two made short work of remotely enabling another
power supply-the RASserver and the LANrover Access Switch from
Shiva. The RAScom unit doesn't use Windows NT for core
hardware functions; it uses its command-line interface instead. The
Shiva box has just one power supply per chassis. In our tests, we
could detect a loss in power remotely, but we couldn't power-cycle
the unit.
RAS Blasters
With our management testing out of the way, we turned to the main
event: scalability. Given that most devices tested handle hundreds
of concurrent calls, the big question for us was whether throughput
would scale linearly. In other words, would caller No. 200 get the
same high throughput as caller No. 1?
To answer that question, we built one of the largest test beds
we've ever used-including more than 200 modems; a telco switch
emulator with up to 11 channelized T1 connections; a fast Ethernet
switch; a Web server hanging off a full-duplex fast Ethernet
connection; and, most important of all, a custom-developed test
application that generated Web requests from up to 200 clients.
One thing we didn't use was 200 individual client PCs (the other
equipment generated enough heat as it was). Instead, we equipped
four large systems (dual-CPU 200-MHz Pentium Pros, each with
256 Mbytes of RAM) with serial port extenders so that each could
generate up to 64 calls.
Ascend argues that individual PCs would offer traffic at a higher
rate than the virtual clients we used. The vendor says Windows
NT does not scale up to this level. It has a point: Individual clients
would offer a larger aggregate load than did our test bed, which
would translate into higher throughput.
Another red flag that Ascend raised had to do with our use of four
separate clocks on the telco switch emulator, which could lead to
timing instabilities. The problem is that the device under test has to
spend time synching up all the lines, thus potentially putting a drag
on performance. We tested the Ascend and Cisco boxes using their
own clocks. In all other cases the clock was furnished by the telco
emulator.
We deliberately chose 33.6-kbit/s modems for our test bed
because there isn't yet a unified specification for the newer 56K
technology (see "56K? No Way," August 1997; data.
com/lab_tests/56k.html). Had we used modems with either of the
two competing, incompatible 56K chip sets, we wouldn't have been
able to test at least some RASs based on the other spec.
Even with the 33.6-kbit/s modems, though, we often saw lower
connection rates-typically at 28.8 kbit/s or 26.4 kbit/s. Some vendors
grumbled that their RASs would fare better had we used another
vendor's modems rather than the ones we used, from Microcom
Inc. (Norwood, Mass.). We don't put much stock in that
contention, since there's no one perfect modem. Besides, even
within a single company, callers are likely to have multiple types of
modems from a variety of vendors.
Heavy Traffic
To generate traffic, we used Intermark, NSTL's custom test
application. In the version developed for this test, the dial-up
clients are all Web surfers, each requesting a file from a Web
server on a fast Ethernet segment. The Intermark script has each
client request the file and then records effective throughput for all
clients. The numbers Intermark records reflect application-layer
throughput; it doesn't record throughput on the wire, which is by
definition somewhat lower because of the overhead added by
packet headers. To ensure we measured all the bits on the wire, we
recorded throughput with a protocol analyzer.
This isn't to say measurement on the wire is better. While
application-layer measurements omit packet overhead, measuring
on the wire doesn't take into account the effect of dropped packets
and retransmissions. Thus, since even retransmit requests
consume bandwidth, the throughput we recorded may be somewhat
higher than what an end-user sees.
Linear Equations
To find out if RAS throughput scales linearly, we took
measurements for 1, 50, 100, and 200 dial-up users. Ideally, our
measurements should climb at a perfect 45-degree angle as we
add clients: two clients should have exactly twice the throughput of
one, four twice two, and so on. To some extent, scalability is a
function of RAS design: Products built around a single CPU don't
scale well. The better approach is to put at least one CPU onboard
each modem module, so that processing power grows as modems
are added.
Besides exploring how well the RASs scaled, we were naturally
interested in measuring how fast the modems would run. Figuring
out the theoretical maximum throughput on our test bed wasn't
exactly straightforward, though. Ordinarily, we'd simply take the
nominal rate of each modem (33.6 kbit/s) and multiply it by the
number of channels per RAS. But in this case, there are a couple of
other considerations. First, because most modems connected at
28.8 kbit/s or 26.4 kbit/s, we chose a middle ground of 27.6 kbit/s as
the nominal connect rate. Second, the file we used was
compressible by a ratio of about 2.8:1, which means the theoretical
maximum on our test bed should be around 77 kbit/s.
The size of the file also was an issue. Modem tests-including the
standard ones recommended by the Telecommunications Industries
Association (TIA)-involve trans- fers of relatively small
(30-kbyte) files. The problem with this approach is that the
compression dictionaries of most modems can cache the entire file,
resulting in artificially high transfer rates on subsequent test runs.
We countered this effect by using a 351-kbyte file, which forced the
modems to compress all data and resulted in somewhat lower
numbers than those often quoted by vendors.
Kicking RAS, Taking Names
The most difficult part of the test- for almost all RASs-was
sustaining 200 simultaneous calls. Ascend's MAX TNT had the
fewest problems: Clients simply made the calls and we offered the
test data. Other vendors had all sorts of configuration problems,
ranging from malfunctioning serial cards to routing loops to faulty
Radius (remote authentication dial-in user service) routines.
In fact, time constraints prevented us from running performance
tests on RAScom's RAServer 2900. And two products couldn't
complete all our tests: Multi-Tech's Commplete Communications
Server and Shiva's LANrover Access Switch. In Multi-Tech's
case, we couldn't offer traffic from any more than 50 clients,
probably because of a faulty WAN hardware module. In Shiva's
case, we could offer traffic from a maximum of 144 dial-up clients.
Despite repeated attempts by Shiva's engineers to diagnose the
problem, we can't say for certain why the system wouldn't work.
We suspect one or more of the chassis used may have been faulty,
but time constraints prevented us from checking this out.
As for most of the other RASs, performance scales in a fairly linear
way-once they're up and running (see Figure 1). For almost the
entire field, all products appear to scale well from 1 to 50 clients.
The exception is 3Com's Total Control Hiper Access System,
which had relatively low aggregate throughput for 50 clients but
more than made up the deficit with 100 or more sessions.
Aggregate throughput for most devices tailed off somewhat when
moving from 50 to 100 dial-up clients. The gain in throughput for
the Ascend and Cisco devices is flat or close to it, and Shiva's
LANrover picks up only moderately. Shiva's device tops out at 72
clients, so doubling up the chassis may have given the LANrover
an edge. But the single-chassis theory doesn't hold in all cases:
3Com's Total Control hub, which can accommodate up to 450 calls
in the same chassis, led the field with the highest aggregate
throughput in the 100-client tests.
When we pushed the number of calls still higher-to 200 clients-all but
a couple of products cried uncle. No product came close to our
scaling ideal: aggregate throughput for 200 clients exactly 200
times greater than single-client throughput. But 3Com's Total
Control did the best, delivering higher aggregate throughput than
any other product. Ascend's MAX TNT was next highest, with
respectable if not totally linear scalability. Cisco's AS5300 trailed
these two, and Shiva's LANrover-although turning in the
second-highest number in the test-was unable to complete with 200
clients. What's more, the aggregate throughput of the Bay box at
200 clients was actually lower than at 100 clients.
We should also note that per-client throughput was about 47 or 48
kbit/s for all devices-far lower than the theoretical maximum of
around 77 kbit/s. This may be a function of impairment introduced
by the telco switch emulator. Some vendors claimed that it was our
modems; Cisco pointed to our use of a single server as a
bottleneck. Others said it was because we used up to 50 clients per
NT test machine. We ran the same test using a standalone 3Com
Sportster modem and a standalone PC; throughput climbed to
approximately 56 kbit/s.
data.com
Lab Test Vendor Participants
Ascend
With support for up to 672 callers per chassis, Ascend's MAX TNT loads a lot of access into
a small box. It also was the easiest product to configure and monitor, especially when using
the vendor's Navis Access management software. This tool shows at a glance which calls are
coming in on which T1s, how many calls the unit is fielding all told, and many other
parameters. Ascend indicates that its aggregate throughput would be significantly higher
when callers are on individual PCs.
Bay
Like Ascend's MAX TNT, Bay's 5399 MSX Concentrator supports up to 672 calls per
chassis. And Bay is the only participating vendor with plans to support both X2 and K56flex
technologies until there's a unified 56K spec (Bay does X2 now and says it will add K56flex
this month). But we ran into configuration difficulties with this device-including a change in
routing tables during a test iteration. As a result, the unit under test was unable to service
200 callers as quickly as it did 100.
Cisco
On the plus side, Cisco's AS5300 will look very familiar to anyone versed in the IOS software
used by all of Cisco's other products. The vendor has added some RAS-specific extensions to
IOS, such as setting up user accounts and viewing various configuration parameters. And
even though it took three chassis to handle our 200 callers, those devices took up less space
than some of the single-chassis solutions. On the minus side, Cisco was outperformed by
other devices in our test. Cisco rightly points out that performance was to some degree a
function of our test application. The company has achieved wire-speed results in other tests.
Multi-Tech
The most distinctive feature of Multi-Tech's Commplete Communication Server is its
management software, Multicom Manager, which allows virtually all configuration and
monitoring to be handled through a Web interface. Unfortunately, the hardware the vendor
supplied wouldn't cooperate with its software. Something, probably faulty serial cards,
prevented us from completing tests with more than 50 callers.
RAScom
RAScom has a novel twist with its RAServer 2900: The device incorporates one or more
Windows NT servers running Microsoft's Remote Access Service software. The use of NT
makes it a snap to define and manage user accounts (all reside within NT domains), and it
also allows the RAS to act as a Web or FTP (file transfer protocol) server. Unfortunately,
however, the device's hardware is managed through a separate command-line utility, which is
somewhat less convenient. Time constraints prevented us from running performance tests on
this device.
Shiva
Even though it's one of the oldest players in the RAS market, Shiva keeps coming up with
new tricks for its LANrover, including Web-based monitoring tools and the ability to define
virtual groups of phone numbers. But what was more notable about Shiva's LANrover this
time around was what it didn't do-handle 200 simultaneous callers. Despite the best efforts of
Shiva's engineers, we were unable to sustain any more than 144 calls. We believe some sort
of hardware problem with our particular test units is probably to blame. Even at 144 callers,
however, Shiva still managed to post the second-highest throughput of any device tested.
3Com
High port density and strong scalability are the hallmarks of 3Com's Total Control Hiper
Access System. The new Hiper modules can handle up to 450 calls per chassis, and our test
showed this device scaled up linearly as calls were added. The vendor also deserves kudos for
its Total Control Manager software, which allowed us to detect hardware failures, run
diagnostics, and perform virtually any other management function.
Top Performers
Ascend
With 672 ports per chassis, Ascend's MAX TNT is one of the biggest and best RASs
available. It also was a breeze to set up for our evaluation and turned in a decent performance.
3Com
"Hyper" is the right word for 3Com's Total Control Hiper Access System. Besides being
the fastest device in the test, 3Com's unit also offers up to 450 ports per chassis and clear,
comprehensive management software
RAS Mgmt Scorecard
data.com
Comparing RAS Scalability
data.com
Test Methodology
data.com |
