A Move From One Big Server To Many. Cisco plans to decentralize its Web site architecture to increase performance and reliability
iw.com
By Todd Spangler, July 13, 1998
The architects of Cisco Systems Inc.'s Web site
had a simple strategy in mind when they worked
out the blueprints for its infrastructure: the
bigger, the better.
"It was a design philosophy to build it once, and
build it big, then not change it," said Mark
Tonnesen, Cisco's IS director of customer
advocacy.
The Cisco Connection Online site, originally
launched back in 1993 primarily as a source for
technical support and product information, has
grown beyond any of its creators' imaginations.
Often cited as perhaps the largest-scale
e-commerce site on the Net, Cisco Connection
Online today tallies an average of $11.2 million
in sales each day, which accounts for about 55
percent of Cisco's overall revenue, according to
the company.
Although the online sales are one of Cisco
Connection Online's most visible components,
its main functions continue to be customer
support, marketing, and training, said Tonnesen,
who oversees management of the site's
infrastructure with Dale Seavey, senior manager
of IS operations, and Al Etterman, IS director
of network operations.
At A Glance
Company: Cisco
Systems Inc.
Headquarters: San Jose,
Calif.
Bandwidth: DS-3 (45
Mbps) in San Jose from
GTE Internetworking;
DS-3 in Raleigh, N.C.,
from Digex Inc. E-1 (2
Mbps) connections in
Sydney, Tokyo, and
Amsterdam
Average bandwidth
utilization: 15 Mbps
during business hours
Web server: Sun
Enterprise 6000 with 24
processors and 10
Gbytes of RAM, running
Solaris 2.5.1 and
Netscape's Web server
Storage: About 750
Gbytes, mirrored on two
Sun disk hot standbys; a
third backup is stored
offsite in Sacramento,
Calif.
As the site's traffic has grown over the past five years--at a rate of about 90
percent per year, now averaging about 15 Mbps of bandwidth utilization
during business hours--Cisco's approach has been to upgrade its main Web
server with bigger and faster hardware.
The site originally was running from a Sun SPARC 20 workstation on an
engineer's desktop, Tonnesen said. About a year and a half ago Cisco
upgraded its Web server, which is located at the company's headquarters in
San Jose, Calif., to a Sun Enterprise 5000 with 14 processors and 4 Gbytes
of RAM.
Though the full capacity of the E5000 wasn't close to being completely used,
Cisco earlier this year moved the site onto an Enterprise 6000 server with 24
processors and 10 Gbytes of RAM, running Solaris 2.5.1 and Netscape's
Web server. "The last thing I want are capacity and response problems,"
Seavey explained.
DIVIDING THE HORSEPOWER
Cisco is now reconsidering its centralized strategy of bigger and faster servers,
with plans to migrate Cisco Connection Online's servers to a more distributed
architecture over the next 18 months.
"We're looking at breaking up the site into subcomponents, using Cisco
LocalDirector [for distributing traffic among several servers at a single
location] and Cisco DistributedDirector [to load-balance among multiple
locations]," Tonnesen said.
By the end of October, Cisco Connect Online will have a Sun Enterprise
6500 server up and running at the San Jose center to provide failover for the
primary E6000 server--a very valuable insurance measure for a site
processing million of dollars in sales every day. Still, even without that
safeguard, Cisco claims its site currently has at least 99.98 percent host
uptime and 99.99 percent Internet connectivity uptime.
In addition, the company plans to roll out mirror sites at several locations
around the globe, beginning in the first quarter of 1999 with a Raleigh, N.C.,
facility, which has a DS-3 (45 Mbps) Internet connection provided by Digex
Inc., as well as a 20-Mbps ATM link from Sprint and several backup T-1
(1.5 Mbps) lines connecting it to Cisco's San Jose data center. From its main
campus Cisco has another DS-3 Internet connection, provided by GTE
Internetworking, which operates a point of presence in nearby Palo Alto.
By next summer, Cisco hopes to have similar mirror sites in Amsterdam and
Japan up and running, followed by one in Sydney. Each of the non-U.S. sites
is connected via E-1 (2 Mbps) lines. Cisco already mirrors a small portion of
its Cisco Connection Online material for international customers and
employees through Digital Island, a Honolulu-based ISP that provides
guaranteed Internet bandwidth to 14 countries.
Tonnesen said Cisco is distributing its servers geographically to provide better
response time to international users, not to provide failover sites in case the
San Jose facility goes entirely dark. "By moving things out to mirror sites, we
can give customers a consistent response wherever they are," he said.
The site's e-commerce infrastructure, however, will not be replicated.
Transactions will still be processed through Cisco's central site in San Jose,
Tonnesen said. Cisco's Web server sits outside the network firewalls, but the
site's commerce activity involves several security components. When
customers place orders they must be authenticated via a Kerberos software
token, then tunnel through two firewalls using an SSL-encrypted session. The
order goes immediately into an e-commerce system running on Oracle
Applications, from which customers can view their order status over the Web.
TRACKING RESPONSE TIMES
To monitor performance, Cisco uses an automated system that checks the
servers in five-minute intervals. The site's engineers also work with Keynote
Systems, an Internet monitoring service (which provides the data for Internet
World's Web Performance Index), to analyze how well Cisco Connection
Online can be accessed from various parts of the country, Tonnesen said.
The company plans to upgrade its San Jose connection to an OC-3 (155
Mbps) connection in September, primarily to reduce the effects of bandwidth
spikes, Seavey said.
"We saw a lot of spikes at different times of day, and we wanted to reduce
the impact," he said. "It's hard for us to predict the kind of traffic we're going
to see minute to minute." The Web site, which serves about 750 Gbytes of
data, is currently mirrored on two Sun disk arrays that are on hot standby,
which means they will kick in instantly if the main hard disk on the Web server
fails. A third copy of the site is backed up daily and trucked off-site to a
storage warehouse in Sacramento, Calif.
Cisco uses a symmetric replication system that pushes out content from the
central repository to its mirrored sites at the same time. Content management
for Cisco Connection Online used to be handled by a custom-written
application, but the site now uses Documentum's document management
system to process several hundred thousand documents.
"We went out and invented a lot of things ourselves to run the site," Seavey
said. "When we first started this thing, nobody had even heard of HTML."
The Cisco network connections, as one would expect, consist entirely of
Cisco products, including four Cisco 7200-series routers at each of its U.S.
sites. All the infrastructure is maintained by a full-time staff of five Webmasters
and three system administrators, although three teams--primary customer
support, marketing, and network commerce--devote a total of 50 people to
developing applications for the site.
At any given time, about 190 projects are in the works for the site, according
to Tonnesen. "It's a very dynamic site," he said. "We find more and more uses
for this site all the time."
RELATED STORIES:
Keywords: infrastructure
Date: 19980713
BROWSERS
WIN 95/98
PLUG-INS
WHAT'S NEW
TOP 25
CWSAPPS
APP SEARCH
WIN NT
WHAT'S COOL
Copyright 1998 Mecklermedia Corporation.
All Rights Reserved. Legal Notices.
|
