Network Pressure -- E-Commerce Has Made Network Availability The Highest Priority And Downtime More Costly Than Ever.
InformationWeek, August 16, 1999 p44
Author
Riggs, Brian; Thyfault, Mary E.
Summary
The rise in electronic commerce means system downtime can cause a company's stock price to fall, as
happened to online auction service eBay, or lead to a government investigation, as happened after
several online stock-trading systems failed. IT departments have many new tools and techniques at their
disposal to keep their organizations' critical systems running. Redundant servers, server farms and
communications links are combined with caching, load balancing and policy networking to protect
against system failures. News service CNN connects each of its two data centers to a different power
grid, and each center has connections through three local telephone companies and several
long-distance companies. CNN Internet Technologies VP Monty Mullig says the company could lose one
link and still get through an average peak. CNN's site has sufficient bandwidth and server capacity to
accommodate three times its average load, which is vital for handling peaks due to breaking news.
Barnesandnoble.com's use of multiple server farms, 1-800-Flowers.com's load balancing strategy, and
the approaches of other electronic commerce sites are described.
Full Text
Downtime is a dirty word for it managers. their goal in building sys- tems and networks has always been
to ensure that bandwidth and computing resources are available when needed. But the growth of
electronic commerce has raised the stakes. The penalty for system crashes and network failures is
greater than ever, and the loss of sales and customers is just the beginning. Now, downtime can result in
falling stock prices, as happened when eBay Inc.'s online auction site crashed in June, or in a government
probe, as when the Securities and Exchange Commission launched an investigation into a series of
crashes involving several online stock-trading systems.
"In the past, if a company's computer system went down for 10 minutes, that would not affect the worth
of its stock," says Norman Dee, director of network services at 1-800-Flowers Inc., which sells floral
arrangements and gifts over the Internet and operates an extranet for business partners. "But
corporations can no longer afford the embarrassment of their commerce servers not being available.
Because of this extraordinary exposure to doing business over the Internet and extranets, companies
such as ours have to ensure that there is constant availability."
To meet that requirement, IT managers are employing a host of new devices, techniques, network
designs, and services to improve the performance and availability of their commerce systems, networks,
and applications and reduce the likelihood that their sites won't be working when a potential customer
comes calling.
Risks And Rewards
The risks and the rewards have never been greater. Companies spend an average of $1 million to launch
an E-commerce site, with many costing between $6 million and $20 million, according to Forrester
Research. Whether they're selling products, services, information, or advertising, these companies are all
trying to grab a piece of a rapidly growing pie. More than $18 billion worth of merchandise will be sold
via E-commerce in the United States this year, a figure that's expected to jump to nearly $53 billion in
2003, according to Forrester.
An obvious tactic for ensuring availability is redundancy-redundant servers, server farms, and
communications links. When redundancy is combined with other products and techniques such as
caching, load balancing, and policy networking, a company can make great strides to ensure its systems
and networks are always available, even if pieces should fail.
A prime example is the Web site operated by CNN, its global operation for news gathering and
distribution. CNN operates two data centers, each connected to a different power grid. Three local
telephone companies-MCI Metro, MFS, and BellSouth-have separate connections into each data center.
Long-distance companies also have high-speed links into each data center: two 155-Mbps pipes from
UUnet, one from Sprint, one from AT&T, and one on order from Qwest Communications, as well as a
45-Mbps circuit from GTE Internetworking.
"We could lose one and get through an average peak," says Monty Mullig, VP of CNN Internet
Technologies in Atlanta. CNN has designed its site, which generates advertising revenue, to have enough
bandwidth and server capacity to handle three times its average load. That's important for a site that
experiences unexpected spikes in traffic because of breaking news events, yet must always be available,
because news breaks around-the-clock.
CNN makes its news available over the Internet in many formats, including Web pages and streaming
audio and video. To get its content to users more quickly, CNN recently contracted with startup Akamai
Technologies Inc. to cache-or store-copies of its content on up to 600 servers located on the networks
of 20 Internet service providers. Akamai also monitors traffic on the Net and routes traffic over
communications links and to servers that are experiencing the least amount of congestion.
Because many of its customers want to hear audio reports or see video clips, CNN also uses aggregators
of streaming media to ensure that content is available when traffic spikes. During President Clinton's
impeachment trial earlier this year, requests for video downloads soared 100-fold. "It's tough enough to
build capacity that is three times our normal mode," Mullig says. "We couldn't deliver the video and serve
up our regular content."
CNN sends its video content to Intervu Inc. and its audio content to Broadcast.com Inc. Intervu has
servers located on the networks of Level 3 Communications Inc. and DBN Corp., and Broadcast.com has
servers on the Level 3 network, providing alternative paths to stream CNN's audio and video news
reports over the Internet.
That may not be enough. CNN is considering setting up a third data center at a carrier's site to improve
the site's performance and availability even more. But Mullig vows to keep most functions in-house. "We
think it's a competitive necessity to know the latest and greatest technology," he says. "Network
providers just know how to provide facilities, but they don't know how to engineer really big sites."
Mullig won't discuss the specific costs of running a multimillion-dollar Web site such as CNN's, but
analysts say the site is one of the best in making use of technology and techniques to ensure availability.
"They understood right up front everything about reliability, scalability, and the handling of events," says
Al Lill, an analyst with Gartner Group. "You can never throw enough money at these things."
Another approach to redundancy is multiple server farms. Barnesandnoble.com LLC, the online arm of
the nationwide bookstore chain, operates a cluster of commerce servers in New York and maintains
replicated servers at hosting facilities operated by America Online and Cable & Wireless plc.
Barnesandnoble.com uses Cisco Systems' DistributedDirector, a load-balancing system that starts at
$17,000, to direct traffic to commerce servers that can best process transactions and serve up requested
information, says Robert Dykman, VP of technical infrastructure. Load-balancing devices intelligently
route traffic to servers or server farms that have the most resources available to process the
transactions and keep traffic from hitting servers that are overly busy or that have failed.
Other load-balancing products work on the local level, routing traffic and maximizing server availability
within a server farm. The Motley Fool Inc., a provider of online investment information, installed a Big/ip
load-balancing appliance from F5 Inc. when it recently updated its commerce site. The $9,990 device
provides redundancy and fault tolerance and makes the commerce site easier to administer and
maintain, says Dwight Gibbs, Motley Fool's chief technologist. "If we want to do maintenance on one box,
we're able to take servers in and out of rotation easily," he adds.
Floral Balance
Load balancing can also help commerce sites that have different classes of customers. Take the
fast-growing 1-800-Flowers.com commerce site, which boosted the number of its commerce servers
from 30 to 80 last year to keep up with increased Internet traffic and transactions. In addition to
processing transactions from consumers, the servers also support an order-processing system called
BloomLink that lets affiliated florists process and fulfill orders as they come into the 1-800-Flowers.com
system. Business partners logged on to the BloomLink extranet can access their orders, download
information to their point-of-sale system, and process other database-intensive information.
To ensure that servers are available when business partners need them, 1-800-Flowers.com has
deployed load-balancing software from IPivot Inc. The software has improved the performance and
overall reliability of the extranet by balancing transactions and requests among multiple servers, says
director of network services Dee.
But the performance of servers accessed by consumers is more important, he says. "I can afford a few
minutes of downtime with my business partners because we're in business together-we're a family. But if
I'm not available for my consumers, they'll go somewhere else."
So Dee is trying a new tactic: offloading encryption processing from the commerce servers. Setting up a
secure, or encrypted, connection to process an E-commerce transaction is a common feature on Web
sites today. But establishing encrypted connections can hurt performance. Dee estimates that every time
a server uses HTTPS, a protocol for accessing a secure Web server, to process an encrypted transaction,
the server's processor is 10 times more active than when it's simply processing requests using HTTP.
To improve performance, 1-800-Flowers is using a Commerce Accelerator from IPivot to intercept
requests for secured connections before they reach the commerce server. The $12,995 device then
establishes the connection and encrypts the traffic using HTTPS or Secure Sockets Layer, another security
protocol, freeing up commerce server resources for processing transactions or providing requested
information.
Another technique for ensuring that certain customers get good service, even when a site is overwhelmed
with traffic, is policy networking. "Until now, we've been able to afford outages-we haven't liked them,
but we could handle them," says John Dodds, a senior systems administrator at Financialweb.com Inc. in
Orlando, Fla., which provides a variety of financial information on its Web site. That attitude changed in
July when the company launched an E-commerce application that pushes real-time stock quotes and
other financial data to paying subscribers. "Now it's absolutely critical to be up 100% of the time," Dodds
says. "We have some real-time traders as customers, and for some of them, just a five-second delay can
amount to 15% profit or 15% loss."
Dodds' goal is to ensure that paying customers have server resources available to them regardless of
how many nonpaying visitors are accessing information from Financialweb.com's systems. To
accomplish that, the company has implemented a policy-based network that can prioritize certain
customer traffic over others using "application-aware" switches from Alteon Inc. that recognize specific
types of traffic and applications. The switches give priority to all traffic that's encrypted, which comes
from paying customers who have secure connections between their PCs and the Financialweb.com
servers. In addition, subscribers to its Premium Professional services will have priority over other paying
subscribers, letting Financialweb.com set up multiple levels of service for different customers.
For some E-commerce site operators, speed is just as important as availability. Losing a customer
because a page takes too long to load is no different than losing one because a commerce site is
unavailable. "We're always trying to increase the speed of delivery of our content," says Motley Fool's
Gibbs.
The traditional-and costly-solution to that problem has been to add more bandwidth to the network. If a
56-Kbps line is overloaded, upgrade to a T1 (1.5-Mbps) line; if the T1 fills up, add another one or two,
or jump to a T3 (45-Mbps) connection. However, that approach can be expensive; a T1 line typically
costs around $1,500 a month, while a T3 line can cost $20,000 a month.
But more bandwidth doesn't solve all problems when it comes to E-commerce because bigger pipes
can't overcome performance bottlenecks caused by the Internet itself. "Much of the delay in getting
content to our customers is latency in the Internet," Gibbs says. "It just takes a long time to get through
all the router hops."
That's why some companies are trying to put frequently accessed data closer to users, especially
bandwidth-intensive content. For example, Infoseek Corp., an Internet information, search, and
commerce site, found that graphical data stored on its commerce servers was slowing the
online-purchasing process.
"Anything that's offered for sale on our site is going to have some kind of visual element to it," says
Infoseek CIO David Chamberlain. "Whether it's a bouquet of flowers or an antique dresser, customers are
going to want to look at images of it, comparedifferent sizes, and examine the front and back and
bottom."
Infoseek and companies such as the Motley Fool and CNN are putting graphical images on servers
distributed throughout a caching network maintained by Akamai or by competitors such as Sandpiper
Networks Inc. and Inktomi Corp. When an Infoseek customer is browsing merchandise, images of the
products will be provided by an Akamai server that's closest to the customer or by one that has the
clearest path to the customer. When the transaction is ready to be placed, the customer will be
connected directly to an Infoseek server. Chamberlain estimates that the technique boosts download
speeds by 20%.
"That way we can intelligently combine our in-house E-commerce application with Akamai's ability to
get images to people faster," he says. "Every time a customer wants something, our servers aren't going
to be tapped. By decentralizing part of the serving, we're making our bandwidth more available for other
things."
Motley Fool is using the same approach. Akamai's caching network stores and delivers to subscribers the
graphical and other bandwidth-intensive material while Motley Fool's servers provide customized
information and process transactions made on its FoolMart commerce site, which sells newsletters,
software, hats, T-shirts, golf balls, and other curios.
Trend Toward Outsourcing
Infoseek's and Motley Fool's use of a service provider to improve performance illustrates the growing
trend to outsource all or part of an E-commerce site. And a host of network operators offers services to
help companies build and manage commerce sites.
Many telecommunications carriers and service providers, such as DBN, Digex, Digital Island, Exodus
Communications, Frontier Communications, Level 3, Qwest, and USinternetworking, let companies
locate commerce servers on their networks. Many of these carriers will manage, and in some cases even
own, the servers and applications. They can also provide fail-safe facilities with redundant power
supplies and fireproof buildings.
Outsourcing E-commerce sites is starting to gain momentum. "We're getting more requests for
high-availability solutions than ever," says Mitch Ferro, director of product management for Internet
hosting at UUnet. "Customers are going to expect more and more high-end operations. They don't want
to be the next eBay."
For some companies, the main appeal of third-party service providers is convenience. Turnkey
commerce services "make a commerce server much easier to maintain," says Mark Barbier, director of
solutions development at ExecuTrain of Phoenix, a unit of nationwide software-training firm ExecuTrain
Corp. The Phoenix operation wants to sell its training classes and training books on the Internet while its
parent company is implementing a companywide E-commerce initiative. But Barbier doesn't want to set
up a server himself; he'd rather subscribe to Yahoo Shopping. "We have people here who could handle it,
but they're in classes eight hours a day," he says.
Doing More With Less
Companies with limited resources facing uncertain demand also find offerings from service providers
appealing. Liveprint.com Inc., a startup online printing company, just launched a Web site that lets
customers quickly create and print custom stationery, business cards, and other documents such as
restaurant menus.
Rick Steele, president and CEO of Liveprint.com, says he hired USinternetworking to operate the
company's commerce site for two reasons. He wants employees to focus on developing new content and
services, rather than on managing servers. Also, he was unsure how much traffic to expect. "I'm not
worried about getting two visitors-I'm worried about getting 20,000 visitors," Steele says.
That problem is in the hands of USinternetworking, which manages the site's hardware, software,
applications, and integration to back-end systems. And USinternetworking, which owns the hardware,
says it can double Liveprint.com's server capacity and triple its 1.5-Mbps network connection within four
hours without human intervention. In addition, USinternetworking stress-tested Liveprint.com's
applications, cutting two months off the company's time to market.
Other commerce-site managers echo Steele's comments. "Outsourcing frees up engineering resources to
improve the quality of the shopping experience on our site," says Tom Chow, VP of technical
development for Reel.com Inc., a movie information site that sells videotapes and DVDs. Reel.com
outsources its site to Exodus.
"We're not a technology company, we're a marketing company," says Steve Furst, CEO of NetGift Registry
Inc. In June, the Durham, N.C., company launched an online gift-registry service that connects customers
to more than 500 retail and charitable organizations. The site can handle 1.5 million members and
1,500 concurrent users.
NetGift signed a three-year, $2 million contract with USinternetworking that lets the service provider
take responsibility for systems hardware and the applications and databases running on six Sun Solaris
servers and EMC storage systems deployed in Annapolis, Md., and Milpitas, Calif. "This is the type of
agreement that enables us to say there is no finger-pointing here," Furst says.
One of the key advantages offered by service providers is experience-both good and bad. "They've been
burned, they've had bad experiences, and they've learned from those things," says Dermot Pope, systems
development manager for Talpx Inc., which created a lumber-exchange site for the $50 billion
softwood, lumber, and paneling industry.
The Chicago company's site uses a Web front end to bring together buyers and sellers and generate
purchase orders, invoices, and payments. But the company ran into problems getting its Web front end
to communicate with its database during the final phases of implementation. UUnet, Talpx's service
provider, brought in engineers who described similar problems other customers had experienced. That
information provided clues for Pope to find a solution. "They were doing more than fulfilling a contract,"
says David Adams, vice chairman of the lumber exchange. "They worked with us."
UUnet also manages Talpx's 10 servers, which are connected by 100-Mbps Fast Ethernet links, its
fault-tolerant infrastructure and load-balancing software, and its PeopleSoft application. Pope likes the
round-the-clock support provided by an outsourcer. "I don't want a million-dollar machine room sitting
in our Chicago offices," he says. "I want somebody watching those machines."
The Right Provider
While outsourcing can relieve many of the headaches involved in running an E-commerce site, the key is
finding the right service provider. BidCom Inc., which provides an online application for managing major
construction projects, says it once employed a carrier-which it would not identify-that was down 10% of
the time and lost packets of customer information. "They didn't see it as a big issue, and they wouldn't
elevate it," says Sal Chavez, co-founder and executive VP of the company.
When BidCom started to provide more services on its site, including the ability to buy materials, it turned
to the Web-hosting service offered by Digital Island. In addition to hosting and managing BidCom's
commerce site and data center, Digital Island helps the company's clients if they are having trouble
accessing the site. Digital Island can also provide detailed reports on who is accessing the site.
A bad service provider can be costly, says Todd Walrath, senior VP of online services for Weather.com,
the Weather Channel's Web site and one of the top-20 revenue-generating sites. Walrath says his
previous hosting service didn't focus enough on uptime, so he switched to Exodus.
"Exodus' entire company is organized to design better Internet connectivity and service to their hosting
customers," he says. Exodus offers a Java tool that lets site managers view the status of their servers from
any Web browser. Exodus also provides a caching service that can duplicate content on servers in
different regions, improving response time for a widely scattered group of users.
Exodus provides the Weather Channel with two to three times more capacity than it needs on an average
day, and can easily add more. The Weather Channel has about 6 million page views a day, but expects
30 million to 40 million views when hurricanes start hitting in the next few weeks. Says Walrath, "As
more people get on the Web, the price of being down for a day or even for a minute is becoming a really
big deal." |
