WOW - stuff that goes on behind the scenes... wonder how it looks at Datek!
nytimes.com
<<As E-Commerce Surges, So Do Technical
Problems
By MATT RICHTEL
As the sun rose on her corner-office window one recent dawn in
San Francisco's financial district, Jan Hier-King punched the start
button on the stopwatch she was clutching and leaned forward in her
chair. Like a track coach training sprinters, she scrutinized the speed and
recovery time of her subject: the Charles Schwab Web site.
Ms. Hier-King, senior vice president
of electronic trading technology for
Schwab, was nose-to-glass with her
computer monitor, putting the
company's site through its early
morning paces. "Everything looks
good," she said, leaning back with
relief after the Web pages loaded in
less than eight seconds. "Hopefully,
today will be uneventful."
She would have a much better idea in
three minutes, when the stock market
opened and 30,000 visitors poured
onto the Web site. This would be
rush hour, information superhighway
style, the kind of onslaught that
makes every day an adventure for
Ms. Hier-King, her colleagues -- and
the entire electronic commerce
industry.
The day before, the site had gone
down for 47 minutes during trading hours after a data-storage device on
one of the mainframe computers locked up. But on this day, when trading
opened back East on the New York Stock Exchange, Schwab's system
roared to life without a sputter.
"It's like leaving a car out in the cold all night, then climbing into it and
revving it up to 100 miles per hour without warming it up," she said.
As Ms. Hier-King knows full well, hardly a week goes by without an
outage at a major E-commerce site, whether online brokerage services
like Schwab, Etrade and Ameritrade or merchants like the bookseller
Amazon.com or the auctioneer Ebay. The week before last, for example,
trading on Ebay was shut down for nearly 22 hours because of a
software glitch.
Colossal as such failures may seem, though, if you pull back the curtains
on so-called mission-critical Web operations, it is a wonder that they do
not crash all the time.
Supporting each Web site and every transaction is a huge technological
undertaking that can involve hundreds of computers, thousands of miles
of network cable, hundreds of software programs and dozens of
engineering employees, some of whom are upgrading systems on the fly.
There are numerous points of vulnerability, ranging from the failure of
multimillion-dollar mainframes, to programming errors, seemingly
innocuous software glitches or human stumbles. The site of
SecureTax.com, for instance, a Web site used by hundreds of thousands
of customers to prepare and file taxes, crashed on April 11 when a $200
network circuit card failed and in another instance in March because an
employee tripped on a power cord.
More often, though, the challenge lies in building
(some would say cobbling together) a collection
of cutting-edge technology that was not
necessarily designed with E-commerce traffic in
mind -- or at least not traffic that grows so
quickly. The Schwab site, for example, had 75
million page hits one day in mid-April, more
than double the peak one-day traffic as recently as December.
"It's like preparing an aircraft carrier to go to war," a Schwab
spokeswoman, Tracey Gordon, said of the daily efforts to keep afloat a
site that has already capsized eight times this year.
Keynote Systems Inc., a company in San Mateo, Calif., that tracks the
reliability of E-commerce sites, has concluded that the major online
brokerage firms, probably the most technologically demanding sites on
today's Web, are accessible much more than 90 percent of the time.
Schwab says that its own site is operable 99 percent of the time.
Given all the variables, achieving such reliability figures is a "mind
boggling" technological feat, in the view of Gene Shklar, Keynote's vice
president for marketing. But he notes that when customers have their
own trading dollars on the line, they expect perfection.
For now, though, the more daunting problem for many E-commerce sites
is not so much the threat of angering or losing customers, but the
prospect of attracting new ones in droves.
James M. Marks, E-commerce specialist and managing director at
Deutsche Bank Securities in New York, said that Web site outages were
largely a result of surges in traffic that seemed inconceivable a year ago.
"They just blew out the trend line," Marks said of brokerage sites in
particular. "They are in a desperate race to stay ahead of the capacity
issues."
Sometimes, before the Web site has a chance to fail, a company will
close it down intentionally. At the personal-finance software publisher
Intuit Inc., for example, the company took down the online tax
preparation and filing portion of its Turbo Tax site for 14 hours on April
12 -- three days before the tax filing deadline -- for a scheduled
maintenance and back-up of the filing data.
The timing was far less than ideal, but Intuit had designed the network to
make weekly maintenance necessary, said Elizabeth Dougherty, product
manager for Web Turbo Tax, an online tax preparation and filing service.
Such precautions might be understandable: for the 1998 tax year, some
388,000 tax returns were filed via the Web site -- a tenfold increase over
the previous year.
Because Intuit's main business is selling packaged software, not offering
online services, a three-day sabbatical may be survivable, if not ideal,
from a customer relations standpoint. In the case of Web companies
whose entire business depends on a continuous bitstream, like the online
auction provider Ebay, even a day's down time can be disastrous.
On Thursday, June 10, Ebay's site crashed at 7:50 p.m. California time
and did not return to service until 5:30 p.m. the next day. Officials of the
Santa Clara, Calif., company estimate their lost transaction revenues at
$3 million to $5 million during the blackout.
The problem -- a corruption of the files used for listing auction items and
submitting bids -- involved software from Sun Microsystems. But Ebay
executives are not ready to assess blame, since they still do not know the
precise cause, even though technicians from Ebay, Sun and other outside
vendors spent the subsequent weekend trying to inoculate the system
against a relapse.
"It could have been any number of things," said an Ebay spokesman,
Kevin Pursglove. "They're still working on root-cause analysis."
In essence, E-commerce is a field so new that the standard operating
manual has yet to be written. That is one reason Ms. Hier-King of
Schwab is on the scene every trading day at 6 a.m., not simply to head
off trouble, but hoping to learn whether there are any patterns to be
gleaned from the seeming chaos.
Although she claims to be continually on edge, Ms. Hier-King presents a
picture of calmness. Her office is immaculate, a single piece of paper on
the desk in front of her reminding her in her tiny, meticulous handwriting
of the events of the day before and the tasks of the day to come. Only
the ball of Play-Doh she keeps next to her keyboard betrays Ms.
Hier-King's inner turmoil. Throughout the day, she will squeeze, twist and
pull the putty -- the intensity varying with the Web activity playing out on
the screen of her Pentium II Compaq Deskpro and the trading activity as
reported by MSNBC or CNNfn on the screen of the television that
hangs from her office ceiling.
Theoretically, Schwab's site is currently designed to accommodate
264,000 simultaneous users. The peak so far was 60,000 simultaneous
users one day in April.
But for each individual customer interaction, there are opportunities for a
crash-causing error to occur at numerous points along the transaction
route, which begins when a customer logs on.
When customers first arrive at the Schwab site, they are actually linking
up with Schwab's data center in Phoenix, some 500 miles away from
Ms. Hier-King's San Francisco monitoring station. It is the data center's
152 Web server computers that essentially store and dish up the main
Schwab Web pages. These so-called gateway servers, which can cost
$60,000 each and are made by IBM, are typically not the cause of an
outage, because of load-balancing software that evenly distributes the
incoming customer traffic.
Customers who choose to trade stocks are sent from these 152 gateway
servers to one of 84 IBM "middleware" computers. The middleware
machines, which can also run $60,000 apiece, act as translators,
conveying the incoming trading data over ultra-high-speed internal lines to
one of seven multimillion-dollar IBM or Hitachi mainframe computers.
The mainframes store and process most of the crucial data in the
transaction, like customer balances, and they include dozens of programs
that can cause a system crash -- whether the programs that control
customer data or account balances, or those that govern links to the
stock exchanges or those that support communication with Schwab's
internal telephone customer service and sales agents.
Keeping the data center running -- or scrambling to the rescue when it
falters -- are more than 250 engineers and technicians, whose managers
are in frequent contact by telephone, pager and e-mail with Ms.
Hier-King back in San Francisco.
On the wall of the data center, a half-dozen color flat-panel monitors,
each six feet tall and five feet wide, display graphical charts reporting the
status of various network elements. The key indicators are sets of bar
graphs representing the level of mainframe activity for such vital functions
as updating customer portfolios and updating trades. A green graph
means the mainframe is nimbly handling its load. But when a graph turns
red, that means the mainframe is laboring or has even seized up -- and
that the Web site may be in danger of crashing.
Sometimes, though, a problem is beyond Schwab's control. The initial
step of a customer transaction, for example, involves logging into the
Schwab sites from the customer's own Internet service provider. Delays
can occur on the service provider's computers or along the way on the
network backbone that carries the service provider's consolidated traffic.
In other cases, a problem with the Schwab data center might the
responsibility of an outside provider of hardware or software. On April
26, for instance, a program on one of Schwab's mainframe computers hit
a snag. The software then did what it was supposed to do under such
conditions -- phone home.
But when the automatic dial-up call reached the East Coast office of the
software's maker -- which Schwab declined to identify -- a technician
there entered a command that was supposed to remotely fix the software
error, but instead caused the program to freeze altogether. That, in turn,
froze the entire Web site for more than 45 minutes during trading hours.
Of course, Schwab must shoulder its share of blame. Back on Jan. 8,
when the site crashed, the fault lay in the failure of the data center's own
technicians to configure software correctly when trying to shift part of the
workload from one mainframe to another. Drawing perhaps from such
experiences, Schwab announced with IBM last week that the companies
planned jointly to develop load-shifting technology for the online
brokerage industry.
On the recent day that began at dawn with a stopwatch, Ms. Hier-King
was able to put her Play-Doh aside as the trading session rang to a close
on Wall Street some six and a half hours later. She took a moment to
remark upon the remarkable: that against considerable odds, the Charles
Schwab online trading system had once again withstood the onslaught.
"Arthur Clarke said that any significant technological achievement is
magic," she said, referring to the science fiction writer. By such measure,
"the Web is magic," she said. "The fact that a single trade can travel the
Net and take place in under 20 seconds is magic.">>
|
