for the record, i'm neither a customer nor a shareholder. just thought you might like the front-page press.
-----
A Cautionary Tale In E-Trade's Glitch
Crash Shows Internet's Vulnerability
By Mark Leibovich
Washington Post Staff Writer
Monday, February 22, 1999; Page A01
The chief technology officer was working on 90 minutes' sleep, a fitful night even by her three-hour norm. Now, shortly after the stock market opened, Debra Chrapaty was seeing red on her monitor, and red meant crisis. It meant hundreds of thousands of online investors could not buy or sell stocks through E-Trade, the Internet brokerage whose computing operations she directed. She took a Tagamet pill to soothe her ulcer.
Meanwhile, in a Silicon Valley schoolyard, E-Trade communications chief Lisa Nash was pledging allegiance to the flag, a ritual she performs with her two daughters every morning. Her beeper pierced the formality with the Code Red news and Nash kissed her daughters goodbye.
On Wednesday, Feb. 3, one of the Internet's highest fliers was effectively grounded by a software glitch. E-Trade's 700,000 account holders were locked out of their online portfolios as the stock market was dropping in heavy trading. Every second E-Trade didn't work could threaten its viability, like a body deprived of oxygen. Internet time is unforgiving, and so are subscribers with money at stake. "Get it together E-Trade," a subscriber railed in an online chat forum, "Or you're going to the E-Grave."
E-Tolerance? The Internet engenders little when problems occur, a reality that was slammed home at E-Trade this month. Online services now are perceived as utilities, not luxuries. In short order, subscribers have come to demand light-switch reliability when they tap their keypads, never mind the unfathomably complex technology underpinning those commands.
As the Internet has bred exhilaration, user expectations likewise have soared, and when something goes wrong, look out. Already, two class action lawsuits have been filed against E-Trade in conjunction with this month's breakdowns.
"Technology fails, and anyone who promises otherwise is full of [it]," Chrapaty said. "Things happen on any given day. My job is not to let our customers notice."
Yet over parts of three days, as its customers could only agonize, E-Trade was laid bare as a public microcosm of Internet frailty. It also became a corporate case study of crisis management in the Internet age, the desperate side of the exhilaration. Chrapaty and Nash were steeped in the ordeal: Chrapaty, 38, a former technology chief for the National Basketball Association, led the high-tech paramedics from E-Trade's offices in Alpharetta, Ga.; Nash, 40, a former State Department intern, headed damage control from the company's headquarters in Palo Alto, Calif.
"We need to perform with militaristic excellence," said Chrapaty, invoking the preferred imagery of chief executive Christos M. Cotsakos, a Vietnam War veteran who was awarded the Purple Heart: "We need to operate with the souls of warriors."
'Wired to Wall Street'
E-Trade is one of the emerging Internet brands to invade contemporary lives, a dual product of the online boom and raging bull market. Now the third-largest Internet brokerage, E-Trade Group Inc. processes 60,000 trades a day. Some subscribers have given up their jobs to ride the gamblers' surge of "day trading," tying fortunes to the stock market and seizing some control of its fluctuations through E-Trade. They pay nothing to subscribe and keep a minimum $1,000 trading balance in their accounts. E-Trade receives $14.95 or $19.95 commissions on each trade, depending on what exchange the stock is listed. Traditional brokerages typically charge twice that.
"Good morning is being replaced by 'Did you buy any Yahoo today?,' " said Jerry Gramaglia, E-Trade's vice president of marketing, whose three children -- ages 10, 12 and 16 -- each have E-Trade portfolios. "We offer the empowerment and thrill of being wired to Wall Street."
Unless subscribers can only see "Error" messages.
Which is why, on that Wednesday morning, Chrapaty's red eyes were fixed on a hideous number. Her monitor revealed that E-Trade's "transaction per second" ratio was well below normal -- and dropping fast. She deduced access problems.
Chrapaty called in her "super-smarts," a team of eight engineers who would root out the cause of the interruption. They laid out the symptoms, analyzed abnormalities and ate Krispy Kreme doughnuts.
It took 30 minutes to isolate the problem, a software upgrade performed the night before. E-Trade technicians had added lines of computer "code" to speed the process by which subscribers had their trades confirmed.
But instead, the new coding infected the site with a most toxic side effect: no one could trade.
Going Into Battle
Lisa Nash arrived at E-Trade's headquarters at 7:30 a.m., Palo Alto time, on Wednesday. She made a beeline for the "Wall Street" conference room, where 12 executives crowded around a speaker phone as the Alpharetta team explained the situation.
Nash wrote "talking points" for customer and media inquiries. This service interruption had nothing to do with site volume, Nash emphasized, and E-Trade could easily handle its daily trade capacity. What's more, only the trading portion of E-Trade didn't work; users could still find stock quotes, financial news and other data. A "reserve team" of 40 customer service managers was activated downstairs, near the cot where Chrapaty often sleeps.
Chief operating officer Kathy Levinson appeared on CNBC early in the afternoon; Cotsakos was flying and essentially out of pocket. Chrapaty explained the technical arcana to Levinson over the phone. Chrapaty then watched Levinson a few seconds later on television.
"It made me realize how closely people were watching us," Chrapaty said, calling the experience "slightly horrifying."
Not as horrifying as what some E-Trade customers were telling company officials, reporters and online chat groups. Nash launched a crisis-specific e-mail address, which received 2,500 inquiries that day. She also led an outbound phone blitz, where company executives personally called hundreds of "high-volume" customers.
Trading was halted for 75 minutes from 10:15 a.m. and 11:30 a.m., New York time. The technical staff placed a "patch" over the bad code to let trading resume. After the market closed, technicians removed the bad code entirely -- or so they thought. Chrapaty ordered pizza for her Alpharetta staff Wednesday night, while Palo Alto staff members ate sushi by the takeout bag.
Nash, who did 50 interviews Wednesday, also took repeated calls from her daughters, ages 6 and 10. "We saw E-Trade on TV," they kept telling her.
Nash slept four hours Wednesday night, Chrapaty two.
Code Red -- Again
They awoke to a bitter deja vu. Trading was down again Thursday.
E-Trade blamed Thursday's troubles on another "aftershock" from the Tuesday upgrade. Some of the troubled code had bled into the site's trading area again, although not all subscribers were affected this time.
Whose fault was this anyway, a reporter asked? "We don't do fault," Chrapaty said briskly.
Chrapaty, who is based in Palo Alto, had previously led her team in "team-building" exercises to instill battle readiness. Rope courses, trust falls and more elaborate sessions, such as giving out water pistols, donning a bull's-eye T-shirt and telling everyone to soak her. This drained aggression and promoted harmony, Chrapaty said. It also forged cohesion and composure during crises. Thursday's crisis lasted more than two hours.
Nash's routine changed little, except that she skipped "flag salute." But this morning reporters were asking about something else: The New York state attorney general's office had just announced it was investigating the online trading industry after "dozens of complaints" from customers.
Such regulatory inquiries, while certainly unwelcome, invite an easy response to the press: "E-Trade is cooperating fully with the investigation," Nash said in many variations Thursday. She emphasized that New York was investigating several online brokers, not just E-Trade, and that it was unrelated to that week's troubles.
After the market closed, Chrapaty held a "town meeting" for her staff. She said she was proud of everyone, told a few they were "champs" and "stud muffins," and gave out hats and T-shirts. Then she signed autographs.
A Shock to the Stock
Friday dawned with sublime quiet and smooth E-trading throughout the morning. It all exploded again at 12:20 p.m. with a 29-minute interruption. "We believe we've got it all fixed," Nash assured Bloomberg News. "But then again, we believed that this morning."
E-Trade staffers complained that the press was picking on them. If they had a 29-minute failure six months ago, no one would have noticed. The attention was a perverse rite of the company's rapid growth and popularity. "I find the attention frustrating," Chrapaty said. "And also flattering."
Either way, Feb. 3, 4 and 5 offered a cautionary lesson in the fallibility of Internet commerce coupled with the volatility of stock-market livelihoods. Wall Street's reaction was not flattering -- or forgiving. Shares of E-Trade closed the week at $48.93 3/4, down from the 52-week high of $62.43 3/4 it reached that Monday. The stock closed Friday at $40.12 1/2 on the Nasdaq Stock Market.
Nash split her weekend between debriefing calls and a family hike. Chrapaty flew to Charlotte Saturday for her great-nephew's birthday, spent $1,000 on presents at Toys R Us, flew back to Alpharetta that afternoon and returned to Silicon Valley on Sunday.
There, Chrapaty found a gift assortment of cookies and oranges, and some neighbors sent over roses.
"Listen guys," she said by way of thanks. "I'm not dead yet."
washingtonpost.com |
