R.I.M. Offers a Reason for BlackBerry Failure
By Ian Austen | April 20, 2007 | NY Times
[ For another gem of a meltdown story, see my next post: "What Really Happened At JetBlue." ]
nytimes.com
OTTAWA, April 19 — The installation of an insufficiently tested piece of software set off a chain reaction that eventually cut off BlackBerry service to more than five million users in North America users, the devices’ maker said late Thursday.
In a statement, the maker, Research In Motion, offered its first explanation of what led to customers being left without service.
From the point when the shutdown began at about 8 p.m. on Tuesday until Thursday night, R.I.M. had left its customers, analysts and journalists guessing about what had gone so wrong.
As was suspected, the problem developed at the company’s Canadian network operations center which processes every e-mail message that travels to or from every BlackBerry in North America.
The company said that the defective software — technically a change to the system’s operating routine — was intended to improve the operation of the BlackBerry handling system’s cache, a temporary holding area for data that is usually used to speed processing.
It was a change that R.I.M. did not believe could upset the overall operation. But the statement said: “the pretesting of the system routine proved inadequate.”
Instead of improving operations, the statement said, the upgrade “triggered a compounding series of interaction errors” between the cache and the database that is used to track operations.
That, however, was just the beginning. When the company determined that the database and the cache were at war, it attempted to switch to a backup system, through something called a “failover process.”
“The failover process did not fully perform to R.I.M.’s expectations and therefore caused further delay in restoring service,” the statement said. It noted that the switch “had been repeatedly and successfully tested previously.”
While the statement said that the system “continues to operate normally,” it also noted that the company has not completed its analysis of the shutdown or fully developed a ways to avoid it.
“Proper analysis can take several days or longer and R.I.M.’s commitment is to provide the most accurate and complete information possible in such situations.”
The company said, however, that it had confirmed that the fault was not related to “core software infrastructure,” hardware, a security breach or a lack of capacity.
------ |
