Problems in the mtnview server farm between 11/10 and 11/18 were caused by our internet service provider to that location. It kept "flapping" (the technical term, according to our ops wizards) which means that every day during that period it disconnected the entire farm from the internet. It did that 10 times on one of those day.
We have two server farms. So, when this flapping happened, half our QCharts/QFeed customer base would be disconnected from the servers there and would start looking for new servers. Since the next couple of servers in the shopping list in the ContinuumClient.ini file were likely to be mtnview servers also, a disconnected user might go through a cycle of trying to reconnect to mtnview a few times before getting down to a server that would connect you to a herndon server (our other farm). But in the mean while, mtnview would re-appear because in flapping, it's "flap on" "flap off" "flap on" type of thing. (Really infuriating for all concerned.)
Needless to say, when half the users are thrashing around trying to get reconnected to another server, it is a recipe for slowing down the servers that are available. Once the user reconnects, they have to get all their data histories for charts refreshed from scratch to make sure it is complete. When hundreds or thousands of users pile on a server or a farm of servers it just can't keep up with such a demand. However, in situations like this the servers are designed to send quotes first but to delay filling historical data requests (for charts) until it can serve them up without delaying the delivery of hte quotes. We figure that the quotes - where the market is now - is more imporant than a snail trail of the past.
We have enough backup capacity in either farm to handle the load if one or the other were to go out and stay out. We even located the farms on either side of the continent to protect the service from geological events that might befall one or the other. (More likely to befall California than Virginia). But there was not enough capacity to handle this particular situation where the servers were up/down/up/down repeatedly during a very busy several days in the markets.
So - that's basically what happened. We've canned the ISP we were using at the mtnview (Mountain View, CA) and have two others in place in mtnview as backup with a large capacity overhead far and above our peak capacity bandiwdth loads. We are also increasing the capacity of servers at both server farms, and are in the process of adding a third farm yet at another location entirely.
Other problems we have had are due to the fact that we get our quotes via satellite from S&P on their 128K feed. This feed is run from their backup server farm and has been the cause of some other issues we have been wrestling with. We are migrating to the much fatter 256K feed from S&P which should solve two issues that have been chronic: missing option trade reports and missing end-of-day resets that cause the Daily data to get screwed up. This is why problems we have been seeing might not be evident to other S&P users on the 256K feed - S&P Comstock are no longer selling the 128K feed - and for good reason.
Our customer service has improved dramatically in the last several months. It isn't perfect - it is hard to handle 800 calls in a day when we have problems like the one described above when our connection to the internet goes up and down 10 times in a day.
The servers are reliable, but the Internet is still a hostile operating environment, and we haven't yet tamed it, though we have quote delivery technology that was designed to handle these fluctuations well. But we continue to work hard to make this service a good value and a useful tool for traders.
Sincerely,
--
Gibbons Burke
Director, Active Trader Products |
