That was happening everywhere, including iHub.
We finally finished up a really pressing project and that error was happening during the last phase of it.
Specifically (if anyone cares about the specifics), the ad serving for both sites used to be handled by iHub's webserver. When that got to be too much, I moved it onto iHub's old db server, having it not only be the ad server's web-server, but also db server for it.
It got to be too much for it, which led to the project we just finished.
Now ad-serving (for both sites) is handled by a relatively powerful webserver (iHub's old webserver, which has a pair of 2.2Ghz Xeon's) and what was ad-delivery's web/db server is now only its db server (with a single 1.9Ghz P4).
The last step was taking that box off the internet and putting it behind its new webserver on the private network. The error you (and everyone else) saw was happening while we configured the db server and webserver to talk to each other via the private network.
So, ad-serving should be in good shape now, with more than enough available horsepower.
Which is why not much has changed on the new SI lately. Been tied up with that and trying to sort out iHub's reliability problem. I think we're pretty close to a solution on that one now, too. I wrote a program to edit all the programs on iHub, ran it, and Matt's going to check it tonight to make sure the development version of that site is working correctly, then put it into production. Once he's done that, I'm going to programmatically make one more change to all of iHub's programs, Matt'll test the site, put it into production, then that problem will hopefully be solved.
Oh, and since I'm getting into specifics, the problem on iHub is that ASP will suddenly shut down. HTML still serves fine, but ASP dies. A reboot fixes it.
There are only 3 possible causes we've found while googling:
1. CPU staying at 100% utilization too long (not the case here -- it runs about 35% during the day).
2. Attempted connection count exceeding the configured limit (not the case -- we're configured for unlimited connections).
3. A lazy programmer (me) usually forgetting to close recordsets, instead counting on ASP to close them since it's supposed to. Supposedly, this results in too many unused recordsets being maintained open and crashes ASP.
So that's what we're working on right now. Today I programmatically made sure no ASP files were closing the database connection, then put code in the last include file that each page uses to close the db connection. Tomorrow I'm going to do the same thing with recordsets.
If that doesn't fix it and ASP is still crashing on iHub next week, I'm going to be REALLY stumped. |
