To: Jay Hartzok who wrote (682 ) 2/23/1999 12:01:00 PM From: Jay Hartzok Read Replies (2) | Respond to of 805
Everyone: I have been debating whether or not to post this information, or just let it go until the fix is released, and have finally decided to go ahead, because like me, all of you have spent money for this software and certainly deserve to know. Before I get into it, I just want say that there is no reason to contact Andrew about this. It is my understanding that he has already been made aware of it, has acknowledged that it exists, and is working on fixing it. Hopefully there will be a updated version out shortly. I have been involved with extensive testing of this software during the last few weeks. Also involved is a computer programmer, who I recently met. Thanks to his expertise in testing procedures, we have proven without a shadow of a doubt, that during back propagation NeuroStock sees the data in the verify period. It also appears the during simulated annealing that Neuro does NOT see the verify period. This flaw in the program explains {for those of you who use both training methods in your nets} why the verify period looks extremely accurate during back prop and seems to collapse when it begins annealing. The verify period that you see when the net has switched to annealing is more representative of what the true verify period really looks like and shows, more so, how accurate your net really is in predicting ahead. We have also determined that as the net becomes more fully trained a "war" ensues between back prop and annealing, the end result usually being that back prop wins and annealing starts "thinking" like back prop's calculations that have been contaminated from data that it should not have ever seen. We have also concluded that if the training start date does not remain constant, the data becomes skewed and Neuro will have quite a job trying to straighten it out, if it can at all. We have also proven, without a doubt, that the Forget button doesn't make the net forget all of the data calculations from the previous net. What does this mean for all the nets that you have created? Well, generally it means that any net that you have created that uses back prop, and used a verify period to ascertain it's accuracy, is flawed. The verify period is not showing the true accuracy of the net. With the software the way it is right now, the only way that I see that anyone can get a true verify period is to alter the data files leaving out the last two or three months of data. Load these files into a net and train that net without any verify period. After the performance levels have stabilized and you believe that it is fully trained, stop training, add the edited data back into the files, but exclude the last ten days, and have the net reread the files. Then move the training start and end dates to their original positions. Do not train. Now you can view the newly added part of the chart, which is now a true verify period, and determine the net's accuracy. If you are satisfied with how it looks then you can move the training end date as far forward as possible and train the newly added data into the net. When the net stabilizes again you can add the last ten days of data and get a true prediction. Hopefully Andrew will have a fix for this shortly. You can find some additional information by visiting the NeuroStock club at Yahoo and reading through the last 20 or 30 posts. Jay