Hi again,
I tried to add the following to the last message, but the 15 minute editing window closed before I finished... so, here is more comments on the verification stuff. Dave
********************
It appears you have figured out the verification issue--as Andrew says, just make the last day of the training set something before the last date of your data. Then you will have a verification set. As you will then note when you look at the scatter plot, it is often the case that the network appears to be quite good on the training set, but when applied to "out of sample" data, does not perform well at all. Further, the confidence level does not take this into account. Thus, my SPX model with 72% confidence was in fact a poor model. If I had not used a verification set, however, I would have expected it to perform well. (That was really the point of my first item in the last posting.)
Also, verification is NOT built in, at least in the conventional way that NNs use verification. The idea of a verification set is that the net is tried out on data that it has never seen before during the the training process. That way, you can possibly get an idea how well it is doing on such "out of sample" data--after all, the predictions must use out of sample data.
Before investing, I like to see some evidence that the net that is built with the training process can actually continue to perform once new data is added. One should not expect 100% success, but it should be better than random. The preponderance of the points in the scatter plot should be on the diagonal, just as they should be for the training data alone. (You can turn the training/verification data on/off separately.)
The downside of using a verification set is that you don't make full use of your data for training the net. Therefore, if the dynamics of the time series changes, it might be weeks before you notice it. (One useful exercise, if you have not already done so, is to download the sine wave data from the NStock web site, which is referenced in the FAQ, I think, and play with that. What it is, is two sequential sine waves with different frequencies. You can use it as an input, and let NS predict when to buy an sell this "signal"--it trains in just a few minutes. Depending on where you set the training/verification boundary, you will see completely different performance.)
There was also a brief discussion here about what it is that NS is actually predicting. Based on the help files as well as examining the *.neu files themselves, it appears that there are two outputs, the direction and size of the expected 10 day change. Depending on which prediction mode you use, it either predicts both or only the direction. Then, for all except Max Profits, I think that it just adjusts the boundaries between buy, sell and hold, depending on your setting for Trading Strategy. Max profit is different somehow, as when you choose that one, it actually re-calculates the outputs, while with the others, it seems to just re-calcuate the B-S-H decisions. (Note: I have no inside info... but you can conclude most of this by the info that is presented plus the structure of the *.neu file.) |
