Len,
Sorry I've been slow to comment. I've been building up to telling my current manager that I was leaving, and finally did so on Friday. I needed to have all my "ducks in a row" so to speak before doing that.
First to continue our discussion:
>>The downside of using a verification set is that you don't make full
>>use of your data for training the net. Therefore, if the dynamics of
>>the time series changes, it might be weeks before you notice it.
>True, not for training, but it still uses all the data through the >last day of all the historical data for its predictions.
>Also, unless I'm mistaken, the volitility bars during the >verification period are based on the out of sample
>predictions, so one can see instantly its accuracy. With no >verification period set, these bars would instead
>indicate net fitness. Also, based on discussions of some that use >other NN programs, a strong argument can be
>made for setting the training period 30-60 days back. They contend >that it actually increases accuracy, and filters
>out "noise".
You are correct (as I read the NS docs, at least) that it does use ALL the data for making the prediction. THAT's OK, since it did not use it in training. I know it's sort of a dirty word to say this, but the best way that I have found to think of NN's is that they do a quite sophisticated form of non-linear curve-fitting. I know it takes the sexiness out of it, but mathematically, that is essentially what it is. They build a mathematical model of your data (whether time series or not) and then use that model to predict the proper value of your chosen output, based on your current inputs. Because of the inherent capability of modelling non-linear systems, NN's are especially useful for these types of systems that are not easily described by just drawing a straight line and extrapolating. HOWEVER, the ease with which they find a "solution" to any problem can be deceiving. This is why I have decided that I want to see how NS does with predicting data it has never seen before--or, in NN terms how it "generalizes"--before I invest. Thus, the verification period.
As an aside, it would be interesting (but I've not done it) to set up several nets for a tradable issue, with different lengths of verification period. One MIGHT find that as long as the nets agreed about the current prediction then it would be a "good" trade, while if they diverged, it would be less reliable. My thinking on this is that it might be a way to give you a signal that the dynamics of the modelled system had changed...so beware. The point is that the model without a verification period is using current data during training, so it might build a different model than the one with verification, which only uses data to say 60 days ago.
As long as we are on the point of verification, etc. I'm curious to know how often you guys retrain. I know in the shareware versions, there was a pop up that told you when to retrain. In the pro version, there does not appear to be a similar popup. This has an impact, since once you have a trained model, and a day has gone by, you automatically start to have a verification period, without omitting any data that you have available during training. (It's not LABELLED "verification", of course, but until you actually do new training, it really is.)
As for a verification period filtering out noise, I think this is just another way of describing the ability to generalize. That is, you don't really want your net to follow every little bump, but rather, to catch the main trend of your timescale of interest as it changes direction. (Unless, of course, you are trying to TRADE every little bump...)
>if you've had the time to browse some our recent discussions you've >noted that we're playing around with
>s/t filter influence settings to attempt to coax Neuro into >analysing and finding relationships between different
>price variances... any thoughts?
I've not studied your posts in detail. At first reading, I didn't understand what you were trying to do. (In fact, I'm still not 100% sure.) My current take is that you are using as inputs several different ST factors. The question I guess I have is how will you know what you are actually learning? (How will you know which the net cares the most about.) Also, I don't remember whether these are all for the predicted issue or if you are also using ST for other inputs (indices or???). Maybe I didn't read it carefully enough.
As for my thoughts, I've never tried this. Frankly, I still don't fully "get" the idea of the ST filter vs. influence period, etc. I know in one of the examples, Andrew shows 3 or 4 inputs that are based on the predicted issue, but I have never understood what those extra inputs actually should be. I also have noticed that he does not specify price or volume for the short term influence, so I really don't have a clear picture of what this even is. (I think one of you also said you were not specifying volume or price.) If either of you can explain why we don't need to choose one or the other, that would be great. (If you have already done this, point me to the post.)
Now, a couple of questions:
As you know, NS uses only the CLOSE information (or volume) that we give it. Based on this, when you are choosing to take a NS recommended trade, do you feel it is necessary to try to buy/sell at the close, since this is presumably the value on which the prediction is based?
I have tentatively decided this is the best thing, but as near as I can tell, it is NOT conclusive.
What VERSIONS of NS are each of you using? It might be useful to know this, since that way we can make sure we are not comparing apples and oranges in terms of capability/operation. I am currently using NS80 (80 node, non shareware). Sometime next year, I think I'm going to have the courage to upgrade to the GA version, but this is it for now.
I've gotta go for tonite...I'll try to check in at least a couple of times a week during the next month. As I already indicated, life is going to be quite "rich" during this time. Once I get re-settled in CO (from NY), I plan to redo some of my NS models as well as explore some new ones. Having this forum should be interesting. I sure wish we could get Andrew to participate. It would make things so much more efficient.
Dave |
