I decided to run an experiment. I took a known good model that has
performed well out of sample for me and pared down the inputs to a
minimum, about 6-8. I know that using more, like 16, causes the CATNN
and TDNN models to not generalize well, so I used the smaller model,
and since I already knew which inputs were the most important, I
eliminated the least significant ones. I selected "make smaller
models" and also "use fewer inputs". On all of the above models most
used 4 inputs and a few used up to 6. Then I ran NGO selecting only
1 type of model, and ran about 150 models, then looked at the 10 best
to see how they performed. I did not make any special selection of
models, but rather used exactly what NGO made. I had to leave to buy
a christmas tree so the CATNN models ran 500 models, and thus CATNN
performed a bit better than the others.
Here are my results by model type:
$/model $/model models/min
In Sample Out of Sample
CATNN $80,002 $15,742 1.8
TDNN 85,127 12,933 2.5
BP 70,439 6,688 8
SOM 73,026 7,997 15
GRNN 189,964 (1,463) 17
The in-sample period was from 7/10/95 to 6/30/98, about 3 years. The
out of sample period was 7/1/98 to current, or 5 months. Since the
out of sample period is only 1/7 as long as the in-sample period many
of the models performed just as well out of sample as in sample.
I would conclude that GRNN does a great job of curve fitting, but has
no predictive value. If you want to run fast, use BP or SOM, but do
not use GRNN. Other than the GRNN models, I don't think that there
is a significant difference between the models, so long as there is
enough data. I ran PNN also (14/minute) but got only out-of-market
models. I started TSOM but it ran extremely slowly and the initial
models were all zero.
This was a small model, and thus ran quicker by a wide margin than
some of the 32 input models I have tried previously. Still a CATNN
only run of 10,000 models would take 91 hours on my Cyrix.
Carl
|
