Dear Tench:
I think the current SPEC benchmark in its current form is about worthless. It allows Major companies like Intel to get far away from the original purpose like using compilers that many have deemed useless for doing real work. The original flavor was to take the typical off the shelf compiler, run it to produce code that is used often by the workstation users community, and run the resulting code on hardware you could pick up today. The current organization has bent over so much to the hardware vendors, that the compiler could be written to see the SPEC source and substitute hand coded and tuned assembly programs for each of the benchmark sources, run it on unreleased hardware in the absolute best conditions on preselected data, and come up with benchmarks almost no one else would when, they would take the mainstream compiler, run it on the top machine they have trying out on data they just randomly selected. AMD does the later, Intel the former.
I would change the rules to include, random data, verifying the results on the same machine using a open source compiler like gcc, making sure the compiler was not optimizing based on the source by first compiling random programs that are handled with no trouble with a OSC like gcc, and then changing the SPEC source programs slightly to see if the size or scores change dramatically (a good compiler should not change the speed for minor changes in the code), and then verify that the hardware and software are able to be bought (if not, mark the benches beta or alpha).
Then compare the two. Outsiders have shown that P3 is not as good as same clocked Athlon.
Pete |
