Sysmark 2001 has fine objectives as a benchmark but I think it is very susceptible to manipulation. It attempts to measure "average response time" for a long series of foreground tasks while it is executing 1-3 background tasks. For reference see bapco.com
The deficiencies I see in this approach are:
1. Nowhere does the documentation say that Sysmark 2001 even measures the performance rate of the background tasks, how much of the virus scanning, WinZipping and Speech-to-Texting is actually completed per unit of time.
2. Because of (1), the benchmark is highly susceptible to the relative priority of foreground and background tasks and could easily be manipulated
3. Because of (1), the benchmark may actually run faster with a slower disk subsystem! The background tasks may be reduced to a crawl by a disk subsystem with a high latency, thus freeing up more CPU time for the foreground tasks, which are the only ones measured.
4. Exactly how the benchmark determines WHEN the foreground request has been completed is a mystery. This is non-trivial and I doubt it works in all situations. For example, the fact that the video routines in Direct X 8 have been "returned from" does not necessarily mean that the screen update is completed.
5. (4) is particularly true for Java webpages. I'm sure we have all experienced the fact that the cursor may be blinking in a dialog box even though Java is still interpreting the last command.
6. Again the choice of background apps is highly suspect.
6a. Winzipping in the background is totally bogus. NO-ONE DOES THAT because WinZip is so fast that it is nearly always the foreground application when it is being used. If someone want to WinZip an entire multi-gigabyte disk for archival purposes, they won't do it more than once a week. The reason BAPCO picked it, is that the data kernel size and program kernel size is small, and will fit into the itsy bitsy P4 L1 data cache (16K, I believe).
6b. Similarly, virus scan in the background is generally done only
a)when starting programs (no significant workload)
b)when downloading something (no workload because virus scan is much faster than any internet connection)
c)done across a network -- this would be useful to simulate in the benchmark, but the scanning rate would be much lower than used in this benchmark.
6c. Text to speech translation - when's the last time you DID THAT? I'm sure you want your computer talking to you while you are editing and creating multimedia documnets! Again, text-to-speech is highly P4-optimizable because it accesses data linearly rather than randomly, allowing the superior memory<-->CPU bandwidth of the P4 to make a difference. But it takes little FPU power (advantage, Athlon) to do text-to-speech.
In short, Sysmark 2001 is a totally rigged benchmark, devoid of any resemblance to real internet content creation or office application usage.
EDIT - PS, I realize I'm probably "preaching to the choir" here, Ali.
Petz |
