"Time to get real and show what's being measured."
This is a really interesting question. I think we all
are confused here.
It appears that the BAPCO sysmarks 2001 and 2002 do
not measure any task completion times, as every
other normal benchmark does.
From BAPCO.com:
" * Average Response time metric: Response time of Applications to user interaction is used as a performance metric rather than end-to-end run time. Users care most about the response time of the application to input,
* Concurrency Performance: BAPCo has extended its state-of-the-art benchmarking technologies by devising methods (first introduced with WebMark 2001 and SYSmark 2001) for accurate and realistic measurement of concurrent tasks. SYSmark 2002 uses scenarios where background applications that detect viruses, compress files, encode video and convert speech to text run concurrently with other business applications
* Realistic execution speed: The benchmark runs at realistic execution speed, with think times between application interactions, in order to emulate a desktop user’s interaction with the operating system and applications. The think time emulates a desktop user’s interaction with the operating system and applications. Operating system behaviour is more realistic when application interaction has think times (just like a real user) as the OS can devote itself to other housekeeping activities (like memory management, scheduling etc)"
bapco.com
So, what the "revolutionary" benchmark does is: it launches
some background task, file compression or whatever.
Plus loads two-three static, solely user-interactive tasks
on top.
Then it pokes the system in 1 second intervals to open
various pop-ups in these other applications like Excel.
It does not measure how fast the Excel parses its macroses,
nor it does not seem to care how soon the compression
ends up. It measures only how soon the pop-up menu is
showing on the desktop. Therefore it measures one thing -
the scheduling and response time of Microsoft Windows,
the application itself does not seem to matter whatsoever,
except maybe as a memory placeholder, or disk exerciser.
Even the video driver is phased out - there is almost
no drawing activity, mostly 1-second "user think time",
very boring to watch even on P4 2.2GHz.
IMO, the benchmark is completely bogus, and should be
totally disregarded from now on. Please let me know
if I got the BAPCO behavior wrong.
- Ali |
