Sun seems unable to fix Java's performance problems. This
press release went out yesterday:
pendragon-software.com
Media Contact:
Ivan Phillips
Pendragon Software
(847) 816-9660
icp@pendragon-software.com
Pendragon Software Clarifies SolarisT CaffeineMarkT Claims
Analysis of test results for Sun Microsystems latest JavaT compiler shows that the compiler specifically scans for Pendragon Software's benchmark
Libertyville, Ill., November 4th, 1997 - Pendragon Software Corporation, the developer of the CaffeineMark Java performance benchmark, today presented evidence that a Sun Microsystems Java compiler specifically identifies part of the CaffeineMark test and produces a misleading score.
On October 20th, 1997, Sun Microsystems issued a press release stating that its Web-enhanced Solaris operating environment delivers the world's fastest Java technology performance. The release also claimed that the new Solaris Just-In-Time (JIT) compiler held a new speed record, 50% faster than the highest Windows NT score, as measured by the CaffeineMark benchmark. Sun provided Pendragon Software Corporation with benchmark results to substantiate their claim.
Analysis by software engineers at Pendragon Software showed that in one of the 9 CaffeineMark benchmark modules, the Logic test, the Solaris JIT compiler scored 50 times higher than any other system to date. There were three possible explanations for this result: either the compiler was performing some impressive optimizations, there was a bug in the compiler, or there was a bug in the benchmark.
To determine if there was a flaw in the compiler or a flaw in the benchmark, Pendragon Software created an alternative version of the benchmark that used a new Logic test. The new Logic test was generated from Java source code that had superficial syntactic changes, and had slightly different bytcodes. There was almost no difference in scores between the original test and the alternate test running in Java environments on Windows 95, Windows NT and MacOS. However, when the Solaris 2.6 JIT compiler ran the alternate test, its scores were 300 times lower. This suggested that the compiler was pattern matching against the benchmark, and performing optimizations that do not work in the general case.
Further analysis of the compiler showed that it contains a block of 600 bytecodes that exactly match part of the LogicAtom.class file used in the CaffeineMark benchmark software. Clearly, the compiler looked for the Logic test module in the CaffeineMark benchmark and produced an extremely high score. This gave the compiler a very good overall CaffeineMark score that would not translate into good Java performance in the general case.
On October 29th, Sun Microsystems removed the JIT compiler from their download page. However, Sun Microsystems' original press release entitled "Sun's Solaris Runs Java Applications 50 Percent Faster than Windows NT" was widely distributed, and is still posted at Sun Microsystems Web site.
"Though we made a request, no retraction of the original press release was issued by Sun Microsystems, so we felt it would be fair to all Java technology vendors to set the record straight," said Ivan Phillips, President of Pendragon Software. "The fastest overall CaffeineMark 3.0 scores we have seen to date are from Windows NT systems running on Intel Pentium II processors at 300 MHz."
The CaffeineMark benchmark can be run directly over the internet at Pendragon Software's Web site at pendragon-software.com. Pendragon Software makes the CaffeineMark software available at no cost via the Web site, and periodically releases submitted results. Pendragon Software does not certify the accuracy of CaffeineMark scores quoted by other individuals and organizations unless specifically noted.
Pendragon Software Corporation is a software development firm based in Libertyville, Illinois. Pendragon Software specializes in software for emerging technologies including JavaT, the 3Comr PalmPilotT platform, and embedded systems. Pendragon Software can be contacted at 847-816-9660; fax at 847-816-9710; e-mail to info@webfayre.com; or through Pendragon Software's Web site at pendragon-software.com.
ÿ###
CaffeineMark is a trademark of Pendragon Software Corporation. Sun, Sun Microsystems, Java and Solaris are trademarks or registered trademarks of Sun Microsystems, Inc. 3Com is a registered trademark, and PalmPilot is a trademark of 3Com corporation or its subsidiaries. All other brands and product names may be trademarks or registered trademarks of their respective holders.
|
