Benchmark Boo-Boos

The history of benchmarking is rife with political and technical squabbles, and sometimes outright cheating. Here are a few examples.

-- Fooling WinBench. When the market for Windows video cards took off a few years ago, some vendors weren't happy with their WinBench results. With about 200 video cards on the market, manufacturers wanted their products to stand out. So a few of them embedded WinBench acceleration code directly in their ASICs or ROMs. This pr oduced blazing benchmark scores, but provided no real benefit to users. Benchmark publisher Ziff-Davis was forced to change WinBench and go on the alert for similar tricks.

-- Int el's SPEC92 snafu. Last January, Intel announced it had overstated the SPECint92 ratings for its Pentium and Pentium Pro microprocessors by an average of 10 percent. For example, Intel rated the 200-MHz Pentium Pro at 366 SPECint92 when the correct rating was 320.2. What happened? Intel compiled the SPEC92 benchmark programs with a beta version of an in-house C compiler. An error caused exaggerated results when running the 023.eqntott program. Within weeks, Intel issued correct results and noted that the error does not affect SPECfp92 or SPEC95 ratings.

-- BYTEmark gets bitten. Late last year, BYTE discovered that its BYTEmark test sometimes reported inaccurate results. Lead developer Rick Grehan traced the problem to the way the C compiler's malloc( ) function allocated memory for arrays of 8-byte floating-point values. In some cases, malloc( ) returned a memory block that wasn't aligned on 8-byte boundaries. This turned out to be a problem when one routine in the benchmark prog ram (a lower/upper decomposition algorithm) repeatedly accessed the nonaligned array. Because nonaligned memory accesses hit Intel x86 processors particularly hard, this led to lower-than-normal numbers for the Pentium Pro. BYTE fixed the problem by placing alignment wrapper code around The 440FX PCIset, code-named Natoma, represents a savings in silicon and system board space--and as our testing reveals, without sacrificing performance. The Bravo's SYSmark NT score compares favorably to those of earlier systems we've tested. However, the Orion chip sets will continue to be a better fit for quad-processor designs because they support four memory controllers and two PCI bridges. malloc( ) .