While the SPEC ratings are probably the most commonly used measure of performance, there is a wide variety of other standardized benchmarks. The list is too long for us to discuss them all in depth here, but we've listed a few below to show the different ways that benchmarks are designed.
Whetstone. An early synthetic benchmark developed by Curnow and Wichman in 1976. It measures floating-point performance and is used to compare architectures and the optimizing compilers that run on them. The code is short and has been translated from its original language (Algol) into many others. Although it is sometimes modified and is not always applied carefully, Whetstone has spread widely throughout the computer community and serves as a useful basis for discussing performance.
Dhrystone. A synthetic benchmark developed by Reinhold Weicker in the earl
y 1980s that focuses on integer and string performance. The name is a play on words, paying respect to the influence of its predecessor, Whetstone. Originally written in Ada, the benchmark has been ported to numerous other languages and has become another popular performance measurement.
Linpack. Jack Dongarra has been a leader in the development of several widely used linear-algebra packages, including Linpack and Lapack. As part of that effort, he maintains a report that measures the performance of a broad variety of machines using various elements of the Linpack library.
NAS Parallel Benchmark. NAS is a branch of the NASA Ames Research Lab; it works with many parallel architectures. Frustrated by the fact that it's extremely hard to compare all the wildly different architectures, members of NAS developed an interesting alternative to traditional benchmarking suites.
On a parallel machine, an application must typically be rewritten completely, in a new form or a new language, for it to
exploit the machine most effectively. The NAS suite provides a straightforward sequential implementation of each algorithm, but the heart of the benchmark is an algorithmic definition of each computation.
Manufacturers are to report how well a simple port of the application runs, but they are also free to develop their own implementations, which are as efficient as they can make them. A set of restrictions on the language can be used to ensure that a machine is not completely unusable by a scientific programmer who is familiar with other parallel architectures.
Using the NAS suite is a great deal more difficult than using something like SPEC, because it involves writing a set of tuned parallel applications. However, the suite gives manufacturers a chance to demonstrate what their machines can do in a way that is impossible with more traditional benchmarks.
Perfect Club. The University of Illinois, which has worked with supercomputing for many years, was not satisfied with the benchmarking
techniques being used. The university's Perfect Club suite takes the same general approach that the SPEC suite does: Assemble a set of real applications donated by interested parties and organize them into a standardized benchmark. In fact, the Perfect Club has recently become part of SPEC.
The Perfect Club programs are floating-point-intensive and are usually executed on supercomputers. A major goal of the Perfect Club project was to characterize applications in terms of their algorithmic behavior, allowing users to get meaningful predictions of the performance they could expect for their own applications.
iCOMP. In the old days, Intel had a few different chips on the market that were relatively easy to compare to each other. Then came the proliferation of 486SX versus 486, clock doubling, and so forth. To help the consumer figure out how different Intel processors compare to one another, the company developed an index that provides a single number to measure them. It is based solely on the pr
ocessor and does not reflect the performance of a particular machine design. The metric hasn't really caught on yet, but Intel's marketing muscle may succeed in making it more prominent.
A World of Benchmarks
