Lying with Statistics

Your November 1993 review of subnotebooks (``Windows Under 4 Pounds'' ) didn't say how the ``overall'' performance numbers were calculated, but the method that you used seems to have arrived at the arithmetic mean of the individual test results. Those results were normalized numbers, whose arithmetic mean is not meaningful. The article ``How Not to Lie with Statistics: The Correct Way to Summarize Benchmark Results'' by P. J. Fleming and J. J. Wallace (Communications of the ACM, March 1986) describes why the geometric mean, not the arithmetic mean, is the only proper average for normalized numbers.

Applying the arithmetic mean can give some pretty silly answers. Benchmark testers commonly misuse the arithmetic mean, but I hope that BYTE will not do so in the future.

Clem Dickey

San Jose, CA

In the article that you cite, we used the geometri c means to compute individual subtest results, but the overall performance value was computed using an arithmetic mean. If we had used a geometric mean to compute the overall values with this particular data set, the relative rankings of systems would have remained unchanged, and magnitudes would have decreased only slightly. However, you are correct in your criticism. We are currently revising our benchmark tests, and future test suites will not use arithmetic means to

compute overall performance scores.

--Eds.