PERFORMANCE
We tested each system under Windows 3.1, DOS 6.0, and SCO Unix 3.2.4. The DOS and Windows test suites consist of a combination of BYTE low-level tests and NSTL application tests.
The application tests use actual business applications to represent a real-world measure of system performance. The DOS performance suite includes WordPerfect 6.0, Lotus 1-2-3 release 2.4, and FoxPro 2.5. The Windows suite uses Microsoft Excel 5.0, Microsoft Word 6.0, WordPerfect 6.0, FoxPro 2.6, AutoCAD 12.0, and PhotoShop 2.5. All applications execute macros that exercise common functions of each application. For instance, the Word for Windows test includes subtests that measure file I/O, search-and-replace functions, changing fonts, scrolling by page and line, checking spelling, print previewing, and printing to a file. We ran Windows tests
in 1024- by 768-pixel resolution and DOS tests in 640- by 480-pixel resolution.
The DOS low-level tests isolate performance of specific subsystems, such as CPU, FPU, memory, video, and hard disk. These tests provide important data for analyzing the results of the application-level tests. The BYTE Windows low-level tests predominantly exercise the Windows GDI (Graphical Device Interface) to determine how well a system can execute basic Windows graphics tasks: drawing a line, displaying text, and executing BitBlt operations.
For the Unix tests, we loaded SCO Unix 3.2.4 on each system and ran a test suite consisting of BYTE's low-level Unix tests and SPEC92 integer and floating-point tests. The Unix tests cover a spectrum of typical scientific and engineering tasks, such as electronic-circuit analysis, architectural analysis, and compilation, as well as typical Unix operating-system commands. These tests primarily stress the CPU/cache/memory architecture of the system. Thus, the Unix test suite eva
luates each system as a generalized workstation.
The SPEC92 suite we use covers a variety of application-based and low-level benchmarks representative of engineering and scientific activities. The integer test (SPECint) contains six CPU integer-intensive benchmarks, mostly written in C. Floating-point benchmarks (SPECfp) contain 14 CPU-intensive floating-point benchmarks, mostly written in FORTRAN. These tests primarily measure the performance characteristics of the processor, the processor cache, and main memory units in processor-intensive applications. They do not attempt to measure display, network, or drive performance. SPEC results are indexed to the performance of a DEC VAX 11/780 computer. A score of 42.3 means that the system ran the test 42.3 times faster than the VAX.
This month, we introduce Unix testing based on XMARK93, an industry-standard benchmark to test performance of X Windows server systems. XMARK93 measures how well a workstation's X server executes primary operations; thes
e operations are weighted according to frequency of use and relative complexity. Results are indexed on a Sun Microsystems SparcStation1 running X11R5 under SunOS 4.1.2 and using a CG3 Color Frame Buffer.
To arrive at overall scores, we scale the results of the DOS and Windows application test scores from one to 10, using the best system's results as a 10. We produce three different performance indexes for each system--one each for DOS, Windows, and Unix.
EASE OF USE
In addition to running performance tests, we examined each system for usability by focusing on two areas: system design and documentation. In looking at system design, we rated each system on how easy it is to insert or extract adapter cards; the presence of labels; whether slots are obstructed or easily accessible; and whether you can disable any integrated components to use their IRQ (interrupt request) settings or to swap to another disk, video, SCSI, or networking arrangement.
We gave top ratings to systems that prov
ided well-integrated manuals with comprehensive indexes. Systems that didn't have manuals for video, disk, and SCSI subsystems were penalized. Relevant jumper and DIP-switch settings had to be detailed for a system to rate highly.
FEATURES
We considered the following features most important for Pentium systems:
-- a flash ROM BIOS for vendor upgrades or bug fixes
-- the number of available adapter slots and drive bays
-- the maximum RAM capacity
-- a parts-and-labor warranty of at least one year
-- a high-performance disk subsystem
-- local-bus graphics
Contributors
Chandrika Mysore, Project Manager/NSTL, has tested peripherals and systems for NSTL since 1989.
John McDonough, Technical Editor/NSTL, has been writing for high-tech publications for the past five years. He can be reached on the Internet at editors@nstl.com.
The Lab Report is an ongoing collaborative project between BYTE magazine and National Software Testing Laboratories (NSTL). BYTE ma
Illustration: The top-right screen shows Excel in the Windows 3.1 application benchmark suite, generating different types of charts from one set of data. This test measures screen I/O speed. The lower screen shows a typical Microsoft PowerPoint slide from the video component of the Windows NT 3.5-based Intermark tests. This test script paints these and other graphics, including Corel clip art and simple lines and circles, in several different sizes. When the tests finish, each system's results appear in columns on the main interface (see the detail at left).
How We Tested
