Our test suite evaluates the 29 10-Mbps Ethernet switches' performance by passing them through two rounds of benchmark tests. For each benchmark, we tested the switching hubs that arrived with fewer than eight ports in a six-port configuration, and those with eight or more ports in an eight-port configuration. The overall scores combine low- and high-level performance scores (75 percent), features (15 percent), and usability (10 percent).
CONFIGURATIONS
The low-level benchmark test setup consists of three dual-analyzer DA-30s (see the sidebar "The Analyzers") connected to the switches with six ports and four DA-30s connected to those with more than six ports. Each of the two analyzers from each DA-30 attaches to one switch port through its AUI (attachment unit interface) connector. We configure one DA-30 as a master, and th
e remaining analyzers in the group of DA-30s serve as slaves. (In the latency test with multicast traffic, the traffic-generating DA-30s have their own master and slave stations separate from the DA-30 that measures latency.) The master initiates each test and collects results by using an out-of-band trigger line hooked to a BNC connector.
We used 10 75-MHz Dell Dimension XPS P75 Pentium systems with IDE drives for the applications tests. We configured one as a file server with a 525-MB Quantum 525AT hard drive and 16 MB of RAM. The other nine systems have 300-MB hard drives and 8 MB of RAM; four serve as traffic generators, and five run the applications tests. The server runs NetWare 3.12, and the workstations run MS-DOS 6.22 and Windows 3.1. While undergoing testing, the switching hubs dedicate ports to the server and to each of the four traffic stations. The five workstations running applications macros share a single switch port via a 16-port Kingston repeater hub.
PERFORMANCE
The performance score of each switching hub is a weighted average of the low-level benchmark scores and the high-level applications test results. The low-level benchmarks provide highly repeatable and consistent performance numbers. But low-level throughput tests rarely represent real-world network activity. Different applications create frames of varying sizes and use irregular timing constraints; our applications tests perpetrate these undesirable performance effects that are inherent in actual use. The applications' setup, environment, and hub configuration also contribute to the high-level performance scores.
LOW-LEVEL BENCHMARKS
Our
latency tests
measure the duration between when a packet arrives at a port and when it is sent out through another port. We involve only two ports in this benchmark and use a FIFO (first-in/first-out) approach--instead of last-out/first-in, which may give misleading results--to measure a sampling of thousands of frame transmission
s.
The latency figure is an average travel time for 64-byte frames (the smallest legal Ethernet frame size), with and without traffic, and for 1518-byte frames (the largest legal Ethernet frame size), with and without traffic. To determine the traffic-handling ability of the switches' backbone, we multicast 64-byte packets among four ports uninvolved in the latency measurement, using five MAC (media access control) addresses and 25 percent of each transmitting unit's bandwidth.
We also run a
port overload test
. Per switching hub, one port receives and the others transmit 64-byte packets. Each port sends from five MAC addresses. We run this test at a 2000-frame-per-second transmission rate and again at 3000- and 4000-fps rates. The port overload score is a ratio of the total packet count the overloaded port acknowledges to the sum of all the packets transmitted to it.
During our
throughput (packet loss)
test, each analyzer unit on a DA-30 is set up with five MAC addresses
and transmits 64-, 128-, and 1518-byte frames to every other analyzer unit on the switch. In other words, on a six-port switch, one port sends to five other ports and receives from five other ports, in crisscross traffic. We test this at 50 percent and 100 percent utilization, once with broadcast traffic and once without. We also test the 64-byte packets with 1 percent and 10 percent broadcast traffic at 50 percent and 100 percent network-utilization levels.
The
behavior test
rates how well the switching hubs handle illegal frames, FCS (frame check sequence) errors, runt frames (shorter than 64 bytes), and run-on frames (longer than 1518 bytes). For each behavior test, we configure each DA-30 to generate 64-byte frames (and 1518-byte frames for the FCS test) at a 10 percent error rate with 10 percent overall network use and one destination MAC address.
APPLICATIONS BENCHMARKS
In the high-level applications benchmarks, four workstations running traffic, each on a d
edicated switch port, engage in two paired "conversations." A sends an IPX packet to B, B sends to A, C sends to D, and D sends to C. In each paired conversation, one station transmits longer frames (1024-byte IPX packets) while the other sends 229-byte IPX packets, for a combined network-utilization level of 100 percent.
This test indicates how the switches operate with no bandwidth to spare. Five workstations running applications tests share a single 10-Mbps port via a repeater hub and communicate with a NetWare 3.12 file server on a dedicated 10-Mbps switch port. The workstations run Excel 5.0, Word for Windows 6.0, and FoxPro for Windows 2.5 macros under the control of a "master" workstation, each for 5 minutes. The benchmarks measure the speed of network file I/O activity when loading and saving files, and under FoxPro, for record lock and unlock actions.
We automate the applications tests through a Windows application manager. Each workstation reads from and writes to a network disk in all
the applications tests. We run this suite once in standard-mode IPX and once in burst-mode IPX for each switching hub. Benchmark results for each application constitute the average transaction time in milliseconds for a specific I/O transaction (e.g., file open and file save), compiled from the five workstations' scores.
EASE OF USE AND FEATURES
To evaluate ease of use, we examine documentation for organization, comprehensiveness, clarity, diagrams, and examples. We also rate how well the LED indicators communicate link status, traffic, collisions, error conditions, and fault isolation. We rate the ease of setup and device management. Basically, the documentation for the majority of these switches applies to operation and the management interface. However, we didn't deem documentation worthy unless it also included a description of and key for LED indicators, including the colors and various sequences that signify fault conditions.
We looked into such features as pluggable
switch modules, hot-swappable and redundant power supplies, expansion options, and support for high-speed backbones and virtual LANs to arrive at a features score. We weight individual features based on our judgment of each one's importance.
Contributors
Tadesse W. Giorgis, Project Manager/NSTL,
has tested NOSes for NSTL for over five years. He holds a Ph.D. in fiber and polymer science from North Carolina State University.
Bruce Levy, Project Manager R&D/NSTL,
has worked at NSTL in performance-test methodology development for over six years. He holds a Ph.D. in mathematics from the University of Michigan.
Samir Abzakh, Technical Analyst/NSTL,
has been testing network products for NSTL since September 1994. He holds an M.S. in electrical engineering from the State University of New York at Binghamton.
John McDonough, Technical Editor/NSTL,
has written about computer-related subjects for several years.
And special thanks to
Selinda Chiquoine
for her thorough research and editing on this report.
Switching Hub with Eight Ports
illustration_link (22 Kbytes)
DA-30s (with two analyzers each)
Switch Test Bench
To monitor a switch, you must define an application to measure it. Our low-level benchmark tests are based on Wandel & Goltermann's Switch Test Bench test suite. The Switch Test Bench applications test Ethernet switches under high-stress, real-world conditions. The application generates random patterns of multidirectional traffic on the ports being tested and then tests the switch for characteristics such as packet loss, throughput, latency, error handling, and overload conditions.
How We Tested
