NT vs. NetWare: File Service Grand Prix

Recently, the Windows NT Magazine Lab handed me this hot assignment: Produce performance benchmarks for Microsoft Windows NT Server 4.0 and Novell NetWare 4.1. Why is this topic so hot? Because chances are really good that at least one of two industry giants will not like the results. Undaunted by the prospect of attracting the wrath of a major corporation, I gathered up my gear and headed to the Lab's speedway.

For my performance tests, I used the Lab's standard configuration: a set of client machines on a 100Mbps Ethernet network that simulates the workload of multiple users. (For details about the Lab's test environment, see the sidebar, "The Benchmarking Speedway," page 65.) One server, running either NT Server 4.0 or NetWare 4.1, tackled the workload of the client machines. Because I wanted to test file-sharing performance, I employed Bluecurve's Dynameasure for File Services 1.5 as the workload engine. (For information about this product, see Lab Reports: "Dynameasure Enterprise 1.5," September 1997.) The combination of Dynameasure and the Lab's test environment let me simulate workloads that users typically perform and pinpoint potential bottlenecks to ensure that I stressed the server, not the network or clients. After a month of testing and reviewing reams of data, graphs, and tables, I determined a clear winner: The checkered flag goes to...NT Server 4.0!

Preparing the Track
I wrestled with several background issues going into these tests. My biggest concern was how to create a fair comparative evaluation. Different people often have different priorities when they evaluate a product. For example, when my wife and I decide to buy a new car, we define our needs and select criteria (e.g., price, speed, appearance, hauling capacity, headroom) to measure a car's potential value. Even if we agree on the criteria, we don't always agree on the importance of each item; picking a clear, overall winner is difficult. Fortunately, we agree that one criterion outweighs all others: performance.

Performance is also a primary criterion that people use to evaluate file servers. Although other factors (price, support, compatibility, interoperability, etc.) matter, performance is of utmost concern. I based my evaluation and conclusion on only how well NT Server 4.0 and NetWare 4.1 performed in the Lab's test environment under Dynameasure for File Services 1.5. If criteria other than performance are more important to you (e.g., your primary concern is to find a file server operating system that runs on a 386), you might come to a different conclusion.

The Speedway Judges
In the performance tests, I measured three main areas: throughput, average response time (ART), and motors per step (MPS). Throughput (measured in kilobytes per second--KBps) is the total number of bytes all the motors copy during the measurement phase of a step divided by the elapsed time of the measurement phase. Throughput measures system capacity. The type of transaction, the number of motors, and the hardware capacity of the system influence throughput. Higher throughput means greater system capacity.

ART is the average time in seconds to complete a transaction during the measurement phase of each step. ART measures the speed of the test system. The type of transaction, the number of motors, and the hardware capacity of the system also influence ART. Lower ART means the system is faster.

The third measurement, MPS, is the number of motors that report results for each step of the test. MPS measures the total number of assigned motors in a step that complete the transaction. MPS is a direct measure of load on the system. Higher MPS means greater load.

I approached the benchmarking process as if I were testing two unique racing teams and pit crews. Each team (NT and NetWare) used the same physical track (the Lab's network) and the same physical cars (clients and motors). Each team had equal time (about 16.5 minutes) to complete as many laps as possible (throughput). I tracked average lap times (ART) and the number of cars that completed the race in a given time frame (MPS).

Start Your Engines
Establishing a test environment that could run both NT and NetWare was the first order of business. For the test server, I used a generic PC clone with the following hardware configuration: a 120MHz Pentium, 64MB of RAM, a master 2.1GB hard disk (EIDE), a slave 2.1GB hard disk (EIDE), and a Novell NE 2000 Socket EA network adapter. Software configuration included NetWare 4.1 with 10 license connections and NT 4.0 with Service Pack 3 (SP3). I partitioned the slave 2.1GB hard disk into two equal areas--one for NTFS and the other for Novell's file system. I added this test system to the Lab test environment to measure performance with the Dynameasure software.

I chose six of the Lab's clients, running NT Workstation 4.0, as my user testbed. I configured each client with eight or nine motors to simulate a total of 50 network users. NWLink IPX/SPX bound all client adapters. I performed some preliminary tests with both the Microsoft NetWare client/protocol software and the Novell-supplied NetWare client/protocol software. I used the same testing specifications I planned to use for the benchmark tests, and I saw no performance difference between the two client/protocol software products. Throughout the benchmark performance tests, I used the Microsoft client/protocol software.

Time Trials
I ran several initial Dynameasure tests just to warm up the track. At this point, I wanted to identify any bottlenecks that could affect the results. In particular, I wanted to eliminate the possibility that the client systems or the network bandwidth could degrade performance. The warm-up tests ran the Dynameasure for File Services Copy All Bidirectional test configured for a 5.6MB dataset; a 1KB block size; 10-second think time; and 6 steps, with the following number of motors assigned to each step: 5, 10, 20, 30, 40, and 50.

The Copy All Bidirectional test consists of 16 different transactions in which compressed data, uncompressed data, binary files, text files, and image files are copied between the server and the clients. Based on this test (and with help from Bluecurve's technical support team), I was able to ensure that I was not overstressing the client workstations or the network. The summary reports for NT and NetWare showed that server performance began to degrade after reaching approximately 25 motors--throughput leveled off, ART rose, and MPS declined.

During these warm-up tests, NT vastly outperformed NetWare, as the Dynameasure graphs in Screen 1 and Screen 2 show. In Screen 1, the left graph displays throughput, and the right graph displays ART. Screen 2 displays throughput in the left graph and MPS in the right graph. NT maintained higher throughput, lower ART, and higher MPS in these tests.

For several reasons, the Lab had hypothesized that NetWare would have higher throughput and be faster during certain types of transactions (such as copying small files). For example, NetWare 4.1 includes Packet Burst technology, which lets a server transmit several packets in a burst, without waiting for verification that each packet has been received. NetWare 4.1 also supports Large Internet Packets, which lets the server and workstation communicate using the largest possible frame size.

The Copy All Bidirectional test includes eight different types of data, with two files for each data type: one file for client-to-server transactions and one file for server-to-client transactions. Because the type of transaction influences the three benchmark measures, I decided to break out individual transactions and compare the results to get additional information. This view of the data would let me compare server performance based on the type of transaction each server completed.

   Previous  [1]  2  Next