Much to my surprise, the detailed results from the tests revealed that for
every type of transaction, NT outperformed NetWare in throughput, ART, and MPS.
The closest throughput values for NT and NetWare occurred during step 5 of the
Copy Compressed Text Bidirectional test: 406KBps for NT and 401KBps for NetWare.
During that step, NT's ART (0.17 seconds) was faster than NetWare's ART (0.51
seconds). The largest gap in throughput values occurred during step 6 of the
Copy Data Bidirectional test: 1.21MBps for NT and 298KBps for NetWare. At that
point, NT 4.0's ART (29 seconds) was more than five times as fast as NetWare's
ART (166 seconds). In all tests, both network operating systems had the same MPS
values, which matched the assigned specifications of 5, 10, 20, 30, 40, and 50
motors.
Reality Check
To ensure that my testing parameters and system configuration were not in
some way slanted toward NT, I presented my findings to Bluecurve's technical
support team and to the other Lab technicians. One major concern pertaining to
NT caching came up in those meetings. For bidirectional transactions,
Dynameasure creates the data files on both the client and the server. For
example, one of the files in the Copy Compressed Binary Files from Client to
Server transaction is identical for every motor during a test. Because of the
small file size and small block size being transported across the network, I
decided to investigate whether NT Server and the NT clients were actually
reading from or writing to the disk for every copy transaction.
On any desktop system, you can see how data caching affects performance
with a simple experiment: Open any large data file stored on a floppy or a hard
disk. The system takes a few seconds to read the requested data, open the
viewing application, and display the data on the monitor. Close the application,
and then open the same file. The data appears almost instantly, and you don't
hear the characteristic spin of the system reading the disk. The system has
cached the data (and possibly the application) in RAM; thus, no disk read
occurs, and the whole operation is substantially faster.
I needed to eliminate any server or client memory caching that could
influence the tests. The idea was to force both NT and NetWare to access the
hard disks as many times as possible during the copy transaction. (In racing
terms, I needed to ensure that the cars made as many pit stops as possible.) I
conducted the next series of tests in the same manner as the warm-up tests,
except I increased the dataset scale and block size. Increasing these settings
increased the memory that the system paged, and flushed the system's RAM.
Armed with this idea, for the next series of tests, I used the following
parameters: a 24.2MB dataset, a 100KB blocksize, a 10-second think time, and six
steps (with 5, 10, 20, 30, 40, and 50 motors, respectively). I again selected
the Copy All Bidirectional test, because the random order of the transactions
makes caching data from one transaction to the next difficult. Graphs 1, 2, and
3 display the results of these tests.
As you can see from the graphs, NT 4.0 came out the clear overall winner in
performance. During the test, peak throughput for NT (741KBps) was double the
peak throughput for NetWare (356KBps). In every step of the test, NT's ART was
more than twice as fast as NetWare's ART under comparable loads (MPS).
To the Winner's Circle
For the last series of benchmarks, I decided to use Dynameasure's Copy All
Files to Server test. This test would eliminate file caching on the clients as a
performance variable. For this test, I used the same test parameters that I used
for the Copy All Bidirectional test: a 24.2MB dataset, a 100KB block size, a
10-second think time, and six steps (with 5, 10, 20, 30, 40, and 50 motors,
respectively). This test and specification set transmitted a large amount of
data across the network and maintained a high frequency of delivery. Graphs 4,
5, and 6 display the test results.
The benchmark data again favors NT. Graph 5 shows that the closest ART
values occurred during step 6 of the test. In step 6, NetWare's ART (122
seconds) was 10 seconds faster than NT's ART (132 seconds), so you might be
tempted to argue that NetWare outperformed NT in the part of the test simulating
the heaviest load (50 motors). However, you must examine the ART data in
relation to the other performance data.
In step 6, the NT racing team completed a maximum 841,000 laps (i.e.,
841KBps peak throughput) , and all 50 cars (MPS) finished the race. The NetWare
racing team completed 167,000 laps (i.e., 167KBps throughput), and only 33 cars
(MPS) finished the race. NT's throughput is five times NetWare's throughput, and
NT has 17 more motors running than NetWare.
What If ...
Someone in the Lab observed that for the last two steps of the test,
NetWare's ART was getting faster, and NT's ART was getting slower. What would
happen if the test continued with more motors? I reran the tests, with a 24.2MB
dataset, a 100KB block size, a 10-second think time, and six steps (with 10, 20,
40, 60, 80, and 100 motors, respectively), and the results again favored NT.
NetWare's maximum throughput was 208KBps with an ART of 142 seconds, running 83
motors. NT reached maximum throughput at 720KBps with an ART of 42 seconds,
running 100 motors.
Post-Race Analysis
To keep the racetrack equal for both teams, I maintained the same physical
network connections, the same protocols, the same physical clients, and the same
physical test server--down to the same physical hard disks. Within Dynameasure
for File Services, I kept identical test specifications (file size, type of
transactions, and number of motors) for each operating system. I gave both
racing teams the same track to race on, the same type and number of cars to
drive, and the same amount of time to complete laps.
After running race after race and watching NT leave NetWare in the dust, I
finally concluded that NT is indeed the better performing operating system for
file services. Does that mean that you should throw out your NetWare servers and
replace them with NT? You be the judge. After all, every garage in the country
does not have a new sports car. But the next time you go shopping, remember
which operating system has the performance edge.
End of Article
Comparing the latest version of NT with all service packs to an old version of NetWare without service packs is not objective by any stretch of the imagination. And, I hope that the prices listed were suggested retail price because I found a lower price for each product on the Web. Not only is NetWare less than half the price you listed for NetWare 4.1, but its price beats NT Server’s.
How can I feel anything but that this comparison was set up so that Novell would fail against Microsoft? If this article were in the April issue, I would think that this review was just some sort of sick joke. Instead, I feel that a fine magazine has lost integrity. How can I know that other articles won’t be so blatantly biased? Will you run a file server test with IntranetWare with Service Pack 3 on one side against NT Server 3.51 with no fixes on the other?<br>
--Eric Quinn
Eric Quinn August 10, 1999