GCN adds benchmarks to test arsenal

By Michael CheekGCN StaffReviews that rely on science and not art tend to forget the human in the equation. So what if a blazing-fast video card or hard drive makes one system perform better than others if you cannot get it onto the office network?The GCN Lab views benchmark tests as one of several important ways to gauge how well a PC, notebook computer or server will perform in a government office.Even as the lab launches a benchmark alliance with Ziff-Davis Benchmark Operation of Morrisville, N.C., concern for the real-world user experience will restrict the ZDBOp benchmark results to a small percentage of the overall grades assigned.For the past 31/2 years, the lab has used the GCNdex32TM, its own exclusive benchmark suite. When the lab put GCNdex32 into action in early 1996, it measured the integer math, floating-point math, video, large-file, small-file and CD-ROM performance of six brand-new 150-MHz Pentium Pro PCs.Soon, however, the Pentium Pro clock speed accelerated and the Pentium MMX emerged, followed by the Pentium II, the Celeron and the Pentium III. Processors today are four times faster and many times more sophisticated than they were just three years ago.Back then, a computer buyer was lucky to get a video card with 2M of memory. Now 32M is commonplace. New hard drives can pack 20G; one-tenth that much was the norm three years ago.In 1997 I realized that the GCNdex32 benchmark would soon need to be retired, and I began looking for a new suite that could better delineate the performance of faster systems. The lab considered updating the GCNdex, but it needed a more comprehensive set of tools.As GCN Lab director, I discovered in talking with GCN readers and other federal users that they often ask computer vendors such as Compaq Computer Corp. and Dell Computer Corp. to provide ZD benchmark results before making a purchase.If readers wanted the ZD benchmarks, I thought, the lab should provide them. It turned out that of the 20 or more mainstream benchmark suites available for measuring performance, ZD's is consistently regarded as the most accurate.Moreover, if you wonder how your office's new Pentium III PCs compare against the lab's test units, it is pretty easy to visit ZDBOp's Web site, at www.zdbop.com, and download for free the benchmarks to run on your own. You can also purchase the benchmarks on CD-ROM for $5.Benchmarks fall into two broad categories: those that isolate various system components and those that take a comprehensive look at the overall system.Isolation benchmarks, sometimes referred to as the synthetic or playback type, examine specific components such as CPU, video and storage. Comprehensive benchmarks take into account how all the components work together. They run scripts or macros to launch a series of real-world application tests.GCNdex32, which is officially retired with this issue, is a purely synthetic benchmark. For example, the floating-point and integer math portions show how fast a processor can complete certain math calculations'one measure of the processor's power.Modern processors, however, also do multimedia, video, sound and other sophisticated computations. The GCNdex32 benchmark cannot measure them.Moreover, a processor can call on other components such as Level 2 cache and primary RAM banks to help it perform faster. Even the connectors between the processor and cache and between the cache and RAM, commonly known as the front-side bus, contribute to computational prowess.You might think of cache and RAM as waiting rooms in a hospital. The front-side bus represents the support staff. The bigger the waiting rooms and the more efficient the support staff, the faster a patient sees the right doctor.The GCNdex32 benchmark could not evaluate all the factors that go into processor performance.ZD's CPUmark 99 does. CPUmark 99 is part of the WinBench 99 Version 1.1, a suite of tests designed to isolate subsystems in a PC or notebook.In the July 12 issue on Page 1, I reviewed three notebooks that had three different processors: a 366-MHz Pentium II, a 333-MHz Pentium II and a 333-MHz Celeron. CPUmark 99 showed exactly what you might expect from the clock rates: The 366-MHz Pentium II performed better than the 333-MHz Pentium II, and both Pentium IIs outperformed the 333-MHz Celeron.The GCNdex32 showed the same thing, but with far less differentiation between processors. Taking the 333-MHz Celeron for comparison, the GCNdex32 showed the 333-MHz Pentium II was about 1 percent faster and the 366-MHz Pentium II was 8 percent faster.Under CPUmark 99, however, the differences were much clearer. The 333-MHz Pentium II gained about 12 percent over the Celeron, and the 366-MHz Pentium II performed 25 percent better.But statistics can be deceiving. Both of the 333-MHz notebooks had 64M of RAM, whereas the 366-MHz unit had 96M. Some of its performance superiority could have come from the larger RAM bank.ZDBOp programmers based CPUmark 99 on the most common processes that occur in real-world applications.'Because modern applications take advantage of larger cache and more memory, CPUmark 99 does, too.Also included in WinBench are graphics and hard drive performance tests. ZDBOp divides these and other tests into Business and High-end portions.Business benchmarks launch common tasks associated with basic applications such as word processors, spreadsheets and databases.High-end benchmarks examine more taxing work such as computer-aided design and graphics generation.In testing video subsystems, the GCNdex32 performs strictly 2-D draw tests. All of ZD's Graphics WinMark 99 video tests are based on real-world processes. Graphics WinMark combines all the common processes encountered with either business or high-end applications.ZDBOp also has an extremely sophisticated 3-D video benchmark that examines particular 3-D features of graphics accelerators. The lab is phasing in ZD's benchmarks and plans to publish 3-D WinBench scores in the future.ZD's Disk WinMark 99 measures a system's hard drive playback in thousands of bytes per second.The most extraordinary ZD benchmark is Winstone 99. It shows how an entire system performs on real-world applications.Like WinBench, Winstone is divided into two tests: Business Winstone and High-end Winstone.Business Winstone uses common office applications: Corel QuattroPro 8, Corel WordPerfect 8, Lotus 1-2-3 97, Lotus Word Pro 97, Microsoft Access 97, Microsoft Excel 97, Microsoft PowerPoint 97, Microsoft Word 97 and Netscape Navigator 4.04.High-end Winstone uses Adobe Photoshop 4.01, Adobe Premiere 4.2, AVS/Express 3.4, Bentley Systems MicroStation SE, PV-Wave 6.1, Microsoft FrontPage 98, Microsoft Visual C++ 5.0 and Sonic Foundry Sound Forge 4.0.Both versions of Winstone run through a series of scripts or macros to perform exactly the same tasks.In my tests of the three notebooks, I discovered that the 333-MHz Pentium II performed worse on Winstone 99 than the 333-MHz Celeron. If I had run only Winstone, I would not know why. But ZD's WinBench explained this surprising result.Both 333-MHz notebooks performed about the same in terms of drive access. Graphics accelerator performance was weaker on the Pentium II portable, however.Winstone 99 results appear as raw numerical scores, such as 15.5. The scores are called Winstone units, and higher is better.'' Winstone uses as a baseline a system with a 233-MHz Pentium MMX processor. Whereas the GCNdex32 has a baseline score of 1.0 for a 66-MHz 486, Winstone's starting score is 10.0.Therefore, a Winstone score of 15.5 means the test system performs tasks about 55 percent faster than a 233-MHz Pentium MMX.The lab will report Business Winstone benchmarks for all CPU units it reviews. It will also report High-end Winstone tests for all desktop PCs running Microsoft Windows NT.To learn about ZD benchmarks visit www.zdbop.com. For independent testing and technical specifications of units tested at the lab, visit www.gcn.com/gcnlab/benchmark.