RSSA word about our Windows Web browser test suite
By Scott M. Fulton, III | Published May 28, 2009, 3:27 PM
Since March 2009, our Windows Web browser performance test suite has utilized four components which score different aspects of the browser engine: the HowToCreate.uk CSS rendering test, the Celtic Kane basic speed comparison, the Web Standards Project's Acid3 standards compliance test, and the SunSpider JavaScript benchmark. We've received approximately equivalent numbers of praises and complaints for our having chosen all four of these independent analyses for our suite, though we see no reason at present to discredit any of them.
We did have to modify the HowToCreate.uk test internally, because the way it typically accounts for its own speed does not take into account how the onLoad event fires differently with various browsers' JavaScript interpreters. Our modifications account for the discrepancy, and we applied those modifications to all the browsers we test, not just those (Google Chrome, Apple Safari) which fire differently.
Each component of our suite counts toward 25% of each browser build's final score. That score is a composite of scores relative to the performance of Microsoft Internet Explorer 7 in Windows Vista Service Pack 2. In a fresh installation of Vista SP2, we noted the times and performance levels of IE7, a relatively slow browser. We then gave IE7 an index score of 1.00, so that we can compare other browsers' performance to IE7's on a general basis. A browser that scores 5.42, for example, performed about 542% better than IE7 in Vista SP2.
The reason we included Acid3 -- a non-speed-oriented test -- in our composite score is because we believe it's important that a browser not only be fast, but do the job it's expected to do. Typically, a browser is expected to adhere by standards, at least from the perspective of the developers who build Web sites for browsers. Thus, a browser that scores 5.42 whose Acid3 score is only 75% conceivably has the opportunity, with a bit of tweaking, to score a 7.23 if it follows the rules. Consider the Acid3 our version of applying "style points."
Currently we test relative performance using an identical machine with a multiple-boot option, enabling us to boot different Windows versions from the same large hard drive. Our platforms are Windows XP Professional SP3, Windows Vista Ultimate SP2, and Windows 7 Release Candidate.
All platforms are always brought up to date using the latest Windows updates from Microsoft, prior to testing. We realize, as some have told us, that this could alter the speed of the underlying platform. However, we expect real-world users to be making the same changes, rather than continuing to use unpatched and outdated software. Certainly the whole point of testing Web browsers on a continual basis is because folks want to know how Web browsers are evolving, and to what degree, on as close to real-time a scale as possible.
Our physical test platform is an Intel Core 2 Quad Q6600-based system using a Gigabyte GA-965P-DS3 motherboard, an Nvidia 8600 GTS-series video card, 3 GB of DDR2 DRAM, and a 640 GB Seagate Barracuda 7200.11 hard drive (among others). The Windows XP SP3, Vista SP2, and Windows 7 RC partitions are all on this drive. Since May 2009, we've been using a physical platform for browser testing, replacing the virtual test platforms we had been using up to that time. Although there are a few more steps required to manage testing on a physical platform, our readers have indicated that the results of physical tests will be more reliable.
Rating explained? In the third paragraph you state: "A browser that scores 5.42, for example, performed about 542% better than IE7 in Vista SP2." This logic is flawed. I think you mean "442% better than IE7". Otherwise based on your explanation of the ratings if a browser scored 0.75 it would be 75% better than 1.0 (rather than 25% worse.)
Score: 0
|The sunspider javascript benchmark isn't a valid test. It attempts to do commands that aren't valid (according to the javascript specification), that the mozilla based browsers have incorrectly implemented. This causes the tests to do different things in other browsers (like IE), and invalidates the results. Not surprising that those portions of the test are written by Mozilla themselves, lol.
A better answer is that opera and firefox should get a "FAIL" for the benchmark, while IE passes. Quite odd that IE does it correctly, and firefox/opera don't.
Score: 0
|"Intel Core 2 Quad Q6600-based system using a Gigabyte GA-965P-DS3 motherboard, an Nvidia 8600 GTS-series video card, 3 GB of DDR2 DRAM, and a 640 GB Seagate Barracuda 7200.11 hard drive (among others)."
6/8GB of RAM and 64bit versions of the OS damnit! I don't care if the results will not be affected (since you're not using x64 tools for testing anyway). We're moving out of the 32bit era. RAM is dead cheap and even the lowest end boards support up to 8/12GB now. Lets see if swapping up the hardware a bit causes any changes in the results.
I don't believe in these kinds of tests unless the most current hardware is used, and though a Q6600 works fine (that's what I've got, actually), I think an I7 would be a more true number cruncher.
Score: 0
|Why does it say that this article was published on May 28, 2010?
Score: 1
|Because it was written in the future, duh.
Score: 0
|I was wondering the same thing....
Score: 0
|One thing that isn't clear here is what the absolute numbers are. It's all well and good to say that this or that particular Javascript snippet executes such and such faster, but that doesn't mean much if it only makes up 2% of the CPU time on your favorite webpage, and that CPU time is only 10% of the total, the rest spent idly waiting on the network. I'd venture to say that a 10% increase in network efficiency will drown a 100% increase in rendering efficiency, when it comes to actually getting a webpage on your screen. This is especially true if the CPU tasks are happening at the same time as network, which I imagine they are. If I have to wait 3 seconds for a file to download, what do I care if you take all 3 of those seconds doing something on the CPU vs doing it in half a second and then sitting idle?
All this concern about script speed seems more like marketing hype than anything a user would care about... but I may be wrong. Do you have any tests that actually show how long it takes to load google.com, msn.com, cnn.com? From empty cache to fully loaded and presented to the screen. And then how much of that time was actually spent in the CPU?
(I'm not dismissing the results, this stuff is important, it's just that I don't think you can take them and make a blanket conclusion that Chrome is 1500% faster than IE7 in any meaningful way)
Score: 0
|"HowToCreate.uk CSS rendering test"
Alas, this is a bad idea. You can't test Safari or Chrome in it accurately. It says so when you load it. They'll get speed advantages in your results that just don't exist in real life.
Score: 0
|Why don't you use Futuremark's Peacekeeper benchmark? It's no longer in beta and provides a well balanced suite of benchmarks. Whenever I'm testing, I use this benchmark (tech test) along with a loading webpages test (basic test) and my results are more useful in terms of the user's speed perception and what a browser's capable of.
Score: 1
|That bench is nice. However, it doesn't include canvas support in the benchmark score, which leaves out some of the newer builds of each browser. (again I think Opera leads the most currently in this realm.)
Score: 0
|"The Vista SP2 and Windows 7 RC partitions are both on this drive."
I wonder if one partition that exists on the outside boundary of the disk might impact speed?
Ideally you should have identical disks, one OS per disk, and defragged and cclean'ed before each run. Sorry to make this difficult. :)
Score: 0
|Since a hard disk drive is comprised of several disks, how exactly would you ensure proper placement of any partition? Besides, with such factors as disk caching and pre-fetching affecting speed at very minimal levels as well (Seagate would attest to the minimal nature of the speed difference), it really isn't worth using a physical disk where the sectors are manually apportioned. That's not a real-world environment anyway.
-SF3
Score: 0
|A hard disk consists of multiple platters. Modern hard disks have an increasing range of disk transfer rates from the inner diameter to the outer diameter of the disk. "Zoned recording."
Like I said, likely insignificant in today's world of mult-terabyte drives, but something to keep aware of if you are putting multiple versions of browsers across these disks. A simple defrag, wipe of free space and extraneous data, defrag should do the trick to reduce rotational differences. I would also, at a minimum, put everything (including browser disk cache) on a separate physical disk from the OS to reduce any access/latency issues with regards to the pagefile/operating system.
A simple test: Fill your 640GB drive with hundreds of GB movies, leave just enough room for your browsers, then install your browsers with the cache on that same drive, and test again. You will get inconsistent results with your test suite.
Score: 0
|Happy 2010!
Score: 0
|Yea, I know, isn't it cool? We're publishing background pieces from the future now.
Actually, it's a temporary fix to a bug in our publishing system, and once that bug is fixed, I'll restore the proper date.
-SF3
Score: 0
|