RSSWindows 7 gives Firefox 3, IE8 speed boosts, while Firefox 3.5 slows down
By Scott M. Fulton, III | Published May 12, 2009, 5:18 PM
In preliminary Betanews tests Tuesday comparing the relative speeds of major Web browsers in Windows Vista- and Windows 7-based virtual machines, not only did the general performance of Microsoft Internet Explorer 8 improve by about 23%, but the latest production build of Firefox 3.0.10 appears to improve its performance by 17.5%. This despite running in a Windows 7-based virtual machine that we estimate to be 12.1% slower overall than a Vista-based VM hosted by the same environment.
These are the initial findings of Betanews' experiments in how the architecture of Windows 7 may or may not influence the performance of major Web browsers. We wanted to see whether Win7 made browsers faster or slower, and doing that meant hosting browsers in virtual environments whose relative speeds with respect to one another could be normalized.
As we discovered, Windows 7 RC Build 7100 runs perceptibly slower on a Virtual PC 2007 platform on XP SP3, than Vista SP2. This does not mean Windows 7 is a slower operating system, but rather that it behaves more slowly in this particular virtualized environment, which after all was designed for Vista. So to make our test fair, we needed to estimate just how much slower our Win7 environment was from Vista, and factor out that difference.
Up to now, we've been comparing relative browser performance in Vista using a relatively slow browser to judge against: IE7. We've used IE7 as our gauge of how much more readily other browsers blow right past it in the performance department, including IE8. But we don't want to install IE7 on Win7 -- although it's technically feasible, doing so would pollute the operating system for running Win8 and other applications. So we needed a new, slow browser that we could rely upon to stand still for us, relatively speaking.
Our first choice was Firefox 1.5, but we learned it had difficulty running in Win7 at all. We ended up using Firefox 2.0.13, not quite the final build of that series of Mozilla's browser. Our aim was to use this browser as a fair gauge of how much slower our Win7 environment was than Vista. This way, we could equalize our indexes, which are based on IE7 -- we can't run IE7 on Win7, but we can estimate how much slower IE7 would be if we could, by measuring how much slower Firefox 2.0.13 is. Though the average speed difference is 12.1% in favor of the Vista VM, for our browser benchmarks, we created differentials for each heat in the competition, to more accurately account for environmental factors between the two environments.
In the Vista VM alone, Firefox 2.0.13 puts in a performance index of 2.49, meaning it performs 249% as well as IE7 in the same environment. Compare that to Firefox 3.0.10's index score of 5.19 in recent Betanews tests in the Vista VM.
Factoring out the speed differentials, we can reliably say that IE8 gives us a performance index of 2.69 in the Win7 VM versus 2.19 in the Vista VM. Meanwhile, Firefox 3.0.10 scores a 6.10 normalized index score in the Win7 VM versus 5.19 in the Vista VM.
The news is not all good for Mozilla, however. Under the same test conditions, Firefox 3.5 Beta 4 slows down in Win7, but only by about 2.5%, scoring a 10.18 normalized index score in the Win7 VM versus 10.44 in the Vista VM. So from this angle, it appears that Windows 7 helps close the gap between Mozilla's production browser and its experimental browser. We're interested to find out whether similar discoveries await us with regard to Google Chrome, and whether Win7 will play nicely with Apple's Safari for Windows. Those results are still forthcoming.
Does anyone else see the following issue on Windows 7 using FireFox 3.5 beta4?
Go to http://fileforum.betanews.com
Ignore some software using the ignore feature. This brings forth a popup.
Wait for popup to confirm the ignore, then try and close it. It hangs FF for about 10-20 seconds every time for me.
Score: 0
|Scott ... You are a moron, get parted magic and install Windows 7 on a separate partition at least and then test it. this voodoo math is garbage and irrelevant.
Score: 0
|Scott...one of the most fundamental things in testing is to NEVER run performance benchmarking in VMs....
flawed testing methodology == Garbage in / Garbage out...
BN Fail
Score: 2
|Interesting. I am sure that the other browser makers will cry foul and take up the issue !! :)
Score: 0
|Scott,
XP made minor adjustments to certain configuration options based on your hardware. Vista took this to a whole new level and made changes to how it handled much of it's IO and memory handling based on your WHEI score....
Windows 7 adjusts nearly everything from processor priority, multimedia handling, buffering, caching, IO....all radically in response to your WHEI score.
Why is this relevant? In a VM, your WHEI score is wildly inaccurate (due to how VM's cache writes, Memory functions and virtualize hardware) and the more the OS depends on these to configure itself, the more wildly inaccurate any benchmark within that VM becomes.
Point being? Unless the tests are done on bare-metal, they are, for the most part, worthless.
Please do future testing without the VM. The fewer variables and configuration inaccuracies involved, the better.
Score: 3
|But PCT, My car performs so much better in a wind tunnel or on a set of rollers! Isn't that indicative of real world performance? Many major automobile manufacturers seem to think so. Are you saying they are all wrong?
Wait, yes they are. And so are these results which have no merit given that I thought betanews was a real news site which could afford actual TEST MACHINES as opposed to TEST VIRTUAL MACHINES. Credibility for BN on this article, ZERO.
Score: 1
|And it's not even like they need test machines. All they need is a swappable drive bay and a few hd's and extra trays.
Score: 0
|Methodology aside, that's a lot of random versions and percentages... visualizing things might be useful...
Score: 1
|Why not compare everything to IE8, since that'll be the dominant Win7 browser? Forming a baseline using a browser that cannot be installed properly (IE7) isn't useful. It might have been with Vista (or XP), but not Win7. And having negative numbers in your results would be perfectly acceptable.
Also, comparing Firefox 2.x or 3.0.x isn't giving us relevant data. Firefox 3.5 will be out long before Win7 is out, and it is very unlikely Win7 users will install the older Firefox versions.
I agree with the other posters about not using VMs as your test environment. The results are not going to be accurate.
Score: 0
|My virtual machine tests conclude that firefox 2.0 runs faster on a non-virtualized machine than firefox 3.5 on a VM. HAH!
Score: -1
|"Microsoft Internet Explorer 8"
If BetaNews are Microsoft brown-nosers, they'd get IE's current title right.
Windows Internet Explorer 8 (has been since 7).
Score: 0
|So wait, can betanews not afford test machines and instead gives results based on VM trials alone? VM Trials are not equal to real world machine results. I don't know very many people who are going to run a browser on their home machine in a VM....
Score: 2
|"I don't know very many people who are going to run a browser on their home machine in a VM...."
Heh...
Never underestimate the paranoid. ;)
Score: 0
|True. Some POWER USERS might but I seriously doubt grandma and grandpa are going to see these "results" and say "I must run this in a VM, Just look how fast it performs!".
I am sorry, but Betanews is losing more and more credibility. From the flawed results, early postings of firefox releases to be the FIRST to do so, to the articles angela and jacqueline write which have no relevance to the software world and appear to just be filler when a slow news day is ongoing.
Score: 0
|Can we get some specs for this VM/host you are running and for future "benchmarks"? How are we to know if some bottleneck in your benchmark isn't causing strange discrepancies?
Score: 1
|Scott, everytime I see one of your posts talking about benchmarks run in virtual machines, it makes me cringe. Some applications will perform totally differently when run in virtualized environment. If you ran the same tests on a native, non-virtualized OS, you might discover totally different results - some products might even switch places in your performance charts.
As a system administrator who manages servers that are virtualized, I can speak from experience there, having run into an issue with a mail server software that lost a LOT of performance when virtualized - more than a competitive product that I was using on another virtual server.
Please consider doing benchmarks on a native machines, reimaging the OS between tests so you will be actually comparing apples with apples.
Score: 8
|LOL! Almost the same post at the same time.
Score: 1
|