RSSAll-new test results: What browser will you use to run Web apps?
By Scott M. Fulton, III | Published September 21, 2009, 11:34 AM
Three laptop computers, all of them cool-looking, all with well-respected brands, all have the features you want, all sell for the same price. This isn't going to be a toy for you; it will be, for at least the next few years, the engine for your work and your livelihood. How do you make a purchasing decision? You check online to see which one is the better performer, and which one other customers prefer.
Five Web browsers, all of them cool-looking, all with well-respected brands, all have the features you want, all of them...are free. But this isn't going to be a newspaper reader or a Twitter feed carrier for you; it will be, for at least the next few weeks, the engine for your work and your productivity. Sure, you'll install all of them. But which one will you install as your default, and which one will you trust with your everyday applications?
When the everyday functionality of Microsoft Office moves to the Web, and PCs are sold not with Office installed but with desktop shortcuts to Office Web Apps instead, and when the applications you run depend on the Web browser you choose, the decision you make about Web browsers will be more important than ever before. If you care about whether an AMD processor that sells for eight dollars less than its Intel counterpart can perform at the same levels under overclocking, or whether adding a second graphics card will crank out ten more frames per second after you add that fourth monitor, then you should care about the performance of your software platform. The differences here are not so incremental.
At Betanews, we've been testing Windows-based Web browsers with greater and greater accuracy throughout this year, with the objective of being able to give you a simple and indisputable way to consider their all-around performance. All through that time, we've been listening to your responses as to how we can improve our methods, and we've been getting a lot of responses.
Here's what we've learned from you:
- You want a simpler, flatter index. Just as the Dow Jones Industrial Average represents the general state of investment in the American economy for any point in time, you need one number that represents the all-around performance of every browser in the field, something you can remember and discuss.
- You also want all the data. Specifically, you want to be able to see exactly how that final index number is obtained. Our verbal explanations haven't always been enough, and you know from personal experience that a browser we've called relatively slow is actually faster in the areas that matter specifically to you.
- You want us to cast our net wider, and find a fairer and more accurate way to assess basic performance. You've told us that page load times, to you, represent basic performance -- if a page loads faster, it's a faster browser. But timing how fast a browser loads Yahoo or Facebook, for example, is a process loaded with uncontrolled variables -- the pages change, the network ebbs and flows, and ads can be textual or in interactive Java 3D. You need a fair and regulated means for assessing real-world page loading speed.
- You warned us not to trust browsers' different methods for reporting their own load times. Since different browsers work in various ways, for example, they fire the JavaScript onLoad event at different times, for different reasons -- so we can't accept onLoad as the complete "finish line." Instead, we need to pay attention to the whole page loading process, specifically with regard to how soon the scripts inside a page can access the elements of that page and start styling and displaying them.
We already had four tests in our previous Web browser suite, and after careful research, we've added four more that focus on areas that you say matter to you, and that some of you say we've overlooked. And as usual, you're right: If we're going to claim to measure "all-around" performance, we need to cover all the bases.
On the next page, we introduce our new solution and our response to your many, many very good suggestions. This is your comprehensive index.
Next: How Betanews will measure Web browser performance...
Mozilla firefox is the best browser to run all kind of web application in internet..
http://www.i-netsolution.com
Score: -2
|You need to add what the X axis represents in each graph and if larger or smaller is better for each one.
Score: -1
|You setup a whole new test bench on a old service pack of an old OS using an old browser as a baseline, yet you put newer browsers, even beta and alpha browsers together. You can't seriously think people will respect this can you?
First of all, when you benchmark, clarify your goals. Are you benchmarking the OS or the browser?
Secondly, I still greatly disagree with the test hardware on three partitions of the same drive. Three drives of the same model, three OS's. We have no idea the interactions of the bootloader or where on the physical disk can impact the performance of your OS.
Score: 0
|I'll use the browser with AdBlock+. Load times on web pages are far more interesting when you cut out CPU and bandwidth hungry Flash apps. It also makes web browsing far less annoying.
I do enjoy using Chrome, but without some ad blocking, I get sick of it in about 30 minutes. :P
Score: 0
|Use a hosts file:
http://someonewhocares.org/hosts/
That way all your browsers are blocked from ads.
Score: 0
|I am getting so incredibly sick of Firefox lately... I really just wish someone would create an addon (free, of course) for Chrome, IE, or Opera that has the listsubscription/blocking functionality of adblock and one for the session management of TMP.
That's It. I can live without everything else... I'd change in a heart-beat.
Anyone know of some 'extension-compatible" firefox forks?
Score: 0
|@mjm01010101
Hosts file doesn't cut it. AdBlock removes the placeholders, updates it's lists, and allows you to block or unblock on-demand. The Hosts file doesn't even come close to that.
People keep bringing it up as an alternative, and I get it..it *is* an alternative...but it's a horrible one.
Score: -1
|I use both.
I agree that firefox + adblock plus is a better overall experience, but chrome + hosts is still faster browsing for what I do day to day.
Score: -1
|@ PC_Tool
http://my.opera.com/Tamil/blog/ad-block
Which also references the following, IIRC:
http://www.fanboy.co.nz/adblock/opera/
http://operawiki.info/FlashBlock
http://my.opera.com/Lex1.../flashblock-for-opera-9
Score: -1
|@Balderstorm:
Closer. We're getting there...
Score: -1
|1) Security?
2) Standards compliance? Really?
3) Bandwidth discrepancies?
4) User visable speed differences?
5) RAM and/or resource usage?
6) Features?
7) Stability?
If you are going to leave in standards in a 'performance' test you should really cover the whole gamma of things affecting why people would use a browser.
For example some things to note:
1) IE7/IE8 on Vista or Win7 runs in a lower security sandbox, protected mode. Since Vista and IE7 was released, the most secure way to browse the Internet is IE7 or IE8 on Vista or Win7 because of protected mode, as the browser can't even harm the logged in user's files.
And I hate that this is factual, as I would love to see Chrome or Firefox implement a sandbox that demotes the level that the browser runs at with regard to the OS's security, on Non-Windows Platforms this would be a good idea as well to protect user data.
Also what 'performance' penalty is this security creating? Was your IE tests ran with protected mode on or off? Was other performance dropping features of each browser also turned off?
For example on IE, if you have Suggested Sites enabled, it does a lookup when you load a page. IE and a couple of other browsers also do extensive Phishing checks on the Web address, that can delay a page's load time depending on the status of the server it is using to check the address. Do you factor in any of this stuff, and enable or disable it accordingly?
Do any browsers 'phone home' additional data to Google or Bing? There was a time when Firefox's agreement with Google had the search feature giving Google more data than they should have been doing. Does Chrome do this still, and if so should that be considered in the rankings?
2) Standards? The Acid test is really a test of how 'hard' standards are implemented, so that 'broken' HTML/CSS also is properly rejected.
This would be like 1998 when comparing Netscape with IE, and the end TABLE tag would be missing on a page (or not received) and Netscape would fail to display the Table at all, yet IE would go ahead and render the Table.
However, by doing this, IE was 'breaking' the standards. Which at the time, especially with flaky ISPs and modem connections was a GOOD THING for users.
Standards are great, but one that demonstrates how 'broken' features are handled is not so bright. And this is basically what Acid is doing, and Microsoft has chosen to not comply with the Acid tests for this very type of reason, just like they chose to render the Table in 1998 even though the browser never got tht end of table tag.
3) How do the various browsers handle flaky connections? This would be important to a user like myself that often is bouncing between Wifi, 3G and 2G connections. I know from my own experience, that Chrome is really bad in a flaky 2G connection compared to Firefox or IE.
4) The difference between 1.0 and 16.0 on your scale is still below what a user can perceive in speed. Can you find any actual speed differences that are even in the realm that a user can notice anymore? Even load times are invisible to a user, especially on Vista or Win7 where superfetch is keeping the browser ready to load.
5) On lower end systems, who is using several times the RAM, and in what situations? I know Chrome can be RAM hungry sometimes where IE and Firefox are light on the same sites. (PS IE no longer has any advantages for load times or RAM usage as in Vista or Win7, as they are fully removed from the Shell)
6) What features do they have and what do you have to give up? When someone uses Chrome they are going to lose their Accelerators and Slices and if they use IE they are going to lose the textbox spell-check features that Firefox offers. How important is the drop down search features of IE, and what add-on/plug-ins are available for each type of browser to offset the feature differences? Also what differences are there when rendering Flash or Silverlight, as they are pretty common now?
7) Stability is important, how often and what sites can trigger failures in any of the browsers? I know of a couple of sites that can lock IE and I also know several sites that will lock Chrome in a nasty loop as well. And again, how well do the browsers handler exceptions in plug-ins/add-ons - how easily can Flash on a HT enabled system kill the browser - which Flash's poor multi-threading code can easily do on virtually every browser. Which browser's recover from a crash the easiest?
My final thought, I think it is good that you are trying to create a 'solid' approach to comparing the browsers, but you need to be all inclusive to be comprehensive or it is going to look like you either don't know what you are doing or are intentionally letting the results be skewed based on favor.
If you want to call this a performance test, you need to do ONLY PERFORMANCE and ensure the metrics of services/features are equal when doing so.
If you want to do a more comprehensive test, stop calling it a 'performance' test and include things like the Security 'factor' and the stability factor. Just as you dinged IE and Firefox for Acid compliance, you should have given IE extra points for the protected mode that sandboxes it to a below 'user' level of security.
I hope you keep working on this project and get some good metrics going.
PS
To everyone out there, you are not going to see a 'faster' experience in ANY of these browsers, as the 1.0 to 16.0 difference is in the millisecond range, so use what you like and the features and security you like.
Score: 2
|Here, Here! and you get a 1. The information you have just provided is why I hate statistics. If anyone is to do a comparison, then it must include all features and restraints.
BetaNews should take this to heart and improve their scoring system, as blind-folding someone and pinning the circle on a number. Then incorporate tests that truly can show comparisons. Being in an informative news business, it should be real news they give. From these numbers one can conclude, erroneously, that Chrome is better and IE is the worse. I say, worse at what? Security or speed?
Score: -1
|Strange that there is an add for IE 8 on this page, when it gets so lambasted here. Looks like speed is inversely related to money spent developing. So MS hires hundreds of programmers who all add "features", each one of which slows the browser down by a little bit.
Score: -2
|An interesting test I ran across:
http://benfirshman.com/projects/jsnes/
I am still trying to figure out why this thing runs so much slower on Firefox & Safari than Chrome. Anyone have any good ideas on some JS benchmarking idea for this one? From the SunSpider test, I can't really say that provides enough info to clearly identify.
Score: 0
|"Creepy"? I think "Crappy" would be better!
Score: -5
|Yawning till penalty for standards non-compliance is removed....
Score: -4
|Yawn
Score: -4
|