RSSTests: Firefox 3.1 JavaScript outpaced by Safari 4, Google Chrome
By Scott M. Fulton, III | Published March 13, 2009, 12:05 PM
Internet enthusiasts today are getting their first glimpse of the TraceMonkey JavaScript interpreter that Mozilla claims will be one of the principal reasons to own and use Firefox 3.5 -- what the latest Firefox will inevitably be called once the numerology gets sorted out. A fresh round of comprehensive Betanews tests Thursday afternoon indicate that Firefox 3.1 Beta 3 will demonstrate close to eight times the general JavaScript calculation and rendering performance of Microsoft Internet Explorer 7 -- a clear performance gain.
But as we were able to verify Thursday, it may not be nearly enough to earn the latest Mozilla beta the performance lead. In a battery of 20 calculation and rendering tests assembled from four different developers' benchmarks and benchmark suites, Apple's Safari 4 beta for Windows perceptibly and appreciably outperformed every other test browser on our virtual machine.
There is no widely accepted industry standard benchmark suite for Web browser performance just yet, but there are quite a few potential nominees. Over the last few weeks, we've been discovering some that were created by independent developers and shared with the rest of the world through Web sites.
In an effort to produce a fair and unbiased rendering of JavaScript engines' relative performance -- something closer to what you'd expect if we were comparing six different grades of CPU to one another -- we gathered a group of respectable benchmarks that appear to cover the gamut for different types of workloads: cryptography, data handling, DOM handling, AJAX declarations, parsing regular string expressions, string manipulation, mathematics, rendering, compliance with standards. And we re-cleaned and re-assembled our Windows Vista SP1-based virtual machine -- again, not the best candidate for a "super-speedway," but a plausible venue for an unencumbered test of how the latest experimental beta, alpha, and release candidate versions of major Web browsers perform in comparison to one another. (Though Google lifted the "beta" designation from its Chrome browser, its development is continually tested and its clients are automatically updated to reflect results from those tests, so we included the latest Chrome edition with the other brands.)
Firefox 3.1 Beta 3 was released to the general public yesterday. We spent a good part of yesterday comparing that release to the first Apple Safari 4 beta for Windows -- which performed astonishingly well in our initial tests -- as well as Google Chrome, Microsoft Internet Explorer 8 Release Candidate 1, and in response to many of our readers' requests, the latest alpha edition of Opera 10.
Knowing that our test venue would be an arguably slow virtual system, our goal here was to test relative performance -- to illuminate the scale of performance differences that all users would likely see on their own, probably faster systems. The way we chose to represent relative performance is by creating a base index -- something we already know to be slow and that we expect will be outpaced.
So we ran our series of tests on the most recently updated version of Internet Explorer 7, prior to upgrading our test VM once again to run IE8 RC1. For each test, we gave IE7 a score of 1.0, to serve as our baseline. We then used four benchmark "suites" for which every final result is rendered relative to that 1.0; so for example, a score of 2.3 on a speed test would mean the browser was 230% the speed of IE7. And for the Acid3 benchmark, which tests compliance, an index score of 8.0 for a browser would mean its rendering score would be eight times better than for IE7.
Next: The tale of the tape...
i think id still have to do my own test to find what im looking for
i need a browser to load fast and have automatic refresh on so i can load faster than other ppl playing internet browser games, ff does it quite well but ill give safari a chance the same as i did with all the others, opera seemed ok but crashed out when you opened a few diffrent windows, with ff i can open 15 windows on automatic refresh every second with only a bit of slowdown to show for it opera and crome crashed when i got past 3 windows
add ons do slow a browser down but some of them essential for saving the fingers from repetitve strain injury
Score: 0
|So let me get this straight:
1) You take 3 speedup ratios and one ratio of Acid3 scores and somehow create a "final score" from them. I have no idea how one munges such disparate quantities into a single number... Even just averaging speedup ratios is wrong, unless you very carefully control the things whose speedups you're measuring. In any case, I see no description of how the overal score was derived.
2) You use a known-broken test (Sean Celtic Kane's), which doesn't measure what it claims to measure.
3) You use a test (HowToCreate CSS Rendering load) on which Safari's lying about when it fires onload makes a big difference (hence the Safari score being 8x lower than that for any other browser, including ones based on the same webkit engine).
4) You use a test (HowToCreate CSS Rendering render) which doesn't actually test CSS rendering performance last I looked at a profile.
Conclusion: Your results are pretty much meaningless. Now if one looks at an actual carefully written test suite which tests what it claims to test (e.g. Sunspider or dromaeo), it's certainly the case that the Safari 4 beta is faster in a lot of cases than the other browsers you tested. But not by nearly as much as your tests show, and there are plenty of other cases where it's slower.
This is probably the worst article on browser performance that I've read recently. Instead of admitting to not knowing what it's talking about, it tries to hide that fact in a lot of pseudo-scientific mumbo-jumbo.
Score: 3
|there were some regression bugs during the merging of TC and M-C. It should be fixed very soon, and should be reflected in 3.5 b4.
Score: 0
|As I just tested minutes ago, there are apparent some bugs related to this have been fixed. the sunspider benchmark improved more than 10% so far from beta 3.
Score: 0
|Interesting. That would explain why beta 3 was slower than beta 2 on Sunspider:
http://davidnaylor.org/b...-slower-but-faster.html
Score: 0
|sounds like including firefox 3.0.x is a good idea for comparison reasons.. the same can go for safari 3 or opera 9, if you were to do it that way though
anyways, we can see alpha/beta and RC versions of different popular browsers, but we don't see such for Chrome.. the Chromium builds are far ahead from the official Chrome releases, scoring 100/100 (with small glitches..) on the acid3 test, so it sounds about right to throw Chromium in the mix if safari beta, firefox 3.1 beta, opera 10 alpha and IE8RC1 are there
Score: 1
|Quite interesting, i checked up the whether TraceMonkey was enabled in the 3.1 b3 on chrome and is wasn't (javascript.options.jit.chrome was set to false). Scott, did you verify that setting before the tests?
Score: 1
|I didn't notice this!
For the others - enter about:config - type, jit - make both the values true
Score: -1
|Uh... no. Don't do that. It's on for web pages (the .content pref). It is NOT ON for the browser user interface (the .chrome pref) because there are known issues with that. If you enable that, expect crashes.
Score: 2
|I would not worry to much about these tests or which is better or faster. In the end, they all have to work correctly for future days to come and pass all the tests or get left behind. So all these things really mean nothing as tomorrow it could all change again. I use IE 8 and Firefox 3.0.7 and I like them both, but fire fox a little better because I can have it the way I want. Firefox will come up to standards in time, that is what they are working on. Just how many browsers do these software makers think we need, couple hundred more?
Score: 0
|I tested three browsers on FutureMark
IE 8.0.7000.0 scored 163
FF 3.07Beta Scored 314
Safari 4 Beta 528.16 scored 528.16
This was done on Windows 7 7000 Build
Score: 0
|Can you add in this benchmark the memory usage? (How much RAM does a browser consume). Any of you who are familiar with these browsers, can you give a rough estimate? Thanks.
Score: 0
|Firefox makes up for its "defects" by being able to get highly customized.
Competition comes only when Safari and Chrome get add-on features.. if they do, I think it will affect the performance!
We'll know as it comes. But as of now, I'd stick with Firefox; the complete browser!
Score: 4
|I agree regarding the customization. I use Firefox for more sensitive matters at home, banking transactions and the like, because I still haven't come to trust the private browsing feature on Safari. However, for general browsing I revert to Safari, as I've noticed Firefox become slower and slower with each revision.
Score: 0
|After reading a recommendation, which said - Completely Uninstall Firefox - I've felt a great improvement in FF3.1 b3 performance.. very quick rendering of pages.. and it somehow feels that the browser seems to understand the web pages better now.. Completely uninstall includes removal from everywhere - locals settings etc and the registry too.. give it a shot!
Score: 0
|That's just great. What version of Chrome is used in the test? Probably 1.0. IE7, IE8 RC1, Opera 10, Safari 4 beta, Firefox 3.1 beta 3 and just Chrome, no version? Probably the slowest one 1.0. Thats is just not right. But there is for sure reason for that, are you guessing????
Here is more realistic bench http://www.futuremark.com/peacekeeper
Score: -3
|Yeah, Betanews has a hidden agenda to crush Chrome. Get real.
Score: 0
|Perhaps you didn't read the article.
"(Though Google lifted the "beta" designation from its Chrome browser, its development is continually tested and its clients are automatically updated to reflect results from those tests, so we included the latest Chrome edition with the other brands.)"
Sounds to me like they used the latest publicly available version of Chrome.
Score: 0
|What we also have are what I see are different philosophies branded into each browser.
IE: The general public. They don't care about other browsers, it is always what they have used, and they aren't even curious about what else is out there, due to ignorance. You also have some geeks that have found add-ons to IE to make it worth their while. You have corps that pretty much mandate this browser, and group policy it to make it more secure and appropriate for their environment. Historically quite insecure, run this puppy in a Vista/7 userland sandbox if you ever do use it.
Firefox: An old geek favorite. It initially was loved for its small size, extensible architecture. The latter is also it's achilles heel, as numerous extensions/themes make the browser less stable, and upgrades aren't always smooth. This browser arguably gives you the most power for viewing online content exactly as you prefer. You can make this browser quite secure with a few select add-ons.
Opera: Always amongst the top performers, the leading feature browser out of the box. Uber-geeks use. Also historically seen as more secure either by being less of a target because of marketshare, but also the very fast response time when they do fix security issues. Sometimes the features are a tad overwhelming, as there are almost too many ways to configure this browser and I bet they scare a lot of newbs off.
Safari/Chrome: I group these together because I see them as so similar. You either like Apple's philosophy or Google's, and if you use these browsers you probably are a fanboy of either. Quite fast in javascript, but javascript is hot now, who knows 5 years from now. These browsers seem to have user interface quirks, you either love 'em or hate 'em. I expect both of these to start to take share growth from IE and Firefox.
My prefs: FF3, Opera, Chrome, in that order. I won't use IE or Safari. If I could just get a better element blocking in Opera I'd probably swap, as I do like its rendering performance. Chrome crashed too much in my tests, but I will give it another try after more testing by users...
Score: 3
|For your element blocking needs in Opera, try this. Fanboy does a great job maintaining both lists:
http://www.fanboy.co.nz/adblock/
Score: 0
|Very good comparison.
Yesterday I compared Firefox 3.1 beta 3 to beta 2 for Javascript performance. The results were interesting:
http://davidnaylor.org/b...-slower-but-faster.html
Score: 0
|You should list what build of chrome you use. There is no "1.0" in common release currently.
Score: 1
|My biggest surprise right now is how few addons have been updated for FF 3.1b3. Isn't this supposed to be a new milestone build? Oh, and great article BTW.
Score: -3
|It's been my experience with the 1.5, 2.0, 3.0, and 3.1 beta nightly builds that the extension developers almost always have their products available at the launch and not a second earlier, probably to assure maximum compatibility with minimal confusion. Besides, there will supposedly be another beta build prior to the release candidate and the final production version.
Score: 0
|How few?
Out of the extensions that I frequently use (Adblock Plus, NoScript, TabMixPlus, StumbleUpon, Gmail Notifier, and Forecastbar Enhanced), the only one that was incompatible with 3.1b3 was TMP.
Though NoScript was compatible, I'm finding it more annoying and less necessary than I used to. Thinking of ditching it altogether.
Sounds like extension developers are on top of things to me.
Score: 0
|"During the era of the so-called "browser wars," the only real matchup was between Netscape Navigator and Microsoft Internet Explorer. And that wasn't even really a battle on equal terms, but rather one between performance and functionality versus convenience."
Nope, Netscape started to really Bloat, crash and Suck.
Score: 0
|Isn't that about the time that AOL took it over and totally raped and pillage it?
Score: 0
|I wonder what FF performance with adblock and noscript is like compared to loading 2+ external ad networks on every Google Chrome and Safari page? Of course, when those browsers get element blocking in as well, test with those features.
Score: 0
|The ultimate test is using the product to find which one really works out best for the individual. I have yet to find a better browser, over all, than Firefox, though Opera has some great features. Firefox renders more pages correctly, of those I use, than Opera. Firefox has better features than Chrome, when the add-ons are used. As for speed, heck with DSL compared to speeds in 1995, the most clunky browser in the world would seem like a speed demon. FF seems speedy enough for this kid. It is the entire package which makes FF the deal for me. Depending on which sites you use, you may also find Opera, very good.
Score: 0
|I care about speed, but compliance is most important, as it should be for everyone. The browsers that can't score 100 on Acid 3 should not be considered in the running.
Score: 0
|Numbers are for the bean counters and to give a developer bragging rights which is AMO the worst way to "sell" their product.
If you look at every aspect of a browser which most important and base them on crashing, security, rendering, speed, and compliance using your train of thought there would be no browsers in the running at all.
Score: 0
|Great, Scott used finally the latest versions available. But (there is allways a but) something smell fishy.
In real life Safari DOES NOT render pages faster than Opera 10 or Firefox 3.1b3... I do not know where the tests were flawed, but they are!
Score: 0
|You should of included the shipping version of Firefox so that Firefox users could tell how much faster it's going to be. Also can you tell the difference between these or are they just fractions of a second.
Score: 0
|I believe in a test prior to this, they did use a final release of FF, but because it did so horribly compared to the alpha and beta versions of other browsers, some people here whined and cried bloody hell murder and that Betanews should've used an alpha or beta of the up and coming FF that had it's latest script engine in it. So it seems in this test they did, and it seems FF still failed to meet or exceed some of the other browsers.
In the end though, people should be more concerned with safety and rendering compliance more than anything. You need to be able to browse without getting jacked by some script kiddie's page hacks. You need to be able to view a web site as it was intended to be viewed by it's author, and not how some giant corporation thinks you should see it...
And to you Opera fans.... Don't sit safe thinking your fav browser there is as safe as you think it is. I myself used to be an Opera fan... that is, until it allowed a web site to download and execute malicious code on my box. While I still appreciate the efforts of the developers behind Opera, I personally won't touch it with a 10 foot pole ever again. You know how it goes.. Once you get burned, you're less likely to want to do it again. It may be a very long time before I trust Opera enough to try it again.
For safety purposes, I'd suggest FF with adblock and noscript. Might be a pain dealing with noscript, but sure as hell it'll save yer butt from just instantly getting hit by haxed page code somewhere.
Chrome(or SRWare Iron for a version that doesn't have tracking code in it. http://www.srware.net/en/software_srware_iron.php) and Safari, both being based on the Webkit, seem to be good for casual browsing, but not for power browsing. They seem relatively safe, but who knows what lurks out there that could hit either one of these.
And that leaves me with IE.... I'm still having flashbacks and nightmares of all the security holes it was riddled with back in the day. Sorry, still don't have the jewels to even think about using it for everyday purposes. Though I do use it locally only for checking my web page designs for compatibility.
Score: 1
|Thank you very much for the SRWare Iron link. I was not aware of that browser.
Score: 0
|