RSSTests: Firefox 3.1 JavaScript outpaced by Safari 4, Google Chrome
By Scott M. Fulton, III | Published March 13, 2009, 12:05 PM
(continued from previous page)
Our test suites include the three benchmarks we used last week to demonstrate Safari 4, including HowToCreate's CSS rendering test (since the rendering times can vary, we averaged the times for five renders for each browser), developer Sean Patrick Kane's octathlon which renders a cumulative speed after a battery of eight tests, and the Web Standards Project's Acid3 compliance test, which renders its score as a percentage.
To these, we added one more suite: the SunSpider JavaScript Benchmark, a pentathlon of real-world simulations that include elements like AES, MD5, and SHA1 cryptography puzzles; ray-tracing formulas; and handling of JavaScript Object Notation (JSON) expressions. With this and the Celtic Kane battery, we'll be able to find out where any browser excels and where it fails.
Each of our four suites produced a cumulative index score, which we then counted toward 25% of the final score.
The final index scores for our tests indicate one thing conclusively: The performance differences between all the major Windows-based Web browsers, compared to one another, are clearly perceptible. No test browser is close to any other; in everyday use, the differences in JavaScript rendering speed are enough to be seen with the naked eye.
And there is a clear leader in that department: Apple Safari 4.
Microsoft can say that its IE8 candidate has more than double the performance of its predecessor, and it can probably smile at that. IE8 RC1's index score is 2.19. However, the latest Safari 4 beta's score under the same conditions is 14.39, blowing away anything Microsoft produces. We expected more from the Opera 10 alpha, given what that browser's faithful followers told us. But with an index score of 5.38, Opera's performance right now is bested by Firefox 3.1 Beta 3, which finished with a 7.85. But Mozilla will have to settle for the bronze for now. Google Chrome is the one vying for the lead, with a second-place score of 11.62.
What were the browsers' strong points? Safari 4 scored well in the HowToCreate box rendering experiment. In the Celtic Kane battery, Safari shined with regular expressions, handling the Document Object Model, and at parsing RegEx expressions -- on that one part alone, Safari scored a 13.94 versus IE7. Its AJAX parser also proved strong. But the SunSpider benchmark was able to highlight some even brighter points for Safari, including just the way it manages control of scripts. In that heat, it scored a 90.94 versus IE7. And scoring 100% on the Acid3 certainly helps.
Chrome has clear strengths and weaknesses. It was not the fastest at the HowToCreate rendering experiment, scoring only 4.73 against IE7. Its 79% score on the Acid3 set it back. But it managed to outperform even Safari 4 in the SunSpider suite, with a 33.36 overall versus 32.81 for Safari. The control flow heat alone produced a 148.37 for Chrome versus IE7, which is testament to how well that browser's unique system architecture pays off.
The browser that will be Firefox 3.5 has some ground to make up. It actually performed rather poorly with the HowToCreate rendering test, actually loading the test slower than IE7 (0.93) and producing a rendering score of only 1.83. Its 93% score on the Acid3 is the best we've ever seen from a Mozilla browser. However, if this were a math quiz, Firefox 3.1 Beta 3 would win hands down, and here is where we're seeing that "orders of magnitude" difference we were promised for TraceMonkey: In the math heat of SunSpider, Firefox pulled off a 17.24, compared to 8.97 for Chrome and 6.93 for Safari.
Since last December, Opera has waved its flag high atop the mountain that is the 100% Acid3 score, and we saw it again yesterday. Opera did win the day on the HowToCreate rendering test, scoring a 5.03 -- the highest of the test browsers, although that score was offset by its slower loading speed. But Opera couldn't keep up in the octathlon and pentathlon, scoring only 3.05 and 6.67, respectively -- the lowest scores for any non-Microsoft browser.
Internet Explorer 8 was never expected to be a breakthrough performer. What we did not expect was a rendering score that is actually slower than IE7 (0.83). IE8 put in an appreciable 5.05 on the SunSpider suite and a less-than-appreciable 1.67 on the Celtic Kane suite. But scoring merely 20% on the Acid3, these days, may not be forgivable.
During the era of the so-called "browser wars," the only real matchup was between Netscape Navigator and Microsoft Internet Explorer. And that wasn't even really a battle on equal terms, but rather one between performance and functionality versus convenience. IE won that battle. So the fact that we are actually comparing Web browsers' performance on an equal scale today is symbolic of the fact that this market, for the very first time in its history, may be becoming truly competitive.
i think id still have to do my own test to find what im looking for
i need a browser to load fast and have automatic refresh on so i can load faster than other ppl playing internet browser games, ff does it quite well but ill give safari a chance the same as i did with all the others, opera seemed ok but crashed out when you opened a few diffrent windows, with ff i can open 15 windows on automatic refresh every second with only a bit of slowdown to show for it opera and crome crashed when i got past 3 windows
add ons do slow a browser down but some of them essential for saving the fingers from repetitve strain injury
Score: 0
|So let me get this straight:
1) You take 3 speedup ratios and one ratio of Acid3 scores and somehow create a "final score" from them. I have no idea how one munges such disparate quantities into a single number... Even just averaging speedup ratios is wrong, unless you very carefully control the things whose speedups you're measuring. In any case, I see no description of how the overal score was derived.
2) You use a known-broken test (Sean Celtic Kane's), which doesn't measure what it claims to measure.
3) You use a test (HowToCreate CSS Rendering load) on which Safari's lying about when it fires onload makes a big difference (hence the Safari score being 8x lower than that for any other browser, including ones based on the same webkit engine).
4) You use a test (HowToCreate CSS Rendering render) which doesn't actually test CSS rendering performance last I looked at a profile.
Conclusion: Your results are pretty much meaningless. Now if one looks at an actual carefully written test suite which tests what it claims to test (e.g. Sunspider or dromaeo), it's certainly the case that the Safari 4 beta is faster in a lot of cases than the other browsers you tested. But not by nearly as much as your tests show, and there are plenty of other cases where it's slower.
This is probably the worst article on browser performance that I've read recently. Instead of admitting to not knowing what it's talking about, it tries to hide that fact in a lot of pseudo-scientific mumbo-jumbo.
Score: 3
|there were some regression bugs during the merging of TC and M-C. It should be fixed very soon, and should be reflected in 3.5 b4.
Score: 0
|As I just tested minutes ago, there are apparent some bugs related to this have been fixed. the sunspider benchmark improved more than 10% so far from beta 3.
Score: 0
|Interesting. That would explain why beta 3 was slower than beta 2 on Sunspider:
http://davidnaylor.org/b...-slower-but-faster.html
Score: 0
|sounds like including firefox 3.0.x is a good idea for comparison reasons.. the same can go for safari 3 or opera 9, if you were to do it that way though
anyways, we can see alpha/beta and RC versions of different popular browsers, but we don't see such for Chrome.. the Chromium builds are far ahead from the official Chrome releases, scoring 100/100 (with small glitches..) on the acid3 test, so it sounds about right to throw Chromium in the mix if safari beta, firefox 3.1 beta, opera 10 alpha and IE8RC1 are there
Score: 1
|Quite interesting, i checked up the whether TraceMonkey was enabled in the 3.1 b3 on chrome and is wasn't (javascript.options.jit.chrome was set to false). Scott, did you verify that setting before the tests?
Score: 1
|I didn't notice this!
For the others - enter about:config - type, jit - make both the values true
Score: -1
|Uh... no. Don't do that. It's on for web pages (the .content pref). It is NOT ON for the browser user interface (the .chrome pref) because there are known issues with that. If you enable that, expect crashes.
Score: 2
|I would not worry to much about these tests or which is better or faster. In the end, they all have to work correctly for future days to come and pass all the tests or get left behind. So all these things really mean nothing as tomorrow it could all change again. I use IE 8 and Firefox 3.0.7 and I like them both, but fire fox a little better because I can have it the way I want. Firefox will come up to standards in time, that is what they are working on. Just how many browsers do these software makers think we need, couple hundred more?
Score: 0
|I tested three browsers on FutureMark
IE 8.0.7000.0 scored 163
FF 3.07Beta Scored 314
Safari 4 Beta 528.16 scored 528.16
This was done on Windows 7 7000 Build
Score: 0
|Can you add in this benchmark the memory usage? (How much RAM does a browser consume). Any of you who are familiar with these browsers, can you give a rough estimate? Thanks.
Score: 0
|Firefox makes up for its "defects" by being able to get highly customized.
Competition comes only when Safari and Chrome get add-on features.. if they do, I think it will affect the performance!
We'll know as it comes. But as of now, I'd stick with Firefox; the complete browser!
Score: 4
|I agree regarding the customization. I use Firefox for more sensitive matters at home, banking transactions and the like, because I still haven't come to trust the private browsing feature on Safari. However, for general browsing I revert to Safari, as I've noticed Firefox become slower and slower with each revision.
Score: 0
|After reading a recommendation, which said - Completely Uninstall Firefox - I've felt a great improvement in FF3.1 b3 performance.. very quick rendering of pages.. and it somehow feels that the browser seems to understand the web pages better now.. Completely uninstall includes removal from everywhere - locals settings etc and the registry too.. give it a shot!
Score: 0
|That's just great. What version of Chrome is used in the test? Probably 1.0. IE7, IE8 RC1, Opera 10, Safari 4 beta, Firefox 3.1 beta 3 and just Chrome, no version? Probably the slowest one 1.0. Thats is just not right. But there is for sure reason for that, are you guessing????
Here is more realistic bench http://www.futuremark.com/peacekeeper
Score: -3
|Yeah, Betanews has a hidden agenda to crush Chrome. Get real.
Score: 0
|Perhaps you didn't read the article.
"(Though Google lifted the "beta" designation from its Chrome browser, its development is continually tested and its clients are automatically updated to reflect results from those tests, so we included the latest Chrome edition with the other brands.)"
Sounds to me like they used the latest publicly available version of Chrome.
Score: 0
|What we also have are what I see are different philosophies branded into each browser.
IE: The general public. They don't care about other browsers, it is always what they have used, and they aren't even curious about what else is out there, due to ignorance. You also have some geeks that have found add-ons to IE to make it worth their while. You have corps that pretty much mandate this browser, and group policy it to make it more secure and appropriate for their environment. Historically quite insecure, run this puppy in a Vista/7 userland sandbox if you ever do use it.
Firefox: An old geek favorite. It initially was loved for its small size, extensible architecture. The latter is also it's achilles heel, as numerous extensions/themes make the browser less stable, and upgrades aren't always smooth. This browser arguably gives you the most power for viewing online content exactly as you prefer. You can make this browser quite secure with a few select add-ons.
Opera: Always amongst the top performers, the leading feature browser out of the box. Uber-geeks use. Also historically seen as more secure either by being less of a target because of marketshare, but also the very fast response time when they do fix security issues. Sometimes the features are a tad overwhelming, as there are almost too many ways to configure this browser and I bet they scare a lot of newbs off.
Safari/Chrome: I group these together because I see them as so similar. You either like Apple's philosophy or Google's, and if you use these browsers you probably are a fanboy of either. Quite fast in javascript, but javascript is hot now, who knows 5 years from now. These browsers seem to have user interface quirks, you either love 'em or hate 'em. I expect both of these to start to take share growth from IE and Firefox.
My prefs: FF3, Opera, Chrome, in that order. I won't use IE or Safari. If I could just get a better element blocking in Opera I'd probably swap, as I do like its rendering performance. Chrome crashed too much in my tests, but I will give it another try after more testing by users...
Score: 3
|For your element blocking needs in Opera, try this. Fanboy does a great job maintaining both lists:
http://www.fanboy.co.nz/adblock/
Score: 0
|Very good comparison.
Yesterday I compared Firefox 3.1 beta 3 to beta 2 for Javascript performance. The results were interesting:
http://davidnaylor.org/b...-slower-but-faster.html
Score: 0
|You should list what build of chrome you use. There is no "1.0" in common release currently.
Score: 1
|My biggest surprise right now is how few addons have been updated for FF 3.1b3. Isn't this supposed to be a new milestone build? Oh, and great article BTW.
Score: -3
|It's been my experience with the 1.5, 2.0, 3.0, and 3.1 beta nightly builds that the extension developers almost always have their products available at the launch and not a second earlier, probably to assure maximum compatibility with minimal confusion. Besides, there will supposedly be another beta build prior to the release candidate and the final production version.
Score: 0
|How few?
Out of the extensions that I frequently use (Adblock Plus, NoScript, TabMixPlus, StumbleUpon, Gmail Notifier, and Forecastbar Enhanced), the only one that was incompatible with 3.1b3 was TMP.
Though NoScript was compatible, I'm finding it more annoying and less necessary than I used to. Thinking of ditching it altogether.
Sounds like extension developers are on top of things to me.
Score: 0
|"During the era of the so-called "browser wars," the only real matchup was between Netscape Navigator and Microsoft Internet Explorer. And that wasn't even really a battle on equal terms, but rather one between performance and functionality versus convenience."
Nope, Netscape started to really Bloat, crash and Suck.
Score: 0
|Isn't that about the time that AOL took it over and totally raped and pillage it?
Score: 0
|I wonder what FF performance with adblock and noscript is like compared to loading 2+ external ad networks on every Google Chrome and Safari page? Of course, when those browsers get element blocking in as well, test with those features.
Score: 0
|The ultimate test is using the product to find which one really works out best for the individual. I have yet to find a better browser, over all, than Firefox, though Opera has some great features. Firefox renders more pages correctly, of those I use, than Opera. Firefox has better features than Chrome, when the add-ons are used. As for speed, heck with DSL compared to speeds in 1995, the most clunky browser in the world would seem like a speed demon. FF seems speedy enough for this kid. It is the entire package which makes FF the deal for me. Depending on which sites you use, you may also find Opera, very good.
Score: 0
|I care about speed, but compliance is most important, as it should be for everyone. The browsers that can't score 100 on Acid 3 should not be considered in the running.
Score: 0
|Numbers are for the bean counters and to give a developer bragging rights which is AMO the worst way to "sell" their product.
If you look at every aspect of a browser which most important and base them on crashing, security, rendering, speed, and compliance using your train of thought there would be no browsers in the running at all.
Score: 0
|Great, Scott used finally the latest versions available. But (there is allways a but) something smell fishy.
In real life Safari DOES NOT render pages faster than Opera 10 or Firefox 3.1b3... I do not know where the tests were flawed, but they are!
Score: 0
|You should of included the shipping version of Firefox so that Firefox users could tell how much faster it's going to be. Also can you tell the difference between these or are they just fractions of a second.
Score: 0
|I believe in a test prior to this, they did use a final release of FF, but because it did so horribly compared to the alpha and beta versions of other browsers, some people here whined and cried bloody hell murder and that Betanews should've used an alpha or beta of the up and coming FF that had it's latest script engine in it. So it seems in this test they did, and it seems FF still failed to meet or exceed some of the other browsers.
In the end though, people should be more concerned with safety and rendering compliance more than anything. You need to be able to browse without getting jacked by some script kiddie's page hacks. You need to be able to view a web site as it was intended to be viewed by it's author, and not how some giant corporation thinks you should see it...
And to you Opera fans.... Don't sit safe thinking your fav browser there is as safe as you think it is. I myself used to be an Opera fan... that is, until it allowed a web site to download and execute malicious code on my box. While I still appreciate the efforts of the developers behind Opera, I personally won't touch it with a 10 foot pole ever again. You know how it goes.. Once you get burned, you're less likely to want to do it again. It may be a very long time before I trust Opera enough to try it again.
For safety purposes, I'd suggest FF with adblock and noscript. Might be a pain dealing with noscript, but sure as hell it'll save yer butt from just instantly getting hit by haxed page code somewhere.
Chrome(or SRWare Iron for a version that doesn't have tracking code in it. http://www.srware.net/en/software_srware_iron.php) and Safari, both being based on the Webkit, seem to be good for casual browsing, but not for power browsing. They seem relatively safe, but who knows what lurks out there that could hit either one of these.
And that leaves me with IE.... I'm still having flashbacks and nightmares of all the security holes it was riddled with back in the day. Sorry, still don't have the jewels to even think about using it for everyday purposes. Though I do use it locally only for checking my web page designs for compatibility.
Score: 1
|Thank you very much for the SRWare Iron link. I was not aware of that browser.
Score: 0
|