RSSApple's Safari 4 Beta for Windows speeds up after security update
By Scott M. Fulton, III | Published May 14, 2009, 1:18 PM
(continued from previous page)
A word on methodology I've gotten a number of comments and concerns regarding the way our recent series of browser performance tests are conducted, many of which are very valid and even important. For the majority of these tests, I use a Virtual PC 2007 VM with Windows Vista Ultimate. Most notably, I've received questions regarding why I use virtual machines in timed tests, especially given their track record of variable performance in their own right.
The key reason I began using VMs was so that I could maintain a kind of white-box environment for applications being tested with an operating system. In such an environment, there are no anti-malware or anti-virus or firewall apps to slow the system down or to place another variable on applications' performance. I can always keep a clean, almost blank environment as a backup should I ever install something that compromises the Registry or makes relative performance harder to judge.
That said, even though using VMs gives me that convenience, there is a tradeoff: One has to make certain that any new tests are being conducted in a host environment that is as unimpeded and functional as for previous tests. With our last article on this subject, I received a good comment from a reader who administers VMs who said from personal experience that their performance is unreliable. With my rather simplified VM environment (nothing close to a virtual data center), I can report the following about my performance observations: If the Windows XP SP3 host hasn't been running other VMs or isn't experiencing difficulties or overloaded apps, then Virtual PC 2007 will typically run VMs with astoundingly even performance characteristics. The way I make sure of this is by running a test I've already conducted on a prior day again (for instance, with Firefox 3.5 Beta 4). If the results end up only changing the final index score by a few hundredths of a point, then I'm okay with going ahead with testing new browser builds.
I should also point out that running one browser and even exiting it often leaves other browsers slower, in both virtual and physical environments. This is especially noticeable with Apple Safari 3 and 4; even after exiting it on any computer I've used, including real ones, Firefox and IE8 are both very noticeably slower, as is everything else in Windows. My tests show Firefox 3 and 3.5 Beta 4 JavaScript performance can be slowed down by around 300%. For that reason, after conducting a Safari test, the VM must be rebooted before trying any other browser.
When VM performance does change on my system, it either changes drastically or little at all. If the index score on a retrial isn't off by a few hundredths of a point, then it will be off by as much as three points. It's never in-between. In that case, I shut down the VM, I reboot XP, and I start over with another retrial. Every time I've done this without exception (and we're getting well into the triple-digits now), the retrial goes back to that few-hundredths-of-a-point shift.
Therefore I can faithfully say I stand behind the results I've reported here in Betanews, which after all are only about browsers' relative performance with respect to one another. If I were testing them in a faster system or on a bare bones physical machine, I'd expect the relative index numbers to be the same. Think of it like geometry: No matter how big a triangle may be, the angles measure up the same, they add up to 180, and its sides have the same proportionate length.
Now, all that having been said, for reasons of reader fidelity alone, there's benefits to be gained from testing on a physical level. You need to trust the numbers I'm giving you, and if I can do more to facilitate that, I should. For that reason, I will be moving our browser tests very soon to a new physical platform (I've already ordered the parts for it). At that point, I plan to restart the indexes from scratch with fresh numbers.
Next, there's been another important question to address concerning one of the tests I chose for our suite of four: It's the HowToCreate.uk rendering benchmark, which also tests load times. On that particular benchmark, Safari and Chrome both put in amazing scores. But long-time Safari users have reported that there may be an unfair reason for that: Safari, they say, fires the onLoad JavaScript event at the wrong time. I've encountered such problems many times before, especially during the late 1980s and early '90s when testing basic compilers whose form redraw events had the very same issues.
This time, the creators of the very test we chose called the issue into question, so we decided to take the matter seriously. HowToCreate.uk's engineers developed a little patch to their page which they said forces the onLoad event to fire sooner, at least more in accordance with other browsers. I applied that patch and noticed a big difference: While Safari 4 still appeared faster than its competition at loading pages, it wasn't ten times as fast, but more like three times. And that's a significant difference -- enough for me to adjust the test itself to reflect that difference. For fairness, I applied the adjusted test to all the other browsers, and noticed a slighter difference in Google Chrome 1 and 2, but a negligible difference in Firefox and IE. Our current round of index numbers reflects this adjustment.
Safar 4 beta in Windows 7 VHD environment may be a more level playing field on an XP machine don't you think? (not an expert opinion here) After downloading both of the Safari beta 4 May 12 updates, they would not install. Rec'd. error messages 'file missing', any ideas? Thanks for reply.
Score: 0
|Hi I am Joseph Letzelter, Your article is very fine.
Score: -1
|People still use Safari on things other they apple control devices? You got an App for that?
Score: 0
|Yeah Safari 4 on Windows is ugly and steals from Chrome. 3.0's UI wasn't bad. But who cares? I stopped using Safari since Apple doesn't get the meaning of "update". Every "update" including a very minor fix means downloading the whole bloated installer again, uninstalling the earlier one and installing this one.
Score: 1
|except not since they're using the same underlying code base minus v8 javascript engine and gears internals rather than html5 unfinished standards.
Score: -1
|Any chance you could post the freaking specs of the machines? Have you EVER seen benchmarks without the specs (hardware AND software, patches/drivers listed?) Typically you want to post something that others can reproduce, scientific method and all.
Score: 0
|Have you actually seen Safari 4 on Windows? It may be the ugliest browser ever. They actually put the tabs in the transparent (Vista/win7) title bar! I'll sacrifice some speed in a javascript test to have something worth looking at.
Score: 0
|looks fine to me on windows but then again, i dont like fugly windows gui since i have taste. living in a dumpster you shouldnt complain about the food you eat.
Score: -1
|This shouldn't come as a surprise. Safari is developed by the world's greatest company of all time. Safari will always be the fastest browser. It is screaming fast on a Mac.
Score: -1
|Speed alone doesn't make a browser excellent. Safari 4 is lesser than it's competitors in all other aspects other than speed, and the speed of Safari 4 at this point in time compared to FF 3.5 is pretty much negligible to the human eye. The only features that are unique to safari 4 at this point in time are:
Topsites- large thumbnails of your most viewed pages
Coverflow bookmarks- what more can I say?
However,
- Top sites is pretty useless unless it's the default page on load.
- Coverflow for bookmarks is eye candy and it's a GUI nightmare IMO. Coverflow is sort of useful in itunes because you can recognise the colors/shapes of your album art but with Safari 4, your websites gets updated automatically, therefore you can't recognise the sites- so you still need to read the list to choose your bookmarks, rendering the coverflow useless.
Safari 4 still provides no way to easily manage addons and still disappoints.
Safari's trump card (speed) is being matched, and it has nothing left to offer in my opinion.
It's not a good browser.
I have OS X and enjoy using it at times, but I'm 100% sure Safari 4 is not part of my "Mac experience" (which is not as awesome as you like to preach) and now it's gathering dust in my applications folder.
Score: 0
|I do like firefox 3.5 on a Mac but Firefox's lack of a native feel keeps me from making it my default browser. Unfortunately Firefox uses XUL which feels clunky, slow and sometimes very foreign on a Mac. Safari is a good browser as well but tabs no longer pop out if you've set all new window links to open in a new tab instead. This wasn't the case in Safari 3.5. Now when I visit a site such as MSNBC to view video news, the videos open in a new tab instead of popping out. It looks awkward. I hope this changes in the final release.
In the meantime, I've grown quite fond of Camino 2 beta 3, which has awesome speed and is a coca app so it is completely native to Mac OS X. It also uses the same gecko 1.9 engine in Firefox. It lacks some features but it is a very lightweight and fast browser. It loads up really fast and blends some of the best features and functionality of both Firefox and Safari.
Score: 0
|Safari 4 is a great browser. It kicks butt with 100% on the latest Acid test (an unheard of score until now) and is NOTICEABLY faster than firefox.
I think it looks great - nice streamlined use of space. However it's still only my backup because of the great plugins I have in firefox.
Nice work Apple. If you can write a browser for a competitor's OS (Windows) that is so overwhelmingly superior to Microsoft's own IE browser, than I can only imagine how much better your OS probably is than Windoze. I think I'll give it a try!
Score: 0
|safari has THE best RSS reading, text searching, and performance. It's built in debugging controls for javascript and html are pretty nice too. people saying it sucks obviously haven't used it enough to know what a real browser is.
Score: -1
|I installed it too on my Mac. Not sure I like that tab bar at the very top though. Sure it's fast but I'm not impress with the layout. And on Windows XP it looks even worse.
I hope the RC or final build will improve.
Score: 0
|"I began using VMs was so that I could maintain a kind of white-box environment for applications being tested with an operating system. In such an environment, there are no anti-malware or anti-virus or firewall apps to slow the system down or to place another variable on applications' performance. I can always keep a clean, almost blank environment as a backup should I ever install something that compromises the Registry or makes relative performance harder to judge."
Install the OSes on bare-metal, ghost or TrueImage or w/e the OS. Re-image the system for each new test. We're talking 10 minutes an image here. VM's post crap results. Period.
"If I were testing them in a faster system or on a bare bones physical machine, I'd expect the relative index numbers to be the same."
Great expectations often lead to great disappointments. Unless you've verified this in testing, your "expectations" mean exactly squat.
"Think of it like geometry: No matter how big a triangle may be, the angles measure up the same, they add up to 180, and its sides have the same proportionate length."
...until the triangle becomes a square.
But since you're getting that test-system, I guess it's all moot. :) Good to hear.
Score: 1
|+1 for not using VMs, safari may still get better results just don't use VM's :P
Score: 0
|As said before, and I see is being rectified in this article, VM's are not a good way to test performance. Performance should be on a real platform not being virtualized. I say also that you should include a graph and breakdown what these numbers mean. Numbers are meaningless, percentages especially, unless we have some frame of reference to understand their meaning with.
Score: 0
|Yes, it's a silly way of testing performance, especially as the host PC can affect performance in unpredictable ways. The only way of doing it properly is to have a clean-install PC with nothing else on it. The installation can be imaged, so getting the same exact base install every time. To do otherwise is foolhardy and simply lazy practice. I, along with other people who really care about performance tests will be ignoring Betanews articles claims on performance from now on.
Score: 0
|@testman:
Even with the knowledge that the methodology behind the tests is changing (to bare-metal vs VM)?
Harsh...
Score: 0
|I'm pretty sure it didn't say that when I posted the comment. Still, at least they're actually realising that bare metal clean-installs are simply better than using VMs as it ensures NOTHING else can skew the results, however small.
Score: 0
|