Firefox Takes the Performance Crown From Chrome

http://news.slashdot.org/story/13/07/02/146251/firefox-takes-the-performance-crown-from-chrome?utm_source=slashdot&utm_medium=twitter

by bpierre • 13 years ago

323 134 13 years ago

news.slashdot.org

leeoniya• 13 years ago

i'm developing some js compute-heavy apps and Chrome still pulls ahead quite significantly (by as much as 5x) in many cases. i've filed js-perf bugs with mozilla that have been accepted but seem to be quite low on the priority list. artificial benchmarks don't always tell the full story :(

if any mozillian js dev is lacking stuff to work on ;)

https://bugzilla.mozilla.org/show_bug.cgi?id=879393

https://bugzilla.mozilla.org/show_bug.cgi?id=858986

33272• 13 years ago

I did not find this objective, the test scope was limited and the sample set of browsers tested were not the latest version. Not that I use IE but why use IE10 when IE11 is available for testing? If you are doing a performance benchmark browser vs browser you will not be taken seriously if your test is not objective. In this case I find the sample set skewed.

msujaws• 13 years ago

In all fairness, the sample set included all "release" builds of the browsers. The vast majority of users only use release builds, so it makes sense to test them when they get released. It also is important because it includes all performance fixes that will possibly go in to the released product.

pix64• 13 years ago

If I understand correctly, IE11 is only a preview version not the final release.

33272• 13 years ago

Correct it is a preview.

If that is the case then I am wrong in my assessment and withdraw my statement.

Roboprog• 13 years ago

I suspect this was "by a nose". But that's good: I hope to see both browsers trading blows in this little war, leap frogging back and forth in the lead.

zaph0d• 13 years ago

Actually not; Firefox beat Chrome comprehensively in the overall test. Chrome actually came 3rd in the list - http://www.tomshardware.com/reviews/chrome-27-firefox-21-ope...

magicalist2• 13 years ago

To be fair, this slashdot story covering the original story (presumably submitted because the original story was submitted and talked about yesterday) mentions the "performance crown", which was just "by a nose". You're right about the overall test, though.

zimbatm• 13 years ago

The issue with Firefox is that some times the whole browser is freezing. That's what keeps pushing me back to Chrome, not performance issues.

mcdoh• 13 years ago

Chrome freezing is why I'm considering switching back to Firefox.

cmpxchg8• 13 years ago

Chrome also phones home all the time, which is why I stopped using it. I don't notice much performance difference between Firefox and Chrome anyway, and always use Noscript, Adblock+ and Disconnect.

matthiasv• 13 years ago

That is your issue with Firefox or your setup. I never had this for at least one or two years now, even with misbehaving scripts.

ksec• 13 years ago

Nope, It is a common issues with Firefox although it has been greatly reduced in the last year or so.

The project to fix all this is called e10s. Moving Content, and UI etc to seperate process as well as OMTx ( Off Main Thread Everything ) where Firefox Dev trys to do everything in async style and move them off Main Thread.

Progress has been slow. But at least they are working on it

driverdan2• 13 years ago

It's a core problem with FF. Everything runs under the same process so a single site with slow / frozen JS will affect the whole browser.

thelukester• 13 years ago

matthiasv, I don't know what your secret is, only 1 tab open in 1 window open at time, JS disabled, or what, but this is a very real issue. I have Firefox installed on all 4 of my machines, work desktop, home desktop, laptop, and a netbook. 2 of which are fresh installs. On all 4 when I visit multiple HTML5 heavy sites, FF's UI gets unresponsive. Chrome and newer versions of IE do not suffer from this, and I'm pretty sure it comes from FF running in a single process.

Chrome and IE have their issues too, like thrashing memory a lot sooner than FF, which is why FF is still the default browser on my laptop and netbook despite its unresponsive UI under heavy load.

yuvadam• 13 years ago

Did you take the time to search through open tickets or create a new one so that Firefox could improve?

cpeterso2• 13 years ago

Do you have any add-ons installed? I'm not shifting the blame from Mozilla to the add-on developer, but it might pinpoint the problem you found. If add-ons are using slow APIs, then Mozilla can do something about it. For example, some synchronous add-on APIs have been replaced with async versions.

kevingadd2• 13 years ago

Are you visiting websites containing the Flash Player plugin? If so, try killing the plugin-container process. Flash continually gets less stable and Firefox can only do so much to compensate when the plugin hangs (plugins are intentionally allowed to insert themselves into the message loop, which makes it possible for them to hang a browser)

ebbv2• 13 years ago

These types of headlines are so meaningless. On what benchmark on what platform?

On OS X Safari outperforms both for many tasks (again, YMMV depending on what you're doing.) But that's meaningless for the billions of Windows and Linux users out there.

I wish our industry would move past these kinds of silly headlines. When it first started when Chrome came out there was a massive difference between Chrome and the other browsers, now all of them are very capable and competitive for pretty much anything you want to do.

mtgx2• 13 years ago

TomHardware's browser benchmarks are pretty comprehensive. However, what annoys me about this test and this headline, is that Chrome 28, with its new faster Blink core, is literally a week away from being released, which means Firefox will only have its 15 seconds of fame (or rather a week).

Why did they make the test immediately after Firefox was out? Or do they repeat the test immediately after each browser version comes out? In that case I'm looking forward to the test with Chrome 28 included.

I'm not saying Chrome 28 will necessarily win in the next one. I just find it a little strange that they did it without waiting a bit more for the next Chrome version, too, before writing a headline like that. It just reminds of me those polls who turn out a certain way depending on how you ask the question.

mariusmg2• 13 years ago

Most likely because it takes time to do those benchmarks. After Chrome 28, FF 23 should be out so why not wait for that either ?

joosters• 13 years ago

These types of headlines are so meaningless. On what benchmark on what platform?

That's what the article is for, you've got no chance fitting those details into the headline!

ebbv2• 13 years ago

But that's my point; the answer is different depending on the platform and the task. The headline is invalid because of that fact. Firefox is not faster at everything, and Chrome is not faster at everything.

evmar• 13 years ago

(Disclaimer: I used to work on Chrome. I'm more than a year out at this point but I still talk to Chrome developers.)

Chrome wins some tests but loses the most on the start-up benchmarks, which is a real bummer. There was a time where we'd spend literally days figuring out why startup regressed by a handful of milliseconds, and this test shows an end-to-end test taking nearly six seconds.

One idea I saw mentioned is that there may be a bug related to how Chrome reads the system proxy settings (something about an API change between Windows 7 and Windows 8). I think most of the Chrome developers are still on Windows 7; the bots that are tracking startup performance are definitely still Windows 7 as well. So maybe that number is more of a single bug than a systematic thing.

All of the above is not intended to diminish Firefox's impressive results -- just thought I'd provide some background! What these sorts of tests show most is that competition pushes browsers ever faster. If the proxy theory above is right, then maybe this test will inspire Google to invest more into Windows 8 testing, which ultimately benefits users.

(Edit: Firefox won other tests too, I also didn't mean to say this was the only factor.)

_pmf_• 13 years ago

Chrome wins some tests but loses the most on the start-up benchmarks

Well, congratulations to FF for being the perfect browser for the 0.01 percent of people who don't keep their browser open all day.

mariusmg2• 13 years ago

Startup time matters (a lot). Even if i open it once a day i still want it to be fast. Nobody likes slow apps.

saint-loup2• 13 years ago

I use Windows at the office and my work doesn't need me to have 300 tabs open at once. So I keep lauching and closing Firefox.

Start-up time matters, thanks.

shardling4• 13 years ago

It's always so cute how everyone thinks the way they use a browser is the only way anyone uses a browser.

Like folk who think that having hundreds of tabs open at once is the norm! :)

Spittie3• 13 years ago

When Firefox was slow to start, it was bashed about it a lot.

Now they have put a lot of effort into that, making it one of the fastest browser to start. And yet it's bashed because they put their effort into "something" that doesn't matter to many people.

cpeterso2• 13 years ago

Mozilla's user stats show that 96% of Firefox Beta users restart their browsers daily and 98% within 2 days. I imagine non-beta Firefox users restart even more often, e.g. they open Firefox to check their Yahoo mail and quit. Most people do not live in their browsers.

krelian• 13 years ago

Chrome is my main browser and I use it all day at work. The browser window opens instantly for me but it only becomes usable about a minute or so later after it has finished grinding on the hard drive. From the looks of it I'd say it's loading all the history, settings and extensions that takes so long.

driverdan2• 13 years ago

Actual benchmark story from yesterday: https://news.ycombinator.com/item?id=5970429

ajays• 13 years ago

I wish they (FF) would also spend some time on the small annoyances:

Lack of a restart button. Now when I upgrade FF, I have to "kill -9" it from my terminal to get it to restore windows upon restart

Memory leaks. Leave yourself logged in to Facebook for a few days, and watch it take up 2GB of memory.

No way to easily filter sites with cookies like you can in the next tab over, where you manage sites with saved passwords. Why is this? Does FF secretly want you to not muck with cookies?

Now, I'm sure there are addons and plugins for the above. But I should _not_ need addons for basic UX. Save the addons for fancier stuff.

padenot• 13 years ago

* For the restart after upgrade, either it is a bug, or you have not seen the little "restore tabs from last time" button in the lower right part of the browser.

* I have not experienced that, I don't use Facebook.

* For the cookie stuff, you'll be pleased to learn about the about:permissions feature (type it in your address bar).

epmatsw2• 13 years ago

Whoa, about:permissions is awesome. Thanks for that!

Recoil42• 13 years ago

  I have not experienced that, I don't use Facebook.

"Area Man Constantly Mentioning He Doesn't Use Facebook"

lazyjones• 13 years ago

about:permissions

Interesting, thanks. But while it's open, the list of sites keeps growing for me, apparently various shady/malware sites get added at the bottom continuously. I haven't managed to find it why this happens, but I'm guessing that it may have something to do with the "safe browsing" feature, perhaps all sites in the blacklist get added to about:permissions for some obscure reason. Any ideas?

epmatsw2• 13 years ago

If you go to Help > About Firefox when there's an update to be installed, there's a restart button that restores tabs.

As for memory leaks, it's been a project for a while now: https://areweslimyet.com/

I'm not entirely sure what the third point is referring to (I just let the browser handle cookies), but in the Privacy tab of Options, you can view and remove individual cookies and search by site. So maybe that helps with what you want?

shardling4• 13 years ago

Leave yourself logged in to Facebook for a few days, and watch it take up 2GB of memory.

It's entirely possible for a website to have a memory leak. (Or, at any rate, consume an ever increasing amount of memory.) If closing the tab and opening it again frees the memory, then there might not be anything FF can do about it. (It's also possible you've got an addon causing the problem.)

Domenic_S• 13 years ago

Doesn't FB constantly update the feed? If you leave the feed open, won't the page grow unbounded? Seems like it's working as intended...

kevingadd2• 13 years ago

Facebook indeed has a long history of really dumb memory leaks that browsers can't do anything about, combined with memory leaks that are exacerbated by browser bugs and require improvements to garbage collectors and stuff. It's rough.

Spittie3• 13 years ago

lack of a restart button.

I don't think Chrome has one. Anyway, either press shift+f2 to bring up the Developer Toolbar and type "restart", or install an addon like Restartless Restart (https://addons.mozilla.org/en-us/firefox/addon/restartless-r...).

You can also just go into the settings, and set "When Firefox starts:" to "Show my windows and tabs from last time".

And I don't see why a "normal" user should want a default button in the UI to restart the browser.

Memory leaks

The situation is much better now, there still are some leaks here and there, but nothing huge, at least with my usage (I usually restart my browser every 1-2 days to update to the latest Nightly, and have about 50 tabs open).

The team responsible for memory leaks is the MemShrink team, you can see their progress here: https://blog.mozilla.org/nnethercote/

I'd also suggest to install the addon "about:addons-memory", and see if any of your addons is leaking memory (https://addons.mozilla.org/en-us/firefox/addon/about-addons-...)

If you still see memory leaks, please report them on bugzilla.

Cookies

While something like this would be nice, I don't think that "normal users" needs to mess up with Cookies.

If you're a power user, and want to enable/disable cookies on a per-sites base, install an addon that let you do it.

I'm personally very happy with Cookie Controller: https://addons.mozilla.org/en-US/firefox/addon/cookie-contro...

Ives• 13 years ago

After chrome upgrades itself, a "Restart Chrome" button appears in the "About Chrome" window.

mccr8• 13 years ago

In "general" under "preferences", you can select whether the browser restores tabs from last time it was exited. Maybe that somehow get set to something besides "Show my windows and tabs from last time"?

omaranto3• 13 years ago

I think Facebook itself has the memory leak: I never leave it open for more than about an hour at a time, but I've heard from people that leave it open for days that it can take up between 1 and 2 GB; some of these people use Firefox, some Chrome and some Safari.

bd• 13 years ago

Ehm, at least one of the tests they used is wrong. They used this one as WebGL performance test:

http://luic.github.io/WebGL-Performance-Benchmark/

Despite the name this program in fact doesn't use WebGL, it runs using three.js 2D canvas renderer (just check the source).

Additionally even the results they got there are suspicious, I got completely opposite results on my system (Windows 7), with Chrome being faster than Firefox by a large margin:

http://www.tomshardware.com/reviews/chrome-27-firefox-21-ope...

They got 777 on Firefox vs 437 on Chrome, I got 290 on Firefox vs 441 on Chrome (this is fully in line with my everyday experience, doing browser graphics).

glomph• 13 years ago

Toms hardware seem to consistently have methodology problems in their tests. I don't understand why they are given so much credence.

brownbat2• 13 years ago

What, no Lynx?

Argorak• 13 years ago

WebGL is not implemented, though 3d renderers to tty exist since the 90s, e.g. ttyquake! A scandal!

http://web.archive.org/web/20021113144923/http://webpages.on... (original site gone)

brownbat2• 13 years ago

Oh, come on. It was a joke.

simonsarris• 13 years ago

Performance is nice to be sure, but I think other things are much more important in 2013, like consistent HTML Canvas implementations.

For one small example the in-canvas kerning on both Firefox and Chrome are awful compared to IE10.

From a while ago: http://i.imgur.com/62WBzVZ.png

You can see for yourself here: http://jsfiddle.net/vVC4s/

Notice also the difference between doubling the font and doubling the scale, esp where each line gets cut off: http://jsfiddle.net/jGcrL/

This makes animations rickety and un-smooth (text animation): http://jsfiddle.net/simonsarris/HZFcR/

Surprisingly IE10 handles all of these perfectly. Of course, IE10 has totally broken canvas clipping (non-rectangular clipping regions are impossible. They worked fine in IE9.)

Firefox does render large text much better than Chrome, which is why I used it when taking screenshots for my book. But scaling the text (as opposed to setting a larger font) is a disaster.

This is just an example. I could go on for days about canvas bugs. I wish there was a bigger push to fix those instead of eking out a performance advantage.

To Firefox's huge credit, I've submitted a lot of bugs to Chromium and the FF team, and the FF team consistently gets back to me within a week and usually fixes the bug within a month. The bug reporting experience with Chrome on the other hand is rather disenchanting.

For a cross-platform bug example, the context's miterLimit is just plain broken by default in Chrome. I reported this (with examples) back in April and have yet to receive any kind of reply. Thank god it's an easy workaround.

https://code.google.com/p/chromium/issues/detail?id=225512

If you want people to use your browser and develop with it in mind, working elements are more important than being slightly faster than the others. (Good developer tools come in close second I think, at least for this crowd).

Aloisius• 13 years ago

I'm using FireFox 23 on a Mac and the kerning looks just fine on that canvas page for the first example. Aliasing is a little ugly though.

kunai• 13 years ago

That's odd, considering Firefox clones the antialiasing algorithm used by the host OS and uses that to render type. Are you sure that perhaps the OS font rendering settings are to blame, and not necessarily Firefox?

Derbasti• 13 years ago

Same for me. No kerning issues like in the png on OSX.

siddboots• 13 years ago

Looks fine on Chromium 25.0/Ubuntu 12.10 and on Firefox 21.0/Ubuntu 12.10

Osmium• 13 years ago

Just checking in to say that your kerning example looks fine on Safari on OS X (latest stable versions). Which is not to say it's perfect, but there's nothing egregious (no overlaps, no large gaps).

triptychs• 13 years ago

Ditto for Chrome.

jgraham2• 13 years ago

This is not a comment to you in particular, but when authors find bugs in browsers there are several things that they can so to ensure that the bug is fixed as quickly as possible:

1. Verify that the bug occurs in the latest available (possibly unstable/nightly) build.

2. Submit the bug to the relevant browser/engine bug tracking system (see below). If you aren't confident about how to write a good bug report, read [1] first.

3. Write a testcase. If at all possible this should be minimal (i.e. not using the whole of jQuery/Angular/whatever) and should certainly be clear (e.g. no minimised code unless it is actually needed to reproduce the issue).

4. If the bug is a conformance bug rather than a QoI issue, submit your testcase to the W3C [2] so that it can be incorporated into the standard testsuite and automatically run by all browser vendors. To do this the test needs to be written in a standard format [3] and submitted using the process at [4]. At the moment the documentation is a bit sucky, but there is a big revamp in the works [5], so that should improve in the near future.

That last one may sound like a lot of effort, but testing is the only way that we will end up with a web platform that is both technologically competitive and open both in spirit and in practice i.e. with multiple interoperable implementations. People put a huge amount of time into working around, and complaining about, cross browser issues. Devoting a small fraction of that time to submitting regression tests instead would dramatically decrease the number of problems in the future. For example submitting a test for canvas clipping both makes it likely that IE will be fixed in the near future, and also ensures that it won't then just regress again one version later.

It seems that it would be in the self-interest of big projects like jQuery to commit to creating a test for each time they have to work around some browser bug, so they can expediate fixing the issue, track which browser versions have the bug, and remove the workaround — and thus reduce their code complexity — as soon as the test passes in all the browsers they are targeting. However this is not just something that matters to big projects. If you have run across a bug that is making it difficult to implement something, you are likely the best person in the world to write a testcase for that issue.

If you are trying to submit bug reports and tests and need help, please ask on #whatwg on Freenode irc or #testing on irc.w3.org [6].

Bug trackers:

  Gecko: https://bugzilla.mozilla.org/

  WebKit: https://bugs.webkit.org/

  Blink: http://crbug.com/

  Trident: http://connect.microsoft.com/IE

  Opera: https://bugs.opera.com/wizard/

[1] http://fantasai.inkedblade.net/style/talks/filing-good-bugs/

[2] https://github.com/w3c/web-platform-tests

[3] https://sites.google.com/site/forthenewbies/home/writing

[4] http://testthewebforward.org/resources/github_test_submissio...

[5] https://github.com/w3c/testtwf-website

[6] http://irc.w3.org/

Silhouette4• 13 years ago

I appreciate the time you spent to make that informative post, and as a professional software developer I also appreciate the value of good bug reports. However, IME this simply isn't true:

People put a huge amount of time into working around, and complaining about, cross browser issues. Devoting a small fraction of that time to submitting regression tests instead would dramatically decrease the number of problems in the future.

The effort required for a regular user to file a bug report in the major browsers is insane. The bug trackers involved are among the most user-hostile and over-complicated interfaces I have ever encountered.

Developers move to six-weekly releases with at least three different flavours in the works at any given time, and then expect people running the regular browser to install a completely different version of the software to test things before filing bug reports. This is insane.

In Firefox's case in particular, users are often asked to create a new profile, which AFAIK still requires command-line hackery and, from direct personal experience, carries a significant risk of corrupting or destroying your regular profile even if you follow the instructions to the letter. This is insane.

And even if users do make the Herculean effort required to create a minimal test case, install the latest development versions of the browser, create a clean profile if necessary, confirm that the bug still happens, and fight through the bug tracker to file a report, there is still no guarantee that anything will actually be done about it. Quality of implementation is obviously not the priority for several of the major browser developers, certainly not Mozilla and Google, and major bugs and site-breaking regressions are frequently ignored for very long periods while shiny new features no-one is actually going to use in production for years and another 0.3642% performance boost in JavaScript are prioritised.

As I said, I'm a professional software developer, and as such I appreciate the value of good bug reports and feedback from users, and I hate it when users make blanket criticisms. But Mozilla and Google are among the most user-hostile organisations on the software landscape in this respect, and until they fundamentally change their approach, I don't see this situation improving. Meanwhile, their users, and those of us trying to develop web sites and applications to support those users against a constantly moving and frequently breaking target, are the ones who lose out.

For the record, I debated whether to post this. Ranting on public forums is rarely constructive. But in this case, I think the presumption that it takes much less time to file good bug reports than to find workarounds must be challenged, or the situation isn't going to improve. For someone who isn't a regular contributor, following the guidelines to file a bug report in the way that Mozilla and co would like isn't the five minute job it should be, it's closer to a five hour job, and I think sometimes the developers/regular contributors/bleeding edge alpha testers don't even realise that.

fmoralesc• 13 years ago

Just to add some data points:

In-canvas kerning seems to be affected by the system rendering. In Arch Linux, using Firefox 22 and the Infinality patches, I get http://imgur.com/eRVru67 IE kerning is better when scaled up, but it looks just as good or better in regular or scaled down sizes.

On the differences between doubling the font and scaling it, I get http://imgur.com/sgu8KYG

mati2• 13 years ago

My experience with reporting bugs in FF/Chrome is a bit different. I reported a bug in canvas text implementation, which occurs in both of those browsers, on Feb 20. I provided all the necessary info and even a minimal test case. The bug has been confirmed in FF quite quickly (within days) and then... nothing. It's quite serious, basically you can't do smooth animation (movement) of text. Similar to the text scaling bug.

https://bugzilla.mozilla.org/show_bug.cgi?id=843310

Reporting to Chrome has been even worse. No word, no nothing. Not even a confirmation. (Although the submitting process itself has been simpler AFAIR).

That's really uncool, after a few of those you lose interest in submitting any more since the feedback feels like "we don't care". So you end up using the time to find workarounds instead.

Now obviously someone will point out everyone can go ahead and submit a patch for the bug itself since it's open source. That's all cool and dandy, I love open source as much as the next guy, but please, let's get serious. Who has the time to dive into a massive codebase like that and fix the bug? Not everyone has plans to become a browser developer.

groby_b2• 13 years ago

Well, no, you're not seriously expected to submit a patch for a code base like that. If you can, that's awesome, of course :)

As for your Chromium bug (http://crbug.com/180300) I'd suspect it fell through the cracks with the whole Blink transition. I've tried to narrow down the tags for it, hopefully that will get somebody's attention.

lxt2• 13 years ago

I note a Mozilla developer has commented on this bug today: https://bugzilla.mozilla.org/show_bug.cgi?id=843310#c3

joedrew• 13 years ago

The kerning differences you're seeing only show up on Windows when not using hardware acceleration (Direct2D). When not using hardware acceleration, Firefox draws text with GDI. GDI doesn't deal well with arbitrary scales applied during text drawing and this explains why the same problems show up in Chrome and Firefox. When Firefox is using DirectWrite/Direct2D (i.e., is using hardware acceleration)—like IE 10—these problems don't show up. Short of heroics, there's not much that can be done to improve text drawing at arbitrary scales with GDI.

homeomorphic• 13 years ago

From a while ago: http://i.imgur.com/62WBzVZ.png

Firefox 22 on GNU/Linux looks a lot better, but still not as good as IE: http://imgur.com/GOCmiN9

youngtaff• 13 years ago

Doing comparative tests of browsers is OK up to a point but in reality the performance problems on the web are due to the way we build and deploy sites.

jumpbug• 13 years ago

In my daily use, I find Chrome to be overall faster and easier on the computer in general. I have recently run into a case during my development where Chrome actually has trouble compared to Firefox ANNNNNNND IE. It involves a large unordered list with several different divs, buttons, and links in it. There is some serious jumpy scrolling compared to both of the others, which surprisingly render smoothly.

This is the only instance I've ever run into where it didn't quite measure up.

humanspecies• 13 years ago

Thanks Google, for investing on the best web browser out there!

visarga• 13 years ago

And that's the day I switched back. I abandoned after FF 3.x

jscheel• 13 years ago

Still has big issues with mouse events between iframes and parent documents. Been an issue since at least 2009. Sad.

Edit: issue was reported in 2001. I can't even comprehend that.

mdhgriffiths• 13 years ago

Websites like "Tom's Hardware" really piss me off.

Pxtl• 13 years ago

I gave up on Firefox a year or two back because of performance concerns - not raw speed but ram consumption (leading to thrashing when tabbing between ram-intensive processes) and its poor single-threaded freezing problems. Nice to hear that FF is getting better and I'll be able to go back to it - its extension community is far better than Chrome's, and Firebug is without peer.

krzyk• 13 years ago

It's funny, I was using Chrome for about a month (when they finally released a Linux version) but was forced back to Firefox mainly because of massive memory consumption (because I keep a lot of open tabs). The other problem I had with it was lack of AdBlock Plus back then - without it the web is unusable.

UberMouse• 13 years ago

Frankly I think Firefox performs a lot better than Chrome for memory usage now. After a few days of having Chrome open it's usually using between 3 and 6GB of RAM (Usually Facebook is using a gigabyte so killing that tab frees up a lot) and this is with <50 tabs open. Friends who use Firefox end up with ~1GB of RAM usage in a similar timeframe.

speedyrev• 13 years ago

Go straight to article referenced in Slashdot http://www.tomshardware.com/reviews/chrome-27-firefox-21-ope...

magicalist2• 13 years ago

I'm really surprised that no one in these comments and the thread on the actual story[1] (instead of this...whatever the equivalent of blogspam is but for slashdot) hasn't mentioned the horrible averaging methodology of this benchmark suite.

The individual tests aren't even a problem (though I would maybe pick some more and/or different ones, especially their somewhat odd benchmark choices in performance and graphics), but the averaging makes no sense at all.

Averaging time-based benchmarks is problematic enough (it feels more right, but it still isn't a good idea), but how on earth do you convert a "number of times we had to refresh a page" result into a number that you can then average with a "standards compliance" count and a measure of memory efficiency?? Even if you normalize (which it doesn't seem like they did from the output numbers, but maybe they did), the numbers still aren't comparable because you've given no account for the relative magnitude of their effect.

e.g. if you have a test of "does the browser have a konami code easter egg?", it doesn't matter if the geometric mean is less sensitive to outliers, because it still doesn't make any sense to take an average of that with "the playback framerate of an HD Trololo video" and then pretend that the average provides any insight. And it's even worse if you then compare that average to other crazy averages!

At best you can look at relative ranking, which they actually mention but then proceed to give exact numbers for their relative ranking. There's no information about "betterness" in there, though, except if you then divide up the numbers again to say "these points came from the win in test A, and these points from test B"...at which point you just have the original tests again. Better to count win/no-wins and use that as your final result. Then at least it's obvious that if some tests are much more non-trivial than others that you'll have to give arbitrary weights for the final result, as opposed to having the arbitrary weights being implicit in the tests themselves.

Sorry for the rant :) This is a good recognition of Mozilla's hard work, though notably they've been winning many of these tests for a while now (especially the memory ones), but it would be nice if tomshardware could drop the basically meaningless overall scores (or we could just collectively ignore them).

[1] https://news.ycombinator.com/item?id=5970429

bzbarsky• 13 years ago

I agree the need to produce a single overall number is nuts. We need a better setup for comparing things, somehow...

That said, if you _do_ need to produce one in a situation like this, then geometric mean is the least bad way to do it because it does not depend on the normalization of the individual tests at all: if you have 'k' tests and change the workload for one of them by a factor of N, then the average will change by a factor of N^{1/k} no matter what the actual numbers are. This is why it's used so commonly in benchmark that want to produce a single number.

Of course this very property is what makes geometric-mean-based benchmarks so gameable: a 20% speedup of an already-fast operation (say eliminating a single instruction) is worth as much for purposes of the benchmark as a 20% speedup in something that's really slow. The result is that browsers keep micro-optimizing already-fast things to win the benchmark game....

Myrth• 13 years ago

I seriously can't understand how people can use it with its constant single threaded UI freezing. I've just tried nightly and still the same issue!

forgotAgain• 13 years ago

I just switched from Chrome to FF. The performance is close enough, and good enough for both, that it's a secondary consideration at this point.

I switched because I got tired of hearing Chrome constantly accessing my hard drive. I wound up going through the list of Chrome switches here (http://src.chromium.org/svn/trunk/src/chrome/common/chrome_s...) to try to alleviate it. Some things helped but not to an acceptable level. I use W7. Using procmon I could see Chrome constantly re-reading keys from the registry and writing to temp files even though caching and pre-fetching were disabled.

I was also was concerned for Chrome accessing my laptop SSD. Even though I couldn't hear it I could see the lifetime allotment of reads and writes being flushed.

rjh29• 13 years ago

On a similar note, my PC's hard drive starts making crazy noises as soon as Windows 7 goes into screensaver mode. I've turned off every disk-related option I can find and it's still doing... something. Maddeningly it stops as soon as the screensaver is interrupted.

Linux is a lot more predictable in that regard...

ericabiz• 13 years ago

Probably the Windows indexing service. It's configured to run when you're not using your computer. You can turn it off and see if that stops the hard drive noises: http://answers.microsoft.com/en-us/windows/forum/windows_7-f...

lesslaw2• 13 years ago

I switched to Chrome to see what the speed was like and I came to the conclusion it was a slicker browser for daily use.

But then I reminded myself that some things are more important than raw speed and responsiveness.

Google's interest is tracking and targeting. Chrome actively worked against me in this regard. Firefox, despite its financial ties to Google, put control and freedom onto my desktop.

A bit of occasional sluggishness is a worthy price to pay.

The beer is free at all the browser bars and while I was drinking it, I remembered I liked free as in speech.

Karunamon• 13 years ago

Google's interest is tracking and targeting.

Last I checked, every bit of this can be turned off in the browser settings if they really bother you that much. (Or heck, use Chromium.)

In my experience, they offer quite a bit. Typo correction, instant and stupidly fast searches from the address bar, that kind of thing.

I'll never understand the pathological hate-on that people have for advertising, I suppose.

lesslaw2• 13 years ago

Advertising is annoying and rude. Flashing away trying to steal my attention from the activity I am trying to pursue.

"don't do that, come and do this instead"

"skip this ad in 5 seconds" - no thanks

The interesting thing about the New Albion was that it was so completely modern in spirit. There was hardly a soul in the firm who was not perfectly well aware that publicity--advertising--is the dirtiest ramp that capitalism has yet produced. In the red lead firm there had still lingered certain notions of commercial honour and usefulness. But such things would have been laughed at in the New Albion. Most of the employees were the hard-boiled, Americanized, go-getting type to whom nothing in the world is sacred, except money. They had their cynical code worked out. The public are swine; advertising is the rattling of a stick inside a swill-bucket. And yet beneath their cynicism there was the final naivete, the blind worship of the money-god. Gordon studied them unobtrusively. As before, he did his work passably well and his fellow-employees looked down on him. Nothing had changed in his inner mind. He still despised and repudiated the money-code. Somehow, sooner or later, he was going to escape from it; even now, after his first fiasco, he still plotted to escape. He was IN the money world, but not OF it. As for the types about him, the little bowler-hatted worms who never turned, and the go-getters, the American business-college gutter-crawlers, they rather amused him than not. He liked studying their slavish keep-your-job mentality. He was the chiel amang them takin' notes.

ekm2• 13 years ago

I just learned something my math teachers never told me from a tech blog:

Geometric mean is useful for comparing when the expected range or units of values is different. For example, startup time is measured in seconds, but BrowsingBench numbers are things like the unitless 6646. The arithmetic mean would fail to "normalize" these values and give disproportionate weight to some over others; the geometric mean is one way of trying to account for this.

mrbill• 13 years ago

I stopped using Firefox and switched to Chrome when, even on an x86 Mac, I could see the browser's UI redraw itself. Not going back at this point.

deepblueq• 13 years ago

So much of any comparison like this is about performance, but I really don't think that's as important as it used to be, and that's coming from someone who generally uses slow computers and optimizes software until they run fast.

I'm typing this on a bit of an exceptional example, a 2.6 GHz Northwood Pentium 4 with 1 GB of single-channel DDR-400 RAM. The one saving grace is that it has an SSD, but I put the swap file on the spinning drive (which is modern). It's running Linux Mint 14, Xfce edition, with a handful of minor OS-level optimizations. Firefox 21, with a fairly standard configuration, flawlessly handles a dozen or more tabs on a daily basis. It's even pretty snappy, more limited by my internet connection (1.5 Mb DSL) than by the hardware it's running on.

If this sorry excuse for a computer does that well, are the relatively minor differences in performance between browsers going to be a big deal on modern hardware? There will be edge cases, such as the people who have hundreds of tabs open at a time, but for the average user I'm having trouble envisioning that.

The things that make a difference anymore are very tough to quantify in tests like Tom's Hardware did. I will always have Firefox around because I think Mozilla actually cares about privacy. I use Presto on my phone because it's the only one I've found that renders things how I want. Many people are tied to a browser because of extensions. Standards support doesn't matter until you find a page where a browser doesn't work, and those pages will be different for different people. Browsers can be rock solid on one computer and worthlessly crashy on another.

I don't think a round of benchmarks has meant anything to me in browser selection for a long time, and when it did I did them myself so as to account for the computer they were running on. I choose by trying to use a variety of them for a while, and a winner always emerges quite quickly.

Shivetya• 13 years ago

I just want to know one thing. When I have only two add ons in Firefox why does it take up nearly 200m of memory with no pages displayed? IE is taking 38m with this page displayed, FF has moved to 208m. Firefox is so damn quick to eat memory it causes my laptop to start caching which decreases performance and eats battery.

tommorris• 13 years ago

The reason I use Firefox is because it has a set of extensions I find useful, and because I like Firefox Sync (end-to-end encryption of bookmarks/history etc.).

Whether the browsers shave a few microseconds off JavaScript performance is neither here nor there for me: but perhaps I'm a weirdo.

Performance is nice, but not enough of a reason for me to switch browser.

Firefox Takes the Performance Crown From Chrome

Related Stories