The Firefox 3.5 fiasco

http://weblogs.asp.net/fbouma/archive/2009/07/09/the-firefox-3-5-fiasco.aspx

by felixmar • 17 years ago

145 108 17 years ago

weblogs.asp.net

felixmar• 17 years ago

I used Procmon to monitor Firefox at startup and it indeed traverses my entire temp folder and Internet Explorer's cache folder.

tptacek2• 17 years ago

Although the interminable feeling the author had while waiting for Firefox 3.5 to start up mirrored my own feeling of waiting for him to get to the f'ing point, he's absolutely right: Firefox should just be using the system's secure random number generator on each platform. If you have a vulnerability in /dev/random or CryptGenRandom(), you have a massively more important finding than Firefox crypto.

maukdaddy• 17 years ago

Apparently the 10,000 eyes of open source missed this huge mistake =( This is exactly the kind of thing I wouldn't except to see in open source software.

jerf2• 17 years ago

This is the 10,000 eyes of open source catching the mistake. It happens in public because there is very little "private". Working As Designed.

How often do you see a forensic analysis of this type on commercial software? Not zero, but less often.

codyrobbins• 17 years ago

This is exactly the reason, as a general rule of thumb, why I don't like end-user software packages that are designed to be cross-platform. While I find Firefox 3.5 on OS X to be way faster than Firefox 3.0, it still doesn't hold a candle to Safari. It may as well be written in Java. For something as fundamental as a web browser, which I'm using all day long, it can't take 30 seconds to open a new window, freeze up and start thrashing halfway through rendering a page, or crash twice a day. (All of which Firefox 3.0 did, and which made me decide to move to Safari.) On top of the performance, the fact that Firefox uses a fake skinned XUL look-alike interface on each platform is the straw that breaks the camel's back, at least for me.

Port the renderer cross-platform, and write separate native apps for each target platform. The amount of complication and cruft that must be in the codebase for making a massively complex software package like Firefox work cross-platform, and the amount of basic OS-level functionality that must have to be reimplemented from scratch because it isn't available on some particular target platform, is probably no small contributor to its performance woes.

lamnk• 17 years ago

Of course Safari loads faster than Firefox on OS X, as IE loads faster in Windows. Their libraries are preloaded by OS ...

Native apps for each platform may give performance boost, but maintaining a consistent UI will be a lot harder

blasdel8• 17 years ago

It gets worse: sometimes they won't use platform libraries because it conflicts with the ideology they're trying to force on the web.

"Of course we won't use DirectShow / QT / GStreamer as an alternate backend for <video> -- that would make it actually useful in the real world, and our Theora evangelism comes first!"

blasdel8• 17 years ago

Gecko used to be used directly by a number of consumers for implementing native applications, but Camino on OS X is the only one left. All others are long-dead or switching to Webkit.

Mozilla threw all of their weight behind XULRunner years ago. It probably wasn't that bad of a decision, given how much of a headache it is to use Gecko directly.

voidpointer3• 17 years ago

I think you made a good point there about the drawbacks of cross-platform end-user apps. The problem is that the idea of developing a single code-base for all the different platforms are so damn compelling to both managers (from a business perspective) and also lots of engineers who've learned to accept code re-use as the holy grail of our craft.

Even doing the core rendering part cross-platform has a lot of technical issues as the platforms provide a lot of functionality that can be leveraged by the rendering engine (typesetting, drawing, animation etc...). Consequently, the more you want to integrate with the native facilities of the platforms, the more interfaces and glue components you need in order to hook the platform independent part up to the native facilities. This leads to a loads and loads of ceremonial code and interfaces that introduce additional layers of complexity.

gms2• 17 years ago

Some background: on Unix, Firefox obtains seed entropy from /dev/urandom, while on Windows it calls RtlGenRandom() for some reason instead of CryptGenRandom(). This seeding material is then fed to the random number generator, a NIST SP800-90 Hash_DRBG.

However they do obtain additional (optional) entropy (if you're familiar with the NIST DRBG's, then I'm talking about the 'Additional Input' parameter) by reading system time while periodically reading a bunch of system dirs to advance time in a non-deterministic manner. And that's where this bug creeps up.

itgoon• 17 years ago

> it calls RtlGenRandom() for some reason instead of CryptGenRandom()

It is calling a kernel interface. The prefix "Rtl" means that it is in the kernel run time library.

Reference: http://blogs.msdn.com/wdkdocs/archive/2008/12/19/windows-ker...

skip• 17 years ago

And this is precisely why the most recent Firefox install I have is from the 2.x series. It used to be that Firefox (then Firebird) was the lean and mean cousin of bloated Netscape. Alas, age has taken it's toll and the amount of built in crud in Firefox that has nothing to do with downloading had display HTML documents is overwhelming...

ScottWhigham• 17 years ago

Hmmm - looking back, I'm not sure why I'm on 3.0x. Were there new features? JSON backups - whoopee! Other than that, from a user's perspective, I'm not sure there is anything better. 2.x was fine by me too.

stepherm• 17 years ago

The "awesome bar" (updated address bar) was added in 3.0 I believe.

thaumaturgy• 17 years ago

2.x has at least one rendering bug that was fixed in 3.x -- whatever rendering bug it was that I discovered the other day but can't now recall.

mr_justin• 17 years ago

Because the JavaScript engine in 3.0 is lightning fast compared to the 2.x engine and the 3.5 is ummm, faster than lightning. By a heck of a lot.

For me personally, the only reason I downgraded back to 3.0 from 3.5 is because Firebug 1.4 is bloody awful at the moment (in beta). Other than that, Firefox 3.5 is great.

pbhj• 17 years ago

I quite like the awesome bar now (was that in 2.0?); some CSS3 support too. Bookmarking tagging, update checking are good as well. Improved SVG support I use occassionally for viewing.

Other than that I suppose faster javascript and optimisations through use of sqlite for history/bookmarks is good.

Probably a few other things if I thought about it.

Main features are in add-ons for me, firebug, yslow, noscript, adblock, seoquake,download toolbar, greasemonkey.

I keep a separate clean profile, without all the footer-bar ("clutter bar") icons for the wife to use whilst surfing

robin_reala2• 17 years ago

Because the number of users on Fx2.0 is low enough that we don’t support it any more. It won’t look broken for the most part, but there’s no guarentees as we don’t actively test in it.

FooBarWidget3• 17 years ago

* Awesomebar. Some people hate it but I love it, and in Firefox 3.5 it's extremely easy to disable via the GUI.

* Bookmarks. I can easily bookmark any page by clicking on the star next to the address bar. Clicking it again allows me to edit it or remove it. I use this all the time.

* History with search.

* Spell check. I used to think it's useless but now I cannot live without it.

Dave_Kean• 17 years ago

Apostrophe fail.

apgwoz• 17 years ago

even leaner was phoenix! Ah, those were the days.

owyn• 17 years ago

To summarize for those who have less spare time than I:

Some firefox developer had a Very Bad (tm) idea to seed a random number generator by scanning the Windows Temp folders, which is now causing a 30 secs to over a minute pause in start up for a lot of users (particularly those that use IE, which creates a LOT of temp files). Yikes.

OperaLover• 17 years ago

At least the good news about the Bad Idea(tm) is that this should be a very small fix for 3.5.x.1 - as opposed to some competing browsers invested development that ignored/contradicted standards, requiring complete re-do's.

michaelneale• 17 years ago

Wow - thanks for the summary. Certainly didn't need wads of text to say that.

Its one of those things "seemed like a good idea at the time, worked well, I feel clever" and then you move on. Then it bites you (or one of your users).

pj• 17 years ago

my question is: If you have NO files in any of your temp folders, do you always get the same random number?

Doesn't seem very random to me.

jrockway6• 17 years ago

One bug is a "fiasco"? How about the 15 years the web has been set back by supporting broken browsers like IE6? Oh, well at least it starts up quickly...

The real issue here is how hard it is to be a consumer of open source software on Windows. On Debian, if I wanted to fix this, I would just "apt-get source" the relevant package, make the change, and have Debian build me a new package with the fix. On Windows, this is apparently not possible, as there is no package management system to install the compiler and source code for you, and the apps check themselves to make sure that you don't modify them.

(I also like how the virus / spyware situation is so out of hand on Windows that you can't even recompile libraries without your own computer assuming you are hacking yourself. Nice.)

Why do people still use Windows?

phatboyslim• 17 years ago

My grandma hacks debian and changes source code all the time. I don't get why people use Windows either.

jrockway6• 17 years ago

This guy is not your grandma. His blog claims he is a "most valuable professional", so executing a Makefile should not be too difficult for him.

I have different expectations for professional programmers than I have for 80-year-old non-programmers.

ErrantX2• 17 years ago

maybe not a fiasco but a serious bug. Im using IE8 at the moment till a fix appears - it's just faster.

wvenable• 17 years ago

Speed isn't everything. For the love of all web developers everywhere, pick any browser (Safari, Chrome, Opera, or Firefox) and dump IE. We would greatly appreciate it.

rbanffy4• 17 years ago

I just let it start up when I log-in. By the time Outlook and the anti-virus (and, presumably, all the malware that was not caught by either my anti-virus or my IT staff), Firefox will be there too.

In the meantime, I walked the floor, said "good morning" to my co-workers and got a nice warm cup of coffee from the espresso machine.

Could I use Linux, I would have to start working only a couple seconds after my computer was turned on. Think of it as a quality of life issue ;-)

cedsav• 17 years ago

I think the author addresses that point. You can't fix that bug yourself because the code needs to be signed by Mozilla (security layer).

jrockway6• 17 years ago

Well, you have to compile the whole thing, not just replace one library. If you are writing the software, you can remove the signature check, obviously.

FiReaNG3L• 17 years ago

When one of the selling point of the update is speed, and you slap a 30 seconds startup penalty for no good reason at all, yeah, its a fiasco.

I was wondering what was taking so damn long to start FF3.5 - on a SSD. I cant imagine ppl on a HD.

jrockway6• 17 years ago

It's a bug. Bugs happen. You encounter one, fix it, and then it isn't a bug anymore.

Have you ever written a computer program before?

a2tech• 17 years ago

Way to completely miss the point-Windows is NOT stopping him from doing anything-Firefox is. The problem lies within the NSS (security stack) inside of Firefox. NSS is integral to Firefox but is not in the main source-it is a separate project. Firefox checks the binaries when it launches to verify that NSS is a Mozilla signed build. So to build Firefox to fix the problem, you must: Download Firefox source and NSS source Make the fix in the NSS source Edit the Firefox source to remove the Mozilla signed binaries check Build everything and hope it works There are MANY perfectly true complaints about Windows and the Windows software development world-but this is problem is all Firefox.

whatusername• 17 years ago

15 years? From 2001 all the way back to 1986.

What browser were you using in 2001 that was better than IE6? Netscape 6?

boucher• 17 years ago

Funny, my reaction to your statement is, why would anyone use Linux? (as a desktop operating system...)

I'm a pretty good programmer. If I had to change the source on a program to use it, I wouldn't. I have better things to do with my brain cycles, and that's all there is to it.

jcsalterego• 17 years ago

line-height: 0.1em ftl

(gross exaggeration)

ironkeith• 17 years ago

As soon as I see eye filth like that I go right to my readability bookmarklet:

http://lab.arc90.com/experiments/readability/

80% of the time, it works every time.

Encosia• 17 years ago

Thank you.

hesdeadjim• 17 years ago

Oh you rock, this is awesome. As soon as I opened the link for this news item I closed it immediately due to the complete lack of readability. Someone buy the man a book on typography, please!

zacharypinter• 17 years ago

This comment was more valuable than the article. Thanks!

rpdillon• 17 years ago

You just made my day. Thank you, sir.

rms2• 17 years ago

There's also always View -> Page Style ->No Style

matthewking2• 17 years ago

A tool as brilliant as this, combined with an audience that reads tons of articles spread all over the internet every day, surely it deserves its own thread?

I can see myself using Readability to format most of the articles I read online now.

Thanks for the link!

surki• 17 years ago

btw, if you are using conkeror(or may be vimperator), use this tip to bind it to a key(I bind it to 'z'). Much easier. http://conkeror.org/Tips#Makethecurrentpagereadablebyremovin...

ajtaylor• 17 years ago

What a fabulous little tool! Thanks for the link.

jayroh• 17 years ago

Before submitting my own comment re:the lack of line-height on this site I searched for "line-height" in the comments because I know Hacker News so damn well.

Forchrissakes people how does anyone think this is even approaching readable!? Someone needs a whack with the UI stick

slater2• 17 years ago

Or just surf around with firebug installed. Then: right-click the text area, choose "inspect element", then right-click the DIV its in, in Firebugs code area, then add this attribute: style="line-height:1.5; font-family: Lucida Grande; width:50%; margin:0 auto" (asp.net's ridiculous CSS is a bit trickier, but that CSS snippet usually gets most sites up to readability scratch)

Oh, and a poor man's arc90 readability thing can be found with the "zap annoyances" bookmarklets:

https://www.squarefree.com/bookmarklets/zap.html

voidpointer3• 17 years ago

Well yes, it's a bug but it's hardly a fiasco. Something like this should be caught in pre-release testing. Still, most users have their browsers running all the time anyway so a slow startup is not really that critical. I've seen far more serious bugs being taken far more lightly.

dave_au• 17 years ago

I'd more alarmed by the fact that they've thrown something new into the mix security-wise than the effect on startup time.

If the system pseudo-random number generator has no problem, what's the use case for not using it? And if it does have a problem I'm sure they would have told someone :)

I just don't know why you add another moving part to the security system if you could avoid it, especially since it seems like a reasonably safe bet that people have had more eyes move over / experience with the alternative.

voidpointer3• 17 years ago

Yeah, it's a stupid bug. Most bugs are once you find them.

I also agree that this got into the security layer is probably the most worrisome part of the whole story. Maye this could build a case for mandatory code reviews for security related modifications.

gizmo• 17 years ago

Maybe it's just me, but I consider this a (serious) security issue as well. The contents of temp files, which Firefox shouldn't even have access to, will now float around in the process memory of Firefox, even after the memory has been deallocated.

Firefox should never, ever, ever, open any file outside its own app directory, user preferences directory, or cache directory.

And a reasonably secure OS shouldn't even allow Firefox to open any of those files.

teilo• 17 years ago

Not necessarily. If this is a typical seeding algorithm, at most there would be one block of data in memory, used for hashing. And one would also think they are smart enough to zero it out before deallocating it. It IS a security library, after all.

__david__2• 17 years ago

If the files are really that important shouldn't they have good permissions set? That is, if "Firefox shouldn't even have access to" them, shouldn't that be enforced by some OS level permissions?

And shouldn't any other programs one has around not be writing confidential information to temp files in the first place?

Seems like more of a dumb bug to me than a security problem.

blasdel8• 17 years ago

It gets worse security-wise -- their recommended fix is to delete all your caches, removing the entropy seed from their RNG: https://support.mozilla.com/tiki-view_forum_thread.php?comme...

And if clearing the caches doesn't affect entropy quality, it's even stupider -- why bother to read them in the first place?

billzeller• 17 years ago

If you want to see the code that does this:

EnumSystemFiles: http://mxr.mozilla.org/mozilla-central/source/security/nss/l...

...called by rng_systemJitter: http://mxr.mozilla.org/mozilla-central/source/security/nss/l...

...called by rng_systemFromNoise: http://mxr.mozilla.org/mozilla-central/source/security/nss/l...

...called by RNG_SystemRNG: http://mxr.mozilla.org/mozilla-central/source/security/nss/l...

...called by (among others) rng_init: http://mxr.mozilla.org/mozilla-central/source/security/nss/l...

andrewf• 17 years ago

My best guess.. RNG_SystemRNG first tries to use a deprecated API (http://msdn.microsoft.com/en-us/library/aa387694(VS.85).aspx), then CryptoAPI, then this temp file entropy gathering approach. But if the deprecated API is _broken_, rather than absent from the DLL, then it will fall back to scanning temp files without ever trying CryptoAPI.

fno2• 17 years ago

I wish the Firefox developers would focus on bug fixing for the next big release. There are so many bugs that are not fixed for many years. For example alt-text on images not correctly shown or disable-output-encoding on XML/XSLT being ignored. Instead that add more and more features. The awesome bar already made me switch to Opera, I miss some extensions though so I do keep an eye if Firefox gets "better" for me.

robin_reala2• 17 years ago

alt-text on images not correctly shown

What is correctly shown? It shouldn’t be shown as a tooltip as that leads people to write alt text that’s specifically useful as a tooltip and useless to the real consumers of alt text: screen reader users and search engines.

fno2• 17 years ago

It was something like no alt text being rendered when images are not displayed or found, I don't fully remember and could not find it in 5 minutes Bugzilla search hell. I did stumble on the still-open bug some days ago though, I think it was 8 years old now.

aceofspades19• 17 years ago

Whats so bad about the awesome bar?

FooBarWidget3• 17 years ago

No idea, it's one Firefox's killer features I miss in other browsers.

That said, Firefox 3.5 allows one to VERY easily disable the awesomebar: Preferences -> Privacy -> Location Bar.

weegee• 17 years ago

I find Safari 3 to run very well on Windows, and I've been using it at home lately. It passes the Acid3 test at 100 percent.

rdoherty• 17 years ago

FYI Mozilla does have an open blocker bug about this problem:

https://bugzilla.mozilla.org/show_bug.cgi?id=501605

schizoidboy• 17 years ago

The "thread dump" feature should be an absolute requirement for any language runtime environment. Java has perfected this with the SIGQUIT signal. Thread dumps give you readable stacks of all threads with a simple user command (kill -3 on Linux/Unix, although a bit more difficult on Windows), without having to install symbol files, install additional software, run "scary" commands (for end users), etc.

This is a textbook example of taking a few thread dumps, a few minutes apart, immediately showing what is going on. IBM has one of these as their Performance or Hang "MustGather" script:

http://www-01.ibm.com/support/docview.wss?uid=swg21115785...

.NET thread dumps suck -- you have to use adplus to attach and they're not easy to read. Native programs (e.g. C/C++) work better with symbols and are just too scary for end users. DTrace on Solaris with Ruby/Python/PHP extensions is nice, but too cumbersome to install, and again too scary for end users.

I'm not aware of built-in thread dumps-by-signal in other non-Java languages (please note in a comment if there is), but this feature is so basic and needs to be baked in to every runtime environment and easy enough for end users to use.

Firefox, being a native program, is in a bad problem determination position here and the haphazard nature of the problem determination process in the forum and bug report shows that. I suggest Firefox create something like IBM's MustGather scripts (starting with a performance one such as IBM's hang MustGather -- http://www-01.ibm.com/support/docview.wss?uid=swg21115785 [click "Show Details" for the steps]). A hang MustGather should go through the process of installing symbol files, getting the thread stacks, submitting the information, etc. This will allow users to at least feel like they can do something valuable to help the developers fix the problem.

The Firefox 3.5 fiasco

Related Stories