Joel on Solid State Disks

http://www.joelonsoftware.com/items/2009/03/27.html

by dmytton • 17 years ago

69 79 17 years ago

inklesspen2• 17 years ago

Ah, the cost of premature optimization: a couple thousand bucks, and a couple days of monkeying around. Better to first figure out where the bottleneck is (and maybe listen to your developer, since he thinks parallelization will help.)

palish• 17 years ago

Bingo! Beat me to it.

pkaler2• 17 years ago

Listen to the last Stack Overflow podcast. Every single month that they ship FogBugz earlier is an extra $200k in revenue share for the developers.

The developer may know how to solve the problem in code. But, Joel is the CEO. He has a better idea of how this affects the Balance Sheet and Income Statement. It's kinda, sorta his job.

inklesspen2• 17 years ago

Except that (a) he spent a couple days of his time being wrong about it, and (b) he then wrote a blog entry about how he was wrong about it.

Imagine if instead he'd said to the guy "okay, spend three hours profiling the build process and get me some suggestions with time estimates", and they'd found some likely prospects, and three days from now he gets to post about how they'll be shipping the next release a month earlier because of the improvements they made to the build process.

mmelin2• 17 years ago

I think this was just an excuse for Joel to get SSD's for his laptop and workstation - and a good one at that!

cubicle67• 17 years ago

Exactly.

"Hmm... Love to get one of those new SSD drives. Just need to justify it somehow"

spolsky6• 17 years ago

Yeah, otherwise my boss never would have let me spend the money ;/

mmelin2• 17 years ago

Yes - I love everything about being self-employed except for the fact that my boss is an asshole.

Timothee2• 17 years ago

Well, in a few months, looking back at Fog Creek's spendings, you might have wondered who the hell authorized this spending... (http://www.joelonsoftware.com/items/2008/12/10.html) ;)

pbz• 17 years ago

What about compiling with VS? Any improvements, and if so by how much?

wheels• 17 years ago

What language is your compiler written in? Most languages have an off the shelf implementation of map / map-reduce or similar these days, which would seem well-suited to compiler tasks and greatly simplifies parallelizing jobs. We use map and map-reduce patterns in C++ heavily which is quite nice since if you've got that relatively small core of code thread safe you can mostly ignore thread safety in the major portions.

You also can often speed up I/O that way since you can saturate both your I/O and CPU bandwidth, rather than alternating between waiting on them.

I also take it that you're not compiling to individual object files since usually compilation since usual compile parallelization is based on farming out single object file compilations to multiple machines or processes.

I wrote some small portions of "icecream" which is used for compilation parallelization by the guys at SUSE and there it would actually transmit over the network a full chroot environment so that even different Linux installs working on compatible architectures (later cross-compiler support was added as well) could execute the compiler jobs of rather different OSes.

brooksbp• 17 years ago

At least you know that the compilation process isn't I/O bound!

mynameishere3• 17 years ago

Seriously? Compiling? What language are they using that requires actual, regular, compile processes?

blasdel11• 17 years ago

Their bullshit private VB dialect that was someone's weekend project: Wasabi http://www.joelonsoftware.com/items/2006/09/01b.html

jcdreads• 17 years ago

Are you angry that they used/wrote something called wasabi? Joel force you to read stuff that makes you mad?

I mean, seriously, what's with all the random ire in the HN comments on this story?

gstar2• 17 years ago

This is just tall poppy syndrome.

Joel runs his successful business, bothers to blog about it since he founded it years ago, graciously comes on here and responds to people - nice of him, huh? So why so bitchy?

You clearly disagree with his company's engineering decisions that you can't possibly have been privy to the reasoning behind. What's your motivation?

smanek6• 17 years ago

What's wrong with that? All else being equal, compiling will generally be faster. Most languages (Python, Java, Ruby, most Common Lisps, etc) do compile - just (usually) to some intermediate and portable bytecode rather than raw machine code. That's probably what their compiler (Wasabi) does too (except, it seems like they are using PHP/VBScript/JS as their bytecode ;-)).

spolsky6• 17 years ago

In FogBugz 6 the compiler output was PHP and VBScript.

In FogBugz 7 the compiler output is .NET/CLR bytecodes.

It also emits compressed JavaScript which runs on the browser.

mynameishere3• 17 years ago

I hate explaining myself, especially when it shouldn't be necessary. Yes, I know Java et al compiles to bytecode. But in practice you never hit a "compile" button because it compiles continuously.

smanek6• 17 years ago

I'm a big fan of incremental builds - no point in recompiling 1M LOC when only a few source files have changed. Image based systems (like Common Lisp) are particularly nice because you can just recompile the functions that have changed and have the changes instantly take effect on a running image. Unfortunately, ASDF (the most common build/packaging system) doesn't have particularly good support for incremental builds yet, but it's an area of active developement (particularly with XCVB and other new build systems).

But don't GCC, Java, and Visual Studio all support parallel builds pretty easily? I've never looked into it, but I know I've read articles about it.

What sort of language/env. does FogCreek use? I vaguely remember hearing that they had written their own compiler in Java that transformed ASP to PHP or some other such ludicrous thing.

Edit: Some googling showed that they decided their software had to run on Windows and Unix boxes, and that the best way to do that was to support ASP (pre .NET) on Windows and PHP on *nix. They already had a large ASP code base, so they wrote a compiler (named Thistle) that could translate a subset of ASP into PHP. Then, it seems like MS deprecated ASP (and they started to realize how much it sucked) so they wrote their own language (Wasabi) that is based on VBScript and can be compiled into ASP, PHP, or JS. My God. I would gouge my eyes out if I had to work there. From the fragments of code I've seen, it frankly seems horrible. See http://www.fogcreek.com/FogBugz/blog/category/Wasabi.aspx

Although, I guess I can't really judge. I recently just wrote some ridiculously convoluted Java code that could automatically and transparently trick the JVM into making certain recursive functions Tail Call Optimizing (i.e., use constant space on the stack, regardless of how deep the recursion is) by transforming the function into automatically issuing/catching Exceptions to manually unwind the stack when I want to. Ah, I miss Common Lisp - I never knew how good I had it.

thwarted• 17 years ago

If you define your dependencies correctly, make can do parallel builds (using the -j option). Of course you will have to use Makefiles in that case, which a lot of people don't like.

smanek6• 17 years ago

Yep, right you are. That seems like a much more reasonable way to go. Parallelize the build - not the compiler.

gord• 17 years ago

Maybe part of that is the pain of dealing with configure / autoconf / autotools?

Ive seen a lot of C/C++ programmers write their own build utilities/scripts and lose proper deps in the process.

I found cmake a reasonable alternative.. scons apparently is too, although I hesitate because it seems to encourage writing your own python build plugins, which can be too tempting for people who like to tinker.

agreed, make -j works a treat, and presumably will gain more as cores multiply.

Its kind of nice to work in a scripting environment and avoid some of these problems [via sensible defaults]

spolsky6• 17 years ago

Seriously? You would gouge your eyes out? Most good hackers that I know actually prefer to work on compilers. We like the fact that we can change the compiler and control our destiny... we've added code generation, active records, lambdas, picture functions, and a lot of other nice stuff to our code without having to rewrite or port any working code.

Does your language let you write one function and have it run on the browser or the server? When you want to change something about your language, can you?

People seem to be pretty quick to criticize the decisions we made without really having all the facts, based on a couple of bits and pieces of obsolete info they gathered from the internet.

smanek6• 17 years ago

That's a fair point - I don't have all the facts.

I do respect that you had a huge (and effective) code base in ASP that you (for very valid business reasons) didn't want to rewrite from scratch. I understand that each of the decisions you have made have probably made you successful and profitable - and in your shoes I probably would have done something similar. From your point of view, I completely understand that your choices were to do a complete rewrite (which was obviously unacceptable), or to come up with these sort of hackish workarounds.

However, I think you have to understand that as a hypothetical programmer thinking about working at Fog Creek, my choice is to work on your large legacy code base in (what appears to me to be) a not very pleasant language - or work on fun little 3 month AI projects in whatever language I choose (which is roughly what I do now). The fact that your compiler infers type information from a variable's name is an ugly hack - and frankly scares me (not only itself, but also more than a little about other design decisions you made). The hoops you guys had to jump through with Wasabi to do metaprogramming would annoy me - I'd rather focus on the fun stuff.

I recognize that, as a business, you made good decisions. But do you understand why I, as a programmer, would not want to work with those sort of technologies?

Oh, and for what it's worth, my language has all the features you've mentioned. For example, we used Parenscript (http://common-lisp.net/project/parenscript/) at my last job - it's a subset of common lisp that can be compiled to Javascript and runs on either the client or server.

Also - sorry about the 'Gouge my eyes out' bit. I meant it to be a bit of funny hyperbole, but guess I failed to convey that properly (humor is hard on the internet). Fogcreek, in my mind, would definitely be above the average Java or .Net shop, but probably below a few fun research labs, prototyping contractors, or small startups.

wglb4• 17 years ago

I remember a chapter or half a chapter in a compiler construction book or article that was effectively titled "Why spend time on compiler optimization", a delightfully ambiguous term. The point was that doing so sped up the compiler which saved time for a large number of users (this was back in the days of compilers as commercial products). Clearly the discussion was targeted to self-compiling compilers.

But the lesson here would be that some effort spent on the speed of compile could positively impact developer productivity or at least negatively affect swordfight time. And yes, I think that very short compile times, like you see in CL (sbcl as an example) or Smalltalk can have an avalanche effect on development. And yes I do believe that small incremental changes are th3e best way to do software development. And yes I would expect to recompile a function in a running image serving web pages as a routine part of development.

But maybe that is just me.

blasdel11• 17 years ago

He doesn't mention that the "compiler" that they're bottlenecking on is the one they wrote themselves, for "Wasabi" -- compiling their private language into VB or PHP.

On top of that, you don't fucking parallelize compilers!

You use a goddamn build system to do that (make, et. al.). Of course, that requires that your compiler be decent enough to be able to compile modules independently, and not have to recompile unmodified source. What do you want to bet that their compiler is just ridiculously awful?

jrockway4• 17 years ago

There’s an open source app called Clonezilla, which, I have to say, is only free if your time is worthless.

What about dd if=/dev/old-disk of=/dev/new-disk bs=16M ?

andrewtj• 17 years ago

It's trivial to folks familiar with nix-a-likes but for others it is too unlike the abstractions they're used to for it to be entertained as anything other than some sort of bizarre incantation.

spolsky6• 17 years ago

If disk sizes are different, which they almost always are, AFAIK dd doesn't quite hack it, otherwise I would have just booted off a Ubuntu live CD and done that.

wmf• 17 years ago

Yes, after the dd you have to run parted to adjust the partition table and then possibly use some other tool to grow the filesystem.

blasdel11• 17 years ago

partimage was the tool you were looking for. It natively understands NTFS, FAT, extX, Reiser, XFS, etc. and can resize as it images.

By default it pops up a somewhat annoying ncurses interface, but it's quite a solid tool.

bemmu• 17 years ago

Wasn't there also some magic required so that bad sectors get zeroed on the target disk, instead of skipped, which would be bad.

blasdel11• 17 years ago

No magic, no bad sectors, no relevance.

blasdel11• 17 years ago

You don't usually migrate between volumes of the same size. If the new one is bigger, you've now got unused space on the end. If the old one was bigger, you've now just fucked your filesystem -- not only will the metadata about the volume size be wrong, but all the inodes stored in blocks past the end of the new disk will be missing.

tsally• 17 years ago

Suggestion: check whether a process is IO bound first before spending hundreds of dollars trying to make it faster with SSDs.

rgoddard• 17 years ago

My guess is that this was the cheaper option. It only cost Joel a couple of days and several hundred on the hard drive. What it did not do is interrupt his programmers. As Joel has already described, he is now a grunt worker. His job is to make it as easy for others to do their jobs. Their time is probably worth 10 to 100 times his, because they are the one actually making something that directly results in more sales and money for the company. So all Joel had to do was save a couple of hours of work for the programmers and he would come out ahead.

Plus these additional side benefits: The programmer who was complaining does not feel like he is being ignored. Joel is now going to upgrade most of their computers, making it better for the entire company. So lots of warm fuzzies to go around, leading to happier programmers, leading to a better product.

litewulf• 17 years ago

The cheaper option would be to run a compile and open any of the many different programs in Windows that tells you whether you're using lots of CPU, RAM or IO-waiting.

Then you go AHA! I'll spend money on CPUs or making my compiler/build parallel.

ighost• 17 years ago

You probably don't even need to check. If it's a compiler, chances are it won't be IO bound.

wglb4• 17 years ago

Not necessarily, I think. If you have lots of memory, a second compile where stuff gets cached might be measurably faster.

ilaksh• 17 years ago

With the current abundance of incremental and/or parallel tools, the fact that the compilation process takes up a significant amount of time is your first clue that there is something fundamentally wrong with the technology selection and overall thought process at this shop.

Also, he could have saved $350 dollars and bought the 80 GB drive (unless he just HAS to store 40+ feature-length pirated movies on his boot/app drive?).

chiffonade• 17 years ago

So... he concludes that CPU bound tasks are CPU bound, and that parallelizing compilation (technically it would be the dispatch of compilation) is better?

Every time I read this guy I fail to get what his allure to developers is.

CRASCH• 17 years ago

I can say my build times went down considerably. However I moved to a completely new machine.

I didn't plan on writing an article so I don't have any official bench marks. I'm guessing they dropped from over a minute and a half for a full recompile to under 30 seconds. Incremental builds used to take about 20 seconds and now take between 2 and ten seconds.

I went from a athlon X64 at 2.2Ghz and 2GB Ram with a 74GB raptor and another 74GB raptor for code to a Core i7 Extreme at 4Ghz and 12GB Ram with 2x80GB intel SSDs in raid 0 and 1 80GB intel SSD for code.

A couple of important things to keep in mind. Intel is still the way to go. The other SSDs can actually be slower in write performance than a normal hard drive if the usage pattern is small files. That is pretty much what you get when you compile.

One other point is that crappy anti-virus software can affect IO performance anywhere from 12X trend micro, 24X Norton. Yes that is Norton will reduce your IO by 2400% in my tests. AVG Free is 19%. So he could have negated the performance gain from the SSD by running bad anti-virus software.

I can say my productivity feels like it nearly tripled.

The important measurement from a programming perspective is not necessarily just the second or two here or there. My work flow changed. I used to hit build and walk out and get drink from the fridge in the garage and it would still be compiling. Now I can't even do anything but glance at my email. These interruptions sometimes take you out of the zone and it can take a long time to get back in the zone. Getting up could easily cost me fifteen minutes or even a half an hour. Just because I lost my train of thought.

Update: I'm running windows 7 and visual studio 8 on this machine.

MichaelApproved• 17 years ago

So he didn't fix the compile speed but he fixed the speed of everything else. I know every second I have to sit and wait for my hard drive to catch up is a second I'm spacing out instead of working. As a developer I just want to get my idea out and code. Anything that can be done to make that faster makes me a happier coder.

richcollins• 17 years ago

I guess you shouldn't always trust your assumptions, but did he really think compilation might be I/O bound?

markessien• 17 years ago

This could have quickly been tested by using a memory disk and trying if the compile would go faster when the files are located there.

jwr• 17 years ago

To summarize the article: yeah, SSDs are faster, but not quite for everything, because if you're CPU bound, SSD won't make your code run faster.

So, why does this get upvoted, again?

rozim• 17 years ago

They should try distcc

Joel on Solid State Disks

Related Stories