Where Lisp Fails: at Turning People into Fungible Cogs

by fogus • 17 years ago

56 90 17 years ago

trapper11• 17 years ago

Surely if lisp makes a programmer N times more efficient then it would be easy to study scientifically. Have there been any studies comparing productivity?

Why is it that in RAD application competitions, "mainstream" languages often win?

Also, if it were truly n times better the library disparity wouldn't exist; libraries should be n times faster to make or duplicate.

I'm asking for evidence here: how can something be so much better and have such a paucity of evidence for it's efficacy? If there really is such scientific evidence for lisps effectiveness I would love to read it & and distribute it to my colleagues in charge of large development teams.

YuriNiyazov• 17 years ago

There has never been a rigorous scientific study done, no, but there's plenty of anecdotal evidence, and, to quote Paul Krugman's tongue-in-cheek title, "the plural of anecdote is data"

trapper11• 17 years ago

That's the same explanation psychics give. If the effect size is large (as is claimed), it should be an extremely simple study. So why has no one done it?

Why don't the lispers organize a RAD competition between lisp and all-comers, surely if lisp is that much more productive you would see the top ten places being lisp dialects.

Would you agree?

jcapote• 17 years ago

Paul Graham writes about this extensively in http://www.paulgraham.com/avg.html

trapper11• 17 years ago

It's computer science not computer woo. I'm asking for evidence not anecdote :)

Surely, there has to be at least some evidence in it's 50 years? Especially given the claimed huge effect size.

randallsquared2• 17 years ago

libraries should be n times faster to make or duplicate

Oh, they often are... and that's why nearly everyone just builds their own. Ease of building it yourself compared to learning someone else's system is a major reason Lisp is languishing, in my opinion.

xpaulbettsx• 17 years ago

I agree - I really think that when it comes down to it, Lisp isn't popular because it's harder to read. Even if it's more terse and far more elegant, the mental tax of trying to understand another person's code is far higher than C, where there's only "the dumb way" to do something.

If people can't understand the code, they can't contribute, so you don't end up with the network effect of F/OSS contributions that make a language popular (note that I say popular, not better)

joe_the_user2• 17 years ago

Ease of building it yourself compared to learning someone else's system is a major reason Lisp is languishing, in my opinion.

Oh, that why Perl is languishing too. Well, the problem is how hard it is to learn someone else's system in Perl. If Lisp has that problem, well, there you are.

silentbicycle4• 17 years ago

Oddly, it can also be a downside. A language that makes reinventing the wheel too easy means that people can spend more time saying, "Psh, I could do that in a couple hours" than sharing. Then, when people want to know what the library is to use for parsing HTTP headers (or whatever), everybody and their nephew shows up to suggest their own half-assed framework. It doesn't inspire confidence.

Take that to its logical extent, and ... wait, which Lisp implementation do you use again? Mine only works with SBCL, crap, nevermind.

When somebody has a really thoroughly debugged C library, it can be like huddling everyone together by a fire for warmth in the tundra. If you don't share, man, that's just cruel.

trapper11• 17 years ago

This is a testable hypothesis due to the number of popular open source projects around we can draw from.

rbanffy• 17 years ago

Ease of building new stuff brings fragmentation:

http://news.ycombinator.com/item?id=1013716

10ren3• 17 years ago

Perhaps one source of data is YC startups. What's the success rate of lisp vs. non-lisp?

Yes, of course, many things contribute to success other than language choice. That's my point.

ams6110• 17 years ago

Maybe it's not that lisp makes any programmer more productive, but that it offers the anecdotal "order of magnitude more productive" programmer features that Java, being designed for the "interchangeable cog" programmer, does not.

naveensundar• 17 years ago

trapper, this study might be in the flavor of what you are seeking. http://www.flownet.com/gat/papers/lisp-java.pdf (2000)

asciilifeform• 17 years ago

> Surely if lisp makes a programmer N times more efficient then it would be easy to study scientifically. Have there been any studies comparing productivity?

I began to write a reply to your question, but decided that it deserved its own post:

http://www.loper-os.org/?p=85

trapper11• 17 years ago

I refer you to:

http://news.ycombinator.com/item?id=1016762

What's wrong with that as evidence for lisps superiority?

plinkplonk6• 17 years ago

"I began to write a reply to your question, but decided that it deserved its own post"

That isn't really a reply to the question posed. Just more ranting and raving.

Sample: "Yes, my dear bean counters, you can measure productivity. I would no more ask you to stop in your attempts at its measurement than I would ask mosquitoes to stop sucking blood. You can measure productivity – even of Lisp; even of political philosophies. You will simply need to secure a very large bag of beans – one deep enough to hold a bean for every twist and turn of a century of tinkering, politicking, and everything else associated with the messy business of successfully thinking new thoughts."

The best interpretation of this drivel is that you want people to accept your claims of superior productivity just on your say so but not ask for any evidence because asking you back up your unsubstantiated claims of superiority would be "bean counting".

Forget the scientific studies. There is another kind of "evidence" in programming - write code that is evidently superior. The best way to show that (say) Linux is an "inferior" operating system is to build one better. Not talk about making one sometime in the misty future. Build. Then talk.

Endless rants aren't as good. As silentbicycle says elsewhere

"I just take issue with how some particularly vocal Lispers have been sitting on the sidelines going, "Our language is THE BEST LANGUAGE" for decades. ("Sick of being a blub programmer and working in Blub? Try our Language For Smart People.") They're so used to being smug because their language was thirty years ahead of its time, fifty years ago."

pg• 17 years ago

Curiously, it's inherently impossible to compare what I think matters most for languages: solving new problems. To compare two languages you have to have people solve the same problem, and that means it has to be a predefined one.

trapper11• 17 years ago

This is what statistics are for, and is exactly the same situation that many other disciplines find themselves in where treatment a influences treatment b.

gruseom• 17 years ago

I agree. Predefined problems are one way people compare languages; code snippet comparisons are the other. Neither touches the important thing, which is how languages differ as a medium for thought. Different languages give rise to different ideas, which give rise to different programs.

The other problem with studying this question formally is that it is so psychologically rooted that people almost invariably "find" that their favorite language turns out the best.

javert3• 17 years ago

I think the comments here about the free software community, and particularly the Linux example, are really off base.

Consider Linux: the poster child of successful free software. It is a knockoff of a 1970s operating system well past its sell-by date.

What does that even mean? Linux (as a kernel) does an excellent job at what it is designed to do. I don't know what is meant by "innovation" here. If the author has a fundamentally different kernel design that is actually better, do tell.

On the other hand, at the subsystem level, there is lots of innovation in the Linux kernel. The Completely Fair Scheduler (CFS) is a good recent example.

This is because a herd simply cannot innovate, whether for fun or for profit. Every innovative work of mankind has been the product of one – sometimes two, rarely three – minds.

Linux development is not driven by a "herd". It's driven by a group of pretty elite programmers known as the subsystem maintainers.

All the major patches are done by one expert, and many of them are fairly innovative. Ingo Molnar did CFS. Ted Ts'o does a lot of really high profile work on filesystems. There are many more examples.

The Linux development process is lightyears ahead of the typical industrial development process. Git, which was created by and for Linux kernel hackers, has been instrumental in helping Linux get ahead.

Distributed version control is a fundamental innovation in software development, and Git is the most successful DVCS by far.

(Other than that, I think the author makes a great point, and I agree with him 100%. That's why I have so far avoided working for big software companies.)

camccann5• 17 years ago

Distributed version control is a fundamental innovation in software development, and Git is the most successful DVCS by far.

Is this is the same git that Torvalds tossed together as a quick hack to replace the proprietary BitKeeper DVCS after the company stopped offering free licenses to Linux developers?

javert3• 17 years ago

It started out that way, but as I said, Git is now by far the leading DVCS, and AFAIK, the Linux community was the first to do DVCS on a grand scale. Furthermore, Git is surely a tremendous improvement over BitKeeper.

That's innovation in software development process.

silentbicycle4• 17 years ago

Nah, I think it's the git that Torvalds wrote when he decided that Monotone had some pretty cool ideas (the whole distributed VCS thing) but was slow and written in icky C++.

tolenka3• 17 years ago

The success of Linux deserves serious consideration, but breathless marvel at Linux and its development practices is entirely misplaced.

All the major patches are done by one expert, and many of them are fairly innovative. Ingo Molnar did CFS. Ted Ts'o does a lot of really high profile work on filesystems. There are many more examples.

Linux is full of examples of poorly designed systems being slipshod replaced for slightly better designed systems that are well behind the state of the art.

In the same time Linux has gone through multiple schedulers (and scheduler flamewars), FreeBSD developers have designed, implemented, tested, and iterated on ULE, and then ULE2, finally (after much review, testing, and careful consideration) merged it and made ULE2 the default scheduler.

The scheduler wasn't adopted until it was investigated from all angles over years, and found to not introduce any significant performance regressions.

The kicker? The previous scheduler was on par with Linux, ULE2 seems to perform considerably better for many workloads (eg, MySQL) on many-core machines.

Innovation isn't magic -- it involves a lot of hard work and careful design by a lot of people.

Really? I'm not sure what innovations you're thinking of here.

  - Solaris' ZFS vs (incomplete) btrfs
  - *BSD's kqueue vs. epoll/inotify/dnotify/
  - Solaris' zones vs ?
  - FreeBSD vimage (network stack virtualization) vs ?

This isn't intended to be a Linux-tirade -- but I wanted to illustrate that Linux is not "lightyears ahead of the typical industrial development process" when compared against other existing kernel projects.

sandGorgon2• 17 years ago

OK I'm confused now..

I think what the commenter meant was the cathedral vs the bazaar model of development. Would'nt FreeBSD, NetBSD, etc all be in the same boat as Linux? Granted that Linux has a benevolent-dictator-aka-linus, but that does not make it deviate too much from the bazaar model.

For all its flaws, git (and github) are becoming the preferred choice of hackers everywhere - I'm not staunchly one of them yet, but I can see the charm. Git only enhances the distributed development model that all *nixes are built upon.

If any, Solaris is the only counter-example to Linux - not FreeBSD.

javert3• 17 years ago

I agree with what sandGorgon said.

I'm not defending the Linux kernel in comparison to other FOSS projects, especially really excellent ones like FreeBSD and possibly Solaris. Linux and FreeBSD have different comparative advantages over each other. My point is that both are way ahead of your typical Java shop, and way ahead of Windows, and that FOSS and Linux do not deserve the beating given to them by the author of the blog post.

AFAIK CFS is considered >= ULE2 by most people, and I'm not aware of significant regressions introduced by CFS, so the high degree of conservatism you pointed out in the FreeBSD project wouldn't have been warranted in this case. I don't have the background to argue the finer points of this issue, but it's off topic for this thread, anyway.

Also, you quote me talking about Git, and then say, "I'm not sure what innovations you're thinking of here." I'm thinking about git and the large-scale distributed development model that was pioneered by the Linux community.

JulianMorrison• 17 years ago

It's true that the FreeBSD process is far more likely to result in an ideal design, but Linux's approach of "build it fast, replace it fast" might average as a win in terms of total number of tolerably performing features per development time, and volume of the search space explored.

mark_l_watson• 17 years ago

I enjoyed reading the article but don't really agree with quite a bit of it. I do agree that single individuals or very small teams most often make great breakthroughs, and that languages like Java do yield more "fungibilty."

I don't really agree with the Lisp-centric discussion because I have worked with many fine programmers who used several different languages. The right person makes a bigger impact than language selection.

As far as productivity goes, building a web portal in rails is far faster than using Lisp, and in my opinion will yield more maintainable code.

That said, I am using Common Lisp a lot right now because I am working on a book about AllegroGraph. (BTW, one good "home" for Lisp is back end services.)

lunchbox• 17 years ago

Lisp is the Ron Paul of programming languages.

plinkplonk6• 17 years ago

I am surprised this piece of trashy fanboi writing is so high on the HN front page. The whole site is full of "lisp is the coolest. Everything else sucks" type borderline trollish posts, endlessly recycling the "lisp is better than everything else" meme. As usual, the HN discussion is better than the crummy article, which may be the desired end result aftr all.

luckydude3• 17 years ago

As a hiring manager, who has managed and fired lisp people, I sort of see both sides of this.

If the problem domain fits lisp, yup, no argument, lisp rocks. I used the Xerox lisp machines and it was really cool in the sense that there was no user land or kernel, it was all one thing. Cool for debugging, cool for performance, very cool when it is a fit.

On the other hand, many problem domains are not a good lisp fit. In fact, most fall into that category, in my not so humble opinion.

As a manager, I look at the technology used to produce a product, I think about development costs (a little) and support costs (a lot). One hot shot lisp hacker that can do whatever is cool, but that's in the "little" category. The "lot" is about support and the hotshots aren't that excited about support, it's boring.

There are other threads all over the intertubes about reading and supporting code and the funny thing is that very few, if any, talk about lisp. All the lisp ones are about "how I can do this cool shit really fast and it makes me feel oh so powerful". Yeah, whatever. I'm happy you are happy. But what I, as a manager, need is the ability to take your cool shit and ship it and debug it and support it for the next decade or two and do all that when you, the studly lisp hacker, have moved on.

Not really liking the prospect of doing that with a big steaming pile of lisp.

Where Lisp Fails: at Turning People into Fungible Cogs

Related Stories