Hacker News as a case study to test the wisdom of the crowd theory

https://venngage.com/blog/why-you-need-to-stop-obsessing-over-comments-on-hacker-news/

by snake_case • 11 years ago

140 112 11 years ago

venngage.com

IIAOPSW• 11 years ago

I think this article will do poorly.

Maken• 11 years ago

I think this comment will do well.

mhurron• 11 years ago

I think you shouldn't care so much about it.

FreakyT• 11 years ago

I'm glad to see that this article mentioned my all-time favorite HN naysayer thread -- the one for Dropbox. Namely because it contains this gem:

For a Linux user, you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem. From Windows or Mac, this FTP account could be accessed through built-in software.

Yes, completely trivial. Every user will definitely have no trouble doing that. (Also, accessing your files with an FTP client? Really?)

thomasahle2• 11 years ago

then using SVN or CVS on the mounted filesystem.

I didn't know Dropbox had versioning control?

pricechild• 11 years ago

https://www.dropbox.com/help/11

Robadob3• 11 years ago

I actually went to try and revert a file in Google Drive this morning and was shocked to find it didn't have this option after using Dropbox for years.

mentos• 11 years ago

haha that is a perfect example

I think articles like op's are good in conditioning the HN crowd to be more aware of their own biases.

Update your model of the world and continue on.

josephpmay• 11 years ago

Fun story:

Someone I know got really stoned one night, FTP'd into a shared Dropbox folder, accidentally uploaded a file, tried to delete that file, and ended up deleting the entire contents of the Dropbox on everyone's computer at the company. We were never able to recover the files, but at least there was nothing too important.

Vendan2• 11 years ago

Utter crap on mobile. There's a side "share" panel that covers the left side of the article with no apparent way to remove it.

noarchy• 11 years ago

Not sure if this is meant to be a sarcastic reply that imitates the usual negativity of HN, due to the subject of the article, or a legit reply.

Vendan2• 11 years ago

Quite legit really. I was unable to read the article on mobile, despite the fact that it had a "responsive design", because of a sidebar covering the left side of the article. I hate how the new standard for "mobile friendly" is to start permanently using portions of the screen for pointless shit.

copperx• 11 years ago

Brace yourselves for the onslaught of negative comments. I can't wait for the statistical analysis to be trashed.

oneJob• 11 years ago

Even if statistical analysis checks out, the premise seems to be all wrong. Speaking generally now. I rarely read HN comments as having predictive intentions, but rather constructive (or not) criticism on things as they stand. Listening to criticism (i.e. negative feedback) about your product can be the difference between an amazing product and a mediocre products with a slow, drawn out fading into mediocre land.

pjc50• 11 years ago

Well, they started out by picking successes. A more valid approach would be to list all startup ideas mentioned on HN and see how they turned out. That includes every acquishutdown mentioned.

But that's vulnerable in the other direction. If 90% of startups fail, it's very easy to predict failure.

An interesting non-statistical inquiry would be to look at the problems people are complaining about and seeing whether they're still present in the successful projects (e.g. dropbox privacy).

bachmeier• 11 years ago

Homejoy is in the article but is no longer in business.[1] So I'm not sure if the point is that there weren't enough negative comments on their launch.

[1] http://blog.homejoy.com/homejoy-says-goodbye/

allencoin• 11 years ago

Certain companies have succeeded despite a sceptical specialized audience. Why is that? The first and most obvious reason is because there is simply not enough diversity in the audience for the wisdom of the crowd theory to hold true.

Right, this almost seems like common sense. I see the Hacker News community as mostly entrepreneurially minded devs who are very invested in the Silicon Valley startup ideal. Obviously there's nothing wrong with that, and there are a lot of really smart people here. But geniuses in a particular field are often inept in others. Hacker News readers don't represent the average consumer, for instance, so I wouldn't put too much stock in their opinions about a primarily average-consumer-facing product like Airbnb. I would, however, put a lot of stock in their opinion about my new dev-facing app like Heroku.

pkorzeniewski• 11 years ago

What irritates me most with HN crowd is that there're several "sacred" topics that you either agree with, or you better shut up because you're clearly wrong and you'll be downvoted to hell. I understand that the voting system can represent your agreement or disagreement with someone, but downvoting valuable opinions just because they're controversial is immature.

danso3• 11 years ago

Like?

CPLX4• 11 years ago

My favorite one is self-driving cars. If you say something factual and obvious, like for example that self-driving cars don't in fact exist yet and zero people are able to use one today, legions of flying monkeys will swoop in to call you a backward thinking luddite.

There are others.

VLM3• 11 years ago

Electric cars are useless for everyone because Real Americans(tm) require (X time 1.25) miles of range where X is whatever is commercially available now and has varied from 25 (lead acid conversions) to 400 or something over the past decade(s). With a side dish of all vehicles must be suitable for all people, despite the differences between mining trucks and pickup trucks and sports cars and commuter cars being order of magnitude greater than the difference between a gas powertrain and an electric powertrain.

Our current political, social, and economic hierarchy is inherently by definition ideal and permanently unchanging and all disagreement is thoughtcrime to be voted down. Everything we've been indoctrinated to believe is right, because might makes right.

In a virtual world, nothing matters more than geography, specifically where you live, and that's not a bug but a feature. With a side dish of urban bicycle riding apartments are the only politically acceptable to discuss solution for humanity. Seriously HN may as well be an urban bicycle blog some days.

There's something inherent about the nature of writing code that means programmers should not only tolerate and expect extremely low corporate social status (think of working conditions and hours and position in the hierarchy and respect (or lack thereof) for ideas), but should embrace the low status and attack outsiders who disagree.

A secular prosperity gospel along the lines of who cares how many become unemployed, let them eat cake, I don't care about historical analogies and guillotines. Sure I was born on third base and think I hit a home run, but they can pull themselves up by their bootstraps, if they really wanted to.

Although nothing is more important than internalizing and understanding technological scaling problems, all macro level problems in economics, business, finance, culture, or international relations merely requires replicated and "turned up to 11" micro level solutions. You know, just like when bubble sort is too slow , the best solution is more faster processors, right?

(added another sacred topic: Much as young adults always believe their generation invented sex, drugs, and rock and roll, they are the inventors of the concept of combining arts and science to earn a buck, nobody has ever earned a buck by putting an artistic face on a boring engineering project)

(and a second added sacred topic: New means better, and new means less bugs than old. New is good inherently because its new. Also technology in IT isn't an endless rotational circular wheel of the same old ideas over and over again with the same old problems over and over again, its strict linear progress like an infinite highway to the promised land of suburban paradise or something)

(and a third sacred topic: voting should indicate how well the voter agrees or disagrees with the opinions expressed in a comment, not how well its written or how interesting it is, voting should solely be a popularity contest.)

zamalek2• 11 years ago

I've found that HN demonstrates this problem far less than other sites (which have a similar format). That's not to say that this isn't a problem on HN, it just seems to be the best of the breed.

tajen• 11 years ago

I used to think feminism was very a sensitive topic, thus untouchable like you describe. It appears not. Most times I've posted skeptical comments against the current form of feminism, I expected to be told off and maybe excluded from HN. I have seen a workmate being fired for such a discussion. In reality, my upvote balance was between 4 and 35 points. It doesn't mean I'm the one who's right, it means it's possible to gain upvotes by echoing part of the people's opinions.

That said, I still feel I'm earning a Godwin point for evoking this topic.

drzaiusapelord2• 11 years ago

Being critical of feminism on a forum full of young single guys isn't the big contrarian display you seem to think it is. HN and reddit are the only places I see this hyper critical attitude and its generally well accepted. Hell, reddit is famous for its popular "red pill" forum.

sageikosa• 11 years ago

Bad tone, non-sequitors or obvious axe-grinding will get you slammed down on HN more than anything else.

bananaoomarang• 11 years ago

The relentless cynicism of the Hackernews community is a large part of the fun. There is usually a lot more thought put into the comments here than other cough Reddit cough places on the ol' internets. A lot of the comments which are criticised here for being later proven wrong are valid concerns.

The degree of 'magic' in Meteor, for instance, is something still discussed a lot, despite it's obvious benefits and success.

mattmanser• 11 years ago

Call me cynical by your example is bad, from experience a lot of us know that excessive magic like meteor is a recipe for great pain later on.

(Boom, boom)

oldmanjay3• 11 years ago

That's pretty definitively cynicism. I don't get the objection.

toxican• 11 years ago

There is usually a lot more thought put into the comments here than other cough Reddit cough places on the ol' internets

I find this place kind of intimidating and as a result I find myself putting a lot more time/effort into my comments than I do elsewhere. I think that's a good thing.

galfarragem• 11 years ago

I remember people even being pissed because Meteor raised more money than other projects. Doesn't matter that their marketing strategy and roadmap are near to perfect allowing them to maximize their market share while other (possible) better projects can't. A typical case of confirmation bias.

dublinben• 11 years ago

In contrast, I find the unrestrained positivity and uncritical acceptance of new 'products' on a site like ProductHunt quite bizarre.

speg• 11 years ago

Is it telling that I come to the comments first on a story about comments?

mccracken• 11 years ago

I even did it on this thread, then realised the irony immediately

carlosrg• 11 years ago

But Hacker News was wrong about Airbnb

Well, about these concerns about security, they weren't totally wrong.[1]

[1] http://www.thelocal.es/20150821/airbnb-to-beef-up-security-a...

Rambunctious• 11 years ago

Is there any data file to show what comment has been marked as positive or negative?

For example, if you look at the Quora thread ( https://news.ycombinator.com/item?id=1197146 ), a comment could say something good about the site and also provide some feedback (e.g. https://news.ycombinator.com/item?id=1197230).

Is this comment marked positive or negative? If feedback is somehow clubbed with negative comments, I would doubt the accuracy of the analysis.

josefresco• 11 years ago

I had this same thought - as many times I comment and then realize that it could be "taken" several ways.

It also amuses me to no end that someone is collecting and analyzing data from HN based (in part) on my sometimes idiotic comments and silly debates with Internet strangers.

danso3• 11 years ago

Not sure if I agree with the conclusion:

Such a specialized community is a great place to get feedback on the technical details of your product, but not to validate your ideas. The responses from people in these communities tend towards scepticism to negativity, even for products they use.

Or with the methodology. Take for example the Dropbox thread [1] which the OP's sentiment algorithm classifies as having a "negative top comment".

This is what the purportedly negative comment says:

The only problem is that you have to install something. See, it's not the same as USB drive. Most corporate laptops are locked and you can't install anything on them. That's gonna be the problem. Also, another point where your USB comparison fails is that USB works in places where you don't have internet access.
My suggestion is to drop the "Throw away your USB drive" tag line and use something else... it will just muddy your vision.
Kudos for launching it!!! Launching/shipping is extremely hard and you pulled it off! Super!

In what universe, besides the one of a sentiment-analysis-algorithm, would that be considered a "negative" comment? Moreover, even if that last, congratulatory line didn't exist, the rest of that comment is very well-worth listening to. It might have been wrong (has Dropbox penetrated corporate IT as well as USB keys? USB keys have definitely taken a beating in reputation and in convenience...but has Dropbox been an easy transition in corporate IT? At Stanford, we have Box)...but it definitely wasn't non-constructive criticism.

If a submission gets a lot of upvotes, to me that's a positive-enough sign of validation. Is it really helpful to the submitter to see a dozen/hundred comments that are merely, "Awesome! I like it!"? I often like reading the comments before I check out the submission, because I don't want to have to parse Press-Releasese to understand what the product does, or who it may compete against, or what its flaws might be...Even if the comments were completely devoid of constructive and insightful criticism -- I'm sure after a big launch, it's helpful, yet annoying when people immediately nitpick grammar and typos -- if you're a founder of a great product, the hemming and hawing of HN is probably the least of your obstacles on the way to success.

[1] https://news.ycombinator.com/item?id=8863

zamalek2• 11 years ago

In what universe, besides the one of a sentiment-analysis-algorithm, would that be considered a "negative" comment?

The algorithm likely has issues identifying if criticism is constructive. If you look at the negative/positive score of each word in isolation (e.g. fail, problem, drop, can't, hard) the comment is overwhelmingly negative.

You actually have to parse and understand the comment to prove that it's very positive. They could have ditched the algorithm, but then they would lose a quantifiable positivity score and instead would have had to rely on human intuition - which has it's problems, e.g. spend the whole day looking at negative comments and neutral comments might start looking positive.

The study may have had fundamental issues at the outset.

thaumasiotes• 11 years ago

instead would have had to rely on human intuition - which has it's problems, e.g. spend the whole day looking at negative comments and neutral comments might start looking positive.

If you assume the people looking to validate their ideas are familiar with the HN community, then this effect makes the analysis more accurate.

mattlutze2• 11 years ago

Well negativity in:

I have a few qualms with this app:
1. For a Linux user, you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem. From Windows or Mac, this FTP account could be accessed through built-in software.
2. It doesn't actually replace a USB drive. Most people I know e-mail files to themselves or host them somewhere online to be able to perform presentations, but they still carry a USB drive in case there are connectivity problems. This does not solve the connectivity issue.
3. It does not seem very "viral" or income-generating. I know this is premature at this point, but without charging users for the service, is it reasonable to expect to make money off of this?

or skepticism in:

It's pretty nice, and I was thinking to myself - hey cool, I could make an online backup of my code. Then it occured to me - who the hell is this guy, and why should I trust my code to be on his server!?
That's a huge issue you should consider. Why would people feel comfortable leaving their valuable stuff on "Drews" server?

Or this[1] conversation. Or this[2] one.

That comments page is, I agree, not a troll-haven or something of the sort. But we do have a tendency to try and pick apart every potential flaw we see in something.

It's not a bad thing! Thorough vetting of an idea is the only good way to find potentially fatal flaws. But if you're looking for what the response of the enthusiast Internet community is going to be, this is definitely not the place to go.

A trial by fire we are.

[1] https://news.ycombinator.com/item?id=8958

[2] https://news.ycombinator.com/item?id=9671

kbenson2• 11 years ago

But we do have a tendency to try and pick apart every potential flaw we see in something.

If as the owner/presenter you aren't looking for that by posting your program/model/business to HN, I think you've not done your research. HN is great for getting constructive criticism. Not all constructive criticism needs to be accepted or acted upon, but if you can't really justify why it doesn't apply to you, then you should think about addressing the problem presented. Otherwise, you're just looking for people to congratulate you, and there's lots of other places to get that. (for the general you, of course).

jasode4• 11 years ago

Or with the methodology. Take for example the Dropbox thread [1] which the OP's sentiment algorithm classifies as having a "negative top comment". [...] In what universe, besides the one of a sentiment-analysis-algorithm, would that be considered a "negative" comment?

To clarify, they said they manually classified the top comment instead of leaving it to the judgement of the algorithm: ", we manually categorized the top comment for each thread."

As for why their human eyes judged it as "negative", I'd speculate it was phrases such as "The only problem is", "That's gonna be the problem.", "your USB comparison fails"

So yes, the top comment's criticism is constructive, but at the same time, it can also be subjectively classified as "negative." The positive back pat at the end of the post was not about the product itself but for the accomplishment of launching something. (You could say it's the attaboy "trophy for participating" consolation prize to soften the previous paragraph's criticism.)

mreiland• 11 years ago

that sentence is technically negative, but not conceptually. constructive criticism is a thing.

rebootthesystem• 11 years ago

This, in a strange way, echo's some of what I said on the thread about Amazon's product review star system: In a crowd the people with the most motivation to participate are those with negative views. This is a problem with voting/rating/forum systems going back decades, HN is no exception.

jmnicolas• 11 years ago

"In a crowd the people with the most motivation to participate are those with negative views."

I'd had the ones that are paid to say good things about a product they probably never used. If we're lucky they compensate each other ;-)

jasode4• 11 years ago

Interesting article. I've thought a lot about the slashdot/HN mispredictions of future successes and my random thoughts:

1) A flawed Theory of Mind applied to the general public: I think it's fairly safe to say that the technical crowd (which includes commentators of slashdot/HN) will lean towards the Asperger/autism spectrum. It's easy for geeks/nerds/experts (especially vocal ones) to misjudge how products could be accepted by the masses (who are not geeks/nerds/experts). The famous examples being the slashdot dismissal of the iPod in 2001 and the iPad in 2009. (And Sara McGuire's article that this thread is about has more examples from HN.)

2) Skepticism/negativity is easier to itemize and write, and as a strange bonus, it is perceived as smarter and more insightful analysis. Looking at all the things that are "wrong" with a product might be described as a type of "closed-ended" thinking. On the other hand, imagining the different ways a product could succeed involves more "open-ended" analysis. The problem of writing open-ended thoughts is that it looks like naive futurism, or unsophisticated cheerleading. (E.g. haha, Back to the Future's Bob Gale thought we'd have hoverboards in 2015! Dean Kamen thought Segways would sell millions of units and would revolutionize the world! etc).

Because of the combinations caused by faulty Theory of Mind and biases towards the shortcomings of products, the commentary on new business ideas will almost always end up being negative.

scrollaway• 11 years ago

It's easy for geeks/nerds/experts (especially vocal ones) to misjudge how products could be accepted by the masses (who are not geeks/nerds/experts).

That's a pretty bad conclusion. Is it any easier for non-"geek/nerds/experts" to judge the iphone on its mass market appeal? I don't think so. In fact that's pretty much what you figured in the BTTF "predictions" failing.

The thing is a lot of people here on HN, and in the tech community, like to judge products based on how easy or hard they are to build, and that has zero relation to how good the product is or how successful it will be. For a non-techie, it's in the realm of the impossible either way, so that stuff doesn't matter at all.

jasode4• 11 years ago

The thing is a lot of people here on HN, and in the tech community, like to judge products based on how easy or hard they are to build, and that has zero relation to how good the product is or how successful it will be.

This is also true but that wasn't my point.

I'm saying that the armchair quarterbacking from nerds is often predicting the business success (the mass acceptance) of a product. There is a huge disconnect between how expert tech communities think and how everyone else thinks. Steve Jobs was one of the few that seemed to bridge this disconnect. But Mike Lazaridis of RIM Blackberry could not. Jobs was more consumer-minded rather than technical. Lazaridis was more engineer-minded rather than consumer (e.g. consumers want a physical keyboard more than a touch screen.)

VLM3• 11 years ago

"to judge the iphone on its mass market appeal"

Note that you can make a lot of money out of a product that flops in the mass market like an iphone. Out of any group of ten Americans, roughly ten drink water, use indoor plumbing, and have access to utility generated electricity, all rather successful products. Its also true that nine out of ten Americans don't have or use an iphone, however, its possible to make an enormous stack of money out of stuff 90% of the population is uninterested in.

For other examples of the same phenomena of making fat stacks of cash off practically no one, see pop music, television, professional sports, AAA video games...

Its a failure to understand the market. Around 9 out of 10 comments about an iphone should be somewhere on the spectrum of "eh" to "that sucks" because a random sampling should indicate iphones are only about twice as cool as Congress. People on /. complained about the ipod because they're normies, not because they're nerds. Normies don't like that stuff, look at the actual sales figures, therefore normie hangout like /. is not going to like the product.

Denigration and name calling of people who don't like a niche product is just Apple fanboyism.

hluska• 11 years ago

In regards to point #1, this is my favourite article about engineers and the autism spectrum. Between a (perhaps) higher incidence of autism spectrum disorders and a focus on systemizing thought, this article would argue that you're onto something:

http://spectrum.ieee.org/biomedical/diagnostics/engineers-an...

It makes me wonder if daycares tuned specifically to the needs of children with two engineers for parents. Maybe a slightly earlier intervention would prevent some of the expected kindergarten issues.

mfoy_2• 11 years ago

A lot of the "negativity" is more like blunt constructive criticism. I think a founder could post a "Show HN", get blasted with "negativity" in the comments, and then address any issues he thought had merit, and end up with a better product.

Also, purely "positive" comments such as "Cool idea!" don't add anything to the conversation.

krapp• 11 years ago

I would be more upset if I posted a Show HN that was ignored than one that met with a lot of negativity. At least on HN, you can expect some of the criticism to come from a competent technical and business background, so a thread that gets any attention at all is bound to have some constructive value.

That said, though, having your ideas validated or condemned on HN shouldn't mean much in the grand scheme.

drzaiusapelord2• 11 years ago

The famous examples being the slashdot dismissal of the iPod

Or more recently, Dropbox, which was Ycombinator financed. Someone linked to an old thread about them and it was largely dismissed as "just FTP" and destined to fail. The famous HN quote is "For a Linux user, you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem. From Windows or Mac, this FTP account could be accessed through built-in software." Yes, trivial for just about anyone, right?

I'd also add a 3rd bullet point about these forums being echo-chambers. I know I personally rarely write anything against the libertarian loving majority nor do I ever criticize any FOSS projects because in the past when I have, its been downvote central. This has a chilling effect on speech and forums like these just become extremists cheering each other on over time.

will lean towards the Asperger/autism spectrum.

I like to think that techies come in all sorts of neurotypicalities, but the ones obsessed with fighting on the internet over politics/tech news are the ones well into weirdo territory. There are lots of techies with moderate views and with great horse-sense. They're just not spending all day on HN and Reddit "correcting" everyone like a real life Sheldon. The non-weirdos seem to be the ones who get shit done and are out there being successful or spending too much time creating to be bothered by online forum debates. The weirdos just have higher visibility.

TheOtherHobbes• 11 years ago

Also, success is not defined by superficial customer appeal, and certainly not by technical details.

Success is defined by a combination of customer appeal, investor persuasion, and ability to execute effective sales and marketing.

An initial Show HN typically includes details about the product and maybe the underlying technology, but doesn't usually tell you much about the other elements.

So I think it's unfair to judge HN on its prescience, when HN is as much about technical critiques as business potential.

It could be interesting would be to look at VC judgements of startups. VCs have a pitch deck, they have some hints about the financials and of customer growth, and they may know something about the reputations of the management team.

With the extra detail, you'd expect better prescience. I have no idea if that's what happens, but I think it would be very interesting to test the hypothesis that VCs make expert decisions.

sillysaurus3• 11 years ago

I think it's fairly safe to say that the technical crowd (which includes commentators of slashdot/HN) will lean towards the Asperger/autism spectrum. It's easy for geeks/nerds/experts (especially vocal ones) to misjudge how products could be accepted by the masses (who are not geeks/nerds/experts).

What you're describing is a startup investor, not someone with autism. Even pg thought Facebook was lame. I thought the iPad was silly.

Predicting the future is hard.

avn2109• 11 years ago

> "...I thought the iPad was silly."

In fairness, the Ipad is silly. The general population is slowly figuring that out. [0]

[0] http://fortune.com/2015/07/16/apple-ipad-watch-analysts/

Retra• 11 years ago

It's easy for geeks/nerds/experts (especially vocal ones) to misjudge how products could be accepted by the masses (who are not geeks/nerds/experts).

I'd bet that a vast majority of users commenting on HN are not trying to predict what other people think, but whether or not it is worthwhile to invest their own efforts into the subject.

I can sit around saying "smartphones are crap" all day long because I don't have a need for them, but that doesn't mean I believe they'll fail because of it. It means I'm personally not going to involve myself in the smartphone business.

jasode4• 11 years ago

I'd bet that a vast majority of users commenting on HN are not trying to predict what other people think, but whether or not it is worthwhile to invest their own efforts into the subject.

If you notice, I wasn't claiming that 51%+ of posters are predicting what others think. I was characterizing a common pattern of commentary from a vocal (and possibly minority) HN crowd.

As an example, the following is 7 quotes from 7 different people pulled from the top 9 parent posts of the HN Docker thread. After reading each one, ask yourself if the poster was talking about himself, or was he talking about others?

- "Most corporate laptops are locked and you can't install anything on them."

- "It does not seem very "viral" or income-generating."

- "It's is problem everyone is having, and everyone knew it."

- "Why would people feel comfortable leaving their valuable stuff on "Drews" server?"

- "If you are looking for a wider audience than those who already know the context of dropbox, make a video where you lay out the case for use of dropbox using simple examples from user point of view(think a college student)"

- "How are you going to scale up your storage to meet the demands of the users? "

- "Your main competition is not USB drives: it is HotMail, GMail, and Yahoo! Mail. Once people are taught the "email it to yourself" trick, they love to use it"

mfoy_2• 11 years ago

There's an XKCD for this, actually: https://xkcd.com/1497/

vatotemking• 11 years ago

HN and reddit in a nutshell

netcan• 11 years ago

I've never read "The Wisdom of Crowds" so maybe I'm missing some stuff, but from the general statement I think it would be a terrible mistake to take it as meaning "commenters are right" or "consensus is right." It's much more subtle than that.

Crowds, in certain structures that yield a useful dynamic, produce wisdom. People transacting in a market allocate resources effectively. Gamblers and bookmakers betting on football matches make good odds. Wiki-editors nitpicking away in wiki-world produce an incredible encyclopedia. Open source, etc. There are also lot of examples of crowds being stupid, and structures that yield a stupid dynamic.

lighthawk• 11 years ago

tl;dr: HN presents a variety of opinion.

Also: misspelled neutral in one of the graphs.

Nice post, though. Keep it up.

rayiner• 11 years ago

What's the saying? They ridiculed Einstein? But they also ridiculed Bozo the Clown?

TillE• 11 years ago

Indeed. From Carl Sagan's wonderful The Demon-Haunted World:

"They laughed at Columbus, they laughed at Fulton, they laughed at the Wright brothers. But they also laughed at Bozo the Clown."

dmalvarado• 11 years ago

That "wisdom of the crowd" thing only works when the crowd isn't influencing each other. Otherwise - groupthink.

danso3• 11 years ago

Obligatory rebuttal of statistical claim: does this analysis suffer from survivor's bias? OK, so these are the few startups and projects that have had reasonable-to-great amount of success...what about all of the startups that we can't think of off the top of our heads [1], because they died? If the number of those is much greater compared to the group OP selected, and they also faced a majority-negative bunch of comments...why shouldn't we conclude that those founders should have obsessed over HN skepticism?

[1] I'll start: Fab https://news.ycombinator.com/item?id=2444387

acqq• 11 years ago

Moreover, it's easy to argue that counting the number of positive and negative comments on any subject on any "social" platform can't be treated as more than immensely biased sample of the sentiment of the users of the platform momentarily interested in commenting the specific topic.

ss6754• 11 years ago

This is an interesting article but very one sided. To really assess "the wisdom of HN community" the author should have assessed all four cells: Successful companies with positive comments, Successful companies with negative comments (which the author did), Unsuccessful companies with positive comments and Unsuccessful companies with negative comments. This would provide a broader and more objective picture of HN community

smikhanov• 11 years ago

The article says that HN failed to predict the success of AirBnb. While factually correct, this says nothing about HN (or anyone in general). Nobody was able to predict the success of AirBnb, even the savviest of investors. AirBnb wasn't the only company of its kind, it wasn't run by the smartest people out there, it wasn't launched at some particularly good time, or anything similar.

They have just survived and succeeded. Why? Who knows.

But obsessing about HN comments is truly not worth it.

codingdave• 11 years ago

Cherry picking posts about the successful companies from the glut of "Show HN" posts that have come through over the years is practically the definition of survivor bias.

Not that I am arguing with their larger point - if you believe in what you are doing, do not let comments on HN stop you from pursuing your vision.

cwilkes• 11 years ago

Exactly. While an interesting read in showing how people can be wrong about later hits, this study is meaningless unless the failures are included. It is like looking at all the winners of the superb owl, finding some negative comments about them, and then saying "see, you shouldn't listen to anyone". What about all the teams that didn't win and had good comments? Or bad?

junto• 11 years ago

I would have thought this was relatively obvious.

Many new ideas and applications are marketed on HN quite validly under Show HN.

However, the first pass at marketing an application may not reflect its form further down the line.

For example, we often see poor landing pages, or simply landing pages that don't convey the problem succinctly enough to quickly satisfy our short attention spans.

A few months or years down the line and these development teams have homed their product, ironed out the bugs and built landing pages that are easy for anyone to understand quickly.

Interesting none the less.

Hence, what is launched on HN != its final form and cannot determine future success.

dandare• 11 years ago

Isn't the sample size too small? I invoke the law of small numbers.

im3w1l• 11 years ago

A positive comment is not a prediction that something will succeed, and a negative comment is not a prediction that something will fail.

mbell• 11 years ago

Three companies, three different responses on Hacker News, and three ultimate outcomes. [speaking of Codecademy, Meteor and Homejoy]

The only one on this list with an 'ultimate outcome' is Homejoy. Codecademy has no business model that I know of, and Meteor isn't even close to profitable.

Quora is also used as a 'success' case and they also don't have a business model.

I find this post does a pretty poor job providing any sort of supporting evidence for it's claims.

Spooky23• 11 years ago

There's definitely heavy cynicism here. But, many of these points are real and are risks that these startups need to factor in.

The AirBnB thread is a great example of this. The risks associated with having strangers in your home is something that AirBnB has to market around and have protections from. (ie. the stories about crazy parties that trash apartments)

ebbv• 11 years ago

I agree with the premise of this post, but it extends beyond HN; don't listen to the internet in general. We commenters don't know as much about whatever you're doing as you do. Maybe you might get some good insight from a random commenter on the web, but usually you won't.

However, their specific example is sort of wrong. AirBnB has indeed blown up, but this has been due to two major factors:

- Traditional landlords, rental properties and yes Bed and Breakfasts moving onto Air BnB, embracing a new, popular platform.

- Extremely high real estate markets in some areas making it attractive and almost necessary to rent out room(s).

It's not like AirBnB has created a change where most or even a significant portion of Americans as a whole rent out their rooms. The reality is that the objections the commenters brought up are real, and most people aren't interested in the hassle the comes with renting out a room, and most people's rooms aren't necessarily attractive to rent.

In the end the original idea people had of how AirBnB would have to be successful; convincing most people to rent out rooms, an convincing travelers to rent rooms in random people's houses, is what was wrong. They only needed that to happen in certain markets, and for landlords in general to realize the advantages they had by renting out their properties on AirBnB.

cuchoi• 11 years ago

I agree with the general analysis but the AirBnb example is not accurate.

Aside from survivor bias, people were just showing their concern with security. And Airbnb success depends mainly on proving a feeling of security for its users (that why they have ID and facebook verification, for example).

minimaxir• 11 years ago

Relevant: if you're looking for a way to scrape all Hacker News submissions and comments, I have a GitHub repository containing Python scripts showing how to do just that: https://github.com/minimaxir/get-all-hacker-news-submissions...

With respect to the article, I've done data analysis on Hacker News myself (e.g. http://minimaxir.com/2014/10/hn-comments-about-comments/ ) and I'm a little skeptical. I was not a fan of using NTLK to classify positive/negative because the models used for training sentiment are based on reviews, not internet site comments, and when I had used NTLK, the results were skewed significantly. I used an alternate model mentioned in the linked post which IMO worked better.

The examples used in the article seem more like correlation-implying-causation and cherry-picking. There are many, many counterexamples of startups people loved on HN which died a painful death.

giarc• 11 years ago

I would love to see what they considered a "negative" comment. Did they clump negative and neutral? Something seems off with the zenefits assessment.

Read the Zenefits post (https://news.ycombinator.com/item?id=5242681). I don't see a single negative comment as implied by the second graph on the venngage.com page (looks like 7 positive, 6 negative). Unless the negative comments have been removed between venngage pulling the data and today.

eevilspock• 11 years ago

Wisdom of the crowd? The HN post that this article cites first had 14 comments. Not a crowd.

It categorizes the first comment on DropBox[1] as negative. What? That comment points out a legitimate potential hurdle to user adoption, gives some marketing advice, and then praises them for launching.

While the article's instinct may be correct, I don't see that it has shown data that supports it.

[1] https://news.ycombinator.com/item?id=8863.

teaneedz• 11 years ago

A lot of thought about measuring an audience that for the most part lives in a bubble. In view of the fact that it's often the same audience which continuously produces bug ridden UX conundrums and ad tech supported mentalities locked in a tech bubble focus, I think that HN might be the wrong place for this study.

I love HN, but one still needs to remember that the audience, although influential, is quite different then a truly diverse mainstream crowd.

gwern• 11 years ago

The HN crowd, or any expert community, don’t have any greater a chance of predicting the success of a startup than a more general crowd.

And what 'more general crowd' are you so confidently comparing to, exactly?

xixi77• 11 years ago

I don't see how analyzing comments on ex-post known successful businesses can tell us anything about anything -- it (at least) needs to be compared to how comments on average startup proposals do, to see if there is any predictive power. If, for example, successful startups had 20% positive comments vs. 10% positive on average, and the difference was significant, that would suggest that HN shows awesome crowd wisdom -- even if 80% of people here hate everything :)

Accounting for confounding variables would be nice too (for example, if field X is generally more liked by HN than field Y, equally positive feedback on a startup doing X would tell us more than that on a startup doing Y).

Still an interesting idea, I just think that people should not forget that "data science" is a subset of statistics, and any kind of data analysis is not really useful without at least some familiarity with it.

yeukhon• 11 years ago

* Most startup will fail.

* Most of the successful startups will be acquired and merged with other more successful startups or established companies.

* Most ideas are repeated and competed aggressively.

* Most of the successful startups primary winning demographic / target audience turns out to be younger crowd, especially with college students, and the young crowd tend to be better at advertising startup ideas.

* Many successful startups you can name right away simply reinvent the existing platform by applying media attraction and sharing among crowds.

* Only way win is make noise. A lot of noise. Yeah. Back to pg's top recommendation, selling your product to users.

* Unless you have billions to burn on your own, don't try to be SpaceX or Telsa. You will never be them (or the same guy who founded both entities).

My own unscientific conclusions.

hartator• 11 years ago

If you believe that 99% of a success of a startup is about execution and 1% is about the idea, sharp guesses about the ideas shouldn't matter that much to gauge one's wisdom.

tlb• 11 years ago

Here's a mechanism by which initial HN sentiment can be negatively correlated with eventual success:

- Great founders describe the MVP accurately, even self-deprecatingly: "Sleep under my kitchen table." They mention the flaws, because they want the most useful feedback.

- Less-confident founders describe the gold-plated, fully-featured potential future version of the product and don't mention the flaws.

At least on a superficial read, the great founder's description seems less promising, but they usually end up with a better product.

I certainly see this dynamic at work in pitches I've been given.

Mz• 11 years ago

The HN crowd, or any expert community, don’t have any greater a chance of predicting the success of a startup than a more general crowd.

For the most part, the purpose of posting something to HN is not to take a vote of that sort. Thus trying to view it through that lens is kind of silly.

venomsnake• 11 years ago

Survivor bias at its best. You must include all companies that have announced themselves on HN.

Hacker News as a case study to test the wisdom of the crowd theory

Related Stories