Business Advice Plagued by Survivor Bias

http://blog.asmartbear.com/blog/business-advice-plagued-by-survivor-bias.html

by yarapavan • 17 years ago

140 36 17 years ago

blog.asmartbear.com

yarapavan• 17 years ago

The article quotes Peter Norvig: When a published paper proclaims "statistically, this could only happen by chance one in twenty times," it is quite possible that similar experiments have been performed twenty times, but have not been published.

ggchappell• 17 years ago

It occurred to me that this means there is a real advantage to academia's "publish or perish" phenomenon. It means that when an experiment's results are not what was hoped, those results are still likely to get published. The experimenter would like to have his name on some ground-breaking results, of course, but, failing that, he still must have publications. So he publishes anyway. Not so true in industry.

jerf• 17 years ago

Are you sure about that? Journals don't (in general) publish negative results. The result of the pressure to publish is not that negative results get published; it's that the scientist has an incentive to create positive results. Not necessarily through data falsification, but through choosing experiments very likely to generate "results", which, unfortunately, also implies that the information content of such experiments is necessarily lower than it could be if the scientist had more freedom to perform an experiment that had a higher likelihood of failure.

zasz• 17 years ago

This article makes everything pg says about the reasons behind Viaweb's success sound rather suspect, doesn't it?

jswinghammer• 17 years ago

Wasn't it always suspect?

biohacker42• 17 years ago

It depends of what you consider suspect. I always saw PG's account of the Viaweb days in the light of the .COM 1.0 bubble.

dejb• 17 years ago

Doesn't it make you less certain of the reasons anyone gives for any success?

mhartl3• 17 years ago

Pretty much every startup success story has an "and then we got spectacularly lucky" part to it. In Viaweb's case, it's more like "and then, for the nth time, we almost died, then somehow didn't", but the idea is the same. It's one of the great unmentionables in the startup world, and understandably so: succeed and you want the credit; fail and any mention of luck sounds like sour grapes.

I've heard a lot of startup founders talk about a lot of startups, and only Marc Andreessen (of Netscape fame) both realized and admitted to the large role luck nearly always plays.

mhartl3• 17 years ago

I should add that great startup success stories almost always involve founders who were awesome as well. (Netscape and Viaweb both qualify on this count.) It's just that being awesome isn't enough. Traf-O-Data had awesome founders, but it failed; those same founders, Bill Gates and Paul Allen, had better luck in their second venture.

10ren2• 17 years ago

At least he failed the first time (websites for galleries), and learnt from it. There was also a control in the form of other similar businesses at the time.

calcnerd256• 17 years ago

Teleology is funny.

smartbear5• 17 years ago

Did I cause you to say that? :-P

mediaman2• 17 years ago

Jason, this was a great piece and more should do themselves a service by reading it.

It's one reason to pursue 'smaller' opportunities with lower variance of return expectations but higher median returns -- for example many niche businesses -- than industries with gigantic return expectations. The latter attracts far more entrants, and while somebody gets rich, the median player does not. But this isn't clear to new entrants because of survivorship bias.

Somewhat related to this is the data availability bias. Big opportunities tend to have much better data sets, and so are easier to study and receive more attention. Many niche industries are closely private and are smaller, so do not have much publicly available data, making study difficult and attracting less attention. Therefore like moths to light bigger industries receive more study and more entrants even though their median expected economics are worse.

mhartl3• 17 years ago

This neatly encapsulates much of my current business strategy.

10ren2• 17 years ago

Warren Buffett talks about this in stock investing: you're more likely to find a bargain in the under-researched companies, because there's greater variance between the stock price and the stock value (price inefficiencies).

He's publicly claimed to be able to double a million dollars in a year with this approach (it won't work for his $40+ billion, because there aren't enough under-researched stocks to buy that are also bargains).

gatormax• 17 years ago

I wish it was still on the front page. Understanding bias is, in my book, key, key, key to an effective understanding (and navigation) of reality.

bigwill• 17 years ago

Very Nassim Taleb. These are great books NT's written that discuss our susceptibility to survivorship bias and other rational failings we have: Fooled by Randomness and The Black Swan.

smartbear5• 17 years ago

Yes! You're right I'm hardly the first people to talk about such things. The Black Swan is a classic although oddly enough right now on the front page of HN there's a critique of Black Swan that pokes holes in it. :-)

But yes it's true there's lots of good material on these subjects for those who want to explore further. Hopefully my piece whetted your appetite -- it's not the final word!

donaldc• 17 years ago

This is a good point. At the same time, I believe there are actual differences between businesses that affect their chance of success.

So the trick is to separate the 'accidental' features of successful businesses that don't increase their chances of success from the 'essential' features that do.

I wish the article went into how one might do this...

mediaman2• 17 years ago

One important way referenced in the article is to evaluate failures as well as successes for the likelihood of a trait: for example Jim Collins describes certain features that successful companies have, but does not evaluate the pool of losers to see if they also featured the same trait.

Another example is when evaluating various corporate traits against stock market performances, many people will choose a list of companies that exist today and then study their history to identify correlations of various traits. This omits those that no longer exist today, and it is a common problem among, for example, beginning hedge fund equity analysts. Instead, one should study the cohort from a random sample as existed at the beginning of the study, not at the end, and then study their performances from that point forward.

smartbear5• 17 years ago

Agreed with mediaman here.

Select both successes AND failures. Where they seem similar, those things may not teach us much. Where there is a pattern shown ONLY IN ONE SIDE (more or less) there might be a lesson.

I have a fear that there's isn't a lot you can do to guarantee success, but I admit this is just my feeling, not based on any facts or stats.

qu1j0t3• 17 years ago

The winners write the histories.

swombat• 17 years ago

Do you read business blogs where the author has failed three times without success?

Am I the only one who regularly reads blog posts by an entrepreneur analysing how he's failed and what he would do differently next time? They pop up on HN quite often.

BrandonWatson• 17 years ago

Are you talking about www.thefailingpoint.com? That's me, and I am happy to share my experiences. They have been in the massive up and down times. Success and failure. I find it therapeutic to write, and since my old engineering team is in Y!Combinator this summer, I got the religion about sharing what I know. They asked me for a compendium of all the stupid mistakes I had made so that they wouldn't make them. =) I joked it was too much work, but it turns out that it makes for some good essays, and I really hope to turn it into a published work. For now, I am posting all the essays online for all to read for free. Chapter 1 and 2 are almost fully posted. Please send feedback.

gommm• 17 years ago

Yes and I love reading those... They are usually much more interesting and insightful

smartbear5• 17 years ago

I admit an amount of hyperbole when I implied that you NEVER read such blogs, and of course it's good that you do!

In general I think it's fair to say that the more popular blogs and books about startups talk about successes rather than failures, so I'll defend my general point.

I did take some license in how I stated it, and thanks for pointing out that it's too harsh and that there ARE good sources of info like that.

seanstickle• 17 years ago

Talk about survivor bias: http://www.dailymail.co.uk/news/article-1165768/The-man-surv...

neilk• 17 years ago

The takeaway here should be: if the book doesn't systematically examine the differences between winners and losers, drop it and walk away. Sentence by sentence, it's making you dumber.

So forget In Search of Excellence. Try books like How To Be a Star at Work, which is based on a pretty rigorous study of employee productivity.

schammy• 17 years ago

This was a great read. I guess I have underestimated the intelligence of bears!

movix• 17 years ago

"Planes with bullets in the cockpit or fuel tanks didn't make it home"

Such a brilliant piece of lateral thinking - thanks for sharing.

ojbyrne• 17 years ago

Jumping on books marketed at airports to harried executives isn't really valid. There's plenty of actual business research done by professionals (and they've heard of "survivorship bias, along with all the other threats to validity). It's like looking at all the "For Dummies" books in your local book store's computer section and deciding that CS research is poor.

And Harvard Business School Case Studies are meant for teaching, and their function is analogous to science textbooks that leave out all the dead ends and fruitless research for pedagogical reasons.

Business Advice Plagued by Survivor Bias

Related Stories