Rethinking the scientific method

http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer

by neilc • 16 years ago

59 34 16 years ago

powera• 16 years ago

Considering that most of these studies are by people with an interest in getting a certain result, it's likely experimental errors/not publishing some results will bias the published numbers in a certain direction.

The scientific method has never been proof against people that lie by omission.

gruseom2• 16 years ago

He's writing about results that were validated through replication. That is the scientific method. If what he's saying is accurate, the problem isn't a few bad apples (that's the proverbial reflexive defense of the status quo anyway).

But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain.

I have no idea if it's accurate, but Lehrer's a credible journalist, less prone to the sensational than most. If he's writing about this, it's probably because serious scientists are concerned about it.

(On another note, when I pasted the above, the following text showed up appended to my selection: Read more http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_... How can such a venerable publication as the New Yorker resort to such tackiness?)

burgerbrain• 16 years ago

He mentions, among other things,studies in ESP as falling prey to this mysterious "well it worked the first time" syndrome.

Now maybe the other examples are more credible, but mixing bullshit like ESP in doesn't help your case at all.

St-Clock• 16 years ago

"But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain."

Good! That's what science is all about. Making theories that can later be "disproved" or "confirmed". It's when you cannot disprove something that it is no longer science.

bugsy3• 16 years ago

The replication they are talking about in these particular pharma studies are of the nature that a pharma corp orders up twelve identical 8 person studies. Three studies come back showing their new pill helped a tiny bit, six show no the pill did nothing, and three show the patients got worse. They then cherry pick the three that showed it helped, and possibly toss in one of the ones that showed the pill did nothing just to cover up what they are doing. They then publish a paper showing that 3 out of 4 studies validated that the pills work. The other 8 studies are set on fire and never mentioned. And there is your multiple studies.

This is not some crazed conspiracy theory either, this (doing multiple very small sample size studies rather than one slightly larger study and discarding and never mentioning some of the studies with the least favorable results) is actually how it is now known to be done, and this methodology is the reason why some journals have started to say that all of the studies have to be registered in advance if they want to publish their results so that the companies can't selectively discard results like they have been doing.

brudgers• 16 years ago

>"He's writing about results that were validated through replication. That is the scientific method."

Experiments confirming phlogiston were validated through replication. The theory of phlogiston was however, incorrect. That's the problem with the scientific method.

vacri• 16 years ago

A rather sensationalist headline given that the scientific method already recognises the problems of human researchers and not only offers some mechanisms to counter those problems, but is eager for more.

rickmode• 16 years ago

I agree. The article's headlines has a distinct anti-intellectual flair, and yet the article itself points out human flaws in research rather than a fundamental flaw in the scientific method.

swombat2• 16 years ago

This is not so much a problem with the scientific method as a problem with self-interested applications of statistics. F still equals m*a hundreds of years later, and that's not about to change.

gruseom2• 16 years ago

Yet I wonder if "the" scientific method has in fact been overapplied way past its sweet spot of mechanics and so on. It's by no means obvious that it works as well in, say, medicine, let alone psychology, let alone sociology, let alone economics. The fact that it's the cult of our age perhaps blinds us from asking the interesting questions about its limits.

swombat2• 16 years ago

Oh yes, there are definitely some problems with over-zealous applications of the scientific method (or rather, over-zealous discarding of other methods of reasoning) - but this article doesn't really cover them. Not everything of value can be measured in a double-blind experiment.

rmc• 16 years ago

What's "the scientific method" you speak of? There main idea behind scientific reasoning is "Look for evidence to test things, don't be afraid to challenge the big boys"

JoachimSchipper3• 16 years ago

Why?

Medical science seems, sad to say, incompetent (http://care.diabetesjournals.org/content/17/2/152.abstract rediscovers integration, and statistical illiteracy among doctors has been repeatedly reported) and corrupt (conferences in lavish locations, vendor-sponsored efficacy research).

I'm sure there are good researchers, don't get me wrong, but I'm also not surprised that some really questionable stuff gets reported.

I'm sadly, even less aware of the state of research in sociology, but I'd like to point out that non-scientifically-tested psychological theories tend to be complete nonsense (cf. lots of Freud's work). With respect to economics: the field has many problems, but a book like Freakonomics is a very readable argument in favour of the use of statistics in (micro-)economical science and related fields. (Do note that the scientist-author, Levitt, actually understands statistics. This is, I suspect, important.)

That said, there are issues with the scientific method, like the fact that it tells us to drop falsified hypotheses but does not, in itself, teach us how to find interesting results. That's not germane to your observation, though.

Raphael• 16 years ago

Or does it? http://en.wikipedia.org/wiki/Special_relativity#Force

defen• 16 years ago

Is there any non-paywall way to view the article? This looks like it might be interesting but I can only see the first couple paragraphs. The title makes me think of three things: Paul Feyerabend, Bayesian inference, and the increasing effectiveness of placebos. I'd be curious to see if any of them are in the article.

michael_dorfman4• 16 years ago

I don't know of any non-paywall way to view the article; I'm a (satisfied) New Yorker subscriber, so I've already paid.

It's a great article, by the way.

The answer to your latter (implied) question is "no"-- the article does not explicitly mention any of those three things.

khafra3• 16 years ago

http://news.ycombinator.com/item?id=1978638 was posted after you commented.

bugsy3• 16 years ago

The scientific method is fine. Note that this is about certain psychiatric pharmaceuticals being found not to have the efficacy that was earlier claimed, in studies done by pharmaceutical companies who doctored results and played statistics games and selectively hid results that didn't support profitability.

It's fascinating to see the level of hubris in the corporate pharmaceutical industry, that after cornered and confronted with the fact that for decades their self-serving fabricated so called research is fraudulent, they would now have a conference to announce that the scientific method has failed.

michael_dorfman4• 16 years ago

Did you read the article? It is not primarily about the pharmaceutical industry, or about doctoring results, although it touches on both of those topics. The argument is far more subtle, and more far-reaching.

drndown20074• 16 years ago

Agreed. The one that has me wondering how far off we might be is the idea that radioactive decay might not be constant:

http://io9.com/5619954/the-sun-is-changing-the-rate-of-radio...

crayz2• 16 years ago

Here's a PDF of the full article: http://crayz.org/science.pdf

JoachimSchipper3• 16 years ago

It's interesting and all, but I think the New Yorker has a right to restrict access to paying subscribers.

EDIT: well, non-paying subscribers apparently. That makes me feel better.

crayz2• 16 years ago

I downloaded it by signing up for a free account, and it's littered with their ads

Rethinking the scientific method

Related Stories