When Science Becomes News, The Facts Can Go Up In Smoke

http://www.npr.org/blogs/13.7/2014/05/02/308926616/when-science-becomes-news-the-facts-can-go-up-in-smoke?ft=1&f=114424647

by jboynyc • 12 years ago

61 36 12 years ago

npr.org

mnw21cam• 12 years ago

Relevant XKCD comic, as required in these situations:

http://xkcd.com/552/

daveslash2• 12 years ago

I use this to try to show people what I mean by "correlation is not causation"

http://twentytwowords.com/funny-graphs-show-correlation-betw...

baddox• 12 years ago

Or directly from the source: http://www.tylervigen.com/

daveslash2• 12 years ago

Thank you.

buckbova• 12 years ago

I think the point should be, this study should not have been published at all. The sample size was way too small with results open to misinterpretation.

After a scan, I think the Boston Globe article is well written.

http://www.bostonglobe.com/lifestyle/health-wellness/2014/04...

The title is "Study finds brain changes in young marijuana users". Maybe it should read "differences" instead of "changes".

idm• 12 years ago

The sample size isn't necessarily the problem. For example: if we wanted to compare 20 cannabis users to 20 non-users, and we measured whether or not they felt "high", I bet our total sample of n=40 would be plenty to show a strong effect of pot use on "feeling high," it would be statistically significant, and it would not be a distortion of the truth at all.

This is an interaction between reporters and exploratory research. Statistical significance works best with confirmatory research, in which there is an a priori hypothesis, a set of falsifiable predictions, and a specific experimental design that manipulates just one or two variables in such a manner as to potentially falsify the predictions. However, confirmation is only half of science.

Before the confirmation stage, there should always be exploration, and that's what this study represents. "Let's put cannabis users in an MRI and see what happens!" It's impossible to draw conclusions from exploratory research in the same way we do from confirmatory research. The main problem is that there's usually no hypothesis, nothing to falsify, and therefore no statistical test exists to help sift through the results.

However, significance testing can still suggest whether an exploratory finding is deviant. If one group mean is different from another group mean, it's totally fair to report that difference. I didn't have to dig too deep into the original article to find out the authors weren't making any outrageous claims. It's merely that journalists reported the exploratory results as if they were the generated by a confirmatory, falsifiable hypothesis. In fact, the scientists were just reporting an interesting difference they observed.

Anyway, the point is not that the study should not have been published at all. The study was fine, the work is valuable, and the reporting was overzealous/misinformed.

Edit: updating to add that the scientists did have some ideas about what kinds of differences they were expecting, consistent with animal models. I might venture to say they had some clearly falsifiable hypotheses, too. The science is fine, and so are the scientists. It's the reporting that is the problem.

SoftwareMaven2• 12 years ago

It absolutely should say differences. Changes implies causality (or, if not causality, at lease differences over time, which also wasn't shown). It is every bit as likely that the brain differences are the reason somebody chose to smoke marijuana. But "Study finds brain differences in young marijuana users" doesn't have any cachet.

squozzer2• 12 years ago

As a medical study, I won't pretend to know whether it was properly designed. As policy fodder, which it is first and foremost - it might have benefited from additional control groups - using substances such as alcohol, nicotine, caffeine, etc. - which might have answered the question of marijuana's harms relative to legal substances.

devindotcom2• 12 years ago

As a reporter I've found that striking a balance between accuracy and reasonable summary can be extremely difficult, and sometimes a headline that has to be less than 45 characters or so ends up being more suggestive than it should. But I hate this kind of story as much as others do, and I'm proud to say I've rejected quite a few stories the editors wanted because of salacious write-ups, and I'm careful to use the language of the study itself or check with the researchers, almost all of whom are happy to discuss their work. We also have a phenomenally well-informed science/space editor where I work, who has been in the press since they were printing the newspaper in the basement of the building. So that helps.

It's worse to have to be the guy who points stuff like this out on Facebook, where you end up sounding like the science equivalent of a grammar nazi - but I've grown to be fine with it, since there's much less room for interpretation in the results of a limited and specific piece of research.

at-fates-hands• 12 years ago

It's quite ironic for NPR to say some news outlets twist facts and headlines to get a point across. Considering the main stream media does this on a daily basis and nobody bats an eye about it.

If you want to talk about misreporting something, you should start there, not a few articles on people casually using weed.

TaylorAlexander• 12 years ago

Actually, this article is great because it proves that most media distorts facts without venturing into controversial territory (the science paper clearly did not say what the articles said it did). This is as good an example as any on the phenomenon you mentioned, and it's a good topic we all remember and may have been convinced by. My parents and I have already had a discussion about it, where they were convinced that marijuana was much worse than believed, and I remained open to the idea that there may be merit to the study they heard about. But this is an article many people were convinced by because it fed into existing bias well, which is exactly the kind of thing we need to expose.

We all know for example that FOX News distorts facts, but if you go into a conversation calling them out, you will lose many people who will claim you are biased. By first pointing out specific, provable stories that have been misreported by all media, you can get a person to believe that news is being poorly reported. Then, after you have convinced them of that, and that it is a problem, you can say "hmmm... now I wonder who is most guilty of perpetuating this." You'll find FOX news in there of course, but if you actually look you will find that all corporate-owned media has turned into fact-indifferent "news entertainment".

Also, it isn't irony to cover something specific instead of the broader issue. It might be ironic for Fox to cover the epidemic of misreporting in the media, but even that would actually be somewhat expected behavior given that they keep insisting they are the only fair network.

hyperion2010• 12 years ago

In cases like this I think it would be helpful if the reporter (probably with the help of the scientist) laid out all the possible explanations for this finding. In this case it would include 'people with a smaller nucleus accumbens are more likely to smoke pot.' Showing people all the possible interpretations of a scientific result is hard, even for scientists. We often miss interpretations that in retrospect seem obvious given the data. Science news needs to report about the science first and about any human elements second.

SoftwareMaven2• 12 years ago

I don't know; I bet that happens pretty well now: Scientist says, "well, foo could be because A, B, or C; we just don't know." Journalist asks, "what would happen if B?" Scientist says, "well, it's unlikely, but D could happen".

Newspaper reports, "Scientists Show D Happens Because Foo Causes B".

The real problem is there is no disincentive to ignore the truth and publish sensationalist drivel, but there are definitive incentives to do so: you get more readers, you tend to embed yourself in whatever philosophical echo chamber you are trying to reach, and, as a result, you get more money. Until there are disincentives, the problem will never go away.

devindotcom2• 12 years ago

Problem is many people read science news for the 'human element,' the big take-away that explains why the research was done — something which rarely exists, but that doesn't stop people from expecting it.

Good science articles at science-focused sites often do go into such detail, but if you read a science article at CNN or gawker, you're going to get a version tweaked for their audience, and abbreviated to boot. Multiple explanations and bouncing possible confounding variables off multiple experts isn't an option if the writer has one hour to create the article and it mustn't exceed 300 words.

jmzbond• 12 years ago

I don't think this problem can be addressed without talking about the fundamental difference in incentives between the groups.

Reporters are incentivized to get the story to sell. Especially as there are more & more freelancers, competition is becoming intense. And let's face it, by nature, we as consumers of information are drawn to the outlandish and sensational. There was an HN a few weeks ago about someone who put out fake crazy headlines and got crazy CTR on Twitter.

Scientists are incentivized to be objective in finding the truth. Scientists avoid making claims about causation until the last possible moment just so they can be sure all the variables have been controlled for and the results are not outliers.

I don't have any well-thought through answers... but thoughts?

waltherg• 12 years ago

Have you ever worked in scientific research yourself?

Scientists sell their story and their version of The Truth just as much as reporters sell their news stories. This image of the objective scientist that prevails outside of academia is mostly false.

If I remember correctly, you'll find more material to think about here (see researching what your boss wants you to, researching what funding agencies want you to, etc. etc.):

https://news.ycombinator.com/item?id=7763737

https://news.ycombinator.com/item?id=122106

SixSigma• 12 years ago

Lots of people smoke pot. They do so, presumably, because it affects their brains, and not despite that fact. It would be astonishing and inexplicable to find that getting high didn't bring about changes in the brain. But are those changes lasting? Are they permanent? We don't know and we'd like to know.

The study didn't even find that the brains had changed at all, just that they were different.

As the sample was so small, they could just have well concluded that brown hair made a difference or people who prefer broccoli to cheese are more likely to smoke pot.

tom_jones• 12 years ago

I've seen many a publication where the scientist urge restraint in interpreting results, but then the newspapers completely ignore their pleas and go instead to proclaim "Study of 10 subjects proves X could make you immortal!". It's irresponsible journalism.

Florin_Andrei2• 12 years ago

You're a reporter. Your career is helped if your articles sell a lot. Of course you have an incentive to put a bit of "kick" in the title, or even in the content of the article, if that will help your baseline.

Money-driven outcomes are not always optimal.

spikels2• 12 years ago

I think blaming "money" for so many problems is wrong. Like any writer reporters want to see their work published and read. This is what it means to be a successful reporter. And while it may eventually lead to more money given the very low compensation of the vast majority of reporters if their behavior was really driven by money they would change professions.

Florin_Andrei2• 12 years ago

Career? Status? Worrying about the kids' college fund? They're all related.

Regardless of how you want to split this particular strand of hair, the bottom line is that media's main incentives nowadays are NOT to inform and educate the public.

Vilkku• 12 years ago

StarTalk Radio (hosted by Neil DeGrasse Tyson) has covered this topic together with Miles O' Brien in the two most recent episodes. I'm a fan of the show, but I can imagine some might find it too humorous for their tastes.

http://www.startalkradio.net/show/reporting-on-science-part-...

EDIT: By "this topic" I meant science reporting, not marijuana.

gtremper• 12 years ago

I listen to StarTalk also and really enjoyed these segments. It seems a lot of these problems have been caused by the transition from "News as information" to "News as entertainment".

garenp• 12 years ago

Humor aside, it really concerns me that so much of the information in mainstream science news articles can mislead readers.

More worrying is that it's often not possible to persuade someone who is swayed in the wrong direction, because they just don't have the base level of knowledge to allow it.

On this point I almost want to say that every person who graduates from High School ought to have gone through a rigorous class in logic and another in statistics. It's all well and good to say everyone should have "critical thinking" skills, but you can't get there without some pretty solid intellectual tools.

hueving• 12 years ago

On this point I almost want to say that every person who graduates from High School ought to have gone through a rigorous class in logic and another in statistics.

Those classes are above the bar for high-school graduation (unless you have an extremely weak definition of rigorous). If you require them, it will just mean lower graduation rates.

glimcat• 12 years ago

The problem with graduation rates is that we then optimize to increase that KPI as much as possible, at the expense of median education quality (which is nominally the social good we're trying to support). You do get more people with "the credential" - which you could also get by handing them out for a buck fifty at Walgreens, if that's what you really wanted to maximize.

It's almost impossible to give politicians a hot-button campaign topic without them over-reducing the problem to a counter-productive KPI, which they will then tie funding to.

spikels2• 12 years ago

Once someone becomes convinced of something it is often very difficult to change their mind despite logic and even strong evidence. I have noticed this even in my own thinking despite being aware of it.

You become invested in an idea by your own thoughts and actions as well as statements to others. Then to change your mind you have to admit to yourself that you are wrong as well as to others who may still hold the mistaken belief. This can be unpleasant so many people just avoid it.

Keep in mind that very few things are black and white so there is always some "reasoning" that can defend the incorrect belief even if it is very unlikely. And there are often people who benefit from the incorrect beliefs (you may disagree but I think the $5 billion US vitamin/supplement industry is a good example).

While if everyone had excellent "critical thinking" tools it should help I'm skeptical this would overcome natural human biases such as pride. And perhaps it is a good thing that at least a few people disagree with any consensus no matter how strongly supported by logic and evidence.

epochwolf• 12 years ago

You become invested in an idea by your own thoughts and actions as well as statements to others.

While I've observed this is true for most people, it isn't true for everyone.

I'd rather be correct than be right. It's one of the characteristics of people with a Myers-Briggs Type Indicator of INTJ. (And perhaps a few others.) I've also heard it called "strong opinions, weakly held" before.

I'm pretty sure both those with this characteristic and those without it spent a fair amount of time being greatly annoyed with eachother.

morbius• 12 years ago

Another terrible example:

Last year, a scientist, particular type of photon capture mechanism to the physics of lightsaber plasma. What was the response?

TIME: "Scientists Finally Invent Real, Working Lightsabers"

The Guardian: "Star Wars Lightsabers Finally Invented"

There are some more examples by CBS and other newspapers, but the point still stands. Scientific journalism has become so misleading that you cannot feasibly trust it anymore. TIME is one of the absolute worst in terms of both real-world and scientific reporting; they sensationalize EVERYTHING.

Whoever writes these abhorrent posts seriously needs to be fired. Saying that the observations of photon behavior in a cloud of atomized rubidium near absolute-zero equal inventing a lightsaber is completely absurd. It's stretched even further by the idiotic TIME headline manager who also felt the need to put the words "Real" and "Working" into the title. This goes beyond yellow journalism into straight-out lie territory.

It's sad that somewhat decent sources like Al Jazeera get completely sidelined by the public in favor of these horrid news-reporting agencies that have the audacity to call themselves journalistic.

GregBuchholz• 12 years ago

More worrying is that it's often not possible to persuade someone who is swayed in the wrong direction, because they just don't have the base level of knowledge to allow it.

I think that may be oversimplifying things. Anyway, I liked chapter 5 of "Probability Theory: The Logic of Science"

http://cdn.preterhuman.net/texts/science_and_technology/Prob...

...the impatient should skip to the last paragraph on page 129 to get to the gist of how rational people can come to different conclusions when presented with the same data.

jasontsui• 12 years ago

To some of the older folks on HN - is this a new problem?

The last few years have brought on a whole different type of newsmedia hybrid (the buzzfeeds, huffpos and gawkers) organization that is driven primarily by clicks and do not hold themselves to the standards of traditional print news. While there were dubious options on paper before (Daily Posts, National Enquirers), the internet is far greater venue for propagating bullshit with clickbait headlines. Some of the newer sites I'm seeing people post on Facebook have skipped the truth part altogether, they go straight to fabricating stories. TV has gone the same direction with news-entertainment.

I'm pretty concerned. When its too hard to find signal in all the noise, I'm afraid folks will give up altogether. With Buzzfeed putting out longform articles and NYT putting up quizzes, its already hard to discern who cares about delivering real news and who will do anything for clicks.

But maybe I'm just young (25), and people have always found echo chambers, and yellow journalism is always something we've had to wade through to find the facts. What do you guys think? Has anything actually changed?

InclinedPlane• 12 years ago

It's not a new problem. If anything I'd say things are better today because good information is so much easier to access.

cafard• 12 years ago

To varying degrees this has always been the case. Publishing and distribution is just easier.

squozzer2• 12 years ago

Not really, at least on the issue of pot. What passes for news turns out to be the most lopsided propaganda war since Rome v. Carthage.

Now that pot has regained something of a legal tolerance, medical studies will once again, as if on cue, provide the last line of defense for the puritans and productivity fetishers.

gizmo686• 12 years ago

What makes this problem even worse is that articles almost never cite the original research (or provide a link). If they did this then it would be far easier to look at the original paper. Just reading the abstract of most papers is often enough to give the lay person a relativly good understanding of the scope and actual result of the research.

When Science Becomes News, The Facts Can Go Up In Smoke

Related Stories