I Fooled Millions Into Thinking Chocolate Helps Weight Loss

http://io9.com/i-fooled-millions-into-thinking-chocolate-helps-weight-1707251800

by brown-dragon • 11 years ago

205 53 11 years ago

oconnor663• 11 years ago

http://slatestarcodex.com/2015/05/30/that-chocolate-study/

For me, the takeaway from this affair is that there is no one-size-fits-all solution to make statistics impossible to hack. Getting rid of p-values is appropriate sometimes, but not other times. Demanding large sample sizes is appropriate sometimes, but not other times. Not trusting silly conclusions like “chocolate causes weight loss” works sometimes but not other times. At the end of the day, you have to actually know what you’re doing. Also, try to read more than one study.

allendoerfer• 11 years ago

Another good indicator is, that in Germany Bild and the Huffington Post reported about it. You generally want to avoid to write a story based on these two. Everybody knows, that Bild consists of sex, hate, tits and the weather forecast as the Ärzte put it in a song mocking people that get their information from that paper.

NyxWulf• 11 years ago

For me the take away is that non-trivial independent replication of results still stands as the gold standard for experimentation. P-hacking shows up clearly when the experiment is replicated. P-values in my experience are only a starting point for further experiments.

Take the cold-fusion debacle. When independent groups attempted to replicate the results they were unable to do so.

Of course the other big take away is that people really don't understand statistics at a basic level, even the "scientific reporters".

digi_owl• 11 years ago

In essence "science" reporters have made it their business to not understand statistics.

That way they can turn weak correlations into strong causations, rake in the eyeballs, and then go "oops" whenever someone points out problems.

mcguire2• 11 years ago

Replication is hard and expensive, particularly in the medical field, and does not get the researcher any real science points.

Besides, it's more fun to do a meta-analysis on a flock of sorta-vaguely-similar studies.

hootener• 11 years ago

For me the take away is that non-trivial independent replication of results still stands as the gold standard for experimentation.

Agreed. Too bad funding agencies rarely if ever give you the money to do it :(.

The takeaway here is that statistics is arguably one of the most nuanced quantitative fields out there. It's really easy to shoot yourself in the foot, particularly with p-values.

I think every statistical test has its place, but my personal favorite lambasting of p-value testing is Steiger and Fouladi's 1997 paper on non-centrality interval estimation [1].

As an aside, Steiger was my graduate statistics professor several years ago, and probably the primary reason I know this paper even exists. If you enjoy the harsh treatment of significance testing in the paper, just imagine hearing it straight from the horse's mouth during lecture :).

[1] http://www.statpower.net/Steiger%20Biblio/Steiger&Fouladi97....

reddog• 11 years ago

Also, try to read more than one study.

Even if I had the time to read several studies and expertise to understand them, I couldn't afford it. Out of curiosity I just went to the Journal of Nutrition (I know nothing about it, picked that journal randomly) and tried to download one random issue. It was $273. Going to multiple original sources isn't practical for most of us.

I doubt that the typical mainstream journalist writing a popular story on a scientific subject could justify this expense to her boss either.

leephillips2• 11 years ago

If you need regular access to the scientific literature you have or make an arrangement with an institution. You don't pay per article or issue. It is a real problem for unaffiliated or freelance folks, though.

takluyver• 11 years ago

I'd hope that news outlets covering any kind of science pay for access to at least some major journals as a matter of course. The subscription prices are probably much lower per issue than the price to buy an issue individually.

Most journals also show you abstracts for free, which are overviews of what each paper says. If you find a paper that you want to read in full, email the author - most scientists are very happy to know that people are interested in their work, so they'll give out copies to anyone who asks.

This isn't to defend the paywalls - I think open access is important. But I don't think the paywalls are a valid excuse for journalists.

hannob• 11 years ago

It is very often possible for journalists to get papers for free. In many cases it's easy to email the author and they'll happily provide you with a copy within short timeframes. Some archives provide free access to journalists, e.g. the cochrane library.

I'm all for open access, because I think everyone should have the right to see publicly funded scientific research. But from my experience as a journalist who's written a number of pieces on scientific works it often isn't a big problem. I never had to pay for a journal article to do my work.

redacted• 11 years ago

Its a serious problem even in academia. I have had to email authors of papers or go begging on reddit.com/r/scholar more than once, even when I had very expensive university subscriptions available to me.

seszett• 11 years ago

By the way, this is the documentary for which this was done:

In French: http://www.arte.tv/guide/fr/052711-000/pour-maigrir-mangez-d...

And in German: http://www.arte.tv/guide/de/052711-000/schlank-durch-schokol...

hartator• 11 years ago

Finally, the most accurate reactions come from the web itself: http://forum.bodybuilding.com/showthread.php?t=166912491

study invalid. no control for calories at all.
Why are calories not counted on any of the individuals?
Free reign is not a way to go about doing a study. They might very well have, we don't know. We all know cocoa has weight loss properties (Theobromine), but to do such a poorly constructed study was pointless.

Jedd• 11 years ago

'everyone'?

I don't recall being fooled into thinking this.

Slightly less flippantly - the assertion, conclusion, and title are wrong.

jessaustin3• 11 years ago

It's difficult to understand why anyone would ever pay any attention to nutrition "science". Everything is wrong. TFA complimented the meatheads on some bodybuilding site for their skepticism. People who pay very close attention to their own habits and results are probably going to be more right about their own nutrition than all the scientists in the world put together.

vezzy-fnord• 11 years ago

I upvoted you because even though you're too far in outright rejecting empiricism about nutrition, the state of nutrition science is indeed quite poor and shrouded in mythology.

jessaustin3• 11 years ago

Of course one can say basic things like "average daily calories should be between 800 and 3000", but even a range that covers 95% of the population isn't right for everyone, and isn't precise enough to really help anyone. Individuals differ not only in terms of lifestyle, which conceivably could be studied, but also in terms of genetics, which no study to date has examined in more than a rudimentary way. The various statistics sins that we complain about on HN are exacerbated when important factors like genetic background are left unstudied. And we haven't even mentioned the fact that most published (and unpublished) nutrition studies are funded by the food and supplement industries!

A rejection of a particularly poor practice of science is not a rejection of science. I doubt we'll see decent nutrition science until we're all equipped with a network of bloodstream nanobots that measures the actual physical response we each display to nutritional input.

beachstartup• 11 years ago

the meatheads on some bodybuilding site

that's not very nice. a lot of people on HN lift weights.

pavlov• 11 years ago

I wouldn't have a problem if posters on said bodybuilding forum referred to HN as "the nerds on some programming site". It's the truth, isn't it?

Mikeb852• 11 years ago

Everything is wrong.

I wouldn't go that far... Nutrition 'science' is quite valid, the only problem is the amount of people in the nutrition space who aren't actually doing 'science'.

People who pay very close attention to their own habits and results are probably going to be more right about their own nutrition than all the scientists in the world put together.

Doubt it. People are often quite irrational about these things. Which is why there's so many people who follow fad diets (and buy the books).

VLM• 11 years ago

Ah the critical distinction is paying attention to your body is not the same thing as buying a book. As per OPs comment, I used to lift, its good exercise, and I saw plenty of demonstrations of data gathering technique and analysis.

pizza234• 11 years ago

It's difficult to understand why anyone would ever pay any attention to nutrition "science"

It's easy to understand, actually (I take your statement literally).

Being in shape is not rocket science: you eat clean, you do sports, and you're done. Although that sounds ambiguous, everybody does know what clean food is, and what sports mean (ahem, walking doesn't count).

The thing is that it's a damn hard lifestyle, if one doesn't have the right mindset/(self)education/upbring. Nutrition "science" is nothing else that a stream of silver bullets, which are easy to apply (avoid this, avoid that), so the public wants to believe in it, because it makes life much easier.

It's obviously a deception; but truth isn't the point - feeling good about the idea is.

Swannie• 11 years ago

The thing is that it's a damn hard lifestyle, if one doesn't have the right mindset/(self)education/upbring.

And budget. To do properly requires a pretty good budget for good quality protein sources, vegetables, etc.

Although that sounds ambiguous, everybody does know what clean food is

Not at all. You project too much. Many people truly believe that healthy food is what the supermarkets & TV tell you is healthy. The popularity of the "NutriBullet" is proof alone - "turn any food into a super food"... Uhh huh.

ddebernardy• 11 years ago

The real story here is that, even after being shown proof of that it was a hoax all along, a chunk of the same population will continue to hold that eating chocolate helps to lose weight.

javajosh3• 11 years ago

I laughed. But more seriously, I wonder if Bohannon's media experiment would have succeeded if you change a detail. E.g. if it was about something truly gross, like drinking curdled milk to lose weight. Journalists want to tell people what they want to hear, because people will tend to buy the paper that says what they are already thinking (weird, I know).

It's human nature that people are more skeptical about claims they don't like.

tdicola• 11 years ago

Imagine how mad the people in the study would be after they found out it was a sham. "I drank curdled milk for 3 weeks so you could get your big break!"

CyberDildonics• 11 years ago

It is a lot easier to convince someone of something they want to believe than of something they don't.

wpietri• 11 years ago

When I read this, I nodded. But now you've got me wondering if I just want to believe you.

ak217• 11 years ago

Actually, the takeaway for me is that John Bohannon behaved unethically and made a bunch of invalid conclusions.

javajosh3• 11 years ago

Bohannon tested a system, it failed, and he reported exactly how he tested the system and how it failed. 'The system' in this case is the global food science and health journalism industry.

He found that ignorance and laziness is rampant, that journal peer review is non-existent, and skepticism about results and methods is non-existent.

He has done a tremendous public service exposing the flaws in a system. With any luck consumers of these information sources will take everything with an enormous, skeptical grain of salt, because the last and best line of defense is your own skepticism critical thinking.

You can't delegate thinking.

leephillips2• 11 years ago

He did not find that "journal peer review is non-existent". He targeted a low-quality journal that does not send out papers for review to make sure he could claim that the research was "published" when sending out his press releases. That's explicit in the article, which you might find an interesting read

alexhill• 11 years ago

that journal peer review is non-existent

This isn't about peer review, because the paper was knowingly submitted only to journals known not to conduct peer review. Quoting from the article:

We needed to get our study published pronto, but since it was such bad science, we needed to skip peer review altogether. Conveniently, there are lists of fake journal publishers.

And later:

The new publisher’s CEO, Carlos Vasquez, emailed Johannes to let him know that we had produced an “outstanding manuscript,” and that for just 600 Euros it “could be accepted directly in our premier journal.”

This is really more about science journalism than about science.

These kinds of journals are difficult for a lay person to tell apart from actual, scientific journals, but distinguishing between the two is exactly the kind of job a competent science journalist would do. Inept science journalists, on the other hand, give these fake journals a reason to exist by citing them unchallenged.

notimetorelax• 11 years ago

If anything I applaud John's work, this is a great social experiment.

mcguire2• 11 years ago

And this attitude is one of the bigger reasons he was able to do it.

spacehome• 11 years ago

@reamworks

[Your post is flagged/dead, so I'm replying here. I hope you see it.]

The relationship between counting calories and weight loss is not quite as simple as the formula [Calories Injested] - [Calories Burned] = [Delta body fat] might indicate.

Details here: http://slatestarcodex.com/2015/01/12/the-physics-diet/

Mikeb852• 11 years ago

To be honest, chocolate may help in an indirect way...

If someone's on a diet, and never gets any 'reward', they might be tempted to cheat or give up.

If they're told a little chocolate every day is good, they'll stick with their diet, have their chocolate, and forego worse 'cheat' items (even though chocolate has plenty of fat and sugar, the amount you consume before you're content is relatively small compared to say, ice-cream, cookies, etc...).

Studies (and anecdotes) have shown that chocolate makes people happy, so that little happiness at the end might just help people stay on their diet and not stress out (stress can cause the body to store fat). Maybe it's something like the French paradox...

The study was pretty funny though, and I appreciate their honesty. And it goes to show why pretty much all research on fad diets is nonsense. Still though, if you choose between drinking a gallon of pop per day or chocolate, chocolate wins. So maybe their 'research' didn't go to waste.

bernardlunn• 11 years ago

My takeaway is you can always sell people something they want to buy.

tgb• 11 years ago

Did any major news organizations retract their articles or apologize?

edc117• 11 years ago

Do they ever? This is likely one of many reasons for eroding confidence in those same organizations.

bjourne2• 11 years ago

It's not news that the peer-review system isn't safe from exploits. People have been using Markov chains to generate studies which have then been published. And journalists uncritically picks up everything they can. All functions in society are based on trust and are easily exploitable if a person is not behaving trustworthy.

So making a fake study about chocolate and getting lots of magazines to publish it I think is a little unethical. It's not news, and the reason most people don't do it is because they want to act trustworthy. For example, someone could, if they wanted, submit patches to some free software projects which would introduce awful security holes and get them merged or just write shit on Wikipedia.

thomasahle• 11 years ago

It wasn't actually a fake study. That's the point.

They used the same statistical methods as too many 'honest' groups out there, and got results thereafter.

bjourne2• 11 years ago

I disagree. The study was fake because the intent was to deceive. He used flawed methods, but could have used 10 times as sophisticated methods and still would have been able to publish completely invalid results.

munro• 11 years ago

To continue this line of though, distributed computation is hard problem, and that's just computers. Here, people have to do real work to statically verify a paper's procedure, which is very cognitive overloading. Imagine reviewing a PR and deciding with certainty if it will work or not. Even then they should also "recompute" the experiment to completely verify it. Of course add human error on top of all this.

It seems there's a problem here, the article made it seem anyone can produce a bad study for lulz, and get everyone's attention. What is more scary is people doing it for vested interests. Maybe there's something that could be learned from projects like BIONC, Wikipedia, or Bitcoin, but better science is going to cost more money, and require more cultural attention. Not sure how to fix those things :)

liquidcool2• 11 years ago

On the flip side, you have some well-heeled corporations and industry groups working to discredit good science that's bad for them. Would be interesting to read a piece on how that's done.

GhotiFish• 11 years ago

Unfortunately the investment to make good science just to get it discredited by interested parties isn't as cost effective.

liquidcool2• 11 years ago

I don't think you need to go that far, you just need a PR person with a conscience to spill the beans on what they did (or dissect what others have done).

I Fooled Millions Into Thinking Chocolate Helps Weight Loss

Related Stories