How science journalism is broken and how can we all fix it

http://medium.com/@nukleosome/how-can-we-fix-research-churnalism-668075268680

17 5 7 years ago

The URL to Scripps Research is obviously working now, after the story got its largest share of public attention from other websites. Truly disappointing.

notafraudster• 7 years ago

I am a little surprised, given the title, that the issue being highlighted is blogspam and the brokenness is basically that science journalists can't fairly capitalize their effort. At the risk of hijacking the discussion to talk about an entirely different issue: science journalism is broken not because spam blogs copy-paste articles. It's broken because the entire ecosystem has the wrong incentives.

Researchers, many of whom have inadequate statistical training, are academics facing publish-or-perish and weird publication barriers (p-value filtering, premium on sexy new results over well-researched incremental advances). The more senior authors who have tenure and are safe from some of the incentives still rely on lab funding, which is similarly competitive, and also tend to people whose methodological training is substantially out of date. Most journal reviewers are subject matter experts, some with limited statistical training.

Universities put out press releases to support authors without being equipped to evaluate the quality of the work or having an incentive to contextualize limitations of the work. Science journalists largely breathlessly report the original press release without reading (or in some cases having access to) the original article. They tend to have a good understanding of jargon but a poor understanding of design. There is no notion of including comment from critical authors, and there is certainly no notion of critiquing the design, replicating the experiment (either a limited replication using the author's code / data, or a more thorough replication to obtain the substantive result). Then the blog-spam people play broken telephone with the initial science reporting. Well-meaning and interested users post these articles to reddit, HN, social media, etc. credulously reporting the headline of the article they read. Most comments in the discussion read the title, skip the article (let alone the study) and proceed assuming it's true, and normally take the form "This validates some other belief I have, so it's true" or "Duh why did they even do this study?", both of which are not useful.

I see very little evidence this process stems from the article's implied undertone that blog spam is costing legitimate journalists money and starving them of resources they need to do a good job. Even very well-funded sites suffer from this. The problem is the incentives. Imagine a science website that posts only 1/10th the number of posts, but deep-dives all of the ones they do.

Let me proposal an alternate model: As a user or a journalist, if you aren't trained enough to read the study, don't post an article about it. Training does not mean subject matter training (as in you understand whatever the specific topic at hand is), although that is useful. Training means enough numeracy to be able to evaluate the work. This should cripple the flow of science communication but vastly increase its quality.

Posting an article you can't understand and relying on the "system" to ensure it's true is the same kind of broken process that leads to people circulating conspiracy theory stuff or stuff we all agree is junk science. This should be especially followed in disciplines like social psychology, applied economics, neuroscience, evolutionary anthropology or evolutionary psychology, medical research, nutrition research, and other fields where experimental or quasi-experimental design are more difficult.

jfengel• 7 years ago

if you aren't trained enough to read the study, don't post an article about it

The problem, as I see it, isn't just that the journalists can't read it, but that their audience don't care about it. The journalists are supposed to serve as intermediaries between the two, but frequently, proper journalism on the story would fail to excite the reader. The vast majority of science is important to other scientists, but science that actually matters even to scientifically-interested readers is not that large.

The readers want more content that interests them, and science fails to provide it. No journalist could bridge the gap between the dull progress of science and the reader's real interest level. The market for "mind-blowing" science -- sweeping technological and philosophical advances -- is far greater than the market for incremental improvements and novel but minor insights.

The niche does exist. I find it filled fairly well by Science News, which has about a dozen pages of real science news every two weeks. I only just got an issue about the black hole photo, which was fine with me: real discoveries are still true and important two weeks later. But most readers would find every other article in that issue terminally dull.

I don't know how to fix that other than to ignore any science "journalism" that comes out more frequently than that. It exists, and I ignore it. It doesn't contribute anything: it's peddling effectively-fictional stories to readers who want sci-fi. I think that the finest science writing promoted well might attract marginally more eyeballs than it currently gets, but most readers simply don't actually care about science, and it's not something journalists can fix.

PierredeFermat3• 7 years ago

I don't think the problem should be approached from a "great market" perspective. The market for Hacker News is quite niche. Yet, it has become a great community for the curious of us. Almost hardly replaceable by any of the communities/websites who have great market potential and wider audience. So niche is probably a good thing here.

Driving a critical discourse (on something like Health or Hacker News) between scientists, journalists and the general public can potentially create more appreciation for scientific progress (which might often be dull and slow indeed) and provide a feedback loop for how science is/should be communicated or even done.

PierredeFermat3• 7 years ago

tldr; I'd like to get in touch and do a deeper dive into the problem so that the solution can be refined. How can I reach out to you?

I am a little surprised, given the title, that the issue being highlighted is blogspam and the brokenness is basically that science journalists can't fairly capitalize their effort.

I think this was slightly highlighted later in the post but agree that it wasn't stressed enough.

Science journalism is broken not because spam blogs copy-paste articles. It's broken because the entire ecosystem has the wrong incentives.

I think the two are closely related; the copy-paste behavior contributes greatly to minimizing the window of capitalization. However, it's definitely not the only factor.

Researchers, many of whom have inadequate statistical training, are academics facing publish-or-perish and weird publication barriers (p-value filtering, premium on sexy new results over well-researched incremental advances). The more senior authors who have tenure and are safe from some of the incentives still rely on lab funding, which is similarly competitive, and also tend to people whose methodological training is substantially out of date. Most journal reviewers are subject matter experts, some with limited statistical training.

That's basically a problem of science, perhaps at the core of it, but not science churnalism I think.

I see very little evidence this process stems from the article's implied undertone that blog spam is costing legitimate journalists money and starving them of resources they need to do a good job.

You don't think aggregators that built some traction (i.e. Phys.org, MedicalXpress) are taking away opportunities of good journalism? I don't have quantitative evidence for this but the qualitative one is pretty strong (talking to journalists, for example).

There is no notion of including comment from critical authors, and there is certainly no notion of critiquing the design, replicating the experiment (either a limited replication using the author's code / data, or a more thorough replication to obtain the substantive result) [...] The problem is the incentives. Imagine a science website that posts only 1/10th the number of posts, but deep-dives all of the ones they do. [...] This should be especially followed in disciplines like social psychology, applied economics, neuroscience, evolutionary anthropology or evolutionary psychology, medical research, nutrition research, and other fields where experimental or quasi-experimental design are more difficult.

Precisely the raison d'être of something like Health News. I guess this was pointed out near the end of the post. Although realizing it will take some time and traction, it looks like a good starting point.

How science journalism is broken and how can we all fix it

Related Stories