Three myths about scientific peer review

by neilc • 18 years ago

19 5 18 years ago

cperciva• 18 years ago

While I agree with the author's main points about peer review, I think it's a bit of a stretch to conclude from those that peer review is useless.

As a peer reviewer, I've sent papers back with comments along the lines of "this work seems quite similar to the published work of X in the following papers (which the author doesn't cite): ... I wonder if he if the author is aware of this earlier work", and subsequently heard nothing more (presumably because, upon looking at said papers, the author came to the conclusion that he had not in fact discovered anything new). On the opposite side of the coin, one of my papers benefited from very similar comments, in that having related work brought to my attention allowed me to revise my paper to better describe how it fit with the rest of the field.

I think it's in cases like this -- where peer reviewers point out things (whether other research, or possible errors) of which the author(s) do not seem to be aware -- that peer review really shines. If reviewers respond merely with a 'yea' or 'nay' they're not doing their job properly; their job is to review, not to judge.

neilc• 18 years ago

I think it's a bit of a stretch to conclude from those that peer review is useless.

I don't think that's what the author concludes. But otherwise, I agree with you.

yummyfajitas• 18 years ago

The quality of peer review varies with the journal. For instance, Journal of Computational Physics tends to give out helpful reviews. From JCP, I usually get back a list of helpful suggestions and weak spots in the paper. Even the incorrect suggestions/criticisms tend to reveal ambiguity in my writing more than anything else.

PRL, not so much. In my experience, at least one referee doesn't even finish reading the paper. It's not just me, this happened to a friend of mine last week:

Referee: "The authors do not demonstrate that their equations apply to any physical example."

Response: "See page 3 paragraph 4."

jderick• 18 years ago

I don't think anyone who has ever reviewed a paper would consider these three things to be "myths" -- they are just obvious facts.

I do think the peer review system will eventually change to something more like hacker news. It is a much more convenient way to do things than trying to find volunteers, mail out papers, collect responses, etc. Not to mention you can potentially draw from a larger audience and you make your work more easily accessible.

One mechanism that does exist in peer review that I'm not sure could be easily reproduced is a sense of duty while reviewing a paper. When selected to review a paper, you know you are one of only a few who will determine the fate of this paper and you feel a responsibility to try hard to understand the paper and give it a fair critique. If a paper were just posted online for anyone to critique, people might not put forth as much effort in their reviews.

aposteriori• 18 years ago

Some people say double-blind reviews make things better since reviewers aren't (theoretically) influenced by the identity of the authors, and authors can't tailor their paper to suit the reviewers (or try to get back at them later for negative reviews).

How about completely open reviewing, like in some of the BiomedCentral medical journals (e.g., http://www.biomedcentral.com/bmcmed)?

If done properly, all the cards are on the table. A paper cannot (should not) be rejected or accepted without legitimate reasons. Reviewers and authors can be made to account (through tarnished reputation and risk of embarrassment, at least) for their actions.

Three myths about scientific peer review

Related Stories