Peer review: the end of an error?

https://www.the-tls.co.uk/articles/public/the-end-of-an-error-peer-review/

by montrose • 8 years ago

164 78 8 years ago

the-tls.co.uk

kelnage2• 8 years ago

The initial examples the author uses really do reflect the magnitude of the third problem. All of them were big breakthrough results in their fields - and hence people cared enough to do proper reviews - especially in the latter two cases. As an academic, I have participated in reviews of papers that I would have never otherwise read, but could still contribute to the process. Without something pushing all academics to participate in the system, then a more open solution risks moving science even closer to “only interesting problems/solutions/results get published”.

aldoushuxley001• 8 years ago

"Only interesting problems/solutions/results get published"

I'm afraid we're already there. Journals are very much strongly incentivized to publish only interesting or novel results.

kelnage2• 8 years ago

I agree there is definitely a bias towards such results - but I fear that completely free-for-all open access reviewing will make the effect even greater.

improbable223• 8 years ago

Isn't this the point of journals? They communicate new knowledge, which changes the status of what is known.

There's a balance to be struck in how certain these things should be before being published, for sure. The p-hacking plague is a symptom of setting this certainty bar too low, or equivalently, the required level of novelty too high.

Things which stay constant are the job of standards bodies, and textbook writers... these are important too, but different.

woliveirajr• 8 years ago

Remarkably, we have arrived at a system where academics feel a moral obligation to perform the thankless task of reviewing the work of other academics, anonymously and unpaid. This undoubtedly makes the literature better than it would otherwise have been, and ensures at least one reader for each paper.

I'm afraid it doesn't ensure that all papers will be read. Some papers can attract more attention, some papers can even receive too much comments with relevant comments being drown in the sea of observations.

John_KZ• 8 years ago

Exactly. It's the same fallacy with open source code. Just because it could be reviewed, doesn't mean it has been. Formal or semi-formal groups that professionally review papers are essential. How are they going to get paid? If you want research to be publicly available (as it should be) you can either put the reviewers on government dime (which isn't a great idea but it's better than nothing) or you can follow the system that was pretty much followed until now:

Research institutions that produce papers to be reviewed, pay up the reviewers, and then we make the research publicly available, because everyone is paid and happy. No need to double-pay the reviewers via per-paper costs etc. The cost of reviewing a paper is fixed. Hosting costs are trivial compared to that. The uni can take them on. In a sense sci-hub fixed the peer-reviewed industry. Formalize it (make all journals open-access) and we're done.

isolli• 8 years ago

Pull-request reviews work the same as peer review: it ensures that at least one person (the reviewer) reads the code or article.

phalangion• 8 years ago

I think the "one reader" refers to the reviewer, who ostensibly should read as a part of the review process.

chrisseaton3• 8 years ago

This article says that peer review is unpaid - it isn't.

If you are an academic or work in industrial research, it's part of your paid job to do peer review as part of the community. You are paid for it - it's part of your job and it's normal to do it during work hours.

heckless2• 8 years ago

Although reviewership is a useful thing that gets put on CVs, I can't think of a position I've ever seen where "review X papers" a year is part of the job requirement.

A more salient point about why it is important to note that peer review is unpaid is that academics, as authors, submit their work---for free---to a journal, where reviewers review it for free, and then the journal turns around and sells the article for $35 apiece.

electricslpnsld2• 8 years ago

I can't think of a position I've ever seen where "review X papers" a year is part of the job requirement.

I work in an industry research lab in the bay, and while there isn't a requirement to review X papers a year, it is expected that you keep up with the literature, and actively engaging in peer review is one way to do so. A few of my colleagues are on the paper committee for our major academic conference, and they end up reviewing 40+ papers a year and using a week or so to take part in the committee meetings.

chrisseaton3• 8 years ago

This is such an adversarial way to look at it.

I see it as a community. We write and share our papers as part of our jobs, we review each others' papers as part of our jobs, we help to run the conference as part of our jobs. Everyone contributes to run the system.

Nobody buys articles for $35 a piece. All major universities and companies subscribe to the ACM or whatever your field's organisation is for you. And guess who runs the ACM and other organisations? The community does! People from the community serve as leadership and help run things.

Yes the ACM gets some money to run their website, conferences and outreach programs. I don't think that's crazy.

woodson• 8 years ago

It is unpaid. A lot of reviewing for conferences is actually being done by Ph.D. students. They have to do it, whether they are on a stipend or not. Equally, a lot of reviewing is being done by (in some cases, former) academics whose current occupation does not provide resources (time) for doing peer review.

electricslpnsld2• 8 years ago

A lot of reviewing for conferences is actually being done by Ph.D. students. They have to do it, whether they are on a stipend or not.

Most (all?) PhD students in science and engineering are on fellowships, at least in the US. Major research universities will not admit unfunded PhD students.

chrisseaton3• 8 years ago

They have to do it, whether they are on a stipend or not.

This is like saying that undergraduate students are unpaid for doing their homework.

It's part of their education to learn how to take part in the community by doing things like reviewing papers.

improbable223• 8 years ago

The point is that it's done largely out of a feeling of obligation to the community. There's a chance that if you declined all of it you'd get a reputation for being an asshole, but there would be no other consequence for you. Whoever pays your salary would probably not notice.

I have actually been paid by some journals, but only a token amount, like $30/paper.

Thrymr• 8 years ago

Engaging in the profession is a part of your job. Doing editorial work for a journal (many of which are for-profit) is arguable. It is certainly accurate to say that reviewers are unpaid by the journal.

scott_s2• 8 years ago

I tend to do most of my paper reviewing during the weekend. No one is going to complain if I do it during my normal work hours, but the demands on my time are high enough that it's hard to focus on it. So in some sense, for me, it is unpaid, outside of my paid day job.

dhilistrian• 8 years ago

That's the way it should be seen, but it's often not.

I was at a conference not too long ago and chatting with a fellow faculty member at another university, pretty well-known in his field (in a medical school department), who told me a story about how his chair berided him for doing peer reviews at all. The colleague was like "but this is part of our job as a scientist" and the chair's response was "but what account number is that coming out of", as in, "who is paying for that?" The chair proceeded to explain that unless the department was being paid for his peer reviewing time, he shouldn't be doing it.

He subsequently explained that he feels like he's checking out of academics per se and trying to shift to public outreach activities because he's tired of trying to make a case for research integrity for its own sake, as opposed to being a vehicle for bringing funds into the department.

Things in higher education are quickly moving toward a state (or have already moved toward a state) where it is implicitly assumed that your job as a researcher is to bring in money, not to do research. That is, if you are paid by the state, you are not being paid to do a job (research) you are being paid to bring in even more money. You're a salesperson or adsperson not a laborer. Grad students are the laborers. I strongly disagree with this model, but it's where we are.

I used to think that conversations like the one with my colleague were kind of fictional dystopian kinds of conversations until I actually had one. And he was dead serious.

Universities are quickly becoming places to rent space for research, basically.

bervlad2• 8 years ago

you are not being paid to do a job (research) you are being paid to bring in even more money

And research funding is going towards big hotel, travel expenses, and conference fees.

Grad students are the laborers

Making the business highly profitable as there is no need to hire a large number of professors where you can offer tuition waivers and cheap stipends to a larger number of people who wants to get another degree

danieltillett5• 8 years ago

Trust me that a minute spent reviewing a paper is one less minute you have toward your actual KPI’s. About the only value of peer review is to give the paper to your students and tell them to find all the mistakes and holes in the paper. This is actually quite a good for the education of grad students as they learn what not to do.

jseliger• 8 years ago

One hopes the present system will go away: http://sappingattention.blogspot.com/2017/09/peer-review-is-...

The problems are in some ways even worse in the humanities: http://www.chronicle.com/article/The-New-Intellectuals/23835... than the sciences. Or see https://jakeseliger.com/2013/02/12/a-lot-of-academic-researc... .

improbable223• 8 years ago

A point not mentioned in the article is that this formal peer review system is itself quite young. I think that sending every paper out to several reviewers only became the default once we had xerox machines -- 1960s? We did a lot of serious science before that.

A bit earlier, the famous anecdote is about Einstein's offence that the Phys Rev editor would dare to send his paper for review at all -- apparently this would not have happened at a German journal of that era: http://www.preposterousuniverse.com/blog/2005/09/16/einstein...

noobermin4• 8 years ago

If anything, that works as a great defense of peer review given Einstein was wrong.

gophile• 8 years ago

Einstein was right about some things and wrong about some things. Which work of Einstein are you referring to when you imply that a peer review could have detected the error in his work?

throwawayyx96• 8 years ago

Author of the article is the same guy who, some years ago, organized a boycott of Elsevier journals due to their extortionate charges.

https://en.wikipedia.org/wiki/The_Cost_of_Knowledge

He has practiced what he preaches with respect to publishing in open access journals.

VHRanger• 8 years ago

Timothy Gowers has the moral courage to do what pretty much everyone else is waiting for other people to initiate

gowld3• 8 years ago

Being recognized as the top living mind in an entire field of science backstops a lot of courage.

jwilk2• 8 years ago

Archived copy, which can be read without JS enabled:

https://archive.is/iPwXi

jwilk2• 8 years ago

(2017)

probably_wrong• 8 years ago

There are two points that I wished were addressed more often.

1. New researchers. The examples he mentions include researchers who were already important in the field, and whose prestige guarantees review. But if you're a new researcher, good luck getting noticed. Blind review doesn't have this problem.

2. The cost of journals. People say "journal" and think of Elsevier, but there are several journals and conferences that publish all papers for free. The ACL, the largest conference on computational linguistics, finances itself with the entry fee to their conference, and you can read all of their papers right now (disclaimer: I'm currently doing work for them). There is a middle ground between Elsevier and Arxiv.

femto• 8 years ago

One way to look at journals are that they are "ready made" reputation for those without their own. A new author can leverage the reputation of the journal to establish their own reputation.

On that basis, maybe the mark of a truly prestigious researcher is one who can post directly to arXiv? Rather than the highest accolade being for a paper in Nature, the highest accolade should be for a widely reviewed paper on arXiv?

CharlesMerriam2• 8 years ago

An interesting parallel can be found in the anonymous review of grant proposals. See "Fund ideas, not pedigree, to find fresh insight", https://bit.ly/2Fy64cP. It ends with "peer review may be the worst system, except for all the others". It can be found in Nature, a peer reviewed journal.

xamuel2• 8 years ago

Not to defend peer review as it is now, but one of the things I've come to value from editorial gatekeepers is that they put the brakes on me publishing things that in later years I might regret. I wrote some pretty mediocre papers as a grad student and I'm grateful some of them were rejected, because I wouldn't want them forever part of my legacy. As a matter of fact, I wrote some "papers" when I was a teenager, which, if I had been allowed to freely publish at that time, might have become permanent albatrosses around my neck :P

danieltillett5• 8 years ago

Didn’t your supervisor step in and say this is not up to scratch? When I was a supervisor a good chunk of my time was spent going through my student’s and postdoc’s manuscripts to ensure nothing was in them that might embarrass us both.

xamuel2• 8 years ago

I was reckless and wrote papers unrelated to my dissertation and submitted without going through my advisor.

Fortunately, the bad ones were rejected. The good ones ended up making me well-recognized in some extremely tiny niches, and ultimately did way more for my career than my PhD work did :D I'm really glad some of the bad ones didn't see the light of day :P

jessaustin• 8 years ago

Do some of the alternative platforms offer a way to repudiate a "published" paper?

kolpa2• 8 years ago

The academic peer review debate has strong parallels to the social-media news debate -- As traditional newspapers/ TV channels are no longer gatekeepers of publication, we have given up a level of vetting in exchange for freedom-from-censorship. But that puts an onus on consumers -- it's not right for consumer to accept everything they see with the same level of default trust they had before. Consumers still must "consider the source", and society to create an effective replacement for the editorial filters we are discarding.

see also: "web of trust"

danieltillett5• 8 years ago

The big problem with peer review is almost no reviewing takes place and it is at best is either a way for your enemies to get revenge or a cheap proofreading service.

The solution is to offer a largish reward to the most constructive peer review for each manuscript chosen by the editor, or better by other reviewers. If I (or more likely one of my starving grad students) knew that I would get $1000 for providing the most constructive peer review then I would put in the required effort to make peer review a worthwhile exercise.

One thing that would help the peer review process is a data dump of the raw data. Many times I have seen some figure or table in a manuscript and felt I could not provide a constructive critic without seeing the underlying data. Going back through the editor to get this data is just too much of a hassle most of the time.

wyattpeak2• 8 years ago

But who decides what's constructive? It's all very well if you're choosing amongst people who basically agree with you - what if they're telling you it's going to take another year of experiments to get it up to standard?

Which reviewers, too, do you respond to? If it's a popular paper you probably can't respond to all criticism, but picking which criticism to interact with isn't going to lead to good science. There's a lot of value, even if it's not perfect, in writers being forced to respond to one or two reviewers' comments.

danieltillett5• 8 years ago

I suggested in the post either the editor or the other reviewers makes the decision of which was the most constructive review.

The suggestion of going back and doing more work exists under the current peer review system. With my proposal you might even get suggestions of what work needs to be done and why you should do it.

In all the papers I have published I have only had a few reviewers agree on anything. The normal approach is you agree to all the reasonable demands even if you don’t personally like them and argue over the important one. Most of the time if the reviewers see you have moved some of the way towards their arbitrary demands they will let you hold onto the important ones.

tom_mellior2• 8 years ago

But who decides what's constructive?

For a start, it would be good if as an author I could give feedback on review quality. Rebuttal periods are becoming standard for computer science conferences, and you can put some feedback into your rebuttal, but if you are too frank about bad reviewers, they could decide to recommend rejection. It would be good to add an option to review reviewers after the final decision on the paper has been made. I do think that that would make (some) reviewers more constructive.

A few conferences do have distinguished reviewer awards, though I've never really seen explained how the winners are chosen.

arca_vorago2• 8 years ago

I've said it before and I'll say it again.

The focus should be on reproducing studies, not on peer-review.

Of course peer review can be of assistance in selecting studies worth putting the effort and funding into reproducing, but peer-review itself is extremely corrupted as a process, but since so many peoples careers depend on it , many of them will not talk about those issues.

I used to hold the peer-review and journal process on a pedestal. Then when working at a genetics lab I ended up having to read a lot of papers to get my sysadmin work done, and started realizing just how bad so many papers are, even though published in "reputable" journals.

For example, since publishing is seen as so vital to some people, I noticed many of them would "consult" on another groups paper just to get their name on the paper, even though they did almost no work, and then those same people would turn around and say stuff like "I've been published X times!" as a sort of reinforcement of authoritativeness (and as advertisement).

People are gaming the system because hardly anyone is actually checking them. Beyond that, the actual peer-review process itself seems to be very lackluster in rigor on it own.

I think there should be a certain amount of public research funding that should be earmarked to go directly to independent reproducability labs.

On top of it all, all this research that the public is funding is often behind a fucking paywall... and that's not just bullshit, it's theft and misuse of public funds. It should be required that publicly funded research should be publicly published, and therefore available for more eyes to see and review.

dragonwriter• 8 years ago

The focus should be on reproducing studies, not on peer-review

The two aren't separable, since a key part of peer review is assuring that a published study is, in principle, reproducible. You can't focus on reproduction withoit focussing on peer review.

arca_vorago2• 8 years ago

I agree, I just worded it slightly poorly. My point is that the biggest focus should be on reproducing studies, not that peer-review shouldn't be revamped also.

I mean, isn't reproducing a study the ultimate peer-review?

squiggleblaz2• 8 years ago

Reproducing is a form of peer review (if you assume that a person who successfully attempts to reproduce your work is ipso facto your peer).

But the fact that your peer (i.e. another academic who works in the same area) has read your paper and commented on it in the process of publication (i.e. customary "formal peer review") does not reflect on its reproducibility. The peer, being a human, has attentional deficits and there's probably true facts that increase the chances of the result occurring by chance which are not considered worthy of publication.

Until a paper's result is reproduced, it can't be regarded as reproducible.

jhbadger2• 8 years ago

I think the mathematical sciences see peer review as less of a requirement than the experimental -- in theory, anyone can verify a proof for themselves. But in experimental science you really need someone familiar with the techniques to make sure that they did all the needed controls and didn't do any shortcuts and that's what peer review is useful for.

pmiller2• 8 years ago

I disagree. Even with the current peer review system in mathematics, false proofs have made it into the literature. Who knows how many more would creep in if we did away with peer review?

tzahola• 8 years ago

One can argue that false proofs made it into the literature because people trusted the reviewers - compare with: appeal to authority.

noobermin4• 8 years ago

Was going to comment to say the exact same thing. OP used a paper by medical professionals to poo-poo formal peer review and then used math papers as examples of successful arxiv papers scrutinized by other scientists. After realizing that, I stopped reading the article then Googled his name, and--surprise, surprise--he's a mathematician.

It will work for mathy and theoretical sciences like CS and theoretical physics, but good luck getting experimentalists to review papers for free (or at least without being asked by a journal). I'm not sure whether peer review is necessary in those cases, but the culture around more applied sciences definitely considers peer review more important, even though we routinely post to arxiv anyway.

gowld3• 8 years ago

It sounds like you're saying that experimentalists rely on peer review as a way to fraudulently cover up their shoddy experimental methods.

pizlonator2• 8 years ago

I've published in CS conferences, CS journals, and on a blog.

Publishing in CS conferences/journals means you get reviews from 3-6 "smart people", and then maybe 50-100 people will read your paper, and most of those people will never contact you.

OTOH if you publish a blog and advertise on hackernews and twitter, you get instant feedback, including from some of the same experts that would have done reviews for conferences/journals.

tincholio2• 8 years ago

The difference is that more often than not, you can't go to the same level of detail and depth in a blog post as you would in a scientific article (and if you do, not that many people will read it). Reading science is hard work, and properly reviewing a "journal length" paper can take upwards of a day. I don't think you can have it both ways. If you write a good vulgarization, or technical blog post, and submit it here, you will likely get many eyeballs. If you post serious science, you'll get much less feedback, if any.

pizlonator2• 8 years ago

My blog posts typically have more scientific content than a CS paper.

wbillingsley• 8 years ago

In some fields, such as HCI (and research through design) I think there could be value in moving to a system where instead you version papers and also publish the criticisms that led to its refinement.

If you've explored a problem, designed a piece of technology, and taken it back to that audience to gain further insights into the problem, what I can do in reviewing it isn't the same kind of analysis as a maths proof. Even if I you were to give me the tech or the code, I wouldn't have the same people to test it with, and would not expect to get exactly the same results. Instead, peer review tends to criticise the writing, missing elements in the argument, pieces of literature that also need considering, aspects of the methodology, and suggest more things you could do.

All of which is very valuable, and sometimes peer review is the best (only) mentoring you'll get if you're in a small university. But it's not quite the same as "we have to review this or a wrong claim might get the imprimatur of being science". Design questions don't make the same claims of completeness or context-independence. It's more of an argument based in the kind of muddy evidence you have to put up with when tackling muddy problems.

Which makes me think that instead of peer review being a hurdle to publication, perhaps we should just make the whole argument and refinement public. Give the reviewers more kudos as instead of being invisible entities whose thoughts are only received by the author and editor, their role in the development of that idea becomes a published item too.

dunk0102• 8 years ago

I am afraid this piece is missing the most massive reason that articles are still published in journals - Impact Factor. Academics live and die not just by the number of articles that they get published, but, crucially, where they get published. The entire academic system of progression is basically controlled by the journals. The reason that the splashy websites never succeed is that they have targeted the wrong problem. If you want to fix the system then you have to come up with a new way of rating academic's publications, which would then be fed back into the career progression system in their individual institutions. Good luck with that.

iandanforth• 8 years ago

While this is true in many institutions there are now useful proxies, specifically a Google Scholar profile or an individual h-index which can be calculated without publication impact factor.

You are correct that it will take a long time to update documentation and policies that reference impact scores, but informally, at least in Machine Learning, people know who the contributors are with high actual impact. In addition because a lot of impactful work is coming out of industry and there is rapid turnover between academia and industry the culture as a whole is evolving.

dunk0102• 8 years ago

Of course there's an untruth in what I said. Ostensibly it's all about Impact Factor, but in reality it's all about how much money you are bringing in. Academia is in a sad state.

freech• 8 years ago

Reddit but only academics can participate? Probably better then peer review, but it won't have the aura.

lowkeyokay• 8 years ago

I really think we should have something like Stack Overflow for academic papers. The comments system works so well. Also, the so called impact factor of journals, which in many cases are directly or indirectly linked with academics and universities financial outcome, could be replaced by the points system from SO.

Of course I'm not proposing using SO directly for academic research but something inspired by it.

hvidgaard• 8 years ago

I think a hybrid between SO and Discourse for papers would be more fitting. There really needs to be an open but moderated discussion for the papers - that'll serve as the peer review.

Peer review: the end of an error?

Related Stories