Should journals pay peer reviewes?

https://www.sciencemag.org/news/2021/03/450-question-should-journals-pay-peer-reviewers

by dariosalvi78 • 5 years ago

40 74 5 years ago

zackkatz• 5 years ago

There’s, ironically, a typo in the title. Should be `reviewers`.

prepend2• 5 years ago

I’d rather just reduce the number of for profit journals. Getting paid $450 seems to be against open access as open journals can’t afford this.

Also the motivations are different when pay is involved. I will donate time to peer review, but $450 is not enough to pay me. So I would be more likely to review as a volunteer than an employee.

RBerenguel2• 5 years ago

I want to contend that to most non-tenured professors (outside of the States at least?) 450$, even if not enough as pay either would be a nice extra from time to time.

michaelmior14• 5 years ago

As a non-tenured professor within the US, I don't really mind reviewing for free. But $450 per review would indeed be a nice extra :)

prepend2• 5 years ago

I like money and $450 is a nice amount (hey, free PlayStation). But my employer doesn’t allow outside payments so I couldn’t accept.

And while I like money, I fee a duty to review a few papers a year, even if they are not good, or boring, or whatever. If I was being paid, I would be more likely to skip those papers because I no longer feel a duty.

This may be a similar motivation described in an experiment that showed how late pickups went up when the daycare introduced late fees. [0]

I think that many in academia are underpaid and should have more money. I just think that adding review fees are not the way.

[0] https://freakonomics.com/2013/10/23/what-makes-people-do-wha...

ta98810• 5 years ago

I agree we don't need the publishers anymore. We have preprint services (arxiv and others), open review systems (pubpeer), sharing platforms (osf, zenodo, dataverse). You have Free software to handle all the manuscript lifecycle (there are a few cant remember the names). We keep the publishers because they still have some prestige and they are blackmailing university libraries. Now there is Scihub to go atound that, but of course there are ethical and legal reasons associated with its use so in my opinion thats not a long term alternative.

ktpsns• 5 years ago

Well, I much more like the question: Should authors pay for their publications? There is a common pun: As a scientist, you pay two times for an article: The one time you publish, and the other time when you subscribe to read your published article. And this is only to access the article as a PDF in the web. There is no print copy or so involved in that payment.

Notorious XKCD: https://xkcd.com/2085/

michaelmior14• 5 years ago

This depends on the field. I have never paid to publish and I doubt I ever would. (This does not count however the conference expenses typically required of an author to present their paper. But attending a conference has significant benefit to me personally, so I don't really see it as a direct cost of publication.)

ta98810• 5 years ago

You usually have three models: - pay for publish and read: big tabloid journals, large medical maf... associations. Cost to publish is usually below $1000, cost to read is $5-$20 a page

- pay for read, free to publish: most journals from the big publishers. Cost is between $5 to $20 a page.

- pay to publish, free to read: the model of most Open Access journals. Some big publishers are proposing that as an option as well. Cost of that option is usually $3000 to $5000 per article.

sokoloff3• 5 years ago

Looking at the real-estate market and its assortment of minimal-value-add/maximum-value-extraction players (appraisals, title insurance firms, 75% of inspectors) as an analog, I suspect that directly paying for reviews will not likely improve the situation.

Perhaps a recognition by institution or university model would work, or softening the dollars (they are credits against APCs or something) might work.

ta98810• 5 years ago

Many universities take reviewing "into account " for the tenure-track process and performance reviews. but it counts really little compared to publications. Data sharing or management of ressources (databases, forums...) is completely ignored usually.

dash2• 5 years ago

I've seen some journals do this. The problem TBH is that the amounts paid aren't enough to be an incentive. Typically £50-100. A good review takes a day to do (maybe more). And academics are probably paid £100/day or more anyway, and aren't very money-motivated.

I think a better approach would be publicity. Academics care about reputation a lot. If good reviewers were acknowledged by the journal, that would be a big incentive, something you could put on your CV.

dairyleia• 5 years ago

This is exactly why they like Publons, it lets them take credit for their reviews.

michaelmior14• 5 years ago

Publons is a nice way of keeping track and I use it myself, but I think institutions are generally willing to take faculty at their word with respect to reviews we have completed. (Although my statement here is really just based on anecdotal evidence, so I could be wrong.

foobarbecue5• 5 years ago

Great point. Offering to pay a small amount could actually reduce the number of people willing to review, since they are currently motivated by a sense of duty. Reminds me of the Freakonomics chapter with the day care: https://freakonomics.com/2013/10/23/what-makes-people-do-wha...

DanielleMolloy4• 5 years ago

Journals should not shed out tiny payments for peer reviews by overworked faculty, they should hire teams of reviewers with diverse specialisations full time. By decent quality assurance on the massive amount of papers each year publishers could actually do something valuable for science.

michaelmior14• 5 years ago

I don't necessarily disagree with paying reviewers, but I think there is some value to having people who are actively working on problems in the same area being the ones doing the reviewing. Yes, it's additional work that we aren't paid for, but personally I enjoy it. I do agree though that for some it can create undue pressure so there may be something in the middle there that works well.

DanielleMolloy4• 5 years ago

I understand this point, yet to correctly assess whether a typical paper in my field - for instance - can be trusted you would need a modern neural networks researcher, a statistician, a cognitive scientist, a neuroscientist and perhaps a physicist. All would have to know about common fallacies in their respective fields and should optimally discuss with another.

Faculty reviewers are not only overworked, they have accumulated biases, can be overconfident about their expertise and in the worst case have malicious incentive to prohibit publication of your results.

I’m certain that a team of specialists who review as their full time job would do a better job than at least half of the reviewers I encountered (especially those at high-profile AI conferences).

I enjoy reviewing too and am doing my best to be constructive, but I’m not sufficiently qualified for most of the offers I receive.

piokoch• 5 years ago

That would be difficult in a fields with a vast range of specializations (physics for sure). They would really need an army of reviewers to cover everything from astronomy to particle physics.

sokoloff3• 5 years ago

Why would society want to encourage people with the deepest specialty knowledge to work for journals and away from the bench?

DanielleMolloy4• 5 years ago

Lots of these highly qualified researches are forced to leave science right now, the reasons all too often not being their qualification. There would be a large pool of good researchers to draw from for this.

gww3• 5 years ago

If the journal is paying a reviewer would there be more pressure in terms of turn around time? Or more pressure to accept contentious papers because the journal expects them to be highly cited or wants to pander to their authors?

I also imagine there would be people out there who would become serial reviewers with the goal of making as much money as possible. I imagine the quality of their reviews wouldn't be great either.

michaelmior14• 5 years ago

For turnaround time, I would probably say yes. Right now it's typically to have several months to review a journal paper. And there's really not much a journal can do if a reviewer is late other than not ask them to review again. Even if reviewers were paid, I suspect it would be difficult from a budgetary perspective to difficult to have serial reviewing be lucrative with the possible exception of adjunct faculty who unfortunately are typically very underpaid.

ta98810• 5 years ago

I'm curious to know in what field you get months for reviewing. In chemistry almost everything is 2-3 weeks, lower for open access journals.

ta98810• 5 years ago

Most journals are asking for 7 to 21 days turnaround these days. Not sure they could stretch that more...

gww3• 5 years ago

I have had a few papers submitted during the pandemic and the time limits given to the reviewer and the time the reviewer actually takes varies drastically. For example, one of my papers was under its second round of peer review for nearly four months. I have colleagues in similar positions with their papers as well.

I do think a monetary incentive would speed up the peer review process if the pay rate was cut for late review submissions.

Mediterraneo103• 5 years ago

No, please. There are still fields where journals are both open-access and not pay-to-publish. This is because they are published by the same non-profit learned societies they always have been for many decades, and overhead costs are low.

Bring in a culture of paying peer reviewers, and suddenly authors (or the learned society) will be under more pressure to get more grant money to pay the new publication costs. Also, as someone who works in academia on an occasional basis, taking years off from funded positions at a time, I appreciate being able to write the occasional article on my own time and get it published without paying hundreds and hundreds of euro out of my own pocket -- fields with pay-to-publish traditions make it hard for non-permanent faculty to contribute.

jariel4• 5 years ago

This is insightful, thanks, but there's a little problem here:

The cost is borne one way or the other, the $500 is a matter of comp, not cost.

If someone is going to review your work, that's an act of very material labour. That they are not really compensated for it, at all, is inconsistent with the basic premise of labour and comp.

You would not be a researcher if you weren't comped in some way, so the absence and/or misalignment of comp is the issue.

It's not free to publish and have 2 reviewers (!!) there is a very material cost that you're just not paying.

Allocating at $1K cost for publication more accurately reflects the labour involved in checking it out (at least a little bit).

It'd seem to me a lot of folks might want to take up the $500/review as a job to earn some reasonable income, they might even get good at it.

It's entirely plausible that one of the 'dysfunctions' of science is 'not enough review' and it's simply because there is no incentive structure for it at all.

We pay people for literally every other function in life. Doctors are paid to 'save lives' even though there is a moral obligation. The notion that we don't pay people for academic work seems odd.

Perhaps journals could identify the papers with the most promise and offer payment for review, or something like that.

Mediterraneo103• 5 years ago

Unpaid peer reviewing is certainly not incentive-free. For example, oftentimes one enjoys peer reviewing because the author cites some publications that are useful or important, but which one wasn't previously aware of. Sometimes the author makes some observation tangential to his/her main argument which proves an inspiration or key to one's own research.

These sort of things may be more common in the humanities versus hard STEM, but in my own field peer review is not generally seen as drudgery or exploitation.

jrochkind15• 5 years ago

Do you realize that the article authors are generally not paid?

It's not free to publish and have 2 reviewers (!!) there is a very material cost that you're just not paying.

Who is the "you"? The author I think? So you do know that article authors don't get paid, you're talking as if they are the "customer" that should be paying?

Generally, for non-open-source journals, which are still the majority in most fields, it is subscribers, the readers, who are the customers who pay the costs (plus enough for a healthy profit for the publisher). (Usually university libraries. The same universities who are paying most of the authors, who don't get paid by the journals, but produce the content that makes a journal valuable).

Elsevier is quite profitable, with a high margin on revenue.

You are painting an economic model of publishing that I'm not sure is connected to the actual real world economics of scientific/academic publishing, it seems to be very hypothetical.

foobarbecue5• 5 years ago

Almost all scientific journals in my fields currently charge authors to publish, but don't pay reviewers. (Earth Science, Planetary Science, Robotics)

I don't think your idea that publication costs need to increase to pay reviewers is valid. Elsevier et al would just cut some of their obscene bloat.

arbitrage2• 5 years ago

I have an inside line to Elsevier. Elsevier will NEVER cut anything, bloat or otherwise.

They will, however, use their market dominance to punish people who rebel against their publishing machine. They've done it consistently every time there's the whiff of shakeup in the publishing industry, going back decades.

arbitrage2• 5 years ago

The system is wickedly broken. Is there a way to fix the reproduction issue in science that has been raging for decades, without affecting the good players? I'm not sure there is.

the_duke• 5 years ago

And what percentage of published papers would be affected by this?

I surmise the amount is anywhere between small and tiny.

There are arguments against paying peer reviewers, but this is not a good one.

pulse7• 5 years ago

It is written somewhere: "Those who work deserve their pay!"

jmakov• 5 years ago

Sholud you pay a programmer for code reviews?

michaelmior14• 5 years ago

If that programmer doesn't work for your company and you want them to do it in a timely manner, then yes, I think it's reasonable to do so. For open access publications, you could argue that reviewing is similar to working on an open source project, but open access is still the exception, not the norm.

jrochkind15• 5 years ago

So, if the problem is that scientists lack motivation to peer review, I think having peer review work recognized as important in tenure-and-promotion would probably be far more motivating than whatever hourly rate they'd end up paying.

Article authors are generally not paid! Yet there is no lack of motivation to write articles.

How to motivate to do peer-review well instead of perfunctorily is another question.

ta98810• 5 years ago

Article authors often pay as well. Either directly because the journal is a science-tabloid or because the authors want to make their work accessible to all. Or indirectly through the indirect cost the universities are taking from grant agencies to pay for their confidential agreements with publishers. Try to do a FOIA of a state university to get that information you will see how hard they will fight the fact that they get scammed.

Authors write because this is how you justify that you are really working, be it for your university, department or your grant agency. Also you want others to build upon your work, but academics would publish a lot less if it was not for the huge pressure from funders and employers (increasing numbers is all that count for many of them).

jrochkind15• 5 years ago

Right.

Peer review is also essential to the production of research articles. And is done largely by the same people who are writing articles; but they are incentivized non-monetarily to write articles, but not as much to review.

To restate my point perhaps more clearly: If reviewing were rewarded/expected as part of one's role at a university, department, or grant agency similarly to authorship, that would do far more to incentivize it than paying a relatively modest amount for a review.

michaelmior14• 5 years ago

I would argue that article authors generally are paid indirectly by the institution which employs them. Most scientific publications are authored by academics or members of research labs who are specifically paid to publish. Of course, pay is not directly tied to each publication, but that is what most authors are paid for. At least in my institution, contributing to the peer review process is also an expected part of the process to achieve tenure. But the incentive is certainly far less than the incentive to publish.

mgr86• 5 years ago

Can you share some of these non profits? I work at one who used to publish a journal for decades, but Sage has handled it in more recent memory. We make our salaries through grants and a library product. But I am not aware of others doing similar work. Partially out of ignorance.

rocqua2• 5 years ago

How about making reviews available under the name of the reviewer. This means good reviewers can get respect of their reviewees, and reviewers are incentivized to write with slightly more consideration.

Similarly, universities should value good reviewers more. Which probably means reviewers need to get more prestige, so universities get prestige for having good reviewers.

ta98810• 5 years ago

This is something that is pushed by some people. With many journals I review for however, I am able to see otger reviewers reviews and I suppose they would be ashamed to have their names to be associated with them (extremely short and with generoc sentences that looks like they didnt even read the paper or just plainly agressive). I've heard many arguments against open review, but the most common is that people are afraid of retaliations if they criticize a work. I understand that people have retaliated to me after comments on PubPeer. But I still think we should do open reviews, that doesnt mean that journals that make people pay for publishing or reading should pay reviewers. We work for free get 0 reward... It is a bit like asking a designer to work for you, and tell them that it is to help the world of design and thank you no you can't sign...

karl11• 5 years ago

Yes. And their names should be attached to anything they review (including the papers they reject) so that they have some skin in the game.

foobarbecue5• 5 years ago

Hunh, this is a great idea. Rejected articles should be published as rejected, and all reviews should be published. I guess this is a bit like the open review model, like The Cryosphere uses, except I think they hide papers after the discussion period if they aren't accepted.

michaelmior14• 5 years ago

I think the problem with this is that it can encourage retaliation. If I know you rejected my paper, I might be incline to reject yours. Now, a nobody like me probably can't get away with doing that to someone with an established reputation. But the other way around is certainly possible.

AcerbicZero• 5 years ago

Can’t we jus my have a GitHub for science and get this over with? It seems silly to maintain this system that may have been functional when it started, but is now clearly behind the technology curve.

michaelmior14• 5 years ago

How would that address the core question being asked which is whether work spent reviewing should be paid? In any case, while there are problems with peer review, I don't think it's related to technology.

TheOtherHobbes• 5 years ago

I don't think the process would be less effective if papers were published without peer review, but were then questioned and challenged in public debate under editorial moderation.

Peer review isn't a single-goal process. It exists to introduce citations of existing work if they aren't already referenced, to question assumptions, and to find obvious errors. (Among others.)

All of these could be done in peer debate.

It's not as if the process is genuinely anonymous anyway. Of course it pretends to be, but it isn't.

Peer review dates from the days when information travelled as fast as a horse, and it made sense to include a review stage for basic quality control.

I'm not convinced it's a good model now, when information exchange is rather quicker.

The real problem is that journal publication is about career status, not about quality of work. Peer review is almost a side issue.

The scientific publishing industry is not just financially parasitic, it also introduces unnecessary delays in idea exchange. Public peer debate would eliminate some of that friction, but it would also have public consequences for reputation - and that might well be more effective for quality control than pseudo-anonymity.

michaelmior14• 5 years ago

At least in my field, I expect this would be a total disaster. One of the major advantages of peer review is that knowing the top tier publication venues, I can look to these to keep up to date on the latest work knowing that what I find is likely to be high quality. In the absence of peer-reviewed publication venues, I see two options:

1. Read everything I can find that is published and try to sort through it all myself to sort out what is good quality. I think in aggregate, this is likely to take more of researchers' time than contributing to the peer review process.

2. Focus only on publications from top quality institutions or researchers whose work I am already familiar with. This has the advantage of saving me time. But it also has the massive disadvantage that I am likely to miss quality work which comes from lesser known institutions or authors.

I don't think either of those two options is attractive. In my field, the delay of idea exchange is not really a problem. If people want to share their ideas, they can often publish a preprint in parallel with submission to a peer-reviewed journal. (Although some journals effectively prohibit this since it has the potential to break the anonymity of the review process.)

Yes, some of the purposes of peer review could be accomplished in the method of "peer debate" you mention. But I think such a process would be much less effective. Of course, that's not to say I don't think there are things broken about peer review that could use fixing :)

jltsiren• 5 years ago

Like many other recent international hires at US universities, I'm on H-1B visa. If paying for reviews was the norm, I believe it would be illegal for me to review papers, even for free.

I'm not sure what to think about this. Peer review is an important way of contributing to the community, and every academic is expected to participate in it. Commercial publishers that make profits from free academic labor are parasites, and they should pay for the services they use. Yet immigration rules often make it difficult to earn any side income, so paying for reviews would exclude many (often less established) academics.

michaelmior14• 5 years ago

That is a good point and one I hadn't considered despite being an H-1B holder myself. I agree with your argument that paying for reviews under current immigration laws would be highly problematic for visa holders. It would either mean we would be expected to do it for free when everyone else gets paid, which would be unfair and potentially still not legal. Or we would be unable to review which I think is an important part of establishing oneself within the community.

taylodl• 5 years ago

Is there a way this can be made double-blind, where the Journal doesn't know the reviewer and the reviewer doesn't know the Journal? Can something akin to Mechanical Turk be utilized? Just trying to think of ways to have paid reviews without incurring all the negatives so typically associated with that arrangement.

Should journals pay peer reviewes?

Related Stories