We shouldn’t take peer review as the ‘gold standard’

https://www.washingtonpost.com/outlook/why-we-shouldnt-take-peer-review-as-the-gold-standard/2019/08/01/fd90749a-b229-11e9-8949-5f36ff92706e_story.html

by danso • 7 years ago

224 137 7 years ago

washingtonpost.com

pacbard2• 7 years ago

What would be the alternative to peer review? White papers on “pop” topics? Opinion pieces by think tanks?

The ideal of peer review is pretty much the gold standard of scientific work. By that, I mean the idea that your work should undergo rigorous scrutiny by your peers before it gets accepted for publications. As any other human system, though, the process is still subject to politics and biases.

My hope is that a truly open review system might one day democratize science and ameliorate the issues with the current review system. For example, it would be cool to put all the revisions, reviews and authors’ comments in the open as an online appendix. That will show how the paper changes as it moves trough the process, the issues that the reviewers brought up, and how the authors responded to them. It will also help show if there is systemic bias against certain researchers or topics.

buboard• 7 years ago

E-life is already doing peer review right. Now you need get others on board

nine_k2• 7 years ago

I think that publication should not be gated by peer review. Release early, release often, publish experimental setup and what you are looking for prior to the actual experiment, etc.

We have arxiv now; the whole process could open up even more, slightly closer to research in the open, like open-source development in the open. This is to make peer review easier.

Then what is currently "publication" becomes what it should be, in the sense of adding value: curation, filtering the most interesting, least dubious results.

chrisseaton3• 7 years ago

arxiv is actually less open than most traditional venues.

Anyone can submit to most conferences and journals. Anyone at all! No PhD or reputation needed and it’s blind. All that matters is the quality of the research.

To submit to arxiv you need to be approved by someone as a legitimate researcher, or beg for reviews from people you’ve never met without any anonymity.

It’s somewhat a step backwards in some ways.

amelius• 7 years ago

Yes all results should get published. Peer review should only affect ranking/discoverability.

unholiness• 7 years ago

You're not the only one to comment this, but from what I know per review is a long iterative process, wher reviewers raise objections and submitted respond with edits.

So, what you're suggesting seems a lot like shipping code before code review, and seems to have many of the same problems. What do you do in the long period before critical issues have been addressed? What do you do if the submitter just says "fuck you, my code works, I have more important things to do"?

tptacek6• 7 years ago

The article doesn't propose we do away with peer review, but rather that lay audiences stop pretending that peer review is a "gold standard" by which the quality of research can be judged. It's "gold standards" the author proposes to eliminate.

pacbard2• 7 years ago

I get that. Sorry my reply wasn’t very clear. The article gives examples of how politicians and policymakers have “weaponized” peer review to silence critics. The author proposes moving away from peer review (and the gold standard it sometimes represents) as a way to solve this issue.

In my reply, I was just pointing out that peer review still plays an important role in the development of knowledge and I was wondering if alternative systems that could replace it would actually be feasible. That lead me somewhere off topic.

By the way, the “weaponization” of peer review is nothing new. You can see it used in the past to discredit research linking smoking or lead to health issues. You can also see of the possible risks or benefits (depending on your point of view) of rejecting peer reviewed research in the anti-vax movement.

tomp• 7 years ago

Personally, I view any study in the less scientific fields (i.e. those that routinely accept p < 0.05 as "proven") as suspect, unless it's preregistered, has full data available for the public to check, and has p < 0.01. (I'm neither aware of any such study, nor have I looked for it... I'm just saying that that should be the minimal acceptable standard for e.g. psychology and medicine.)

JumpCrisscross2• 7 years ago

What would be the alternative to peer review?

Replication. If science has a gold standard, it is this.

tptacek6• 7 years ago

Replication is slow and can be extremely expensive. It's a good thing, but it doesn't serve the same function peer review does.

analog312• 7 years ago

It would be interesting to see if replication is really the glue that holds our greatest scientific achievements together. I suspect not. In my view a higher standard is progress towards powerful unifying theories, even if getting there is a messy business.

I'm thinking of something like relativity or quantum mechanics. Suppose a study in those fields fails replication. The whole thing still holds together, to the point where controversies at the part per billion level make it to the front page of the newspaper. Perhaps even most studies, taken in isolation, would be found to have problems when subjected to the strictest criteria for replication. Choosing replication as a silver bullet would be an unnecessary distraction.

Now, what about fields where there is no unifying theory on the horizon? If replication is all we've got, then sure. I can certainly see the point, especially if the results affect personal decisions (diet, medications, etc.) or public policy.

I suspect that "gold standards" can hurt science as much as help. Telling people that science is bunk because of the "replication crisis" contradicts the fact that messy science has produced results of astounding accuracy and predictive power. Learning from success should be at least as important as installing safeguards against failure.

snowwrestler• 7 years ago

The gold standard of science is a different experiment whose findings agree with the findings of the first.

Replication is much less valuable, scientifically and professionally, than non-scientists think it is. Simply repeating a published investigation runs a much higher risk of repeating any experimental errors or faulty assumptions that might have harmed the first one.

The whole point of science is to find knowledge that persists beyond one particular perspective; knowledge that is independently verifiable. Rote replication is not the best way to find this type of knowledge.

As for peer review, its purpose is simply to sharpen the communication of a completed study. Even if every study was replicated, the papers would still benefit from peer review.

SubiculumCode• 7 years ago

Some journals do this. It is an interesting approach.

I am however ambivalent, or even perhaps against allowing public comments directly into a paper. We've all experienced, no doubt, the a naive but loud internet com-mentor spreading FUD on comment sections and social media on topics they simply know nothing about, and do so prolifically. It is nice to think that good arguments will float to the top, but often it is the loudest and least nuanced argument, right or wrong, which gets taken up.

aeternus• 7 years ago

I would love to see an open review system like that. Comments out in the open, along with the experiments and experimental data which supports or does not support the work.

Science is all about experimentation, yet we currently measure papers by number of citations rather than by experimental support.

o09rdk• 7 years ago

I'm an editor of a journal that has one of (the?) highest IFs in its field, among other things. Lots of experience with scientific publishing.

I've had lengthy discussions about this with my colleagues, and I don't think the problem is with peer review per se, it's with how peer review, and by extension, scientific literature, is interpreted and utilized. It's not just with the lay community, it's among scientists themselves, as well as scientific consumers as in engineering, healthcare, etc.

People now treat science, and scientific publications, as if it's a farm or factory. The produced commodity is the peer-reviewed paper. Once research, via a paper, is peer-reviewed, it's blessed as the truth, and if it's not, it's worthless. The goal of many institutions is now not to produce sound research per se, but to receive money (largely due to indirect cost income) to produce the commodity of peer-reviewed paper. Once a paper is accepted as in press, it can be cited as if it were truth, which provides a building block for something else. The truth per se doesn't matter, only the citability of it as peer-reviewed.

I realize this sounds a bit conspiratorial but I do think this is basically how it works at this point in time, at least in certain fields. Peer-review is overvalued, as are single papers, and contributions of particular researchers in many cases. That doesn't mean they don't have any value, just that their real value is probably much less than we assume.

I agree that peer review can be improved a lot, and will likely always have an important role in some form, but I don't think fundamental problems in the field will go away unless there's a change in perspective on the real scientific process as a whole, to something more nuanced and gradual.

elihu• 7 years ago

Test results are very easy to fake. The "gold standard" is whether outside groups can run the same tests and reproduce the same results. Peer review doesn't mean much more than "this paper seems interesting and is well-written and properly formatted."

Retric• 7 years ago

Self consistent is another major check. A lot of math and similar errors are caught in peer review.

petschge4• 7 years ago

Peer review isn't (and never was meant to be) a golden seal declaring a result to be unchangeable truth. It is a process that makes sure that at least one competent colleague has read it and caught all the embarrassing Monday morning mistakes that might have been in there. This is done before wider publication to avoid that the 100 people in your field all stumble over the same stupid little mistake and can spent their time thinking about the actual content.

Think of it as the type checking by the compiler or a code review by somebody on your team, NOT a papal announcement of unchangeable dogma.

dnautics3• 7 years ago

Depends on the journal, for example

https://en.m.wikipedia.org/wiki/Organic_Syntheses

mxcrossb• 7 years ago

I disagree. The gold standard is if outside groups subject your findings to new experiments and reproduce the same results.

tfha• 7 years ago

Well, there's replicating the experiment and then there's challenging the conclusion. Those are two separate ideas, and are actually fairly independent.

Fomite10• 7 years ago

I agree with you.

"I can run your code/experiment/etc. in your population and get your results" isn't the gold standard either. "We approached it from a different way, and with a different population, and we get a consistent answer" is both harder and more compelling.

tptacek6• 7 years ago

Just to clarify: this article is against gold standards, not against peer review. It concludes by observing that peer review is a necessity. The argument it raises is that peer review doesn't mean what lay audiences think it means.

I think they're clearly correct, and that even on HN, in a community that (roughly) prides itself on being scientifically literate, there are broad misunderstandings of what peer review means (during the bogus "Sokal Squared" hoax, for instance, many commenters implied that peer review prior to publication was meant to encompass replication). Also, while I'm not a "scientist", I've gotten to do some peer review work for ACM and Usenix, and even in the little bit of review I did, I seen some shit. There is much less formality and oversight to review than you might expect.

ZhuanXia• 7 years ago

Is there any proof peer review has increased scientific productivity? It is relatively new, and I don't see any reason prima facie why it would work better than an editorial teaming willing to delegate when necessary. Anecdotally, I see lots of academics complain about very important results being unable to make it through peer review. Ralph Merkel's early groundbreaking cryptography work was suppressed for years by unthinking peer review. From Merklel:

My first paper (and, in fact, the first paper) on public key cryptography was submitted in 1974 and initially rejected by an unknown "cryptography expert" because it was "...not in the main stream of present cryptography thinking...."

sokoloff• 7 years ago

Isn’t peer review at most journals done by “an editorial team delegating as necessary”?

I married into the field, didn’t work in it, but that’s what I’ve observed from the side.

michaelt• 7 years ago

  Is there any proof peer review has
  increased scientific productivity?
  It is relatively new

According to [1] "The council minutes of 1 March 1665 made provisions for the tract to be revised by members of the council of the Royal Society, providing the framework for peer review to eventually develop, becoming fully systematic as a process by the 1830s."

Peer review is a relatively new procedure in the same way that California is a relatively new state :-p

[1] https://en.wikipedia.org/wiki/Philosophical_Transactions_of_...

lonelappde5• 7 years ago

You are thinking of journal review, which hijacked peer review until arxiv and the internet in general rescued it.

nextos4• 7 years ago

You are totally right about peer review being misunderstood as replication. That's why preprints have gained so much traction lately. Things have become so complex that sometimes peer review is only good enough to filter results that are obviously wrong.

But replicating a proof or a complex experiment is way beyond what reviewers can accomplish.

Another problem with peer review, especially in experimental fields, are malicious reviewers. Big PIs typically gain control of a particular subfield by publishing something first, then getting lots of papers to review and blocking said papers and/or stealing good ideas from them.

kleiba• 7 years ago

Big PIs typically gain control of a particular subfield by publishing something first, then getting lots of papers to review and blocking said papers and/or stealing good ideas from them.

That's a bold claim, especially the "typically" part. How can you back it up?

lonelappde5• 7 years ago

Proofs aren't very hard to review, especially because math doesn't have the proprietary problems of wet science so drafts aren't hidden. Such is the benefit of working in a field no one can profit from :-) Math doesn't have a replication crisis; false papers are rare.

jedberg5• 7 years ago

Couldn't agree with you more. I too have done peer review, and the most disappointing thing in the world is when you point out something that is inaccurate and the response from the publisher is, "if we made it accurate there is too much nuance for the layperson to understand, so we'll keep it the way it is even though it's wrong because they'll get the gist of it".

petschge4• 7 years ago

There is a lot of frustrating things in being the reviewer, but "that would be too much nuance for a layperson" is not one I have encountered at any of the four journals I have reviewed for.

esyir• 7 years ago

Wait, when does this happen. In my experience, the sort of document undergoing peer review is rarely the sort to be read by laypeople to begin with.

apathy• 7 years ago

As a machine learning / probabilistic programming friend put it, (journal) peer review is boosting with three weak learners.

Better than nothing, but very far from ironclad. Only replication really verifies scientific findings. Everything else is just window dressing.

(I say this as a regular reviewer; for whatever reason, this particular week I’m reviewing for both Science and Nature.)

anm892• 7 years ago

What exactly was bogus about sokal squared? They made stuff up, they got it peer reviewed and they got it published. Yeah the journals weren't great but none of them were publication mills and not everything published gets published in Nature.

dnautics3• 7 years ago

I wouldn't consider nature to be necessarily so reputable either. If you want an almost unassailable example of peer review, check this out:

https://en.m.wikipedia.org/wiki/Organic_Syntheses

Fomite10• 7 years ago

There were a couple issues with the Sokal Squared hoax. The biggest one was that they relied a lot on their own definition of "obviously absurd" rather than using some form of empirical or unbiased "made stuff up" generator.

In the process, they wrote some papers that, especially if you weren't assuming the person made up the data underlying them, were...middling compelling.

I read one of the papers - the dog park one - and my conclusion was much the same as one of the peer reviewers. That it was an interesting concept, presented slightly too stridently (a common thing with papers written by say...students), and interesting enough as a frame to jump off from.

ChuckMcM• 7 years ago

Yup exactly correct. I've also been a reviewer from time to time on various ACM and IEEE papers and there are clearly authors who are trying to get reviewed by their 'friends' rather than by objective reviewers. And there is also reviewer biases, I reviewed a paper where one of the other reviewers was extremely harsh, and that reviewer turned out to have a stake in the outcome. Now a good journal is supposed to curate these sorts of conflicts of interest but my experience is that isn't always the case.

elcritch• 7 years ago

To be cynical, sorting through conflicts of interests would take more effort and Elsevier wouldn’t find that profitable. More realistically running an unbiased journal will always be difficult unless human nature changes significantly. Nevertheless it’s still worthwhile to do.

Same as you, I’ve had reviewers reject research based on a reviewer’s stake in the outcome, and also had good reviews with constructive feedback. As a reviewer I’ve also helped "tear apart" an article that had good results but was poorly structured to help them clean up their article.

tomlue3• 7 years ago

I'm a recent PhD graduate that focused heavily on publication and collected ~20 publications. I think the situation with peer review is worse than discussed here.

Peer review:

1. dramatically reduces the pace of progress

2. exacerbates publication bias

3. creates a false sense of accomplishment

4. creates a false meritocracy

5. creates many perverse incentives

(3) and (4) unfortunately wrap into grant financing as well. The merit of your research isn't measured on its impact on the human condition, but on it's 'impact' factor (a measure derived from your publications).

The gatekeeper problem here is pernicious as well. If you become a highly cited author your ability to get/maintain financing improves. You also become a gatekeeper as a peer reviewer. Which means that you are now strongly incentivized to accept papers from people who cite your work or align themselves with you, and reject everything else.

What is absolutely amazing here, is that the peer review process is opaque. It is my belief that if you knew who reviewed which papers you would quickly discover that mild to severe abuse of peer review is the norm, not action by a handful of bad actors. This is because the entire academic reward system is wrapped into the process. Getting your name on a big paper can have lifelong ramifications on your ability to get grants, start companies, do consulting work, etc.

Peer reviewers should probably get paid for the work. If they don't get paid then their incentive to do review must come from somewhere else, vague notions of improving the field don't cut it. Peer review should obviously be transparent. Some people might be uncomfortable signing their name to a paper rejection, but its time to get over that. A small payment might help reviewers overcome this discomfort. It is bizarre that peer reviewers don't get paid. Peer review is valuable work if done right, and without payment all the reward of being a peer reviewer comes for the wrong reasons.

Finally, I agree with some other comments. Publication should not be contingent on peer review, it should come first. This would increase the pace of progress, reduce publication bias, reduce the false meritocracy, reduce the ability for bad actors to censor research, and more. The cost would be a larger number of publications, but perhaps this would help people realize that many of the publications coming out right now are of little value.

Transparency is badly needed in many facets of academic research. My company made a site that helps bring transparency into literature review (sysrev.com).

sgillen2• 7 years ago

I totally agree with the transparency thing. I would love to see something like open review (https://openreview.net/forum?id=ryQu7f-RZ) become more common, and even have something like this replace the traditional peer review system.

We would all have to think long and hard about how to align incentives to get the desired results.As a first pass I think hiring some amount of people to review every paper, and then allowing anyone who wants to come by and leave their comments as well might be a place to start. I think some sort of reputation system similar to stack overflow might be a motivator as well.

Fomite10• 7 years ago

I've badly soured on open peer review, having had some experiences with it. I think you'll have very differential levels of review, and very differential values of how free people feel to go after authors.

I'd much prefer double-blind peer review.

cassowary372• 7 years ago

Those are all very real problems (and as you point out, they impact grant review as well). On the other hand, fully transparent peer review doesn't necessary address most of them. Reviewers who have to sign their reviews may be reluctant to anger colleagues (or try to curry favor with them).

As you presumably know a many journals have experimented with open peer review, but editors still need to police the reviews to look for bias. It solves some problems but creates others.

tomlue3• 7 years ago

transparent review isn't a perfect solution. But the problem you suggest of reluctance to anger colleagues or currying favor are worse under the current system than they would be under a transparent one.

Just because other journals failed in the past doesn't mean we shouldn't try again in the future! Maybe there were some mistakes we could learn from? Also the internet is becoming a more familiar tool used more often by more people every day, maybe it just wasn't the right time previously.

wtvanhest2• 7 years ago

I'm going to propose something that is unconventional today, but I believe will be standard practice at somepoint.

Overtime we have shown that many mistakes are made in labs and we know that many experiments are not replicable.

I think its time that we ask researchers using public funds to film their entire experiment from beginning to end. Every pipette, every prep etc.

Peer review should also have a video audit. Then a replication in another lab. The cost of ensuring results are what they say they are would be offset by less wasted research dollars on false rabbit holes.

sgillen2• 7 years ago

I do actually agree that we should apply more funding to replicating results. I disagree that a video log is an efficient way to ensure replicatability. In many disciplines this is not useful at all (math, high energy physics). And in most others, like your biology example, I think it adds an undue burden without much gain.

fwip• 7 years ago

What evidence do you have that suggests that the cost of doing a video audit (an expert carefully watching hundreds of hours of experimentation) and a replication study for every submitted article would be cheaper than the occasional paper that leads to a second experiment that doesn't pan out?

Nemo_bis7• 7 years ago

Are you trying to re-invent https://en.wikipedia.org/wiki/Reproducibility_Project ?

wtvanhest2• 7 years ago

No. Reproducibility requires double the investment and may also have the same or different mistakes. A video log would help researchers find errors and confirm results.

Video is simply a better solution.

dan-robertson2• 7 years ago

A curious thing that’s happened in some parts of mathematics is that formal peer review, needing coordination of journals, editors, and reviewers, can happen after many people have already read the paper (effectively informally peer reviewing it but with fewer comments), and even after people have cited the paper.

This peer review then seems to not serve much purpose.

An example is the recent first finite prime gap paper where the result was quickly strengthened several times by other authors long before the paper made it into any journal.

Another example here: https://mobile.twitter.com/adrianprw/status/1156534906618597...

mathgenius• 7 years ago

A 2015 study of 1,000 medical journal submissions found that of the papers that were eventually published, the 14 that became the most frequently cited were initially rejected. Groundbreaking studies by Sir Frank MacFarlane Burnet, Rosalind Yalow, Baruch Blumberg and others were rejected by peer reviewers, yet later led to Nobel Prizes.

Yup, if you are actually doing good work it is likely to be so different to what is going on that people have a hard time evaluating it.

dmurray• 7 years ago

If you are actually doing ground-breaking work. There's plenty of scope for doing good work that is boring and mainstream and still high quality enough to be published.

analog312• 7 years ago

Indeed, my work was of the latter type. It was rejected by Physical Review Letters but accepted by another journal.

One feature of peer review that's overlooked, is that there are different journals with different standards, and you can choose a journal that fits what you're trying to accomplish.

magicalhippo• 7 years ago

I can't read the article, but being rejected doesn't have to mean it's bad work.

My mom has helped many PhD students take their degree over the years, and in the process helped with their papers. Several have been rejected, some multiple times. Frequently it was down to presentation issues (the student had not explained the certain things clearly enough or similar), or there was disagreement about how strong conclusions one could draw from the results.

For example, recently a paper was rejected because the experimenters had forgotten or overlooked an important detail which meant they couldn't control for a certain variable. One of the reviewers picked this up, and rejected the paper because the findings could possibly be explained by this uncontrolled variable. So they had to resubmit with a weaker conclusion (this is when my mom got involved).

In this case the review process was harsh but fair. The experimenters had goofed, and the reviewer caught it.

Of course when doing ground breaking experiments, I guess the process might not be optimal. But the majority of scientists are not doing that.

dagw4• 7 years ago

The question is why where they rejected? My wife is a researcher and does a fair amount of peer review and sometimes she has to reject papers, not because their necessarily wrong, but because they're so badly written its hard to tell if they're right or wrong. Other times the research and paper is fine, but the conclusions are overstated, for example claiming that A is better than B, when all they've actually proven is that A is not worse than B. Other times it's great paper in every way, but covering a topic that really isn't relevant for that particular journal.

Basically there are lots of reasons to get rejected that have nothing to do with the quality or validity of the research.

Fomite10• 7 years ago

I recently rejected a paper as "This is fine, and really interesting, but no one who reads this journal is going to get it".

Theodores• 7 years ago

So you have to be a generation ahead of your time for your work to be truly enlightening. Rejection is therefore to be expected if your work is really good rather than something churned out. Science progresses one funeral at a time.

A simpler and more elegant solution is hard for the old guard to take in. You are sweeping away years of knowledge that they have built up about their earth-centred universe to propose an all new sun-centred solar system.

If you are in the position of having written some work that is ahead of its time then how do you know that? Rejection isn't good enough. The reasons why are important. If you have not presented your work neatly or used language that others don't like then that won't help. If you haven't talked about some of the smaller points that the learned people think are important about the topic then you could be done for. We need something more than 'rejection', failed for presentation reasons could be helpful so we can know if the jury is out of some 'rejected' ideas.

Nothing ever gets rejected for being twenty years ahead of its time. It would be helpful if this was an option.

scythe• 7 years ago

We don’t.

Peer review is a very helpful component of a system that slowly filters and refines academic work. It’s like the first air intake stage of a jet turbine.

cassowary372• 7 years ago

For everyone complaining about peer review, another alternative to open review is paid reviewers who focus on a particular topic. For example, I'm an editor at a journal where statistical reviewers are paid and evaluated, while the remaining (content) reviewers are not. This ensures that we have a rigorous review of the statistical methods, at least. I do think payment increases the reviewers' responsibility and conscientiousness, and the overall quality of the subsequent articles. (nb This probably has been tested somewhere.)

BUT - paid reviews are not feasible for many journals, especially open access/low-cost journals where margins may be thin. (Conversely, Elsevier should be able to pay their reviewers in gold bullion, rather than taking advantage of the scientific community's altruism, but that's another conversation).

jedberg5• 7 years ago

Somewhat tangential, but are there any journals that will only accept a paper if someone unrelated to the author can replicate the study? Or a journal of only replicated studies?

Nemo_bis7• 7 years ago

Sure. My favourite is IPOL Journal · Image Processing On Line https://www.ipol.im/, which is doing it since 2011: «IPOL is a research journal of image processing and image analysis which emphasizes the role of mathematics as a source for algorithm design and the reproducibility of the research. Each article contains a text on an algorithm and its source code, with an online demonstration facility and an archive of experiments. Text and source code are peer-reviewed and the demonstration is controlled. IPOL is an Open Science and Reproducible Research journal».

More recently, some megajournals have joined the trend and are trying to expand it, particularly eLife: https://elifesciences.org/labs/ad58f08d/introducing-elife-s-... (also covered in https://www.nature.com/articles/d41586-019-00724-7 ).

For some context see: * https://en.wikipedia.org/wiki/Replication_crisis * https://www.frontiersin.org/articles/10.3389/fninf.2017.0006...

neonate• 7 years ago

http://archive.is/UCSof

typon2• 7 years ago

I think the peer review system is ripe for, as SV people call it, "disruption".

Even something like an app that you submit papers to, which then anonymizes them, standardizes formatting and presents them to randomly selected experts for review in a certain time period would go a long way compared to the current process. In my lab at a prestigious Canadian university, the papers' authors names were visible as well as the reviewers names because any back and forth post review would happen through email. I think journals and universities need to buy into a system that standardizes this process so at least these basic requirements of impartiality can be met.

tastroder2• 7 years ago

While I'd apreciate a little fresh air in the process as well...

an app that you submit papers to, which then anonymizes them, standardizes formatting and presents them to randomly selected experts for review in a certain time period

Except for that it's not an app (that would be kind of pointless, why would I want that on a phone instead of a webapp), that's pretty much the standard in my field. Anonymization is done through the authors (if the conference/journal wants it to be) and ensured by the conference submission system. Formatting is standardized anyway.

The problem is the "randomly selected experts" part, for many (most?) subfields suitable reviewers are a scarce resource, to the point that in quite a few double-blind processes I've seen one or both parties could recognize each other's work or review.

I've always found openreview [1] a nice way to at least make the process public, especially when anonymity is an illusion anyway.

https://openreview.net/

typon2• 7 years ago

By app I meant software not a phone app.

Secondly, I think the problem of having few experts in the field is a legitimate problem but I don't think the status quo is a good solution either. I think it erodes trust in the scientific community which is very hard to build back up.

p1necone2• 7 years ago

If peer review in Academia is anything like code review in corporate software dev it's probably next to worthless.

jdnenej• 7 years ago

Peer review never seems to catch actual bugs but does give you helpful tips like "we already have a function that does this" or "the language has a feature for this"

p1necone2• 7 years ago

Oh code review done properly is really great - I'm just jaded by repeatedly seeing code go up and getting that green tick with no comments and knowing that the reviewer barely even attempted to read it.

chr12• 7 years ago

The whole scientific publishing field is still stuck in previous century. We desperately need a github for science, where any reader would be a reviwer, who would be able to rate the article and comment on it, and then journals would use these ratings combined with credibility rankings of reviewers to create indexes of interesting new article-repositories.

HNisCurated• 7 years ago

I have people citing research with useless AMA standard. If you want both parties to agree, they need to see the Data.

Most studies become uninteresting after you read their terrible data.

chr12• 7 years ago

Absolutely agree, articles without data and without the sourcecode, are not science, and that's another reason why we need github for science.

Nemo_bis7• 7 years ago

More on this idea: http://bjoern.brembs.net/2015/04/what-should-a-modern-scient... http://bjoern.brembs.net/2018/01/why-academic-journals-need-...

fastaguy88• 7 years ago

The idea that in scientific publication, more is better, makes little sense to me as a practicing scientist. The scientific literature is already filled with papers with conclusions that are not well supported by the data, and that is after 50-80% of submissions are rejected.

To argue that publication should have fewer restrictions, one needs to show that the modest number of papers that are improperly rejected by every journal (not just the first submission) equals or outnumbers the overwhelming number of submissions that should not be published because they reflect a misunderstanding of their field or a misinterpretation of the data. For many (most) scientists, the problem is not that important results are unpublished, the problem is that it is almost impossible to keep up with the current gated literature, particularly when that literature is full of mistakes. While it is certainly true that some reviewers are biased and some ill-informed, most of the time papers are rejected because the authors did not communicate well (for papers that should be published) or because the authors did not understand that their data did not support their conclusion (papers that should be rejected).

The scientific literature needs a better signal to noise ratio, not more noise.

Nemo_bis7• 7 years ago

The rejection of a submission does cancel a work, it merely shifts elsewhere. The question is whether peer review gives the right incentives; many suspect that publication venues with an ostensibly stricter peer review actually encourage the production of more noise, not less. See https://www.frontiersin.org/articles/10.3389/fnhum.2013.0029...

alexmlamb• 7 years ago

"Studies have shown that journal editors prefer reviewers of the same gender, that women are underrepresented in the peer review process, and that reviewers tend to be influenced by demographic factors like the author’s gender or institutional affiliation. "

Most venues are double blind, so reviewers wouldn't know about the latter things. Having fewer women in the reviewer pool is unfortunate, but it's hard to see how it would effect the mean review quality in a direct way assuming that men and women are equally good at reviewing (maybe in an indirect way it could harm reviews by increasing the load per-reviewer).

bmc7505• 7 years ago

Not sure if this is the case, but I'd like to play devils advocate. While most venues are double blind, it is still possible there is a subconscious gender bias. Many studies have shown that gender can be readily detected from anonymized authors based style alone.

https://www.aclweb.org/anthology/W11-0310

http://cs224d.stanford.edu/reports/BartleAric.pdf

https://www.sciencedirect.com/science/article/pii/S187705091...

xioxox• 7 years ago

Whether venues are double blind is extremely field dependent. In my field no journals are to my knowledge. However, many people use an initial rather than a first name, so it's difficult to assign gender unless you already know the person.

abdullahkhalids• 7 years ago

Depends on the field. In my field, I know pretty much the twenty groups who have the expertise to review my work and every paper of mine I could identify with high certainity from the referee's comment which group they belonged to.

known• 7 years ago

Is https://en.wikipedia.org/wiki/Wisdom_of_the_crowd better?

We shouldn’t take peer review as the ‘gold standard’

Related Stories