Peer Community in X: A free recommendation process of scientific papers

by eternalban • 9 years ago

74 18 9 years ago

fiatjaf2• 9 years ago

Best scientific initiative of the entire history.

Just saying it because I had this idea 4 months ago (not that I'm claiming ownership of the idea, since it's a pretty simple idea that probably was thought of by thousands) and presented it to a friend which is a scientist (I'm not, thank you, God) and he promptly dismissed it.

nmca• 9 years ago

I really, really hope this takes off. Current system is so flawed...

phreeza• 9 years ago

Interesting. Romain Brette has been doing something similar by himself for computational neuroscience (and other subjects that catch his interest) http://journal.romainbrette.fr/

jessriedel2• 9 years ago

ArXiv-overlay journals like Quantum seem like a strictly better approach.

http://quantum-journal.org

eternalban• 9 years ago

Stated end goal of Quantum:

"The point of “becoming an attractive journal for researchers in every stage of their career” is crucial: Quantum should become an attractive publishing venue for the whole community of researchers in the field, including those who support Quantum’s vision but who, due to external constraints, cannot afford to take the risk of not publishing in a high-impact journal. Arguably this can be achieved in one of two ways: either the whole incentive system is overturned, or Quantum must become comparable to high-impact journals for the purposes of career advancement. In the short and medium term, only one of these options seems realistic. The decision to create Quantum as a selective journal is to a good extent based on the belief that it will give Quantum a much higher chance of making a difference." [source: http://quantum-journal.org/should-quantum-be-selective/]

I think it's an apples and oranges case given one (PCiX's) is a platform and the other (Q's) is a journal. It also seems to me that PCiX's goals are explicitly aligned with facilitating the sharing of results in a trust/reputation network, whereas Quantum seeks to address additional e.g career issues as well (which can limit it to more conservative decisions.)

jessriedel2• 9 years ago

As plainly written, that's a case of Quantum actually trying to consider real-world constraints, not a different end goal. It demonstrates seriousness.

dibanez• 9 years ago

One of the problems with the current peer review system is the ease with which a reviewer can recommend rejecting a manuscript based on personal conflicts of interest (for example, viewing the work as competitive with their own). Making reviews open, and possibly de-anonymizing them, could alleviate this (allowing the public to review the review itself). Since the review requires the manuscript itself as context, the manuscript then needs to become publicly visible regardless of the reviewers' decisions.

Another problem is the difficulty that editors have in finding reviewers (to work for free essentially). One solution could be to require authors to act as reviewers before they can submit their own work. For example, if a manuscript requires N favorable reviews to be accepted, then for each manuscript an author submits, they must provide N reviews of other manuscripts.

There is still something missing from the above in terms of "reviewing a review". The closest thing I can think of is comment threads where the post is the manuscript... There needs to be a way for low-quality reviews to be reported as such and invalidated. Right now all this is on the shoulders of the editor, who doesn't have time to do more than add up the recommendations of the reviewers.

chrisseaton2• 9 years ago

to work for free essentially

I don't understand this. If you are an academic, or in industry R&D, then it's part of your job to take part in the community - you are being paid for it.

jorgemf2• 9 years ago

You pay to publish in a journal, you pay to have access to the papers of the journal and you review for the journal for free. Do you see the problem? You review for free but pay to access the content and to publish.

Note: the journal is not related with the people who pay you to do research.

carbocation3• 9 years ago

Is there a reason to think this would be better than just computing the pagerank of every scientific paper based on the citation networks? (Optionally, use signals to predict future pagerank of recently published papers.)

I would expect that using the citation data from published papers would be more valuable, in the same way that revealed preferences are more valuable than polls due to the cost.

While Google Scholar implicitly uses pagerank to rank academic papers, I've put together a tool that explicitly shows you pagerank for each paper, available at [1]. Currently, this contains only pubmed, which means that it does leave out a significant set of physics/computer science papers, and also doesn't yet capture preprint archives.

1 = https://pubrank.carbocation.com/

SubiculumCode2• 9 years ago

To discuss how a paper is wrong you have to first cite the paper. The need to discuss bad papers can arise for a number of reasons. So some papers cite rank != quality.

carbocation3• 9 years ago

True. Pagerank is "importance" rather than "quality". But as your network of recommenders becomes larger and starts to approximate the scientific community as a whole, I'd imagine that there will be convergence. If not, that would be a welcome and (to me) surprising result.

SubiculumCode2• 9 years ago

Curious. Just looked up my papers, most published in the last few years. How does the pubrank take in account time, or does it?

carbocation3• 9 years ago

Pubrank currently is just pagerank based on the citation network. The common way to take time into account is to compute, for example, a "pagerank at 2 years". There are other ways to adjust for time (from linear modeling to more complex approaches) that I'd like to explore.

On average, for reasons that are not clear to me but which emerge from the data, pagerank seems to stabilize around 8 years after publication, for most publications.

fiatjaf2• 9 years ago

I expected other people to suggest here, or maybe come up with a link of, a system totally decentralized of peer-reviews.

veddox• 9 years ago

I like the idea, but I am afraid that as long as there are no "big names" behind it, it isn't going to make much headway in the conservative world of science. As tough as it sounds: reputation is the currency of science, and PCiX needs to get some before it can become a viable publishing alternative to the established journals.

Peer Community in X: A free recommendation process of scientific papers

Related Stories