Arxiv.org reaches a milestone and a reckoning

https://www.scientificamerican.com/article/arxiv-org-reaches-a-milestone-and-a-reckoning/

by pseudolus • 5 years ago

157 86 5 years ago

I legit don’t get how come there isn’t some government funded publication platform. No other endeavor has as much bang for buck. Arxiv needs $2.5m a year https://arxiv.org/about/reports-financials which is chump change.

thrwn_frthr_awy3• 5 years ago

I also don't understand how there can so much money to throw at startups and not one person to "adpot-a-highway" for a year.

ahelwer• 5 years ago

Because you cannot rely on the rich to fund common goods through voluntary donations. It is not a good idea and never will be. Other than some medical research, charity has never solved a single social ill.

zarzavat• 5 years ago

Much of the money for startups comes ultimately from pension funds. They have a fiduciary duty to make sure the money is invested in something that is expected to make a return.

manquer2• 5 years ago

Arxiv is a indirect threat to publications. Publications lobby the government and also influence university grants and other bodies.

We still don't have a proper open /free tax filing too from the IRS or tax computed by IRS only for us to verify/confirm as in some of other countries, primarily because intuit (and others) lobbied to make sure the government would never develop one (encoded in law no less), so it is not that surprising that a community benefiting project that threatens some big business revenues has no government funding available.

onion2k2• 5 years ago

Arxiv is a indirect threat to publications.

It's incredibly annoying that publications try to continue to exist in order to act as a gatekeeper to exclusive papers. If journals added real value by peer-reviewing, collating, curating, offering editorial opinion etc then the fact papers are also available in the raw state somewhere else wouldn't much of a threat.

IAmEveryone• 5 years ago

If you mean some government should fund arXiv, specifically, then I don't disagree but would guess the government's default position of "it exists without public money, and we think that's rather excellent".

If you're suggesting it as alternative for academic publishing in general, it would cost a lot more.

Arxiv isn't a "publication platform" in the sense that peer-reviewed journals are. The cost of hosting and distributing PDFs is close to zero, obviously. What does cost money is the copyediting and running the peer review process.

People think Elsevier are the worst people in the world (not entirely wrong) and taking in millions while exploiting the free labor of authors and reviewers. But it's relatively easy to quantify the value they do add: their financials are public and, last I checked, they have operating margins of about 30 %.

Let's say we could do without their marketing and billing departments and that gets us to a margin of 50 %. That's pretty good, but it isn't quite the rip-off people sometimes suggest it is. Replacing the commercial publishers with government-funded open-access journals would be a billion-dollar endeavor, not a few million for servers and a bit of software.

eru• 5 years ago

Perhaps it's best for governments not to get involved in arxiv. They are heavily involved in most other aspects of academia, and the results haven't been universally pretty.

erwincoumans• 5 years ago

Interestingly, the peers doing the hard work (of actual reviewing) for free.

Running the reviewing process can also happen through https://openreview.net

Not sure why we need Elsevier exactly, could you be more specific?

tsbischof2• 5 years ago

Part of the challenge is the reputation issue. Academics want to publish where there is prestige, which means there has to be a perception of the venue as good. This typically either comes from reasonable gatekeeping and high-quality peers (e.g. arXiv) or pushing metrics (e.g. Nature Publishing Group journals). Generating an attractive journal is non-trivial and few people with the appropriate knowledge and expertise take the plunge would do so without some guarantee of profit.

Vinnl• 5 years ago

Academics want to publish where there is prestige

Important to note that this is not some innate "want"; their careers depend on it. It's a means to an ends, the ends being being able to keep practicing science.

(Of course prestige-by-proxy will play a role as well, but it's not what's keeping the system in place.)

mananaysiempre2• 5 years ago

Funding for government-funded archival projects tends to begin with a bang but gradually fall through the cracks due to the project becoming politically unexciting, frequently with all data just completely lost one day with no warning, so I’d be disinclined to trust any such initiative. The French government made an attempt with HAL[1], but given how INRIA just unceremoniously deleted the software repository Gforge with no backups or redirects (and broke every GCC build out there, because the GCC dependency ISL was hosted on it[2]), I’m not sure it’ll survive once the nationalistic fervor is exhausted. Besides, a country-specific repository seems to be strictly worse than a country-neutral one.

On the other hand, PubMed exists.

My guess is, for a preprint archive, network effects are paramount, and ArXiv was there first. It’s not directly a government project, but is still funded by US government grants and hosted by Stanford (now that the international mirror network is shut down, which does give me a slightly jittery feeling).

[1] https://hal.archives-ouvertes.fr/

[2] https://groups.google.com/g/isl-development/c/JGaMo2VUu_8

goodcanadian• 5 years ago

The Canadian Astronomy Data Centre is a good counter example of a successful government funded archival project:

https://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/en/

It probably helps that it is run under the auspices of the Herzberg Institute of Astrophysics.

lN5uhiNxWPBi• 5 years ago

There is, right? In many cases around the world.

Here in the UK, there is a big research evaluation exercise called the REF. The REF is important because it largely determines how the government allocates funds. In order to be eligible to be evaluated, researchers have to essentially submit their pre-prints into open access.

What this has meant for the last 10+ years is that most universities participating in the REF have an e-prints server, like the arxiv. Obviously, being publically funded, universities are then using government money to pay for these costs.

Now your question might be whether it's sensible for the entire country (or research councils) to operate their own e-prints server. I don't know the answer to this.

All I know is that the more bureaucracy you involve, the more bullshit results.

henrydark2• 5 years ago

The two sites I start my day with: hacker news and arxiv.

gmadsen• 5 years ago

same

onion2k2• 5 years ago

Why have you never submitted an article you've found on Arxiv to HN? Are you not finding anything interesting there? (Based on https://news.ycombinator.com/submitted?id=henrydark ... I guess you could use a different account?)

hasjekyll5• 5 years ago

I do a similar thing, but the papers I'm interested in are usually a bit too niche for a general audience. That's one of the things I don't like about academic papers, particularly in my field (astro), we don't tailor our abstracts and introductions to a wider audience.

buro9• 5 years ago

Do you read the /new submissions for particular topics?

HN is a nice tidy list of top things, is there an equivalent for ArXiv?

hasjekyll5• 5 years ago

You can use things like arxivsorter and benty-fields for this. You give them a list of papers you've published, or authors you are interested in and then get recommendations. The recommendations can be up and down voted to improve future recommendations.

gsvclass• 5 years ago

https://42papers.com

telxosser2• 5 years ago

arXiv advanced search is great.

I just have a few bookmarked searches for what I am interested in.

I don't see how you could generalize things though across all papers. Like randomly clicking on what was submitted to High Energy Physics - Lattice yesterday. I don't have the first clue what any of that is and that would be the same for most topics for most people I imagine.

henrydark2• 5 years ago

Yup, I read the daily abstracts of Representation Theory (math.RT), Number Theory (math.NT) and Programming Languages (cs.PL)

HerrMonnezza2• 5 years ago

From the article:

But moderators frequently intervene, delaying posting by days or weeks, reclassifying papers or even outright rejecting submissions.

Definitely not my experience. I haven't posted to arXiv in a while now, but I wasn't even aware a moderation queue exists -- none of my papers were delayed, reclassified, or other...

EDIT: I see the article provides a few examples, but then they also provide numbers: "At arXiv, roughly 6 percent of submissions receive a hold, and about 2 percent are rejected." -- 8% of heavy-handed moderation intervention wouldn't qualify as "frequently" IMHO.

hasjekyll5• 5 years ago

I guess it just stops arxiv from turning into vixra.org

domenicrosati• 5 years ago

Just anecdotally, Ive had a paper rejected once from arxiv. I am glad they did... it was not a very good one. Every other time thought has been fine

prof-dr-ir• 5 years ago

The arxiv is a treasure, and such preprint servers not only streamline the academic debate but are also a necessary first step towards a future without for-profit academic publishers.

The arxiv moderators are like human spam filters. And as anyone who dealt with email should know, spam filters are essential for a platform to remain useful. I think the article puts undue emphasis on the few edge cases that invariably arise with any such filtering system.

And being a moderator is a thankless job indeed. The efforts of a moderator are largely focused on the absolute worst papers, and the job is therefore very different from being a referee or editor of a reasonable academic journal. So personally I would also focus elsewhere any efforts at increasing diversity in academic institutions.

_def• 5 years ago

I'm in no way involved in academic research, and I don't now how it works in detail, but a question that popped into my head these days was: is there a place where people can publish papers even without any association with anything? Like, if I manage to do proper research, even without any qualifications on paper, would it still be useful? Does this make any sense?

elcomet• 5 years ago

You can submit your work to any conference that is double-blind and get accepted if your paper is good. You don't need to be affiliated to a university.

tsbischof2• 5 years ago

In principle, submission is viable at many venues. PNAS has a few mechanisms for having established authors endorse your submission, for example. The main challenge is going to be the appearance of credibility to get past the starting point, since many non-affiliated submissions tend to be unpublishable without serious rework, at least in lab-based disciplines.

SiempreViernes2• 5 years ago

Good research is always useful, where good means something between "reliable and flawlessly explained" and "plausible and very novel".

So on one end you could takes something like Newtons Principa, that synthesised a lot of the known facts and laws at the time into one single coherent framework, and on the other you can have Galileo recognising that he was seeing surface features on the moon and not just patterns on a flat disk and announcing this will some illustrations showing the principle and not what he actually saw.

As to the first part of the question, if you talk of "publishing" in academia you usually mean you want your work to be know by other credible researchers in the hope they find it useful.

The most reliable way to achieve that without credentials is to befriend someone respected in the field you are interested in before you send them your results, because they are more likely to have patience with any errors or confusion in your presentation that way.

There are of course some places that pride themselves on being completely free from filtering, but the concept of "spam folder" along should tell you how good a chance you have of getting any serious attention in those places.

lynndotpy• 5 years ago

As others have said, you can post to academic conferences. But each field has different peculiarities and patterns. You would start by doing your research, finding a relevant conference, and writing a paper to match their style.

A more approachable avenue could be blogposts and videos. One good example of a scientific blogpost I recall is here: ( https://dynomight.net/2020/12/15/some-real-data-on-a-DIY-box... ) Someone tested very simple air purifiers with limited experiments. It follows the same "abstract-methods-results-discussion-conclusion" format you might expect. Some obvious caveats, this data could have all been faked, and it isn't peer reviewed for methodological flaws. But you can reproduce the experiments yourself for ~$150.

I am in academia but I like blogposts a lot more. I wish they were more popular. I think HTML makes more sense than PDF, and to some degree they cut the middleman out between communicating to peers and communicating to other academics.

YeGoblynQueenne7• 5 years ago

What you lose on arxiv is peer review. That might sound like a feature- you get to let others know of your results faster- but if you make a mistake it will not be caught until it's out in the wild (and possibly until everyone has already cited it and based their own work on it). With peer-review you know that someone will read your work that has the background to understand it and to catch errors.

So what, you'll say, independent experts can catch your mistake if they read the paper on arxiv and contact you to suggest a correction. Yes, but there's no incentive to do the hard work to correct mistakes if your paper is already on arxiv and people are already citing it (who didn't do their due dilligence). Then errors start cascading. Science is self-correcting, sure, but it's much easier to avoid errors in the first place. That's an important function of modern peer review.

I'm speaking from experience, of course. Peer review has certainly helped improve my work. I didn't enjoy having to redo the work, but the end result was something I could be confident about. Also, I'm only speaking about publishing in journals, because peer review in conferences is a very different beast. In my field, of machine learning and artificial intelligence research, I'd go as far as to say that peer review in conferences is broken, getting published or not is a lottery and you're better off putting your stuff on arxiv: less hassle and more people will read it. You might even see your article posted on HN. When was the last time a paper published on a conference server was posted on HN? See?

EGreg• 5 years ago

Well, this is exactly why we should have more WikiNews and less Twitter. I explore these issues in far more depth here:

https://www.youtube.com/watch?v=gv5mI6ClPGc

(Discussion w Noam Chomsky)

IshKebab• 5 years ago

With peer-review you know that someone will read your work that has the background to understand it and to catch errors.

That's overly optimistic. What you mean is "with peer-review you know that someone will at least skim your work and maybe catch errors."

I agree with the rest of your points though. Zero peer review is probably not a great solution.

marginalia_nu3• 5 years ago

Honestly the esteem of peer-review does not hold up to the reality of peer-review. It's elevated to this gold standard for quality science, but in practice, it's just not. Not only is some of the real landmarks in science older than peer review as we know it today (until the middle of 20th century, articles were reviewed by an editor), in several fields, 50-90% of peer-reviewed studies fail to replicate.

It's very possible to just keep submitting papers to journals until you get someone who shrugs and lets it through. Reviewers aren't paid, and while reviewing is encouraged it basically does bupkis to further your career (and academia is very cut-throat).

SiempreViernes2• 5 years ago

There's actually some incentive: you get to feel smart by telling someone else is wrong. But it is indeed not a very strong one, and there's plenty of other places to act smart on the internet where you'll actually have an audience too!

hasjekyll5• 5 years ago

Agreed. Peer-review is essential. I do like the idea of a public peer review however, similar to that in the https://joss.theoj.org/ - I had a really pleasant experience reviewing a paper for JOSS.

I think sometimes reviewers hide behind their anonymity to give snarky responses, and using their power over you, knowing you will try your very best to implement all of their suggestions in order to publish faster. I just think the whole process should be more transparent.

atomack• 5 years ago

The way we used arxiv worked well in physics, though this is 15 years ago now so might have changed since.

arxiv was about distribution. It didn't replace peer review - articles were still submitted to journals and published there too.

If an article was posted to arxiv and not a journal, the odds of a citation went down massively. And the journal it was submitted to was a factor in whether or not we read it. When articles were eventually published, most authors also updated the preprint with the post peer review version.

Basically it meant that (1) it was easy to keep up to date with what everyone was working on, and pick up interesting new stuff (2) most citations, post 80s, you saw in whatever paper you were reading, you could look up on arxiv and be reading it in seconds.

kgwgk2• 5 years ago

arxiv was about distribution. It didn't replace peer review - articles were still submitted to journals and published there too.

I'm surprised that they no longer use the term "preprint" at all, at least it's nowhere to be found on the homepage or "about" section.

The consequences of this amnesia are hilarious: https://twitter.com/gustavnilsonne/status/138948729731431219...

Why do we call it "preprints"? The term seems to imply that work is preliminary or unfinished. As far as I can tell, the term introduced by @arxiv , the first online repository for scientific manuscripts, is "e-print". Is "preprint" a marketing device invented by publishers?

YeGoblynQueenne7• 5 years ago

This is not the case in computer science and particularly machine learning, especially in recent years. You'll find many papers where a majority of references are to preprints that stay preprints for ever. You'll also find many papers that have hundreds of citations, all while remaining preprints forever (and many of those citations are from forever-preprints themselves).

In machine learning, for the most part, arxiv is used to avoid peer-review. Or a way to "publish" work that has been rejected by a peer-reviewed publication, of course.

And to be more cynical, it's also a convenient source of references to pad up a Related Work section and make it look like incremental work is part of a growing body of groundbreaking new work. /jaded

Edit: well, I'm not just being cynical. The fact that everyone can put their half-baked papers on arxiv means that the 90% of work that is crap, per Sturgeon's Law, is now a much bigger quantity than ever before and one must sift through reams and reams of crap before finding work that has any meaningful results to report. Again, that's the case in machine learning specifically. I don't know about other fields.

currymj• 5 years ago

In ML at least, the arXiv version is the “canonical” one. This is because the conferences have onerous page limits, so the official conference version is usually mangled with lots of cut content. The arXiv will have content restored and be more readable.

In theory you should read arXiv and cite the conference version, but often people cite arXiv and nobody cares because google scholar mostly combines things properly.

HerrMonnezza2• 5 years ago

What you lose on arxiv is peer review.

Or, instead, you get a much larger audience reviewing your article, instead of the 2 to 4 reviewers involved in the paper publishing process :-)

You can get comments from critical and interested readers, you upload a new version of the paper to arXiv, and repeat the process until the article is ready for submission/publishing.

In other words: I think that an article on arXiv (on average and as a whole, individual articles may be exceptional) is subject to much wider and more extensive scrutiny and verification than the average article during the paper publication process.

YeGoblynQueenne7• 5 years ago

Well, if all one wants is an "audience" one can simply report the results of their research on twitter. Much bigger audience. But that's not the point, is it? The point is that you want someone who knows their shit, to review your paper, and who will have both the knowledge and the motivation to engage with your work critically and help you see what is behind all those big blind spots we all have for our own work. I'm sure this happens with papers put on arxiv, occasionally, but my expectation is that just because the paper is on arxiv most people will not be that interested in making a serious effort to review it, and that it is much more likely that such a serious effort will be made by a reviewer in a peer-reviewd venue, particularly one of the top journals (conference reviews are a mess).

So, yeah, I disagree. The level of scrutiny one gets from putting their work on arxiv doesn't compare with peer review by experts in one's field.

indigochill• 5 years ago

Borrowing from software engineering, you can publish code to a git repository and someone else can make a pull/merge request with a proposal to make fixes/improvements. There's still work involved in reviewing, revising, and merging those proposals, but it can be less work than needing to make every change yourself.

I wonder if academic papers could benefit from a similar process of communal development (though it flies in the face of the dead-tree format of academic papers, but we're talking about Arxiv here, not Elsevier). Even if suggestions don't get merged, having a repository that captures discussion around the central artifact is also useful (such as when a maintainer decides that a feature doesn't belong in a project).

nuclearnice1• 5 years ago

Some of the sentiment of your last sentence is embodied in open reviews.

https://openreview.net/group?id=ICLR.cc/2022/Conference&refe...

dandanua• 5 years ago

but there's no incentive to do the hard work to correct mistakes if your paper is already on arxiv and people are already citing it

Peer reviewing is essential, of course. But note that in most journals reviewers are not getting paid. They have even less incentives to find mistakes in your work than you are. I have no idea how such an utterly broken system emerged, where journals siphon billions of tax money exploiting the work of scientists.

YeGoblynQueenne7• 5 years ago

The incentive to freely review others' work is that others will also freely review your work.

This is quite separate than the exploitative aspects of the big scientific publishers. Reviewers are certainly exploited by publishers, because publishers make money from reviewers' free work. But reviewers would do the work for free anyway because that's what they expect others to do.

thereddaikon• 5 years ago

Peer review is important but lets be sure to not conflate it with the established way it operates. There is nothing that says peer review can only happen through submitting to an established Journal that conducts a traditional peer review process. We already know that process isn't perfect so there is no harm in looking to improve or replace that. Does that mean ArXiv as it is works as a replacement? No, and you make some good points why it doesn't.

time_to_smile• 5 years ago

With peer-review you know that someone will read your work that has the background to understand it and to catch errors.

This is a remarkably naive (and charmingly optimistic) view of peer-review.

For the vast majority of the history of academic research peer review did not play a major part. It wasn't until the 1970s that the modern day peer review system emerged and it did so as a reaction to a funding crisis in academia to produce the illusion of legitimacy.

It of course has done no such thing. It has done nothing to limit the reproducibility crisis which has shown it's ugly head in nearly all scientific fields of study. One could even argue that peer review coupled with a publish or perish culture is the cause of such a failure of science.

This also means just about any great scientific discovery you can think of prior to 1970 did not go through the peer review process as we know it today. If science seems less exciting today, peer review is certainly one of the reasons for this. I'll leave you with Geoffrey Hinton's words on the subject:

Now if you send in a paper that has a radically new idea, there's no chance in hell it will get accepted, because it's going to get some junior reviewer who doesn't understand it. Or it’s going to get a senior reviewer who's trying to review too many papers and doesn't understand it first time round and assumes it must be nonsense. Anything that makes the brain hurt is not going to get accepted. And I think that's really bad.

Laking peering review is a feature, and not just because it speeds up time to results.

YeGoblynQueenne7• 5 years ago

> This is a remarkably naive (and charmingly optimistic) view of peer-review.

Great way to join a convesation. Well done.

I didn't say anything about "the vast majority of history of academic research". I talked about my experience of peer review of my work. Are you responding to someone else's comment but quoting mine?

simplestats2• 5 years ago

I'm only speaking about publishing in journals, because peer review in conferences is a very different beast. In my field, of machine learning and artificial intelligence research, I'd go as far as to say that peer review in conferences is broken, getting published or not is a lottery and you're better off putting your stuff on arxiv: less hassle and more people will read it.

The situation for conferences in CS is basically how it works in journals in other fields. There may be more low-quality submissions to journals, but at the same time, the extreme competitiveness means most high-quality submissions are rejected also. The "solution" has been the appearance of open-access journals which are less concerned with subjective reasons for rejection like "impact", and scale to accept more papers. But they cost thousands of dollars. Open archives are a more democratic solution.

throwaway81523• 5 years ago

I wonder if anyone remembers Advogato and its certification/reputation system. It claimed to be very difficult to subvert. I'm sure that was naive and it held up because nobody back then was that motivated to game it, but even today, a little human oversight could probably steady it. So I wonder if something like it could be used for arxiv, relieving some of the endorser/moderation stuff that they have now.

raphlinus• 5 years ago

Yes, I bet you could do something interesting there, but one of the lessons is that social graphs can become very noisy, especially if people have an incentive to add edges. Don't expect a purely technical (reputation) system to solve what is a social/cultural problem.

politician• 5 years ago

TLDR: 6% of submissions receive a hold, 2% are rejected. The problem, the article says, is that only 13% of the moderators are women and there is not enough diversity of nationality.

My opinion here, but it sounds like instead of a 92% immediate acceptance rate, the author would prefer an immediate acceptance rate closer to 100%. I’m not sure why diversity would get them there, but I am encouraged that the current moderators are able to produce 92% immediate acceptance rate. On the other hand, I am distraught by the suggestion that a highly diverse moderator base would presumably only account for a few percent reduction in holds and rejections. It’s my understanding that that should have a far more significant impact.

Arxiv.org reaches a milestone and a reckoning

Related Stories