Giant, free index to world's research papers released online

https://www.nature.com/articles/d41586-021-02895-8

by webmaven • 5 years ago

477 121 5 years ago

"Malamud says that he did have to get copies of the 107 million articles referenced in the index in order to create it; he declined to say how,"

It's clearly Sci-Hub, as this 2019 article all-but-confirms:

https://www.nature.com/articles/d41586-019-02142-1

"And around the same time that he heard about the Rameshwari judgment, he had come into possession (he won’t say how) of eight hard drives containing millions of journal articles from Sci-Hub [...] Malamud began to wonder whether he could legally use the Sci-Hub drives to benefit Indian students [...] Asked directly whether some of the text-mining depot’s articles come from Sci-Hub, he said he wouldn’t comment"

(It'd be nice if there was a coverage source other than this scientific publisher, whose biases are obvious).

bane2• 5 years ago

Out of interest, is sci-hub generally available to get a copy of for...research purposes?

cinntaile• 5 years ago

There are torrents available. I'm unsure if I can link to the site directly without getting banned so you'll have to check the Gizmodo article for the link to the site containing the torrents. [0] It's missing some of the 2021 papers though, judging by the dates.

[0] https://gizmodo.com/archivists-want-to-make-sci-hub-un-censo...

AlexCoventry• 5 years ago

Taking you at your word that you're just asking out of interest, I hear (but haven't verified) that libgen mirrors scihub.

http://libgen.rs/dbdumps/

throwaway879080• 5 years ago

see this reddit thread on Scihub rescue mission https://www.reddit.com/r/DataHoarder/comments/nc27fv/rescue_...

there is a link there to 851 torrents , ~77TB of data (compressed zip files), and also an index (sql db dump)

vmception• 5 years ago

also, I want a version that lets me browse papers. Unlike people in academia or other research fields, I don't know the paper that I want to pirate yet.

from my lived experience, this request has been seen as absurd and invalidated by many because their lived experience is always having feeds of papers and abstracts they can start from. or how they see a browser extension that doesn’t do what I’m asking at all to be good enough (there’s some that automatically give you the scihub link to any paper you are looking at, which still puts me at square one, which paper?).

its a pretty basic feature though.

m0zg2• 5 years ago

Scihub has been a life saver for me once I started working on some of the more obscure areas of AI such as signal/time series processing. Anything off the beaten path is locked up behind paywalls, and I'm sorry I ain't paying $40 just to see if someone's paper sucks or not (which 95% of them do, in this particular niche, especially the ones shielded from scrutiny by paywalls).

liketochill• 5 years ago

95% of all papers I have read have sucked. Maybe they just weren’t what I was looking for but a lot of them I couldn’t believe got published as anything novel

wolverine8765• 5 years ago

Nature has provided straight coverage of Sci-Hub:

http://www.nature.com/news/pirate-research-paper-sites-play-...

As has Science:

http://www.sciencemag.org/news/2016/04/whos-downloading-pira...

bane2• 5 years ago

For n-gram indexes where n>1, I wonder if some of the sequence alignment techniques from bioinformatics could reconstruct the original documents?

anitil• 5 years ago

Exactly my thought. I haven't delved in to what's here but I suspect this copyright workaround is not meant to be foolproof

rmbyrro• 5 years ago

Also crossed my mind. We'd still would be left without illustrations, table organization, charts, etc, which in a considerable number of cases are of extreme relevance, though

marginalia_nu• 5 years ago

You would typically require some sort of filtering in an N-gram index to prevent the index from growing larger than the corpus. Maybe something like a tf-idf-threshold, or some other heuristic.

Otherwise you get a lot of useless "of that", "in an", "as a"-type word salad that is of no use to anyone, while what's actually useful sounds more like N-grams out of Tamarian.

Luisvel• 5 years ago

I was looking for a big and novel dataset to work on as a personal project. Can devote a couple months after work to it. What are interesting things that could be done with this now as a non-scientist, but that could be useful to share too?

markdown• 5 years ago

Create a search engine, and put it online. I'd love to search the index but can't download 7 terabytes of data to do so.

eigenman• 5 years ago

I would be interested in seeing how different branding terms evolve in the literature; e.g., "machine learning" vs "artificial intelligence" vs "neural net" or "surrogate model" vs "digital twin" vs "response surface". There a number of terms of art that have substantial overlap, but which term ends up being used depends on the audience, which often includes grant providers. I suspect the popularity of these terms evolves according to what terms appeal the most to funding agencies.

mandor• 5 years ago

Provide examples of uses of words and possibly help us rephrase in a more idiomatic way. Basically, answer "how do people use this word usually?"

A bit like linguee but for scientists: (at the bottom of the page): https://www.linguee.fr/francais-anglais/search?source=auto&q...

exikyut5• 5 years ago

An annoyingly broad answer, but: broadcast an opportunity in which you provide vetoed access to the server/instance holding the data (eg, with standard user accounts, private home directories and the data located read-only in a dir off of /). Would create a small storm of overhead ("please install ..." ad nauseum), but provide interesting opportunities for open-ended creativity.

Also, while unusual, you could potentially extract value by lightly imposing requests ;) on others to (where viable/a good fit) at least have a go at helping you with problem-solving/busywork-type steps in the research you're doing (however experimental/uncertain). Since everyone would be looking at the same dataset, this may bring interesting/unexpected ideas to the table (and maybe even shed light on potential opportunities for collaboration down the road). For individuals who are reasonably independent and self-directed but have no experience working with huge amounts of data, this would also provide a cool chance to play around in a real-world-scale environment without the failure risks attached to eg fulfilling^Wfiguring out business requirements etc.

(Now I'm reminded of https://tilde.club/, which operates a shared Linux box a (very) large bunch of people have collective access to. It's a stab in the dark (ie, the one reference I'm aware of), but maybe the admins there would have interesting insight about managing shared access Linux boxes.)

mindcrime3• 5 years ago

I was looking for a big and novel dataset to work on as a personal project

As an aside, one interesting data-set that is out there, legally and freely available, which is decent sized (not as big as this, of course) is the United States Fire Administration's NFIRS[1] data.

Roughly speaking, NFIRS is a record of all fire calls run in the USA, including details like date, time, location, responding department, incident type, dollar value of damages, number of deaths, number of injuries, etc.

I say "roughly speaking" because strictly speaking participation in NFIRS is voluntary and not every department in the USA participates. If memory serves correctly, for example, I think FDNY does not. But as far as I know, a significant majority of the fire departments in the US do send their data to NFIRS and so it becomes a pretty interesting snapshot of fire activity across the entire country.

Edit: from the NFIRS page:

The NFIRS database comprises about 75% of all reported fires that occur annually.

[1] https://www.usfa.fema.gov/nfirs/

minihat• 5 years ago

I wonder if this data could be used to index papers based on the type of mathematics they employ.

The same applied math turns up across domains - I would love to be able to see who else is using the same math I already know on their problems.

tomrod• 5 years ago

Brilliant. Let us dig in!

hashin• 5 years ago

That sounds super interesting! The results will depend upon the kind of information this database provides about each paper. We don't know yet the kind of processing each paper goes through and the quality of the "description" in the database. Assuming that it is good enough, this would be a really interesting project to work on.

amitlzkpa• 5 years ago

Sign me up for that!

pddpro• 5 years ago

This is an amazing idea !

lordnacho• 5 years ago

Hey this is an interesting idea. How would it be done though?

For example I've noticed certain similarities across classical statistics, Machine Learning and digital signal processing. But they all seem to come at it from different angles, and they often use different terminology. Sometimes a field will rediscover something discovered elsewhere a long time ago, eg the trapezium rule.

ausbah• 5 years ago

even worse, a field will take term from another field and misuse it enough such that "wrong" definition" becomes the default definition

Vinnl• 5 years ago

Slightly relevant classic: https://fliptomato.wordpress.com/2007/03/19/medical-research...

NmAmDa3• 5 years ago

Here we return to models based on LaTeX stuff and the lack of development in this OCR area to generate that index.

michaelmior• 5 years ago

One of my colleagues, Richard Zanibbi, leads the Document and Pattern Recognition Lab[0] here at the Rochester Institute of Technology. They're doing a lot of interesting work on formula recognition and search.

[0] https://www.cs.rit.edu/~dprl/

wolverine8765• 5 years ago

Where is "here"?

gopiandcode• 5 years ago

“We have seen some initiatives run into trouble, however, when the necessary rights have not been secured to enable their sustainability,”

I love the double-speak that these corrupt publishers use - it really highlights how mafia-esque the whole situation is. "Pay your protection fee, or we won't be able to /protect/ you". Intellectual slavery and copyright is just another tool of opression used to keep the wealthy in their positions of power.

charcircuit8• 5 years ago

Anyone can create IP. It's not something only the wealthy can do.

pugworthy5• 5 years ago

Indeed. This is my new strategy at the place I work, where I can't get acceptance of new interesting "a few years out there" ideas for our product line, but I can file for IP for such ideas. I'm actually encouraged to. It's a great outlet for the creative if you've got the organization to help out.

beckman4663• 5 years ago

Anyone can create IP. It's not something only the wealthy can do.

No. First you need privileged access to humanity’s techno-scientific inheritance. New discoveries and their negative research are increasingly being monopolized through trade secret protection (which, unlike patents, means that some discoveries can be forever monopolized).

https://www.ipwatchdog.com/2019/02/19/dont-fooled-patent-pur...

Yes I agree this isn’t the best source so if you have a good source that explains it better, please share?

ramblenode• 5 years ago

Anyone can create a billion dollar company or win a Nobel prize while we're at it. All technically true but largely meaningless for lack of nuance and perspective.

If you're an independent inventor you'll need to hire a patent attorney for your patent to have a meaningful chance, which will run several grand. The patent office is overwhelmed and as a result tends to reject the first draft of an application so that only the most tenacious filers stick around. So you'll spend several grand more on each of probably 3-5 iterations of the application. $10k is a realistic estimate for attorneys' fees if the process goes well.

Then you'll need to monetize it. If you want to sell a product you'll deal with all the other IP holders and their "patent thickets" that intersect with trivial aspects of your product. To sell your product you'll need to license their IP, eroding the value of your own.

Finally you'll have to defend your IP from infringement, which is a major expense---much larger than filing. If you threaten a company with infringement they will make it their strategy to grind you down with legal expenses. If it goes to litigation, you should expect to spend at least several hundred grand.

So overall I would not say just "anyone" has meaningful access to IP. As with many legal/bureaucratic processes, your mileage will vary with how deep your pockets are.

Here is a book written by two economists on how the current IP regime concentrates power and stifles innovation: http://www.dklevine.com/general/intellectual/againstfinal.ht...

tinus_hn3• 5 years ago

Creating IP is useless if you’re not wealthy enough to enforce its exclusivity.

dukeofdoom• 5 years ago

I would like to identify as a wealthy capitalist, where do I get my oppressed slaves to write patents for me? Monty Python skit voice

pugworthy5• 5 years ago

We aren't slaves, we are partners. Nor are we all startup leaders.

We have ideas, we don't necessarily have the time, money or even ability to capitalize on them directly as a business.

We also have no need to come up with ideas which have direct application right now. We can come up with "here's a good idea (maybe)"... stuff, and it really doesn't matter so much if it doesn't pan out.

We still get the bonus, the credit, and the feeling that our idea got recorded and acknowledged.

It's making lots of small bets to get ahead, rather than go for broke on one big one.

wyldfire2• 5 years ago

I wonder -- there's lots of machine learning designed for making inferences and there's some for language and transfer. How easy is it to train an oracle that you can ask questions? Or could you make an ontology? Seems like a wealth of scientific papers might be a neat thing to train on. Not to emit gibberish new papers, but to make something useful. Somehow.

jmnicholson5• 5 years ago

check out our work in this area: https://scite.ai/search/citations

We have extracted 918M citation statements from 27M full text articles.

wyldfire2• 5 years ago

This is interesting but AFAICT it is "merely" indexing.

dash2• 5 years ago

Back button is a bit broken there, if you want to back out of the login page.

jmnicholson5• 5 years ago

We (scite.ai) have extracted 918M citation statements (3 full sentences) from 27M full-text articles (more than half are from paywalled articles from indexing agreements).

We've taken this an extra step and have classified the citation statements as providing supporting or contrasting evidence (https://direct.mit.edu/qss/article/doi/10.1162/qss_a_00146/1...)

And, recently, have made these citation statements easily searchable to find expert analyses and opinions on nearly any topic extracted from the literature: https://scite.ai/search/citations

pedalpete• 5 years ago

Nice work on Scite, I'm not sure if this is a different use case than I have, but my searches are listing duplicates of the same papers many times. Is there a reason to not collapse duplicates into a single result?

jmnicholson5• 5 years ago

Is this on Citation Statement search? I think it is probably because the citation context contains two or more citations in it. We look at citations per sentence so it is duplicated there. I can see how that is confusing though.

a3_nm• 5 years ago

This looks very nice! is it possible to download the dataset?

(PS: comparable initiative of citation datasets that are downloadable: S2ORC https://github.com/allenai/s2orc and Internet Archive Refcat https://blog.archive.org/2021/10/19/internet-archive-release...)

jmnicholson5• 5 years ago

We can't release the full dataset as our licensing agreements with publishers restrict it. We do have an API though that can be used: https://api.scite.ai/docs

kragen• 5 years ago

I wonder if this project is what Carl's friend Aaron Swartz was working on that he got prosecuted for until he committed suicide.

2Gkashmiri2• 5 years ago

Scihub is what 100 tb ? That is like inr 80,000 + a server to hold drives say another 30,000 so in total around 110,000 or 120,000. That is around USD 1500. If I get a windfall of sorts, this is the thing I am going to spend on.

Donations, eh yeah but I feel like scihub needs everyone's support.

exikyut5• 5 years ago

I happened to check Amazon a few minutes ago while writing another comment and found 18TB drives seem to be hovering around US$350. 6*18=108TB (and US$2100), however, that does not account for redundancy.

I recently learned the fun way that ZFS performance (incl read-only) absolutely tanks if the pool is full (the box was unusable until I deleted the data), so to use ZFS you'd want to add in a few TB of spare capacity. https://wintelguy.com/zfs-calc.pl says 10 drives ($3500) should provide 100TB of space.

Alternatively you could use dm-integrity (which is very new and does not yet have optimized I/O routines) and either layer RAID on top of that or manually duplicate the data yourself, which would let you just get exactly the number of disks. It's possible you could alternatively run Ceph with ECC, but I don't know if Ceph likes having some free space to work with.

This also does not take into account a server, but this does not need to be too expensive, probably US$750 or potentially even less.

I respond entirely out of curiosity; I also dream of backing up large datasets one day :D

2Gkashmiri2• 5 years ago

my argument is just to provide redundancy. we do not need additional local redundancy because the data is out there. it would be a problem if my copy or your copy somehow becomes the only copy. then we would be concerned.

https://www.amazon.in/Western-Digital-DigitalBBGB0060HBK-BES...

this says INR 11,000.00 which is $137.raw 100 would need around 18 so $2500.

i think there can be variations of disk sizes and amazon fluctuations but in cases like these i would have a business buy them because that way input tax of 18% would pass on to the business and conversely to me for lowering the final bill. so, $2118 would be for these drives. same for the server. a tax benefit of 18%.

as i said, i would love to have this

wolverine8765• 5 years ago

How does this differ from Google Scholar's search index, or JSTOR's? Is it that it's open?

allturtles• 5 years ago

From the article:

And although free search engines such as Google Scholar have — with publishers’ agreement — indexed the text of paywalled literature, they only allow users to search with certain types of text queries, and restrict automated searching. That doesn’t allow large-scale computerized analysis using more specialized searches, Malamud says.

ant6n• 5 years ago

Can this new index used for search. A quick glance makes it seem like a huge list of ngrams, which isn't exactly easy to with with.

6510• 5 years ago

We need a generic effort to take out old ideas that bother all but a few people.

modernerd• 5 years ago

Carl Malamud's Aaron Swartz memorial speech is relevant and moving:

https://www.youtube.com/watch?v=VllJDnMcTzM

ozymandias12• 5 years ago

Thanks for this. I also noticed the bottom thanks on the index:

In_memoriam Shamnad Basheer ; Aaron Swartz

wolverine8765• 5 years ago

Looking around, I see Shamnad Basheer was found dead in a car. Is there evidence of how that happened?

noiwillnot3• 5 years ago

How people goes about downloading/mirroring sci-hub? I assume you cannot start torrenting from your home in Texas without getting into problems. Do you get a seedbox? You are in a state like China or Russia?

Getting these > 70 TB to a residential address does not seems easy.

krageon3• 5 years ago

If you are serious: you can download it to a seedbox, connect to it and download it to your residential address over scp. The storage is a solved problem, not cheap but then you are trying to store a significant chunk of all papers :)

d110af5ccf• 5 years ago

download it to a seedbox

Or if you already have a VPN provider, check their terms to see what their torrent policies are. Last I checked mine permitted such traffic in a few specific datacenters.

exikyut5• 5 years ago

That solves the routing problem, but not the pipe size problem.

70TB over 1Gbps: 6+ days

70TB over 500Mbps: 12+ days

70TB over 100Mbps: 2+ months

70TB over 50Mbps: 4+ months

70TB over 25Mbps: 8+ months

70TB over 10Mbps: 21+ months

70TB over 5Mbps: 3 years

70TB over 1Mbps: 17 years

70TB via mail: equivalent to 100Mbps (international) to 1Gbps+ (local) depending on distance, but only cost-effective if an equivalent connection is unavailable due to average weight (and price) of sufficient HDDs, eg, 3x18TB = ~$1200, combined weight approx 1kg.

fbanon• 5 years ago

I assume you cannot start torrenting from your home in Texas without getting into problems.

Why?

noiwillnot3• 5 years ago

Legal threats from some IP holders. I assume that the situation with the other part of libgen (the book one) should be similar.

neurobot1232• 5 years ago

Get a VPN

brainwipe• 5 years ago

This is awesome; did a quick ego search and didn't find mine in there but found some references to it. Nonetheless, an excellent resource!

heinrichhartman• 5 years ago

How about learning a language model on ~~the SciHub texts~~ some large private repository of articles, that can auto-complete a paper from the title ...

tasogare• 5 years ago

There is enough irrelevant and dumb papers written by humans not to add more generated artificially.

plg• 5 years ago

Is there a search interface?

capableweb• 5 years ago

Direct link: https://archive.org/details/GeneralIndex

If you want to perform searches against the index: https://archive.org/search.php?query=creator%3A%22Public+Res...

Great to see a large "Signatories to the Declaration of Support" and that it's hosted by the Internet Archive. Keep fighting the good fights my friends! :)

johnnybaptist• 5 years ago

Any advice on how to search against this index? I can't seem to figure out the right search queries, everything I try gives no results.

For example, I'm trying to find articles with "neuroscience" or "brain-computer interface" among the keywords.

IshKebab• 5 years ago

This doesn't look especially useful. You can search for small sentence fragments and get the paper title I guess? But you can already do that on Google Scholar.

You can't read the papers or see their citations or cited-bys. What's it for? Poking Elsevier?

someonewhocar3s• 5 years ago

Still can't construe it as ethical or right to prevent access to scientific papers. There's something about universities pooling cash for publishing, but whatever's happening is something else.

RNCTX• 5 years ago

Reminder that these journal publishers have egregious and exploitive contracts with universities, and should not be commended for turning any blind eyes to such databases.

The state university I attended publishes their subscription expenditures, link to the document here [1].

Besides the outrageous prices the other thing that jumps off the pages is how many of them have managed to hide the amount of tax money they receive from a state institution with contract terms.

1. https://digital.library.unt.edu/ark:/67531/metadc1132742/

Giant, free index to world's research papers released online

Related Stories