Sci-Hub statistics and database

by bluish29 • 4 years ago

547 127 4 years ago

p1esk4• 4 years ago

Interesting. Medical field dominates research in terms of publications. Chemistry produces double the papers compared to physics, and humanities are smaller than biology but larger than physics. I wonder where machine learning papers fit in - CS or Math or both?

sgillen• 4 years ago

I would imagine it depends on the particular paper, the more experimental ones in CS, the more theory ones in math.

Do note though that most math and ML practitioners use arxiv over sci-hub.

remuskaos3• 4 years ago

It is note worthy that most of physics (at least high energy physics) ist published on arxiv.org and open access.

I don't know if sci hub bothers with publications that are available freely from an official source.

p1esk4• 4 years ago

Good point. If so, this data is a lot less interesting :(

philipkglass• 4 years ago

Sci-hub will grab and serve anything with a DOI (or at least used to; I don't know if they have started ingesting papers again after turning it off a while ago). I have found open access papers there before. It's simpler to just paste the DOI into sci-hub than to check to see if it's one of the few open access articles in a mostly paywalled journal.

anon_123g9873• 4 years ago

published on arxiv.org and open access

Don't use the term "open access" like this. A paper published on arXiv is free to read, and was freely published. "Open access" is a scam by the big publishers, where they don't take money from the readers, but make the authors pay. Or, putting it another way, anyone can pay their way in those journals and publish (sometimes sub par) papers.

The_rationalist2• 4 years ago

Machine learning publication rate is small, at least by assuming that paperswithcode contains most of the publications.

mmettler• 4 years ago

Alexandra should get the Nobel prize.

allisdust• 4 years ago

With the rent seeking companies being from Europe? Not a chance.

Nobel is a political tool that's mostly there to make a point (especially that peace prize).

anon_123g9873• 4 years ago

He said "should", not "will". Both of you are right.

The_rationalist2• 4 years ago

How come we don't have extensive software for helping doctor decision making by making use e.g of bayesian inference while feeding on the available superintelligence that enable those 24 millions paper? Expert systems long passed the hype curve and it's time for them to cycle up again!

nefitty3• 4 years ago

I think Watson does something like this.

roywiggins• 4 years ago

It didn't seem to actually work though.

https://slate.com/technology/2022/01/ibm-watson-health-failu...

f6v2• 4 years ago

Because research can be controversial. There’re papers in my field saying patients have increased frequency of certain cells. There’re other papers saying they’re not. Go figure.

Qem3• 4 years ago

Nailed it. With publish or perish incentivizing shenanigans like "p-hacking", many of those papers are the research-equivalent of spam.

kilburn• 4 years ago

An older comment of mine https://news.ycombinator.com/item?id=30049522 fits well here. I'll adapt it to your question ;)

Basically: medicine as a whole is already some sort of expert system.

- Data collection and cleanup: Researchers conduct experiments to produce meaningful data and extract conclusions from that data.

This part isn't more automated because we have strict rules that prevent medical data collection and analysis without a clear purpose. Otherwise we'd be able to collect a lot more information to try and extract results from it using more inference-oriented techniques (deep learning and the like).

- Modeling & training: Expert panels produce guidelines from the results of that research. These panels are the "training part" of the system.

As a sibling comment said, replacing these panels with ML-based techniques isn't trivial because the data produced in the previous step is fairly noisy (p-value hacking, difficulty of capturing all the variables, etc.). Furthermore, the techniques that yield best results nowadays also produce them without clear explanations on why they hold, which is not something we are prepare to accept in medicine.

- Execution: Doctors diagnose and treat following said guidelines. In fact, they use decision flows that they themselves call... algorithms!

The main reason why execution is not automated is that we do not have the technology for machines to capture the contextual and communication nuances that doctors pick up on. There can be a world of difference between the exact same statement given by two different patients or even the same patient in two different situations. Likewise, the effect of a doctors' statement can be quite literally the opposite depending on who the patient is and their state of mind. One of the most important aspects of the GP's job is to handle these differences to achieve the best possible outcomes for their patients.

All that being said, there are companies trying to produce expert systems to help doctors diagnose. See https://infermedica.com/product/infermedica-api for instance.

pmoriarty2• 4 years ago

It's interesting how sci-hub's papers on medicine dwarf those in many other fields like comp-sci, math, and physics. I wonder if that reflects the number of papers in those fields, or if sci-hub just has a non-representative sample. If the latter, why?

p1esk4• 4 years ago

It does appear to be the latter. I just searched for several famous ML papers (attention is all you need, lottery ticket hypothesis, capsules, etc) and they are not there. I think if someone counted all papers that have been ever published anywhere, the picture would be a lot different.

pmoriarty2• 4 years ago

So does that mean that vastly more people in medicine use sci-hub than do people from other fields?

Or is there some other reason for the discrepancy?

Qem3• 4 years ago

I guess today is much easier to find new noteworthy, publishable facts in medicine than physics. New diseases are discovered every year, and old diseases are poorly understood (e.g Alzheimer disease), and the treatments for many of them are still sub-optimal, or even inexistent. Every patient is different, individual cases are research-worth. We only got antibiotics in the 1940s. On the other hand, most big breakthroughs of physics happened between the 17th century and the first decades of the 20th century. After the general case is cracked in physics, individual cases have very little publishing value.

lucb1e4• 4 years ago

much easier to find new noteworthy, publishable facts in medicine than physics

Plus I would imagine that everyone wants their illness looked into, thus that's where funding tends to go. I care much less for physicists to figure out what dark matter is than how to treat health problem X that bothers me daily. (Just an example. In my particular case I'm healthy and would actually be quite excited about dark matter findings compared to any individual illness solution... but still.)

This comment would be worth a lot more with some stats about funding going towards the different fields, though. Not sure where to find that.

_Wintermute• 4 years ago

I think it's due to the sheer number of biomedical papers published each year, coupled with comp-sci, maths and physics papers being less likely to be behind a paywall.

octoberfranklin• 4 years ago

... which is due to the sheer amount of cash being dumped into medical research. Because congressmen get cancer.

There are several medical subfields that on their own get more funding than the entire NSF (i.e. all other science).

pessimizer• 4 years ago

I suspect that a lot of ill people, and people with loved ones who are ill, spend an enormous amount of time reading every paper they can find on the illness, or trying to narrow symptoms down to a particular illness. That's also a situation where you can go through 50 papers easily for every useful one (or even intelligible one, if you're a layman) you find, so a definite sci-hub situation unless you're independently wealthy. It's the requests that draw (or once drew, right now I guess) stuff into the database.

agumonkey• 4 years ago

I don't even know the scale of medical research vs CS in terms of number of schools.

abliefern• 4 years ago

The answer is right there on the page: the database covers >95% of publications among major publishers.

yewenjie• 4 years ago

Math and comp-sci researchers use https://arxiv.org heavily and thus people usually do not need something like sci-hub.

Relevant xkcd - https://xkcd.com/2085/

nefitty3• 4 years ago

Here's a notebook that fetches Sci-Hub mirrors from Wikidata and tests them. I also included an iOS Shortcut to add to your Share screen. When you're on a site that Sci-Hub recognizes and you use the shortcut it will try to fetch the paper.

https://observablehq.com/@iz/sci-hub

ok123456• 4 years ago

What's the most popular paper on all of scihub? By field?

2Gkashmiri3• 4 years ago

i asked this question here and at many places before. why do people "rely" on an organization that sifts through hundreds of thousands of papers and then charge exorbitant prices for providing this service? if we use the amazon analogy, is amazon with millions of products worse than a boutique cat food seller that specializes in a specific cat food for a specific cat breed? maybe. but what about the "rest" of products?

why are our scientists made to rely on elsevier et al to sift through the junk and find for them the perfect paper instead of doing it themselves? is science now such a cutthroat quick competition that it requires you to give a company the priviledge to work for you so that you dont have to do your own due diligence?

in india, we have a lot of local research that is done on open databases like shodh ganga and many more. but if you have to access foreign research material, better luck your university has an agreement with elsevier and others to pay them millions for a login. the alternative, go to scihub and find what you need.

i understand the whole quality/delivery debate but doesnt the average user already know who the big players in the specific domain are and who are trusted? or you want discoverability at the hands of a "trusted third party" without doing the legwork yourself.

then at the other end you have non-academics like me. I might have heard of a research paper in some article and i cannot read it without paying an arm and a leg. why? if we use the whole ebook/book argument that compensation is commensurate to the sales so more popular book means more money to the author but here authors arent compensated but elsevier so why should i pay elsevier? because they filtered through 1000 papers to provide 10 and for that privilege, they require unlimited royalty for ever? why?

slater• 4 years ago

There's some truth to "publish or perish". Scientists are expected to publish in prestigious journals.

Qem3• 4 years ago

Not only expected, but actually forced. In many places, a streak of a few years with no publications in prestigious journals can unrecoverably sink a researcher career.

OmicronCeti4• 4 years ago

A typical PhD dissertation these days is 3 publications in high-quality journals. It is explicitly required at most schools.

f6v2• 4 years ago

Because otherwise you’ll have to sift through tons of garbage “research”. It’s already a common knowledge that many articles coming from certain countries are fraudulent. There’s a lower chance of having those in journals like Nature Medicine.

Hendrikto• 4 years ago

why are our scientists made to rely on elsevier et al to sift through the junk and find for them the perfect paper instead of doing it themselves?

Scientists do do that themselves. That‘s why it is called peer review. Journals take scientists work for free, they just pre-select papers, but don‘t do the review.

2Gkashmiri3• 4 years ago

are you saying you rely on "pre-selection" of the journals and for that the demand exorbitant prices? why dont you rely on peer-review data on scihub itself?

is the pre-selection such an important thing that these companies necessarily have to remain in business?

belter• 4 years ago

It looks like the torrents have all subjects. Anybody aware if there are torrents only for Math or Comp.sci ?

modeless• 4 years ago

Did Sci-Hub start working again? Last time I checked it wasn't adding new papers because of some legal thing going on in India.

phoe-krk• 4 years ago

Yes - see https://twitter.com/ringo_ring/status/1492419986291408898

Ansil8498• 4 years ago

The site is working in the sense of you can download old papers, but I don't believe any new papers from the last year are accessible.

lamontcg• 4 years ago

AFAIK its not really working again? I think there was an upload of a bunch of papers in a batch recently, but not ones that I was hoping for. I'm sort of worried about the past-tense language in this page suggesting that it isn't starting back up again.

aurizon8• 4 years ago

If only the Nobel Committee would say:- The Nobel Committee will only consider research published under an Open Source Access repository in reviewing published papers for consideration for the Nobel Prize after ~~ June 30, 2022. This would unleash a horde of hungry cats among those fat pigeons that are the paywalled journals. There would be a crying and wailing - ending with piles of feathers,(and purring cats), and researchers all over the world, and especially in the many 'third world' Universities whose minds are currently held hostage to budgets and local politics. The world would gain immeasurably by this simple act!

Ansil8498• 4 years ago

Is there any word about when sci hub is going to start adding new articles again? It's currently only useful as an archive of old research articles. New papers from the last year are not available. I never understood the rationale for stopping new content, though I believe it had some relation to some court case in India...but I don't understand why that was a reason to stop adding articles, and why it hasn't been restarted yet.

joshuaissac2• 4 years ago

They had resumed adding articles after receiving legal advice that the Delhi High Court injunction only applied for a few months.

https://mobile.twitter.com/ringo_ring/status/143435621720862...

Ansil8498• 4 years ago

I saw that tweet, but it doesn't change the material reality: try plugging in some DOIs from recent article from the last year, and they will not be there.

Scihub used to be a great resource, now it's only a resource for old research. Still useful for background material, but not for current work.

I also don't understand why the Indian court case has any impact on new article availability. The owner is not Indian. The servers and domains are not Indian. There doesn't seem to be any actual reason to stop adding new articles, other than some idiotic halfbaked point that only hurts the people who need the articles, like when Project Gutenberg banned anyone from a German IP, except this is much worse since there is no way around it for people who need new papers.

derbOac2• 4 years ago

What I read was that the Indian judicial system tends to be favorable to things like Sci Hub in its interpretation of copyright, and Sci Hub wanted to act in good faith with regard to that court, so as to have a fairly solid basis in international law for operating, should it rule in Sci Hub's favor. I might be off in this understanding, but that's what I understood.

Ansil8498• 4 years ago

Yeah, I have heard this reasoning, but it seems muddled. How is keeping the site online so old articles are available but no new articles are added acting in "good faith"? It's not like the old articles are any less copyrighted than the new articles, so this doesn't make sense to me.

The court case has also been delayed for over a year now, so if it is delayed indefinitely, like it seems to be, then we will also not get access to new articles, also indefinitely? That's ridiculous. The last update from the court proceedings claimed that there would be a new update over a month ago, which in turn got delayed yet again to a few days ago, and there's been nothing [1].

[1] https://delhihighcourt.nic.in/dhc_case_status_oj_list.asp?pn...

gw67• 4 years ago

How they are able to store data without being seized?

gumby2• 4 years ago

100 TB is pretty small. I wonder if she will start torrenting it so people can back it up and share the load.

logifail• 4 years ago

100 TB is pretty small. I wonder if she will start torrenting it so people can back it up and share the load

This has been ongoing for a while now:

Rescue Mission for Sci-Hub and Open Science: We are the library https://www.reddit.com/r/DataHoarder/comments/nc27fv/rescue_...

gumby2• 4 years ago

Excellent, thanks!

beambot• 4 years ago

IPFS or ARWeave...?

raziel2701• 4 years ago

Alexandra Elbakyan is a titan and a saint. I couldn't have been able to finish my research without access to papers my institution wasn't subscribed to.

OmicronCeti4• 4 years ago

I snuck her into my own dissertation acknowledgements: https://imgur.com/bDgtBAE

jdrc• 4 years ago

Now i feel foolish for not acknowledging her, especially in elsevier papers.

ahmed_ds2• 4 years ago

Isn't it fantastic that we are alive and seeing the resurrection of the great library of Alexandria right before our eyes?

She has done more than any other organization or individual in the history of mankind when helping people in second and third world country pursue advance research since the advent of internet. Well she and the people who pirate and distribute MS Office. Faculties around the world recommend scihub as the main and only source of research and journals.

userbinator• 4 years ago

The great library of Alexandra, you mean.

intunderflow• 4 years ago

Remember to donate to sci-hub to keep it going! Even a small donation helps and is way more than the extortionate prices we'd all have to pay without it :D

derbOac2• 4 years ago

I have to be honest and say this is the one time crypto has made a lot of sense to me.

KyleLewis• 4 years ago

I hope more disciplines can shift the publishing culture towards the norm found in physics, where arxiv is the go to. I'm not sure why that is the case but it's pretty great.

pxtail• 4 years ago

It's kind of ironic twist of fate and disgrace that something like SciHub needs to be hosted/served from Russia in order to avoid getting squished and destroyed by rent seekers greedy hands.

gdy2• 4 years ago

If anything is a disgrace it's your russophobic comment.

Alexandra Elbakyan, who created SciHub, was born is Kazakhstan (exUSSR) and later studied in Russia where she lives now. I'm proud that Russia is one of the few countries on the planet where American exterritorial law doesn't apply.

trasz3• 4 years ago

I'm not sure the preceding comment was actually rusophobic. To me it seems you're mostly in agreement in that this content needs to be hosted somewhere safe from the US. It's just that you are proud - because it's hosted in your country - and the previous person is annoyed, because it cannot be hosted in theirs.

aurizon8• 4 years ago

The publishers now encode their papers with individual identifiers, that generationP calls UUIDs on all pdf's. That means they can trace it back to the institution(and perhaps the actual prof?) They have then sent nastygrams to threaten them with fees or loss of access to the institution - potentially serious punishment. What is needed is a way to run the papers through an OCR recognition program to create renewed text and to further process to randomly vary a large number of adjacent letter spacings. (this is called 'micro kerning' and it allowed a second order of unique document recognition where the document is scanned by the journal to look for these fractional kerning gaps in the letter spacing which can lead back to the institution). I suppose a program flow could be made to OCR + random micro kerning changes - it would take time, but once set up it would be a rapid computer based document flow process. Photos/charts could be sanitised as well, but legends etc on them would also need OCR and micro kerning adjustments. With 25 words of text, each with 6 letters, micro kerning can easily create 10,000 unique ID's, easily enough to cope with all subscribed institutions - and that is on one chart/photo. I suspect the subscribing institutions have acted with firm words to their profs and grad students to block this. This can easily be told to us if a few people in assorted institutions let us know if they have been read firm words about this?

I am not sure how Sci-hub can get past this, unless they get a good Indian court ruling and can use Indian friends to scan printed copies of journals - if they exist in this online age?

robwwilliams2• 4 years ago

No “firm words” that I heard in my neck of the USA woods.

I was struck by very low number of German downloads. Did I miss something?

aurizon8• 4 years ago

No idea about Germany, but they may well be more law abiding?

lucb1e4• 4 years ago

Makes me wonder how much they'd have to offer me to accept the task of implementing such tracking algorithms to modify other people's scientific papers for the purpose. Certainly I won't be the cheapest, but still. Either some developer is very vested in the idea of keeping science a secret or someone got a very nice bonus.

Edit: also the sysadmin that keeps this database safe without 'accidental' data loss on UUID to downloader mappings.

aurizon8• 4 years ago

It seems to me that this type of tracking, under the guise of guarding rights is against the GDPR = unlawful tracking that they are harassing Google and FB with? Is not justice's sword 2 edged?

hda2• 4 years ago

I am not sure how Sci-hub can get past this

Quite trivially, actually, thanks to the good old analogue hole: https://news.ycombinator.com/item?id=30084193

All sci-hub would have to do in this case is download the same paper through three or more accounts (different institutions, networks, countries?) at three different times, rasterize them and keep the common denominator. If a pixel has no common denominator, they'd have to fall back to a default value. This is by no means a perfect method and it has its weaknesses and pitfalls, but the resulting PDF will be far less useful as a means to de-anonymize accounts using information from the PDF itself.

Publishers still have other sources of information to de-anonymize accounts if the multiple accounts/downloads aren't truly isolated from one another.

aurizon8• 4 years ago

Yes, that would mush their secret sauce...

pishpash• 4 years ago

Do a reflow of the entire document? As long as it's semantically recognizable it can be done. Figures are harder. ML can really help here: someone make a pdf2tex!

trasz3• 4 years ago

Does this mean it is trivially easy for any student or employee to make their institution stop paying publishers by getting it banned?

aurizon8• 4 years ago

LOL, student sabotage!. I think the paywalls would extract a penance that maintains their cash flow - remember, Greed is their mantra, may it never ever cease.

sockaddr• 4 years ago

Anyone notice the logo update? The key loop has been exchanged for a hammer and sickle.

lucb1e4• 4 years ago

I did not, that's somewhat interesting (not hard to explain if you read the about pages, though). It's also well within the domain of politics and that seems to be something HN generally avoids.

dankle• 4 years ago

Getting 403 Forbidden. Did my isp block it, or it this an error on their end?

periheli0n5• 4 years ago

Sci-hub is sometimes the last resort to obtain a resource that is otherwise unobtainable. But what has become of the old way of obtaining unaccessible papers: Ask the authors for a copy?

Sites like ResearchGate make this very easy. And often a simple email does the job, too.

Advantages:

* It is legal

* The author gets feedback that someone out there reads their research

* Making direct contact to your peers is a good thing.

abcc8• 4 years ago

It's too time consuming and has an undefined likelihood of success. People will naturally flock to alternative methods, such as sci-hub, that are faster and until recently were near guaranteed to have the desired content.

periheli0n5• 4 years ago

Agreed, sci-hub is so much more convenient. But when the publishers finally shut it down for good, we'll have to find another solution.

A community of scientists sharing their papers would be a good thing already now.

I personally know active scientists who don't even try anymore to look up the paper, but rather go directly to sci-hub for any doi they need. I can understand why, but I also think that this doesn't lead to a sustainable publishing culture.

nottorp• 4 years ago

It is legal

Are you sure that the usual suspects don't make authors assign copyright or at least distribution rights? I wouldn't put it past them...

periheli0n5• 4 years ago

I have signed quite a few of those copyright transfer forms, and there are always a clause that would allow sharing on a personal basis.

yellowcake02• 4 years ago

Except a researcher could easily need to skim 30+ papers in a day, this is not a solution.

periheli0n5• 4 years ago

Agreed, in that case that wouldn't work.

But honestly, how often does one have to skim that many papers in a day, to a level where the freely available abstract is not sufficient?

Perhaps every once in a while when one compiles a survey of a new field they enter. Once the project is set on the rails, one rarely has to read that much.

machiaweliczny• 4 years ago

It's shame that US or EU don't run open DB for journals that every university should be able to publish as back-up. Also they could create some tools to make research more productive that's free to use.

That we have to rely on single person effort that's risking their freedom to do so is fucking insane.

Sci-Hub statistics and database

Related Stories