Sci-Bay: Google Scholar plus Sci-Hub

by mycoborea • 8 years ago

466 106 8 years ago

Just tested it by searching for “spline”. This is great! Can someone elaborate on how it was made? Specifically, how it integrates with Google Scholar. Is that done client side or server side? If server side how come it hasn’t been blocked by Google seeing as Google don’t seem to like robots using their regular search function so my guess would be they wouldn’t like it for Scholar either. Perhaps it is proxying requests and would also pass on any CAPTCHAs presented? Still in that case I would expect all requests to get hit with a CAPTCHA. Perhaps it just hasn’t had enough traffic yet?

k5hp• 8 years ago

Interesting that Hacker News allows links to pirate sites... This will probably get removed soon

veridies• 8 years ago

I wouldn't think Hacker News would be in any trouble with this one. The link is to a noteworthy site, not to any infringing content; you'd have to take action and use a specific search term to find copyrighted content. And the implication of posting it is clearly that people should find it interesting and discuss it, not that they should use it to download articles.

zeth___• 8 years ago

And it's not like searching in google scholar, getting the doi and putting it in scihub was any great hardship before.

petra• 8 years ago

How did you integrate scholar in sci-bay ? Does scholar have an API ?

And about the future, how do you see google responding?

shakna3• 8 years ago

I don't know how they are doing it, but Google Scholar does not have an API, and scraping is against their TOS.

Don’t misuse our Services. For example, don’t interfere with our Services or try to access them using a method other than the interface and the instructions that we provide.

Despite this, there is scholar.py [0], which can extract files from Google Scholar, though it explicitly doesn't work around the rate limits.

[0] https://github.com/ckreibich/scholar.py

crispyporkbites• 8 years ago

Http is an interface with implicit instructions (especially if restful), provided by google

userbinator2• 8 years ago

or try to access them using a method other than the interface

Unless this actually exploits something and hacks into Google's servers to get to the content, which would be something quite different, it wouldn't really be distinguishable from someone manually visiting the site in a browser, volume aside.

IMHO the pervasive attitude today of somehow requiring permission or an explicitly sanctioned "API" to access what is otherwise publicly accessible data is rather troubling for the freedom and flexibility of the Web as a whole. It encourages walled-garden content models and centralisation.

jrochkind13• 8 years ago

Google Scholar definitely and intentionally offers no API.

I don't see this lasting long...

mchannon• 8 years ago

google.com/scholar doesn’t work for you?

wyldfire• 8 years ago

The issue is regarding: this service (Sci-Bay) depends on Google Scholar, yet there's no public API for Google Scholar that it could leverage. If it's scraping Google Scholar results, then it's likely a ToS violation and unlikely to last long.

gpm2• 8 years ago

At a glance it looks like it's really just a proxy, that was limited to scholar.google.com and mutates the page slightly (adds a header, sci-hub links).

Does google generally block proxy servers?

jrochkind13• 8 years ago

I don't know if they generally do, but I'm sure they can/will if they want to.

Spivak• 8 years ago

They don't block startpage and they've been around for a while.

Vinnl5• 8 years ago

I'd imagine that would be quite hard: many university libraries have their own proxies which make sure that visitors to the library are able to access the content that the library has paid for, and often modifies Google Scholar (with their cooperation, I believe) to list links to accessible versions of the content next to search results.

danielecook• 8 years ago

Why is that? Seems like it would be really beneficial to the scientific community.

shakna3• 8 years ago

They likely don't have legal permission to allow for third-party access to the data they provide.

matheusmoreira2• 8 years ago

The page's HTML is the API. It's pretty easy to download a web page, parse the HTML and then extract specific bits of information from it. The browser does the same thing on the user's behalf, which is why it is called the user agent.

hrasyid2• 8 years ago

An API is a contract. HTML can be tweaked and become incompatible with your parser at the developer's whim.

jrochkind13• 8 years ago

Home page currently says:

See you later
Too much attention is a bad thing, Sci-Bay decides to stop service for a while. Sorry.

Apparently I was not wrong.

This could be developed as a browser plugin that would be much harder or almost impossible for Google to prevent. Well, a Firefox browser plugin, a Chrome browser plugin presumably they wouldn't allow.

jmnicholson• 8 years ago

How is this any better than using the sci-hub plugin?

agumonkey• 8 years ago

reminder that this went up not long ago https://whereisscihub.herokuapp.com

bcaa7f3a8bbc• 8 years ago

If sci-hub is going to scrape publishers, they could put in a bit more effort.

+1, especially for the Onion site. Onion service supposed to be a primary mean to host uncensored websites instead of having to look for the latest domain name everyday, unfortunately it seems nobody cares about it. Most of the time I access it from my browser, the front-end proxy was malfunctioning, or the back-end Tor daemon has dead... Tor network itself do have capacity problem, but they could do much better than a broken front-end proxy... e.g. with Onion Balance.

Vinnl5• 8 years ago

Thanks for sharing; note that I've moved that to a somewhat simpler URL: https://whereisscihub.now.sh/

I should also add that I am also working on a project to incentivising authors to make their work freely available: https://flockademic.com/

(More info here: https://medium.com/p/the-holy-grail-in-open-access-sharing-t... )

Aelius• 8 years ago

Sci hub seems like the perfect candidate for ipfs, I'm astounded the mirrors haven't implementated that yet.

eruci• 8 years ago

It does not work.

tomrod• 8 years ago

Oh, this is wonderful!

fwgwgwgch3• 8 years ago

Can someone help dispel my ethical concerns over using papers like this? Eg sci-hub.

Any and all arguments, on both sides are very welcome.

lvs• 8 years ago

Well, the research and the researchers are all almost entirely funded with public funds from around the world. The fact that they are forced for professional reasons to funnel all their work into a for-profit publishing industry for distribution is a historical anachronism that dies hard.

In a broad sense, there's only very weak analogy to music or film piracy in terms of the process of creative work. Everyone has already paid for scientists to do this science. Scientists and institutions have even paid the journals to publish their work in almost all cases, and they won't reap any reward if you read the work through the legitimate distribution channel. You only pay the publisher, not a starving researcher.

Personally, I'd feel very happy if someone read my papers by any means necessary. That was the whole point of writing them, setting aside the practical annoyance of simply trying to keep my career afloat.

Edit: The lifetime value of a single paper to the journal can actually be priced, since they started charging researchers for open access a few years ago (often in addition to other fees). This number is usually in the $2-5K range per paper, in my field.

theptip3• 8 years ago

A timely request:

http://slatestarcodex.com/2018/03/19/the-dark-rule-utilitari...

"So this is my argument that Sci-Hub can be ethical. Universalized it would destroy the system – but the system is bad and needs to be destroyed. And although this would break the law, a very slight amount of law-breaking might be a beneficial solution to inadequate equilibria that could be endorsed even when universalized."

lsh3• 8 years ago

but the system is bad and needs to be destroyed

The system needs to be reformed, and that is currently happening, peacefully. Perhaps not everywhere at once and perhaps not as quickly as you might prefer, but open access publishing has made great strides in just a few short years.

This doesn't liberate the millions (probably) of academic articles whose authors relinquished their copyright to big publishers, but it is resulting in new tensions like Germany and Elsevier butting heads (https://www.nature.com/articles/d41586-018-00093-7) and SciHub and exciting new legislation

No destruction or 'very slight law-breaking' necessary, nor mealy-mouthed 'inadequate equilibria' lies-to-self necessary either. Change is happening, the world doesn't need to go all Mad Max.

fwgwgwgch3• 8 years ago

Are you the author? "We have no idea how to create Original movies anymore ". This is funny. It should be a sequel to tropic thunder. Aptly named.

Well written article btw.

Mononokay4• 8 years ago

It allows for open-access of the epitome of human progress, which is something unavailable to low-income communities. It's copying data, not stealing from a museum, so it's not as if any journal is losing much, if anything.

There's no real ethical downside - allowing people who wouldn't normally have access to research helps to reduce inequality, and allows human progress to advance a lot quicker than if it weren't open. There are brilliant people of which have no access to research, who could - and would - more likely do something with it than not.

Given that we all pay for science, [0] from people in the lowest drudges of poverty, to people in the highest skyscrapers in New York, we should all have access to it. Imagine you paid taxes meant to improve roads in your area, but ended up going to a place hundreds of miles away from you. You'll never get to use these roads, and you'll be told you're unethical and unlawful if you even consider using them. It wouldn't be very fair, would it?

[0] https://undsci.berkeley.edu/article/who_pays

lsh3• 8 years ago

Unless the article has been published as open access under a CC-BY (or similar) licence, it's not open access, please don't confuse the two.

I agree with you in spirit - tax payer funded research outputs should be publicly available - but this is what got Aaron Swartz: it is unlawful.

Fins8• 8 years ago

One could easily come up with quite a few ethical arguments against Sci-Hub -- it's there to stick it to the "West". It has nothing to do with science, human progress, or anything as lofty. Basically, just another department of the Olgino Troll Factory.

That it lets you freely access some scientific materials regardless of whether the rights owner (or authors, for that matter) agree to it or not is just a minor side-effect.

lalaland1125• 8 years ago

One interesting thing about sci-hub is that it's actually in a researcher's best interests to have their articles uploaded to sci-hub and widely distributed. The more people that have access to an article, the more people can build off the ideas contained within, resulting in potentially more citations and advances that the original author can take advantage of.

So, you reading articles off of scihub and expanding on the contained work within directly benefits the author. Just make sure to cite the author if you yourself publish! (Which is actually much less difficult than you might think. Even a blog post can contribute to the field).

fwgwgwgch3• 8 years ago

I like your point. It's something I've heard the first time

Vinnl5• 8 years ago

Authors are often allowed to share at least their preprints (i.e. submitted manuscripts that have not been peer reviewed yet) elsewhere, but this rarely happens in many academic disciplines. In fact, they are often required to do so by their institution, to deposit their work in its Institutional Repository, yet even that requirement is often flaunted.

(In my view, this is because the personal benefit is not that clear yet, especially if the work is just hidden in some long list of other research. Which is why I'm working on helping them use their research to raise their academic profile: https://medium.com/flockademic/the-holy-grail-in-open-access... )

jadedhacker• 8 years ago

Profit motive is holding scientific progress back. It's time to disrupt the status quo. It's fully possible to design a non-profit system that makes scientific publishing possible. Sci-hub may provide the structural perturbation required to fully dismantle the existing industry so that we can make a new one in the shell of the old.

pmyteh3• 8 years ago

I'm an academic. I've never met one who has objected to Sci-hub. Many of us use it, even to get papers we do have legit access to: journal publishers' access control systems are a pain in the arse.

Leaving aside the societal merits, our incentives are to get published, then get read. Sci-hub helps with the latter, and doesn't harm anyone except the publishers (who most academics cordially dislike anyway).

pbhjpbhj2• 8 years ago

I imagine even the most socialist/communist/whatever is at least partly incentivised by earning a living too.

In fact, isn't that the problem, that quantity of publications (and citations) is used as a metric for continued employment? Those awarding research grants, etc., don't provide a free open-access publication as the place publication is required.

It seems governments could fix this in favour of the public quite easily; require publication in an open access journal of all research (papers and data) with any funding from the public purse.

gkya5• 8 years ago

I'm preparing for a master's thesis (and then a research career), my university's library is rather scarce. Local uni's with superb libraries like Bogazici and Koc only allow graduate students from other unis. W/o sci-hub, I'd not have been able to read a single paper I've read in the past couple of years. Would have had to wait until I graduate, and had to expect that I made my way into a uni with a nice library, or at least be in a city where I can access to another uni with a rich library.

Information is produced mostly on tax money, and even when it's not, it's based on tech and info developed on tax money, and all of that is based on millennia of science one way or another the global society produced. I believe it's fair to think I and every other person "own" all of the scientific information already anyways. Even if you don't believe in that, I can't see how the other option is to pour mostly taxpayer money into cancerous organisations that sell what they have not even helped produce.

Vinnl5• 8 years ago

In favour: the fact that costs of access to research haven't plummeted with the rise of the internet as they have for other media, and that the traditional publishers are making profit margins of 30-40% (typically associated with luxury brands) indicates that the academic publishing market is not working as it should, and that the traditional publishers are making more profit off publicly funded work than you'd expect them to "deserve" based on the value they add.

Against, from a societal point of view: Sci-Hub is no longe-term solution, as it still depends on articles being published elsewhere. Furthermore, there is not yet a proper alternative to the traditional publisher's main contribution: credentialing of academics.

Against, from a point of view of the shareholders of the traditional publishers: if this leads to cancelled subscriptions, they lose money. A case could be made that Sci-Hub has contributed to e.g. most German university libraries cancelling their Elsevier subscriptions this year.

zaarn2• 8 years ago

In short Elsevier.

The movie industry has it's fair share of copyright protectionism (DRM is popular) and same goes for gaming.

Elsevier is worse. They get paid by scientists so they publish their papers. They get paid massive amounts of money by universities (sometimes even the same that paid the scientist) to be granted access to research. Research largely funded by tax money. Research results that had already been paid for to even be published there.

There is a very good article in the SlateStarCodex if you're interested in a more in-depth article on it.

paulhilbert2• 8 years ago

I read this quite often and frankly do not understand it. How is Elsevier worse than ACM, IEEE or Springer? I found them to be equally intolerable but still Elsevier gets routinely picked as the prime example...

lihan• 8 years ago

We'll see how long this one last.

Tomminn• 8 years ago

Honeypot?

Myrmornis2• 8 years ago

I'm 100% in favour of sci-hub.

However, note that they are very anarchic when it comes to commercial books, not just journal articles!

E.g. from the Sci-Bay search results, this is $131 on amazon.com, and quite possibly the authors do want the royalties.

[BOOK] Intelligent optimisation techniques: genetic algorithms, tabu search, simulated annealing and neural networks D Pham, D Karaboga - 2012 - books.google.com ... Cited by 916 Related articles All 3 versions [Download Book]

gkya5• 8 years ago

I believe people in academia are paid to write these books anyways, so they might as well not receive the royalties. As a prospective academic myself, I find it unethical.

dagw2• 8 years ago

I believe people in academia are paid to write these books anyways

While that does occasionally happen, it's definitely not true in general. The people I know who wrote academic books did so by taking a sabbatical from their university job and/or working evenings and weekends for the actual writing. Occasionally they can apply for a separate writing grant to cover their lost salary, but that's completely separate from their day job.

Myrmornis2• 8 years ago

Yeah, that's not really how it works. See dagw's sibling reply, and note that there's a continuum between academic monographs that would only ever be in university libraries, to popular science books, in between which is a large textbook market, and note that you can download all the above for free right now from libgen. And I am not moralizing. It's an amazing free resource! Transformative for people in nations at an economic disadvantage, and transformative for people in the first world too -- it would cost thousands of dollars or hours of inefficient library visits to be able to consult such a breadth of resources. I just don't know how exactly we/society should think of this.

xstartup• 8 years ago

It works well, OT: Anyone knows how to remove the top header in this Scihub link:

https://sci-bay.org/article?link=https://pdfs.semanticschola...

gpm2• 8 years ago

Download the pdf, open it in your browser (or another pdf reader) directly.

lsh3• 8 years ago

If sci-hub is going to scrape OA publishers, they could put in a bit more effort.

For example this (which sucks): https://sci-bay.org/article?link=https://www.ncbi.nlm.nih.go...

Versus the actual article: https://elifesciences.org/articles/24234

n4r9• 8 years ago

I think the issue is with sci-bay rather than sci-hub. Searching sci-hub for the title of the article brings you to the second webpage you linked.

PokemonNoGo• 8 years ago

I'm sorry but I don't see how it works?

https://sci-bay.org/scholar?hl=en&as_sdt=0%2C5&q=entropy+sha...

-> Please show you're not a robot

samat• 8 years ago

This reminds me of Popcorn time so much.

Rightholders do not fear torrents as long as they are unusable for the general population.

The second they see something usable — they go berserk.

Gonna need some popcorn to watch this one.

Vinnl5• 8 years ago

Likewise, the traditional publishers often respond to demands by funders to make research available by e.g. allowing researchers to share their work elsewhere, and often only after a year or so after publication [0]. This makes the barrier to do so higher, and makes the research less findable. It's not odd to expect that when initiatives like Unpaywall [1] make that research more discoverable, things like embargo periods will get worse.

[0] https://medium.com/flockademic/how-open-can-open-access-be-c...

[1] https://unpaywall.org/

moomin• 8 years ago

This is a clever mashup. Of course, if you want it to last a week, I'd be making some effort to distribute the source far and wide...

rkskejfj• 8 years ago

Beyond close vs. distant ties: Understanding post-service sharing of information with close, exchange, and hybrid ties

irundebian• 8 years ago

Aaaaand it's down.

s2th4d• 8 years ago

And it's down.

"See you later Too much attention is a bad thing, Sci-Bay decides to stop service for a while. Sorry. Anyone who knows how Sci-Bay works and wishes this tool benefits more academics, please contact: info@sci-bay.org"

aysus• 8 years ago

BarOn, R. (1997). EQ-i Baron Emotional Quotient Inventory: A Measure of Emotional Intelligence : Technical Manual. Toronto, ON: MHS.

abhishekjha• 8 years ago

Is the website down? It just says "too much attention" caused it to shut down.

Sci-Bay: Google Scholar plus Sci-Hub

Related Stories