How to scrape data from sites you can't log into

http://ssscripting.wordpress.com/2009/02/24/how-to-scrape-data-from-sites-you-cant-log-into/

by geoscripting • 17 years ago

37 37 17 years ago

ssscripting.wordpress.com

Mystalic• 17 years ago

I wonder if this works every time...

And if it did, how long until a defense is made...

cliffy• 17 years ago

You could make it very onerous to move around the site even if you're logged in. For example, after every few links you follow you have to pass another Turing test. That would be completely unreasonable though.

I think once a user is logged into your site, you'd have an extremely hard time defeating this sort of behavior without degrading the quality of user interaction or treading on legitimate use of your site. You may be able to defeat egregious abuses, such as scraping entire photo galleries in seconds, but even that can be defeated by a script that randomizes requests/times between requests.

geoscripting11• 17 years ago

More and more websites are adding Turing tests after relatively short intervals of time. I consider that to be a good thing, and there are some good ways to stop/identify spiders. You can check out this site: http://stackoverflow.com/questions/450835/how-do-you-stop-sc... . It shows some pretty good techniques that one can use to defend from spiders.

Retric• 17 years ago

There is nothing magic about web browsers, telent to port 80 at www.google.com and with a simple get request they will spit back their website. You can make it a little harder to do this stuff but a packet sniffer is always going to let you pretend to be any software you want unless they are using encryption. Also because Firefox is open source you can't prevent people from scripting using it anyway.

PS: I recommend all aspiring coders to telent to www.google.com at least once just to feel the magic.

timf4• 17 years ago

"Firefox is open source you can't prevent people from scripting using it"

There are frameworks for doing that already, check out Selenium: http://seleniumhq.org/

geoscripting11• 17 years ago

That's how it all starts :)

radu_floricica3• 17 years ago

> PS: I recommend all aspiring coders to telent to www.google.com at least once just to feel the magic.

I'm pretty sure it's illegal in a couple of ways. </bitter>

But yes, the magic is there.

philh• 17 years ago

Some sites will expire the cookie after a period of inactivity, in which case this will stop working. But it can easily be refreshed by getting a new set of cookies, which as other people have pointed out is a process that can be automated.

spoiledtechie• 17 years ago

interesting.

timf4• 17 years ago

Seems to me this is just saying: log in via the browser (with the right password), then use the generated cookie with the scraping code. You can also do that with wget --load-cookies

geoscripting11• 17 years ago

True, you can use wget as well, but this way you have more control over your code.

timf4• 17 years ago

For sure, I was responding to the title, it sounded like a security thing.

tarmac• 17 years ago

You're still technically logging in by providing the copied cookie. I don't see any difference here.

Or say the title should be changed to "How to scrape your data from your sites that require login"

almost2• 17 years ago

Agreed, this more of a "how to scrape data from sites you can log into". It's not even a very useful example at that.

For anyone who wants to scrape sites that require login I'd recomend Python with Twill. That lets you do the whole thing with ease.

geoscripting11• 17 years ago

Twill is an option indeed, but this way you don't miss out on the javascript. You can take advantage of all of your browser's features.

apgwoz3• 17 years ago

But, if the site is using only HTTP and cookies, there's no reason not to first make a request to the login page with the username/password and retrieve the cookie via the "cookie" header that comes back... Did I totally misread the article, or was it just dumb?

timf4• 17 years ago

A lot of scraping frameworks will handle it for you, too. It's even the front page example code @ http://jwebunit.sourceforge.net/

I think most people are reacting to the title... I for example thought it was a security posting.

apgwoz3• 17 years ago

Of course there are frameworks to do this for you, and I'm sure jwebunit will work perfectly well. I guess I'm disappointed that the author didn't understand the fact that Firefox doesn't perform magic, and that a login page is no different than anything else being scraped.

geoscripting11• 17 years ago

I think you misread the article.

apgwoz3• 17 years ago

I definitely didn't. The article mentions that you login, pull out the cookies, copy the value INTO your code, and since his examples are using java, recompile it.

babo• 17 years ago

I do my scrapping with BeautifulSoup from Python, actually from an iPython shell. With a urllib2 opener you could easily handle cookies and UserAgent pretty easy, the later is also important for some sites.

geoscripting11• 17 years ago

There's also mechanize for python. It can be found here : http://wwwsearch.sourceforge.net/mechanize/ . It handles cookies by default, and is a pretty good tool.

sho3• 17 years ago

Huh? What he did is completely automatable, you don't have to touch Firefox. And if you really did need JS to log in, which I have never seen outside misguided banks, there's tools for that too. Selenium and JSSH come to mind but for 99% of sites you'd just need Mechanize.

And Java? Why the hell would anyone write a script in a compiled language like Java? Desperate for that 2ms time saving between 10 second waits for the pages to come down, eh? And any for-real scraping script would have a time delay built in anyway.

The guy doesn't know what he's talking about.

geoscripting11• 17 years ago

The example wasn't written in Java for performance gain. It so happened that I had NetBeans open , and it was easier for me to write it in Java at the moment :).

sho3• 17 years ago

Easier!? You wrote pages and pages describing the most inefficient way imaginable to do something I can do in 5 lines of Ruby, and you call it easier? And unless I'm very much mistaken, you'd have to compile the code anew whenever the cookie changed?

Well, good luck to you, and the more script kiddies you confuse the better I guess, but there are seriously much better ways to do this. Go look at Ruby Mechanize (I think it's also available for python); coming from Java you will be blown away by just how easy this kind of thing is. How do you think we all test? ; )

Update: Oh I see you know Mechanize from another article. So why not just use that ... you do know it can do all that logging in stuff for you, right?

rams• 17 years ago

Can someone suggest a browser-independent way of handling login pages that require javascript ? Twill doesn't handle javascript.

geoscripting11• 17 years ago

You could give HtmlUnit a try.

fugue88• 17 years ago

man wget

wget --save-cookies cookies.txt ...

wget --load-cookies cookies.txt ...

How to scrape data from sites you can't log into

Related Stories