Why you should use curl and firebug to scrape

http://nachivpn.blogspot.com/2015/05/why-you-should-use-curl-firebug-to.html

by nachivpn • 11 years ago

2 2 11 years ago

nachivpn.blogspot.com

marvel_boy• 11 years ago

Newbie here. cURL is a command line tool, firebug is a browser tool. So how can I use the two tools at the same time?

nachivpn• 11 years ago

HTML of a page is received in response to a HTTP request. A page load might involve multiple HTTP requests. Often parts of a web page that interest you might be loaded specifically by a request. By using firebug, you could examine which HTTP request fetched you the content you need and what parameters (headers, cookies, etc) where associated with it. Using this information, you could simulate a HTTP request of your own using cURL (in fact, firebug provides a "copy as cURL command" feature which can be very useful). Firebug is used only in the initial stages to understand. Post which, cURL can be used in your shell script (or a suitable language of choice) to implement your bot.

Why you should use curl and firebug to scrape

Related Stories