Web Scraping with Python

https://www.scrapingbee.com/blog/web-scraping-101-with-python/

by daolf • 4 years ago

195 77 4 years ago

f311a• 4 years ago

That's just a basic introduction. I would not call this article "everything you need to know".

Yeah, it’s as if someone posted an article “everything you need to know about cooking” and just explained the concepts of plates, pans, and some of the kitchen appliances.

I guess the fact that it’s currently very high on the front page of HN kind of confirms this type of post works, though, which is unfortunate.

daolf6• 4 years ago

Hi there, co-author here.

Always to improve the content we're writing here. What else would you have expected to read in such an article?

danmur• 4 years ago

I think it's a pretty good article personally, sounds like the complaint is just about the title :P

edent2• 4 years ago

I think it is good. You have to remember that some of the loudest voices on here take things extremely literally. They have no concept of hyperbole for emphasis. Or, indeed, anything which makes writing interesting to read.

Is your guide everything someone needs to know? No. But anyone literate in the ways of modern English understands what you mean.

It is an excellent guide and I think you should consider expanding it & perhaps creating a book.

Please don't be discouraged by the people on here who don't have the skill or courage to write or submit anything.

CJefferson• 4 years ago

Nowadays I would jump straight to selenium (or similar), as most websites feature AJAX or similar, so need a full browser.

Then, you don't actually do anything with selenium, click a button / link, or anything interesting.

is_true3• 4 years ago

I think it's actually not that bad. Scrapping is a topic that is as broad as the number of sites on the web, a minefield of corner cases.

daolf6• 4 years ago

Thank you!

ohyoutravel2• 4 years ago

Why do these types of low quality, seemingly spam / SEO articles always rise to the top on weekend mornings? Is it lack of competition? Easier to manipulate votes?

mateuszbuda• 4 years ago

Maybe they use their product to generate upvotes O.O

daolf6• 4 years ago

"low quality".

I'm hurt :(

PS: we spend tens of hours writing those piece of content and even pay a technical editor to spot the typo and make it more readable since we're not native English. You might not like this post, but I can assure that genuine care was put into writing this!

degenerate• 4 years ago

Scrolling through your article I disagree, it's high quality content. What converts it to "low quality" is the bait-n-switch title. This is not "everything you need to know" -- this is "how to get started from scratch".

Metaphor would be "Everything you need to know about fixing cars" and the article shows you how to check the engine light, change oil, rotate tires, and replace spark plugs. There's just no way to make a promise that large and have your article be considered high quality.

wintercarver• 4 years ago

I thought it was a nice summary, concise, organized, with examples and references. Will revisit it should I need a reminder on scraping. Would not call it low quality at all.

Would recommend you ignore passing comments with no constructive criticism. The title is going to be a point of contention as it’s a big claim and probably being misinterpreted as not “everything you need to know [to get started]” but rather “everything you need to know [ever is in this one article and you’ll need not read anything else]”.

tsukurimashou• 4 years ago

change the title to "everything you need to get started..." instead of "everything you need to know" and try to go easy on SEO optimization

most of negative comments will go away

ohyoutravel2• 4 years ago

I don’t think it’s very good and many other highly-rated top level comments seem to agree that not only does it have a scammy SEO “top ten best ${X} in ${CURRENT_YEAR}” but there is a mismatch between what the article is attempting to do with how it is attempting to explain and do it.

While I’m glad it’s not GPT-3 level spam, or outsource to third world country for copy level spam, in my opinion the article fails in several fundamental ways, noted above. Putting “genuine care” into something is commendable, but is not a substitute for quality, relevant content.

OTOH you’re getting lots of clicks and views for whatever product you’re selling, and even my comments help the “traction” HN gives it, so it doesn’t actually matter what I think.

is_true3• 4 years ago

Probably a tiny bit of manipulation: https://news.ycombinator.com/item?id=30025005

sixhobbits2• 4 years ago

It can take 100 hours or more to put together a guide like this. It is painful to do but helps many people, so its ultimately very rewarding. Every developer I know has learned more from free guides written for free than they have from paid courses, bootcamps, and often university degrees.

But thanks for your contribution I guess..

is_true3• 4 years ago

I think it's a nice article, just needs an "almost" and everyone should be happy. Did you write it?

pahn2• 4 years ago

The article is an overview aimed at beginners, but as such actually pretty good and helpful. It does not seem to overtly promote their product. Classifying this as spam / SEO only because one is not the target group is not fair.

Luc• 4 years ago

The article was obviously written for SEO purposes.

mywaifuismeta• 4 years ago

I'm pretty sure there is some manipulation going on to get this article to the top. It's extremely basic and "baity" compared to what you usually find here. I hope HN doesn't turn into the next Medium. It has been getting worse recently, which makes me think people have found better and better ways to create and manage spam/upvote accounts. Perhaps GPT-3 allows them to automate the karma generation for new accounts :)

idk1• 4 years ago

What makes you think this is low quality. I code ruby and not much python but it looked really good to me.

pfranz3• 4 years ago

I end up needing to do something like this every few years. Even when I use Python at my day job every day, it's easy to miss changing best practices when it's slightly outside your domain. So, like I did with this article, skim it to see how credible it seems and take note of things that look like they've changed since I last had to do it (and take a mental note in case it comes up in the near future and I need to get something up relatively quickly).

srvmshr• 4 years ago

We should have some community guidelines to keep out Medium/Towards Data Science and similar low-effort article sources from HN.

Genuinely in favor of lesser submission vs. increased noise in submissions. Beginner articles are not taboo, but goes against having high quality insights in general.

PS: Flagging is mechanism to filter by community efforts. Guidelines set some general preconditions to the quality of articles for larger dissemination.

edent2• 4 years ago

You can either hit the "flag" link, or submit something better.

pedrovhb2• 4 years ago

That's nice, but I don't see much value in learning about sockets for scraping; it's way too low a level. The lowest level I found useful was using a requests/httpx for requests and using regex to parse data when the data you're scraping has a constant enough structure and you're scraping a large number of pages, as regex is a lot faster than parsing html.

I'd add that it's often worth spending some time looking at the website for alternate ways than the obvious one of getting the data you're after. sitemap.xml sometimes give useful hints.

Another golden trick is to learn reverse engineering mobile app APIs with mitmproxy or something like it. Nowadays it's kind of a pain to do since Android has been locking things down more and more, but it's still quite possible. Apps very often provide endpoints that give you structured data when the web version is server-rendered HTML only, have fewer anti-scraping measures and rate limiting, and even provide data that isn't available at all for the web version.

Toxygene• 4 years ago

as regex is a lot faster than parsing html

This person would like a word with you -- https://stackoverflow.com/a/1732454

hashmush• 4 years ago

There's a big difference between parsing HTML and

using regex to parse data when the data you're scraping has a constant enough structure

Regex is fine, just don't parse the HTML itself.

melenaboija• 4 years ago

The first comment says a lot about it:

I think it's time for me to quit the post of Assistant Don't Parse HTML With Regex Officer. No matter how many times we say it, they won't stop coming every day... every hour even. It is a lost cause, which someone else can fight for a bit. So go on, parse HTML with regex, if you must. It's only broken code, not life and death

pedrovhb2• 4 years ago

Well, yes - he's saying "regex is not appropriate for parsing html", and I'm saying "regex is faster than parsing html" - they're not contradictory statements, and both are true :)

To be clear, I'm not talking about building a syntax tree or a way to generically extract elements based on a CSS path selector. I'm saying if you're only interested in a couple of data points in a 3 MB HTML document, and you're sure they're always between some other specific text or even tags, then it's more efficient to use a simple regex than it is to parse the entire thing, which is computationally expensive when running over a large number of large files.

matheusmoreira2• 4 years ago

I love this answer so much. I'm surprised it hasn't been deleted yet like many of my other favorites.

pfranz3• 4 years ago

For personal projects I generally follow the steps this blog lays out; start with the light and low-level APIs and work my way up as needed. I do usually skip over sending raw sockets when I start, but I think knowing them is worthwhile for troubleshooting and optimizing. I often find myself jumping to different levels when navigating scraping--from http headers to javascript rendering. While you can touch most of those things with requests, I find it easier to reproduce exactly what I see my browser doing with lower level APIs. The backend might be tightly-coupled with the front end. So you might get stuck on a specific header, user-agent string, or something often related to sessions or login.

shahidkarimi• 4 years ago

Scrapy is there to make all these happening under a single framework.

bschne• 4 years ago

Aside: The last time I had to scrape a lot of data from the web, I additionally used SQLite, which is a breeze to use with Python (basically one import statement and you're set). It might be overkill for some cases, but I found it a huge boon for keeping track of which pages were scraped, which failed, and doing subsequent data processing and parsing "offline". It made it so much easier to recover from the inevitable random error or different markup somewhere deep in your list of pages to scrape etc.

stall84• 4 years ago

This is great.. Mainly because the very first thing he does is explain the network requests themselves, focussing on the (somehow often left-out) fact that you are going to have to spoof a browser (or headers associated with it) almost always these days to get around bot-protections.

kaycebasques• 4 years ago

I think the overall software architecture approach of this post is fundamentally backwards. Given how much of the web is rendered client-side these days you need to start out with a headless option. Headless means that you fire up a true browser and then automate the actions that you need to perform. It's indistinguishable from a real person using a browser. If you try to use urllib3 on a webpage that does heavy client-side rendering then you're going to get incomplete HTML returned from the server (because the website intends to use JavaScript to complete the rendering of the page). On the more rare occasions when you are dealing with static HTML (i.e. there is no rendering on the client; the HTML returned from the server is the complete content) then you can use something like urllib3.

Re: which headless library to use this post mentions Selenium which was one of the first headless libs but from what I've heard probably not the best (in terms of developer experience or reliability or robustness) but that's only hearsay... I've never used Selenium myself. Playwright seems like the best option in town if you want to use Python. Built by the former Chrome DevTools team (meaning those people really know how browser internals work). https://playwright.dev/python/docs/intro

datalopers• 4 years ago

Headless scraping quickly becomes a very expensive approach when you try and scale the effort. I only employ it when absolutely necessary. And it’s most definitely distinguishable by any modern (incapsula, perimeterx, cloudflare) WAF.

apienx• 4 years ago

The Apify library tries to address most of these issues (I'd say quite successfully). Apify.com provides a platform you can deploy the scrapers on. And yes, at scale.

There's also a software marketplace where you can order custom scrapers. 98% of the projects ran thru it have a 5-star rating (Disclaimer: I moderate that marketplace). Pro-tip: submit your project with a Gmail address to skip sales and reach me directly.

lapser2• 4 years ago

Or you could use the API that the web app has to inevitably use.

chrsig2• 4 years ago

this is assuming that it's documented...otherwise you're just hand evaluating javascript to figure out what it would call...and then you get to thinking that you should just embed a javascript interpreter and evaluate it...and at that point, you've gone down the path of implementing a headless browser.

matheusmoreira2• 4 years ago

Love this approach. We can just bypass all the normal web scraping and get the structured data straight from the source. These APIs are usually no less stable than the ever changing HTML structure anyways.

Case study: YouTube.js

https://news.ycombinator.com/item?id=31021611

https://github.com/LuanRT/YouTube.js

daolf6• 4 years ago

We've decided to go this route because based on billions of web-scraped page, Headless-based scraping is still a minority. And, it's way harder and more expensive to do at scale.

elorant2• 4 years ago

From my experience news sites are the one category that requires a headless browser the most. With e-commerce sites, or anything else, it's like 80% of the cases will work with a normal http request.

samwillis2• 4 years ago

From my experience headless browser scraping is in the order of 100x slower due to increased bandwidth, cpu and memory. I would seriously suggest the other, start with traditional scraping and if you can’t make it work then go headless. The difference in speed is to the point that going a longer route (maybe having to scrape 10x more pages for example) to the content that enables you to not use headless will probably still end up quicker.

On the memory side, headless you end up with far more memory leaks, having to manage stopping and starting new browser instances while maintaining scraping state. The devops overhead is probably 10x more with headless.

taosx2• 4 years ago

Agreed (especially on the devops overhead), some options to lower the bandwidth usage using request blocking: - block ads - blocks videos, images - css stylesheets

In the past I've built scraper infrastructure (headless pools, credential stores, proxy managers, agent profiles) and managed to get a pretty efficient service by tracking cpu,memory,network usage per each job and writing specialized versions. I got pretty far trying to automatically generate specialized scrapers from previous requests but I moved to other projects.

Scraping becomes boring really fast if you don't use the data in meaningful ways.

MonaroVXR• 4 years ago

What is traditional?

elorant2• 4 years ago

If you don't load images, and you disable 3rd party JavaScript you can save a lot of loading time with a headless. I'm using a preconfigured profile in Firefox with uMatrix installed and sites load at least 50% faster. Add another 20-30% from not loading images, and suddenly using a headless becomes affordable.

1vuio0pswjnm7• 4 years ago

"Given how much of the web is rendered client-side these days you need to start out with a headless option."

What does "rendered client-side" mean.

Assuming that "rendered client-side" means interpretation and execution of Javascript is necessary to read a site's textual content, then how much of the web is rendered client-side.

If the focus is on textual content, e.g.,, someone is primarily "scraping" text as opposed to images and video, I would guess that only a minority of the web is "rendered client-side". How would we prove otherwise.

This guess I am making would not be an uneducated one. I have been accessing the web without using Javascript for over 30 years. Today, I still use a text-only browser to render HTML as text/hypertext. This allows me to read the site's textual content, quickly and easily. I initiate most HTTP requests with TCP clients, not the browser. All requests, whether from TCP client, browser, or otherwise, are made through a localhost forward proxy. If most websites were truly dependent on Javascript, it stands to reason I would not be able to read much of the web. In other words, another web user who reads the web with a Javascript-enabled browser should be able to read websites that I could not read. This has not been the case. In fact, I often see commenters on HN complaining that they cannot read a site that I am having no trouble reading. The culprit is often Javascript.

The truth is that I rarely encounter a site that cannot be read with the text-only browser. For example, I can read the content of almost every site submitted to HN. A very small minority of sites I find are, more or less, empty shells with links to some Javascripts but no textual content for the visitor to read. These "landing pages" expect a Javascript-enabled browser that automatically follows links in the page (e.g., to remote Javascript files), and that retrieves, interprets and executes Javascript automatically and indiscriminately.[FN1] In what some might see as a Rube Goldberg design pattern, the scripts then make HTTP requests to the "real" site. In such cases it generally only takes me a few minutes to find the "real" site, often what some refer to as a "JSON endpoint".[FN2] However this process has not lead me to rely on a "headless" browser to read websites.

Honestly, if a majority of sites adopted the "JSON endpoint" approach to serving textual content it would make reading websites even easier for me. I could just retrieve JSON and reformat it to a uniform brand of simple HTML that I prefer, as I already do for some sites. I could make the format of all websites 100% identical. IME, a web of uniformly-formatted content is much easier and faster to digest. I would imagine it would easier for machines to digest as well. The text-only browser I use currently makes the format of all sites look almost the same, since it only uses a single font and so many websites use similar designs. Because it does not automatically follow links or execute Javascript, it also tends to make the "load" time of all sites very similar. For me, this uniformity speeds up the ability to digest web content as compared to using a graphical browser for the same purpose.

FN1. Today we see "modern" browsers incorporating an ever-changing array of "features" and options to try to mitigate the risks of this behaviour.

FN2. Generally, IME, these "endpoints" serve the textual content with minimal markup or sometimes no mark up at all. Thus, the end user is free to format the text into whatever design suits their personal tastes. As a website visitor, this is relatively more efficient IMO than trying to read an infinite number of possible "web designs" which is the approach we currently see on today's www. It is more predictable. With the later approach, visiting a new website with a Javascript-enabled, graphical browser is always a "surprise". It might be easy to read or it might not. Visiting "endpoints" generally does not suffer from this problem.

dmortin• 4 years ago

How do scrapers deal with randomized classes in web pages which is more and more common these days?

Relying on the page structure only is not a robust alternative.

edmundsauto• 4 years ago

I've had some success with running a meta-scraper that will search for known value on a page, then back out the page structure from there. It won't help with randomly generated class names, but 95% of tasks I've written aren't this complex.

For sites that are hard to scrape (usually bigger sites that get scraped a lot), I pivot towards buying a data feed. Economies of scale incentivize these data companies towards putting someone on maintaining the feed full-time.

chasd00• 4 years ago

What I’ve done is pay very close attention to the network traffic in your browsers dev tools. The data has to get to the browser somehow. Once you’re able to get a session token/cookie then you can figure out what GETs or POSTs you need to get the data you want by watching the requests your browser makes.

photochemsyn• 4 years ago

One problematic thing that pops out immediately for a Python-centric approach is that they don't mention that this is all best done in some kind of Python virtual environment, like miniconda or virtualenv. They just suggest 'pip install package', which is not a good approach for anyone (and particularly not beginners) - unless you want to end up with this:

https://xkcd.com/1987/

Looking around a bit with the requirement that the online tutorial mention this rather important fact, I found this alternative option, which helpfully notes:

We want to run all our scraping projects in a virtual environment, so we will set that up first.

https://python-adv-web-apps.readthedocs.io/en/latest/scrapin...

Compare and contrast that discussion with the one presented in this post - the above is far superior. Also, I don't understand why one would suggest PostGreSQL to a beginner when sqlite3 is included already in Python, and is going to be easier to use for small databases. Towardsdatascience seems to have a nice intro-to-sqlite3 tutorial.

mynameismon• 4 years ago

Perhaps the only issue I would have with this blogpost is using Postgres. By all means, SQLite can do the exact same thing, just easier for a beginner, since they don't have to wade through a mess of networking. Just add the binary to the PATH and one is good to go.

taosx2• 4 years ago

Just don't forget to optimize it for writes (WAL-mode...etc) when having lots of sources.

ducktective• 4 years ago

Consider pup https://github.com/EricChiang/pup

inshadows• 4 years ago

How do scrapers deal with being nice to a website these days? I'm talking multiple IPs, request rate, exponential backoff. Is there any body of knowledge for this?

TBurette• 4 years ago

Is there a good way to combine Scrapy framework (retry, rate limiting,..) with a headless browser such as selenium (to get full js-loaded client-side data)?

When I had to do it I ended up duplicating each page request twice. Once for scrapy and once again with selenium.

ihartley• 4 years ago

You can use something like scrapy-playwright[0] to run a headless browser framework as your download handler. I think there are versions for some of the other headless systems, if you prefer those.

[0] https://github.com/scrapy-plugins/scrapy-playwright

samwillis2• 4 years ago

scrapy-playwright is good, and Playwright is awesome. However due to the architecture of Playwright it just keeps accumulating memory until it crashes. You will want to set up your scraper to save its state regularly, cleanly shut down and restart. But once you have that working it does work well.

holografix• 4 years ago

How do people get around browser finger printing by “Sign in with Google” these days?

All I get is “your browser is not safe” etc which blocks me completely.

Web Scraping with Python

Related Stories