mnmkng

Crawlee isn’t any less configurable than Scrapy. It just uses different, in my personal opinion more approachable, patterns. It makes it easier to start with, but you can tweak whatever you want. Btw, you can add middleware in Crawlee Router.

Technically it can. You can log in with the PlaywrightCrawler class without issue. The question is if there’s 2FA as well and how that’s handled. Crawlee does not have any abstraction for handling 2FA as it depends a lot on what verification options are supported on the SSO side. So that part would need a custom implementation within Crawlee.

We tried a self hosted OCR model a few years ago, but the quality and speed wasn’t great. From experience, it’s usually better to reverse engineer the APIs. The more complicated they are, the less they change. So it can sometimes be painful to set up the scrapers, but once they work, they tend to be more stable than other methods.

Data pollution is real. Also location specific results, personalized results, A/B testing, and my favorite, badly implemented websites are real as well.

When you encounter this, you can try scraping the data from different locations, with various tokens, cookies, referrers etc. and often you can find a pattern to make the data consistent. Websites hate scraping, but they hate showing wrong data to human users even more. So if you resemble a legit user, you’ll most likely get correct data. But of course, there are exceptions.

In one word. Nothing.

But I personally think it does some things a little easier, a little faster and little more conveniently than the other libraries and tools out there.

Although there’s one thing that the JS version of Crawlee has which unfortunately isn’t in Python yet, but it will be there soon. AFAIK it’s unique among all libraries. It’s automatically detecting whether a headless browser is needed or if HTTP will suffice and using the most performant option.

It uses Playwright under the hood, so yes, it can do all of that, and more.

It’s an “old” law that did not consider many intricacies of internet and the platforms that exist on it and it’s mostly made obsolete by EU case law, which has shrunk the definition of a protected database under this law so much that it’s practically inapplicable to web scraping.

(Not my opinion. I visited a major global law firm’s seminar on this topic a month ago and this is what they said.)

Sorry about the confusion. Some features, like the tiered proxies, are not documented properly. You’re absolutely right. Updates will come soon.

We wanted to have as many features in the initial release as possible, because we have a local Python community conference coming up tomorrow and we wanted to have the library ready for that.

More docs will come soon. I promise. And thanks for the shout.

Oh wow, thanks! Will fix it right away. Crawlee is originally a JS library.

finally!

Yeah I agree, keeping the source HTML is great for debugging or retro-fixing issues. We also like to take screenshots on important errors, when running headless.

Exactly. The dynamic websites need to pull the data from somewhere as well. There's no magic behind it. Either all the data is in the initial payload in some form (not necessarily HTML), or it's downloaded later, again, over HTTP.

Headless browsers are useful when the servers are protected by anti-scraping software and you can't reverse engineer it, when the data you need is generated dynamically - not downloaded, but computed, or simply when you don't have the time to bother with understanding the website on a deeper level.

Usually it's a tradeoff between development costs and runtime costs. In our case, we always try plain HTTP first. If we can't find an obvious way to do it, we go with browsers and then get back to optimizing the scraper later, using plain HTTP or a combination of plain HTTP and browsers for some requests like logins, tokens or cookies.

If it doesn’t, please make an issue. We know it works from the community but we don’t have tests specifically for Lambda. But it should work, so we’ll help if it doesn’t.

Yes

I get your point and I don't have an objective answer to it. We believe that internet is an open medium and there's immense value for humankind waiting to be discovered and unlocked in all its data. After all, many of the big tech companies in the world utilize web scraping heavily.

Rate limits can be applied for different reasons. If they protect the website from being overloaded, they are good in our opinion. If they protect it from competition, research or building new non-competitive, but valuable products that are not harmful to the original website, they are not ideal.

We leave that to the user to decide the ethics of their project and just provide the tools.

Yeah, sure. We don't advertise it, but if you get in touch with us on support@apify.com or through the chat widget, we can create a proxy-only plan for you.

With fingerprints it's a tradeoff between having enough of them for large scale scraping and staying consistent with your environment. E.g. you can get exponentially more combinations if you also use Firefox, Webkit, MacOS and Windows user-agents (and prints) when you're running Chrome on Linux, but you also expose yourself to the better detection algorithms. If you stick to Linux Chrome only prints (which is what you usually run in VMs), you'll be less detectable, but might get rate limited.

Nice! Good luck with your project. The parsers are available under the utils.social namespace: https://crawlee.dev/api/utils/namespace/social The headless browser utils are under puppeteer and playwright utils https://crawlee.dev/api/puppeteer-crawler/namespace/puppetee...

The example uses Crawlee already, you can just remove the

import { Actor } from 'apify';

and then all references to Actor and either remove them or replace them with Crawlee functions.

E.g. await Actor.openKeyValueStore() should be replaced with KeyValueStore.open()

It makes sense to add a separate example for Crawlee though. But it's true that it does not exist yet.

You can use a headless browser (would recommend PlaywrightCrawler) to log in once and then use the session cookie until it expires in any crawler. When it expires, you can re-login and repeat the process.

Yeah, exactly. Using pure HTTP needs much less resources than running headless browsers, so unless you really need to use a browser, you can save a lot of compute power (and money) by using plain HTTP.

Funny, I never knew about node-osmosis. I like the API in the example. It's a shame that it's no longer maintained.

Thanks! We really love open source and wanted to give back to the community. Crawlee is built on top of other great open-source libraries and projects. It's the best thing about building software.

word!

You can use any captcha solving service with Crawlee, but we plan to add a plugin to make its use much easier. It's on our roadmap.

The ideal approach would depend on your architecture. It's really easy and cheap to create new queues on the Apify platform (we create ~500k every day) so we usually run a crawler per domain. It performs the best and it's the easiest to set up.

On Crawlee level, you can open new queues with one line of code and name them with the hostname, so the most straightforward solution would be to run multiple Crawler instances with multiple queues and then rate limit using the options explained here https://crawlee.dev/docs/guides/scaling-crawlers and push the new URLs to the respective queues using the URLs' hostname.

If you'd like to discuss this a bit more in depth, you can join our Discord or ask in GitHub discussions. Both are linked from Crawlee homepage.

In crawlee, you can use the useIncognitoPages option to create a separate context for each page https://crawlee.dev/api/browser-pool/class/LaunchContext#use... Not sure if it will be enough to offset your RAM requirements.

From our experience, RAM is not the limiting factor. It's the CPU. You need at least 1 CPU core for the modern browsers to work reliably at scale so if you're using a container that has 1GB ram and 0.25 core, it's just not worth it. If you have access to containers that have strong CPUs and not a lot of RAM, then it's a different story.

Thank you ;)

Stealth is already included by default, but I love the name :D

Thank you! :)

While I'm not particularly happy to see JavaScript begin taking over another field as it truly is an awful language, more choice is always better and this project looks valuable enough to make dealing with JS a worthwhile tradeoff.

Love that comment :D

Yeah, the ability to switch between headless and http is very important to us in production. We often hack something up quickly with headless and then later optimize it to use HTTP when we find the time.