rustdeveloper

Happy to suggest another web scraping API alternative I rely on: https://scrapingfish.com

Interesting data for sugar in products scraped from Walmart: https://scrapingfish.com/blog/scraping-walmart

This is correct, my friends from Scraping Fish are hosting https://compareproxy.com to help people find proxy for web scraping. I'm happy to "push" for Scraping Fish as I'm also a satisfied user who received a lot of help from the founders for my web scraping projects.

And many more: https://compareproxy.com/

For web scraping at scale you want to get lost in the crowd. This usually means being (or pretending to be) chromium on windows. Unusual browsers are suspicious, detected or have very distinct fingerprint.

This is a terrible news :( I know it was an option for web scraping and I used in once. I’m curious what is the real reason they took it down.

Surprisingly, according to you tool, HN is neutral on “web scraping”. I noticed others also reported bias for neutral on other keywords.

There are also SaaS products with usage based pricing. It depends on what SaaS or software or product it is. Different pricing model works for different things.

Actually, they do allow this. I store my photos and iPhone backup on synology NAS. You just have to devote some time to set it up yourself.

I'm using Scraping Fish because of their pay-as-you-go style pricing as opposed to subscription with monthly scraping volume commitment. And they don't charge extra credits for JS rendering or residential proxies because the cost of each request is the same: https://scrapingfish.com

This reminds me of https://scrapingfish.com/blog/are-most-rust-jobs-in-crypto :)

Do we know how LLMs available in OpenLLM and other open source LLMs compare to different versions of GPT models? I know there’s a leaderboard on huggingface: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... but it doesn’t contain GPT models.

Does it give access to the internet? Could it be used with this setup: https://scrapingfish.com/blog/byo-mobile-proxy-for-web-scrap... ?

For $20 you can get over 10k requests from scraping fish: https://scrapingfish.com/buy

I don’t see how any LLM would help me with a high quality proxy, which is what I actually need in web scraping and I’m using https://scrapingfish.com/ for this.

There are tutorials for building you own mobile proxy pool so it’s very accessible these days: https://scrapingfish.com/blog/byo-mobile-proxy-for-web-scrap...

I'll be working on my own mobile proxy build: https://scrapingfish.com/blog/byo-mobile-proxy-for-web-scrap...

I initially built a system for web scraping but was constantly running into issues of getting blocked, even when using good quality residential proxies. I had to constantly investigate why I'm getting blocked and update tools. Sometimes the effort was significant when I had to switch to a different framework which was giving me a better success rate.

Then, I switched to web scraping API (I'm using https://scrapingfish.com as they have convenient pricing for my use case, but there are other alternatives). Now I only have to maintain parsing logic in scrapers. It also actually reduced my costs of scraping since I no longer pay for proxies which are more expensive for my scale than a web scraping API.

This looks really cool! There was a tutorial posted on HN about building mobile proxy pool with RPI that had obvious limitations: https://scrapingfish.com/blog/byo-mobile-proxy-for-web-scrap... It seems this could be a solution to scale capabilities of a single RPI.

For web scraping, I recommend using a web scraping API, e.g. https://scrapingfish.com. This solves all potential problems with getting blocked and can make data extraction easier as well.

For the app, I've recently started using Remix (https://remix.run) and so far it seems to have been a good choice for me. There is a good integration with Remix in Mantine for front end: https://mantine.dev/guides/remix/. I think it's a good full stack choice if you just want to quickly build an app for your project/product.

I use Scraping Fish API: https://scrapingfish.com/

“This industry makes scraping available to individuals and companies that otherwise would not have the capabilities.” - seems like web scraping companies are doing a good job :)

Looks interesting. Will try the script for web scraping.

I think that if you don’t want to invest a lot of time into learning web scraping and money to get a pool of residential, or even better mobile, proxies it’s easy to quickly get good results with web scraping API like https://scrapingfish.com They have good blogposts, for example, for how to scrape public Instagram profiles: https://scrapingfish.com/blog/scraping-instagram

I don’t see this happening. If you know rust you can use it elsewhere.

I didn’t find so many crypto job offers for python. But your right, I’m curious how it looks for go or cpp.

I was looking for a rust job recently and got the same impression that most of them are in crypto :|