jancurn

Added.

Here's the comparison table: https://github.com/apify/mcpc?tab=readme-ov-file#related-wor...

Exactly. Once you start looking at MCP as a protocol to access remote OAuth-protected resources, not an API for building agents, you realize the immense value

Cool, adding this to my list of MCP CLIs:

  - https://github.com/apify/mcpc
  - https://github.com/chrishayuk/mcp-cli
  - https://github.com/wong2/mcp-cli
  - https://github.com/f/mcptools
  - https://github.com/adhikasp/mcp-client-cli
  - https://github.com/thellimist/clihub
  - https://github.com/EstebanForge/mcp-cli-ent
  - https://github.com/knowsuchagency/mcp2cli
  - https://github.com/philschmid/mcp-cli
  - https://github.com/steipete/mcporter
  - https://github.com/mattzcarey/cloudflare-mcp
  - https://github.com/assimelha/cmcp

Yes!

Thank you! I guess I was fed up with lack of clients supporting enough MCP capabilities, so I had to build new one :)

You sent me down a rabbit hole: https://esolangs.org/wiki/APGsembly is mentioned in the book

Hey, this is Jan, founder of Apify.

We’re running a marketplace of 8,000+ tools called Actors for all kinds of web data extraction and automation use cases (see https://apify.com/store). Just last month, we paid out more than $500k to community developers who publish these Actors on the Apify platform.

The unit economics work for niche tools: scrapers for specific platforms, packaged open-source tools, MCP servers, or API wrappers. Too small for building SaaS around it, but developers earn a few thousand dollars per month as passive income.

We believe there can be many more such Actors. So we're putting $1M in prizes on the table to motivate developers to build new, useful Actors. Our bet is that 10,000 new specific tools can widely expand the capabilities of many AI agents and unlock a lot of value.

Hey HN,

This is Jan, the founder of Apify (https://apify.com/) — a full-stack web scraping platform.

With the help of Python community and the early adopters feedback, after an year of building Crawlee for Python in beta mode, we are launching Crawlee for Python v1.0.0.

The main features are:

- Unified storage client system: less duplication, better extensibility, and a cleaner developer experience. It also opens the door for the community to build and share their own storage client implementations.

- Adaptive Playwright crawler: makes your crawls faster and cheaper, while still allowing you to reliably handle complex, dynamic websites. In practice, you get the best of both worlds: speed on simple pages and robustness on modern, JavaScript-heavy sites.

- New default HTTP client `ImpitHttpClient` (https://crawlee.dev/python/api/class/ImpitHttpClient), powered by the Impit (https://github.com/apify/impit) library): fewer false positives, more resilient crawls, and less need for complicated workarounds. Impit is also developed as an open-source project by Apify, so you can dive into the internals or contribute improvements yourself: you can also create your own instance, configure it to your needs (e.g., enable HTTP/3 or choose a specific browser profile), and pass it into your crawler.

- Sitemap request loader: easier to start large-scale crawls where sitemaps already provide full coverage of the site

- Robots exclusion standard: not only helps you build ethical crawlers, but can also save time and bandwidth by skipping disallowed or irrelevant pages

- Fingerprinting: each crawler run looks like a real browser on a real device. Using fingerprinting in Crawlee is straightforward: create a fingerprint generator with your desired options and pass it to the crawler.

- Open telemetry: monitor real-time dashboards or analyze traces to understand crawler performance. easier to integrate Crawlee into existing monitoring pipelines

For details, you can read the announcement blog post: https://crawlee.dev/blog/crawlee-for-python-v1

Our team and I will be happy to answer here any questions you might have.

Hey all,

we’re publishing this whitepaper that describes a new concept for building serverless microapps called Actors, which are easy to develop, share, integrate, and build upon. Actors are a reincarnation of the UNIX philosophy for programs running in the cloud.

Our goal is to make Actors an open web standard. We’d love to hear your thoughts.

Here’s the corresponding GitHub repo: https://github.com/apify/actor-whitepaper

For this use case, you might use this ready-made Actor: https://apify.com/apify/website-content-crawler

For sure, simply store cookies after login and then use them to initiate the crawl.

Not yet, but it’s on the roadmap

Thank you!

There are many legitimate and legal use cases where one might want to circumvent blocking of bots. We believe that everyone has the moral right to access and fairly use non-personal publicly available data on the web the way they want, not just the way the publishers want them to. This is the core founding principle of the open web, which allowed the web to become what it is today.

BTW we continuously update this exhaustive post covering all legal aspects of web scraping: https://blog.apify.com/is-web-scraping-legal/

The main advantage (for now) is that the library has a single interface for both HTTP and headless browsers, and bundled auto scaling. You can write your crawlers using the same base abstraction, and the framework takes care of this heavy lifting. Developers of scrapers shouldn't need to reinvent the wheel, and just focus on building the "business" logic of their scrapers. Having said that, if you wrote your own crawling library, the motivation to use Crawlee might be lower, and that's fair enough.

Please note that this is the first release, and we'll keep adding many more features as we go, including anti-blocking, adaptive crawling, etc. To see where this might go, check https://github.com/apify/crawlee

Probably not :)

Last week we "officially" launched something that we worked on for the past few years.

The real motivation for this project comes from a simple frustration that it's not easy to make living out of developing your hobby open source software, even though it's being used by thousands of companies and brings them significant value. The only accessible way to make living from your software is by selling it as SaaS, which in turn is still overly complicated. We believe there is a better way and today we'd like to show it to you. I'm looking forward to hear what you think.

Today we're "officially" launching something that we worked on for the past few years.

The real motivation for this project comes from a simple frustration that it's not easy to make living out of developing your hobby open source software, even though it's being used by thousands of companies and brings them significant value. The only accessible way to make living from your software is by selling it as SaaS, which in turn is still overly complicated. We believe there is a better way and today we'd like to show it to you.

I'm looking forward to hear what you think.

Apify | Marketing & Growth, Product, ... | Prague | Full-time ONSITE | https://apify.com/jobs

Apify builds software technology and infrastructure that helps anyone leverage the full potential of the largest source of information ever created in the history of humankind - the Web. Our mission is to let people automate mundane tasks on the web and spend their time on things that matter. We strive to keep the web open as a public good and a basic right for everyone, regardless of the way you want to use it, as its creators intended.

Join Apify and help us make the web more programmable!

PayPal only just dodged this bullet. After they merged with X.com, the new CEO wanted to switch to Windows — even after PayPal cofounder Max Levchin showed that their software scaled only 1% as well on Windows as Unix. Fortunately for PayPal they switched CEOs instead.

It’s pretty funny to see that in 2006 Elon Musk didn’t even deserve a mention of his name. How things have changed...

Awesome, thanks!

Hello HN! This is Jan, co-founder of Apify. Web Scraper (apify/web-scraper) is our new actor in Apify Store which aims to be a general-purpose tool for web crawling and scraping. Under the hood, it uses headless Chrome and the data extraction is done using a provided piece of front-end JavaScript code, potentially with the help of libraries such as jQuery.

Please give it a try, it's free. We'd love to hear what you think - both good or bad things :)

Yes, it can! See https://apify.com/marketplace

Disclaimer: I'm a co-founder of Apify :)

Another sign the interweb as we know it is falling apart, https://www.example.com is not working anymore.

Apologies for sneaking into this thread. You can schedule batch jobs using Apify Actors and Scheduler. The actor can run any Docker image at times specified using a cron expression. Just have a look at:

https://www.apify.com/docs/actor https://www.apify.com/docs/scheduler

Disclaimer: I'm a co-founder of Apify

Thank you so much!

The SDK runs anywhere where you have Node running. And if you can run headless Chrome with Puppeteer there too, than you can use it in the SDK too. This might require several libraries and configuration settings. If I’m not mistaken, Google Cloud Functions support Puppeteer by default, AWS Lambda does not. With any Docker-based serverless platform such as Zeit Now or Apify Cloud you just need to use the right Docker image.

Awesome, looking forward to hear what you think :)

Hey guys, today we’re showing HN a new open-source library that we have been working on for almost a year. It incorporates lessons learned from scraping of thousands of websites over the last 4 years. We figured there was no such universal library for JavaScript, while for example Python has one (https://scrapy.org/). That wasn’t fair, because JavaScript is THE language of the web :)

Anyway, we hope you’ll give it a shot and we’re really looking forward to hear what you think about it. All feedback welcome!

It's nice to see serverless platforms adding support for headless Chrome. But there's still one problem with AWS Lambda / Cloud Functions / Zeit Now - the run time is limited to a few minutes. If you want to run any longer job, e.g. a web crawler, you need to either spin up the instances yourself or use platform like Apify, which allows running arbitrary-long jobs, provides pre-built Docker images for headless Chrome or XVFB, and provides SDK to simplify state persistence, access to proxies etc.

For example, a simple actor to convert HTML to PDF looks like this:

https://www.apify.com/jancurn/url-to-pdf

More info:

https://www.apify.com/docs/actor

https://www.apify.com/docs/sdk/apify-runtime-js/latest

https://www.apify.com/library?type=acts

Disclaimer: I'm a co-founder of Apify