HN user
The Chrome extension approach may shift some (most?) of the risk to the end user, since technically they are now the one scraping. Theoretically getdex would be relatively better off in this arrangement, while putting their customers into a legal gray area.
Related research on past litigation in this area for anyone that wants to go deeper:
https://deepnoodle.ai/research/linkedin-legal-battles-tos-vi...
Currently, yes. However it's quite early and I'm exploring whether this is the best option.
The reason I was posting here on HN is to see if anyone more technical may be interested in this as well. E.g. to leverage an API like this with AI agents or otherwise.
https://github.com/diveagents/dive
Dive orchestrates multi agent workflows in Go. Take a look and let me know what you think.
+1 to the article.
Along these lines, I'm building Dive:
https://github.com/diveagents/dive
When building a SaaS with a Go backend, it's nice to be able to have the option of the agents and workflows being in the same process. And being confident in the ability of that to scale well.
While it's true that Go lacks good ML libraries, for some this isn't too consequential if your app is primarily using Anthropic or OpenAI and a database that offers semantic or hybrid search for RAG. The ML is done elsewhere. Plus it could be that you can leverage MCP servers and at that point you're language agnostic.
Regarding the concurrency model approach with Go and agents, I initially baked a message based approach (a la the Actor model, with one goroutine per agent) into Dive Agents, but eventually found that this would be better implemented as another layer. So currently in Dive it's the user's choice on how to implement concurrency and whether to use messaging. But I anticipate building that back in as an optional layer.
Risor is an embedded scripting library and CLI for Go. This release adds new modules for web crawling, background scheduling, and more. Happy scripting.
Just a quick release announcement for Risor, an embedded scripting library and CLI for Go. Plenty of additions relating to web crawling, background scheduling, and more. Happy scripting.
I'm definitely with you that sites should be leveraging CDNs and similar. But I get that many don't want to do any work to support bots that they don't want to exist in the first place.
To me it seems like the companies actually doing the crawling have an incentive to leverage centralized caching. It makes their own crawling faster (since hitting the cache is much faster than using Playwright etc to load the page) and it reduces the impact on all these sites. Which would then also decrease the impact of this whole bot situation overall.
An aspect I find interesting is that these crawlers are all doing highly redundant work. As in, thousands of crawlers are running around the world, and each crawler may visit the same site and pages multiple times a week.
This seems like an opportunity for a company like Firecrawl, ScrapingBee, etc to offer built-in caching with TTLs so that redundant requests can hit the cache and not contribute to load on the actual site.
Even if each company that operates a crawler cached pages across multiple runs, I'd expect a large improvement in the situation.
For more dynamic pages, this obviously doesn't help. But a lot of the web's content is more static and is being crawled thousands of times.
I built something for my own company that crawls using Playwright and caches in S3/Postgres with a TTL for this purpose.
Does this make sense to anyone else? I'm not sure if I'm missing something that makes this harder than it seems on the surface. (Actual question!)
Nice. I was wondering if r53 latency-based routing would work as an optional add-on to this. I didn't look into it but was thinking it might not work because the latency based routing records are A records that have to point at a fixed IP address right (rather than the function url in my case). But presumably API gateway just gives you hostnames as well so maybe I'm mistaken here?
Oh I realized I was thinking of Cloudformation Stacksets, since those like you automate deployment of a Stack across multiple regions.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...
Aha, indeed. A few interesting notes on this. And thanks for the link to your Makefile.
I see you're a Cloudformation Stacks user (good stuff), so you may know some of what I say below already, but maybe the Terraform specific notes will be of interest.
Many AWS deployments (Terraform or otherwise) are single region unfortunately, with a primary factor there being that AWS APIs are for the most part region-specific. The AWS Terraform provider was built with this mindset too, unfortunately. Then the situation is complicated by Terraform's lack of support for dynamic providers, which is one of the most discussed still-open issues in Terraform.
Some related open issues:
https://github.com/hashicorp/terraform/issues/24476 https://github.com/hashicorp/terraform/issues/25244 https://github.com/hashicorp/terraform/issues/19932
My understanding is that people that have this as a first-class concern use something like Terragrunt to dynamically generate the HCL as needed.
https://terragrunt.gruntwork.io/
For Burrow here, I was was focused on shipping something and didn't want to spend time on generating that code just yet. But the project has gotten some attention so if this is a recurring request I'll probably do it.
Note that I intentionally chose the 17 AWS regions that are default enabled in every new AWS account, so it should only be if you want to customize it that this becomes important.
A Cloudformation Stacks based deployment to achieve multi-region would be another good (and perhaps better) option here. I could add that to the Makefile, with inspiration from your link there.
Cloudflare Workers are great in general. The way they route client requests to the nearest Cloudflare data center means that a single client will most often end up with a small number (perhaps even 1) of public IP addresses coming out of Cloudflare. And to my knowledge, there's not a way to ask for your client to be routed in a different way. So due to this, AWS Lambda's inherent region-specific behavior is actually an advantage for Burrow, since we can ask to be routed through a specific region (no matter how distant) by virtue of hitting the per-region Lambda URLs.
That said, you could actually layer the two together if you were so inclined, and/or optionally route through Cloudflare and AWS.
Making the Cloudflare Worker implementation of this would be trivial and there's no dependency on Go in any way for the proxy itself.
Source code for the testing is available here:
Risor is an embeddable scripting language for the Go ecosystem. This blog post introduces its opcodes and shows how it was influenced by CPython.
GitHub project here:
Source code available here:
https://github.com/pgedge/pgedge-northwind
You can also read more about how this works in the companion blog post:
https://www.pgedge.com/blog/pgedge-cloud-cloudflare-workers-...
Hi HN!
You might be interested in pgEdge Cloud's new free offering, which includes 3 vCPU, 12 GB storage, and a 3 region cluster.
This is distributed, multi-region, multi-writer PostgreSQL.
The situations where our approach resonates with people the most:
* Ultra-high availability for SaaS
* Low-latency access, especially in multi-region scenarios
* Meeting data residency requirements
* Use with edge platforms like Cloudflare Workers
Our engineering team is a mix of folks that were previously at AWS RDS, plus others coming from a cloud security and IaC background. We're really interested in getting feedback from the community and would love you to give our free "Developer Edition" of pgEdge Cloud a try.
You can check out my blog post to learn more about the launch and see links to our various open source projects.
A quick demo video is available here: https://www.pgedge.com/video/pgedge-cloud-developer-edition-...
Thanks for taking a look! - Curtis (VP Product, pgEdge)
Take a look at Risor and its pipes capability.
https://github.com/risor-io/risor#quick-example
Stay in the Go ecosystem, but gain pipes, Python-like f-strings, and more.
(I'm the author)
Agreed!
Shameless plug: this is why I built Risor.
https://github.com/risor-io/risor
Keep in the Go ecosystem, retain compatibility with the Go programs you already have, but have a much more concise scripting capability at your disposal.
Yeah. I know Elixir has its pipe operator |> which operates similarly. I don't know how much this really plays a part, but because "|" is so commonly used for the bitwise-or operator, you then have to choose another token to use for it. With Risor I made the judgement that bitwise-or can be handled in some other way, since it's less important than getting a concise pipe expression.
Yes indeed. FWIW, I'd love any feedback on the project if there are things you like or dislike, in your first impression.
All good questions. Risor hasn't really fully "launched" yet and its documentation is incomplete at this point.
GCP and Azure to be added in the coming weeks. Here's a related GitHub issue:
https://github.com/risor-io/risor/issues/65
For now, you can browse the available modules in the source tree here:
https://github.com/risor-io/risor/tree/main/modules
Please open an issue on GH if you have other module requests! MySQL is a logical one but I was waiting for someone to ask :-)
What this is referring to is that the AWS SDK is built into the Risor binary. This means that using Risor you can call any AWS API without needing to install extra packages/dependencies/etc.
Nice. Somehow this is the first time I'm seeing Zygomys.
If you're not in love with the lisp syntax, you might want to take a look at Risor as well.
https://github.com/risor-io/risor#quick-example
array := ["gophers", "are", "burrowing", "rodents"]
sentence := array | strings.join(" ") | strings.to_upper
print(sentence)Spock may be of interest to you: https://pgedge.github.io/spock/
I’ve been working on Tamarin, a new embedded scripting language for Go projects.
Great book. This was the inspiration for Tamarin. I've been working on this as a side project for about a year now.
+1 to this. In our case we say "deploy all" but use Zim[1] to automatically determine which services have associated changes. This keeps the overall deploy quick.
This is comparable to CloudFormation or Terraform in terms of determining whether something is up-to-date, but more general purpose.
Zim was inspired by all of these and has drastically cut build times for our team. We use it to build hundreds of Go, Python, and Node artifacts for use in AWS. https://github.com/fugue/zim