Rive, Fast and reliable background jobs in Go

https://github.com/riverqueue/river

by mountainview • 1 month ago

44 15 1 month ago

github.com

jeffbee2• 1 month ago

They avoided all those pesky distributed systems problems by making a system that is not distributed. Hell of a claim.

brandur3• 1 month ago

Blake wrote a nice page on the benefits of using transactional-based enqueuing here:

https://riverqueue.com/docs/transactional-enqueueing

It's true that it's not distributed, but there are a lot of benefits to not going distributed immediately, like extremely predictable data consistency. I would hazard to guess that the _vast_ majority of apps that are not built by the superscalers are already using a database like Postgres or SQLite to store their data, and River merely suggests that you hook your job queue into the database that you already have.

jedberg• 1 month ago

DBOS recently wrote a great blog post about why you should colocate your workflow data with your application data:

https://www.dbos.dev/blog/co-locating-workflow-state-with-yo...

jeffbee2• 1 month ago

I just think it's an odd line of argument. The system also avoids most if not all of the pitfalls of hydroponic cultivation. Because it categorically is not that.

havercosine2• 30 days ago

Well spotted but I don’t think it’s bad trade off. A beefy postgres instance (with standby configured), couple of worker nodes running dbos / river directly as a library backed by same db . This system can go surprisingly far.

I’ve seen and used airflow , spark , temporal for many systems. I’d def pick this simpler choice for 95% workloads these days.

leetrout• 1 month ago

Typo in title... it's River not Rive

brandur3• 1 month ago

Yes, if one of the admins/mods could correct this, it'd be amazing! The rest of the title is fine.

mountainview• 30 days ago

Ah, sorry about this. I cannot change it anymore.

hasyimibhar• 1 month ago

How does it compare to a full-fledged durable execution platform like DBOS[0], which follows the same philosophy? Looks like River does have workflows, but it's locked behind Pro [1].

[0] https://dbos.dev

[1] https://riverqueue.com/docs/pro/workflows

brandur3• 1 month ago

There's some similarities with DBOS for sure like using Postgres as a backend. Here's some differences based on my browsing of their docs for a little while:

* River's built around an entirely open core with its bread and butter being background jobs rather than workflows, with basic background jobs being good enough for most apps in most situations. It's true that workflows are gated behind Pro, but a lot of users will find they won't even need them, or won't need them until much later.

* River's aimed more solidly at Go, especially for the running of the background jobs themselves. Blake and I are both experienced Go developers, and we've gone through great pains to make the API as elegant as possible and as easy-to-use as possible, aiming for things like consistency and predictable + well-documented APIs. DBOS supports Go as well, but I believe our API compares very favorably [1], though you can be the judge.

* I might be missing something in the DBOS docs, but especially pertaining to background jobs, I believe River's feature set is quite a lot more comprehensive. e.g. Bulk insertion, unique jobs, periodic/cron jobs, job snoozing, job scheduling, unique jobs, test helpers, etc. We've tried to include everything that people would need when building out with background jobs, including all the edge cases.

Lastly, to be fair, DBOS is price-gated as well [2], and pricing is based on usage whereas River's is not.

[1] https://docs.dbos.dev/golang/programming-guide [2] https://dbos.dev/dbos-pricing

whinvik• 1 month ago

Curious how people use systems like this or DBOS etc.

Do you self-host? Do you use the same DB as the application DB? Do you use this for API background jobs, batch jobs, cron jobs?

As someone who uses Airflow a lot but not for background jobs, I am interested in the pros and cons of various approaches.

havercosine2• 30 days ago

Yes and yes. DBOS self hosted , pointed to same application DB. Systems like these (durable execution) fit well with agents or LLM driven on demand workflows. You might be attaching multiple flaky local or remote api call tools. Many complex LLM workflows that you don’t want to one shot but break down into chain of prompt can be added to river/dbos + LLM , providing much needed retries and bit of concurrency control.

Airflow perhaps fits better with scheduled recurring ETL workflows.

absoluteunit1• 1 month ago

I was just asking on the Go subreddit for suggestions and River came up.

My only hesitation is that many features are locked behind Pro.

I normally only settle for completely open source tools that I can self host and try to avoid these “partially” open tools.

For now, going to try to migrate my current stack to RabbitMQ. Seems like a very mature and completely open source tool.

River does look cool but I’m not a fan of vendor lock in. Will always avoid when possible

dwrodri• 1 month ago

I was asking around a while ago about better single-node solutions to "durable workflows" than Airflow before I was familiar with the terminology. Admittedly, Go + durable workflows in the cloud feels like the perfect marriage. Seeing the gated features makes me hesitant, unfortunately, but definitely considering this!

TN1ck• 30 days ago

I’ve used asynq [1] at my past job. And although development is not as active anymore, they have a new contributor and we never encountered any bugs. It has been really solid and we relied on it heavily.

[1] https://github.com/hibiken/asynq

Rive, Fast and reliable background jobs in Go

Related Stories