sorentwo

As a library maintainer, skill and taste are almost equally important. If I can’t recognize inefficiencies, difficult to maintain code, or generally unpleasant code smells, then people lose trust in my libraries/products and it’s no better off than some recently generated slop.

Years of production experience, wisdom, and using something in anger matters for both skill and taste.

The efforts we've undergone to make Oban (and Pro) work with CRDB have been ridiculous. Feature detection all over because of a lack of common operators and functions that can't be used in indexes. The worst is the rampant "serialization_failure" errors that force continual transaction retries. Not how I'd suggest scaling Postgres.

That said, as a predecessor to dbos in building durable workflows just using Postgres, I concur with the overall sentiment.

The moment that OpenCode, after helping fix a Dockerfile issue, decided it was time to deploy to prod without asking for consent, I was out.

Nearly this, but using ghostty instead of tmux. You don’t get the remote connection aspect of tmux, but for splitting/zooming/preserving windows it is fantastic. The best part is you can configure natural shortcuts rather than using a leader for everything.

There is one for elixir that is _mostly_ compatible with oban-py. Full compatibility, and potentially native hosting, are goals before 1.0

https://github.com/oban-bg/oban_web

With a typical Redis or RabbitMQ backed durable queue you’re not guaranteed to get the job back at all after an unexpected shutdown. That quote is also a little incorrect—producer liveness is tracked the same way, it’s purely how “orphaned” jobs are rescued that is different.

You can have jobs that run as long as you like. The difference is purely in how quickly they are restored after a crash or a shutdown that doesn’t wait long enough.

This is absolutely true (except we went OSS + Web initially, Pro came later). You were an inspiration, always helpful in discussion, and definitely paved the way for this business model.

The vast, vast majority of Python libraries are not async-friendly and most still rely on the GIL. On the other hand, Celery has absolutely no asyncio support at all, which sets the pro feature apart.

That's great advice. Wish we'd been in contact before =)

Transactions around fetching/updating aren't trivial, that's true. However, the work that you're doing _is_ regular activity because it's part of your application logic. That's data about the state of your overall system and it is extremely helpful for it to stay with the app (not to mention how nice it makes testing).

Regarding overall throughput, we've written about running one million jobs a minute [1] on a single queue, and there are numerous companies running hundreds of millions of jobs a day with oban/postgres.

[1] https://oban.pro/articles/one-million-jobs-a-minute-with-oba...

There are other projects that implement the ideas in OSS, but that's the same in Elixir. Not that we necessarily invented DAGs/workflows, but our durable implementation on the Elixir side predates DBOS by several years. We've considered it an add-on to what Oban offers, rather than the entire product.

Having an entirely open source offering and selling support would be an absolute dream. Maybe we'll get there too.

It supports workflows, rate limiting, unique jobs, bulk operations, transactional enqueuing, etc. Why not move these things to the OSS version to be competitive with existing options, and focus on dedicated support and more traditional "enterprise" features, which absolutely are worth $135/month (the Oban devs provide world-class support for issues).

We may well move some of those things to the OSS version, depending on interest, usage, etc. It's much easier to make things free than the other way around. Some Pro only features in Elixir have moved to OSS previously, and as a result of this project some additional functionality will also be moved.

Support only options aren't going to cut it in our experience; but maybe that'll be different with Python.

There are many more options available in the Python ecosystem than Elixir, so you're competing against Temporal, Trigger, Prefect, Dagster, Airflow, etc etc.

There's a lot more of everything available in the Python ecosystem =)

Oban has been a lifesaver for me and it is the tool I miss the most from the Elixir ecosystem when doing work in Python

That's wonderful to hear! Hopefully you can make use of Oban in both places now =)

I have one question: are there any plans for interop between Oban and the new Django Tasks[1]?

There aren't any plans at this point, but it's certainly possible.

Pleased to see this posted! A lot of design time and effort behind this project (something we'll be speaking about this year). Happy to answer any questions people may have.

Seconded. I was going to say the exact same thing. Brilliant thought exercise that I still think about on a weekly basis 20 years later.

Before looking at the zoo I figured there would be a dozen or so engines compared. Seeing the actual comparison is astounding!

The amount of work just to aggregate and compare is admirable, let alone the effort behind the engines themselves.

It is an old US military term that means “F*ked Up Beyond All Recognition”

Just as a whole the Erlang and Elixir primitives allow oban to be built truly in the most retarded, obvious way and get away with it.

Maybe it is obvious in retrospect…

This is absolutely true.

I can confirm, from firsthand knowledge, that Elixir is used at dozens of Fortune 500 companies in the US.

The roadmap is purely about AI, and reads like it was written by AI. It’s purely trendy and myopic.

The architecture is remarkable. The lengths they’ve gone to for language version compatibility, and protecting app namespaces is especially impressive.

https://github.com/elixir-lang/expert/blob/main/pages/archit...

Absolutely love monodraw for diagrams in documentation! All of the diagrams for Oban and Oban Pro are done this way:

Job Lifecycle: https://hexdocs.pm/oban/job_lifecycle.html

Composition: https://oban.pro/docs/pro/1.6.4/composition.html

Oban doesn't use advisory locks for fetching jobs (unless there is uniqueness involved)—it uses `FOR UPDATE SKIP LOCKED` as well to pull jobs.

The problem was with restrictive connections, not DNS based discovery for clustering. It wasn't possible (as far as I'm aware) to connect directly from one dyno to another through tcp/udp.

Ping requires something persistent to check. That requires creating tuples, and most likely deleting them after they’ve been consumed. That puts pressure on the database and requires vacuuming in ways that pubsub doesn’t because it’s entirely ephemeral.

Not to mention that pubsub allows multiple consumers for a single message, whereas FOR UPDATE is single consumer by design.

In app notifications, which can be disabled. Our triggers were only used to get subsecond job dispatching though.

Very very few applications use mnsesia. There’s absolutely no way I would recommend it over Postgres.

We have Postgres based pubsub, but encourage people to use a distributed Erlang based notifier instead whenever possible. Another important change was removing insert triggers, partially for the exact reasons mentioned in this post.

Postgres LISTEN/NOTIFY was a consistent pain point for Oban (background job processing framework for Elixir) for a while. The payload size limitations and connection pooler issues alone would cause subtle breakage.

It was particularly ironic because Elixir has a fantastic distribution and pubsub story thanks to distributed Erlang. That’s much more commonly used in apps now compared to 5 or so years ago when 40-50% of apps didn’t weren’t clustered. Thanks to the rise of platforms like Fly that made it easier, and the decline of Heroku that made it nearly impossible.

Shaun Ryder sets a low bar. Definitely the best maracas player though.