Faktory, a new background job system

http://www.mikeperham.com/2017/10/24/introducing-faktory/

by mperham • 9 years ago

194 53 9 years ago

Looks very similar to beanstalkd, the only difference I can see is that Faktory comes with a bundled GUI. Is there anything else I'm missing?

aarondf• 9 years ago

From the FAK:

Faktory aims to be more feature-rich and better supported. Many of Faktory's OSS competitors are "dead" and no longer supported. I am fortunate enough to have both expertise in background jobs and a business model to support Faktory long-term.

falcolas• 9 years ago

Background job systems rarely need extensive support (or, for that matter, a constant stream of features) I've used Gearman for years, to great success.

Not saying that Gearman is a better solution, just saying that "better supported" for such a relatively simple tool is not always necessary.

jsjohnst• 9 years ago

That it’s written by the same guy behind Sidekiq, a very successful Ruby framework for background jobs.

switchbak• 9 years ago

I (unfortunately) use beanstalkd, and it seems to be pretty dead, at least on the Java client side.

My question is why to we even need a 'background job' system, isn't this just a message broker / queue? RabbitMQ (and friends) can do much of this, no? Maybe I'm missing some of the future features they aim to implement.

sandGorgon2• 9 years ago

It uses Facebook's high-performance RocksDB embedded datastore internally to persist all job data, queues, error state, etc

This breaks the devops story for me. The difference in featureset between Rocksdb and redis is not that big.. However redis is hugely supported on the cloud and in high-availability fully-managed mode.

Its so convenient to use sidekiq on heroku or aws. I really hope you build this on redis rather than a new persistence server.

deedubaya4• 9 years ago

Correct me if I'm wrong, but RocksDB is embedded withink faktory, so this would actually be one less service to manage than using an external redis server. Isn't that a good thing?

From my experience, when running high throughput, quickly executing sidekiq workers on heroku, the expense often doesn't come from dynos, but the redis instance as the limiting scale factor usually comes down to connections. That won't be a problem with Faktory and an embedded datastore.

wilde• 9 years ago

It depends on how comfortable you are developing a monitoring and failover system for something stateful. Right now AWS handles failover of our redis instances. With Faktory, I’m not sure what that looks like yet.

meritt2• 9 years ago

Can you provide a bit more details about the limits you're encountering? Connections? As in the sheer quantity? Or speed to connect? Are you hitting linux open files limits? I have a tough time understanding how Redis would be the bottleneck among a bunch of Ruby workers.

sandGorgon2• 9 years ago

This is not the state of mind right now - and this is the same thing that blockchain devs seem to think. Everybody uses the cloud for anything reasonably important. And if it is not important, the difficulty delta between an embedded datastore and "./redis" is miniscule.

Which means that leveldb/rocksdb gives zero path to scalability at the cost of saving a maximum of a few minutes of effort.

At worst, this could have been done in redis with embedded lua packaged together. That would have solved both your external redis problem as well as a reasonable path to scale.

rubiquity• 9 years ago

When you peel back a layer or two from the background job onion you land smack dab in the land of message queues. I’ll be curious to see what trade offs this library makes around classic message queue problems such as delivery semantics, visibility windows, sharding or replication for multi node, etc.

There’s definitely a lot of message queues out there these days and things like Kafka which turn into message queues. That said, a lot of them are a pain to operate so there’s room for one that is easy to deploy and operate.

odammit• 9 years ago

I’ve used Sidekiq for years. Great project. Looking forward to giving this a swing.

I really respect what Mike’s done being able to monetize his work on cool open source projects.

deedubaya4• 9 years ago

When Mike released Sidekiq for Crystal-lang, I thought this might be in the works. From the readme:

The Ruby and Crystal versions of Sidekiq must remain data compatible in Redis. Both versions should be able to create and process jobs from each other. Their APIs are not and should not be identical but rather idiomatic to their respective languages.

It makes sense to unify the protocol for background job processing, making them language agnostic. Some languages tackle different problems better than others, so this will be a really useful tool.

Great work, Mike. Keep the hits rolling.

manigandham8• 9 years ago

You can just use Redis, Kafka, or better yet use Google's Cloud Pub/Sub or Azure Service Bus for an extremely cheap and highly-reliable system that already has all of these features included.

What exactly is the use-case for an entirely new and separate system that looks like it's single-node only?

karmajunkie3• 9 years ago

I had a similar question, given that I'm happily processing jobs in Elixir that are enqueued by a Ruby app via Sidekiq. After taking a look at the README's, it looks like what this gets you is dumb clients, with the logic around retries, etc built into the server, rather than forcing the client to handle these details. That's great for polyglot systems—the dumber the client the better, in my book. As another commenter posted, I'll be interested to see more about what tradeoffs it makes in distribution.

If your current system is already something like SNS/SQS or an actual message queue with acks, then this probably isn't aimed at you.

manigandham8• 9 years ago

Why run something extra when you can use a single database table which is already universally accessible by any dumb client?

Push, fetch, ack = insert, select, delete. A few lines of SQL or a stored procedure gets it done. Postgres basically made skip locked for queues.

jitl2• 9 years ago

It seems nuts to me to introduce a stateful queue system like this where the persistence story is “data’s out there on a single node in some kind of disk format, hope it doesn’t go missing!”

I guess this style significantly reduces setup friction, but it’s almost an irresponsible design in a universe where the average cloud provider is telling you, “your compute instances may vanish at any time.” If this used Kafka, Redis, MySQL, or another well-known stand-alone data store, I know my data is replicated, and I can recover my enqueued jobs if Somethimg Bad happens.

I like the nice UI and the simple API, but there’s no way this will replace Resque or any Kafka-based-job-whatever at my place of work.

mperham6• 9 years ago

It supports backup and restore today but RocksDB doesn't have a replication story at the moment.

Side note: Resque will lose jobs if it crashes, it doesn't use RPOPLPUSH. I hope you'll take another look at Faktory in a few months, maybe we'll have addressed your issues.

koolba2• 9 years ago

Using RPOPLPUSH there's no guarantee you won't lose the job if the Redis server crashes prior to publishing it. Even with persistence enabled (AOF) there's still a small window of time between receiving the message and the next AOF fsync. The sender would think the message is queued as it got an OK response back but the message will be lost.

There are three common solutions to this:

First is to pretend it doesn't exist (quite common!).

Second is to understand that losing messages is a possibility and only use it for message types that the loss of a message wouldn't be critical (which covers quite a bit).

The third solution, which actually addresses the problem, separates the persistence of message details to transactional store from the event notification of a new message. A "sweeper" type task can thing check the persisted message list for messages that haven't been processed and re-publish them.

(Disclaimer: I'm a huge fan of Redis and think it's the bee's knees of data structure servers.)

jitl2• 9 years ago

Fair, and I will :-) Thanks for your response. Are you married to RocksDB/instance storage, or are you considering adapters or something to a pluggable persistence layer?

zellyn• 9 years ago

Do you have plans for multi-node persistence / geographic redundancy? We'd love something like this, but right now it would probably be easier to roll something custom on top of either Kafka or CockroachDB…

nikolay2• 9 years ago

How does this compare to https://github.com/antirez/disque?

manigandham8• 9 years ago

Disque was never officially released and is considered deprecated now. Redis already does well as a queue, v4.0 came with modules which add even more functionality, and future releases will include a new streams datatype, similar to Kafka.

nikolay2• 9 years ago

Thanks for sharing this - I've obviously missed these developments.

lobster_johnson• 9 years ago

This looks nice, but I'm also confused by why anyone would build a single-node data store in 2017. I can't find any information about whether replication and HA/failover is planned. Writing a queue on top of RocksDB is arguably trivial; it's the other stuff that is difficult.

The other thing about job systems that a lot of people seem to ignore is client-side scaling. We run our apps on Kubernetes, where you'd naturally want to tune worker scheduling dynamically to accomodate queue size. Feeding custom queue metrics into the horizontal pod autoscaler is one way to do this.

pmontra• 9 years ago

Many web applications are so low traffic that a single node is enough to handle everything: application server, database, queue system, static content, etc. It's up to the business deciding if they want to spend the money to buy high availability. In my experience almost everybody don't and have lived years with only occasional downtimes that didn't harm their business.

sscarduzio2• 9 years ago

yes but downtime is different than data loss

mperham6• 9 years ago

For those of you pointing out flaws, remember you are picking apart a pre-1.0, just launched project. We're all hackers and startup people right? Think MVP, no one can ship a perfect system completely finished.

manigandham8• 9 years ago

The only flaw here seems to be a lack of use-case... why not just add an HTTP API to Sidekiq instead, if it doesn't already have it? That way the value of the queue logic and semantics can be offered to any external client.

mperham6• 9 years ago

Sidekiq is not a server, it's a Ruby worker process. Redis is the server and I can't build a fast, easy to use embedded queue system on top of Redis.

geetfun• 9 years ago

Thanks Mike for creating Sidekiq and this. pre-1.0 is alright with me for showing it on HN.

blairanderson• 9 years ago

Just remember, grumpy HN comments are .001% of people viewing the homepage and clicking the link to your site.

Your potential customers know better than to get their opinions from reading these comments :)

twic• 9 years ago

Aren't the flaws flaws in the concept, rather than the implementation, though? Are those likely to be improved on?

yeukhon• 9 years ago

This is a strong defensive stand, especially coming from the author. Picking things apart with you is better than a "great work and now next post" comment.

continuations• 9 years ago

On a related matter, how does job queue differ from message queue?

What can a job queue do that Kafka or RabbitMQ cannot do?

manigandham8• 9 years ago

There is no difference. A "job" is just a message to a worker process describing what to do and maybe some data or arguments to be used in doing that work.

There are some useful features like priorities, scheduling in the future, claiming and then acknowledging a job as done or returning to pool, etc... but these can all be found in or coded on top of existing systems.

mperham6• 9 years ago

Background jobs are specialized messages. You can build a decent job system on top of a message queue but you'll lose the specifics. For instance, Rabbit and Kafka won't give you the built in retry system with error tracking and UI. Faktory enforces a specialized message format (the job payload) and can do many things with that data; message queues that treat messages as a simple byte array can't do that.

https://github.com/contribsys/faktory/wiki/The-Job-Payload

perfect_kiss• 9 years ago

One of the key features I love beanstalkd for, is a support for assigning different priorities for the jobs in the same queue, so the jobs with larger priority assigned will be processed first. Looks like this feature is missing from Faktory.

mperham6• 9 years ago

https://github.com/contribsys/faktory/issues/new

languagehacker• 9 years ago

Mike makes some amazing software in this space. I have some mostly pragmatic concerns about building out a framework that requires some specialized server to handle state for workers, though.

Maybe for something like RabbitMQ or SQS, this would be a satisfactory replacement, since these seem to be relatively monotasked persistence servers. So for your traditional Celery+RabbitMQ deployment, for instance, this could be a good replacement.

But let's we consider cases like Redis, Memcached, or Kafka, where the persistence store we're using is often also being utilized as a cache or linear log in other aspects of the same product. This would make Faktory troublesome, because it introduces additional maintenance costs compared to a service that we already need. Furthermore, if I can use an off-the-shelf, hosted storage solution for enqueued job definitions, like we do with Elasticache, I reduce my operational costs even more.

So it's not a question of whether or not Faktory works, but whether it's worth the cost of building, deploying, and maintaining a specialized monotasker server instance on top of the worker pool I already need to build and maintain. I'd be interested in understanding where the long-term value add would be in most large-scale practical SOAs, and how folks excited about this project anticipate implementing it might go.

sscarduzio2• 9 years ago

This would be great as a layer on top of SQS and/or GCE PubSub. And I'd host Faktory in a Lambda or CloudFunction.

Faktory, a new background job system

Related Stories