Ask HN: Good pub/sub setup for personal projects?

https://news.ycombinator.com/item?id=32077690

by rschachte • 4 years ago

14 12 4 years ago

news.ycombinator.com

I’m messing around with a single node k8s homelab I’m running for fun side projects and micro service development.

I’ve done a lot of work with Kafka, but it’s pretty heavy and I really just want a decent pub/sub queue setup between services that would be somewhat reliable.

Is there anything you’d recommend? Open to thoughts on just using Postgres as a queue as well.

fendy30022• 4 years ago

uhm, rabbitMQ?

the problem is just Kafka being heavy?

metadat2• 4 years ago

Or a directory of text files, or a relational DB like postgres...

The options are many. It doesn't really matter, just do whatever will meet your needs!

fendy30022• 4 years ago

while possible, those are hard to develop to be a reliable pub/sub messaging though.

hallway_monitor• 4 years ago

Probably this. Rabbit is super easy to set up. Super lightweight. You can give it a try in half an hour.

wdb• 4 years ago

Maybe Redpanda or Redis pub/sub?

rektide• 4 years ago

Not entirely a real suggestion, but if you want really ultra-fun & easy, patchbay is the most awesome, simplest, easiest-to-use & most versatile/multi-purpose http service for relaying things around/queuing that I know.

Atm it's all in-memory, afaik. There are go & node.js implementations, which I hope some-day we add persistence to.

I'm still working on k8s Persistent Volumes, but in terms of bigger/better software, I hope soon-ish to have a Pulsar Operator[2] going. Kafka is very very exacting & specific, inflexible, but Pulsar's architecture seems much more general & flexible, capable of things like great geo-distribution by virtue of it's underlying Bookkeeper storage layer. It'd be my production go-to for any pubsub details.

[1] https://github.com/patchbay-pub https://patchbay.pub/ https://news.ycombinator.com/item?id=21639066 (362 points, 3 years ago, 79 comments)

[2] https://docs.streamnative.io/operators/pulsar-operator/pulsa...

remram• 4 years ago

Redis is extremely easy (single binary), though that might not be super compelling if you can run containers. RabbitMQ is good too and has additional features like automatic requeueing if the client stops before acknowledging (and more complex routing rules etc). Both have clients in many languages.

pramsey• 4 years ago

I had a good time with the pq module for a pgsql/python [1] queue. There's a certain brutal simplicity to just having one persistent process, the database, to manage.

[1] https://pypi.org/project/pq/

winrid• 4 years ago

Why do you want to use a queue between services for a personal project? Just use RPC or similar?

Unless you want to play with queues... otherwise I would use the database, and add redis or rabbitmq later.

boloust• 4 years ago

At one end of the spectrum: ZeroMQ[1]. No broker to run, and everything can stay within your local network. Just add the library to your microservices and expose a port. It gives you a socket interface with well-engineered queuing primitives. It's enough for most side projects in its simplest form, while also giving you the flexibility to implement a huge number of distributed queuing patterns.

At the other end of the spectrum: just use a managed queue from your preferred big cloud provider. All are reliable enough for this use case, and the cost for most projects would be zero or negligible.

[1] https://zeromq.org/

joshxyz• 4 years ago

redis streams are cool

Ask HN: Good pub/sub setup for personal projects?

Related Stories