ZeroMQ: High-Performance Concurrency Framework

https://zeromq.org/

by klaussilveira • 2 years ago

95 55 2 years ago

zeromq.org

thesuperbigfrog• 2 years ago

ZeroMQ was followed by nanomsg (https://nanomsg.org/) and nng (https://nng.nanomsg.org/)

Some of Martin's rationale here:

https://250bpm.com/blog:23/index.html

ZeroMQ is still widely used and popular, but I am not sure if it is still actively developed.

jvanderbot2• 2 years ago

I've used zeromq, nanomsg, and nng. The differences are subtle and focused on native library support, background threading model, and other systems level things.

All of them are based on specs that are widely published. I've had zero problems implementing real robotic systems in nng, zmq, etc.

And it is so damn easy to use it's amazing to me the whole world doesn't use it.

throwaway984393• 2 years ago

I'm pretty sure the world doesn't use it more because it doesn't have a flashy marketing campaign and trendy developer tools/libraries don't have a plugin for it. If a whole bunch of developers aren't writing blog posts about it, does it even exist? Plus, it's old, so it's bad.

AnonHP• 2 years ago

And it is so damn easy to use it's amazing to me the whole world doesn't use it.

Perhaps many of them are locked into “the cloud” and “serverless”, by default choosing the proprietary solutions offered by these providers on their platforms?

rcxdude2• 2 years ago

I've had some bad experiences with zeromq in robotic systems contexts: it's very assert-happy, and therefore tends to bring down the whole process in corner cases, and it's quite difficult to debug. It caused me quite a lot of headache and I'm no longer particularly enamored of the approach (the internal architecture is one which makes error propagation very difficult, so even if the individual bugs were fixed there's not a good overall handling strategy).

taspeotis• 2 years ago

it's amazing to me the whole world doesn't use it.

I Googled nng and apparently it's this? [1]

It's written in C so if you click the issues tab the second and third issues are "IPC - Use After Free" and "Setting TLS config option via nng_socket_set_ptr causes access violation if you free config."

Why would you want the world to build other software on this?

[1] https://github.com/nanomsg/nng

mindslight5• 2 years ago

I looked at 0mq et al, and what I couldn't understand is why sockets have a single exclusive type. Like say I want a process that generally sends out streaming broadcast updates, but is also controlled through request-reply. It would need two separate sockets. Then I'd have to reinvent some sort of ordering protocol across the two (as well as program logic to handle the partially-connected state), which would defeat much of the point of using 0mq?

nurettin• 2 years ago

Without knowing any details, it sounds like a hard problem whether you use 0mq or not.

exe343• 2 years ago

news = -setup broadcast socket-

orders = -setup req rep socket-

while 1:

----order = orders.read (with timeout)

----orders.send(ack)

----if order:

--------do stuff differently

----news.send(status)

nlnn• 2 years ago

I've been looking at these recently for a project.

nng looks promising, but the guide from zmq seemed like a killer feature. It describes all sorts of high level patterns, gotchas, etc.

For nng I mostly found API documentation, which made me a bit more cautious (though to be fair, I've not tried it yet).

masfoobar3• 2 years ago

I have used ZeroMQ with C, Dlang, and Python -- mostly for learning.

However, I have used NetMQ.. a C# implementation of ZeroMQ in live software and the results are very positive!

I used a Pub-Sub pattern for one program to keep users informed on progress of a task, which could have taken hours to complete. They had a GUI program which spits out updates. It worked really well.

I was also tasked updating a Till software, which the integration of orders to the central system was extremely slow. I used NetMQ which was looking extremely successful but was put on hold due to IT manager not understanding Software Developers -- if something starts taking more than a week to do (which I stated I needed a month) they get itchy and move onto to something else. Sadly, that never got completed.

Now, I have played with NNG and there are some interested articles (or hidden pages from memory) about companring NNG to ZeroMQ - it seems the "patterns" are simplified.

I am currently in the progress of creating bindings for NNG. Seems to be pretty good, so far. I plan to move away from NetMQ (C#) in favour of this language moving forward.

Whether you use ZeroMQ or NNG - I dont think you can go wrong. It is all about the process more than anything, ensuring you do not lose data.

sidcool2• 2 years ago

Anything new with ZeroMQ? It's been around for a long time.

joezydeco4• 2 years ago

The licensing was changed about nine months ago. LGPL-3.0+ is out and MPL 2.0 is in. Very thankful for that.

sidcool2• 2 years ago

Ah ok. Was wondering why this was on HN front page today. :)

josephcsible3• 2 years ago

Why are you thankful for the MPL instead of the LGPL? Is there any advantage to the MPL other than being easier to incorporate MPL code into proprietary software?

masfoobar3• 2 years ago

I always get confused when I see MPL... part of me panics thinking ZeroMQ went with the "Microsoft Public License"

:-)

jobulanus• 2 years ago

NATS is the ZeroMQ of today.

lgas• 2 years ago

They aren't really the same thing. NATS is a message queue/MOM, ZeroMQ is a smarter abstraction over sockets.

BiteCode_dev• 2 years ago

NATS looks a lot like crossbar.io, what's the differences?

kromnomo• 2 years ago

I think https://zenoh.io is more of a successor or replacement.

packetlost3• 2 years ago

They're not even remotely the same thing. You could build NATS on ZeroMQ but not the other way around. ZeroMQ is more like a network/message-passing abstraction library while NATS is a service.

veggieroll2• 2 years ago

I'd 90% agree with this, except that NATS does have an embedded mode.[0] That plus the clustering, gateway, and leaf-node features gets it very close to what ZeroMQ is doing, though at a much higher complexity level that's maybe unnecessary depending on what you're doing. Embedding is exclusive to Go programs, so that's very limited compared to ZeroMQ.

[0] https://www.youtube.com/watch?v=cdTrl8UfcBo

veggieroll2• 2 years ago

People are replying that NATS isn't the same as ZeroMQ. But, I think the missing piece here is that most people who are looking at ZeroMQ really just want something like NATS. There is a decent amount of effort required to make ZeroMQ work. Whereas, NATS just does the thing you wanted in the first place without all that effort.

sigmonsays• 2 years ago

I would rather use sockets. Not getting errors when a client times out is a bad api design to me. I've used zeromq an only kept it around for IPC.

I may have been doing it wrong, but i personally want to know when clients disconnect/reconnect/etc. the API seems to hide all that from you and your send or recv just block.

haneul2• 2 years ago

Imo, ZMQ is more of an abstraction with which to design protocols, rather than a message queue ready to use, Kafka-style. I found unexpectedly that I needed to really read the entire manual and work through the worked examples and some of my own to start getting it, rather than the usual incremental read.

So, you can set up a protocol using ZMQ such that you become aware when a client times out, and you can set behavior regarding the high water mark, and other things, but you have to actually do it explicitly - it's not required that you do it, because you can choose to maximize throughput or minimize latency instead.

But, whatever protocol behavior / performance you want, you can pretty much build it with ZMQ. In Python, ZMQ was the only "feasible for my not-a-network-engineer-self" to get a system with 100μs latency with sufficient throughput and guarantees (when used for IPC, although not using the IPC transport type). gRPC was a lot less performant for me, granted it would've been more convenient, but the low latency was a hard need.

Although, networks are one of my noobier areas, so I might be blind in many ways here.

otabdeveloper4• 2 years ago

I don't see what ZMQ abstracts that you don't get in TCP already.

Is this all just about having a common cross-language API for TCP? Wasn't "BSD sockets" supposed to be that?

ohnoesjmr• 2 years ago

Pretty much this.

I always found it crazy how zmq gained any traction at all.

"Oh, I have a req/resp workload" - one of the sides restarts, goes out of rhythm with the state of the connection (whether its req or resp), unrecoverable errors.

Every system I've seen use zmq usually use it without these fancy patterns (use yeet messages in any order), and usually have some sort of "Is anybody there on the other side?" message to combat the fact there is no way to introspect connection state (otherwise your writes just block at the high water mark), at which point it would have been easier to just use tcp.

The whole thing to me reeks of mongo. It's great if you are completely incapable/incompetent of solving the problem properly.

bheadmaster• 2 years ago

at which point it would have been easier to just use tcp

Funny, I always used ZeroMQ as a message-oriented TCP-like protocol that reconnects automatically.

It's great if you are completely incapable/incompetent of solving the problem properly.

Oof. I think it's great if you don't want to mess with low-level socket details and just want to write a actor-like messaging protocol.

packetlost3• 2 years ago

We've used ZMQ rather successfully, but step 1 of ZMQ is don't use REQ/REP at all, ever. It has advantages primarily around throughput and abstracting details across different mediums (which is sometimes actually a detriment).

aitchnyu2• 2 years ago

As a new dev in 2011, I was unsuccessfully making a http worker thing in Python with Zeromq and I was distracted by their docs were encouraging replacing callback architecture with message passing with inproc queues.

notachatbot1234• 2 years ago

Is this criticism still relevant? 2011 is 13 years (as in: a teenager's life) ago.

aitchnyu2• 2 years ago

FWIW most discussion about inproc is around 2010, per $SEARCHENGINE. Guess that architecture didnt catch on.

mafuy• 2 years ago

I've never used ZeroMQ. How does it compare to MPI? I assume it is easier to use?

1vuio0pswjnm7• 2 years ago

Appreciate the rosettacode-like (chrestomathy) approach to documentation:

https://zguide.zeromq.org/

mharig• 2 years ago

If an app needs a DB anyway, has ZeroMQ then advantages over a DB based MQ, like PostgreSQL LISTEN/NOTIFY or SQLite with update_hook?

Did anybody compare throughput/latency for these approaches? Edit: ... for the basic zmq patterns PUB/SUB, REQ/REP, Client/Server

ZeroMQ: High-Performance Concurrency Framework

Related Stories