matticakes

I'm one of the original authors, happy to answer any questions.

NSQ is a full-featured messaging platform out-of-the-box, whereas zeromq and nanomsq are lower level libraries that you could use to build (the same) functionality.

You can embed the daemon into your application.

That sentence is missing an important word. No centralized brokers, meaning the typical/recommended deployment topology is an nsqd node on all hosts producing messages.

Thanks!

This is something that I've always tried to stress when talking about NSQ...

The "message queue" is the most boring and uninteresting aspect of the system.

It's the combination of out-of-the-box tooling and conceptually simple primitives that really differentiate it from other systems.

Yea, in the original announcement blog post (which became the design doc):

http://nsq.io/overview/design.html

(project author)

I appreciate all the work that Tyler has put into his series of blog posts on messaging systems (and NSQ bug reports!) but I think this particular article is one that misses the mark [1].

While raw performance is important, I think comparing the guarantees and related operational and development semantics are far more interesting and useful.

These systems vary a great deal in the interfaces and functionality they expose to users building on top of them and operators dealing with them when they're deployed in production (and inevitably break!). This is ultimately what matters most.

P.S. single node (localhost) performance is essentially useless. Most of these tools are designed to be deployed as the backbone of large distributed systems (NSQ in particular), so scalability and performance in that context is a better indication of real-world performance.

[1] He's since posted this follow-up that outlines many of the same things http://www.bravenewgeek.com/benchmark-responsibly/

(project author)

NSQ is as much about what it doesn't do as it is what it does. To a certain extent this mirrors, and was inspired by, the language's philosophy (Go) [1].

Also, NSQ was designed to replace an existing home-grown system deployed at scale. This dictated a lot of the initial requirements (and in certain cases excluded off-the-shelf tools).

When we left the experimental phase we realized we had built something that was useful to others, and it turns out that despite not having the features you've identified it can be incredibly effective in lots of use cases that don't need stronger guarantees.

[1] If I'm being honest, NSQ was a vehicle for adoption of Go at bitly as well as the project we used to learn the language. This was a huge risk at the time (almost 3 years ago) but one that has certainly paid off.

check out NSQ, designed for (in part) exactly this use case (and also written in Go)

http://bitly.github.io/nsq

NSQ treats the message data as an opaque blob so the format wouldn't directly affect it (except on some lower level related to overall message size I suppose). It would impact your producers (encoding) and consumers (decoding), obviously.

re: de-duping - there are lots of things to consider, I highly recommend reading through http://cs.brown.edu/courses/csci2270/archives/2012/papers/we..., it's a fantastic paper. At a high level the answer is idempotency.

What sort of use case are you thinking of? (context would help answer your de-dupe question)

there are a few (growing) number of production installations of NSQ...

our (bitly's) cluster spans a a few datacenters and hits peaks of 80k messages/second.

I can answer any questions you have (one of the authors)

You can swap out the "metrics system" described in the post with any downstream system.

I have a lot of experience with NSQ but have not used NewRelic. I can answer any specific questions you might have...

Thanks.

For the record... NSQ is a generic messaging platform and is data format agnostic, there is nothing specific about metrics collection (it was just my contrived example).

disclaimer: co-author of NSQ [1] here

Agreed. Message queues play an important role for us (bitly) in being a layer of fault tolerance, buffering, and a means to perform various operational tasks.

They're so important to us that we decided to build something that worked exactly like we wanted.

NSQ is a realtime distributed message processing system where we've taken the approach of focussing on making it ops friendly and easy to get started.

IMO, solutions that make it easy to develop on and administrate are most important... because things break. NSQ is straightforward to deploy (limit dependencies), simple to configure (runtime discovery), and client libraries provide a lot of functionality important for handling failures (like backoff, deferred messages, etc.) for a variety of use cases.

We've written an in-depth introductory blog post [2] about NSQ that has more details.

[1] https://github.com/bitly/nsq [2]: http://word.bitly.com/post/33232969144/nsq

It's a bit more explicit in NSQ but we essentially support multicast-like routing through "channels".

A "channel" receives a copy (at the source) of all the messages for a "topic" and has its own set of clients subscribed.

The focal point is discovery. Not other queues (or other libraries that can build queues). This is an interesting way to approach it, thanks for open sourcing.

We chose to solve the discovery problem a bit differently in NSQ (https://github.com/bitly/nsq) but I could certainly see some interesting opportunities to experiment with a distribute hash table approach as well.

The truth is we wanted to use ZeroMQ initially. It would have been really awesome to have all that flexibility on the client side.

The ZeroMQ documentation is fantastic as well. It inspired and helped shape some of the design choices we made.

The further along we got from design to implementation it became obvious that it would be important to "own" the socket. Generally speaking, this is exactly what ZeroMQ prevents you from doing (and rightly so, it aims to abstract all of that away).

The choice to use Go had an impact here as well. Language features like channels and the breadth of the standard library made it really easy to translate our NSQ design into working code, offsetting the benefit of ZeroMQ's abstractions.

That's fair, you certainly can use rabbitmq in a distributed and decentralized fashion.

You bring up an important point though... What is important is the topology we're promoting and the platform we've built to support it. The actual queue is less important and it made sense for us to own that piece to achieve our goals.

We looked into AMQP based solutions. Our understanding is that slaving, master-slave failover, or other strategies are used to mitigate the fact that there is a broker responsible for a given stream of messages.

We wanted a brokerless, decentralized, and distributed system. NSQ has no broker or SPOF so it is able to route around failures.

That said, I think RabbitMQ is a good tool depending on your requirements. I can imagine a broker proving useful in situations where you may want strict ordering or no-duplication. Those were tradeoffs we were willing to make for fault tolerance and availability.

Also, given the fact that we were already operating infrastructure built on simplequeue (which is also distributed and brokerless) we found it more appealing to evolve rather than replace.

beanstalkd is a great project and it certainly inspired some of our thinking. I think of beanstalkd as a more fully functional alternative to simplequeue (what we built/used prior to NSQ - more details in our blog post http://word.bitly.com/post/33232969144/nsq).

Some of the goals of NSQ transcended just replacing our specific daemon that buffered and delivered messages (most importantly the interactions with the lookup service). Because of that, we felt that owning that piece would make it easier to achieve those goals.

Additionally, one of the most important properties of nsqd (the queue component of NSQ) is that data is pushed to the client rather than polled (like in beanstalkd).

Agreed, a comparison would be helpful for highlighting the tradeoffs behind the choices we made.

We do talk about the evolution of our infrastructure and the genesis of NSQ in our blog post, http://word.bitly.com/post/33232969144/nsq.

it is data format agnostic so you can certainly serialize using the gob package.

one of the developers here...

The protocols that exist in NSQ now are designed to be the simplest implementation that worked.

You've correctly pointed out some of the issues. At this stage, the distinction of producer vs consumer was mostly so that you could publish at all without having to use the HTTP interface. For our use cases, in particular taking advantage of the /mput endpoint, we aren't even using the TCP based publishing protocol.

Re: your point on sending metadata with SUB commands. I agree its a bit ugly. We actually intend on improving that aspect by instead sending the data in the form of an IDENTIFY type command upon initial connection. That information is used in the various administrative UIs and endpoints.

I'm going to do a bit more reading on MQTT, thanks for the link.

perhaps its time to bring "the game programming megasite" back, too

bitly is overwhelmingly C and Python

For Sierra (kings quest, etc) fans there are a couple good options:

http://sarien.net/ (web based!) http://www.agdinteractive.com/ (high-res remakes) http://www.tsl-game.com/ (The Silver Lining - a completely new game continuing the Kings Quest story line)

https://github.com/mreiferson/py-wkhtmltox

https://github.com/mreiferson/php-wkhtmltox

bump for an awesome startup / architect

Not bad (this done without specifying any options which might improve conversion)...

http://www.visionaryrenesis.com/nytimes.pdf

Sure, it's certainly not perfect. In the cases where I've used it the benefits far far outweigh the negatives you've pointed out.

I feel like you need to enter with an open mind for any conversion to PDF...

if you don't mind my asking - what's involved in the process that generates the PDF's?