High-Performance server for NATS.io, the cloud and edge native messaging system

https://github.com/nats-io/nats-server

by Kinrany • 3 years ago

146 72 3 years ago

github.com

jitl2• 3 years ago

With all the message queue whatever thingies, as an outside I’m quite confused about ideal use cases.

NATS vs MQTT vs Kafka vs Redis Queue vs Amazon SQS - how do they all stack up?

Xeoncross• 3 years ago

Queues are often used for two reasons: as a way to throttle processing (just throw it into the queue and drain as you have time) and a way to avoid state loss (process the queue item and only ack/clear it once you've done X, Y, & Z. Of course there are more uses, but these are the two most common.

Depending on the throughput you need, the number of clients pulling/putting into the queue, and other things you can rule out certain options. For example, Redis queue will be one of the fastest choices - but is very limited in capacity (memory size). SQS is basically unlimited capacity, but can often have 150ms or more of time elapse between when you put the item in and when it's available.

paulgb• 3 years ago

MQTT is a protocol rather than an alternative to NATS; NATS supports MQTT (https://docs.nats.io/running-a-nats-service/configuration/mq...)

As far as ideal use cases, we use NATS for https://plane.dev in two ways:

- As a message bus, it is a layer of abstraction on top of the network. Instead of each node needing to establish a connection to every node it needs to connect to, it just connects to a NATS cluster and messages are routed by subject. This is great for debugging because we can "wiretap" messages on a given subject pattern and verify what's being sent. We even have a service that listens on NATS subjects and conditionally turns events into Slack messages.

- It has a built-in RAFT implementation (via JetStream), which we piggyback on when we need to create consensus among nodes.

atombender3• 3 years ago

NATS is not a queue. It's distributed pub/sub message broker for communicating between applications.

NATS's only responsibility is to route messages in near-real-time from publishers to consumers. Messages are ephemeral and dropped immediately after delivery; if nobody is listening, the messages vanish. Messages are only queued temporarily in RAM if the consumer is busy, and they can get dropped if a consumer doesn't handle them fast enough. In short, NATS is very lightweight and fast, and designed for things that are lightweight and fast. It's like a kind of distributed socket mechanism, and works best as a communication primitive you build stuff on top of (like TCP or UDP) rather than a fully fledged system.

So it's very different from Kafka and other types of queues that are durable and database-like. Kafka is good for "fat pipes" that centralize data from producers into a log which is then consumed by massively parallel sets of consumers, and you don't constantly change this topology. NATS is good for networks of fast-changing producers and consumers that send small messages to each other, often one-on-one, although any fan-out topology works. It's great for firehose-type routing. For example, imagine you want your app to produce lots of different kinds of telemetry. Your app just sends messages to a topic "telemetry" or maybe a dotted topic like "telemetry.iostats" or "telemetry.errors". Then a client can "tap" into that topic by listening to it. If no client is listening, the firehose goes nowhere. But then a client can tap into "telemetry.errors" and get just the stream of error messages. Topics are just strings, so you can create unique topics for temporary things; an app can send a message to another app like "hey, do some work and then send the result to my temporary topic foobar726373".

NATS is particularly notable for its "just works" design. The clustering, for example, ties together brokers with no effort. Clients typically don't need any configuration at all, other than the name of a NATS server.

NATS can be used as a low-level component to build stateful stuff. NATS Jetstream is a Kafka-like solution that stores durable logs, and uses NATS as its communication protocol. Liftbridge is another one.

zerbinxx4• 3 years ago

It is worth noting that the jetstream API (to me) (not that experienced) to lack some of the features of Kafka w.r.t. replayability - for instance I can’t easily say “go back and re-run messages from X point-in-time”. Instead it may be necessary to write custom handlers for replay or safeguard your systems to be very idempotent (a good pattern in event driven systems, but not one that is explicitly required by Kafkaesque messaging systems).

jadbox• 3 years ago

Isn't it a big problem if readers miss messages if either the topic readers go offline briefly or message throughout exceeds readers?

sofixa2• 3 years ago

NATS vs MQTT vs Kafka vs Redis Queue vs Amazon SQS - how do they all stack up

I'm curious why are RabbitMQ and AMQP no longer part of such comparisons (not only your comment, nobody else on this thread has mentioned them).

jitl2• 3 years ago

At my past job RabbitMQ was an infrequent source of painful surprises. I think it's kinda like HAProxy, a venerable solution that's well known but being replaced in new work more trendy implementations of the same kind of thing.

pnpnp• 3 years ago

I have been a huge advocate of NATS.

For anyone looking to support multiple message patterns on one message bus, this is what you want to check out.

In AWS terms, it’s like SNS/SQS/Kinesis all rolled into one bus & very intuitive to work with.

vosper3• 3 years ago

Any thoughts on NATS' JetStream? Looks quite compelling, to me.

https://docs.nats.io/using-nats/developer/develop_jetstream

protochron3• 3 years ago

Jetstream is the Kafka or Kinesis bit built out on top of the NATS core protocol. It's much easier to work with than either of them in my experience

SebastianM2• 3 years ago

Jetstream (includes Streams (better as in Kafka IMO), a powerful KV Store and also an Object Store to store files) is really powerful. It earns much more attention than it currently has.

Jetstream is also much easier to operate than Kafka IMO. Just a simple single binary with an easy to understand config.

clumsysmurf• 3 years ago

I'm curious how this compares with MQTT, there seems to be some overlap.

I was looking to stream sensor data from a mobile device using MQTT, but the Eclipse Paho Java client (1.2.5) hasn't seen a release in 3 years and I found it to be pretty buggy. There are lots of open issues on the GitHub page.

ozarker• 3 years ago

MQTT is a protocol. Nats is a server that supports MQTT among other things

xupybd3• 3 years ago

I'm using this in production. So far it's worked very well. Took a few hours to setup with certificates. Now it just works.

zerbinxx4• 3 years ago

What do you run it on?

xupybd3• 3 years ago

We have a manufacturing operation with a public B2B ecommerce platform.

It's just a dumb communications buss for us. It's replaced most of our ETL needs with Realtime events. If an order is placed online an event is raised. Our accounting system can consume that event to create the sales order. Then the production system consume the same event to add the job to the next production batch. Each production step produces events that can be used to update other systems, including Realtime updates on the ecommerce system.

We use Jetstream so consumers don't need to be awake when producers create events.

This system spans the cloud and 3 physical locations. But, to the consumers and producers it's one buss that they only have to authenticate with once.

gjvc• 3 years ago

not to be confused with https://www.nats.aero/

zmmmmm• 3 years ago

Struggling to figure out what "cloud native" actually means. Unfortunately their own web site doesn't say and the CNCF web site they link to leaves me none the wiser.

At a guess they are talking about applications being built from the ground up to dynamically allocate resources using cloud providers APIs directly rather than relying on an assumption fixed resources are already provisioned and the application runs within them.

I wonder if I'm right ...

jerrygenser• 3 years ago

There are a bunch of OSS that solve problems when running distributed scale out workloads in "the cloud". For example prometheus is a metric db for storing monitoring data.

Ceph is a storage backend. Most of them tend to be built for or work well with kubernetes.

re-thc2• 3 years ago

Struggling to figure out what "cloud native" actually means.

- very fast startup

- low memory

- can be easily distributed or is stateless

xuancanh2• 3 years ago

That is not what cloud native means. It is an overloaded term, but the basic idea is that the application is designed to operate in the cloud environment, utilizing cloud's capabilities such as elastic and automated infrastructure management.

lyu07282• 3 years ago

applications being built from the ground up to dynamically allocate resources using cloud providers APIs directly

If that indeed is what cloud native means, it sounds interesting. But the problem is that all these APIs and especially "managed" services are super proprietary and you'll vendor lock yourself pretty hard. But I suppose that ship has sailed a long time ago.

mprime12• 3 years ago

NATS is 100% open source

pjmlp• 3 years ago

Cloud native usually means targeting the "Cloud OSes", while being agnostic from the underlying OS, or legacy stuff like POSIX.

Or to put it more clear, applications written in modern language runtimes, packaged in containers that can be run on top of whatever orchestration is available, and using provider APIs and resources.

otabdeveloper4• 3 years ago

Struggling to figure out what "cloud native" actually means.

Same as "web scale" and "big data".

mprime12• 3 years ago

The non-cynical, snark-free definition would be “built from the ground up to be used and deployed in various kinds of cloud based environments”.

Whether that’s just marketing BS or real depends on the project. Wether it fits your particular kind of cloud environment is also a different story.

In the specific case of NATS I love how I can start with a single server on localhost, then maybe upgrade to a single fly.io instance, then later move to a larger AWS, instance, then later add some fault-tolerance by turning a single server into a cluster, then later have multiple clusters in various AZs around the world, hosted on different cloud providers.

NATS makes all of these changes (and a lot more) a breeze. Any component or application using pub/sub, KeyVal, durable streams, request/response will just keep working without a single any changes.

Disclaimer: I love NATS, it’s the most promising piece of infrastructure technology I have seen in a long time.

lakomen2• 3 years ago

I got into it again about 2 weeks ago. Been doing a small project with the microservice api.

Compared to HTTP/REST I miss the debugabilty of it. Since all messages are sent over websocket and received, in binary form, since there is no support from Chrome or Firefox, one has to tediously manually extract the json payload and try to make sense of it.

Jetstream is still quite the mystery. It just doesn't want to work the way I want it to, especially on the JS client side, blocking script execution and timing out.

Then there's auth, after a few shutdowns and reboots I got locked out of my local installation. Oidc subject to nats user mapping doesn't exist and has to be done manually.

So TL;DR the core functionality is great. Everything else seems to be WIP.

devmunchies• 3 years ago

You can subscribe to all topics by using a *. Why not have a debug subscriber that listens to all messages and plays it in the terminal?

I agree that the auth system is cumbersome. I wanted to use it on the edge for IoT devices where the device only has the same permissions as the user who the device belongs to, but not very easy. Their auth isn’t very customizable.

lakomen2• 3 years ago

This is actually a good idea, I'll try that, thanks :)

But, you know, it's still a bit more complicated than opening the browser's debug console and inspecting request and response in the networking tab.

Yeah you have operators, which are essentially the auth admins or orgs, however you'd like to look at it, then there are accounts which are an alias for "project name" and then you have users, which are "client name"s. And that's fine for their infrastructure only. The problem is, in the real world you have an external identity service (OIDC, IAM,...) and the JWT this service creates includes a subject, but the NATS auth system has no support for external bridges. Also you have to decide, would you like to use operator/account/user or single JWT, or passphrase or manually managed user/password accounts? In a professional world, you'd have various customer applications each divided by their operators, accounts and users. And then fine grained user capabilities.. headache time.

So what you have to do is distribute a default user certificate with a client (client meaning actual client in the client/server sense), then do the signin/sso process and have some middleware checking this token from the OIDC auth process and check is a user with this subject exists, if not, create a new user and cert and send that cert back to the client which will then create a new connection with the new certificate.

Very complicated.

The initial default client certificate is required for "logging in" via NATS. I guess it could be a standalone program, the oidc to nats user mapper/manager.

So compared to HTTP/REST, more work involved.

SebastianM2• 3 years ago

Nats provides a nice CLI that can help with debugging: https://github.com/nats-io/natscli

Besides that I'm also working on a UI solution, that will help to get better overview of your cluster: https://qaze.app/

gedw994• 3 years ago

this sounds really awesome.

npace12• 3 years ago

nats is phenomenal. Libraries and docs are not great, but you won't need them much as you just forget it's even there.

protochron3• 3 years ago

I work quite a bit with CNCF wasmCloud (https://wasmcloud.com/) and that project wouldn't exist without NATS. You can think of wasmCloud as a distributed compute lattice using WebAssembly on the server for providing the compute and NATS to provide the interconnectivity between your various WebAssembly modules.

NATS certainly has its quirks, but I can't recommend it highly enough if you need any sort of pub/sub or stream processing. It even has built-in key-value and object storage for when you need to store larger messages or content. I definitely prefer Jetstream to Kafka in pretty much every use case I can think of. At my current employer (https://cosmonic.com) we use NATS not only for wasmCloud, but we also stream log data and metrics and it keeps up with everything we throw at it with a very low footprint. Auth is kind of counterintuitive until you've spent some time with it, but NATS provides you with a ton of flexibility (docs here: https://docs.nats.io/running-a-nats-service/configuration/se...).

https://natsbyexample.com/ is a great resource, and can do a better job than I can in illustrating the various ways NATS can be used along with different deployment topologies.

no_circuit2• 3 years ago

TIBCO Rednezvous, https://www.tibco.com/products/tibco-rendezvous, is the first thing that came to my mind from previous experience in the financial industry working with real-time market data. Although I'm not sure if it has built-in KV support for dealing with large payloads like NATS does, TIBCO RV and their related software packages are worth checking out to see what an long time established commercial product offers. Which leads me to...

... the protocol is text-based like HTTP with CR LF for field both for the client, https://docs.nats.io/reference/reference-protocols/nats-prot..., and cluster protocols, https://docs.nats.io/reference/reference-protocols/nats-serv... -- which means encoding overhead if your payloads are binary. So depending on your definition of performance, ymmv.

I really do not see how implementing an API across multiple languages is easier by making a new linefeed-based protocol, https://github.com/nats-io/nats-server/blob/0421c65c888bf381..., than just using code-generated JSON or gRPC (Protobuf or Flatbuffers). One could then write subscriptions/clustering algorithms in a protocol-neutral library.

c-cube• 3 years ago

I don't think there's encoding overhead. It's like http, like you said, but just like http body encoding it seems that actual payloads are preceded with their length so that you just read n followed by n bytes. Look at the docs for `pub` for example. Json would be less efficient; gRPC adds tons of complexity and overhead.

klabb32• 3 years ago

Correct. Text based is more than fine these days, as long as you use length prefix so you can avoid escaping. But you do still need to parse lines which can be tricky in bare or poorly tooled environments. It’s the best trade off today, because debugging binary formats is a serious obstacle.

Json would be less efficient; gRPC adds tons of complexity and overhead.

Indeed. There aren’t many suitable specs around, and this protocol, albeit custom, is very easy to implement. Which is proven by the fact that there are well maintained Nats clients in many different languages.

coder543• 3 years ago

I'm not sure how much it will reduce your skepticism, but I believe NATS was actually created by the same person who created TIBCO RV: https://twitter.com/derekcollison/status/1163299967089254402

Derek seems like the kind of person who might know a thing or two about messaging systems.

I really like what NATS has become, and I do appreciate the simplicity of the protocol.

jensneuse• 3 years ago

We've recently built an adapter for NATS KV so you can easily expose it through an HTTP-based GraphQL API. (https://docs.wundergraph.com/docs/databases/kv) We're also looking at supporting streams in the future.

What's great is that NATS is written in go and we can easily embed it for testing and dev purposes. Furthermore, Synadia makes it super easy to run NATS across multiple regions.

throw140820202• 3 years ago

Would it ever make sense for NATS to support UDP or SRTP?

throw140820202• 3 years ago

Ahh, they may work on QUIC this year: https://github.com/nats-io/nats-server/issues/457

gedw994• 3 years ago

i think they are waiting for QUIC to be merged into golang

debuggerpk• 3 years ago

Big fan of nats, especially how they are doing tenant isolation and geo-distributed queue.

ensocode• 3 years ago

What are good resources to learn about this new trend of using messaging? We already had this stuff MoM (message oriented middle ware). Pretty interesting.

evnix• 3 years ago

We have had pretty bad experience with NATS. Apart from core NATS which is just in memory, you need their sub modules to build any kind of serious business on top of it.

We initially built our code around NATSStreaming, they then went ahead and deprecated that.

But since we saw so many big companies using NATS, they thought it might be a good idea to stick with it and we did a year long migration to their shiny new NATSJetstream Push based approach, but from what I see now in the conversations they are going to deprecate that too in favour of Pull based approach which is architecturally very different, now we will have to somehow convince management for another rewrite. I am not sure if we should even rewrite or just move to another product at this point.

Dear NATS, please stop throwing away and rewriting protocols and products. Or make it such that the end client libraries would handle that upgrade automatically with a library upgrade.

We should have just stuck with the more traditional Kafka or RabbitMQ.

What I have also learnt is that when companies put big brand logos on their websites, it just means some random Dev from that company is using it for their side project or experimental mini project.

Kinrany3• 3 years ago

From what I've seen it's too early to migrate from Push. With the last update they emphasized Pull as the default, but Push is not deprecated yet.

And even NATS Streaming still works: it is deprecated, not removed.

Unlike databases, a good thing about messaging and streaming solutions is that you don't have to pick one: you can make them talk to each other as long as there are bridges. This also applies to different approaches to messaging/streaming provided by a single platform.

bzzzt• 3 years ago

Just like you don't want to support tens of different databases you shouldn't build your solution on lots of bridged message busses. If stuff goes down at 2 am you don't want to have to find out which message queue was the problem or what tooling you should use to debug it.

rcombatwombat5• 3 years ago

(note that NATS Streaming is a now deprecated predecessor to NATS JetStream)

Pull does have advantages over push (e.g. one-to-one flow control since the transfer of the messages is initiated by the client (pull requests)), and they are basically functionally equivalent (only thing push can do that pull can not is send a copy of all the message to all the subscribers, should you ever need it). They both exists because historically push came first and then pull later).

As a developper using NATS JetStream you should really not have to worry about push or pull, you should just care whether you want to consume the messages via call back or via an iterator or via fetching batches, after that whether pull or push is being used underneath the covers is irrelevant to you.

And this is exactly how it is in the new JetStream API (https://github.com/nats-io/nats.go/tree/main/jetstream#readm...) you don't have to worry about push/pull anymore and you can consume in any of the 3 ways described above (callback, iterator, fetch batch) it's all a lot simpler and easier to use.

jonathanstrange• 3 years ago

Can someone tell me which transport protocols and ports this uses or can be configured to use? Moreover, do clients have NAT traversal capabilities (no pun intended)?

This is always so hard to figure out for B2B libraries.

rcombatwombat5• 3 years ago

It supports clients connecting over TCP (which can be encrypted) and Websockets (can also be encrypted), you can configure it to pick on any port you want and it works through Network Address Translation (see https://docs.nats.io/running-a-nats-service/configuration#co...). Even supports MQTT clients directly (https://docs.nats.io/running-a-nats-service/configuration/mq...)

High-Performance server for NATS.io, the cloud and edge native messaging system

Related Stories