FZambia

Thanks! Our users mostly run Redis or Valkey, so we optimize for them. We also have Dragonflydb in our test suite. But good to know there is one more option which may be used as a drop in replacement for Redis and provide extra properties, guarantees or performance benefits.

Many here recommend using Kafka or RabbitMQ for real-time notifications. While these tools work well with a relatively stable, limited set of topics, they become costly and inefficient when dealing with a large number of dynamic subscribers, such as in a messaging app where users frequently come and go. In RabbitMQ, queue bindings are resource-intensive, and in Kafka, creating new subscriptions often triggers expensive rebalancing operations. I've seen a use case for a messenger app with 100k concurrent subscribers where developers used RabbitMQ and individual queues for each user. It worked at 60 CPU on Rabbit side during normal situation and during mass reconnections of users (due to some proxy reload in infra) – it took up to several minutes for users to reconnect. I suggested switching to https://github.com/centrifugal/centrifugo with Redis engine (combines PUB/SUB + Redis streams for individual queues) – and it went to 0.3 CPU on Redis side. Now the system serves about 2 million concurrent connections.

Client SDKs are often a major challenge in systems like these. In my experience, building SDKs on top of asynchronous protocols is particularly tricky. It's generally much easier to make the server-side part reliable. The complexity arises because SDKs must account for a wide range of usage patterns - and you are not controlling the usage.

Asynchronous protocols frequently result in callback-based or generator-style APIs on the client side, which are hard to implement safely and intuitively. For example, consider building a real-time SDK for something like NATS. Once a message arrives, you need to invoke a user-defined callback to handle it. At that point, you're faced with a design decision: either call the callback synchronously (which risks blocking the socket reading loop), or do it asynchronously (which raises issues like backpressure handling).

Also, SDKs are often developed by different people, each with their own design philosophy and coding style, leading to inconsistency and subtle bugs.

So this isn't only about NATS. Just last week, we ran into two critical bugs in two separate Kafka SDKs at work.

For real-time notifications, I believe Nats (https://nats.io) or Centrifugo (https://centrifugal.dev) are worth checking out these days. Messages may be delivered to those systems from PostgreSQL over replication protocol through Kafka as an intermediary buffer. Reliable real-time messaging comes with lots of complexities though, like late message delivery, duplicate message delivery. If the system can be built around at most once guarantees – can help to simplify the design dramatically. Depends on the use case of course, often both at least once and at most once should co-exist in one app.

Hi everyone!

I'd like to share that we've just released Centrifugo v6 - a major update of scalable WebSocket server. The release addresses some usability pain points and adds nice features and more observability.

Centrifugo is an open-source standalone server written in Go – https://github.com/centrifugal/centrifugo. Centrifugo can instantly deliver messages to application online users connected over supported transports (WebSocket, HTTP-streaming, Server-Sent Events (EventSource), GRPC, WebTransport). Centrifugo has the concept of a channel – so it's a user-facing PUB/SUB server. Everything implemented in a language-agnostic way – so Centrifugo can be used in combination with any frontend or backend stack.

These days we also provide Centrifugo PRO version – and trying to find a balance to be sustainable.

The server is based on the open-source Centrifuge library - https://github.com/centrifugal/centrifuge, so many improvements mentioned in Centrifugo v6 release blog post (even those for Centrifugo PRO) may be used just as a library in Go application.

We provide real-time SDKs for popular client environments – for browser and mobile development – they connect to both Centrifuge library based servers and Centrifugo server.

Generally Centrifugal ecosystem provides a good alternative to Socket.IO and cloud services like Pusher.com and Ably.com

Will be happy to answer on any questions

Yep, and in addition to that the ephemeral ports problem will araise at some scale with long-lived connections and infrastructure balancer/reverse proxy chain. So it's still required to tune.

Wow, it's fascinating how a single HN comment can drive meaningful traffic to a project! I'm the author of Centrifugo, and I appreciate you mentioning it here.

Let me share a bit more about Centrifugo transport choices. It’s not just about supporting multiple transports — developers can also choose between bidirectional and unidirectional communication models, depending on their needs.

For scenarios where stable subscriptions are required without sending data from the client to the server, Centrifugo seamlessly supports unidirectional transports like SSE, HTTP-streaming, unidirectional gRPC streams, and even unidirectional WebSockets (this may sound kinda funny for many I guess). This means integration is possible without relying on client-side SDKs.

However, Centrifugo truly shines in its bidirectional communication capabilities. Its primary transport is WebSocket – with JSON or Protobuf protocols, with SSE/HTTP-streaming fallbacks that are also bidirectional — an approach reminiscent of SockJS, but with more efficient implementation and no mandatory sticky sessions. Sticky sessions is an optimization in Centrifugo, not a requirement. It's worth noting that SSE only supports JSON format, since binary is not possible with it. This is where HTTP-streaming in conjuction with ReadableStream browser API can make much more sense!

I believe Centrifugo gives developers the flexibility to choose the transport and communication style that best fits their application's needs. And it scales good out of the box to many nodes – with the help of Redis or Nats brokers. Of course this all comes with limitations every abstraction brings.

Hello, I am author of https://github.com/centrifugal/centrifugo. Our users can choose from WebSocket, EventSource, WebTransport (experimental for now, but will definitely stabilize in the future). WebRTC is out of scope as the main purpose is central server based real-time json/binary messaging, and WebRTC makes things much more complex since it shines for peer-to-peer and rich media communications.

What I'd like to add is that Centrifugo also supports HTTP-streaming – not mentioned by the OP – but this is a transport which has advantages over Eventsource - like possibility to send POST body on initial request from web browser (with SSE you can not), it supports binary, and with Readable Streams browser API it's widely supported by modern browsers.

Another thing I'd like to mention about Centrifugo - it supports bidirectional WebSocket fallbacks with EventSource and HTTP-streaming, and does this without sticky sessions requirement in distributed scenario. I guess nobody else have this at this point. See https://centrifugal.dev/blog/2022/07/19/centrifugo-v4-releas.... Which solves one more practical concern. Sticky sessions is an optimization in Centrifugo case, not a requirement.

If you are interested in topic, we also have a post about WebSocket scalability - https://centrifugal.dev/blog/2020/11/12/scaling-websocket - it covers some design decisions made in Centrifugo.

Wondering whether coroutines may be a step towards async event-based style APIs without allocating read buffers for the entire connection. I.e. a solution to problems discussed in https://github.com/golang/go/issues/15735. Goroutines provide a great way to have non-blocking IO with synchronous code – but when it comes to effective memory management with many connections Go community tend to invent raw epoll implementations: https://www.freecodecamp.org/news/million-websockets-and-go-.... So my question here – can coroutines somehow bring new possibilities in terms of working with network connections?

Every time I read criticism of WebSockets it reminds me about WebSuckets (https://speakerdeck.com/3rdeden/websuckets) presentation :)

I am the author of Centrifugo server (https://github.com/centrifugal/centrifugo) - where the main protocol is WebSocket. Agree with many points in post – and if there is a chance to build sth without replacing stateless HTTP to persistent WebSocket (or EventSource, HTTP-streaming, raw TCP etc) – then definitely better to go without persistent connections.

But there are many tasks where WebSockets simply shine – by providing a better UX, providing a more interactive content, instant information/feedback. This is important to keep - even if underlying stack is complicated enough. Not every system need to scale to many machines (ex. multiplayer games with limited number of players), corporate apps not really struggle from massive reconnect scenarios (since number of concurrent users is pretty small), and so on. So WebSockets are definitely fine for certain scenarios IMO.

I described some problems with WebSockets Centrifugo solves in this blog post - https://centrifugal.dev/blog/2020/11/12/scaling-websocket. I don't want to say there are no problems, I want to say that WebSockets are fine in general and we can do some things to deal with things mentioned in the OP's post.

Hi! Could you please extend the meaning of word "global-scale" a bit? Does this only mean that users will connect to the nearest server or there are more tricks on backend to scale PUB/SUB? You are writing that "messages are optimized for speed across our global network" – could you also elaborate more what do you mean saying this?

Thanks for feedback, I thought the article left fully unnoticed here:) Go also has a small disadvantage for writing WebSocket apps – you don't have full control on read buffer reuse (since its got allocated for entire lifetime of connection) thus memory consumption per connection is more than it could be. Otherwise I am pretty happy with Go for this task.

I also don't really like new design of main repo page. Repo short description on the right – so inconvenient. BTW this is how it looks on mobile: https://i.imgur.com/SqV23WN.jpg - the most interesting things about repo usually located in first lines of README - now we can't see it.

A couple of days ago Centrifugo got version 1.0. It was 3 years ago when I started developing this project. Originally written in Python (Tornado) and called Centrifuge it migrated to Go language earlier this year (https://news.ycombinator.com/item?id=10068403).

I personally believe that Centrifugo is quite distinctive server with some unique features and simple to use. It already helped adopting real-time events on several sites in Mail.Ru Group where I work and several other web projects around the world. It's especially helpful for applications with backend that can't painlessly work with lots of simultaneous connections - for example Django backed web projects. And it's MIT licensed - can help to save several hundred of dollars per month as a replacement of paid services.

Two additional links here, one to documentation – https://fzambia.gitbooks.io/centrifugal/content/ – and second to demo instance on Heroku (use password: demo) – https://centrifugo.herokuapp.com – so you don't need to run your own process to play with (it's on free Heroku dyno with all the consequences).

As usually any feedback much appreciated.

Hi!

1) It was very interesting and challenging sometimes. I remember several difficulties during migration: many usages of interface{} values at first stages which then be replaced with more strong types, several possible races which Mr Klaus Post (https://github.com/klauspost) helped to find, he also noticed at some problems in code style and channel buffer overflow problem. But the greatest part of this migration is that I don't need anymore think "Is library I use non-blocking or not?" as Go's built-in concurrency allows to use all libraries written for language. And also after Tornado it's great to have tools to inspect active goroutines. I also personally don't like how many locks I use at moment - I hope to refactor several parts of code to use channels for synchronization.

2) Elixir is great, but unfortunately I had no experience with it - just read several articles. Centrifugo is not massively scalable at moment - it does not support Redis cluster yet - this is an area for investigating. So now it is limited to Redis throughput and suites well for small or medium projects. So the answer here: Erlang is absolutely great for software like this but I have chosen a language in which I could implement all that needed in a reasonable time. Go has also its unique features so at moment I am pretty happy that I started migration.

Thanks! You are looking at administrative web interface, the goal of this interface is just to show messages coming into channels in real-time just to see that Centrifugo works fine and maybe inspect messages being sent. So when you are reloading a page web interface is just start waiting for new messages from scratch. Web interface must be used as just a tool to monitor your Centrifugo installation and is absolutely optional. It allows to send API commands: publish messages, see presence and history information etc

The goal of Centrifugo is just to deliver messages instantly to all connected clients. It has channel history which can be used to show last N messages from channel but it's not related to what you see in web interface. Also note that Centrifugo is not a persistent storage and channel history messages will expire after configurable timeout. So it's possible to implement what you want using channel history but if you take into account 2 things: message history is limited to N messages and message history will expire after M seconds.

Hello, thanks for sharing, yes - it looks similar in its core idea, will definitely look at code base and hopefully find something interesting to learn from. One of current Centrifugo problems is that lots of features out of the box result in not so trivial client. So as far as I personally don't know Java and Objective-C/Swift there are no clients for mobile apps yet. Hopefully this will change in future with help of open-source community. Or maybe Go 1.5 mobile apps support will allow to create shared library for using from those languages.

Hello!

- at moment logging is quite unstructured - just text entries, written into stdout or file. In future I am planning to move to something like this (https://github.com/Sirupsen/logrus) and improve log structure.

- metrics exist in Centrifuge but the next step for Centrifugo (the only thing left to be fully compared with Python version). At moment external monitoring tools can be used for CPU and memory monitoring. I am planning to utilize this library for metrics - https://github.com/rcrowley/go-metrics

- you can run several instances on different machines connected via Redis and load balance clients between them. But no sharding support out of the box.

What do you think about this? I highly appreciate any feedback

Here is a link to Centrifugo – real-time messaging server in Go. This is a central part of Centrifugal stack (http://fzambia.gitbooks.io/centrifugal/content/). It supports Websocket and SockJS client connections. Originally server was written in Python (https://github.com/centrifugal/centrifuge), but recently the entire code base moved to Go language. This is a server that works as a service – can be used with web sites written in any language. Feedback is highly appreciated.

I don't see nothing wrong with separate asynchronous server which handles real-time for your Django site.

When event generated by user happens on your site - you just handle it in a traditional manner i.e. - POST via AJAX, validate, save if necessary and then publish into asynchronous server which broadcasts event to all connected clients. In this way you have a graceful fallback in case of async server downtime, so your user doesn't even notice something went wrong. You are not mixing things which were not developed to be mixed. In this case you are just writing your site as usual and then add real-time elements where necessary.

Using Gevent together with Django seems like monkey patching entire web site to me.

I really respect the work of guys developing uWSGI. But at moment it does not seem to be usable in a simple obvious way. Maybe in future their real-time support will become mature and convenient enough.

Of course, Meteor and Derby like approach is another level of problem solution. But in context of Django I don't think we should consider them as examples. We use python, not javascript - we have no native solution for browser environment and I personally think we do not even need it.

yep, when you want real-time page updates on your site (chat, comments, counters, games) you should generally use some kind of asynchronous backend to achieve this. Centrifuge is a message broker to which clients from browsers connect, after connecting clients subscribe on channels. And every message which was published into channel will be broadcasted to connected clients. There is documentation - https://centrifuge.readthedocs.org/en/latest/, also screencast - http://www.youtube.com/watch?v=RCLnCexzfOk