Million User Webchat with Full Stack Flux, React, Redis and PostgreSQL

https://blog.rotenberg.io/million-user-webchat-with-full-stack-flux-react-redis-and-postgresql/

by elierotenberg • 11 years ago

129 37 11 years ago

blog.rotenberg.io

thomasfl• 11 years ago

This could be the inspiration for a great open source project, and become something that could easily be deployed to a cloud hosting platform. Basically it's the same as firebase, but with some of the react and flux goodness like server side rendering. Or somebody could package this as a product.

ylesaout• 11 years ago

Where did you see there is server side rendering? Flux and React does not imply server side rendering out of the box.

elierotenberg3• 11 years ago

Actually this whole stack is designed with SSR in mind, see the related package react-nexus (which I will soon blog about, too!) :)

bsaul2• 11 years ago

Great post, but has this design been implemented to effectively handle something close to a million users ( or even 100k and show that no part is overheating) ?

As in every new and complicated design, i'm a bit skeptical of rules of thumb calculations. You never know what the wrong latency issue at the wrong place can do...

moe3• 11 years ago

but has this design been implemented to effectively handle something close to a million users

Obviously not. Some of his numbers are off by an order of magnitude.

E.g. he claims "10 million messages/sec" for a single redis instance.

In reality redis tops out at well under one million messages/sec; http://redis.io/topics/benchmarks

The design is almost comically bad (single source of truth for a "scalable" chat app?!). This is either an attempt at parody or this guy must be suffering from a rather severe case of second system effect...

[1] http://en.wikipedia.org/wiki/Second-system_effect

elierotenberg3• 11 years ago

Oh, you're absolutely right, 10M/s is a mistake, which I've corrected.

Few remarks though:

- it's still way below the number of actions we're talking about here (~100k/s)

- since redis is only used as a generic MQ and not as a store, it can be sharded at the app level without the pain usually associated with redis clustering

- I've deployed a similar (but less performant) design for the player of a gaming website, which is in use in production for more than a year, and works like a charm (we're talking ~5-50k users per channel on a daily basis). This is definitely a "second-system" pattern, but I try to avoid the associated pitfalls :)

I'd be genuinely interested by your feedback!

shripadk• 11 years ago

I agree. "Number of messages a single Node broker can handle: ~10k per second without JSON stringification memoization, ~100k per second with JSON stringification memoization (Nexus Flux socket.io does that for you)." <- Can't be possible. The numbers should be lower with JSON stringification/parsing instead of being 10x.

pothibo• 11 years ago

I thought it was someone from Slack talking about their infrastructure (or similar).

Instead, it's some abstract theory. I believe everyone can dream of any architecture.

cha_os• 11 years ago

This is so interesting - keep up the good work!

vonklaus• 11 years ago

Do you have anything like this up on github?

jacques_chester2• 11 years ago

GameRanger has six million users, tens of thousands to hundreds of thousands of whom will be active simultaneously. His chat problem involved has moderate fan-out.

Scott Kevill does it on a single machine (last time I checked) with hand-rolled C++ and close attention to the details of how the Linux networking stack works.

marktangotango• 11 years ago

Very interesting, do you have links available?

pratikch• 11 years ago

I am also interested to know, Do you've any links on this?

jacques_chester2• 11 years ago

Scott gets very little press -- mostly when he picks up support for games after the publisher or matchmaking service abandons one.

I used to live in Perth and we'd sometimes hang out and mock the front page of HN.

http://www.gameranger.com/about/

itsbits• 11 years ago

Single Machine??...really...would love to know more about the stack..

fauigerzigerk3• 11 years ago

One problem I have with using postgres listen/notify as a general purpose message queue is that it requires polling (At least that was the case when I last looked at it). Of course you can use a blocking wrapper around the polling code but it still causes unnecessary roundtrips.

nunwuo• 11 years ago

Huh? The entire point of listen/notify is exactly that you don't have to poll.

fauigerzigerk3• 11 years ago

And yet in previous versions of libpq polling was the only way to use listen/notify. I'm not sure if it was a limitation of libpq or the server implementation itself: http://www.postgresql.org/docs/9.4/static/libpq-notify.html

It's still not a particularly simple interface as you have to check for notifications after every single SQL command if I understand it correctly.

Erwin• 11 years ago

Your database connection is just a socket, so you can add that file descriptor to the set of file descriptors you are waiting IO on, if you are using a classic select/poll based system. See an example in the pscyopg2 docs here: http://initd.org/psycopg/docs/advanced.html#asynchronous-not...

Once that FD is active, you call the poll() method and your notify payload becomes available to you.

fauigerzigerk3• 11 years ago

You are right. It seems that this was an issue with older versions of libpq: "In prior releases of libpq, the only way to ensure timely receipt of NOTIFY messages was to constantly submit commands" http://www.postgresql.org/docs/9.4/static/libpq-notify.html

remon2• 11 years ago

I suppose there is some value to this as a thought experiment but pretty much every tier in that architecture has breaking flaws. The most relevant being that anything that has an audience of 1 million users cannot run on an architecture that has single points of failure.

aantix• 11 years ago

Physically the hardware can't handle it or are you just advocating that it shouldn't?

remon2• 11 years ago

Hardware fails. This architecture has single points of failures which means if one of those instances fails the entire system goes down. This is generally not acceptable for environments with 1 million active users (which implies high business value)

chucksmart• 11 years ago

TLDR;'thought experiment', feh, if this was actual tested result would be much more interesting.

_almosnow• 11 years ago

But if the author didn't "thought experiment" about that, how else would he bait you into clicking...

megaman821• 11 years ago

The Flux terminology is confusing to me. It looks like the Observer pattern to me.

* Stores contain Observables

* Components (or Views) contain Observers

* Actions are Proxies

So the article is basically saying the Observer pattern is scalable, but uses the buzz-phrase "Full Stack Flux" instead. To make it even worse it is only a theoretical application of this pattern.

gfodor• 11 years ago

AFAICT, Facebook engineered their client side code around Flux in order to eliminate two-way data binding in their user interface code, which leads to all sorts of issues. I don't imagine they push the pattern into the server. Their relay stuff still relies upon a smart data tier which understands a query language called GraphQL.

I definitely don't think they considered implementing a dumb dispatcher and store layer on the database server using stored procedures. (This seems terrifying to me, I don't see the upside.)

It's an interesting experiment but I think this might be an example of being too aggressive in trying to generally apply a design pattern that was motivated by a specific problem.

polskibus• 11 years ago

As far as I understand, this design is based on similar goals as the Meteor/MongoDB project - instead of reading oplog, you listen for notifications.

I wonder how would Meteor be better than this proposal given that both are node based.

razzi• 11 years ago

Anybody know how to efficiently make smexy graphs like that? Did Adobe have a hand in their creation?

alxndr• 11 years ago

OT: as someone trying to learn my way around Erlang's OTP framework, I'd be interested in a walkthrough of an OTP based system compared to this.

Million User Webchat with Full Stack Flux, React, Redis and PostgreSQL

Related Stories