Scaling Secret: Real-Time Chat

https://medium.com/@davidbyttow/scaling-secret-real-time-chat-d8589f8f0c9b

by danielalmeida • 11 years ago

136 57 11 years ago

I recently discovered Pusher and was able to build and ship an entirely new feature in under a week. They have good documentation, thought provoking example apps and simple pricing.

Couldn't be more impressed with them. Keep it up!

graffitici• 11 years ago

Out of curiosity, if you were to implement the features of Pusher yourself, would it have added too much to your implementation? I guess you would need to set up your own Socket.IO server, and push data from that.

I've never implemented this, which I think is the reason I am underestimating the service that Pusher provides. It would be great if someone that did implement it would convince me that it would be a large hassle to do that internally.

morgante2• 11 years ago

A single Socket.IO server is easy.

A scalable cluster of Socket.IO servers isn't.

nulltype• 11 years ago

I haven't used Pusher or Socket.IO, so I don't know specifically what they offer. However, I have made a simple unidirectional websocket-type server and it was not a terrible hassle.

With a go server on a GCE n1-standard-2 you should be able to handle over 100k connections per server. I haven't done any benchmarks for throughput, but using redis as the messaging backend you can do pretty well.

You take 100k connections on each of the endpoint servers and make a single redis connection, using redis pubsub to subscribe to channels. Publishers can connect to the same redis and publish to those channels.

If you wanted bi-directional it could work a similar way I think, I'd guess the specific architecture depends on your requirements.

sondh• 11 years ago

I have come across Slanger[1] while researching for a good websocket server and it worked well for us. It's basically an open source version of Pusher backend. You may be interested in their code.

[1] https://github.com/stevegraham/slanger

powera• 11 years ago

If they really had 1 million concurrent active users "assuming 5% of our simultaneous actives were chatting, that we needed to be ready to support up to 50,000 users chatting at once", and were seeing growth "chat was an instant hit. It grew week-over-week to well over 50,000 concurrent connections", why did they shut down?

hayksaakian• 11 years ago

the simple answer is "not enough growth"

carrotleads• 11 years ago

I would say Secret did not have enough imagination to monetise those numbers or had a moral reason against that kind of money making..

blissofbeing• 11 years ago

So basically they used pusher to scale to around 50,000 concurrent users.

notrollinghere• 11 years ago

This isn't great for Layer. I think most companies want to own their chat stack. Its not complex enough, like say Stripe, to need a 3rd party. Has anyone used Layer and felt they added value?

Animats• 11 years ago

They used Google's database service and Pusher, so only in a limited sense do they "own their chat stack".

Are there any true peer to peer chat apps left, or do they all go through some server now, even after call setup?

StavrosK• 11 years ago

I really doubt it. Remember when you could just send someone a file, because their IM client allowed you to connect to each other directly?

I wonder if there's a good way to do this now, as some plug in somewhere... Even XMPP's file forwarding functionality fails for me.

EvanPlaice• 11 years ago

"Are there any true peer to peer chat apps left"

Bittorrent recently released 'Bleep' which supports encrypted peer-to-peer chat including image transfer. It holds the message locally until a direct connection to the receiver is established.

fideloper• 11 years ago

Really interesting that Pusher didn't choke on that!

Also patting my own back for thinking about chat session based on sorted user id's (for 2+ person chat) on my own once also. Felt like a nice way to do it, unsure if that breaks down at any point.

daurnimator2• 11 years ago

50000 concurrent users? That's small enough to do with a single thread on a modern server.... What are they scaling?

Xorlev• 11 years ago

50,000 people chatting + persistence, albeit temporary.

davidbyttow3• 11 years ago

Your point is valid. This isn't ultra-high scale, just the name of the series I intend to publish more into. It's more about building a simple service for instant usage upon launch.

We also used Pusher for global in-app notifications, which had to scale past 1,000,000 concurrent connections.

Aside: 50000 concurrent requests on a single thread, huh? :) Unless I'm missing something, that's 50 seconds assuming a request takes only 1ms to process. Sounds like magic.

davidbyttow3• 11 years ago

Clarified the 1MM number in the post.

nostrademons2• 11 years ago

1ms is high for a request, assuming that you touch only RAM in a pattern with high locality (which you can usually manage when routing chat messages). Take a look at the TechEmpower benchmarks: most of the Java and C++ frameworks can manage 1M req/sec for JSON serialization:

https://www.techempower.com/benchmarks/#section=data-r10&hw=...

daurnimator2• 11 years ago

Aside: 50000 concurrent requests on a single thread, huh? :) Unless I'm missing something, that's 50 seconds assuming a request takes only 1ms to process. Sounds like magic.

1ms to route a chat message is plenty.

e.g. on a single AWS m1.small prosody[1] can process 40,000 stanzas per second.

[1] http://prosody.im

jongdubois3• 11 years ago

It really depends on a lot of things such as how often users send messages on average and what kinds of messages (JSON or plain text?). The latest version of SocketCluster can handle 25000 concurrent users per CPU core each sending a JSON message every 6 seconds so the 50K number does match my own benchmarks if you assume that each user sends a JSON message every 12 seconds on average.

calgaryeng• 11 years ago

Step 1) Take investor's money Step 2) Take $3MM off the table Step 3) Profit Step 4) Shut down

DigitalSea• 11 years ago

There isn't enough of this information out there. Scaling things is incredibly hard, a lot more difficult than some would think. Not only do you have limitations around the software, but hardware and the problems that come with scaling hardware. Great article, looking forward to some more specifics (load balancing, memory/cpu/disk issues) in future articles.

allthetime4• 11 years ago

How is this news? Chat has been around for a long time and Pusher is nothing more than a cash grab for lazy developers. You can easily get better performance, scalability, and customizability, for much cheaper with socket.io and and a caching and/or messaging service (Redis, *MQ).

enraged_camel2• 11 years ago

I find it distressing that a "mid-brow dismissal" such as yours is the top comment.

allthetime4• 11 years ago

Is it "mid-brow dismissal"? I'll submit to your criticism of my tone, but I believe a strong argument lies behind it. Its not that I mean to discredit what Secret has accomplished, only that people shouldn't be wooed into thinking Pusher is the way to accomplish it. Its a waste of money in my opinion, and the DIY alternatives will, I assume, prove to be much better options in the long run.

Confusion• 11 years ago

Lazy is smart. Allows you to focus on adding value. Not needing better performance, scalability and customizability is not only reasonable: it's highly probable. Building and maintaining is not cheap

mrinterweb3• 11 years ago

"It grew week-over-week to well over 1,000,000 concurrent connections (chat + notifications). Luckily, Pusher worked well and was fairly inexpensive" I'd love to find out what they mean by inexpensive because from pusher.com/pricing 10,000 connections is $399/month. I have no idea what 100x connections would cost (hopefully not $39,900/month). Frankly, the pricing of pusher and other competitors seems insane to me. I was considering pusher for my startup, but I realized that if my startup became mildly successful, I'd be stuck with a bill that would force my business to rapidly build another solution or take on excessive debt. Why begin with prototype technology that could lead your startup (if successful) to become insolvent?

It's not as if there is a shortage of available tech for handling socket connections. It took about a week for me to learn enough node.js and build a service with socket.io that did everything I needed.

true_religion• 11 years ago

10,000 connections is $399/month is about the price of a dedicated server just for chat.

Depends on your application. Generally, if you have 1,000,000 concurrent users, shouldn't you be making some money or have investment to tide you over?

My app has ~3000 concurrent users, which comes out to 250,000 users per day making in the order of $XX,XXX per month.

At 1,000,000 concurrent, we'd expect to have 83 million users per day visiting the site. At that rate, we'd be making at the very least hundreds of thousands of dollars per month and could easily afford a $39,000/mo chat service.

jongdubois3• 11 years ago

Some apps have lower profit margins than others. Also some companies need to be able to integrate with their realtime channels on the backend (E.g. for doing custom realtime analytics, integrating with third-party services, applying transformations/filtering to messages in realtime, capturing certain messages for storing in your database, launching various realtime-based background tasks, etc... You do lose a lot of flexibility by going with a service like Pusher. Though it is well designed and very convenient.)

mrinterweb3• 11 years ago

"10,000 connections is $399/month is about the price of a dedicated server just for chat." So you're saying that pusher is breaking even on their service and that a custom socket server should also expect to require similar hardware in order to serve 10,000 connections? If so, I highly doubt either is the case. I've seen 50,000 sockets served on a $20/month VPS using socket.io without issue.

jparreira• 11 years ago

For those who think Pusher $399 for 10k concurrent is a lot you have Realtime (http://framework.realtime.co/messaging) with $250 for 25k (including 251 million messages every month). Just saying...

jbrooksuk• 11 years ago

My friend has started making Pushman, a free alternative to Pusher. Worth looking at http://pushman.dfl.mn/

morgante2• 11 years ago

Secret raised $35M.

$399/month is a tiny drop in that bucket. It is almost certainly not a good use of engineering resources for Secret to build their own in-house Pusher alternative, even if it would only take a single engineer a few days.

mrinterweb3• 11 years ago

I suppose if you have boat loads of cash to burn, a couple thousand a month would not be a big deal. For a bootstrapped startup, money is a concern.

jongdubois3• 11 years ago

SocketCluster's API is similar to that of Pusher so it can be a good self-hosted alternative: http://socketcluster.io/ - It's also designed with scalability in mind. You do have to manage presence yourself though but that shouldn't be too difficult (You can keep track of user presence in your own database - That could actually be an advantage since you can go ahead and use that presence data easily on your back-end to do other stuff - Easy to integrate).

We're currently working with a couple of fast growing startups so it's getting traction (still early adopter stage though). We'll be announcing our native iOS client near the end of this week. (disclosure: I'm the main author)

amelius• 11 years ago

Why not use WebRTC for chat these days?

ozkatz2• 11 years ago

Secret was a mobile app. WebRTC is not supported on any version of iOS and is supported only on very recent android (5.x)

edit: source: http://caniuse.com/#search=webrtc

albertwang• 11 years ago

That caniuse link refers to mobile safari. WebRTC in iOS is supported for native apps

JonCox• 11 years ago

I'm making a real-time communication iOS app, we're using Firebase[1] to quickly build it's real-time features.

There's to be a lot of hate on here towards using such services. Is it really a better option (flexibility, owning your data & price wise) to spend the time building your own system and having to hire engineers later on to manage/develop it?

[1] https://www.firebase.com

boothead2• 11 years ago

Hey Pusher guys; how much of that flow went through Haskell? Are you using it for your main service yet?

mdpye3• 11 years ago

None of it sadly, though the application fetches its config from a haskell service.

The bits we are preparing for the main service are quite large (architecturally) but we hope to be beta testing them soon.

boothead2• 11 years ago

Make sure you keep blogging - I really enjoyed your last post on testing with Haskell!

Scaling Secret: Real-Time Chat

Related Stories