How We Manage a Million Push Notifications an Hour

https://blog.gojekengineering.com/how-we-manage-a-million-push-notifications-an-hour-549a1e3ca2c2

by shadykiller • 7 years ago

140 88 7 years ago

blog.gojekengineering.com

srameshc• 7 years ago

This is cool, but I found this things called MQTT protocol and Emitter.io is one implementation of it. You can build your own notification system on top of it for a much cheaper cost.

lachlan-sneff• 7 years ago

1 million events an hour is just 278 a second, which is a much more surmountable number than 1 million.

fourier763• 7 years ago

What if you get one push event on the first second, and then 999,999 on the last second? Still one million per hour. Don’t assume the events are evenly distributed. I see this a lot on HN. Yes, 1,000,000 divided by 3600 is indeed about 278, but why would we assume it was 278 notifications exactly every second?

Alupis• 7 years ago

Nothing says you must handle them all at once though. You could queue and handle as your system allows... still reaching a 1 million per hour throughput.

pkaye• 7 years ago

Then they should have said they can handle a million push notifications a second.

nwallin• 7 years ago

Queue it. It's not particularly urgent that push notifications go out instantaneously. A 99th percentile latency of 5 minutes is fine.

bt848• 7 years ago

Same reaction. You could handle this kind of traffic with, just as examples, procmail or inetd on a single PC.

messe• 7 years ago

Hell, modern microcontrollers (which run into the several hundred megahertz) could handle that load.

lwansbrough• 7 years ago

I'm currently implementing a notification system for our network of sites. Aside from the HTTP microservice, my implementation is about the same. Which isn't much of a surprise - I expect this is a fairly common way of doing it.

With that said, I was surprised to learn how complicated it can be to send notifications cross platform. Fortunately we're targeting the web, so I really only have to worry about two push provider implementations: APNs and VAPID. I really hope Apple agrees to implement VAPID sooner than later so developers can stop wasting their time.

We have used OneSignal in the past, but there's something so satisfying about delivering the notification yourself. Also, a word of caution for people new to push notifications: we discovered early on that sending a million push notifications all at once is a really good way of crashing your site, as they have a surprisingly high click through rate in the first minute of sending!

philjohn• 7 years ago

Why not use Amazon SNS? They abstract away the differences between APNS and whatever Google are calling their push infrastructure these days.

kondro3• 7 years ago

I don’t mean to discount the work done here, but why would a dev team build this themselves these days? There are many practically infinitely scalable services that do just this with various levels of sophistication.

From the simple AWS SNS ($0.50/million messages) or Google Firebase (free) solutions that do multi-device push messaging whilst managing keys, redelivery and delivery responses to the more managed services like Urban Airship and OneSignal.

The actual Android delivery component has to end up going through Firebase anyway.

The only part of this solution SNS and Firebase don’t provide are the user fan-out functionality.

Surely there are other more important features delivery business value to be done rather than re-inventing (and then scaling and supporting) the delivery of mobile notifications.

9nGQluzmnq3M• 7 years ago

Did you read the post? They're using Firebase messaging, but that only covers a part of their workflow.

kondro3• 7 years ago

They’re only using Firebase for Android delivery (which is mandatory anyway). But it is just as happy to deliver to APNS at $0 as well with near infinite scalability.

The actual delivery component of their messaging system is unnecessary.

thewarrior5• 7 years ago

Pretty soon your app is just a bunch of proprietary cloud provider products glued together and now there’s vendor lock-in and much less flexibility.

You might want to throttle messages based on user settings or say ML models to avoid too many annoying messages. For critical stuff you might want to switch over to email or SMS.

These are startups that have the funding to do this kind of thing. This is a core part of the stack for any such company.

kondro3• 7 years ago

If that's your concern, lean on Firebase.

You have to use Firebase to deliver messages to Android anyway, having it deliver to APNS as well you get for free with that implementation.

Maybe you do want to do those things, but this implementation doesn't, nor do they mention the intention to do that. But if you did, SNS also provides implementations for delivering SMS and Email (preferably through SES).

Startups shouldn't waste their money on this type of thing, especially if you consider most startups will fail. Vendor lock-in is a pretty stupid thing to worry about for an early-stage start-up (and I argue it's stupid to worry about long-term too if that vendor is AWS, Google or Azure who are all very competitive with each other).

tanilama• 7 years ago

I don't think vendor lock-in is what small startup should worry about.

discordance• 7 years ago

Efficiently handling million of events per second (not per hour) using whatever cloud PaaS is a no brainer these days. The lock-in is worth it when you can reliably move that fast.

If you're really worried about lock-in, most event based PaaS support running containers. Moving between clouds would then just be the glue-work around your containers.

SkyPuncher2• 7 years ago

Honestly, how many companies DON'T rely on some proprietary cloud provider at some point?

* SMS - You can't just interface directly with some cell company (which is proprietary anyways). * EMAIL - Have you tried to send your own emails in the past decade? Just use a provider. * Push Notifications - Have to use Firebase for Android anyways.

----

Your core stuff should limit proprietary stuff. Edges are going to change over time no matter what, might as well use stuff that makes you faster now.

basilgohar2• 7 years ago

One big factor is being free from bundling proprietary, vendor-specific libraries that may also leak user's information or otherwise increase the security risk footprint of your app. Having control over the services that make up the core of your business, as opposed to outsourcing and relying on 3rd-parties, can be very freeing and also cost effective in the long run.

harryh• 7 years ago

Once you're hosting on AWS you've pretty much already crossed that particular bridge.

gdeglin5• 7 years ago

I cofounded OneSignal for this reason. We were previously a game studio that built an in-house solution. It was a huge maintenance headache and a distraction. Eventually, we realized that so many companies were one-off push implementations and decided to focus our efforts on building a really great third-party solution.

Firebase and SNS have since entered this market as well. I'm obviously biased, but I honestly can't recommend them. They work for some of the basics, but there's been very little innovation in either product and occasional service issues that go unsolved for a long time. They also don't make any money from them, which should be a concern for anyone that plans to use these services for such a core part of their business.

hw• 7 years ago

It's not hard to do what they did, so I'm not surprised they did it themselves. It's just simple async queue processing, and doesn't really need a lot of sophistication, especially at just 1M notifications per hour. Now if they need to process a few orders of magnitude more, then AWS SNS would probably be a good next step.

dragonsh4• 7 years ago

Gojek is a super app in Indonesia serving a population of 264 million. They can afford this level of engineering and are required to do it, to stay competitive. They have millions of motorcycle driver (called gojeks) for whom they are trying to find jobs while idling waiting for passengers to transport.

If uber built these no one will question them why don't they use some off the shelf solution. Gojek is a google cloud customer and under their startup program - surge. So I am sure they are aware of the solution proposed.

They don't call themselves technology company but company which helps gojek drivers to find jobs and earn extra. In the process they became a food delivery, transportation, payments, courier company. In spite of this they still focus on their primary objective to find jobs for gojek and let them earn extra for better living.

Before Gojek app most of the drivers were just idling waiting for passengers, now with a $25 Kai OS feature phone [1], they can just receive a message and earn instead of just idling, they don't have to use iPhone or Android or some heavy OS. Indeed based on social survey some of the drivers could increase there earning 3-4 times based on how willing are they to take up additional jobs.

I believe the beauty of Gojek lies in its model to focus on finding jobs and developing solution to help those motorcycle drivers using technology. Although they extended it to cars. I am intrigued by their idea since its not a copy of Uber, but a solution to a problem local to Indonesia, which can extend to any country in the world finding jobs to do while idling and earn more.

[1] https://en.wikipedia.org/wiki/KaiOS

hello_moto5• 7 years ago

They can afford this level of engineering and are required to do it, to stay competitive
If uber built these no one will question them why don't they use some off the shelf solution

Uber and Gojek don't have the same talent pool so we can't compare them as Apple-to-Apple.

It is a risky move for Gojek and not for Uber because of the talent pool in their respective area.

ppsreejith• 7 years ago

I don't think the other answers here question the impact Gojek has on millions of people.

If uber built these no one will question them why don't they use some off the shelf solution.

If it's over-engineered, they probably would.

Disclaimer: I work at Gojek.

hello_moto5• 7 years ago

Engineering Pride (and exercise for future employment).

While this might sound condescending but hear me out:

- if you keep doing the same thing like everybody else out there, gluing APIs from cloud-vendor, you won't stand out.

- if you're a company of _that_ size (Uber, Gojek), you gotta do something big because the name/brand is associated with "Engineering" excellence

- if you're a leader (or in management), would you tell your employee to "suck it up" and do the boring stuff like everybody else? or would you empower them to do more? hype them up? this is a strategy for keeping them happy too.

I don't begrudge that. That's how companies evolve: they start small and scrappy, they grow very fast, and they're at the level where "doing simple stuff" won't get you promotion or you won't have something cool to show during the interview for your next job.

It's work dynamic.

shanipribadi2• 7 years ago

``` Surely there are other more important features delivery business value to be done rather than re-inventing (and then scaling and supporting) the delivery of mobile notifications. ``` the business value the service provided is that this allow the engineering organization to scale. Gojek has couple hundred engineers, split across multiple teams, where each teams own multiple services/products/applications, each with their own development cycle. Having a single abstraction provided as a service helps to mitigate the risks involved in changes to implementation details and external dependencies (e.g. GCM deprecation) since it abstracts it away completely.

gdeglin5• 7 years ago

We had a similar challenge supporting sending this many notifications at OneSignal, which we solved using Rust.

We recently hit a peak of 850 Million notifications per second, and 5 billion notifications per day. Here's a blog post on how we do it. Written back when we were at "only" 2 billion notifications per week: https://onesignal.com/blog/rust-at-onesignal/

sk5t• 7 years ago

The one-day record could have been serviced in 5.8 seconds of peak per-second throughput, is that right?

gdeglin5• 7 years ago

It's not a perfect comparison because we benefit slightly by batching messages with the same content when sending to Android devices, and we have some backend features in place to reduce storage usage when the content is the same across many messages.

iOS, however, is still quite challenging at scale since there's no batching mechanism for APNS, so even bandwidth becomes a bottleneck. We do web push too, which requires a lot of CPU cycles to encrypt each payload for its recipient.

broth• 7 years ago

I really like the diagram in this blog post. Is that diagram custom made or did you use software to create it?

bsaul• 7 years ago

ROFL : I was about to post a message here bragging how i achieved 2 millions push a minute with a simple kafka queues + a cluster of push servers running on go architecture, but now i feel humbled !

tempguy99993• 7 years ago

No offence but is this correct?

We recently hit a peak of 850 Million notifications per second

per second?

From your blog "OnePush is fast - We've observed sustained deliveries up to 125,000/second and spikes up to 175,000/second."

I think you may have a typo. The bandwidth would be incredible too, if it were an unlikely 10 bytes per delivery that would be 8.5 GB/Sec.

gdeglin5• 7 years ago

This post is now a couple years old, so we've grown a lot since then. Here's a newer (more marketing-centric) post where we announced our 850k/second milestone: https://onesignal.com/blog/throughput-record/

It's generally under 4 bytes per delivery, depending on the content, and we have several delivery servers. APNS, for example, doesn't support payloads larger than 4 bytes.

gdeglin5• 7 years ago

Oops, you're right. There is a typo. It should say "850,000/second"

ian0• 7 years ago

I used to have pretty much the same reaction as most of the comments here on gojek engineering posts. Coming from a telco background the RPS figures were quite low while the complexity of architecture seemed completely over the top.

However, I now know more about the company. They have one of the most lean engineering teams of comparable companies here. Their CTO & VP Eng are incredibly practical guys whose advice would resonate with pretty much everyone here (Eg interview here[0] where they repeat the dangers of scaling too fast). They have small multifunctional product teams and the microservices architecture fits to this (as opposed to building it for fanciness / resumes sake). And they did it. Theres dozens of "features" in the app which are entire giant businesses in their own right.

So Id imagine the shortest path to getting this up and running this was definitely considered, would have been nice to go into more detail as to why simpler / outsourced solutions would work but id guess theres a good reason.

[0] https://www.youtube.com/watch?v=He0XBBfCEVk

mkagenius• 7 years ago

but id guess theres a good reason.

Sometimes there are no good reasons, and we should have the ability to question it sometimes, that's how progress is made, in science and in engineering.

didibus• 7 years ago

I've seen a few good example blog posts discussing this scale, the 10 to 500 transactions per second and learned a lot from all of them.

Does anyone know of any description of architectures scaling to the next magnitude? 1000 to 10 000 TPS ? And even higher to say 100 000 to 1 000 000 TPS ?

random42• 7 years ago

Its difficult to "guess" ahead of the fact how you are going to scale for the next 10x, because you need to know.

a.) Which component is going to start breaking, and that depends on the usage pattern.

b.) Which business/tech compromises are ok to make, and that depends on a.)

Generally speaking though, you'd try to benchmark the system to find the bottleneck component, and based on the nature of it, either try to throw more hardware (horizontal or vertical scaling) or optimize the software.

blackoil• 7 years ago

We handle peaks of 800k tps in few systems. It is for an analytical platform. Partition in Kafka by some evenly distributed key, Create simple apps that read from a partition and process it, commit offset. Avoid communication between process/threads. Repartition using kafka only. For some cases we had to implement sampling wherein usecase required highly skewed partitions.

jzl• 7 years ago

I appreciate the positive effort to communicate something useful, but this basically came down to "We use RabbitMQ".

kaycebasques• 7 years ago

I had a blast using Go-Jek and Grab when I visited Indonesia. It’s basically Uber/Lyft, except you sit on the backseat of a scooter rather than a car.

https://www.instagram.com/p/BZHty8NnTVZ/?igshid=5qg5ndm9r9ui

Of course as others mentioned it’s a super app so they do many other things, but I believe this is one of their main businesses.

thelittleone2• 7 years ago

Go-Jek has been huge for us as holiday rental managers in Bali. But lately there seems to be some sort of racket by drivers:

Example for Go-send (sending a package): "I have motorbike problem please can you cancel".

A minute after cancelling I get a message on WhatsApp: "Hey fixed my motorbike, still need?". Same thing with food orders.

Example for the Go-food: "Hey I'm at the restaurant, the phone is down. Can you cancel order? I will do manually".

Obviously the restaurant and the driver gets the full customer payment this way. Go-jek gets zero. This depends on the customer cancelling as otherwise the drivers rating takes a hit.

reportgunner2• 7 years ago

Wow that's really clever!

uvu• 7 years ago

I have done similar like. I think I do much better than those. But, I don't pass their interview haha.

jbverschoor• 7 years ago

Why is less than 300tx/sec a such a big deal?

finchisko• 7 years ago

I thought this would be about implementing custom notification server and not using google's or apple's.

reportgunner2• 7 years ago

Is that a lot of notifications ?

londons_explore2• 7 years ago

1 Million per hour is only 300 per second. On a 1.5 Ghz 4 core raspberry pi, that gives you 21 million clock cycles to deal with each message.

The architecture seems rather overengineered considering a single raspberry pi could do the job, even after 100x scaling!

lmilcin• 7 years ago

I wrote an algorithmic trading framework to work on Warsaw Stock Exchange.

WSE back then had a limit of 10k messages per second (as in one message every 1/10000th of a second). Messages came by two separate network operators so that was 20k messages to be deduplicated to 10k operations. Incomming messages came in compressed so they required uncompressing.

Responding to the message required complex processing and then may have resulted in an order to market which, again, had to be constructed and validated.

All this worked on a single server (regular two-socket Xeon-based server).

There was 10us (microseconds..) time budget to send response to the market and it had to work every time even under maximum possible load (10k messages per second).

Routing and forwarding 300 messages per second doesn't seem like something to brag about...

cristaloleg• 7 years ago

Hi from Warsaw, care to tell something more about architecture, used technologies, numbers? Thanks in advance.

thewarrior5• 7 years ago

And what happens if the raspberry pi goes down for an hour ?

londons_explore2• 7 years ago

You spend $35 on a spare.

Merg• 7 years ago

You buy ten more before hand and have all of them running side by side, to retake workload on any outage.

Replace raspberry pi with any local server configuration.

danpalmer2• 7 years ago

I think this is a disingenuous or naive view, while you may not have been serious, I think it's worth unpacking why it's naive as I think a lot of engineers have similar opinions.

To start, there's nothing here about what machine this architecture runs on, it could be running on a Raspberry Pi for all we know.

Then, this ignores the cost of database lookups for the keys. That data is probably small enough to be on the one machine, but then you have to have service support for (reliably, in real time) syncing that data to the service. A separate database is therefore probably the right solution here, which means you're doing networking in each message send, which makes it unlikely that a Pi could do this.

Next up you've got the issues of reliability. The message queue separation gives you better reliability in the face of issues such as upstream APIs going down or erroring, or issues for a specific user. All the business logic around handling this, the message queue handling persistence and ACID semantics (or parts of it), this all takes additional resources, not to mention potentially a fair bit of disk space (for a Pi) to queue up undelivered messages should an upstream API slow down or stop accepting new messages.

Then you have hardware failure, at this scale you don't want a single machine failure to wipe out your primary communication method with millions of customers. You'd therefore want to have a distributed system, even if that's only for reliability rather than performance.

Lastly, 21 million clock cycles might sound like a lot, and might go a long way with C/C++/Rust, but as you move up to more dynamic languages that will reduce significantly. It happens that they are using Go here, and that's likely to get pretty good performance out of the hardware, but writing this service in Python/Ruby would be a very valid choice for developer productivity, or based on existing skills they have in the team. That might be 1/10th the performance, but since you need a distributed system for reliability here anyway, adding a few more machines to the pool might be a better choice than introducing a lower level language that takes longer to develop and exposes you to memory safety or threading issues.

There may well be other factors I haven't considered here, but I think for the use case of delivering that scale of messages, the reliability options you get with a system like this are well worth the additional hardware requirements and architecture overhead.

Edit: lmilcin makes a good point about trading systems, but there are several differences – that system still has a single point of failure, it was probably written in a low level language with input from experts on performance, and it was running on a much faster machine. The single point of failure of a server-grade machine like that is probably an acceptable risk if you own the hardware, but in a cloud environment (which brings other benefits) hardware is less reliable so probably not an acceptable risk there. I don't think it's an apples-to-apples comparison, although it is interesting.

tempguy99993• 7 years ago

I partly agree with you but you're probably over-engineering it. They talk about "GoLife booking notifications" and "promotional notifications". Neither of these require (edit: I mean sound like they require) once-only semantics. If you're OK with very rarely failing over then restarting from scratch on the failover machine is ok. Even if not, pushing batched IDs ("I've done IDs xyz..) over a network to the failover machine would remove most of that. I don't think it needs ACID semantics.

Then you have hardware failure, at this scale

It's really, really, not scale!

Lastly, 21 million clock cycles might sound like a lot, and might go a long way with C/C++/Rust, but as you move up to more dynamic languages that will reduce significantly

Then you're prob using the wrong lang.

I don't really accept that level of engineering is necessary all round, unless the business case requires it, and then I'd speak to whoever put those business requirements together and ask hard questions.

tempguy99993• 7 years ago

Yep, I hit the calculator first thing too, but I think your criticism is not quite fair, after all the data has to be pulled off disk and acknowledgements written back.

That said, it really doesn't sound too difficult with straightforward architecting (says the armchair critic).

jiveturkey• 7 years ago

My thoughts exactly.

One million per hour? I forgot how to count that low.

dis-sys• 7 years ago

1 million notifications per hour, or ~300 per second, seriously?

shanipribadi2• 7 years ago

The title of the blog is actually a click-bait.

The point of the article was more that when you have multiple teams, each owning multiple products, having a single well-defined abstractions over external dependencies provided as a service that you control is important to manage risks posed by those external dependencies and to make overall maintenance easier.

How We Manage a Million Push Notifications an Hour

Related Stories