OpenPush: A Free, Decentralized Push Messaging Framework for Android

https://f-droid.org/en/2020/02/03/openpush-talk.html

by blendergeek • 6 years ago

347 95 6 years ago

gdeglin• 6 years ago

This is an interesting idea. Sadly, applications that use this are probably not eligible to be distributed through Google Play. One of the main reasons, as I understand, is that Google wants to discourage lots of apps from each having their own network connections open in the background.

https://news.ycombinator.com/item?id=20145206

JoshTriplett• 6 years ago

There's really no good reason that a network connection open in the background needs to use extra battery, if configured carefully. If configured incorrectly, it absolutely would, but a background connection doesn't need to regularly wake the phone up, just as Google's own push messaging service doesn't.

_-___________-_4• 6 years ago

More connections probably does mean more wakeups, in general, though. FCM will coalesce notifications that happen close together in time, for example. Even if OpenPush does that (I haven't read all the details yet), it still means two coalesced batches of notifications rather than one if you have some apps using FCM and some using OpenPush. The situation obviously gets even worse if every app implements push itself.

I still think it's worth it, to be clear, and for people who want a fully de-Googled device, hopefully eventually all their apps will use OpenPush and the battery life situation is the same as a device where all the apps use FCM. (Assuming OpenPush implements similar coalescing.)

izacus2• 6 years ago

You're right, there isn't. But what years of unrestricted background execution have shown us is that developers these days will choose the lazy solution over the right solution about 9 times out of 10.

I'd be the richest guy if I'd get a penny every time I've seen an Android app do background polling every 5 minutes (!) instead of using push notifications or even long polling. The usual response to recommendation to improve the design was "Why? It works and it's a lot of work to run a push server!".

So as much as I dislike restrictions on my operating systems, Google forcing developers to push notifications has massively improved battery life across the ecosystem.

boudin• 6 years ago

I think the target will mainly be developers who already provide a version of their app on f-droid.

I don't know of it's an obligation to use google's firebase pish notifications though. I might be mistaken, but I think that Conversations use its own push notifications already: https://play.google.com/store/apps/details?id=eu.siacs.conve...

_-___________-_4• 6 years ago

This will probably be used for apps that distribute through, e.g., F-Droid. Apps that also distribute on Google Play could use FCM for the version distributed through Google Play, and OpenPush for the version distributed elsewhere.

rohan1024• 6 years ago

Then give the user choice between the service he wants to use. How we can't see through this monopoly is beyond me.

tuukkah3• 6 years ago

Counter-example: Telegram is available in Google Play and implements its own push service (kept alive by a permanent background notification).

izacus2• 6 years ago

The Play Store version users FCM and there's no notification.

est315• 6 years ago

There is still a centralized push notification app on the phone. So it's not every app having its own network connection to the push service. The decentralization refers to the fact that the notification app likely will get the ability to configure different push services. That configuration is then sent to the apps that use it to upload it to their upstream backends. Those upstream backends now have to send push notifications according to that configuration (token + hostname).

osamagirl69• 6 years ago

I am really excited about this project! Having a source of push notifications without routing them through the google servers will really advance the state of open source and/or google free android. It is great to see it being backed by f-droid and apparently funded by the German government (through prototype fund and Federal Ministry of Education)

TrickyRick2• 6 years ago

Considering the BND's (German equivalent of NSA) love of intercepting traffic and gathering metadata about citizens, I wouldn't put "funded by the German government" in the Pro column.

tuukkah3• 6 years ago

One, end-to-end encrypted push service can make any intercepted data useless, as you cannot even know if the traffic corresponds to a message or is just a ping, let alone which app or contact the message is from.

TuringTest• 6 years ago

It may be a case of German government's right hand not knowing what German government's left hand is doing. An open source distributed framework can't be coerced into routing its messages through a central server for surveillance.

dr_hooo• 6 years ago

I see no reason make this negative link. The BMBF is a research agency which gives out grants for research through public calls (somewhat similar to the NSF in the US).

There are no strings attached, besides providing reports on scientific contributions achieved throughout the project.

It's not like the BND will come knocking on your door saying make algorithm X weak or you will not receive funding (or worse).

me551ah2• 6 years ago

Running background services in Android is tricky. Most device manufacturers put their own battery management software on devices, which makes it almost impossible to run long-running services in the background. And since this framework would either perpetually run in the background or run frequently to check for new messages, your app is guaranteed to be throttled(force-killed) by most devices. And being force killed means that your app cannot receive push notifications even from FCM.

I'm curious to know if anyone is using this framework in the wild.

Yizahi• 6 years ago

My previous phone was Oneplus with such an extreme battery manager (rated by some resource in top 3 of most aggressive managers). There were messaging apps which worked just fine in background and send push notifications even after prolonged periods in background (e.g. Telegram and others) and some apps which were killed in minutes after being minimized (e.g. official Skype app). Point is - I suspect that there is a correct way to make inoffensive apps which will work in background, but certain minority of developers simply don't give a damn about battery usage and proper coding guidelines.

me551ah2• 6 years ago

Oneplus incidentally maintains a whitelist of apps which are allowed to run in the background and only apps in that list can run for extended periods. While it is possible for users to add apps to that list, it comes with a set of apps already whitelisted ( like whatsapp). I found it out the hard way while working on Android.

You can read more about it here in case you are curious: https://hackernoon.com/notifications-in-android-are-horribly...

EwanToo• 6 years ago

The reason why some apps work is they've been whitelisted by the manufacturer, not a choice by the developers (except maybe a choice not to pay to be whitelisted?)

If you're not whitelisted, there's very little you can do except for ask users to take various manual steps to change operating systems.

ignoramous• 6 years ago

My previous phone was Oneplus with such an extreme battery manager (rated by some resource in top 3 of most aggressive managers).

This site ranks Android OEMs that kill apps in the name of battery optimization: https://dontkillmyapp.com/

I believe this aggressive behaviour tends to come from user-feedback, given users care a lot about battery endurance and heat dissipation.

tuukkah3• 6 years ago

There are ways around the battery management mis-features, but you need to check the instructions phone by phone here: https://dontkillmyapp.com/

dmos62• 6 years ago

Under Nokia it says that its Android One phones are aggressively killing apps as well. Isn't Android One a phone with vanilla Android install without vendor crap by definition?

I'm happy that I'm using LineageOS. My phone is turning 7 years old, and it's running Android 9, and I've no reason to switch so far.

BubuIIC• 6 years ago

Author here (of that post and the framework).

The solution to the background limitations for now is running as a foreground service. This shows a persistent notification (which can be hidden by the user) but can then run indefinitely. The other option is to integrate this at the system level (via a custom rom, root, magisk, etc.).

I'm curious to know if anyone is using this framework in the wild.

The client side implementation of this is still WiP, so it's not used yet in the wild. I hope to be at a point where I can work with some apps on integrating it there in the next few months.

yonixw• 6 years ago

but can then run indefinitely

Just bought a Xiaomi Redmi Note 7 and it kills everything including foreground services with a notification.

A simple user still can allow it but it will require some tech knowledge to follow instruction that are described here: https://www.reddit.com/r/Xiaomi/comments/amcck5/miui_10_keep...

smichel172• 6 years ago

Maintainer of Red Moon, a screen filter app, here. I've struggled with similar problems as others describe for a long time.

As far as I know, the only truly 100% reliable way to never be killed is to run as an accessibility service. I haven't implemented this in Red Moon but will likely add it as an option. I believe it has a large impact on bettery life.

You can also set alarms using AlarmManager to restart the service if it has been killed, and optionally wake up the device. This is what DNS66 does.

aabbcc1241• 6 years ago

look forward to get access from a cordova plugin

DanCarvajal• 6 years ago

(which can be hidden by the user)

I wish this was more obvious to people. Had to use my retire parent's phone to long into Netflix for them and it was overflowing with of persistent notifications.

ForHackernews• 6 years ago

Have you looked at integration with microg, yet? https://microg.org/ It seems like a natural candidate wanting a replacement of GCM/FCM.

kwanbix• 6 years ago

I noticed that. That said, how does whatsapp is never affected by this? How do they achieve it?

finchisko• 6 years ago

i believe whatsapp and few other most common apps are on a white list of apps allowed to run in the background.

gruez• 6 years ago

Does whatsapp roll their own push? I believe they use gcm, which is whitelisted by android.

mrtksn• 6 years ago

Wow, I didn't know that the Apps on Android do implement their own push services in the background.

For all the edge cases out there, OpenPush is great news.

However, this is a huge no-no for me as it means that I would need to actively manage the device to make sure that things do work as intended.

Not happy with Google "owning" the push notifications? Then maybe the solution should be a non-profit independent corporation running it.

Having a service-per-app app cannot be as efficient as having a single service serving them all. Both in terms of device resources and user time & experience.

Surely having an alternative is a good thing but the issue seems to be institutional(You don't want to have Google in charge), not technical(Google's service works great), therefore the solution out to be institutional too. Or something that acts like an institution, something like bl_______n.

est315• 6 years ago

This is a great and important project for F-Droid.

The problem is harder than it seems on the surface as idle connections can be terminated by the network operator. There's a paper about this: https://ieeexplore.ieee.org/document/8214213

I guess ultimately one should build a database about phone networks and store measurements of their maximum heartbeat intervals in there so that new connections can use those heartbeat intervals to save battery.

Also ideally the app should send its data via WiFi if available. So it has to detect when a WiFi connection is established and open a connection wia WiFi. Ideally the mobile connection is kept alive though for the case the WiFi proves unreliable. E.g. when I leave my home on foot I get further and further away from the WiFi and there's a short time span where my phone keeps trying to use the WiFi connection even though nothing gets through any more.

Detecting all these edge cases can be tricky. Fortunately the design makes the protocol between the push server and the on-phone app open to iteration and innovation. It's a good design!

awelkie• 6 years ago

Excellent point. So how does Google solve the problem? Do they just coordinate the heartbeat intervals with the carriers, or do they bypass the Internet altogether and use some cell-network-specific push mechanism? If the latter, is this mechanism available to everyone or just partners of the carriers?

est315• 6 years ago

Do they just coordinate the heartbeat intervals with the carriers, or do they bypass the Internet altogether and use some cell-network-specific push mechanism?

IIRC they use the xmpp protocol, so standard TCP over IP. In threads about this topic you often hear claims that Google has special deals with carriers. Whether this is true or not, no idea. I haven't seen any confirmation of this. It would be interesting to conduct measurements whether FCM sends less frequent heartbeat messages than the timeout is for "normal" long lived tcp connections.

In any case, deals or not, I guess Google has built a database as I've mentioned. But I'm only guessing here.

netsharc• 6 years ago

AFAIK these disconnects still happen (my knowledge might be outdated). It would be cool if the phone figures out by the weakening signal + location that it should turn on the data over cell connection.

toast0• 6 years ago

If you can build a database, great, but you can probably get by with a device based adaptive ping frequency, reset when connectivity changes.

Start at a 1 second ping, and after each successful ping, double the interval until you get to over a minute. Then add a minute between each ping until you get to 10 minutes, then add ten seconds between each ping. If a connection drops, use the last successful interval for tenish pings, then probe with an additional one second. If the connection drops again while probing, reconnect and wait longer to reprobe. If the first ping failed, then assume the network changed and reprobe from the top.

And yes, you do need to start at one second; I've seen networks with a ten second NAT timeout, which was amazingly disruptive, and thankfully temporary.

TheAceOfHearts• 6 years ago

I have a tangentially related tip or suggestion of which many appear to be unaware: most (all?) cellphone service providers (at least in the US) let you send and receive emails over text message. For example, with Project Fi you'd use $NUMBER@msg.fi.google.com [0]. You can use this as a super simple alert or notification system for scripting and small self-hosted apps without having to setup any additional services or tools. However, it's important to keep in mind that both email and text messaging are unencrypted by default, so you should be careful with what you send.

Let's consider a situation in which this might be useful: You have a 15 to 30 minute task running, such as downloading a large file (e.g. Linux ISOs) from a remote server or compiling a small C++ project. Instead of sitting around waiting you could go outside for a short walk in order to get a bit of exercise and sunlight, which helps you get to the recommended minimum of 10k daily steps. Once the task is finished you receive a text message which prompts you to head back inside.

If you just want to send yourself notifications then you can actually get pretty far only using email and text messaging. However, I'm aware that OpenPush solves a few different problems, and I love that we're finally getting an open-source self-hosted alternative to GCM.

[0] https://support.google.com/fi/answer/6356597?hl=en&ref_topic...

est315• 6 years ago

The protocol between the push server and the push app on the device is entirely up to those two. You can use e-mail over SMS if you want.

commoner• 6 years ago

OpenPush could be very valuable for open source Android apps once it is fully developed. Currently, open source Android apps that do not depend on Google Play Services are using WebSocket or Server-Sent Events (SSE) for push notifications. For example, Signal falls back to WebSocket when Firebase Cloud Messaging (FCM) is not available.

Tutanota published a blog post explaining how they deliver push notifications to their Android app with SSE instead of FCM:

https://f-droid.org/en/2018/09/03/replacing-gcm-in-tutanota....

OpenPush would reduce the amount of work needed to implement push notifications compared to the made-from-scratch implementations developed by Signal and Tutanota.

hjkgfdfgh• 6 years ago

I wish something like this existed that could additionally proxy FCM notifications. Allowing push notifications without any Google involvement is the ideal long term goal, but one transitional path I dream about would be forcing Google to proxy all their notifications through a host of your choice (ideally that you control).

I think I have about 30 false start projects with names like PushMux and rough outlines on potential ways to accomplish it.

Tangentially, I also dream of having true one-way push notifications. Send a minimal notification to the target device via FLEX/POCSAG or even one-way Iridium, and allow the device to connect to the internet as needed to fetch the rest of the notification.

Andrew_nenakhov7• 6 years ago

I don't think it'll fly, here is why:

Our product is a XMPP client, which can't work if it is not actively running in the background, so we were working with making apps run in background on Android for over 10 years, since version 2.0.

The trend is clear: with every update, we get more restrictions. Not only bg services are killed off from memory by aggressive memory managers of crappy phone vendors, Google themselves push developers to rely on proprietary FCM services. The writing is on the wall, and one day background services in Android will simply cease to work, all in the name of better battery life and customer care. So this OpenPush service will be short lived, it'll likely be disallowed to work in this form in Android 12 or 13.

The only proper long-term solution here is an anti-monopoly legislation, that would make Google allow Android users to subscribe to a third-party push notification service. Such service is, actually, a relatively simple: current FCM is just a modified XMPP connection that a phone keeps, and it will be easy to replicate and deploy.

Apple is guilty of this, too. A proper anti-monopoly legislations should target them as well, and not only make them allow third-party app stores, but third party push notifications as well.

This control of push notification services and restriction creep that occurs on both platforms makes Google's and Apple's grip on their respective OSs much tighter than Microsoft's grip on Windows ever was. This tyranny must be stopped, and I don't believe it can be done without legal action, by technical means only.

est315• 6 years ago

one day background services in Android will simply cease to work, all in the name of better battery life and customer care.

In that instance, having an app which centralizes the push service needs of other apps would be highly useful, as it could then be installed as a system app with the same kind of access as the FCM service. For many phones, custom ROMs are available. Average people probably won't have the skills to install custom ROMs. But the goal of OpenPush is not to replace FCM from all phones everywhere. It's to meet the needs of the subcommunity of people who want to use degooglified Android, maybe also later on integrate with non-Android mobile OSs. In that subcommunity, people likely already use custom ROMs, and among those who don't, the readiness is higher to install one.

Andrew_nenakhov7• 6 years ago

You see, unless this app is built in a custom ROM, it will cease to work, too! Because it needs to be running in the background and maintain connection to push server.

anderspitman3• 6 years ago

The filesystem APIs are moving in a similar direction if I'm not mistaken? Less and less control for developers. It makes me sad, but also gives me hope that there might be a legitimate opening for a 3rd mobile OS to enter the market, built around OSS and privacy. Sometimes it's hard to keep the dream alive though. So many attempts have been made.

greatjack613• 6 years ago

100%, I was one of the many vocal devs on the android bug report who complained about the change of scoped storage.

They caved in for the current release, but they are going to force it in android 11.

On the bright side I think this will cause android engineers to become more of a specialty as the platform migrates more and more from just another java implementation to a whole new world of apis!

snarf21• 6 years ago

The problem is that every app thinks that they must run in the background. Every app must have millisecond push updates for their social media network that another Like was just received. Worse yet, most of them don't know how to do so in a symbiotic way. It makes complete sense that OS manufacturers have taken a stand to protect users from bad developers. They don't want the blame and they want to keep selling devices.

It is a tough spot. How do you have open app stores and not have them filled with malware? Android hasn't figured this out yet. Users are dumb and will just click on anything. They need to be protected from themselves sometimes.

Andrew_nenakhov7• 6 years ago

Well, it's understandable that many apps want to run as a regular process. Good old UNUX was and it's GNU/Linux descendants are quite OK with it. The users have control and can choose which apps to run and which not. If some app misbehaved, it was up to a user to identify it and remove a problem.

But Google decided that it will be our nanny and will guard all users against misbehaving apps, whether the users asked about it, or not, and you can't refuse this care. That is tyranny and should be fought against.

Now, I don't think that push notifications are bad. They are rather good idea and let them be. It is the monopoly on push notifications which is bad.

How do you have open app stores and not have them filled with malware?

Long before the 'app store' term was coined, Linux had software repositories, which worked like magic, and without issues:

    sudo aptitude install libreoffice

Boom, you now have libreoffice on your PC! If repositories had massive problems with malware, I'm not aware of them. So it was done decades ago and can be done again.

jayd16• 6 years ago

I'm conflicted. Its not like I want the same situation of many background services and tray icons you see in Windows on my mobile device.

jml7c5• 6 years ago

I think it's alright to have the _option_ to keep a background service running, as long as the user is kept informed of the power impact (preferably with hard numbers). There are some use cases where it's the most reasonable option.

swiley• 6 years ago

I really like the web push notification architecture: you register with the service you want to be notified from (like a chat server or an IRC not) and give them an ID from the push notification server your device listens to.

There’s no extra control outside of the user and the services they want to talk to (I think you can even host your own on Firefox), no vendor lockin, and the power and bandwidth usage are all great. There’s too much money to be made breaking useful push notifications and charging for ads though so I’ll be shocked if it ever gets done right on popular consumer devices.

yencabulator• 6 years ago

so I’ll be shocked if it ever gets done right on popular consumer devices.

Seems to work nicely on my Android. Which would imply it's already done. (I'm sure greed will ruin it at some point, somehow.)

shujito• 6 years ago

Can websockets be used as a self-hosted push notification alternative?

OpenPush: A Free, Decentralized Push Messaging Framework for Android

Related Stories