Cloudflare Pub/Sub: Programmable MQTT Message Brokers

https://blog.cloudflare.com/announcing-pubsub-programmable-mqtt-messaging/

by elithrar • 4 years ago

128 47 4 years ago

blog.cloudflare.com

0xy2• 4 years ago

Cloudflare is really fast putting the pieces together to become a modern simpler AWS. By focusing on edge networks and programmability from the start they've been able to outmaneuver AWS in product development and pricing.

AWS today is stable, reliable but expensive and rigid. A lot of their products feel like they've been taken out of the oven 15 minutes too early. AppSync, API Gateway, CloudFormation and others all feel lethargic, stagnant or not fit for purpose in the current world.

I don't think AWS is in a position to compete with Cloudflare's strategy without massively slashing bandwidth prices and improving their product development. They also aren't really a credible edge platform. Lambda@Edge is really bad, CloudFront is slow and terrible compared to Akamai, Fastly and Cloudflare (and Amazon's retail business doesn't even use it -- because it sucks).

Super interesting to watch. ($NET shareholder)

astlouis44• 4 years ago

Great take, I agree. Thoughts on a Fastly or Cloudflare acquisition by one of the tech giants?

0xy2• 4 years ago

If Akamai were smart they'd buy Fastly yesterday, because they're rapidly falling behind. Cloudflare is harder to see because of antitrust concerns. With Fastly's current position, they should sell, and someone should buy them for the network and progress on edge compute alone.

nicoburns2• 4 years ago

They have no non-function compute though, which is arguably AWS’s core feature.

jbverschoor• 4 years ago

It's also the things that's easiest to migrate away from to a cheap-and-performant VPS or dedicated server

fasteo3• 4 years ago

cloudflarer around here ?

Did you wrote the MQTT broker from scratch or are you using some existing broker (Mosquitto , Nanomq, etc). Just curious.

elithrar5• 4 years ago

(I'm the PM for this at Cloudflare & the blog author)

We looked at a number of existing options — trying to strike a balance between languages we run at scale (tooling, ops, security), memory safety, and long-term maintainability as a growing company.

We ended up writing an MQTT broker from scratch in Rust, with our own Durable Objects[1] for consistency and co-ordination across our hundreds of edge locations (and tens of thousands of machines). The MQTT spec itself[2] is very approachable, which made this undertaking easier still.

One of the increasingly cool parts of Cloudflare is that we use more and more of what our customers use: our Broker service sits behind Spectrum[3], and we use Workers + DO for all of the metadata/session handling. The next set of new products will invariably start to build on Pub/Sub itself, too :-)

[1] https://developers.cloudflare.com/workers/learning/using-dur... [2]: https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html [3]: https://developers.cloudflare.com/spectrum/

fasteo3• 4 years ago

Thanks, make sense.

Actually, I asked because it surprised me that D1 - sqlite - was not written the same way.

[1] https://blog.cloudflare.com/introducing-d1/

maxpert• 4 years ago

Do you plan to opensource the implementation some time?

fisian• 4 years ago

Some time ago I was at a talk about verifying different MQTT brokers so they fulfill the spec. They presented several inconsistencies they found (in OSS brokers) and submitted fixes for.

I looked up if there's some info online and found that paper (unfortunately paywalled): https://graz.pure.elsevier.com/en/publications/learning-base...

kitd• 4 years ago

The MQTT spec itself[2] is very approachable, which made this undertaking easier still.

Indeed. I've found writing an MQTT client is a good tutorial project when learning a new language.

rafaelturk• 4 years ago

Not clear for me if I can `push` messages from my backend to client? At lest for me this is a killer feature.

elithrar5• 4 years ago

In the MQTT world, in which Pub/Sub is based, every publisher is a client, and every subscriber is a client. You can be both, too!

Assuming your existing backend is in $SOME_CLOUD_PROVIDER, publishing messages to Pub/Sub is just a few lines of code in your favorite language - we have a couple of examples up in JS + Rust - e.g. https://developers.cloudflare.com/pub-sub/examples/connect-j...

Subscribers receiving those messages can be anywhere you can run an MQTT client, which turns out to be... a lot of places :-)

(I'm the PM for this, and we'll be adding more examples to our docs in the coming weeks)

hack_tmmm• 4 years ago

This is such a wonderful news

VectorLock3• 4 years ago

I like how Cloudflare is taking a serious run at AWS.

And you can't even subscribe a program to a SNS topic and Cloudflare is giving us pub/sub.

nullwarp• 4 years ago

FWIW with AWS, despite the naming, the IoT Core MQTT broker is an excellent product for exactly this - it's just sold as IoT specific. It's really just an MQTT broker at it's core and could probably be split off into it's own product without the IoT branding.

manigandham4• 4 years ago

AWS has several "pub/sub" like services: SNS, IoT MQTT, Managed Kafka, Kinesis, EventBridge, etc.

VectorLock3• 4 years ago

And they're all kinda fucky, like SNS handles a lot but you can't write a program to receive events.

sieabahlpark3• 4 years ago

Sounds like a SQS if you want to subscribe. SNS is for push notifications?

losvedir• 4 years ago

Wow, another neat Cloudflare release. Is there some sort of "innovation week" type thing going on right now?

One question that the example given raises for me, is how to manage that function I see there in a robust, "proper software engineering" sort of way. This has been a challenge for me on AWS when trying out Lambdas. The individual pieces are cool, but the system that connects them all together is a little opaque. How do you version control the code? How do you keep track of which pieces work with which other pieces?

We use terraform to manage the AWS resources, but it's kind of clunky, and not official. Is there some Cloudflare-supported way to maybe declaratively define your different resources?

Someone who has built a somewhat complex app with a lot of workers, and durable objects, and different pieces of the ecosystem: how do you manage that? How do people review changes?

Since it sounds like from another comment that Cloudflare dogfoods its own services internally, I'm curious if there's a write-up about how they actually do so? The little function code snippets give a nice taste of the code a worker might run, but that's still kind of a long way clearing up for me how you would actually use this in practice in a standard production environment.

brightball• 4 years ago

Platform Week

jffry• 4 years ago

Is there some sort of "innovation week" type thing going on right now?

Yes. "Platform Week 2022": https://blog.cloudflare.com/platform-week-2022/

piperswe• 4 years ago

At least on my team at Cloudflare, great Terraform support is a basically required part of any API change. We have an official Terraform provider (https://registry.terraform.io/providers/cloudflare/cloudflar...), and nearly every API feature is provisionable using that provider.

anonymousDan• 4 years ago

How does this compare to something like Ably?

knodi• 4 years ago

I love it!! There needs to be more message brokers as a service providers.

shikivi• 4 years ago

Pub/Sub, Workers, KV, Durable Objects, R2... yes, these all sounds promising. Awesome tool for developers. But there is one huge obstacle in the middle that prevents most to actually try this stack.

Logs and Account Access control.

Cloudflare is a very powerful platform, you can't let everyone be Admin. If you run anything, you need to be able to see logs it to ship it somewhere.

For some reason, Cloudflare thinks that these are enterprise feature. I got news for you, these are basic features for anyone who wants to build more than just a simple script for individual use.

You work so hard to build these features but let only a small subset of users really do anything meaningful with it

sitkack• 4 years ago

Honest question, not a snark attack. But would it be possible to build those things on top of Cloudflare itself, esp the logging? For access control, you would have to have a separate process controlling the CF api for configuration.

okaybuy2• 4 years ago

Not so simple when a lot of the tooling that facilitates easy development is tied to CF API, are you meant to build your own cloudflared and wrangler just so you dont have to grant everyone in the org super admin?

This is where AWS wins, AWS gives you everything from $0/month, IAM in the above case, but you don't stay at $0 for long, where as CF gives you partial bits but you remain at $0-low/pm for longer

Its actually really ironic the same company that says SAML for all for free, can't even get basic RAC on their own admin controls

imobbscf• 4 years ago

What would you like to see?

Disclaimer: Engineer on the IAM team at Cloudflare

manigandham4• 4 years ago

IAM/RBAC for members of an organization to control who can change settings, deploy scripts, add domains, etc.

sieabahlpark3• 4 years ago

Just provide an Aws IAM type interface and everyone would be happy. Also don't charge for it, because that just states your systems literally were not designed with authorization in mind.

jplevine• 4 years ago

We just announced something called the Workers Analytics Engine today: https://blog.cloudflare.com/workers-analytics-engine/

More coming soon!

Happy to chat more - jpl at cloudflare dot com

okaybuy2• 4 years ago

Yes, fully agree, just to give Fred access to develop a worker in dev environment we have to grant them access to the whole account, it's batshit insane. We created acmecorpx.com for dev and put this on a seperate CF account but it still sucks

michaelrhansen• 4 years ago

How many of these commenters work at Cloudflare? :)

haroldadmin• 4 years ago

I like that Cloudflare is steadily becoming a serious alternative to the existing cloud providers. If you folks can build a decent IAM tool for access control to Cloudflare resources, much more people will start using your services.

VirusNewbie• 4 years ago

huh, Fastly is doing pub/sub lately too with fanout now:

https://www.fastly.com/blog/fastly-fanout-why-real-time-mess...

eska• 4 years ago

Since you guys seem to be using Rust, what client library do you recommend for MQTTv5 support?

cehrig• 4 years ago

(Pub/Sub Eng at Cloudflare)

We are testing the broker using the Paho C wrapper https://github.com/eclipse/paho.mqtt.rust

.. and have added a client example to our dev-docs https://developers.cloudflare.com/pub-sub/examples/connect-r...

Cloudflare Pub/Sub: Programmable MQTT Message Brokers

Related Stories