Writing a chat application in Django 4.2 using async StreamingHttpResponse

https://valberg.dk/django-sse-postgresql-listen-notify.html

by ipmb • 3 years ago

103 33 3 years ago

pmontra• 3 years ago

This implementation is probably a little different but we were using long poll in the late 90s and early 2000s. The problem was that you were committing one thread (or worse, one process) to each client. That obviously doesn't scale unless threads are extremely light on RAM and either the OS or the runtime support a large number of them. I remember that a way out was using continuations. Jetty was a Java application server that supported them (random link [1]) One thread -> many connections. I didn't investigate how Django is implementing this now but CPUs and RAM are still CPUs and RAM.

[1] https://stackoverflow.com/questions/10587660/how-does-jetty-...

klabb3• 3 years ago

The problem was that you were committing one thread (or worse, one process) to each client.

This is mostly true today as well, although we do have beefier machines and more efficient thread pooling.

I did some personal research for infrequent real-time notifications. My conclusion was that many stateful connections are poorly memory-optimized by language runtimes and reverse proxies. Even with lightweight tech like Golang, gRPC, nginx, etc, it’s hard to push anything less than 30 kB for an idle conn, mostly from thread/goroutine stacks, user space buffers and (easy to overlook) a bunch of HTTP headers and TLS handshake state that often remain after establishing the conn. That’s without any middleware or business logic wants their share of the cake.

The only mature project I found that really took this stuff seriously is uWebSockets. It’s extremely lightweight, around an OOM better than most alternatives. Highly recommend – they also have a socket library so you can implement other protocols if needed.

Anyway, it’s important to be aware that massive amounts of long running connections is not just about adding a websocket/SSE library. Chances are you’ll need a mostly-separate serving stack for that purpose.

Ralfp2• 3 years ago

Just a heads up that currently Django is not cleaning up open PostgreSQL connections when ran in ASGI mode, leading to too many open connections error: https://code.djangoproject.com/ticket/33497

whalesalad2• 3 years ago

that’s kind of a dealbreaker

thefreeman• 3 years ago

could this be addressed by something like pgbouncer or another connection pooler?

pdhborges3• 3 years ago

I'll have zero trust running async Django until all sync_to_async and async_to_sync calls are removed from the code base.

lisasays3• 3 years ago

Care to elaborate?

tomwojcik• 3 years ago

Same here, but without these weird utils it doesn't get any better.

I have 7 YoE with Django. Its great at so many things. You see some code, like middlewares, and immediately understand what's going on.

Now, we also have Starlette. The base of all new, fancy asgi libraries. Here's the base middleware class.

https://github.com/encode/starlette/blob/8d7a1cacfb3e1a30cbb...

In the last couple of years I heard 'we're running fastapi on production. Wanna join us?' so many times... but the reality is that it's still not suitable for prod. Who wants to work with a code like that if you have a readable, stable Django? I'm clueless.

baq• 3 years ago

Running Postgres without a connection bouncer is a huuuge no-no already, but if this breaks the default mode of pgbouncer it’s basically a showstopper.

winrid• 3 years ago

Your average django app running on one or two servers with four workers each (which use connection polling) does not need pgbouncer.

kolanos• 3 years ago

Reading that bug report it looks like Django hasn't been isolated as the source of the problem? Looks like Carlton Gibson closed it until it is reproducible. Problem could be in uvicorn, psycopg2, etc

Ralfp2• 3 years ago

It was. Its an implementation change made in Django v4 that caused this.

etimberg• 3 years ago

Yeah, I've definitely seen odd things in Django ASGI. Had at least one instance where it looked like it leaked data between requests but couldn't get a clear enough reproduce. Switching back to WSGI worked fine though

samwillis• 3 years ago

All the new asyncIO stuff in Django is awesome, they are doing a phenomenal job retrofitting it all to an inherently sync designed framework.

One thing to note, it's been possible to build these sort of things with Django by using Gevent and Gunicorn for over a decade, and it works well.

In many ways I wish Gevent had been adopted by Python rather than AsyncIO.

jononomo• 3 years ago

Where can I learn more about this? I've been thinking of trying to integrate Supabase Realtime (https://github.com/supabase/realtime) into my Django app (without the rest of Supabase), but I'd also like to keep things even simpler if possible.

Also, what was the reason not to go with Gevent?

lastofus• 3 years ago

Also, what was the reason not to go with Gevent?

The biggest downside of Gevent IMO is that it enables the magic of turning sync code into async code via monkey patching things like the socket lib. This lack of explicitness can make things a bit difficult to reason about, without a good mental model of what Gevent is doing underneath the hood.

bastawhiz2• 3 years ago

I'd strongly recommend django-channels if you want to change very little. It feels like Django and it integrates without a lot of headache (at least in my experience).

btown• 3 years ago

Can second the Django/Gevent/Gunicorn stack - we use it in production and IMO it's much easier to reason about and code in than asyncio. Just write synchronous Python, including long-running `requests` calls, and the system will yield control whenever you're blocked on network or disk I/O, no matter how deep that is in your stack. The entire ecosystem of Python libraries just works. I also wish more people knew about gevent - it's truly magical, especially if you need to work with APIs with unpredictable latency!

jwtnb• 3 years ago

Hi, thank you for your comment. I have been looking for correct ways to integrate websockets on my django app. Currently I'm using gevent gunicorn setup. Can you elaborate a bit more on how to make websockets work with an existing app? Thanks.

bastawhiz2• 3 years ago

I did this. If you're on a modern version of Django, it's actually pretty easy: swap out gunicorn for an async alternative and install django-channels. I even got it working with GraphQL subscriptions (using graphene) without much hassle.

kiraaa• 3 years ago

https://github.com/sysid/sse-starlette makes token streaming so much easier in python

dnadler• 3 years ago

I’m using this for an internal ChatGPT UI clone and it’s working great.

The actual biggest pain for me has been the front end handling of it with the fetch api. But that’s likely just due my inexperience with it.

Thaxll• 3 years ago

Do modern chat actually use websocket and the like? Discord / Slack on the web ( browser ) what do they use?

vorticalbox• 3 years ago

The slack_bolt library has support for web sockets.

https://api.slack.com/apis/connections/socket

holler• 3 years ago

yes they use websockets

jhgg• 3 years ago

Discord uses websockets on all clients/platforms.

Waterluvian• 3 years ago

“ The idea is that the client "subscribes" to an HTTP endpoint, and the server can then issue data to the client as long as the connection is open.”

To those who have been around longer than me: isn’t this just long polling, that predates websocket?

Writing a chat application in Django 4.2 using async StreamingHttpResponse

Related Stories