Disqus: Scaling the World’s Largest Django Application (2010) [video]

http://ontwik.com/python/disqus-scaling-the-world%E2%80%99s-largest-django-application/

by pajju • 14 years ago

109 39 14 years ago

What struck me instantly was the use of Slony. I haven't listened to the whole thing yet but I am interested in their justification. Perhaps they just haven't moved to 9.2 yet.

d0ugal2• 14 years ago

I don't think they could move to 9.2 in 2010 ;)

sugarcode• 14 years ago

slony offers some compelling advantages over streaming replication - even on new databases we setup, we still like slony for several reasons:

* Version upgrades (streaming replication requires the same PG version between master/slave, slony does not)

* Logical replication gives us finer-grained control over how data is replicated across the cluster

* Ability to create additional indexes on slaves

slony isn't perfect and it has caused us some headaches, but its flexibility makes it our go-to replication tool for postgres.

lewispb• 14 years ago

Largest? Is Disqus really 'larger' than Instagram?

simonw• 14 years ago

I think it was in 2010 when they gave this talk.

saevarom• 14 years ago

Yeah these figures are really outdated. This one from PyCon 2011 is newer: http://www.slideshare.net/zeeg/pycon-2011-scaling-disqus-725...

mibbitier• 14 years ago

Also "large" is fairly meaningless when you don't include a measure of complexity.

Sharing images is a fairly simple task.

stef25• 14 years ago

Painful spam in the disqus comments at the bottom of that page. Don't they have something in place against this?

beaumartinez• 14 years ago

"+1"s are hardly spam, they're just low quality comments. If they filtered those, the web would be a very commentless place

d0ugal2• 14 years ago

kordless• 14 years ago

At least they haven't translated the slug into Russian. Every time I get email from them about a comment on my post, it shows up translated into Russian.

zerop• 14 years ago

All with apache and haproxy? No nginx or uwsgi/gunicorn or Redis? How old is this article?

ihsw• 14 years ago

Sept. 7 2010, according to their slideshare link.

http://www.slideshare.net/zeeg/djangocon-2010-scaling-disqus...

recuter5• 14 years ago

In the newer (2011) slides they mention growth by ~100 Million comments in 6 months. Most comments are tiny, I'd be surprised if this whole dataset was much larger then ~100GB.

I don't really understand why Disqus and Reddit for that matter don't just switch to redis. There has got to be a good reason because I can't think of one. Sounds like you could run the whole thing from just one box and only have slaves for redundancy.

Why isn't vertical scaling more in vogue?

lamby• 14 years ago

Because people like things like transactions and are generally risk-adverse when choosing storage technologies?

bretthoerner2• 14 years ago

If you really think that's the case then you don't understand Disqus/Reddit OR Redis.

antirez• 14 years ago

Disqus is currently using some Redis AFAIK, but probably not to store the bulk comment data (and probably for good reasons since I can imagine an enormous difference between working set and total data set in this case of Disqus).

Kilimanjaro3• 14 years ago

Using django for a distributed commenting system?

Hammer and screws.

legutierr• 14 years ago

Just wondering (I'm not necessarially objecting to your statement):

* what would you see as the right they of app for Django, if not this? Obviously Django is handling it, so what's the issue?

* what is the right tool to build a distributed comment system?

Kilimanjaro3• 14 years ago

What is a distributed commenting system? An HTML script tag to load a static file that will ping an api that will serve you comments.

What is django? A framework with strong routing, OR-mapping, templates and admin modules.

You need none to build it. Your main concern shouldn't be a coding framework, but load balancing, caching, failover control, more sysadmin stuff than code.

Any language could do. Framework? not needed.

Now, for the backend system to control that monster, then yes, you may use django.

So, use django for complex apps that need routing, data management and UI presentation, plus a powerful admin module.

* I am a python/django developer.

riffraff• 14 years ago

how is disqus distributed? All the comments go to a central service.

TommyDANGerous• 14 years ago

Great watch.

zeeg5• 14 years ago

As these slides are very old, here's some updates:

* We use Flask and nginx in various areas now (the main app is still Django). Our realtime app, for example, is powered off of uwsgi and Flask.

* There are nearly 1b monthly uniques across the network serviced by the platform.

* ~300 servers

* Still Postgres (with Slony, multiple clusters), Redis, Memcache, and some Cassandra for newer things (not comments).

Also mostly confident we're still the "largest Django app" in terms of traffic.

recuter5• 14 years ago

Interesting.

Care to update on the other stats like requests at peak? Also, I wonder - if you're at 1b uniques is there anywhere to grow from here other then more engagement?

Oh, also: Would you say the 1% contribute, 9% curate, 90% consume heuristic holds true in your experience at this scale or is it even more skewed?

reinhardt• 14 years ago

Virtual servers or bare metal? What do you use for herding them?

lucian3032• 14 years ago

I second that. Would be interesting to know a bit more about the systems setup.

Also why the decision to use an ORM? To manage the partitions?

Disqus: Scaling the World’s Largest Django Application (2010) [video]

Related Stories