Rewriting Reddit (2005)

http://www.aaronsw.com/weblog/rewritingreddit

by amalantony06 • 12 years ago

345 114 12 years ago

aaronsw.com

linuxydave3• 12 years ago

Whatever happened to that Lisp clone?

mineshaftgap• 12 years ago

It must be around here somewhere.

valarauca1• 12 years ago

<joke>

Likely consumed by meta-patterns, like every other project who's primary reason to exist is to be a LISP clone of something.

bhhaskin2• 12 years ago

I wounder...

swartkrans2• 12 years ago

I'm not sure if linuxydave is joking, but for those not in on the joke, Hacker News is written in arc, a Lisp variant. Hacker News resembles early versions of reddit quite a bit and Reddit was a ycombinator funded startup. Early on there was a big overlap in the two communities, although Hacker News was quite a bit more exclusive and limited in scope.

27182818284• 12 years ago

I knew HN was written in Arc, but I wasn't aware that it is the Lisp clone referred to by this piece. Was it?

geerlingguy• 12 years ago

Just wondering (as someone who only uses/touches Python indirectly, through some projects that use it, but is not currently a 'Python developer' by any means); was this posted in response to some recent drama in the Django community, or as an argument against Lisp?

I'm just wondering why it seems this story is quickly getting upvoted on the front page of HN, when it seems there are no other frontpage stories remotely related to Django, Reddit, or Lisp?

rrradical• 12 years ago

I upvoted just because it's a neat, historical document that I missed the first time around. I have no particular stake in Lisp or Python.

bhhaskin2• 12 years ago

Yep, I knew that Reddit was originally written in Lisp and made the change to Python at some point. It is neat to learn exactly why and how.

p4bl0• 12 years ago

when it seems there are no other frontpage stories remotely related to Django, Reddit, or Lisp

That may be true about Django, but not the other two.

Reddit being a YC-backed company, it always has been a hot topic here on HN. Maybe less now than a few years back but the same HNers are still here.

And as for Lisp, it has always been a hot topic on HN too. Many people know HN via PG who they discovered for his Lisp writings. HN is itself written in a Lisp dialect. And Lisp related stories appear monthly on the frontpage, maybe weekly if we take all Scheme and Emacs related links into account. Just take a look at the search results for the past week with any Lisp related keyword.

erkose• 12 years ago

Aaron Swartz

swalsh• 12 years ago

A combination of an interesting essay that most haven't read, and the fact that it makes us remember Aaron Schwartz

Ntrails• 12 years ago

Until I saw this I didn't even click that it was him.

Wow.

PublicEnemy1112• 12 years ago

"was this posted in response to some recent drama in the Django community"

What drama?

PublicEnemy1112• 12 years ago

Why the downvote? My question was sincere. I have kept up with the framework, but not the community

mikepurvis• 12 years ago

Note that reddit has rewritten at least once since the web.py rewrite. cf. https://news.ycombinator.com/item?id=51078

I recall reading a more scathing post from spez at one point about frustrations with web.py, but I don't seem to see it now. It may have been removed. In any case, Flask seems to be the inheritor of the "write webapps in Python" description.

thearn4• 12 years ago

In any case, Flask seems to be the inheritor of the "write webapps in Python" description.

Flask definitely fills that role for me. It's very simple to get up and going with it, and deploys easily.

thalesfc3• 12 years ago

Me too. This post is making me so happy to see that others like Flask as I do. :)

mox1• 12 years ago

web.py is still my go to for lightweight webapps. I won't start a big discourse on the pros and cons vs. other technologies, but I will say it is incredibly easy to setup and use.

The current github page is here: https://github.com/webpy/webpy

If you are looking to get up and running quickly, I would recommend forking this skeleton: https://github.com/jzellman/webpy-skeleton

Finally, the built in database framework is a bit simplistic. I like to pair this with Peewee (included in the webpy-skeleton).

rdtsc3• 12 years ago

Yeah same here. It has the cleanest mapping from REST resources to Python class + method. It just made sense to me:

The router block at the top plus:

    class MyResource:

       def GET(self,...): ...

       def POST(self,...): ...

       def PUT(self,...): ...

       def DELETE(self,...): ...

Looks really nice. Haven't tried serving or rendering templates from it, so never tested that code.

tiglionabbit• 12 years ago

Check out Flask's MethodView

euphemize2• 12 years ago

FWIW, many other python web frameworks use the very same pattern.

jedberg7• 12 years ago

web.py never really worked well, so we ended up rewriting it again. The interesting thing was that Django still wasn't up to the task when we did that, mostly because it's templating engine was too slow. So we chose Pylons instead.

It should be noted that Django has since fixed that issue.

swartkrans2• 12 years ago

Was there one specific problem with web.py or multiple issues? Was there ever a write up about what you guys learned from trying web.py? Would be an interesting read.

jedberg7• 12 years ago

Keysersosa answered this better below, but basically maintainability. There was no writeup, I guess because doing write-ups about internal software switches wasn't really a thing back then. :)

Igglyboo• 12 years ago

Do you guys wish you were using Django right now?

jedberg7• 12 years ago

I don't work there so I don't really know, but I do know that I usually don't use Django for the reasons listed in the blog post -- namely that you have to live in their world view to write Django.

aquilaFiera• 12 years ago

No, the reddit project itself is still based on Pylons. However redditgifts (the reddit marketplace) is based in on Django so reddit does have Django.

rdtsc3• 12 years ago

Do you recall what didn't work? So far web.py looks like a really nice rest-ful api backed framework. I rather like it.

http://webpy.org/

jedberg7• 12 years ago

That version of web.py has little relation to the one that Aaron wrote. The current version looks pretty good but I have no idea what it's scalability and modularity story is.

keysersosa8• 12 years ago

A lot of the decisions made in designing web.py were also carried over to tornado's web framework (which not coincidentally is also a web.py).

At the time (and I have to admit my memory is a little foggy as it was 7 years ago), I believe we mostly switched away from it for maintainability reasons. It wasn't under active development, and we were in the middle of a complete MVC rewrite to impose some order on some otherwise organically grown code.

general_failure3• 12 years ago

Since you guys rewrote over the weekend, I don't understand why anything cannot work well...

TylerE2• 12 years ago

Django templating is still pretty slow.... the last time I benchmarked it (maybe 6 months ago) it was a fairly large amount slower than Jinja2 (3-4x maybe?)

dangayle2• 12 years ago

What's cool is that there is a legitimate push to make Jinja a first class citizen in Django, with other third party templating engines being available via a new API.

Spiritus• 12 years ago

Did you make sure to enable to cached template loader[1]?

Enabling the cached template loader often improves performance drastically, as it avoids compiling each template every time it needs to be rendered.

[1] https://docs.djangoproject.com/en/dev/ref/templates/api/#dja...

euphemize2• 12 years ago

Used Tornado, django and flask quite a bit, web.py only once. As others have mentioned, it didn't work that well, and the community around it is smaller than other frameworks. Again today, Django seems a bit cumbersome, Tornado works well but has a very low bus factor.

Pretty sure flask is currently the front runner in "python microframeworks".

thalesfc3• 12 years ago

Flask has my preference 2. I was wondering what was reddit's folks opinion for flask.

NSCFType• 12 years ago

Flask (2010) wasn't around when reddit was (re)written (2005).

gsk• 12 years ago

What's bus factor?

jms18• 12 years ago

If the author got hit by a bus, there's no one else around to maintain it.

Generally refers to "how many people can continue to develop/support this software if original maintainer (and close co-developers) get hit by a bus."

dragonwriter• 12 years ago

What's bus factor?

The number of developers that can be suddenly and totally lost from a project (e.g., by being hit by a bus) without the project itself becoming unable to proceed effectively.

joesmo2• 12 years ago

I think you can leverage these criticisms at about 90% of the framework software out there written in just about any language. Most developers like to create complex, complicated software mainly to show off their "cleverness" without thinking through the implications for the people that have to use it. It's as if they think that by cramming in every design pattern they can think of and using many different libraries together for no perceptible reason makes the software better when they could've achieved the same thing much more simply and cleanly if they had put a little thought into it. Such over-engineering is equally a problem as the spaghetti code of programmers who don't know better.

sbilstein• 12 years ago

I'm not too familiar with Django but with several other frameworks and I definitely feel like this applies to lots of them out there.

In my experience though, I've worked on teams with developers who criticize frameworks for those reasons, choose a minimal or no framework at all, and spend a ton of extra, unnecessary time reinventing a worse version of a framework they decided 'sucked' once they realize they need security, error handling, form processing, an ORM etc... Over-engineering is an epidemic.

LeafStorm• 12 years ago

I almost went down this route on a project once. As soon as I noticed myself writing an identity map, I slapped myself in the face and just downloaded Symfony.

crdoconnor• 12 years ago

Django actually seems to be one of the least offensive frameworks to me in that respect. It didn't really try to be clever and it did things that made sense with a minimal amount of magic.

joesmo2• 12 years ago

Great point. That's probably the worst case, IMO, along with over-engineering and plain spaghetti. I should have mentioned it, but I think my PTSD from dealing with such engineers and projects is still overwhelming.

thathonkey• 12 years ago

And I think you can leverage THESE criticisms at about 90% of software out there written in just about any language.

ryandrake• 12 years ago

One of the more simple litmus tests I use when evaluating a framework: Does it provide (and require you use) its own versions of data types already provided by the language (or the language's standard libraries)? If so, it's a big red flag. Not necessarily a reason to disqualify, but definitely cause to apply extra scrutiny.

The thought process is: If I have to spend precious development time converting all my std::strings to and from YourStrings, then I can bet that there is plenty more un-necessary cleverness and complexity waiting to bite me once your framework gets its claws into my code.

sillysaurus3• 12 years ago

There are many reasons not to use std::string, some of them more valid than others.

Judging a C++ framework by whether it implements its own string type probably isn't a good idea. C++ is inherently limited, and a string type is often the only way to have certain design guarantees. In a performance-critical context, it's sometimes crucial to control the exact pattern of allocation. And not just how they're allocated in memory, but also how often they're allocated. You can control std::string's pattern of allocation by using a custom allocator, but you can't control how often std::strings are created, for example, because std::string implements an interface which is guaranteed to construct temporary strings. This normally isn't a problem, but in contexts where every millisecond is crucial (like gaming) this can be disastrous, since it's very hard to optimize the performance characteristics of std::string once it becomes pervasively used throughout your codebase.

Another reason to make your own string type is for proper unicode support. Maybe C++11 or C++14 added features to help with that, so maybe this is less valid nowadays. I haven't kept up.

coldtea4• 12 years ago

One of the more simple litmus tests I use when evaluating a framework: Does it provide (and require you use) its own versions of data types already provided by the language (or the language's standard libraries)?

That's a pretty low bar...

temuze• 12 years ago

The first Python web framework I used was Django. I'm really glad I came across it first because I was inexperienced at the time and it forced me to adopt a lot of good practices.

However, now that I'm more experienced, I can relate a little to what Aaron is saying here. Learning all of this boilerplate stuff is annoying! Recently, I started contributing to a friend's rails app and I found the process of learning how to use each separate little tool tiresome.

This is why I like Pyramid - it starts off dirt simple. Hello World is almost as short in Pyramid as it is in Flask. As you grow, you can start organizing and adding different things as you need them. It has absolutely 0 magic. I find Pyramid to be a happy medium between Flask's minimalism and Django's extensibility.

Yuioup• 12 years ago

I wonder. How popular is Pyramid? I ask because I found Pyramid to be quite intimidating the first time (I'm not an exprienced Python developer). It took me four attempts to finally 'get' it - the first three times giving up in frustration and leaving it alone for 2-3 months before trying again.

Now that I get it though, it seems like an interesting framework but not a productive one. I've only dabbled in Pyramid with my free time and have never built a serious project with it, but I can't imagine building anything large scale with it.

(I'm a .NET developer BTW, MVC is what I work with in my day job)

achillean• 12 years ago

Shodan is a decently-sized website and runs on Pyramid (https://www.shodan.io). I don't know how popular it is in the general Python community, but it's been amazing to work with as a framework.

sontek• 12 years ago

We (https://www.surveymonkey.com/) use it for all our services. I can't imagine using another framework! Pyramid is great for what we do.

I also know http://www.cars.com/ is heavily powered by Pyramid.

dangayle2• 12 years ago

I think a lot of that is in the presentation of the tutorials and the documentation. There's a book on Oreilly called "Lightweight Django" that strips away all of the absolute non-essential cruft of Django and starts from scratch, slowly adding pieces from the framework as it goes along. It shows you why you need each piece and why it goes where it goes.

Django doesn't need the magic, it's just annoyingly presented that way.

gprasant• 12 years ago

They say hell hath no fury like a lisp fan scorned. I wonder if they hadn't been acquired, would they have still used Lisp ?

bensw• 12 years ago

This blog post refers to a rewrite that happened before the Condé Nast acquisition.

keysersosa8• 12 years ago

We switched to python about 4 months into the startup and a full year before getting acquired.

raldi2• 12 years ago

And some people still haven't forgiven you.

zoomerang• 12 years ago

I can remember this switch back in the day, but for some reason my memory has failed me. I could have sworn the original was in Haskell, not Lisp.

Selective memory I guess.

alexis• 12 years ago

Great post on that when Steve made the switch. http://www.redditblog.com/2005/12/on-lisp.html

andyzweb• 12 years ago

RIP Aaron ;_;

general_failure3• 12 years ago

Is reddit that simple to rewrite over the weekend?

wcummings2• 12 years ago

In 2005, very probably

keysersosa8• 12 years ago

Yup that was the case. Though, I believe it was also a long weekend -- I think it might have been Veteran's day because I remember putting the finishing touches on it around Thanksgiving.

wcummings2• 12 years ago

What Lisp webserver was used, originally? Hunchentoot? What web servers/stack are Lispers using these days?

keysersosa8• 12 years ago

At the time it was "TBNL" (http://weitz.de/tbnl/), which I believe has indeed evolved into Hunchentoot.

elwell• 12 years ago

What web servers/stack are Lispers using these days?

If you're using Clojure, ring + tomcat, etc...

eudox• 12 years ago

What web servers/stack are Lispers using these days?

Clack (https://github.com/fukamachi/clack)

orthecreedence• 12 years ago

I use Wookie a lot (http://wookie.lyonbros.com/) but I'm biased because I built it. It's a non-blocking CL app server.

_stem2• 12 years ago

For all people still thinking that Django is not "lean" or flexible enough for their next project, let me just leave this here: https://speakerdeck.com/mstepniowski/django-best-python-micr...

A single-file web application in Django: https://github.com/mstepniowski/microdjango/blob/master/poll...

keysersosa8• 12 years ago

Yeah it should be noted that all of the referenced decisions we made around web frameworks in this thread predate django's 1.0 release. I think I was comparing pylons to django 0.96 iirc, and in 2005 when aaron wrote web.py, I don't think django was on anybody's radar (or at least it wasn't on ours).

Django has definitely come a long way!

INTPenis2• 12 years ago

A single-file web application in Django: https://github.com/mstepniowski/microdjango/blob/master/poll....

That still looks very complicated compared to most microframeworks I've used, including web.py.

And all that django-specific logic that needs to be in there makes it harder to read unless you're familiar with the django ecosystem.

coldtea4• 12 years ago

And all that django-specific logic that needs to be in there makes it harder to read unless you're familiar with the django ecosystem.

What "django-specific" logic? Those are standard imports and a few helper objects, the kind of which you'll find in any web framework.

[edit: importants => imports]

pshc2• 12 years ago

We currently use web.py + gevent to run our API where I work. It's really easy to work with since it's so tiny and simple, and since we only need JSON in and out for now, it's perfectly sufficient.

Before web.py, we were on a scary Tornado implementation. The API was rewritten to web.py in a week with a minimal delta. We won't be on web.py forever, and gevent sometimes kicks up minor fusses, but I think web.py has given us a nice malleable form that helped us get out of a terrible place, and we can easily hop off once we decide where we want to be.

RIP aaronsw.

quasse• 12 years ago

I'm not very familiar with Tornado (we've switched to mostly Flask now) but I'd be interested to know what kinds of problems cropped up that lead you to switch to web.py

pshc2• 12 years ago

Tornado requires asynchronous I/O. Almost all of the existing I/O was done synchronously. Yeah.

tim333• 12 years ago

For another take on the Reddit rewrite from Steve Huffman see:

http://www.pyvideo.org/video/234/pycon-2009--keynote--reddit...

25m 40s

gcb0• 12 years ago

"It means readable HTML with the proper HTTP headers."

said all the other frameworks's authors a few months before him that are now churning out deprecated headers just like that one will in a couple months, while gallantly trying to protect the user from handling headers.

voltagex_• 12 years ago

Huh?

83457• 12 years ago

Would clojure fare better now?

closetnerd• 12 years ago

Its incredible how intelligent this kid was.

ocb• 12 years ago

Just 19 when he wrote this!

coldtea4• 12 years ago

The Lisp newsgroup, comp.lang.lisp, was upset about the switch that they’re currently planning to write a competitor to reddit in Lisp, to show how right they are or something.

That must have ended as a huge success, because as we all know the important thing behind a website is the backend language...

Perceval• 12 years ago

Isn't Hacker News written in Lisp?

dllthomas• 12 years ago

I think you miss the point of the parent comment, which is that backend language is an implementation detail and generally shouldn't matter to users. You can disagree with that point, it just is a very different point than anything addressed by noting that HN has a Lisp backend.

rpdillon• 12 years ago

I've used CherryPy on a number of projects, and I really like it, but rarely see it mentioned in discussions of Python web programming. It's light and minimal, and seems equally deft for writing APIs as conventional web page driven apps. Flask is mentioned all the time, however. Does anyone have insight into the popularity of Flask vs. CherryPy?

Rewriting Reddit (2005)

Related Stories