The switch: Python to Node.js

http://journal.paul.querna.org/articles/2011/12/18/the-switch-python-to-node-js/

by pquerna • 15 years ago

207 134 15 years ago

journal.paul.querna.org

espeed2• 15 years ago

The depth of libraries available in the Python or JVM ecosystems is going to dwarf what's available for Node. Were you able to find everything you need for Node?

jahewson• 15 years ago

OTOH JavaScript is the most popular language on GitHub

ergo142• 15 years ago

And how does this relate to the statement above? How many of this is DOM manipulation code for browser? i'll make a wild guess of 90%?

insin• 15 years ago

I always wonder what percentage of that consists of included dependencies.

clojurerocks12• 15 years ago

Part of this though is because web apps are for better or worse increasingly being almost nothing but javascript. And it becomes tedious moving back and forth between javascript and another language. Even if that language is really good. So i think people are just going with javascript for "everything" and rewriting their libraries in it.

nivertech• 15 years ago

HTML is the most popular language on GitHub ;)

substack• 15 years ago

Node is catching up pretty fast, although its modules might not be as mature yet:

    102257  perl    http://www.cpan.org/
     33270  java    http://search.maven.org/#stats
     31921  ruby    https://rubygems.org/
     18068  python  http://pypi.python.org/pypi
      5732  node    http://search.npmjs.org/

espeed2• 15 years ago

What's the quality like of most of those Ruby gems? I keep seeing people refer to the gem number, but how sophisticated are they?

Last time I looked there wasn't much along the line of SciPy, NumPy, NLTK, or Matplotlib.

matwood• 15 years ago

Count of modules seems like a useless metric. A better one would be modules that are considered ready for production. Even then, the only ones that matter are the ones you will end up needing for a project.

hello_moto4• 15 years ago

The actual number for Java is a little bit skewed probably. You're only using one maven central repository (which is already big enough). But there are other repositories as well such as RedHat's (JBoss, Hibernate, etc). SpringSource's (Spring frameworks, grails probably, and everything under SpringSource umbrella).

And also keep in mind that scripting languages have higher number largely because people use them for writing scripts/command-line as well. And if they insist to use Java for scripting well then... they deserve to be punished by walking up and down the stairs 20x per library per decision.

ak2173• 15 years ago

although its modules might not be as mature yet

a bit of an understatement.

clojurerocks12• 15 years ago

Of course it depends what libraries youre looking or. If youre looking for libraries for build the "basics" of a web app then there are plenty of libraries available. If yore doing other things though and looking for libraries that dont exist maybe its because youre using the wrong technology.

malandrew7• 15 years ago

Not necessarily. Sometimes the lack of libraries isn't solely because it is the wrong technology, but because it is the wrong community. Language and community are pretty intertwined.

LISP for example is a great technology for a lot of things, but library-wise it suffers because within that community there is a tendency for people to "roll their own" solutions.

Java has tons of libraries, but I'm not sure most people would say it's the right technology for a lot of things. It's generally the right technology for projects that involve lots of developers with mixed levels of ability (before some Java person down votes me for this, consider that Gosling himself made this point back in the mid-90s when he invented the language)

pquerna3• 15 years ago

No, we were not able to find everything we needed out of the box.

We had to write many things. There is a list of our dependencies at the bottom of this post:

http://journal.paul.querna.org/articles/2011/12/17/technolog...

Having said that, the only ones I consider moderately painful is when we need to deal with node-thrift, which on the Python or JVM is well maintained by existing groups.

baghali• 15 years ago

Did you evaluate EventMachine? If yes would you please share your findings.

pimeys4• 15 years ago

I've used fibers-enabled EventMachine in one part of our application. It's pretty nice, because you don't have to do callbacks and the Ruby fibers are pretty nice actually.

We're using it for handling jobs from Resque, which reads and writes from our database and do a ping to a 3rd party server.

But. The amount of bugs in the em-enabled libraries is just amazing. For example em-activerecord just didn't work so well under huge load (dropping SQL-connections), so I had to write my own mini ORM for the workers. Also Resque blocks by default, so of course doing a non-blocking and non-forking version was a priority.

The worst thing here are the error messages and backtraces. There are none (DeadFiberException for failing assert) and with hacks you can get out a bit more information if the job crashes.

Now thinking this again, it would be a good idea to write it again with Node.js. Just because it's still more mature compared to EventMachine, and all of it's libraries are reactor core friendly by default.

toisanji• 15 years ago

I'm using em in production also and I am seeing quite few bugs too. I wanted to ask you some eventmachine related questions, how can I get ahold of you, I didn't see your info in your profile.

mattgreenrocks• 15 years ago

It's a shame that fibered Ruby feels like such a ghetto. I'd really like to see it be a viable concurrency choice, especially for apps that call a lot of third party services.

sandGorgon2• 15 years ago

>For example em-activerecord just didn't work so well under huge load (dropping SQL-connections), so I had to write my own mini ORM for the workers.

That's interesting - could you elaborate (here or a blog post) on your findings. what was different between your ORM and em-activerecord that made it more performant.

The thing is - my first thought would have been to leave the ORM alone and focus on the DB side (like more sophisticated connection pooling/pgbouncer, etc. ). Which is why I'm interested in what went wrong in the ORM that made it screw up when used in a non-blocking kind of a setup.

lordlarm• 15 years ago

I've come up with the same conclusion regarding the Django ORM, as they have. An excerpt:

I believe our single biggest mistake from a technical side was not reigning in our use Django ORM earlier in our applications life. We had Twisted services running huge Django ORM operations inside of the Twisted thread pool. It was very easy to get going, but as our services grew, not only was this not very performant, and it was extremely hard to debug

Django has in many ways become the industry standard in regards to fast development - but it, as so many other ORMs, does not scale and produces horrible SQL-statements. Also for many applications I find Django to be a huge overkill - and in more recent times I've focused a lot more energy using microframeworks such as Flask and Bottle.

ergo142• 15 years ago

"Django has in many ways become the industry standard in regards to fast development - but it, as so many other ORMs, does not scale and produces horrible SQL-statements."

I disagree - many ORM's in python world? Try using sqlalchemy it will put django's ORM to shame, it will build fast good looking queries most of time, and if you have a rare case of something not good enough - you can easly tweak it. Can't say anything about Storm but saying that "industry standard" django ORM is OK because "others have same flaws" is just distorting reality to fit this thesis ;-) Ugly truth is django ORM is not ok and it bites many people when they are at the point where it's real pain to fix this - the more developers are aware of this the better.

jpk2• 15 years ago

I'm in the process of building an api in Django (GeoDjango, actually, with PostGIS). What are the gripes with the ORM and how could it be better? (Not that I'm disagreeing; I'm curious. I'm not experienced enough to know what makes an ORM okay or not.)

malandrew7• 15 years ago

Django (and Rails)(and their ORM) are easy solutions, but they are solutions that complect and therefore are not simple.

malandrew7• 15 years ago

I now have two down votes for this answer. Can someone please explain why this deserved a downvote.

viraptor2• 15 years ago

I'd really like to see that spreadsheet, since there are many python frameworks other than twisted available. There are eventlets/greenlets, there's gevent, there's diesel and monocle for minimal things, etc. Makes you wonder what actually made them change the whole technology... Than again, they could probably even reuse pyjamas to still write node.js code in python...

OT: "We should of used SQLAlchemy. We should of built ..." I thought this expression was Bristol-specific. Apparently it's taking US too.

tikhonj• 15 years ago

I don't know if you missed the link or if it was added later, but it's here: https://docs.google.com/spreadsheet/ccc?key=0AvBGESHWxhk2dHJ...

It seems they looked at Gevent, Go, C++ and the JVM. As usual, no love for Haskell :(, not that they're entirely to blame there.

reinhardt• 15 years ago

Too bad that the comparison doesn't make much sense: Node/Twisted/Gevent (language specific frameworks) vs Go/C++ (languages) vs JVM (platform).

viraptor2• 15 years ago

Thanks, either I'm going crazy, or this wasn't a link before...

lucian1900• 15 years ago

It's very obviously skewed to get Node.JS as a winner.

LeafStorm• 15 years ago

In the Southern US, at least, "should have" commonly gets shortened into "should've," which sounds sorta like "should of." That's still no excuse for writing it.

andrewcooke3• 15 years ago

so what makes javascript callbacks easier to understand than twisted's deferreds? aren't they basically the same thing?

and how does "we picked Node.js" logically follow from "It is obvious that the JVM platform is one of the best ways to build large distributed systems right now"?

dextorious7• 15 years ago

There probably a middle step "we also wanted to play with the latest shiny technology", that's not mentioned.

clojurerocks12• 15 years ago

Id have to disagree with that myself. While node and mongodb are both "shiny" they are because they solve a problem that people have and offer solutions to problems that are very appealing. Including as i mentioned i blieve in another comment here that using node allows you to only have ot work with one language. Which is great. So its true that python has similiar technologies such as tornado and twisted. But if node works then why not just stick to one technology to have to maintain rather then multiple ones. Also node has a great community that is increasing. Many python libraries dont have this. Meaning really documentation. For example tornado has very little documentation.

andrewcooke3• 15 years ago

looking at the spreadsheet, "shiny newness" seems to be "velocity"; it has the highest weight of all (twice the average).

i can only begin to imagine the conversations a year down the line:

- how did you decide this?

- we used a spreadsheet. science!

- so you weighted against the asynchronous approach that confused your developers earlier?

- actually, we put most weight on it being an exciting new technology.

- oh.

https://docs.google.com/spreadsheet/ccc?key=0AvBGESHWxhk2dHJ...

amix2• 15 years ago

I blame them and not the technology stack they used. It's idiotic to mix blocking technologies with non-blocking technologies as blocking the IO loop is fatal for performance as you won't be able to process any events while the IO loop is blocked.

And this person still has not learned from their mistake, they think that using SQLAlchemy would have solved their problem. If they wanted to make Twisted work they should have picked a database that can be used in a non-blocking way.

The reason why I like node.js is that everything is built in a non-blocking way. Twisted has the same philosophy until people think they can get away using a blocking library inside the IO loop.

dextorious7• 15 years ago

SO, basically you just learned about non-blocking and think node.js is a golden bullet. It's not.

And it's exactly the same situation as anything they could have used in Python, as soon as they add actual code to get work done (as opposed to hand over work and wait for a callback).

"""The reason why I like node.js is that everything is built in a non-blocking way. Twisted has the same philosophy until people think they can get away using a blocking library inside the IO loop."""

That's not what they were doing, though. At the end of the non-blocking fiesta you want to actually return results to a waiting connection -- actual results, rendered templates etc. That will block and take time -- especially in node.js which is single threaded.

So they just traded Python's threads/processes/lightweight threads for node.js processes.

taf2• 15 years ago

Single threaded, but small enough processes that multiple processes in node makes up for it. http://nodejs.org/docs/latest/api/child_processes.html

build a process pool and queue jobs to background processes, it's the same arch, you'd use if you had threads so what's your point?

amix2• 15 years ago

No, I did not learn about non-blocking approach recently. I have built solutions in Java and in node that scaled to over 300.000 open connections pocessing billions of messages. Besides that you are missing my point, which basically is to not mix blocking with non-blocking code.

clojurerocks12• 15 years ago

This is another reason why i moved off of django. This arrogant attitude of belittling other people when they have problems. I expect this from the rails community not from python. However django is increasingly becoming a "its the django way or the highway buddy". Which is dissapointing frankly.

clojurerocks12• 15 years ago

Nice how i get voted down. Thus proving my point about the python and django communities. So glad i moved to node. I will still consider using python for backend stuff though.

clojurerocks12• 15 years ago

After developing a v1.0 of a project with django i ended up also moving to node.js as well as mongodb as opposed to postgresql. I found myself increasingly wrestling with django. I really like python and the django community and think django is good for certain types of applications. But at least for what im building it wasnt the right solution. Im also using node.js for another project which is also currently built with django. It has different requirements and i might use python on the backend whereas the other project will most likely just be node.js and mongodb.

peregrine2• 15 years ago

I'm curious as to why Go wasn't a final contender? It started off with Go being a crowd favorite and ended with it nowhere. I assume it has to do with the young package repository but what's the reasoning you used?

pquerna3• 15 years ago

Go's core packages are good.

Go's not-in-core packages are a mixed bag. Lots of "one off" experiment projects. Which is fine, thats where it is in its life. Node.js was in the same place packages wise 18 months ago.

Additionally, at the time we were looking at this, Go was still doing releases every 2 weeks -- this was months before the Go 1.0 plan was even announced: http://blog.golang.org/2011/10/preview-of-go-version-1.html

peregrine2• 15 years ago

Thanks! I am playing around with Go and you are right the packages are a mixed bag. Not to mention using a search engine is basically impossible.

sneak6• 15 years ago

Python is a programming language. Node.js is not. What you meant to say is that you switched from Python (presumably running on top of a Real Web Server) to javascript running on Node.

The argument could be made that they are both interpreters, but please, get it right - you are writing javascript, which is a pretty shitty language in which to develop apps these days. The only reason we use it is because it has near-universal browser support - and then only for UIs.

Also, not to parrot, but it's true - node is cancer. It's for fad chasers who have no idea how real servers manage to serve volume.

Fortunately, most people inexperienced enough to choose node are inexperienced enough to never need to scale, so it'll work out okay for them - for implementation-specific values of 'okay'. :)

karterk• 15 years ago

> you are writing javascript, which is a pretty shitty language in which to develop apps these days.

Woah. I do both Python and JS, and guess what - there are a fair number of people who hate Python's significant whitespace. And to them, it's also a "hacky" language (with all kinds of __foo__ stuff). So, shitty-ness is in the eyes of the beholder.

> It's for fad chasers who have no idea how real servers manage to serve volume.

There are real world apps like Trello, which is on Node, and are scaling well. Get your facts right, or stop trolling.

hello_moto4• 15 years ago

I'm not going to argue too much between JS and Python.

But whitespace as debate topic over corner cases in JS language? really?

And also Trello is just launched and suddenly it scales off miles away? really?

malandrew7• 15 years ago

Yup, shitty-ness is in the eye of the beholder. I wish that every time a discussion of syntax came up, we'd be as quick to make reference to Wadler's law as we are to make reference to Godwin, when Nazi's are brought into the picture.

http://www.haskell.org/haskellwiki/Wadlers_Law

Arguing syntax is more often then not pretty absurd, unless you are arguing how the syntax actually changes semantics (like with S-expressions and overloaded parens).

StavrosK2• 15 years ago

What __foo__ stuff?

sneak6• 15 years ago

I think it's amusing that when I claim that javascript is a shitty language, instead of trying to claim that javascript isn't shitty, you point out the shittier attributes of the language I was comparing it to.

Regardless of whether you like python or not, javascript pretty much sucks. Coffeescript helps a bit, but if you aren't running in browser, why hobble yourself like that?

rdtsc2• 15 years ago

> fair number of people who hate Python's significant whitespace

if that is their only reason to claim JS is a better language, i wouldn't bother listening to their opinion too much.

jalanco2• 15 years ago

Wow. What are you so frightened of? It's obvious from your post that you have invested a great deal of time in python and you are terrified that Node is going to take away your livelihood. That is not going to happen. Python is here to stay, just as Javascript is. Both will continue to be improved over time, but there is no reason to freak out and lash out at other people. Relax.

And I'd like to see the results of your study where you have proven that "most people inexperienced enough to choose node are inexperienced enough to never need to scale". You should publish such a bombshell study.

But in truth, your fear is causing you to make nasty attacks and, well, just make shit up. Come on, just admit it, Sneak.

adolfojp• 15 years ago

"It's obvious from your post that you have invested a great deal of time in python and you are terrified that Node is going to take away your livelihood."

No it's not. You're just making one hell of an assumption.

sneak6• 15 years ago

Were it that Python was my livelihood my life would be a lot simpler. Instead, my livelihood comes from unfucking bad decisions like "hey, let's use PHP!" or "hey, let's rewrite the whole thing in javascript and use node!" or "MongoDB seems great, let's switch from MySQL!"

It's not fear of new, it's fear of broken. I encourage you to use whatever shiny stack-of-the-month you enjoy, as it basically amounts to my competitive advantage over people like that. I'll be over here using Apache and nginx and MySQL and python and perl and other languages that don't make me write abominations like '==!', same way I have forever. Best of luck to you.

davidw2• 15 years ago

> It's for fad chasers who have no idea how real servers manage to serve volume.

I think that pquerna has some pretty good ideas about how "real servers" work, but leaving that be, you could actually explain what's wrong with the system, rather than attack the people.

Having some Erlang experience myself, I can find plenty not to like about node.js. However, the Erlang guys have fired round after round into their own feet to the point where even if it's inferior, node.js has probably already surpassed Erlang for many server tasks.

therockhead• 15 years ago

Would you mind explaining "the Erlang guys have fired round after round into their own feet"?

simonw• 15 years ago

"It's for fad chasers who have no idea how real servers manage to serve volume"

If you look at who's building serious systems with Node.js, that statement doesn't hold up. CloudKick / the Rackspace monitoring team are hardly amateurs - neither are Joyent, LinkedIn, eBay or Yahoo.

FraaJad2• 15 years ago

Yes. Those teams have sharp people who can probably write good software using COBOL if they had to.

Also see facebook and PHP. Just because Facebook uses PHP, it doesn't mean that PHP is a good language.

hello_moto4• 15 years ago

If they believe in Node.JS, good for them. I assume they have the right people to write good and modular javascript code.

If you're a good front-end JS developers but never ever wrote a minimum reusable JS library, you've got a lot to learn before diving to JS.

As long as the code is waaay more modular than this: https://github.com/marijnh/CodeMirror2/blob/master/lib/codem...

They should be alright in terms of adding new features. And if Node.JS hits scalability issue? well tough luck I suppose.

I see that JVM is a very serious contender but as already noted: License issue. And let me add this: I'm betting my ass that there's the "I don't want to write Java" whisperers as well. Now now, don't BS me. I know how the so-called "engineers" think of Java these days. Sure, you could use Clojure, Scala, or JRuby. But seeing the competitor is Node.JS, pretty easy choice don't you think?

My experience writing JS code for Node.JS based platform (ExpressJS, etc) is that there's a bigger chance that you'll write more code. Code to make the code more modular (sounds weird isn't it?). Code to make sure you work-around the warts of JS. Each line of code should be highly scrutinized due to JS warts.

This thread is going to be a typical fun nerds-fight. I guarantee you. I'm going to grab a popcorn and watch nerds doing keyboard slamming.

wulczer• 15 years ago

Sounds like a weird reason to switch the language and framework used for your product. Blocking libraries block just the same in Node and in Twisted. Why not just stop using the Django ORM or more generally, stop writing code that blocks the main thread for long periods of time?

The problem of the main thread waiting on a response from the database is easily solved without rewriting all your code, which incidentally will have the same issues, since there's no new magic there.

If your software is a giant, ugly hairball because you wrote it in a hurry (I know I did this many times!) don't blame it on the language or framework. What I'm wondering is: Python/Twisted/Django rank equally or similarly on all the points from the list, except for Team Experience, which has to be much higher with the old stack than with a completely new technology and set of tools. This makes me think that there was a "play with shiny things" point after all.

andrewvc• 15 years ago

Well, I agree and I don't agree.

I'm personally not a huge fan of node, and I prefer to write non-blocking code in Aleph/Netty (which is closer to twisted in terms of potential to block due to blocking java libs) myself.

But, I can understand on a large team not wanting to deal with the potential for blocking libs.

That said, if performance is that important, it should be part of your test suite, and should catch that.

rictic• 15 years ago

It's been a while since I've written nodejs in anger so I may be out of date, but as a rule libraries in Node don't block, they present async interfaces and do what they have to in order to keep from blocking.

Python has no such conventions, making many libraries into dangerous minefields for the Twisted developer.

kyledrake• 15 years ago

If you want to use a reactor pattern and you don’t want to use JavaScript, you can use EventMachine on Ruby. Twisted is the same thing for Python. I’m sorry to hear you were having problems with Twisted, but EventMachine is absolutely rock solid. I have gotten a lot of work done with it.

Now, let’s say you don’t want to do callbacks, but still take advantage of the Reactor pattern in EM. I wrote a patch for Sinatra that uses Fibers to wrap callbacks, hence allowing you to continue to program synchronously as you used to, and it’s called Sinatra Synchrony (http://kyledrake.net/sinatra-synchrony).

But in order to do non-blocking IO, you don’t need to use a Reactor pattern, because Ruby internally does not block on IO (the GIL does not affect this). And of course you get solutions like JRuby (built on the JVM), which provide for threads (and Rubinius 2 coming soon).

My APIs written with any and all of these concurrency options can get thousands of hits per second. It’s quite scalable, but still provides all the rich libraries, reusable code, and testing support that makes my APIs high quality, which to me (and my users) is more important than making them fast. With this approach, my APIs are both high quality and fast. And I have never experienced a single crash of any worker processes in production. I would know, because my process monitor is smart enough to observe the workers in my deploy code and informs me, while adding another worker to replace the fallen one. Did I mention I can do zero-downtime hot deploys? Again, all implemented in Ruby.

I’m not trying to brag here, but there’s this weird current of arguments that these existing solutions don’t scale. It’s a myth, I don’t know where it comes from, and I’m worried that if I don’t speak out against these arguments, I’m going to wake up in a giant pile of half-baked Javascript spaghetti written by people that didn’t even understand the real problem in the first place (blocking IO people, research it!).

The switch: Python to Node.js

Related Stories