Python at Netflix

http://techblog.netflix.com/2013/03/python-at-netflix.html

by Lightning • 13 years ago

232 74 13 years ago

techblog.netflix.com

jedberg12• 13 years ago

As one of the main drivers of Python at Netflix, this one is near and dear to my heart. Let me know if you have any questions, or come visit our booth!

arpineh• 13 years ago

Hi.

There was many interesting things listed at the blog post. I have been planning a system similar in idea to Sting, your OLAP server (?).

I'm planning a light RPC style system on top of Pandas dataframe objects, transferring HTTP requests to Pandas method calls. Memory usage of such a system with our data is too expensive if all data is in use. So, I'm investigating possible caching and database solutions.

Could you perhaps elaborate on the implementation of Sting's in memory database?

[posted from yet another HN account]

jedberg12• 13 years ago

I just posted this above: "I don't have any info about Sting, but we're working on some blog posts that go deeper into each of these topics. I'll make sure that the interest in Sting is accounted for in the prioritization."

So now that's +2. :)

pwang6• 13 years ago

Have you looked at the Cubes project?

http://databrewery.org/cubes.html

yellow• 13 years ago

I'm pushing for more python use at my company as well. It's tough as everyone is drinking the ruby kool-aid.

P.S. I really enjoyed your talk at QCon SF last year!

irahul3• 13 years ago

> It's tough as everyone is drinking the ruby kool-aid.

Ruby and Python are similar in terms of execution speed and developer productivity. If your work requires something that Python has and Ruby doesn't(say numpy), then it might make sense; otherwise what would you accomplish by switching from Ruby to Python?

e1ven• 13 years ago

Keep it up!

Posts like these are great on a tech level, but they're also good recruitment - Seeing Python, along with hackers that I respect (like you) at Netflix make me at least the littlest bit tempted to go there ;)

jedberg12• 13 years ago

You've caught on to our ploy!

But in all seriousness, we like to share what we do because sharing is cool and it attracts other people who like to share cool technology, which are the kind of people we like to work with.

pwang6• 13 years ago

You should also come by our booth (Continuum Analytics) and we can talk shop!

DoggettCK• 13 years ago

We were actually talking about your company in our sprint planning meeting a few minutes ago, since we're doing in-house Python analytics. You might be getting a call.

bsimpson3• 13 years ago

You know how Google claims that by saving fractions of a second for every search that billions of people do, they've saved many man-lives of productivity? As a leading engineer at both Reddit and Netflix, you've undone all of that. Well done.

qxf22• 13 years ago

Unfair tone. I hope it was in jest. My life is not all about productivity. There is a leisure aspect as well:

http://en.chessbase.com/home/TabId/211/PostId/4008728

jedberg12• 13 years ago

I always tell people that I've spent the last six years of my life as a digital crack dealer.

BrianEatWorld• 13 years ago

Where can I find more info about Sting? I didn't see a link in the article and some quick Google searches haven't proved fruitful.

jedberg12• 13 years ago

I don't have any info about Sting, but we're working on some blog posts that go deeper into each of these topics. I'll make sure that the interest in Sting is accounted for in the prioritization.

pwang6• 13 years ago

Kurt Brown had some slides about data & Python at the NYC Python meetup last October. He did a Sting demo there and some of the slides talk about it (I think... it might be a different thing, though. October was a long time ago..)

http://www.meetup.com/nycpython/files/

tootie2• 13 years ago

I have to say that after doing a python project I totally don't get why people love it so much. The syntax is slightly cleaner, but the lack of threading, first-class functions and dependency management tools make it waste at least as much time as it saves. I honestly don't believe in the concept of a programming language being able to make code more or less readable. A Java project with a POM file can be up and running on any OS in about 5 minutes.

brown9-2• 13 years ago

Python doesn't have first class functions?

nightpool• 13 years ago

a) python-pip combined with virtualenv gives very good requirement support.

b) python does have first class functions?

c) threading still happens, you just can't take advantage of multiple cores. This isn't as big of a problem as you'd think though, you can still write very fast code.

frankwiles• 13 years ago

Ummm, not sure where you're getting your information but Python has threads (http://docs.python.org/2/library/threading.html), first-class functions (http://en.wikipedia.org/wiki/First-class_function), and dependency management tools (google for pip, easy_install, etc.)

mixmastamyk4• 13 years ago

It has its share of warts, but doesn't lack any of those things you mention.

Java shines at the huge end, but for most everything else python is a more productive choice as it is so much more concise. Perhaps you are not yet using it optimally?

bryanwb• 13 years ago

would you be interested to talk about these tools on the http://foodfightshow.org devops podcast? we would love to discuss them with you!

jedberg12• 13 years ago

I would, but I could probably find you people who know more about the tools, like the folks who wrote them. :)

Hit me up on email (it's in my profile).

cpenner461• 13 years ago

Hey jedberg - any plans/timeline for open sourcing aminator? It sounds pretty cool...

voidlogic2• 13 years ago

As a Java/Python shop I think Netflix should evaluate Go after 1.1 RC comes out in April. Go is very popular with people who have heavily used both Java and Python (as well as C).

robertfw• 13 years ago

Just curious, why would this be downvoted?

Afforess• 13 years ago

If I had to guess, I would say because GP didn't actually give any real reasons why adding complexity (a new language) would improve or solve a problem. Advocating languages is fine, we all have our pet languages, but it is important to remember they each are just one tool in the toolbox and that what is more important is how the language solves the problem well, instead of just blindly advocating one language over another.

ketralnis• 13 years ago

Because it's not helpful to anyone. It just says "they should use my favourite toy instead of their favourite toy". No details about why or how it would help anyone anywhere ever. Just "Someone else should do a lot of work. Because I said so."

jedberg12• 13 years ago

I believe we do have one or two people here looking at Go.

Given our culture, there really is nothing stopping anyone from using it, other than the fact that they would have to figure out how to interact with the platform.

rdtsc2• 13 years ago

Anyone used or uses Erlang?

elq2• 13 years ago

I've used go at netflix before to hack up a quick load test, but recently the vast majority of my non java/c++ code at netflix has been python for writing glue to interact with our ops infrastructure and julia, which recently supplanted python/pandas/scikit, for my my research oriented stuff.

But overall go is a really neat "systems" level language.

netcraft4• 13 years ago

Re: Web applications in python - I am curious about the overall feeling for them - Do you write web apps in python because rapid development makes up for any performance problems and if you run into problems you would rewrite it in something faster, or do you feel like a python web app can be just as good as alternatives when it comes to speed?

Put another way - when you think of python for web applications / services - do you think rapid development and speed, or just rapid development and you would go somewhere else for speed?

irahul3• 13 years ago

I don't think anybody equates Python with execution speed. Writing web applications in Python facilitates rapid development, and since most of the web applications are io bound, slow execution speed doesn't make noticeable difference.

netcraft4• 13 years ago

So when you think of execution speed in web applications / services - what do you think of?

jedberg12• 13 years ago

I like Python for the speed of development. Also, at least for these internal apps, speed is generally of secondary concern, since these aren't typically high load applications.

At reddit we solved the problem by writing the most often called parts in C and using c extensions. There is definitely more work to do there, but overall I think Python gets a bad rap as far as speed is concerned.

It's certainly not the quickest, but it isn't a dog either.

netcraft4• 13 years ago

Thanks for the information!

daeken• 13 years ago

> At reddit we solved the problem by writing the most often called parts in C and using c extensions.

I'm wondering if anyone has solved these sorts of problems with Cython in the real world. It looks like a great solution for a large set of problems in terms of Python performance, since it becomes a CPython extension, but I never hear anyone talk about it. Can anyone chime in?

mixmastamyk4• 13 years ago

I'm thinking pure speed is not a big differentiator these days... as long as it isn't horrible.

Why? The apps are mostly IO bound, and any large site is going to be caching, deferring slow operations, and sharding/breaking into tiers, cdns, etc, etc. The amount of time a web framework actually runs is relatively small.

Others have already mentioned what to do when performance is a problem.

jkldotio2• 13 years ago

Python really does hit a sweet-spot for many types of operations. Devops capacity in system scripting, crawling and scraping with excellent web frameworks (Django through to Bottle). On top of all that there are great machine learning libraries as well as good APIs to almost anything not written in Python. My site gets huge amounts of NLP, machine learning and information extraction done in just a few lines because of libraries available in python that aren't in the languages that it's usually compared to.

I've also pretty much replaced bash with the ipython shell at this point, as you can use all of the bash commands anyway and even seamlessly mix them with python, file_list = !ls, it's a excellent environment for data analysis and manipulation.

jedberg12• 13 years ago

You know, it never occurred to me to use iPython as my shell. I think that would make me more productive. I'll have to try that, thanks!

jkldotio2• 13 years ago

I have 'ipython --profile pysh' in my terminal's custom command, on servers I'm a bit more cautious and have a 'pu' (power up, or python up) command which is basically the first thing I type. You have to watch out for the shell history not working quite the same and that newly installed programs won't be seen until you do a '%rehashx'. Other than that it's pretty much all gravy. Now I just need to get around to transitioning to that python tiling window manager.

While I've got you, and seeing you are not longer at Reddit, what do you think of jkl.io (launched today)?

pwang6• 13 years ago

I think someone asked the question at a Scipy conference a few years back, "Who here uses IPython as their primary shell", and IIRC about 30% of the people raised their hands.

yen223• 13 years ago

Someone once said that Python is the second-best language for everything. I found that to be true.

mixmastamyk4• 13 years ago

Any remote jobs or must you live near Los Gatos?

jedberg12• 13 years ago

Generally we want people to be in Los Gatos, but there are exceptions, mostly in the CDN team.

donutdan41142• 13 years ago

Why does Netflix only allow 4-10 character passwords? My weak password is 14 characters. There is no reason to have a length limit.

belandrew2• 13 years ago

My Netflix password is 20 characters. Where did you get that it only allows 4-10?

donutdan41142• 13 years ago

Trying to update my password: http://i.imgur.com/wLNpR0C.png

Python at Netflix

Related Stories