Web Service Efficiency at Instagram with Python

https://engineering.instagram.com/web-service-efficiency-at-instagram-with-python-4976d078e366#.s1st93t8g

by ksashikumar • 10 years ago

195 46 10 years ago

engineering.instagram.com

pm90• 10 years ago

Wish they would share some of that profiling code...

fovc• 10 years ago

I wonder if they looked into using existing APM solutions before building out the profiler and monitor

lifeisstillgood3• 10 years ago

I am digging into a gnarly bug in thousands of lines in python 2.6 right now and, well, a custom cProfile, C types support and auto-reporting sounds ... Lovely :-)

But I know that they will have their rough edge cases and developers banging their heads on the desk making this custom code for their needs.

But the grass does seem greener today :-)

lifeisstillgood3• 10 years ago

I think on reflection however we need to remember, with all our sprints and agile iterations, that just in Instagram billions of web requests are being served by just dozens of developers based on code written over 25 years now, following good practises and hard work - kudos to the Python Dev team on 25 years.

dalke• 10 years ago

To be fair, that's more like "started 25 years ago". Very little of the original Python 1.0 code exists (especially given the great renaming when the "Py"/"PY"-prefix occurred.)

I like to think of the thousands of developers involved. ;)

For example, assuming Instagram uses gcc to build Python or any of the other components, gcc started in 1987.

lifeisstillgood3• 10 years ago

Of course ...

sorenjan• 10 years ago

If engineering time wasn't an issue, how much would a company of Instagram's scale save by switching from Python to a lower level language like C++? Pretend you could wave a magic wand and overnight have the entire code base transformed from Python to well written C++, how would that affect the hardware requirements, and future development?

I guess what I'm really asking is what pros and cons are there with using dynamic and relatively slow languages compared to faster languages like C++, Java, or C#? My uninformed reaction is that it's easy for startups to get started with these easy languages but that it leads to problems when it's time to scale up. Dropbox is writing their own Python implementation, and Facebook have their PHP to C++ compiler.

nemothekid• 10 years ago

how much would a company of Instagram's scale switching from Python to a lower level language like C++

I guess one thing you have to consider is

1. How many engineers do you have to hire to make Python "fast enough"

2. What does it cost to deploy a full C++ team? Is it easier for new programmers to learn & write safe C++ code vs. Python.

I wonder what is cheaper?

SEJeff• 10 years ago

    s/C\+\+/golang/g

kirkdouglas2• 10 years ago

From the users perspective they're doing fine (haven't had any problems with Instagram performance).

draw_down• 10 years ago

Major con: Good luck finding people who want to write a web app in C++.

weberc2• 10 years ago

This is probably an extension of the "if developer time wasn't an issue"; at least that's a major factor for me (not being able to develop quickly).

Scramblejams• 10 years ago

It's important to remember that for a lot of network server purposes, your code only has to be fast enough to saturate your network interface, and after that there isn't much point in trying to speed things up.

Not sure how often that's the case in Instagram's use, but I've been using Python and Twisted to saturate network interfaces effectively since 2003 :-), and in those situations I've been more interested in the correctness advantages of a good statically-typed language rather than the performance.

abiox• 10 years ago

time-to-saturation is also worth considering.

nine_k• 10 years ago

С++ is not exactly a comfortable language. While recent changes may be great, and there are quite nice libraries, the language still does little to prevent you from shooting yourself in the foot (and sometimes downright provokes it).

If I were about high speed, I'd take Java (which regularly rivals, and sometimes surpasses, C++ in speed), or Rust, or maybe Go. Other worthy contenders would be Haskell, OCaml, D, maybe even Elixir (unless they are CPU-heavy), though with these hiring becomes much harder than with Python or Java (and probably even C++).

pbreit2• 10 years ago

Developer productivity is crucial to figuring out a business and then in the rare event that you actual get product/market fit, scaling is trivial (ie, no one ever failed to scale, not even Friendster).

sametmax2• 10 years ago

Even when engineering time is not an issue, Python stays. Look at dropbox: they convert a part of they code to rust for performance reasons. Yet they have no plan to replace the whole Python code base, because Python is very convenient to use for no performance critical part of the app that are affected by it's dynamic nature. And they are not that numerous. In the end, it's not just time, it's also a matter of confort and even pleasure.

acd• 10 years ago

Is Pypy used?

http://speed.pypy.org/

piquadrat3• 10 years ago

Instagram is a great story for Django, it's the best argument to bring against doubters of Django's scalability. But I wonder how much Instagram contributes to Django itself. I follow Django's development relatively closely, and I'm not aware of any major work contributed by Instagram engineers. They aren't listed on Django's fundraising page, neither.

https://www.djangoproject.com/fundraising/

kirkdouglas2• 10 years ago

As far as I know, Disqus (https://disqus.com/) and Eventbrite (https://www.eventbrite.com/) are also major users of Django.

piquadrat3• 10 years ago

Some of the most prolific Django developers work at Eventbrite. I can't say how much, if any, of their contributions to the Django ecosystem are on Eventbrite's dime, though.

fefifofu• 10 years ago

The fundraising page is depressing. They're only asking for US$200,000 and they only have 19% at half way through the year. In other news... Instagram hits half a billion monthly active users... but not on Django's fundraising page.

I'm not knowledgeable about open-source issues, pros/cons, etc., but man... only 19% of $200K. Doesn't seem right given Python is one of the major languages and I believe Django is the most popular web framework in Python.

J0-onas• 10 years ago

You'd think that companies who profit from open source projects would support them financially. Considering how big django is there should be many other companies who profit from the projekt.

200k/year should be collected in no time...

sametmax2• 10 years ago

That's the state of the entire Python community. To get money, you need to get hype, and Python is anti-hype. We are people prioritizing on robustness, clarity, documentation, etc. Proven useful things. But because it disregard what we consider superficial and frivolous, we cut ourself from the same enthousiam and energy you can see in other community like JS or Ruby. And from money.

Look at the PSF budget, it's so small it makes you cry.

tomchristie• 10 years ago

My thoughts on this:

* Let's stop talking about 'Donations' and start talking about 'Investment' - the conversation needs to be about the pay-off that businesses will get, not some altruistic "give-back"

* Make the case that collaborative investment has a great cost-reward, as the cost is spread over a bunch of companies.

* Make it clear that ecosystems that are well invested will be more competitive that those that are not. If your company has $$$ worth of engineers experienced in Django, you'd be daft not to ensure that Django remains competitive vs other frameworks.

It's an atypical pitch to make, and as with all investments it's difficult for folks to judge, because the pay-off is at a distance. I still believe it's still possible for us to change how collaboratively funded OSS is seen, once companies start to recognize well-invested ecosystems as a competitive advantage.

And yeah, this is pretty much the pitch I'm making for funding Django REST framework, where I'm currently 1/3 of the way towards being able to focus on it 100% of the time, with a full-time salary... https://fund.django-rest-framework.org/topics/funding/

munro2• 10 years ago

From the perspective of scraping their website, it was React & GraphQL that I was interfacing with. So what does Instagram use Django for? Perhaps that's why their contributions to the project are low.

piquadrat3• 10 years ago

From Instagram's medium post

Instagram currently features the world’s largest deployment of the Django web framework

I think it is fair to assume that they use Django pretty heavily from that.

meteorfox• 10 years ago

This part I'm skeptical:

  "Compared to CPU time, CPU instruction is a better metric,
  as it reports the same numbers regardless of CPU models
  and CPU loads for the same request."

CPU instructions will be more stable than CPU time for sure, and it does show that their metric is stable based on their chart but a single CPU instruction can take multiple cycles, especially if there are stalls in the pipeline or other processes are "polluting" the cache. Depending on the CPU model, the number of uops that can be issued concurrently varies, and the latency of these instructions will also vary on the size, and access pattern to their memory hierarchy.

Also, what about a change in compiler version? That can also vary the number of instructions. Unless they are referring to a Python bytecode instruction as a CPU instruction.

Would measuring CPI be a better indicative of their efficiency? They could also track both, no need to settle for one.

zerohp• 10 years ago

I agree with your skepticism. They will not detect a performance regression when there is a change in instruction mix or cache miss ratio.

wheaties• 10 years ago

Wow, CPU is the limiting factor? I've never been CPU bound in a Web app, only I/O bound. Am I doing something wrong?

Granted, I don't use python anymore for Web dev. I'm using languages that can handle 10x the traffic per box that Django used to be able to handle. This isn't a knick against python. I love python. Just such a different perspective.

hardwaresofton• 10 years ago

This also immediately popped out to me as the wrong assumption to make (and start off with, for that matter)... But I could see it being true if:

- You are essentially maxing out incoming and outgoing network IO

- You have no database to speak to (or your DB is in-memory)

In that case, I guess it is just a race to do some computation and fetch the right data from RAM, then the limiting factors would be RAM access and the number of the computations you could do to try and keep up with inbound requests?

dchuk• 10 years ago

What languages/frameworks are you using now?

pbreit2• 10 years ago

I'm guessing that you're working on something that is fractionally as successful as Instagram. Getting to the point where you need to scale is hard, scaling is easy.

instakill• 10 years ago

If any Instagram people are reading this, please for the love of all that is sane, please I beg you add some kind of progress indicator for your API.

3 weeks in, I find myself loading my app's API client page to see if it's still stuck in a greyed out "go live" mode.

I've got an "Instagram API" search saved for twitter and StackOverflow to check checking if anyone else gets to slip through but others struggle just as much.

Just a little transparency and predictability is all that we ask for.

spullara• 10 years ago

They should really show what they are doing with their CPU. How many requests per second and their characteristics? Generally Python (Ruby, etc) is terribly inefficient relative to the next tier of languages like Java and Go. At Twitter we were able to reduce number of servers by 10x and decrease latency by 10x by moving the web application from Ruby to Scala/Java with no architectural changes, essentially byte for byte (and bug for bug) compatible. My guess is they have done this math and would rather pay for more servers and take the latency hit rather than rewrite it.

deno• 10 years ago

Did you at the time compare it with a JIT implementation like PyPy (for Python, not sure what would be the Ruby equivalent)?

pritambaral• 10 years ago

A JITing implementation of Ruby would by JRuby. Depending on JRuby's state at that time, Twitter could have used JRuby and gradually rewritten parts of it in Scala. Perhaps.

Web Service Efficiency at Instagram with Python

Related Stories