Why Is Python Growing So Quickly?

https://stackoverflow.blog/2017/09/14/python-growing-quickly/

by lainon • 9 years ago

254 267 9 years ago

Why Is Python Growing So Quickly?

Really easy to answer: It is stupid fun to program with Python. There are libraries for everything you can imagine and they are generally very easy to use. Once you grok the virtual environment thing, writing apps/programs/scripts is just a matter of creating a new env and installing the libs you need. Testing ideas with Jupyter notebook is fun, fast and rewarding. Pycharm is awesome. VSCode with Python just works. You can automate tons of boring stuff (Thanks Al!)... and the list goes on... and on...

pageandrew• 9 years ago

for whatever reason, the word "grok" annoys me to no end

castis• 9 years ago

In other words, you don't grok "grok"?

mark-r4• 9 years ago

The origin of the term is fascinating: https://en.wikipedia.org/wiki/Grok

antod2• 9 years ago

Heh. There was a python web framework called grok.

Zarath• 9 years ago

For me it's the fact that no English words end in "ok".

drcongo• 9 years ago

I thought it was just me.

pvg• 9 years ago

I think a big part of the answer for SO is the popularity and inscrutable nature of pandas. 'Popular, useful and incomprehensible' make for excellent Stack Overflow growth.

newen4• 9 years ago

Got to agree. I used to google pandas problems for remotely complex stuff because it wasn't really intuitive how to do it in pandas. Now I have a better context for how simple such things can be because of data.table in R.

rpazyaquian3• 9 years ago

You're listing off tooling, libraries, and frameworks written with or targeting Python, which isn't Python itself. As soon as you want to go off the rails in any of those things, Python gets as complicated as any other language.

My take on the matter is that Python is gaining traction due to its market share of the Data Science/Machine Learning field, which is what's really gaining traction and growing quickly. Had Perl or Ruby been the premier language to use for DS/ML, they would be growing instead of Python.

schneems2• 9 years ago

This. Most schools are moving away from C or Java being their primary language. Most comp sci professors acknowledge the usefulness of a scripting language and so when they need to pick one, they pick one they have the most familiarity with. Due to numpy and scipy this tends to be Python over Ruby.

Also many things about python that I don't like such as `len` being a function rather than a method on a list are very trivial when coming from a language like C which has hordes of tiny inconsistencies.

The whitespace thing in python is pretty gimmicky but does also hold a certain amount of pedantic appeal.

empath75• 9 years ago

You still have to explain why python is the premier language for that.

andrewflnr• 9 years ago

Come on, we all know a language and its library ecosystem go hand in hand. Trying to separate them from the standpoint of adoption is pointless. TFA comes to its conclusion by looking at libraries. In your hypothetical where Perl or Ruby take Python's place, that would manifest as Perl or Ruby having better data science libraries. The causality would probably be just as cyclical as it is for any "best language/library for task X" question.

msla2• 9 years ago

The reason C is so popular is because it's the scripting language of Unix. C++ is the scripting language of Windows. Objective-C, Macs and iOS.

Well, Python is the scripting language of a ton of useful libraries which are relevant to a lot of very interesting fields.

R sits in a similar spot, but Python is a better, friendlier language, which means actual programmers are more apt to recommend it as an offhand solution to a simple problem someone who isn't seen as a programmer is having: "Oh, that's easy. Just copy and paste this Python code, and maybe hack at it until it does something." You can get scientists programming with recommendations like that.

kbenson• 9 years ago

Had Perl or Ruby been the premier language to use for DS/ML, they would be growing instead of Python.

Well Perl was the premiere data science language for a while. Python made inroads and eventually took over. I think you're mostly right through, it's just a matter of other factors affecting the communities at the time causing Perl to not grow nearly as quick as Python, and Python getting a good numeric library. Had Perl not lost a lot of community in the early 2000's, it might have retained this area.

dekhn• 9 years ago

I'd be using lua if it had python's library breadth. Or maybe julia. Not python.

guelo• 9 years ago

Are you suggesting that we ignore the article's data-driven conclusion that it is because of pandas and the rise of data science and machine learning?

rpazyaquian3• 9 years ago

That is exactly what I am thinking myself.

fish_fan2• 9 years ago

I think you're reading the article wrong—pandas is a symptom, not a cause.

That said, it seems foolish to ignore the clear data science use case. But that's also been a theme for at least five years now.

njharman2• 9 years ago

No they're explaining why Pandas was written in Python and why data science and machine learning projects choose Python over R or Java or Haskell, etc.

CJefferson2• 9 years ago

I have come to the conclusion that part of the reason python is so fun is because it is slow.

There are so many lovely and easy to use libraries in Python. I took a couple and tried to be inspired by them to write a C++ library, maybe for addition to boost.

However, if you are writing C++, people expect your library to be fast and memory efficient -- you can't go around sticking the entirety of a parsed file in a std::string. Also, people then start saying your code better interoperate with their templated classes well, and they want to replace the memory management entirely, and it goes on...

Of course, the resulting libraries are very fast and very clever, but they have lost the joy. They also take much longer to write, maintain and use.

paulddraper• 9 years ago

It's a funny way of phrasing it, but you're not wrong.

C++ libraries should allow people to squeeze every bit of performance juice out of them (e.g. custom allocator).

Python libraries should be intuitive to use.

---

Naturally, there are exceptions, like numpy.

harshgupta• 9 years ago

Part of the reason is same side network effect. It makes sense to create new libraries in languages having great existing libraries.

barrkel• 9 years ago

Watch out for conflating abstraction level with speed. Coding at a low level of abstraction means sweating the small stuff, but it's easy to fool yourself into thinking you're making things fast - when actually, you're micro-managing.

Painting with broader brushstrokes with less concern for micro-efficiencies lets you block out a solution faster. Another part of the problem with C++ is that the components come with so many rough edges that need to be filed down that you can't help getting swamped in a certain level of detail, unless you start with a large toolbox or framework that lets you work back at a higher level again.

You ought to have the equivalent of Java's file stream, buffer and text decoder as three separate bits that you can compose, so that putting a whole file in a string is never even a temptation. But if you start out with a sparse toolbox, it's a slog.

xapata14• 9 years ago

One of my clients recently replaced a major chunk of their C++ code with Python and achieved a speedup from multi-hour to sub-millisecond. That's a big win.

Python encourages readable code, which enables smarter algorithms, which ultimately swamps all those little memory optimizations. Oh, and PyPy/Numba/Cython let you get those little optimizations, too.

shalabhc2• 9 years ago

python is so fun is because it is slow

C++ pushes you to conflate optimization with the higher level meaning of what you're trying to implement. In Python you can write the higher level meaning in a more 'pure' way, which is fun of course. The really interesting question is if you can program at the higher level while still running fast, perhaps by specifying optimizations separately?

(BTW, for Python code that can be as fast at C++ see https://kratzert.github.io/2017/09/12/introduction-to-the-nu... - it doesn't sacrifice clarity, but is limited to a subset, for now).

fsloth2• 9 years ago

"you can't go around sticking the entirety of a parsed file in a std::string"

Sure you can. C++ is at best programmed like python. Just throw the working first version together - and once you hit a performance gap, you can optimize it.

Using a profiler.

Premature optimization is mostly pointless because for most non-trivial performance critical apps the bottleneck wont be where you expect it.

The faster you find the actual problematic hotspots with production content the better. The way to get there fast is to treat C++ like filthy python.

Don't trust anyone trying to goad you into premature optimization unlesa they understand the specific problem you are trying to solve, and only then with skepticism.

drewrv• 9 years ago

Part of it is just that Python has an awesome, supportive community. Sounds like the C++ community is kinda critical and demanding, in your experience.

RivieraKid• 9 years ago

Julia is fast and more fun than Python.

tw10104• 9 years ago

That description doesn't explain why there has been such a sudden growth in recent years. Python is an old language. The real reason it is growing so much just recently is of course due to ML.

dwaltrip3• 9 years ago

The real reason it is growing so much just recently is of course due to ML.

This is certainly a potential explanation. But I don't think it is self-evident that the trend of recent growth is entirely due to this.

Edit: The article supports the claim - I should have read it before commenting.

jdc0589• 9 years ago

Once you grok the virtual environment thing

it's incredible how hard it is to get some people on board with this. I've gotten used to good project-local package management in other langauges, I can't imaging working with anything that depended on stuff happening to be available in a global namespace ever again.

whipoodle2• 9 years ago

It is a bit more of a pain to work with than similar features in other languages. Like as far as I know you can't just move a venv, you have to rebuild it in the new place. Whereas `node_modules` is just a dir. Also, there is a weird layering where venv abstracts away installed eggs/wheels/whatever they're calling them these days, but then since that's still painful to work with, you need another layer on top like virtualenvtools or burrito or whatever it is. (Sorry, it's been a couple years so my memory is fuzzy)

wooly_bully• 9 years ago

Have them try Pipenv!

https://github.com/kennethreitz/pipenv

While virtualenv management isn't that difficult, Pipenv is VERY intuitive.

gilrain• 9 years ago

As someone who has largely moved from Python to Node: it's not the concept of project-local packages that is the problem. It's virtualenv's awful implementation and UI.

dom05• 9 years ago

There are still widely used packages that don't work with pip, like PyQt.

gdulli• 9 years ago

It is stupid fun to program with Python

Around 2009 I started questioning if I even enjoyed coding anymore. I was primarily using PHP and Java. Rather than quit the profession I tried Python and suddenly it was fun again. 8 years later, it still is. Python just spoke my language.

My career is based around using Python because it's the way I enjoy the work. Everything else, like the actual properties of Python relative to other languages, is irrelevant to me if using another language would make the work unenjoyable again. (Though if Python didn't measure up very well compared to other languages it wouldn't have become my go-to in the first place.)

cutler5• 9 years ago

Try Clojure and I promise you will have even more fun.

alvivar• 9 years ago

Maybe you should try F#, to me both are kind of fun compatible (plus the functional focus that is kind of fun by itself).

Daishiman2• 9 years ago

Same thing here. I worked with PHP and Java apps and it was just pain having to deal with the awkwardness of PHP's designs and Java's boilerplate. I actually went into sysadmin roles because of that. Python gave me hope that there's fun development roles to be had.

Nowadays the panorama for good languages, thankfully, is much better. PHP is no longer that much of a dumpster fire with modern frameworks, and Java has decided to adopt better defaults and drop the XML madness. Not to mention all the good languages that have come up (Elixir, Clojure, Rust, etc).

blauditore3• 9 years ago

I recently worked with Python for a couple of months, for the first time. Even though reading code and writing simple parts is fairly easy, I found several details to be quite confusing, like variable scope, or the various underscore conventions/variables. Also, I struggled to find a good free IDE (there was no money for PyCharm), which made debugging a hassle. Also, dependency management with pip feels a bit quirky compared to e.g. Maven or npm.

All in all, I found it neither much more or less fun to program with than e.g. Java, C# or TS. It might be a better fit for small projects than others, though.

wslh• 9 years ago

IDE? Visual Studio Code.

Python is not perfect but the final code is simple to write and understand years later.

pletnes3• 9 years ago

Pycharm and visual studio have free editions. Vim, emacs, Atom, VS Code, sublime - there are many options.

NoahTheDuke• 9 years ago

variable scope

Which parts did you find confusing? Was it the lack of a block-level scope, such as within for loops and if statements? I personally find the LEGB (Local, Enclosing, Global, Builtin) system to be pretty intuitive.

[0] https://stackoverflow.com/a/23471004/3023252

chrshawkes2• 9 years ago

Free excellent Python IDE is Visual Studio Code with a Python Extension installed by Don Jayamayne. VS Code is a solid Python experience.

walterstucco2• 9 years ago

The real answer is: data scientist everywhere

They usually don't know better

wwarner• 9 years ago

zing!

copperx• 9 years ago

What would they be using if they knew better? R?

vturner2• 9 years ago

This is an honest question, what's the size of projects you have written in Python? For me, once a project gets beyond some size, the aggravation only grows of debugging old code and adding new. Mostly, I see this is as a lack of compilation and types in Python. How do others overcome this and enjoy large Python projects?

andybak4• 9 years ago

OK. Bear with me here. This might be overstating my case...

In some sense - there are no large projects. Only projects that have failed at being modular.

Maintainability involves keeping clear boundaries between functional units so that you can keep enough code in your head at any one time to reason about it.

If we assume that Python is capable of handling more code than you can keep in your head at any given time, then surely the problem is simply one of making sure your modules are nicely self-contained from each other.

flavio816• 9 years ago

Mostly, I see this is as a lack of compilation and types in Python. How do others overcome this and enjoy large Python projects?

Divide code into modules.

If you are worried about types, use object oriented programming to define the classes you want and then use the assert() instruction to check variables to comply with said classes/types.

A good IDE like PyCharm will take advantage of such assert() statements.

crdoconnor3• 9 years ago

A combination of:

* Realistic integration tests that cover as many user stories as possible.

* Raise exceptions ASAP for any kind of invalid data or state (could be checking types, but could also be checking for file/directory existence). Could also be pip installing and using 'schema'.

* Do not build on poor quality modules and strive to decouple and uninvent poorly reinvented wheels.

* Just in general: loosely couple everything.

I don't see this as a static typing issue at all. Static typing only helps with a small part of the problem and it usually does that at some expense (usually more verbose & less flexible code).

I'm a little skeptical that super-strong type systems like haskell necessarily help all that much either. They clearly help eliminate classes of bugs, but the overhead is very high (the amount of everyday software written in haskell is seemingly small relative to its apparent popularity).

mmagin• 9 years ago

I actually love forcing people into a fairly consistent coding style, but everything else about the language irks me compared to my previous extensive use of Perl (on a large enterprise software product).

One particularly big thing, that it doesn't have much of a distinct compilation phase. You'll write some script you want to run to do some one-off task that will take the next hour, only to find it crashed after 20 minutes when it hits some boneheaded syntax error you made.

makecheck• 9 years ago

For that though you can use linting tools such as "pylint"; catches a lot of errors that might otherwise be runtime exceptions.

dwaltrip3• 9 years ago

For this reason, I generally test scripts like that with a small slice of the data, or small set of dummy data. Then one can also verify that the output is reasonable -- which is good to do even with a compiled language that would catch all syntax errors ahead of time.

shalabhc2• 9 years ago

Take a look at mypy and pytype - there is a drive in the Python community to add a verification system to the language.

cbr• 9 years ago

    only to find it crashed after 20 minutes when it hits
    some boneheaded syntax error you made

I find mypy to be pretty useful for this.

IamNotAtWork• 9 years ago

I haven't overcome the virtual env thing. There are a few virtual environment libraries, not sure which one to use. Then it is awkward to have to change your entire environment over by activating it and for it to be global. What if I want to work on two different projects in two different terminals? Why can't I activate an environment by just CDing into a directory with my project?

Then there is the GIL. I still don't know if multithreaded python apps are really MT or just faking it. And yes, I know you can do event driven stuff and that does look very interesting.

orf• 9 years ago

Use "python3 -m venv"

You can activate two different environments in two different terminals. Why wouldn't you be able to?

You can automatically activate venvs when you cd into a directory with virtualenvwrapper.

Rule of thumb: if a thread goes into C it's releasing the GIL. Not always, but often. Like reading a file, or a library like like numpy or Pillow.

hyperpallium• 9 years ago

Java could have libraries almost as easy to use as python, but doesn't.

However, some things can't be as easy as in Python, such as sympy, which does symbolic processing of in-python expressions (differentiation, integration etc).

I actually don't find python easy to use - every time I use it, I have to look up how to do for loops (what? range? zero-based? does it include the last one?), and the docs don't have links for arguments (even though they require specific types), because no static types.

But the libraries absolutely are easy to use. And because many are written in C (unlike most of Java's librries), you're arguably running C.

submeta3• 9 years ago

+1 for PyCharm. With it I realized how much I was missing with Vim/Emacs/Sublime (even though I configured a lot in those editors).

mixmastamyk2• 9 years ago

Once you grok the virtual environment thing

I wish this would die, it just scares off newbies. In my twenty years of Python I encountered conflicting reqs once or twice. It's easy to direct folks to the documentation in those circumstances rather than cramming it down every newbie's throat---YAGNI.

Now that pip can install with --user and containers are popular, the problem isn't even possible in production in many circumstances.

donarb• 9 years ago

Virtual environments are not just for conflicting requirements. I write code for clients on my laptop and don't want to clog up my packages directory with project specific libraries that I'll never use myself. Once the project is done and I turn code over to a client, I can delete the virtual env and all the installed libraries. Virtual envs also allow me to have multiple versions of Python installed on my machine.

This discussion is a lot like version control. How many people made daily (hourly) duplicates of project folders before learning the benefits of version control?

alienreborn• 9 years ago

because readability is one of the most, if not the most, important quality for mainstream programming language adoption.

cremp• 9 years ago

Golang forces readability; might want to give it a shot.

lilbobbytables2• 9 years ago

Just the other day I started looking back into Golang. Syntactically, and with Gofmt, it is quite readable.

But then they have to go and ruin it by making single letter variables a common thing.

pyre• 9 years ago

But significant whitespace means that I can't write code like this:

   sub (@) { return map { "@{[ $_[0] . $_ ]}" } @$_ };

Disclaimer: The majority of the programmers that I have known in real life that have a significant (and very vocal) dislike of Python's whitespace requirements have all written code like that above (this is my from-memory reproduction of actual code that one of them wrote). Python sucks because it's "restricting" them from being "expressive" with their code.

minimaxir• 9 years ago

For statistical programming in particular, readability of your code by others is the most important part of the coding style.

lilbobbytables2• 9 years ago

Glad I read your disclaimer before responding. :P

Was just reviewing colleague's code with them the other day. Hit some blocks like this (not quite as offensive, but tackling a bit in one line), and hell if they knew what was going on in their own code.

AlexCoventry3• 9 years ago

What does that do?

seanmcdirmid• 9 years ago

Alternative syntax for short one liner functions (e.g. As recently added to C#, always in scala) can reduce a lot of this friction without sacrificing formatting guidelines. However, I'm not sure if it's in python's philosophy to explore such things.

someone1012• 9 years ago

Have you ever looked e.g. at the distutils code or tried to figure out how to use it without using Stackoverflow?

That function is easy to parse if you take 20 min learning the syntax. Object oriented monstrosities aren't.

wyldfire3• 9 years ago

Phew -- it's an automatic filter.

Sadly, you can write code just like that in Python. I don't know what that perl snippet does, my caches have long since flushed ;) -- but here's a Python snippet that's relatively dense:

   def foo(y): return lambda x: {str(i) : '_{}'.format(i**y) for i in range(x)}

reificator4• 9 years ago

I've never posted about whitespace sensitivity before today, so maybe I don't fit in the group you're pointing at, but I don't like Python's whitespace sensitivity either.

Not because I want to write everything on one line, but because it's an extra step when trying to understand what code does.

For instance, a piece of code is misbehaving, not being called as part of the block it visually appears part of. In a bracket/paren language I can highlight one bracket and my editor shows which one matches. In Python I have to figure out what the indentation is built from, spaces or tabs, and make sure there's no ambiguity there.

We have developers here that are claiming they wrote things in Python for their PHD, and yet when I look at their code in any language there are tabs and spaces mixed in the same line, with inconsistent indentation. Multiple people have tried to express the issue or teach ways to deal with it, to no avail.

The fact of the matter is that whitespace is by definition invisible, and that makes it hard for people to grasp its importance. Basing control flow on invisible characters is asking for trouble.

To quote PEP 20:

Explicit is better than implicit.

Brackets are explicit, whitespace is implicit. You only can see whitespace by the gaps between visible characters, and then you have to guess. Is it an actual U+0020 like you see 99% of the time? Is it an nbsp? It it a tab that happens to stop after the width of a space? You don't know until you encounter an error or turn on whitespace rendering.

Was it copied and pasted from www.python-help-7575.com, which uses a platform that favors punctuation spaces? Was it copied from a site that didn't wrap their code in <code> or <pre>? (I see this all the time in comment systems on programming blogs) Should we claim that copying and pasting code in a beginner targeted language is something you do at your own risk?

mark-r4• 9 years ago

As much grief as Python gets for significant whitespace, I think it really helps the readability. Not only does it enforce what should already be good indenting habits, but it eliminates the noise of braces.

wickawic• 9 years ago

I always figured the people complaining about the whitespace thing are people whose code I really don't want to be reading anyway

agumonkey• 9 years ago

It does cut a large chunk of issues off. Also helps "canonicalize" idioms. That said you can still have mess from time to time. But that's alright.

blunte2• 9 years ago

Lack of braces is visually appealing, but I really miss brace hopping in Vi - % to bounce between open and close brace. That also applies to languages with do/end instead of braces.

dbcurtis2• 9 years ago

I've always had the opinion that computers should serve the needs of humans, not the other way around.

In languages where white space is insignificant, it is pretty much universal that we use syntactic elements to communicate block structure to the compiler, and white space to communicate block structure to humans. Having two different ways of communicating block structure is: a) redundant and wasteful, and b) a source of bugs.

So, if you buy into the idea that having two different ways of communicating block structure is somewhere between an annoyance and a language design defect, and also buy into the idea that computers should serve the needs of humans, then it follows that the compiler should extract block structure from source the same way that humans do. That being: significant white space.

The {} languages seem very primitive to me any more.

ufmace• 9 years ago

I've had mixed feelings about it, but I think one of the better things about it is that it forces less experienced scientist and data analysis programmers to use consistent whitespace.

tw10104• 9 years ago

Why do us engineers have a tendency to oversimplify everything to one big major cause like this? I see it all the time. Obviously in this case this is not the only reason. The real reason is a mix of many influences.

acomjean2• 9 years ago

While better than perl, as someone who has to maintain others python code the tendency to want to write stuff compactly makes it difficult to maintain than other languages in my opinion (the 5 others I work with). I've found other languages easier to read.

It might just be the code base I'm working with..

kodablah• 9 years ago

I think there are two forms of readability: localized readability and whole-program readability. Python only satisfies the former. Your statement about readability being one of the most important qualities I believe is true, but more for whole-program readability in my experience.

JoeMalt• 9 years ago

Agreed. Working on a large Python codebase, I find myself greatly missing Java's improved static analysis, even for simple things like "find the definition of this function".

Alex3917• 9 years ago

With Python even the code I wrote as a junior developer basically stands up five years later, looking back as a senior developer. There are definitely things I would have done differently, but there isn't any code where it would take several weeks just to figure out what the hell was going on and then another several weeks to make some modest improvements.

binthere• 9 years ago

I've started to use python a lot for 3d scripting on Blender and Maya. On Blender this is the only option.

gallerdude• 9 years ago

As someone introduced to Python and C two semesters ago, I'm not surprised at all. If I want to be really nerdy and crank some performance, I write C. If I want to test ideas, write to a text file, or basically anything else, I use Python. Python just makes it all super easy and nice. I respect the performance gains of C, but ease of use will always win.

acomjean2• 9 years ago

It's the libraries I think. that's why I started using it. I don't think it's particularly different syntactically from most other languages. The built in data types are nice, though why they insist on using different names from other languages (lists, dictionaries ...)

Better editors help too.

redblacktree2• 9 years ago

I may be biased, but I much prefer "dictionary" to "hash table" and "list" to "array." They provide much better metaphors to real-life things, instead of requiring people to learn and use additional jargon.

baldfat• 9 years ago

I have mostly left Python and moved to R, Racket and Lua for my needs. It is hard to see how Python won't stay the predominant first language for decades to come just due to momentum and the good reputation.

My first language was Basic and my second language was C64 Assembly and Pascal. I would have killed for Python as a kid.

tw10104• 9 years ago

The really interesting question though is how big of an influence meta-posts like this one has on the growth of one programming language over another. How would you model that?

mark-r4• 9 years ago

Have you ever known someone to pick a language based on a blog post?

tw10104• 9 years ago

Javascript programmers base a lot of their choice of framework on where the hypetrain is heading, which blog posts can have a lot of influence over.

deepnotderp• 9 years ago

Because programming in Python is great for productivity, has lots of useful libraries, is rather bearable/enjoyable depending on your view point and for 90% of people/companies you don't need the speed of anything more than PyPy or Cython because, believe me, you're not at Facebook or Google scale, and even those companies often use Python (Google most famously).

Btw, for those of you who dislike the dynamic typing (e.g. myself) of Python, check out Cython, it's really heaven.

thisisit• 9 years ago

Mostly because the whole Data Science and ML stuff is catching on. Everyone and their grandma now wants to do something related to AI, ML etc. The easiest language for these is? Python. Hence the growth.

Once upon a time it was all about cloud, SaaS and building a webapp. Hence the growth of JS.

choxi• 9 years ago

The chart makes a pretty compelling case that this is the right answer. The top trending libraries are pandas, numpy, and matplotlib (both flask and django are probably trending at the same baseline rate of any top 10 popular language). These are all libraries heavily used in data science.

We also know that ML and AI are new macro trends in the tech industry. Python's elegant syntax isn't really evidence for why Python is trending now as opposed to any other time in the last decade.

bsilvereagle• 9 years ago

These are all libraries heavily used in data science.

These are all libraries heavily used in any technical computing field. The cost of MATLAB, the simplicity of Jupyter, and the community around Python have caused Python to slowly be a language used for general technical computing, not just data science.

h4l0• 9 years ago

Python has been continuously becoming the entry point of computer programming. I think the language reached that goal and now people who started with python are contributing in different fields.

Personally I met with python thanks to course assignments. It helped me to finish my projects way faster than any other language. I loved its simplicity and now even in my complex projects, I tend to solve my problems in "python way".

Also modularity, almost non-existent boiler-plate code, etc kind of things attract data scientists without computer science background.

alunchbox• 9 years ago

the colleges and programming intro courses for my university are starting off with python now. When I went to uni it was a language called processing which is a simple version of Java. College started out with JavaScript moving into .Net. Some my teacher I kept in contact with moved over the courses to python which might also be a factor in python popularity. Not to mention High schools programming classes also starting with python or js instead of QBasic.

maxerickson• 9 years ago

Related discussion of the previous post: https://news.ycombinator.com/item?id=15186025

AlexCoventry3• 9 years ago

TL;DR, machine learning.

goatlover8• 9 years ago

ML plus Pandas, Matplotlib, SciPy, Jupyter Notebooks, etc.

beamatronic• 9 years ago

Easy package management. If Java made it just as easy to add packages, and run in an interpreted mode, it might have had similar success. Actually years ago there was something called "jython"

outworlder2• 9 years ago

It's not only because of data science of machine learning. Big, high profile projects are being done in Python (eg. Openstack).

I am at a big company, and it seems most new projects are being done in Python. No one asks why, it is just accepted.

Now the question is why. I have no explanation, other than maybe that's because universities have been teaching python more often, and it happens to be practical.

I just wish everyone were using Python 3.

deckar01• 9 years ago

Is stack overflow traffic a good indicator of library growth? It seems like libraries with really good documentation would require fewer stack overflow questions regardless of their popularity.

csense• 9 years ago

Now that powerful machines are commonplace, the biggest limiting factor of software creation is developer productivity.

Python's so popular because it maximizes developer productivity.

igravious3• 9 years ago

I say all the following as an avowed Rubyist

1) Computational Linguistics libraries

2) Machine Learning libraries

3) Still one of the blessed languages internal to Google? and BFDL was employed for a while there

4) Jupyter Notebooks

5) Coder Dojo approved language

I used to think that all dynamic languages (Ruby/Python/Javascript/Perl/…) were of a par but I have to admit that Python has an awful lot of mindshare and it's beginning to exhibit strong network effects.

aphextron• 9 years ago

I used to think that all dynamic languages (Ruby/Python/Javascript/Perl/…) were of a par but I have to admit that Python has an awful lot of mindshare and it's beginning to exhibit strong network effects.

Ruby is a wonderful language with a really unfortunate standard implementation. It's easily 5 times slower than Node at practically anything, which is sad because Ruby is infinitely better than Javascript. I really think this is what it comes down to and why Rails has been dying off.

goatlover8• 9 years ago

Agreed on Ruby as a language, but Node also has the advantage of being isomorphic. It's hard to beat JS when it's the browser language, and then you provide a really nice server side platform for it.

Plus there has been a reaction to Rails size and complexity. I kind of hate that everyone thinks of Rails when Ruby is mentioned. But I thought Sinatra was a really nice microframework.

igravious3• 9 years ago

All that beautiful meta-programming flexibility has come at a price. The research projects sponsored by the likes of IBM and Oracle to give Ruby a much needed speed boost are very slow in the offing – it feels like V8 was developed overnight in comparison. Aesthetically I think Ruby beats Python/Javascript/Perl but unfortunately that's not enough to move the needle on the dial.

It'll be interesting to see whether Python or Ruby or Ruby get to WebAssembly first – I'm correct in saying that none has of yet, yes?

jhbadger• 9 years ago

That's why I have high hopes for Crystal. Yes, it isn't exactly Ruby, but it is a compiled language that has a very Ruby-ish feel.

cutler5• 9 years ago

Ruby 2.4 and Python 3.6 are the same speed unless you're using a super-optimised library such as numpy. In fact, for string parsing Ruby is faster than Python due to its superior regex implementation. PHP 7, on the other hand, is much faster than Python or Ruby. I don't know how Zend pulled it off.

flavio816• 9 years ago

I used to think that all dynamic languages (Ruby/Python/Javascript/Perl/…) were of a par

Try Lua, Common Lisp, Julia, and Racket, for a surprise.

ben_jones• 9 years ago

Because it's easy for people to learn with no prior knowledge in programming. It's taught in many intro classes now and as time goes on the simple fact will cause the language to be used more in production environments as well.

ralmidani• 9 years ago

Python has a reputation for being a good "beginner" or "scripting" language. Describing it as such is really a disservice.

One of things that makes Python a killer language (apart from its readable syntax and straightforward programming model) is how it enables you to jump in at any skill level and be immediately productive, without hampering your ability to get things done quickly and elegantly as your skills grow.

There are faster, trendier languages out there, but I suspect Python will still be thriving when many of them have become footnotes.

runevault3• 9 years ago

The interesting thing, to me, with Python is it can fall out of favor but it seems very good at coming back. Look when Ruby got big and became the most popular scripting language. yet Python kept trucking on, until the data science explosion that made it having things like Numpy and Scipy matter so much (and then from there libs like Tensorflow).

blhack• 9 years ago

This has been such a frustration of mine when mentoring people through learning to code.

Python is a great beginner language, but it isn't a language that is only for beginners.

ihaveajob• 9 years ago

This just about sums it up for me: https://xkcd.com/353/

Animats• 9 years ago

based on Stack Overflow question visits

Is Python use really growing? Or is this a Stack Overflow thing? I thought it was being displaced by Javascript Everywhere.

Python had a 5-year setback from the Python 3 transition, but we're now mostly past that.

ZanyProgrammer• 9 years ago

I think if anything the SO stats underestimate the pervasiveness and popularity of Python, even outside the data science world.

fish_fan2• 9 years ago

Well, most devs I know would rather use python for server-side and scripting, but you're not wrong. Python never found a way to do cross platform UI as well as react native; it's on every browser; you don't have to unlearn brace syntax.

That said; the idea of something like numpy in javascript is years off and laughable.

goatlover8• 9 years ago

Javascript isn't displacing Python or any other language in data science. The web doesn't encompass all of computing.

cutler5• 9 years ago

It's certainly growing here in the UK. From being level with Ruby 5 or 6 years ago the Python:Ruby jobs ratio is now 2.9. I've also watched Django recently edge ahead of Rails for the first time.

jackschultz2• 9 years ago

The most interesting part of the post is the conclusion where the author says that he's a data scientist who uses R and doesn't feel the need to switch to Python!

He also has very good thoughts, and very important for beginners to know, about how there isn't a single right way to learn programming.

In any case, data science is an exciting and growing field, and there’s plenty of room for multiple languages to thrive. My main conclusion is to encourage developers early in their career to consider building skills in data science. We’ve seen here that it’s among the fastest-growing components of the software development ecosystem, and one that’s become relevant across many industries.

make3• 9 years ago

How it that interesting? Statisticians and economists everywhere use R instead of Python

jackschultz2• 9 years ago

The fact that the author wrote a sizable post about how big Python is and is growing, but then says he's more of an R type of person instead of Python. Obviously that's interesting to see when authors aren't giant fans of what they're trying to promote.

CuriouslyC• 9 years ago

What's particularly interesting is that you can tell someone's academic pedigree from whether they use R or Python. As you mentioned, R is very popular in stats, while Python is very popular in computer science. In fact, you can tell whether other academic fields are more influenced by stats or comp. sci by this measure as well. R in econ is a great example of this, while fields that build simulations to do research tend to skew Python.

alephnil• 9 years ago

Python is what BASIC was in the 1970s and 80s, in a good way.

It enables people that normally would not program to do things that they otherwise could not be done with traditional software with a GUI, that being web based, apps or desktop applications.

It is a stepping stone into computing that otherwise would have been inaccessible for many. Unlike BASIC, the design of the language is good enough that it can be used in professional applications, and not only being a stepping stone.

darkmagnus• 9 years ago

The community is very welcoming, in my experience.

INTPenis• 9 years ago

It's dead simple and people complain about speed but I use it for massive distributed backends handling hundreds of messages each second and scaling without problem.

And this was way back when twisted was the only asynchronous game in town. So I'm not surprised it's growing.

aaronchall• 9 years ago

I'm a Python answerer on Stack Overflow, I teach Python at NYU, and whaddaya know, I write Python for my day job.

Why is it growing so quickly? I speculate:

Lots of people want to learn to program - and Python is easy to learn and teach.

I think it blows other languages out of the water. I'm going to compare it to its closest competitor, Ruby. Ruby is also open-source, high level, popular relative to other languages (which means you can probably get help if you need it), it has lots of libraries (meaning you can do a lot with it), ok tutorials, and a decent package manager. On the downside, Ruby is also a very large language, which makes it much harder to learn to mastery.

I think Python blows Ruby out of the water in terms of how easy Python is to learn relative to Ruby.

I don't encourage people to learn Ruby. I spent a long time (months) trying to learn it. I still don't tell people I know it.

I used to think that Ruby's syntax is twice as much to learn - now I think it might be even more than that.

Ruby's YACC file is ~918 lines long[0], while Python's grammar file is 149 lines long.

    ~$ wc cpython/Grammar/Grammar 
     149  879 6472 cpython/Grammar/Grammar

Even if 1/2 of Ruby's grammar output is whitespace, that's more than twice as much grammar to learn as Python's grammar[1].

I could compare Python to other languages, but I don't think they come close to matching the upsides of Python.

I still want to learn other languages, like Haskell, lisps, and C, to a level of mastery - but since I can be so productive so fast with Python, the other goals are something I work on in my spare time without a bunch of urgency.

[0] https://stackoverflow.com/a/16629318/541136

[1] https://docs.python.org/3/reference/grammar.html

PS I can see that this comparison to Ruby isn't popular due to the downvoting - but nobody is contradicting my point.

cbanks• 9 years ago

Cough. Python's cpython/Grammar/Grammar uses a compact EBNF notation. Converted to yacc it is about 850 lines long.

sinemetu11• 9 years ago

How do people create and manage internal python packages that work together as an ensemble? I've found the packaging system with setup.py dependency_links to be pretty cumbersome and not well supported. How do people version these internal packages and manage them? Running your own pypi seems nontrivial and using github eggs/releases isn't very functional either.

My experience with python has been positive but I don't understand how people who want to create separate libraries for specific functionality tie these codebases together without using deprecated features or containerizing everything.

IanCal• 9 years ago

Gemfury provide a service which might be useful.

If you need to keep things on premise, is running your own pypi particularly hard? I've never tried so maybe there's a bunch of hidden complexity but there are some packages to run one and even a docker image: https://hub.docker.com/r/codekoala/pypi/builds/bwe6cdn4swgyi...

ipmb• 9 years ago

pip2pi makes it really easy to run your own PyPI. All you need is something that can serve static files (Nginx, S3, etc.)

https://github.com/wolever/pip2pi

microDude• 9 years ago

Really simple answer...

University undergraduate classes in a variety of courses use it as their default language, so it is a bunch of students trying to complete their homework assignments.

Derbasti• 9 years ago

I don't like how Numpy, Matplotlib and Pandas are portrayed as "Data Science". There is nothing particular to "Data" in these three packages. All I see is "Science".

All scientists need numerical simulations or evaluations, plotting, and statistics. That's what these libraries do. Not "Data Science", but "Science".

The correlation with TensorFlow/Keras suggests Data Science, maybe. But I bet there is a similar correlation with scikit-learn, Scipy, scikit-image, basemap, and heck, Matlab, Julia and R--and those each suggest different scientific endevours.

TL;DR: In my opinion, Python is becoming the language of Science, which includes, but is not limited to, Data Science.

cjbillington2• 9 years ago

I'm still at a loss as to what type of science there is that doesn't have data.

"Data science" to me seems like a buzzword that has connotations of a field or industry starting to use science where they previously didn't. Over here in academia it's just "science", or possibly "data analysis" if you want to emphasise that that's the part of your project you're working on right now as opposed to data collection.

xapata14• 9 years ago

Theoretical physicists don't always use data. Many scientists sometimes use case studies instead of systematic data collection.

PeterisP• 9 years ago

You seem to treat "data science" as a subset of science, but it's really not - it generally is used to refer to the application of scientific methods to data analysis outside of "the field of science" in various business environments, and that is a much wider field with more people than those actually working in science.

dasil003• 9 years ago

The obvious answer to me is the slow trickle down of people from universities as CS programs have migrated from Java/Scheme to Python. This also overlaps with Stack Overflow usage, so I think that explains a lot.

blunte2• 9 years ago

#1 - it comes installed on most Linux and on Mac (even though different versions can be a bit of a pain sometimes)

#2 - it's dead simple to do things; the syntax is, to me at least, about as direct and uncomplicated as you can get (and without being overly wordy!)

#3 - so many handy libraries... it's good for full projects, and it's great for integrating and gluing things together

I wouldn't call programming in Python fun, per se, but I do call "getting things done quickly and easily" fun.

russellbeattie• 9 years ago

Python is a great language, despite its quirks. Imagine how popular Python would be if it had decent basic documentation and a single active version? The confusion caused by Python 3 caused at least a decade of slow adoption and the docs, well, that's pretty much a lost cause. (Compare https://docs.python.org/3/library/os.path.html to https://nodejs.org/api/path.html - the latter has a nice summary at the beginning, an index of methods at the top of the page listing everything below, method inputs and return values are noted for each method clearly, concise non-interactive terminal examples that you can copy/paste for each method is given, the exact version the method was added, a collapsed history of changes and any error types thrown. My only assumption is that the Python maintainers must have a side deal with O'Reilly to keep the docs in the worst format ever as a source of continued revenue for both.)

mjw1007• 9 years ago

I'd say that, over the last twenty years, Python's docs have gone from excellent to good to middling to excellent to good to middling (dropping as the language expanded and the generally expected standard went up, and improving when they did one major revamp).

So I agree that another revamp would be good now, but I historically the docs have been a plus as much as they've been a minus, relative to Python's alternatives.

dom05• 9 years ago

Much plain reference data is missing from the docs (like return types). Its neat to describe a function, but if you don't hand out the API, then the docs are not all that useful.

antod2• 9 years ago

I'm doing mostly Ruby now and I miss the Python docs.

apple4ever• 9 years ago

Python's documentation is terrible! I do like Python (only been using it for about a year), but the docs make it so hard to figure out how to do something.

Will_Do• 9 years ago

I don't think SO questions is the best way to gauge the popularity of a language and especially not a package.

I big reason why Pandas is so popular is because its API is so complicated. I probably have 10x more experience using Pandas than Django. And yet, when I'm doing something in Pandas I'm visiting the documentation far more often than Django. Let's compare the top level APIs:

>> import pandas as pd
>> len(dir(pd))

177

>> import django
>> len(dir(django))

Also, I find the data manipulation Pandas provides is far inferior to the R Hadleyverse and is one of the few things that R does better than Python.

dom05• 9 years ago

Django has a very deep package tree, there are tons of stuff in there — a lot of things you wouldn't expect.

yen223• 9 years ago

Django admin could fill a section by itself.

davesque• 9 years ago

My simple answer to this, apart from the arguments laid down in the article, is that the Python core devs (and community) have found a unique balance between sobriety and adventurousness. I think this explains why the language was adopted by data scientists and ML researchers. My complaints with other languages usually center around them being too preoccupied with theoretical or academic concerns (e.g. Haskell) or being too loose in adopting new features or opaque syntax (e.g. Javascript).

Also, Python is probably the most concise and easy-to-read imperative language that has gained wide acceptance.

rdiddly• 9 years ago

OK Stackoverflow, that's enough charticles about this language or that, and which one's growing and which one gets used by whom at what time of day and in relation to which other query etc. It's like they didn't expect the attention they got for these, and now they're addicted to it. Nothing against Python... because if anything my underlying point might be that language doesn't ultimately matter that much.

But besides, there still remains the same question as always: Doesn't the posing of Stackoverflow questions tend to reflect more who's starting to use a language, than the sum total of who uses it per se? (Since presumably questions taper off with time.) If you're measuring growth or adoption, as in this case, sure, it might seem like a reasonable approximation, but it doesn't tell you anything about long-term use (by people who become experts and stop visiting Stackoverflow) or about attrition (people who quit using it and move on to something else). Thus any trend Stackoverflow can see, probably overemphasizes short-term phenomena. Which is great if you're doing algorithmic trading on the stock market and make your money by responding to the market's every short-term boom & bust (and other investors' overconfidence & diaper-crappings respectively), but not so great if you're choosing a language to which you'll devote months of study.

mountaineer• 9 years ago

Python has been in the top 3 of hiring posts within "Who is hiring" threads for at least 4 years now[1] so interesting to see other datasets noting Python's popularity as well.

1. https://hntrends.com

rurban• 9 years ago

Besides the obvious answer: pseudocode alike, also don't forget: worse is better.

Of all major languages is by far the slowest, hardest to extend and by now easily surpassed PHP and JavaScript with its number of design quirks.

njharman2• 9 years ago

It sucks less than the alternatives.

cleandreams2• 9 years ago

Why is Python So Aggravating?

That's a question that bugs me. I can't believe the level of obviousness of the things that are wrong. That is because we write complex code. Things like direct access to data members (which is 'pythonic'!) are cruel blows to software productivity for complex code. Ugh, Ugh, let me count the ways: No compiler to check the code. The other day I dumped 800 extra lines into a file as a typo. Many, many duplicate functions. Of our whole test suite, how many tests failed? One!!! Ugh. This is not even to mention the topics of Black.Holes.In.Your.Code because you can't tell what types the functions receive. I really think python is driven by non-programmers who consume rather than produce complex code and the teachers who teach these people. I believe the problems will increase over time. Does anyone know the story of Hack at Facebook? I heard the lead developer give a talk once. Be afraid.

flavio816• 9 years ago

This is not even to mention the topics of Black.Holes.In.Your.Code because you can't tell what types the functions receive.

Use aptly named parameters, always, add assert() statements for type checking whenever you feel mistakes could happen.

I can bet, based on my experience, that it's much "safer" (less error-prone) to use a language that allows calling functions with named parameters (where the order of parameters doesn't matter and you know exactly which parameter are you sending a value into), than a language like C++ or Java which has a static typing system but has no named parameteres, plus depends on argument order to call a function.

You should try using Python in this way.

cleandreams2• 9 years ago

We do that. Try developing a complex code base with a medium sized team. It becomes very hard to make changes!!! You just need compiler support, IMHO. Python offers ease and fun, and that has led to an explosion of functionality. But there is a tradeoff. And at some point we will hit that wall, as an industry.

wyldfire3• 9 years ago

Keep in mind that there's python development practices which might be adequate to solve your problems.

Things like direct access to data members

I think you're complaining about (lack-of-) encapsulation. Between methods, functions, and properties, there's ample opportunity for encapsulation. The convention is for internal-use-only members to be prefixed with an underscore.

No compiler to check the code.

In fact there are many static checkers. pylint, pyflakes, etc.

This is not even to mention the topics of Black.Holes.In.Your.Code because you can't tell what types the functions receive.

In fact, you can use type hints [1] to do this.

The other day I dumped 800 extra lines into a file as a typo. Many, many duplicate functions. Of our whole test suite, how many tests failed? One!!! Ugh.

This isn't quite explicit enough for me to understand. You unintentionally redefined a function multiple times in the same file, and your test suite caught the error. You think that instead Python should've refused to execute/import this module w/redefined functions? Hmm, I suppose it's a very uncommon need to redefine functions but it's not uncommon to assign a function to a file-scope name. Also keep in mind that both pylint and pyflakes detect this mistake.

[1] https://docs.python.org/3/library/typing.html

dom05• 9 years ago

I'm not sure that you've taken the time to understand python development practices.

I think this tone is misplaced here.

bdcravens• 9 years ago

Many have listed out the technical reasons, but I think there's a cultural aspect: I think many people are turned off by the community of languages like Ruby. Sometimes you just want a great tool, not unicorns and kittens.

discreteevent• 9 years ago

Your comment is a little bit snarky. But I have to admit that stuff does put me off

swat535• 9 years ago

What exactly are you talking about here?

Are you referring to Github, Shopify, Basecamp and many other functional businesses scaled and built on the shoulders of ruby "unicorns" and "kittens" ?

Having worked with both languages, I can confirm that they simply have different view points with tradeoffs, and that in itself is a matter of personal opinion.

But claiming one community does not build great tools because you view them as "kittens" is ridiculous.

Also please do not generalize things like "most people" prefer X.

[EDIT] I sincerely hope this comment does not come off as rude. I am just trying to understand OP's point of view

newen4• 9 years ago

I think he means those intro to ruby tutorials where they make ruby (and programming in general) sound all magical, where it's so easy to program and programming is just unicorns and kittens or whatever. It becomes tiring and dumb after a while.

Daishiman2• 9 years ago

From my outsider perspective, equivalent Ruby tools advertise themselves a the Most Awesome Thing Ever(TM), with well-designed pages, funny characters and exceedingly strong developer personalities.

In comparison, big Python projects have a lower profile, a more serious, adult tone, and generally don't brag too much about their capabilities.

In practice I've seen that the Python community's greater skepticism towards new packages means that libraries have evolved more slowly but with more certainty, whereas Ruby libs have been swapped out as fashions come an go. Witness the fact that Rails decided to package Coffeescript at its core, whereas Django has no strong opinions on frontend matters. Also look at Django's mature deprecation policy compared to pretty much everything else.

vgy7ujm• 9 years ago

It's beyond my comprehension... it's so .. ugly .. and dead .. and unexciting ... zzzzzzz

luord• 9 years ago

Heh, most of the questions I've answered are in the Flask and/or Vue tags; not surprised to see them related and I'm happy that python's popularity keeps on growing.

Why Is Python Growing So Quickly?

Related Stories