PyCon 2015: Are we still changing the world?

http://blog.lisnr.com/2015/05/11/pycon-2015-are-we-still-changing-the-world/

by tedmiston • 11 years ago

45 75 11 years ago

blog.lisnr.com

Right now, a huge opportunity for Python seems to be data science. I didn't notice it in the list of places where Python shows up. Also, in that space, it more than just shows up, it brings unique capability to the space with its libraries and the support major tool developers have for it.

tedmiston12• 11 years ago

I agree with you 100%. I also got a comment from a NASA engineer that other strong additions are NASA itself and the Department of Energy. If you know of any specific companies, I'd definitely welcome the suggestion.

mpdehaan23• 11 years ago

I think folks just don't talk about Python, they get things done.

While I think, say, other areas also get things done, Rails developers are more likely to blog and have pretty fonts. Nothing against that, it just seems to be the case.

While metrics show there's more Python out there, it's just not exciting, it doesn't change a lot, and it just works. And that's fine too.

tedmiston12• 11 years ago

I find myself thinking (hoping) this is the case too. In fact, writing the post to talk about Python at this level was indirectly inspired by a talk I heard at Code Genius (http://code.genius.com/), put on by the Ruby-based Rap Genius Engineering team.

With respect to metrics, I think having the baseline is important, and you're right. I also found myself looking for more granular metrics then what I was able to find. I really want to be able to answer questions like

- "Within Python, how is usage shifting between web apps, scientific work (both research prototyping and in practice), open source, machine learning, etc.?" - "How are the 'frameworks on top of frameworks' like Flask-RESTful and Django Rest Framework changing in adoption over time?" - "Is anyone actually adopting Python 3 as their default yet?"

I wasn't quite able to find that data, but if anyone has ideas for first steps in getting there, I'd be happy to contribute (email in profile).

baldfat5• 11 years ago

Python is still over shadowed by R in data science.

I stopped using Python for data science and use R and it really worked out great for me. R has gained a lot of momentum and I think that Pandas with Wes leaving isn't being developed as quickly. (Most of it is re-doing what R has done). I am amazed at all the FUD I was told to not use R and just use Python but after hitting so many walls I tired it and I am so glad I did. Your millage may vary.

jules3• 11 years ago

    <-- Maturity -- Fundamentals well designed -->
           R       Python      Julia

PaulHoule2• 11 years ago

R is an awful language with great libraries.

My main beef about R (and similar tools) is that I summarize my data I get from Hadoop and find it is still too big for R to handle and I then have to summarize it some more.

Lofkin6• 11 years ago

Thanks for your insight. I have some questions for your, if you don't mind.

What walls and why did you switch? Did you find the shift in syntax and mindset difficult? Have you used rpy2?

There definitely seems to be more innovation in R world, but blaze, numba and bokeh have no R equal.

realusername• 11 years ago

I also have the same feeling. Python is very mathematics-friendly compared to other imperative languages and is used a lot more than other mainstream languages for mathematics related projects. On Github, there is a lot of good quality machine learning and statistics projects built with Python.

woah• 11 years ago

Python is a very nice language, but a big stumbling block is the poor package management. Contrasting this with node, which makes it as easy as possible to build modular software, python's packages are a big step backwards.

mpdehaan23• 11 years ago

I'm not following this. Can you give examples of features in node?

Given, the quality of things in any community repo, and the namespacing, are often terrible, but this hasn't ever kept me from using good things out of it.

I'm also not sure how it can be a step backwards, as Python is quite a bit older :)

One thing I don't like about npm is folks are starting to store non-npm things in it, which I think may because of the weird situation of npm getting funding. I don't understand that just as much as I wouldn't understand PyPi or CPAN getting funding :)

carrja99• 11 years ago

The key difference between pypi and npm is that if you pull a package down and it depends on something==1.0.2 and you depend directly on the incompatible something==0.8.9 both version live together in peace because only request will use 1.0.2. With pypi (and most other dependency management systems) you'll have to make a choice on which one you want to use and in this example you'll have to choose an older version of the library depending on something==1.0.2.

tedmiston12• 11 years ago

I've done just a small amount of work with node, but as a professional Python dev, I very much agree that package management is one of (if not the most) significant downsides of our ecosystem.

One feature I'd love to have personally is a clear way to manage only direct dependencies, but have reproducible builds in pip. @nvie has done some great work in this space [0][1], but not all of us have caught up yet. :)

0: http://nvie.com/posts/pin-your-packages/

1: http://nvie.com/posts/better-package-management/

Lofkin6• 11 years ago

anaconda solves this

vegabook5• 11 years ago

For me, the only place where Python truly excels versus the competition is Numpy. The easy python scaffolding gets you to the heart of a problem in a friendly and flexible way, and then you can interface easily to this big V8 of a compute engine (especially if you combine it with the MKL libraries). The ecosystem around Numpy is astronomically good.

I have fallen in love again with Python (2) for this reason. I tried 3 but I don't need all it's accountant-like seriousness. It's less fun to code in. When I need to design for servers or for performance I will do golang or ocaml. When I need simple scaffolding for numerical work, and scripting for prototyping integration of various systems, Python 2.7 is like an old, unfussy, friend, helping me along my long and winding data wrangling road with a minimum of cognitive tax. It just works. Long may it live.

In my view the Python powers should look at numerical / data science and throw all of Python's weight behind that, and forget about the web, where they've already lost.

alexhill• 11 years ago

Can you articulate what you mean by Python 3's "accountant-like seriousness"? Is there more to it than print-as-a-function?

I code in both every day and just don't see what the fuss is about. To me, they're barely different dialects; more like different accents.

jMyles4• 11 years ago

The many more generator-style iterables give it a strange dramatic tone to me. Also, the type hints syntax looks utterly un-fun.

vegabook5• 11 years ago

First of all I don't think the print statement is a small issue. I bet 5% of my code is print statements. I love the print statement's unfussy "can do" mentality. Hey, the whole reason I got into Python was because it was utterly barebones. Print "hello world". that's it. You can almost feel the fun a young GvM might have had making it.

Then all this u"xx" stuff on strings. What's that about? I don't care about unicode. 256 ascii characters is fine for me. If I need Unicode I can do it, but I don't need it by default.

Iterators instead of ranges. Fine. Why again? To save memory? I've got 32 billion bytes. I don't need it. It's complicating something to please the computer scientists and it's messing with my mind which has far bigger problems to solve, so it's unpragmatic. If I see htop moaning about memory, I can easily change my code.

Generally, I just like the 2.0 attitude. It's carefree. It just works. Three is trying too hard. Python is just a tool for me. My really hard problems are my daily battles with nonconvex optimization of vast data sets.

tedmiston12• 11 years ago

Thanks for taking the time to read. I agree with most of what you've said, but of course I still don't think Python has lost on the web, though that was my concern that sparked writing this post.

What are your thoughts on the future of Python + NumPy vs. MATLAB or R? It seems the numeric and scientific community is embracing Python more, but what I've heard is mostly anecdotal.

vegabook5• 11 years ago

I use R extensively, even more than Python, but it is not a robust language when your problem moves from exploration to anything resembling production. The memory management appears to be the problem. Hugely inefficient and slow even with 3.x. I get crashes frequently, even under Linux and I run production on R. Moreover, Numpy's two orders-of-magnitude speed increase opens up a new class of problem that R just can't handle, especially in the streaming data space. Pandas has made a big improvement in taking Python towards R, though it would be great if it were multidimensional. Where R has Python beat hands down is in its "vectors everywhere" functional philosophy. List comprehensions just don't match the power, and vector indexing is uneven even in Pandas.

I have less experience with Matlab for cost, and lock in reasons. I need to have the option to distribute my code easily.

jaimebuelta2• 11 years ago

As a Python developer myself, I think that the community is very "pragmatic engineering" oriented (explicit better than implicit, get things done, simplicity, maintainability, etc)

The focus is on getting things done, today and tomorrow; not be trendy or be on the first page of HN or disrupt the language space using a new paradigm.

Python is "silently?" used A LOT. Not only on web development (which has more the spotlight on the Internet) but in areas like systems scripting, data analysis or science research. I think that is a great language with a lot of opportunities (including career opportunities) that is still growing.

Yes, it's not the new kid in town. I think of it more as an advantage, really.

tedmiston12• 11 years ago

Our pragmatism is one of the biggest factors that draws me to Python as a dev.

I agree about our significant use outside of web dev -- even though I know a lot of it is "out there", I had a hard time gathering it with many major orgs not being on CrunchBase, AngelList, or StackShare, (and some with a footprint that doesn't represent their true size on GitHub). If there's a source for this data out there, I'd love to find it.

amyjess• 11 years ago

Yes, I can confirm that I've done a prodigious amount of system scripting in Python.

ianbicking• 11 years ago

Is Python changing the world, or Go, or Ruby, or Javascript? If you are asking about "changing the world" that's a big thing to ask for. Simply being useful doesn't make a technology world-changing. I feel very confident that Go will not change the world. People may use Go to make things that change the world, but Go won't change the world. If Python has changed the world – that is the language itself has done so, not just the people using the language – then I'd look mostly to its community, governance, and development process. For Ruby I'd look to its culture, philosophy, and discourse. Javascript actually has a technical claim to changing the world: it's created a programming platform that requires no installation of software, and has brought sandboxed development into the world in a broad and practical way. But if we look at Node.js/IO.js, a Javascript platform with similar scope to Python, then I personally see a generational reinvention: a platform that a new generation of budding developers can call their own, and make their own decisions. Asking a lot of questions that no one actively considers in these older platforms.

But there's lots of other things that can change the world besides programming languages. Are programming languages really the best place to hang your hat? Do you want to compare Python to Ruby or Node.js, or compare Python a web application or wiki or the cloud; i.e., entirely different categories?

tedmiston12• 11 years ago

Ian -- I'm honored for you taking the time to read through and comment. Your post (understandably) has the highest click-through of anything I linked. At first I was a bit taken back by the words myself (borrowed from the PSF [0]), but as you've pointed out, all of these languages have demonstrably changed the world through different aspects. Thank you for your insight and efforts in sparking the conversation.

0: http://pyfound.blogspot.com/2015/02/psf-python-brochure-sold...

bsg752• 11 years ago

"People may use $LANGUAGE to make things that change the world, but $LANGUAGE won't change the world."

coldtea• 11 years ago

Which makes total sense.

It's the things (end product) that matter, and unless they could only/better/easier be implemented in $LANGUAGE, then it could have been many other languages that could have been used as well.

Whereas something like C did change the world.

PaulHoule2• 11 years ago

I think there is a certain kind of sloppiness in Python culture that holds it back.

For instance, the split between Python 2 and Python 3 has kept from me getting more into Python because I have to choose one or the other in terms of which libraries I want to be compatible with (and the libraries are a big reason I'd want to to use Python.)

If you look at Java or PHP or .NET there is a lot of concern about keeping compatibility between versions, but going from 2 to 3 they changed a lot of APIs that they didn't have to.

The most disturbing thing is that I have been shouted down for years on places like proggit for bringing this issue up but it is only in the last year that I have seen it talked about by pythonistas, and that's a general problem I see with Python, that it ducks the hard questions which Java, in particular, faces before anyone else. (i.e. having a memory model for threads which is mature)

On one hand, the GIL means that Python is not scalable in terms of threads. Today 4 core machines are mainstream, even for phones, and with a language that supports scalable threads, you can get a 4x speedup easily for many real apps.

The other option is go to the Node route, which is balls-to-the-walls async operations in a single process, but (i) the async people also have had a culture of "kicking the can down the road", (ii) it is hard to get programmers who are used to sync APIs to switch to async (this is one reason why the "Modern" interface in Win 8 was DOA... Microsoft was pushing people to use Javascript because rewriting Win32 apps to be async defeats the purpose of using Win32 which is being able to use all the old stuff and not learning anything new.)

msellout4• 11 years ago

There are many ways to get around the GIL. Use PyPy. Use processes. And are you really doing that much computation? Threads on standard CPython are perfectly fine for I/O.

actsasbuffoon2• 11 years ago

PyPy still has a GIL: http://pypy.readthedocs.org/en/latest/faq.html#does-pypy-hav...

blt• 11 years ago

Yes, I really am doing that much computation. Not everyone works on database-dependent web apps.

The GIL sucks. It holds Python back from wider use. I wanted to use Python for the GUI/scripting layer of a high performance desktop application, but dealing with the GIL in a mixed C++/Python multithreaded event-driven architecture was too annoying. When Python code isn't in charge of the main application loop, things get ugly.

These days, I mainly see Python as a source of syntax inspiration for other languages. I say this with the highest amount of respect. I think Python's best qualities - list comprehensions, generators, itertools - are largely independent of its interpreted, dynamic nature. They will eventually find a better home in a language with more industrial-grade semantics.

"The GIL isn't that bad, get over it" is not an attitude that will sustain a language for decades.

jules3• 11 years ago

You get a 4x speedup with multicore, and get a 100x slowdown for using Python. Furthermore, Python supports multiple cores perfectly well as long as you use multiple processes. Python threads also work fine for I/O. The only thing it doesn't do is shared memory parallelism. Maybe staying within one process gives you an x% speedup compared to the overhead of sending data between processes, but it simply does not make much sense compared to the 100x. If you need that kind of performance you would not use Python.

jaimebuelta2• 11 years ago

GIL worries are not founded, most of the time.

Multithreads in Python works just fine for I/O bounded operations. Switch a multithread program to use multiple processes (and avoid GIL issues) is quite simple with the multiprocessing module[1] if you need to achieve 100% CPU usage.

In my experience, complains about the GIL are mostly FUD.

[1]https://docs.python.org/2/library/multiprocessing.html

Lofkin6• 11 years ago

Numba allows multithreaded gil free numerical code.

actsasbuffoon2• 11 years ago

Some languages manage to have a really nice blend between OS threads and lightweight concurrency. Erlang/Elixir can spin up hundreds of thousands of lightweight "processes" (their word for green threads), which get split up between a number of real OS threads by schedulers. It means you can have great performance on IO with green threads, but also have great performance on CPU bound tasks with honest-to-goodness threads.

Clojure's core.async can pull off the same trick, though I don't know as much about how it works.

rkuykendall-com• 11 years ago

Python is changing the world more than ever before, by becoming the new default.

I know undergrad CS students, masters CS students, non-CS professionals well into their career, and non-CS hobbyists. And they are all writing Python. Five years ago when I was doing my undergrad, that would have been Java, or C++.

It's not exactly an exciting change from a HackerNews viewpoint, but it's pretty amazing for programming as a whole. The people I know are so much more excited to be learning. There is frustration, but much less than when I knew people learning on C++. Obviously that's still a fantastic language, but I'm happy Python is the new default.

tedmiston12• 11 years ago

I was excited when MIT switched to Python for setting the trend for other schools to follow that it's okay to teaching computing from the top down.

EToS• 11 years ago

Im currently migrating a large FTSE100 web application stack from Java/SpringMVC over to Python/Django.. Its been a long journey (4 years!) to unhinge Java as the given option, but Pythons proven time and time again to save millions per project, and often with better results because developers can concentration on product rather than framework problems.

rwhitman2• 11 years ago

Is the Python 2 vs Python 3 thing ever going to be resolved?

If you look at Python adoption from the perspective of someone picking it up for the first time, the very first issue they encounter is the "which version of Python should I use" debate, which pretty much frames the experience from that point onward.

The pragmatic attitude that's kept Python 2.X afloat is a key part of why Python has had longevity but it also doesn't set a great tone for "changing the world" or keeping up with the latest sexy fads

techdragon• 11 years ago

Ubuntu 16.04 LTS is going to be Python 3 by default. I switched to Python 3 only a year ago. The war is over, Python 3 won, and now it's just the "too big to fail" guys who still think Python 2 has a future. Google, Dropbox and a few others who will probably give it another5 years before they make the switch using 3.5's type hints to help automate the process of moving to Python 3.

xorcist2• 11 years ago

There was never a war, and Python 3 didn't win it.

We're still not seeing any mainstream Python 3 adoption (no, I don't count your own project). PyPI downloads was still < 5% last I checked. The big web frameworks are still better tested and better performant with 2. Scientific computing is stuck with 2. Same with Ansible. And the big companies you mention have no (public) intention to migrate.

The core developers decided long ago that 3 is what everyone should use, so I have no doubt that will be the case. The open question is how much is left of the ecosystem by then. It's commonly discussed at conferences but there is no straightforward answer. At least we got the TLS stuff backported to 2 which might be indicative of a coming thawing.

streamline92• 11 years ago

The Python GIL problem could been solved in Python3 by just moving all global interpreter instance variables into a single struct and changing the C API to take a pointer to this interpreter struct as the first argument in all C API functions. No thread local variable hacks need be used and all interpreter instances could have been completely independent of each other, allowing for seamless and efficient share-nothing multi-threading. Python3 was not intended to be 100% compatible with Python2 anyway, so it was a perfect opportunity to fix this long standing design bug. But this opportunity was lost.

Strangely, the Google V8 team repeated Python's API design error and also relies on global interpreter variables and thread-local variables to swap interpreter contexts, or "isolates" in V8-speak. Although my understanding is that the V8 team recognizes the problem and are (slowly) addressing it and changing their APIs accordingly with each successive release.

copsarebastards• 11 years ago

To be honest, I think the influence of the Ruby/Rails ecosystem is hurting Python. In the last few years, it's become impossible to run many Python projects without doing a pip install -r requirements.txt. That in itself would not be a problem, except that many pip packages are buggy and poorly maintained. Few people take the time to vet such libraries because the effort involved in a pip install is so low. The result is that we often spend hours configuring a buggy library instead of writing code that would do a better job solving our problems. "Convention over configuration" is no less important than it always was, but it's certainly less fashionable.

There are exceptions. Django rest framework, for example, is an extremely mature, well-made library. But these are becoming less of a percentage of the whole and are harder to find in the chaff.

And there are a few projects still carrying the torch--but typically these are old projects that have been around for a long time, still maintained by older developers who were writing Python when Python was young. We still have at least a decade before these old masters start retiring, so I don't think Python will die any time soon. But it has become more important than ever to be careful what you let into your dependencies.

tedmiston12• 11 years ago

This one resonates with me.

Though I haven't used either myself yet, I've noticed the trend in automated code quality tooling becoming more popular in Python. Specifically, the ones that come to mind are: Codacy, Code Climate (both free for open source); and Landscape.

Personally I'm still using strong classics like coverage and pylint, but plan to explore the new gen stuff soon.

PyCon 2015: Are we still changing the world?

Related Stories