Yes, Python is Slow, and I Don’t Care

https://hackernoon.com/yes-python-is-slow-and-i-dont-care-13763980b5a1

by mithunmanohar1 • 9 years ago

136 194 9 years ago

hackernoon.com

VHRanger3• 9 years ago

The problem is not so much that python is slow. It's that in some scenarios python can't be made fast.

Fast prototyping is great but being stuck with a prototype for deployment isn't.

ColinWright• 9 years ago

Did you read the section on "Optimizing Python" ?? If so, I'm assuming you read about using CPython to migrate your package/program/module painlessly to C.

So given that, can you elaborate on your objection? I'd like to know why you think the article is wrong about optimising your Python to make it fast enough.

VHRanger3• 9 years ago

"migrate your package/program/module painlessly to C"

Do you or anyone on your python using team can use C professionally for things you'll use in production?

I work with python everyday on a team that uses python everyday, and I worked professionally in C++ for a >2 years time, and I still wouldn't trust myself to write a reliable/safe C backend to something.

And using C++ behind python with something like pybind11 is anything but painless on the other end; it requires careful consideration.

Python is "good enough" in nearly all cases, but there are some instances where you sort of hate yourself for using it for the prototype since you're forced to effectively rewrite afterwards.

traverseda5• 9 years ago

It's that in some scenarios python can't be made fast.

Can you give some examples of this? I mean, obviously with enough effort you can "make python fast" since it has good C bindings, and can just be a thin wrapper around fast stuff. Similar to how command line tools can be ridiculously fast[^1] despite, ostensibly, running in bash.

So I'm a bit confused about what you're claiming. Organizational issues, it's difficult to get management on board with an optimization pass?

[^1]: https://aadrake.com/command-line-tools-can-be-235x-faster-th...

VHRanger3• 9 years ago

My point is do you have competent C programmers on your team?

That said, there is a point in some python prototypes where you "hit the performance wall". For whatever reason, you'll need to look at one of the options to make python faster and none of them are painless unless you're already a serious C programmer.

kerkeslager• 9 years ago

In ten years of Python development I have yet to come across an instance where Python couldn't be made fast. In some cases critical sections had to be delegated to C but even that is very rare.

fnord123• 9 years ago

One of the best features of Python is the almost free FFI into C. This means you can prototype and then dial up the performance to 11 if required.

wyldfire• 9 years ago

Your bottleneck is most likely not CPU or Python itself.

I've found that this is often the case. Nearly always disk or network. But it's sometimes surprising how little work you need to do to become CPU-bound. This is the price we pay for such a tremendously dynamic language.

Indeed, the article's suggestions of C/Cython/PyPy are good ones to remedy the problem when it occurs.

booshi• 9 years ago

This keeps getting posted, and while it makes some valid points, it's a lot of handwaving.

Arguably, other languages can get code out faster depending on the dev, language, etc.

blumomo2• 9 years ago

Definitely depending on the developer. That's why his disclaimer on the article's beginning: "I'm a Python fan boy". Nevertheless a very good read to me.

0xcde4c3db• 9 years ago

Agreed. Things that are handwaved include:

1) Performance can be a genuine requirement of the product, i.e. if it's not fast enough, it doesn't ship. You can't ship faster and cheaper by sacrificing the thing you need to ship (well, you can, but then you're shipping a different product, not meeting the same requirements sooner; it's no different than cutting a feature).

2) Many processes can't be horizontally scaled in an efficient way, period. Not because the programmer is ignorant of some cool algorithm, but because the problem is fundamentally expensive to parallelize. Maybe you end up getting something like a 20% boost by having twice as many nodes, even after applying all the cool algorithms. And you don't necessarily get that scalability in your code base for free, either.

3) "Speed" in the mobile and embedded spaces is often as much about energy efficiency and thermal management as getting done sooner.

4) The metrics for deciding that Python is faster to develop in only measure small problems. People tend to shy away from Python for bigger projects, and the reasons for this are pretty hotly debated.

mangecoeur• 9 years ago

Python is also heavily used in science, where performance really does matter. It's successful because of how highly ergonomic python apis can be built on top of optimised C/C++/Fortran libraries.

That said, there is clearly a desire to write 'fast' code in python itself without swapping to C. Cython helps, but to get really fast Cython code you actually have to write with C-semantics (so you are basically writing C with Python syntax).

Projects like numba JIT are interesting in that they can optimise domain-specific code (i.e. numerical/array code) that's written in normal python style. It also means jumping through a few hoops (although with the latest version in many cases all you need is a single decorator on your hot function). You can even do GIL-less multithreading in some cases.

Overall things are looking promising, with the addition of the frame evaluation API and possible improvements to the python C-api that could make JIT and similar extentions easier.

dom963• 9 years ago

The fact that Python is slow isn't its only problem. What I care more about nowadays is wasting my time hunting bugs that could have been avoided by a static type system.

blumomo2• 9 years ago

Please tell me, do you write tests? I've learned that it's necessary. I do 100% code coverage and I'm enjoying TDD a lot.

virmundi• 9 years ago

The whole point of static type systems is that they give you the "type tests". 100% code coverage just to get what the compiler would give you is a waste of time. If python is supposed make developers more productive, this is just dragging them down.

adwn• 9 years ago

Time/cost to locate and fix a bug: at compile time < in unit test < in integration test < in production.

Besides, tests cannot find all the bugs a good static type system can find, and vice versa. Tests, even at 100% coverage, are not a complete substitute.

vog• 9 years ago

If you haven't already tried, I recommend to port [1] a Python program to OCaml, F# or Haskell.

There are so many tests you can throw away, so many corner cases you don't need to test anymore. (I'm talking about tests that you can't even write down without getting a compiler error.) Not to mention the assertions within your functions that aren't needed anymore.

And those little type annotations are so much simpler and easier to write down than corresponding tests.

If you really head for 100% testing, not just code coverage, but also all corner cases, you will love the modern static type systems. (However, you should really use an ML type system, because doing the same with Java or C++ is cumbersome and not much fun.)

[1] I did so for my mathematics diploma thesis, starting implementing my formalism in Python, having lots of trouble when refactoring, then moving all code to OCaml and then being able to refactor the code alongside the developing formalism in the thesis.

VT_Drew• 9 years ago

I'm enjoying TDD a lot

Meh, test are great in so far as they can help you not break things if you upgrade the framework, or langauge in the future. But other than that I have to agree with Kent Beck's sentiment of "I get paid for code that works, not for tests".

https://istacee.wordpress.com/2013/09/18/kent-beck-i-get-pai...

If I had started out trying to learn Django using TDD as in this tutorial:

http://chimera.labs.oreilly.com/books/1234000000754/index.ht...

I never would have learned Django. I would have given up with the ridiculous slow progress of results and functionality.

jsd1982• 9 years ago

100% coverage is meaningless with respect to correctness and the ability to prove facts about your code. 100% coverage only means that you've been able to exercise all relevant lines of code at least once, not that you've tested all possible execution paths through the code.

The idea that you could somehow replace the utility of static type checking with a suite of human written tests is laughable. Static typing systems allow you to prove facts about your code which make reasoning about correctness much easier than with dynamic type systems which practically disallow this (unless gradual typing is allowed in your language).

flogic• 9 years ago

Static types and tests check different things. Tests are for making things function as desired. Types are for making sure they actually connect sensibly where desired. Sometimes you can use them for the other purposes though that often produces ugly tests/types.

gipp• 9 years ago

Mypy / the PyCharm typechecker have come a long way. It's not the same, of course, but you can get a good bit of mileage out of the new gradual typing system these days.

nerdwaller2• 9 years ago

Have you checked out MyPy and the optional type hinting? I've been going that route in my python recently and been liking the both/and of having an optional type checker.

dom963• 9 years ago

I have, and I do like that Python is heading in this direction. But this feature does still feel half-baked to me, the error messages are often confusing and I do recall running into a few bugs along the way. I'm also not certain that the type hinting will ever be 100%, the dynamic nature of Python seems to prevent that. This could end up giving some people a false sense of correctness.

ageofwant• 9 years ago

You only pay the the price while debugging. In your static language you pay the price continually. You are wasting orders of magnitude more time fighting your language's type system every day, you have to read reams of boilerplate code that are unrelated to the problem at hand, each extra line increases the attack surface and complexity of your code.

christophilus• 9 years ago

Having worked in both dynamic and statically typed systems, I don't think this is true. A good statically typed language doesn't get in your way very often, and it doesn't produce any more boilerplate than doc comments do in dynamic languages. You do write doc comments, right?

In fact, I'd say that most of the time, a good statically typed language does the opposite of get in your way. It helps you w/ refactoring tools, better intellisense and editor help, self-documentation, etc. The older a codebase gets, the more helpful a static type system is.

That said, there are statically typed languages which produce a ton of boilerplate (Java used to, at least), and there are those that don't (F# comes to mind).

The only dynamically typed language I've used where I didn't regularly miss static typing is Clojure. (Elixir may fit the bill, but the jury's still out on that.)

asah• 9 years ago

This.

Let me add a radical point: static typing is often thrown out anyway, when you encode your data as strings, serialize/marshal it, create polymorphic lists and hierarchical data structures, etc etc

Meanwhile, disciplined use of python is perfectly easy to get right: - use pylint - be ruthlessly consistent in naming classes, methods and variables (which pylint helps btw,) - for large projects, consider mypy etc

carlmr3• 9 years ago

You are wasting orders of magnitude more time fighting your language's type system every day

I program C# a lot, I never fight the type system. It helps me. It tells me while coding that something is wrong. In Python I'll catch this as well...eventually, when I run the code. A type error in C# would be a runtime error in Python. That's very inconvenient.

The static types are also documentation showing intent. It's hard to read old dynamically typed code bases, because I have to do the type processing in my head to understand what is what. Much simpler if I have nicely defined interfaces.

autokad• 9 years ago

I see this stated as if its a universal fact, but it really true that static types reduce over-all bug density?

I myself have found this to not be true, and I have found the same in reading / talking to others. However, if you have found a resource that has data on the contrary, I would find it very interesting to read.

https://medium.com/javascript-scene/the-shocking-secret-abou...

contravariant3• 9 years ago

A static type system doesn't protect you from choosing the wrong abstraction, and all the bugs that result from it. But if used properly I think it does help.

icebraining3• 9 years ago

"It doesn't matter than Python is slow, besides we can use compiled libraries to speed it up"

"People saying it doesn't matter that Python is slow are deluding themselves and preventing Python from getting faster like JS did"

"Python is inherently harder to optimize than JS since it has <very dynamic features>"

"Smalltalk/Lisp/etc are also very dynamic yet are much faster"

"The slowness of Python is harming the planet by being inefficient and therefore wasting more energy/producing more pollution"

Did I miss any arguments? I know certain topics are bound to attract some repetitive discussion, but "Python is slow" has been one of the worst.

jrs95• 9 years ago

"Because Guido said so" isn't on your list, but otherwise I think you got them all

yen223• 9 years ago

"The slowness isn't worth the metaclass abuse" would be my take

dom02• 9 years ago

"Python is inherently harder to optimize than JS since it has <very dynamic features>"

Python is not a very dynamic language in the sense that you actually can't change a lot of stuff (and a number of the things you can change just segfault CPython). I think JS is more dynamic, for example. Or Ruby.

icebraining3• 9 years ago

These are not my arguments, mind you; I don't know enough to make them.

You've piqued my interest, though: can you give me an example of those things that you can't change or that break CPython?

jayflux• 9 years ago

I get the point this guy is making, but if you need something parallel for a cpu bound task, throwing more hardware at the problem isn't the most efficient solution if you can just use more cores. For example adding another quad core when the first cpu is only using one core anyway is inefficient and expensive.

Right tool for the right job I suppose.

nhumrich4• 9 years ago

Python does multiprocess very well. You can easily use all cores on your machine. Pythons main "disadvantage" is threading because of the GIL. But each process gets its own GIL. So when you multi process, your not limited to one core.

classybull2• 9 years ago

This. I had a problem where I needed to scrape roughly 20,000 html documents daily, which is normally a pretty slow task. You have to open the file, load it into memory, parse the DOM, and then run all of your selection methods. Sequentially, it took about 60 minutes daily. Multithreading slowed it down because it was CPU bound. Multiprocessing allowed me to run 12 processes across 8 cores. That took the total processing time down to about 4 minutes or so. And I was able to write the code in a day. Writing something similar in Java or C++ would have taken me a week.

aeorgnoieang• 9 years ago

The point is that (modifying existing code or writing new code to) "just use more cores" may be less efficient, for the business or organization that is employing the programmer, given that programmer salaries, over even a fairly short amount of time, can be more expensive than hardware.

Waterluvian• 9 years ago

Python is my Swiss army knife. I love it because it is a single tool that can aid in almost every project I do. But if I'm doing one specific thing a lot, I want that thing to be done well and done efficiently, so I'll reach for the specific screwdriver I need.

Also most of my problems are IO bound so single threaded concurrency is fine.

But I represent a very small portion of the global problem space.

_pmf_• 9 years ago

Somewhat ironically, Python is used a lot for things that would benefit from raw speed (data processing pipelines) and do not benefit at all from dynamic typing (since the kind of property bags / data frame views over data are easily replicated in statically typed languages). But Python's C extension API is quite a bit easier than p.e. Matlab's MEX API (to me at least); can typical Python IDEs compile and relink extension modules without an external build step?

Your bottleneck is most likely not CPU or Python itself.

With applications that are dominated by raw data processing, it's very, very easy to be CPU dominated. Hell, I had one quite trivial data converter for logfiles where the "parsing the printf string" part of Java's printf dominated processing and writing a custom formatter halved processing time (while regexes can be compiled, the format string cannot be precompiled and will be interpreted each time); it's one of those things where I would intuitively say "why did this moron write his custom formatter" if I stumbled upon it in a code review. Intuitively, you'd expect this to be a simple case of an IO dominated task (which it is now once the bottleneck has been removed).

If it's fire-and-forget batch jobs, you can get away with it, but if the converter is part of a user facing fat client application that runs on a old office laptop, you don't have that luxury.

freetime2• 9 years ago

I pretty much agree with everything in the article - except for the bit where he tries to quantify why python is better from a developer efficiency perspective than other languages.

The main example he cites is a study that compares the amount of time writing string processing routines in different languages - which is quite a bit different from the work I do every day. I develop web apps which means I generally work in very large code bases, and spend most of my time modifying existing code rather than writing fresh code from scratch. I have found that statically typed languages (java + typescript) and the fantastic IDE support that comes along with them make it really easy to navigate around the code and refactor things. Also - the compiler tends to catch and prevent a whole class of bugs that you might otherwise only catch at runtime in a dynamically typed language.

Of course there are other situations where I prefer to use Ruby as my scripting language of choice - it all comes down to using the right tool for the job at hand. Unfortunately I don't think the author gives enough consideration to the trade-offs between static vs. dynamically typed languages, and I think he would have been better just leaving that section out as it isn't really necessary to prove his point that CPU efficiency isn't important in a lot of applications.

Ultimately though I completely agree with his main point: "Optimize for your most expensive resource. That’s YOU, not the computer."

scarface746• 9 years ago

I was all ready to savage his opinion after reading the headline but I agree looking at my architecture that I designed for the company I work for, CPU isn't the bottleneck. Every time I try to increase performance by multi threading as much as possible, the databases start screaming.

On the other hand, the idea that dynamic languages are more productive than static languages are laughable. Statically type languages prevent a lot of bugs and allow for a lot of automated provably correct refactorings that simple cannot be done with a statically typed languages. You can't even reliably do a "find usages" of classes using a dynamically typed language.

simonh2• 9 years ago

'Dynamic languages' is too often used as a shorthand, or interchangeable with 'scripting language', as here. There are a ton of more relevant language features when it comes to productivity, automatic memory management being the biggest IMHO. Interpreted vs compiled makes a difference too because your code->launch->test cycle can be so fast. But I agree, I'm a huge Python fan and even though I appreciate not having to gum up my code with type declarations, it's not really that big of a win in terms of productivity.

rockostrich4• 9 years ago

But with a statically typed language, you don't need to go through that code -> launch -> test cycle as often because everything is type-safe.

dec0dedab0de• 9 years ago

You could have static duck typing, so declarations wouldn't be an issue. Though, for the most part I like Python just the way it is. I actually hate when a library tries to check types, instead of just using whichever object I passed it.

rockostrich4• 9 years ago

I came here to make the same point about statically typed languages. My previous job mostly involved writing web servers/services in Python. A lot of the existing unit tests in our codebase were solely there to check for type safety (one of the developers that had been there for most of the lifetime of the codebase over engineered a bit with OOP in Python). Now at my current job, we use Scala. I still prefer Python for scripting, but it is very nice to be able to write code and have the compiler do most of the work when it comes to making sure everything fits nicely together.

freedomben• 9 years ago

This has been exactly my experience after working for years in Ruby and returning to Java & Golang. It truly does take me longer to write code without my precious Ruby, but that time is more than made up for by not having to write tests for every single possible code path, purely to check for type safety.

I've also experienced an unexpected benefit in that it is a lot easier to read other people's code since types are explicit and easy to see.

thekaleb• 9 years ago

MyPy provides some useful type-checking for python. I have not had a chance to use it in production code because we are still stuck on python 2 due to Google AppEngine standard not supporting python 3.

bpicolo• 9 years ago

http://mypy.readthedocs.io/en/latest/python2.html

You can use comment annotations to typecheck with mypy in py2

carlmr3• 9 years ago

On the other hand, the idea that dynamic languages are more productive than static languages are laughable. Statically type languages prevent a lot of bugs and allow for a lot of automated provably correct refactorings that simple cannot be done with a statically typed languages. You can't even reliably do a "find usages" of classes using a dynamically typed languag

Exactly, I get quick and precise code completion, I catch plenty of errors beforehand etc. I'd say I'm about 10x as productive in C# as in Python, with similar amount of experience. Python only shines when there is a library that does something really well that you need. For me any productivity advantage in Python is from lots and lots of libraries.

Also in terms of maintainability, I find my C# code easy to read and modify a year later when I've forgotten completely about it. In Python I need to rescan all of the types into my head until I can understand what the program does.

I mean with var and dynamic, C# offers everything you need for duck typing efficiency, while preserving the very important statically typed interfaces.

daemin• 9 years ago

Of course that depends how much more you have to write in one language compared to another. Say you're 10x more productive but have to write 5x code, that sort of cuts down the advantage somewhat.

barrkel2• 9 years ago

    dynamic f = x => x * 2;

Duck typing doesn't seem to be offering everything.

Newtopian• 9 years ago

If you annotate your types in python with the appropriate type hinting comments I find I get quite satisfactory code completion and error detection from my IDE (pycharm). Perhaps not as much as VisualStudio might do for me with C#, true, but then again I don't use them for the same thing either.

Each task has it's set or preferred tools, to write a quick command line tool to bridge a gap between two systems and automate our workflow a bit further python is just perfect for me and much more productive than C#. However to create a system with a team of dozens spanning the whole skill gamut I feel much more comfortable having the compiler as an active member of my team policing the architectural vision for us. In which case C#/Java will win hands down.

If however speed and responsiveness are prime worries then I'll roll up my sleeves and bring C/C++ in. sure my productivity will take a hit, it will likely be harder to maintain as well but if this effort makes the difference for my customers in their productivity then it's well worth the investment.

Often times the final system will be a mix of all these technologies, each used where it's strength shines to maximise their impact.

Yet the technology is worthless when faced with inadequate code structure and system architecture. Coupled convoluted code will be a PITA no matter the language used.

bluntfang• 9 years ago

In Python I need to rescan all of the types into my head until I can understand what the program does.

Couldn't that be solved with sane variable naming conventions and docstrings/documentation?

dogweather• 9 years ago

I feel this about Haskell compared to Ruby: Really easy to jump back in to code I set aside months ago.

barrkel2• 9 years ago

You sound like someone who hasn't used dynamic languages in anger, or you'd mention some of the things that dynamic languages do well that statically typed languages aren't so great at, to prevent your argument sounding like a straw man.

For example: dropping into a debugger (binding.pry / pry_remote in Ruby) to write code interactively in the context of the application, transferring that code to the source, and continuing on with my next fragment of functionality with a refresh of the web page, no recompiles or restarts required. You can do this with some difficulty and certain caveats in a statically compiled language, but only if things have been set up in just the right way, with automatic dependency recompilation and reloading, hot code replacement, etc. And even then, there are limitations on the kind of code you can write in the editor, depending on the underlying technology. Typically you can't create whole new classes, or introduce new fields etc.

Or consider levering up your language. Dynamic languages often give you the ability to create "new" syntax via expressive literals. This opens up more avenues for declarative programming; construct a data structure that models your problem more directly, and then write code that interprets the data structure. You want extremely lightweight literals for this, readable with minimal ceremony, including lambdas for when you need an escape hatch, a little pocket of custom code embedded in the bigger data structure. Statically typed languages have little context to infer types for such lambdas, unless they can generalize from the operators applied, but then you have a generic function, not a piece of data. So you end up infecting your DSL with type annotations and cruft, and before you know it the whole thing is hiding the forest behind the trees.

Thing is, if you aren't used to using these tools, you might not even know they exist, and you may be missing out on productivity you didn't know you could have.

brightball• 9 years ago

I've come to the realization that static vs dynamic is more of a way people think about problems than anything else.

Personally, I've never met a static language that was remotely close to dynamic in terms of productivity...but a big portion of that is the sheer volume of extra code necessary in a static language combined with personal workflow (as you describe).

golergka• 9 years ago

You're describing tools usable in writing new code.

What static languages thrive in, however, is maintaining old code that was written by somebody else.

skybrian• 9 years ago

Hot reload works well in Java and Dart (using IntelliJ).

As you say, you generally can't add new fields or classes, but it's not clear to me how important that is? Doing a full restart occasionally is not that big a deal.

I am skeptical about the long-term maintenance of systems with custom DSL's. Unless it's something popular like JSX, it seems like you end up with your own language dialect that nobody else understands?

notacoward2• 9 years ago

I'd take the point about literals a half-step further. It's really nice to be able to throw a string at a function and have that function compare it to a literal value using == instead of strcmp - and oh yeah, someone was just about to reply saying I should always use strncmp for extra productivity loss. Or I can throw a list/dict of any random type around without lots of ceremony. Do I lose type checking? Sure. Then again, those "expected int, got unsigned int" warnings aren't very helpful sometimes. "Must compile without warnings" is a good idea for long-term maintainability of long-term code, but when you're writing a simple script or still experimenting with an algorithm that crap just gets in the way. A minute or two defining and aligning types might not seem to much, but when you have to do it ten times in the course of what was only a one-hour task to begin with it's a pretty significant hit.

scarface746• 9 years ago

You've been able to do that with ASP.Net MVC for years. You can change your views hit save without having to start over.

But on the back side, why is "restarting" such a heavyweight operation? If I'm iterating over a piece of back end code that takes a while to get to debug, I use the testing framework as a harness to run small bits of code - not a real unit test just an easy way to setup the prerequisites to step through a piece of functionality.

This opens up more avenues for declarative programming; construct a data structure that models your problem more directly, and then write code that interprets the data structure.

Even C++ has a level of reflection these days. But why wouldn't you be able to do that with c# and attributes or Java with annotations?

You want extremely lightweight literals for this, readable with minimal ceremony, including lambdas for when you need an escape hatch, a little pocket of custom code embedded in the bigger data structure. Statically typed languages have little context to infer types for such lambdas, unless they can generalize from the operators applied, but then you have a generic function, not a piece of data.

C# has had type safe lambda for a decade. And as far as a generic function, you can define a lambda/Linq expression that can be reused and that can generate c# IL code at runtime, a MongoQuery when using a the Mongo driver or SQL Expression when using Entity Framework. Ling expressions and the expression trees they generate are very powerful, versatile and typessafe.

abhirag3• 9 years ago

"the idea that dynamic languages are more productive than static languages are laughable." -- being statically typed or dynamically typed comes with its own set of tradeoffs and what a person is more productive in is a highly subjective matter. Lispers are more productive in Lisp than Haskell and vice versa.

"Statically type languages prevent a lot of bugs and allow for a lot of automated provably correct refactorings that simple cannot be done with a statically typed languages." -- not true, Clojure is trying to do that with Clojure.spec and specifications being checked at runtime can get you closer to things you could have automatically proved correct only with languages with dependent types, nothing against statically typed languages but I feel that your sweeping generalizations hurt the point you are trying to make.

hellofunk12• 9 years ago

I recently saw an analysis of frequency of bugs per language according to a breakdown of Github repositories, using issues and branches as a way to quantify bug reports.

It was very interesting to see that Clojure projects were among the least buggy of all languages represented. The report looked at a variety of factors to explain "bugginess", including typing and LOC, among other things. The conclusion in Clojure's case was that the very low line count necessary to write Clojure programs was vastly more influential on bugginess than for languages that were statically typed.

And I suppose that's true: a language that requires you write very little code is going to produce programs that are easier to write (and thus maintain), regardless of whether it is statically or dynamically typed.

scarface746• 9 years ago

Clojure is trying to do that with Clojure.spec and specifications being checked at runtime can get you closer to things you could have automatically proved correct only with languages with dependent types, nothing against statically typed languages but I feel that your sweeping generalizations hurt the point you are trying to make.

"Checking at runtime* is exactly the problem. Why would checking at runtime be more reliable than compile time?

jacquesm• 9 years ago

that simple cannot be done with a statically typed languages

You meant to write 'dynamically'. Not too late to edit!

shakna• 9 years ago

You can't even reliably do a "find usages" of classes using a dynamically typed language.

Let me introduce you to Racket. It started out in life as Scheme, and became something more.

DrRacket, it's IDE, is fantastic. The main reason why? You can trace every function call, and every macro, to its definition with a quick mouse-over, even if said definition is in another file or library, even if said definition is only conditionally created.

There isn't really a reason a dynamic language can't do tracing like that, or mass renames across files... Because some already do.

fiddlerwoaroof• 9 years ago

Yeah, Common Lisp, Racket and Smalltalk all are dynamically-typed languages that offer these sorts of features. Racket can solve this through its powerful reader/macroexpander. The others do it by having you develop inside your run-time environment where you can take advantage of all sorts of interesting information only available at run-time.

lyschoening• 9 years ago

From my experience, statically typed Python is quite a bit slower to write. It pays off for modules that see a lot of use in the codebase, but typing every bit of code can be a lot of work for little gain.

coldtea• 9 years ago

On the other hand, the idea that dynamic languages are more productive than static languages are laughable.

And yet it was common understanding just a couple decades ago. And not just from naive "script kiddies". Read what PG writes about Lisp in ViaWeb for example.

And it's not because we know we have faster to develop static languages, type inference, faster compiles (and also not because of Haskell etc). Those apply to insignificant minorities using Go and Haskell (insignificant compared to the huge hordes using Java, C#, Python, PHP, C/C++, and JS).

It's because of the fad mostly -- JS transpilers and everybody else going for moar types.

Statically type languages prevent a lot of bugs and allow for a lot of automated provably correct refactorings that simple cannot be done with a statically typed languages.

Which is almost irrelevant when it comes to produce something -- you can get it to market faster with those bugs than with the extra rigidness.

flavio813• 9 years ago

Statically type languages prevent a lot of bugs

Correction:

Statically typed languages prevent a lot of the bugs that total beginners to programming would make; while they are useful in preventing the one, two or tree trivial, easy to solve bugs that a seasoned programmer would make once in a week.

They are also not useful at all to prevent the real-life, critical bugs that have nothing to do with types at all, and that are the ones that will negatively impact the product the most and will take serious time to get fixed.

You know what static typing is good for? It has nothing to do with "preventing bugs". It has to do with performance and compiler optimizations.

dragonwriter4• 9 years ago

They are also not useful at all to prevent the real-life, critical bugs that have nothing to do with types at all,

Most bugs are about types, just most languages’ type systems are too weak to express the logical types relevant to many important bugs (and, if they weren't, then you'd probably have to worry more the about bugs in the type-level code.)

traverseda5• 9 years ago

There's are still some big gains python could make, if python implementations were better.

Micropython is equivalent to a real-time cooperative-multitasking OS. If it had ~~better~~ support for things like cffi, you could implement posix on top of it. I can imagine a laptop that runs gnu+python in the next few years.

That's a whole new usecase, simply because that implementation uses a lot less ram. What usecases would we discover for a faster python?

Shared objects and proper sandboxing would also be huge.

jerf2• 9 years ago

"There's are still some big gains python could make, if python implementations were better."

At this point, I would find it far easier to believe that you are underestimating the difficulty involved in what it takes to speed up Python than that there are enormous gains yet to be had in speeding up Python. I suspect JS has had more optimization effort expended overall, but Python has still had a ton of work by lots of smart people, and generally got an earlier head start on optimization. (They didn't start trying to make JS "fast" right away; they spent rather a lot of time getting JS's hookup to the DOM in the face of things like .innerHTML working first, before anyone even cared to do what we today do routinely without thought in plain ol' Javascript, let alone with our glorious frameworks.)

There are enormous gains to be had in speeding up "a language that is like Python except certain things are banned", but people have already done that analysis too and discovered that broadly speaking, if you do that, too much existing Python breaks. If you want to see something like that, check out the RPython aspect of the PyPy project, which successfully implements a fast subset of Python. But it is a noticeably restricted subset of Python; AIUI it's not even close to something you can just drop in to your code and get faster speeds.

One of the things that I've learned from Python and the other attempts to speed up the scripting languages is that despite the mantra, yes, there is in practice such a thing as an intrinsically slow language. (The theoretical existence of a Python interpreter that can run all existing Python code at C speeds doesn't do us much good if we have no idea after decades of very smart people banging on the problem how to manifest it. Personally I'd suspect that while such a beast theoretically exists it has an exponential complexity startup cost or, if you prefer, exponential compilation costs. And probably a pretty decent code and/or RAM bloat cost, too.) And Python is one of them. Some of the reasons why it is so much fun to use are part of that intrinsic slowness. Some of them really aren't.

I personally think there's a lot of up-and-coming languages that are exploring the space of how to get those nicer programming abstractions and programmer-convenient code without paying anywhere near the runtime cost that the dynamic scripting languages of today do; it's one of the more exciting developments I see coming up. People complain a lot about code bloat and poor performance of our code since right now we have to choose between "fairly inconvenient but fast" and "convenient but slow and bloated". Patience! Better choices are developing, but they're still young.

monkmartinez• 9 years ago

Will you please share the languages you thunk are up and coming?

e12e2• 9 years ago

I think the work being done on ruby with the truffle/graal project is interesting in this regard - it's my understanding that ruby is very dynamic too - but it's interesting to see what can be achieved by inclining c to ruby to c etc. It's a crazy stack, compared to something like red (redlang) - but it's amazing how close they are able to get to "magically fast compiler/runtime".

If they manage to combine tracing/warm-up with ahead-of-time compilation to executables... It'll be very interesting indeed.

traverseda5• 9 years ago

I consider "python implementations being better" to be some simple things.

* pypy-level performance by default

* Much faster startup times (I'm looking at you namedTuple)

* Better libraries for specific use-cases, like interacting with opengl

* Rpython actually being documented so I can write library code in it

I'd also really like to see good sandboxing and fast-enough object sharing.

Those are the kinds of things I think can open up whole new market-segments. And python being used by more market-segments means more support for python.

hellofunk12• 9 years ago

Yeah? Well, Jimmy Crack Corn and I don't care.

agentgt7• 9 years ago

Unless we are talking like circa 1999 I don't think I have heard a complaint yet that Python is slow. I'm curious who or where the author heard that from (not specifically the people themselves but the domain they are in).

What I have heard complaints about Python are (and I don't agree with all these points):

* Its not statically typed

* The python 2/3 compatibility

* It has some design flaws: GIL, variable assigning, mutable variables, lambdas, indentation (I don't agree with all these but this is complaints I have heard).

* The plethora of packaging (ie its not unified)

I guess one could argue its slow because it can't do concurrency well but that really isn't raw speed.

Then the author started comparing string processing of programmer time from a study which... doesn't help the authors point at all.

* Python has and will always be fast at string processing and most people know this

* The people that complain about python speed are almost certainly not doing string processing

* I have serious questions about the study in general (many languages have changed quite a bit since then)

wheaties• 9 years ago

Oh, I've heard "Python is slow" from Node folks, Java folks, Scala folks (in terms of productivity) and of course, die hard C++ fans. I'm a fan of Python but i also like languages with a stronger FP bent (and statically typed.)

agentgt7• 9 years ago

Oh, I've heard "Python is slow" from Node folks, Java folks, Scala folks (in terms of productivity)

I have hard time believing Java folks made an argument on productivity unless you meant for that parenthetical statement to be only applied to Scala. Regardless productivity is a far more complicated than raw speed (particularly if you get into maintenance as some languages are easy to pump out code but harder to maintain... oh perl.. the pain...).

And yeah I have seen the C++ folks complain about Python but this in my experience has been video game programming which is clearly not the domain the author is in or talking about (I think given the microservice discussions). Hence why I would like to know more about these folks complaining.

rockostrich4• 9 years ago

Most comparisons of popular web development languages will show a table where every other language out performs Python (except maybe Ruby) [1]. In reality, like the author pointed out, the benchmarks don't really matter when you consider network calls.

[1] https://www.techempower.com/benchmarks/

nerdponx• 9 years ago

For some data processing tasks Python can be brutally slow, especially text processing. NumPy is only fast because it's written in C and offloads hard numerical calculations to BLAS.

agentgt7• 9 years ago

Yes but at least Python has very good native access to libraries.

For example I am sure Java is faster at numerical processing than pure Python at least with Python you have the option of writing in native. Python has really good C interop and it maybe considered by some not part of the language but I consider that it is.

Yeah you could write native code for Java via JNI but its not easy (granted its been a while) and for reasons I can't remember why but JNI is slow (probably moving stuff on and off the heap... I can't recall).

pg3144• 9 years ago

I'm curious who or where the author heard that from (not specifically the people themselves but the domain they are in).

In the telecom domain, I've dealt with data big enough that Python wasn't really feasible. Think 100 of millions of records in CSV format that need to be parsed and processed. Doing that in Python is going to be painful.

classybull2• 9 years ago

Python is insanely fast at data processing and analysis because it has very fast libraries.

As a matter of fact, don't know if you've heard, but data processing it kind of like.. Python's thing...

agentgt7• 9 years ago

Its interesting you mention CSV as I had an issue with CSV with Java many years ago.

I was trying to cut a couple of columns out of a CSV and tried to write it in Java and for some reason either the CSV library was slow or maybe I didn't have the right buffered input but I ended up writing a python script that outperformed my Java code (which reminds me I meant to revisit that).

Anyway here is the python code (it might not be the exact code I used as I didn't update it but whatever):

https://gist.github.com/agentgt/1383185

Acalyptol• 9 years ago

Could you elaborate on what are the complaints you've heard about "variable assigning, mutable variables, lambdas"? Just curious.

agentgt7• 9 years ago

It unclear if you are introducing a new variable to the scope or mutating an existing one.

    # Am I defining foo for the first time or am I mutating
    foo = "stuff"

Python if I recall in the OOP sense combats this with requiring self.foo but this is not the case with other forms of lexical scoping such as loops with in loops etc.

And of course you can't define constants.

IIRC for lambdas its that they can't span multiple lines and I believe they are getting removed (I'm not a python 3 expert)?

nhumrich4• 9 years ago

Author here. I mostly work with a lot of Java devs. They are hesitant to try python because "java is faster". Some use the static typing excuse, but oddly, the most common reason I hear people wont do python is because its "too slow". Perhaps I live in a weird bubble, but that's the motivation behind the article.

agentgt7• 9 years ago

I'm a Java dev (and the one you replied to) and in the past when I didn't work at my own company the real reason Python was often not picked was because it was considered "a toy language" or "scripting language" (ie hard to maintain) and also because it didn't have vendor support like Java did. Of course this was like 8-10 years ago.

I still program and sadly prefer writing code in Java over Python even though I have tons of Python scripts in my ~/bin that I have written over the years. Mainly because I just know the ecosphere better.

And thats what most people should just admit why they choose a particular programming language. As long as its good enough... better... is just what you know more of... the cost of learning something new just to replace something new for marginal speed increase isn't worth it.

So when people say Java is slow to learn, not cool or slow to develop in because the language is verbose... I say just like you: "I Dont' Care" because learning the whole ecosphere of another language is goddamn expensive (that is languages are relatively easy to learn but all the libraries, best practices and tools is another thing entirely).

odiroot• 9 years ago

* Its not statically typed

Yes, that's why we use it.

dahart3• 9 years ago

I'd agree with all the complaints you list, and I love Python, but it is definitely kinda slow. I'd put the speed of python below several of the items on your list in terms of priorities I care about.

I've done a lot of image processing in Python using libraries like PIL, numpy and opencv. Doing per-pixel operations is extremely easy in PIL - improving my dev time, but CPU wise the slowest of the bunch. I love prototyping that way, but I have to move to numpy or opencv or another language to speed it up. A recent program to do a slightly complicated color transform on a 512x512 image was taking me over 60 seconds with PIL. It was 5-10 seconds in JavaScript, and less than 1 second using numpy.

et2o• 9 years ago

Wouldn't numpy be the default choice for this type of work in Python?

bane• 9 years ago

Python is very developer productivity friendly, but degrades performance in weird ways and the methods to increase performance don't always make lots of obvious sense and often are the "idiomatic" way.

I haven't looked into the guts myself, but I'd bet that for similar or equivalent operations, the way the different syntaxes are handled under the hood are wildly different. e.g. function calls are much faster than method calls on objects.

Optimized pure python can look very ugly and non-idiomatic.

nadam• 9 years ago

"It used to be the case that programs took a really long time to run. CPU’s were expensive, memory was expensive. Running time of a program used to be an important metric."

As hardware gets faster we give it new tasks that could not be achieved before. Like rendering high resolution stereoscopic images using physically based shading at 90 FPS on relatively cheap consumer hadware (VR). There are still quite a lot of code that we call 'performance critical'. Most of that code is written in C/C++ (and CUDA and glsl, and hlsl, etc...) today.

carlmr3• 9 years ago

Yeah, also we have a lot of code that a lot of engineers in our company run. Some of it takes hours instead of a few minutes to run, because it's a bunch of slowness combined. Sure you can say the engineer doesn't have to watch. But sometimes you need this result, and can't do much else until it's there. Doing a few loops at a few hours at a time is very time consuming.

Being lazy about slowness is bad in the long run, if you or your customers waste inordinate amounts of time, because one of your devs didn't want to think.

Sure you should optimize from the bottleneck, look at hot path, measure where optimization makes sense. But using tools that make some stuff 10x faster without much effort is not premature optimization. It's just not being stupid.

ivm3• 9 years ago

It's still expensive on client machines because most of the persons in the world are NOT software engineers with 6-digit salaries.

They run cheap computers with HDDs and Windows polluted by a ton of 3rd party crap. They don't know how to fix it and silently suffer.

I was cleaning a local vet clinic's devices recently – they were literally switching between two computers to not wait 5 minutes of non-responsiveness because some bloated software was occasionally consuming 100% of CPU.

mark-r3• 9 years ago

A lot of businesses these days prefer web apps. It's not hard to understand why - all the hassle of system maintenance falls to the people who host the app and can afford to know their stuff. If your Windows PC is suffering from rot just replace it with a Chromebook.

fiatjaf• 9 years ago

It’s more important to get stuff done than to make it go fast.

This is not a real absolute. It is only valid when what you have to run will not benefit a lot from performance or suffer a lot from lack of it.

The real guidance you can have in these matters is: how many times is my code going to run per second?

Some programs are written to be run once a day, others 10000000 times in a second. The first ones should be written in the language you're most productive in, the second ones in the fastest possible language.

donatj• 9 years ago

I know it's not trendy, but I would argue PHP is as productive for developers as Python and has a MUCH faster runtime, particularly after 7.

__s• 9 years ago

without getting stuck in the weeds of the small things such as whether you should use a vector or an array

Yes, instead get into the weeds of tuple vs list

Not included in the graph of time-to-solve-problem static languages: statically typed languages with type inference

scbrg• 9 years ago

Given that they have exactly the same interface, that choice is really easy. You go with one until it turns out to be insufficient, and then you switch to the other and not a single line of code has to change, except at the point where you create the thing.

Incidentally, the same is true in many situations in Python, and that is (IMO) one of its strengths.

snarfy• 9 years ago

Slow doesn't matter when you scale horizontally.

nhumrich4• 9 years ago

Author here. Surprised to see this toping HN. Appreciate all the feedback. Let me know if you have any questions.

kodablah• 9 years ago

The article could be titled: "Yes, Python is Slow To Refactor and Maintain, and I Still Don't Care".

I never understand why dynamic language enthusiasts primarily focus on new code only. You have to discuss all sides of increased or decreased productivity to make a rational argument.

hasenj• 9 years ago

Python is optimized for getting interesting things done in a few lines of code. Small scripts you write once and then forget.

For serious projects? IMO python is a disaster.

Thaxll• 9 years ago

Giving EvE Online as an example is bad because that game artificially slow the game loop to keep up with the number of players, would this happen with C++ on a recent architecture? Probably not.

nervous123• 9 years ago

This sentiment is the reason for almost all software (especially on the web) beeing a load of crap. It's slow, it's buggy and developers always give the same excuse: CPUs and memory are cheap, therefore we can waste our customers time.

Imagine what we could do with the amazing hardware we have, if people started to do the sane thing and actually use the hardware to do things efficiently.

dahart3• 9 years ago

Python's value to me has always been that it's easier to get things done, not it's speed. One time when I was interviewing a candidate for a coding job, the candidate said she loved Python the most "because you can just yell at it and it'll work."

It's both the breadth of the standard library and ecosystem, and the simple language design, that make developing things in Python faster for me.

Doing problems on Project Euler has been an education for me in how algorithm matters more than speed. Lots and lots of people spend hours writing long C++ codes that are easily beaten by a few lines of Python. It certainly goes the other way too, and the wrong algorithm in Python is even that much slower and more painful than the right algorithm in C++. But when the right algorithm is used and the problem is solved in a few milliseconds, it really doesn't matter which language uses more CPU cycles, all that matters is whether you saw the insight that let you skip 99% of the search space, and how much time you spend writing code.

progman• 9 years ago

Yes, time to market is important. However, you don't need to compromise convenience of development for the sake of performance. If you twist your Python code to get performance it takes time. If you need performance, and like the syntax of Python then you should take a look at Nim [1]. With Nim I develop as quickly as in Python while I get the performance of C.

[1] https://nim-lang.org

I believe application performance is important on servers. It makes a difference if your Shop software written in Python is able to handle 50 requests per second, or if the same software written in Nim can handle 500 rps. And by the way, Nim provides static typing which helps a lot to catch errors at compile time.

cup-of-tea2• 9 years ago

Nim seems really good, but does it have a decent REPL these days? I'm not sure if it would be as convenient with a statically typed language, but I like the incremental development approach so much that I only use C if I absolutely have to.

dom963• 9 years ago

It doesn't. But you can grab Aporia (or some other tool) to quickly compile and run some code, it replaces a REPL very well in my experience.

bluedino• 9 years ago

Many times when Python is blamed for being slow, it's the programmers fault. Python is great that you can 'regular' people writing code in it quickly. The problem is, these regular people don't always understand algorithms or things like caches, threads, databases...

A lot of these users can just say "My department needs a $40,000 24 CPU server with maximum RAM from MicroWay/SuperMicro, we need to run our codes faster", when they are just trying to brute force things.

They understand the problem domain but don't have the programming skills to use a computer to efficiently solve it.

But, these guys are all a step ahead of the ones who are stuck in the mindset of "C is the only language fast enough for my work", while not even understanding pointers and basic syntax and getting stuck on silly things like text processing, which could be done in minutes in Python.

sedlich• 9 years ago

In "What if CPU time is an issue?" we could also mention the nim language (and not only cython) because it compiles (not only) to C and feels like python.

deadsy• 9 years ago

Premise: It's more important to be productive than to have fast code. Conclusion: Use Python. Is the premise true? For many cases- yes, but it depends. If you are running an application on the cloud and your metric is $/user/year and you have many users then saving some compute resources for each user gets attractive and you don't want to just throw another VM at it.

Is the conclusion true? Garbage collection gives big productivity gains. Other languages have GC. It's not nice to see your Python code die after a few days because you messed up the type passed to a function. Other languages fix that at compile time. Multicore is now. Other languages are built with better multicore awareness.

boomlinde• 9 years ago

The author argues from his professional experience as a Python developer that it's fast enough, that you'll spend most time waiting for I/O anyway, that you can just throw more servers at the problem etc.

The problem is that his experience as a Python developer doesn't accurately reflect the prevalence of problems where runtime CPU performance actually is an issue. Of course not, because who in their right mind would make an informed decision to solve such a problem in Python? Python has worked for him because it is only useless for a category of problems that he hasn't had the opportunity to solve because he's a Python developer. Outside this professional experience, not everything is a trivially parallel web service that you can just throw more servers at if CPU time exceeds I/O waiting.

It all really boils down to what your requirements are, whether you have all the time and memory of a whole server park at your hands, or a fraction of the time available in a smaller embedded system, how timely the delivery of the software has to be and how timely it needs to deliver runtime results once it's up and running. There are times where Python just isn't fast enough, or where getting it fast enough is possible, but more convoluted and tricky than implementing the solution in a more performant language. Developer time may be more expensive than the platform that my solution is for, but that doesn't get around the fact that it eventually will need to run with the available resources.

agnivade• 9 years ago

However, this is no longer true, as silicon is now cheap. Like really cheap. Run time is no longer your most expensive resource.

Our client won't spend more money than a t2.medium instance on aws. Nothing we can do about it. In that case, run time does become an expensive resource.

But I get the point that OP is trying to make. Just wanted to mention that not all of us have the comfort of having enough resources on which our app runs.

karmakaze• 9 years ago

Putting aside the discussion regarding productivity, there is a case where I have found execution time to matter. Scaling an application which uses an unsharded database. The long transaction durations and number of connctions were bottlenecking db throughput. The particular app was a Ruby/Rails monolith.

Yes, Python is Slow, and I Don’t Care

Related Stories