SciPy 1.0: fundamental algorithms for scientific computing in Python

https://www.nature.com/articles/s41592-019-0686-2

by Anon84 • 6 years ago

387 98 6 years ago

dragonsh5• 6 years ago

Our policy for evolving the API over time is that new functionality can be added, while removing or changing existing functionality can only be done if the benefits exceed the (often significant) costs to users and only after giving clear deprecation warnings to those users for at least one year. In general, we encourage changes that improve clarity in the API of the library but strongly discourage breaking backward compatibility, given our position near the base of the scientific Python computing stack.

This is one of the nuggets out of it. I have been using Python with scipy and more extensively with it's sci-kit library since 2014. Started with it by solving discrete traveling sales man problem for solution of sitting arrangements and was astonished with its ease of use of matrix algebra with these libraries and got hooked to it. It helped me to do scientific computing and create a user friendly user interface on web to help end-user to just focus on siting arrangement and conference Organization. Can solve it with Matlab or GNU octave, but couldn’t have developed easily the whole system with user interface in them.

If Swift wanted to be used in scientific computing it needs to follow the similar policy of managing its scientific stack.

Julia is good for scientific computing, but it’s UI and web development libraries are isn’t that powerful. So will see how it develops.

I constantly see on HN criticism of Python and something rewritten in Rust as something revolutionary or ground breaking that it will change the whole world. Earlier this fad was with go language. Probably in time to come Rust will be used for some core library development which brings substantial benefits like scipy, not something rewritten as wrapper around C library with unsafe code which can be done with Python much better.

pmarin3• 6 years ago

Hacker News always has been driven by fashion and novelty. It's a news site after all.

The main innovation of recent programming languages is the use of social media for marketing and PR by their companies to promote them.

jolux3• 6 years ago

The main innovation of recent programming languages is the use of social media for marketing and PR by their companies to promote them.

For some languages I think yes, others no. Rust in particular has a memory management philosophy which was basically novel when it came out; it existed in research papers and such but not very many people were using those languages to do mainstream software engineering. It’s a big deal.

gigatexal3• 6 years ago

That discrete traveling salesman problem of seating arrangements ... how did you model that? I have a similar problem with organizing committees where committees have members that are in other committees and so committees with overlapping members can’t meet together etc etc.

bhl• 6 years ago

Not exactly TSP, but something similar on solving seating arrangements: it’s with simulated annealing; it might help with formulating your optimization problem.

https://news.ycombinator.com/item?id=16548012

ylk• 6 years ago

I think you could also see this as a graph coloring problem https://en.m.wikipedia.org/wiki/Graph_coloring#Scheduling

Edit: to elaborate, each committee would be a vertex and a committee-vertex is connected to another vertex if one or more of their members overlap. Once you’ve got such a graph you simply have to a assign a color (or number/letter/anything) to every vertex in the graph, with the requirement that no neighboring (as defined by being connected by an edge) vertexes are allowed to have the same color. You also try to use the minimum amount of colors needed. Then you can safely schedule the committees with the same colors for the same timeslots, and be sure that everybody will be able to attend.

DataDaoDe• 6 years ago

Probably in time to come Rust will be used for some core library development which brings substantial benefits like scipy, not something rewritten as wrapper around C library with unsafe code which can be done with Python much better.

I think a nice situation would be to have all the core scientific libraries and tools written in rust with clean APIs, independent of any particular scripting language but with the clear goal of providing a clean API to wrapper languages. Then any scripting language can wrap these libs and go - essentially something like a universal numpy/scipy/scikit, etc. This would mean Ruby/Python/JavaScript/Julia, etc. would all be able to get to state of the art immediately while minimizing the duplication of effort across languages and creating a standard "ML" or "Scientific" API.

woodson• 6 years ago

You mean like currently BLAS/LAPACK? Written in FORTRAN, wrapped by any scripting language, e.g. Python's NumPy/SciPy, Julia's LinearAlgebra (part of stdlib, https://docs.julialang.org/en/v1/stdlib/LinearAlgebra/), R, Octave, Javascript (BLASjs etc.). That's the standard API for now.

mxcrossb• 6 years ago

Can solve it with Matlab or GNU octave, but couldn’t have developed easily the whole system with user interface in them.

This is a very key point. I still find that Matlab often has a larger set of features and better efficiency for linear algebra. But thanks to Scipy, you really can expand your mind about what kinds of systems to interface with.

enriquto8• 6 years ago

GNU Octave is indeed faster than numpy. This may be because GNU Octave uses highly optimized solvers (like the famous modified Cholesky method). Due to scipy's license fundamentalism, they refuse to use many of these solvers because they are GPL-licensed. It is a bit sad to see how these ridiculous ideological constraints taint an otherwise excellent software package.

Dowwie2• 6 years ago

Rust isn't going to replace Python because it isn't an interactive language that can be used for adhoc analysis, but it will be used to write the libraries that python imports when the existing ones are no longer sufficient for people's needs.

Python isn't criticized as much as it is recognized as the lesser tool for certain categories of development. That's fine because it is a general purpose language with many great uses.

YawningAngel• 6 years ago

What do you mean? There's an RFC for a proper REPL in Rust and a bunch of good-enough hacks in the meanwhile, there just isn't that much demand for one. You can absolutely write ad-hoc scripts in Rust if you want to

pjmlp4• 6 years ago

For Swift to be taken seriously, it needs to be actually usable across macOS, GNU/Linux and Windows, and not be something that still resembles the heyday of GNUStep experience.

warmvanilla• 6 years ago

Very cool. So what are your go-to tools for putting a front end on top of your models?

dragonsh5• 6 years ago

For the seating arrangement problem in conference everything is in Python and used flask with simple bootstrap and jinja templates.

Feuilles_Mortes• 6 years ago

Qt is cross-platform, very stable, and has sane Python bindings. No other reasonable choice (imo).

gumby• 6 years ago

I agree this is a great philosophy. I have to point out as well, though, that one of the major objections to C++ is that it is "too big" and has too many "dangerous" constructs, both of which are a consequence of this approach.

As a developer using the language this is great: I can write using clean modern approaches* and still use older packages. (Making this all continue to work is a burden on the implementation developers of course.)

* not to argue which modern features are clean or not -- this is about SciPy.

nrclark• 6 years ago

Isn't Scipy on 1.4.1?

kingbirdy• 6 years ago

The article appears to have been written in 2018, and just published recently. From the Discussion section:

SciPy has a strong developer community and a massive user base. GitHub traffic metrics report roughly 20,000 unique visitors to the source website between 14 May 2018 and 27 May 2018 (near the time of writing)

SciPy 1.0 was released towards the end of 2017, so that timing makes sense for a 1.0 retrospective. Not sure why it took so long to get published.

Areading314• 6 years ago

Peer review takes a long time. There is often back and forth among reviewers, who are volunteers and not beholden to deadlines.

lrem• 6 years ago

I have on my desk a 2017 best paper award, for a paper printed in 2016, submitted 2015, being an extended version of a 2013 paper for a problem we solved in 2012. Academic publishing is like that.

geoalchimista2• 6 years ago

Because it took half a year for the paper to get accepted. You can see that the manuscript was sent to Nature Methods on July 28, 2019.

philshem2• 6 years ago

For the record, July 2019 is still at least 18 months after the release of SciPy 1.0 (late 2017)

https://docs.scipy.org/doc/scipy/reference/release.1.0.0.htm...

philshem2• 6 years ago

After the recent release of Pandas 1.0, I was curious to see if/when Conda would upgrade my SciPy to 1.0. Turns out I'm already on 1.4.1 :)

ddavis• 6 years ago

Congratulations to all involved with this paper, an awesome accomplishment for the SciPy community-- pretty wild that it's almost 20 years old.

timkam• 6 years ago

Good that they got a paper in Nature. It's important such accomplishments get rewarded and acknowledged in the traditional academic way. I hope these people pile up citations. The engineers who build the foundations of modern scientific research get far less credit than they deserve.

enriquto8• 6 years ago

Indeed! This publication is very good news, and I am happy that free software acquires such a prominent place in the academic citation space.

On the other hand, notice that scipy is in large part an interface. The underlying solvers (e.g., numpy.linalg.solve) are often routines written in Fortran several decades ago, and already duly cited and academically recognized.

jononor2• 6 years ago

I'm looking forward to the second-order effects of open source scientific software being publishable in "high-impact" publications. It makes it much more attractive for scientists to use and develop open source, benefiting everyone.

neonate• 6 years ago

https://web.archive.org/web/20200207035109/https://www.natur...

tobmlt• 6 years ago

SciPy, and especially numpy: the “free drugs”of the scientific computing world. In the best possible way.

enriquto8• 6 years ago

(...) In the best possible way.

I do not see it that way. A whole generation of students is growing with the wrong impression that writing a "for" loop is inevitably inefficient. Also, they believe that it is OK for large arrays of numbers not to be a core language construct, requiring an external library like numpy. These two incorrect beliefs are incredibly damaging in my view.

In a sane programming environment (e.g., with jit compilation), writing a matrix product using three loops should be just as efficient than calling a matrix product routine. The matrix product should also be rightly available without need to "import" anything.

chpatrick• 6 years ago

I don't think that's currently true in almost any mainstream language though. Even in C++, it's faster to use an optimized BLAS library working on a large set of data than your own for loop.

visarga• 6 years ago

Well, you got to think about optimisation - loop unrolling, parallelisation on all available cores, efficient data layout that works well with caching, doing it on GPU, adapting to the type of GPU you have. It's not as easy as dumping three for loops.

threatofrain• 6 years ago

Optimization will never end up that way unless the system understands both your intention and the domain, but why would you want to express your intentions on matrices through 3 for-loops, as opposed to a product operation already available to you?

And if a matrix product operation were available to a language as a standard construct, wouldn't it be every bit as opaque, encouraging students to think that hand-written for-loops aren't as effective as using black-box things?

jacobolus• 6 years ago

A whole generation of students

Compared to the previous 2 generations of Matlab students, Python/scipy is a night-and-day improvement.

JoeSmithson• 6 years ago

I think what you generally learn is that code you write yourself is potentially slow whereas code from dedicated libraries is probably faster. I don't think it's best practice in any language to write your own matrix multiplication with for loops.

(P.S. Python has some JITed implementations and also has a built in matrix product - @)

iddan• 6 years ago

Writing a “for” will always be less efficient as it enforces a flow. Fully declarative code can always be optimised more

geoalchimista2• 6 years ago

Even in a language where writing three layers of loops is an okay choice (e.g., Fortran), I'd argue against using it everywhere because it is poor abstraction and it will likely be less performant than calling the relevant BLAS routines. I believe the philosophy of NumPy is to outsource lower-level numerical operations to C. When NumPy isn't enough, one can always go to Cython or Numba.

_fbpt• 6 years ago

I agree with your comment to some extent. I'm kinda allergic to writing loops, even in C++ where it's the best solution.

On the other hand, in C++, hand-rolled matrix multiplication is both slower and an order of magnitude less accurate than MKL (or possibly OpenBLAS too).

raverbashing• 6 years ago

they believe that it is OK for large arrays of numbers not to be a core language construct

Well, the Python array is good for most of the use cases. Adding other types to the core language could be useful, but then you get into "which type would be better for me"? Somebody else then would say "but what about a sparse matrix, etc"

I think it's ok to look to numpy for those things.

writing a matrix product using three loops should be just as efficient than calling a matrix product routine

Well, yes, but actually no.

Because of code vectorization, the loop is going to be more inefficient. Yes, compilers are getting better, but they can't do miracles (especially in languages like C)

By using matrix products you're explicit in your desired end result. This can then be optimized by the libraries and you don't have to worry about all the tricks behind it.

plafl• 6 years ago

It's certainly true sometimes one must do acrobatics to avoid writing a simple for loop. The main problem for me is usually the unnecessary use of memory (I use memmap quite liberally). In that regard Julia at least is able to chain together multiple operations to avoid temporary arrays.

On the other hand, and I know is a little niche application, consider writing code for execution on GPUs. Now it's great you are capable of expressing your algorithms using the language of matrix and vector operations and can forget about moving memory in and out of the GPU, threads, blocks, warps and other arcana.

I think that what you call "a sane programming environment" just means "an easy programming environment for me". I don't mind at all having to import libraries. Have you ever tried to structure a big Matlab project?

So basically I agree you with about having fast for loops and disagree on everything else.

SiempreViernes• 6 years ago

Wait, large arrays should be a core construct, but it's ok that matrix multiplication has to be hand rolled?

kortex• 6 years ago

It sounds like you aren't choosing your domain of focus properly. Are you teaching computer science, data science, a hard science, or a social science? You can't have your cake and eat it too. Do you want a language that's easy to teach, easy to solve scientific data problems with? Python with libraries.

Do you want something fast using for loops and other primitives? Don't use python!

Do you want something with great stats and easy graphing but little else? Use R or similar.

Do you want all the benefits of python but want to compile the hot path? Use bindings/cython/any of the numerous inline voodoo python tools.

The matrix product should also be rightly available without need to "import" anything

Why? Even the python stdlib requires imports. The bare language needs keywords and primitives, that's it. Any matrix operation code that loads without import is more overhead.

It sounds like you want a DSL * . Use matlab/octave/R/SPSS. But don't be shocked when you wanna wrap a gui/webpage/api around your code and it's painful.

* DSL in the loosest sense, a specialized, less general language

eesmith• 6 years ago

As a minor point related to "large arrays of numbers not to be a core language construct"

Python added the ellipsis notation, used in arr[..., 0], as a result of feedback from the Numeric project in the 1990s (Numeric is the precursor to NumPy).

So while the implementation of "large arrays of numbers" is not part of core Python, the syntax of one way to index large arrays of number is.

FWIW, I agree with others - there's no way that a matrix product using three loops will be as efficient as calling a matrix product routine. Efficient versions of the latter also consider memory access patterns and cache use. The example tiled version at https://en.wikipedia.org/wiki/Matrix_multiplication_algorith... uses 6 loops.

stelfer• 6 years ago

I remember going to the first SciPy conference at Caltech in 2002. IIRC Most of the talk was about needing better package management. What happened since is just amazing.

xianwen• 6 years ago

May I ask what the status of parallel computation is with SciPy?

fock• 6 years ago

OpenMP threaded blas-routines

mud_dauber• 6 years ago

I learned about SciPy via Numerical Python (Apress/GitHub: https://github.com/apress/numerical-python). That was a ton of fun, especially after I recovered from freshman calculus flashbacks.

maxnoe• 6 years ago

Shameless plug:

Here is a grand tour of scipy as a jupyternotebook me and a colleague used as intro to scipy for PhDs in astrophysics:

https://github.com/maxnoe/scientific_python_notebooks

You can try it without installing the software by clicking on the binder button.

trentnelson• 6 years ago

Just wanted to say I really like and appreciate this, great work!

complex_pi• 6 years ago

There is also https://scipy-lectures.org/ :-)

valgor• 6 years ago

Great read!

I'm ignorant of numerical computing, so I have to ask: is Fortran still used because it is that much faster than C or any other specialized language? I assume that is the case because I'm sure someone would have rewritten those routines by now otherwise.

coliveira• 6 years ago

Fortran still is the language for numerical computing, most algorithms in this area have been implemented in Fortran. And this is very specialized code that has been test by experts in their areas. That's why a project like SciPy cannot simply replace it with something written in C or Python.

SciPy 1.0: fundamental algorithms for scientific computing in Python

Related Stories