Scipy - the embarrassing way to code

http://www.vetta.org/2008/05/scipy-the-embarrassing-way-to-code/

by dood • 18 years ago

66 37 18 years ago

scott_s2• 18 years ago

The bottleneck in writing code isn’t in the writing of the code, it’s in understanding and conceptualising what needs to be done.

Good observation, succinctly stated. Most of my days are spent trying to figure out what I need to do, not how to do it.

noahlt• 18 years ago

Exception: interfacing with 3rd-party libraries.

brent4• 18 years ago

This is exactly why I have urged people to learn Matlab (on this forum and others). Specifically, I'm sick of seeing hundreds of lines of code written in Java that are essentially a poor (slow) implementation of a one liner in Matlab that uses a 20 year old fortran library. I have to go back all of 8 days to find the following quote in response to my plea:

"...and I question the usefulness of learning MATLAB unless you actually need it."

The problem is, most people don't realize they need it and just plod along writing hundreds of lines of python, perl, ruby, java, etc. implementing a decade old numerical computation.

ejs• 18 years ago

When you write a lot of matlab it becomes a different way of thinking. Instead of the for loops multiple levels deep you can write it all as operations to multi-dimensional matrices.

I would agree that many problems people spend time on can be found as single commands in matlab.

ivankirigin• 18 years ago

I've done lots of Matlab development, and this is too true.

Python is going to replace Matlab, but the pattern in the same.

People aren't really used to thinking about matrices and vectors as primitive types and operations.

By the way, this is an excellent resource for learning Matlab. http://www.andrew.cmu.edu/course/16-720/handouts.html

Look at the "working with images" section. If you manually touch every pixel in an image, you're doing something wrong. You code will be 5X as long, and 100X as slow.

edw5192• 18 years ago

This is a familiar story, regardless of technology.

The more you work, the less you appear to have.

The efficiency paradox.

bridgetroll2• 18 years ago

"Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away." -Antoine de Saint Exupéry

edw5192• 18 years ago

Bulletin board material.

ola2• 18 years ago

The article seems to have two messages: one that it's better to fully comprehend the problem before you start coding; the other that learning a powerful library will save you pages of code. Both well rehearsed truths of computer programming.

As for myself I often find that I have to fully explore the problem on my own terms before I can maturely optimize my source.

mhartl• 18 years ago

it's better to fully comprehend the problem before you start coding

Often the best way to understand a problem is to try to implement a solution.

ola2• 18 years ago

That's a very true observation mhartl and I try to touch on it in my last sentence although it is enigmatically worded as I was trying to sneak a "Premature optimization is the root of all evil" reference into it. Alas not all wordplay is destined for upmods!

Prrometheus• 18 years ago

Would it be advisable to try to implement a solution on a chalkboard first? Mathematical notation and diagrams can be amazingly succinct.

carterschonwald5• 18 years ago

I think the deeper idea to take away from this is that knowing more theory is always a good thing

carterschonwald5• 18 years ago

which begs the question of why it seems that even in pretty technical domains, theres still an aversion to aggressive use of theory tools?

brent4• 18 years ago

What does this have to do with theory?

carterschonwald5• 18 years ago

if you don't know any of the theory behind a problem, you don't even necessarily know if theres a solution thats useful in practice.

A good example in Computer Science is the various reductions (efficient or not )between different problems, where you relate the difficulty of the problem to any other computational problem which you find you're able to encode within the structure of the orginal problem.

gruseom3• 18 years ago

The comments here on Matlab have me intrigued. I'm ignorant and have a few newb questions. How does it compare to the APL family of vector languages (J/K/Q)? How does it compare to Mathematica? What roughly are the licensing terms for commercial use (i.e. if you want to build a product with it)?

carterschonwald5• 18 years ago

in terms of interface, it has more in common with mathematica (both good and bad) than do the APL style languages, basically the language semantics are C style with dynamic shape checking for array operations and a nice syntax for many different linear algebraic operations.

It'd be nice if there was a nice type system to aid in getting comfortable with matlab's semantics, in my opinion

tokipin3• 18 years ago

it's an algol language with array manipulation. APL languages start with array manipulation as a primitive in the design of the language, so they are much better at these sorts of things

Matlab is for engineers, Mathematica is for scientists. Matlab is designed for speedy matrixy stuff. Mathematica uses a lisp-like structure 'replacement' pattern matching evaluation scheme, and intrinsically supports symbolic mathematics and is the the only language i would really call multiparadigmatic

gruseom3• 18 years ago

Thanks. My project is calculation-intensive and performance is important. I would love to use kdb. Arthur Whitney's work is stunning. Unfortunately, their licenses are draconian. Can you offer any insight about these various options from the standpoint of building a commercial product on top of them?

randomhack• 18 years ago

I would suggest looking into NumPy/SciPy instead of Matlab. Matlab has a very good library for numerical computation but in my experience, Matlab is absolutely terrible as a language for anything except matrix stuff.

NumPy gives you a good matrix syntax while building on an excellent general purpose language. Personally I felt NumPy is a lot cleaner and Python a lot more fun than Matlab though Matlab has many many "toolboxes"/libraries not in NumPy.

carterschonwald5• 18 years ago

I second this sentiment. The difference in natural expressiveness is almost painful

maximilian• 18 years ago

matlab costs boatloads of money for commercial stuff. At least from my perspective.

I wish there were an open alternative to matlab that was fast. Matlab is kinda fast if you do exactly the right thing, but its interpretedness gets in the way. Most of the people I know doing computational stuff write everything in c using some outside libraries to speed things up. Often they just code up all their own algorithms from scratch. Its lame, but they usually halve the computation time. I could imagine that SciPy and NumPy is much much faster than Matlab.

Matlab is crazy easy to code for though. Its really handy for prototyping.

llimllib• 18 years ago

> I could imagine that SciPy and NumPy is much much faster than Matlab.

SciPy and NumPy tend to be just about as fast as Matlab, when used in their default manner. When you drop into inline Blitz/C for your inner loops using Scipy though, you can achieve orders of magnitude improvements.

Check it out: http://www.scipy.org/PerformancePython

tokipin3• 18 years ago

give J (an APL descendant) a try, it has some good tutorial thingies that appear right in the console thingie:

http://www.jsoftware.com/stable.htm

and here's a post about it:

http://www.squidoo.com/jprogramming

scott_s2• 18 years ago

The introduction to J that peaked my interest: http://prog21.dadgum.com/14.html

Haven't done anything with it yet, but it's on my radar.

plinkplonk• 18 years ago

> give J (an APL descendant) a try,

J is not open source. Might be a factor to consider.

From the J site,

"Jsoftware licenses source for all the Jsoftware binaries. There are several factors in determining the cost of source licenses. Primary is the source itself: J Engine (portable C); Windows GUI support (C++); and Unix GUI support (Java). Secondary is the scope of use: internal; distributed products; and competitive (to Jsoftware) products. And finally the platforms: Windows; Unix; PocketPC; etc. The price range is from $10,000 to $400,000. Regular updates to our current source levels are available for separate update fees."

mooneater• 18 years ago

careful, not free: http://www.jsoftware.com/source.htm

rapind• 18 years ago

Great article. Learn the hard way and it'll stick. Plus it's fun.

bsaunder• 18 years ago

I've noticed a similar conceptual pattern while investigating meta-programming. I think there's something fundamental going on here. Something that conventional programming constantly misses. Something remarkably simple, consistent and powerful. It's where code is data along a continuous path of transformations.

ambition• 18 years ago

Read Godel Escher Bach. Given the comment above, you'll like it.

JulianMorrison• 18 years ago

I can't believe nobody has mentioned Sage Math (http://www.sagemath.org/).

It's sort of a set union of every good free math program.

Scipy - the embarrassing way to code

Related Stories