Programming Languages for Machine Learning

by ahalan • 15 years ago

67 35 15 years ago

djacobs3• 15 years ago

We're using Matlab/Octave in the Stanford ML course, and it's certainly elegant but not necessarily comfortable for functional programmers. I'm reimplementing our assignments with Clojure/Incanter and am finding my code just as performant and even cleaner. In my opinion, the ML domain maps to Lisp code almost as well as AI as a whole.

tikhonj• 15 years ago

Choosing a language for a class has different constraints than choosing one for research or production systems. I imagine performance is less important while ease of learning the language is probably worth much more (probably means not using Haskell for a class :().

I think the ML class at my university--it's in "beta" right now--is using Python following similar logic.

tensor• 15 years ago

The place where performance really matters is generally in the core learning algorithms. Since these are all written in C or Java, you can very easily get away with programming the rest of the code in a higher level language. This is why Python is a feasible choice for machine learning at all.

That said, Clojure is certainly a lot faster than Python for native code and the performance gap with Java is continuing to drop. It's also worth pointing out that you should not ignore C or C++ libraries just because your are running on the JVM. It's not very hard to interface to a core C library with JNI, although be warned, there is a slight trick to doing this via clojure rather than Java.

option1138• 15 years ago

I'm in the course as well and don't really understand what the fuss about Octave is. The help system is abysmal and third-party library support seems far behind other popular platforms. It's also quite unstable on Windows.

I've been playing around with R for the past two weeks and have been more or less happy (with the exception of the memory and speed limitations in the GNU implementation).

Nevertheless, this will be a really exciting field for the next decade or so. Amazing possibilities right now!

ericlavigne• 15 years ago

The instructor emphasized the performance benefit of Octave's optimized matrix operations. Did you use Colt's matrix operations, or were you able to match the performance of vectorized Octave code with non-vectorized Clojure code?

rincewind• 15 years ago

I also implemented some of the exercises in clojure, and my naive implementation without colt or incanter ran orders of magnitude slower than my octave code.

Then again i did not code it with performance in mind. I overloaded algo.generic for matrix operations, so the fixnum math was probably not inlined.

aheilbut2• 15 years ago

I think the real magic comes from understanding the algorithms as matrix operations, and then implementing them by more or less just writing down the algebra.

djacobs3• 15 years ago

I used Colt. I'd be interested to see if Clojure parallelization and under-the-hood optimizations could match vectorization performance. Note, I haven't run official benchmarks yet. My comparisons have been more like "back propagation and gradient checking took about the same length of time in both".

spariev• 15 years ago

I believe Incanter uses Colt/Parallel Colt under the hood

nosignal• 15 years ago

I'm doing the course as well and have been wanting to become more familiar with Clojure. I'd be very interested to see your implementations of the algorithms in a functional style. After the course is finished, I hope you consider putting up a blog or repo with the solutions.

djacobs3• 15 years ago

I'll be happy to do that. If you mail me your contact info, I'll ping you when the class is over.

cbo• 15 years ago

This article doesn't really describe what the title implies. It states issues and concerns for several languages, names several that have large ML libraries written for them already, and even gives his own approach for how to implement new ML algorithms, but it doesn't actually give real options for "Programming Languages for Machine Learning".

I've found that Matlab/Octave is a decent substitute for a "high-level language" to sketch out new approaches with. They're significantly fast, as well as significantly suited to matrix algebra that they can give decent results, even though they have some less-than-beautiful code. Matlab appears to be the language of choice for AI at the University of Toronto.

Personally, I think the best option would be to roll with a functional language (or at least, a language with functions as first-class objects), since a lot of ML algorithms can be reduced to recursion on several matrices, often using very similar functions. For example, ANNs frequently have very similar structures and training strategies, but simply use different learning functions.

Everything can be done in C/C++ though, and while it'd be harder, ML is an area where the gain in speed and efficiency is so significant that extra development time pays boatloads in terms of ROI. Even basic ML examples often involve dealing with 300x236 dimensional data, so you can imagine how much that data would scale up significantly in production environments.

wladimir• 15 years ago

Everything can be done in C/C++ though, and while it'd be harder, ML is an area where the gain in speed

Don't forget machine learning generally involves a lot of experimentation, and this is easier with higher-level languages. Hand-optmization always makes assumptions about specific data structures and details which can be hard to change later.

You can get very close with code-generating high-level languages though. See Theano (http://deeplearning.net/software/theano/), for example. It uses a functional approach to compose the formulas for (c-)ANNs, and has advanced features such as automatic differentiation. Scalable generated GPU code can beat the pants of even the fastest hand-written C loops. Sure, hand-optimized GPU code can perform even faster but in my experience that is usually not worth the trouble.

the_cat_kittles• 15 years ago

scikits.learn, pyMC, scipy, numpy, milk and more in python make a pretty formidable toolkit, R is also (probably more powerful, albeit awkward) excellent! Who needs to read a huge article...

cageface2• 15 years ago

Scala doesn't have great native libraries for this but otherwise I've found it a very good language for basic ML stuff. You can express the algorithms very succinctly and readably but still get good performance.

mattmiller• 15 years ago

Check out Mahout: http://mahout.apache.org/

cageface2• 15 years ago

Of course the great thing about Scala is that you can pretty transparently use all the Java stuff out there. Thanks for reminding me about Mahout.

mindcrime• 15 years ago

No, but there is a very rich assortment of ML libraries available in Java. That alone makes Scala, Clojure, Java, Groovy, etc. reasonable choices for some ML tasks. Mahout is particularly interesting since most of the Mahout algorithms are available in an implementation that's designed to run on Hadoop.

jey• 15 years ago

This is a five year old post. These days Python is probably the way to go with NumPy, SciPy, Theano, IPython, matplotlib, Cython, etc. I'm also really looking forward to PyPy's NumPy support -- should allow you to write in pure Python code but still get most of the optimization benefits of writing in a statically typed compiled language like C++.

glimcat2• 15 years ago

Python and Haskell for development. Weka and R for saving loads of time. Some competency in Java and C helps.

sunkencity• 15 years ago

Waffles for speed http://waffles.sourceforge.net/ command line utilities and c++ lib. Beats the .... out of R on larger datasets.

microtonal2• 15 years ago

Did you ever do number crunching in Haskell [1]? Experiments by the developers of the repa matrix library show that repa (which exploits data parallelism) does not even outperform a single threaded C++ implementation when 8 CPU cores are used:

http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-hask...

The most glaring problem is that GHC currently does not support SIMD vector instructions. A problem which is now, fortunately, being addressed:

http://hackage.haskell.org/trac/ghc/wiki/SIMD

But currently, Haskell is not really an option if you want to write serious ML software that rely on vector or matrix operations.

[1] I did, for maximum entropy modeling. Believe me, it's not near competitive to C or C++ yet. But one day, it will probably be.

sramsay• 15 years ago

Considering what happened to Lisp when machine learning was called AI, I'm not sure I'd want my language to win this contest. ;)

brianobush• 15 years ago

but machine learning isn't promising the world. So far ML has a good track record thus I believe we are safe.

srean• 15 years ago

Given that there are many lisp aficionados here at HN, some may find lush http://lush.sourceforge.net/ quite interesting. Lush has been discussed at HN before but here is a short summary: You get lisp syntax, optimized multi-dimensional arrays, extremely easy integration with C code, common numerical optimization libraries, and the option of translating (and then compiling) lispy code into a C dynamic library.

The language features that the interpreter supports is subtly (and sometimes not so subtly) different from the compiled version though they share the same syntax.

In the context of scaling machine learning code you often hear that one should/could write most of it in matlab/octave and the critical parts in C. But anyone who has done it would know it is such a butt-hurting nuisance. In comparison the C integration is a pleasure in Lush.

purplebear• 15 years ago

I highly recommend to consider probabilistic languages, which constitute a natural framework for probabilistic reasoning and modelling. For example, have a look at this:

http://research.microsoft.com/en-us/um/cambridge/projects/in...

or at this:

http://projects.csail.mit.edu/church/wiki/Church

Have a great day!

Programming Languages for Machine Learning

Related Stories