What languages and tools do you use for cluster computing?

https://news.ycombinator.com/item?id=309300
by etal • 18 years ago
4 7 18 years ago

I've recently come into possession of an account on a research computing cluster. This is for bioinformatics work at a university. For the past few years I've been interested in parallelism-friendly functional languages like Haskell, but in my actual work I haven't come across anything that C, Python and Bash can't handle easily. So this seems like a great opportunity to try the weird stuff.

Most scientists seem to get by using MPI with C, C++ or Fortran. Naturally I'll learn how to do that, too, but I strongly suspect that this traditional route tends to result in sprawling, complex code that's difficult to debug. There must be a better way. Ocaml and Python have MPI interfaces, too; Scala seems to be designed for distributed computing on existing platforms.

And the platform is a limitation, too. The interactive node on this cluster shows a few stable packages to work with: GCC 3.4.6, Java 1.6, Python 2.3 (with a broken numpy), and of course the MPI suite. Nothing exotic; CVS and SVN for version control. I think this means Erlang and Haskell are ruled out, unless they're compiled to C or JVM bytecode first -- any VM must be available on all the nodes if the code's going to run. Right?

So, what do you use? Or, what would you use in this situation?

Related Stories

Loading related stories...

Source preview

news.ycombinator.com