Julia parallel task runtime (WIP)

https://github.com/JuliaLang/julia/pull/22631

by boromi • 8 years ago

70 10 8 years ago

jrevels• 8 years ago

This work has been a monumental collaborative effort and I think the eventual outcome will be pretty spectacular.

I gave a talk at an MIT Julia meetup a few months ago that the author of this PR also spoke at; I found his talk to be an insightful overview of this work: https://www.youtube.com/watch?v=YdiZa0Y3F3c

fundamental• 8 years ago

I really look forward to using improvements in parallel execution within Julia. I've been experimenting with it for years and in practice parallel execution has been fairly hacky, though it has been improving. When it works, it does seem to work quite well.

eigenspace• 8 years ago

Very excited to see this come to fruition. Julia is really nice to use on parallel clusters and this will really improve that experience. Especially since that exception stack PR just merged: https://github.com/JuliaLang/julia/pull/28878

iopiop• 8 years ago

It's just me or everytime Julia is in HN a veritable army of shills follows

boromi2• 8 years ago

Care to elaborate? I think a lot of HN posts around popular technologies will clearly have a strong group of followers, not sure why it would be appropriate to call them `shills`.

So maybe there is or maybe it's just genuine interest in the novel parallel task runtime. I'd like to think it's the later.

dang• 8 years ago

Posting like this breaks the site guidelines. Please read https://news.ycombinator.com/newsguidelines.html and follow them when commenting here.

3JPLW• 8 years ago

Is there a non-developer overview of what this patch does? Pull requests often aren't the best thing for Hacker News submissions as they assume a very deep level of understanding of a project… especially when they chronicle long development tasks.

boromi2• 8 years ago

checkout https://www.youtube.com/watch?v=YdiZa0Y3F3c which explains the new features of the task scheduler in a simple fashion.

Basically Julia is employing a new algorithm to schedule tasks, where traditional methods would get stuck/blocked. Essentially, if you run a job with parallel tasks and one of the threads finishes, it's extremely difficult to schedulate the free thread, since the data may still be used by the other threads that are busy; thus you need to have clever way to determine how to allocate computational resources. Julia is implementing results of a new paper on how to optimally do this.

roknovosel• 8 years ago

I've been researching parallel programming languages for my thesis and Julia has been really fun to work with. Alongside Chapel, they really make parallel programs fun to write. Although, in my experience, it's been really hard to make them run as fast as C+OpenMP. Mostly because I got trapped in a lot of performance pitfalls as a newcomer to the language. E.g.: parallel TSP branch-and-bound algorithm where I can't seem to improve the performance - https://github.com/novoselrok/parallel-algorithms/blob/maste...

ChrisRackauckas• 8 years ago

Judging from the linked code, you're using multiprocessing. Using multiprocessing is not similar to OpenMP. If you want something like C+OpenMP, then you'd need to use threading. This parallel task runtime will use multithreading like OpenMP, and should be similar in speed.

Julia parallel task runtime (WIP)

Related Stories