joajoa

Hey, thank you for your question. In fact it is not one CPU program driving thousands of GPUs but rather the opposite: independent processes, each driving one GPU, and all running the same program on different data. The host process (CPU) is just moving the data to the GPU and driving the network. The tensor (sub)tiles are allocated to individual nodes.

I also want to be clear and I hope I was upfront enough about it in the post: I haven't validated any of this at scale myself. Phobos is a learning project, and the numbers in the post are single-GPU. This is my understanding of how the big clusters work, not something I've run as I do not have access to a large cluster.

Make.TV | Engineering | Cologne, Germany / Seattle, WA | Onsite / Remote | Full-Time

We're building the only live video router in the cloud. Make.TV is hiring in Cologne, Germany as well as Seattle, WA.

Check out https://make.tv/ for more info and https://make.tv/career/ for current vacancies.

Don't hesitate to contact me directly: jebert (ät) make.tv

With Java, and any other VM that doesn't specify an exact GC algorithm, it always depends.

IIRC the train algorithm used in some JVMs improves locality. Most GCs use a pointer-bump scheme anyways, leading to pretty good locality for objects that have been created together.

So yes, the JVM _may_ have some pretty cool mechanisms to minimize those. I would be also interested in G1s behavior and whether or not it improves locality somehow.

Absolutely. See [1] for instance. Superoptimization has been around since the 80s and at the first superoptimizer by Alexia Massalin targeted the 68000 instruction set.

[1] http://superoptimization.org/w/images/f/f1/Wp4.pdf

It's not that simple. See [1] and [2]

[1] https://bugs.openjdk.java.net/browse/JDK-8054307 [2] http://shipilev.net/blog/2015/black-magic-method-dispatch/

It was written by André Michelle (http://andre-michelle.com/)

I know it's a shameless plug, but if you want to convert any Java bytecode with less semantic differences than scala.js and full reflection support you might want to give https://www.defrac.com/ a try.