High-Performance GPU Computing in the Julia Programming Language (2017)

https://devblogs.nvidia.com/gpu-computing-julia-programming-language/

122 54 7 years ago

devblogs.nvidia.com

systems• 7 years ago

Just pointing out that this is an article from 2017 and was discussed before on hn https://news.ycombinator.com/item?id=15564639

vasili111• 7 years ago

I always glad to see topics about Julia. I think it has good potential to replace several other languages with one better language.

adamnemecek• 7 years ago

Julia is one of my fav languages. For numerical computing, neither python + numpy, nor matlab come even close. The interop is nuts. To call, say numpy fft, you just do

using PyCall

np = pyimport("numpy")

res = np.fft.fft(rand(ComplexF64, 10))

No casting back and forth. This is a toy example, julia ofc has fftw bindings.

Interop with C++, MATLAB, Mathematica etc is similarly simple.

siproprio5• 7 years ago

In theory Julia is supposed to be fantastic.

In practice, things either don't exist, or are poorly implemented:

Plotting simple things take 30 seconds.

And that's if you don't count the time it takes to `] add Plots`, especially on Windows!

And the REPL is broken.

And the editor is slow and annoying (Juno or vscode).

And documentation ranges from poor (no examples, buggy between platforms, broken links due to version updates) to non-existent. For example, lots of tutorials will often link to broken links to official documentation, links that one time were thought to be working but now aren't.

And so on...

xiaodai3• 7 years ago

Sounds like road to maturity problems. The key is to think, "are these issues solvable?" if they are it's only a matter of time. The next thing is when those things are fixed, does Julia offer something above and beyond what Python and R can do (easily). If the answer is for you, thne it's a matter of whether Julia provides value now for you.

ddragon3• 7 years ago

Your 1st, 2nd and 4th points seem to be fundamentally the same, which is compile time latency making interactive use slow. That's definitely a problem for a language trying to solve the two language problem of having high interactivity and high performance at the same time, and the compiler team [1] are now focusing on that issue on versions 1.4 and beyond hopefully it will get to the point where it isn't a problem anymore.

Documentation is always a problem (especially for smaller packages), but I don't feel I had more issue with Julia than other languages (with a few exceptions, some languages managed to have an exceptional culture in terms of great documentation). Most of the issues were due to the fact that Julia just got to 1.0 a year ago, and the breaking changes made so most documentation became outdated, but this will only become less of a problem since the language became stable.

Julia has one of the best REPL of any language I used, and thankfully I didn't meet with any problems with it. Might be a good idea to create an issue on the github.

[1] https://discourse.julialang.org/t/compiler-work-priorities/1...

zzleeper• 7 years ago

Similar experience here. Did a comparison of a bunch of statistical tools (R, Matlab, Julia, Python, etc.) on small-ish datasets. Used the latest versions in all cases, in Windows 10. All but Julia ran the regressions in <1s, while Julia took 20+ seconds, mostly importing the required libraries and just starting up.

Sure, the usual answer is "well its an initial cost, its faster after that" but not all my code would otherwise take days to run. As long as "using CSV" takes 10 seconds, I'm out.

cbkeller• 7 years ago

Things are still improving, but I already find it quite usable in practice. Yes, precompilation can take a while, but once you start using the language regularly you hardly notice it, since you only need to re-precompile after installing updates -- which ends up being a small proportion of the time. Same with "time to first plot", since I almost always have a session already running already these days.

stabbles• 7 years ago

And the REPL is broken.

You should really back up that claim, because in my experience it's absolutely great

socialdemocrat• 7 years ago

Yes plotting in Julia is slow upon first invocation due to the JIT. Annoys me too, but to say the REPL is broken is profoundly puzzling to me. It is the best REPL I have ever used. It beats anything I have used for Python, Ruby, JavaScript, Lua etc.

Also your documentation issue is also strange. Yes certain things don’t exist but I would say the Julia docs is quite well made. In particular if you use the REPL documentation I find it much better than Python. Tends to be quite nice examples, color coding etc.

sytelus3• 7 years ago

Why this infrastructure is so tightly coupled with CUDA? CUDA is very specific and closed APIs for NVidia hardware only. Programming languages should focus on more general primitives that might work on NVidia or TPUs or something else. PyTorch also has CUDA all over in its APIs and its frustrating to see such tight binding with closed one company API. Also take a look at OpenCL.

darknoon• 7 years ago

It's because CUDA performs better. It's not nice, but it's the situation we're living in. Particularly AMD support and performance are lot.

sytelus3• 7 years ago

Sure, I don't mind CUDA backend as first class citizen. I'm talking about having my code sprinkled with word "cuda" all over. Why can't I write my code that is bit more abstract and potentially compilable to different backends? That is, think about the primitives instead of tightly getting married to cuda forever. AMD performance might not be good today but how about 10 years later? How about using TPUs instead? or FPGAs (if someone creates backend for it)?

shmerl7• 7 years ago

It doesn't perform better than what you can do in Vulkan. It's simply more entrenched.

Athas3• 7 years ago

Are you certain that the story is as simple as "CUDA performs better"? It's common folklore, but I have seen little evidence. The only situations I know of when CUDA performs better is when CUDA-specific features are used (if they are relevant for whatever problem is at hand). Also, CUDA libraries (like cuBLAS or cuFFT) tend to be more efficient than their OpenCL equivalent, which is likely because much more work has gone into them. I have also noted that the CUDA compiler is willing to use less accurate (but faster) floating-point instructions by default (for things like e.g. inverse square root), where you need to pass options to the OpenCL compiler for it to do the same. This will matter for some programs.

In fact, I have run tens of thousands of lines of essentially equivalent CUDA and OpenCL code (automatically generated) on the same hardware, and performance was in all cases very similar[0]. If anything, CUDA was actually slower than average (but in the cases I investigated, this was down to arbitrary differences like the CUDA compiler not unrolling some loops as aggressively and such).

[0] https://futhark-lang.org/blog/2019-02-08-futhark-0.9.1-relea...

dlphn___xyz• 7 years ago

there are specific benefits of cuda over opencl: see https://arxiv.org/vc/arxiv/papers/1005/1005.2581v1.pdf

sytelus3• 7 years ago

Yes, but is cuda going to keep its edge 10 years down the line? Do I want to hardcode my algorithms so tightly with today's cuda APIs? Can there be better more generic primitives that are agnostic of propitiatory cuda APIs but would support it as backend without too much perf hit?

idnefju• 7 years ago

OpenCL isn't in a good place. CUDA has become the industry standard.

shmerl7• 7 years ago

Because Nvidia likes lock-in. It totally doesn't have to. Today we have Vulkan for general purpose GPU programming.

llukas2• 7 years ago

OpenCL, hip, Vulkan, what tomorrow? Or alternatively, there were cl* libraries, roc* libraries and hip* libraries for AMD? Which ones are supported?

CUDA doesn't require rewriting code with ${OSS} framework of the year, every year. They need to earn that lock-in with future compatibility guarantees which none of OSS projects has.

jlebar• 7 years ago

Why this infrastructure is so tightly coupled with CUDA?

It's not. It uses LLVM, which can easily target AMD GPUs. (Whether the Julia folks have invested in making this work, I dunno, but it's not Extremely Hard.)

Understandably nvidia gives you the wrong impression.

wallnuss• 7 years ago

We are indeed interested in targeting AMD GPUs. There is a prototype backend available at https://github.com/JuliaGPU/AMDGPUnative.jl and we are closely following the status of SPIR-V and Intel GPUs in LLVM.

The focus on CUDA comes from the fact that most HPC systems for scientific computing are using Nvidia GPUs. That is finally slowly changing.

maleadt2• 7 years ago

Our view is that to get performance out of a system (here CUDA), it's better not to start abstracting it right away. So we have CUDAnative.jl and CUDAdrv.jl for fairly low-level CUDA programming, albeit in a high-level language. However, with CuArrays.jl we implement the Julia array interface for CUDA GPUs. That means you can write array code for one platform (CPU using Base.Array) and start using hardware accelerators by just switching the array type (CUDA GPU using CuArray). Of course, real-life applications might still need to use CUDA specific functionality for one reason or another, but at least you can get most of the way without platform-specific programming.

pjmlp3• 7 years ago

Because Khronos up to a little while lived on a bubble that we have to use C, write our own compiler and linking logic to use GPGPUs and collect debugging toolchains from each OEM.

Only when they started getting a beating of PTX bytecode and multi-language deployment on CUDA did they woke up and came up with SPIR (later SPIR-V) and SYCL, which still isn't widely deployed.

shmerl7• 7 years ago

The performance possibilities of GPUs can be democratized by providing more high-level tools that are easy to use by a large community of applied mathematicians and machine learning programmers.

How exactly CUDA is "democratizing" anything, if it's tied to Nvidia? Vulkan backend would make more sense for that purpose.

rrss• 7 years ago

That sentence explains perfectly well what it means by democratizing, and how is independent of the platform being tied to nvidia.

shmerl7• 7 years ago

Can you elaborate please? I was under the impression that CUDA is tied to Nvidia, unless you mean there are now working shims for other GPUs.

pjmlp3• 7 years ago

It allows us to use any programming language with PTX backend.

OpenCL on the other hand is C FTW and now kind of supports C++ if one has luck with the drivers.

From that point of view is democratizing GPGPU programming to anyone that doesn't want to deal with either C or C++.

shmerl7• 7 years ago

That's not democratizing GPU computing, that's "democratizing" Nvidia lock-in. Totally different thing, so their claim was hypocritical, since they made it sound like a general thing.

maleadt2• 7 years ago

Author here, happy to answer any questions! We've been developing and maintaining this toolchain for a while now, so the relevant packages (CUDAnative.jl for kernel programming, CuArrays.jl for a GPU array abstraction) are much more mature. Our focus has recently been on implementing a common base of array operations that can be used across devices (GPU, CPU, etc), so that users can develop using the base CPU array type, quickly benefit from a GPU by switching to CuArrays, only to rely on specific CUDA-specific functionality from CuArrays/CUDAnative when they need custom functionality.

m4r35n357• 7 years ago

Julia is presented as a simple language, but is is anything but that in practice.

ddragon3• 7 years ago

Julia is not presented as a simple language, it's presented as a "I want everything" language [1], a Python-Ruby-Perl-C-Fortran-Lisp-Matlab crossover with it's own unique spice. Which is completely opposite from something like Go. You can start programming knowing only one of Julia's inspiration, for example programming Julia like Python, but if you want all the language brings you'll have to dive in a lot of the other sides (which might clash a little with the cleverness of the compiler, as it will accept such varied styles it will not guide you to the one through way of idiomatic Julia code).

Still the Julia team did a great job in making all those diverse features feel part of one connected philosophy instead of an ad hoc pile of functionality, even if it does take a little while to fully internalize it.

[1] https://julialang.org/blog/2012/02/why-we-created-julia

High-Performance GPU Computing in the Julia Programming Language (2017)

Related Stories