CuPy Accelerates NumPy on the GPU? Hold My Cider, Here's Clojure

https://dragan.rocks/articles/20/Clojure-Numpy-Cupy-CPU-GPU

by signa11 • 6 years ago

120 46 6 years ago

dragandj12• 6 years ago

Uses free Clojure library Neanderthal: https://github.com/uncomplicate

Also uses the book Numerical Linear Algebra for Programmers: https://aiprobook.com/numerical-linear-algebra-for-programme...

akssri8• 6 years ago

The benchmark is flawed. Cupy follows Numpy in doing a type-promotion of the data to float64 before computing the covariance.

https://github.com/cupy/cupy/blob/b19577199d3f75c76e23c75194...

The following function in PyTorch completes the float32 computation in 27 ms (1080 Ti). I'm fairly sure that a similar function in Cupy would achieve very similar results.

  def corr(dat):
      cov = (dat @ dat.T).div_(dat.shape[-1]-1)
      sigma = torch.diag(cov).sqrt_()    
      return cov.div_(sigma[None]).div_(sigma[None].T)

The heart of the computation SGEMM is in anycase provided by CUBLAS, and the only areas in which PyT and Cupy differ are in the tricks used to unroll the loop for the broadcasting & L1 op. The performance difference b/w PyT, Cupy and others should be negligible IMO.

Note: np.corrcoef takes about 1.12 s and the above float32 version takes 325ms on the CPU.

dragandj12• 6 years ago

So, you first state that the benchmark is flawed, then you confirm the results of the benchmark, and repeat everything that the OP describes as the solution, just with a different tool (PyTorch instead of Clojure)?

BTW did you check that your corr function gives the correct result? It looks to me that you forgot a few steps.

akssri8• 6 years ago

The PyT version above and the both the CPU/GPU versions on Neantherdal work completely in float32. I confirm that the performance of Neantherdal GPU is similar to that in the above PyT version. Yes.

The part that is flawed is that you're comparing this to Numpy (CPU)/ Cupy (GPU) both of which coerce the input array to float64 (for precision reasons) before computing the covariance and correlation. You only need to check the output type of the result to verify this (if the pointer to the code is not sufficient).

jsdavo• 6 years ago

Thanks for sharing! Why does CuPy switch to float64 in this case? Is it a bug?

akssri8• 6 years ago

Numpy/Cupy do explicit coercing to float64. There is no documentation for why this is done, but since GEMM is used to compute the covariates (summation over data points), it makes sense to increase the precision.

You could get away using a double to only accumulate the sum, but it's a pain to write such mixed-precision (slow) function in C and then wrap it in Python, esp. for something like this.

(See, https://github.com/numpy/numpy/blob/v1.17.0/numpy/lib/functi...)

fluffything• 6 years ago

CuPy is a drop in replacement for NumPy. Ask Numpy why they do that.

didibus6• 6 years ago

I'm not sure it is flawed. It just seems that you found the root cause of why NumPy/CuPy are slower, but it seems there is no way as a user of NumPy/CuPy to perform the float32 computation, even though you were explicit and said `:dtype "float32"`.

All these libraries under the hood delegate to the same things for the actual computation, so the only advantage of one over the other would be this type of stuff.

That NumPy/CuPy coerce to a float64 under the hood, and gives us no way to force it not to do so, is a disadvantage that they have in my opinion.

In any case, like the article says, the performance of NumPy/CuPy is still good enough, and maybe NumPy thinks it knows better then it's users that they should be using higher precision always.

Edit: I know I'll face the brigade of NumPy users. But I invite people to engage in the discussion unbiased. With a switch from CPU to GPU you'd expect more than 29% performance boost. If CuPy went with a float32 proc, it would give you a 96% performance increase! So ya, I think we can start to argue CuPy isn't doing the right thing here.

akssri8• 6 years ago

user of NumPy/CuPy to perform the float32 computation

This is just getting tiring.

Numpy and Cupy are perfectly capable of doing float32 computation - their only "fault" is that they coerce data to float64 in this one fairly unimportant function (which you can implement to your liking in 3-4 LOC). Hell, PFNs entire deep learning system, Chainer (which mind you, both predated and inspired PyTorch, and is still quite competitive with it), is built entirely on top of Cupy!

Benchmarks make sense only when the outputs are the same - in this case, they certainly are not. It's the responsibility of the author to make sure that the outputs are the same (a real benchmark), or to argue that Numpy/Cupy are wasting time by using float64 in xp.cov (an issue of implementation). He does neither.

xvilka• 6 years ago

Or just use Julia where the speed is out of the box.

bearzoo3• 6 years ago

Oh? I don't suppose that julia wraps lapack functions for no good reason. What makes you say this

ChrisRackauckas• 6 years ago

A lot of the wrappers are largely legacy at this point. BLAS1 and BLAS2 operations are handled in pure Julia through the broadcast system, there's no performance degradation there and the fusion of operations actually makes for a lot of performance advantages over simple BLAS calls. For BLAS3, tools like LoopVectorization let you write a pure BLAS3 in Julia and get good performance, Gaius.jl is a good example of this by matching OpenBLAS with pure Julia code. Of course you still have to write out how to do blocking, but that's about it. At this point it's mostly needing more coverage before deprecating BLAS bindings in a Julia 2.0 down the line.

https://github.com/MasonProtter/Gaius.jl

chenzhekl• 6 years ago

CuPy is not developed by NVIDIA, but by Preferred Networks, the company behind Chainer.

rrss3• 6 years ago

Power users even know that Nvidia provides a drop-in replacement, CuPy, that brings the power of GPU acceleration.
but Nvidia provides CuPy
It's Nvidia's stuff

Is CuPy actually developed by Nvidia? It looks like the original authors were at Preferred Networks, and none of the top few contributors on github work at Nvidia.

eslaught• 6 years ago

If anyone's looking for NumPy that works on distributed machines with GPUs, see also Legate, which is produced directly by NVIDIA:

https://research.nvidia.com/publication/2019-11_Legate-NumPy...

https://developer.nvidia.com/legate-early-access

Disclaimer: I work on Legion, the runtime system that Legate sits on top of, but otherwise don't have anything to do with the Legate project.

hellofunk2• 6 years ago

One of the things I never understood about the hype around these Clojure GPU libraries is that there is a lot of marketing jumbo about how high-level it all is. Even the book for these libraries has the words "no C++!" half a dozen times on its ordering page [0].

However, these are just high-level libraries and you cannot write the GPU shaders in Clojure. You still must use C or C++ for that (depending on if you are going the CL or Cuda route), and your C/C++ code is embedded or called from the Clojure side. This is no different than using another high-level language like Python -- I mean it's focusing on a rather misleading and irrelevant detail. You can use Clojure or Python (or many other languages) to call OpenCL or Cuda shaders, or you can just use the libraries that call them for you. Clojure is not special in this regard.

[0] https://aiprobook.com/deep-learning-for-programmers/

dragandj12• 6 years ago

The book you've linked contains exactly 0 lines of C or C++ code. 100% of the code in the book is Clojure.

hellofunk2• 6 years ago

I think you missed the point of my comment. You don’t have to write your own C++ code when using high-level python GPU libraries either.

didibus6• 6 years ago

Hum.. I'm not sure there's that much hype. It's just saying that Clojure can also be used as a high level layer for fast linear algebra. I do see some claims that as a high level layer, Clojure works even better, it gives you a full REPL, and the power of macros and all that.

It also goes to show you that there's barely any overhead added by the Clojure layer, which is something you want from a higher level layer.

Maybe you need some additional context, but in the Clojure world, there are many offerings for that layering, core.matrix, neanderthal, NumPy, CuPy and others can all be used. So I guess a lot of what draganj is trying to show is he believes neanderthal is the layer with the least overhead, which everyone can agree to disagree on.

You're never going to beat an assembly level implementation, or 50 year's worth of optimization done to a Fortran implementation, or straight openCL/cuda implementation in a higher level language, especially not in Clojure. So in the market of high level languages like Julia, Clojure, Python, R, Swift, etc. and their corresponding libraries, they all are just competing on being more user friendly, having better more useful/flexible abstractions, a better development flow, and the least amount of overhead. So that's what you'll be hearing them all fight over.

akssri8• 6 years ago

Common Lispers have had the ability to write CUDA kernels (with shaders) on-the-fly with a DSL in cl-cuda for many years now. See,

https://github.com/takagi/cl-cuda

Cupy (a project in which takagi is/was actively involved) also has the ability to compile kernels in the Python REPL, but the kernel code needs to be written in C and passed in as a string (pyOpencl can do the same for OpenCL). In fact, not too many years ago, the kernels for gradient descent, max pooling etc. were all entirely in Python in Chainer. Projects like JAX take this to another level by having the ability to transform the bytecode of a restricted class of Python functions straight into CUDA kernels.

I can't speak much about Clojure/Neanderthal, but I'd advise the people to stop dissing projects they are not familiar with, least of all using silly benchmarks like the above.

dragandj12• 6 years ago

Good advice. And, yet, you've taken it pretty seriously to diss Clojure/Neanderthal and my blog post, mostly by talking about unrelated stuff and projects that the post didn't even mention. And while introducing these themes left and right you didn't even bother to show some code related to this topic, just a suggestion of great projects by cool people.

Yes, you showed the PyTorch code related to the blog post that confirms what the blog post says, but when I pointed out that the code has incorrect functionality (by missing some calculations) you didn't even bother to correct it, or to confirm that the code is good and that I'm wrong.

So, it seems that your standard is that it is enough for one side to throw bits and pieces around and call it a day, and for the other to run around and prove that their stuff is better than everything that could possibly be done in every technology.

I choose to stick to the theme. The theme is CuPy, NumPy, Clojure & Neanderthal. The related theme could be code in another technology. Great - write about it. But, even if every other technology were a million times better than what I describe in the article, it does not change the fact that CuPy and NumPy have the issue I've described.

bernardv• 6 years ago

Great article. I learned a bunch of things I didn’t know about Numpy and Clojure. As far as I’m concerned the Dragan does rock.

CuPy Accelerates NumPy on the GPU? Hold My Cider, Here's Clojure

Related Stories