Intel will add deep-learning instructions to its processors

http://lemire.me/blog/2016/10/14/intel-will-add-deep-learning-instructions-to-its-processors/

by ingve • 10 years ago

389 125 10 years ago

Hugged. Google cache link: http://webcache.googleusercontent.com/search?q=cache:lemire....

sbierwagen2• 10 years ago

Blogspam, kinda. Post just links to https://software.intel.com/sites/default/files/managed/69/78... and says it mentions two new instructions: AVX512_4VNNIW (Vector instructions for deep learning enhanced word variable precision) and AVX512_4FMAPS (Vector instructions for deep learning floating-point single precision)

On Intel's part, it seems kinda... late? It's like adding Bitcoin instructions when you already know everyone's racing to make Bitcoin ASICs. How could it beat dedicated hardware, or even GPUs, on ops/watt? Maybe it's intended for inference, not training, but that doesn't sound compelling either.

ninjin• 10 years ago

I am not so sure if it is "late". Sure, for training models it is far too late, we have semi-dedicated hardware for that at this point. But it is possible that it will become increasingly common to run these models on end-user systems and in that case these instructions could provide a bit of a speed boost.

valarauca12• 10 years ago

>On Intel's part, it seems kinda... late?

Not exactly AVX512 is used on the Xeon Phi which is Intel's Floating Co-Processor for Deep Learning/OpenCL tasks. But that being said, GPU's blow the Xeon Phi out of the water in terms of FLOPS, local RAM, and RAM bandwidth.

astrodust5• 10 years ago

Isn't Phi better at dealing with branch-heavy code? GPUs absolutely kill on straight-forward compute, but they seem to flail helplessly if the logic is highly conditional.

I've read that in some cases GPUs evaluate both branches of a condition and discard the unwanted results. A CPU doesn't do this.

obmelvin2• 10 years ago

As the other poster mentions, I assume their view is that these types of operations will become increasingly prevalent in consumer software applications. Lots will be powered by the cloud, but it would be useful if some inference can be done on a laptop CPU.

Eridrus• 10 years ago

"The cloud" still has to run the models on something, and as long as Intel roughly keeps up with the cost of specialised hardware for doing inference, many people will just keep doing inference on Intel CPUs, rather than bothering with specialised hardware.

dsabanin4• 10 years ago

Intel is also adding FPGAs[1] right next to their CPUs, which can possibly become a next big thing in HPC.

[1] http://www.extremetech.com/extreme/184828-intel-unveils-new-...

yid3• 10 years ago

This aspect of their new chips is massively underrated. An FPGA is the future-proof solution here, not chip-level instructions for the soup-du-jour in machine learning.

Edit: which is not to say that I'm not welcoming the new instructions with open arms...

sargun• 10 years ago

Any idea when this is going to land?

snippyhollow• 10 years ago

[2014]

tankXos• 10 years ago

Might be a dumb question, but what is the implication of adding FPGAs[1] right next to their CPUs?

ericjang• 10 years ago

Not exactly late. There's no physical law that says a CPU can't beat a GPU for a certain family of operations, and in fact in practice, small toy networks with non-dense operations (for instance, if you are doing research on optimization algorithms for DNNs or Bayesian inference algorithms) tend to run faster on the CPU than GPU.

adevine• 10 years ago

I am not familiar with how deep learning algorithms would benefit from these instructions in a way different from how they can just use vector instructions optimized for graphics processing. So my question is, how are these "Vector instructions for deep learning enhanced word variable precision" different from other vector instruction sets?

sbierwagen2• 10 years ago

Good question. It doesn't say! The PDF has absolutely no detail on what it actually does. Googling the instruction brings up people updating CPUID fields, but nothing to do with actually using it.

vegabook• 10 years ago

disingenuous to use a highly charged term like "blogspam" and then qualify it as "kinda". kettle/black. Why say "blogspam" if you don't mean it?

And while I agree that the OP is not indepth, it is not blogspam. AVX512 is a very real, very very real, order of magnitude, performance improvement. To give you an idea, going from standard 64-bit to AVX-256 with MKL Numpy (under ipython) you will go from 6-7 seconds, to under 1 second, on an identical 2012+ processor, on this code:

  import numpy as np
  np.random.seed(1234)
  xx = np.random.rand(1000000).reshape(1000, 1000)
  %timeit np.linalg.eig(xx)

The best way to test this is to run the above on bog-standard Ubuntu 16.04 (sudo apt-get install python(3)-pip / sudo pip(3) install numpy first), and then run it again after installing Anaconda Python.

AVX512 will halve this timing again, and before you tell me that Cuda will do this much faster, I'll remind you that there is a point at which 4000 cuda stream processors operating on 64bit data can, in many algos, be a vastly too large window. Phi KL's 72 branching cores with 4 threads and 2x512-bit AVX each, can be a very strong competitor to the flatspace GPUs in nonlinear algos. There is clear blue sky between the scalar architectures and the massively parallel architectures, which has not been targeted yet and this is where KL fits in. After all, the GPUs are still, first and foremost, designed to rotate and translate massive-amounts of triangles in 3d shooters, a task for which a vast number of very dumb little cores is optimal. As much as Nvidia make like you to believe different, that is still their primary market and it doesn't always map well to everything.

Between the current extremes of a small number of very flexible and clever cores (xeon/i7), and 4000 very stupid but numerous cores (GPU), there is a lot of room something in between.

Intel isn't so much late to the party as attacking an obliquely different segment of the market, which may prove very interesting indeed.

wpietri• 10 years ago

On Intel's part, it seems kinda... late? It's like adding Bitcoin instructions when you already know everyone's racing to make Bitcoin ASICs.

Nah. You're thinking they have to be the best, that they have to win. That is true when the choice is buying an Intel CPU vs buying a GPU. But most developers will already have a late-model Intel CPUs around.

For Intel, this is a defensive move. They're bending the performance curve for particular use cases. All this has to do is to keep a certain number of developers from bothering to go the extra mile to use special hardware and software. People working at Google's scale won't care. But there will be other people who try running something and say, "Current hardware's fine. We could do better with GPUs, but not so much better that it's worth the time and money."

I'm sure it's also the start of a learning cycle for them. Some people told them their current processors were not enough, so they added this. They'll now hear why this isn't adequate either and adapt again if the business case is there.

joelthelion• 10 years ago

So Intel CPUs will be able to perform 16 operations at a time, when Nvidia can do thousands?

sitkack• 10 years ago

Latency

j1vms• 10 years ago

Latency

This. Also, trying to slow the loss of customers purchasing what are essentially Nvidia GPU-heavy systems with Nvidia CPUs tacked on to act as the controllers.

wmu2• 10 years ago

On Knights Landing CPU a single core is be able to perform 2 x 16 operations on single-precision numbers, as a core has two AVX512 execution units. Each core can run 4 thread, and KNL has 64 physical cores.

visarga• 10 years ago

Intel is hoping that the Xeon+FPGA package is enticing enough to convince enterprises to stick with x86, rather than moving over to a competing architecture (such as Nvidia’s Tesla GPGPUs ).

Then let it prove its performance on a few deep learning benchmarks. If is so fast and accessible, we'll be impressed.

antirez6• 10 years ago

That's a good news! About that, two days ago I modified the implementation of neural networks inside Neural Redis in order to use AVX2. It was a pretty interesting experience and in the end after modifying most of it, the implementation is 2x faster compared to the vanilla C implementation (already optimized to be cache obvious).

I never touched AVX or SSE in the past, so this was a great learning experience. In 30 minutes you can get 90% of it, but I think that to really do great stuff you need to also understand the relative cost of every AVX operation. There is an incredible user at Stack Overflow that replies to most AVX / SSE questions, if you check the AVX / SSE tag you'll find it easily.

However I noticed that when there were many load/store operations to do, there was no particular gain. See for example this code:

    #ifdef USE_AVX
            __m256 es = _mm256_set1_ps(error_signal);

            int psteps = prevunits/8;
            for (int x = 0; x < psteps; x++) {
                __m256 outputs = _mm256_loadu_ps(o);
                __m256 gradients = _mm256_mul_ps(es,outputs);
                _mm256_storeu_ps(g,gradients);
                o += 8;
                g += 8;
            }
            k += 8*psteps;
    #endif

What I do here is to calculate the gradient after I computed the error signal (error * derivative of the activation function). The code is equivalent to:

    for (; k < prevunits; k++) *g++ = error_signal*(*o++);

Any hint about exploiting AVX at its max in this use case? Thanks. Ok probably this was more a thing for Stack Overflow, but too late, hitting enter.

dfbrown• 10 years ago

I would try unrolling 2-4 iterations of the loop. Multiple sequential loads isn't much slower than a single load, so batching your loads and stores together will let you do more arithmetic operations for each time you hit memory.

YZF2• 10 years ago

It might help to try and squeeze more work between the load and the store. E.g. do two of these in parallel (load/load/mul/mul/store/store). Think about latency vs. throughput, the load and the stores may have high latency but also high throughput so you need to be issuing more of them concurrently (without any dependencies between them). Just off the top of my head.

nkurz5• 10 years ago

Actually, in most cases instruction ordering isn't really a factor on modern Intel processors. They operate "out of order" (often abbreviated OoO), which means that they execute instructions (µops really) from a "reorder buffer" as soon as dependencies are satisfied. Since the loads have no dependencies, they will already execute several (possibly many) iterations ahead of the multiplies and stores.

The latency of the store doesn't matter to use, since we aren't immediately using the result. This doesn't mean that you won't see a benefit from your suggestion, but if you do it will probably be from the reduced loop overhead rather than the improved concurrency.

makmanalp• 10 years ago

I'm not the most experienced, so I may be talking out of my ass.

I imagine that what you're hitting is IO. Getting stuff from lower levels of the hierarchy is slow. To benefit from vector ops, you want your operators to be already in registers. The question is, I wonder whether you're hitting the bandwidth limit of that operation, basically. If not, I wonder if there is something that helps you help the hardware prefetch.

You might also want to try and see if unrolling the loop helps, gets rid of the iteration conditionals, but I guess you're adding more instructions, so perhaps more instruction cache misses.

Also, are the pointers to where you're grabbing the block of vectors aligned? Looks like it could be. Try something like _mm256_load_ps perhaps.

antirez6• 10 years ago

The problem I've is exactly lack of alignment... unfortunately why the vectors themselves are aligned, the way I need to jump inside the vectors is not... Padding would complicate quite a bit the implementation so I was not sure if implement it or not, also I'm not sure about the effects on the cache... I wonder what's the penalty of loadu vs load, maybe that's the problem indeed. Thanks.

nkurz5• 10 years ago

This should be a very fast loop. If the input and output are larger than cache, you should be limited by RAM bandwidth. And if your outputs and gradients fit in L1 cache, it should be possible to get your loop down to single cycle per iteration: load, multiply, store, increment, and test can all execute in the same cycle. It will probably be difficult to convince a compiler to produce the necessary code, though, and if you do, it will likely be specific to that particular compiler and flags.

If you want to make this as fast as it can be, the first step is to measure what you do have, and see how fast it is in terms of cycles per iteration. Then you'll want to look at the generated assembly, and what the compiler has done to your carefully crafted loop. The better you've optimized it to begin with, the more likely that it will have done something to make it worse. If you want guaranteed performance on your target platform, you need to lock down the assembly.

The fastest loop structure here will involve using a single negative offset, so that you can increment it by 8 and test if you have reached zero. This allows the addition and loop test to "fuse" into a single µop (ADD-JNZ). The loads will be (end_outputs + neg_offset), and the stores (end_gradients + neg_offset). Unrolling should not be necessary if you can get the right assembly.

In the absence of making it "perfect", though, you may benefit by unrolling it a few times, although your compiler is probably already doing this. You might try to encourage the compiler to generate a single instruction "load-multiply" instruction by using a dereferenced pointer in the multiply. The processor is limited to decoding 4 instructions per cycle, and depending on what the compiler produces, saving an instruction per iteration here might avoid this bottleneck.

And as someone else mentioned, it will help if your inputs and outputs are 32B aligned. In particular, stores that cross 4KB boundaries can be expensive enough to slow down the whole algorithm (although I think this has gotten better on Skylake). It shouldn't be a big difference, though, unless everything else is extremely optimized. I explore it here: http://www.agner.org/optimize/blog/read.php?i=415#419

antirez6• 10 years ago

Thanks you for the interesting advices. Unfortunately no 32B alignment. The start address of the weights vectors is aligned, but the problem is that as I jump from here to there to get the weights, the alignment is broken. I bet that if I analyze the way I loop trough the weights, I can figure out a way to make this whole thing a linear aligned access. Probably that's my starting point! Thank you. I'll make sure to try different cycles numbers and unrolling, I don't want to reach the level of writing the assembler, but to have at least a good speedup for what's possible.

About RAM bandwidth, basically it depends a lot on the neural network size, for small/medium networks it's all inside L1, for very large not, and of course the bigger they are, the slower to train, the more interesting to get the speedup. Thanks.

qb454• 10 years ago

However I noticed that when there were many load/store operations to do, there was no particular gain. See for example this code:

You may be running out of cache/memory bandwidth, compare your throughput with numbers from benchmarks or "Software Optimization Manual" of your CPU. If you are on Linux, try

  perf stat -d ./whatever

which reports, among other things, the rate of cache and memory hits, obtained from hardware "performance counters".

_mm256_loadu_ps

Some years ago and with SSE, unaligned was slower than aligned. On some chips only if the address happened to really be unaligned, on others in every case. Not sure how it is with AVX.

zbjornson2• 10 years ago

On recent processors (last few years), as long as you're not crossing a cache line, unaligned is essentially the same as aligned in terms of performance.

gok3• 10 years ago

Assuming this is your code: https://github.com/antirez/neural-redis/blob/master/nn.c

You're really just implementing a vector*matrix multiply. You probably want to just use BLAS's sgemv routine. On macOS, link Accelerate and use cblas_sgemv(); on Linux consider installing Intel MKL or OpenBLAS.

If you're just looking to learn what a reasonably state-of-the-art SGEMV kernel looks like for a modern chip like Haswell, check out OpenBLAS's code:

https://github.com/xianyi/OpenBLAS/blob/develop/kernel/x86_6...

ajtulloch• 10 years ago

https://github.com/Maratyszcza/NNPACK/blob/master/src/x86_64...

(from https://github.com/Maratyszcza/NNPACK) is also a pretty reasonable AVX2 GEMM implementation written in a Python assembler, it's a bit easier to follow than the OpenBLAS kernels and has very reasonable performance vs OpenBLAS/MKL.

I second the guidance to just use a BLAS library. I'd guess you'd see a 2-10x speedup at least. It looks like you're only implementing a classical MLP library (which is fine for learning, but not the best choice for something like this, integrating VW into Redis would be a much better choice IMO).

antirez6• 10 years ago

That sounds a great advice, I'll check this library. I've the feeling that the fact I've non aligned addresses in the current weights scheme will be a problem and padding will be required. That's why I used AVX "loadu" that deals with non aligned addresses, but perhaps I'm paying a lot of performances because of this. Thanks.

EDIT: apparently on modern CPUs that's not the case and magically LOADU can be as fast as LOAD.

happycube• 10 years ago

Not holding my breath, since they haven't gotten AVX512 into desktops yet! (Kaby Lake might be worth something if they did... but nope, just more lie7's mostly...)

unsignedqword• 10 years ago

Cool stuff, but unfortunately Intel has really delayed AVX512 instructions for their main consumer processors (ffs, it was supposed to hit on Skylake). It looks like we have another die shrink to go after Kaby Lake before we get that sweet ultra-wide SIMD:

https://en.wikipedia.org/wiki/Cannonlake

p1esk3• 10 years ago

Skylake Xeons are still on track to have AVX512, last I checked.

jostmey• 10 years ago

How much will this help? Neural networks work so well on distributed computing systems because the operations run in parallel. With a neural network, you can leverage tens of thousands of separate computing cores. Top intel processors have around a dozen cores or so.

What am I missing here?

wmu2• 10 years ago

Xeon Phi have got more cores, 50 or 60, and the cores share memory. Unlike GPUs memory space is uniform, there is no need to do explicit transfers to/from GPU's memory.

arcanus• 10 years ago

Xeon Phi have got more cores, 50 or 60, and the cores share memory

60 cores. Furthermore, the cores have multiple (4) hardware threads per core. So on the latest Phi (Knights Landing) you can have several hundred threads (240) running concurrently without contention (in principle).

steego• 10 years ago

What am I missing here?

There may be a market sweet spot between dedicated high-performance machine learning that's ideal for experts who are willing to buy/lease dedicated hardware and regular developers who want to use simpler machine learning toolkits to solve easier problems. Toolsets like Julia, NumPy, R, etc would take advantage of the new instruction and basically give your average a speed boost for simply having an Intel PC.

eb0la• 10 years ago

This reminds me when you had to buy crypto accelerator cards for your webservers (does anyone remember Soekris?)...

The market stalled after Intel started adding support for crypto operations on-die.

This looks like Intel is protecting his server chip business shooting at nVidia before they start selling Titan-X with other server chip (what chip - I don't know; but I bet there's a business plan on a spreadsheet somewhere).

I guess training neural nets will a GPUs / custom chips task for some time; but as soon as you have developed the classifier / predictor you have to run it on whatever hardware you have and that means you might need less parallel computing power and it doesn't make (financial) sense buying GPUs for that.

gok3• 10 years ago

Let's say you want to run a service that does scene classification in images using deep convolutional networks. It takes 100ms on an Intel chip today, and 3ms on a GPU. But you can't run your entire service on a GPU...you're going to need that CPU anyway. So if Intel can close the gap even a little bit (say, down to 25ms), it might be worth it to avoid the headache of running a GPU server and just buy more Intel chips, which can run anything.

zenobit256• 10 years ago

Great, another extension to the ever-growing instruction x86 instruction set...

I'd question why, but at this point, we're just going to keep bolting things on to an existing architecture.

"Microcode it. Why not?"

partycoder• 10 years ago

You can also try a binarized neural network.

https://arxiv.org/abs/1602.02830

eveningcoffee• 10 years ago

I did tests with Torch7 few days ago to compare it with OpenBLAS library on multiple cores against CUDNN.

Single core difference was around 100x.

Using multiple cores improved the situation but there still was about 15x difference that did not improve by adding more cores (I tested with up to Xeon 64 cores on a single machine).

I did not test with Intel MKL library as I did not have time for this.

I wonder how much these instructions would improve the situation and does anybody here have experience with Intel MKL library.

AvenueIngres• 10 years ago

What are the implications of this? Faster processing?

gragas2• 10 years ago

TL;DR: You can basically do 8 operations at once (but of course, it's not that simple).

Think about a normal 64-bit register: you can add and subtract 64-bit numbers, OR and AND them, etc. Now think about a 64-byte register. What could you do with that? Well, suppose you're looping through contiguous memory and changing each word AND the change that your making to each word is independent of the change you make to any other word. With a normal 64-bit register you'd have to do one operation on each word. But with a 64-byte "register", you can load 8 words into it and do just one operation for the same effect as applying that operation 8 times (once to each word). Thus, some code---vectorizable code---can be sped up nearly 8x.

zbjornson2• 10 years ago

Really curious how Intel (others?) determined that these new instructions are for deep learning. The instructions are for permutation, packed multiply+low/high accumulate, and a sort of masked byte move. Are these super common in deep learning?

p1esk3• 10 years ago

Deep learning is all about GEMM operations: matrix-matrix, or matrix-vector multiplications. And the values in these matrices are typically low precision (16 or even 8 bits is enough). So if you can pack many of such low precision values into vectors, and perform a dot product in parallel, you will get a pretty much linear speedup.

schappim2• 10 years ago

There is actually an undocumented neural net engine within the Intel chip on the Arduino 101.

schappim2• 10 years ago

I guess it's documented now: http://www.general-vision.com/products/curieneurons/

yongjik• 10 years ago

Apologies if I'm dumb, but did Intel actually tell exactly what operations these "deep learning" instructions do?

I skimmed through the linked Intel manual, but it seems that it just defines two instruction family names (AVX512_4VNNIW and AVX512_4FMAPS) without actually saying what they do.

* It almost feels like a marketing term, in the same way everything was "multimedia instructions" back in the 90s.

iaw• 10 years ago

It's been a long time since I look at machine language but, if I understand correctly, the difference is the capacity of the operation. Instead of performing an addition operation on 1 or 2 bytes at a time the operation can be performed on 64 bytes at a time. Given that 'deep learning' requires a number of parallel simple operations expanding the volume of parallel operations in a single CPU instruction boost capacity.

Seriously, just supposition until someone with better understanding chimes in.

bradhe• 10 years ago

heh, if this is the case then we're going to benefit in a lot of areas from this.

denim_chicken• 10 years ago

I hope they will be more useful than MPX.

ktamiola• 10 years ago

Sounds good to me. Let's hope this is not just a marketing move.

betolink• 10 years ago

Is this the moment when Sarah Connor shows up to ruin the party?

rosstex• 10 years ago

Could someone explain how these are deep-learning optimized? All I see is that they're part of the 512-bit vector family that has previously existed.

tomrod• 10 years ago

Well. I guess I'm going to be using Intel!

gragas2• 10 years ago

Isn't this basically just wider SIMD? Is this in anyway actually tailored specifically to deep learning?

sunstone• 10 years ago

"get me a beer' would be an appreciated instruction to add.

thasaleni• 10 years ago

Excuse my ignorance, but isn't "deep learning instructions" an oxymoron

Intel will add deep-learning instructions to its processors

Related Stories