Accelerating Neural Networks with Binary Arithmetic

https://software.intel.com/en-us/articles/accelerating-neural-networks-with-binary-arithmetic

by pplonski86 • 9 years ago

99 12 9 years ago

software.intel.com

quotemstr• 9 years ago

It's been known for a while now that quantization of neural networks greatly increases efficiency while not decreasing inference power much --- NNs are great at dealing with noise, and squashing a float weight into a byte is just adding noise. These people seem to have quantized "all the way" as it were, which makes sense.

yigitdemirag• 9 years ago

IBM's TrueNorth does the same, it is a common practice in neural network chips for inference. I think doing that kind of tricks for training is much more valuable. Doing training at hundreds of GPUs and inference with chips at mobile, can only be a temporary solution for deep learning market.

deepnotderp2• 9 years ago

One thing that people conveniently forget when discussing binary nets is that they take a steep hit in object detection accuracy.

ktta2• 9 years ago

How much longer do they have to be trained (the article talked about this but didn't give numbers)? Do they need more data than neural nets with FP ops? Any estimates?

One thing that looks attractive to me is that FPGAs can be used very effectively for this purpose because these will take up much less LUTs and nets for specific purpose can be manually programmed in HDL.

regularfry2• 9 years ago

You don't even need FPGA's for this sort of network, necessarily. Take a look at how WISARD worked: https://scholar.google.co.uk/scholar?q=igor%20Aleksander%20W... It's "just" a bunch of RAM.

deepnotderp2• 9 years ago

It's not just being trained longer, that in itself isn't too much of a worry, but rather the issue is that they aren't very good even after retraining, object detection accuracy is essentially the cost of extreme quantization.

For example, see Deephi's presentation at HotChips.

tc• 9 years ago

Geoffrey Hinton described doing this back in 2012. He noted that it learned more slowly but generalized better, he described how this is nearly isomorphic with dropout, and he presented a biological motivation:

https://www.youtube.com/watch?v=DleXA5ADG78#t=45m32s

https://www.youtube.com/watch?v=DleXA5ADG78#t=4m24s

wpower32• 9 years ago

I'm new to ANNs. Is the slowdown in training due to the fact that the activation functions have such a sharp shape? That is the binary version of activation doesn't look like it has a nice differential to use in something like gradient decent.

deepnet• 9 years ago

Binarisation is not for backpropagation which requires a gradient (binary is a cliff) for the calculus backprop to work - this gradient is provided by the slope of the sigmoid.

The OP is only suggesting binarisation for the feedforward use of nets, not the learning and training phase.

from the OP : " Real valued gradients are required for SGD to work. The weights are stored in real valued accumulators and are binarized in each iteration for forward propagation and gradient computations. "

wpower32• 9 years ago

Awesome, thanks.

Accelerating Neural Networks with Binary Arithmetic

Related Stories