Intel Gets Serious About Neuromorphic, Cognitive Computing Future

https://www.nextplatform.com/2017/02/11/intel-gets-serious-neuromorphic-cognitive-computing-future/

by Katydid • 9 years ago

149 52 9 years ago

paulsutter3• 9 years ago

The article misses the core issue: neural network architectures are still in flux. "Neuromorphic" chips are hardwired to one architecture, which makes them power efficient but less flexible. When designs are more stable, such chips could be more practical.

Meanwhile, upcoming (non-neuromorphic) AI processors are taking two directions: larger numbers of simplified GPU-type cores (such as NVIDIA Xavier and Intel's Lake Crest/Nervana chips), and FPGAs.

Simplifying cores means lower precision, as fp32 and fp64 are overkill for neural networks and take up lots of silicon. The current NVIDIA Pascal added fp16 and byte operations such as the DP4A convolution instruction[1]. Even smaller precision is practical (down to 1 bit with XNORnet[2], and the DoReFa paper[3] gives an excellent summary of the falloff in accuracy through 32-8-4-2-1 bits for weights, activations, and gradients).

[1] https://devblogs.nvidia.com/parallelforall/mixed-precision-p...

[2] XNORnet, https://arxiv.org/abs/1603.05279

[3] DoReFa, https://arxiv.org/abs/1606.06160

anigbrowl• 9 years ago

Seems like the sort of thing FPGAs could be good for. I have a few synthesizers that use them; elaborate feedback loops are a staple of sound design.

deepnotderp13• 9 years ago

XNOR and DoReFa take pretty big hits though, and they can't be easily trained without full precision, or at least "shadow weights".

paulsutter3• 9 years ago

For XNORnet, yes, but DoReFa has training with low bitwidth activations[1]. I don't cite these papers as proof that few bits are needed, but as evidence that precision is an open question.

[1] "We propose DoReFa-Net, a method to train convolutional neural networks that have low bitwidth weights and activations using low bitwidth parameter gradients. In particular, during backward pass, parameter gradients are stochastically quantized to low bitwidth numbers before being propagated to convolutional layers" (from the abstract)

emcq8• 9 years ago

What's wrong with training in full precision? There's a huge market for efficient inference in robotics, devices, and autonomous vehicles.

deepnotderp13• 9 years ago

This is a common fallacy, although the architectures themselves may be changing (quite a bit!), the basic computationally intensive operations, such as convolutions and matrix multiplies aren't. The simple way is to switch from FP32 to 16-bit fixed point, and you're good to go, and you just saved almost 10X power. This is the strategy Nervana/Intel, even Nvidia, and other startups such as Wave are pursuing.

paulsutter3• 9 years ago

Exactly - evolving network architectures suggest using a non-neuromorphic design (such as Xavier, Lake Crest), which trades less precision for more cores given the same power/real estate.

Neuromorphic designs like IBM's True North are more hardwired, and that's the limitation towards general purpose use. Yann LeCun's remarks on True North:

https://www.facebook.com/yann.lecun/posts/10152184295832143

pizza• 9 years ago

see: discussion for "Why are Eight Bits Enough for Deep Neural Networks?" 511 days ago

https://news.ycombinator.com/item?id=10244398

russdill• 9 years ago

I think the Intel neuromorphic computing will be a combination of 3D XPoint and technologies gained by purchasing Altera, including routing, DSPs, and reconfigurablity.

emcq8• 9 years ago

For training networks, designing en efficient chip is a scary proposition.

However for inference tasks, low precision GEMM (as you said) goes a long way and better than what you often get. That's why chips like Movidius' Myriad are getting popular and are more similar to DSPs than neuromorphic designs.

I agree that Intel's neuromorphic group doesn't get it, but other groups have taken neuromophic design principles that lead to efficient designs. For example, TrueNorth is very low precision, has great data locality, and though it was designed over 5 years ago can still use modern convolutional networks only imagined afterwards [0]. But its silicon implementation is not very brain like.

[0] https://arxiv.org/pdf/1603.08270.pdf

m3kw9• 9 years ago

Seem like a fancy name for parallel computing chip that specialize in efficient parallel computing

adevine• 9 years ago

It's that what it actually is? Not familiar with the field, so is "neuromorphic chip design" really "we wired a bunch of GPUs together?"

Don't mean to minimize the work involved, just trying to decipher the marketing speak.

DesiLurker• 9 years ago

neuromorphic chips try to mimic human brain's neuronal structure for computing, entirely new paradigm. checkout IBM truNorth architecture.

mrstone• 9 years ago

They are a chip architecture that communicates similarly to how neurons do - i.e. spiking behaviours. Makes it easy to approximate some neural systems (such as a neuromorphic retina).

pzone• 9 years ago

I think the point is that even GPUs are suboptimal for neural network computation and we need something more specialized.

justinpombrio• 9 years ago

Could someone explain what a neuromorphic chip is?

The article assumes we know, but I haven't heard of it. And the wikipedia article on "neuromorphic engineering" talks about stuff like analog circuits, copying how neurons work, and memristors, none of which seem that related.

mattlevan• 9 years ago

All of the subjects you mention at the end of your comment, especially memristors, are indeed the focus of neuromorphic computing. It's a nascent technology field, but there are plenty of papers available on IEEExplore or ACM Digital Library to satisfy curiosity! I'll take a look at the survey paper I wrote a couple summers back-- if it's decent I'll edit this post with a link.

Edit, links: Paper: https://docs.google.com/file/d/0B7QHR9a8j1iiU3RxSHZSNFh2cEdv... Slides: https://docs.google.com/file/d/0B7QHR9a8j1iiSE1ET2ZTb09aNFBP...

mljoe• 9 years ago

The best definition I can come up with is hardware that implements neural network architecture directly, especially McCulloch–Pitts spiking neurons (which have a temporal component). In neuromorphic chips, neurons are an actual component in the hardware, you can ask questions like "how many neurons does this chip have?". Contrast to how neural nets as we use them today which are actually implemented as a computation graph on tensors. It turns out that a special kind of neural network can be abstracted well as a series of tensor ops (dense feedforward layered networks[1]), but this is not necessarily the case for any neural network. So neuromorphic chips have a possibility of being far more general.

[1]: Which are wired something like this: http://neuralnetworksanddeeplearning.com/images/tikz40.png - Notice that dense connections and layered architecture. For all intents and purposes, this what neural nets look like today because of how easily it is to treat a NN with this specific wiring as a chain of tensor computations and thus execute on more conventional hardware.

return02• 9 years ago

Loosely defined, it's various chips that implement simplified computational models of neurons, and some plasticity functions. They are usually simplified because going into the greatest detail (modeling hodgkin-huxley type channels) would require too much computation. In the neuroscience community there is not yet an acceptable model for a simplified neuron so everyone picks some spiking neuron model or they make up one. Even less is known for plasticity functions.

I believe the idea is that if you simulate too many of them, something useful will happen.

nomailing• 9 years ago

What I remember from my neuromorphic engineering course (or analog VLSI course) is that we designed the silicon layout (with n and p doping regions) in a way that the transistors are operating in the subthreshold regime in the IV characteristics. If I remember correctly the IV characteristic is linear in the subthreshold region? In contast in normal digital chips only the super-threshold region is used (voltage above a certain saturation threshold switches the transistor completely on). Using the subthreshold region it is possible to implement spiking neuons with only very few transistors. It works completely different than digital circuits. The connections between the transistors don't transmit just 0's and 1's. Instead all wires transmit analogue signals where the exact voltage matters. This makes these chips extremely energy and space efficient. These chips can also work much faster even in comparison to biological neurons (obviously using some assumptions and simplifications, such as neglecting certain special kinds of ion channels found in real neurons).

petra• 9 years ago

Theoretically , Analog is by far the best for neural networks. But why aren't we starting to see chips offered ? Heck even an old process like 130nm could have some practical uses .

bobsil12• 9 years ago

Asynchronous, analog, spike-based, co-located memory. Closer NN sim than von Neumann architecture, orders of magnitude more power-efficient.

One human brain-equivalent NN on classic architecture costs ~$70M and uses ~100 houses worth of power.

alexmlamb22• 9 years ago

The rigor of this claim is very questionable. I assume it's based on neuron == hidden unit.

In any case though my guess is that a human neuron accomplishes way more than a hidden unit in a NN so it may be fair to view that as a lower-bound.

espeed• 9 years ago

Are these the Neuromorphic chips [1] Jeff Hawkins of Numenta [2] has been talking about?

[1] Neuromorphic Chips https://www.technologyreview.com/s/526506/neuromorphic-chips...

[2] Numenta papers/videos http://numenta.com/papers-videos-and-more/

It would also be good to see a major chip manufacturer or cloud provider that makes its own chips (Google/IBM) get serious about graph processing chips [3,4] and moving beyond floating point [5].

[3] Novel Graph Processor Architecture https://www.ll.mit.edu/publications/journal/pdf/vol20_no1/20...

[4] Novel Graph Processor Architecture, Prototype System, and Results https://arxiv.org/pdf/1607.06541.pdf

[5] Stanford Seminar: Beyond Floating Point: Next Generation Computer Arithmetic https://www.youtube.com/watch?v=aP0Y1uAA-2Y

mrfusion• 9 years ago

Would it be possible to build an analog neural network with hardware?

emcq8• 9 years ago

Yes, this is traditionally a big effort in the neuromorphic community originating in Carver Meads lab at Caltech.

Perhaps the best success of such analog computation efforts are with Neurogrid (full disclosure I worked with this group): https://web.stanford.edu/group/brainsinsilicon/

One thing to keep in mind is that communication is often still in digital spikes, which some may argue is more neuron like than an analog encoding.

deepnotderp13• 9 years ago

Yes, I work at a startup that's doing this, but it's very dangerous with the amounts of noise, which seem fine on MNIST and even CIFAR, but you die on imagenet.

The key to circumventing this is very complicated and our "secret sauce".

38kkdiu• 9 years ago

Can you say a bit more about what you mean by "it's very dangerous with the amounts of noise"?

adamnemecek• 9 years ago

I'm hoping that we could see photonic computers.

return02• 9 years ago

Do 'Neuromorphic' chips do anything useful? At least neural networks have well known utility, but afaik, neuromorphic (i.e. heavily simplified models of neurons that one only hopes - but cannot prove are correct) have no useful applications - or even theoretical functions.

bobsil12• 9 years ago

Low-power video object recognition for military drones, security cams, etc.

emcq8• 9 years ago

The near future is not in putting learning inside a chip. We're a long ways off from the one shot learning needed to make a device actually interesting with localized learning.

Instead, the future is recording and uploading your observations to the cloud, with data scientist and neural net wizards training over this dataset on a cluster with tons of GPUs, and then deploying an optimized model to scrappy low precision inference chips.

This is why FPGA based designs will fail to be stunning. Specialized low precision ASICs more similar to DSPs like Movidius' Myriad (in the Phantom drones and Google's Project Tango devices), Google's TPU, upcoming Qualcomm chips, or Nervana's will become increasingly popular.

aj7• 9 years ago

For laughs, I searched this document for the term Nvidia.

Intel Gets Serious About Neuromorphic, Cognitive Computing Future

Related Stories