Link Between Deep Neural Networks and the Nature of the Universe

https://www.technologyreview.com/s/602344/the-extraordinary-link-between-deep-neural-networks-and-the-nature-of-the-universe/

by taeric • 10 years ago

91 17 10 years ago

technologyreview.com

alehander42• 10 years ago

I think the point is that all the patterns we deal with are a result of physic laws, so that somehow limits the solution space

akvadrako• 10 years ago

That seems to be concise summary of the articles point, but my first reaction is that it's a superficial distinction.

One of the researches is Max Tegmark, author of Our Mathematical Universe. In short, he suggests every theory becomes it's own universe; physical laws are representations of patterns in some kind of calculation or proof.

As he explains, although every possible theory is a universe, "most" complex theories can be reduced to simple ones. So it seems plausible that those simple patterns are more common and more likely to be found in a random (mathematical or physical) universe.

This means that neural networks are likely to work with those common patterns. One might say it because that's closer to the nature of physical laws. But more directly, that's because of the nature of mathematical laws.

rbanffy• 10 years ago

I wonder why the article got so long.

faragon2• 10 years ago

Neural networks are trained with information from the Universe. So, in my opinion, it should not be shocking at all that neural networks store some projection/model/echo of the universe (e.g. physical laws, maths, etc.).

adityab• 10 years ago

You should watch Max Tegmark's talk "Connections between Physics and Deep Learning" if this interests you [1].

Additionally, a paper that has everyone excited about deep connections between the mathematical analysis of physical systems and the hierarchical feature learning paradigm speaks of the connection in terms of the Renormalization Group [2].

Regardless of it's practical utility, the philosophical implications do tickle the intellect. On a dreamy note, I wonder if it would be possible to draw Category-theoretic parallels between some physical theories and statistical learning theory. There is so much to learn, and I am trying my best to teach myself (on the side) the mathematics that they don't teach in my CS grad school. So much to learn, such little time. :)

[1] https://youtu.be/5MdSE-N0bxs

[2] http://arxiv.org/abs/1410.3831

mamon• 10 years ago

I think that theoretical physics is an example of confusing map with the territory. Just because you can work out some mathematical mumbo-jumbo does not necessarily mean that it has any connection to reality.

This applies especially well to string theory which is just layers of pointless mathematics, each layer trying to make up for obvious deficiencies of lower layers.

Houshalter• 10 years ago

I wrote a better explanation for why neural nets work here: http://houshalter.tumblr.com/post/120134087595/approximating...

The answer is that neural networks are a somewhat crude approximation of Solomonoff induction. Solomonoff induction is an ideal, perfect machine learning method. It only assumes that the universe is a random computer program, and tries to infer which one.

faragon2• 10 years ago

Also interesting is the relation between Solomonoff induction and Kolmogorov Complexity.

rdlecler12• 10 years ago

It's not that no one understands, it's that journalists don't know that their's research that has addressed this question. The problem with intuitively understanding neural networks is that most of the weighted edges are spurious. Meaning you can remove most of them (WiJ -> 0) without negatively affecting the mapping function solution. Failing to do this we stare at a fully connected neural network which gives us no clues why it works. However , once you get rid of these spurious interactions (through perturbation analysis), then This will reveal the underlying circuit diagram for the network.

Often these threshold integration neurons are just approximating a simpler Boolean function so it really comes down to digital logic.

Techniques for pruning networks can be found here:

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2538912/

smaddox• 10 years ago

Pruning works for understanding sparse networks, but not for holographic networks, which can encode a much higher information density. Understanding a holographic network requires a large amount of effort, similar to determining the content of an optical holograph. You can try to develop a model, but if the information density is very high, it's no faster than observing the response to various inputs.

The problem becomes even more difficult if there is no clear distinctions between layers in the network, as in the brain.

rdlecler12• 10 years ago

A network should be no more complex than it needs to be. If you're not pruning spurious interactions then you're implicitly assuming that each interaction is information bearing. If you're not pruning spurious interactions from holographic networks then you're assuming that they're all functionally relevant. I promise you they're not.

You're also assuming that holographic networks are somehow necessary for most of the tasks we require. The human brain, while having billions of neurons is still quite sparsely connected.

p1esk2• 10 years ago

Pruning the network leads to a smaller network. How does that help with understanding? Does compressing the network from 100M parameters to 10M parameters reveal "the underlying circuit diagram" that you can understand?

ankurdhama• 10 years ago

Take any two fields, keep abstracting away stuff and you will reach a stage where you find a link between the two fields.

divan• 10 years ago

Despite the huge success of deep neural networks, nobody is quite sure how they achieve their success.

Is that true? After reading Jeff Hawkings - "On intelligence" book (2004), for me it's pretty much clear why they are so successful on that set of tasks. Despite of HTM model built by Jeff being kind of different from DNN, the idea and origin of brain and consciousness described pretty well. All modern advances in neuroscience just prove how strong the theory in this book is.

bbctol• 10 years ago

Jeff Hawkins has some pretty heterodox theories of intelligence, many of which are not backed up by modern neuroscience. Furthermore, as neuroscience advances, the similarities between so-called artificial neural networks and actual neurons are rapidly receding. Much of the current success of convolutional neural networks is (loosely) inspired by a particular part of the anatomy of the eye, but non-visual processing and abstract reasoning seems to take place under much more complicated patterns that ANNs may not be able to replicate as easily.

p1esk2• 10 years ago

for me it's pretty much clear why they are so successful on that set of tasks

What's less clear is why HTM has not been successful on any tasks since 2004. I remember a few years ago they entered Kaggle competition to detect anomalies in EEG. Their team finished in ~150th place (out of 500). Even though anomaly detection in time series data was supposed to be their strongest feature.

Tycho• 10 years ago

Going a bit further, Chris Langan's CTMU characterised the universe itself as a sort of back-propagation algorithm with a global utility function.

Link Between Deep Neural Networks and the Nature of the Universe

Related Stories