Neural Networks Are Impressively Good at Compression

https://probablydance.com/2016/04/30/neural-networks-are-impressively-good-at-compression/

by ingve • 10 years ago

244 64 10 years ago

tacos10• 10 years ago

This is both a fascinating read of an obviously bright and clever fellow and also a TERRIBLE way to approach building intuition around the emergent behaviors of neural networks. If you want to learn about them without repeating the terrible mistakes this guy is about to make if he continues down this path, pick up a book.

CardenB• 10 years ago

What sort of terrible mistakes are you referring to?

tacos10• 10 years ago

A fundamental misunderstanding of the space, focusing on the wrong things, thinking he can visualize (or should visualize) the mechanics, hyperfocus on topologies, misunderstanding of key distance functions, sigmoids, training/overtraining, add buzzword here.

Right now he's a chef who obsessively spent six months thinking about knives. But it's about the meat and the heat, not the chef knife.

Just pick a problem and start playing with it! If you lean to this sort of pedantic "I MUST UNDERSTAND ALL FUNDAMENTALS" (I do!) I still recommend Matlab (available for $49-$99 in student editions) because the tools are so good and the docs are so damn readable. This post is basically a matlab doc page minus the links to the stuff that actually works.

astazangasta• 10 years ago

Lossy compression. Reminds me of this hilarious bit: http://web.archive.org/web/20050402231231/http://lzip.source...

svantana• 10 years ago

I thought this would be about something like giving the Hutter Prize [1] a go using character RNNs [2]. Instead, it's a somewhat confused "gentle introduction" to neural nets (which there are plenty of already, of higher quality) and compression is sort of handwavely discussed, not properly with bits and entropy like us information theorists would have it :)

[1] http://prize.hutter1.net [2] https://github.com/karpathy/char-rnn

blazespin• 10 years ago

It's an interesting idea though, as sophomore as the attempt was. I'd be curious to see images lossy compressed via a neural net. Do you know of any better attempts than this?

tacos10• 10 years ago

Naive backprop is just another way of expressing the same algorithm. This corollary was actually used to analyze early nets in the 90s. I can't pull up papers for some reason, they're out there though.

kaffeinecoma3• 10 years ago

A "kind of" compression using polygons: https://rogeralsing.com/2008/12/07/genetic-programming-evolu...

I believe it was discussed here a few years back.

petra• 10 years ago

Not excatly what you've asked, but even more interesting:There's a startup that uses deep learning to take a compressed video and knows how add details and improve the quality, just by understanding what we are expecting.

Can't find the startup's name though.

walrus2• 10 years ago

It's been done before. Alex Graves achieved 1.33 bits per character using stacked LSTMs.[1] The current Hutter Prize record is 1.28 bpc. Note, however, that the 1.33 bpc value was calculated on a held-out validation set and doesn't include the network weights.

[1] http://arxiv.org/abs/1308.0850

twotwotwo• 10 years ago

Man, the RNN-generated Wikipedia data around page 14 of http://arxiv.org/pdf/1308.0850v5.pdf is fascinating.

Some things you'd expect from a modern language model--like, your phone keyboard's predictor already generates some grammatical-looking phrases, and there's only room to do so much in a phone. I'm more impressed that (as the author notes) it gets syntax right, including matched pairs over long distances, occasionally makes words up using plausible components, suffixes, etc. ("quanting"), and learns other quirks (e.g. there's Cyrillic after the [[ru:]] link and some Hangul after the [[ko:]]). Those aren't intuitively things you'd expect from applying arithmetic to a bunch of characters.

pigscantfly• 10 years ago

I was expecting to read about some experiments with autoencoders here, not this tutorial. I'm not sure how the author is learning about neural nets or where they are now, but anyone at this level of knowledge who wants to know more would be well served by going through Geoff Hinton's class on Coursera.

jg8610• 10 years ago

It's good to see people write up their experiments, it's useful for the rest of us to test how we understand neural nets.

I think there are a few mistake in your maths though. You can learn a 1-1 discrete mapping through a single node where you are using a one-hot vector. You just assign a weight to each of the input nodes, and then use a delta function on the other side. If I understood correctly, this is what you are doing.

Also, if you use a tanh in your input layer, but keep a linear output layer (as you start off with), you are still doing a linear approximation because you have a rank H (where H is the hidden layer) matrix that is trying to linearly approximate your input data. This is done optimally using PCA.

I'd second the advice to look into the coursera courses, or the nando de freitas oxford course on youtube (that actually has a really nice derivation of backprop).

rdlecler1• 10 years ago

I'm sure with perturbation analysis you could also remove even more edges from the ANN.

brendanofallon• 10 years ago

I think some of his statements are a little off-base, for instance, regarding the choice of tanh() for the activation function he says: "But mostly it’s 'because it works like this and it doesn’t work when we change it.'" People have spent a fair amount of time investigating the properties of different activation functions, and my understanding is that tanh() is generally not in favor, and ReLU (rectified linear) is probably a better choice for many applications, for reasons that are well understood. Maybe the author isn't all that familiar with the field?

tgflynn3• 10 years ago

It depends on who you read. LeCun certainly advocated tanh for a long time. ReLU seems to be currently favored, especially for deep nets but I'm not sure it's being used universally. In my own experiments I am seeing it train faster than logistic, but not necessarily give better results (maybe because you don't want the net to train too fast to reach the best optimum).

His overall point that the field is largely empirical with little solid theory to guide one's choices is, in my opinion, not wrong. It may be that a general theory doesn't exist and different network architectures and hyperparameters are needed for different types and sizes of problems.

argonaut2• 10 years ago

It doesn't depend on who you read. ReLUs (and their variants) are virtually universal now. LeCun advocated tanh but that was years and years ago before ReLUs came out (they came out in 2011) and he advocated ReLUs as early as three years ago (http://www.cs.nyu.edu/~yann/talks/lecun-ranzato-icml2013.pdf). Empirically they give better results and are faster.

(I say "almost" because while I haven't seen a single state of the art result without ReLUs, I'm sure there is probably a random paper out there)

walrus2• 10 years ago

In my totally unscientific experiments, tanh worked better on shallow networks.

pjbrunet• 10 years ago

I suppose that one episode of POI (the chain of laptops on ice with Pink Floyd) could have implied a neural network of sorts, manifesting at the hardware level. If you remember the episode, they had to compress the AI to fit in a briefcase.

robbiep• 10 years ago

Ok so 3 layers increases the complexity but simplifies the connections. The human cortex has 6 layers.

Now create 6 layers, classify different sets of inputs as 'different' to represent different neurochemicals (you need several excitatory and several inhibitory and then a couple of very small master neurochemicals that have major excitatory and inhibitory responses to represent the dopamine network and a whole system for the amygdala), cluster different groups to either respond to inputs or create outputs, and set it loose on an environment. How close would we come to something that behaves as if conscious?

zamalek• 10 years ago

How close would we come to something that behaves as if conscious?

The human brain is recursive in nature - the neurons and synapses form a cyclic graph. You can't do that with a simple perceptron.

otempomores• 10 years ago

If the human brain is recursive... whats stopping it from going in infinite circles ... Chemistry? If you inhibit that reaction..do you crash? can you canbecome a master of a field by having the specialized neurons keep there cycle support ed?

kaffeinecoma3• 10 years ago

People are knocking this guy for not being an expert and maybe getting some details wrong. Maybe it's a little bit like watching a non-programmer stumble their way through a blog post about learning to program- experienced programmers may cringe a bit.

But I really appreciate these kinds of write-ups: he declares his non-expertise up-front, and then proceeds to document his understanding as he goes along. There's something useful about this kind of blog post for non-experts.

I'm working my way through Karpathy's writeup on RNNs (http://karpathy.github.io/2015/05/21/rnn-effectiveness). I've mechanically translated his Python to Go, and even managed to make it work. But I still don't entirely understand the math behind it. Now obviously Karpathy IS an expert, but despite his extremely well-written blog post, a lot of it is still somewhat impenetrable to me ("gradient descent"? I took Linear Algebra oh, about 25 years ago). So sometimes it's nice to see other people who are a bit bewildered by things like tanh(), yet still press on and try to understand the overall process.

And FWIW I had the same reaction as the author when I started toying around with neural nets- it's shocking how small the hidden layer can be and still do useful stuff. It seems like magic, and sometimes you have to run through it step-by-step to understand it.

karpathy• 10 years ago

Sorry about that! There's a lot to cover for one blog post to do satisfyingly. I encourage you to check CS231n for a more thorough treatment where we also discuss, for example, the tradeoffs of different activation functions like tanh(), have a more gentle introduction on gradient descent, I devote a whole lecture to char rnn, assignment #1 (they are available) would demystify the backward pass, etc.

Also definitely +1 for not putting down people who write similar posts. I encourage everyone who is trying to learn to do it through blog posts because it lets you explain/organize thoughts. I also enjoy reading them quite a bit because it illustrates the kinds of conceptual problems beginners face (which is not at all obvious once you've been in the area for a few years). And it's also interesting to see many different interpretations of the same concepts, as everyone has different background and the way they reason through things is usually quite unique. Granted, this one could have been named something more appropriate!

kaffeinecoma3• 10 years ago

No need to apologize- I learned SO much from your blog, thank you. I didn't realize the course was online (https://www.youtube.com/watch?v=NfnWJUyUJYU). Also, looks like there's a subreddit for it as well: https://www.reddit.com/r/cs231n

It's really wonderful that all of this is freely available, thank you.

pmarreck• 10 years ago

FYI, gradient descent is covered in one of the very first weeks of Andrew Ng's Coursera machine learning class, so perhaps just watch those lessons (free)

Gradient descent is the approximation solution basically because getting the exact solution requires a good computation of inverse matrices which is apparently not yet doable (it's too slow)

toxik• 10 years ago

I think the reason people do gradient descent is that the datasets are too large to solve for all inputs simultaneously. It isn't impossible in theory, really.

SixSigma• 10 years ago

http://cs224d.stanford.edu/syllabus.html

covers RNN on lec 8

pvnick• 10 years ago

Nando De Freitas has a great youtube channel with videos that you might find helpful, including this one in particular on unconstrained optimization: https://www.youtube.com/watch?v=QGOda9mz_yA&list=PLE6Wd9FR--...

StavrosK2• 10 years ago

People are knocking this guy for not being an expert and maybe getting some details wrong.

I think this style of teaching has great value. Someone who's learning something themselves is the person most suitable to teach it to others, since they know exactly what a novice user doesn't know. For example, I wanted to write something up for monads the other day, since it's a simple concept that's made super confusing by people who dive into mathematical notation right away. The downside with this approach is that the novice lacks experience, so what they're learning may not be entirely accurate.

I think the best approach is a hybrid: Someone who is learning the material explains it, and someone who already knows it points out mistakes. In this case, HN can serve as the expert, and we all end up with a very informative post.

tobz3• 10 years ago

One of my great regrets with leaving my last job was that: I had little FP experience, and we had one guy with a lot of it. We had done informal teaching sessions, and had planned to try to write blog posts / record podcasts of our little sessions, precisely for the reasons you mentioned, hoping that it would help others absorb the material more easily.

jcoffland• 10 years ago

This is not compression. The author focuses on the small number of nodes and calls it compression but it's the edges that encode the information and there are a lot of them.

tgflynn3• 10 years ago

You can get (lossy) compression by using an autoencoder with a hidden layer that has fewer nodes than the number of inputs, since then you only need to store the hidden layer excitations for each example. The weights and network architecture itself are part of the compressor, not the compressed data.

As for how well this works compared to other approaches, I don't know, and this blog post certainly doesn't address the question.

zardeh• 10 years ago

If you count the number of connections on that last picture you will notice that there are fewer than there were in the first network. There were 55=25 at first, now there are 52*2=20. That reduction would be larger if I had more input and output nodes. For 10 nodes that reduction would be from 100 connections down to 40 when inserting two nodes in the middle. That’s where the compression in the title of this blog post comes from.

algorithm314• 10 years ago

Neural networks are used for over a decade in Context Mixing compressors like paq https://en.wikipedia.org/wiki/PAQ

frozenport• 10 years ago

How can the author draw conclusions without citing any code or experiments. For example, if we try to use the features DeepDream leaned we can see that our image will have terrible compression artifacts (faces of cats). DNNs do dimensional reduction, it is not clear if this correctly preserves image features.

IamFermat• 10 years ago

This is really cool. Totally can see a way where the algo would adapt based on object getting compressed

john_reel• 10 years ago

ZPAQ and pngwolf have very nice approaches to learning algorithms for compression. Especially ZPAQ.

Neural Networks Are Impressively Good at Compression

Related Stories