“The Unreasonable Effectiveness of Deep Learning Representations”

https://blog.insightdatascience.com/the-unreasonable-effectiveness-of-deep-learning-representations-4ce83fc663cf

by neuhaus • 8 years ago

85 35 8 years ago

blog.insightdatascience.com

robius3• 8 years ago

What I find unreasonable is doing all this without knowing what the model is doing. It's blind with no way to steer and correct it.

That is what feed forward networks and back propagation do for us. So why do we keep using them?

Then there's the statistics of it all.. what are we actually modeling? 'The real world' you say? Think again.

Data has to be changed and manipulated into i.i.d. form, or the algorithms won't work. How does an independent set of random variables give us a model of the actual dataset which is a very limited representation of the real world? It doesn't. It's modeling something else.

Okay, why don't we take dependence into account? Surely that would represent the real world better. Good question! (Shirley has nothing to do with it.)

It's because there is no formal definition of dependence in statistics. Let that sink in for a minute.

So the math needs work, statistics needs a revolution, and then we can begin to change AI enough for it to finally start making sense. Focus on explainable algorithms and actual ability to validate that what models generate make sense and will not be unlawfully biased or have outliers that will cause harm.

There appears to be only one company who has something like this. But few actually care.

kornish• 8 years ago

Which company, and what do they have?

robius3• 8 years ago

The company is a small startup with an amazing breakthrough called Optimizing Mind.

They have magical ways of 'explaining' black box models.

But it's not what DARPA is pushing (box remains black), rather the opposite, illuminating what's inside the box, making it a transparent open box. So much so, that the models they make you can edit by hand, since they make sense (to mere humans). Has rather immense implications.

Here's their crappy website: https://optimizingmind.com

nerdponx3• 8 years ago

https://cran.r-project.org/web/packages/datarobot/vignettes/...

wadkar2• 8 years ago

So the math needs work

Finally! I thought I was alone (and stupid) for thinking like this.

Is there any literature or any meta-work that discusses the notion of probability itself? What is expectation? What is dependence?

ssivark• 8 years ago

Er... maybe not that much reworking?

robius3• 8 years ago

There is very little, but what I have found is good work from The Math Citadel folks: https://www.themathcitadel.com/category/spire/probability-th... ,, also one in the stats category.

nerdponx3• 8 years ago

Any probability textbook would answer that.

I would be surprised if Khan Academy didn't cover at least expectation.

meikos• 8 years ago

what do you mean by the notion of probability itself?

probability was mastered far before computers were a thing

soVeryTired• 8 years ago

What is expectation?

There is a formal mathematical definition:

Let (\Omega, \mathcal{F}, P) be a probability space, and let X: \Omega -> S be a random variable taking values in some measurable space (S, \mathcal{S}).

Then the expectation is \int X(\omegs)dP

In computer science terms, do an experiment with every possible random seed and average the outcome (set \Omega to be the set of all seeds, and set P to be the uniform measure on them).

nerdponx3• 8 years ago

It's because there is no formal definition of dependence in statistics. Let that sink in for a minute.

What? Statistical dependence (of random variables) is defined clearly and precisely.

Data has to be changed and manipulated into i.i.d. form, or the algorithms won't work

Neural networks don't use the iid assumption.

I downvoted you because it seems like you don't really know what you're talking about and you're currently the top post in the thread. Please don't spread misinformation.

srean• 8 years ago

Strongly agreed. It seems robius really is clue less when he/she's talking about modeling independence or modeling the lack of independence.

avaku• 8 years ago

Agree. I can't downvote, so I just agree :)

bjourne• 8 years ago

All statistical learning, which neural nets are a form of, use the iid assumption. See https://stats.stackexchange.com/questions/213464/on-the-impo...

etaioinshrdlu• 8 years ago

I would try to remember... Just because you don't like it, doesn't mean it doesn't work. Deep learning is creating an awful lot of actual value right now, and I think we're just getting started.

JoeAltmaier• 8 years ago

Looking at pictures and categorizing them by appearance has very limited application. The human context can't be in grasped that way, can it?

For instance, 'wedding pictures'. A cake being cut; a cute kid throwing flower petals; a black-clad clergyman; a hand with a ring on it. Any human could categorize a pile of pictures into those that are in the 'wedding' category, and those that aren't. But no strategy based on weighting pixels is ever going to get there.

Likewise, 'cute' or 'scary' or 'funny'. And on and on.

jchw• 8 years ago

There are definitely networks in our brain that classify what we're seeing. An uneducated guess, though, is that the networks in our brains have many, many intermediate representations and don't go directly from image -> words, but rather go to abstract classifiers that can go back to words.

kujaomega• 8 years ago

I see a good explanation of the problem and a good evolution of the done steps. But I see a problem in the approach. When you are getting the most similar result, you are supposed to compute high cosine similarity between all the embeddings. If you have more than a billion of embeddings and the embeddings have 1k dimensions, it will take a lot of time. How would you solve this problem? Clustering the embeddings?

bglazer• 8 years ago

This is generally called the k-Nearest Neighbors problem. You should check out the various data structures for doing this, like the ball tree and kd tree.

bunderbunder2• 8 years ago

There are off-the-shelf libraries like ANNOY and nmslib that index the vectors in a way that allows for fast (possibly approximate) nearest neighbors searches.

“The Unreasonable Effectiveness of Deep Learning Representations”

Related Stories