vinn124

for a good introduction to order and complexity (including intelligence) arising from nothing, read "the origins of order" by stuart kauffman.

who cares?

At the core RL is just updating a table of values, and then using function approximation (aka, machine learning) for more complex cases.

this seems to be a common assertion about ml. other refrains include "ml is just matrix multiplication" and "ml is just affine transformations followed by nonlinearity".

while technically true, it is an unhelpful comment as it doesnt shed any light on the salient questions, such as:

* how do you reduce bias/variance, since youre sampling a minuscule slice of the state/action space? * how do you construct a value function (or something like it) in a sparse reward environment, with possibly thousands of time steps? * how do you know your policy network is exploring new states?

it is the equivalent of learning how to draw an owl: draw a few circles, then draw the fucking owl.

particularly when most microsoft investors value microsoft stock for its earnings potential, as opposed to growth/revenue potential.

Increased accuracy per unit wallclock time is what you want.

yah, especially for a framework for distributed learning!

3) Test that the solution works on toy examples, like MNIST, simple block worlds, simulated data, etc.

youre right: mnist, imagenet, etc are toy examples that do not extend into the real world. but the point of reproducible research is to experiment on agreed upon, existing benchmarks.

What are the use cases for adding yet another layer to the stack?

in my limited experience with horovod, horovod is most useful when youre running large clusters of workers/ps. in those situations, you typically have to manually find the appropriate balance of workers/ps. (otherwise youd run into blocking or network saturation issues.) horovod addresses this issue with their ring allreduce implementation.

having said all of that, im sticking with distributed tf for now.

Usually you throw everything and see what sticks.

most practitioners start with the simplest possible learner, then gradually, and thoughtfully, increase model complexity while paying attention to bias/variance. this is far from a "kitchen sink" approach.

Many losses which don't seem differentiable can be reformulated as such...

agreed, especially with policy gradients.

If the dimensionality is small, second-order methods (or approximations thereof) can do dramatically better yet.

i have not seen second order derivatives in practice, presumably due to memory limitations. can you point me to examples?

not exactly what you asked for, but if youre looking for a gentle academic introduction to the intersection of ai and games, i recommend [ AI Researchers, Video Games Are Your Friends!](https://arxiv.org/abs/1612.01608)

the quality and preciseness of this answer is why i read hn every day.

Why should we expect there to be any mathematical foundation to this stuff?

i would be surprised that "this stuff" would be exception to the unreasonable effectiveness of mathematics. mathematics underpins virtually every observed phenomenon, including theoretical physics, computer science, economics. in fact, the mathematical structure of any physical theory often points the way to further advances in that theory and even to empirical predictions.

to not expect that "this stuff" should not have any mathematical foundation is a fantastically naive view.

youve missed the point.

the point is: anything complex can be dismissed as "just x,y,z" if you dont appreciate the massive body of work behind it.

i made that point because OP observed that "ml is just affine transformations" or something to that effect. yes, that's one way to frame it - if youre okay overlooking roughly 30 years of research.

Then my next question will be how can I trust something that I don't understand.

ive always found this mentality silly. do you understand aerodynamic principles? laws of electromagnetism? mendelian genetics? or information theory?

how does this solve anything? if a simple decision tree could predict the outputs of more complex deep nets, why not use the decision tree in the first place? also, what do you do when a decision tree isnt powerful enough, as in the case of many interesting problems such as speech, computer vision, etc.

it was the ability of steve jobs to create an organizing principle (and its corresponding organizational structure) that deserves credit for apple's success.

It's just alternating layers of affine transformations and nonlinearity with lots of tricks and improved routing.

and a computation is just 0s and 1s, with lots of if/then statements.

pytorch, not tf, seems to be winning the hearts and minds of ml researchers. then again, things change quickly.

If you're not waymo you probably use ROS.

is this true? ros works well for prototyping, but is it actually used in prod?

I am curious to know what major innovations in search engines happened since the page rank algorithm, or were there only incremental improvements?

a ton has happened! since pagerank, theres been a ton of advances around nlp that has changed the way queries are processed prior to information retrieval. for example, google's rankbrain seems to do a lot of the heavylifting around word similarity.

Note that the article has a comment by Yann LeCun (hopefully it's not an impersonator).

i wouldnt be surprised if it was lecun. colah's illustrations on nonlinear transformations have made it into several lecun papers, including the following [nature review]( https://www.nature.com/articles/nature14539).

agreed

that is true: popular rl techniques (eg policy gradients) are very similar to "vanilla" supervised learning techniques and architectures, but they are unsupervised in the sense that they required zero human input.

alphago zero is the canonical example of tabula rasa machine learning.

sounds like a job for [adversarial networks](https://arxiv.org/pdf/1406.2661.pdf)!

The test of believability yep! that is the fundamental limitation of adversarial networks. theres no good measure or "loss", as it's highly subjective.

but isn't that the whole point of GANs?

not quite, but youre on the right path.

think about it this way: you (the generative model) are trying to predict a unit gaussian, which is just a fancy way to say bell curve. you get +1 if you predict a number in this distribution (eg 0.1 or -0.5, which is within one standard deviation of the mean of 0); you get -1 if you predict a number thats "far" from this distribution (something like 40 - which has an infinitesimally low probability of being drawn from a unit gaussian).

mode collapse, then, is when you predict 0 all the time. yes, you are technically correct but youve failed to learn the true distribution.

obviously ive simplified this quite a bit and have anthropomorphize the model, but i hope you get the gist. otherwise, the [original paper](https://arxiv.org/abs/1406.2661) is refreshingly easy to read.

deployment is not the only reason. distributed ml (not deepmind scale but we're still talking about several instances, usually for something like hyperparameter tuning) is fairly common in practice. without docker, id waste most of my time provisioning and getting things to work - even if it's just for myself!

what a lovely project - thank you for sharing

theres research outside ai/ml - it's a big world out there.

as an aside, i think dan luu's writings are some of the best on interweb: articulate, methodical, concise.