TherML – Thermodynamics of Machine Learning

https://arxiv.org/abs/1807.04162

by selimthegrim • 8 years ago

154 24 8 years ago

arxiv.org

mrcactu52• 8 years ago

Google's Data Centre literally generates heat. So anything we can do in this direction, is helpful.

https://deepmind.com/blog/deepmind-ai-reduces-google-data-ce...

selimthegrim2• 8 years ago

I’m not sure this is what the authors meant by their title.

mrcactu52• 8 years ago

One of the authors is at Google, so it wouldn't surprise me. Thermodynamics is clearly a metaphor, but information entropy they are studying be bounded by the literal heat generated by their computations.

mlthoughts20184• 8 years ago

Does anyone really find it surprising that you can do contortions around KL divergence among candidate probabilistic models to the point of shoe-horning some analogue of thermodynamic laws and equilibria into machine learning? I don’t find this either surprising, illuminating, or interesting.

It reminds me of the “information geometry of boosting” section of [0].

[0] https://pdfs.semanticscholar.org/2fad/679058e465fc07f942cfed...

gog-ma-gog• 8 years ago

...yes? Why is it not surprising?

danielmorozoff2• 8 years ago

Fundamentally, thermodynamics and information theory share a slew of similar ideas. Because in a lot of ways information theory came out of a thermodynamic thought process applied to signals. Even more recent work in ml like Boltzmann machines are directly taken from this overlap. That being said stronger theoretical connections are well worth the effort as these fields do have some different ideas that if shown to be similar can yield discoveries and push our understanding forward. Some thoughts that come to mind are compression, model generalizability/ convergence.

kgwgk3• 8 years ago

Minor nitpick: Boltzmann machines are hardly recent. They were introduced in the eighties.

WhitneyLand2• 8 years ago

Does anyone? What % of people make any connection at all between the fields?

Firstly this is not a CS or Physics only forum so I would guess the subject could easily have never come up for many (if so for the simpler/non-ML connections see https://en.wikipedia.org/wiki/Entropy_in_thermodynamics_and_...).

Secondly I think general awareness beyond formal education has increased in a non-linear way only over the last 20 years only because there has been so much science press around the celebrity of Steven Hawking.

His celebrity alone raises awareness, however it was compounded by drama depicted between himself and others and some really interesting observations along the way (https://en.wikipedia.org/wiki/Holographic_principle).

When all of this publicity has various inevitable Kevin Bacon trails back to the fundamental connections, I’d say the population is not only not everyone but could be much less if things unfolded a little differently.

mlthoughts20184• 8 years ago

I mean it more from a Joel Spolsky sort of too much abstraction point of view, e.g. from [0]:

“When great thinkers think about problems, they start to see patterns. They look at the problem of people sending each other word-processor files, and then they look at the problem of people sending each other spreadsheets, and they realize that there’s a general pattern: sending files. That’s one level of abstraction already. Then they go up one more level: people send files, but web browsers also “send” requests for web pages. Those are both sending operations, so our clever thinker invents a new, higher, broader abstraction called messaging, but now it’s getting really vague and nobody really knows what they’re talking about any more.

And if you go too far up, abstraction-wise, you run out of oxygen. Sometimes smart thinkers just don’t know when to stop, and they create these absurd, all-encompassing, high-level pictures of the universe that are all good and fine, but don’t actually mean anything at all.”

[0]: < https://www.joelonsoftware.com/2000/07/22/microsoft-goes-bon... >

orbifold• 8 years ago

Whether you find something like this surprising largely depends on your prior knowledge. There are quite different sets of people that work with statistics, theoretical physicists are one of them. So a recent advance in one field (machine learning), will inevitably draw in the hyenas from other fields, especially if one of their favorite toys (building increasingly complicated high energy particle physics models) has been largely taken away from them given the recent zero results at the LHC. Which is how you get review articles like this one (https://arxiv.org/abs/1803.08823).

kgwgk3• 8 years ago

Thanks for the link, it seems a nice review.

c3534l• 8 years ago

I doubt even 1% of the population can decode that sentence, let alone find it so blindingly obvious it need not be said.

mlthoughts20184• 8 years ago

It seems clear that this paper (and many papers like it that try to produce higher level abstractions casting one field’s concepts into theorems of some other field) is only meant for that 1% you mention. In fact this is why I don’t get excited about these sorts of results. For people outside the main fields, it ends up sounding like some dramatic philosophical discovery, but for people in the field it’s borderline meaningless. If you play around with definitions long enough, you can always find some notion of X that looks like “equilibrium” or “convergence” or “entropy” or “orthogonality” or some decomposition or some transformation for some other concept Y. Finding these results is almost always just about rejiggering definitions so that something mundane can be reinterpreted as something significant “in the right space.”

henripal2• 8 years ago

Self plug: If you enjoy this and want some (less technical) insight on connections between ML and thermodynamics, you might enjoy my series of blog posts: http://henripal.github.io/blog/stochasticdynamics

SmooL• 8 years ago

Oh I'm going to enjoy this, thank you!

henripal2• 8 years ago

Happy to talk some more about this :)

Another self plug (would have put it in first post but the video was just uploaded):

Here's my SciPy 2018 talk on the subject (from last week!!) https://m.youtube.com/watch?v=WUs0u2PJ2UU&index=46&t=0s&list...

mlazos• 8 years ago

This is why I enjoyed a computational probability class in college - the professor introduced all of the concepts starting with a problem in physics, a lot of methods in modern stats were built to explain physical behavior of particles with certain constraints.

I-like-food• 8 years ago

Related paper/blog post by baez! It's quite interesting and very readable (assuming a background in ugrad thermo). He ends up creating algorithmic analogs to measures like temperature, energy etc much like in this.

https://johncarlosbaez.wordpress.com/2010/10/12/algorithmic-...

TherML – Thermodynamics of Machine Learning

Related Stories