Neural Nets Optimized for Kolmogorov Complexity / Minimum Description Length

https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00489/112499/Minimum-Description-Length-Recurrent-Neural

by puttycat • 4 years ago

2 1 4 years ago

direct.mit.edu

presheaf• 4 years ago

This is interesting. Recently had the idea of making up an instruction set for bit-strings so that I could generate a bunch of programs w/ the instruction set to compress bit-strings where short instruction sequences would get high scores and longer ones would get low scores.

The tricky part is designing the feedback loop to properly train the instruction generator and like in this paper I needed to also include some non-differentiable stack operations. It's surprisingly hard to find work that combines neural networks and compression algorithms even though they seem like an obvious fit. This also allows for downstream tasks that are not possible with just vector spaces because the network that can compress bit-strings must be encoding some non-trivial features of the data set and can be used to augment downstream tasks with differentiable compression.

Neural Nets Optimized for Kolmogorov Complexity / Minimum Description Length

Related Stories