Neural ODE's: Understanding how they model data

https://jontysinai.github.io/jekyll/update/2019/01/18/understanding-neural-odes.html

by jsinai • 8 years ago

170 16 8 years ago

jontysinai.github.io

The diagrams here really helped to explain neural ODEs in an intuitive fashion. Does anyone know the best library that implements them so I can play around in an iPyNB? :-)

ChrisRackauckas• 8 years ago

https://news.ycombinator.com/item?id=18979541 shows (with code) how to build and train neural ODEs.

jsinai4• 8 years ago

Thanks. Looks fantastic. Also appreciate the feedback from parent.

contravariant• 8 years ago

y(x) = f(x,y), y(x0) = y0

If this is supposed to be the ODE definition, shouldn't it be y'(x) = f(x,y)? Otherwise I don't quite understand the definition of 'f'.

dfan• 8 years ago

I noticed the same thing. I presume it is a typo. Later on f is defined as you would expect.

jsinai4• 8 years ago

Thanks for pointing this out. It is indeed a typo.

GistNoesis2• 8 years ago

Seems like it will be hard to get it to work in Tensorflow, because it needs to compute the gradient in an unusual way, which afaik don't play nice with the existing architecture.

My guess is it will need some deep wizardry of the same kind as OpenAI gradient check-pointing.

Neural ODE, is a nice trick to reduce the memory usage to O(1) instead of O(nb timesteps). But the implementation cost and complexity cost probably mean we are better using gradient check-pointing on a forward dynamic and pay the memory cost.

It will also probably won't play well with noise.

Are there any implementation of it in tensorflow yet?

jsinai4• 8 years ago

There is a PyTorch implementation by the authors https://github.com/rtqichen/torchdiffeq and there is a Julia implementation due: https://news.ycombinator.com/item?id=18676986#18691054. As for TensorFlow, we will have to wait and see if it can be done. The adjoint method is matrix multiplication, but the tricky part will also be integrating with numerical ODE solvers.

317070• 8 years ago

I have one in tf, so I can confirm it is possible.

GistNoesis2• 7 years ago

Cool. This mean that hopefully there will be some tf open source implementations in the future. I guess there is something I don't see. I'm intrigued, does your code run inside a single sess.run() so that it can be composed nicely? If so did you use a "special trick"?

liuliu• 7 years ago

This is the first time I seriously read the paper and the code. Does this mean for ODE net, we are sharing weights for f? (even though we evaluate f at different y and t points)?

If that is the case, seems you can implement a full static version of ODE net with tf.While.

briga• 8 years ago

Thanks for this--science needs more writers who are able to distill complex subjects into clear and readable form.

jsinai4• 8 years ago

Thanks, I appreciate the kind words.

lucidrains• 7 years ago

Second this! Thank you for the wonderful explanation

buboard• 8 years ago

a quick intro to the concept is given by an author of the paper here: https://www.reddit.com/r/MachineLearning/comments/a65v5r/neu...

madhadron• 7 years ago

Can anyone comment on the relationship between this and the differential equations used by the connectionist neuroscientists back in the 1980's?

Neural ODE's: Understanding how they model data

Related Stories