Clarifying exceptions and visualizing tensor operations in deep learning code

https://explained.ai/tensor-sensor/index.html

by parrt • 6 years ago

62 12 6 years ago

parrt5• 6 years ago

One of the biggest challenges when writing code to implement deep learning networks is getting all of the tensor (matrix and vector) dimensions to line up properly, even when using predefined network layers. This article describes a new library called TensorSensor that clarifies exceptions by augmenting messages and visualizing Python code to indicate the shape of tensor variables. It works with Tensorflow, PyTorch, and Numpy, as well as higher-level libraries like Keras and fastai.

parrt5• 6 years ago

BTW, here's a link to the twitter thread.

https://twitter.com/the_antlr_guy/status/1313569854918590465

jph00• 6 years ago

Be sure to read the last section - really deep work involved, including writing a Python parser! Here's a snippet:

"I used the built-in Python tokenizer, but built my own parser for the subset of statements (assignments, return) and operations supported by TensorSensor. There is a built-in Python parser, but it generates a different AST than I wanted; plus, filtering the built-in tree structure for just the parts I care about would be about the same amount of work as writing the parser. I built the parser using simple recursive descent, rather than adding another dependency (my friend antlr) to TensorSensor.

Next, we need to augment the AST with the values of all subexpressions so it looks like the AST from the previous section. This is a straightforward bottom-up walk of the AST calling eval() on each subexpression, saving the result in the associated node. In order to execute the various bits of the expression in the proper context after an exception, though, TensorSensor has to walk the call stack back down to the specific context that directly invoked the tensor library. The trick is not chasing the call stack too far, down into the tensor library.

Once the AST is augmented with partial values, we need to find the smallest subexpressions that evaluate to tensors for visualization purposes. That corresponds to the deepest subtrees that evaluate to tensors, which is the role of tsensor.analysis.smallest_matrix_subexpr()."

(Note: the author is also the creator of ANTLR.)

parrt5• 6 years ago

Thanks, Jeremy. :) I didn't go into super huge detail in the article on the implementation part as most readers won't have interest in language nerd details like you and I do.

nemoniac• 6 years ago

This kind of code transformation would be an everyday task in a Lisp. Would it make sense to use a Lisp FFI to interface one of tensor libraries and write the transformations there?

hamolton• 6 years ago

This looks like a neat tool, especially for students taking courses like CS 231N! I imagine this already works in Jupyter Notebook? Perhaps this could be integrated into an extension for the VSCode, maybe in the debugger.

parrt5• 6 years ago

I wondered about that. It might work just calling my internal pyviz("some python code in string) function from the debugger. It'll execute in context (maybe?)

blackbear_2• 6 years ago

This looks really neat, but I've only felt the need for such a tool with tensorflow. With pytorch and numpy you can just use ipdb or any other debugger to be parachuted right at the exception and inspect shape and value of all the variables you need.

parrt5• 6 years ago

True, but you have to do it one at a time to look at the variables. Also, the exception message doesn't tell you which operator among several is the issue. This just makes it easier. :)

blackbear_2• 6 years ago

Of course, but debugging is a general skill that will work in every case.

Not trying to detract anything from this tool, I'll definitely give it a try it next time I have something that doesn't work :)

bwasti• 6 years ago

I believe problems solved by this tool generally disappear when using named tensors

modeless• 6 years ago

Are people using named tensors a lot these days? I haven't seen a lot of code using them but I also haven't been looking at a lot of tensor code lately.

Clarifying exceptions and visualizing tensor operations in deep learning code

Related Stories