The Reformer – Pushing the limits of language modeling

https://colab.research.google.com/drive/15oP52_7W5dRcAnbgX3tYADsu4R3cjMIf?usp=sharing

by datashrimp • 6 years ago

62 13 6 years ago

colab.research.google.com

bitforger• 6 years ago

A alternative to this has recently been published at ICML that claims to be faster. The website and tutorial video are very nice, too.

https://linear-transformers.com/

yahyaheee• 6 years ago

This is amazing!

cgearhart• 6 years ago

Thanks for sharing. This was great.

I wonder how much of a limitation it poses that the gradient requires some massaging to preserve efficient training; does that only work for some kernels, or can it be automated for arbitrary kernels?

blueblimp• 6 years ago

Are the results actually good? Table 2 reports 3.40 bits/dim on CIFAR-10, but PixelRNN in 2016 got 3.06 bits/dim (Table 3 in https://arxiv.org/abs/1601.06759). I would like to compare the MNIST results also but I'm having trouble converting between bits/dim and nats in a way that gives a sensible result. It's a bit annoying that the paper does not compare to previously-reported numbers on these benchmarks.

joeddav• 6 years ago

IMO the theoretical insight w.r.t. transformers as RNNs through the kernel formulation of self-attention is more interesting than the experimental results.

algo_trader3• 6 years ago

Are these reformer/linformer mosty space-efficient or also inference-runtime improved?

101101001010• 6 years ago

They are also efficient at inference-time. On GPUs the difference is noticeable only for sequences of length > 1024 (2048 for reformer since it adds some operations for hashing) thanks to the massive parallelism of GPUs amortizing the quadratic effect of the "usual" self-attention mechanism.

[edit] Linformer (https://arxiv.org/pdf/2006.04768.pdf) is a different project from the one linked in https://linear-transformers.com/ (Transformers are RNNs https://arxiv.org/pdf/2006.16236.pdf).

Der_Einzige• 6 years ago

What is the performance of reformer or linformer or any of these other new models in practical applications (not the benchmarks that researchers game)? Is it better than BERT?

The_rationalist• 6 years ago

I actively follow the state of the art pre trained models on paperswithcode.com and Nlp progress. The state of the art (often outperforming BERT by far) is XLnet and sadly is from 2019. 2020 has been stagnating (except for the special case of generative tasks with GPT3) I have observed that zero researchers have tried to improve on top of XLnet. While BERT has had ~20 alternatives implementations that improve upon it. Researchers are often unaware of what is the current state of the art,this induce a lag in research progress.

algo_trader3• 6 years ago

Are reformer/linformer more space-efficient or also inference-runtime improved?

For me the greatest trick is improved runtime compared to older seq/RNN techniques

ThomThom• 6 years ago

"zero researchers have tried to improve on top of XLnet" I question this assertion.

In particular at least the Roberta model by Facebook is already improving significantly upon XLNet.

random_savv• 6 years ago

This could be a transformative way to share research papers - it seems you can run all their models inside the paper!

The Reformer – Pushing the limits of language modeling

Related Stories