Transformers Are Multi-State RNNs

by DreamGen • 3 years ago

41 9 3 years ago

Icko• 3 years ago

I've seen at least 6 such papers, all being like "<popular architecture> are actually <a bit older concept>". Neural networks are generic enough that you can make them equivalent to almost everything.

civilized• 3 years ago

Has anybody proved that transformers are just kernel SVM yet?

blackbear_• 3 years ago

I hope you're satisfied with Gaussian Processes: https://arxiv.org/abs/1806.07572

haswell• 3 years ago

Linking the latest generation tech to the previous generations is actually really helpful from my perspective.

All of the terminology for this tech is still emerging, and it can be quite difficult to formulate a reliable mental model for any of it due to how quickly it’s changing.

If hammers were more difficult to understand, I could imagine someone writing about the fact that hammers are in fact, just a piece of steel mounted on a handle made of wood.

Neural networks are generic enough that you can make them equivalent to almost everything.

Which to me is why papers like this are useful. They help newcomers conceptualize what the latest <popular architecture> is actually made of in terms of <a bit older concept> that the reader may already understand.

It will take some time for this information space to stabilize.

howrar• 3 years ago

It's also helpful in that you can now take insights from <older concept> and apply it to <popular architecture>. Many things are easier to reason about when framed a little differently.

dboreham• 3 years ago

See: https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Arnold_repr...

nh23423fefe• 3 years ago

oblique

joewferrara• 3 years ago

They show that a decoder only transformer (which gpts are) are rnns with infinite hidden state size. Infinite hidden state size is a pretty strong thing! Sounds interesting to me.

visarga• 3 years ago

not infinite, just scaling linearly with length

Transformers Are Multi-State RNNs

Related Stories