Attention Mechanism in Deep Learning

https://blog.floydhub.com/attention-mechanism/

by ReDeiPirati • 7 years ago

195 24 7 years ago

blog.floydhub.com

When we think about the English word “Attention”, we know that it means directing your focus at something and taking greater notice. The Attention mechanism in Deep Learning is based off this concept of directing your focus, and it pays greater attention to certain factors when processing the data.

I actually think they should rename it to 'Focus Mechanism'

elcomet• 7 years ago

Why do you think Focus Mechanism is more appropriate than Attention?

nerdponx2• 7 years ago

Less anthropomorphism in machine learning is good IMO.

joe_the_user• 7 years ago

Well, attention implies something that can shift rapidly, that might have some meaning and that can lead to action.

Focus is broadly like attention but without those implications.

I don't know enough to say which one is more appropriate here but that's how the terms seem to "color" the concepts.

codesushi42• 7 years ago

I disagree.

"Attention" is all you need.

physicsyogi2• 7 years ago

Focus is usually directed at a single thing though, it's a narrowing of attention. Attention mechanisms can pay attention to more than one thing at a time.

kuu• 7 years ago

In case you find Attention (and specially transformers) interesting I have some saved links introducing the topic:

http://www.peterbloem.nl/blog/transformers

https://nostalgebraist.tumblr.com/post/185326092369/the-tran...

https://papers.nips.cc/paper/7181-attention-is-all-you-need....

http://jalammar.github.io/illustrated-transformer/

https://arxiv.org/pdf/1807.03819.pdf

BiasRegularizer2• 7 years ago

While Transformer's Self Attention(SA) is great, there are many applications where SA doesn't apply. For a more comprehensive overview of attention mechanisms, I often find myself coming back to Lilian Weng's post:

https://lilianweng.github.io/lil-log/2018/06/24/attention-at...

phkahler• 7 years ago

Would it be more accurate to use the word "importance" than "attention"? I feel like the later is encroaching on "intention" and conciousness more than these techniques warrant.

BiasRegularizer2• 7 years ago

"Importance" is a fairly overused word in DL. e.g. importance sampling and importance weighted gradients.

"Attention" works by creating inductive bias for the upstream network, which is analogous to human attention, and the word itself is much more intuitive.

Keep in mind machine learning is largely a descriptive science(modeling the behavior), whereas neuroscience is more prescriptive. So from the behavioral perspective, attention is better suited than importance.

msamwald• 7 years ago

On the other hand the word "attention" in deep learning is often used non-intuitively, e.g. "one token attends to another token" in self attention is hardly analogous to human cognitive processes.

abhgh2• 7 years ago

In the context of attention, there is a very interesting recent paper that warns against conflating attention and token importance - "Is Attention Interpretable?" [1]. This is an accepted paper in ACL-2019:

[1] https://www.aclweb.org/anthology/P19-1282

stochastic_monk• 7 years ago

See also Attention is Not Explanation [0].

[0] https://www.aclweb.org/anthology/N19-1357

abhgh2• 7 years ago

Thanks!

physicsyogi2• 7 years ago

There's a rebuttal to this as well: Attention is not not Explanation. https://arxiv.org/abs/1908.04626

eanzenberg• 7 years ago

Is there any work on attention for cnns or other computer vision algo's?

lucidrains• 7 years ago

https://arxiv.org/abs/1904.09925

ilaksh• 7 years ago

Did a FAANG company patent it already? If so, can we safely assume that since such a patent is ridiculous, it should be ignored in relation to any commercial service that might use these techniques?

abakus• 7 years ago

For a very easy to understand explanation of the transformer and attention mechansim, see:

https://blue-season.github.io/transformer-in-5-minutes/

Attention Mechanism in Deep Learning

Related Stories