> For the complete documentation index, see [llms.txt](https://www.mlcompendium.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://www.mlcompendium.com/deep-learning/attention.md).

# Attention

These notes moved. The illustrated guides and papers on the Transformer architecture and its variants are now in [Transformers](https://www.mlcompendium.com/deep-learning/transformers.md), and the notes on ELMo, ULMFiT, BERT, XLNet, GPT-2, and GPT-3 style models are now in [Pretrained Language Models](https://www.mlcompendium.com/natural-language-processing-nlp/pretrained-language-models.md).

## Attention

An encoder-decoder squeezes a whole input sequence into one fixed vector, and attention lets the decoder look back at every input step instead. This page starts with the illustrated guides and the papers that introduced attention, then walks how scoring and context vectors work in encoder-decoder RNNs, and ends with Keras attention code; deprecated addresses stay last.

Attention grew out of sequence-to-sequence models, so it shares their notes. The same notes are in [SEQ2SEQ SEQUENCE TO SEQUENCE](https://www.mlcompendium.com/natural-language-processing-nlp/neural-nlp.md#seq2seq-sequence-to-sequence).

The best way in is visual. Illustrated attention- AMAZING and Illustrated self attention - great are the two illustrated guides. The Illustrated attention link is at the end of the body, where its address now serves a different article, and the Illustrated self attention address is kept in the deprecated list. Next, [Jay alamar on attention, the first one is better.](http://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/) is Jay Alammar's Visualizing A Neural Machine Translation Model, the mechanics of seq2seq models with attention, animated.

From the pictures to the source: [Attention is all you need (paper)](https://arxiv.org/abs/1706.03762) is the paper, and [The annotated transformer - reviewing the paper](http://nlp.seas.harvard.edu/2018/04/03/attention.html) is The Annotated Transformer. For the whole family at once, [Lilian weng on attention](https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html), self, soft vs hard, global vs local, neural turing machines, pointer networks, transformers, snail, self attention GAN.

Before the transformer, attention lived inside RNNs. [Understanding attention in rnns](https://medium.com/datadriveninvestor/attention-in-rnns-321fbcd64f05) starts from why even LSTMs, which overcome vanishing gradients, still hit a limit on long sequences. Another good intro with gifs to attention sat beside it. The WildML post, now at <https://dennybritz.com/posts/wildml/attention-and-memory-in-deep-learning-and-nlp/>, is the [Clear insight to what attention is, a must read](http://www.wildml.com/2016/01/attention-and-memory-in-deep-learning-and-nlp/). Google's announcement, [Transformer NN by google](https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html), is Jakob Uszkoreit's post on the Transformer as a novel architecture for language understanding, set against RNNs. An Intuitive explanation to attention is the self-attention post linked at the end of the page. [Attention by vidhya](https://www.analyticsvidhya.com/blog/2019/11/comprehensive-guide-attention-mechanism-deep-learning/) is Himanshi Singh's guide to the attention mechanism, its introduction in deep learning, an implementation in Python using Keras, and its applications in computer vision.

Attention is one of several ways to extend an RNN. [Augmented rnns](https://distill.pub/2016/augmented-rnns/) - including turing / attention / adaptive computation time etc. general overview, not as clear as the one below.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-0293737b920b83aef3d7bbbc1a01b8d1d216a40a%2Fgimg-04ce8fbcf2fe.png?alt=media" alt=""><figcaption><p>Augmented RNNs overview figure.</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-4914e35681edbc0db7e3eae375e4e6c589ec3448%2Fgimg-d20387353f65.png?alt=media" alt=""><figcaption><p>Augmented RNNs overview figure.</p></figcaption></figure>

The one below is the clearer one. [A really good REVIEW on attention and its many forms, historical changes, etc](https://medium.com/@joealato/attention-in-nlp-734c6fa9d983) starts with the attention mechanism as Bahdanau introduced it, then goes through self-attention, two-way attention, key-value-predict models and hierarchical attention. Whether attention helps depends on the data: [Medium on comparing cnn / rnn / han](https://medium.com/jatana/report-on-text-classification-using-cnn-rnn-han-f0e887214d5f) - will change on other data, my impression is that the data is too good in this article.

The mechanics come from Jason Brownlee's Machine Learning Mastery. Mastery on [rnn vs attention vs global attention](https://machinelearningmastery.com/global-attention-for-encoder-decoder-recurrent-neural-networks/) introduces global attention as a simplification of attention for encoder-decoder RNNs that is easier to implement. Mastery on [attention](https://machinelearningmastery.com/how-does-attention-work-in-encoder-decoder-recurrent-neural-networks/) - this makes the whole process clear, scoring encoder vs decoder input outputs, normalizing them using softmax (annotation weights), multiplying score and the weight summed on all (i.e., context vector), and then we decode the context vector. Its variants are:

1. Soft (above) and hard crisp attention
2. Dropping the hidden output - HAN or AB BiLSTM
3. Attention concat to input vec
4. Global vs local attention

Mastery on [attention with lstm encoding / decoding](https://machinelearningmastery.com/implementation-patterns-encoder-decoder-rnn-architecture-attention/) - a theoretical discussion about many attention architectures. This adds make-sense information to everything above. It frames the problem attention solves:

1. Encoder: The encoder is responsible for stepping through the input time steps and encoding the entire sequence into a fixed length vector called a context vector.
2. Decoder: The decoder is responsible for stepping through the output time steps while reading from the context vector.
3. A problem with the architecture is that performance is poor on long input or output sequences. The reason is believed to be because of the fixed-sized internal representation used by the encoder.

The implementation patterns it compares are:

1. Enc-decoder
2. Recursive
3. Enc-dev with recursive

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-a0c1b1817a743945774423fddc761f94898cb757%2Fgimg-0fde387d704d.png?alt=media" alt=""><figcaption><p>Encoder-decoder attention architecture patterns.</p></figcaption></figure>

With the patterns clear, the Code on GIT follows. HAN - [GIT](https://github.com/richliao/textClassifier) is a text classifier for Hierarchical Attention Networks for Document Classification. [Non penalized self attention](https://github.com/uzaymacar/attention-mechanisms/blob/master/examples/sentiment_classification.py) is the sentiment classification example from a family of attention mechanism implementations compatible with TensorFlow 2.0 and Keras. LSTM [BiLSTM attention](https://github.com/gentaiscool/lstm-attention) is an attention-based bidirectional LSTM for a classification task (ICASSP), and its [paper](https://arxiv.org/pdf/1805.12307.pdf) is Attention-Based LSTM for Psychological Stress Detection from Spoken Language Using Distant Supervision. Tushv89 [Keras layer attention implementation](https://github.com/thushv89/attention_keras) is a Keras layer implementation of attention for sequential models. Richliao, hierarchical [Attention code for document classification using keras](https://github.com/richliao/textClassifier/blob/master/textClassifierHATT.py) is the HAN script in that same repo, and his [blog](https://richliao.github.io/supervised/classification/2016/12/26/textclassifier-HATN/) is Richard Liao's Text Classification, Part 3 - Hierarchical attention network. The Keras users [group chatter](https://groups.google.com/forum/#!topic/keras-users/IWK9opMFavQ) is the discussion around it. The note on all of these: word level then sentence level embeddings.

Self-attention in Keras has its own packages. [Self Attention pip for keras](https://pypi.org/project/keras-self-attention/) is the package, and its [git](https://github.com/CyberZHG/keras-self-attention) repo describes an attention mechanism for sequential data that considers the context for each timestamp. [Phillip remy on attention in keras, not a single layer, a few of them to make it.](https://github.com/philipperemy/keras-attention-mechanism) is a Keras attention layer with Luong and Bahdanau scores.

Position is the last piece. [Self attention with relative positiion representations](https://medium.com/@_init_/how-self-attention-with-relative-position-representations-works-28173b8c245a) explains the Shaw et al. paper that modified the Transformer's self-attention to consider the relative distances between sequence elements. The original alignment idea is [nMT - jointly learning to align and translate](https://arxiv.org/abs/1409.0473), Neural Machine Translation by Jointly Learning to Align and Translate. [Medium on attention plus code, comparison keras and pytorch](https://medium.com/huggingface/understanding-emotions-from-keras-to-pytorch-3ccb61d5a983) is Thomas Wolf's Hugging Face post on detecting emotions, sentiments and sarcasm, moving the model from Keras to PyTorch.

The Towards Data Science posts close the page. The Intuitive explanation to attention, an-intuitive-explanation-of-self-attention-4f72709638e1, is at <https://towardsdatascience.com/an-intuitive-explanation-of-self-attention-4f72709638e1>. Illustrated attention, attn-illustrated-attention-5ec4ad276ee3, is at <https://towardsdatascience.com/attn-illustrated-attention-5ec4ad276ee3>; that address now opens Alex Dremov's Understanding Flash Attention, which writes the algorithm from scratch as a GPU kernel in Triton. For attention inside BERT, deconstructing-bert-part-2-visualizing-the-inner-workings-of-attention-60a16d86b5c1 is at <https://towardsdatascience.com/deconstructing-bert-part-2-visualizing-the-inner-workings-of-attention-60a16d86b5c1>.

### Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* Super fast transformers. This address no longer opens: <http://transformer>
* Bert multilabel classification. This address no longer opens: <http://towardsdatascience>
* Bert with keras, blog post. This address no longer opens: <https://www.ctolib.com/Separius-BERT-keras.html>
* Explain bert - bert visualization tool.. This address no longer opens: <http://exbert.net/>
* "GAN" using xgboost and gmm for density sampling. This address no longer opens: <https://edge.skyline.ai/data-synthesizers-on-aws-sagemaker>
* fine-tuning. This address no longer opens: <http://wiki.fast.ai/index.php/Fine\\_tuning>
* Towards Data Science: 2019-year-of-bert-and-transformer-f200b53d05b9. This address no longer opens: <https://towardsdatascience.com/2019-year-of-bert-and-transformer-f200b53d05b9>
* Towards Data Science: 3-ways-to-make-new-language-models-f3642e3a4816. This address no longer opens: <https://towardsdatascience.com/3-ways-to-make-new-language-models-f3642e3a4816>
* Towards Data Science: bert-is-not-good-at-7b1ca64818c5. This address no longer opens: <https://towardsdatascience.com/bert-is-not-good-at-7b1ca64818c5>
* Towards Data Science: beyond-word-embeddings-part-2-word-vectors-nlp-modeling-from-bow-to-bert-4ebd4711d0ec. This address no longer opens: <https://towardsdatascience.com/beyond-word-embeddings-part-2-word-vectors-nlp-modeling-from-bow-to-bert-4ebd4711d0ec>
* Towards Data Science: breaking-bert-down-430461f60efb. This address no longer opens: <https://towardsdatascience.com/breaking-bert-down-430461f60efb>
* Towards Data Science: build-a-bert-sci-kit-transformer-59d60ddd54a5. This address no longer opens: <https://towardsdatascience.com/build-a-bert-sci-kit-transformer-59d60ddd54a5>
* Towards Data Science: deconstructing-bert-distilling-6-patterns-from-100-million-parameters-b49113672f77. This address no longer opens: <https://towardsdatascience.com/deconstructing-bert-distilling-6-patterns-from-100-million-parameters-b49113672f77>
* Towards Data Science: demystifying-generative-adversarial-networks-c076d8db8f44. This address no longer opens: <https://towardsdatascience.com/demystifying-generative-adversarial-networks-c076d8db8f44>
* Towards Data Science: elmo-contextual-language-embedding-335de2268604. This address no longer opens: <https://towardsdatascience.com/elmo-contextual-language-embedding-335de2268604>
* Towards Data Science: elmo-helps-to-further-improve-your-word-embeddings-c6ed2c9df95f. This address no longer opens: <https://towardsdatascience.com/elmo-helps-to-further-improve-your-word-embeddings-c6ed2c9df95f>
* Towards Data Science: examining-berts-raw-embeddings-fd905cb22df7. This address no longer opens: <https://towardsdatascience.com/examining-berts-raw-embeddings-fd905cb22df7>
* Towards Data Science: explainable-data-efficient-text-classification-888cc7a1af05. This address no longer opens: <https://towardsdatascience.com/explainable-data-efficient-text-classification-888cc7a1af05>
* Towards Data Science: gan-ways-to-improve-gan-performance-acf37f9f59b. This address no longer opens: <https://towardsdatascience.com/gan-ways-to-improve-gan-performance-acf37f9f59b>
* Towards Data Science: identifying-the-right-meaning-of-the-words-using-bert-817eef2ac1f0. This address no longer opens: <https://towardsdatascience.com/identifying-the-right-meaning-of-the-words-using-bert-817eef2ac1f0>
* Towards Data Science: illustrated-self-attention-2d627e33b20a. This address no longer opens: <https://towardsdatascience.com/illustrated-self-attention-2d627e33b20a>
* Towards Data Science: testing-bert-based-question-answering-on-coronavirus-articles-13623637a4ff?source=email-4dde5994e6c1-1586483206529-newsletter.v2-7f60cf5620c9-----0-------------------b506d4ba\_2902\_4718\_9c95\_a36e33d638e6---48577de843eb----20200410. This address no longer opens: <https://towardsdatascience.com/testing-bert-based-question-answering-on-coronavirus-articles-13623637a4ff?source=email-4dde5994e6c1-1586483206529-newsletter.v2-7f60cf5620c9-----0-------------------b506d4ba\\_2902\\_4718\\_9c95\\_a36e33d638e6---48577de843eb----20200410>
* Towards Data Science: transfer-learning-using-elmo-embedding-c4a7e415103c. This address no longer opens: <https://towardsdatascience.com/transfer-learning-using-elmo-embedding-c4a7e415103c>
* Towards Data Science: understanding-bert-is-it-a-game-changer-in-nlp-7cca943cf3ad. This address no longer opens: <https://towardsdatascience.com/understanding-bert-is-it-a-game-changer-in-nlp-7cca943cf3ad>
* Towards Data Science: understanding-language-modelling-nlp-part-1-ulmfit-b557a63a672b. This address no longer opens: <https://towardsdatascience.com/understanding-language-modelling-nlp-part-1-ulmfit-b557a63a672b>
* paper. This address no longer opens: <https://www.cs.cmu.edu/\\~diyiy/docs/naacl16.pdf>
* A survey of long term context in transformers.. This address no longer opens: <https://www.pragmatic.ml/a-survey-of-methods-for-incorporating-long-term-context/>
* tutorial. This address no longer opens: <https://allennlp.org/tutorials>
* Elmo code on git. This address no longer opens: <https://github.com/allenai/allennlp/blob/master/tutorials/how\\_to/elmo.md>
* Medium on how - unclear. This address no longer opens: <https://blog.frame.ai/learning-more-with-less-1e618a5aa160>
* Fast NLP on how. This address no longer opens: <http://nlp.fast.ai/classification/2018/05/15/introducting-ulmfit.html>
* sparse bert. This address no longer opens: <https://github.com/huggingface/transformers/tree/master/examples/movement-pruning>
* post. This address no longer opens: <https://www.ai21.com/pmi-masking>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://www.mlcompendium.com/deep-learning/attention.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
