NLP Zero to One: Attention Mechanism (Part 12/30)

Bottle Neck Problem, Dot-Product Attention

Kowshik chilamkurthy
Mar 2 · 4 min read
Generated by Author

Introduction..

Illustration of Bottleneck issue, generated by author

Attention Mechanism..

illustration of vanilla and attention encoder-decoder model, generated by author

Weights of Attention

hidden state calculation at decoding step t, generated by author

Dot-Product Attention

Score equation, generated by author
Dynamic context vector equation, generated by author

Parameterised Attention

Note:

Generated by Author

Nerd For Tech

From Confusion to Clarification

Nerd For Tech

NFT is an Educational Media House. Our mission is to bring the invaluable knowledge and experiences of experts from all over the world to the novice. To know more about us, visit https://www.nerdfortech.org/.

Kowshik chilamkurthy

Written by

RL | ML | ALGO TRADING | TRANSPORTATION | GAME THEORY

Nerd For Tech

NFT is an Educational Media House. Our mission is to bring the invaluable knowledge and experiences of experts from all over the world to the novice. To know more about us, visit https://www.nerdfortech.org/.