1. A primer on transformer/1.04.  Understanding Self-attention mechanism.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Understanding Self-attention mechanism\n",
    "\n",
    "We learned how to compute query $Q$, key $K$, and value $V$ matrices and we also learned that they are the different projections of our input matrix $X$. Now let’s see how the query, key, and value matrices are used in the self-attention mechanism.\n",
    "\n",
    "We learned that in order to compute a representation of a word, the self-attention mechanism relates the word to all the words in the given sentence. Consider the sentence 'I am good'. To compute the representation of the word 'I', we relate the word 'I' to all the words in the sentence as shown in the following figure: \n",
    "\n",
    "![title](images/8.png)\n",
    "\n",
    "But why do we need to do this? Understanding how a word is related to all the words in the sentence helps us to learn better representation. Now, let's learn how the self-attention mechanism relates a word to all the words in the sentence using the query, key, and value matrices. The self-attention mechanism includes four steps, let's take a look at them one by one.\n",
    "\n",
    "\n",
    "## Step 1\n",
    "\n",
    "The first step in the self-attention mechanism is to compute the dot product between the query matrix $Q$ and the key matrix $K^T$:\n",
    "\n",
    "![title](images/9.png)\n",
    "\n",
    "The following shows the result of the dot product between the query matrix $Q$ and the key matrix $K^T$:\n",
    "\n",
    "![title](images/10.png)\n",
    "\n",
    "But what is the use of computing the dot product between the query and key matrix? What exactly does $Q.K^T$ signifies? Let's understand this by looking at the result of $Q.K^T$ in detail.\n",
    "\n",
    "Let's look into the first row of the matrix $Q.K^T$ as shown in the following figure.  We can observe that we are computing the dot product between the query vector $q_1$ (I) and all the key vectors - $k_1$ (I), $k_2$(am),  $k_3$(good).  Computing dot product between two vectors tells us how similar they are.\n",
    "\n",
    "Thus computing dot product between the query vector $q_1$ and the key vectors ($k_1, k_2, k_3$) tells us how similar the query vector $q_1$ (I) is to all the key vectors - $k_1$ (I), $k_2$(am),  $k_3$(good). By looking at the first row of the matrix  $Q.K^T$  , we can understand that the word 'I' is most related to itself than the words ('am', 'good') since the dot product value is high for $q_1.k_1$ compared to  $q_1.k_2$  and  $q_1.k_3$ :\n",
    "\n",
    "![title](images/11.png)\n",
    "\n",
    "Now, let us look into the second row of the matrix $Q.K^T$. As shown in the following figure, we can observe that we are computing the dot product between the query vector $q_2$  (am) and all the key vectors -  $k_1$ (I), $k_2$(am),  $k_3$(good).  This tells us how similar the query vector  $q_2$ (am) is to the key vectors - $k_1$ (I), $k_2$(am),  $k_3$(good).\n",
    "\n",
    "By looking at the second row of the matrix $Q.K^T$, we can understand that the word 'I' is most related to itself than the words ('am', 'good') since the dot product value is high for $q_2.k_2$ compared to  $q_2.k_1$  and  $q_2.k_3$ :\n",
    "\n",
    "\n",
    "![title](images/12.png)\n",
    "\n",
    "\n",
    "Similarly, let's look into the third row of the matrix $Q.K^T$. As shown in the following figure, we can observe that we are computing the dot product between the query vector  $q_3$(good) and all the key vectors - $k_1$ (I), $k_2$(am),  $k_3$(good). This tells us how similar the query vector  $q_3$ (good) is to all the key vectors - $k_1$ (I), $k_2$(am),  $k_3$(good).\n",
    "\n",
    "By looking at the third row of the matrix $Q.K^T$, we can understand that the word 'good' is most related to itself than the words ('am', 'good') in the sentence since the dot product value is high for  $q_3.k_3$ compared to  $q_3.k_1$  and  $q_3.k_2$ :\n",
    "\n",
    "![title](images/13.png)\n",
    "\n",
    "\n",
    "\n",
    "Thus, we can say that computing the dot product between the query matrix $Q$ and the key matrix $K^T$ essentially gives us the similarity score which helps us to understand how similar each word in the sentence to all other words. \n",
    "\n",
    "\n",
    "## Step 2\n",
    "\n",
    "\n",
    "The next step in the self-attention mechanism is to divide the matrix  $Q.K^T$ by the square root of the dimension of the key vector. But why do we have to do that? This is useful in obtaining stable gradients.\n",
    "\n",
    "Let $d_k$ be the dimension of the key vector. Then we divide  $Q.K^T$   by $\\sqrt{d_k}$ . The dimension of the key vector is 64. So taking the square root of it, we will obtain 8. Hence, we divide $Q.K^T$ by 8 as shown in the following figure:\n",
    "\n",
    "\n",
    "![title](images/14.png)\n",
    "\n",
    "## Step 3\n",
    "\n",
    "By looking at the above similarity scores, we can understand that they are in the unnormalized form. So, we normalize them using the softmax function. Applying softmax function helps in bringing the score to the range of 0 to 1 and the sum of scores equals to 1 as shown in the following:\n",
    "\n",
    "\n",
    "![title](images/15.png)\n",
    "\n",
    "We can call the preceding matrix a score matrix. With the help of these scores, we can understand how each word in the sentence is related to all the words in the sentence. For instance, look at the first row in the preceding score matrix, it tells us the word 'I' is related to itself by 90%, to the word 'am' by 7%, and to the word 'you' by 3%.\n",
    "\n",
    "\n",
    "\n",
    "## Step 4\n",
    "\n",
    "Okay, what’s next? We computed the dot product between the query and key matrix, obtained the scores and then we normalized the scores with the softmax function. Now the final step in the self-attention mechanism is to compute the attention matrix $Z$.\n",
    "\n",
    "The attention matrix contains the attention values for each word in the sentence. We can compute the attention matrix $Z$ by multiplying the score matrix $\\text{softmax}\\bigg(\\frac{QK^T}{\\sqrt{d_k}}\\bigg) $  with the value matrix $V$ as shown in the following:\n",
    "\n",
    "\n",
    "![title](images/16.png)\n",
    "![title](images/17.png)\n",
    "\n",
    "As we can observe, the attention matrix $Z$ is computed by taking the sum of value vectors weighted by the scores. Let us understand this by looking into row by row. First, let us see how the first row,  $z_1$ self-attention of the word 'I' is computed: \n",
    "\n",
    "\n",
    "![title](images/18.png)\n",
    "\n",
    "From the preceding figure, we can understand that, $z_1$ , self-attention of the word 'I' computed as a sum of value vectors weighted by the scores. Thus, the value of  $z_1$ will contain 90% of values from the value vector $v_1$  ('I') and 7% of the values from the value vector $v_2$  ('am') and 3% of values from the value vector $v_3$  ('good').\n",
    "\n",
    "But how this is useful? To answer this question, let’s take a little detour to the example sentence we saw earlier, 'A dog ate the food because it was hungry'. Here the word 'it 'indicates 'dog'. To compute the self-attention of the word 'it' , we follow the same above steps. Suppose, we have the following:\n",
    "\n",
    "![title](images/19.png)\n",
    "\n",
    "From the preceding figure, we can understand that the self-attention value of the word 'it' contains 100% of values from the value vector $v_2$ ('dog'). This helps the model to understand what the word 'it' actually refers to 'dog' and not 'food'. Thus by using a self-attention mechanism, we can understand how a word is related to all other words in the sentence. \n",
    "\n",
    "Now, coming back to our example, $z_2$, self-attention of the word 'am' is computed as the sum of value vectors weighted by the scores as shown in the following:\n",
    "\n",
    "\n",
    "\n",
    "![title](images/20.png)\n",
    "\n",
    "As we can observe from the preceding figure, the value of $z_2$ will contains 2.5% of values from the value vector  $v_1$ ('I') and 95% of the values from the value vector  $v_2$ ('am'), and 2.5% of values from the value vector  $v_3$('good').\n",
    "\n",
    "Similarly, $z_3$ , self-attention of the word 'good' is computed as sum of value vectors weighted by the scores as shown in the following:\n",
    "![title](images/21.png)\n",
    "This implies that the value of  will contains 21% of values from the value vector $v_1$ ('I') and 3% of the values from the value vector $v_2$ ('am') and 76% of values from the value vector $v_3$ ('good').\n",
    "\n",
    "Thus, the attention matrix $Z$  consists of self-attention values of all the words in the sentence and it is computed as:\n",
    "\n",
    "$$Z = \\text{softmax} \\bigg( \\frac{Q K^T}{\\sqrt{d_k}} \\bigg) V $$\n",
    "\n",
    "To get a better understanding of the self-attention mechanism, the steps involved are summarized below:\n",
    "\n",
    "- First, we compute the dot product between the query matrix and the key matrix $Q.K^T$ and get the similarity scores.\n",
    "- Next, we divide   $Q.K^T$  by the square root of the dimension of the key vector $\\sqrt{d_k}$ . \n",
    "- Then, we apply the softmax function to normalize the scores and obtain the score matrix, $\\text{softmax} (QK^T/\\sqrt{dk}) $\n",
    "- Finally, we compute the attention matrix $Z$  by multiplying the score matrix with the value matrix $V$. \n",
    "\n",
    "The self-attention mechanism is graphically shown below: \n",
    "\n",
    "![title](images/22.png)\n",
    "\n",
    "The self-attention mechanism is also called scaled dot product attention. Since here we are computing the dot product (between the query and key vector) and scaling the values (with $\\sqrt{d_k}$ ). \n",
    "\n",
    "Now that we have understood how the self-attention mechanism works, in the next section, we will learn about the multi-head attention mechanism.  \n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}