word_rnn_ramayana.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i43FN7Hv0Vvj"
      },
      "source": [
        "# Text generation using RNN - Word Level\n",
        "\n",
        "To generate text using RNN, we need a to convert raw text to a supervised learning problem format.\n",
        "\n",
        "Take, for example, the following corpus:\n",
        "\n",
        "\"Her brother shook his head incredulously. He was not aware of the situation at all.\"\n",
        "\n",
        "First we need to divide the data into tabular format containing input (X) and output (y) sequences. In case of a character level model, the X and y will look like this:\n",
        "\n",
        "|      X                |  Y      |\n",
        "|-----------------------|---------|\n",
        "|    < word1 >< word2 > | < word3 > |\n",
        "|    Her brother        |  shook  |\n",
        "|    brother shook      |  his    |\n",
        "|    shook his          |  head   |\n",
        "|    his head           | incredulously |\n",
        "|    head incredulously |    .    |\n",
        "|    ..                 |    .    |\n",
        "|    situation at       |  all    |\n",
        "|    at all             |    .    |\n",
        "\n",
        "Note that in the above problem, the sequence length of **X is two words** and that of **y is one word**. Hence, this is a many-to-one architecture. We can, however, change the number of input words to any number depending on the problem.\n",
        "\n",
        "A model is trained on such data. To generate text, we simply give the model any two words using which it predicts the next word. Then it appends the predicted word to the input sequence (to the extreme right of the sequence) and discards the first word (word on extreme left of the sequence). Then it predicts again using the new sequence and the cycle continues until a fix number of iterations. An example is shown below:\n",
        "\n",
        "Seed text: \"Did I\"\n",
        "\n",
        "|      X                                            |  Y                       |\n",
        "|---------------------------------------------------|--------------------------|\n",
        "|                        Did I                      |    < predicted word 1 >  |\n",
        "|               I < predicted word 1 >              |    < predicted word 2 >  |\n",
        "|       < predicted word 1 > < predicted word 2 >   |    < predicted word 3 >  |\n",
        "|       < predicted word 2 > < predicted word 3 >   |    < predicted word 4 >  |\n",
        "|                      ...                          |            ...           |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KfPrTidP0Vvk"
      },
      "source": [
        "# Notebook Overview\n",
        "1. Preprocess data\n",
        "2. Build LSTM model\n",
        "3. Generate text"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "n5R1APadwANc",
        "scrolled": true
      },
      "outputs": [],
      "source": [
        "# import libraries\n",
        "import warnings\n",
        "warnings.filterwarnings(\"ignore\")\n",
        "\n",
        "import re\n",
        "import numpy as np\n",
        "import nltk\n",
        "from nltk.tokenize import word_tokenize\n",
        "import requests\n",
        "from nltk.tokenize import word_tokenize\n",
        "\n",
        "from gensim.models import KeyedVectors\n",
        "\n",
        "from keras.preprocessing.text import Tokenizer\n",
        "from keras.utils import to_categorical\n",
        "from keras.preprocessing.sequence import pad_sequences\n",
        "# from keras.utils.vis_utils import plot_model\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense\n",
        "from keras.layers import LSTM\n",
        "from keras.layers import Embedding"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VrfTnFwC0Vvl"
      },
      "source": [
        "# 1. Preprocess data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "collapsed": true,
        "id": "PXzkqw1ipoPg"
      },
      "outputs": [],
      "source": [
        "# download ebook\n",
        "url = \"https://www.gutenberg.org/files/24869/24869-0.txt\"\n",
        "book = requests.get(url)\n",
        "data = book.text"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DLxJu5p70Vvm",
        "outputId": "ba74ace7-bc43-43a2-bcb0-629a351900aa"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "ï»¿The Project Gutenberg EBook of The Ramayana\r\n",
            "\r\n",
            "\r\n",
            "\r\n",
            "This eBook is for the use of anyone anywhere at no cost and with almost no\r\n",
            "restrictions whatsoever. You may copy it, give it away or re-use it under\r\n",
            "the terms of the Project Gutenberg License included with this eBook or\r\n",
            "online at http://www.gutenberg.org/license\r\n",
            "\r\n",
            "\r\n",
            "\r\n",
            "Title: The Ramayana\r\n",
            "\r\n",
            "\r\n",
            "\r\n",
            "Release Date: March 18, 2008 [Ebook #24869]\r\n",
            "\r\n",
            "Language: English\r\n",
            "\r\n",
            "Character set encoding: UTF-8\r\n",
            "\r\n",
            "\r\n",
            "***START OF THE PROJECT GUTENBERG EBOOK THE\n"
          ]
        }
      ],
      "source": [
        "# let's look at the text\n",
        "print(data[:500])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uLkurmgw0Vvm",
        "outputId": "97608b21-3a47-4a85-ded9-573597ea9807"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "19177\n"
          ]
        }
      ],
      "source": [
        "# subset the book from the first chapter, that ism INVOCATION - everything before first chapter is irrelevant data\n",
        "start_index = re.search(\"invocation.\\(1\\)\", data, re.I)\n",
        "print(start_index.start())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "collapsed": true,
        "id": "fEMxH9SD0Vvm"
      },
      "outputs": [],
      "source": [
        "# Let's see how does the text look like\n",
        "data = data[start_index.start():]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "N6gCm8wi0Vvm",
        "outputId": "20703a65-8dae-4efe-b9ad-0ce01407b9bb"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "INVOCATION.(1)\r\n",
            "\r\n",
            "\r\n",
            "Praise to VÃ¡lmÃ­ki,(2)bird of charming song,(3)\r\n",
            "  Who mounts on Poesyâs sublimest spray,\r\n",
            "And sweetly sings with accent clear and strong\r\n",
            "  RÃ¡ma, aye RÃ¡ma, in his deathless lay.\r\n",
            "\r\n",
            "Where breathes the man can listen to the strain\r\n",
            "  That flows in music from VÃ¡lmÃ­kiâs tongue,\r\n",
            "Nor feel his feet the path of bliss attain\r\n",
            "  When RÃ¡maâs glory by the saint is sung!\r\n",
            "\r\n",
            "The stream RÃ¡mÃ¡yan leaves its sacred fount\r\n",
            "  The whole wide world from sin and stain to free.(4)\r\n",
            "T\n"
          ]
        }
      ],
      "source": [
        "# let's look at the text\n",
        "print(data[:500])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HsMN8HJr0Vvm"
      },
      "source": [
        "## Clean text"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import nltk\n",
        "nltk.download('punkt')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "N8L9l_yd1UNQ",
        "outputId": "71f98ab3-4b0e-4ae1-95a0-c4af5233e64b"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Package punkt is already up-to-date!\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "collapsed": true,
        "id": "DJBSAbry0Vvm"
      },
      "outputs": [],
      "source": [
        "# define a function to clean text data\n",
        "def clean_document(document, char_filter = r\"[^\\w]\"):\n",
        "    '''\n",
        "    input:\n",
        "    document          :  string\n",
        "    char_filter       :  regex pattern - removes those characters from the text that match the pattern\n",
        "\n",
        "    output: clean document\n",
        "    '''\n",
        "\n",
        "    # convert words to lower case\n",
        "    document = document.lower()\n",
        "\n",
        "    # tokenise words\n",
        "    words = word_tokenize(document)\n",
        "\n",
        "    # strip whitespace from all words\n",
        "    words = [word.strip() for word in words]\n",
        "\n",
        "    # join back words to get document\n",
        "    document = \" \".join(words)\n",
        "\n",
        "    # remove unwanted characters\n",
        "    document = re.sub(char_filter, \" \", document)\n",
        "\n",
        "    # replace multiple whitespaces with single whitespace\n",
        "    document = re.sub(r\"\\s+\", \" \", document)\n",
        "\n",
        "    # strip whitespace from document\n",
        "    document = document.strip()\n",
        "\n",
        "    return document\n",
        "\n",
        "data = clean_document(data)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "len(data)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BNuvWse00wnq",
        "outputId": "3f92ee4d-34ed-4294-ef1d-d346736a317d"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "2195566"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Note: Reducing the size of the data. Since, it causes RAM consumption error when we use the whole data"
      ],
      "metadata": {
        "id": "r0vVQ6oF015_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "data = data[:400000]"
      ],
      "metadata": {
        "id": "qfqqR_ag0yVP"
      },
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cnNU1hg90Vvm",
        "outputId": "ccb478e1-08a6-4126-d030-4bc650e6013d"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of words in document: 78188\n"
          ]
        }
      ],
      "source": [
        "# length of text\n",
        "words = word_tokenize(data)\n",
        "print(\"Number of words in document: {}\".format(len(words)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lStmpXvE0Vvm"
      },
      "source": [
        "## Convert characters to integers"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "collapsed": true,
        "id": "PBkfQnPGqV_4"
      },
      "outputs": [],
      "source": [
        "# use Keras' Tokenizer() function to encode text to integers\n",
        "word_tokeniser = Tokenizer()\n",
        "word_tokeniser.fit_on_texts([data])\n",
        "encoded_words = word_tokeniser.texts_to_sequences([data])[0]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rga0jJb-0Vvm",
        "outputId": "f0b13a1b-6d93-48a1-a791-d10298dead59"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Vocabulary Size: 6713\n"
          ]
        }
      ],
      "source": [
        "# check the size of the vocabulary\n",
        "VOCABULARY_SIZE = len(word_tokeniser.word_index) + 1\n",
        "print('Vocabulary Size: {}'.format(VOCABULARY_SIZE))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L5YLA14G0Vvn"
      },
      "source": [
        "## Divide data in X and y"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6MLG5FhI0Vvn"
      },
      "source": [
        "### Create sequences\n",
        "\n",
        "In each training sample, X will have a sequence of 5 words and y will have the sixth word. In other words, this means that use previous five words of a sequence to predict next word."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "collapsed": true,
        "id": "hwJLfNn3qX56"
      },
      "outputs": [],
      "source": [
        "sequences = []\n",
        "MAX_SEQ_LENGTH = 5  # X will have five words, y will have the sixth word\n",
        "\n",
        "for i in range(MAX_SEQ_LENGTH, len(encoded_words)):\n",
        "    sequence = encoded_words[i-MAX_SEQ_LENGTH:i+1]\n",
        "    sequences.append(sequence)\n",
        "sequences = np.array(sequences)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gxNtFH0D0Vvn",
        "outputId": "3de1bc30-7910-4b80-dd04-f6ccdbb34c7d"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total number of training samples: 78183\n",
            "\n",
            "Sample sequences: \n",
            "[[3864 3865  650    3  107  720]\n",
            " [3865  650    3  107  720  937]\n",
            " [ 650    3  107  720  937 3866]]\n"
          ]
        }
      ],
      "source": [
        "print('Total number of training samples: {}'.format(len(sequences)))\n",
        "print('\\nSample sequences: \\n{}'.format(sequences[0:3]))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "collapsed": true,
        "id": "N3baVAiOqZpj"
      },
      "outputs": [],
      "source": [
        "# divide the sequence into X and y\n",
        "sequences = np.array(sequences)\n",
        "\n",
        "X = sequences[:,:-1]  # assign all but last words of a sequence to X\n",
        "y = sequences[:,-1]   # assign last word of each sequence to y"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "He2LhHAt0Vvn",
        "outputId": "e7709281-97e8-4b50-a000-cc486d668833"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Input of the first data point: [3864 3865  650    3  107] \n",
            "\n",
            "Output of the first data point: [ 720 ]\n"
          ]
        }
      ],
      "source": [
        "# Look at the first training example\n",
        "print(\"Input of the first data point:\", X[0], \"\\n\")\n",
        "print(\"Output of the first data point: [\", y[0], \"]\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zWQsaUBj0Vvn"
      },
      "source": [
        "### One-hot encode y"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QxXOFjTs0Vvn",
        "outputId": "c0ea2ee8-9492-4434-9dd7-9e686963cb54"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(78183,)"
            ]
          },
          "metadata": {},
          "execution_count": 18
        }
      ],
      "source": [
        "y.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "collapsed": true,
        "id": "Mf5_9fXD0Vvn"
      },
      "outputs": [],
      "source": [
        "y = to_categorical(y, num_classes=VOCABULARY_SIZE)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HX_Npp_cVWyW",
        "outputId": "9eb47216-e7f6-44a2-9dc7-03c9a9688af5"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(78183, 5)\n",
            "(78183, 6713)\n"
          ]
        }
      ],
      "source": [
        "print(X.shape)\n",
        "print(y.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oWRV0n3U0Vvn"
      },
      "source": [
        "There are 410241 sequences (data points) in total.\n",
        "\n",
        "Remember that to use an RNN data has to be of the shape (#samples, #timesteps, #features)\n",
        "\n",
        "In X, the third dimension, that is, number of features is missing because we're going to use the Keras' Embedding Layer. Hence we don't need to explicitly reshape the data to incorporate the third dimension. That will be done automatically by Keras.\n",
        "\n",
        "In y, the second dimension is missing, that is, the number of timesteps because y is not a sequence, it's just a single word. The number of features are represented by a one-hot encoded vector whose length is the VOCABULARY_SIZE."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NWVCMTSv0Vvo"
      },
      "source": [
        "### Pad sequences"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LxuPfShZ0Vvo",
        "outputId": "e7d99a64-5df2-4faa-907f-82b447b6caec"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Input sequence length: 5\n"
          ]
        }
      ],
      "source": [
        "X = pad_sequences(X, maxlen=MAX_SEQ_LENGTH, padding='pre')\n",
        "print('Input sequence length: {}'.format(MAX_SEQ_LENGTH))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0zESJhKf0Vvo"
      },
      "source": [
        "# 2. LSTM"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "collapsed": true,
        "id": "kKqsoGdR0Vvo"
      },
      "outputs": [],
      "source": [
        "# create model architecture\n",
        "\n",
        "EMBEDDING_SIZE = 100\n",
        "\n",
        "\n",
        "model = Sequential()\n",
        "\n",
        "# embedding layer\n",
        "model.add(Embedding(VOCABULARY_SIZE, EMBEDDING_SIZE, input_length = MAX_SEQ_LENGTH))\n",
        "\n",
        "# lstm layer 1\n",
        "model.add(LSTM(128, return_sequences=True))\n",
        "\n",
        "# lstm layer 2\n",
        "model.add(LSTM(128))\n",
        "\n",
        "# output layer\n",
        "model.add(Dense(VOCABULARY_SIZE, activation='softmax'))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EqooNN-g0Vvo",
        "outputId": "ca6f23f6-e4a8-4f2f-97a4-bf782eabe185"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " embedding (Embedding)       (None, 5, 100)            671300    \n",
            "                                                                 \n",
            " lstm (LSTM)                 (None, 5, 128)            117248    \n",
            "                                                                 \n",
            " lstm_1 (LSTM)               (None, 128)               131584    \n",
            "                                                                 \n",
            " dense (Dense)               (None, 6713)              865977    \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 1786109 (6.81 MB)\n",
            "Trainable params: 1786109 (6.81 MB)\n",
            "Non-trainable params: 0 (0.00 Byte)\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "# compile network\n",
        "model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
        "\n",
        "# summarize defined model\n",
        "model.summary()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cvY3R74rqbOu",
        "outputId": "a9345f3c-04df-4627-89ce-a09875c885e4"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/10\n",
            "306/306 - 20s - loss: 7.0135 - accuracy: 0.0570 - 20s/epoch - 64ms/step\n",
            "Epoch 2/10\n",
            "306/306 - 5s - loss: 6.5848 - accuracy: 0.0588 - 5s/epoch - 18ms/step\n",
            "Epoch 3/10\n",
            "306/306 - 5s - loss: 6.3931 - accuracy: 0.0721 - 5s/epoch - 15ms/step\n",
            "Epoch 4/10\n",
            "306/306 - 5s - loss: 6.2261 - accuracy: 0.0813 - 5s/epoch - 15ms/step\n",
            "Epoch 5/10\n",
            "306/306 - 4s - loss: 6.0985 - accuracy: 0.0853 - 4s/epoch - 14ms/step\n",
            "Epoch 6/10\n",
            "306/306 - 4s - loss: 5.9840 - accuracy: 0.0921 - 4s/epoch - 13ms/step\n",
            "Epoch 7/10\n",
            "306/306 - 4s - loss: 5.8744 - accuracy: 0.1007 - 4s/epoch - 14ms/step\n",
            "Epoch 8/10\n",
            "306/306 - 4s - loss: 5.7769 - accuracy: 0.1081 - 4s/epoch - 15ms/step\n",
            "Epoch 9/10\n",
            "306/306 - 5s - loss: 5.6906 - accuracy: 0.1147 - 5s/epoch - 17ms/step\n",
            "Epoch 10/10\n",
            "306/306 - 4s - loss: 5.6091 - accuracy: 0.1192 - 4s/epoch - 13ms/step\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.src.callbacks.History at 0x7b8226bd3ac0>"
            ]
          },
          "metadata": {},
          "execution_count": 24
        }
      ],
      "source": [
        "# fit network\n",
        "model.fit(X, y, epochs=10, verbose=2, batch_size=256)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rGwvJq3o0Vvo"
      },
      "source": [
        "### Load word embeddings to represent the input words"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "collapsed": true,
        "id": "SsmdygFq0Vvo"
      },
      "outputs": [],
      "source": [
        "# word2vec download link (Size ~ 1.5GB): https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit\n",
        "\n",
        "path = '/content/drive/MyDrive/GoogleNews-vectors-negative300.bin.gz'\n",
        "\n",
        "# load word2vec using the following function present in the gensim library\n",
        "word2vec = KeyedVectors.load_word2vec_format(path, binary=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "collapsed": true,
        "id": "IxbH4WpW0Vvo"
      },
      "outputs": [],
      "source": [
        "# assign word vectors from word2vec model\n",
        "\n",
        "EMBEDDING_SIZE  = 300  # each word in word2vec model is represented using a 300 dimensional vector\n",
        "VOCABULARY_SIZE = len(word_tokeniser.word_index) + 1\n",
        "\n",
        "# create an empty embedding matix\n",
        "embedding_weights = np.zeros((VOCABULARY_SIZE, EMBEDDING_SIZE))\n",
        "\n",
        "# create a word to index dictionary mapping\n",
        "word2id = word_tokeniser.word_index\n",
        "\n",
        "# copy vectors from word2vec model to the words present in corpus\n",
        "for word, index in word2id.items():\n",
        "    try:\n",
        "        embedding_weights[index, :] = word2vec[word]\n",
        "    except KeyError:\n",
        "        pass"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RXlMv1rF0Vvo",
        "outputId": "d1ee94b2-8a2c-41da-804f-935f53609341"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Embeddings shape: (6713, 300)\n"
          ]
        }
      ],
      "source": [
        "# check embedding dimension\n",
        "print(\"Embeddings shape: {}\".format(embedding_weights.shape))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "collapsed": true,
        "id": "NZcZVrU_0Vvo"
      },
      "outputs": [],
      "source": [
        "# create model architecture\n",
        "\n",
        "model_wv = Sequential()\n",
        "\n",
        "# embedding layer\n",
        "model_wv.add(Embedding(VOCABULARY_SIZE, EMBEDDING_SIZE, input_length = MAX_SEQ_LENGTH,\n",
        "                    weights = [embedding_weights], trainable=True))\n",
        "\n",
        "# lstm layer 1\n",
        "model_wv.add(LSTM(128, return_sequences=True))\n",
        "\n",
        "# lstm layer 2\n",
        "# when using multiple LSTM layers, set return_sequences to True at the previous layer\n",
        "# because the current layer expects a sequential intput rather than a single input\n",
        "model_wv.add(LSTM(128))\n",
        "\n",
        "# output layer\n",
        "model_wv.add(Dense(VOCABULARY_SIZE, activation='softmax'))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ggm7q8yd0Vvp",
        "outputId": "4b584320-43b8-4e62-888c-d2c3d70eaeb7"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"sequential_1\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " embedding_1 (Embedding)     (None, 5, 300)            2013900   \n",
            "                                                                 \n",
            " lstm_2 (LSTM)               (None, 5, 128)            219648    \n",
            "                                                                 \n",
            " lstm_3 (LSTM)               (None, 128)               131584    \n",
            "                                                                 \n",
            " dense_1 (Dense)             (None, 6713)              865977    \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 3231109 (12.33 MB)\n",
            "Trainable params: 3231109 (12.33 MB)\n",
            "Non-trainable params: 0 (0.00 Byte)\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "# compile network\n",
        "model_wv.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
        "\n",
        "# summarize defined model\n",
        "model_wv.summary()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "89054041-5caa-4f09-8907-a21d8af01126",
        "id": "BQR9oizO0Vvp"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/10\n",
            "306/306 [==============================] - 20s 50ms/step - loss: 6.9891 - accuracy: 0.0573\n",
            "Epoch 2/10\n",
            "306/306 [==============================] - 6s 18ms/step - loss: 6.5936 - accuracy: 0.0590\n",
            "Epoch 3/10\n",
            "306/306 [==============================] - 6s 19ms/step - loss: 6.3512 - accuracy: 0.0677\n",
            "Epoch 4/10\n",
            "306/306 [==============================] - 4s 15ms/step - loss: 6.1281 - accuracy: 0.0864\n",
            "Epoch 5/10\n",
            "306/306 [==============================] - 5s 16ms/step - loss: 5.9453 - accuracy: 0.0985\n",
            "Epoch 6/10\n",
            "306/306 [==============================] - 6s 19ms/step - loss: 5.8042 - accuracy: 0.1066\n",
            "Epoch 7/10\n",
            "306/306 [==============================] - 5s 15ms/step - loss: 5.6737 - accuracy: 0.1169\n",
            "Epoch 8/10\n",
            "306/306 [==============================] - 5s 15ms/step - loss: 5.5453 - accuracy: 0.1245\n",
            "Epoch 9/10\n",
            "306/306 [==============================] - 5s 16ms/step - loss: 5.4215 - accuracy: 0.1305\n",
            "Epoch 10/10\n",
            "306/306 [==============================] - 4s 15ms/step - loss: 5.3036 - accuracy: 0.1366\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.src.callbacks.History at 0x7b80e22760b0>"
            ]
          },
          "metadata": {},
          "execution_count": 30
        }
      ],
      "source": [
        "# fit network\n",
        "model_wv.fit(X, y, epochs=10, batch_size=256)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KmeOjpb_0Vvp"
      },
      "source": [
        "# 3. Generate text"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "collapsed": true,
        "id": "2NiH8h5oqkX5"
      },
      "outputs": [],
      "source": [
        "def generate_words(model, word_tokenizer, MAX_SEQ_LENGTH, seed, n_words):\n",
        "    output_text = seed\n",
        "    seed_text = seed.split()\n",
        "\n",
        "    for _ in range(n_words):\n",
        "\n",
        "        # Convert words to a sequence of indices\n",
        "        encoded_words = word_tokenizer.texts_to_sequences([seed_text])[0]\n",
        "\n",
        "        # Pad the sequence\n",
        "        padded_words = pad_sequences([encoded_words], maxlen=MAX_SEQ_LENGTH, truncating='pre')\n",
        "\n",
        "        # Predict next word (using `predict` instead of `predict_classes`)\n",
        "        prediction = model.predict(padded_words, verbose=0)\n",
        "\n",
        "        # Get the index of the highest probability class\n",
        "        predicted_index = np.argmax(prediction, axis=-1)[0]\n",
        "\n",
        "        # Find corresponding word in tokenizer and append to seed_text\n",
        "        for word, index in word_tokenizer.word_index.items():\n",
        "            if index == predicted_index:\n",
        "                output_text += ' ' + word\n",
        "                seed_text.append(word)\n",
        "                break\n",
        "\n",
        "        # Slide the window\n",
        "        seed_text = seed_text[1:]\n",
        "\n",
        "    return output_text"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Afv2bgMo0Vvp"
      },
      "source": [
        "### Let's look at some text generations"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gmhTo5KgjPFk",
        "outputId": "c4c91544-0e67-4dbc-8a3c-35ad2142f7cb"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "rama never told anyone about the king the king and all the king and all the king and all the king and all the king and all the king and all the king and all the king and all the king and all the king and all the king and all the king and all the king and all the king and all the king and all the king and all the king and all the king and all the king and all the king and all the king and all the king and all the king and all the king and all the king\n"
          ]
        }
      ],
      "source": [
        "# Suppose your model expects sequences of length 5\n",
        "MAX_SEQ_LENGTH = 5\n",
        "\n",
        "# text generation using first model - model without word embeddings\n",
        "seed_text = \"rama never told anyone about\"\n",
        "num_words = 100\n",
        "print(generate_words(model, word_tokeniser, MAX_SEQ_LENGTH, seed_text, num_words))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 33,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ba67b675-5812-4ecf-fd3e-dc85473fe087",
        "id": "acvWsEoD0Vvp"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "rama never told anyone about the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of all\n"
          ]
        }
      ],
      "source": [
        "# text generation using second model - model with word embeddings\n",
        "seed_text = \"rama never told anyone about\"\n",
        "num_words = 100\n",
        "print(generate_words(model_wv, word_tokeniser, MAX_SEQ_LENGTH, seed_text, num_words))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5caaaf4a-a6a2-4266-cc4f-94568ca789d3",
        "id": "1jaaDFQE0Vvp"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "how are you doing of the king to all the king and all the king and all the king and all the king and all the king and all the king and all the king and all the king and all the king and all the king and all the king and all the king and all the king and all the king and all the king and all the king and all the king and all the king and all the king and all the king and all the king and all the king and all the king and all the king and\n"
          ]
        }
      ],
      "source": [
        "# text generation using first model - model without word embeddings\n",
        "seed_text = \"how are you doing\"\n",
        "num_words = 100\n",
        "print(generate_words(model, word_tokeniser, MAX_SEQ_LENGTH, seed_text, num_words))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2da09899-7b60-4d68-ca71-37d7c7cdc01a",
        "id": "uTe71ZWn0Vvq"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "how are you doing spoke the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of all the king of\n"
          ]
        }
      ],
      "source": [
        "# text generation using second model - model with word embeddings\n",
        "seed_text = \"how are you doing\"\n",
        "num_words = 100\n",
        "print(generate_words(model_wv, word_tokeniser, MAX_SEQ_LENGTH, seed_text, num_words))"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.5"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}