ntuoss_nlp.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "ntuoss-nlp.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/wayneczw/ntuoss-nlp-workshop/blob/master/ntuoss_nlp.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "Fls3MxImeqWm",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#Introduction to NLP"
      ]
    },
    {
      "metadata": {
        "id": "akqUinl1eujc",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**What is [NLP](https://towardsdatascience.com/an-easy-introduction-to-natural-language-processing-b1e2801291c1)?**\n",
        "\n",
        "*\n",
        "Natural Language Processing (NLP) is a sub-field of Artificial Intelligence that is focused on enabling computers to understand and process human languages, to get computers closer to a human-level understanding of language. Computers don’t yet have the same intuitive understanding of natural language that humans do. They can’t really understand what the language is really trying to say. In a nutshell, a computer can’t read between the lines.\n",
        "*"
      ]
    },
    {
      "metadata": {
        "id": "Hm83222DpUgp",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**NLP Tasks**\n",
        "\n",
        "* [Information Retrieval](https://nlp.stanford.edu/IR-book/information-retrieval-book.html)\n",
        "* [Information Extraction](https://www.ontotext.com/knowledgehub/fundamentals/information-extraction/)\n",
        "* [Document Classification](https://cloud.google.com/blog/products/gcp/problem-solving-with-ml-automatic-document-classification)\n",
        "* [Summarization (Extractive/Abstractive)](https://towardsdatascience.com/a-quick-introduction-to-text-summarization-in-machine-learning-3d27ccf18a9f)\n",
        "* [Question Answering](https://medium.com/lingvo-masino/question-and-answering-in-natural-language-processing-part-i-168f00291856)\n",
        "* [Machine Translation](https://www.andovar.com/machine-translation/)\n",
        "* **Sentiment Analysis**\n",
        "* [Parsing](http://stp.lingfil.uu.se/~nivre/master/NLP-Parsing.pdf)\n",
        "* [Part of Speech (POS) Tagging](https://medium.freecodecamp.org/an-introduction-to-part-of-speech-tagging-and-the-hidden-markov-model-953d45338f24)\n",
        "* [Word Sense Disambiguation (WSD)](https://web.stanford.edu/~jurafsky/slp3/slides/Chapter18.wsd.pdf)\n",
        "* [Named Entity Recognition (NER)](https://towardsdatascience.com/named-entity-recognition-with-nltk-and-spacy-8c4a7d88e7da)\n",
        "* [Topic Modeling](https://www.analyticsvidhya.com/blog/2016/08/beginners-guide-to-topic-modeling-in-python/)\n",
        "* and more...."
      ]
    },
    {
      "metadata": {
        "id": "re9z__rvfqGw",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##Agenda\n",
        "\n",
        "\n",
        "*   Basic understanding to train/validation/test dataset and model selection.\n",
        "*   Basic techniques to preprocess/clean the text data.\n",
        "*   Brief introduction to commonly used text embeddings.\n",
        "*   Brief introduction to commonly used scoring metrics.\n",
        "*   Simple sentiment analysis toy experiment.\n",
        "*   Brief introduction to Universal Sentence Encoder\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "F5BetYmCkbRp",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#Task 1 - Prepare Data"
      ]
    },
    {
      "metadata": {
        "id": "3WX2BozCf95t",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##TASK 1.1 - Load In Data"
      ]
    },
    {
      "metadata": {
        "id": "3p01mxZPpHte",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "'''\n",
        "These data are adopted from\n",
        "https://github.com/Seh83/ML_Sentiment_Label_Model/tree/master/data\n",
        "'''\n",
        "\n",
        "from google.colab import files\n",
        "uploaded = files.upload()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "GHZFuOscpPZy",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!ls"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "zbkua6NPtG1O",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "with open(\"imdb_labelled.txt\", \"r\") as f:\n",
        "    str_data = f.read().split(\"\\n\")\n",
        "\n",
        "with open(\"amazon_cells_labelled.txt\", \"r\") as f:\n",
        "    str_data += f.read().split(\"\\n\")\n",
        "\n",
        "with open(\"yelp_labelled.txt\", \"r\") as f:\n",
        "    str_data += f.read().split(\"\\n\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "MHZviOcQhQZn",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Let's see how the raw text looks like:"
      ]
    },
    {
      "metadata": {
        "id": "sWQ90DiUhYpT",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "print(str_data[0])\n",
        "print(type(str_data[0]))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "NowDtXCLhhqE",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Manipulation of data types:"
      ]
    },
    {
      "metadata": {
        "id": "QoJ_Ss74uLrk",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "data = [line.split(\"\\t\") for line in str_data if len(line.split(\"\\t\")) == 2 and line.split(\"\\t\")[1]]\n",
        "print(data[0])\n",
        "print(type(data[0]))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "rxILnOxChrXb",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Split the data into X and Y:"
      ]
    },
    {
      "metadata": {
        "id": "IkmWS8TzvKW6",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "X = [line[0] for line in data]\n",
        "Y = [line[1] for line in data]\n",
        "print(\"X: {}\".format(X[0]))\n",
        "print(\"Y: {}\".format(Y[0]))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "tLWyzHLuolm5",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##Task 1.2 - Split Data"
      ]
    },
    {
      "metadata": {
        "id": "Dd4ij31dh4QA",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Train-Test Split\n",
        "**Train set**: \n",
        "Train the model.\n",
        "\n",
        "**Test set**:\n",
        "Assess the model performance + Model Selction\n",
        "\n",
        "**Validation set (sometimes optional but usually strongly encouraged)**: \n",
        "Model selection + Early Stopping\n",
        "\n",
        "The usual ratio of train/validation/test goes as follow:\n",
        "- *raw data* -> 80% Train + 20% Test OR 70% Train + 30% Test\n",
        "- *Train* -> 80% Train + 20% Validation OR 90% Train + 10% Validation\n",
        "\n",
        "### Model Selection\n",
        "\n",
        "**Overfitting**\n",
        "When the model trained is fitted too closely to the train set, the model may lose its ability to generalize to untrained data. This essentially means that an overfitted model is unlikely to make accurate prediction on unseen data.\n",
        "\n",
        "**Underfitting**\n",
        "When the model fails to identify the pattern in the train set, it it very unlikely that it can do well in predicting unseen data.\n",
        "\n",
        "<!-- Models are trained on data, and data is supposedly a good representation of the *true population*. However, more than often, the data available for training are not good enough due to:\n",
        "- insufficient data\n",
        "- unbalance data mix\n",
        "- etc\n",
        "\n",
        "Due to the inevitable defect in the data itself, we need to apply some techniques to select the best model among different models: -->\n",
        "\n",
        "Common methods:\n",
        "- **K-fold Cross Validation**\n",
        "- Leave-One-Out (LOO) Cross Validation\n",
        "- K Data Splits Random SubSampling\n",
        "- Three-Way Data Splits Method\n",
        "\n",
        "Want to know more: https://towardsdatascience.com/train-test-split-and-cross-validation-in-python-80b61beca4b6"
      ]
    },
    {
      "metadata": {
        "id": "m2fNsUOEv09r",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "np.random.seed(7)\n",
        "\n",
        "X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2)\n",
        "\n",
        "print(len(X_train))\n",
        "print(len(X_test))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "i7azJCuyk8QD",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##Task 1.3- Preprocess Data"
      ]
    },
    {
      "metadata": {
        "id": "ounlw0Tp1_cf",
        "colab_type": "code",
        "outputId": "c55bb3d4-2f40-4927-d325-d07b1963736f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "print(X_train[1])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Also, the fries are without a doubt the worst fries I've ever had.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "TjuGaW0Oo-wi",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##Preprocessing Techniques\n",
        "\n",
        "**Regex**:  a tool to clean up unwanted text.\n",
        "\n",
        "Eg1 re.sub(r'[^\\w]', '', text), which will remove all non-word tokens; 'This is a !@#!$ token.' -> 'This is a  token.'\n",
        "\n",
        "**Stopwords**: words which are filtered out before or after processing of natural language data (text). Usually the commonly seen words such as *I, we, you, and, this*.\n",
        "\n",
        "**Stemming**: stemming is the process of reducing inflected (or sometimes derived) words to their word stem, base or root form—generally a written word form.\n",
        "\n",
        "Eg1 ideas -> idea\n",
        "Eg2 policy -> polic\n",
        "        police -> polic\n",
        "\n",
        "**Lemmatization**: algorithmic process of determining the lemma for a given word. Lemmatization is very similar to stemming, except that stemming is a lot more brutal.\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "CK9W0NrYxk7u",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import nltk\n",
        "import re\n",
        "nltk.download('stopwords'),nltk.download('snowball_data')\n",
        "from nltk.stem.snowball import SnowballStemmer\n",
        "from nltk.corpus import stopwords\n",
        "\n",
        "stop_words = set(stopwords.words(\"english\"))\n",
        "stemmer = SnowballStemmer(\"english\")\n",
        "\n",
        "def pre_process(text):\n",
        "    if not isinstance(text, str): text = str(text)\n",
        "\n",
        "    z = re.sub(r'[^\\w\\d\\s]', ' ', text)\n",
        "    z = re.sub(r'\\s+', ' ', z)\n",
        "    z = re.sub(r'^\\s+|\\s+?$', '', z.lower())\n",
        "    return ' '.join(stemmer.stem(token) for token in z.split() if token not in set(stop_words))\n",
        "#end def\n",
        "\n",
        "X_train_processed = [pre_process(x) for x in X_train]\n",
        "X_test_processed = [pre_process(x) for x in X_test]\n",
        "\n",
        "print(X_train_processed[1])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "JEid7x1S4BwZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#Task 2 - Train\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "XUMzuMr3rl-a",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##Common Text Embeddings\n",
        "\n",
        "Text is an unstructure data, so computer cannot understand them. Therefore, there is a need to transform the text into a numerical vector/matrix.\n",
        "\n",
        "\n",
        "**CountVectorizer**: aka Bag of Words/N-gram\n",
        "\n",
        "\n",
        "\n",
        "**TfidfVectorizer**: Term-frequency-inverse-document-frequency. This technique takes into account of the document level frequency, and penalise tokens that appear often among different documents."
      ]
    },
    {
      "metadata": {
        "id": "Pr0etrR_4Jw1",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "\n",
        "count = CountVectorizer(ngram_range=(1, 2))\n",
        "tfidf =TfidfVectorizer(ngram_range=(1, 2))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "crf_29Jk5xwx",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "X_train_count = count.fit_transform(X_train_processed)\n",
        "X_test_count = count.transform(X_test_processed)\n",
        "\n",
        "X_train_tfidf = tfidf.fit_transform(X_train_processed)\n",
        "X_test_tfidf = tfidf.transform(X_test_processed)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "SfDs0MOn7FIg",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from sklearn.linear_model import LogisticRegressionCV\n",
        "from sklearn.metrics import f1_score\n",
        "\n",
        "# Using CounterVectorizer\n",
        "classifier = LogisticRegressionCV(cv=5, random_state=0, multi_class='ovr')\n",
        "classifier.fit(X_train_count, Y_train)\n",
        "\n",
        "count_predicts = classifier.predict(X_test_count)\n",
        "\n",
        "# Using TfidfVectorizer\n",
        "classifier.fit(X_train_tfidf, Y_train)\n",
        "tfidf_predicts = classifier.predict(X_test_tfidf)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "z20ms-XItjau",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Logistic regression** is a statistical method for analyzing a dataset in which there are one or more independent variables that determine an outcome. The outcome is measured with a dichotomous variable (in which there are only two possible outcomes).\n",
        "\n",
        "**LogisticRegressionCV** is a sklearn implementation of logistic regression classifier, with **Cross Validation** technique being used."
      ]
    },
    {
      "metadata": {
        "id": "qrc65vpt_bw3",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#Task 3 - Evaluation"
      ]
    },
    {
      "metadata": {
        "id": "z-YXK_KA-Y9m",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from sklearn.metrics import classification_report\n",
        "\n",
        "print(\"Classification Report for CountVectorizer:\\n{}\".format(classification_report(Y_test, count_predicts)))\n",
        "\n",
        "print(\"Classification Report for TfidfVectorizer:\\n{}\".format(classification_report(Y_test, tfidf_predicts)))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "z5X3DRtJuJCH",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Precision** is the number of True Positives divided by the number of True Positives and False Positives. Put another way, it is the number of positive predictions divided by the total number of positive class values predicted.\n",
        "\n",
        "**Recall** is the number of True Positives divided by the number of True Positives and the number of False Negatives. Put another way it is the number of positive predictions divided by the number of positive class values in the test data.\n",
        "\n",
        "**F1 Score** is the **2*((precision*recall)/(precision+recall))**. It is also called the F Score or the F Measure. Put another way, the F1 score conveys the balance between the precision and the recall.\n",
        "\n",
        "Want to know more: https://machinelearningmastery.com/classification-accuracy-is-not-enough-more-performance-measures-you-can-use/"
      ]
    },
    {
      "metadata": {
        "id": "bCx8I-wn_994",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Task 4 - USE Embedding with Neural Network"
      ]
    },
    {
      "metadata": {
        "id": "n8NFNWfpG3gv",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!pip3 install --quiet tensorflow-hub\n",
        "!pip3 install keras\n",
        "import tensorflow as tf\n",
        "import tensorflow_hub as hub\n",
        "from keras.layers import Dense\n",
        "from keras.layers import Input\n",
        "from keras.layers import Lambda\n",
        "from keras.models import Model\n",
        "from keras import backend as K\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "UfHzYU7tbDl4",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##Task 4.1 - Introduce to USE"
      ]
    },
    {
      "metadata": {
        "id": "gJAkudOMuq-n",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The [Universal Sentence Encoder](https://tfhub.dev/google/universal-sentence-encoder/1) encodes text into high dimensional vectors that can be used for text classification, semantic similarity, clustering and other natural language tasks.\n",
        "\n",
        "There are many other encoders:\n",
        "- [elmo](https://tfhub.dev/google/elmo/1)\n",
        "- [GloVe](https://nlp.stanford.edu/projects/glove/)\n",
        "- [Word2Vec](https://towardsdatascience.com/introduction-to-word-embedding-and-word2vec-652d0c2060fa)\n",
        "- [Doc2Vec](https://medium.com/scaleabout/a-gentle-introduction-to-doc2vec-db3e8c0cce5e)\n",
        "- etc"
      ]
    },
    {
      "metadata": {
        "id": "6oPF5ogaAEzU",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "'''Codes in this cell are adopted, with slight modifications, from \n",
        "https://colab.research.google.com/github/tensorflow/hub/blob/master/examples/colab/semantic_similarity_with_tf_hub_universal_encoder.ipynb\n",
        "'''\n",
        "\n",
        "module_url = \"https://tfhub.dev/google/universal-sentence-encoder-large/3\"\n",
        "embed = hub.Module(module_url)\n",
        "\n",
        "# Reduce logging output.\n",
        "tf.logging.set_verbosity(tf.logging.ERROR)\n",
        "\n",
        "with tf.Session() as session:\n",
        "    session.run([tf.global_variables_initializer(), tf.tables_initializer()])\n",
        "    X_train_use = session.run(embed(X_train))\n",
        "\n",
        "    for i, embedding in enumerate(np.array(X_train_use).tolist()):\n",
        "        print(\"Original: {}\".format(X_train[i]))\n",
        "        print(\"Embedding size: {}\".format(len(embedding)))\n",
        "        embedding_snippet = \", \".join(\n",
        "            (str(x) for x in embedding[:3]))    \n",
        "        print(\"Embedding: [{}, ...]\\n\".format(embedding_snippet))\n",
        "        \n",
        "        if i == 5: break"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "o1wGU7xEa5C1",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Task 4.2 - Build a NN Model with USE Embedding"
      ]
    },
    {
      "metadata": {
        "id": "T9oWQ4XNMcGw",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "X_train, X_validation, Y_train, Y_validation = train_test_split(X_train, Y_train, test_size=0.2)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "jjc76d6LS5V8",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def _batch_iter(X, Y, batch_size=32, **kwargs):\n",
        "    data_size = len(Y)\n",
        "    num_batches_per_epoch = int((data_size - 1) / batch_size) + 1\n",
        "\n",
        "    def data_generator():\n",
        "        while True:\n",
        "            # Shuffle the data at each epoch\n",
        "            shuffled_indices = np.random.permutation(np.arange(data_size, dtype=np.int))\n",
        "\n",
        "            for batch_num in range(num_batches_per_epoch):\n",
        "                start_index = batch_num * batch_size\n",
        "                end_index = min((batch_num + 1) * batch_size, data_size)\n",
        "                X_batch = [X[i] for i in shuffled_indices[start_index:end_index]]\n",
        "                Y_batch = [Y[i] for i in shuffled_indices[start_index:end_index]]\n",
        "\n",
        "                yield ({'x_input': np.asarray(X_batch)}, {'output': np.asarray(Y_batch)})\n",
        "            #end for\n",
        "        #end while\n",
        "    #end def\n",
        "\n",
        "    return num_batches_per_epoch, data_generator()\n",
        "#end def\n",
        "\n",
        "train_steps, train_batches = _batch_iter(X_train, Y_train)\n",
        "validate_steps, validate_batches = _batch_iter(X_validation, Y_validation)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "CVnFxo7vLsd_",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "USE_MODULE_URL = \"https://tfhub.dev/google/universal-sentence-encoder/2\"\n",
        "USE_EMBED = hub.Module(USE_MODULE_URL, trainable=True)\n",
        "\n",
        "\n",
        "def USE_Embedding(x):\n",
        "    return USE_EMBED(tf.squeeze(tf.cast(x, tf.string)), signature=\"default\", as_dict=True)[\"default\"]\n",
        "#end def\n",
        "\n",
        "\n",
        "# Initialize session\n",
        "with tf.Session() as session:\n",
        "    K.set_session(session)\n",
        "    session.run([tf.global_variables_initializer(), tf.tables_initializer()])\n",
        "    tf.logging.set_verbosity(tf.logging.ERROR)\n",
        "\n",
        "    x_input = Input(shape=(1,), dtype=tf.string, name='x_input')\n",
        "    x_embed = Lambda(USE_Embedding, output_shape=(512,))(x_input)\n",
        "    x = Dense(256, activation='relu')(x_embed)\n",
        "    output = Dense(1, activation='sigmoid', name='output')(x)\n",
        "\n",
        "    model = Model(inputs=[x_input], outputs=output)\n",
        "    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
        "    model.summary()\n",
        "    model.fit_generator(\n",
        "            epochs=10,\n",
        "            generator=train_batches,\n",
        "            steps_per_epoch=train_steps,\n",
        "            validation_data=validate_batches,\n",
        "            validation_steps=validate_steps)\n",
        "    \n",
        "    X_test = np.array(X_test, dtype=object)\n",
        "    Y_test = np.array(Y_test, dtype=int)\n",
        "    threshold = sum(Y_test)/Y_test.shape[0]\n",
        "    use_predicts = model.predict(X_test)\n",
        "    use_predicts = [1 if i > threshold else 0 for i in use_predicts]\n",
        "    print(\"Classification Report for USE:\\n{}\".format(classification_report(Y_test, use_predicts)))\n"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}