train_and_predict.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Yelp Review Quality Prediction based on Text, Useful, Funny, and Cool Counts\n",
    "\n",
    "Authors: Kushal D'Souza and Linda Wong\n",
    "Date: 12/10/2019\n",
    "\n",
    "The first step in our project is to import the data in order to analyze the dataset. We believe that analyzing the data before preparing the model is an important aspect of any machine learning task. Please provide the location of the Yelp dataset in the input box below after which, we will extract the reviews that are stricly related to restaurants.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip3 install gensim\n",
    "!pip3 install keras\n",
    "!pip3 install nltk\n",
    "!pip3 install numpy\n",
    "!pip3 install pandas\n",
    "!pip3 install sklearn\n",
    "!pip3 install tensorflow==1.15\n",
    "!pip3 install wordcloud\n",
    "!pip install gensim\n",
    "!pip install keras\n",
    "!pip install nltk\n",
    "!pip install numpy\n",
    "!pip install pandas\n",
    "!pip install sklearn\n",
    "!pip install tensorflow==1.15\n",
    "!pip install wordcloud\n",
    "\n",
    "import gensim\n",
    "from gensim.models import word2vec\n",
    "from itertools import cycle\n",
    "import json\n",
    "from keras.utils import np_utils\n",
    "import math\n",
    "import matplotlib.pyplot as plt\n",
    "import nltk\n",
    "import numpy as np\n",
    "import os\n",
    "import pandas as pd\n",
    "import re\n",
    "from sklearn import preprocessing\n",
    "from sklearn.cluster import KMeans\n",
    "from sklearn.manifold import TSNE\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.neighbors import KDTree\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "import sys\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras import Sequential\n",
    "from tensorflow.keras.layers import Bidirectional, Dense, Embedding, SimpleRNN, LSTM\n",
    "from wordcloud import WordCloud, ImageColorGenerator\n",
    "from sklearn.externals import joblib\n",
    "\n",
    "\n",
    "nltk.download('all')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Please run then cell below and then follow the instructions provided hereafter. On running the cell a text input box will appear where you SHOULD enter the path to your dataset folder. A dropdown box to select one of the options from useful, funny or cool that you want to predict will also appear. Please select one of the options from 'useful', 'funny' or 'cool' in the dropdown box.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import display\n",
    "from ipywidgets import widgets\n",
    "text = widgets.Text()\n",
    "display(text)\n",
    "\n",
    "dataset_folder = '/Users/lindawong/Documents/Github/yelp_review_prediction/yelp_dataset'\n",
    "\n",
    "def handle_change(change):\n",
    "    if change['type'] == 'change' and change['name'] == 'value':\n",
    "        global dataset_folder\n",
    "        dataset_folder = change['new']\n",
    "\n",
    "text.observe(handle_change)\n",
    "\n",
    "\n",
    "dd_widget = widgets.Dropdown(\n",
    "    options=['useful', 'funny', 'cool'],\n",
    "    value='useful',\n",
    "    description='Data Type:',\n",
    "    disabled=False,\n",
    ")\n",
    "\n",
    "display(dd_widget)\n",
    "\n",
    "data_type = 'useful'\n",
    "def on_change(change):\n",
    "    if change['type'] == 'change' and change['name'] == 'value':\n",
    "        global data_type\n",
    "        data_type = change['new']\n",
    "        \n",
    "\n",
    "dd_widget.observe(on_change)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please verify that your folder_location and data type are as intended after running the cell below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(data_type)\n",
    "print(dataset_folder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Extraction\n",
    "In the following cells, we extract data from the review.json and business.json files. The main data that we will work with is the review text for each review along with its count for 'useful', 'funny' and 'cool'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# functions for data extraction\n",
    "def extract_data(business_data_folder, review_data_folder):\n",
    "    output_data = {}\n",
    "    with open(business_data_folder) as bd_handle:\n",
    "        business = bd_handle.readline()\n",
    "\n",
    "        while business:\n",
    "            bd_data = json.loads(business)\n",
    "\n",
    "            if bd_data['categories']:\n",
    "                output_data[bd_data['business_id']] = {\n",
    "                    'categories': [cat.strip() for cat in bd_data['categories'].split(',')],\n",
    "                    'reviews': []\n",
    "                }\n",
    "\n",
    "            business = bd_handle.readline()\n",
    "\n",
    "        bd_handle.close()\n",
    "\n",
    "    with open(review_data_folder) as rd_handle:\n",
    "        review = rd_handle.readline()\n",
    "\n",
    "        while review:\n",
    "            rv_data = json.loads(review)\n",
    "\n",
    "            if rv_data['business_id'] in output_data:\n",
    "                review = {}\n",
    "                review['review_id'] = rv_data['review_id']\n",
    "                review['text'] = rv_data['text']\n",
    "                review['useful'] = rv_data['useful']\n",
    "                review['funny'] = rv_data['funny']\n",
    "                review['cool'] = rv_data['cool']\n",
    "                output_data[rv_data['business_id']]['reviews'].append(review)\n",
    "\n",
    "            review = rd_handle.readline()\n",
    "\n",
    "        bd_handle.close()\n",
    "        \n",
    "    return output_data\n",
    "\n",
    "def write_extracted_data(output_data, output_file):\n",
    "    data_to_write = []\n",
    "    for business_id in output_data:\n",
    "        categories = output_data[business_id]['categories']\n",
    "\n",
    "        if 'Restaurants' in categories and len(output_data[business_id]['reviews']) > 0:\n",
    "            cat_to_write = ':'.join(categories)\n",
    "            for review in output_data[business_id]['reviews']:\n",
    "                review_id = review['review_id']\n",
    "                text = review['text']\n",
    "                useful = str(review['useful'])\n",
    "                funny = str(review['funny'])\n",
    "                cool = str(review['cool'])\n",
    "                data_to_write.append({\n",
    "                    'business_id': business_id,\n",
    "                    'categories': cat_to_write,\n",
    "                    'review_id': review_id,\n",
    "                    'text': text,\n",
    "                    'useful': useful,\n",
    "                    'funny': funny,\n",
    "                    'cool': cool\n",
    "                })           \n",
    "\n",
    "    with open(output_file, 'w') as of_handle:\n",
    "        json.dump(data_to_write, of_handle)\n",
    "        of_handle.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "business_data_folder = os.path.join(dataset_folder, 'business.json')\n",
    "review_data_folder = os.path.join(dataset_folder, 'review.json')\n",
    "output_data = extract_data(business_data_folder, review_data_folder)\n",
    "dataset_file = './dataset.json'\n",
    "write_extracted_data(output_data, dataset_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, the data we need from review.json and business.json are stored in the file \"dateset.json\". Next, we will clean up the data and write the preprocessed datasets to \"normalised_data.json\".\n",
    "\n",
    "## Data Analysis and Cleansing:\n",
    "\n",
    "In the next cell, we analyze the data, specifically the distribution of values within the 'useful', 'funny' and 'cool' columns. We notice that the data is heavily skewed towards values between 0 to 5. Most of the values for 'useful', 'funny' and 'cool' lie between 0 and 20. There are outliers that occur in the data with some values going up to more than 500. \n",
    "\n",
    "In order to enable our model to accurately predict how 'useful', 'funny' or 'cool' a certain review is, we apply feature clipping on each of these columns. We also normalised these values so that they lie in the range of 0 to 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# functions for data preprocessing\n",
    "def preprocess_data(data_file):\n",
    "\n",
    "    df = pd.read_json(data_file)\n",
    "    df['useful'] = df['useful'].astype(float)\n",
    "    df['funny'] = df['funny'].astype(float)\n",
    "    df['cool'] = df['cool'].astype(float)\n",
    "    print(\"Dataset size: {0:d} rows x {1:d} columns\".format(df.shape[0], df.shape[1]))\n",
    "\n",
    "    vote_labels = ['useful', 'funny', 'cool']\n",
    "    useful_min, useful_max = df['useful'].min(), df['useful'].max()\n",
    "    funny_min, funny_max = df['funny'].min(), df['funny'].max()\n",
    "    cool_min, cool_max = df['cool'].min(), df['cool'].max()\n",
    "\n",
    "    # print out distribution of reviews by their vote counts\n",
    "    for i in range(len(vote_labels)):\n",
    "        ranges = np.arange(-1, 100, 10)\n",
    "        counts = df.groupby(pd.cut(df[vote_labels[i]], ranges)).count()[vote_labels[i]]\n",
    "        print(counts)\n",
    "        print(\"Out of range counts: \", df.shape[0]-counts.sum())\n",
    "\n",
    "    # save raw vote counts\n",
    "    df['useful_raw'] = df['useful']\n",
    "    df['funny_raw'] = df['funny']\n",
    "    df['cool_raw'] = df['cool']\n",
    "    # clip vote counts to specific ranges\n",
    "    df['useful'] = df['useful'].clip(3.0, 27.0)\n",
    "    df['funny'] = df['funny'].clip(3.0, 20.0)\n",
    "    df['cool'] = df['cool'].clip(3.0, 24.0)\n",
    "\n",
    "    # normalize data with min-max scaling\n",
    "    min_max_scaler = preprocessing.MinMaxScaler()\n",
    "    useful_scaled = min_max_scaler.fit_transform(np.array(df['useful']).reshape(-1,1))\n",
    "    df['useful'] = pd.DataFrame(useful_scaled)\n",
    "    funny_scaled = min_max_scaler.fit_transform(np.array(df['funny']).reshape(-1,1))\n",
    "    df['funny'] = pd.DataFrame(funny_scaled)\n",
    "    cool_scaled = min_max_scaler.fit_transform(np.array(df['cool']).reshape(-1,1))\n",
    "    df['cool'] = pd.DataFrame(cool_scaled)\n",
    "\n",
    "    ranges = [-0.1, 0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]\n",
    "    processed_data = []\n",
    "\n",
    "    # balance dataset for each of vote category\n",
    "    # make sure there is an even number of reviews in each increment of 0.1\n",
    "    # from range 0.0 to 1.0\n",
    "    for i in range(len(vote_labels)):\n",
    "        counts = df.groupby(pd.cut(df[vote_labels[i]], ranges)).count()[vote_labels[i]]\n",
    "        min_count = counts.min()\n",
    "        new_df = pd.DataFrame()\n",
    "        for j in range(10):\n",
    "            lower = j/10\n",
    "            upper = (j+1)/10 if j < 9 else 1.1\n",
    "            new_data = df[(df[vote_labels[i]] >= lower) & (df[vote_labels[i]] < upper)]\n",
    "            new_data = new_data.sample(frac=1)\n",
    "            new_data = new_data.head(min_count)\n",
    "            new_df = new_df.append(new_data)\n",
    "\n",
    "        processed_data.append(new_df)\n",
    "    \n",
    "    return processed_data\n",
    "\n",
    "\n",
    "def write_extracted_data(useful, funny, cool):\n",
    "    output_dict = {\n",
    "        'useful': useful.to_json(orient = 'records'),\n",
    "        'funny': funny.to_json(orient = 'records'),\n",
    "        'cool': cool.to_json(orient = 'records')\n",
    "    }\n",
    "\n",
    "    with open('normalised_data.json', 'w') as op_handle:\n",
    "        json.dump(output_dict, op_handle)\n",
    "        op_handle.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "processed_data = preprocess_data(dataset_file)\n",
    "write_extracted_data(processed_data[0], processed_data[1], processed_data[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Word2Vec Model:\n",
    "\n",
    "In the next step, we cleanse each review by removing any punctuation and extra spaces. We also lowercase all the words in the review and remove any stopwords from the data. We then use this cleansed dataset to train a Word2Vec model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# functions for creating the word2vec model\n",
    "def read_data(dataset_file):\n",
    "    print(\"Reading data...\")\n",
    "    with open(dataset_file, 'r') as fh:\n",
    "        full_data = json.load(fh)\n",
    "        useful_data = pd.DataFrame(json.loads(full_data['useful']))\n",
    "        funny_data = pd.DataFrame(json.loads(full_data['funny']))\n",
    "        cool_data = pd.DataFrame(json.loads(full_data['cool']))\n",
    "        all_data = useful_data.append([funny_data, cool_data])\n",
    "    \n",
    "    return useful_data, funny_data, cool_data, all_data\n",
    "\n",
    "\n",
    "def clean_data(tokenized_data_file, all_data, tokenizer, stop_words):\n",
    "    print(\"Cleaning data...\")\n",
    "    with open(tokenized_data_file, 'w') as fh:\n",
    "        reviews = all_data['text'].tolist()\n",
    "\n",
    "        for index in range(len(reviews)):\n",
    "            review = reviews[index]\n",
    "            no_tabs = str(review).replace('\\t', ' ').replace('\\n', '')\n",
    "            alphas_only = re.sub(\"[^a-zA-Z\\.]\", \" \", no_tabs)\n",
    "            multi_spaces = re.sub(\" +\", \" \", alphas_only)\n",
    "            no_spaces = multi_spaces.strip()\n",
    "            clean_text = no_spaces.lower()\n",
    "            sentences = tokenizer.tokenize(clean_text)\n",
    "            sentences = [re.sub(\"[\\.]\", \"\", sentence) for sentence in sentences]\n",
    "\n",
    "            if len(clean_text) > 0 and clean_text.count(' ') > 0:\n",
    "                for sentence in sentences:\n",
    "                    sentence = sentence.split(' ')\n",
    "                    pruned_sentence = [word for word in sentence if word not in stop_words]\n",
    "                    sentence = ' '.join(pruned_sentence)\n",
    "                    if sentence: \n",
    "                        fh.write(\"%s\\n\" % sentence)\n",
    "\n",
    "            if (index % 5000) == 0:\n",
    "                fh.flush()\n",
    "        fh.close()\n",
    "\n",
    "\n",
    "def create_wve(tokenized_data_file):\n",
    "    embedding_size = 256\n",
    "    min_word_count = 50\n",
    "    context = 30\n",
    "    downsampling = 1e-2\n",
    "    print(\"Creating word vectors...\")\n",
    "    model = word2vec.Word2Vec(\n",
    "        word2vec.LineSentence(tokenized_data_file),\n",
    "        size=embedding_size,\n",
    "        min_count=min_word_count,\n",
    "        window=context,\n",
    "        sample=downsampling)\n",
    "    model.init_sims(replace=True)\n",
    "    model.save('./wve.model')\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "useful_data, funny_data, cool_data, all_data = read_data('normalised_data.json')\n",
    "tokenized_data_file = './tokennized.txt'\n",
    "tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')\n",
    "stop_words = nltk.corpus.stopwords.words()\n",
    "clean_data(tokenized_data_file, all_data, tokenizer, stop_words)\n",
    "wve_model = create_wve(tokenized_data_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regression and Classification Model with LSTM:\n",
    "\n",
    "In the following sections, we attempt to predict the preprocessed values of labels (‘useful’, ‘funny’, and ‘cool’) for each review. In order to do that, we create a recurrent neural network model with LSTM units with review text represented as word vector embeddings for the input and a predicted value for the output.\n",
    "\n",
    "The model below also includes another version which does binary classification instead of regression, and we will experiment with both versions.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# implementation of LSTM model\n",
    "class LSTMModel:\n",
    "    def __init__(self, data_file, label):\n",
    "        self.data_file = data_file\n",
    "        self.label = label\n",
    "        self.df = []\n",
    "        self.text = []\n",
    "        self.num_reviews = 0\n",
    "        self.max_length = 0\n",
    "        self.embedding_dim = 256\n",
    "        self.batch_size = 32\n",
    "        self.lstm_dim = 100\n",
    "        self.text_embedding = None\n",
    "        self.model = None\n",
    "        self.train_text = None\n",
    "        self.test_text = None\n",
    "        self.train_labels = None\n",
    "        self.test_labels = None\n",
    "\n",
    "    def read_data(self):        \n",
    "        # read normalized datatsets\n",
    "        with open(self.data_file, 'r') as f:\n",
    "            full_data = json.load(f)\n",
    "            self.df = pd.DataFrame(json.loads(full_data[self.label]))\n",
    "            self.df = self.df.sample(frac=1).reset_index(drop=True)\n",
    "\n",
    "    def create_input_embedding(self):\n",
    "        # preprocess text\n",
    "        # remove newlines, spaces, and punctuations\n",
    "        # lowercase words and separate by commas\n",
    "        for i in range(self.df.shape[0]):\n",
    "            line = gensim.utils.simple_preprocess(self.df['text'].iloc[i])\n",
    "            self.text.append(line)\n",
    "\n",
    "        length = max([len(s) for s in self.text])\n",
    "        self.num_reviews = len(self.text)\n",
    "        \n",
    "        # convert text to word vector embedding\n",
    "        text_embedding = np.zeros((self.num_reviews, length, self.embedding_dim))\n",
    "        for i in range(len(self.text)):\n",
    "            idx = 0\n",
    "            for j in range(len(self.text[i])):\n",
    "                # get jth word of ith review\n",
    "                word = self.text[i][j]\n",
    "                # if word is in vocab, add its vector embedding, else skip\n",
    "                if word in wve_model.wv.vocab:\n",
    "                    vec = wve_model.wv[word]\n",
    "                    text_embedding[i,idx,:] = vec\n",
    "                    idx += 1\n",
    "            if idx > self.max_length:\n",
    "                self.max_length = idx\n",
    "        \n",
    "        self.text_embedding = text_embedding[:,:self.max_length,:]\n",
    "        print(\"Text embedding size: \", text_embedding.shape)\n",
    "        \n",
    "    def build_regression_model(self):\n",
    "        # construct regression model\n",
    "        self.model = Sequential()\n",
    "        self.model.add(Dense(self.batch_size, input_shape=(self.max_length, self.embedding_dim)))\n",
    "        self.model.add(Dense(300, activation='relu'))\n",
    "        self.model.add(Dense(150, activation='relu'))\n",
    "        self.model.add(LSTM(self.lstm_dim, return_sequences=False))\n",
    "        self.model.add(Dense(1, activation='linear'))\n",
    "\n",
    "        self.model.compile(loss='mae', \n",
    "                           optimizer='adam', \n",
    "                           metrics=['mae', 'acc'])\n",
    "        \n",
    "    def train_regression_model(self):\n",
    "        labels = self.df[self.label].values\n",
    "        self.train_text, self.test_text, self.train_labels, self.test_labels = \\\n",
    "            train_test_split(self.text_embedding, labels, test_size=0.3)\n",
    "\n",
    "        print(\"Train data size: \", self.train_text.shape, self.train_labels.shape)\n",
    "        print(\"Test data size: \", self.test_text.shape, self.test_labels.shape)\n",
    "\n",
    "        i = self.num_reviews // self.batch_size\n",
    "        i *= self.batch_size\n",
    "        self.train_text = self.train_text[:i,:,:]\n",
    "        self.train_labels = self.train_labels[:i]\n",
    "\n",
    "        self.model.fit(self.train_text,self.train_labels, \n",
    "                  batch_size=self.batch_size, shuffle=True, epochs=7)\n",
    "        \n",
    "    def test_regression_model(self):\n",
    "        self.model.evaluate(self.test_text, self.test_labels)\n",
    "        \n",
    "    def build_classification_model(self):\n",
    "        print(\"Building model...\")\n",
    "        # construct classification model\n",
    "        self.model = Sequential()\n",
    "        self.model.add(Dense(self.batch_size, input_shape=(self.max_length, self.embedding_dim)))\n",
    "        self.model.add(Dense(300, activation='relu'))\n",
    "        self.model.add(Dense(150, activation='relu'))\n",
    "        self.model.add(LSTM(self.lstm_dim, return_sequences=False))\n",
    "        self.model.add(Dense(1, activation='sigmoid'))\n",
    "\n",
    "        self.model.compile(loss='binary_crossentropy',\n",
    "                           optimizer='adam',\n",
    "                           metrics=['acc'])\n",
    "        \n",
    "    def train_classification_model(self):\n",
    "        # convert labels to 0 and 1\n",
    "        # 0 = not useful, 1 = useful\n",
    "        binary_labels = []\n",
    "        raw_labels = self.df[self.label].values\n",
    "        for l in raw_labels:\n",
    "            if l < 0.5:\n",
    "                binary_labels.append(0)\n",
    "            else:\n",
    "                binary_labels.append(1)\n",
    "        encoder = LabelEncoder()\n",
    "        encoder.fit(binary_labels)\n",
    "        labels = encoder.transform(binary_labels)\n",
    "\n",
    "        # divide into train and test sets\n",
    "        self.train_text, self.test_text, self.train_labels, self.test_labels = \\\n",
    "            train_test_split(self.text_embedding, labels, test_size=0.3)\n",
    "\n",
    "        print(\"Train data size: \", self.train_text.shape, self.train_labels.shape)\n",
    "        print(\"Test data size: \", self.test_text.shape, self.test_labels.shape)\n",
    "\n",
    "        i = self.num_reviews // self.batch_size\n",
    "        i *= self.batch_size\n",
    "        self.train_text = self.train_text[:i,:,:]\n",
    "        self.train_labels = self.train_labels[:i]\n",
    "\n",
    "        print(\"Training model...\")\n",
    "        self.model.fit(self.train_text,self.train_labels, \n",
    "                  batch_size=self.batch_size, shuffle=True, epochs=7)\n",
    "        \n",
    "    def test_classification_model(self):\n",
    "        print(\"Predicting123...\")\n",
    "        self.model.evaluate(self.test_text, self.test_labels)\n",
    "        pred = self.model.predict(self.test_text)\n",
    "        pred[pred < 0.5] = 0\n",
    "        pred[pred >= 0.5] = 1\n",
    "        \n",
    "        print(classification_report(self.test_labels, pred)) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we create an instance of the regression model. After reading the input dataset, we convert the preprocessed text data to word vector embeddings using our Word2Vec model. We then build the LSTM model, feed the embeddings as input to train it, and evaluate it with the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "LSTM_reg_model = LSTMModel('normalised_data.json', data_type)\n",
    "LSTM_reg_model.read_data()\n",
    "LSTM_reg_model.create_input_embedding()\n",
    "LSTM_reg_model.build_regression_model()\n",
    "LSTM_reg_model.train_regression_model()\n",
    "LSTM_reg_model.test_regression_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we repeat the same steps to create a binary classification version of our LSTM model. Again, we train and evaluate it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "LSTM_class_model = LSTMModel('normalised_data.json', data_type)\n",
    "LSTM_class_model.read_data()\n",
    "LSTM_class_model.create_input_embedding()\n",
    "LSTM_class_model.build_classification_model()\n",
    "LSTM_class_model.train_classification_model()\n",
    "LSTM_class_model.test_classification_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model with Cluster Inputs:\n",
    "As seen above, the accuracy from a vanilla LSTM model is low. We will now try a different approach wherein we change the input features that we feed into the network. The thinking behind this is that a large portion of the review text contains words that are irrelevant or make no contribution towards predicting whether a review is 'useful', 'funny' or 'cool'. Therefore, instead of converting the words in the review into word vector embeddings and feeding these as input to the network, we are going to handpick a subset of words that we think are crucial to having a good review. For example, if we think that the words: 'cook', 'bake' and 'fry' would help determine whether a review is useful or not, then for each review, we would look at each word within the review and determine which of the above three words it is closest to in terms of its word vector embedding. Let us say that a review contains three words that are closest to 'cook', four closest to 'bake' and 'one closest to 'fry'. The input to the network is then the word vector embeddings for 'cook', 'bake' and 'fry' weighted by how many words in the review are closest to them.\n",
    "\n",
    "In order to determine what words we should use as crucial words, we take the word vector embeddings of the whole vocabulary that we calculated above and run K-Means clustering on these embeddings with the number of clusters set to 120. The intuition behind doing this is that we can seperate the vocabulary into 120 different groups, wherein each group represents some kind of terminology related to restaurants and cooking. This should become clear on running the following code.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.cluster import KMeans\n",
    "from sklearn.neighbors import KDTree\n",
    "\n",
    "normalised_data_path = './normalised_data.json'\n",
    "print(\"Clustering work vectors...\")\n",
    "clustering = KMeans(n_clusters = 120, init='k-means++')\n",
    "clustering_index = clustering.fit_predict(wve_model.wv.vectors)\n",
    "joblib.dump(clustering, './cluster.model')\n",
    "cluster_centers = clustering.cluster_centers_\n",
    "center_labels = [wve_model.wv.most_similar(positive=[vector], topn=1)[0][0] for vector in cluster_centers]\n",
    "centroid_map = dict(zip(wve_model.wv.index2word, clustering_index))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(center_labels)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see in the above cell, our cluster centers happen to be terms that are related to food. In order to demonstrate how each cluster center might contribute towards determining whether a review is useful or not, we plot the wordclouds for four of the centers below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Getting top words for each cluster...\")\n",
    "tree = KDTree(wve_model.wv.vectors)\n",
    "closest_points = [tree.query(np.reshape(x, (1, -1)), k=20) for x in cluster_centers[0:4]]\n",
    "closest_words_ids = [x[1] for x in closest_points]\n",
    "closest_words = {}\n",
    "closest_ids = {}\n",
    "\n",
    "for i in range(0, len(closest_words_ids)):\n",
    "    closest_words['Cluster #' + str(i)] = [wve_model.wv.index2word[j] for j in closest_words_ids[i][0]]\n",
    "    closest_ids['Cluster #' + str(i)] = [wve_model.wv[wve_model.wv.index2entity[j]] for j in closest_words_ids[i][0]]\n",
    "\n",
    "wd_df = pd.DataFrame(closest_words)\n",
    "id_df = pd.DataFrame(closest_ids)\n",
    "top_words = wd_df\n",
    "top_ids = id_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def display_cloud(cluster_index, color_map):\n",
    "    print(\"Word clouds for each cluster: \")\n",
    "    if len(top_words['Cluster #' + str(cluster_index)])>0:\n",
    "        wc = WordCloud(background_color=\"black\", max_words=2000, max_font_size=80, colormap=color_map)\n",
    "        wordcloud = wc.generate(' '.join([word for word in top_words['Cluster #' + str(cluster_index)]]))\n",
    "        plt.subplots(1,1, figsize=(15, 15))\n",
    "        plt.plot()\n",
    "        plt.imshow(wordcloud, interpolation='bilinear')\n",
    "        plt.axis(\"off\")\n",
    "\n",
    "\n",
    "cmaps = cycle([\n",
    "            'flag', 'prism', 'ocean', 'gist_earth', 'terrain', 'gist_stern',\n",
    "            'gnuplot', 'gnuplot2', 'CMRmap', 'cubehelix', 'brg', 'hsv',\n",
    "            'gist_rainbow', 'rainbow', 'jet', 'nipy_spectral', 'gist_ncar'])\n",
    "for i in range(4):\n",
    "    col = next(cmaps);\n",
    "    display_cloud(i, col)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Saving cluster centroids...\")\n",
    "output = {}\n",
    "output['embedding_size'] = 256\n",
    "output['num_clusters'] = 120\n",
    "output['cluster_centers'] = cluster_centers.tolist()\n",
    "\n",
    "with open('./cluster_data.json', 'w') as of_handle:\n",
    "    json.dump(output, of_handle)\n",
    "    of_handle.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next section, we process our review text into a form where for each review, we convert the review into its weighted centroid embedding form that was described earlier. This section of the code takes a **considerable time to run**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from keras.utils import np_utils\n",
    "\n",
    "class ClusterModelInput:\n",
    "    def __init__(self):\n",
    "        self.wve_model = gensim.models.Word2Vec.load('./wve.model')\n",
    "        self.cluster_data_file_name = './cluster_data.json'\n",
    "        self.dataset_location = './normalised_data.json'\n",
    "        self.cluster_data = None\n",
    "        self.useful_data = None\n",
    "        self.tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')\n",
    "        self._stop_words_ = nltk.corpus.stopwords.words()\n",
    "        self.clustering = joblib.load('./cluster.model')\n",
    "        self.train_test_data_file_path = './train_test_data.json'\n",
    "\n",
    "    def load_data(self):\n",
    "        with open(self.cluster_data_file_name) as fh:\n",
    "            self.cluster_data = json.load(fh)\n",
    "            fh.close()\n",
    "\n",
    "        with open(self.dataset_location, 'r') as fh:\n",
    "            full_data = json.load(fh)\n",
    "            self.useful_data = pd.DataFrame(json.loads(full_data[data_type]))\n",
    "            fh.close()\n",
    "\n",
    "    def prepare_input(self):\n",
    "        df = self.useful_data.sample(frac=1).reset_index(drop=True)\n",
    "        text_input = df['text'].tolist()\n",
    "        transformed_text = []\n",
    "        cluster_count = []\n",
    "\n",
    "        for _, review in enumerate(text_input):\n",
    "            no_tabs = str(review).replace('\\t', ' ').replace('\\n', '')\n",
    "            alphas_only = re.sub(\"[^a-zA-Z\\.]\", \" \", no_tabs)\n",
    "            multi_spaces = re.sub(\" +\", \" \", alphas_only)\n",
    "            no_spaces = multi_spaces.strip()\n",
    "            clean_text = no_spaces.lower()\n",
    "            sentences = self.tokenizer.tokenize(clean_text)\n",
    "            sentences = [re.sub(\"[\\.]\", \"\", sentence) for sentence in sentences]\n",
    "            review_weights = range(self.cluster_data['num_clusters'])\n",
    "            review_weights = dict(Counter(review_weights))\n",
    "            review_weights = dict.fromkeys(review_weights, 0)\n",
    "\n",
    "            if len(clean_text) > 0 and clean_text.count(' ') > 0:\n",
    "                for sentence in sentences:\n",
    "                    sentence = sentence.split(' ')\n",
    "                    pruned_sentence = [self.get_cluster_bucket(word) for word in sentence if word not in self._stop_words_]\n",
    "\n",
    "                    if len(pruned_sentence) > 0:\n",
    "                        pruned_sentence = dict(Counter(pruned_sentence))\n",
    "                        for key in review_weights:\n",
    "                            if key in pruned_sentence:\n",
    "                                review_weights[key] = review_weights[key] + pruned_sentence[key]\n",
    "            \n",
    "            sorted_review_weights = np.array([review_weights[key] for key in sorted(review_weights.keys())])\n",
    "            final_weights = self.cluster_data['cluster_centers'] * sorted_review_weights[:, None]\n",
    "            transformed_text.append(final_weights)\n",
    "            cluster_count.append(sorted_review_weights)\n",
    "        \n",
    "        self.input_data = np.array(transformed_text)\n",
    "        self.rf_input_data = np.array(cluster_count)\n",
    "        self.labels = np.array(df[data_type].tolist())\n",
    "        self.alternate_labels = np.array(df[data_type + '_raw'].tolist())\n",
    "\n",
    "    def get_cluster_bucket(self, word):\n",
    "        if word in self.wve_model.wv.vocab:\n",
    "            word_embedding = self.wve_model.wv[word]\n",
    "            return self.clustering.predict([word_embedding])[0]\n",
    "        else:\n",
    "            return -1\n",
    "\n",
    "    def save_training_data(self):\n",
    "        rows, _ = self.useful_data.shape\n",
    "        split_point = int(math.floor(rows * 0.7))\n",
    "        train_input = self.input_data[0:split_point, :]\n",
    "        test_input = self.input_data[split_point:rows, :]\n",
    "        rf_train_input = self.rf_input_data[0:split_point, :]\n",
    "        rf_test_input = self.rf_input_data[split_point: rows, :]\n",
    "        train_labels = self.labels[0:split_point]\n",
    "        test_labels = self.labels[split_point:rows]\n",
    "        train_alternate_labels = self.alternate_labels[0:split_point]\n",
    "        test_alternate_labels = self.alternate_labels[split_point:rows]\n",
    "        \n",
    "        class_labels_train = []\n",
    "        for l in train_labels:\n",
    "            if l < 0.33:\n",
    "                class_labels_train.append('low')\n",
    "            elif l >= 0.33 and l < 0.66:\n",
    "                class_labels_train.append('medium')\n",
    "            elif l >= 0.66 and l <= 1.0:\n",
    "                class_labels_train.append('high')\n",
    "        \n",
    "        encoder_train = LabelEncoder()\n",
    "        encoder_train.fit(class_labels_train)\n",
    "        encoded_labels_train = encoder_train.transform(class_labels_train)\n",
    "        # convert integers to dummy variables (i.e. one hot encoded)\n",
    "        one_hot_labels_train = np_utils.to_categorical(encoded_labels_train)\n",
    "        \n",
    "        class_labels_test = []\n",
    "        for l in test_labels:\n",
    "            if l < 0.33:\n",
    "                class_labels_test.append('low')\n",
    "            elif l >= 0.33 and l < 0.66:\n",
    "                class_labels_test.append('medium')\n",
    "            elif l >= 0.66 and l <= 1.0:\n",
    "                class_labels_test.append('high')\n",
    "\n",
    "        encoder_test = LabelEncoder()\n",
    "        encoder_test.fit(class_labels_test)\n",
    "        encoded_labels_test = encoder_test.transform(class_labels_test)\n",
    "        # convert integers to dummy variables (i.e. one hot encoded)\n",
    "        one_hot_labels_test = np_utils.to_categorical(encoded_labels_test)\n",
    "        \n",
    "        oh_class_labels_train = []\n",
    "        for l in train_labels:\n",
    "            if l < 0.5:\n",
    "                oh_class_labels_train.append('not ' + data_type)\n",
    "            elif l >= 0.5:\n",
    "                oh_class_labels_train.append(data_type)\n",
    "        \n",
    "        oh_encoder_train = LabelEncoder()\n",
    "        oh_encoder_train.fit(oh_class_labels_train)\n",
    "        oh_encoded_labels_train = oh_encoder_train.transform(oh_class_labels_train)\n",
    "        # convert integers to dummy variables (i.e. one hot encoded)\n",
    "        oh_one_hot_labels_train = np_utils.to_categorical(oh_encoded_labels_train)\n",
    "        \n",
    "        oh_class_labels_test = []\n",
    "        for l in test_labels:\n",
    "            if l < 0.5:\n",
    "                oh_class_labels_test.append('not ' + data_type)\n",
    "            elif l >= 0.5:\n",
    "                oh_class_labels_test.append(data_type)\n",
    "\n",
    "        oh_encoder_test = LabelEncoder()\n",
    "        oh_encoder_test.fit(oh_class_labels_test)\n",
    "        oh_encoded_labels_test = oh_encoder_test.transform(oh_class_labels_test)\n",
    "        # convert integers to dummy variables (i.e. one hot encoded)\n",
    "        oh_one_hot_labels_test = np_utils.to_categorical(oh_encoded_labels_test)\n",
    "        \n",
    "        return (train_input, test_input, train_labels, test_labels, one_hot_labels_train, one_hot_labels_test, rf_train_input, rf_test_input, oh_one_hot_labels_train, oh_one_hot_labels_test)\n",
    "\n",
    "model = ClusterModelInput()\n",
    "model.load_data()\n",
    "model.prepare_input()\n",
    "train_input, test_input, train_labels, test_labels, train_alternate_labels, test_alternate_labels, rf_train_input, rf_test_input, oh_train_labels, oh_test_labels = model.save_training_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have our input data and labels in the correct format, we will train a dense neural network model to classify the data into one of three categories, wherein the values of 0 to 0.33 for 'useful', 'funny' and 'cool' fall into a class called 'low', the values for 0.33 to 0.66 fall into another class called 'medium' and finally the values for 0.66 to 1.0 fall into a class called 'high'. We thus hope to classify a given review as being low, medium or high in terms of 'usefulness', 'funniness' or 'coolness'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import json\n",
    "import tensorflow.keras.backend as K\n",
    "from keras.activations import softmax\n",
    "import numpy as np\n",
    "\n",
    "class ClusterModel:\n",
    "    def __init__(self, train_data, test_data, train_labels, test_labels):\n",
    "        self.model = None\n",
    "        self.embedding_size = None\n",
    "        self.num_clusters = None\n",
    "        self.cluster_data_file_name = './cluster_data.json'\n",
    "        self.train_data = train_data\n",
    "        self.test_data = test_data\n",
    "        self.train_labels = train_labels\n",
    "        self.test_labels = test_labels\n",
    "        self.read_data()\n",
    "        self.create_model()\n",
    "        self.train_model()\n",
    "        self.predict_and_evaluate()\n",
    "\n",
    "    def create_model(self):\n",
    "        print('Building model...')\n",
    "        model = tf.keras.Sequential()\n",
    "        dense_layer = tf.keras.layers.Dense(500, activation='relu', kernel_initializer='glorot_uniform', input_shape=(self.num_clusters, self.embedding_size))\n",
    "        model.add(dense_layer)\n",
    "        dense_layer_2 = tf.keras.layers.Dense(300, activation='relu')\n",
    "        model.add(dense_layer_2)\n",
    "        flatten_layer = tf.keras.layers.Flatten()\n",
    "        model.add(flatten_layer)\n",
    "        dense_layer_3 = tf.keras.layers.Dense(3, activation='softmax')\n",
    "        model.add(dense_layer_3)\n",
    "\n",
    "        model.compile(loss = 'categorical_crossentropy', optimizer = tf.keras.optimizers.Adam(learning_rate = 0.01), metrics = ['mse', 'mae'])\n",
    "        self.model = model\n",
    "        print(model.summary())\n",
    "\n",
    "    def read_data(self):\n",
    "        print('Reading data...')\n",
    "        with open(self.cluster_data_file_name) as fh:\n",
    "            data = json.load(fh)\n",
    "            self.embedding_size = data['embedding_size']\n",
    "            self.num_clusters = data['num_clusters']\n",
    "            fh.close()\n",
    "    \n",
    "    def train_model(self):\n",
    "        print('Training model...')\n",
    "        self.model.fit(self.train_data, self.train_labels, batch_size = 128, epochs = 5, validation_split = 0.3)\n",
    "\n",
    "    def predict_and_evaluate(self):\n",
    "        print('Predicting...')\n",
    "        target_names = ['class 0', 'class 1', 'class 2']\n",
    "        pred = self.model.predict(self.test_data)\n",
    "        pred_labels = []\n",
    "        true_labels = []\n",
    "        \n",
    "        for value in pred:\n",
    "            pred_labels.append(np.argmax(value))\n",
    "        \n",
    "        for value in self.test_labels:\n",
    "            true_labels.append(np.argmax(value))\n",
    "        \n",
    "        print(classification_report(true_labels, pred_labels, target_names=target_names))\n",
    "        \n",
    "        \n",
    "cluster_model = ClusterModel(train_input, test_input, train_alternate_labels, test_alternate_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model with random forests:\n",
    "As we can see above, we get a marginal improvement in accuracy. Next, we will try using a random forest to classify this data. Random forest is an ensemble method that consists of multiple decision trees. For this classifier, the input data is only the weights that are assigned to each of the cluster centers. For example, if we decide that the words 'cook', 'bake' and 'fry' would help determine whether a review is useful or not, then for each review, we would look at each word within the review and determine which of the above three words it is closest to in terms of its word vector embedding. Let us say that a review contains three words that are closest to 'cook', four closest to 'bake' and one closest to 'fry'. The input to the network is then the array [3, 4, 1]. Again, we try to classify each review into one of three categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import numpy as np\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "\n",
    "class RandomForest:\n",
    "    def __init__(self):\n",
    "        self.cluster_data_file_name = './cluster_data.json'\n",
    "        self.rf_train_data = rf_train_input\n",
    "        self.train_labels = train_alternate_labels\n",
    "        self.rf_test_data = rf_test_input\n",
    "        self.test_labels = test_alternate_labels\n",
    "        \n",
    "        self.train_model()\n",
    "        self.predict_and_evaluate()\n",
    "\n",
    "    def train_model(self):\n",
    "        print('Training model...')\n",
    "        self.classifier = RandomForestClassifier(n_estimators=100, random_state=0)\n",
    "        self.classifier.fit(self.rf_train_data, self.train_labels)\n",
    "    \n",
    "    def predict_and_evaluate(self):\n",
    "        print('Predicting...')\n",
    "        target_names = ['class 0', 'class 1', 'class 2']\n",
    "        pred = self.classifier.predict(self.rf_test_data)\n",
    "        pred_labels = []\n",
    "        true_labels = []\n",
    "        \n",
    "        for value in pred:\n",
    "            pred_labels.append(np.argmax(value))\n",
    "        \n",
    "        for value in self.test_labels:\n",
    "            true_labels.append(np.argmax(value))\n",
    "        \n",
    "        print(classification_report(true_labels, pred_labels, target_names=target_names))\n",
    "        \n",
    "\n",
    "model = RandomForest()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, this model does not perform much better than the previous model. Therefore, we will rephrase our classification task as a binary classification task wherein we determine if a review is 'useful', 'funny' and 'cool' or it is 'not useful', 'not funny' and 'not cool'. Therefore, any values for the 'useful', 'funny' and 'cool' columns that fall below 0.5 are treated as not useful and any values in the range of 0.5 to 1 are treated as useful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import numpy as np\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "\n",
    "class RandomForest:\n",
    "    def __init__(self):\n",
    "        self.cluster_data_file_name = './cluster_data.json'\n",
    "        self.rf_train_data = rf_train_input\n",
    "        self.train_labels = oh_train_labels\n",
    "        self.rf_test_data = rf_test_input\n",
    "        self.test_labels = oh_test_labels\n",
    "        \n",
    "        self.train_model()\n",
    "        self.predict_and_evaluate()\n",
    "\n",
    "    def train_model(self):\n",
    "        print('Training model...')\n",
    "        self.classifier = RandomForestClassifier(n_estimators=100, random_state=0)\n",
    "        self.classifier.fit(self.rf_train_data, self.train_labels)\n",
    "    \n",
    "    def predict_and_evaluate(self):\n",
    "        print('Predicting...')\n",
    "        target_names = ['class 0', 'class 1']\n",
    "        pred = self.classifier.predict(self.rf_test_data)\n",
    "        pred_labels = []\n",
    "        true_labels = []\n",
    "        \n",
    "        for value in pred:\n",
    "            pred_labels.append(np.argmax(value))\n",
    "        \n",
    "        for value in self.test_labels:\n",
    "            true_labels.append(np.argmax(value))\n",
    "        \n",
    "        print(classification_report(true_labels, pred_labels, target_names=target_names))\n",
    "\n",
    "model = RandomForest()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that the accuracy increases significantly which is expected because we reduced the dimensionality of the output classifications. We can try the same classification task with the dense neural network model to see whether it improves the accuracy of the binary classification task. However, we do not expect to see a significant improvement using this network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import json\n",
    "import tensorflow.keras.backend as K\n",
    "from keras.activations import softmax\n",
    "import numpy as np\n",
    "\n",
    "class ClusterModel:\n",
    "    def __init__(self, train_data, test_data, train_labels, test_labels):\n",
    "        self.model = None\n",
    "        self.embedding_size = None\n",
    "        self.num_clusters = None\n",
    "        self.cluster_data_file_name = './cluster_data.json'\n",
    "        self.train_data = train_data\n",
    "        self.test_data = test_data\n",
    "        self.train_labels = train_labels\n",
    "        self.test_labels = test_labels\n",
    "        self.read_data()\n",
    "        self.create_model()\n",
    "        self.train_model()\n",
    "        self.predict_and_evaluate()\n",
    "\n",
    "    def create_model(self):\n",
    "        print('Building Model...')\n",
    "        model = tf.keras.Sequential()\n",
    "\n",
    "        dense_layer = tf.keras.layers.Dense(500, activation='relu', kernel_initializer='glorot_uniform', input_shape=(self.num_clusters, self.embedding_size))\n",
    "        model.add(dense_layer)\n",
    "        dense_layer_2 = tf.keras.layers.Dense(300, activation='relu')\n",
    "        model.add(dense_layer_2)\n",
    "        flatten_layer = tf.keras.layers.Flatten()\n",
    "        model.add(flatten_layer)\n",
    "        dense_layer_3 = tf.keras.layers.Dense(2, activation='softmax')\n",
    "        model.add(dense_layer_3)\n",
    "\n",
    "        model.compile(loss = 'categorical_crossentropy', optimizer = tf.keras.optimizers.Adam(learning_rate = 0.01), metrics = ['mse', 'mae'])\n",
    "        self.model = model\n",
    "        print(model.summary())\n",
    "\n",
    "    def read_data(self):\n",
    "        print('Reading data...')\n",
    "        with open(self.cluster_data_file_name) as fh:\n",
    "            data = json.load(fh)\n",
    "            self.embedding_size = data['embedding_size']\n",
    "            self.num_clusters = data['num_clusters']\n",
    "            fh.close()\n",
    "    \n",
    "    def train_model(self):\n",
    "        print('Training model...')\n",
    "        self.model.fit(self.train_data, self.train_labels, batch_size = 128, epochs = 5, validation_split = 0.3)\n",
    "\n",
    "    def predict_and_evaluate(self):\n",
    "        print('Predicting...')\n",
    "        target_names = ['class 0', 'class 1']\n",
    "        pred = self.model.predict(self.test_data)\n",
    "        pred_labels = []\n",
    "        true_labels = []\n",
    "        \n",
    "        for value in pred:\n",
    "            pred_labels.append(np.argmax(value))\n",
    "        \n",
    "        for value in self.test_labels:\n",
    "            true_labels.append(np.argmax(value))\n",
    "        \n",
    "        print(classification_report(true_labels, pred_labels, target_names=target_names))\n",
    "cluster_model = ClusterModel(train_input, test_input, oh_train_labels, oh_test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, this model provides a similar accuracy to the previous one.\n",
    "\n",
    "## Conclusion:\n",
    "\n",
    "In this project, we set out to create a model which can help Yelp in determining how useful, funny or cool a new review is. To start off, we tried to frame this as a regression task wherein we determine the usefulness, funniness or coolness of a review on a scale of 0 to 1, 0 being not useful and 1 being very useful. However, we soon learned that this was not feasible and we had to reframe our problem as a classification task. On running various models to classify each review, we have come to the conclusion that due to some inherent bias's present in the review data, it is difficult to model a good estimator for how useful, funny or cool a given review is. This happens due to various factors, some of which are explained below:\n",
    "\n",
    "* Two reviews that are equalliy similar in terms of content and length have vastly different counts for 'useful' and 'funny' within the training data. This might be due to the fact that some restaurants are located in more populated areas and as a result have a larger number of upvotes on their reviews.\n",
    "* Some reviews which are quite useful, funny, or cool simply do not have many upvotes. This could be due to the fact that they were not online for a large amount of time before the data was harvested or simply because the restaurants did not get many visitors and thus do not get many upvotes on Yelp.\n",
    "* Usefulness, funniness and coolness are subjective criteria by which one measures the quality of a review. What one person or demographic finds as useful might not be useful at all in another demographic. From personal experiences of using Yelp in London, it was quite important to see the proximity of a certain restaurant to a local train station or underground station in London. We would consider a review that mentions such details as more useful than one that does not simply because public transport is the most widely used form of transport in London. However, when reading through reviews in LA for example, the proximity of the restaurant to public transport is not important at all. Due to the precense of such bias's in the data, we surmise that there is no clear underlying pattern in the data that can help us in modeling a good estimator of 'usefulness', 'funniness' or 'coolness' of a review."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}