HW5_Kushal.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Yelp Review Analysis based on Useful, Funny and Cool Counts\n",
    "\n",
    "The first step in our process is to import the data in order to analyse the dataset. We believe that analysing the data before preparing the model is an important aspect of any Machine Learning Task. Please provide the location of the Yelp DataSet in the input box below after which, we will extract the reviews that are stricly related to restaurants.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install gensim\n",
    "!pip install keras\n",
    "!pip install nltk\n",
    "!pip install numpy\n",
    "!pip install pandas\n",
    "!pip install sklearn\n",
    "!pip install tensorflow==1.15\n",
    "!pip install wordcloud\n",
    "\n",
    "import gensim\n",
    "from gensim.models import word2vec\n",
    "from itertools import cycle\n",
    "import json\n",
    "from keras.utils import np_utils\n",
    "import logging\n",
    "import matplotlib.pyplot as plt\n",
    "import nltk\n",
    "import numpy as np\n",
    "import os\n",
    "import pandas as pd\n",
    "import re\n",
    "from sklearn import preprocessing\n",
    "from sklearn.cluster import KMeans\n",
    "from sklearn.manifold import TSNE\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.neighbors import KDTree\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "import sys\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras import Sequential\n",
    "from tensorflow.keras.layers import Bidirectional, Dense, Embedding, SimpleRNN, LSTM\n",
    "from wordcloud import WordCloud, ImageColorGenerator\n",
    "from sklearn.externals import joblib\n",
    "import math\n",
    "\n",
    "nltk.download('all')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Please run then cell below and then follow the instructions provided hereafter. On running the cell a text input box will appear where you SHOULD enter the path to your dataset folder. A dropdown box to select one of the options from useful, funny or cool that you want to predict will also appear. Please select one of the options from 'useful', 'funny' or 'cool' in the dropdown box**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import display\n",
    "from ipywidgets import widgets\n",
    "text = widgets.Text()\n",
    "display(text)\n",
    "\n",
    "dataset_folder = '/Users/kldsouza/Documents/CS 544/code/project'\n",
    "\n",
    "def handle_change(change):\n",
    "    if change['type'] == 'change' and change['name'] == 'value':\n",
    "        global dataset_folder\n",
    "        dataset_folder = change['new']\n",
    "\n",
    "text.observe(handle_change)\n",
    "\n",
    "\n",
    "dd_widget = widgets.Dropdown(\n",
    "    options=['useful', 'funny', 'cool'],\n",
    "    value='useful',\n",
    "    description='Data Type:',\n",
    "    disabled=False,\n",
    ")\n",
    "\n",
    "display(dd_widget)\n",
    "\n",
    "data_type = 'useful'\n",
    "def on_change(change):\n",
    "    if change['type'] == 'change' and change['name'] == 'value':\n",
    "        global data_type\n",
    "        data_type = change['new']\n",
    "        \n",
    "\n",
    "dd_widget.observe(on_change)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please verify that your folder_location and data type are as intended after running the cell below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(data_type)\n",
    "print(dataset_folder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Extraction\n",
    "In the following cells, we extract data from the review.json and business.json files. The main data that we will work with is the review text for each review along with its count for 'useful', 'funny' and 'cool'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# functions for data extraction\n",
    "def extract_data(business_data_folder, review_data_folder):\n",
    "    output_data = {}\n",
    "    with open(business_data_folder) as bd_handle:\n",
    "        business = bd_handle.readline()\n",
    "\n",
    "        while business:\n",
    "            bd_data = json.loads(business)\n",
    "\n",
    "            if bd_data['categories']:\n",
    "                output_data[bd_data['business_id']] = {\n",
    "                    'categories': [cat.strip() for cat in bd_data['categories'].split(',')],\n",
    "                    'reviews': []\n",
    "                }\n",
    "\n",
    "            business = bd_handle.readline()\n",
    "\n",
    "        bd_handle.close()\n",
    "\n",
    "    with open(review_data_folder) as rd_handle:\n",
    "        review = rd_handle.readline()\n",
    "\n",
    "        while review:\n",
    "            rv_data = json.loads(review)\n",
    "\n",
    "            if rv_data['business_id'] in output_data:\n",
    "                review = {}\n",
    "                review['review_id'] = rv_data['review_id']\n",
    "                review['text'] = rv_data['text']\n",
    "                review['useful'] = rv_data['useful']\n",
    "                review['funny'] = rv_data['funny']\n",
    "                review['cool'] = rv_data['cool']\n",
    "                output_data[rv_data['business_id']]['reviews'].append(review)\n",
    "\n",
    "            review = rd_handle.readline()\n",
    "\n",
    "        bd_handle.close()\n",
    "        \n",
    "    return output_data\n",
    "\n",
    "def write_extracted_data(output_data, output_file):\n",
    "    data_to_write = []\n",
    "    for business_id in output_data:\n",
    "        categories = output_data[business_id]['categories']\n",
    "\n",
    "        if 'Restaurants' in categories and len(output_data[business_id]['reviews']) > 0:\n",
    "            cat_to_write = ':'.join(categories)\n",
    "            for review in output_data[business_id]['reviews']:\n",
    "                review_id = review['review_id']\n",
    "                text = review['text']\n",
    "                useful = str(review['useful'])\n",
    "                funny = str(review['funny'])\n",
    "                cool = str(review['cool'])\n",
    "                data_to_write.append({\n",
    "                    'business_id': business_id,\n",
    "                    'categories': cat_to_write,\n",
    "                    'review_id': review_id,\n",
    "                    'text': text,\n",
    "                    'useful': useful,\n",
    "                    'funny': funny,\n",
    "                    'cool': cool\n",
    "                })           \n",
    "\n",
    "    with open(output_file, 'w') as of_handle:\n",
    "        json.dump(data_to_write, of_handle)\n",
    "        of_handle.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "business_data_folder = os.path.join(dataset_folder, 'business.json')\n",
    "review_data_folder = os.path.join(dataset_folder, 'review.json')\n",
    "output_data = extract_data(business_data_folder, review_data_folder)\n",
    "dataset_file = './dataset.json'\n",
    "write_extracted_data(output_data, dataset_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, the data we need from review.json and business.json are stored in the file \"dateset.json\". Next, we will clean up the data and write the preprocessed datasets to \"normalised_data.json\".\n",
    "\n",
    "## Data Analysis and Cleansing:\n",
    "\n",
    "In the next cell, we analyse the data, specifically the distribution of values within the 'useful', 'funny' and 'cool' columns. We notice that the data is heavily skewed towards values between 0 to 5. Most of the values for 'useful', 'funny' and 'cool' lie between 0 and 20. There are outliers that occur in the data with some values going up to more than 500. \n",
    "\n",
    "In order to enable our model to accurately predict how 'useful', 'funny' or 'cool' a certain review is, we apply feature clipping on each of these columns. We also normalised these values so that they lie in the range of 0 to 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# functions for data preprocessing\n",
    "def preprocess_data(data_file):\n",
    "\n",
    "    df = pd.read_json(data_file)\n",
    "    df['useful'] = df['useful'].astype(float)\n",
    "    df['funny'] = df['funny'].astype(float)\n",
    "    df['cool'] = df['cool'].astype(float)\n",
    "    print(\"Dataset size: {0:d} rows x {1:d} columns\".format(df.shape[0], df.shape[1]))\n",
    "\n",
    "    vote_labels = ['useful', 'funny', 'cool']\n",
    "    useful_min, useful_max = df['useful'].min(), df['useful'].max()\n",
    "    funny_min, funny_max = df['funny'].min(), df['funny'].max()\n",
    "    cool_min, cool_max = df['cool'].min(), df['cool'].max()\n",
    "\n",
    "    # print out distribution of reviews by their vote counts\n",
    "    for i in range(len(vote_labels)):\n",
    "        ranges = np.arange(-1, 100, 10)\n",
    "        counts = df.groupby(pd.cut(df[vote_labels[i]], ranges)).count()[vote_labels[i]]\n",
    "        print(counts)\n",
    "        print(\"Out of range counts: \", df.shape[0]-counts.sum())\n",
    "\n",
    "    # save raw vote counts\n",
    "    df['useful_raw'] = df['useful']\n",
    "    df['funny_raw'] = df['funny']\n",
    "    df['cool_raw'] = df['cool']\n",
    "    # clip vote counts to specific ranges\n",
    "    df['useful'] = df['useful'].clip(3.0, 27.0)\n",
    "    df['funny'] = df['funny'].clip(3.0, 20.0)\n",
    "    df['cool'] = df['cool'].clip(3.0, 24.0)\n",
    "\n",
    "    # normalize data with min-max scaling\n",
    "    min_max_scaler = preprocessing.MinMaxScaler()\n",
    "    useful_scaled = min_max_scaler.fit_transform(np.array(df['useful']).reshape(-1,1))\n",
    "    df['useful'] = pd.DataFrame(useful_scaled)\n",
    "    funny_scaled = min_max_scaler.fit_transform(np.array(df['funny']).reshape(-1,1))\n",
    "    df['funny'] = pd.DataFrame(funny_scaled)\n",
    "    cool_scaled = min_max_scaler.fit_transform(np.array(df['cool']).reshape(-1,1))\n",
    "    df['cool'] = pd.DataFrame(cool_scaled)\n",
    "\n",
    "    ranges = [-0.1, 0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]\n",
    "    processed_data = []\n",
    "\n",
    "    # balance dataset for each of vote category\n",
    "    # make sure there is an even number of reviews in each increment of 0.1\n",
    "    # from range 0.0 to 1.0\n",
    "    for i in range(len(vote_labels)):\n",
    "        counts = df.groupby(pd.cut(df[vote_labels[i]], ranges)).count()[vote_labels[i]]\n",
    "        min_count = counts.min()\n",
    "        new_df = pd.DataFrame()\n",
    "        for j in range(10):\n",
    "            lower = j/10\n",
    "            upper = (j+1)/10 if j < 9 else 1.1\n",
    "            new_data = df[(df[vote_labels[i]] >= lower) & (df[vote_labels[i]] < upper)]\n",
    "            new_data = new_data.sample(frac=1)\n",
    "            new_data = new_data.head(min_count)\n",
    "            new_df = new_df.append(new_data)\n",
    "\n",
    "        processed_data.append(new_df)\n",
    "    \n",
    "    return processed_data\n",
    "\n",
    "\n",
    "def write_extracted_data(useful, funny, cool):\n",
    "    output_dict = {\n",
    "        'useful': useful.to_json(orient = 'records'),\n",
    "        'funny': funny.to_json(orient = 'records'),\n",
    "        'cool': cool.to_json(orient = 'records')\n",
    "    }\n",
    "\n",
    "    with open('normalised_data.json', 'w') as op_handle:\n",
    "        json.dump(output_dict, op_handle)\n",
    "        op_handle.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "processed_data = preprocess_data(dataset_file)\n",
    "write_extracted_data(processed_data[0], processed_data[1], processed_data[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Word2Vec Model:\n",
    "\n",
    "In the next step, we cleanse each review by removing any punctuation, full stops and extra spaces. We also lowercase all the words in the review and remove any stopwords from the data. We then use this cleansed dataset to train a Word2Vec model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# functions for creating the word2vec model\n",
    "def read_data(dataset_file):\n",
    "    logging.info(\"Reading data...\")\n",
    "    with open(dataset_file, 'r') as fh:\n",
    "        full_data = json.load(fh)\n",
    "        useful_data = pd.DataFrame(json.loads(full_data['useful']))\n",
    "        funny_data = pd.DataFrame(json.loads(full_data['funny']))\n",
    "        cool_data = pd.DataFrame(json.loads(full_data['cool']))\n",
    "        all_data = useful_data.append([funny_data, cool_data])\n",
    "    \n",
    "    return useful_data, funny_data, cool_data, all_data\n",
    "\n",
    "\n",
    "def clean_data(tokenized_data_file, all_data, tokenizer, stop_words):\n",
    "    logging.info(\"Cleaning data...\")\n",
    "    with open(tokenized_data_file, 'w') as fh:\n",
    "        reviews = all_data['text'].tolist()\n",
    "\n",
    "        for index in range(len(reviews)):\n",
    "            review = reviews[index]\n",
    "            no_tabs = str(review).replace('\\t', ' ').replace('\\n', '')\n",
    "            alphas_only = re.sub(\"[^a-zA-Z\\.]\", \" \", no_tabs)\n",
    "            multi_spaces = re.sub(\" +\", \" \", alphas_only)\n",
    "            no_spaces = multi_spaces.strip()\n",
    "            clean_text = no_spaces.lower()\n",
    "            sentences = tokenizer.tokenize(clean_text)\n",
    "            sentences = [re.sub(\"[\\.]\", \"\", sentence) for sentence in sentences]\n",
    "\n",
    "            if len(clean_text) > 0 and clean_text.count(' ') > 0:\n",
    "                for sentence in sentences:\n",
    "                    sentence = sentence.split(' ')\n",
    "                    pruned_sentence = [word for word in sentence if word not in stop_words]\n",
    "                    sentence = ' '.join(pruned_sentence)\n",
    "                    if sentence: \n",
    "                        fh.write(\"%s\\n\" % sentence)\n",
    "\n",
    "            if (index % 5000) == 0:\n",
    "                fh.flush()\n",
    "        fh.close()\n",
    "\n",
    "\n",
    "def create_wve(tokenized_data_file):\n",
    "    embedding_size = 256\n",
    "    min_word_count = 50\n",
    "    context = 30\n",
    "    downsampling = 1e-2\n",
    "    logging.info(\"Creating word vectors...\")\n",
    "    model = word2vec.Word2Vec(\n",
    "        word2vec.LineSentence(tokenized_data_file),\n",
    "        size=embedding_size,\n",
    "        min_count=min_word_count,\n",
    "        window=context,\n",
    "        sample=downsampling)\n",
    "    model.init_sims(replace=True)\n",
    "    model.save('./wve.model')\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "useful_data, funny_data, cool_data, all_data = read_data('normalised_data.json')\n",
    "tokenized_data_file = './tokennized.txt'\n",
    "tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')\n",
    "stop_words = nltk.corpus.stopwords.words()\n",
    "clean_data(tokenized_data_file, all_data, tokenizer, stop_words)\n",
    "wve_model = create_wve(tokenized_data_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LSTM Model with Cluster Inputs:\n",
    "As seen above, the accuracy on using a vanilla LSTM model is low. We will now try a different approach wherein we change the input features that we feed into the network. The thinking behind this is that a large portion of the review text contains words that are irrelevant or make no contribution towards predicting whether a review is 'useful', 'funny' or 'cool'. Therefore, instead of converting the words in the review into word vector embeddings and feeding these as input to the network, we are going to handpick a subset of words that we think are crucial to having a good review. For example, if I decide that the words: 'cook', 'bake' and 'fry' are going to be words that determine whether a review is useful or not, then for each review, I will look at each word within the review and determine which of the above three words it is closest to in terms of its word vector embedding. Let us say that my review contained three words that are closest to 'cook', four closest to 'bake' and 'one closest to 'fry'. My input to the network is then the word vector embeddings for 'cook', 'bake' and 'fry weighted by how many words in the review were closest to them.\n",
    "\n",
    "In order to determine what words I should use as crucial words, I take the word vector embeddings of the whole vocabulary that we calculated above and run K-Means clustering on these embeddings with the number of clusters set to 120. The intuition behind doing this is that, we can seperate the vocabulary into 120 different groups, wherein each group represents some kind of terminology related to restaurants and cooking. This should become clear on running the following code.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "from sklearn.cluster import KMeans\n",
    "from sklearn.neighbors import KDTree\n",
    "logging.basicConfig(level=logging.INFO)\n",
    "\n",
    "normalised_data_path = './normalised_data.json'\n",
    "logging.info(\"Clustering work vectors\")\n",
    "clustering = KMeans(n_clusters = 120, init='k-means++')\n",
    "clustering_index = clustering.fit_predict(wve_model.wv.vectors)\n",
    "joblib.dump(clustering, './cluster.model')\n",
    "cluster_centers = clustering.cluster_centers_\n",
    "center_labels = [wve_model.wv.most_similar(positive=[vector], topn=1)[0][0] for vector in cluster_centers]\n",
    "centroid_map = dict(zip(wve_model.wv.index2word, clustering_index))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(center_labels)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see in the above cell, our cluster centers happen to be terms that are related to food. In order to demonstrate how each cluster center might contribute towards determining whether a review is useful or not, we have plotted the wordclouds for four of the centers below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logging.info(\"Getting top words for each cluster\")\n",
    "tree = KDTree(wve_model.wv.vectors)\n",
    "closest_points = [tree.query(np.reshape(x, (1, -1)), k=20) for x in cluster_centers[0:4]]\n",
    "closest_words_ids = [x[1] for x in closest_points]\n",
    "closest_words = {}\n",
    "closest_ids = {}\n",
    "\n",
    "for i in range(0, len(closest_words_ids)):\n",
    "    closest_words['Cluster #' + str(i)] = [wve_model.wv.index2word[j] for j in closest_words_ids[i][0]]\n",
    "    closest_ids['Cluster #' + str(i)] = [wve_model.wv[wve_model.wv.index2entity[j]] for j in closest_words_ids[i][0]]\n",
    "\n",
    "wd_df = pd.DataFrame(closest_words)\n",
    "id_df = pd.DataFrame(closest_ids)\n",
    "top_words = wd_df\n",
    "top_ids = id_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def display_cloud(cluster_index, color_map):\n",
    "    logging.info(\"Printing word clouds for each cluster\")\n",
    "    if len(top_words['Cluster #' + str(cluster_index)])>0:\n",
    "        wc = WordCloud(background_color=\"black\", max_words=2000, max_font_size=80, colormap=color_map)\n",
    "        wordcloud = wc.generate(' '.join([word for word in top_words['Cluster #' + str(cluster_index)]]))\n",
    "        plt.subplots(1,1, figsize=(15, 15))\n",
    "        plt.plot()\n",
    "        plt.imshow(wordcloud, interpolation='bilinear')\n",
    "        plt.axis(\"off\")\n",
    "#         plt.savefig('clusters/cluster_' + str(self.wve_model.most_similar(positive=[self.cluster_centers[cluster_index]], topn=1)[0][0]), bbox_inches='tight')\n",
    "\n",
    "\n",
    "\n",
    "cmaps = cycle([\n",
    "            'flag', 'prism', 'ocean', 'gist_earth', 'terrain', 'gist_stern',\n",
    "            'gnuplot', 'gnuplot2', 'CMRmap', 'cubehelix', 'brg', 'hsv',\n",
    "            'gist_rainbow', 'rainbow', 'jet', 'nipy_spectral', 'gist_ncar'])\n",
    "for i in range(4):\n",
    "    col = next(cmaps);\n",
    "    display_cloud(i, col)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logging.info(\"Saving cluster centroids\")\n",
    "output = {}\n",
    "output['embedding_size'] = 256\n",
    "output['num_clusters'] = 120\n",
    "output['cluster_centers'] = cluster_centers.tolist()\n",
    "\n",
    "with open('./cluster_data.json', 'w') as of_handle:\n",
    "    json.dump(output, of_handle)\n",
    "    of_handle.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next section, we process our review text into a form where for each review, we convert the review into its weighted centroid embedding form that was described earlier. This section of the code takes a considerable time to run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from keras.utils import np_utils\n",
    "\n",
    "class LSTMModelInput:\n",
    "    def __init__(self):\n",
    "        self.wve_model = gensim.models.Word2Vec.load('./wve.model')\n",
    "        self.cluster_data_file_name = './cluster_data.json'\n",
    "        self.dataset_location = './normalised_data.json'\n",
    "        self.cluster_data = None\n",
    "        self.useful_data = None\n",
    "        self.tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')\n",
    "        self._stop_words_ = nltk.corpus.stopwords.words()\n",
    "        self.clustering = joblib.load('./cluster.model')\n",
    "        self.train_test_data_file_path = './train_test_data.json'\n",
    "        logging.basicConfig(level=logging.INFO)\n",
    "\n",
    "    def load_data(self):\n",
    "        with open(self.cluster_data_file_name) as fh:\n",
    "            self.cluster_data = json.load(fh)\n",
    "            fh.close()\n",
    "\n",
    "        with open(self.dataset_location, 'r') as fh:\n",
    "            full_data = json.load(fh)\n",
    "            self.useful_data = pd.DataFrame(json.loads(full_data[data_type]))\n",
    "            fh.close()\n",
    "\n",
    "    def prepare_input(self):\n",
    "        df = self.useful_data.sample(frac=1).reset_index(drop=True)\n",
    "        text_input = df['text'].tolist()\n",
    "        transformed_text = []\n",
    "        cluster_count = []\n",
    "\n",
    "        for _, review in enumerate(text_input):\n",
    "            logging.info('Currently processing review %s:', review)\n",
    "            no_tabs = str(review).replace('\\t', ' ').replace('\\n', '')\n",
    "            alphas_only = re.sub(\"[^a-zA-Z\\.]\", \" \", no_tabs)\n",
    "            multi_spaces = re.sub(\" +\", \" \", alphas_only)\n",
    "            no_spaces = multi_spaces.strip()\n",
    "            clean_text = no_spaces.lower()\n",
    "            sentences = self.tokenizer.tokenize(clean_text)\n",
    "            sentences = [re.sub(\"[\\.]\", \"\", sentence) for sentence in sentences]\n",
    "            review_weights = range(self.cluster_data['num_clusters'])\n",
    "            review_weights = dict(Counter(review_weights))\n",
    "            review_weights = dict.fromkeys(review_weights, 0)\n",
    "\n",
    "            if len(clean_text) > 0 and clean_text.count(' ') > 0:\n",
    "                for sentence in sentences:\n",
    "                    sentence = sentence.split(' ')\n",
    "                    pruned_sentence = [self.get_cluster_bucket(word) for word in sentence if word not in self._stop_words_]\n",
    "\n",
    "                    if len(pruned_sentence) > 0:\n",
    "                        pruned_sentence = dict(Counter(pruned_sentence))\n",
    "                        for key in review_weights:\n",
    "                            if key in pruned_sentence:\n",
    "                                review_weights[key] = review_weights[key] + pruned_sentence[key]\n",
    "            \n",
    "            sorted_review_weights = np.array([review_weights[key] for key in sorted(review_weights.keys())])\n",
    "            final_weights = self.cluster_data['cluster_centers'] * sorted_review_weights[:, None]\n",
    "            transformed_text.append(final_weights)\n",
    "            cluster_count.append(sorted_review_weights)\n",
    "        \n",
    "        self.input_data = np.array(transformed_text)\n",
    "        self.rf_input_data = np.array(cluster_count)\n",
    "        self.labels = np.array(df[data_type].tolist())\n",
    "        self.alternate_labels = np.array(df[data_type + '_raw'].tolist())\n",
    "\n",
    "    def get_cluster_bucket(self, word):\n",
    "        if word in self.wve_model.wv.vocab:\n",
    "            word_embedding = self.wve_model.wv[word]\n",
    "            return self.clustering.predict([word_embedding])[0]\n",
    "        else:\n",
    "            return -1\n",
    "\n",
    "    def save_training_data(self):\n",
    "        rows, _ = self.useful_data.shape\n",
    "        split_point = int(math.floor(rows * 0.7))\n",
    "        train_input = self.input_data[0:split_point, :]\n",
    "        test_input = self.input_data[split_point:rows, :]\n",
    "        rf_train_input = self.rf_input_data[0:split_point, :]\n",
    "        rf_test_input = self.rf_input_data[split_point: rows, :]\n",
    "        train_labels = self.labels[0:split_point]\n",
    "        test_labels = self.labels[split_point:rows]\n",
    "        train_alternate_labels = self.alternate_labels[0:split_point]\n",
    "        test_alternate_labels = self.alternate_labels[split_point:rows]\n",
    "        \n",
    "        class_labels_train = []\n",
    "        for l in train_labels:\n",
    "            if l < 0.33:\n",
    "                class_labels_train.append('low')\n",
    "            elif l >= 0.33 and l < 0.66:\n",
    "                class_labels_train.append('medium')\n",
    "            elif l >= 0.66 and l <= 1.0:\n",
    "                class_labels_train.append('high')\n",
    "        \n",
    "        encoder_train = LabelEncoder()\n",
    "        encoder_train.fit(class_labels_train)\n",
    "        encoded_labels_train = encoder_train.transform(class_labels_train)\n",
    "        # convert integers to dummy variables (i.e. one hot encoded)\n",
    "        one_hot_labels_train = np_utils.to_categorical(encoded_labels_train)\n",
    "        \n",
    "        class_labels_test = []\n",
    "        for l in test_labels:\n",
    "            if l < 0.33:\n",
    "                class_labels_test.append('low')\n",
    "            elif l >= 0.33 and l < 0.66:\n",
    "                class_labels_test.append('medium')\n",
    "            elif l >= 0.66 and l <= 1.0:\n",
    "                class_labels_test.append('high')\n",
    "\n",
    "        encoder_test = LabelEncoder()\n",
    "        encoder_test.fit(class_labels_test)\n",
    "        encoded_labels_test = encoder_test.transform(class_labels_test)\n",
    "        # convert integers to dummy variables (i.e. one hot encoded)\n",
    "        one_hot_labels_test = np_utils.to_categorical(encoded_labels_test)\n",
    "        \n",
    "        oh_class_labels_train = []\n",
    "        for l in train_labels:\n",
    "            if l < 0.5:\n",
    "                oh_class_labels_train.append('not ' + data_type)\n",
    "            elif l >= 0.5:\n",
    "                oh_class_labels_train.append(data_type)\n",
    "        \n",
    "        oh_encoder_train = LabelEncoder()\n",
    "        oh_encoder_train.fit(oh_class_labels_train)\n",
    "        oh_encoded_labels_train = oh_encoder_train.transform(oh_class_labels_train)\n",
    "        # convert integers to dummy variables (i.e. one hot encoded)\n",
    "        oh_one_hot_labels_train = np_utils.to_categorical(oh_encoded_labels_train)\n",
    "        \n",
    "        oh_class_labels_test = []\n",
    "        for l in test_labels:\n",
    "            if l < 0.5:\n",
    "                oh_class_labels_test.append('not ' + data_type)\n",
    "            elif l >= 0.5:\n",
    "                oh_class_labels_test.append(data_type)\n",
    "\n",
    "        oh_encoder_test = LabelEncoder()\n",
    "        oh_encoder_test.fit(oh_class_labels_test)\n",
    "        oh_encoded_labels_test = oh_encoder_test.transform(oh_class_labels_test)\n",
    "        # convert integers to dummy variables (i.e. one hot encoded)\n",
    "        oh_one_hot_labels_test = np_utils.to_categorical(oh_encoded_labels_test)\n",
    "        \n",
    "        return (train_input, test_input, train_labels, test_labels, one_hot_labels_train, one_hot_labels_test, rf_train_input, rf_test_input, oh_one_hot_labels_train, oh_one_hot_labels_test)\n",
    "\n",
    "model = LSTMModelInput()\n",
    "model.load_data()\n",
    "model.prepare_input()\n",
    "train_input, test_input, train_labels, test_labels, train_alternate_labels, test_alternate_labels, rf_train_input, rf_test_input, oh_train_labels, oh_test_labels = model.save_training_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have our input data and labels in the correct format, we will train a deep LSTM model to classify the data into one of three categories, wherein the values of 0 to 0.3 for 'useful', 'funny' and 'cool' fall into a class called 'low', the values for 0.3 to 0.6 fall into another class called 'medium' and finally the values for 0.6 to 1.0 fall into a class called 'high'. We thus hope to classify a given review as being low, medium or high in terms of 'usefulness', 'funnyness' or 'coolness'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import json\n",
    "import tensorflow.keras.backend as K\n",
    "from keras.activations import softmax\n",
    "import numpy as np\n",
    "import logging\n",
    "\n",
    "class LSTMModel:\n",
    "    def __init__(self, train_data, test_data, train_labels, test_labels):\n",
    "        logging.basicConfig(level=logging.INFO)\n",
    "        self.model = None\n",
    "        self.embedding_size = None\n",
    "        self.num_clusters = None\n",
    "        self.cluster_data_file_name = './cluster_data.json'\n",
    "        self.train_data = train_data\n",
    "        self.test_data = test_data\n",
    "        self.train_labels = train_labels\n",
    "        self.test_labels = test_labels\n",
    "        self.read_data()\n",
    "        self.create_model()\n",
    "        self.train_model()\n",
    "        self.calculate_accuracy()\n",
    "\n",
    "    def create_model(self):\n",
    "        logging.info('Creating Model')\n",
    "        model = tf.keras.Sequential()\n",
    "#         input_layer = tf.keras.layers.InputLayer(batch_size=128, input_shape=(self.num_clusters, self.embedding_size))\n",
    "#         model.add(input_layer)\n",
    "\n",
    "        dense_layer = tf.keras.layers.Dense(500, activation='relu', kernel_initializer='glorot_uniform', input_shape=(self.num_clusters, self.embedding_size))\n",
    "        model.add(dense_layer)\n",
    "        # dense_middle = tf.keras.layers.Dense(500, activation='tanh', use_bias=True, kernel_initializer='glorot_uniform', bias_initializer='zeros')\n",
    "        # model.add(dense_middle)\n",
    "        dense_layer_2 = tf.keras.layers.Dense(300, activation='relu')\n",
    "        model.add(dense_layer_2)\n",
    "        flatten_layer = tf.keras.layers.Flatten()\n",
    "        model.add(flatten_layer)\n",
    "        dense_layer_3 = tf.keras.layers.Dense(3, activation='softmax')\n",
    "        model.add(dense_layer_3)\n",
    "\n",
    "        model.compile(loss = 'categorical_crossentropy', optimizer = tf.keras.optimizers.Adam(learning_rate = 0.01), metrics = ['mse', 'mae'])\n",
    "        self.model = model\n",
    "        print(model.summary())\n",
    "\n",
    "    def read_data(self):\n",
    "        logging.info('Reading data')\n",
    "        with open(self.cluster_data_file_name) as fh:\n",
    "            data = json.load(fh)\n",
    "            self.embedding_size = data['embedding_size']\n",
    "            self.num_clusters = data['num_clusters']\n",
    "            fh.close()\n",
    "    \n",
    "    def train_model(self):\n",
    "        logging.info('Training Model')\n",
    "        self.model.fit(self.train_data, self.train_labels, batch_size = 128, epochs = 5, validation_split = 0.3)\n",
    "\n",
    "    def calculate_accuracy(self):\n",
    "        predicted = self.model.predict(self.test_data)\n",
    "        total = len(predicted)\n",
    "        correct = 0\n",
    "    \n",
    "        for index, value in enumerate(predicted):\n",
    "            if self.test_labels[index][np.argmax(value)] == 1:\n",
    "                correct = correct + 1\n",
    "        \n",
    "        print(\"Final accuracy is %s\", (correct/total))\n",
    "\n",
    "lstm_model = LSTMModel(train_input, test_input, train_alternate_labels, test_alternate_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see above, we get a marginal improvement in accuracy. Next, we tried using a random forest to classify this data. Random forest is an ensemble method that consists of multiple decision trees. For this classifier, the input data is only the weights that are assigned to each of the cluster centers. For example, if I decide that the words: 'cook', 'bake' and 'fry' are going to be words that determine whether a review is useful or not, then for each review, I will look at each word within the review and determine which of the above three words it is closest to in terms of its word vector embedding. Let us say that my review contained three words that are closest to 'cook', four closest to 'bake' and 'one closest to 'fry'. My input to the network is then the array [3, 4, 1]. Again, we try to classify each review into one of three categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import json\n",
    "import numpy as np\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "\n",
    "class RandomForest:\n",
    "    def __init__(self):\n",
    "        logging.basicConfig(level=logging.INFO)\n",
    "        self.cluster_data_file_name = './cluster_data.json'\n",
    "        self.rf_train_data = rf_train_input\n",
    "        self.train_labels = train_alternate_labels\n",
    "        self.rf_test_data = rf_test_input\n",
    "        self.test_labels = test_alternate_labels\n",
    "        \n",
    "        self.train_model()\n",
    "        self.calculate_accuracy()\n",
    "\n",
    "    def train_model(self):\n",
    "        logging.info('Training Model')\n",
    "        self.regressor = RandomForestClassifier(n_estimators=100, random_state=0)\n",
    "        self.regressor.fit(self.rf_train_data, self.train_labels)\n",
    "    \n",
    "    def calculate_accuracy(self):\n",
    "        logging.info('Predicting')\n",
    "        predictions = self.regressor.predict(self.rf_test_data)\n",
    "        correct = 0\n",
    "        total = len(self.test_labels)\n",
    "\n",
    "        for index, prediction in enumerate(predictions):\n",
    "            if self.test_labels[index][np.argmax(prediction)] == 1:\n",
    "                correct = correct + 1\n",
    "        \n",
    "        accuracy = correct/total\n",
    "        print('Final accuracy is ' + str(accuracy))\n",
    "\n",
    "model = RandomForest()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, this model does not perform much better than the previous model. Therefore, we will rephrase our classification task as a binary classification task wherein we either determine if a review is 'useful', 'funny' and 'cool' or it is 'not useful', 'not funny' and 'not cool'. Therefore, any values for the 'useful', 'funny' and 'cool' columns that fall into the range of 0 to 0.5 are treated as not useful and any values in the range of 0.5 to 1 are treated as useful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import json\n",
    "import numpy as np\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "\n",
    "class RandomForest:\n",
    "    def __init__(self):\n",
    "        logging.basicConfig(level=logging.INFO)\n",
    "        self.cluster_data_file_name = './cluster_data.json'\n",
    "        self.rf_train_data = rf_train_input\n",
    "        self.train_labels = oh_train_labels\n",
    "        self.rf_test_data = rf_test_input\n",
    "        self.test_labels = oh_test_labels\n",
    "        \n",
    "        self.train_model()\n",
    "        self.calculate_accuracy()\n",
    "\n",
    "    def train_model(self):\n",
    "        logging.info('Training Model')\n",
    "        self.regressor = RandomForestClassifier(n_estimators=100, random_state=0)\n",
    "        self.regressor.fit(self.rf_train_data, self.train_labels)\n",
    "    \n",
    "    def calculate_accuracy(self):\n",
    "        logging.info('Predicting')\n",
    "        predictions = self.regressor.predict(self.rf_test_data)\n",
    "        correct = 0\n",
    "        total = len(self.test_labels)\n",
    "\n",
    "        for index, prediction in enumerate(predictions):\n",
    "            if self.test_labels[index][np.argmax(prediction)] == 1:\n",
    "                correct = correct + 1\n",
    "        \n",
    "        accuracy = correct/total\n",
    "        print('Final accuracy is ' + str(accuracy))\n",
    "\n",
    "model = RandomForest()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that the accuracy has increase significantly which is expected because we reduced the dimensionality of the output classifications. We can try the same classification task with the deep learning network to see whether the deep learning network improves the accuracy of the binary classification task. However, we do not expect to see a significant improvement using this network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import json\n",
    "import tensorflow.keras.backend as K\n",
    "from keras.activations import softmax\n",
    "import numpy as np\n",
    "import logging\n",
    "\n",
    "class LSTMModel:\n",
    "    def __init__(self, train_data, test_data, train_labels, test_labels):\n",
    "        logging.basicConfig(level=logging.INFO)\n",
    "        self.model = None\n",
    "        self.embedding_size = None\n",
    "        self.num_clusters = None\n",
    "        self.cluster_data_file_name = './cluster_data.json'\n",
    "        self.train_data = train_data\n",
    "        self.test_data = test_data\n",
    "        self.train_labels = train_labels\n",
    "        self.test_labels = test_labels\n",
    "        self.read_data()\n",
    "        self.create_model()\n",
    "        self.train_model()\n",
    "        self.calculate_accuracy()\n",
    "\n",
    "    def create_model(self):\n",
    "        logging.info('Creating Model')\n",
    "        model = tf.keras.Sequential()\n",
    "#         input_layer = tf.keras.layers.InputLayer(batch_size=128, input_shape=(self.num_clusters, self.embedding_size))\n",
    "#         model.add(input_layer)\n",
    "\n",
    "        dense_layer = tf.keras.layers.Dense(500, activation='relu', kernel_initializer='glorot_uniform', input_shape=(self.num_clusters, self.embedding_size))\n",
    "        model.add(dense_layer)\n",
    "        # dense_middle = tf.keras.layers.Dense(500, activation='tanh', use_bias=True, kernel_initializer='glorot_uniform', bias_initializer='zeros')\n",
    "        # model.add(dense_middle)\n",
    "        dense_layer_2 = tf.keras.layers.Dense(300, activation='relu')\n",
    "        model.add(dense_layer_2)\n",
    "        flatten_layer = tf.keras.layers.Flatten()\n",
    "        model.add(flatten_layer)\n",
    "        dense_layer_3 = tf.keras.layers.Dense(2, activation='softmax')\n",
    "        model.add(dense_layer_3)\n",
    "\n",
    "        model.compile(loss = 'categorical_crossentropy', optimizer = tf.keras.optimizers.Adam(learning_rate = 0.01), metrics = ['mse', 'mae'])\n",
    "        self.model = model\n",
    "        print(model.summary())\n",
    "\n",
    "    def read_data(self):\n",
    "        logging.info('Reading data')\n",
    "        with open(self.cluster_data_file_name) as fh:\n",
    "            data = json.load(fh)\n",
    "            self.embedding_size = data['embedding_size']\n",
    "            self.num_clusters = data['num_clusters']\n",
    "            fh.close()\n",
    "    \n",
    "    def train_model(self):\n",
    "        logging.info('Training Model')\n",
    "        self.model.fit(self.train_data, self.train_labels, batch_size = 128, epochs = 5, validation_split = 0.3)\n",
    "\n",
    "    def calculate_accuracy(self):\n",
    "        predicted = self.model.predict(self.test_data)\n",
    "        total = len(predicted)\n",
    "        correct = 0\n",
    "    \n",
    "        for index, value in enumerate(predicted):\n",
    "            if self.test_labels[index][np.argmax(value)] == 1:\n",
    "                correct = correct + 1\n",
    "        \n",
    "        print(\"Final accuracy is %s\", (correct/total))\n",
    "\n",
    "lstm_model = LSTMModel(train_input, test_input, oh_train_labels, oh_test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, this model provides a similar accuracy to the previous one.\n",
    "\n",
    "## Conclusion:\n",
    "\n",
    "In this project, we set out to create a model which can help Yelp in determining how useful, funny or cool a new review is. To start off, we tried to frame this as a regression task wherein we determine the usefulness, funnyness or coolness of a review on a scale of 0 to 1, 0 being not useful and 1 being very useful. However, we soon learnt that this was not feasible and we had to reframe our problem as a classification task. On running various models to classify each review, we have come to the conclusion that due to some inherent bias's present in the review data, it is difficult to model a good estimator for how useful, funny or cool a given review is. This happens due to various factors, some of which are explained below:\n",
    "\n",
    "* Two reviews that are equalliy similar in terms of content and length have vastly different counts for 'useful' and 'funny' within the training data. This might be due to the fact that some restaurants are located in more populated areas and as a result have a larger number of upvotes on their reviews.\n",
    "* Some reviews which are quite useful, funny, or cool simply do not have many upvotes. This could be due to the fact that they were not online for a large amount of time before the data was harvested or simply because the restaurant does not get many visitors and thus does not get many upvotes on Yelp.\n",
    "* Usefulness, funnyness and coolness are subjective criteria by which one measures the quality of a review. What one person or demographic finds as useful might not be useful at all in another demographic. From my personal experience of using Yelp back home in London, it was quite important for me to see the proximity of a certain restaurant to a local train station or underground station in London. I would consider a review that mentions such details as more useful than one that does not simply because public transport is the most widely used form of transport in London. However, when reading through reviews in LA for example, the proximity of the restaurant to public transport is not important to me at all. Due to the precense of such bias's in the data, we surmise that there is no clear underlying pattern in the data that can help us in modelling a good estimator of 'usefulness', 'funnyness' or 'coolness' of a review."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}