Twitter US Airline Sentiment - 2. Classifier with ULMFiT.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Challenge Part 2: Sentiment Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai.text import *\n",
    "import my_utils"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Having already trained the language model, it is time to use it for the actual task!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "VERSION = 'v1'\n",
    "\n",
    "DATA_PATH = 'data/%s/' % VERSION\n",
    "\n",
    "import os\n",
    "os.makedirs(DATA_PATH, exist_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the dataset\n",
    "\n",
    "We will load the already pre-processed text data. However for the classification task, we only care about the strongly positive and negative examples, so we will filter out the neutral ones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sentiment</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>negative</th>\n",
       "      <td>9178</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>neutral</th>\n",
       "      <td>3099</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>positive</th>\n",
       "      <td>2363</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           text\n",
       "sentiment      \n",
       "negative   9178\n",
       "neutral    3099\n",
       "positive   2363"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with open('data/lm/tweet_data.pkl', 'rb') as f:\n",
    "    df = pickle.load(f)\n",
    "\n",
    "df.groupby('sentiment').count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sentiment</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>negative</th>\n",
       "      <td>9178</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>positive</th>\n",
       "      <td>2363</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           text\n",
       "sentiment      \n",
       "negative   9178\n",
       "positive   2363"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_labeled = df[df['sentiment'] != 'neutral']\n",
    "\n",
    "df_labeled.groupby('sentiment').count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>plus you've added commercials to the experienc...</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>it's really aggressive to blast obnoxious \" en...</td>\n",
       "      <td>negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>and it's a really big bad thing about it</td>\n",
       "      <td>negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>seriously would pay $ 30 a flight for seats th...</td>\n",
       "      <td>negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>yes , nearly every time I fly VX this “ ear wo...</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Well , I didn't … but NOW I DO ! :-D</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>it was amazing , and arrived an hour early . Y...</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>I &lt;3 pretty graphics . so much better than min...</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>This is such a great deal ! Already thinking a...</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>xxhan I'm flying your fabulous Seductive skies...</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 text sentiment\n",
       "1   plus you've added commercials to the experienc...  positive\n",
       "3   it's really aggressive to blast obnoxious \" en...  negative\n",
       "4            and it's a really big bad thing about it  negative\n",
       "5   seriously would pay $ 30 a flight for seats th...  negative\n",
       "6   yes , nearly every time I fly VX this “ ear wo...  positive\n",
       "8                Well , I didn't … but NOW I DO ! :-D  positive\n",
       "9   it was amazing , and arrived an hour early . Y...  positive\n",
       "11  I <3 pretty graphics . so much better than min...  positive\n",
       "12  This is such a great deal ! Already thinking a...  positive\n",
       "13  xxhan I'm flying your fabulous Seductive skies...  positive"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_labeled.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to have reproducible results, I will set a fixed random seed, and set the number of workers to 1, as suggested in the [documentation](https://docs.fast.ai/dev/test.html#getting-reproducible-results)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_utils.set_all_seeds(123)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create the DataBunch for the classification task. We will use 10% of the data for the validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs = 48\n",
    "\n",
    "data_lm = load_data(path='data/lm', file='data_lm.pkl')\n",
    "\n",
    "data_clas = (TextList.from_df(df_labeled, cols=['text'], vocab=data_lm.vocab, path=DATA_PATH)\n",
    "           .split_by_rand_pct(valid_pct=0.1, seed=0)\n",
    "           .label_from_df(cols='sentiment')\n",
    "           .databunch(bs=bs, num_workers=1))\n",
    "\n",
    "data_clas.save('data_clas.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>text</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj the xxmaj fact xxmaj that u xxmaj see xxmaj black xxmaj history xxmaj month 12 xxmaj months a xxmaj year xxmaj is xxmaj xxunk ! xxmaj we xxup will xxup be xxmaj an xxmaj xxunk xxmaj base xxmaj for xxmaj corp . xxmaj like u xxmaj in xxmaj future !</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxup xxunk u xxup us xxup airways xxup with xxup yo xxup shitty xxup chicken xxup xxunk xxup xxunk xxup that xxup so xxup xxunk xxup and u xxup xxunk xxup make xxup me xxup wait xxup in a 6 xxup hr xxup layover xxup xxunk u xxup and</td>\n",
       "      <td>negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos our ft xxunk xxup xxunk / xxup dfw ( w / cx to xxup lga ) tomorrow xxmaj cancelled xxmaj flighted , 4 hrs so far waiting for call back . xxmaj what should we do ? xxmaj need to get to nyc</td>\n",
       "      <td>negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxup mia - xxup ewr # xxunk 😄 😄 😄 excellent crew . xxup ewr - xxup iad # xxunk 😡 😡 😡 xxmaj no crew to load bags - waiting w / door open freezing . 20 mins past departure .</td>\n",
       "      <td>negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj cancelled xxmaj flights flt from xxup ewr . \" xxmaj no crew \" . xxmaj tells wife &amp; 4 yr old to \" get to xxup ny to catch xxhan \" to xxup yyz ! xxmaj good customerservice is dead .</td>\n",
       "      <td>negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj thanks . xxmaj still booked our trip 3 / xxunk 17 xxup lb to xxup slc to see grand kids . xxmaj just very frustrating . xxmaj tried app , web , etc . xxmaj still love u guys !</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos i know . i ’ m sure you hear it all the time — don ’ t like the new seats . i ’ m a small guy , 5 ’ 6 ” and i feel the new planes are tight</td>\n",
       "      <td>negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos file loc : xxup xxunk . xxmaj bag at airport since last xxunk . xxmaj not \" scheduled \" to get xxup xxunk xxunk xxunk 8 ? xxup why ? u xxunk b ashamed . i 'm disgusted w / u</td>\n",
       "      <td>negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj hi ! xxmaj just got my xxup tsa pre - check # &amp; put it in my acct . xxmaj but it 's not on my boarding pass for a flight this xxunk . xxmaj can it be updated ?</td>\n",
       "      <td>negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj step 1 : xxmaj cancelled xxmaj flight flight . xxmaj step 2 : xxmaj do n't notify customer . xxmaj step 3 : xxmaj charge them for food while they try to xxunk their wait . xxmaj brilliant .</td>\n",
       "      <td>negative</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# show a few examples\n",
    "data_clas.show_batch(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's finally create the classifier, that will take the already fine-tuned encoder from the previous part."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "def initialize_text_classifier():\n",
    "    learn = text_classifier_learner(data_clas, AWD_LSTM, drop_mult=0.5, path=DATA_PATH)\n",
    "    # a trick to be able to load from another folder...\n",
    "    learn.path = Path('data/lm')\n",
    "    learn.load_encoder('fine_tuned_enc')\n",
    "    learn.path = Path(DATA_PATH)\n",
    "    return learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = initialize_text_classifier()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Min numerical gradient: 2.75E-02\n",
      "Min loss divided by 10: 3.02E-02\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl4XVXZ///3nXlOkybpkLRNOtLSkYaW0jIpQ+FRZhRwYJBBEVQU/IkD+oAog8qDCPxEBBEQZFILAqUCVSjUTjSdaOncpmOaNvOcrO8f56SchqQnbc6YfF7Xda6cvfbae987ycmdtdfea5lzDhERkcOJCXcAIiIS+ZQsRETELyULERHxS8lCRET8UrIQERG/lCxERMQvJQsREfFLyUJERPxSshAREb/iwh1AoOTk5LjCwsJwhyEiElWWLl26zzmX669er0kWhYWFLFmyJNxhiIhEFTPb2p16ugwlIiJ+KVmIiIhfShYiIuKXkoWIiPilZCEiIn4pWYiIiF9KFiIi4peShYhIFHtpaSl/+e+2oB9HyUJEJIq9tKyUl5aVBv04QU0WZjbbzNaZ2QYz+0En6+83s+Xe18dmVuGzrtVn3ZxgxikiEq1qGltITwr+YBxBO4KZxQIPAWcApcBiM5vjnFvTXsc5d7NP/ZuAKT67qHfOTQ5WfCIivUF1QwtDs1OCfpxgtiymARucc5ucc03Ac8B5h6l/GfBsEOMREel1qhtaSE+KD/pxgpks8oHtPsul3rJPMbNhQBHwtk9xkpktMbOFZnZ+8MIUEYle1Q3N0X0ZCrBOylwXdS8FXnTOtfqUDXXO7TSz4cDbZrbSObfxkAOYXQdcBzB06NBAxCwiEjWaWtpobGkjLTH4ySKYLYtSYIjPcgGws4u6l9LhEpRzbqf36yZgPof2Z7TXedQ5V+ycK87N9Tscu4hIr1Lb2AIQkpZFMJPFYmCUmRWZWQKehPCpu5rMbAyQBXzgU5ZlZone9znATGBNx21FRPqy6gZPsghFyyJoR3DOtZjZjcBcIBZ43Dm32szuAJY459oTx2XAc84530tUY4Hfm1kbnoR2t+9dVCIiAtWNzQAh6eAOajpyzr0GvNah7PYOyz/rZLv3gQnBjE1EJNq1tyyi/TKUiIgEUY2ShYiI+NN+GSra74YSEZEgam9ZpKllISIiXanyJouMKH+CW0REgqimsYW4GCMxLvh/ypUsRESiVE2DZ8RZs84GzAgsJQsRkShV3dAckv4KULIQEYlaNY0tpCcGv78ClCxERKJWVUOLWhYiInJ4NQ0tZChZiIjI4VQ3NofkgTxQshARiVo1ugwlIiKH45zzdHCH4IE8ULIQEYlKjS1tNLc6XYYSEZGuVR8c6kPJQkREulDd4B1xVslCRES6UtM+/7YeyhMRka5Uh3B4clCyEBGJSqGcUhWULEREolJ7n4UuQ4mISJfa+yx0GUpERLp0cEpVPWchIiJdqW5sITEuhoQQzJIHShYiIlGpuiF0Q32AkoWISFSqbmgO2Z1QoGQhIhKVPIMIKlmIiMhhVDe0hKxzG5QsRESiUk2DWhYiIuJHTWMLaSF6IA8gdGmpl9m8r5Z/LN/BnJKdJMfHct/Fkxg3OCPcYYlIH1EV4g7uPp8sGppbOfd371HYP5WinFQKc1IZlZfG1GFZmNmn6r+7voxfzV1HSWklZjC9KJtNZbWc//ACfnTOWL46Y1in2wHsqWpg9c5KJhX0o39aYrBPTUR6qU9myVOyCJmaxhaG9U9l075a5q8ro6m1DYAzxg3gvosn0i8l4WDdpz7Ywk/nrKawfyo/Omcsn580mIGZSZTXNHLLCyX8dM5q3tuwj2tPGk5tYwtVDc1U1jezsrSSRVv2s7W8DoCEuBgumJzPVbMKOWagpzXS2ubYcaCeyvpmxgxMD9mDNiISfWqbWnEudE9vA5hzLmQHC6bi4mK3ZMmSHu2jtc2xs6Ke11ft4r6568hLT+LBy6cwqaAfd/3zIx5fsJnTx+bxwKVTSO3wQ2prczy+YDP3vLGW5tZDv6f9UuI5vjCb6UXZjBmYzhurdvPSslIamtuYVJBJXVMrW8vrDiaq5PhYji/KZsbw/uRnJbN2VxWrd1axZlcVGUlxXHvScC44Lp/EuNguz6WppY0PNpXT2NxKYnwsCbExpCXGMXpg2mG3E5HIt7uygRN++Ra/uGACl08f2qN9mdlS51yx33pKFp0r2V7Bjc8uY1dFA8cOzqCktJKrZxbxo/8ZS2xM55eZwNOXsW1/HelJcWQkxZGeFE9uWiIxHbapqGvi2UXbmbt6N3npiRTlpjI8J5XUxDiWbDnA+xv38fGeGgDiYoyReWkcOziTj/dUs3JHJXnpiVxzUhGfnzSYAelJB/d/oLaJvyzaxp8/2MKeqsZPxZcYF8PkIf2YVpTN9KL+HF+UpeQhEmXW76nmjPv/w28vm8K5kwb3aF9KFgFQWd/MD15awZtr9vCzz4/jKzMKA7p/f/ZWN1BW3ciI3DSS4j1/0J1zLNhQzsPzN/D+xnIAkuJjGJadysDMJP67uZyG5jZOGpXDFTMKGZiZRGNLK40tbVTUNbN06wEWb9nPqh2VtHmbsSeNyuGzYwdw6phccrroS9ld2UC/lPiDcYhI+CzbdoALH36fJ646ntPG5PVoX91NFn2+z+JwMpPjefhLx3k7kkJ3i1q7vPQk8tKTDikzM2aNymHWqBxW7ajkw+0VbN1Xy5byWkoP1HPepHyunlXEmIHpne7znAmDAE9fzcKN5by1di9vr93D66t2AzA8J5XiwiyKh2XjcPx3037+u3k/OyrqGZyZxB3njef0cQMOG3dlXTPvrNvL1vI6th+oY/v+OirqmslMiScrJZ6slISDraminDSKclLJTA7991ckWh2c+CiEfRZKFn6YWVgSRXeMz89kfH7mUW2blhjH6eMGcPq4ATg3nlU7qliwcR9Ltuxn3po9PL+kFID+qQlMK8rmqzOG8dKyUq758xLOHj+Qn517LAMyPklkzjlKSit5euFWXinZSWOLp/9lQEYiQ7NTGNY/hcr6Zjbvq2VZXQXlNY20+TRqB2QkMn5wJsfmZzJ+cAaThvQ7ZP8i8omaEE+pCkoWgichTijIZEJBJpwyAuccG8tqAceI3LSDtwJfNbOIP7y7iQfeWs976/cxZmA67X/vK+qa2FhWS0pCLBdNLeCLxUMYMzC9y8tWjS2tbN9fx6ayWjbtq2Xd7mpW76zknXV7DyaRwZlJTBmaxYSCTKobmtm4t5aNZTXsrKhnWP9UJnpjHpadyrb9dXy8p5oNe2soq24kPSmOzOR4MpLjGZGbykVTCxiUmRz8b6ZICNQ0emfJC+E/skHtszCz2cADQCzwmHPu7g7r7wdO8y6mAHnOuX7edVcAP/au+7lz7snDHSsYfRbSuS37avnNvI/ZV+PpQDeDuJgYTh+bx/lT8nv0C1zf1MqaXVWUbK9g2bYDfLitgh0V9cTGGMP6pzAiN43BmUls2lfLyh2VVNQ1H9w2JSGWkXlpDMhIoqbhk1uXSw/UE2PwmWPyuGzaUE4ZnUtcrG5Nluj12Lub+Pk/P6Lkp2f2+BJu2PsszCwWeAg4AygFFpvZHOfcmvY6zrmbferfBEzxvs8GfgoUAw5Y6t32QLDile4rzEnlt5dNCcq+kxNimTosi6nDsriaIsBzh1dqYtynnj1xzlF6oJ7t++sY2j+FwZnJn7rrDGBbeR3PLd7G80tK+ddHS0hPjKO4MItpRf2ZVpTNxIJM4pU8JIpUh3iWPAjuZahpwAbn3CYAM3sOOA9Y00X9y/AkCICzgHnOuf3ebecBs4FngxivRKis1IROy82MIdkpDMlOOez2Q/un8P3Zx3DzGaN566O9/PvjMhZtLueddWUApCbEMn14f2aOzOGE4dkYRllNI2XVjVQ3NHPs4EwmDcnULcYSMaobWkhNiD3sbfyBFsxkkQ9s91kuBaZ3VtHMhgFFwNuH2TY/CDFKHxIfG8Ps8QOZPX4gAPtqGlm0eT/vb9zHgg3lvL12b5fbJsXHMHVYFscXZlOUk+rttE8lKyW+y+FdRIKlprE55DfeBDNZdPYJ6qqD5FLgRedc65Fsa2bXAdcBDB3as6cYpe/JSUvknAmDDt5OXHqgjqVbD5AQG0NueiK56Ykkx8fy4fYKFm4q54ON5fzfv9Yfso8BGYl8dUYhl08b2mULSCTQqhtaQnonFAQ3WZQCQ3yWC4CdXdS9FPhmh21P7bDt/I4bOeceBR4FTwf30YcqAgVZKRRkffqS1lnHDuSsYz2tkfqmVrYfqGNbeR3b9tfxzrq93Dd3HQ++vZ6LpxZw/ckj/F4WE+mpUA8iCMFNFouBUWZWBOzAkxAu71jJzMYAWcAHPsVzgV+YWZZ3+UzgtiDGKtItyQmxjB6QzugBnocer55VxLrd1fzxvU08v7iU11bu5tWbZjG4n27TleCpDvHERxDEyY+ccy3AjXj+8H8EPO+cW21md5jZuT5VLwOecz738Ho7tu/Ek3AWA3e0d3aLRJoxA9O59+JJvPbtWTQ2t3LDM8tobGn1v6HIUaoO8VwWEOSH8pxzrwGvdSi7vcPyz7rY9nHg8aAFJxJgI/PS+dUlk/jGM8v4+asfcef548MdkvRSNY0tpIdwljzQtKoiAXX2hEFcd/Jwnlq4lb99WBrucKSXCkcHt5KFSIB9/6wxTC/K5raXV7JmZ1W4w5FeprXNUdfU2nv6LET6qrjYGB68fAqZyfFc/thC3lu/L9whSS9SE4ant0HJQiQo8tKT+Ot1M8hLT+Srj/+XR/+zkd4yd4yEV7V3EMGMED+Up2QhEiSFOan87YaZnHXsQH7x2lq+/dxy6pt0l5T0THUYhicHJQuRoEpNjOPhLx3HrWeN4ZUVO7nlhRK1MKRHahq9Ex8pWYj0LmbGN08byS1njuGfK3cxp6SrgQxE/FOfhUgvd/3Jw5kytB+3/2M1e6oawh2ORKmqhvaJj5QsRHqluNgYfvOFyTS2tPL9F1focpQclU8uQ6mDW6TXKspJ5YfnjOXfH5fxl0Xbwh2ORKFwTHwEShYiIffl6cOYNTKHu/75EZvKasIdjkSZmoYWYswzjXAoKVmIhFhMjHHvxRNJjIvhyicWU1bdGO6QJIqU1zaRkRz6SbeULETCYHC/ZB6/8njKqhu56k+LDl6HFvFnzc5KjhmYHvLjKlmIhMmUoVk8/OXj+GhXNV9/ailNLW3hDkkiXFNLGx/tqmZSQb+QH1vJQiSMThuTx70XTeS9Dfv43gsltLbpDinp2trdVTS1tjExDMkitN3pIvIpF00toKymkbtfX0t1QzMPfHEKmSmhvS1SokNJaSUAEwsyQ35stSxEIsDXTxnBXReMZ8GGfZz70Hus210d7pAkApVsr6B/agIFWaGftlfJQiRCfGn6MJ699gTqmlq54OEFvL5yV7hDkgizorSCiQWZIb8TCpQsRCJKcWE2r940i9ED0vnGM8u47eWV1OpOKQFqG1vYsLcmLP0VoGQhEnEGZCTx1+tP4PpThvPc4m2c/cC7LN6yP9xhSZit2lFJm4NJQ0LfXwFKFiIRKTEultvOHsvz18/A4fjC7z/gzlfXUF6jB/j6qhUHO7fVshCRDo4vzOb1b5/MZdOG8viCzcy65x3u+uca9lZr1Nq+pqS0gvx+yeSkJYbl+EoWIhEuLTGOX1wwgXk3n8LZ4wfy+IItnHTPO/z2rfUaubYPWVFaGZZbZtspWYhEiZF5afzmi5N5+3uncPq4Afxm3sc89u7mcIclIXCgtolt++vCdgkK9FCeSNQZ1j+VBy+dAsBdr31EXkYi503OD3NUEkwrdnj6KyaFsWWhZCEShWJijF9fMomy6kZueaGE3LREThyZE+6wJEhWbK8AYLwuQ4nIkUqKj+UPXymmKCeV659ayuqdleEOSYKkpLSS4bmpZIR4djxfShYiUSwzJZ4/XTWN1MQ4LnrkfZ5euFWd3r3QitIKJoexvwKULESi3uB+ycy5cSbHF2bz47+v4to/L9HzGL3I7soG9lY3hvVOKFCyEOkV8jKSePKqadz+uXH85+N9zH7gXZ5dtI26Jg0VEu1KSj39FROHqGUhIgEQE2NcPauIf9w4kwEZidz28kqm3/UWt/9jlUaxjWJby2sBz63T4aS7oUR6mbGDMnjlxlks3XqAZ/67jecWbefPH2zlwin5/ORz48hKTQh3iHIEqupbiDFITwzvn2u1LER6ITOjuDCb+784mYU//Cw3nDqCOSU7Of03/2ZOyU51gkeRqoZmMpLjwzIsua9uJQszG2Fmid73p5rZt8wsvBfQRKRbslMT+P7sY3jlplkUZCXzrWc/5GtPLmG59959iWxV9c1kJod/5sTutixeAlrNbCTwR6AI+EvQohKRgBs7KIOXb5jJj/9nLIs27+f8hxZw3u/e46WlpTQ0t4Y7POlCVUNLWJ+vaNfdZNHmnGsBLgD+zzl3MzAoeGGJSDDExhjXnDSchT/8LHeedyy1Ta1874USTvvVfHZV1oc7POlEZX0zGcnh717ubrJoNrPLgCuAV71l4U91InJU0hLj+MqMQubdfDJPXj2Nyvpmvv3sclpa28IdmnRQVd8cVS2Lq4AZwF3Ouc1mVgQ8HbywRCQUzIxTRudy1wXjWbRlPw+8tT7cIUkHVQ1RlCycc2ucc99yzj1rZllAunPubn/bmdlsM1tnZhvM7Add1PmCma0xs9Vm9hef8lYzW+59zen2GYnIEbtgSgGXTC3gd+9s4L31+8Idjvioqm8hMyVKkoWZzTezDDPLBkqAJ8zsN362iQUeAs4GxgGXmdm4DnVGAbcBM51zxwLf8Vld75yb7H2d2/1TEpGj8b/nHcvI3DS+89flmokvQjS1tFHf3EpGUvT0WWQ656qAC4EnnHNTgdP9bDMN2OCc2+ScawKeA87rUOda4CHn3AEA59ze7ocuIoGUkhDHQ186jppGT/9Fs/ovwq6qoRmAjCi6dTbOzAYBX+CTDm5/8oHtPsul3jJfo4HRZrbAzBaa2WyfdUlmtsRbfn5nBzCz67x1lpSVlXUzLBHpyugB6dx1/gQ+2FTOT/6+Sg/vhVlVvTdZRECfRXfbNncAc4EFzrnFZjYc8NcT1tnjhh1/8+KAUcCpQAHwrpmNd85VAEOdczu9x3rbzFY65zYesjPnHgUeBSguLtZvtUgAXDS1gC3ltTz49gaGZKfwzdNGhjukPquqwTMQZCQ8lNetZOGcewF4wWd5E3CRn81KgSE+ywXAzk7qLHTONQObzWwdnuSx2Dm3s/1YZjYfmAJsRESC7rtnjGbb/jrum7uOgqxkTdsaJpXtLYtoec7CzArM7G9mttfM9pjZS2ZW4GezxcAoMysyswTgUqDjXU1/B07zHiMHz2WpTWaW5TO8SA4wE1jT/dMSkZ4wM+69eCLTCrO59YUVLNq8P9wh9UmRdBmqu30WT+D5Qz8YT7/DK96yLnmf+L4Rz+Wrj4DnnXOrzewOM2u/u2kuUG5ma4B3gFudc+XAWGCJmZV4y+92zilZiIRQYlwsj351KgXZyVz75yVs2FsT7pD6nEjq4LbudGCZ2XLn3GR/ZeFUXFzslixZEu4wRHqdbeV1XPjIAhLjYvnbDSeSl5EU7pD6jEfmb+SeN9ay9s7ZJMXHBuUYZrbUOVfsr153Wxb7zOzLZhbrfX0ZKO9ZiCISDYb2T+GJK6dxoK6JK59YTLX3v10JvqqGZhJiY0iMC/9sEt2N4Go8t83uBnYBF+MZAkRE+oAJBZk8/KXjWLenmhueWUZTi57BCIX2QQTDPZcFdH+4j23OuXOdc7nOuTzn3Pl4HtATkT7i1DF53H3hBN5dv4/PP/gev//3Ro1UG2SRMogg9GymvO8GLAoRiQqXFA/hgUsnk5wQyy9fX8uJd7/NF3//Aev3aI7vYKhqaCE9Ajq3oWfJIvztIhEJufMm5/P3b85k/i2ncvPpo/l4TzW3vLiCtjY9FxtokTJLHvQsWeg3Q6QPK8xJ5VufHcWP/mccJdsrmFPS8Zlb6SnPZajwP5AHfpKFmVWbWVUnr2o8z1yISB934ZR8JuRncs8ba6lv0vSsgVTV0BwRz1iAn2ThnEt3zmV08kp3zkVGuhORsIqJMX7yuXHsqmzgD+9uCnc4vYZzjqr6yJh/G3p2GUpEBIBpRdmcPX4gj8zfyJ4qzYURCI0tbTS1tvWKPgsRkYNuO3ssrW2O++auC3covUIkDSIIShYiEiBD+6dw1cxCXlpWyuItGniwpyJpEEFQshCRAPrmZ0YyLDuF659aytby2nCHE9UiaRBBULIQkQDKSIrniaum0eYcV/1pMRV1TeEOKWpV1XsmPoqKW2dFRI5UUU4qj36lmNL99Vz/1FIaW3Q77dFob1mog1tEeq1pRdncd8lE/rt5P7e9tFJzeR+FTzq4IyNZREb7RkR6nfMm57O1vI7fzPuYY/Mz+dqsonCHFFXaO7jTdRlKRHq7mz4zkjPGDeCXr33Esm0Hwh1OVKlqaCEpPobEuOBMenSklCxEJGjMjF9dMolB/ZK48ZllHKhVh3d3RdIggqBkISJBlpkcz8OXT2VfTRM3P79co9N2U2UEzWUBShYiEgITCjL5yefHMX9dGY/8e2O4w4kKkTSIIChZiEiIfHn6UM6dNJjfzPuYDXs1WZI/nkEEI6NzG5QsRCREzIyffn4cyfGxGj+qG6oa1GchIn1U/7RErj95OHNX72HpVo0fdTiV9boMJSJ92NdOKmJSUzn7vnoNLiMDYmIgIwNuuAE2qj8D2ueyUAe3iPRhKW/N46VHvs5p783BqqvBOaiuhsceg4kT4fXXwx1i2NU2tdLmImd4clCyEJFQ2rgRLr6YuIZ6Eto6jBnV3Ax1dXDxxX2+hRFpw5ODkoWIhNKvf+1JCofT3Az33x+aeCJUpA0iCEoWIhJKTz/dvWTx1FOhiSdCVdZF1iCCoGQhIqFUUxPYer1UVUP7XBZKFiLSF6WlBbZeL1UVYfNvg5KFiITSl78M8X7+W46Ph698JTTxRCj1WYhI3/a973UvWdx8c2jiiVDtEx+lJaplISJ90YgR8OKLkJLyqaTRFBNLc1KyZ/2IEWEKMDJU1beQlhhHXGzk/ImOnEhEpG84+2xYsQKuu87z5HZMDC4jg7dPOo/Lvvl7Gs84M9wRhl1VQ3NEDSIIShYiEg4jRsDvfgeVldDailVWkv7Y71kSl829b6xj+fYKahpbwh1l2FRF2LhQoDm4RSRCnDiiP6eNyeWP723mj+9tBiC/XzLHF2Yxe/wgThmdS3JCZEwxGmyRNoggKFmISIQwMx674ni2lteyfm8NG/bWsHZ3NfM/LuPvy3eSHB/LacfkcvrYAZw6Jo/s1IRwhxw0VQ0t5PdLDncYhwhqsjCz2cADQCzwmHPu7k7qfAH4GeCAEufc5d7yK4Afe6v93Dn3ZDBjFZHwi40xhuemMTw3jbOO9ZQ1t7axaPN+Xl+1izdX7+G1lbsxg+OGZnHmuAFcPauI+AjqCA6Eqvpmxg5KD3cYhwhasjCzWOAh4AygFFhsZnOcc2t86owCbgNmOucOmFmetzwb+ClQjCeJLPVueyBY8YpIZIqPjWHmyBxmjszhjnPHs3pnFW+t3cNbH+3ll6+vJTbGuOak4eEOM6A8HdyRdRkqmOl4GrDBObfJOdcEPAec16HOtcBD7UnAObfXW34WMM85t9+7bh4wO4ixikgUiIkxJhRk8p3TR/PKTbM4vjCLP72/hZbWtnCHFjCtbY7qhpaIeiAPgpss8oHtPsul3jJfo4HRZrbAzBZ6L1t1d1sR6eO+Nms4pQfqeXPNnnCHEjA17eNC9aFkYZ2UuQ7LccAo4FTgMuAxM+vXzW0xs+vMbImZLSkrK+thuCISbc4YN4Bh/VN47N1N4Q4lYNqH+uhLz1mUAkN8lguAnZ3U+Ydzrtk5txlYhyd5dGdbnHOPOueKnXPFubm5AQ1eRCJfbIxx9cwilm2rYOnW3tGlWVkfecOTQ3CTxWJglJkVmVkCcCkwp0OdvwOnAZhZDp7LUpuAucCZZpZlZlnAmd4yEZFDXDy1gIykOP74Xu9oXVRG4Cx5EMRk4ZxrAW7E80f+I+B559xqM7vDzM71VpsLlJvZGuAd4FbnXLlzbj9wJ56Esxi4w1smInKI1MQ4Lp8+jDdW7Wb7/rpwh9Njq3ZUAjAiLzXMkRwqqDcnO+dec86Nds6NcM7d5S273Tk3x/veOee+65wb55yb4Jx7zmfbx51zI72vJ4IZp4hEtytOHEaMGU8s2BLuUHps4aZyRuSmkpeeFO5QDtG7nmQRkT5pUGYyn5s4iL8u3sbe6oZwh3PUWlrbWLzlACcM7x/uUD5FyUJEeoUbPzOSNgc3PL2MppbofO5i9c4qahpblCxERIJlZF4691w8kSVbD/C/r6wOdzhHZeGmcgCmD88OcySfFlk38oqI9MC5kwazemclv//3JibkZ3LptKHhDumIRGp/BahlISK9zPfPOoaTRuVw+z9Ws2xb9Dx7Ecn9FaBkISK9TGyM8eBlUxiYmcQ3nl4aNZMoRXJ/BShZiEgv1C8lgQcuncyeqkYemb8h3OF0SyT3V4CShYj0UlOGZnH+5MH84d3NlB6I/If1Irm/ApQsRKQXu3X2MRhw7xvrwh3KYUV6fwUoWYhIL5bfL5nrTh7OnJKdEd3ZHen9FaBkISK93NdPGUFueiJ3vroG5z4100FYlFU30tDcenA50vsrQMlCRHq51MQ4bj1zDB9uq+CVFbvCHQ7VDc185lfzOfHut7l/3sfsq2mM+P4KULIQkT7goqkFjBuUwT2vrz3kP/pweG3lLqobWxiZl8YDb63nxLvfZsGG8oi+BAVKFiLSB8TGGD/53Dh2VNTz6H/CO+/FS8t2MDw3lb9edwL/+u4pXHRcPglxMZw9flBY4/JHyUJE+oQZI/pzzoSBPDJ/I7sq68MSw/b9dSzavJ+LjivAzBiZl8YvL5zIqv89i1mjcsISU3cpWYhIn3Hb2WNpdY57Xl8bluO/vGwHZnD+lPywHL8nlCxEpM8Ykp3CdScN5+/Ld4Z8zm7nHC9/WMqM4f3Oq2PjAAANlklEQVTJ75cc0mMHgpKFiPQp3zh1BHnpidzxymra2kJ3K+3SrQfYWl7HRccVhOyYgaRkISJ9SmpiHD84+xhKSit5+cMdITvuS8tKSUmIZfb4gSE7ZiApWYhIn3P+5HyOG9qPn/9zDbsrgz8Na0NzK6+u2MXs8QNJTYzOaYSULESkz4mJMX51ySQam9v43gvLg345at6aPVQ3tETtJShQshCRPmp4bho//fw4Fmwo54/vbQ7qsf724Q4GZSZF/IN3h6NkISJ91hePH8JZxw7g3rlrWb2zMmjHWbbtAKeOySM2xoJ2jGBTshCRPsvMuPvCiWSnJvDt55ZT3xT4oUD21zZRUdfMyLy0gO87lJQsRKRPy0pN4NeXTGbD3hrueSPwD+ttLKsBYERuasD3HUpKFiLS580alcNXZwzjyQ+28GGA573YuLc9WahlISIS9W49awwD0pO47eWVNLe2BWy/m/bVkhgXw+AofGrbl5KFiAiQnhTPneePZ+3u6oCOTLtxbw1FOalR3bkNShYiIgedMW4A50wYyANvrWeTt6+hpzaW1UT9JShQshAROcTPPn8siXEx/PBvK3s8DWtjSyvb9tdFfec2KFmIiBwiLyOJH54zloWb9vPysp6NHbWtvI42ByOi/LZZULIQEfmULxYPYdKQfvzqzXU9moa1/bbZ4TlKFiIivU5MjHHb2cewq7KBJxZsOer9bCyrBWC4LkOJiPROJwzvz+lj83j4nQ3sr206qn1sLKthUGZS1I4060vJQkSkC//f7GOobWrhwbfXH9X2G8tqe0WrApQsRES6NGpAOl88fghPL9zK1vLaI9rWOcemvb3jtllQshAROazvnD6auJgY7pu77oi2K6tupLqxRclCRKQvGJCRxLUnFfHqil289dGebm/Xmzq3IcjJwsxmm9k6M9tgZj/oZP2VZlZmZsu9r2t81rX6lM8JZpwiIodz/SkjmJCfyQ3PLOODjeXd2uaT0WbVsjgsM4sFHgLOBsYBl5nZuE6q/tU5N9n7esynvN6n/NxgxSki4k9qYhxPXj2NodkpXPPkYpZvr/C7zcayGlISYhmYkRSCCIMvmC2LacAG59wm51wT8BxwXhCPJyISNNmpCTx9zXT6pyVyxeOLWLu76rD1N5XVUpSTSkyUDyDYLpjJIh/Y7rNc6i3r6CIzW2FmL5rZEJ/yJDNbYmYLzez8IMYpItItAzKSeOaa6STFx/CVPy5iX01jl3V7ywCC7YKZLDpLpx1H5XoFKHTOTQT+BTzps26oc64YuBz4PzMb8akDmF3nTShLysrKAhW3iEiXhmSn8Oerp1NZ18wdr6zptE5Dcys7KuqVLLqpFPBtKRQAO30rOOfKnXPtqfkPwFSfdTu9XzcB84EpHQ/gnHvUOVfsnCvOzc0NbPQiIl0YMzCdb542kjklO3ln3d5Prd+8rxbnes+dUBDcZLEYGGVmRWaWAFwKHHJXk5kN8lk8F/jIW55lZone9znATKDzFC4iEgbfOHUEo/LS+PHfVlHb2HLIut52JxQEMVk451qAG4G5eJLA88651WZ2h5m13930LTNbbWYlwLeAK73lY4El3vJ3gLudc0oWIhIxEuJi+OWFE9hRUc+v3/z4kHUb99ZiBkU5vadlEdTRrZxzrwGvdSi73ef9bcBtnWz3PjAhmLGJiPRUcWE2Xz5hKH96fzP/M3EQ4Pj3x/t4aWkpgzOTSU6IDXeIARP9QyGKiITR92cfw7w1e7jokfcBiDGYPKQfV5xYGN7AAkzJQkSkBzKS4vntpVOYU7KTGSP6M2tkDv1SEsIdVsApWYiI9ND04f2ZPrx/uMMIKg0kKCIifilZiIiIX0oWIiLil5KFiIj4pWQhIiJ+KVmIiIhfShYiIuKXkoWIiPhlznWcYiI6mVklsL6TVZlApZ8y3+XO3rd/zQH2HWWIncXRnfWK/9Cyoz0Hf/Efrs7h4u247O+94j/yOv5+h7o6n0DGf7j4/K2P9M/wMOec/zkenHO94gU82t3yjmW+y5299/m6JNDxKf7ux9+Tc/AX/5Gcw5HGH4ifgeLvuqyr8wlk/N05h2j/DPt79abLUK8cQXnHslf8vO9q30fC3z4Uf9fvQxH/4eocLt6Oy915fzQUf9dlXZ1PIOPvzj56w2egS73mMlQomNkS55nqNSpFe/wQ/eeg+MNL8R+93tSyCIVHwx1AD0V7/BD956D4w0vxHyW1LERExC+1LERExK8+myzM7HEz22tmq45i26lmttLMNpjZb83MfNbdZGbrvHOL3xvYqA+JIeDxm9nPzGyHmS33vs4JfOQHYwjK99+7/hYzc2aWE7iIO40jGD+DO81shff7/6aZDQ585AdjCEb895nZWu85/M3M+gU+8oMxBCP+S7yf3TYzC0rfQE/i7mJ/V5jZeu/rCp/yw35OjlhPbqWK5hdwMnAcsOootl0EzAAMeB0421t+GvAvING7nBdl8f8MuCVav//edUOAucBWICfazgHI8KnzLeD/j7L4zwTivO/vAe6JsvjHAmOA+UBxJMXtjamwQ1k2sMn7Ncv7Putw53i0rz7bsnDO/QfY71tmZiPM7A0zW2pm75rZMR23M7NBeD7QHzjPT+TPwPne1d8A7nbONXqPsTfK4g+ZIMZ/P/B9IOidccE4B+dclU/VVIJ4HkGK/03nXIu36kKgIMri/8g5ty5YMfck7i6cBcxzzu13zh0A5gGzg/E577PJoguPAjc556YCtwAPd1InHyj1WS71lgGMBk4ys/+a2b/N7PigRvtpPY0f4EbvJYTHzSwreKF2qkfxm9m5wA7nXEmwAz2MHv8MzOwuM9sOfAm4PYixdiYQv0PtrsbzH20oBTL+UOpO3J3JB7b7LLefS8DPUXNwe5lZGnAi8ILPpb3Ezqp2Utb+318cnqbgCcDxwPNmNtyb2YMqQPE/AtzpXb4T+DWeD3zQ9TR+M0sBfoTnMkhYBOhngHPuR8CPzOw24EbgpwEOtVOBit+7rx8BLcAzgYzxcAIZfygdLm4zuwr4trdsJPCamTUBm51zF9D1uQT8HJUsPhEDVDjnJvsWmlkssNS7OAfPH1TfpnUBsNP7vhR42ZscFplZG56xXMqCGbhXj+N3zu3x2e4PwKvBDLiDnsY/AigCSrwfuAJgmZlNc87tDnLs7QLxO+TrL8A/CVGyIEDxeztZPwd8NhT/KPkI9Pc/VDqNG8A59wTwBICZzQeudM5t8alSCpzqs1yAp2+jlECfYzA6cKLlBRTi08kEvA9c4n1vwKQutluMp/XQ3nF0jrf868Ad3vej8TQPLYriH+RT52bguWj6/neos4Ugd3AH6WcwyqfOTcCLURb/bGANkBvs730wf4cIYgf30cZN1x3cm/Fc0cjyvs/uzjkeccyh+IFG4gt4FtgFNOPJwl/D85/pG0CJ9xf+9i62LQZWARuB3/HJw40JwNPedcuAz0RZ/E8BK4EVeP4DGxRN8Xeos4Xg3w0VjJ/BS97yFXjG8smPsvg34Pknabn3Fcy7uYIR/wXefTUCe4C5kRI3nSQLb/nV3u/7BuCqI/mcHMlLT3CLiIhfuhtKRET8UrIQERG/lCxERMQvJQsREfFLyUJERPxSspBezcxqQny8x8xsXID21Wqe0WdXmdkr/kZwNbN+ZnZDII4t0pFunZVezcxqnHNpAdxfnPtkoLyg8o3dzJ4EPnbO3XWY+oXAq8658aGIT/oWtSykzzGzXDN7ycwWe18zveXTzOx9M/vQ+3WMt/xKM3vBzF4B3jSzU81svpm9aJ65G55pnyvAW17sfV/jHRSwxMwWmtkAb/kI7/JiM7ujm62fD/hkwMQ0M3vLzJaZZ76C87x17gZGeFsj93nr3uo9zgoz+98Afhulj1GykL7oAeB+59zxwEXAY97ytcDJzrkpeEZ7/YXPNjOAK5xzn/EuTwG+A4wDhgMzOzlOKrDQOTcJ+A9wrc/xH/Ae3+94Pd6xjT6L56l6gAbgAufccXjmUPm1N1n9ANjonJvsnLvVzM4ERgHTgMnAVDM72d/xRDqjgQSlLzodGOczwmeGmaUDmcCTZjYKzwid8T7bzHPO+c5BsMg5VwpgZsvxjPXzXofjNPHJYIxLgTO872fwydwCfwF+1UWcyT77XopnrgLwjPXzC+8f/jY8LY4BnWx/pvf1oXc5DU/y+E8XxxPpkpKF9EUxwAznXL1voZk9CLzjnLvAe/1/vs/q2g77aPR530rnn6Vm90mnYFd1DqfeOTfZzDLxJJ1vAr/FM89FLjDVOddsZluApE62N+CXzrnfH+FxRT5Fl6GkL3oTzzwRAJhZ+9DQmcAO7/srg3j8hXgufwFc6q+yc64SzxSrt5hZPJ4493oTxWnAMG/VaiDdZ9O5wNXe+RIws3wzywvQOUgfo2QhvV2KmZX6vL6L5w9vsbfTdw2eoeUB7gV+aWYLgNggxvQd4LtmtggYBFT628A59yGeEUkvxTOhULGZLcHTyljrrVMOLPDeanufc+5NPJe5PjCzlcCLHJpMRLpNt86KhJh3Vr9655wzs0uBy5xz5/nbTiSc1GchEnpTgd9572CqIERT14r0hFoWIiLil/osRETELyULERHxS8lCRET8UrIQERG/lCxERMQvJQsREfHr/wEeb6tHPPo3XAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot(suggestion=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.318212</td>\n",
       "      <td>0.226600</td>\n",
       "      <td>0.902946</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "alpha = 1e-2\n",
    "learn.fit_one_cycle(1, alpha, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('first')\n",
    "learn.load('first');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From here on, we start to gradually unfreeze the layers and re-train the network, following the findings of the paper."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.307525</td>\n",
       "      <td>0.185602</td>\n",
       "      <td>0.928943</td>\n",
       "      <td>00:08</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.freeze_to(-2)\n",
    "alpha /= 2\n",
    "learn.fit_one_cycle(1, slice(alpha/(2.6**4),alpha), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('second')\n",
    "learn.load('second');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.255565</td>\n",
       "      <td>0.178561</td>\n",
       "      <td>0.933276</td>\n",
       "      <td>00:12</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.freeze_to(-3)\n",
    "alpha /= 2\n",
    "learn.fit_one_cycle(1, slice(alpha/(2.6**4),alpha), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('third')\n",
    "learn.load('third');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.219360</td>\n",
       "      <td>0.160707</td>\n",
       "      <td>0.937608</td>\n",
       "      <td>00:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.189047</td>\n",
       "      <td>0.157505</td>\n",
       "      <td>0.940208</td>\n",
       "      <td>00:18</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.unfreeze()\n",
    "alpha /= 5\n",
    "learn.fit_one_cycle(2, slice(alpha/(2.6**4),alpha), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's save the final model for quicker future use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('final')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Checking results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "if learn is None:\n",
    "    learn = initialize_text_classifier()\n",
    "learn.load('final');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train score = 0.961227, test score = 0.940208\n"
     ]
    }
   ],
   "source": [
    "score_train = learn.validate(data_clas.train_dl)[1].item()\n",
    "score_test = learn.validate(data_clas.valid_dl)[1].item()\n",
    "\n",
    "print(\"Train score = %f, test score = %f\" % (score_train, score_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's interpret the results some more. We can get the confusion matrix and the \"most confused\" samples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "interp = TextClassificationInterpretation.from_learner(learn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAARoAAAEmCAYAAAC9C19sAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAG9BJREFUeJzt3Xm4VWX5xvHvzXxQcAicRRwQHGNyol8mopamJqIgooZDipqVikOaiUOEQ2pqmloZTjmmpuFQ5IhggOKUhBqaioqQCigo4PP7Y61DWwLOAXn3gnXuz3Xt6+w1P5tzuPf7vmvtvRQRmJml1KjoAsys/Bw0Zpacg8bMknPQmFlyDhozS85BY2bJOWjsS5FUI+k+SR9JuuNL7GeApIeXZ21FkfR1Sf8suo4ViXwdTcMg6WDgJKATMBOYAPwsIp78kvs9FDgB6BER8750oSs4SQF0iIhXi65lZeIWTQMg6STgMmAosDbQDrgK+M5y2P1GwKSGEDL1IalJ0TWskCLCjxI/gNWAWcCBS1inOVkQTckflwHN82W7AG8BJwNTgXeAw/Nl5wCfAXPzYxwJDAFuqth3eyCAJvn0QOBfZK2qycCAivlPVmzXAxgLfJT/7FGx7FHgPGBUvp+HgTaLeW219Z9aUf9+wF7AJOA/wBkV628PjAY+zNe9EmiWL3s8fy0f56+3X8X+TwPeBW6snZdvs2l+jK759HrANGCXov82qvp3WHQBfiT+BcO3gHm1/9EXs865wBhgLaAt8BRwXr5sl3z7c4Gm+X/QT4A18uULB8tigwZYBZgBdMyXrQtslT9fEDTAmsAHwKH5dv3z6a/kyx8FXgM2B2ry6WGLeW219f80r/97wPvALUArYCtgDrBJvn43YMf8uO2Bl4EfVewvgM0Wsf8LyAK7pjJo8nW+l++nJfAQcHHRfxfVfrjrVH5fAabFkrs2A4BzI2JqRLxP1lI5tGL53Hz53IgYQfZu3nEZ6/kc2FpSTUS8ExEvLWKdbwOvRMSNETEvIv4ATAT2qVjn+oiYFBGzgduBzks45lyy8ai5wK1AG+CXETEzP/5LwLYAETE+Isbkx30duAb4Rj1e09kR8WlezxdExHXAK8DTZOF6Zh37Kx0HTflNB9rUMXawHvBGxfQb+bwF+1goqD4BVl3aQiLiY7LuxiDgHUl/ltSpHvXU1rR+xfS7S1HP9IiYnz+vDYL3KpbPrt1e0uaS7pf0rqQZZONabZawb4D3I2JOHetcB2wNXBERn9axbuk4aMpvNFnXYL8lrDOFbFC3Vrt83rL4mKyLUGudyoUR8VBE7E72zj6R7D9gXfXU1vT2Mta0NK4mq6tDRLQGzgBUxzZLPHUraVWyca/fAkMkrbk8Cl2ZOGhKLiI+Ihuf+JWk/SS1lNRU0p6SLsxX+wPwE0ltJbXJ179pGQ85AdhZUjtJqwE/rl0gaW1J+0paBfiUrAs2fxH7GAFsLulgSU0k9QO2BO5fxpqWRiuycaRZeWvr2IWWvwdsspT7/CUwPiKOAv4M/PpLV7mScdA0ABFxCdk1ND8hGwh9E/g+cE++yvnAOOB54AXgmXzeshzrL8Bt+b7G88VwaER29moK2ZmYbwDHLWIf04G983Wnk50x2jsipi1LTUtpMHAw2dms68heS6UhwHBJH0rqW9fOJH2HbEB+UD7rJKCrpAHLreKVgC/YM7Pk3KIxs+QcNGaWnIPGzJJz0JhZcv4AWB3UpCbUrFXRZViuyxbtii7BKjzzzPhpEdG2rvUcNHVQs1Y071jnWUyrklFPX1l0CVahpqkWvoJ7kdx1MrPkHDRmlpyDxsySc9CYWXIOGjNLzkFjZsk5aMwsOQeNmSXnoDGz5Bw0Zpacg8bMknPQmFlyDhozS85BY2bJOWjMLDkHjZkl56Axs+QcNGaWnIPGzJJz0JhZcg4aM0vOQWNmyTlozCw5B42ZJeegMbPkHDRmlpyDxsySc9CYWXIOGjNLzkFjZsk5aMwsOQeNmSXnoDGz5Bw0Zpacg8bMknPQmFlyDhozS85BY2bJOWjMLDkHjZkl56Axs+QcNGaWnIPGzJJz0JhZck2KLsCWjxMG9GRg7x5EBC+9OoWjz76JHb+6CT8/sTfNmjbm2ZffZNA5NzN//ueceFgv+u21HQBNGjei08brsOGup/PBjE8KfhXl8+abb3LU4Yfx3nvv0qhRI4448mi+/4MfAnDVlVfw66uvpEmTJnxrz28zdNiFBVebzkobNJJWBw6OiKvy6fWAyyPigGIrq7712q7Gcf2/QZc+P2POp3O56YIj6Ldnd84a9G32POYKXv33VM469tscss8ODL9nNJfeMJJLbxgJwF47b80JA3o6ZBJp0qQJwy78BV26dmXmzJn02KEbvXbbnalT3+P+++5l7DPP07x5c6ZOnVp0qUmtzF2n1YHjaiciYkpDDJlaTRo3pqZ5Uxo3bkRNi2Z8MvszPv1sHq/+O/sD/tuYiezXq/P/bNf3W925/cHx1S63wVh33XXp0rUrAK1ataJTpy2YMuVtrr3magafejrNmzcHYK211iqyzOSSBY2k9pJelnSdpJckPSypRtKmkh6UNF7SE5I65etvKmmMpLGSzpU0K5+/qqSRkp6R9IKk7+SHGAZsKmmCpIvy472Yb/O0pK0qanlUUjdJq0j6XX6MZyv2tVKb8v5HXHbDSCY9cB6T//IzZsyazZ0PP0PTpo3pumU7AHrv1pkN1l7jC9vVtGjK7j224J6RE4oou8F54/XXmTDhWbbbfgdenTSJUU8+wdd77MDuu36DcWPHFl1eUqlbNB2AX0XEVsCHQB/gWuCEiOgGDAauytf9JfDLiNgOmFKxjzlA74joCvQEfiFJwOnAaxHROSJOWei4twJ9ASStC6wXEeOBM4G/5cfoCVwkaZWFi5Z0tKRxksbFvNnL4Z8hrdVb1bD3Ltuwxd5ns8keZ7JKTTMO2ms7Djv9ei48eX+euHEwMz/+lHnz539hu2/vvA2jJ/zL3aYqmDVrFv379uGiX1xG69atmTd/Hh988AGPjxrD0GEXccjBfYmIostMJvUYzeSIqH27HA+0B3oAd2RZAUDz/OdOwH7581uAi/PnAoZK2hn4HFgfWLuO494O/AU4myxw7sjn7wHsK2lwPt0CaAe8XLlxRFxLFog0arnWCv/b33WHTrw+ZTrTPpgFwD1/e44dv7oxt44Yy25HXgZArx070WGjLzbPD/xmN+5wtym5uXPn0r9vH/r1H8B+vfcHYP31N2C/3vsjie22355GjRoxbdo02rZtW3C1aaRu0Xxa8Xw+sCbwYd4KqX1sUcc+BgBtgW4R0Rl4jywgFisi3gamS9oW6EfWwoEstPpUHLtdRLy82B2tJN589z9sv83G1LRoCkDP7Tvyz8nv0XaNVQFo1rQJJw/cnevufHLBNq1XbcH/dduM+x59vpCaG4qIYND3jqRjpy344YknLZi/z7778egjfwPglUmT+Oyzz2jTpk1RZSZX7bNOM4DJkg6MiDvyLtC2EfEcMIasa3UbcFDFNqsBUyNirqSewEb5/JlAqyUc61bgVGC1iHghn/cQcIKkEyIiJHWJiGeX38srxtgX3+Duvz7L6FtOY978z3lu4lv89q5RDDl+b/b8+tY0aiSuu+MJHhs7acE2+/b8KiPHTOSTOZ8VWHn5PTVqFLfcfCNbb70NO3TLBuPPOX8o3z38CI456gi6dd6aZk2b8ZvfDaeilV86StUvlNQeuD8its6nBwOrAsOBq4F1gabArRFxrqQOwE1krY4/A0dHxPqS2gD35etOAL4G7BkRr0u6BdgWeAD41ULHWxt4GzgvIs7J59UAl5F13wS8HhF7L+l1NGq5VjTv2Hf5/KPYl/bB2CuLLsEq1DTV+IjoXtd6yYJmaUlqCczOWxoHAf0jovCzQg6aFYuDZsVS36BZkS7Y6wZcmXenPgSOKLgeM1tOVpigiYgngK8WXYeZLX8r85XBZraScNCYWXIOGjNLzkFjZsk5aMwsOQeNmSXnoDGz5Bw0Zpacg8bMknPQmFlyDhozS85BY2bJOWjMLDkHjZkl56Axs+QcNGaWnIPGzJJz0JhZcg4aM0vOQWNmyTlozCw5B42ZJeegMbPkHDRmlpyDxsySc9CYWXIOGjNLzkFjZsk5aMwsOQeNmSXXZHELJN0HxOKWR8S+SSoys9JZbNAAF1etCjMrtcUGTUQ8Vs1CzKy8ltSiAUBSB+DnwJZAi9r5EbFJwrrMrETqMxh8PXA1MA/oCdwA3JiyKDMrl/oETU1EjAQUEW9ExBBg17RlmVmZ1Nl1AuZIagS8Iun7wNvAWmnLMrMyqU+L5kdAS+AHQDfgUOC7KYsys3Kps0UTEWPzp7OAw9OWY2ZlVJ+zTo+wiAv3IsLjNGZWL/UZoxlc8bwF0IfsDJSZWb3Up+s0fqFZoyT5Yj4zq7f6dJ3WrJhsRDYgvE6yilYwnbdox+NPXV50GZZ75d1ZRZdgy6A+XafxZGM0IusyTQaOTFmUmZVLfYJmi4iYUzlDUvNE9ZhZCdXnOpqnFjFv9PIuxMzKa0nfR7MOsD5QI6kLWdcJoDXZBXxmZvWypK7TN4GBwAbAL/hv0MwAzkhblpmVyZK+j2Y4MFxSn4i4q4o1mVnJ1GeMppuk1WsnJK0h6fyENZlZydQnaPaMiA9rJyLiA2CvdCWZWdnUJ2gaV57OllQD+PS2mdVbfa6juQkYKen6fPpwYHi6ksysbOrzWacLJT0P7EZ25ulBYKPUhZlZedT3BnLvAp+TfXK7F/BysorMrHSWdMHe5sBBQH9gOnAb2fcG96xSbWZWEkvqOk0EngD2iYhXASSdWJWqzKxUltR16kPWZXpE0nWSevHfq4PNzOptsUETEXdHRD+gE/AocCKwtqSrJe1RpfrMrATqHAyOiI8j4uaI2Jvsc08TgNOTV2ZmpVHfs04ARMR/IuIafzG5mS2NpQoaM7Nl4aAxs+QcNGaWnIPGzJJz0JhZcg4aM0vOQWNmyTlozCw5B42ZJeegMbPkHDRmlpyDxsySc9CYWXIOGjNLzkFjZsk5aMwsOQeNmSXnoDGz5Bw0Zpacg8bMknPQmFlyDhozS85BY2bJOWjMLDkHjZkl56Axs+QcNGaWnIOmpObPn8/XdujGAb33AeC4Y45ip+26sGP3zhzS/0BmzZpVcIXl9dOTj+UbnTemd6/tF8yb+NLzDNi3Jwd+swcH7bUzLzw7DoCZMz7i+4cfyAF77ETvXttxz203FlV2Uitd0EgaJOmw/PlASetVLPuNpC2Lq27FcdWVl9OxY6cF08MuuoTRY59lzLgJbLBhO665+lcFVldu+x44gKtvvPsL8y792VkMOvHH3PHQUxw/+EwuHXoWALcOv5ZNO3TizodH89vbR3DxeWcy97PPiig7qZUuaCLi1xFxQz45EFivYtlREfGPQgpbgbz91ls89MAIvnv4kQvmtW7dGoCIYM7s2UgqqrzS677j/7Ha6mt8YZ4kPp45E4CZM2bQdu11/zt/1iwigk8+/pjVVl+Dxk2aVL3m1Kr6iiS1Bx4Enga6AJOAw4CdgIvzesYCx0bEp5KGAfsC84CHI2KwpCHALOB1oDtws6TZ+T4eAAYD2wEbR8Sp+XEHAt0i4gRJhwA/AJrldRwXEfNTv/ZqOu2UEzlv6DBm5X/YtQZ97wgefugBOnXakqEXXFxQdQ3TqUOGMeiQ3vzi/DOJzz/nhnv+CkD/gcfwgyP60at7Bz6eNYuLrvo9jRqtdO//dSriFXUEro2IbYEZwEnA74F+EbENWdgcK2lNoDewVb7u+ZU7iYg7gXHAgIjoHBGzKxbfCexfMd0PuE3SFvnzr0VEZ2A+MGDhAiUdLWmcpHHT3n9/ubzoanlgxP20bbsWXbp2+59lv77ud7wy+S06durEXXfcVkB1DdftN/6WU84exl/+PpFTzh7G2accD8Cox0bSccttGTnuFe54cBRDzxrMrJkzCq52+SsiaN6MiFH585uAXsDkiJiUzxsO7EwWQnOA30jaH/ikvgeIiPeBf0naUdJXyMJtVH6sbsBYSRPy6U0Wsf21EdE9Irq3adt2mV5kUcY89RQj/nwfW22+CQMPO5jHH32EowYeumB548aN6XNAX+69548FVtnw/OnOW9htz30B2GPv3rw4YTwA995+I7323AdJtNt4U9bfcCMmvzppSbtaKRURNFGvlSLmAdsDdwH7kXW5lsZtQF+gD3B3RAQgYHjeAuocER0jYshS7neFds75Q/nna//mpUn/4vc33MLOu/Tkuutv4LXXXgWyMZoRI+5n84qBYkuv7drrMG7MkwA8Peox2m28KQDrrLchT496DIDp70/ljddeYYON2hdVZjJFjDq1k7RTRIwG+gN/BY6RtFlEvAocCjwmaVWgZUSMkDQGeHUR+5oJtFrMcf4InAm8AZyWzxsJ3Cvp0oiYmnfPWkXEG8vv5a14IoJjjjycmTNnEBFss822XHrFVUWXVVqnHn8448Y8wYf/mc5u23XkuJPP4OwLruCCIacxf948mjVvwdnDLgfgmB+exlknDWL/3XYgIvjRGeeyxpptCn4Fy5+yN/oqHSwbDB4BPA70AF4hC5b/GQwG1gTuBVqQtUQujojhtYPBEXGxpD7AUOALg8ERMS4/3v3AlhGxoHskqR/wY7LW3Fzg+IgYs7iau3brHo8/9ffl9C9gX9bk9+vdg7Yq2HbDVuMjontd6xURNPdHxNZVO+iX5KBZsThoViz1DZrynUczsxVOVcdoIuJ1YKVpzZjZ8uEWjZkl56Axs+QcNGaWnIPGzJJz0JhZcg4aM0vOQWNmyTlozCw5B42ZJeegMbPkHDRmlpyDxsySc9CYWXIOGjNLzkFjZsk5aMwsOQeNmSXnoDGz5Bw0Zpacg8bMknPQmFlyDhozS85BY2bJOWjMLDkHjZkl56Axs+QcNGaWnIPGzJJz0JhZcg4aM0vOQWNmyTlozCw5B42ZJeegMbPkHDRmlpyDxsySc9CYWXIOGjNLzkFjZsk5aMwsOQeNmSXnoDGz5BQRRdewQpP0PvBG0XUsB22AaUUXYQuU5fexUUS0rWslB00DIWlcRHQvug7LNLTfh7tOZpacg8bMknPQNBzXFl2AfUGD+n14jMbMknOLxsySc9CYWXIOGjNLzkFjZsk5aEpOUo2kjkXXYaDMIZJ+mk+3k7R90XVVg4OmxCTtA0wAHsynO0v6U7FVNWhXATsB/fPpmcCviiunehw05TYE2B74ECAiJgDtC6ynodshIo4H5gBExAdAs2JLqg4HTbnNi4iPii7CFpgrqTEQAJLaAp8XW1J1OGjK7UVJBwONJXWQdAXwVNFFNWCXA3cDa0n6GfAkMLTYkqrDVwaXmKSWwJnAHvmsh4DzI2JOcVU1bJI6Ab0AASMj4uWCS6oKB02JSeoSEc8WXYdlJP0SuC0iGlyr0l2ncrtE0kRJ50naquhijGeAn0h6VdJFkhrM99G4RVNyktYB+gL9gNZk76jnF1tVwyZpTaAPcBDQLiI6FFxScm7RlFxEvBsRlwODyK6p+WnBJRlsBnQiu9RgYrGlVIdbNCUmaQuylswBwHTgVuCuiJhaaGENlKQLgP2B14DbgT9GxIfFVlUdTYouwJK6HvgDsEdETCm6GGMysFNElOFLyZeKWzRmiUnqFBETJXVd1PKIeKbaNVWbg6aEJN0eEX0lvUB+FWrtIiAiYtuCSmuQJF0bEUdLemQRiyMidq16UVXmoCkhSetGxDuSNlrU8ogow32qVjqSWix8seSi5pWRzzqVUES8kz89LiLeqHwAxxVZWwO3qAv1GsTFew6actt9EfP2rHoVDZykdSR1A2okdZHUNX/sArQsuLyq8FmnEpJ0LFnLZRNJz1csagWMKqaqBu2bwEBgA+CSivkzgTOKKKjaPEZTQpJWA9YAfg6cXrFoZkT8p5iqTFKfiLir6DqK4KBpACStBbSonY6IfxdYToMj6ZCIuEnSyXzxLCAAEXHJIjYrFXedSiz/Ks9LgPWAqcBGwMuAP2BZXavkP1cttIoCuUVTYpKeA3YF/hoRXST1BPpHxNEFl2YNjM86ldvciJgONJLUKCIeAToXXVRDJelCSa0lNZU0UtI0SYcUXVc1OGjK7UNJqwKPAzfnX7w0r+CaGrI9ImIGsDfwFrA5cEqxJVWHg6bcvgPMBk4ku+XKa8A+hVbUsDXNf+4F/KEhnQH0YHCJRcTHFZPDCyvEat0naSJZ+B+X3wWh9B8/AA8Gl5qkmfzv6dSPgHHAyRHxr+pX1bBJWgOYERHz8y+Pbx0R7xZdV2pu0ZTbJcAU4BayT24fBKwD/BP4HbBLYZU1QJKaAocCO0sCeAz4daFFVYlbNCUm6emI2GGheWMiYkdJz0XEV4uqrSGS9BuycZrabuyhwPyIOKq4qqrDLZpy+1xSX+DOfPqAimV+h6m+7RYK97/l1zqVns86ldsAsnfNqcB7+fNDJNUA3y+ysAZqvqRNayckbQLML7CeqnHXyaxKJPUi+x7n2kH49sDh+YWUpeYWTYlJ2jy/AvXFfHpbST8puq4GbBRwDfB5/rgGGF1oRVXiFk2JSXqM7MrTayKiSz7vxYjYutjKGiZJtwMzgJvzWf2BNSLiwOKqqg4PBpdby4j4e34qtZY/glCcjgsNBj/iwWArg2n54GMASDoAeGfJm1hCz0rasXZC0g40kG88dNepxPKzGtcCPYAPyG5gNsB3QSiGpJeBjkDtF4+1I/t+oM8p+W1wHDQlJqk52bUz7YE1ycYHIiLOLbKuhmpxt7+pVeY3AI/RlNu9wIfAM2QfRbAClTlI6uIWTYn5DJOtKDwYXG5PSdqm6CLM3KIpMUn/ADYjGwT+FN972wrioCkx33vbVhQOGjNLzmM0Zpacg8bMknPQ2HInab6kCZJelHRH/t24y7qvXSTdnz/fV9LpS1h3dUnHLcMxhkgavKw1Wt0cNJbC7IjonF/D8xkwqHKhMkv9txcRf4qIYUtYZXVgqYPG0nPQWGpPAJtJai/pZUlXkV2pvKGkPSSNlvRM3vJZFUDStyRNlPQksH/tjiQNlHRl/nxtSXdLei5/9ACGAZvmramL8vVOkTRW0vOSzqnY15mS/inpr2SfP7KEHDSWjKQmwJ7AC/msjsAN+XfjfAz8BNgtIrqS3QLmJEktgOvIbnT3dbK7NizK5cBj+dcudAVeAk4HXstbU6dI2gPoAGxPdivgbpJ2ltSN7I4QXciCbLvl/NJtIf6sk6VQI2lC/vwJ4LfAesAbETEmn78jsCUwKv++nGZk3zbXCZgcEa8ASLoJOHoRx9gVOAwgIuYDH+X3TKq0R/54Np9elSx4WgF3R8Qn+TH+9KVerdXJQWMpzI6IzpUz8jCpvHOmgL9ERP+F1uvM8rtDg4CfR8Q1Cx3jR8vxGFYP7jpZUcYAX5O0GYCklpI2ByYCG1fcLaD/YrYfCRybb9tYUmtgJllrpdZDwBEVYz/rS1oLeBzoLalGUit8P/LkHDRWiIh4HxgI/EHS82TB0yki5pB1lf6cDwYv7uMSPwR6SnoBGA9sFRHTybpiL0q6KCIeJrtL5+h8vTuBVhHxDHAbMAG4i6x7Zwn5IwhmlpxbNGaWnIPGzJJz0JhZcg4aM0vOQWNmyTlozCw5B42ZJff/Yu49wSFF4J4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "interp.plot_confusion_matrix()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Text</th>\n",
       "      <th>Prediction</th>\n",
       "      <th>Actual</th>\n",
       "      <th>Loss</th>\n",
       "      <th>Probability</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>xxbos last 2 times i checked a bags they were xxunk damaged . xxmaj no one answers the baggage call line for status ? xxunk</td>\n",
       "      <td>negative</td>\n",
       "      <td>negative</td>\n",
       "      <td>5.22</td>\n",
       "      <td>0.99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj its not just xxup phl , at xxup btv today made to deplane because \" it 's xxmaj sunday and we do n't know if maintenance will be available \"</td>\n",
       "      <td>negative</td>\n",
       "      <td>negative</td>\n",
       "      <td>4.13</td>\n",
       "      <td>0.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxup dfw xxup gate 16 to xxup xxunk agent seriously is beyond incompetent . xxmaj never flying xxup aa again if can be xxunk fail pathetic</td>\n",
       "      <td>negative</td>\n",
       "      <td>negative</td>\n",
       "      <td>3.76</td>\n",
       "      <td>0.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos and btw , the xxhan and xxhan managed on time departures same time same destination - &amp; would n't take away a xxunk old kids bag !</td>\n",
       "      <td>negative</td>\n",
       "      <td>negative</td>\n",
       "      <td>3.21</td>\n",
       "      <td>0.96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj thank you ! ! ! xxmaj this whole crew has xxunk through bad weather and xxunk . xxmaj pilot keeping us well informed . customerservice</td>\n",
       "      <td>negative</td>\n",
       "      <td>positive</td>\n",
       "      <td>3.16</td>\n",
       "      <td>0.04</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "interp.show_top_losses(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But this is not so informative, so I made a small modification to just show the ones with misclassified labels :-)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Text</th>\n",
       "      <th>Prediction</th>\n",
       "      <th>Actual</th>\n",
       "      <th>Loss</th>\n",
       "      <th>Probability</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj thank you ! ! ! xxmaj this whole crew has xxunk through bad weather and xxunk . xxmaj pilot keeping us well informed . customerservice</td>\n",
       "      <td>negative</td>\n",
       "      <td>positive</td>\n",
       "      <td>3.16</td>\n",
       "      <td>0.04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos i listed my baggage claim number . xxmaj have them call me , my number is on the ticket . i had to use xxmaj lady xxmaj speed xxmaj stick this morning .</td>\n",
       "      <td>positive</td>\n",
       "      <td>negative</td>\n",
       "      <td>2.91</td>\n",
       "      <td>0.05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos you guys are killing me . xxurl</td>\n",
       "      <td>positive</td>\n",
       "      <td>negative</td>\n",
       "      <td>2.60</td>\n",
       "      <td>0.07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxhan they did n't address the assult or drinks xxunk on her the drunk passenger had to have a wheelchair to get off plane</td>\n",
       "      <td>positive</td>\n",
       "      <td>negative</td>\n",
       "      <td>2.02</td>\n",
       "      <td>0.13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos - thanks to the agent boarding xxunk , xxup atl - xxup aus . xxmaj left my wallet on the inbound . xxmaj he found it for me !</td>\n",
       "      <td>negative</td>\n",
       "      <td>positive</td>\n",
       "      <td>1.98</td>\n",
       "      <td>0.14</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "my_utils.show_top_losses_misclassified(interp, 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's show the predicted sentiment of some hand-made inputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-\t0.998\t0.002\tAwful flight\n",
      "+\t0.005\t0.995\tGreat trip, thanks to all\n",
      "-\t0.570\t0.430\tattendant was rude!\n",
      "-\t0.907\t0.093\tI thought I was going to have some great vacations, but everything was ruined by the delay and the rude service. Way, to go @VirginAmerica!\n",
      "+\t0.480\t0.520\tfirst time with you, pretty fair :-) Food was ok\n"
     ]
    }
   ],
   "source": [
    "examples = [\n",
    "    \"Awful flight\",\n",
    "    \"Great trip, thanks to all\",\n",
    "    \"attendant was rude!\",\n",
    "    \"I thought I was going to have some great vacations, but everything was ruined by the delay and the rude service. Way, to go @VirginAmerica!\",\n",
    "    \"first time with you, pretty fair :-) Food was ok\"\n",
    "]\n",
    "\n",
    "for example in examples:\n",
    "    pred = learn.predict(example)\n",
    "    label = pred[1].item()\n",
    "    probs = pred[2].tolist()\n",
    "    print('%s\\t%.3f\\t%.3f\\t%s' % ('-+'[label], probs[0], probs[1], example))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see which words of our examples had the greatest contribution in the classification decision:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span style=\"font-family: monospace;\"><span title=\"0.040\" style=\"background-color: rgba(254, 238, 230, 0.5);\">xxbos</span> <span title=\"0.118\" style=\"background-color: rgba(254, 225, 211, 0.5);\">xxmaj</span> <span title=\"1.000\" style=\"background-color: rgba(103, 0, 12, 0.5);\">awful</span> <span title=\"0.348\" style=\"background-color: rgba(252, 154, 123, 0.5);\">flight</span></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<span style=\"font-family: monospace;\"><span title=\"0.036\" style=\"background-color: rgba(254, 239, 231, 0.5);\">xxbos</span> <span title=\"0.132\" style=\"background-color: rgba(253, 222, 208, 0.5);\">xxmaj</span> <span title=\"0.849\" style=\"background-color: rgba(172, 16, 22, 0.5);\">great</span> <span title=\"0.830\" style=\"background-color: rgba(178, 18, 23, 0.5);\">trip</span> <span title=\"0.326\" style=\"background-color: rgba(252, 162, 132, 0.5);\">,</span> <span title=\"1.000\" style=\"background-color: rgba(103, 0, 12, 0.5);\">thanks</span> <span title=\"0.327\" style=\"background-color: rgba(252, 162, 132, 0.5);\">to</span> <span title=\"0.528\" style=\"background-color: rgba(248, 94, 66, 0.5);\">all</span></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<span style=\"font-family: monospace;\"><span title=\"0.087\" style=\"background-color: rgba(254, 230, 219, 0.5);\">xxbos</span> <span title=\"1.000\" style=\"background-color: rgba(103, 0, 12, 0.5);\">attendant</span> <span title=\"0.361\" style=\"background-color: rgba(252, 150, 119, 0.5);\">was</span> <span title=\"0.801\" style=\"background-color: rgba(186, 20, 25, 0.5);\">rude</span> <span title=\"0.451\" style=\"background-color: rgba(251, 121, 89, 0.5);\">!</span></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<span style=\"font-family: monospace;\"><span title=\"0.023\" style=\"background-color: rgba(254, 241, 235, 0.5);\">xxbos</span> <span title=\"0.132\" style=\"background-color: rgba(253, 222, 208, 0.5);\">i</span> <span title=\"0.679\" style=\"background-color: rgba(223, 44, 37, 0.5);\">thought</span> <span title=\"0.209\" style=\"background-color: rgba(252, 199, 177, 0.5);\">i</span> <span title=\"0.183\" style=\"background-color: rgba(253, 207, 188, 0.5);\">was</span> <span title=\"0.285\" style=\"background-color: rgba(252, 176, 148, 0.5);\">going</span> <span title=\"0.143\" style=\"background-color: rgba(253, 219, 203, 0.5);\">to</span> <span title=\"0.254\" style=\"background-color: rgba(252, 185, 159, 0.5);\">have</span> <span title=\"0.377\" style=\"background-color: rgba(251, 145, 113, 0.5);\">some</span> <span title=\"0.559\" style=\"background-color: rgba(245, 83, 59, 0.5);\">great</span> <span title=\"1.000\" style=\"background-color: rgba(103, 0, 12, 0.5);\">vacations</span> <span title=\"0.194\" style=\"background-color: rgba(252, 204, 183, 0.5);\">,</span> <span title=\"0.248\" style=\"background-color: rgba(252, 187, 162, 0.5);\">but</span> <span title=\"0.636\" style=\"background-color: rgba(236, 56, 42, 0.5);\">everything</span> <span title=\"0.268\" style=\"background-color: rgba(252, 181, 154, 0.5);\">was</span> <span title=\"0.910\" style=\"background-color: rgba(145, 10, 18, 0.5);\">ruined</span> <span title=\"0.260\" style=\"background-color: rgba(252, 184, 157, 0.5);\">by</span> <span title=\"0.149\" style=\"background-color: rgba(253, 216, 200, 0.5);\">the</span> <span title=\"0.309\" style=\"background-color: rgba(252, 167, 138, 0.5);\">delay</span> <span title=\"0.193\" style=\"background-color: rgba(252, 204, 183, 0.5);\">and</span> <span title=\"0.255\" style=\"background-color: rgba(252, 185, 159, 0.5);\">the</span> <span title=\"0.901\" style=\"background-color: rgba(151, 11, 19, 0.5);\">rude</span> <span title=\"0.393\" style=\"background-color: rgba(251, 140, 108, 0.5);\">service</span> <span title=\"0.201\" style=\"background-color: rgba(252, 201, 180, 0.5);\">.</span> <span title=\"0.161\" style=\"background-color: rgba(253, 213, 195, 0.5);\">xxmaj</span> <span title=\"0.575\" style=\"background-color: rgba(243, 77, 55, 0.5);\">way</span> <span title=\"0.242\" style=\"background-color: rgba(252, 189, 163, 0.5);\">,</span> <span title=\"0.292\" style=\"background-color: rgba(252, 173, 145, 0.5);\">to</span> <span title=\"0.648\" style=\"background-color: rgba(232, 52, 41, 0.5);\">go</span> <span title=\"0.494\" style=\"background-color: rgba(251, 107, 75, 0.5);\">xxunk</span> <span title=\"0.435\" style=\"background-color: rgba(251, 126, 94, 0.5);\">!</span></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<span style=\"font-family: monospace;\"><span title=\"0.090\" style=\"background-color: rgba(254, 229, 218, 0.5);\">xxbos</span> <span title=\"0.695\" style=\"background-color: rgba(217, 38, 35, 0.5);\">first</span> <span title=\"0.693\" style=\"background-color: rgba(219, 39, 35, 0.5);\">time</span> <span title=\"0.548\" style=\"background-color: rgba(246, 87, 62, 0.5);\">with</span> <span title=\"0.486\" style=\"background-color: rgba(251, 110, 78, 0.5);\">you</span> <span title=\"0.315\" style=\"background-color: rgba(252, 166, 137, 0.5);\">,</span> <span title=\"0.907\" style=\"background-color: rgba(147, 10, 18, 0.5);\">pretty</span> <span title=\"1.000\" style=\"background-color: rgba(103, 0, 12, 0.5);\">fair</span> <span title=\"0.865\" style=\"background-color: rgba(167, 15, 21, 0.5);\">:-)</span> <span title=\"0.298\" style=\"background-color: rgba(252, 171, 142, 0.5);\">xxmaj</span> <span title=\"0.980\" style=\"background-color: rgba(112, 2, 14, 0.5);\">food</span> <span title=\"0.352\" style=\"background-color: rgba(252, 154, 123, 0.5);\">was</span> <span title=\"0.699\" style=\"background-color: rgba(217, 38, 35, 0.5);\">ok</span></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.cm as cm\n",
    "\n",
    "for example in examples:\n",
    "    interp.show_intrinsic_attention(example, cmap=cm.Reds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can do the same with some random training examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span style=\"font-family: monospace;\"><span title=\"0.044\" style=\"background-color: rgba(254, 237, 229, 0.5);\">xxbos</span> <span title=\"0.150\" style=\"background-color: rgba(253, 216, 200, 0.5);\">xxmaj</span> <span title=\"1.000\" style=\"background-color: rgba(103, 0, 12, 0.5);\">thanks</span> <span title=\"0.607\" style=\"background-color: rgba(240, 65, 48, 0.5);\">!</span></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<span style=\"font-family: monospace;\"><span title=\"0.052\" style=\"background-color: rgba(254, 236, 227, 0.5);\">xxbos</span> <span title=\"0.322\" style=\"background-color: rgba(252, 163, 134, 0.5);\">they</span> <span title=\"0.716\" style=\"background-color: rgba(212, 33, 32, 0.5);\">sent</span> <span title=\"0.380\" style=\"background-color: rgba(251, 144, 112, 0.5);\">her</span> <span title=\"0.276\" style=\"background-color: rgba(252, 178, 151, 0.5);\">a</span> <span title=\"0.864\" style=\"background-color: rgba(167, 15, 21, 0.5);\">letter</span> <span title=\"0.250\" style=\"background-color: rgba(252, 186, 160, 0.5);\">and</span> <span title=\"0.476\" style=\"background-color: rgba(251, 114, 82, 0.5);\">said</span> <span title=\"0.329\" style=\"background-color: rgba(252, 160, 131, 0.5);\">they</span> <span title=\"0.319\" style=\"background-color: rgba(252, 164, 135, 0.5);\">would</span> <span title=\"0.774\" style=\"background-color: rgba(194, 22, 27, 0.5);\">talk</span> <span title=\"0.231\" style=\"background-color: rgba(252, 192, 168, 0.5);\">to</span> <span title=\"0.308\" style=\"background-color: rgba(252, 168, 139, 0.5);\">the</span> <span title=\"1.000\" style=\"background-color: rgba(103, 0, 12, 0.5);\">stewardess</span> <span title=\"0.392\" style=\"background-color: rgba(251, 140, 108, 0.5);\">about</span> <span title=\"0.903\" style=\"background-color: rgba(149, 11, 19, 0.5);\">serving</span> <span title=\"0.765\" style=\"background-color: rgba(198, 22, 28, 0.5);\">drunks</span> <span title=\"0.552\" style=\"background-color: rgba(245, 86, 61, 0.5);\">drinks</span> <span title=\"0.575\" style=\"background-color: rgba(243, 77, 55, 0.5);\">how</span> <span title=\"0.564\" style=\"background-color: rgba(244, 81, 58, 0.5);\">does</span> <span title=\"0.417\" style=\"background-color: rgba(251, 132, 100, 0.5);\">that</span> <span title=\"0.699\" style=\"background-color: rgba(217, 38, 35, 0.5);\">help</span></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<span style=\"font-family: monospace;\"><span title=\"0.080\" style=\"background-color: rgba(254, 231, 221, 0.5);\">xxbos</span> <span title=\"0.460\" style=\"background-color: rgba(251, 119, 87, 0.5);\">have</span> <span title=\"0.566\" style=\"background-color: rgba(244, 81, 58, 0.5);\">been</span> <span title=\"0.915\" style=\"background-color: rgba(143, 9, 18, 0.5);\">waiting</span> <span title=\"0.313\" style=\"background-color: rgba(252, 166, 137, 0.5);\">for</span> <span title=\"0.250\" style=\"background-color: rgba(252, 186, 160, 0.5);\">a</span> <span title=\"0.540\" style=\"background-color: rgba(247, 90, 64, 0.5);\">call</span> <span title=\"0.300\" style=\"background-color: rgba(252, 171, 142, 0.5);\">back</span> <span title=\"0.306\" style=\"background-color: rgba(252, 168, 139, 0.5);\">for</span> <span title=\"0.450\" style=\"background-color: rgba(251, 121, 89, 0.5);\">over</span> <span title=\"0.371\" style=\"background-color: rgba(252, 148, 116, 0.5);\">two</span> <span title=\"0.546\" style=\"background-color: rgba(246, 89, 63, 0.5);\">hours</span> <span title=\"0.905\" style=\"background-color: rgba(149, 11, 19, 0.5);\">jan</span> <span title=\"0.278\" style=\"background-color: rgba(252, 177, 150, 0.5);\">to</span> <span title=\"0.835\" style=\"background-color: rgba(177, 17, 23, 0.5);\">die</span> <span title=\"0.632\" style=\"background-color: rgba(237, 57, 43, 0.5);\">tomorrow</span> <span title=\"0.392\" style=\"background-color: rgba(251, 140, 108, 0.5);\">at</span> <span title=\"0.334\" style=\"background-color: rgba(252, 159, 129, 0.5);\">6</span> <span title=\"0.383\" style=\"background-color: rgba(251, 143, 111, 0.5);\">am</span> <span title=\"0.190\" style=\"background-color: rgba(252, 205, 185, 0.5);\">xxmaj</span> <span title=\"0.665\" style=\"background-color: rgba(227, 47, 39, 0.5);\">cancelled</span> <span title=\"0.165\" style=\"background-color: rgba(253, 212, 194, 0.5);\">xxmaj</span> <span title=\"0.590\" style=\"background-color: rgba(242, 72, 52, 0.5);\">flightled</span> <span title=\"0.734\" style=\"background-color: rgba(207, 28, 31, 0.5);\">what</span> <span title=\"0.602\" style=\"background-color: rgba(241, 66, 49, 0.5);\">are</span> <span title=\"0.493\" style=\"background-color: rgba(251, 107, 75, 0.5);\">my</span> <span title=\"1.000\" style=\"background-color: rgba(103, 0, 12, 0.5);\">options</span> <span title=\"0.648\" style=\"background-color: rgba(232, 52, 41, 0.5);\">?</span></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<span style=\"font-family: monospace;\"><span title=\"0.027\" style=\"background-color: rgba(254, 241, 234, 0.5);\">xxbos</span> <span title=\"0.119\" style=\"background-color: rgba(254, 225, 211, 0.5);\">xxmaj</span> <span title=\"0.678\" style=\"background-color: rgba(223, 44, 37, 0.5);\">way</span> <span title=\"0.249\" style=\"background-color: rgba(252, 187, 162, 0.5);\">to</span> <span title=\"0.777\" style=\"background-color: rgba(194, 22, 27, 0.5);\">suck</span> <span title=\"0.244\" style=\"background-color: rgba(252, 189, 163, 0.5);\">at</span> <span title=\"0.283\" style=\"background-color: rgba(252, 176, 148, 0.5);\">customer</span> <span title=\"0.204\" style=\"background-color: rgba(252, 200, 179, 0.5);\">service</span> <span title=\"0.140\" style=\"background-color: rgba(253, 220, 205, 0.5);\">,</span> <span title=\"0.164\" style=\"background-color: rgba(253, 213, 195, 0.5);\">xxmaj</span> <span title=\"0.706\" style=\"background-color: rgba(215, 36, 34, 0.5);\">dallas</span> <span title=\"0.310\" style=\"background-color: rgba(252, 167, 138, 0.5);\">is</span> <span title=\"1.000\" style=\"background-color: rgba(103, 0, 12, 0.5);\">trapped</span> <span title=\"0.262\" style=\"background-color: rgba(252, 182, 156, 0.5);\">in</span> <span title=\"0.250\" style=\"background-color: rgba(252, 187, 162, 0.5);\">an</span> <span title=\"0.490\" style=\"background-color: rgba(251, 109, 77, 0.5);\">ice</span> <span title=\"0.421\" style=\"background-color: rgba(251, 131, 99, 0.5);\">storm</span> <span title=\"0.334\" style=\"background-color: rgba(252, 159, 129, 0.5);\">but</span> <span title=\"0.296\" style=\"background-color: rgba(252, 172, 144, 0.5);\">you</span> <span title=\"0.238\" style=\"background-color: rgba(252, 191, 166, 0.5);\">did</span> <span title=\"0.269\" style=\"background-color: rgba(252, 181, 154, 0.5);\">n&#x27;t</span> <span title=\"0.174\" style=\"background-color: rgba(253, 209, 191, 0.5);\">xxmaj</span> <span title=\"0.359\" style=\"background-color: rgba(252, 151, 120, 0.5);\">cancelled</span> <span title=\"0.080\" style=\"background-color: rgba(254, 231, 221, 0.5);\">xxmaj</span> <span title=\"0.175\" style=\"background-color: rgba(253, 209, 191, 0.5);\">flight</span> <span title=\"0.228\" style=\"background-color: rgba(252, 193, 169, 0.5);\">7</span> <span title=\"0.171\" style=\"background-color: rgba(253, 211, 192, 0.5);\">am</span> <span title=\"0.130\" style=\"background-color: rgba(253, 222, 208, 0.5);\">flight</span> <span title=\"0.155\" style=\"background-color: rgba(253, 215, 199, 0.5);\">&amp;</span> <span title=\"0.210\" style=\"background-color: rgba(252, 199, 177, 0.5);\">been</span> <span title=\"0.150\" style=\"background-color: rgba(253, 216, 200, 0.5);\">on</span> <span title=\"0.360\" style=\"background-color: rgba(252, 150, 119, 0.5);\">hold</span> <span title=\"0.319\" style=\"background-color: rgba(252, 164, 135, 0.5);\">all</span> <span title=\"0.349\" style=\"background-color: rgba(252, 154, 123, 0.5);\">day</span> <span title=\"0.238\" style=\"background-color: rgba(252, 191, 166, 0.5);\">.</span></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<span style=\"font-family: monospace;\"><span title=\"0.087\" style=\"background-color: rgba(254, 230, 219, 0.5);\">xxbos</span> <span title=\"1.000\" style=\"background-color: rgba(103, 0, 12, 0.5);\">platinum</span> <span title=\"0.511\" style=\"background-color: rgba(250, 102, 71, 0.5);\">flyer</span> <span title=\"0.459\" style=\"background-color: rgba(251, 119, 87, 0.5);\">bought</span> <span title=\"0.457\" style=\"background-color: rgba(251, 119, 87, 0.5);\">tkt</span> <span title=\"0.128\" style=\"background-color: rgba(253, 223, 209, 0.5);\">in</span> <span title=\"0.086\" style=\"background-color: rgba(254, 231, 220, 0.5);\">xxup</span> <span title=\"0.210\" style=\"background-color: rgba(252, 199, 177, 0.5);\">aa</span> <span title=\"0.072\" style=\"background-color: rgba(254, 233, 223, 0.5);\">.</span> <span title=\"0.074\" style=\"background-color: rgba(254, 233, 223, 0.5);\">xxmaj</span> <span title=\"0.308\" style=\"background-color: rgba(252, 168, 139, 0.5);\">also</span> <span title=\"0.261\" style=\"background-color: rgba(252, 184, 157, 0.5);\">both</span> <span title=\"0.430\" style=\"background-color: rgba(251, 127, 95, 0.5);\">accounts</span> <span title=\"0.149\" style=\"background-color: rgba(253, 216, 200, 0.5);\">are</span> <span title=\"0.261\" style=\"background-color: rgba(252, 184, 157, 0.5);\">linked</span> <span title=\"0.065\" style=\"background-color: rgba(254, 234, 224, 0.5);\">.</span> <span title=\"0.043\" style=\"background-color: rgba(254, 237, 229, 0.5);\">xxmaj</span> <span title=\"0.103\" style=\"background-color: rgba(254, 227, 215, 0.5);\">so</span> <span title=\"0.070\" style=\"background-color: rgba(254, 233, 224, 0.5);\">xxup</span> <span title=\"0.137\" style=\"background-color: rgba(253, 220, 205, 0.5);\">why</span> <span title=\"0.158\" style=\"background-color: rgba(253, 214, 197, 0.5);\">ca</span> <span title=\"0.095\" style=\"background-color: rgba(254, 229, 217, 0.5);\">n&#x27;t</span> <span title=\"0.075\" style=\"background-color: rgba(254, 232, 222, 0.5);\">i</span> <span title=\"0.186\" style=\"background-color: rgba(253, 206, 186, 0.5);\">check</span> <span title=\"0.085\" style=\"background-color: rgba(254, 231, 220, 0.5);\">in</span> <span title=\"0.102\" style=\"background-color: rgba(254, 227, 215, 0.5);\">for</span> <span title=\"0.163\" style=\"background-color: rgba(253, 213, 195, 0.5);\">flight</span> <span title=\"0.329\" style=\"background-color: rgba(252, 160, 131, 0.5);\">tomorrow</span> <span title=\"0.297\" style=\"background-color: rgba(252, 172, 144, 0.5);\">?</span></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df_sample = df_labeled.sample(5)\n",
    "\n",
    "for _, row in df_sample.iterrows():\n",
    "    interp.show_intrinsic_attention(row.text, cmap=cm.Reds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Putting all together\n",
    "\n",
    "Finally, we can compare with the baseline results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = {}\n",
    "with open('data/results.pkl', 'rb') as f:\n",
    "    prev_results = pickle.load(f)\n",
    "    results.update(prev_results)\n",
    "\n",
    "results['ulmfit_wt103'] = (score_train, score_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>method</th>\n",
       "      <th>score_train</th>\n",
       "      <th>score_test</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>ulmfit_wt103</td>\n",
       "      <td>0.961227</td>\n",
       "      <td>0.940208</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>SVC_linear_10.0</td>\n",
       "      <td>0.951906</td>\n",
       "      <td>0.914982</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>nb_multi_counts</td>\n",
       "      <td>0.943385</td>\n",
       "      <td>0.903896</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>dtree</td>\n",
       "      <td>0.952051</td>\n",
       "      <td>0.880519</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>nb_gaussian_tf_idf_feature_sel</td>\n",
       "      <td>0.865107</td>\n",
       "      <td>0.811255</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>nb_gaussian_tf_idf</td>\n",
       "      <td>0.829097</td>\n",
       "      <td>0.687446</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>random_with_class_dist</td>\n",
       "      <td>0.674658</td>\n",
       "      <td>0.658874</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           method  score_train  score_test\n",
       "6                    ulmfit_wt103     0.961227    0.940208\n",
       "4                 SVC_linear_10.0     0.951906    0.914982\n",
       "1                 nb_multi_counts     0.943385    0.903896\n",
       "5                           dtree     0.952051    0.880519\n",
       "3  nb_gaussian_tf_idf_feature_sel     0.865107    0.811255\n",
       "2              nb_gaussian_tf_idf     0.829097    0.687446\n",
       "0          random_with_class_dist     0.674658    0.658874"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_df = pd.DataFrame([(k, *v) for k, v in results.items()], columns=['method', 'score_train', 'score_test'])\n",
    "results_df = results_df.sort_values(by='score_test', ascending=False)\n",
    "results_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7f12d82cb5f8>"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsgAAAI8CAYAAADsqF15AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3XnYXeO9//H3V2IKERLhUGNrJiIkMbRVqmIqOvyqKC0d1FE60ob2oNOhR6ulpT3aqpppndOjlUNKKVoqOGpoiKFKpCXSiiBK+P7+uNdO7uw8jzyRZ8jwfl1Xrjx77bXXvvfae6/9Wfe0IjORJEmSVCzT1wWQJEmSFiUGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqWJAlqRFTESsFxHPRUS/vi5Lb4qIGyLiowu5jfsiYpduKtLref7zIuJrffX8krqHAVlayjWh5B8RsXxfl2VpFRGPRsQ7Wrcz87HMXDkzX+mDsmREbNTbz9tdMnPLzLxhQR4TEbtExOQeKpKkxZABWVqKRcQGwFuBBPbr5efu35vPJ0lSVxmQpaXbB4FbgfOAD9V3RMSKEfGtiPhLREyPiJsjYsXmvrdExO8j4pmIeDwiDmuWz9VEHhGHRcTN1e2MiE9ExIPAg82yM5ptPBsRd0TEW6v1+0XECRHxcETMaO5fNyLOiohvtZX3lxHx6Y5eZETsFBETmtcxISJ2qu67ISK+GhG/a55jfESs3sl2Vo+IXzWv++8RcVNELNPct3ZEXBERUyPizxHxyepxJ0fE5RFxfvMc90XEyOa+C4D1gF823So+HxEbNPuqf1XGrzX7/LnmtQ6JiIua/TahOdlpPd9mEfHrpowPRMQB1X3nNfvvqqYsf4iINzX33dis9sfmed7fwT54U0T8JiKmRcTTTRlWre5/NCKOjYi7m/19WUSs0Ny3WrP/pjatFr+KiHU6eI7lm7IPq5atEREzI2LofN6H2bXxETE6Im5v9tGTEXF6B8+1EvC/wNrNa36ueS9HR8QtzXP8NSK+FxHLNY+JiPh2RDzVvMa7I2KrDrY9MCKuj4gzIyLm+UBJWmQZkKWl2weBi5p/e0TEmtV93wS2A3YCBgOfB16NiPUogeK7wFBgG+CuBXjOdwHbA1s0tyc02xgMXAz8rBWogM8CBwF7A6sAHwZeAH4KHFSFotWB3YBL2p8sIgYDVwFnAkOA04GrImJItdrBwOHAGsBywLGdlP1zwOTmda8JnABkU45fAn8E3tCU5dMRsUf12P2AS4FVgSuB7wFk5qHAY8C+TbeK/+jkuQ8EDm22/ybgFuAnzX6bCJzUvN6VgF9T9uUazf47OyK2rLZ1EPBlYDXgIeDrTVl2bu4f3pTlsg7KEcApwNrA5sC6wMlt6xwA7AlsCGwNHNYsX6Yp8/qUk4KZrf1Qy8x/UvbVIW1lvjYzp9LJ+9BBWc8AzsjMVSj77PIOnut5YC9gSvOaV87MKcArwGeA1YEdKe/pUc3DxgA7A5tQ3s/3A9Pm2knl83Ud8LvM/GRmdlQ+SYsoA7K0lIqIt1CCyuWZeQfwMCUo0gS+DwOfyswnMvOVzPx9E1w+QAkql2Tmy5k5LTMXJCCfkpl/z8yZAJl5YbONWZn5LWB5YNNm3Y8CX8rMB7L4Y7PubcB0SmiBEh5vyMwnO3i+fYAHM/OC5jkuAe4H9q3W+UlmTmrKdDklsHfkZWAtYP3mtd/UBJ9RwNDM/EpmvpSZjwA/bMrVcnNmjmv6FV8ADF+AfdYq48OZOZ1ygvJwZl6bmbOAnwEjmvXeCTyamT9pXu+dwBXA/6u29V+ZeVvz2Ite4/XOIzMfysxfZ+Y/m7B6OvC2ttXOzMwpmfl3yonDNs1jp2XmFZn5QmbOoATz9se2/BQ4uHUSRDk5uKD5u7P3od3LwEYRsXpmPpeZty7A67wjM29t9uGjwH9WZX0ZGAhsBkRmTszMv1YPXxv4LfCzzPxSV59T0qLDgCwtvT4EjM/Mp5vbFzOnm8XqwAqU0Nxu3U6Wd9Xj9Y2I+FxETGyaqp8BBjXPP7/n+ilzahgPYU54arc28Je2ZX+h1MS2/K36+wVg5U62dRqlxnV8RDwSEWOb5etTmuifaf2j1GrWNfLtz7FCLFg/7Dr8z+zgdqvM6wPbt5XlA8C/vEZZOnu982i6OlwaEU9ExLPAhcx5v15z+xExICL+M0q3nWeBG4FVo4PZOjLzD8DzwNsiYjNgI0rNO3T+PrT7CKWW9/6mG8o7F+B1btJ04/hbU9Z/b73OzPwNpeb7LODJiDgnIlapHr4PsCLwg64+n6RFiwFZWgpF6Ut8ACV8/C0i/kZpTh4eEcOBp4EXKc3S7R7vZDmUQDOguv0vHawzu6YvSn/jLzRlWS0zV6XUDLf6a77Wc10I7N+Ud3PgF52sN4USGmvrAU90sn6nMnNGZn4uM99IqYH+bETs1pTzz5m5avVvYGbu3dVNL2hZXsPjwG/byrJyZv5rN23/FEp5t266LhzCnPdrfj5HaR3Yvnlsq0tHZ49vnQQdCvw8M1+E13wf5pKZD2bmQZSuJt8Aft50QZln1Q6WfZ/S0rBxU9YT6nJm5pmZuR2wJSWEH1c99ofA1cC4Tp5P0iLOgCwtnd5F6WO5BaX5extKyLwJ+GBmvgqcC5zeDFjqFxE7RpkK7iLgHRFxQET0jzJYrNVEfxfwnqamcCNKDd5rGQjMAqYC/SPiREpf45YfAV+NiI2bgVFbt/oOZ+ZkSv/lC4ArWl02OjAO2CQiDm7K+/7mdf+qqzurJSLeGREbNQOunqXsw1eA24BnI+ILUQY39ouIrSJiVBc3/STwxgUtTyd+RXm9h0bEss2/URGxeTeVZSDwHPBMRLyBuYPh/Ayk1HY/0/QNP2k+618AvJsSks9vLXyN92EuEXFIRAxtPs/PNIs7mjrvSWBIRAxqK+uzwHNNDfbsE4xmf24fEctSTgpf7GC7RwMPAL9qTkglLUYMyNLS6UOUPq2PZebfWv8ozcYfaJr+jwXuoYTQv1Nq4JbJzMcog+Y+1yy/izn9ab8NvEQJHD+lhOnXcg2lP+0kSreHF5m7C8bplD7B4ylh5ceUpuuWnwLD6Lx7BZk5jdIv93OUgVSfB95ZdS1ZEBsD11IC4i3A2Zl5Q9OveF/KicafKTXwP6J0F+mKU4AvNV0iOhsg2CVN394xlP7PUyjdHb5B6dvdFScDP23KckAH938Z2JZS038V8F8LULzvUN6/pymzp1z9Wis3J0F3Ump4b6ru6vB96GATewL3RcRzlAF7B7Zqodue537KAM9Hmte9NuXzfzAwg1IjXA9YXKVZ9g/K53YaZVBrvc0EjqB8nv+nGngqaTEQDqyVtLiKiJ0pXS02aGoJtYSJiHMpM0w42E1Sr3GifkmLpaZ5+1PAjwzHS6Yoczu/hzkzdEhSr5hvF4uIODfKZOj3dnJ/RJkE/aEok6Vv2/3FlKQ5mv60z1Cm+vpOHxdHPSAivgrcC5yWmX/u6/JIWrrMt4tF04T5HHB+ZnZ0paC9gWMofRK3p0zKvn0PlFWSJEnqcfOtQc7MGykDcTqzPyU8ZzMJ+6oRsVZ3FVCSJEnqTd0xi8UbmHvU+WTmnoBfkiRJWmx0xyC9jiZ477DfRkQcQZn2hpVWWmm7zTbbrBueXpIkSercHXfc8XRmDu3q+t0RkCdTLgfbsg5l7s15ZOY5wDkAI0eOzNtvv70bnl6SJEnqXET8ZUHW744uFlcCH2xms9gBmJ6Zf+2G7UqSJEm9br41yBFxCbALsHpETKZcGnRZgMz8AeUyrnsDDwEvAIf3VGElSZKknjbfgJyZB83n/gQ+0W0lkiRJkvqQV9KTJEnqQS+//DKTJ0/mxRdf7OuiLPFWWGEF1llnHZZddtmF2o4BWZIkqQdNnjyZgQMHssEGGxDR0eRf6g6ZybRp05g8eTIbbrjhQm2rOwbpSZIkqRMvvvgiQ4YMMRz3sIhgyJAh3VJTb0CWJEnqYYbj3tFd+9mALEmSJFXsgyxJktSLNhh7Vbdu79FT9+nW7XW3Z555hosvvpijjjpqgR+79957c/HFF7Pqqqv2QMk6Zw2yJEmSumzWrFkLtP4zzzzD2Wef3eF9r7zyyms+dty4cb0ejsGALEmStMR7/vnn2WeffRg+fDhbbbUVl112GRMmTGCnnXZi+PDhjB49mhkzZvDiiy9y+OGHM2zYMEaMGMH1118PwHnnncf73vc+9t13X8aMGQPAaaedxqhRo9h666056aSTOn3usWPH8vDDD7PNNttw3HHHccMNN7Drrrty8MEHM2zYMADe9a53sd1227HllltyzjnnzH7sBhtswNNPP82jjz7K5ptvzsc+9jG23HJLxowZw8yZM3tsf9nFQpIkaQl39dVXs/baa3PVVaV7x/Tp0xkxYgSXXXYZo0aN4tlnn2XFFVfkjDPOAOCee+7h/vvvZ8yYMUyaNAmAW265hbvvvpvBgwczfvx4HnzwQW677TYyk/32248bb7yRnXfeeZ7nPvXUU7n33nu56667ALjhhhu47bbbuPfee2dPx3buuecyePBgZs6cyahRo3jve9/LkCFD5trOgw8+yCWXXMIPf/hDDjjgAK644goOOeSQHtlf1iBLkiQt4YYNG8a1117LF77wBW666SYee+wx1lprLUaNGgXAKqusQv/+/bn55ps59NBDAdhss81Yf/31Zwfk3XffncGDBwMwfvx4xo8fz4gRI9h22225//77efDBB7tcntGjR881V/GZZ57J8OHD2WGHHXj88cc73NaGG27INttsA8B2223Ho48++rr2RVdYgyxJkrSE22STTbjjjjsYN24cxx9/PGPGjOlwSrTM7HQbK6200lzrHX/88Xz84x9/XeWpt3XDDTdw7bXXcssttzBgwAB22WWXDucyXn755Wf/3a9fvx7tYmENsiRJ0hJuypQpDBgwgEMOOYRjjz2WW2+9lSlTpjBhwgQAZsyYwaxZs9h555256KKLAJg0aRKPPfYYm2666Tzb22OPPTj33HN57rnnAHjiiSd46qmnOnzugQMHMmPGjE7LNn36dFZbbTUGDBjA/fffz6233rqwL3ehWYMsSZLUi/piWrZ77rmH4447jmWWWYZll12W73//+2QmxxxzDDNnzmTFFVfk2muv5aijjuLII49k2LBh9O/fn/POO2+umtuWMWPGMHHiRHbccUcAVl55ZS688ELWWGONedYdMmQIb37zm9lqq63Ya6+92GefuV//nnvuyQ9+8AO23nprNt10U3bYYYee2QkLIF6rKr0njRw5Mm+//fY+eW5JkqTeMnHiRDbffPO+LsZSo6P9HRF3ZObIrm7DLhaSJElSxS4WkiRJWmjTpk1jt912m2f5ddddN8+UbYs6A7IkSZIW2pAhQ2bPdby4s4uFJEmSVDEgS5IkSRUDsiRJklQxIEuSJKnHPPPMM5x99tmv+/Hf+c53eOGFF7qxRPPnID1JkqTedPKgbt7e9O7d3nzMmjWL/v27HiFbAfmoo456Xc/3ne98h0MOOYQBAwa8rse/HtYgS5IkLeGef/559tlnH4YPH85WW23FZZddxoQJE9hpp50YPnw4o0ePZsaMGbz44oscfvjhDBs2jBEjRnD99dcDcN555/G+972PfffdlzFjxgBw2mmnMWrUKLbeemtOOumkTp977NixPPzww2yzzTYcd9xxnT62ozKeeeaZTJkyhV133ZVdd921h/fSHNYgS5IkLeGuvvpq1l57ba666ioApk+fzogRI7jssssYNWoUzz77LCuuuCJnnHEGUC5Nff/99zNmzBgmTZoEwC233MLdd9/N4MGDGT9+PA8++CC33XYbmcl+++3HjTfeyM477zzPc5966qnce++9s6eA6+yxU6dOnaeMgwYN4vTTT+f6669n9dVX741dBViDLEmStMQbNmwY1157LV/4whe46aabeOyxx1hrrbUYNWoUAKussgr9+/fn5ptv5tBDDwVgs802Y/31158dkHfffXcGDx4MlJA7fvx4RowYwbbbbsv999/Pgw8+2KWydPbY9jIOGtTNXVEWgDXIkiRJS7hNNtmEO+64g3HjxnH88cczZswYImKe9TKz022stNJKc613/PHH8/GPf3yBy/Jaj20v44knnrjA2+8O1iBLkiQt4aZMmcKAAQM45JBDOPbYY7n11luZMmUKEyZMAGDGjBnMmjWLnXfemYsuugiASZMm8dhjj7HpppvOs7099tiDc889l+eeew6AJ554gqeeeqrD5x44cCAzZsyY72Pby3jnnXd2+PjeYA2yJEnSEu6ee+7huOOOY5lllmHZZZfl+9//PpnJMcccw8yZM1lxxRW59tprOeqoozjyyCMZNmwY/fv357zzzmP55ZefZ3tjxoxh4sSJ7LjjjgCsvPLKXHjhhayxxhrzrDtkyBDe/OY3s9VWW7HXXntx2mmndfjYhx56aJ4yAhxxxBHstdderLXWWrMHDfa0eK2q9J40cuTIvP322/vkuSVJknrLxIkT2Xzzzfu6GEuNjvZ3RNyRmSO7uo0ltgZ5g7FX9ej2Hz11nx7dviRJkvrGEhuQJUmS1HumTZvGbrvtNs/y6667jiFDhvRBiV4/A7IkSZIW2pAhQ2bPdby4cxYLSZIkqWJAliRJ6mF9NSnC0qa79rMBWZIkqQetsMIKTJs2zZDcwzKTadOmscIKKyz0tuyDLEmS1IPWWWcdJk+ezNSpU/u6KEu8FVZYgXXWWWeht2NAlqRu5BSTktotu+yybLjhhn1dDC0AA7J6hCFBkiQtruyDLEmSJFUMyJIkSVLFgCxJkiRVDMiSJElSxYAsSZIkVQzIkiRJUsVp3qQljFPsSZK0cKxBliRJkioGZEmSJKliF4vX6+RBPbz96T27fUmSJHXIGmRJkiSpYkCWJEmSKgZkSZIkqWJAliRJkioGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqWJAliRJkioGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqWJAliRJkioGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqdK/rwsgvS4nD+rBbU/vuW1LkqRFnjXIkiRJUsWALEmSJFUMyJIkSVLFgCxJkiRVDMiSJElSxYAsSZIkVQzIkiRJUsWALEmSJFW6dKGQiNgTOAPoB/woM09tu3994FxgKPB34JDMnNzNZZW0KOjJi7SAF2qRJPW5+dYgR0Q/4CxgL2AL4KCI2KJttW8C52fm1sBXgFO6u6CSJElSb+hKF4vRwEOZ+UhmvgRcCuzfts4WwHXN39d3cL8kSZK0WOhKQH4D8Hh1e3KzrPZH4L3N3+8GBkbEkIUvniRJktS7uhKQo4Nl2Xb7WOBtEfF/wNuAJ4BZ82wo4oiIuD0ibp86deoCF1aSJEnqaV0JyJOBdavb6wBT6hUyc0pmviczRwBfbJbNM9ImM8/JzJGZOXLo0KELUWxJkiSpZ3QlIE8ANo6IDSNiOeBA4Mp6hYhYPSJa2zqeMqOFJEmStNiZb0DOzFnA0cA1wETg8sy8LyK+EhH7NavtAjwQEZOANYGv91B5JUmSpB7VpXmQM3McMK5t2YnV3z8Hft69RZMkSZJ6n1fSkyRJkioGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqWJAliRJkioGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqdK/rwsgSVoAJw/q4e1P79ntS9JiwBpkSZIkqWJAliRJkioGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqWJAliRJkioGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKv37ugCSJHWXDcZe1aPbf/TUfXp0+5IWDdYgS5IkSRUDsiRJklQxIEuSJEkVA7IkSZJUMSBLkiRJFQOyJEmSVDEgS5IkSRUDsiRJklQxIEuSJEkVA7IkSZJUMSBLkiRJFQOyJEmSVDEgS5IkSRUDsiRJklQxIEuSJEkVA7IkSZJUMSBLkiRJFQOyJEmSVDEgS5IkSRUDsiRJklQxIEuSJEkVA7IkSZJUMSBLkiRJFQOyJEmSVDEgS5IkSRUDsiRJklQxIEuSJEkVA7IkSZJUMSBLkiRJFQOyJEmSVOnf1wWQJEmLvw3GXtWj23/01H16dPtSzRpkSZIkqWJAliRJkioGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqWJAliRJkioGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqWJAliRJkioGZEmSJKnSv68LIEnSYuPkQT247ek9t21JC8QaZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqdKlgBwRe0bEAxHxUESM7eD+9SLi+oj4v4i4OyL27v6iSpIkST1vvgE5IvoBZwF7AVsAB0XEFm2rfQm4PDNHAAcCZ3d3QSVJkqTe0JUa5NHAQ5n5SGa+BFwK7N+2TgKrNH8PAqZ0XxElSZKk3tOVC4W8AXi8uj0Z2L5tnZOB8RFxDLAS8I5uKZ0kSZLUy7pSgxwdLMu22wcB52XmOsDewAURMc+2I+KIiLg9Im6fOnXqgpdWkiRJ6mFdCciTgXWr2+swbxeKjwCXA2TmLcAKwOrtG8rMczJzZGaOHDp06OsrsSRJktSDuhKQJwAbR8SGEbEcZRDelW3rPAbsBhARm1MCslXEkiRJWuzMNyBn5izgaOAaYCJltor7IuIrEbFfs9rngI9FxB+BS4DDMrO9G4YkSZK0yOvKID0ycxwwrm3ZidXffwLe3L1FkyRJknqfV9KTJEmSKgZkSZIkqWJAliRJkioGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqWJAliRJkioGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqWJAliRJkioGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqWJAliRJkioGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqWJAliRJkioGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqWJAliRJkioGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqWJAliRJkioGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqWJAliRJkioGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqWJAliRJkioGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqWJAliRJkioGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqWJAliRJkioGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqWJAliRJkioGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqWJAliRJkioGZEmSJKliQJYkSZIqBmRJkiSp0qWAHBF7RsQDEfFQRIzt4P5vR8Rdzb9JEfFM9xdVkiRJ6nn957dCRPQDzgJ2ByYDEyLiysz8U2udzPxMtf4xwIgeKKskSZLU47pSgzwaeCgzH8nMl4BLgf1fY/2DgEu6o3CSJElSb+tKQH4D8Hh1e3KzbB4RsT6wIfCbhS+aJEmS1Pu6EpCjg2XZyboHAj/PzFc63FDEERFxe0TcPnXq1K6WUZIkSeo1XQnIk4F1q9vrAFM6WfdAXqN7RWaek5kjM3Pk0KFDu15KSZIkqZd0JSBPADaOiA0jYjlKCL6yfaWI2BRYDbile4soSZIk9Z75BuTMnAUcDVwDTAQuz8z7IuIrEbFftepBwKWZ2Vn3C0mSJGmRN99p3gAycxwwrm3ZiW23T+6+YkmSJEl9wyvpSZIkSRUDsiRJklQxIEuSJEkVA7IkSZJUMSBLkiRJFQOyJEmSVOnSNG+SJEl96uRBPbz96T27fS1WrEGWJEmSKgZkSZIkqWJAliRJkioGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqWJAliRJkioGZEmSJKnipaYlSZIWcxuMvapHt//oqfv06PYXNdYgS5IkSRUDsiRJklQxIEuSJEkVA7IkSZJUMSBLkiRJFQOyJEmSVDEgS5IkSRUDsiRJklQxIEuSJEkVA7IkSZJUMSBLkiRJFQOyJEmSVOnf1wWQJEnSIu7kQT28/ek9u/0FZA2yJEmSVDEgS5IkSRUDsiRJklQxIEuSJEkVA7IkSZJUMSBLkiRJFQOyJEmSVDEgS5IkSRUDsiRJklQxIEuSJEkVA7IkSZJUMSBLkiRJFQOyJEmSVDEgS5IkSRUDsiRJklQxIEuSJEkVA7IkSZJUMSBLkiRJFQOyJEmSVDEgS5IkSRUDsiRJklQxIEuSJEkVA7IkSZJUMSBLkiRJFQOyJEmSVDEgS5IkSRUDsiRJklQxIEuSJEkVA7IkSZJUMSBLkiRJFQOyJEmSVDEgS5IkSRUDsiRJklQxIEuSJEkVA7IkSZJUMSBLkiRJFQOyJEmSVDEgS5IkSRUDsiRJklQxIEuSJEkVA7IkSZJUMSBLkiRJFQOyJEmSVDEgS5IkSRUDsiRJklQxIEuSJEkVA7IkSZJU6VJAjog9I+KBiHgoIsZ2ss4BEfGniLgvIi7u3mJKkiRJvaP//FaIiH7AWcDuwGRgQkRcmZl/qtbZGDgeeHNm/iMi1uipAkuSJEk9qSs1yKOBhzLzkcx8CbgU2L9tnY8BZ2XmPwAy86nuLaYkSZLUO7oSkN8APF7dntwsq20CbBIRv4uIWyNiz+4qoCRJktSb5tvFAogOlmUH29kY2AVYB7gpIrbKzGfm2lDEEcARAOutt94CF1aSJEnqaV2pQZ4MrFvdXgeY0sE6/5OZL2fmn4EHKIF5Lpl5TmaOzMyRQ4cOfb1lliRJknpMVwLyBGDjiNgwIpYDDgSubFvnF8CuABGxOqXLxSPdWVBJkiSpN8w3IGfmLOBo4BpgInB5Zt4XEV+JiP2a1a4BpkXEn4DrgeMyc1pPFVqSJEnqKV3pg0xmjgPGtS1iyk4SAAAgAElEQVQ7sfo7gc82/yRJkqTFllfSkyRJkioGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqWJAliRJkioGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqWJAliRJkioGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqWJAliRJkioGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqWJAliRJkioGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqWJAliRJkioGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqWJAliRJkioGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqWJAliRJkioGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqWJAliRJkioGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqWJAliRJkioGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqWJAliRJkioGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqWJAliRJkioGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqWJAliRJkioGZEmSJKliQJYkSZIqBmRJkiSpYkCWJEmSKgZkSZIkqdKlgBwRe0bEAxHxUESM7eD+wyJiakTc1fz7aPcXVZIkSep5/ee3QkT0A84CdgcmAxMi4srM/FPbqpdl5tE9UEZJkiSp13SlBnk08FBmPpKZLwGXAvv3bLEkSZKkvtGVgPwG4PHq9uRmWbv3RsTdEfHziFi3W0onSZIk9bLIzNdeIeJ9wB6Z+dHm9qHA6Mw8plpnCPBcZv4zIo4EDsjMt3ewrSOAI5qbmwIPdM/L6BOrA0/3dSGWYu7/vuO+71vu/77l/u877vu+tbjv//Uzc2hXV+5KQN4RODkz92huHw+Qmad0sn4/4O+ZOajLRV4MRcTtmTmyr8uxtHL/9x33fd9y//ct93/fcd/3raVt/3eli8UEYOOI2DAilgMOBK6sV4iItaqb+wETu6+IkiRJUu+Z7ywWmTkrIo4GrgH6Aedm5n0R8RXg9sy8EvhkROwHzAL+DhzWg2WWJEmSesx8AzJAZo4DxrUtO7H6+3jg+O4t2iLvnL4uwFLO/d933Pd9y/3ft9z/fcd937eWqv0/3z7IkiRJ0tLES01LkiRJFQOyJEmSVDEgS1psRET0dRkWRRHhsVw9KiKGRMRKfV2ORU1EbBYRHV08TYs5D6qLmIjYJCIOiojN+7osmltEbB4Rq1S3DWu9JIp+WQ2acP8XEbFMZr7a/L1ctdz9o24REVsAXwP26euyLEoi4kPAdcB6fV0WdS4iln09lQgG5EVE8wZ+G7gC2AG4KiI26NNCCYCIWDkifgL8N/DNiPg8QDrCtddk8UpErBERx0REuP+LzHw1IlaLiPOAMyNibLPc/aOFUoWKh4DHgS0iYv0+LNIiISKWi4j+wMbA0UBGxCZ9XCx1ICK2BnZrjpNdvooeGJAXJdsAywO7ZuangDuBXcCaoEXAMGAosDnwPWDf5rLprStHqgdExIC2258Hfkt5L1ZcWrsVtL/u5qB/GfAHYCzw9Yg4sC/KpiVD02Izu2UiM18Cfg0MBnbt08L1oYjoHxHnAAdn5ixgIHAu8HlgRp8WTnOpfpunAic0FQi3RsSburqNLs2DrJ4RETtQ3rwpmTmBctVCImIMsBVwZ0RsmJl/7sNiCt4E3NvUyN0dEV8ELo+IH2fmK31ctiVOc2B7G/A88IemJSWAHTJz82q95YF/Lk21yXVoqQyltG78CTgf+AVwfW+XTUuO5vuUEbERcAjl6rm3A1sA20TENpl5V1+WsbdFxHKZ+VJE3AC8KyJuAZ4GHgbOz8y/9mkBBcyuQMjqt/lF4DlgJ2DPzHy4q9taKmtg+lpEDIiIHwCXACcAF1b3rQIcDHybcmXCL0fETn1S0KVcVXP/CPCu1vLMvBH4PXBq23rqBs2BbQDwpYj4PfB1yonk9hFxfkScGRG/AH7WrL9UhGOY3Z1ii4j474j4fETsBcyk7KOxwFmZ+d7MfDIiNgM/n+qaVstE6/MSEccCFwF/Az5H+Yz9HHgFeFvd331J15yYvtTcXI7StWLvzPwq8EXgo83JhPpIq8UxM1/NzIyIXZpa412BT1C6CW0UESs068/3uGhA7hvrAhtl5oaZ+RGg1a9y+cx8NjMPy8z/zMz/AFYBtgR/6HpSRKwaEe+PiOGtZc2XLDLz98BjEfGN6iFfBzaPiIFLU0DrKR10l5gMDKf0e/x4Zj4HvB/4X8qJ5aeAjSNiVK8WtJe1f+cjYjSlSfcC4C5KYHkS+B0wITOvadY7GzgqIpb186nXEhHLNMe5VsvECs3//YGdgT9TfoNezczngduADYC393ZZe1NErB8R58PsE9NNI+JWSjgeBxwfEVs237npwLubVi31sqZL2WGtbhUR8RngFEpr2pVNK/wVwBHAIOhaxYoBuW8E8GTVF+bzwJ6UPq7tHqM0NS9VNWW9qamhnwiMAX4WEe+OiFWbu1vdkP6V8gXcqvUw4KHMnOGJy8KrZmE4MiLeTvncHwQ8A4xqmjdvzsxLKMFwe0rN1j/6qsy9ofWdj4gPNV2vNge+QGk2PAk4NzNfAD4LDI+ISyPiTiCBsZn5ch8VXYuJqsbt7RFxAfDWpjbuQ5Q+/x8HjsrMLzW1b79qHrrGEj4G4zHgMxExqLm9JvBEZn4xM79ICVwfa+77FvBeylgi9ZKI2CciNgauyMyzgX9p7lqD0jL/a0qXoB0z88fAP4FDI+LsiDh+ftu3D3IPa2q4fgJ8OTN/1nb34Ih4JDNviYiHgQ8AdzXdLIZTmkxXAL6BetLbgVMy88yIeDelSWZl4ILMfLmphXs4Ir4JjI2IJyk1KzctTf1fu1P7fmu+J++h7NdHgb9k5gkRsTvl/bkDeCki1gUuB6YAn8jMh3q98D0oyjyzr2bmzOb2jsBIYFvgI5Qg/CNK0/cnM/OO5mTur5m5f0RsA8xYkH52Wro1J/ifo3z/TgGua2aMuQx4T2a+p1lvHeCTlNazr2bm3/uqzL2hOWkYANwcEW8FVgSejoihmTmVEor/EBHXZOb/RsQ4SvcT9YLmZO1QyuDkHza1yB+IiMOAaynjMbajZKhNm5O/z1COowMprXCv/Rz+tvecKNO+nEg581wZ2LlVoxMR36LUAp2RmU9FxBsp8yluTRl082/ATZl5bp8UfgnWnIC80IxCJiI+BWzddHchIo4E3kgJyPe0PfZNlBOZOzPzV2ihRRmE9wilxvM/mrB8OGWg2c3AacC9lPB8MiUATmwe29GgtcVOE3J3onzuxgH9KC1876V0JzkgM3/bBOh7gcOa22tRfqivysyL+qb0Wtw1J/+TgIsprRTLAn+kfP/GU0LGGOCHmXl69bglooIgIvbKzP/t5L5vU76PJ1BqjU+vujLdDdxKOVm3taYX1J+5iDgUeAvwY0olyi+AyzPzgohYqzVwMiLeD6yXmadFRP/qt/81P792segB1eCFxyhfpt2BVjNoy3cpB6K9mh/5R4AbKU33j2Xm4a1wvIQ3Y/WaKHNN/4jS7LJdddeDwPSIGNHcvo5yQHxj9bh3RMTKmflwZn6lFY476DurTsS8g4A+HBEfyMxHKf2KW9NHTQJuolyU4EXKYMghwO8z87YlMBwH8EHKgf4tlCbsayjdSE6h1JavAdD0Af034GsRcTElTN9rONZCupZy8nkGpTb5u5TP5BjKgOTngbfX4RgW/25/EbFmREyg1J63lrV3mfsm8FZgfeAs4EMRcWFEXEdpyTmhqviyu10Pa2r2V2rGWRxAmfHoX5vB3f8JHNxUOE6LiA0i4ixKd7Rbm8e3wvEy8/v8+uPejSKiX0S8E1inWbRqZt7Z/P0F4GNNjQ9NKPghMBq4IiLuBV4Cnq/fwGZdm20WUnPg2pPSQf9hYHRErNHcPYkSxHZpulM8SJmTujUAbDPK9EYz27bHkhDQeku1r1au/n9X8z4cDbwlIrbKzOmUecBfAD6VmfcCn8/Mr3WyvcVWzLk64C+Alymve3nga5l5fnPfmcCxrRPlzLwQeDdwHmUC/H/vk8JrsdBewdLRSX1mXg2MzsyPZOaBlFaJFYGnM/N/MvOEzPxrNAP6eqfkvWI1SsXUhRHxPpg39GfmE5QaylMy80rg05SBil/OzG9k5tPVb/VifcKwGPl/wNqZuS9lhoqhEbFfU3H11+b+lSkneC8C22TmTfUGuvL7YReLbtKqto+Ij1BG2y8D/L31d9On61xK/8KPVo9bFngf8Hj7G6ju1ZycPEW5UuGRlLkrf93c927KhVkmZeZZEXE05eD5NQ96r0/zQzp7dHxErEgZRTwkM09sll0E3J6Z346IL1HmqXxL84MzijJH+OPV9pa4H6Eog0Q/Q/m8TaFMaTeU0tfz4ea4cj0wPjNPWVJqztV7ImIw8A5Ki8MLWWZlaB8HsBylcuc4Snemj2bmLdX9i313iua4sj5ljMOrUa4KeB1lPNbXMvNHHX2/mq5Nv6N0ifxJ232L/X5ZXLT2dUR8mDIvfuuCXWMpF/Q6gjLLyLmU2uU/tyoYm8qIBapstAZ5IVU/2rOaM/X7KV0nnsjMA9q+OJ8Bdmya60+MiLdm5suZeXErHNtk36P+lpmvZObvKAPBdm6aYqA0aV9CmRrrSkqN/1VtPyBLUs1Jj2odyJofodUjYkCWgWdPAP8Sc+b2/gmwf5Tpkr4GjGxqAl7NzD+0wjHMvtz0EvVDFGXg05cp++FoyrSOk4G1KLXDs5pVxwL7RZkK0nCsTrX/hkTEUZS+xDsD/05pfejoRLM/sDflinDD63DcyfqLo9Uo37Nto8zzfDBwFaWLya+h45rFpmvTQZQuFcCSe8K+KOisW2m1r18CpkZEa+avK4B3Ah/OcgGbAzLzoSocx+tpiTeMLaS2L8fFlH6THwIGRplHcfnWG9M0Hf+DEsbWpDTTADbZd6fOTjKaM8/WfRdQ+hhv29z3QmbeCuwLfDMz1626x8x+fA8We4nQ/qMRESdS+jBe0Jzl/4ZycrJPs961lB+t45pNDG+aMZcWy1COBTdl5v3AOZT+xrcAb4+In0bEXygtGztm5j/7sKxaxDVBoNVi84Zm8aaZuQVwKeX4NrSTY+RM4AeZ+fmqwmeJ0eybaZQZDK6ltBhekJmfojTDvzcihnT0uKZWeWKWK+m1ujr5e9DN6m6lEbF8tF2Mpqqk+g2lC8Vnoly7YD/KYNI7msfPNbvR632vDMivQ12TGBE7RMSpzc2LgP6Z+RvKWfjBrR+0pn/yZyj9KrfKzE/UP3Z+2RZe9eVq/UAMaF+nalqcBFxNmWP36xFxYXP/I1mulEdEOA1iF8Wciw3UNe77UmZm2Ioym8u7KbO03AqsH2XO47dRZrC4K0r/7weaxy4ttfUzKLN17AqQmeOA3Sk/4ldQ5qHdJTOX6Pme1T2aSoB1IuL7wPcjYigwIiKuAb5KmW3hB8y5GEgrAPZrGmhmRUT/11vjtihr9k0/ynfqYUp3icnN3VdSmuiHtdZv2y+vNq09jgnqAe0VhBHxr8CfKLXCszXvYWTmFMpA0knA6cCGwIeyXNSr2xgAXofmTRrU1AhvB3wyIu6iHHSejdLX8kzgPyJiMqVp5ljgx5n5bZg3zGnhVV+uN1JGZP+ccuDrzNOUKbTupEzh0769WfM8Qh2q9v32wJgsl2AdBDyQ5RKt90WZdeEjmXloczw8gXJBi+OaZrF6e0vLCeMzlFlU9o+Iv1JOFiZTZsC5PjP/1peF0+IlykUTfkTp179fRKxM+Uy9knOmsXwDpXvZL4B/Nt/dV6L0sz0V+PWS2orThNuLIuIV4Mgo1yF4ODPHR7lA0f4R8VBmTm6OQa9EuTreScDeEbFbUwutbtAE4/oqjq1pVvemjEd5sP0xrd+Gppb4mxHxwyaLdfvMRtYgvw5ROvZfEGUS/99TaoBmAnsAOwJrZOYdlClHdqQcrP6Umc82j++XpY+l4XghxZxLS/ZrajGPp4Te33R2kG9OcAZT+p99NjN3yswblqJay27X7P/vUM7qWxcQ6AcsF3NmC/k+sF1ErN20shyYmbu3wnFnXWOWZM3B/sfA/1H6h95Imcfz24Zjdab9WBURh0TE+5pA8TBN1zHK9GzXA4Mi4uimZu4aSo3bi9WJ7ccp3Q5+B/yyl15Gt2tqfdv3TbT/nZmXNot2a45dmwI/AO6jXKGztf6RlP13NzDCcNy9qtr59SLiE83iN1GmuhwdEe+PiBOa3+u5RJnggMycXnWD6dZMtdT9IC2IzgJTZv4F+BnlB+1ZYFVKLdAfgb2Ao5pVfwYck82I/erxNtEspGibAi/L4LtXgQGUeRFbc+V2FnqfycxDs1yesp5uS6/PEGAw8LbMPKtZdmmz/JMRsTVwDOWqR08CZObTMOckZ2k9YczMmZn5PeDDlOmILu7rMmnR1jpWNbWbUKYIPKf5+z+AGVEufpGUwcdnUmZGeSvw/sz896aiYK0o8wCvSenKc+niehxsAlI2r+stEXFg+3G9ua+Ve86g9EO+C/geZcacH+WcaVb3BjaiDJZdbPfLoi4ivkqZXWW1ZtGvgRGU7nnrUmb5+kir60/1e/FyUyk2pBW0u71svufziojVgBWyuQpLs2yeqVwi4jRKU+jbKQP0xlMORvcAX8q2ycP9gnW/KFfI+SClJv98SreJ/6FM6P7L9i9NR11b4nVM/7I0iojVW6G2g/u2ppz1b5RlIMuKmTkzIral9K/dkzJ92QlZ5gCX1EVNbdlGlEuKP9N0nfgwpTvOlTlnKsBbslyi/WhK6+UhVZiur0A2u7atCRhLRM1o01r1QeBASv/+mykD8SZ18hu+AaXFd64B860gvbSetPeE5rc36/cgIraizCf93nq9tt/nsZSpQY9r+wwfTLlo0tGZeV2PlNnMNkdz0OkHbEmZYPq/KF+2E7L0o2ytt0zTLDCY0on8W8BFmfnpiFgzM5/sg+Iv8ar9vgywHHA8sE3z/xcpAyCPpXR1eT/wySyTvLdOUpbJOdO+bE6ZMeHSeZ9J7SLicMr+/Xn7yURr31L6Pt6Zmd9tlm8HPJqZ0+rvhT880oKJiNUpV3tbDlgbuIFSMbMMcG5m3t+EvfsptW5JqTC4IjN/3LatJeL7116xEWVQ9jjKpej3jYiNgI8BT2bbFQCb9dsHFc++BLF6TpQr1q6ZmVdH6ff+Z8pUe88CmwD/yMyDowzy/gQlk306M+9rHv9myrSXDwH/lpnP9VRZ7WLRiIh3AIdS5iGd0Pz9K2BWUyM2e1+1Di6Z+ffMPJ/S7/IvzbLZIaB3X8GSq671jYjlsvTffpEyAO8DlEA8mjIo7LDMvJzyA3FoNDNRNE0wr0TEyhHxLcpVDCf0xetZXDTNWSs1N68ALgfeFG3TPzU/Mq9SAvJxEXFYRJxBaVXZvFnHcCwtgKiuWte03GxKufT4tpk5njJ6f03KnL6DmpaZPwI/bdY/kfKdncvi/v1rjkuzZ9mIiE0iYmBmvkA5To2A2YO47gLWjYidq8fONeNORAxs1jccd6OIGBAR744yZouIWC3KxdLOBg6IiC9TugbtT+kXfxHlKoWjoswYEsAPs4xTaYXjdSkX+fp0Zn6mJ8MxGJCJiFERcQewHmWwTD9Kv6TzKP2KvwKvfVDJzK9kMztFtWyxPggtSqqmwE8D4yPiUxHxjiwDIfenTCO2CXAT8O6I2AI4jVKTMPug19SC3kwZCPXWzHy4l1/KYqM5KXkn5XsBZc7JT1Bq6zduX785AfkdpcZmIPBPyqCWm9vW83shzUfrRLJp6v+XZvGVlFbNK5vazqcoNclvo1yAamvK8W35povT7Zk5oxWyF3cRMTIi1miONRkR20fEbygnDT9vuoB9F/hjRLTmVf8tZaDifq2T/Wq/jo6Iy5gzoFHdawPgI8zZv7sDf8jMHSnjUA6kXLRlQtOSO5HSLegvlIrJKzPzCpjrwiGTs4wd6pXf7qU+IFMuv/mbzDyX8uYcSxko81XKBQ0+3TrDrA80zZnofK9xrwXXqiFoW/ZJSg3Khym1kp+NiBUog05eqJrKXgRGZrkK21nV49cEngO2z8z/ae+LpiLKHKr/RmnKXZ0yjc59lBaVX1BqirePDuaYBsjMazLzu1kuNvBC+3dE0vw1rWVrR8RPgPMj4hhKgPgepUJg/2a9n1CmqfwkpcXmvzNztyxXrWxta0k51r2ZcoVJmu6NJwCnNv1XtwC+2az3DeDAiBiaZb7c3wCXZubzTTBePco80SdRxgr9ttdfyRKqqZ1vtfj+CfhfygWP3ti07P4sIn5JCc9nAMMps1UMpZzc7U6ZCnSuGXxyzmD8Xv0sL5WB7v+3d97hdpTVF35XCj0EBGIApQjSNEAgEiWKoBCR0BFBRY0/RFBAkRhQepUaVKRIDSCoEARFQJEOFtAYIISmKKAGhNBDCyRZvz/2N8nk5KbBbedkv8+T596ZMzOZc86dmf3tb+21618i0c3uQEnjiYKHPwKrlNH4KOIL61++wDXK/r1qU/bLKpqF9Mrs2DtHs1YiL1rW9SKC4iuJXusbAKOKzOJM4NMKI/ydgO8V2Ut1vGqK8mnbY5ydyObFSsCqROXwv4EPA7+1fZLDVP82orp4vcYd2xowOosfk2SetHHtrEnYjt1BTDtvC4yw/VfgP8AASWsU6UDlL/7BasamVQamCgu26h7+I2AtSZ+y/TxRH4SkcURn1CGKNvV3EFrs75f9brE9tmw7hJCdXG17mNvw2U3ePiU7P13SBxS+0hcRM4qbK/pDrEvYC34OuBT4KNEn4gVgV9tfsP1Ed0k2douT6Gw8qwfxxkQ3lgklbX8j8WXtUEZAvweOJuQWW5X9KxuYAwj3hOmpX3r7VDfAogubrrBzOQU4X9LQ8tm+TkwvTnS03L25SCkeAnYhWqR+wvY99WO2UPaksxgH3Exk5p8hJBM9FQV3EA/jN4lGA/1gZsfBMmDsrbBYWjwHjEkyb+oDSUkrltUTCUnT34hA+TnCQ/wThFazNyEp24OYjn6q7F9ZYDX9wLT6XEqyZK2yemWifgRHc4jPAifZPoQYvJ9SArERhAa7OlY1I3kfsJVDw520M5IWkXQGEfyuZPtVomPtpoQ079/AjmWgcjDhMf0z21Md3W27VZ3KQhEglxn7HrXl3SSdL+nztk+2vS4wTNLGjuKGPxL96ne0fQIxZbO+7bPL/ttKuoP4/NZ3zSImWTBKpmQ0zPCoHEw8AKYQ38NJiqrXW4lK1ypD8g1iSm112/fZvrqsrx4QGRjPg8ZRuqShhJ7xGmKQ+CnbVwGLEyP96sF7PTCY8DiuDxi/TDS86Fuf4k2SZFYkbSxpZJEBTJe0kaTfEgHe6YTN6H8I7/CzbH8eeBI4qPw8HNjC9tca5BRNHxhXlM/lg5KuB0Yp7Oh+CDwr6cCy2XRgRYUr0RQiYbK07f/Zfqo23V91X3ullT6jbsjHCNu8gbYvhRlNWapGahOJWeBDiKL6b9j+c/0A3SU4hoUgQK5N2U8vcohVgeFE16BTJH2ubHo6oe+CyKC9BKwuabESgP27BNpLEdPOO9k+zcXrOFkwiiylr6PSeKSkpcvNbDeiSvsw2z8hsgLbEd7SNwFnS7qNaEV5ROMUWd785h/P6jXZi5AQVXZ5NwEfkDQAuBgYKOkoSdcAjwDfsl01YxlSJC7rAZvYvq6T30qSNA1lNuYSoqXzJIWe/3uEpO9AQiq2t8LabQPgjiI3e50oOFunzII+0pj8aWbakJn0JRyixtjezjO9mvcDDinb/4rITv4auMP2jq7ZrHanYKuVqWXolyPcJyp5TO+y/nLiexrq0M3vans/NziEdTda0ge5POwXtz25LPcmZBKfITJcV9m+XGE0/R3gQ2V6eCJwpO3zFdWyzzQct9uk/psdRfXwsraHSno/MSjZmOj2dCJh7j6mZJhHAefavk7S0sD7PLM98Wzm78mcUfEOLddID6IC/HjbzxfJxCii2cBZkg4h9GOHEBnkPYgq5AsbjnksYcfz7059M0nShJRg9yEi+JsM/ItIAvwF+DZRVHZECR7OIArTPkzcF89s9WeQouXwWCJTPpIolp9ETNHfX54LlwE9be+uqA+a7KhJycZPHci8YiBJuwCDgNE1ycTaZTC3D3Cr7UfKehExaLf9e265ALlcLN8nfPWuIjKNuxF+iDcSBQ+vERmwaZJ+B9xl+6iSTf6A7cNqx8uguB2pPk+FA8W/ge1t3yXpHEK8/y1FxfY6wFElw3I0UTw2wvbLtWPljfBtUg0sSub34fK59wA2A44imhKsQgTHV9i+so1j5LWRJAtAucbWJLTFLxGzNj2Ie+FYovPdc2Xbj9j+s6R1gCm2H6uO0SrXXT3BUaR05xNFiIcBDxKFeJ8kan02IDKU3ycK6s8lnu1TyzMlnwcdQMnU93UURs5pm+p58m5CW7wiMSu/AzAA2NdN2EG126a23w7lS5pEjMg3Ii6ml4jR+UO2xxP6VgE7lt2OAL5dMsY/rwfHkFM07U3tRvYGcDJwdnnpBGALSRsS02ZTKVXKwPHAyHpwXI6VN8N5oFqzgbK8k6SxwA/KYPKLwLaSPlj+1h8j7gvfLVn6c4lC1RnHq37PayNJ5oykdSStXF9XrpmngK8SQd5GRUN8NuHB/6Kk9RRWWDsril0ftv2YZja4aJnrrgRVfcviVoRt2462JwBL2r7I4Xt7NtGTYBlgmu1Jtney/Wb1eeTzoP2RtCzR2GPzsvwNScdI2qosz6j5KX+bTxOzkHcC3yBisC9UwXH9WdQMtFSAXL6k3kT1fX9ga9t3EgHXVmWzcT5nN6UAABgdSURBVMTIdKii/e1fgI/bfqb68rqzJqaZkDRAYca+eH29Z3oangosJWnPcgH9jJC4/AcYDyxe9p1q+8X8XhYMzdpsYFVJHyCyMd8hfI6/DbxIVIV/v3y+yxOuLsuVh/NNtl+uro1WejgnSUehKFi9mZmNdqr1sj3Z9uWEDvlEANtHAM8Szap+Snjzj/SsBXjTm11OVt3Daz/XA85R1AYtBuwr6YoiLXlQUd+wqqTjic/zupLoqo7XEnZ23Q1FF7zetl8gkiabSjoZ+DTR5OPnklYvs/CzFMbbnmj7LGBv23vVn93N9vfbihKLrxLT84sTVa1nE4UNfwS2sz1B0qaENdilLrZgSftRRp2nElqkC4BxbuioptL3XtLWwEW2+0tahCjG+w5wfWYE3jmKAqAjia5F/wAeKHKKwWXd32xfKuliom3takRl8S1ddc5J0qyUe9h0Qqb0NyJb/Hylx2zYdgmiNfK1ts8syZ0lgTcdbZNbSk5RR9JKtp9U1JgMJ+R1x0n6NDHr+yBRsNibmEHcBfjN3Kb5k3dOCWSXAbYgklT/JGqD9gX6AV+y/aykUYSD11Z1mUw5RvVsr7LFTTvr0RIZOUlDJe1RFm8HnieyyMsSWeQniUr8yqXibqKhRAbH7UyZLhtN8e20fXpjcAxhDVYurN8B4yWdb/tNwtdyRnCcWeO3j6SNgNOAPrZXJeRFuwPYvpsoFBpU5BVfJgpiBlTBcWZnkmT+UHi3nwt8zmF72Ae4kLBlm9zWPiUIPgs4SNKitt+y/aJrHSibNbCoUNBoJ7krkQjB4WJ0M/B+hef9b4G7iGZFg4DnHNZsFzsKiXvWAq+kHZH0KcKC7SVCono+ESC/TriETCUafWB7BLCOpG3KDGWv2t9sFRwv76Bp/4ZbJfjoCxxWbjL/ILromKh8XUfRbehoYGIJ4KY7KoTzQmsnFAb2EKPMKcQAZGptKq2tz7r6+/s6IbXo6bDUm5ZT+u3Ca0RGqg/MaEv7qKTjyuu3AJW9G7bvt/2WWqjZQJJ0NJIWKUHxbYSf/tqEXOKfwCUuTTzawvZvCMvQKQ3rm/7aqzKLjrqTfkVOge0xwH8VjbYgssWPErUQPQnp13lEe+hT68d0aRzSiW9joaA8bx8ikls9gWnA6oTD0QRHn4H/EgmVqpnNScQgEEIXXiW1dgMeIDLPTU2rBMhXEhY5R5Xl24kvZyoxbTywXKhfsP1STSuTF1o7UKYVq+mx9wJr1HTGVQHFbJ91CYR72f6n7d3rD4X8buaPuQ3ybD9MNFeZJOnjZfXewMGKotRHgWNt/7phv6Z/OCdJZ1AkEG+WxUUIK7JtbB9LeIp/tcgI2tq3GoiO65ST7SRqyY3KneJwosHTwYoOqRBT9oeX+//TREJrCOE6dYHtTWxfUvZvlTil21K+q2cIjfEvCFnLCcDKiiYsEF7G6wMbln3OYGZ3YUv6iMInfwgwuMwONzUt8YdXvtxTge0kreqwyXmduGGd4+jhDuTF1kEsTkyZvUr4TL8qaUuYYRZeZZEHKBqtUL3mWovuzOjPH6oZsM9pIFH7LG8mroXBik5UE4hR/8Zl/6cbtk+SZC6UorFLYIYrz9qS7iKC4+uB70laz/YNxHT1jgrv42r/yo2iyritUn429TVY5BSNetQvACvYXotww9lb0meK7O52YLSkLwIDgR8QzT5eLPu2hMykO1KPg8rX9hWii+qjhIZ+MyK5sgLRNRXbtxOzkgMkVbOS95f9VyZcK0bY/qZLD4pmp6WK9BSVrlsQU8o3ENmxl8pr2VCinZA0DFjR9vm1dZcBT9g+RNJ3iW5Pw2uvL0pIKa4A/udZu7gdSDhVnN5Z76FZKdNbPwEOcpiv7w+8Adxje6zaKOopmf1dCW33bH7GSZLMPyWQfRdxz3qpSPi+ZXuX8vrZhG/xAYo6gLPK63fXr09JmxDSvxttn9Y176Z9aHhfawFftn1oCcSWBk4BVibkj+sQ/sUiZrQ+CpzqcJxKOpDG54OkQeW5cRzhdby/pP8DhhHf0R5EMqUf8ASRXHmkMZZq1fiqV1efQHtSLsjBhMb4rzCLDqrlvrzOpuiMv0dYg62m6Lx2UsmEXEJY9PQmpmLOkHQe8XBYCjiWsIs5r3Yj3YJoRHE3cFzj/5fMpDyUe9h+StJzwNckvU6Y598F/ErSFm5ovV24mahMvr1+vLwmkmTBKdPJSwB/kPQxYgbtWUkrOHz4RwF3S7rB9m8lXU/I/aqMcz/ifrci0UDhX130VtqN8r76EgH/YsT96Qnb5yo6pa5kexsASZOBbxJB8SlE8Ex5Le9LHUhDcDwCOF7S14ieBLdKWokoyNsU2NP2OZKeBbYHzvYcmtW06nfWUhnkOiWgaFp7ke6EouvdKGAT4DTbPy+ZkR8B1xIOIf2B/YH9HVXYKwMjiEzLWmW/K8vxFgUuIix89vdciliSNkf9yxHe3pOA3RyFdScAa9nepf6QaSujnCTJ/CHp0w5nhbZe+wFR0HQIYdd2WpFVIGk8MXDd1/ZbtX36EZ3grrD9+9mP2rwofHL7A18B9iPu/+sC7wPOJDTZqwOfJ3TGY2r75n2qgykxUV/gcNsjSjB8MlGkXbU0f9z2lZK2JboZfsb2f+vHaNVguC1aNkBO2g9JawCn2x5Wliufw0HA54ib4F7An4GP2J5Y23cp26/Ulns6ivNWr0ajybwpU5XHEK07jwQ+DKwNHGj7n+X1x4Edq6If1VqvlteXckM3wiRJZkfRMvda4F7be5V1jfralcs2exBBxu6EfnNFQm97ge1n6/sWXW3TN/yoUwKvRYhg6/raIOEG4F4iMN4T2IdwShhZf0YknYukXxKmBv8i6tDeJAY1Bn5n+2xFl9X3OexAq/0WukFMFqwl88MbRFe7zSUNBb6uqExegei7/jqhJXuT0JPVeRVmKbio3C0yOJ5PisbxCuBlwld1y/L7MsCHJC1Zblx3AK+Uool6EdCXgZsIb9EkSebNssT1dKnCt3e2aeQS5F0AnGD7GuAAIvA42vZJjoYKs3QQcxPblJVAeDaKgnEKkZ0cWHtpNCGleLftcwg7u8/bnljdozr+rBc+NAcjAs30tf8mUTy6A/BFYAIxC/JeotAORyvvu+v7L2zBMWSAnMwfkwjrl4sJ78PViIrXvYiR5xeJ9pPvo0HXXn8wdN7pthzLATsDY8p0762ET+WjxHdwvKSriUKKJyvNvaJN63XAAKKL5P1ddP5J0q1ROEusXgsuXicCiIuJwG9OgcdoYBVJX7H9jKMx0h1l+6aX+Gn+HHOqz+U04LOSdpLUn5BbPEDokrH9eLV91gV1DHPL8taSUxMJieNY4FPAwQ53ox2JjoVJISUWyXwjaR2iknUx2y8o2nqvC3y36GAH2R7btWfZmkj6NfCw7YMlLU+0Z12K8A29kZgau7O2/VAieB5ZPZiSJGmbous/BPg5sDlRH9GfaLRzrO0n5rLvusA/XfyQW0WnqQVwzKl+l/QlIos8DDgduI/QHO/vmqVn0r5odheRvYFrgD+VZ3Obf5OSrgReAQ7wTHu9hU5KMScyQE7eNgov0LFusGfLC6z9kbQBcCmwi+2/S9qGaMU61vb1te0qffgSjla2SZLMhZo++Fyi1f0fgH1s/1fSGYRW82KHv/4s+1ErBK9r/puZ8r56lFqRC4EXiIx65ZjzdWAWx5w29Nl9gPcQLkZX2f5xZ76HhYU26kz6EQV3zxHuKk/YPryNQu+qFqhyXknaIAPkZL6R1IvQKQ0jZBVVwUVeYJ2AwqtyoO1h5Wa4uO1K452DkiR5mxR95u7AgcSM2I1l/VCiEPli27eVdTMCyLL8Htcq/ZuZNgKp+XbMKdtXgddQwr7taNtXdfb7WNhQNFsZDjwJ3Gr7QknrAz8lvreH28r41/bvlRn+2UkNcjLflAuoD9Fu8mDbw21PymKLTuNM4AVJywLYfrX67DM4TpK3Tymeu4wI6vZROPfgsGJ7GthB0nvKOpcgcFFJ3weuLYFk01NkEj0kHSfpfqLW5M+End0qZbNDieLgjarguF6EXQbv99jeIIPj9qWxuFHSexVNutYn2kOvTPQoWNb2eMLT+EiY5butB8orltcyOG6DDJCTBcL2eNtfs31buVZ7tILerhmw/ZTtPWy/UN3g8rNPknnTlmtCQ6BRDTR/UVZ9shSorU3ocB8A/lfbfh+iWHY8Maszi/yiWdE7d8wZTjjm9O+K829lSnbeRQ5UOVL0Jgq4H7F9CyFpeTewXnn9x8BgSVtCBMklUF5R0gXAiDIznLRBSiySt0VO6Xcd+dknyfzTkDH7KKGNHdOoF64Vmm1GePYOIILiYVUBXtluG+ATRMOF1zvrfXQGknYimp6sYfsxhcXdmuXfasD9hF3kksDOLh73koYQRY4PAUdW0q+kfahLWSQdDSwB3Gn7GknfIVyKPl5ePw14lpAFTZS0oe17a8c6gnCvOMT27bP9Z8kMMkBOkiRJWhpFB7svETrjyUQh3k9LwetsFf6SVgP62f5LbV1VzNfSA1SlY063oEjprgM+b/txSUsTziAGbiGcKi4BfkPYtv3C9mhF+/NvA8c0BMbvIrrj/YtoG930BaUdTQbISZIkScvQ6CYhaQngemCy7e0krUkEdE/bPq2N/RsLzxaqAialY06XI6l3KYg8D5hmex9F58ariW610yRtB3wMuIwonj8FGFC+kzYLRyUtUp8NSeZOapCTJEmSpqcNPexakvqU4O2XlC5vth8lWiC/t8gpqn17NExl9ynbLzTBMYDt+4jirh+UVb8DRlXBsWZ2B5xafmZw3M6U4Lg3cA+wXckKTwIeBrYum90GbAy8y/a1wJ+IYj3m5KqSwfGCkQFykiRJ0rRIGiSpX62AabCkW4ATgCslLe/w4b1P0siy2+3Aq8D2kpaEGQVMlrSJpMuBjbrkDXUP0jGnE2mjgHQbQu+9GFEgOopwEvk7sKmk/rYnEx7VANje0/a4zjvr1iclFkmSJEnTIulbwG227ys6y9HAmbZ/L+k/wM22h5ds8Q+ArR32lJ8AXnbp/ln0tscSdmYHuNYII0k6irY07ZL2A5axfVxZfpD427wN+C7hUrEM8A9g7xIst0wXx+5CBshJkiRJU1FsrqbX5BC7EsHuDZL6AoOBEwl5wK7AiFLxfxnwmu29Go43hAhATnR4HyekY05H0uCu8kHi73SM7QmSRgH/sf3D8voexN/z2iWbPwSYavvurjr/hYEMkJMkSZKmoSGwWKsUkh0AHGh7lbL+fOBG25eXQqfNgA2BvsRz76myXeVMsRTwelb2J52JpMWALYCRwETC1/hK4L/Aj4AdbP9P0gBgLGEteHLDMXIQ00GkQXSSJEnSNBSv4g8CJwPTJA23/UNJX5J0YHGmmA6sKGldYArhz7u07f/BzKCiykBXfr5J0lHMIZA9A/gosIftsZKGASMIS707gZMkvUY0XvkqETzPQgbHHUdmkJMkSZJuSxu2bX2BXwGX2B5dW78pcA3RSexThH/vhsBhtq/o1JNOkkKtuLGSA21UlsdJei/habyXozttX+Bo4Hnbx5RtPwP8zPaEsn9mjDuJDJCTJEmSbo+kfYlp5ieJKenHCeur9wP32x5TNMY9be8uaQXC+/iNsn/PlFAknUmDHGglotHH8sBzwLWlsccxhH/xTmW7gcDFwPC6K0VjoJ10PGnzliRJknQr6rZXkgZK+huwFWHNNhEYR/gaLwX0Az5bAosDgEUlLQI8Z/uNUtBHBsdJZ1PkQD0U7aFPAn5pe3PgLWB/SZ8EjgRWl7RD2e0hYM+G4LhHZWPYyW9hoSYzyEmSJEm3Q1Jf2y9JOgh4zPaYsr5PZWtVllciMm4jbI/votNNktko1oL7EkHvWcSA7hLgQeBlYEnbX5c0HDjI9npdda7J7GSRXpIkSdKlVFPRtZ/rAUdIOpholrBvsXJ7BthB0u5Epf/XgJ2Bc+rBccopkm7CcoR923q2nylZ4ntt7yfpI8BvJI2zfZ6kP3XtqSaNZICcJEmSdCm1oqP+hMb4TeBR4IulWOmvwEtE5u15YFvgeKKz2BDbzzccL4PjpMuxfbWk64A9Cd38JGDL4q6yM3AtcHfZ9u/Z6KN7kRrkJEmSpFNR0KNh3a5Ee11sPwrcDLxf0lDbvwXuAlYFBhH64ldsX2z7eUk9G9v1Jkk34TBga0lr2P4V8EvgIqJhzfD6zEcGx92L1CAnSZIknUY9SyapH7C87QfL8n3A6OJr/G5CQrEC8G2iAG834Azbl3TN2SfJgiPpOGBD29uW5Rk6+rRt675kBjlJkiTpcNrwgz0c+ANwsKRTymb7AodL6mX7acDAEKJxwgW2N6mC48YMdJJ0Y84EXpS0bAmIJxd3C2Vw3H3JDHKSJEnSYbTl3yrpC8Bg298sv58N/J/tKyVdRdi5/R7YEbga+I3tl8q+WYCXJEmHkwFykiRJ0iE0NEpYC/iy7UNL9ndp4BRgZeBhYB1CQiFgb6IF76m27+ySk0+SdiblFM1FBshJkiRJh1Frn7sYoSnex/a5kj4EHGV7WNluMnAiERRPaThGVvcnSdKppM1bkiRJ0pEcCrwL+ArRMOGw0hL6DaCPpI8BqwN/BP5eD46rjFsGx0mSdDZZ5JAkSZK0O8XKbVFgUeAy29Ns/4gIko8oPy8DTge2JtrrjqkfI6ejkyTpKlJikSRJkrxt5iV/kHQR8LDtE8vy7sBoYE3bEyWtZvvx6liQfrBJknQ9mUFOkiRJFojSmKM3zDmYrdmwnQZ8VtJOkvoT3fIeIHTJ1ILjHi509PknSZLMiwyQkyRJkvlG0orAVcD7yvL+kvaSNKgs94CQR5SgdzzwQ2Az4A5gKtH44y1JM+pgUk6RJEl3IiUWSZIkyTwp8ocetqdJuhB4AXgd2IBoA/11YAvb/6jv0+B/3Ad4D3AWcJXtH3fme0iSJJlfMkBOkiRJ5kqjf6uk5YBfAZOA3Wy/JekEYC3bu7QRGPcsgfVQwvv4aNtXdfb7SJIkmV9SYpEkSZLMlUouIek4SfcDHwf+DPQEVimbHQp8SNJGtXbSPcv+04r04h7bG2RwnCRJdycD5CRJkmSuSNoMuAJ4GTgI2LL8vgwRFC9ZMsx3AK8UizdVLaElDQduIgr0kiRJuj3ZKCRJkiSZF8sBOwMjbT8maSlgTeBRYC/gw5JWBZYEnqxlkIcAhxCex9vZfrVLzj5JkmQBSQ1ykiRJMk8k/ZrwMz5Y0vLAcGApYBvgRuB3tu+sbT+UCJ5HVlZuSZIkzUIGyEmSJMk8kbQBcCmwi+2/S9oGGASMtX19bbtetqdKWsL2a111vkmSJO+EDJCTJEmS+ULSccBA28NK0d3ilWyi0ekiSZKkmckivSRJkmR+ORN4QdKyALZfrbWHzuA4SZKWITPISZIkSZIkSVIjM8hJkiTJAlG1k06SJGlVMoOcJEmSJEmSJDUyC5AkSZIkSZIkNTJATpIkSZIkSZIaGSAnSZIkSZIkSY0MkJMkSbo5kjYsjTmq5aMkfecdHO8d7Z8kSdLqZICcJEnS/dmQaOmcJEmSdAIZICdJknQCklaT9LCk8yVNkHSZpC0l/VHSPyRtImlJSRdK+qukeyTtIGkR4BhgN0n3StqtHHI9SbdJ+pekb9b+nwPL8SdIOqC2/lBJj0i6CVi7c999kiRJc5E2b0mSJJ2ApNWAR4GBwAPAX4H7gD2B7YGvAA8CD9q+VNIywF/K9rsCg2zvV451FDAU2ALoAzwC9AfWBy4CPgwIuBvYg0iGXAQMBnoB44Cf2D61I99zkiRJs9Krq08gSZJkIeIx2/cDSHoAuNm2Jd0PrAa8B9i+pg9eDFhlDse6zvYUYIqkZ4B3Ax8Frrb9avk/rgI+RgTIV9t+ray/pkPeXZIkSYuQAXKSJEnnMaX2+/Ta8nTifjwN2MX2I/WdJA2ex7Gmlf01l/87pwuTJEnmk9QgJ0mSdB9uAPaXJABJA8v6yYSUYl7cAewoaQlJSwI7AXeW9TtJWlxSH2C79j/1JEmS1iED5CRJku7DsUBvYLykCWUZ4FaiKK9epDcbtscRWuO/EPrj823fU9ZfDtwL/JIImpMkSZI5kEV6SZIkSZIkSVIjM8hJkiRJkiRJUiMD5CRJkiRJkiSpkQFykiRJkiRJktTIADlJkiRJkiRJamSAnCRJkiRJkiQ1MkBOkiRJkiRJkhoZICdJkiRJkiRJjQyQkyRJkiRJkqTG/wMHjWoYEBjwCQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "results_df.plot.bar(x='method', rot=30, ylim=(0.5, 1.0), figsize=(12, 8), title='Accuracy on sentiment analysis task')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('data/results2.pkl', 'wb') as f:\n",
    "    pickle.dump(results, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discussion: weaknesses, potential improvements\n",
    "\n",
    "- We can see that the ULMFiT model is superior to the other options, with a test accuracy slightly above 94%, which is around the state of the art.\n",
    "- The Twitter dataset itself is not very big; with more training data and investing more time in tuning the hyperparameters, better results can be achieved.\n",
    "- Since we are using transfer learning from a very powerful language model (WT-103), it is not necessary to invest too much time in NLP pre-processing of the text (as opposed to what was done with the baseline models in part 0).\n",
    "- However, _some_ pre-processing had to be done due to the specific characteristics/idiosincracies of Twitter text:\n",
    "  - hashtags had to be mostly converted into \"regular\" words\n",
    "  - handlers (e.g. `@VirginAirlines`) have to be accounted for, since they can be too specific, they are mapped to dedicated token. The same applies to URLs, something that is not done out-of-the-box by the data loader.\n",
    "  - Emojis and other characters can convey meaning, so we should try to keep them as \"words\".\n",
    "- In the same line, the feature engineering of the tokens shows some weaknesses, that could be improved later:\n",
    "  - Many people write in UPPERCASE in tweets to convey emotion; this is very different from the original Wikipedia language style. I suspect that this degrades the performance of the model (because the tokenizer actually detects and remembers those changing in casing). To solve it I would either:\n",
    "    - fine-tune the language model with more training data, in order to learn those idiosincracies; or\n",
    "    - pre-process the text to convert the uppercased words to \"normal\" (we would keep normally capitalized words though), although some sentiment information could be lost there.\n",
    "    "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}