Twitter US Airline Sentiment - 1. Language Model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Challenge Part 1: Language Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai.text import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The idea of this challenge is to use deep learning (more specifically, the ULMFiT) so let's start. In this notebook I will load a pre-trained language model and fine-tuning with our specific Tweeter dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "INPUT_FILE = 'twitter-airline-sentiment/Tweets.csv'\n",
    "VERSION = 'lm'\n",
    "\n",
    "DATA_PATH = 'data/%s/' % VERSION\n",
    "\n",
    "import os\n",
    "os.makedirs(DATA_PATH, exist_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text normalization\n",
    "\n",
    "As opposed to the classic approach that might require careful pre-processing and feature extraction, a deep neural network gives us much more freedom. And fast.ai's `TextList` does most of the work for us: tokenization, cleaning up the strings, building the dictionary, numeralization/vectorization, etc.\n",
    "\n",
    "In any case, I considered convenient to apply a very light pre-processing over the raw strings, namely:\n",
    "- All the tweets have a handler with the airline's company at the beginning (e.g. `@VirginAmerica`), I am removing them as they bring no additional information and actually could make the models to overfit, learning to classify according to the airline's name instead of the actual sentiment of the text.\n",
    "- `nltk` has a very handy `TweetTokenizer` ([documentation](https://www.nltk.org/api/nltk.tokenize.html)) that helps to clean up HTML tags and to respect/normalize emojis.\n",
    "- Any other handlers are replaced by a placeholder `xxhan`.\n",
    "- Any hashtag that looks like a word (e.g. `#angry`) has the initial `#` removed, since they normally carry regular meaning. Other hashtags are just kept, if they are too rare they will be replaced by UNK anyway.\n",
    "- All URLs are replaced by `xxurl`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.tokenize.casual import TweetTokenizer\n",
    "\n",
    "REGEX_URL = re.compile(r'https?://|www.')\n",
    "TOKEN_URL = 'xxurl'\n",
    "TOKEN_HANDLE = 'xxhan'\n",
    "\n",
    "def normalize_token(token):\n",
    "    if token.startswith('@'):\n",
    "        return TOKEN_HANDLE\n",
    "    elif token.startswith('#') and token[1:].isalpha():\n",
    "        return token[1:]\n",
    "    elif re.match(REGEX_URL, token):\n",
    "        return TOKEN_URL\n",
    "    else:\n",
    "        return token    \n",
    "\n",
    "tokenizer = TweetTokenizer(preserve_case=True)\n",
    "\n",
    "def normalize_text(text):\n",
    "    tokens = tokenizer.tokenize(text)\n",
    "    \n",
    "    # if the first token is a handler, it's normally irrelevant\n",
    "    if tokens[0].startswith('@'):\n",
    "        tokens.pop(0)\n",
    "    \n",
    "    return ' '.join(normalize_token(t) for t in tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's load the original CSV, extract the columns we care about and apply the pre-processing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DATA SIZE: (14640, 2)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What xxhan said .</td>\n",
       "      <td>neutral</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>plus you've added commercials to the experienc...</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>I didn't today ... Must mean I need to take an...</td>\n",
       "      <td>neutral</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>it's really aggressive to blast obnoxious \" en...</td>\n",
       "      <td>negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>and it's a really big bad thing about it</td>\n",
       "      <td>negative</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text sentiment\n",
       "0                                  What xxhan said .   neutral\n",
       "1  plus you've added commercials to the experienc...  positive\n",
       "2  I didn't today ... Must mean I need to take an...   neutral\n",
       "3  it's really aggressive to blast obnoxious \" en...  negative\n",
       "4           and it's a really big bad thing about it  negative"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raw_data = pd.read_csv(INPUT_FILE, header=0)\n",
    "\n",
    "df = raw_data.copy()[['text', 'airline_sentiment']]\n",
    "df = df.rename(columns={'airline_sentiment': 'sentiment'})\n",
    "df_raw = df.copy()\n",
    "\n",
    "df['text'] = df['text'].apply(normalize_text)\n",
    "\n",
    "print(\"DATA SIZE: \" + str(df.shape))\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also show a few tweets in more detail, to check the effect of the normalization:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NEUTRAL: @VirginAmerica What @dhepburn said.\n",
      ">> What xxhan said .\n",
      "\n",
      "POSITIVE: @VirginAmerica plus you've added commercials to the experience... tacky.\n",
      ">> plus you've added commercials to the experience ... tacky .\n",
      "\n",
      "NEUTRAL: @VirginAmerica I didn't today... Must mean I need to take another trip!\n",
      ">> I didn't today ... Must mean I need to take another trip !\n",
      "\n",
      "NEGATIVE: @VirginAmerica it's really aggressive to blast obnoxious \"entertainment\" in your guests' faces &amp; they have little recourse\n",
      ">> it's really aggressive to blast obnoxious \" entertainment \" in your guests ' faces & they have little recourse\n",
      "\n",
      "NEGATIVE: @VirginAmerica and it's a really big bad thing about it\n",
      ">> and it's a really big bad thing about it\n",
      "\n",
      "NEGATIVE: @VirginAmerica seriously would pay $30 a flight for seats that didn't have this playing.\n",
      "it's really the only bad thing about flying VA\n",
      ">> seriously would pay $ 30 a flight for seats that didn't have this playing . it's really the only bad thing about flying VA\n",
      "\n",
      "POSITIVE: @VirginAmerica yes, nearly every time I fly VX this “ear worm” won’t go away :)\n",
      ">> yes , nearly every time I fly VX this “ ear worm ” won ’ t go away :)\n",
      "\n",
      "NEUTRAL: @VirginAmerica Really missed a prime opportunity for Men Without Hats parody, there. https://t.co/mWpG7grEZP\n",
      ">> Really missed a prime opportunity for Men Without Hats parody , there . xxurl\n",
      "\n",
      "POSITIVE: @virginamerica Well, I didn't…but NOW I DO! :-D\n",
      ">> Well , I didn't … but NOW I DO ! :-D\n",
      "\n",
      "POSITIVE: @VirginAmerica it was amazing, and arrived an hour early. You're too good to me.\n",
      ">> it was amazing , and arrived an hour early . You're too good to me .\n",
      "\n"
     ]
    }
   ],
   "source": [
    "def show_normalization(_df):\n",
    "    for i, row in _df.iterrows():\n",
    "        print(row.sentiment.upper() + \": \" + row.text + '\\n>> ' + normalize_text(row.text) + '\\n')\n",
    "\n",
    "show_normalization(df_raw.head(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we will save the already normalized data, for future use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(DATA_PATH + 'tweet_data.pkl', 'wb') as f:\n",
    "    pickle.dump(df, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the base model\n",
    "\n",
    "As indicated in the paper and in fast.ai's documentation, I am using a pre-trained language model from a subset of Wikipedia (the famous [wikitext-103](https://einstein.ai/research/blog/the-wikitext-long-term-dependency-language-modeling-dataset)).\n",
    "\n",
    "Note that for training the LM, we can use the **entire** dataset, even with the neutral tweets. Everything is useful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# creating the data source\n",
    "# let's use a batch size of 48 as suggested\n",
    "bs = 48\n",
    "\n",
    "data_lm = (TextList.from_df(df, cols=['text'], path=DATA_PATH)\n",
    "           .split_by_rand_pct(valid_pct=0.1, seed=42)  # just for repeatibility\n",
    "           .label_for_lm()\n",
    "           .databunch(bs=bs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see a few entries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>idx</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>$ 30 a flight for seats that did n't have this playing . it 's really the only bad thing about flying xxup va xxbos yes , nearly every time i fly xxup vx this “ xxunk xxunk ” won ’ t go away :) xxbos xxmaj really missed a prime opportunity for xxmaj men xxmaj without xxmaj hats xxunk , there . xxurl xxbos it was amazing , and</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>xxbos trying to check - in ... but looks like your site is down ? xxbos spending my birthday night with you , xxup dal - xxup dca . xxmaj get me home ! xxbos xxmaj sent a couple messages to the email you provided but to no avail . xxmaj will try again next year . xxmaj cheers . xxbos first time flying xxmaj virgin , went to xxunk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>i got an email at xxunk am that the flight was xxmaj cancelled xxmaj flightled , does n't matter now flight into atl is delayed , thx xxbos 1k and had problem getting out of xxup fll to xxup iah sent xxup dm to you about making my connection please let me know xxbos xxmaj done and done xxbos thanks , just sent :) xxbos i 'm grounded in xxmaj</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>an apology . i will never fly with your airline again . unitedairlines xxbos has the xxup worst customer xxunk ! xxmaj this xxmaj xxunk rep at call center must be investigated . xxmaj my baggage 's lost and there 's no help but argument ! xxbos should xxup not sell tickets for xxhan on xxurl . has terrible service and xxup ua staff can not reach xxmaj silver xxbos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>taking my 1 k status to xxhan neveragain xxup worstcustomerservice xxbos xxmaj hi ! what is the phone number for reservations in xxmaj xxunk ? xxmaj thanks xxbos can i request a ticket change through twitter ? xxbos xxup xxunk xxmaj flight was a nightmare ! ! xxmaj from poor customer service , having my confirmed seat given away ... + more issues xxbos to start daily # xxup xxunk</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_lm.show_batch()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_lm.save('data_lm.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fine tuning\n",
    "\n",
    "We can leverage fast.ai's `language_model_learner` that already includes the pre-trained model based on WT103, and the architecture as described in the paper, using a AWD_LSTM with drop-out and an AdamOptimizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = language_model_learner(data_lm, AWD_LSTM, drop_mult=0.3, path=DATA_PATH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use standard tooling to find our hyper-parameters (in this case, the learning rate) for the fine tuning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Min numerical gradient: 2.75E-02\n",
      "Min loss divided by 10: 4.37E-02\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8XHW9//HXJ8lkX9uka1pKC5SltLQNS0EriAKiolX0h4oKqJWr4r5wr9ddr7uIVsFaL4qiXgW5CF4LuAACbaGFtrSltKUtbVraZk+zJzOf3x9zkoaQNmmbk5lJ3s/HYx6ZOed75nwymcxnvusxd0dERAQgLdEBiIhI8lBSEBGRHkoKIiLSQ0lBRER6KCmIiEgPJQUREemhpCAiIj2UFEREpIeSgoiI9MhIdABHq7S01KdNm5boMEREUsqaNWuq3b1soHIplxSmTZvG6tWrEx2GiEhKMbMXBlNOzUciItJDSUFERHooKYiISA8lBRER6aGkICIiPZQURESkh5KCiIj0UFIQEUkBP/zbFv61tSr08ygpiIgkOXfnR3/fysrtNaGfS0lBRCTJNXdEiTkUZkdCP1eoScHMis3sTjPbbGbPmtmCPvuLzOxeM1tnZhvN7Now4xERSUWNrZ0AFOaEnxTCXvvoZmC5u19pZplAbp/9HwY2ufsbzawMeM7M7nD3jpDjEhFJGQfbuoDhqSmElhTMrBBYCFwDEHzQ9/2wd6DAzAzIB2qBrrBiEhFJRY1t8ZpCQXb4a5iG2Xw0HagCbjOzp81smZnl9SmzBDgN2As8A3zM3WMhxiQiknKGs/kozKSQAcwDbnH3uUAzcGOfMpcCa4FJwFnAkqCG8RJmttjMVpvZ6qqq8IdkiYgkk+6aQmGK1xQqgUp3XxU8vpN4kujtWuBPHrcN2AGc2veJ3H2pu1e4e0VZ2YDXiBARGVF6+hRSuabg7vuA3WY2M9h0MbCpT7FdwXbMbDwwE9geVkwiIqmou/loOPoUwj7DDcAdwcij7cC1ZnY9gLvfCnwN+KWZPQMY8Dl3rw45JhGRlNLY1kVWRhpZGemhnyvUpODua4GKPptv7bV/L3BJmDGIiKS6g22dw9J0BJrRLCKS9Bpbu4alkxmUFEREkl5jWycFwzBxDZQURESSXmOrmo9ERCRwsE3NRyIiElDzkYiI9Ghs7aIwRzUFEZFRr60zSkc0NiwrpIKSgohIUutZ90gdzSIi0tjafS0FNR+JiIx6h1ZIVU1BRGTUO7RCqmoKIiKjXs8FdlRTEBGRQ5fiVFIQERn1ejqa1XwkIiIH2zrJSDNyIuFfSwGUFEREklpjcC0FMxuW8ykpiIgkscbWrmG5DGc3JQURkSR2sK1z2EYegZKCiEhSa2wbvsXwQElBRCSpNbaqpiAiIoH4tRRGSE3BzIrN7E4z22xmz5rZgn7KXGhma81so5k9HGY8IiKpJn7VteGrKYSdfm4Glrv7lWaWCeT23mlmxcBPgcvcfZeZjQs5HhGRlNEZjdHSER22ZbMhxKRgZoXAQuAaAHfvADr6FHsn8Cd33xWUORBWPCIiqaZnMbwR0nw0HagCbjOzp81smZnl9SlzClBiZg+Z2Roze09/T2Rmi81stZmtrqqqCjFkEZHk0b0Y3nCtewThJoUMYB5wi7vPBZqBG/spMx94PXAp8AUzO6XvE7n7UnevcPeKsrKyEEMWEUkeh5bNHhlJoRKodPdVweM7iSeJvmWWu3uzu1cDjwBzQoxJRCRlHLrAzghoPnL3fcBuM5sZbLoY2NSn2D3AK80sw8xygXOBZ8OKSUQklSSi+Sjs9HMDcEcw8mg7cK2ZXQ/g7re6+7NmthxYD8SAZe6+IeSYRERSwnBfdQ1CTgruvhao6LP51j5lvgt8N8w4RERSUU/z0QjpUxARkePQ2NqJGeRnjoA+BREROT6NbV3kZ2WQljY811IAJQURkaTVOMzLZoOSgohI0mps7RrW/gRQUhARSVrxmsLw9SeAkoKISNJqbO0c1jkKoKQgIpK0Dg7zVddASUFEJGmpo1lERACIxZymdnU0i4gI0NTRhfvwLoYHSgoiIkmpezE8NR+JiAiNrcO/GB4oKYiIJKVD11JQTUFEZNTrXjZb8xRERORQn4Kaj0RERM1HIiLSo7ujOV9DUkVE5GBbJ7mZ6UTSh/djWklBRCQJ1bd2UjTMs5lBSUFEJClVN7VTmp817OdVUhARSUI1TR2Mzc8c9vOGmhTMrNjM7jSzzWb2rJktOEy5s80samZXhhmPiEiqqGlqZ2ze8NcUwu7WvhlY7u5XmlkmkNu3gJmlA98G7g85FhGRlODuVDd3UFowgmoKZlYILAR+AeDuHe5e30/RG4C7gANhxSIikkoOtnfR0RWjNAE1hTCbj6YDVcBtZva0mS0zs7zeBcxsMrAIuPVIT2Rmi81stZmtrqqqCi9iEZEkUNPUATDi+hQygHnALe4+F2gGbuxT5ofA59w9eqQncvel7l7h7hVlZWXhRCsikiRqmtoBGJuA0Udh9ilUApXuvip4fCcvTwoVwO/NDKAUuNzMutz9f0OMS0QkqVUHSaE0ATWF0JKCu+8zs91mNtPdnwMuBjb1KXNi930z+yVwnxKCiIx21UHzUSLmKYQ9+ugG4I5g5NF24Fozux7A3Y/YjyAiMlp19ymMyRtBNQUAd19LvImot36TgbtfE2YsIiKpoqa5neLcyLCvewSa0SwiknSqm9oZm4BaAigpiIgkneqmjoSMPAIlBRGRpFPT1E6ZkoKIiEB3TUHNRyIio15HV4yG1s6ELIYHSgoiIkmlriVxS1yAkoKISFKpOtg9m1k1BRGRUa+muXs2s2oKIiKjXiIXwwMlBRGRpFLTpJqCiIgEqpvaycxIIz8r7KXp+qekICKSRKqbOijNyyS4pMCwU1IQEUkiNc3tlBYkpj8BlBRERJJKIhfDAyUFEZGkUpPAxfBASUFEJGm4e5AUVFMQERn1Gtu66IjGErZCKigpiIgkjUMT11RTEBEZ9bqXuEjUCqkwyKRgZjPMLCu4f6GZfdTMisMNTURkdOmuKSRqMTwYfE3hLiBqZicBvwBOBH4bWlQiIqNQVYKXuIDBJ4WYu3cBi4AfuvsngIkDHWRmxWZ2p5ltNrNnzWxBn/3vMrP1we1xM5tz9L+CiMjI0F1TKEngPIXBLq7RaWbvAN4LvDHYFhnEcTcDy939SjPLBHL77N8BvMrd68zsdcBS4NxBxiQiMqLUNHVQkhshkp647t7BnvlaYAHwDXffYWYnAr850gFmVggsJN7chLt3uHt97zLu/ri71wUPVwLlRxO8iMhIUt3UntCJazDImoK7bwI+CmBmJUCBu39rgMOmA1XAbUGz0BrgY+7efJjy7wP+2t8OM1sMLAaYOnXqYEIWEUk5NU0dCV3iAgY/+ughMys0szHAOuIf9D8Y4LAMYB5wi7vPBZqBGw/z/BcRTwqf62+/uy919wp3rygrKxtMyCIiKae6uT2hI49g8M1HRe7eCLwFuM3d5wOvGeCYSqDS3VcFj+8kniRewsxmA8uAN7l7zSDjEREZcaoPtid05BEMPilkmNlE4O3AfYM5wN33AbvNbGaw6WJgU+8yZjYV+BPwbnffMshYRERGnI6uGI1tXanRpwB8FbgfeMzdnzSz6cDWQRx3A3BHMPJoO3CtmV0P4O63Al8ExgI/DS4o0eXuFUf5O4iIpLza7tnMCa4pDLaj+Y/AH3s93g68dRDHrQX6fsjf2mv/+4H3DypSEZERrLp73aMELnEBg+9oLjezu83sgJntN7O7zEzDR0VEhsie+lYAJhVnJzSOwfYp3Ab8GZgETAbuDbaJiMgQqKyLJ4UpJX3n+A6vwSaFMne/zd27gtsvAY0NFREZIrtrW8jLTKc4dzCLRYRnsEmh2syuNrP04HY1oOGjIiJDpLKulSljcgkG3STMYJPCdcSHo+4DXgSuJL70hYiIDIHKuhbKS3ISHcbgkoK773L3K9y9zN3HufubiU9kExGR4+Tu7K5toTzB/QlwfFde++SQRSEiMorVt3TS3BFlypjUTgqJbfgSERkhdte1AKRO89Fh+JBFISIyiiXLcFQYYEazmR2k/w9/AxKf0kRERoDdtUFNYUziP1aPmBTcvWC4AhERGa1217VQlBOhMDuxcxTg+JqPRERkCMTnKCS+lgBKCiIiCbe7toXy4sT3J4CSgohIQrm7agoiIhJX1dROe1csKSaugZKCiEhC7a4NhqOqpiAiIpXBxLVkmKMAg78cp4wyXdEYe+vbAIhkGJnpaRRkR8jM0PcIkaHUPXFtchLMZgYlhZdxd57aVc/K7TW4H5q3lx1Jpzg3k5LcCGUFWZw5uSjhS9wer20HDrJiey0dXTE6ozE6umLsrm3h2X2NbNnfREdX7CXlI+nGaRMLOWtKMXPKiynKidAVc6IxxwxKcjMpK8ikND8Lw6hpbqemuYOGlk7KCrKYVppHUU7ix2GLJJPKuhZK8zPJzUyOj+PkiCLB3J26lk7uWbuH3z2xiy37mwY8puKEEr58xRnMmlw0DBEOnY6uGA9s2sdvVr7Ayu21L9tfmp/JaRMLueb8aZxUlk9amtEZjSeNvfVtrN1dx51rKrl9xQvHdP4xeZlMKcmhMCcSn6yTE+GEMbnMLi/mzPIi8rOO/Jasb+lg495GNu1t5IXaZrIy0smJpJOTmU4k3UgzIz3NyEgz8rIyes5RmB0hNzOdvKwM8rLSyUxPGzCpd78vaps7aGzrpLG1k/auGBMKs5lcksPYvMyU/2Igibe7tpXJSdJ0BCEnBTMrBpYBs4gvl3Gdu6/otd+Am4HLgRbgGnd/Kqx4aps7eGJHLat21LBqey2761po74q95BvxnPIivvmWM7n8zInkZqYD4A6tnVHqWzqoa+nkmT0N/PDBLVyx5FHecc5UPn3JTEryMgcdh7vT2hnlYFsXY/MyyUg/cpNMW2eUHdXNvFDTzK7aFl6oaaG+pZOMdCMjLY1IutHeFaOpvYvm9i46umKMzc9kQmE24wqzicacnTXN7KppYeuBJhpaOykvyeFzl53KG+dMjDcLpaeRkW5EBogFIBpztlc10doZJSMtflzMndqmDqqa2qlu6sDdGZufyZi8LAqzM9jf2M7OmmZ2Vjezp76VxrYu9tS30tDSSU1zBwBmcMKYXBxo7YjS2hmlK+qkp8U/6M3iq0l2K86N0BV1Wjq6iB3lSlyZ6WkU5mRQmB2hIDuDnMx0siPxBNPeFaOyroXKulZaOqKHf46MNMrysygKElxJXoQTS/OYOaGQUycUcGJp3qBeTxndKutakurLZdg1hZuB5e5+pZllAn3T4euAk4PbucAtwc8hd8/aPXzs92sByI6kMW9qCWdPm0x2ZjpZGenkZqbzipNKD/vHycxIoygnwglj4awpxVwxZxI3PbiFX698gbuf3sPlZ07kyvnlnHviGNxhy4GDPL6thvWV9dS3xr9lNrZ10dDaSUNLJx3ReCLKTE9jelkeJ48voLwkh1jM6QiacvbWt7KtqonKulZ6tWRRlBNhbF4mXTGnKxqjM+ZkR9LIy8wgPyuDSHoa26uaWfF8DY1tXQBMLMpm6phcLjtjApfNmsDCU8pITzu2b7npacbJ44duBZSapnbW72lg3e56tu5vIj3NyA0+pCPpRjQG0ViMmMfbXc+YVMjpEwsZm58FxJNsRzRGZzTelBWLOV0xp6m9K3jdO2ls7aKlI540mzuivfbFf7Z2Rqlr7mBvkOimjc3jFSeVUV6Sw9j8zJ7aRiTd2NfQxt76VvbUt1LT1BH/m7Z2snnfQR7YuJ+uIEPlRNI558QxvPLkUl5xcikzxxeoZiEvEY05e+pbuWzWxESH0sN6t5sP6RObFQLrgOl+mJOY2c+Ah9z9d8Hj54AL3f3Fwz1vRUWFr169+qjj2VXTwr3r93LuiWOYXV48ZB2mz+07yG2P7eC+9S/S1N5FeUkOrR3Rnm+/k4qyKS3IojA70vPNtDg3k+LcCHmZ6VTWt7J1fxNb9h/kxYY2IsG39Uh6GuMKsjhpXD4njctnRlk+08bmMXVMLkVHcQ3X1o4oZvE+EQlfe1eU7VXNPLfvIGt31/OvrVU8X9UMxGs2c6cUM29qCfOnlXD2tDGqSYxye+tbOf9b/+Abi2bxrnNPCPVcZrbG3SsGKhdmTWE6UAXcZmZzgDXAx9y9uVeZycDuXo8rg22HTQrHaurYXD580UlD/bTMnFDAt946my+98QyWb3yR+9a9SFFOhAUzxrJgxtiET0jJyVQyGE5ZGemcNrGQ0yYW8ua5k4H4P/6j26pZs7OOp3bV8c/nqgAoyY1w2awJvGH2JM6bPvaYa26SupJpyexuYSaFDGAecIO7rzKzm4EbgS/0KtPff8HLahVmthhYDDB16tQQQj1+OZnpLJpbzqK55YkORZLMpOIc3l4xhbdXTAGgobWTldtr+Mv6F7ln7V5+98RuJhVl875XTueqs6eQN0Bnu4wcPUtmJ8lwVAg3KVQCle6+Knh8J/Gk0LfMlF6Py4G9fZ/I3ZcCSyHefDT0oYoMn6KcCJeeMYFLz5hAa0eUf2w+wK9W7ORr923iR3/fynsWnMB7z59GadBnIiNXss1RgBBnNLv7PmC3mc0MNl0MbOpT7M/AeyzuPKDhSP0JIiNNTmY6r589kT98cAF/+tD5nDd9DEv+uY0LvvUPPn/3M+ysbh74SSRl7a5rYXxhFlkZydPMG3Y99QbgjmDk0XbgWjO7HsDdbwX+j/hw1G3Eh6ReG3I8Iklr3tQSfvbuCp6vamLZv7bzx9WV/PaJXbxu1gQ+8ZpThnTElySH3bUtSdWfACGOPgrLsY4+Ekk1Bw628avHd/Krx1+gpaOLK+eX8/HXnMKk4uRpapDjc/Y3/sbCk8v4/tvnhH6uwY4+0ng4kSQ1riCbz1x6Ko989iKuveBE/vfpvVz4vYf43v3P0d51+El1khr2N7ZRdbCdMycXJjqUl1BSEElyY/Iy+cIbTucfn34Vl8+awJJ/buONP36U9ZX1iQ5NjsP6ygYAzixPntnMoKQgkjLKS3L54VVzue2as2ls7WLRTx/nO8s3q9aQop6prCfN4PSJSgoichwuOnUc939iIW+ZO5mfPvQ8V/z4MTbsaUh0WHKU1u9p4JTxBUk3wVRJQSQFFeVE+O7b5nDbNWdT19LBm3/yGDc9uIXOaGzggyXh3J0Nexo4M4kWwuumpCCSwi46dRwPfGIhb5g9kZv/vpU3LVGtIRW82NBGdVNH0vUngJKCSMorzs3kh1fN5dar53PgYDtv+sljfGf5Zto61deQrHo6mVVTEJGwXDZrAn/75EIWBX0Nl//oX6zbrRFKyeiZPfVkpMWvZJhslBRERpDi3Ey+97Y53H7dObR3xvh/S1fwwMZ9iQ5L+lhfGe9kTsYl7ZUUREaghaeUcc9HLmDmhEI++Js1/PKxHYkOSQLdncyzk7A/AZQUREas0vwsfv+B83jNaeP58r2b+Pp9m4ge7XVLZchV1rVS19KZVJfg7E1JQWQEy8lM59ar5/PeBSew7NEdvO9XT9LQ6zrXMvyeCUaHqaYgIgmRnmZ8+Yoz+PqbZ/HYtmqu+MmjbN7XmOiwRq31lQ1E0o2ZE5Jz1VslBZFRwMy4+rwT+P3i82jtiLLoJ4/zl/W6dEkibNjTwKkTCpPqGgq9KSmIjCLzTxjDfTe8gtMnFfKR3z3F7St2JjqkUcXdWV9Zn7T9CaCkIDLqjCvM5o73n8vFp47ni/ds5Oa/bSXVrquSqnbVttDY1pW0/QmgpCAyKmVH0rn16nm8dV45N/1tC1+5dxMxjUwKXTLPZO4W9uU4RSRJZaSn8d0rZ1OcG+EXj+5gX0Mb33v7HPKz9LEQlmf2NJCZkcYpSXxpVdUUREaxtDTjP19/Gv/5+tN4YNM+Fv3kMXZWNyc6rBFr7e56Tp9YSGZG8n70Jm9kIjIszIz3v3I6t193LlVN7Vyx5FH++dyBRIc14kRj8ZnMc5K4PwGUFEQk8IqTS7n3I69gckku7/vlk/zhyd2JDmlEeb6qiZaOKLPLixMdyhGFmhTMbKeZPWNma81sdT/7i8zsXjNbZ2YbzezaMOMRkSObMiaXu/5tARecVMpn71rPLx7VmklDpXvF2jlTVFO4yN3PcveKfvZ9GNjk7nOAC4Hvm1nmMMQkIoeRm5nBsvdW8LpZE/jafZu46cEtGrI6BNZXNpCflcH00vxEh3JEiW4+cqDAzAzIB2qBrsSGJCJZGen8+B1zuXJ+OTf/fSvf/OtmfNs2+NCHoLAQ0tLiPz/0IXj++USHmxLWVdZz5uQi0tIs0aEcUdhJwYEHzGyNmS3uZ/8S4DRgL/AM8DF310VmRZJARnoa33nrbN593glsue0PdM2aDcuWwcGD4B7/uWwZzJ4Nf/1rosNNau1dUZ59sZHZSd50BOHPU7jA3fea2TjgQTPb7O6P9Np/KbAWeDUwIyjzL3d/yWpdQUJZDDB16tSQQxaRbmlpxlfOyKbz3m8RaW97eYHOzvjtyith/XqYMWP4g0wBm188SGfUmZPkncwQck3B3fcGPw8AdwPn9ClyLfAnj9sG7ABO7ed5lrp7hbtXlJWVhRmyiPSRdtMPyIwNcL3nzk646abhCSgFra+MdzIn8/IW3UJLCmaWZ2YF3feBS4ANfYrtAi4OyowHZgLbw4pJRI7Bb36DdQ5wDYbOTvj1r4cnnhS0rrKB0vxMJhfnJDqUAYXZfDQeuDveh0wG8Ft3X25m1wO4+63A14BfmtkzgAGfc/fqEGMSkaPV1DS05Uah9ZX1zC4vJvg8TGqhJQV33w7M6Wf7rb3u7yVegxCRZJWfH+9UHkw5eZmm9i62Hmji8jMnJjqUQUn0kFQRSXZXXw2RyJHLRCLw7ncPTzwpZsOeBtxJiU5mUFIQkYF86lMDJoVoRgQ+8YlhCii1pFInMygpiMhAZsyAO++E3NyXJQePRGiNZPORRf/OhuzSBAWY3NZVNlBeksPY/KxEhzIoSgoiMrDXvS4+D2Hx4pfMaLbFi2lYtZr1Z57Pe/77CbbuH0TfwyizvrI+ZZqOQElBRAZrxgxYsgQaGiAajf9csoQJc8/gjvefS0aa8c5lq9ih6zH0qDrYzu7a1pRpOgIlBREZAtNK87jj/ecSjTlXLV3BtgMangrw6LYqAM6fkTpNa0oKIjIkTh5fwO8+cB7RGFy1dAXP7VNT0sPPVTE2L5MzJhUmOpRBU1IQkSEzc0IB//PB80hPM65auoINexoSHVLCxGLOI1urWXhKWdKvjNqbkoKIDKkZZfn84YMLyM3M4J0/X9kzJHO02bC3gdrmDl51Smqt16akICJD7oSxefzPB8+jKDfC1ctWjcoawyNbqjCDV56cOv0JoKQgIiEpL8nldx84j4LsCO9atoqNe0dXYnh4SxVnTi5KmfkJ3ZQURCQ05SW5/H7xeeRnZfCuZavYtLdx4INGgIbWTp7aVc/Ck1Or6QiUFEQkZFPGxGsMOZF0rlq6glXbaxIdUuge31ZNNOa8aqaSgojIy0wdm8sfPriAsoIs3v2LJ/jzur2JDilUD2+poiA7g7lTUmcmczclBREZFlPG5HLXv53PWVOK+ejvnuaWh57H3RMd1pBzdx7eUsUrTiolIz31PmJTL2IRSVnFuZnc/r5zeMPsiXx7+Wa+/pdnR1xi2HqgiRcb2lJuKGq3MK+8JiLyMtmRdH501VxK87P4xaM7iMacL73x9JS4KtlgPLIlvrTFQiUFEZHBSUszvvTG00lPM37x6A66YjG+esWslJr5ezj/fO4AJ4/LZ1IKXI+5P0oKIpIQZsZ/vv40MtKNnz28nWgMvv7mWaSncGJoaOlk5fZaPrhweqJDOWZKCiKSMGbGjZedSiQtjSX/3EZtczs3XzWX7Eh6okM7Jv94bj/RmHPJGRMSHcoxU0eziCSUmfHpS2fypTeezgOb9vPOn6+ktrkj0WEdkwc27md8YRazJ6fO9RP6CjUpmNlOM3vGzNaa2erDlLkw2L/RzB4OMx4RSV7XXnAit7xrHhv3NvLWWx7nhZrUulhPW2eUh7dU8drTx6d038hw1BQucvez3L2i7w4zKwZ+Clzh7mcAbxuGeEQkSV02ayK//cC51Ld08KafPMbjz1cnOqRBe2xbNS0dUS45PXWbjiDxzUfvBP7k7rsA3P1AguMRkQSbf8IY/vfDF1CaH5/9/OsVOxMd0qA8sHE/BVkZnDd9bKJDOS5hJwUHHjCzNWa2uJ/9pwAlZvZQUOY9IccjIinghLF53P2h83nVKWV84Z6N/Mfdz9AZjSU6rMOKxpy/Pbufi04dR2ZGor9rH5+wRx9d4O57zWwc8KCZbXb3R/qcfz5wMZADrDCzle6+pfeTBAllMcDUqVNDDllEkkFBdoSfv6eC797/HLc+/Dw7q5u55V3zKcqNJDq0l3lqVx01zR1ccsb4RIdy3EJNae6+N/h5ALgbOKdPkUpgubs3u3s18Agwp5/nWeruFe5eUVaWmrMEReTopacZN77uVH7w9jms3lnHop8+xs7q5OuAfmDjPjLT01J2aYveQksKZpZnZgXd94FLgA19it0DvNLMMswsFzgXeDasmEQkNb1lXjm/ef+51LV08OafPsbKJFp+2925f+N+zj9pLAXZyVeLOVph1hTGA4+a2TrgCeAv7r7czK43s+sB3P1ZYDmwPiizzN37Jg4REc45Md4BPTYvk3f/YhV3P12Z6JAA2PRiI7tqW7g0hSes9WaptkJhRUWFr17d75QHERkFGlo7uf7Xa1ixvYZPvfYUPvLqkxK2mF57V5S33bqCHdXNPPTpC5P60ptmtqa/qQF9pXY3uYiMOkU5EX513TksmjuZ7z+4hRvvStzIpK/eu4n1lQ18/21zkjohHA2tfSQiKSczI40fvH0O5SU5/Pgf29hW1cT33zaHaaV5wxbD3U9XcseqXXzwVdNTeq2jvlRTEJGUZGZ86pKZ3HzVWWzZf5DLf/Qv7lj1wrBctOe5fQf5jz9t4JwTx/CZS2aGfr7hpKQgIintTWdN5v6PL2Te1BI+f/cGrrntSTbsaQjtfFv3H+QDt6+kgAyGAAAJbklEQVQmLyuDJe+Ym5KX3DySkfXbiMioNKk4h9uvO4evXHEGq3fW8oYfP8o7lq7kH5v3E4sNXc3hnrV7eNNPHqOlo4ul75nPuMLsIXvuZKHRRyIyojS0dvL7J3Zx22M72dfYxrSxubxlXjmL5k5mypjcY3rOts4o//V/z3L7ihc4e1oJS945j/EplhAGO/pISUFERqTOaIy/rH+R/3lyNyuCyW7nTBvD/GklnDg2j2mleUwdk8uYvMyXrFfU1hllX0MbL9S2sGZnLat21LJ2dz3tXTE+8MoT+exlpxJJwSYjJQURkUBlXQv3rN3Lvev2su1AE119mpTyszIozo3Q1hmjuqm9Z3uawRmTijh72hhee/p4FsxI3RVQlRRERPrRFY2xp76VHdXNVNa1UtfcQV1LJ3UtHWRlpDG5OIdJxTlMLslh1uQi8rNGxsj9wSaFkfHbiogMUkZ6GieMzeOEscM3pyGVpF7DmIiIhEZJQUREeigpiIhIDyUFERHpoaQgIiI9lBRERKSHkoKIiPRQUhARkR4pN6PZzKqAeqDv2rhFA2wb6H73z1Kg+hhC6+/8g9nfd/uRHveNtfe2Y4l7OGPufT8Rr7XeH3p/HGl/Kr4/jiZmgJPdvWjASNw95W7A0qPdNtD9Xj9XD1VMg9nfd/uRHveN9XjjHs6YE/1a6/2h98dIe38cTcyDOUf3LVWbj+49hm0D3e/v+OONaTD7+24/0uP+Yj2euIcz5t73E/Fa6/1x9PT+GPz9ZI95MOcAUrD5KGxmttoHsWhUsknFuBXz8EnFuBVzYqRqTSFMSxMdwDFKxbgV8/BJxbgVcwKopiAiIj1UUxARkR4jOimY2X+b2QEz23AMx843s2fMbJuZ/cjMrNe+G8zsOTPbaGbfGdqow4nbzL5sZnvMbG1wuzzZY+61/9Nm5mZWOnQRh/Y6f83M1gev8QNmNikFYv6umW0O4r7bzIqHMuYQ435b8D8YM7Mha8c/nlgP83zvNbOtwe29vbYf8X2fMMcyfCpVbsBCYB6w4RiOfQJYABjwV+B1wfaLgL8BWcHjcSkS95eBT6fSax3smwLcD7wAlCZ7zEBhrzIfBW5NgZgvATKC+98Gvp0K7w/gNGAm8BBQkehYgzim9dk2Btge/CwJ7pcc6fdK9G1E1xTc/RGgtvc2M5thZsvNbI2Z/cvMTu17nJlNJP7PvcLjf73bgTcHu/8N+Ja7twfnOJAicYcqxJhvAj4LDHnnVxgxu3tjr6J5Qx13SDE/4O5dQdGVQPlQxhxi3M+6+3PJEuthXAo86O617l4HPAhclsj/1YGM6KRwGEuBG9x9PvBp4Kf9lJkMVPZ6XBlsAzgFeKWZrTKzh83s7FCjPeR44wb4SNBE8N9mVhJeqD2OK2YzuwLY4+7rwg60l+N+nc3sG2a2G3gX8MUQY+02FO+NbtcR/9Y6HIYy7rANJtb+TAZ293rcHX+y/F4vM6qu0Wxm+cD5wB97Nd9l9Ve0n23d3/gyiFcDzwPOBv5gZtODbB+KIYr7FuBrweOvAd8n/gEQiuON2cxygc8Tb9oYFkP0OuPunwc+b2b/DnwE+NIQh3ookCGKOXiuzwNdwB1DGWN/hjLusB0pVjO7FvhYsO0k4P/MrAPY4e6LOHz8Cf+9DmdUJQXiNaN6dz+r90YzSwfWBA//TPwDtHcVuhzYG9yvBP4UJIEnzCxGfL2TqmSO29339zru58B9IcYLxx/zDOBEYF3wj1gOPGVm57j7viSNua/fAn8hxKTAEMUcdIC+Abg4zC84vQz1ax2mfmMFcPfbgNsAzOwh4Bp339mrSCVwYa/H5cT7HipJ/O/Vv0R3aoR9A6bRq8MIeBx4W3DfgDmHOe5J4rWB7k6gy4Pt1wNfDe6fQrxqaCkQ98ReZT4B/D7ZY+5TZidD3NEc0ut8cq8yNwB3pkDMlwGbgLKhjnU43h8McUfzscbK4TuadxBvXSgJ7o8Z7Ps+EbeEBxDqLwe/A14EOoln5vcR//a5HFgX/CN88TDHVgAbgOeBJRya6JcJ/CbY9xTw6hSJ+9fAM8B64t/AJiZ7zH3K7GToRx+F8TrfFWxfT3ytmckpEPM24l9u1ga3IR0xFWLci4Lnagf2A/cnMlb6SQrB9uuC13gbcO3RvO8TcdOMZhER6TEaRx+JiMhhKCmIiEgPJQUREemhpCAiIj2UFEREpIeSgowIZtY0zOdbZmanD9FzRS2+quoGM7t3oFVKzazYzD40FOcW6UtDUmVEMLMmd88fwufL8EOLxIWqd+xm9itgi7t/4wjlpwH3ufus4YhPRhfVFGTEMrMyM7vLzJ4MbhcE288xs8fN7Ong58xg+zVm9kczuxd4wMwuNLOHzOxOi19v4I7uNe+D7RXB/aZgEbx1ZrbSzMYH22cEj580s68OsjazgkMLAuab2d/N7CmLr7v/pqDMt4AZQe3iu0HZzwTnWW9mXxnCl1FGGSUFGcluBm5y97OBtwLLgu2bgYXuPpf4Kqb/1euYBcB73f3VweO5wMeB04HpwAX9nCcPWOnuc4BHgA/0Ov/NwfkHXNcmWPfnYuIzzgHagEXuPo/4dTy+HySlG4Hn3f0sd/+MmV0CnAycA5wFzDezhQOdT6Q/o21BPBldXgOc3mtly0IzKwCKgF+Z2cnEV6aM9DrmQXfvvZb+E+5eCWBma4mvifNon/N0cGiBwTXAa4P7Czi0Rv5vge8dJs6cXs+9hvia+xBfE+e/gg/4GPEaxPh+jr8kuD0dPM4nniQeOcz5RA5LSUFGsjRggbu39t5oZj8G/unui4L2+Yd67W7u8xztve5H6f9/ptMPdc4drsyRtLr7WWZWRDy5fBj4EfHrMZQB892908x2Atn9HG/AN939Z0d5XpGXUfORjGQPEL+eAQBm1r30cRGwJ7h/TYjnX0m82QrgqoEKu3sD8Ut4ftrMIsTjPBAkhIuAE4KiB4GCXofeD1wXrPuPmU02s3FD9DvIKKOkICNFrplV9rp9kvgHbEXQ+bqJ+LLnAN8BvmlmjwHpIcb0ceCTZvYEMBFoGOgAd3+a+EqcVxG/2E2Fma0mXmvYHJSpAR4LhrB+190fIN48tcLMngHu5KVJQ2TQNCRVJCTB1eNa3d3N7CrgHe7+poGOE0kk9SmIhGc+sCQYMVRPiJc/FRkqqimIiEgP9SmIiEgPJQUREemhpCAiIj2UFEREpIeSgoiI9FBSEBGRHv8flsvsiC5aX9IAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot(suggestion=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both the plot and the suggested value give us the hint of `3E-02` for the learning rate. The empirically found values for momentum seem to be working fine as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.344635</td>\n",
       "      <td>3.991299</td>\n",
       "      <td>0.253111</td>\n",
       "      <td>00:13</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(1, 3e-2, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('fit_head')\n",
    "learn.load('fit_head');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Time to fine tune the model with our Twitter data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.unfreeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.829806</td>\n",
       "      <td>3.930231</td>\n",
       "      <td>0.261445</td>\n",
       "      <td>00:17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.768957</td>\n",
       "      <td>3.874933</td>\n",
       "      <td>0.270292</td>\n",
       "      <td>00:17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.685932</td>\n",
       "      <td>3.825279</td>\n",
       "      <td>0.278003</td>\n",
       "      <td>00:17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.595220</td>\n",
       "      <td>3.784717</td>\n",
       "      <td>0.284416</td>\n",
       "      <td>00:17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.485023</td>\n",
       "      <td>3.764398</td>\n",
       "      <td>0.288258</td>\n",
       "      <td>00:17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.340658</td>\n",
       "      <td>3.766773</td>\n",
       "      <td>0.289096</td>\n",
       "      <td>00:17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.144660</td>\n",
       "      <td>3.794972</td>\n",
       "      <td>0.287744</td>\n",
       "      <td>00:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.916090</td>\n",
       "      <td>3.872813</td>\n",
       "      <td>0.283414</td>\n",
       "      <td>00:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.648367</td>\n",
       "      <td>3.958725</td>\n",
       "      <td>0.281006</td>\n",
       "      <td>00:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.397522</td>\n",
       "      <td>4.078024</td>\n",
       "      <td>0.278788</td>\n",
       "      <td>00:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.141704</td>\n",
       "      <td>4.217356</td>\n",
       "      <td>0.260687</td>\n",
       "      <td>00:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.912413</td>\n",
       "      <td>4.312204</td>\n",
       "      <td>0.268155</td>\n",
       "      <td>00:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.694399</td>\n",
       "      <td>4.413949</td>\n",
       "      <td>0.267560</td>\n",
       "      <td>00:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.538969</td>\n",
       "      <td>4.526304</td>\n",
       "      <td>0.266883</td>\n",
       "      <td>00:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.384210</td>\n",
       "      <td>4.607045</td>\n",
       "      <td>0.266017</td>\n",
       "      <td>00:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>15</td>\n",
       "      <td>1.255685</td>\n",
       "      <td>4.705038</td>\n",
       "      <td>0.263366</td>\n",
       "      <td>00:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>16</td>\n",
       "      <td>1.147895</td>\n",
       "      <td>4.760036</td>\n",
       "      <td>0.263880</td>\n",
       "      <td>00:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>17</td>\n",
       "      <td>1.055195</td>\n",
       "      <td>4.807643</td>\n",
       "      <td>0.260931</td>\n",
       "      <td>00:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>18</td>\n",
       "      <td>0.989845</td>\n",
       "      <td>4.847697</td>\n",
       "      <td>0.263312</td>\n",
       "      <td>00:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>19</td>\n",
       "      <td>0.926222</td>\n",
       "      <td>4.875401</td>\n",
       "      <td>0.259903</td>\n",
       "      <td>00:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>20</td>\n",
       "      <td>0.874113</td>\n",
       "      <td>4.896961</td>\n",
       "      <td>0.261147</td>\n",
       "      <td>00:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>21</td>\n",
       "      <td>0.847419</td>\n",
       "      <td>4.899154</td>\n",
       "      <td>0.262148</td>\n",
       "      <td>00:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>22</td>\n",
       "      <td>0.814785</td>\n",
       "      <td>4.909819</td>\n",
       "      <td>0.261282</td>\n",
       "      <td>00:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>23</td>\n",
       "      <td>0.795107</td>\n",
       "      <td>4.903749</td>\n",
       "      <td>0.262419</td>\n",
       "      <td>00:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>24</td>\n",
       "      <td>0.770134</td>\n",
       "      <td>4.924276</td>\n",
       "      <td>0.261959</td>\n",
       "      <td>00:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>25</td>\n",
       "      <td>0.756493</td>\n",
       "      <td>4.914553</td>\n",
       "      <td>0.261905</td>\n",
       "      <td>00:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>26</td>\n",
       "      <td>0.762684</td>\n",
       "      <td>4.922577</td>\n",
       "      <td>0.262256</td>\n",
       "      <td>00:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>27</td>\n",
       "      <td>0.736371</td>\n",
       "      <td>4.917683</td>\n",
       "      <td>0.261418</td>\n",
       "      <td>00:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>28</td>\n",
       "      <td>0.734408</td>\n",
       "      <td>4.912556</td>\n",
       "      <td>0.262365</td>\n",
       "      <td>00:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>29</td>\n",
       "      <td>0.730752</td>\n",
       "      <td>4.916226</td>\n",
       "      <td>0.262284</td>\n",
       "      <td>00:18</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(30, 1e-3, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('fine_tuned')\n",
    "learn.load('fine_tuned');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the same spirit than the fast.ai examples, we can quickly check how good is our model in \"predicting\" the next word of a sentence, given a start:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "My flight was Cancelled Flightled and i 'm calling o waiting for me to talk to someone about my flight . Should i be at work today . xxbos i am still sitting in the airport in 2 minutes . i 'm sorry , but i called to reserve the\n",
      "---\n",
      "My flight was Cancelled Flightled and i 'm needing some help reflight Booking Problems . xxbos i really want it together . Buying Early Bird was AMAZING . You guys are the best ! xxbos It 's not the weather . The\n",
      "---\n",
      "My flight was Cancelled Flightled Monday . Just hung up on me . xxbos But we are and are , being put on security , faithful . Staff is great ! xxbos xxhan Do you know when the last chance is ? xxbos since when does\n",
      "---\n",
      "I tried to book a non - profit ( they only gave me credit card ) and told me to call the website that i paid . Terrible service xxbos Why have n't you issued a travel advisory for Charlotte Wednesday night and Thursday ? 4 - 6\n",
      "---\n",
      "I tried to check in , but they redirect me to call in to xxhan , which i also ca n't get into contact with xxbos i was just called abt my flight Cancelled Flightlation - a flight that was Cancelled Flighted at 10:30 a Sunday .\n",
      "---\n",
      "I tried to call your customer service line only to be kept in a blankets for 45 mins ... then there for 5 mins when they told me they 'd be fine on the incorrect ticket . xxbos i keep looking for help on xxbos Hi . i 'm missing a conference\n",
      "---\n"
     ]
    }
   ],
   "source": [
    "N_WORDS = 50\n",
    "N_SENTENCES = 3\n",
    "TEXTS = [\"My flight was\", \"I tried to\"]\n",
    "\n",
    "for text in TEXTS:\n",
    "    for _ in range(N_SENTENCES):\n",
    "        sentence = learn.predict(text, N_WORDS, temperature=0.75)\n",
    "        print(sentence + '\\n---')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save_encoder('fine_tuned_enc')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}