3. Working with text files.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with text files\n",
    "\n",
    "### Bogumił Kamiński\n",
    "\n",
    "In this notebook we will show how one can interact with CSV files when working with DataFrames."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "using DataFrames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "using CSV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "using Arrow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "using Statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we download the data set we will work with and save it as auto.txt file in a current working directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"auto.txt\""
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "download(\"https://archive.ics.uci.edu/ml/machine-learning-databases/auto-mpg/auto-mpg.data-original\",\n",
    "         \"auto.txt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us check how the file looks inside using the `readlines` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "406-element Vector{String}:\n",
       " \"18.0   8.   307.0      130.0   \" ⋯ 25 bytes ⋯ \" 1.\\t\\\"chevrolet chevelle malibu\\\"\"\n",
       " \"15.0   8.   350.0      165.0   \" ⋯ 17 bytes ⋯ \"5   70.  1.\\t\\\"buick skylark 320\\\"\"\n",
       " \"18.0   8.   318.0      150.0   \" ⋯ 18 bytes ⋯ \"   70.  1.\\t\\\"plymouth satellite\\\"\"\n",
       " \"16.0   8.   304.0      150.0      3433.      12.0   70.  1.\\t\\\"amc rebel sst\\\"\"\n",
       " \"17.0   8.   302.0      140.0      3449.      10.5   70.  1.\\t\\\"ford torino\\\"\"\n",
       " \"15.0   8.   429.0      198.0      4341.      10.0   70.  1.\\t\\\"ford galaxie 500\\\"\"\n",
       " \"14.0   8.   454.0      220.0      4354.       9.0   70.  1.\\t\\\"chevrolet impala\\\"\"\n",
       " \"14.0   8.   440.0      215.0   \" ⋯ 17 bytes ⋯ \"5   70.  1.\\t\\\"plymouth fury iii\\\"\"\n",
       " \"14.0   8.   455.0      225.0      4425.      10.0   70.  1.\\t\\\"pontiac catalina\\\"\"\n",
       " \"15.0   8.   390.0      190.0   \" ⋯ 18 bytes ⋯ \"   70.  1.\\t\\\"amc ambassador dpl\\\"\"\n",
       " \"NA     4.   133.0      115.0   \" ⋯ 20 bytes ⋯ \" 70.  2.\\t\\\"citroen ds-21 pallas\\\"\"\n",
       " \"NA     8.   350.0      165.0   \" ⋯ 32 bytes ⋯ \"evrolet chevelle concours (sw)\\\"\"\n",
       " \"NA     8.   351.0      153.0      4034.      11.0   70.  1.\\t\\\"ford torino (sw)\\\"\"\n",
       " ⋮\n",
       " \"25.0   6.   181.0      110.0   \" ⋯ 21 bytes ⋯ \"82.  1.\\t\\\"buick century limited\\\"\"\n",
       " \"38.0   6.   262.0      85.00   \" ⋯ 33 bytes ⋯ \"smobile cutlass ciera (diesel)\\\"\"\n",
       " \"26.0   4.   156.0      92.00   \" ⋯ 26 bytes ⋯ \"1.\\t\\\"chrysler lebaron medallion\\\"\"\n",
       " \"22.0   6.   232.0      112.0      2835       14.7   82.  1.\\t\\\"ford granada l\\\"\"\n",
       " \"32.0   4.   144.0      96.00      2665.      13.9   82.  3.\\t\\\"toyota celica gt\\\"\"\n",
       " \"36.0   4.   135.0      84.00   \" ⋯ 17 bytes ⋯ \"0   82.  1.\\t\\\"dodge charger 2.2\\\"\"\n",
       " \"27.0   4.   151.0      90.00      2950.      17.3   82.  1.\\t\\\"chevrolet camaro\\\"\"\n",
       " \"27.0   4.   140.0      86.00      2790.      15.6   82.  1.\\t\\\"ford mustang gl\\\"\"\n",
       " \"44.0   4.   97.00      52.00      2130.      24.6   82.  2.\\t\\\"vw pickup\\\"\"\n",
       " \"32.0   4.   135.0      84.00      2295.      11.6   82.  1.\\t\\\"dodge rampage\\\"\"\n",
       " \"28.0   4.   120.0      79.00      2625.      18.6   82.  1.\\t\\\"ford ranger\\\"\"\n",
       " \"31.0   4.   119.0      82.00      2720.      19.4   82.  1.\\t\\\"chevy s-10\\\"\""
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "readlines(\"auto.txt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this exercise we have chosen a typical file, which in practice means that things are not trivial.\n",
    "* first we note that it has no header with column names.\n",
    "* second, we see that the last column is tab separated, while earlier columns are separated with varying number of spaces\n",
    "* finally we see that missing values are encoded by \"NA\" in this file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will show several options how the file can be parsed to a `DataFrame`.\n",
    "The first one is to replace tabs with spaces in the source file, and then load it using `CSV.File` command."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start by getting the contents of the file into a single string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"18.0   8.   307.0      130.0      3504.      12.0   70.  1.\\t\\\"chevrolet chevelle malibu\\\"\\n15.0   8.   350.0      165.0      3693.      11.5   70.  1.\\t\\\"buick skylark 320\\\"\\n18.0   8.   318.0      150.0      3436.      11.0   70.  1.\\t\\\"plymouth satellite\\\"\\n16.0   8.   304.0    \" ⋯ 31610 bytes ⋯ \"00      2130.      24.6   82.  2.\\t\\\"vw pickup\\\"\\n32.0   4.   135.0      84.00      2295.      11.6   82.  1.\\t\\\"dodge rampage\\\"\\n28.0   4.   120.0      79.00      2625.      18.6   82.  1.\\t\\\"ford ranger\\\"\\n31.0   4.   119.0      82.00      2720.      19.4   82.  1.\\t\\\"chevy s-10\\\"\\n\""
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raw_str = read(\"auto.txt\", String)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we replace all tabs in this string by spaces\n",
    "\n",
    "(note that in general it is not a safe operation as theoretically if you had columns that are strings they might have contained quoted tabs; fortunately in this case they do not have them so we are safe)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"18.0   8.   307.0      130.0      3504.      12.0   70.  1. \\\"chevrolet chevelle malibu\\\"\\n15.0   8.   350.0      165.0      3693.      11.5   70.  1. \\\"buick skylark 320\\\"\\n18.0   8.   318.0      150.0      3436.      11.0   70.  1. \\\"plymouth satellite\\\"\\n16.0   8.   304.0    \" ⋯ 31610 bytes ⋯ \"00      2130.      24.6   82.  2. \\\"vw pickup\\\"\\n32.0   4.   135.0      84.00      2295.      11.6   82.  1. \\\"dodge rampage\\\"\\n28.0   4.   120.0      79.00      2625.      18.6   82.  1. \\\"ford ranger\\\"\\n31.0   4.   119.0      82.00      2720.      19.4   82.  1. \\\"chevy s-10\\\"\\n\""
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "str_no_tab = replace(raw_str, '\\t'=>' ')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally we create an `IOBuffer` backed by the string we have just created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "IOBuffer(data=UInt8[...], readable=true, writable=false, seekable=true, append=false, size=32149, maxsize=Inf, ptr=1, mark=-1)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "io = IOBuffer(str_no_tab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can think of variable `io` as an in-memory I/O stream. Therefore we can pass this stream to the `CSV.File` function to read it as-if it were a CSV file. Note that in the options we choose that:\n",
    "* the delimiter is space\n",
    "* we ignore repeated (consecutive) occurences of the delimiter (so we correctly handle our file which has columns padded by spaces)\n",
    "* we explicitly pass column names via `header` keyword argument\n",
    "* we specify that missing values are represented using `\"NA\"` string in our file\n",
    "\n",
    "Finally note that we pass the result of `CSV.File` operation to a `DataFrame` constructor using `|>`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class=\"data-frame\"><p>406 rows × 9 columns (omitted printing of 1 columns)</p><table class=\"data-frame\"><thead><tr><th></th><th>mpg</th><th>cylinders</th><th>displacement</th><th>horsepower</th><th>weight</th><th>acceleration</th><th>year</th><th>origin</th></tr><tr><th></th><th title=\"Union{Missing, Float64}\">Float64?</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Union{Missing, Float64}\">Float64?</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th></tr></thead><tbody><tr><th>1</th><td>18.0</td><td>8.0</td><td>307.0</td><td>130.0</td><td>3504.0</td><td>12.0</td><td>70.0</td><td>1.0</td></tr><tr><th>2</th><td>15.0</td><td>8.0</td><td>350.0</td><td>165.0</td><td>3693.0</td><td>11.5</td><td>70.0</td><td>1.0</td></tr><tr><th>3</th><td>18.0</td><td>8.0</td><td>318.0</td><td>150.0</td><td>3436.0</td><td>11.0</td><td>70.0</td><td>1.0</td></tr><tr><th>4</th><td>16.0</td><td>8.0</td><td>304.0</td><td>150.0</td><td>3433.0</td><td>12.0</td><td>70.0</td><td>1.0</td></tr><tr><th>5</th><td>17.0</td><td>8.0</td><td>302.0</td><td>140.0</td><td>3449.0</td><td>10.5</td><td>70.0</td><td>1.0</td></tr><tr><th>6</th><td>15.0</td><td>8.0</td><td>429.0</td><td>198.0</td><td>4341.0</td><td>10.0</td><td>70.0</td><td>1.0</td></tr><tr><th>7</th><td>14.0</td><td>8.0</td><td>454.0</td><td>220.0</td><td>4354.0</td><td>9.0</td><td>70.0</td><td>1.0</td></tr><tr><th>8</th><td>14.0</td><td>8.0</td><td>440.0</td><td>215.0</td><td>4312.0</td><td>8.5</td><td>70.0</td><td>1.0</td></tr><tr><th>9</th><td>14.0</td><td>8.0</td><td>455.0</td><td>225.0</td><td>4425.0</td><td>10.0</td><td>70.0</td><td>1.0</td></tr><tr><th>10</th><td>15.0</td><td>8.0</td><td>390.0</td><td>190.0</td><td>3850.0</td><td>8.5</td><td>70.0</td><td>1.0</td></tr><tr><th>11</th><td><em>missing</em></td><td>4.0</td><td>133.0</td><td>115.0</td><td>3090.0</td><td>17.5</td><td>70.0</td><td>2.0</td></tr><tr><th>12</th><td><em>missing</em></td><td>8.0</td><td>350.0</td><td>165.0</td><td>4142.0</td><td>11.5</td><td>70.0</td><td>1.0</td></tr><tr><th>13</th><td><em>missing</em></td><td>8.0</td><td>351.0</td><td>153.0</td><td>4034.0</td><td>11.0</td><td>70.0</td><td>1.0</td></tr><tr><th>14</th><td><em>missing</em></td><td>8.0</td><td>383.0</td><td>175.0</td><td>4166.0</td><td>10.5</td><td>70.0</td><td>1.0</td></tr><tr><th>15</th><td><em>missing</em></td><td>8.0</td><td>360.0</td><td>175.0</td><td>3850.0</td><td>11.0</td><td>70.0</td><td>1.0</td></tr><tr><th>16</th><td>15.0</td><td>8.0</td><td>383.0</td><td>170.0</td><td>3563.0</td><td>10.0</td><td>70.0</td><td>1.0</td></tr><tr><th>17</th><td>14.0</td><td>8.0</td><td>340.0</td><td>160.0</td><td>3609.0</td><td>8.0</td><td>70.0</td><td>1.0</td></tr><tr><th>18</th><td><em>missing</em></td><td>8.0</td><td>302.0</td><td>140.0</td><td>3353.0</td><td>8.0</td><td>70.0</td><td>1.0</td></tr><tr><th>19</th><td>15.0</td><td>8.0</td><td>400.0</td><td>150.0</td><td>3761.0</td><td>9.5</td><td>70.0</td><td>1.0</td></tr><tr><th>20</th><td>14.0</td><td>8.0</td><td>455.0</td><td>225.0</td><td>3086.0</td><td>10.0</td><td>70.0</td><td>1.0</td></tr><tr><th>21</th><td>24.0</td><td>4.0</td><td>113.0</td><td>95.0</td><td>2372.0</td><td>15.0</td><td>70.0</td><td>3.0</td></tr><tr><th>22</th><td>22.0</td><td>6.0</td><td>198.0</td><td>95.0</td><td>2833.0</td><td>15.5</td><td>70.0</td><td>1.0</td></tr><tr><th>23</th><td>18.0</td><td>6.0</td><td>199.0</td><td>97.0</td><td>2774.0</td><td>15.5</td><td>70.0</td><td>1.0</td></tr><tr><th>24</th><td>21.0</td><td>6.0</td><td>200.0</td><td>85.0</td><td>2587.0</td><td>16.0</td><td>70.0</td><td>1.0</td></tr><tr><th>25</th><td>27.0</td><td>4.0</td><td>97.0</td><td>88.0</td><td>2130.0</td><td>14.5</td><td>70.0</td><td>3.0</td></tr><tr><th>26</th><td>26.0</td><td>4.0</td><td>97.0</td><td>46.0</td><td>1835.0</td><td>20.5</td><td>70.0</td><td>2.0</td></tr><tr><th>27</th><td>25.0</td><td>4.0</td><td>110.0</td><td>87.0</td><td>2672.0</td><td>17.5</td><td>70.0</td><td>2.0</td></tr><tr><th>28</th><td>24.0</td><td>4.0</td><td>107.0</td><td>90.0</td><td>2430.0</td><td>14.5</td><td>70.0</td><td>2.0</td></tr><tr><th>29</th><td>25.0</td><td>4.0</td><td>104.0</td><td>95.0</td><td>2375.0</td><td>17.5</td><td>70.0</td><td>2.0</td></tr><tr><th>30</th><td>26.0</td><td>4.0</td><td>121.0</td><td>113.0</td><td>2234.0</td><td>12.5</td><td>70.0</td><td>2.0</td></tr><tr><th>&vellip;</th><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td></tr></tbody></table></div>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccccccccc}\n",
       "\t& mpg & cylinders & displacement & horsepower & weight & acceleration & year & origin & \\\\\n",
       "\t\\hline\n",
       "\t& Float64? & Float64 & Float64 & Float64? & Float64 & Float64 & Float64 & Float64 & \\\\\n",
       "\t\\hline\n",
       "\t1 & 18.0 & 8.0 & 307.0 & 130.0 & 3504.0 & 12.0 & 70.0 & 1.0 & $\\dots$ \\\\\n",
       "\t2 & 15.0 & 8.0 & 350.0 & 165.0 & 3693.0 & 11.5 & 70.0 & 1.0 & $\\dots$ \\\\\n",
       "\t3 & 18.0 & 8.0 & 318.0 & 150.0 & 3436.0 & 11.0 & 70.0 & 1.0 & $\\dots$ \\\\\n",
       "\t4 & 16.0 & 8.0 & 304.0 & 150.0 & 3433.0 & 12.0 & 70.0 & 1.0 & $\\dots$ \\\\\n",
       "\t5 & 17.0 & 8.0 & 302.0 & 140.0 & 3449.0 & 10.5 & 70.0 & 1.0 & $\\dots$ \\\\\n",
       "\t6 & 15.0 & 8.0 & 429.0 & 198.0 & 4341.0 & 10.0 & 70.0 & 1.0 & $\\dots$ \\\\\n",
       "\t7 & 14.0 & 8.0 & 454.0 & 220.0 & 4354.0 & 9.0 & 70.0 & 1.0 & $\\dots$ \\\\\n",
       "\t8 & 14.0 & 8.0 & 440.0 & 215.0 & 4312.0 & 8.5 & 70.0 & 1.0 & $\\dots$ \\\\\n",
       "\t9 & 14.0 & 8.0 & 455.0 & 225.0 & 4425.0 & 10.0 & 70.0 & 1.0 & $\\dots$ \\\\\n",
       "\t10 & 15.0 & 8.0 & 390.0 & 190.0 & 3850.0 & 8.5 & 70.0 & 1.0 & $\\dots$ \\\\\n",
       "\t11 & \\emph{missing} & 4.0 & 133.0 & 115.0 & 3090.0 & 17.5 & 70.0 & 2.0 & $\\dots$ \\\\\n",
       "\t12 & \\emph{missing} & 8.0 & 350.0 & 165.0 & 4142.0 & 11.5 & 70.0 & 1.0 & $\\dots$ \\\\\n",
       "\t13 & \\emph{missing} & 8.0 & 351.0 & 153.0 & 4034.0 & 11.0 & 70.0 & 1.0 & $\\dots$ \\\\\n",
       "\t14 & \\emph{missing} & 8.0 & 383.0 & 175.0 & 4166.0 & 10.5 & 70.0 & 1.0 & $\\dots$ \\\\\n",
       "\t15 & \\emph{missing} & 8.0 & 360.0 & 175.0 & 3850.0 & 11.0 & 70.0 & 1.0 & $\\dots$ \\\\\n",
       "\t16 & 15.0 & 8.0 & 383.0 & 170.0 & 3563.0 & 10.0 & 70.0 & 1.0 & $\\dots$ \\\\\n",
       "\t17 & 14.0 & 8.0 & 340.0 & 160.0 & 3609.0 & 8.0 & 70.0 & 1.0 & $\\dots$ \\\\\n",
       "\t18 & \\emph{missing} & 8.0 & 302.0 & 140.0 & 3353.0 & 8.0 & 70.0 & 1.0 & $\\dots$ \\\\\n",
       "\t19 & 15.0 & 8.0 & 400.0 & 150.0 & 3761.0 & 9.5 & 70.0 & 1.0 & $\\dots$ \\\\\n",
       "\t20 & 14.0 & 8.0 & 455.0 & 225.0 & 3086.0 & 10.0 & 70.0 & 1.0 & $\\dots$ \\\\\n",
       "\t21 & 24.0 & 4.0 & 113.0 & 95.0 & 2372.0 & 15.0 & 70.0 & 3.0 & $\\dots$ \\\\\n",
       "\t22 & 22.0 & 6.0 & 198.0 & 95.0 & 2833.0 & 15.5 & 70.0 & 1.0 & $\\dots$ \\\\\n",
       "\t23 & 18.0 & 6.0 & 199.0 & 97.0 & 2774.0 & 15.5 & 70.0 & 1.0 & $\\dots$ \\\\\n",
       "\t24 & 21.0 & 6.0 & 200.0 & 85.0 & 2587.0 & 16.0 & 70.0 & 1.0 & $\\dots$ \\\\\n",
       "\t25 & 27.0 & 4.0 & 97.0 & 88.0 & 2130.0 & 14.5 & 70.0 & 3.0 & $\\dots$ \\\\\n",
       "\t26 & 26.0 & 4.0 & 97.0 & 46.0 & 1835.0 & 20.5 & 70.0 & 2.0 & $\\dots$ \\\\\n",
       "\t27 & 25.0 & 4.0 & 110.0 & 87.0 & 2672.0 & 17.5 & 70.0 & 2.0 & $\\dots$ \\\\\n",
       "\t28 & 24.0 & 4.0 & 107.0 & 90.0 & 2430.0 & 14.5 & 70.0 & 2.0 & $\\dots$ \\\\\n",
       "\t29 & 25.0 & 4.0 & 104.0 & 95.0 & 2375.0 & 17.5 & 70.0 & 2.0 & $\\dots$ \\\\\n",
       "\t30 & 26.0 & 4.0 & 121.0 & 113.0 & 2234.0 & 12.5 & 70.0 & 2.0 & $\\dots$ \\\\\n",
       "\t$\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ &  \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m406×9 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m mpg       \u001b[0m\u001b[1m cylinders \u001b[0m\u001b[1m displacement \u001b[0m\u001b[1m horsepower \u001b[0m\u001b[1m weight  \u001b[0m\u001b[1m acceleration \u001b[0m\u001b[1m \u001b[0m ⋯\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Float64?  \u001b[0m\u001b[90m Float64   \u001b[0m\u001b[90m Float64      \u001b[0m\u001b[90m Float64?   \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64      \u001b[0m\u001b[90m \u001b[0m ⋯\n",
       "─────┼──────────────────────────────────────────────────────────────────────────\n",
       "   1 │      18.0        8.0         307.0       130.0   3504.0          12.0   ⋯\n",
       "   2 │      15.0        8.0         350.0       165.0   3693.0          11.5\n",
       "   3 │      18.0        8.0         318.0       150.0   3436.0          11.0\n",
       "   4 │      16.0        8.0         304.0       150.0   3433.0          12.0\n",
       "   5 │      17.0        8.0         302.0       140.0   3449.0          10.5   ⋯\n",
       "   6 │      15.0        8.0         429.0       198.0   4341.0          10.0\n",
       "   7 │      14.0        8.0         454.0       220.0   4354.0           9.0\n",
       "   8 │      14.0        8.0         440.0       215.0   4312.0           8.5\n",
       "   9 │      14.0        8.0         455.0       225.0   4425.0          10.0   ⋯\n",
       "  10 │      15.0        8.0         390.0       190.0   3850.0           8.5\n",
       "  11 │\u001b[90m missing   \u001b[0m       4.0         133.0       115.0   3090.0          17.5\n",
       "  ⋮  │     ⋮          ⋮           ⋮            ⋮          ⋮          ⋮         ⋱\n",
       " 397 │      26.0        4.0         156.0        92.0   2585.0          14.5\n",
       " 398 │      22.0        6.0         232.0       112.0   2835.0          14.7   ⋯\n",
       " 399 │      32.0        4.0         144.0        96.0   2665.0          13.9\n",
       " 400 │      36.0        4.0         135.0        84.0   2370.0          13.0\n",
       " 401 │      27.0        4.0         151.0        90.0   2950.0          17.3\n",
       " 402 │      27.0        4.0         140.0        86.0   2790.0          15.6   ⋯\n",
       " 403 │      44.0        4.0          97.0        52.0   2130.0          24.6\n",
       " 404 │      32.0        4.0         135.0        84.0   2295.0          11.6\n",
       " 405 │      28.0        4.0         120.0        79.0   2625.0          18.6\n",
       " 406 │      31.0        4.0         119.0        82.0   2720.0          19.4   ⋯\n",
       "\u001b[36m                                                  3 columns and 385 rows omitted\u001b[0m"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1 = CSV.File(io,\n",
    "               delim=' ',\n",
    "               ignorerepeated=true,\n",
    "               header=[:mpg, :cylinders, :displacement, :horsepower,\n",
    "                       :weight, :acceleration, :year, :origin, :name],\n",
    "               missingstring=\"NA\") |>\n",
    "      DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that not all columns of the data frame have been displayed and we see 30 rows (they do not fit on my screen).\n",
    "\n",
    "It is easy to change the default maximum width and height of the output by setting appropriate values in `ENV` dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(200, 15)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ENV[\"COLUMNS\"], ENV[\"LINES\"] = 200, 15"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class=\"data-frame\"><p>406 rows × 9 columns</p><table class=\"data-frame\"><thead><tr><th></th><th>mpg</th><th>cylinders</th><th>displacement</th><th>horsepower</th><th>weight</th><th>acceleration</th><th>year</th><th>origin</th><th>name</th></tr><tr><th></th><th title=\"Union{Missing, Float64}\">Float64?</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Union{Missing, Float64}\">Float64?</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"String63\">String63</th></tr></thead><tbody><tr><th>1</th><td>18.0</td><td>8.0</td><td>307.0</td><td>130.0</td><td>3504.0</td><td>12.0</td><td>70.0</td><td>1.0</td><td>chevrolet chevelle malibu</td></tr><tr><th>2</th><td>15.0</td><td>8.0</td><td>350.0</td><td>165.0</td><td>3693.0</td><td>11.5</td><td>70.0</td><td>1.0</td><td>buick skylark 320</td></tr><tr><th>3</th><td>18.0</td><td>8.0</td><td>318.0</td><td>150.0</td><td>3436.0</td><td>11.0</td><td>70.0</td><td>1.0</td><td>plymouth satellite</td></tr><tr><th>4</th><td>16.0</td><td>8.0</td><td>304.0</td><td>150.0</td><td>3433.0</td><td>12.0</td><td>70.0</td><td>1.0</td><td>amc rebel sst</td></tr><tr><th>5</th><td>17.0</td><td>8.0</td><td>302.0</td><td>140.0</td><td>3449.0</td><td>10.5</td><td>70.0</td><td>1.0</td><td>ford torino</td></tr><tr><th>6</th><td>15.0</td><td>8.0</td><td>429.0</td><td>198.0</td><td>4341.0</td><td>10.0</td><td>70.0</td><td>1.0</td><td>ford galaxie 500</td></tr><tr><th>7</th><td>14.0</td><td>8.0</td><td>454.0</td><td>220.0</td><td>4354.0</td><td>9.0</td><td>70.0</td><td>1.0</td><td>chevrolet impala</td></tr><tr><th>8</th><td>14.0</td><td>8.0</td><td>440.0</td><td>215.0</td><td>4312.0</td><td>8.5</td><td>70.0</td><td>1.0</td><td>plymouth fury iii</td></tr><tr><th>9</th><td>14.0</td><td>8.0</td><td>455.0</td><td>225.0</td><td>4425.0</td><td>10.0</td><td>70.0</td><td>1.0</td><td>pontiac catalina</td></tr><tr><th>10</th><td>15.0</td><td>8.0</td><td>390.0</td><td>190.0</td><td>3850.0</td><td>8.5</td><td>70.0</td><td>1.0</td><td>amc ambassador dpl</td></tr><tr><th>11</th><td><em>missing</em></td><td>4.0</td><td>133.0</td><td>115.0</td><td>3090.0</td><td>17.5</td><td>70.0</td><td>2.0</td><td>citroen ds-21 pallas</td></tr><tr><th>12</th><td><em>missing</em></td><td>8.0</td><td>350.0</td><td>165.0</td><td>4142.0</td><td>11.5</td><td>70.0</td><td>1.0</td><td>chevrolet chevelle concours (sw)</td></tr><tr><th>13</th><td><em>missing</em></td><td>8.0</td><td>351.0</td><td>153.0</td><td>4034.0</td><td>11.0</td><td>70.0</td><td>1.0</td><td>ford torino (sw)</td></tr><tr><th>14</th><td><em>missing</em></td><td>8.0</td><td>383.0</td><td>175.0</td><td>4166.0</td><td>10.5</td><td>70.0</td><td>1.0</td><td>plymouth satellite (sw)</td></tr><tr><th>15</th><td><em>missing</em></td><td>8.0</td><td>360.0</td><td>175.0</td><td>3850.0</td><td>11.0</td><td>70.0</td><td>1.0</td><td>amc rebel sst (sw)</td></tr><tr><th>&vellip;</th><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td></tr></tbody></table></div>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccccccccc}\n",
       "\t& mpg & cylinders & displacement & horsepower & weight & acceleration & year & origin & name\\\\\n",
       "\t\\hline\n",
       "\t& Float64? & Float64 & Float64 & Float64? & Float64 & Float64 & Float64 & Float64 & String63\\\\\n",
       "\t\\hline\n",
       "\t1 & 18.0 & 8.0 & 307.0 & 130.0 & 3504.0 & 12.0 & 70.0 & 1.0 & chevrolet chevelle malibu \\\\\n",
       "\t2 & 15.0 & 8.0 & 350.0 & 165.0 & 3693.0 & 11.5 & 70.0 & 1.0 & buick skylark 320 \\\\\n",
       "\t3 & 18.0 & 8.0 & 318.0 & 150.0 & 3436.0 & 11.0 & 70.0 & 1.0 & plymouth satellite \\\\\n",
       "\t4 & 16.0 & 8.0 & 304.0 & 150.0 & 3433.0 & 12.0 & 70.0 & 1.0 & amc rebel sst \\\\\n",
       "\t5 & 17.0 & 8.0 & 302.0 & 140.0 & 3449.0 & 10.5 & 70.0 & 1.0 & ford torino \\\\\n",
       "\t6 & 15.0 & 8.0 & 429.0 & 198.0 & 4341.0 & 10.0 & 70.0 & 1.0 & ford galaxie 500 \\\\\n",
       "\t7 & 14.0 & 8.0 & 454.0 & 220.0 & 4354.0 & 9.0 & 70.0 & 1.0 & chevrolet impala \\\\\n",
       "\t8 & 14.0 & 8.0 & 440.0 & 215.0 & 4312.0 & 8.5 & 70.0 & 1.0 & plymouth fury iii \\\\\n",
       "\t9 & 14.0 & 8.0 & 455.0 & 225.0 & 4425.0 & 10.0 & 70.0 & 1.0 & pontiac catalina \\\\\n",
       "\t10 & 15.0 & 8.0 & 390.0 & 190.0 & 3850.0 & 8.5 & 70.0 & 1.0 & amc ambassador dpl \\\\\n",
       "\t11 & \\emph{missing} & 4.0 & 133.0 & 115.0 & 3090.0 & 17.5 & 70.0 & 2.0 & citroen ds-21 pallas \\\\\n",
       "\t12 & \\emph{missing} & 8.0 & 350.0 & 165.0 & 4142.0 & 11.5 & 70.0 & 1.0 & chevrolet chevelle concours (sw) \\\\\n",
       "\t13 & \\emph{missing} & 8.0 & 351.0 & 153.0 & 4034.0 & 11.0 & 70.0 & 1.0 & ford torino (sw) \\\\\n",
       "\t14 & \\emph{missing} & 8.0 & 383.0 & 175.0 & 4166.0 & 10.5 & 70.0 & 1.0 & plymouth satellite (sw) \\\\\n",
       "\t15 & \\emph{missing} & 8.0 & 360.0 & 175.0 & 3850.0 & 11.0 & 70.0 & 1.0 & amc rebel sst (sw) \\\\\n",
       "\t$\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m406×9 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m mpg      \u001b[0m\u001b[1m cylinders \u001b[0m\u001b[1m displacement \u001b[0m\u001b[1m horsepower \u001b[0m\u001b[1m weight  \u001b[0m\u001b[1m acceleration \u001b[0m\u001b[1m year    \u001b[0m\u001b[1m origin  \u001b[0m\u001b[1m name                      \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Float64? \u001b[0m\u001b[90m Float64   \u001b[0m\u001b[90m Float64      \u001b[0m\u001b[90m Float64?   \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64      \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m String63                  \u001b[0m\n",
       "─────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n",
       "   1 │     18.0        8.0         307.0       130.0   3504.0          12.0     70.0      1.0  chevrolet chevelle malibu\n",
       "   2 │     15.0        8.0         350.0       165.0   3693.0          11.5     70.0      1.0  buick skylark 320\n",
       "   3 │     18.0        8.0         318.0       150.0   3436.0          11.0     70.0      1.0  plymouth satellite\n",
       "  ⋮  │    ⋮          ⋮           ⋮            ⋮          ⋮          ⋮           ⋮        ⋮                 ⋮\n",
       " 404 │     32.0        4.0         135.0        84.0   2295.0          11.6     82.0      1.0  dodge rampage\n",
       " 405 │     28.0        4.0         120.0        79.0   2625.0          18.6     82.0      1.0  ford ranger\n",
       " 406 │     31.0        4.0         119.0        82.0   2720.0          19.4     82.0      1.0  chevy s-10\n",
       "\u001b[36m                                                                                                         400 rows omitted\u001b[0m"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the display is much nicer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us discuss an alternative way to read in the original file.\n",
    "This time we will first read the data in directly from the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class=\"data-frame\"><p>406 rows × 2 columns</p><table class=\"data-frame\"><thead><tr><th></th><th>metrics</th><th>name</th></tr><tr><th></th><th title=\"String\">String</th><th title=\"String63\">String63</th></tr></thead><tbody><tr><th>1</th><td>18.0   8.   307.0      130.0      3504.      12.0   70.  1.</td><td>chevrolet chevelle malibu</td></tr><tr><th>2</th><td>15.0   8.   350.0      165.0      3693.      11.5   70.  1.</td><td>buick skylark 320</td></tr><tr><th>3</th><td>18.0   8.   318.0      150.0      3436.      11.0   70.  1.</td><td>plymouth satellite</td></tr><tr><th>4</th><td>16.0   8.   304.0      150.0      3433.      12.0   70.  1.</td><td>amc rebel sst</td></tr><tr><th>5</th><td>17.0   8.   302.0      140.0      3449.      10.5   70.  1.</td><td>ford torino</td></tr><tr><th>6</th><td>15.0   8.   429.0      198.0      4341.      10.0   70.  1.</td><td>ford galaxie 500</td></tr><tr><th>7</th><td>14.0   8.   454.0      220.0      4354.       9.0   70.  1.</td><td>chevrolet impala</td></tr><tr><th>8</th><td>14.0   8.   440.0      215.0      4312.       8.5   70.  1.</td><td>plymouth fury iii</td></tr><tr><th>9</th><td>14.0   8.   455.0      225.0      4425.      10.0   70.  1.</td><td>pontiac catalina</td></tr><tr><th>10</th><td>15.0   8.   390.0      190.0      3850.       8.5   70.  1.</td><td>amc ambassador dpl</td></tr><tr><th>11</th><td>NA     4.   133.0      115.0      3090.      17.5   70.  2.</td><td>citroen ds-21 pallas</td></tr><tr><th>12</th><td>NA     8.   350.0      165.0      4142.      11.5   70.  1.</td><td>chevrolet chevelle concours (sw)</td></tr><tr><th>13</th><td>NA     8.   351.0      153.0      4034.      11.0   70.  1.</td><td>ford torino (sw)</td></tr><tr><th>14</th><td>NA     8.   383.0      175.0      4166.      10.5   70.  1.</td><td>plymouth satellite (sw)</td></tr><tr><th>15</th><td>NA     8.   360.0      175.0      3850.      11.0   70.  1.</td><td>amc rebel sst (sw)</td></tr><tr><th>&vellip;</th><td>&vellip;</td><td>&vellip;</td></tr></tbody></table></div>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cc}\n",
       "\t& metrics & name\\\\\n",
       "\t\\hline\n",
       "\t& String & String63\\\\\n",
       "\t\\hline\n",
       "\t1 & 18.0   8.   307.0      130.0      3504.      12.0   70.  1. & chevrolet chevelle malibu \\\\\n",
       "\t2 & 15.0   8.   350.0      165.0      3693.      11.5   70.  1. & buick skylark 320 \\\\\n",
       "\t3 & 18.0   8.   318.0      150.0      3436.      11.0   70.  1. & plymouth satellite \\\\\n",
       "\t4 & 16.0   8.   304.0      150.0      3433.      12.0   70.  1. & amc rebel sst \\\\\n",
       "\t5 & 17.0   8.   302.0      140.0      3449.      10.5   70.  1. & ford torino \\\\\n",
       "\t6 & 15.0   8.   429.0      198.0      4341.      10.0   70.  1. & ford galaxie 500 \\\\\n",
       "\t7 & 14.0   8.   454.0      220.0      4354.       9.0   70.  1. & chevrolet impala \\\\\n",
       "\t8 & 14.0   8.   440.0      215.0      4312.       8.5   70.  1. & plymouth fury iii \\\\\n",
       "\t9 & 14.0   8.   455.0      225.0      4425.      10.0   70.  1. & pontiac catalina \\\\\n",
       "\t10 & 15.0   8.   390.0      190.0      3850.       8.5   70.  1. & amc ambassador dpl \\\\\n",
       "\t11 & NA     4.   133.0      115.0      3090.      17.5   70.  2. & citroen ds-21 pallas \\\\\n",
       "\t12 & NA     8.   350.0      165.0      4142.      11.5   70.  1. & chevrolet chevelle concours (sw) \\\\\n",
       "\t13 & NA     8.   351.0      153.0      4034.      11.0   70.  1. & ford torino (sw) \\\\\n",
       "\t14 & NA     8.   383.0      175.0      4166.      10.5   70.  1. & plymouth satellite (sw) \\\\\n",
       "\t15 & NA     8.   360.0      175.0      3850.      11.0   70.  1. & amc rebel sst (sw) \\\\\n",
       "\t$\\dots$ & $\\dots$ & $\\dots$ \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m406×2 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m metrics                           \u001b[0m\u001b[1m name                      \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m String                            \u001b[0m\u001b[90m String63                  \u001b[0m\n",
       "─────┼──────────────────────────────────────────────────────────────\n",
       "   1 │ 18.0   8.   307.0      130.0    …  chevrolet chevelle malibu\n",
       "   2 │ 15.0   8.   350.0      165.0    …  buick skylark 320\n",
       "   3 │ 18.0   8.   318.0      150.0    …  plymouth satellite\n",
       "  ⋮  │                 ⋮                              ⋮\n",
       " 404 │ 32.0   4.   135.0      84.00    …  dodge rampage\n",
       " 405 │ 28.0   4.   120.0      79.00    …  ford ranger\n",
       " 406 │ 31.0   4.   119.0      82.00    …  chevy s-10\n",
       "\u001b[36m                                                    400 rows omitted\u001b[0m"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_raw = CSV.File(\"auto.txt\", header=[:metrics, :name]) |> DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that this time CSV.jl auto-detected that tab is the right delimiter to split the columns\n",
    "\n",
    "(it was the only delimiter that produced consistent number of columns)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will split `:metrics` column manually now"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "406-element Vector{Vector{SubString{String}}}:\n",
       " [\"18.0\", \"8.\", \"307.0\", \"130.0\", \"3504.\", \"12.0\", \"70.\", \"1.\"]\n",
       " [\"15.0\", \"8.\", \"350.0\", \"165.0\", \"3693.\", \"11.5\", \"70.\", \"1.\"]\n",
       " [\"18.0\", \"8.\", \"318.0\", \"150.0\", \"3436.\", \"11.0\", \"70.\", \"1.\"]\n",
       " [\"16.0\", \"8.\", \"304.0\", \"150.0\", \"3433.\", \"12.0\", \"70.\", \"1.\"]\n",
       " [\"17.0\", \"8.\", \"302.0\", \"140.0\", \"3449.\", \"10.5\", \"70.\", \"1.\"]\n",
       " ⋮\n",
       " [\"27.0\", \"4.\", \"140.0\", \"86.00\", \"2790.\", \"15.6\", \"82.\", \"1.\"]\n",
       " [\"44.0\", \"4.\", \"97.00\", \"52.00\", \"2130.\", \"24.6\", \"82.\", \"2.\"]\n",
       " [\"32.0\", \"4.\", \"135.0\", \"84.00\", \"2295.\", \"11.6\", \"82.\", \"1.\"]\n",
       " [\"28.0\", \"4.\", \"120.0\", \"79.00\", \"2625.\", \"18.6\", \"82.\", \"1.\"]\n",
       " [\"31.0\", \"4.\", \"119.0\", \"82.00\", \"2720.\", \"19.4\", \"82.\", \"1.\"]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "str_metrics = split.(df_raw.metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let us create an empty `df1_2` data frame that we will populate with appropriate columns.\n",
    "The pattern we use here is typical when you e.g. perform repeated computations whose results you want to store in a `DataFrame`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class=\"data-frame\"><p>0 rows × 8 columns</p><table class=\"data-frame\"><thead><tr><th></th><th>mpg</th><th>cylinders</th><th>displacement</th><th>horsepower</th><th>weight</th><th>acceleration</th><th>year</th><th>origin</th></tr><tr><th></th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th></tr></thead><tbody></tbody></table></div>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccccccc}\n",
       "\t& mpg & cylinders & displacement & horsepower & weight & acceleration & year & origin\\\\\n",
       "\t\\hline\n",
       "\t& Float64 & Float64 & Float64 & Float64 & Float64 & Float64 & Float64 & Float64\\\\\n",
       "\t\\hline\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m0×8 DataFrame\u001b[0m"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1_2 = DataFrame([col => Float64[] for\n",
    "                  col in [:mpg, :cylinders, :displacement, :horsepower, :weight, :acceleration, :year, :origin]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have a data frame that has 8 columns and 0 rows. It accepts floating point values. However in columns `:mpg` and `:horsepower` we have to allow the data frame to hold missing values. We do it using `allowmissing!` function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class=\"data-frame\"><p>0 rows × 8 columns</p><table class=\"data-frame\"><thead><tr><th></th><th>mpg</th><th>cylinders</th><th>displacement</th><th>horsepower</th><th>weight</th><th>acceleration</th><th>year</th><th>origin</th></tr><tr><th></th><th title=\"Union{Missing, Float64}\">Float64?</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Union{Missing, Float64}\">Float64?</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th></tr></thead><tbody></tbody></table></div>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccccccc}\n",
       "\t& mpg & cylinders & displacement & horsepower & weight & acceleration & year & origin\\\\\n",
       "\t\\hline\n",
       "\t& Float64? & Float64 & Float64 & Float64? & Float64 & Float64 & Float64 & Float64\\\\\n",
       "\t\\hline\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m0×8 DataFrame\u001b[0m"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "allowmissing!(df1_2, [:mpg, :horsepower])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the element type of columns `:mpg` and `:horsepower` changed to `Float64?` which signals that that columns allows missing values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are ready to populate our data frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "for row in str_metrics\n",
    "    push!(df1_2, [v == \"NA\" ? missing : parse(Float64, v) for v in row])\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class=\"data-frame\"><p>406 rows × 8 columns</p><table class=\"data-frame\"><thead><tr><th></th><th>mpg</th><th>cylinders</th><th>displacement</th><th>horsepower</th><th>weight</th><th>acceleration</th><th>year</th><th>origin</th></tr><tr><th></th><th title=\"Union{Missing, Float64}\">Float64?</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Union{Missing, Float64}\">Float64?</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th></tr></thead><tbody><tr><th>1</th><td>18.0</td><td>8.0</td><td>307.0</td><td>130.0</td><td>3504.0</td><td>12.0</td><td>70.0</td><td>1.0</td></tr><tr><th>2</th><td>15.0</td><td>8.0</td><td>350.0</td><td>165.0</td><td>3693.0</td><td>11.5</td><td>70.0</td><td>1.0</td></tr><tr><th>3</th><td>18.0</td><td>8.0</td><td>318.0</td><td>150.0</td><td>3436.0</td><td>11.0</td><td>70.0</td><td>1.0</td></tr><tr><th>4</th><td>16.0</td><td>8.0</td><td>304.0</td><td>150.0</td><td>3433.0</td><td>12.0</td><td>70.0</td><td>1.0</td></tr><tr><th>5</th><td>17.0</td><td>8.0</td><td>302.0</td><td>140.0</td><td>3449.0</td><td>10.5</td><td>70.0</td><td>1.0</td></tr><tr><th>6</th><td>15.0</td><td>8.0</td><td>429.0</td><td>198.0</td><td>4341.0</td><td>10.0</td><td>70.0</td><td>1.0</td></tr><tr><th>7</th><td>14.0</td><td>8.0</td><td>454.0</td><td>220.0</td><td>4354.0</td><td>9.0</td><td>70.0</td><td>1.0</td></tr><tr><th>8</th><td>14.0</td><td>8.0</td><td>440.0</td><td>215.0</td><td>4312.0</td><td>8.5</td><td>70.0</td><td>1.0</td></tr><tr><th>9</th><td>14.0</td><td>8.0</td><td>455.0</td><td>225.0</td><td>4425.0</td><td>10.0</td><td>70.0</td><td>1.0</td></tr><tr><th>10</th><td>15.0</td><td>8.0</td><td>390.0</td><td>190.0</td><td>3850.0</td><td>8.5</td><td>70.0</td><td>1.0</td></tr><tr><th>11</th><td><em>missing</em></td><td>4.0</td><td>133.0</td><td>115.0</td><td>3090.0</td><td>17.5</td><td>70.0</td><td>2.0</td></tr><tr><th>12</th><td><em>missing</em></td><td>8.0</td><td>350.0</td><td>165.0</td><td>4142.0</td><td>11.5</td><td>70.0</td><td>1.0</td></tr><tr><th>13</th><td><em>missing</em></td><td>8.0</td><td>351.0</td><td>153.0</td><td>4034.0</td><td>11.0</td><td>70.0</td><td>1.0</td></tr><tr><th>14</th><td><em>missing</em></td><td>8.0</td><td>383.0</td><td>175.0</td><td>4166.0</td><td>10.5</td><td>70.0</td><td>1.0</td></tr><tr><th>15</th><td><em>missing</em></td><td>8.0</td><td>360.0</td><td>175.0</td><td>3850.0</td><td>11.0</td><td>70.0</td><td>1.0</td></tr><tr><th>&vellip;</th><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td></tr></tbody></table></div>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccccccc}\n",
       "\t& mpg & cylinders & displacement & horsepower & weight & acceleration & year & origin\\\\\n",
       "\t\\hline\n",
       "\t& Float64? & Float64 & Float64 & Float64? & Float64 & Float64 & Float64 & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & 18.0 & 8.0 & 307.0 & 130.0 & 3504.0 & 12.0 & 70.0 & 1.0 \\\\\n",
       "\t2 & 15.0 & 8.0 & 350.0 & 165.0 & 3693.0 & 11.5 & 70.0 & 1.0 \\\\\n",
       "\t3 & 18.0 & 8.0 & 318.0 & 150.0 & 3436.0 & 11.0 & 70.0 & 1.0 \\\\\n",
       "\t4 & 16.0 & 8.0 & 304.0 & 150.0 & 3433.0 & 12.0 & 70.0 & 1.0 \\\\\n",
       "\t5 & 17.0 & 8.0 & 302.0 & 140.0 & 3449.0 & 10.5 & 70.0 & 1.0 \\\\\n",
       "\t6 & 15.0 & 8.0 & 429.0 & 198.0 & 4341.0 & 10.0 & 70.0 & 1.0 \\\\\n",
       "\t7 & 14.0 & 8.0 & 454.0 & 220.0 & 4354.0 & 9.0 & 70.0 & 1.0 \\\\\n",
       "\t8 & 14.0 & 8.0 & 440.0 & 215.0 & 4312.0 & 8.5 & 70.0 & 1.0 \\\\\n",
       "\t9 & 14.0 & 8.0 & 455.0 & 225.0 & 4425.0 & 10.0 & 70.0 & 1.0 \\\\\n",
       "\t10 & 15.0 & 8.0 & 390.0 & 190.0 & 3850.0 & 8.5 & 70.0 & 1.0 \\\\\n",
       "\t11 & \\emph{missing} & 4.0 & 133.0 & 115.0 & 3090.0 & 17.5 & 70.0 & 2.0 \\\\\n",
       "\t12 & \\emph{missing} & 8.0 & 350.0 & 165.0 & 4142.0 & 11.5 & 70.0 & 1.0 \\\\\n",
       "\t13 & \\emph{missing} & 8.0 & 351.0 & 153.0 & 4034.0 & 11.0 & 70.0 & 1.0 \\\\\n",
       "\t14 & \\emph{missing} & 8.0 & 383.0 & 175.0 & 4166.0 & 10.5 & 70.0 & 1.0 \\\\\n",
       "\t15 & \\emph{missing} & 8.0 & 360.0 & 175.0 & 3850.0 & 11.0 & 70.0 & 1.0 \\\\\n",
       "\t$\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m406×8 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m mpg      \u001b[0m\u001b[1m cylinders \u001b[0m\u001b[1m displacement \u001b[0m\u001b[1m horsepower \u001b[0m\u001b[1m weight  \u001b[0m\u001b[1m acceleration \u001b[0m\u001b[1m year    \u001b[0m\u001b[1m origin  \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Float64? \u001b[0m\u001b[90m Float64   \u001b[0m\u001b[90m Float64      \u001b[0m\u001b[90m Float64?   \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64      \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64 \u001b[0m\n",
       "─────┼────────────────────────────────────────────────────────────────────────────────────────\n",
       "   1 │     18.0        8.0         307.0       130.0   3504.0          12.0     70.0      1.0\n",
       "   2 │     15.0        8.0         350.0       165.0   3693.0          11.5     70.0      1.0\n",
       "   3 │     18.0        8.0         318.0       150.0   3436.0          11.0     70.0      1.0\n",
       "  ⋮  │    ⋮          ⋮           ⋮            ⋮          ⋮          ⋮           ⋮        ⋮\n",
       " 404 │     32.0        4.0         135.0        84.0   2295.0          11.6     82.0      1.0\n",
       " 405 │     28.0        4.0         120.0        79.0   2625.0          18.6     82.0      1.0\n",
       " 406 │     31.0        4.0         119.0        82.0   2720.0          19.4     82.0      1.0\n",
       "\u001b[36m                                                                              400 rows omitted\u001b[0m"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1_2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "finally, let us add a column `:name` from the `df_raw` data frame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "406-element Vector{String63}:\n",
       " \"chevrolet chevelle malibu\"\n",
       " \"buick skylark 320\"\n",
       " \"plymouth satellite\"\n",
       " \"amc rebel sst\"\n",
       " \"ford torino\"\n",
       " ⋮\n",
       " \"ford mustang gl\"\n",
       " \"vw pickup\"\n",
       " \"dodge rampage\"\n",
       " \"ford ranger\"\n",
       " \"chevy s-10\""
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1_2.name = df_raw.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class=\"data-frame\"><p>406 rows × 9 columns</p><table class=\"data-frame\"><thead><tr><th></th><th>mpg</th><th>cylinders</th><th>displacement</th><th>horsepower</th><th>weight</th><th>acceleration</th><th>year</th><th>origin</th><th>name</th></tr><tr><th></th><th title=\"Union{Missing, Float64}\">Float64?</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Union{Missing, Float64}\">Float64?</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"String63\">String63</th></tr></thead><tbody><tr><th>1</th><td>18.0</td><td>8.0</td><td>307.0</td><td>130.0</td><td>3504.0</td><td>12.0</td><td>70.0</td><td>1.0</td><td>chevrolet chevelle malibu</td></tr><tr><th>2</th><td>15.0</td><td>8.0</td><td>350.0</td><td>165.0</td><td>3693.0</td><td>11.5</td><td>70.0</td><td>1.0</td><td>buick skylark 320</td></tr><tr><th>3</th><td>18.0</td><td>8.0</td><td>318.0</td><td>150.0</td><td>3436.0</td><td>11.0</td><td>70.0</td><td>1.0</td><td>plymouth satellite</td></tr><tr><th>4</th><td>16.0</td><td>8.0</td><td>304.0</td><td>150.0</td><td>3433.0</td><td>12.0</td><td>70.0</td><td>1.0</td><td>amc rebel sst</td></tr><tr><th>5</th><td>17.0</td><td>8.0</td><td>302.0</td><td>140.0</td><td>3449.0</td><td>10.5</td><td>70.0</td><td>1.0</td><td>ford torino</td></tr><tr><th>6</th><td>15.0</td><td>8.0</td><td>429.0</td><td>198.0</td><td>4341.0</td><td>10.0</td><td>70.0</td><td>1.0</td><td>ford galaxie 500</td></tr><tr><th>7</th><td>14.0</td><td>8.0</td><td>454.0</td><td>220.0</td><td>4354.0</td><td>9.0</td><td>70.0</td><td>1.0</td><td>chevrolet impala</td></tr><tr><th>8</th><td>14.0</td><td>8.0</td><td>440.0</td><td>215.0</td><td>4312.0</td><td>8.5</td><td>70.0</td><td>1.0</td><td>plymouth fury iii</td></tr><tr><th>9</th><td>14.0</td><td>8.0</td><td>455.0</td><td>225.0</td><td>4425.0</td><td>10.0</td><td>70.0</td><td>1.0</td><td>pontiac catalina</td></tr><tr><th>10</th><td>15.0</td><td>8.0</td><td>390.0</td><td>190.0</td><td>3850.0</td><td>8.5</td><td>70.0</td><td>1.0</td><td>amc ambassador dpl</td></tr><tr><th>11</th><td><em>missing</em></td><td>4.0</td><td>133.0</td><td>115.0</td><td>3090.0</td><td>17.5</td><td>70.0</td><td>2.0</td><td>citroen ds-21 pallas</td></tr><tr><th>12</th><td><em>missing</em></td><td>8.0</td><td>350.0</td><td>165.0</td><td>4142.0</td><td>11.5</td><td>70.0</td><td>1.0</td><td>chevrolet chevelle concours (sw)</td></tr><tr><th>13</th><td><em>missing</em></td><td>8.0</td><td>351.0</td><td>153.0</td><td>4034.0</td><td>11.0</td><td>70.0</td><td>1.0</td><td>ford torino (sw)</td></tr><tr><th>14</th><td><em>missing</em></td><td>8.0</td><td>383.0</td><td>175.0</td><td>4166.0</td><td>10.5</td><td>70.0</td><td>1.0</td><td>plymouth satellite (sw)</td></tr><tr><th>15</th><td><em>missing</em></td><td>8.0</td><td>360.0</td><td>175.0</td><td>3850.0</td><td>11.0</td><td>70.0</td><td>1.0</td><td>amc rebel sst (sw)</td></tr><tr><th>&vellip;</th><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td></tr></tbody></table></div>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccccccccc}\n",
       "\t& mpg & cylinders & displacement & horsepower & weight & acceleration & year & origin & name\\\\\n",
       "\t\\hline\n",
       "\t& Float64? & Float64 & Float64 & Float64? & Float64 & Float64 & Float64 & Float64 & String63\\\\\n",
       "\t\\hline\n",
       "\t1 & 18.0 & 8.0 & 307.0 & 130.0 & 3504.0 & 12.0 & 70.0 & 1.0 & chevrolet chevelle malibu \\\\\n",
       "\t2 & 15.0 & 8.0 & 350.0 & 165.0 & 3693.0 & 11.5 & 70.0 & 1.0 & buick skylark 320 \\\\\n",
       "\t3 & 18.0 & 8.0 & 318.0 & 150.0 & 3436.0 & 11.0 & 70.0 & 1.0 & plymouth satellite \\\\\n",
       "\t4 & 16.0 & 8.0 & 304.0 & 150.0 & 3433.0 & 12.0 & 70.0 & 1.0 & amc rebel sst \\\\\n",
       "\t5 & 17.0 & 8.0 & 302.0 & 140.0 & 3449.0 & 10.5 & 70.0 & 1.0 & ford torino \\\\\n",
       "\t6 & 15.0 & 8.0 & 429.0 & 198.0 & 4341.0 & 10.0 & 70.0 & 1.0 & ford galaxie 500 \\\\\n",
       "\t7 & 14.0 & 8.0 & 454.0 & 220.0 & 4354.0 & 9.0 & 70.0 & 1.0 & chevrolet impala \\\\\n",
       "\t8 & 14.0 & 8.0 & 440.0 & 215.0 & 4312.0 & 8.5 & 70.0 & 1.0 & plymouth fury iii \\\\\n",
       "\t9 & 14.0 & 8.0 & 455.0 & 225.0 & 4425.0 & 10.0 & 70.0 & 1.0 & pontiac catalina \\\\\n",
       "\t10 & 15.0 & 8.0 & 390.0 & 190.0 & 3850.0 & 8.5 & 70.0 & 1.0 & amc ambassador dpl \\\\\n",
       "\t11 & \\emph{missing} & 4.0 & 133.0 & 115.0 & 3090.0 & 17.5 & 70.0 & 2.0 & citroen ds-21 pallas \\\\\n",
       "\t12 & \\emph{missing} & 8.0 & 350.0 & 165.0 & 4142.0 & 11.5 & 70.0 & 1.0 & chevrolet chevelle concours (sw) \\\\\n",
       "\t13 & \\emph{missing} & 8.0 & 351.0 & 153.0 & 4034.0 & 11.0 & 70.0 & 1.0 & ford torino (sw) \\\\\n",
       "\t14 & \\emph{missing} & 8.0 & 383.0 & 175.0 & 4166.0 & 10.5 & 70.0 & 1.0 & plymouth satellite (sw) \\\\\n",
       "\t15 & \\emph{missing} & 8.0 & 360.0 & 175.0 & 3850.0 & 11.0 & 70.0 & 1.0 & amc rebel sst (sw) \\\\\n",
       "\t$\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m406×9 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m mpg      \u001b[0m\u001b[1m cylinders \u001b[0m\u001b[1m displacement \u001b[0m\u001b[1m horsepower \u001b[0m\u001b[1m weight  \u001b[0m\u001b[1m acceleration \u001b[0m\u001b[1m year    \u001b[0m\u001b[1m origin  \u001b[0m\u001b[1m name                      \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Float64? \u001b[0m\u001b[90m Float64   \u001b[0m\u001b[90m Float64      \u001b[0m\u001b[90m Float64?   \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64      \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m String63                  \u001b[0m\n",
       "─────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n",
       "   1 │     18.0        8.0         307.0       130.0   3504.0          12.0     70.0      1.0  chevrolet chevelle malibu\n",
       "   2 │     15.0        8.0         350.0       165.0   3693.0          11.5     70.0      1.0  buick skylark 320\n",
       "   3 │     18.0        8.0         318.0       150.0   3436.0          11.0     70.0      1.0  plymouth satellite\n",
       "  ⋮  │    ⋮          ⋮           ⋮            ⋮          ⋮          ⋮           ⋮        ⋮                 ⋮\n",
       " 404 │     32.0        4.0         135.0        84.0   2295.0          11.6     82.0      1.0  dodge rampage\n",
       " 405 │     28.0        4.0         120.0        79.0   2625.0          18.6     82.0      1.0  ford ranger\n",
       " 406 │     31.0        4.0         119.0        82.0   2720.0          19.4     82.0      1.0  chevy s-10\n",
       "\u001b[36m                                                                                                         400 rows omitted\u001b[0m"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1_2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we move forward we should stress one very important thing that is related to `df1_2.name = df_raw.name` assignment. After this operation columns `:name` in both `df1_2` and `df_raw` are the same objects.\n",
    "We can easily check it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "true"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1_2.name === df_raw.name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Such behavior is allowed for performance reasons. In this case we accepted it as we want to discard `df_raw` data frame and not use it in later analysis.\n",
    "However, in general it would be safer to create a copy of `:name` column when assigning it ti `df1_2` data frame.\n",
    "\n",
    "It can be achieved like this:\n",
    "```\n",
    "df1_2[:, :name] = df_raw.name\n",
    "```\n",
    "or like this\n",
    "```\n",
    "df1_2.name = df_raw[:, :name]\n",
    "```\n",
    "We could also write:\n",
    "```\n",
    "df1_2[:, :name] = df_raw[:, :name]\n",
    "```\n",
    "but this time there would be one unnecessary copy made (one when reading the data from `df_raw` the other when writing data to `df1_2`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can check that `df1` and `df1_2` data frames are equal using `isequal` funtion, so that we ended up with identical data frames."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "true"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "isequal(df1_2, df1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that as the data frames contain missing values comparing them with `missing` would produce `missing`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "missing"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1_2 == df1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us investigate yet another way to create our data frame, this time in one-shot (this line is more difficult to understand, but it shows you the power that DataFrames.jl gives you when working with data, in part 4 of the tutorial we will give some more examples of supported transformations):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class=\"data-frame\"><p>406 rows × 9 columns</p><table class=\"data-frame\"><thead><tr><th></th><th>mpg</th><th>cylinders</th><th>displacement</th><th>horsepower</th><th>weight</th><th>acceleration</th><th>year</th><th>origin</th><th>name</th></tr><tr><th></th><th title=\"Union{Missing, Float64}\">Float64?</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Union{Missing, Float64}\">Float64?</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"String63\">String63</th></tr></thead><tbody><tr><th>1</th><td>18.0</td><td>8.0</td><td>307.0</td><td>130.0</td><td>3504.0</td><td>12.0</td><td>70.0</td><td>1.0</td><td>chevrolet chevelle malibu</td></tr><tr><th>2</th><td>15.0</td><td>8.0</td><td>350.0</td><td>165.0</td><td>3693.0</td><td>11.5</td><td>70.0</td><td>1.0</td><td>buick skylark 320</td></tr><tr><th>3</th><td>18.0</td><td>8.0</td><td>318.0</td><td>150.0</td><td>3436.0</td><td>11.0</td><td>70.0</td><td>1.0</td><td>plymouth satellite</td></tr><tr><th>4</th><td>16.0</td><td>8.0</td><td>304.0</td><td>150.0</td><td>3433.0</td><td>12.0</td><td>70.0</td><td>1.0</td><td>amc rebel sst</td></tr><tr><th>5</th><td>17.0</td><td>8.0</td><td>302.0</td><td>140.0</td><td>3449.0</td><td>10.5</td><td>70.0</td><td>1.0</td><td>ford torino</td></tr><tr><th>6</th><td>15.0</td><td>8.0</td><td>429.0</td><td>198.0</td><td>4341.0</td><td>10.0</td><td>70.0</td><td>1.0</td><td>ford galaxie 500</td></tr><tr><th>7</th><td>14.0</td><td>8.0</td><td>454.0</td><td>220.0</td><td>4354.0</td><td>9.0</td><td>70.0</td><td>1.0</td><td>chevrolet impala</td></tr><tr><th>8</th><td>14.0</td><td>8.0</td><td>440.0</td><td>215.0</td><td>4312.0</td><td>8.5</td><td>70.0</td><td>1.0</td><td>plymouth fury iii</td></tr><tr><th>9</th><td>14.0</td><td>8.0</td><td>455.0</td><td>225.0</td><td>4425.0</td><td>10.0</td><td>70.0</td><td>1.0</td><td>pontiac catalina</td></tr><tr><th>10</th><td>15.0</td><td>8.0</td><td>390.0</td><td>190.0</td><td>3850.0</td><td>8.5</td><td>70.0</td><td>1.0</td><td>amc ambassador dpl</td></tr><tr><th>11</th><td><em>missing</em></td><td>4.0</td><td>133.0</td><td>115.0</td><td>3090.0</td><td>17.5</td><td>70.0</td><td>2.0</td><td>citroen ds-21 pallas</td></tr><tr><th>12</th><td><em>missing</em></td><td>8.0</td><td>350.0</td><td>165.0</td><td>4142.0</td><td>11.5</td><td>70.0</td><td>1.0</td><td>chevrolet chevelle concours (sw)</td></tr><tr><th>13</th><td><em>missing</em></td><td>8.0</td><td>351.0</td><td>153.0</td><td>4034.0</td><td>11.0</td><td>70.0</td><td>1.0</td><td>ford torino (sw)</td></tr><tr><th>14</th><td><em>missing</em></td><td>8.0</td><td>383.0</td><td>175.0</td><td>4166.0</td><td>10.5</td><td>70.0</td><td>1.0</td><td>plymouth satellite (sw)</td></tr><tr><th>15</th><td><em>missing</em></td><td>8.0</td><td>360.0</td><td>175.0</td><td>3850.0</td><td>11.0</td><td>70.0</td><td>1.0</td><td>amc rebel sst (sw)</td></tr><tr><th>&vellip;</th><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td></tr></tbody></table></div>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccccccccc}\n",
       "\t& mpg & cylinders & displacement & horsepower & weight & acceleration & year & origin & name\\\\\n",
       "\t\\hline\n",
       "\t& Float64? & Float64 & Float64 & Float64? & Float64 & Float64 & Float64 & Float64 & String63\\\\\n",
       "\t\\hline\n",
       "\t1 & 18.0 & 8.0 & 307.0 & 130.0 & 3504.0 & 12.0 & 70.0 & 1.0 & chevrolet chevelle malibu \\\\\n",
       "\t2 & 15.0 & 8.0 & 350.0 & 165.0 & 3693.0 & 11.5 & 70.0 & 1.0 & buick skylark 320 \\\\\n",
       "\t3 & 18.0 & 8.0 & 318.0 & 150.0 & 3436.0 & 11.0 & 70.0 & 1.0 & plymouth satellite \\\\\n",
       "\t4 & 16.0 & 8.0 & 304.0 & 150.0 & 3433.0 & 12.0 & 70.0 & 1.0 & amc rebel sst \\\\\n",
       "\t5 & 17.0 & 8.0 & 302.0 & 140.0 & 3449.0 & 10.5 & 70.0 & 1.0 & ford torino \\\\\n",
       "\t6 & 15.0 & 8.0 & 429.0 & 198.0 & 4341.0 & 10.0 & 70.0 & 1.0 & ford galaxie 500 \\\\\n",
       "\t7 & 14.0 & 8.0 & 454.0 & 220.0 & 4354.0 & 9.0 & 70.0 & 1.0 & chevrolet impala \\\\\n",
       "\t8 & 14.0 & 8.0 & 440.0 & 215.0 & 4312.0 & 8.5 & 70.0 & 1.0 & plymouth fury iii \\\\\n",
       "\t9 & 14.0 & 8.0 & 455.0 & 225.0 & 4425.0 & 10.0 & 70.0 & 1.0 & pontiac catalina \\\\\n",
       "\t10 & 15.0 & 8.0 & 390.0 & 190.0 & 3850.0 & 8.5 & 70.0 & 1.0 & amc ambassador dpl \\\\\n",
       "\t11 & \\emph{missing} & 4.0 & 133.0 & 115.0 & 3090.0 & 17.5 & 70.0 & 2.0 & citroen ds-21 pallas \\\\\n",
       "\t12 & \\emph{missing} & 8.0 & 350.0 & 165.0 & 4142.0 & 11.5 & 70.0 & 1.0 & chevrolet chevelle concours (sw) \\\\\n",
       "\t13 & \\emph{missing} & 8.0 & 351.0 & 153.0 & 4034.0 & 11.0 & 70.0 & 1.0 & ford torino (sw) \\\\\n",
       "\t14 & \\emph{missing} & 8.0 & 383.0 & 175.0 & 4166.0 & 10.5 & 70.0 & 1.0 & plymouth satellite (sw) \\\\\n",
       "\t15 & \\emph{missing} & 8.0 & 360.0 & 175.0 & 3850.0 & 11.0 & 70.0 & 1.0 & amc rebel sst (sw) \\\\\n",
       "\t$\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m406×9 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m mpg      \u001b[0m\u001b[1m cylinders \u001b[0m\u001b[1m displacement \u001b[0m\u001b[1m horsepower \u001b[0m\u001b[1m weight  \u001b[0m\u001b[1m acceleration \u001b[0m\u001b[1m year    \u001b[0m\u001b[1m origin  \u001b[0m\u001b[1m name                      \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Float64? \u001b[0m\u001b[90m Float64   \u001b[0m\u001b[90m Float64      \u001b[0m\u001b[90m Float64?   \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64      \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m String63                  \u001b[0m\n",
       "─────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n",
       "   1 │     18.0        8.0         307.0       130.0   3504.0          12.0     70.0      1.0  chevrolet chevelle malibu\n",
       "   2 │     15.0        8.0         350.0       165.0   3693.0          11.5     70.0      1.0  buick skylark 320\n",
       "   3 │     18.0        8.0         318.0       150.0   3436.0          11.0     70.0      1.0  plymouth satellite\n",
       "  ⋮  │    ⋮          ⋮           ⋮            ⋮          ⋮          ⋮           ⋮        ⋮                 ⋮\n",
       " 404 │     32.0        4.0         135.0        84.0   2295.0          11.6     82.0      1.0  dodge rampage\n",
       " 405 │     28.0        4.0         120.0        79.0   2625.0          18.6     82.0      1.0  ford ranger\n",
       " 406 │     31.0        4.0         119.0        82.0   2720.0          19.4     82.0      1.0  chevy s-10\n",
       "\u001b[36m                                                                                                         400 rows omitted\u001b[0m"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1_3 = select(df_raw,\n",
    "               :metrics =>\n",
    "               ByRow(x -> something.(tryparse.(Float64, split(x)), missing)) =>\n",
    "               [:mpg, :cylinders, :displacement, :horsepower, :weight, :acceleration, :year, :origin],\n",
    "               :name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us understand what is going on in the above expression.\n",
    "\n",
    "The general rule os specifying transformations in `select` (also `transform` and `combine`) is:\n",
    "\n",
    "`source_columns => transformation => destination_columns`\n",
    "\n",
    "In this case we take `:metrics` column from the source data frame. We process it element by element,\n",
    "which is signaled by `ByRow` (you can think of this as broadcasting).\n",
    "Now each element is first `split` and then we try parsing it as `Float64`.\n",
    "If we fail `tryparse` returns `nothing`, which we convert to `missing` using the `something` function.\n",
    "As a result we obtain a vector of `Float64` or `Missing` values.\n",
    "Since the last element of our transformation is a vector of culumn names this vector is expanded into\n",
    "that many columns.\n",
    "\n",
    "Finally we add the `:name` column from the source data frame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As above we check that the result is the same as `df1`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "true"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "isequal(df1_3, df1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can easily count the number of missing values in the `df1` data frame using the `eachcol` function that returns the iterator over columns of the data frame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "14"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(count(ismissing, col) for col in eachcol(df1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we could alternatively transform our data frame to a `Matrix` and use `count` on it (it would be slower though as unnecessary copies of data would be performed):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "14"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "count(ismissing, Matrix(df1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "or if you like iterators:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "14"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "count(ismissing, Iterators.flatten(eachcol(df1)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively could use the `mapcols` function to get the number of missings per column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class=\"data-frame\"><p>1 rows × 9 columns</p><table class=\"data-frame\"><thead><tr><th></th><th>mpg</th><th>cylinders</th><th>displacement</th><th>horsepower</th><th>weight</th><th>acceleration</th><th>year</th><th>origin</th><th>name</th></tr><tr><th></th><th title=\"Int64\">Int64</th><th title=\"Int64\">Int64</th><th title=\"Int64\">Int64</th><th title=\"Int64\">Int64</th><th title=\"Int64\">Int64</th><th title=\"Int64\">Int64</th><th title=\"Int64\">Int64</th><th title=\"Int64\">Int64</th><th title=\"Int64\">Int64</th></tr></thead><tbody><tr><th>1</th><td>8</td><td>0</td><td>0</td><td>6</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr></tbody></table></div>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccccccccc}\n",
       "\t& mpg & cylinders & displacement & horsepower & weight & acceleration & year & origin & name\\\\\n",
       "\t\\hline\n",
       "\t& Int64 & Int64 & Int64 & Int64 & Int64 & Int64 & Int64 & Int64 & Int64\\\\\n",
       "\t\\hline\n",
       "\t1 & 8 & 0 & 0 & 6 & 0 & 0 & 0 & 0 & 0 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m1×9 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m mpg   \u001b[0m\u001b[1m cylinders \u001b[0m\u001b[1m displacement \u001b[0m\u001b[1m horsepower \u001b[0m\u001b[1m weight \u001b[0m\u001b[1m acceleration \u001b[0m\u001b[1m year  \u001b[0m\u001b[1m origin \u001b[0m\u001b[1m name  \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Int64 \u001b[0m\u001b[90m Int64     \u001b[0m\u001b[90m Int64        \u001b[0m\u001b[90m Int64      \u001b[0m\u001b[90m Int64  \u001b[0m\u001b[90m Int64        \u001b[0m\u001b[90m Int64 \u001b[0m\u001b[90m Int64  \u001b[0m\u001b[90m Int64 \u001b[0m\n",
       "─────┼────────────────────────────────────────────────────────────────────────────────────────\n",
       "   1 │     8          0             0           6       0             0      0       0      0"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mapcols(x -> count(ismissing, x), df1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "also it is easy to find the rows containing missing values using the `filter` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class=\"data-frame\"><p>14 rows × 9 columns</p><table class=\"data-frame\"><thead><tr><th></th><th>mpg</th><th>cylinders</th><th>displacement</th><th>horsepower</th><th>weight</th><th>acceleration</th><th>year</th><th>origin</th><th>name</th></tr><tr><th></th><th title=\"Union{Missing, Float64}\">Float64?</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Union{Missing, Float64}\">Float64?</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"String63\">String63</th></tr></thead><tbody><tr><th>1</th><td><em>missing</em></td><td>4.0</td><td>133.0</td><td>115.0</td><td>3090.0</td><td>17.5</td><td>70.0</td><td>2.0</td><td>citroen ds-21 pallas</td></tr><tr><th>2</th><td><em>missing</em></td><td>8.0</td><td>350.0</td><td>165.0</td><td>4142.0</td><td>11.5</td><td>70.0</td><td>1.0</td><td>chevrolet chevelle concours (sw)</td></tr><tr><th>3</th><td><em>missing</em></td><td>8.0</td><td>351.0</td><td>153.0</td><td>4034.0</td><td>11.0</td><td>70.0</td><td>1.0</td><td>ford torino (sw)</td></tr><tr><th>4</th><td><em>missing</em></td><td>8.0</td><td>383.0</td><td>175.0</td><td>4166.0</td><td>10.5</td><td>70.0</td><td>1.0</td><td>plymouth satellite (sw)</td></tr><tr><th>5</th><td><em>missing</em></td><td>8.0</td><td>360.0</td><td>175.0</td><td>3850.0</td><td>11.0</td><td>70.0</td><td>1.0</td><td>amc rebel sst (sw)</td></tr><tr><th>6</th><td><em>missing</em></td><td>8.0</td><td>302.0</td><td>140.0</td><td>3353.0</td><td>8.0</td><td>70.0</td><td>1.0</td><td>ford mustang boss 302</td></tr><tr><th>7</th><td>25.0</td><td>4.0</td><td>98.0</td><td><em>missing</em></td><td>2046.0</td><td>19.0</td><td>71.0</td><td>1.0</td><td>ford pinto</td></tr><tr><th>8</th><td><em>missing</em></td><td>4.0</td><td>97.0</td><td>48.0</td><td>1978.0</td><td>20.0</td><td>71.0</td><td>2.0</td><td>volkswagen super beetle 117</td></tr><tr><th>9</th><td>21.0</td><td>6.0</td><td>200.0</td><td><em>missing</em></td><td>2875.0</td><td>17.0</td><td>74.0</td><td>1.0</td><td>ford maverick</td></tr><tr><th>10</th><td>40.9</td><td>4.0</td><td>85.0</td><td><em>missing</em></td><td>1835.0</td><td>17.3</td><td>80.0</td><td>2.0</td><td>renault lecar deluxe</td></tr><tr><th>11</th><td>23.6</td><td>4.0</td><td>140.0</td><td><em>missing</em></td><td>2905.0</td><td>14.3</td><td>80.0</td><td>1.0</td><td>ford mustang cobra</td></tr><tr><th>12</th><td>34.5</td><td>4.0</td><td>100.0</td><td><em>missing</em></td><td>2320.0</td><td>15.8</td><td>81.0</td><td>2.0</td><td>renault 18i</td></tr><tr><th>13</th><td><em>missing</em></td><td>4.0</td><td>121.0</td><td>110.0</td><td>2800.0</td><td>15.4</td><td>81.0</td><td>2.0</td><td>saab 900s</td></tr><tr><th>14</th><td>23.0</td><td>4.0</td><td>151.0</td><td><em>missing</em></td><td>3035.0</td><td>20.5</td><td>82.0</td><td>1.0</td><td>amc concord dl</td></tr></tbody></table></div>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccccccccc}\n",
       "\t& mpg & cylinders & displacement & horsepower & weight & acceleration & year & origin & name\\\\\n",
       "\t\\hline\n",
       "\t& Float64? & Float64 & Float64 & Float64? & Float64 & Float64 & Float64 & Float64 & String63\\\\\n",
       "\t\\hline\n",
       "\t1 & \\emph{missing} & 4.0 & 133.0 & 115.0 & 3090.0 & 17.5 & 70.0 & 2.0 & citroen ds-21 pallas \\\\\n",
       "\t2 & \\emph{missing} & 8.0 & 350.0 & 165.0 & 4142.0 & 11.5 & 70.0 & 1.0 & chevrolet chevelle concours (sw) \\\\\n",
       "\t3 & \\emph{missing} & 8.0 & 351.0 & 153.0 & 4034.0 & 11.0 & 70.0 & 1.0 & ford torino (sw) \\\\\n",
       "\t4 & \\emph{missing} & 8.0 & 383.0 & 175.0 & 4166.0 & 10.5 & 70.0 & 1.0 & plymouth satellite (sw) \\\\\n",
       "\t5 & \\emph{missing} & 8.0 & 360.0 & 175.0 & 3850.0 & 11.0 & 70.0 & 1.0 & amc rebel sst (sw) \\\\\n",
       "\t6 & \\emph{missing} & 8.0 & 302.0 & 140.0 & 3353.0 & 8.0 & 70.0 & 1.0 & ford mustang boss 302 \\\\\n",
       "\t7 & 25.0 & 4.0 & 98.0 & \\emph{missing} & 2046.0 & 19.0 & 71.0 & 1.0 & ford pinto \\\\\n",
       "\t8 & \\emph{missing} & 4.0 & 97.0 & 48.0 & 1978.0 & 20.0 & 71.0 & 2.0 & volkswagen super beetle 117 \\\\\n",
       "\t9 & 21.0 & 6.0 & 200.0 & \\emph{missing} & 2875.0 & 17.0 & 74.0 & 1.0 & ford maverick \\\\\n",
       "\t10 & 40.9 & 4.0 & 85.0 & \\emph{missing} & 1835.0 & 17.3 & 80.0 & 2.0 & renault lecar deluxe \\\\\n",
       "\t11 & 23.6 & 4.0 & 140.0 & \\emph{missing} & 2905.0 & 14.3 & 80.0 & 1.0 & ford mustang cobra \\\\\n",
       "\t12 & 34.5 & 4.0 & 100.0 & \\emph{missing} & 2320.0 & 15.8 & 81.0 & 2.0 & renault 18i \\\\\n",
       "\t13 & \\emph{missing} & 4.0 & 121.0 & 110.0 & 2800.0 & 15.4 & 81.0 & 2.0 & saab 900s \\\\\n",
       "\t14 & 23.0 & 4.0 & 151.0 & \\emph{missing} & 3035.0 & 20.5 & 82.0 & 1.0 & amc concord dl \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m14×9 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m mpg       \u001b[0m\u001b[1m cylinders \u001b[0m\u001b[1m displacement \u001b[0m\u001b[1m horsepower \u001b[0m\u001b[1m weight  \u001b[0m\u001b[1m acceleration \u001b[0m\u001b[1m year    \u001b[0m\u001b[1m origin  \u001b[0m\u001b[1m name                             \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Float64?  \u001b[0m\u001b[90m Float64   \u001b[0m\u001b[90m Float64      \u001b[0m\u001b[90m Float64?   \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64      \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m String63                         \u001b[0m\n",
       "─────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n",
       "   1 │\u001b[90m missing   \u001b[0m       4.0         133.0       115.0   3090.0          17.5     70.0      2.0  citroen ds-21 pallas\n",
       "   2 │\u001b[90m missing   \u001b[0m       8.0         350.0       165.0   4142.0          11.5     70.0      1.0  chevrolet chevelle concours (sw)\n",
       "   3 │\u001b[90m missing   \u001b[0m       8.0         351.0       153.0   4034.0          11.0     70.0      1.0  ford torino (sw)\n",
       "  ⋮  │     ⋮          ⋮           ⋮            ⋮          ⋮          ⋮           ⋮        ⋮                    ⋮\n",
       "  12 │      34.5        4.0         100.0 \u001b[90m  missing   \u001b[0m  2320.0          15.8     81.0      2.0  renault 18i\n",
       "  13 │\u001b[90m missing   \u001b[0m       4.0         121.0       110.0   2800.0          15.4     81.0      2.0  saab 900s\n",
       "  14 │      23.0        4.0         151.0 \u001b[90m  missing   \u001b[0m  3035.0          20.5     82.0      1.0  amc concord dl\n",
       "\u001b[36m                                                                                                                   8 rows omitted\u001b[0m"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filter(row -> any(ismissing, row), df1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assume we are interested in the brand of each car. We can extract it from `:name` column using broadcasting like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "406-element Vector{SubString{String63}}:\n",
       " \"chevrolet\"\n",
       " \"buick\"\n",
       " \"plymouth\"\n",
       " \"amc\"\n",
       " \"ford\"\n",
       " ⋮\n",
       " \"ford\"\n",
       " \"vw\"\n",
       " \"dodge\"\n",
       " \"ford\"\n",
       " \"chevy\""
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1.brand = first.(split.(df1.name))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class=\"data-frame\"><p>406 rows × 10 columns</p><table class=\"data-frame\"><thead><tr><th></th><th>mpg</th><th>cylinders</th><th>displacement</th><th>horsepower</th><th>weight</th><th>acceleration</th><th>year</th><th>origin</th><th>name</th><th>brand</th></tr><tr><th></th><th title=\"Union{Missing, Float64}\">Float64?</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Union{Missing, Float64}\">Float64?</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"String63\">String63</th><th title=\"SubString{String63}\">SubStrin…</th></tr></thead><tbody><tr><th>1</th><td>18.0</td><td>8.0</td><td>307.0</td><td>130.0</td><td>3504.0</td><td>12.0</td><td>70.0</td><td>1.0</td><td>chevrolet chevelle malibu</td><td>chevrolet</td></tr><tr><th>2</th><td>15.0</td><td>8.0</td><td>350.0</td><td>165.0</td><td>3693.0</td><td>11.5</td><td>70.0</td><td>1.0</td><td>buick skylark 320</td><td>buick</td></tr><tr><th>3</th><td>18.0</td><td>8.0</td><td>318.0</td><td>150.0</td><td>3436.0</td><td>11.0</td><td>70.0</td><td>1.0</td><td>plymouth satellite</td><td>plymouth</td></tr><tr><th>4</th><td>16.0</td><td>8.0</td><td>304.0</td><td>150.0</td><td>3433.0</td><td>12.0</td><td>70.0</td><td>1.0</td><td>amc rebel sst</td><td>amc</td></tr><tr><th>5</th><td>17.0</td><td>8.0</td><td>302.0</td><td>140.0</td><td>3449.0</td><td>10.5</td><td>70.0</td><td>1.0</td><td>ford torino</td><td>ford</td></tr><tr><th>6</th><td>15.0</td><td>8.0</td><td>429.0</td><td>198.0</td><td>4341.0</td><td>10.0</td><td>70.0</td><td>1.0</td><td>ford galaxie 500</td><td>ford</td></tr><tr><th>7</th><td>14.0</td><td>8.0</td><td>454.0</td><td>220.0</td><td>4354.0</td><td>9.0</td><td>70.0</td><td>1.0</td><td>chevrolet impala</td><td>chevrolet</td></tr><tr><th>8</th><td>14.0</td><td>8.0</td><td>440.0</td><td>215.0</td><td>4312.0</td><td>8.5</td><td>70.0</td><td>1.0</td><td>plymouth fury iii</td><td>plymouth</td></tr><tr><th>9</th><td>14.0</td><td>8.0</td><td>455.0</td><td>225.0</td><td>4425.0</td><td>10.0</td><td>70.0</td><td>1.0</td><td>pontiac catalina</td><td>pontiac</td></tr><tr><th>10</th><td>15.0</td><td>8.0</td><td>390.0</td><td>190.0</td><td>3850.0</td><td>8.5</td><td>70.0</td><td>1.0</td><td>amc ambassador dpl</td><td>amc</td></tr><tr><th>11</th><td><em>missing</em></td><td>4.0</td><td>133.0</td><td>115.0</td><td>3090.0</td><td>17.5</td><td>70.0</td><td>2.0</td><td>citroen ds-21 pallas</td><td>citroen</td></tr><tr><th>12</th><td><em>missing</em></td><td>8.0</td><td>350.0</td><td>165.0</td><td>4142.0</td><td>11.5</td><td>70.0</td><td>1.0</td><td>chevrolet chevelle concours (sw)</td><td>chevrolet</td></tr><tr><th>13</th><td><em>missing</em></td><td>8.0</td><td>351.0</td><td>153.0</td><td>4034.0</td><td>11.0</td><td>70.0</td><td>1.0</td><td>ford torino (sw)</td><td>ford</td></tr><tr><th>14</th><td><em>missing</em></td><td>8.0</td><td>383.0</td><td>175.0</td><td>4166.0</td><td>10.5</td><td>70.0</td><td>1.0</td><td>plymouth satellite (sw)</td><td>plymouth</td></tr><tr><th>15</th><td><em>missing</em></td><td>8.0</td><td>360.0</td><td>175.0</td><td>3850.0</td><td>11.0</td><td>70.0</td><td>1.0</td><td>amc rebel sst (sw)</td><td>amc</td></tr><tr><th>&vellip;</th><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td></tr></tbody></table></div>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccccccccc}\n",
       "\t& mpg & cylinders & displacement & horsepower & weight & acceleration & year & origin & name & brand\\\\\n",
       "\t\\hline\n",
       "\t& Float64? & Float64 & Float64 & Float64? & Float64 & Float64 & Float64 & Float64 & String63 & SubStrin…\\\\\n",
       "\t\\hline\n",
       "\t1 & 18.0 & 8.0 & 307.0 & 130.0 & 3504.0 & 12.0 & 70.0 & 1.0 & chevrolet chevelle malibu & chevrolet \\\\\n",
       "\t2 & 15.0 & 8.0 & 350.0 & 165.0 & 3693.0 & 11.5 & 70.0 & 1.0 & buick skylark 320 & buick \\\\\n",
       "\t3 & 18.0 & 8.0 & 318.0 & 150.0 & 3436.0 & 11.0 & 70.0 & 1.0 & plymouth satellite & plymouth \\\\\n",
       "\t4 & 16.0 & 8.0 & 304.0 & 150.0 & 3433.0 & 12.0 & 70.0 & 1.0 & amc rebel sst & amc \\\\\n",
       "\t5 & 17.0 & 8.0 & 302.0 & 140.0 & 3449.0 & 10.5 & 70.0 & 1.0 & ford torino & ford \\\\\n",
       "\t6 & 15.0 & 8.0 & 429.0 & 198.0 & 4341.0 & 10.0 & 70.0 & 1.0 & ford galaxie 500 & ford \\\\\n",
       "\t7 & 14.0 & 8.0 & 454.0 & 220.0 & 4354.0 & 9.0 & 70.0 & 1.0 & chevrolet impala & chevrolet \\\\\n",
       "\t8 & 14.0 & 8.0 & 440.0 & 215.0 & 4312.0 & 8.5 & 70.0 & 1.0 & plymouth fury iii & plymouth \\\\\n",
       "\t9 & 14.0 & 8.0 & 455.0 & 225.0 & 4425.0 & 10.0 & 70.0 & 1.0 & pontiac catalina & pontiac \\\\\n",
       "\t10 & 15.0 & 8.0 & 390.0 & 190.0 & 3850.0 & 8.5 & 70.0 & 1.0 & amc ambassador dpl & amc \\\\\n",
       "\t11 & \\emph{missing} & 4.0 & 133.0 & 115.0 & 3090.0 & 17.5 & 70.0 & 2.0 & citroen ds-21 pallas & citroen \\\\\n",
       "\t12 & \\emph{missing} & 8.0 & 350.0 & 165.0 & 4142.0 & 11.5 & 70.0 & 1.0 & chevrolet chevelle concours (sw) & chevrolet \\\\\n",
       "\t13 & \\emph{missing} & 8.0 & 351.0 & 153.0 & 4034.0 & 11.0 & 70.0 & 1.0 & ford torino (sw) & ford \\\\\n",
       "\t14 & \\emph{missing} & 8.0 & 383.0 & 175.0 & 4166.0 & 10.5 & 70.0 & 1.0 & plymouth satellite (sw) & plymouth \\\\\n",
       "\t15 & \\emph{missing} & 8.0 & 360.0 & 175.0 & 3850.0 & 11.0 & 70.0 & 1.0 & amc rebel sst (sw) & amc \\\\\n",
       "\t$\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m406×10 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m mpg      \u001b[0m\u001b[1m cylinders \u001b[0m\u001b[1m displacement \u001b[0m\u001b[1m horsepower \u001b[0m\u001b[1m weight  \u001b[0m\u001b[1m acceleration \u001b[0m\u001b[1m year    \u001b[0m\u001b[1m origin  \u001b[0m\u001b[1m name                      \u001b[0m\u001b[1m brand     \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Float64? \u001b[0m\u001b[90m Float64   \u001b[0m\u001b[90m Float64      \u001b[0m\u001b[90m Float64?   \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64      \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m String63                  \u001b[0m\u001b[90m SubStrin… \u001b[0m\n",
       "─────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n",
       "   1 │     18.0        8.0         307.0       130.0   3504.0          12.0     70.0      1.0  chevrolet chevelle malibu  chevrolet\n",
       "   2 │     15.0        8.0         350.0       165.0   3693.0          11.5     70.0      1.0  buick skylark 320          buick\n",
       "   3 │     18.0        8.0         318.0       150.0   3436.0          11.0     70.0      1.0  plymouth satellite         plymouth\n",
       "  ⋮  │    ⋮          ⋮           ⋮            ⋮          ⋮          ⋮           ⋮        ⋮                 ⋮                  ⋮\n",
       " 404 │     32.0        4.0         135.0        84.0   2295.0          11.6     82.0      1.0  dodge rampage              dodge\n",
       " 405 │     28.0        4.0         120.0        79.0   2625.0          18.6     82.0      1.0  ford ranger                ford\n",
       " 406 │     31.0        4.0         119.0        82.0   2720.0          19.4     82.0      1.0  chevy s-10                 chevy\n",
       "\u001b[36m                                                                                                                    400 rows omitted\u001b[0m"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Earlier we have shown how one can manually find rows that have missing values. A common operation is the reverse - i.e. selecting only rows that do not contain missing values. This can be achieved like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class=\"data-frame\"><p>392 rows × 10 columns</p><table class=\"data-frame\"><thead><tr><th></th><th>mpg</th><th>cylinders</th><th>displacement</th><th>horsepower</th><th>weight</th><th>acceleration</th><th>year</th><th>origin</th><th>name</th><th>brand</th></tr><tr><th></th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"String63\">String63</th><th title=\"SubString{String63}\">SubStrin…</th></tr></thead><tbody><tr><th>1</th><td>18.0</td><td>8.0</td><td>307.0</td><td>130.0</td><td>3504.0</td><td>12.0</td><td>70.0</td><td>1.0</td><td>chevrolet chevelle malibu</td><td>chevrolet</td></tr><tr><th>2</th><td>15.0</td><td>8.0</td><td>350.0</td><td>165.0</td><td>3693.0</td><td>11.5</td><td>70.0</td><td>1.0</td><td>buick skylark 320</td><td>buick</td></tr><tr><th>3</th><td>18.0</td><td>8.0</td><td>318.0</td><td>150.0</td><td>3436.0</td><td>11.0</td><td>70.0</td><td>1.0</td><td>plymouth satellite</td><td>plymouth</td></tr><tr><th>4</th><td>16.0</td><td>8.0</td><td>304.0</td><td>150.0</td><td>3433.0</td><td>12.0</td><td>70.0</td><td>1.0</td><td>amc rebel sst</td><td>amc</td></tr><tr><th>5</th><td>17.0</td><td>8.0</td><td>302.0</td><td>140.0</td><td>3449.0</td><td>10.5</td><td>70.0</td><td>1.0</td><td>ford torino</td><td>ford</td></tr><tr><th>6</th><td>15.0</td><td>8.0</td><td>429.0</td><td>198.0</td><td>4341.0</td><td>10.0</td><td>70.0</td><td>1.0</td><td>ford galaxie 500</td><td>ford</td></tr><tr><th>7</th><td>14.0</td><td>8.0</td><td>454.0</td><td>220.0</td><td>4354.0</td><td>9.0</td><td>70.0</td><td>1.0</td><td>chevrolet impala</td><td>chevrolet</td></tr><tr><th>8</th><td>14.0</td><td>8.0</td><td>440.0</td><td>215.0</td><td>4312.0</td><td>8.5</td><td>70.0</td><td>1.0</td><td>plymouth fury iii</td><td>plymouth</td></tr><tr><th>9</th><td>14.0</td><td>8.0</td><td>455.0</td><td>225.0</td><td>4425.0</td><td>10.0</td><td>70.0</td><td>1.0</td><td>pontiac catalina</td><td>pontiac</td></tr><tr><th>10</th><td>15.0</td><td>8.0</td><td>390.0</td><td>190.0</td><td>3850.0</td><td>8.5</td><td>70.0</td><td>1.0</td><td>amc ambassador dpl</td><td>amc</td></tr><tr><th>11</th><td>15.0</td><td>8.0</td><td>383.0</td><td>170.0</td><td>3563.0</td><td>10.0</td><td>70.0</td><td>1.0</td><td>dodge challenger se</td><td>dodge</td></tr><tr><th>12</th><td>14.0</td><td>8.0</td><td>340.0</td><td>160.0</td><td>3609.0</td><td>8.0</td><td>70.0</td><td>1.0</td><td>plymouth &apos;cuda 340</td><td>plymouth</td></tr><tr><th>13</th><td>15.0</td><td>8.0</td><td>400.0</td><td>150.0</td><td>3761.0</td><td>9.5</td><td>70.0</td><td>1.0</td><td>chevrolet monte carlo</td><td>chevrolet</td></tr><tr><th>14</th><td>14.0</td><td>8.0</td><td>455.0</td><td>225.0</td><td>3086.0</td><td>10.0</td><td>70.0</td><td>1.0</td><td>buick estate wagon (sw)</td><td>buick</td></tr><tr><th>15</th><td>24.0</td><td>4.0</td><td>113.0</td><td>95.0</td><td>2372.0</td><td>15.0</td><td>70.0</td><td>3.0</td><td>toyota corona mark ii</td><td>toyota</td></tr><tr><th>&vellip;</th><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td></tr></tbody></table></div>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccccccccc}\n",
       "\t& mpg & cylinders & displacement & horsepower & weight & acceleration & year & origin & name & brand\\\\\n",
       "\t\\hline\n",
       "\t& Float64 & Float64 & Float64 & Float64 & Float64 & Float64 & Float64 & Float64 & String63 & SubStrin…\\\\\n",
       "\t\\hline\n",
       "\t1 & 18.0 & 8.0 & 307.0 & 130.0 & 3504.0 & 12.0 & 70.0 & 1.0 & chevrolet chevelle malibu & chevrolet \\\\\n",
       "\t2 & 15.0 & 8.0 & 350.0 & 165.0 & 3693.0 & 11.5 & 70.0 & 1.0 & buick skylark 320 & buick \\\\\n",
       "\t3 & 18.0 & 8.0 & 318.0 & 150.0 & 3436.0 & 11.0 & 70.0 & 1.0 & plymouth satellite & plymouth \\\\\n",
       "\t4 & 16.0 & 8.0 & 304.0 & 150.0 & 3433.0 & 12.0 & 70.0 & 1.0 & amc rebel sst & amc \\\\\n",
       "\t5 & 17.0 & 8.0 & 302.0 & 140.0 & 3449.0 & 10.5 & 70.0 & 1.0 & ford torino & ford \\\\\n",
       "\t6 & 15.0 & 8.0 & 429.0 & 198.0 & 4341.0 & 10.0 & 70.0 & 1.0 & ford galaxie 500 & ford \\\\\n",
       "\t7 & 14.0 & 8.0 & 454.0 & 220.0 & 4354.0 & 9.0 & 70.0 & 1.0 & chevrolet impala & chevrolet \\\\\n",
       "\t8 & 14.0 & 8.0 & 440.0 & 215.0 & 4312.0 & 8.5 & 70.0 & 1.0 & plymouth fury iii & plymouth \\\\\n",
       "\t9 & 14.0 & 8.0 & 455.0 & 225.0 & 4425.0 & 10.0 & 70.0 & 1.0 & pontiac catalina & pontiac \\\\\n",
       "\t10 & 15.0 & 8.0 & 390.0 & 190.0 & 3850.0 & 8.5 & 70.0 & 1.0 & amc ambassador dpl & amc \\\\\n",
       "\t11 & 15.0 & 8.0 & 383.0 & 170.0 & 3563.0 & 10.0 & 70.0 & 1.0 & dodge challenger se & dodge \\\\\n",
       "\t12 & 14.0 & 8.0 & 340.0 & 160.0 & 3609.0 & 8.0 & 70.0 & 1.0 & plymouth 'cuda 340 & plymouth \\\\\n",
       "\t13 & 15.0 & 8.0 & 400.0 & 150.0 & 3761.0 & 9.5 & 70.0 & 1.0 & chevrolet monte carlo & chevrolet \\\\\n",
       "\t14 & 14.0 & 8.0 & 455.0 & 225.0 & 3086.0 & 10.0 & 70.0 & 1.0 & buick estate wagon (sw) & buick \\\\\n",
       "\t15 & 24.0 & 4.0 & 113.0 & 95.0 & 2372.0 & 15.0 & 70.0 & 3.0 & toyota corona mark ii & toyota \\\\\n",
       "\t$\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m392×10 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m mpg     \u001b[0m\u001b[1m cylinders \u001b[0m\u001b[1m displacement \u001b[0m\u001b[1m horsepower \u001b[0m\u001b[1m weight  \u001b[0m\u001b[1m acceleration \u001b[0m\u001b[1m year    \u001b[0m\u001b[1m origin  \u001b[0m\u001b[1m name                      \u001b[0m\u001b[1m brand     \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Float64 \u001b[0m\u001b[90m Float64   \u001b[0m\u001b[90m Float64      \u001b[0m\u001b[90m Float64    \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64      \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m String63                  \u001b[0m\u001b[90m SubStrin… \u001b[0m\n",
       "─────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n",
       "   1 │    18.0        8.0         307.0       130.0   3504.0          12.0     70.0      1.0  chevrolet chevelle malibu  chevrolet\n",
       "   2 │    15.0        8.0         350.0       165.0   3693.0          11.5     70.0      1.0  buick skylark 320          buick\n",
       "   3 │    18.0        8.0         318.0       150.0   3436.0          11.0     70.0      1.0  plymouth satellite         plymouth\n",
       "  ⋮  │    ⋮         ⋮           ⋮            ⋮          ⋮          ⋮           ⋮        ⋮                 ⋮                  ⋮\n",
       " 390 │    32.0        4.0         135.0        84.0   2295.0          11.6     82.0      1.0  dodge rampage              dodge\n",
       " 391 │    28.0        4.0         120.0        79.0   2625.0          18.6     82.0      1.0  ford ranger                ford\n",
       " 392 │    31.0        4.0         119.0        82.0   2720.0          19.4     82.0      1.0  chevy s-10                 chevy\n",
       "\u001b[36m                                                                                                                   386 rows omitted\u001b[0m"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2 = dropmissing(df1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let us find all rows that correspond to `\"saab\"` brand. You can do it in two ways, either indexing or using `filter` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class=\"data-frame\"><p>4 rows × 10 columns</p><table class=\"data-frame\"><thead><tr><th></th><th>mpg</th><th>cylinders</th><th>displacement</th><th>horsepower</th><th>weight</th><th>acceleration</th><th>year</th><th>origin</th><th>name</th><th>brand</th></tr><tr><th></th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"String63\">String63</th><th title=\"SubString{String63}\">SubStrin…</th></tr></thead><tbody><tr><th>1</th><td>25.0</td><td>4.0</td><td>104.0</td><td>95.0</td><td>2375.0</td><td>17.5</td><td>70.0</td><td>2.0</td><td>saab 99e</td><td>saab</td></tr><tr><th>2</th><td>24.0</td><td>4.0</td><td>121.0</td><td>110.0</td><td>2660.0</td><td>14.0</td><td>73.0</td><td>2.0</td><td>saab 99le</td><td>saab</td></tr><tr><th>3</th><td>25.0</td><td>4.0</td><td>121.0</td><td>115.0</td><td>2671.0</td><td>13.5</td><td>75.0</td><td>2.0</td><td>saab 99le</td><td>saab</td></tr><tr><th>4</th><td>21.6</td><td>4.0</td><td>121.0</td><td>115.0</td><td>2795.0</td><td>15.7</td><td>78.0</td><td>2.0</td><td>saab 99gle</td><td>saab</td></tr></tbody></table></div>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccccccccc}\n",
       "\t& mpg & cylinders & displacement & horsepower & weight & acceleration & year & origin & name & brand\\\\\n",
       "\t\\hline\n",
       "\t& Float64 & Float64 & Float64 & Float64 & Float64 & Float64 & Float64 & Float64 & String63 & SubStrin…\\\\\n",
       "\t\\hline\n",
       "\t1 & 25.0 & 4.0 & 104.0 & 95.0 & 2375.0 & 17.5 & 70.0 & 2.0 & saab 99e & saab \\\\\n",
       "\t2 & 24.0 & 4.0 & 121.0 & 110.0 & 2660.0 & 14.0 & 73.0 & 2.0 & saab 99le & saab \\\\\n",
       "\t3 & 25.0 & 4.0 & 121.0 & 115.0 & 2671.0 & 13.5 & 75.0 & 2.0 & saab 99le & saab \\\\\n",
       "\t4 & 21.6 & 4.0 & 121.0 & 115.0 & 2795.0 & 15.7 & 78.0 & 2.0 & saab 99gle & saab \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m4×10 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m mpg     \u001b[0m\u001b[1m cylinders \u001b[0m\u001b[1m displacement \u001b[0m\u001b[1m horsepower \u001b[0m\u001b[1m weight  \u001b[0m\u001b[1m acceleration \u001b[0m\u001b[1m year    \u001b[0m\u001b[1m origin  \u001b[0m\u001b[1m name       \u001b[0m\u001b[1m brand     \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Float64 \u001b[0m\u001b[90m Float64   \u001b[0m\u001b[90m Float64      \u001b[0m\u001b[90m Float64    \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64      \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m String63   \u001b[0m\u001b[90m SubStrin… \u001b[0m\n",
       "─────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────\n",
       "   1 │    25.0        4.0         104.0        95.0   2375.0          17.5     70.0      2.0  saab 99e    saab\n",
       "   2 │    24.0        4.0         121.0       110.0   2660.0          14.0     73.0      2.0  saab 99le   saab\n",
       "   3 │    25.0        4.0         121.0       115.0   2671.0          13.5     75.0      2.0  saab 99le   saab\n",
       "   4 │    21.6        4.0         121.0       115.0   2795.0          15.7     78.0      2.0  saab 99gle  saab"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2[df2.brand .== \"saab\", :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class=\"data-frame\"><p>4 rows × 10 columns</p><table class=\"data-frame\"><thead><tr><th></th><th>mpg</th><th>cylinders</th><th>displacement</th><th>horsepower</th><th>weight</th><th>acceleration</th><th>year</th><th>origin</th><th>name</th><th>brand</th></tr><tr><th></th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"String63\">String63</th><th title=\"SubString{String63}\">SubStrin…</th></tr></thead><tbody><tr><th>1</th><td>25.0</td><td>4.0</td><td>104.0</td><td>95.0</td><td>2375.0</td><td>17.5</td><td>70.0</td><td>2.0</td><td>saab 99e</td><td>saab</td></tr><tr><th>2</th><td>24.0</td><td>4.0</td><td>121.0</td><td>110.0</td><td>2660.0</td><td>14.0</td><td>73.0</td><td>2.0</td><td>saab 99le</td><td>saab</td></tr><tr><th>3</th><td>25.0</td><td>4.0</td><td>121.0</td><td>115.0</td><td>2671.0</td><td>13.5</td><td>75.0</td><td>2.0</td><td>saab 99le</td><td>saab</td></tr><tr><th>4</th><td>21.6</td><td>4.0</td><td>121.0</td><td>115.0</td><td>2795.0</td><td>15.7</td><td>78.0</td><td>2.0</td><td>saab 99gle</td><td>saab</td></tr></tbody></table></div>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccccccccc}\n",
       "\t& mpg & cylinders & displacement & horsepower & weight & acceleration & year & origin & name & brand\\\\\n",
       "\t\\hline\n",
       "\t& Float64 & Float64 & Float64 & Float64 & Float64 & Float64 & Float64 & Float64 & String63 & SubStrin…\\\\\n",
       "\t\\hline\n",
       "\t1 & 25.0 & 4.0 & 104.0 & 95.0 & 2375.0 & 17.5 & 70.0 & 2.0 & saab 99e & saab \\\\\n",
       "\t2 & 24.0 & 4.0 & 121.0 & 110.0 & 2660.0 & 14.0 & 73.0 & 2.0 & saab 99le & saab \\\\\n",
       "\t3 & 25.0 & 4.0 & 121.0 & 115.0 & 2671.0 & 13.5 & 75.0 & 2.0 & saab 99le & saab \\\\\n",
       "\t4 & 21.6 & 4.0 & 121.0 & 115.0 & 2795.0 & 15.7 & 78.0 & 2.0 & saab 99gle & saab \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m4×10 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m mpg     \u001b[0m\u001b[1m cylinders \u001b[0m\u001b[1m displacement \u001b[0m\u001b[1m horsepower \u001b[0m\u001b[1m weight  \u001b[0m\u001b[1m acceleration \u001b[0m\u001b[1m year    \u001b[0m\u001b[1m origin  \u001b[0m\u001b[1m name       \u001b[0m\u001b[1m brand     \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Float64 \u001b[0m\u001b[90m Float64   \u001b[0m\u001b[90m Float64      \u001b[0m\u001b[90m Float64    \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64      \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m String63   \u001b[0m\u001b[90m SubStrin… \u001b[0m\n",
       "─────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────\n",
       "   1 │    25.0        4.0         104.0        95.0   2375.0          17.5     70.0      2.0  saab 99e    saab\n",
       "   2 │    24.0        4.0         121.0       110.0   2660.0          14.0     73.0      2.0  saab 99le   saab\n",
       "   3 │    25.0        4.0         121.0       115.0   2671.0          13.5     75.0      2.0  saab 99le   saab\n",
       "   4 │    21.6        4.0         121.0       115.0   2795.0          15.7     78.0      2.0  saab 99gle  saab"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filter(:brand => ==(\"saab\"), df2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the `:brand => ==(\"saab\")` syntax means that we take elements of `:brand` column and pass it to `==(\"saab\")` function.\n",
    "\n",
    "Now `==(\"saab\")` is just a shorthand for `x -> x == \"saab\"`.\n",
    "\n",
    "Alternatively we could do the filtering operation in the following way (this is a bit slower but you might find it more readable):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class=\"data-frame\"><p>4 rows × 10 columns</p><table class=\"data-frame\"><thead><tr><th></th><th>mpg</th><th>cylinders</th><th>displacement</th><th>horsepower</th><th>weight</th><th>acceleration</th><th>year</th><th>origin</th><th>name</th><th>brand</th></tr><tr><th></th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"Float64\">Float64</th><th title=\"String63\">String63</th><th title=\"SubString{String63}\">SubStrin…</th></tr></thead><tbody><tr><th>1</th><td>25.0</td><td>4.0</td><td>104.0</td><td>95.0</td><td>2375.0</td><td>17.5</td><td>70.0</td><td>2.0</td><td>saab 99e</td><td>saab</td></tr><tr><th>2</th><td>24.0</td><td>4.0</td><td>121.0</td><td>110.0</td><td>2660.0</td><td>14.0</td><td>73.0</td><td>2.0</td><td>saab 99le</td><td>saab</td></tr><tr><th>3</th><td>25.0</td><td>4.0</td><td>121.0</td><td>115.0</td><td>2671.0</td><td>13.5</td><td>75.0</td><td>2.0</td><td>saab 99le</td><td>saab</td></tr><tr><th>4</th><td>21.6</td><td>4.0</td><td>121.0</td><td>115.0</td><td>2795.0</td><td>15.7</td><td>78.0</td><td>2.0</td><td>saab 99gle</td><td>saab</td></tr></tbody></table></div>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccccccccc}\n",
       "\t& mpg & cylinders & displacement & horsepower & weight & acceleration & year & origin & name & brand\\\\\n",
       "\t\\hline\n",
       "\t& Float64 & Float64 & Float64 & Float64 & Float64 & Float64 & Float64 & Float64 & String63 & SubStrin…\\\\\n",
       "\t\\hline\n",
       "\t1 & 25.0 & 4.0 & 104.0 & 95.0 & 2375.0 & 17.5 & 70.0 & 2.0 & saab 99e & saab \\\\\n",
       "\t2 & 24.0 & 4.0 & 121.0 & 110.0 & 2660.0 & 14.0 & 73.0 & 2.0 & saab 99le & saab \\\\\n",
       "\t3 & 25.0 & 4.0 & 121.0 & 115.0 & 2671.0 & 13.5 & 75.0 & 2.0 & saab 99le & saab \\\\\n",
       "\t4 & 21.6 & 4.0 & 121.0 & 115.0 & 2795.0 & 15.7 & 78.0 & 2.0 & saab 99gle & saab \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m4×10 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m mpg     \u001b[0m\u001b[1m cylinders \u001b[0m\u001b[1m displacement \u001b[0m\u001b[1m horsepower \u001b[0m\u001b[1m weight  \u001b[0m\u001b[1m acceleration \u001b[0m\u001b[1m year    \u001b[0m\u001b[1m origin  \u001b[0m\u001b[1m name       \u001b[0m\u001b[1m brand     \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Float64 \u001b[0m\u001b[90m Float64   \u001b[0m\u001b[90m Float64      \u001b[0m\u001b[90m Float64    \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64      \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m String63   \u001b[0m\u001b[90m SubStrin… \u001b[0m\n",
       "─────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────\n",
       "   1 │    25.0        4.0         104.0        95.0   2375.0          17.5     70.0      2.0  saab 99e    saab\n",
       "   2 │    24.0        4.0         121.0       110.0   2660.0          14.0     73.0      2.0  saab 99le   saab\n",
       "   3 │    25.0        4.0         121.0       115.0   2671.0          13.5     75.0      2.0  saab 99le   saab\n",
       "   4 │    21.6        4.0         121.0       115.0   2795.0          15.7     78.0      2.0  saab 99gle  saab"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filter(row -> row.brand == \"saab\", df2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To finish this part of the tutorial let us save the `df2` file to auto2.csv and auto2.arrow files. We will use them later in the next parts of the course."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"auto2.csv\""
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CSV.write(\"auto2.csv\", df2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"auto2.arrow\""
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Arrow.write(\"auto2.arrow\", df2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us just quickly inspect what we have written to disk in auto2.csv (auto2.arrow file is binary) before we finish:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "393-element Vector{String}:\n",
       " \"mpg,cylinders,displacement,horsepower,weight,acceleration,year,origin,name,brand\"\n",
       " \"18.0,8.0,307.0,130.0,3504.0,12.0,70.0,1.0,chevrolet chevelle malibu,chevrolet\"\n",
       " \"15.0,8.0,350.0,165.0,3693.0,11.5,70.0,1.0,buick skylark 320,buick\"\n",
       " \"18.0,8.0,318.0,150.0,3436.0,11.0,70.0,1.0,plymouth satellite,plymouth\"\n",
       " \"16.0,8.0,304.0,150.0,3433.0,12.0,70.0,1.0,amc rebel sst,amc\"\n",
       " ⋮\n",
       " \"27.0,4.0,140.0,86.0,2790.0,15.6,82.0,1.0,ford mustang gl,ford\"\n",
       " \"44.0,4.0,97.0,52.0,2130.0,24.6,82.0,2.0,vw pickup,vw\"\n",
       " \"32.0,4.0,135.0,84.0,2295.0,11.6,82.0,1.0,dodge rampage,dodge\"\n",
       " \"28.0,4.0,120.0,79.0,2625.0,18.6,82.0,1.0,ford ranger,ford\"\n",
       " \"31.0,4.0,119.0,82.0,2720.0,19.4,82.0,1.0,chevy s-10,chevy\""
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "readlines(\"auto2.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that CSV.jl by default used a comma to separate the fields in our file and written a header in the first row of the file."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 1.7.0",
   "language": "julia",
   "name": "julia-1.7"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}