1100.Intro-to-neurons.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Intro to neurons\n",
    "\n",
    "At this point, we know how to build models and have a computer automatically learn how to match the model to data. This is the core of how any machine learning method works.\n",
    "\n",
    "Now, let's narrow our focus and look at **neural networks**. Neural networks (or \"neural nets\", for short) are a specific choice of a **model**. It's a network made up of **neurons**; this, in turn, leads to the question, \"what is a neuron?\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Models with multiple inputs\n",
    "\n",
    "So far, we have been using the sigmoid function as our model. One of the forms of the sigmoid function we've used is\n",
    "\n",
    "$$\\sigma_{w, b}(x) = \\frac{1}{1 + \\exp(-wx + b)},$$\n",
    "\n",
    "where `x` and `w` are both single numbers. We have been using this function to model how the amount of the color green in an image (`x`) can be used to determine if an image shows an apple or a banana.\n",
    "\n",
    "But what if we had multiple data features we wanted to fit?\n",
    "\n",
    "We could then extend our model to include multiple features like\n",
    "\n",
    "$$\\sigma_{\\mathbf{w},b}(\\mathbf{x}) = \\frac{1}{1 + \\exp(-w_1 x_1 - w_2 x_2 - \\cdots - w_n x_n + b)}$$\n",
    "\n",
    "Note that now $\\mathbf{x}$ and $\\mathbf{w}$ are vectors with many components, rather than a single number.\n",
    "\n",
    "For example, $x_1$ might be the amount of the color green in an image, $x_2$ could be the height of the object in the picture, $x_3$ could be the width, and so forth. We can add information for as many features as we have! Our model now has more parameters, but the same idea of gradient descent (\"rolling the ball downhill\") will still work to train our model.\n",
    "\n",
    "This version of the sigmoid model that takes multiple inputs is an example of a **neuron**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the video, we see that one huge class of learning techniques is based around neurons, that is, *artificial neurons*. These are caricatures of real, biological neurons. Both *artificial* and *biological* neurons have several inputs $x_1, \\ldots, x_n$, and a single output, $y$. Schematically they look like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"600\" height=\"400\" viewBox=\"0 0 2400 1600\">\n",
       "<defs>\n",
       "  <clipPath id=\"clip170\">\n",
       "    <rect x=\"0\" y=\"0\" width=\"2400\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip170)\" d=\"\n",
       "M0 1600 L2400 1600 L2400 0 L0 0  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip171\">\n",
       "    <rect x=\"480\" y=\"0\" width=\"1681\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip170)\" d=\"\n",
       "M192.941 1486.45 L2352.76 1486.45 L2352.76 47.2441 L192.941 47.2441  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip172\">\n",
       "    <rect x=\"192\" y=\"47\" width=\"2161\" height=\"1440\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polyline clip-path=\"url(#clip172)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:0.5; fill:none\" points=\"\n",
       "  769.671,1365.85 977.332,1365.85 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip172)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:0.5; fill:none\" points=\"\n",
       "  769.671,966.505 977.332,966.505 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip172)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:0.5; fill:none\" points=\"\n",
       "  769.671,567.159 977.332,567.159 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip172)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:0.5; fill:none\" points=\"\n",
       "  769.671,167.812 977.332,167.812 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip172)\" d=\"\n",
       "M1153.04 1365.85 L1152.65 1357.88 L1151.45 1349.98 L1149.48 1342.25 L1146.74 1334.75 L1143.27 1327.56 L1139.09 1320.75 L1134.26 1314.4 L1128.82 1308.56 L1122.82 1303.29 \n",
       "  L1116.33 1298.64 L1109.4 1294.67 L1102.12 1291.41 L1094.54 1288.89 L1086.75 1287.15 L1078.82 1286.18 L1070.84 1286.02 L1062.88 1286.65 L1055.03 1288.07 L1047.35 1290.27 \n",
       "  L1039.94 1293.23 L1032.85 1296.91 L1026.17 1301.28 L1019.96 1306.29 L1014.28 1311.9 L1009.19 1318.05 L1004.74 1324.68 L1000.97 1331.72 L997.92 1339.1 L995.625 1346.74 \n",
       "  L994.105 1354.58 L993.374 1362.53 L993.442 1370.51 L994.305 1378.45 L995.957 1386.26 L998.381 1393.87 L1001.55 1401.2 L1005.44 1408.17 L1010 1414.72 L1015.2 1420.78 \n",
       "  L1020.97 1426.3 L1027.26 1431.21 L1034.02 1435.46 L1041.16 1439.03 L1048.63 1441.86 L1056.34 1443.93 L1064.22 1445.22 L1072.19 1445.72 L1080.16 1445.42 L1088.07 1444.32 \n",
       "  L1095.83 1442.44 L1103.36 1439.8 L1110.59 1436.41 L1117.45 1432.33 L1123.87 1427.57 L1129.78 1422.2 L1135.12 1416.27 L1139.84 1409.84 L1143.9 1402.96 L1147.25 1395.71 \n",
       "  L1149.86 1388.17 L1151.71 1380.4 L1152.77 1372.49 L1153.04 1365.85  Z\n",
       "  \" fill=\"#008000\" fill-rule=\"evenodd\" fill-opacity=\"0.5\"/>\n",
       "<polyline clip-path=\"url(#clip172)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:0.5; fill:none\" points=\"\n",
       "  1153.04,1365.85 1152.65,1357.88 1151.45,1349.98 1149.48,1342.25 1146.74,1334.75 1143.27,1327.56 1139.09,1320.75 1134.26,1314.4 1128.82,1308.56 1122.82,1303.29 \n",
       "  1116.33,1298.64 1109.4,1294.67 1102.12,1291.41 1094.54,1288.89 1086.75,1287.15 1078.82,1286.18 1070.84,1286.02 1062.88,1286.65 1055.03,1288.07 1047.35,1290.27 \n",
       "  1039.94,1293.23 1032.85,1296.91 1026.17,1301.28 1019.96,1306.29 1014.28,1311.9 1009.19,1318.05 1004.74,1324.68 1000.97,1331.72 997.92,1339.1 995.625,1346.74 \n",
       "  994.105,1354.58 993.374,1362.53 993.442,1370.51 994.305,1378.45 995.957,1386.26 998.381,1393.87 1001.55,1401.2 1005.44,1408.17 1010,1414.72 1015.2,1420.78 \n",
       "  1020.97,1426.3 1027.26,1431.21 1034.02,1435.46 1041.16,1439.03 1048.63,1441.86 1056.34,1443.93 1064.22,1445.22 1072.19,1445.72 1080.16,1445.42 1088.07,1444.32 \n",
       "  1095.83,1442.44 1103.36,1439.8 1110.59,1436.41 1117.45,1432.33 1123.87,1427.57 1129.78,1422.2 1135.12,1416.27 1139.84,1409.84 1143.9,1402.96 1147.25,1395.71 \n",
       "  1149.86,1388.17 1151.71,1380.4 1152.77,1372.49 1153.04,1365.85 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip172)\" d=\"\n",
       "M1153.04 966.505 L1152.65 958.532 L1151.45 950.638 L1149.48 942.902 L1146.74 935.403 L1143.27 928.214 L1139.09 921.408 L1134.26 915.052 L1128.82 909.211 L1122.82 903.942 \n",
       "  L1116.33 899.298 L1109.4 895.325 L1102.12 892.064 L1094.54 889.547 L1086.75 887.798 L1078.82 886.836 L1070.84 886.67 L1062.88 887.302 L1055.03 888.725 L1047.35 890.925 \n",
       "  L1039.94 893.88 L1032.85 897.561 L1026.17 901.931 L1019.96 906.946 L1014.28 912.557 L1009.19 918.706 L1004.74 925.333 L1000.97 932.371 L997.92 939.75 L995.625 947.397 \n",
       "  L994.105 955.234 L993.374 963.184 L993.442 971.168 L994.305 979.105 L995.957 986.915 L998.381 994.522 L1001.55 1001.85 L1005.44 1008.82 L1010 1015.37 L1015.2 1021.44 \n",
       "  L1020.97 1026.95 L1027.26 1031.86 L1034.02 1036.12 L1041.16 1039.68 L1048.63 1042.51 L1056.34 1044.58 L1064.22 1045.87 L1072.19 1046.37 L1080.16 1046.07 L1088.07 1044.97 \n",
       "  L1095.83 1043.09 L1103.36 1040.45 L1110.59 1037.07 L1117.45 1032.98 L1123.87 1028.23 L1129.78 1022.86 L1135.12 1016.92 L1139.84 1010.49 L1143.9 1003.61 L1147.25 996.367 \n",
       "  L1149.86 988.822 L1151.71 981.055 L1152.77 973.142 L1153.04 966.505  Z\n",
       "  \" fill=\"#008000\" fill-rule=\"evenodd\" fill-opacity=\"0.5\"/>\n",
       "<polyline clip-path=\"url(#clip172)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:0.5; fill:none\" points=\"\n",
       "  1153.04,966.505 1152.65,958.532 1151.45,950.638 1149.48,942.902 1146.74,935.403 1143.27,928.214 1139.09,921.408 1134.26,915.052 1128.82,909.211 1122.82,903.942 \n",
       "  1116.33,899.298 1109.4,895.325 1102.12,892.064 1094.54,889.547 1086.75,887.798 1078.82,886.836 1070.84,886.67 1062.88,887.302 1055.03,888.725 1047.35,890.925 \n",
       "  1039.94,893.88 1032.85,897.561 1026.17,901.931 1019.96,906.946 1014.28,912.557 1009.19,918.706 1004.74,925.333 1000.97,932.371 997.92,939.75 995.625,947.397 \n",
       "  994.105,955.234 993.374,963.184 993.442,971.168 994.305,979.105 995.957,986.915 998.381,994.522 1001.55,1001.85 1005.44,1008.82 1010,1015.37 1015.2,1021.44 \n",
       "  1020.97,1026.95 1027.26,1031.86 1034.02,1036.12 1041.16,1039.68 1048.63,1042.51 1056.34,1044.58 1064.22,1045.87 1072.19,1046.37 1080.16,1046.07 1088.07,1044.97 \n",
       "  1095.83,1043.09 1103.36,1040.45 1110.59,1037.07 1117.45,1032.98 1123.87,1028.23 1129.78,1022.86 1135.12,1016.92 1139.84,1010.49 1143.9,1003.61 1147.25,996.367 \n",
       "  1149.86,988.822 1151.71,981.055 1152.77,973.142 1153.04,966.505 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip172)\" d=\"\n",
       "M1153.04 567.159 L1152.65 559.185 L1151.45 551.291 L1149.48 543.556 L1146.74 536.056 L1143.27 528.867 L1139.09 522.061 L1134.26 515.705 L1128.82 509.864 L1122.82 504.595 \n",
       "  L1116.33 499.951 L1109.4 495.978 L1102.12 492.717 L1094.54 490.2 L1086.75 488.451 L1078.82 487.489 L1070.84 487.323 L1062.88 487.955 L1055.03 489.378 L1047.35 491.578 \n",
       "  L1039.94 494.534 L1032.85 498.215 L1026.17 502.584 L1019.96 507.6 L1014.28 513.21 L1009.19 519.359 L1004.74 525.986 L1000.97 533.024 L997.92 540.403 L995.625 548.05 \n",
       "  L994.105 555.887 L993.374 563.838 L993.442 571.821 L994.305 579.758 L995.957 587.568 L998.381 595.175 L1001.55 602.502 L1005.44 609.476 L1010 616.027 L1015.2 622.09 \n",
       "  L1020.97 627.604 L1027.26 632.514 L1034.02 636.771 L1041.16 640.332 L1048.63 643.162 L1056.34 645.233 L1064.22 646.524 L1072.19 647.022 L1080.16 646.722 L1088.07 645.626 \n",
       "  L1095.83 643.747 L1103.36 641.103 L1110.59 637.72 L1117.45 633.631 L1123.87 628.879 L1129.78 623.51 L1135.12 617.577 L1139.84 611.141 L1143.9 604.266 L1147.25 597.02 \n",
       "  L1149.86 589.475 L1151.71 581.708 L1152.77 573.795 L1153.04 567.159  Z\n",
       "  \" fill=\"#008000\" fill-rule=\"evenodd\" fill-opacity=\"0.5\"/>\n",
       "<polyline clip-path=\"url(#clip172)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:0.5; fill:none\" points=\"\n",
       "  1153.04,567.159 1152.65,559.185 1151.45,551.291 1149.48,543.556 1146.74,536.056 1143.27,528.867 1139.09,522.061 1134.26,515.705 1128.82,509.864 1122.82,504.595 \n",
       "  1116.33,499.951 1109.4,495.978 1102.12,492.717 1094.54,490.2 1086.75,488.451 1078.82,487.489 1070.84,487.323 1062.88,487.955 1055.03,489.378 1047.35,491.578 \n",
       "  1039.94,494.534 1032.85,498.215 1026.17,502.584 1019.96,507.6 1014.28,513.21 1009.19,519.359 1004.74,525.986 1000.97,533.024 997.92,540.403 995.625,548.05 \n",
       "  994.105,555.887 993.374,563.838 993.442,571.821 994.305,579.758 995.957,587.568 998.381,595.175 1001.55,602.502 1005.44,609.476 1010,616.027 1015.2,622.09 \n",
       "  1020.97,627.604 1027.26,632.514 1034.02,636.771 1041.16,640.332 1048.63,643.162 1056.34,645.233 1064.22,646.524 1072.19,647.022 1080.16,646.722 1088.07,645.626 \n",
       "  1095.83,643.747 1103.36,641.103 1110.59,637.72 1117.45,633.631 1123.87,628.879 1129.78,623.51 1135.12,617.577 1139.84,611.141 1143.9,604.266 1147.25,597.02 \n",
       "  1149.86,589.475 1151.71,581.708 1152.77,573.795 1153.04,567.159 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip172)\" d=\"\n",
       "M1153.04 167.812 L1152.65 159.838 L1151.45 151.944 L1149.48 144.209 L1146.74 136.709 L1143.27 129.52 L1139.09 122.714 L1134.26 116.358 L1128.82 110.517 L1122.82 105.248 \n",
       "  L1116.33 100.604 L1109.4 96.6314 L1102.12 93.3702 L1094.54 90.8528 L1086.75 89.1043 L1078.82 88.1423 L1070.84 87.9763 L1062.88 88.6079 L1055.03 90.031 L1047.35 92.2312 \n",
       "  L1039.94 95.1866 L1032.85 98.8676 L1026.17 103.237 L1019.96 108.253 L1014.28 113.863 L1009.19 120.012 L1004.74 126.639 L1000.97 133.677 L997.92 141.056 L995.625 148.703 \n",
       "  L994.105 156.54 L993.374 164.491 L993.442 172.474 L994.305 180.411 L995.957 188.222 L998.381 195.828 L1001.55 203.155 L1005.44 210.129 L1010 216.68 L1015.2 222.743 \n",
       "  L1020.97 228.257 L1027.26 233.167 L1034.02 237.424 L1041.16 240.985 L1048.63 243.815 L1056.34 245.886 L1064.22 247.177 L1072.19 247.675 L1080.16 247.375 L1088.07 246.279 \n",
       "  L1095.83 244.4 L1103.36 241.756 L1110.59 238.373 L1117.45 234.284 L1123.87 229.532 L1129.78 224.163 L1135.12 218.23 L1139.84 211.795 L1143.9 204.919 L1147.25 197.673 \n",
       "  L1149.86 190.128 L1151.71 182.361 L1152.77 174.448 L1153.04 167.812  Z\n",
       "  \" fill=\"#008000\" fill-rule=\"evenodd\" fill-opacity=\"0.5\"/>\n",
       "<polyline clip-path=\"url(#clip172)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:0.5; fill:none\" points=\"\n",
       "  1153.04,167.812 1152.65,159.838 1151.45,151.944 1149.48,144.209 1146.74,136.709 1143.27,129.52 1139.09,122.714 1134.26,116.358 1128.82,110.517 1122.82,105.248 \n",
       "  1116.33,100.604 1109.4,96.6314 1102.12,93.3702 1094.54,90.8528 1086.75,89.1043 1078.82,88.1423 1070.84,87.9763 1062.88,88.6079 1055.03,90.031 1047.35,92.2312 \n",
       "  1039.94,95.1866 1032.85,98.8676 1026.17,103.237 1019.96,108.253 1014.28,113.863 1009.19,120.012 1004.74,126.639 1000.97,133.677 997.92,141.056 995.625,148.703 \n",
       "  994.105,156.54 993.374,164.491 993.442,172.474 994.305,180.411 995.957,188.222 998.381,195.828 1001.55,203.155 1005.44,210.129 1010,216.68 1015.2,222.743 \n",
       "  1020.97,228.257 1027.26,233.167 1034.02,237.424 1041.16,240.985 1048.63,243.815 1056.34,245.886 1064.22,247.177 1072.19,247.675 1080.16,247.375 1088.07,246.279 \n",
       "  1095.83,244.4 1103.36,241.756 1110.59,238.373 1117.45,234.284 1123.87,229.532 1129.78,224.163 1135.12,218.23 1139.84,211.795 1143.9,204.919 1147.25,197.673 \n",
       "  1149.86,190.128 1151.71,182.361 1152.77,174.448 1153.04,167.812 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip172)\" d=\"\n",
       "M1552.39 766.832 L1551.99 758.858 L1550.8 750.964 L1548.82 743.229 L1546.09 735.729 L1542.61 728.541 L1538.44 721.734 L1533.61 715.379 L1528.17 709.537 L1522.17 704.268 \n",
       "  L1515.68 699.624 L1508.75 695.652 L1501.46 692.391 L1493.89 689.873 L1486.1 688.125 L1478.17 687.163 L1470.19 686.997 L1462.23 687.628 L1454.38 689.051 L1446.7 691.252 \n",
       "  L1439.28 694.207 L1432.2 697.888 L1425.52 702.258 L1419.31 707.273 L1413.63 712.883 L1408.53 719.032 L1404.08 725.659 L1400.31 732.697 L1397.27 740.077 L1394.97 747.723 \n",
       "  L1393.45 755.561 L1392.72 763.511 L1392.79 771.494 L1393.65 779.431 L1395.3 787.242 L1397.73 794.849 L1400.9 802.176 L1404.78 809.15 L1409.35 815.701 L1414.54 821.763 \n",
       "  L1420.32 827.277 L1426.61 832.187 L1433.36 836.444 L1440.51 840.006 L1447.98 842.836 L1455.69 844.907 L1463.56 846.198 L1471.53 846.695 L1479.51 846.395 L1487.42 845.3 \n",
       "  L1495.18 843.421 L1502.71 840.776 L1509.94 837.393 L1516.8 833.305 L1523.21 828.552 L1529.12 823.183 L1534.47 817.251 L1539.19 810.815 L1543.25 803.94 L1546.6 796.693 \n",
       "  L1549.21 789.149 L1551.05 781.381 L1552.11 773.468 L1552.39 766.832  Z\n",
       "  \" fill=\"#ff0000\" fill-rule=\"evenodd\" fill-opacity=\"0.5\"/>\n",
       "<polyline clip-path=\"url(#clip172)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:0.5; fill:none\" points=\"\n",
       "  1552.39,766.832 1551.99,758.858 1550.8,750.964 1548.82,743.229 1546.09,735.729 1542.61,728.541 1538.44,721.734 1533.61,715.379 1528.17,709.537 1522.17,704.268 \n",
       "  1515.68,699.624 1508.75,695.652 1501.46,692.391 1493.89,689.873 1486.1,688.125 1478.17,687.163 1470.19,686.997 1462.23,687.628 1454.38,689.051 1446.7,691.252 \n",
       "  1439.28,694.207 1432.2,697.888 1425.52,702.258 1419.31,707.273 1413.63,712.883 1408.53,719.032 1404.08,725.659 1400.31,732.697 1397.27,740.077 1394.97,747.723 \n",
       "  1393.45,755.561 1392.72,763.511 1392.79,771.494 1393.65,779.431 1395.3,787.242 1397.73,794.849 1400.9,802.176 1404.78,809.15 1409.35,815.701 1414.54,821.763 \n",
       "  1420.32,827.277 1426.61,832.187 1433.36,836.444 1440.51,840.006 1447.98,842.836 1455.69,844.907 1463.56,846.198 1471.53,846.695 1479.51,846.395 1487.42,845.3 \n",
       "  1495.18,843.421 1502.71,840.776 1509.94,837.393 1516.8,833.305 1523.21,828.552 1529.12,823.183 1534.47,817.251 1539.19,810.815 1543.25,803.94 1546.6,796.693 \n",
       "  1549.21,789.149 1551.05,781.381 1552.11,773.468 1552.39,766.832 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip172)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:0.5; fill:none\" points=\"\n",
       "  1126.34,1286.11 1419.36,846.578 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip172)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:0.5; fill:none\" points=\"\n",
       "  1158.9,923.643 1386.8,809.694 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip172)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:0.5; fill:none\" points=\"\n",
       "  1158.9,610.021 1386.8,723.97 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip172)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:0.5; fill:none\" points=\"\n",
       "  1126.34,247.558 1419.36,687.086 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip172)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:0.5; fill:none\" points=\"\n",
       "  1568.37,766.832 1776.03,766.832 \n",
       "  \"/>\n",
       "</svg>\n"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "include(\"scripts/draw_neural_net.jl\")\n",
    "\n",
    "number_inputs, number_neurons = 4, 1\n",
    "\n",
    "draw_network([number_inputs, number_neurons])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We should read this as showing how information flows from left to right:\n",
    "- 4 pieces of input information arrive (shown in green on the left);\n",
    "\n",
    "- the neuron (shown in red on the right) receives all the inputs, processes them, and outputs a single number to the right."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In other words, a neuron is just a type of function that takes multiple inputs and returns a single output.\n",
    "\n",
    "The simplest interesting case that we will look at in this notebook is when there are just two pieces of input data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"600\" height=\"400\" viewBox=\"0 0 2400 1600\">\n",
       "<defs>\n",
       "  <clipPath id=\"clip210\">\n",
       "    <rect x=\"0\" y=\"0\" width=\"2400\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip210)\" d=\"\n",
       "M0 1600 L2400 1600 L2400 0 L0 0  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip211\">\n",
       "    <rect x=\"480\" y=\"0\" width=\"1681\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip210)\" d=\"\n",
       "M186.274 1486.45 L2352.76 1486.45 L2352.76 47.2441 L186.274 47.2441  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip212\">\n",
       "    <rect x=\"186\" y=\"47\" width=\"2167\" height=\"1440\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polyline clip-path=\"url(#clip212)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:0.5; fill:none\" points=\"\n",
       "  247.59,1172.34 669.337,1172.34 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip212)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:0.5; fill:none\" points=\"\n",
       "  247.59,361.292 669.337,361.292 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip212)\" d=\"\n",
       "M1026.2 1172.34 L1025.39 1156.15 L1022.97 1140.12 L1018.95 1124.41 L1013.39 1109.18 L1006.34 1094.58 L997.867 1080.75 L988.054 1067.84 L977.002 1055.98 L964.821 1045.28 \n",
       "  L951.632 1035.85 L937.567 1027.78 L922.767 1021.16 L907.38 1016.04 L891.559 1012.49 L875.463 1010.54 L859.253 1010.2 L843.089 1011.49 L827.134 1014.38 L811.548 1018.84 \n",
       "  L796.486 1024.85 L782.098 1032.32 L768.528 1041.2 L755.912 1051.38 L744.376 1062.78 L734.035 1075.27 L724.993 1088.72 L717.339 1103.02 L711.151 1118.01 L706.489 1133.53 \n",
       "  L703.402 1149.45 L701.919 1165.6 L702.055 1181.81 L703.81 1197.93 L707.164 1213.8 L712.086 1229.24 L718.525 1244.12 L726.418 1258.29 L735.686 1271.59 L746.235 1283.91 \n",
       "  L757.961 1295.1 L770.747 1305.08 L784.464 1313.72 L798.975 1320.96 L814.136 1326.7 L829.796 1330.91 L845.797 1333.53 L861.979 1334.54 L878.182 1333.93 L894.243 1331.71 \n",
       "  L910.002 1327.89 L925.301 1322.52 L939.987 1315.65 L953.914 1307.35 L966.943 1297.69 L978.943 1286.79 L989.794 1274.74 L999.388 1261.67 L1007.63 1247.71 L1014.44 1232.99 \n",
       "  L1019.74 1217.67 L1023.49 1201.89 L1025.64 1185.82 L1026.2 1172.34  Z\n",
       "  \" fill=\"#008000\" fill-rule=\"evenodd\" fill-opacity=\"0.5\"/>\n",
       "<polyline clip-path=\"url(#clip212)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:0.5; fill:none\" points=\"\n",
       "  1026.2,1172.34 1025.39,1156.15 1022.97,1140.12 1018.95,1124.41 1013.39,1109.18 1006.34,1094.58 997.867,1080.75 988.054,1067.84 977.002,1055.98 964.821,1045.28 \n",
       "  951.632,1035.85 937.567,1027.78 922.767,1021.16 907.38,1016.04 891.559,1012.49 875.463,1010.54 859.253,1010.2 843.089,1011.49 827.134,1014.38 811.548,1018.84 \n",
       "  796.486,1024.85 782.098,1032.32 768.528,1041.2 755.912,1051.38 744.376,1062.78 734.035,1075.27 724.993,1088.72 717.339,1103.02 711.151,1118.01 706.489,1133.53 \n",
       "  703.402,1149.45 701.919,1165.6 702.055,1181.81 703.81,1197.93 707.164,1213.8 712.086,1229.24 718.525,1244.12 726.418,1258.29 735.686,1271.59 746.235,1283.91 \n",
       "  757.961,1295.1 770.747,1305.08 784.464,1313.72 798.975,1320.96 814.136,1326.7 829.796,1330.91 845.797,1333.53 861.979,1334.54 878.182,1333.93 894.243,1331.71 \n",
       "  910.002,1327.89 925.301,1322.52 939.987,1315.65 953.914,1307.35 966.943,1297.69 978.943,1286.79 989.794,1274.74 999.388,1261.67 1007.63,1247.71 1014.44,1232.99 \n",
       "  1019.74,1217.67 1023.49,1201.89 1025.64,1185.82 1026.2,1172.34 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip212)\" d=\"\n",
       "M1026.2 361.292 L1025.39 345.098 L1022.97 329.065 L1018.95 313.355 L1013.39 298.124 L1006.34 283.524 L997.867 269.701 L988.054 256.793 L977.002 244.929 L964.821 234.228 \n",
       "  L951.632 224.796 L937.567 216.729 L922.767 210.105 L907.38 204.993 L891.559 201.441 L875.463 199.488 L859.253 199.15 L843.089 200.433 L827.134 203.323 L811.548 207.792 \n",
       "  L796.486 213.794 L782.098 221.27 L768.528 230.145 L755.912 240.331 L744.376 251.725 L734.035 264.213 L724.993 277.672 L717.339 291.966 L711.151 306.953 L706.489 322.483 \n",
       "  L703.402 338.401 L701.919 354.547 L702.055 370.761 L703.81 386.88 L707.164 402.743 L712.086 418.192 L718.525 433.073 L726.418 447.237 L735.686 460.541 L746.235 472.854 \n",
       "  L757.961 484.053 L770.747 494.025 L784.464 502.67 L798.975 509.903 L814.136 515.651 L829.796 519.857 L845.797 522.479 L861.979 523.49 L878.182 522.88 L894.243 520.656 \n",
       "  L910.002 516.839 L925.301 511.468 L939.987 504.597 L953.914 496.294 L966.943 486.642 L978.943 475.738 L989.794 463.69 L999.388 450.619 L1007.63 436.655 L1014.44 421.938 \n",
       "  L1019.74 406.616 L1023.49 390.84 L1025.64 374.77 L1026.2 361.292  Z\n",
       "  \" fill=\"#008000\" fill-rule=\"evenodd\" fill-opacity=\"0.5\"/>\n",
       "<polyline clip-path=\"url(#clip212)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:0.5; fill:none\" points=\"\n",
       "  1026.2,361.292 1025.39,345.098 1022.97,329.065 1018.95,313.355 1013.39,298.124 1006.34,283.524 997.867,269.701 988.054,256.793 977.002,244.929 964.821,234.228 \n",
       "  951.632,224.796 937.567,216.729 922.767,210.105 907.38,204.993 891.559,201.441 875.463,199.488 859.253,199.15 843.089,200.433 827.134,203.323 811.548,207.792 \n",
       "  796.486,213.794 782.098,221.27 768.528,230.145 755.912,240.331 744.376,251.725 734.035,264.213 724.993,277.672 717.339,291.966 711.151,306.953 706.489,322.483 \n",
       "  703.402,338.401 701.919,354.547 702.055,370.761 703.81,386.88 707.164,402.743 712.086,418.192 718.525,433.073 726.418,447.237 735.686,460.541 746.235,472.854 \n",
       "  757.961,484.053 770.747,494.025 784.464,502.67 798.975,509.903 814.136,515.651 829.796,519.857 845.797,522.479 861.979,523.49 878.182,522.88 894.243,520.656 \n",
       "  910.002,516.839 925.301,511.468 939.987,504.597 953.914,496.294 966.943,486.642 978.943,475.738 989.794,463.69 999.388,450.619 1007.63,436.655 1014.44,421.938 \n",
       "  1019.74,406.616 1023.49,390.84 1025.64,374.77 1026.2,361.292 \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip212)\" d=\"\n",
       "M1837.25 766.818 L1836.44 750.624 L1834.02 734.591 L1830.01 718.881 L1824.45 703.65 L1817.39 689.05 L1808.92 675.227 L1799.11 662.319 L1788.05 650.455 L1775.87 639.754 \n",
       "  L1762.68 630.322 L1748.62 622.255 L1733.82 615.631 L1718.43 610.518 L1702.61 606.967 L1686.52 605.014 L1670.3 604.676 L1654.14 605.959 L1638.19 608.849 L1622.6 613.318 \n",
       "  L1607.54 619.32 L1593.15 626.796 L1579.58 635.671 L1566.96 645.856 L1555.43 657.25 L1545.09 669.739 L1536.04 683.198 L1528.39 697.492 L1522.2 712.479 L1517.54 728.009 \n",
       "  L1514.45 743.926 L1512.97 760.073 L1513.11 776.287 L1514.86 792.406 L1518.22 808.269 L1523.14 823.718 L1529.58 838.599 L1537.47 852.763 L1546.74 866.067 L1557.29 878.38 \n",
       "  L1569.01 889.579 L1581.8 899.551 L1595.52 908.196 L1610.03 915.429 L1625.19 921.177 L1640.85 925.383 L1656.85 928.005 L1673.03 929.016 L1689.23 928.406 L1705.3 926.182 \n",
       "  L1721.05 922.365 L1736.35 916.994 L1751.04 910.123 L1764.97 901.82 L1777.99 892.168 L1789.99 881.264 L1800.85 869.216 L1810.44 856.145 L1818.68 842.181 L1825.49 827.464 \n",
       "  L1830.79 812.142 L1834.54 796.366 L1836.69 780.296 L1837.25 766.818  Z\n",
       "  \" fill=\"#ff0000\" fill-rule=\"evenodd\" fill-opacity=\"0.5\"/>\n",
       "<polyline clip-path=\"url(#clip212)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:0.5; fill:none\" points=\"\n",
       "  1837.25,766.818 1836.44,750.624 1834.02,734.591 1830.01,718.881 1824.45,703.65 1817.39,689.05 1808.92,675.227 1799.11,662.319 1788.05,650.455 1775.87,639.754 \n",
       "  1762.68,630.322 1748.62,622.255 1733.82,615.631 1718.43,610.518 1702.61,606.967 1686.52,605.014 1670.3,604.676 1654.14,605.959 1638.19,608.849 1622.6,613.318 \n",
       "  1607.54,619.32 1593.15,626.796 1579.58,635.671 1566.96,645.856 1555.43,657.25 1545.09,669.739 1536.04,683.198 1528.39,697.492 1522.2,712.479 1517.54,728.009 \n",
       "  1514.45,743.926 1512.97,760.073 1513.11,776.287 1514.86,792.406 1518.22,808.269 1523.14,823.718 1529.58,838.599 1537.47,852.763 1546.74,866.067 1557.29,878.38 \n",
       "  1569.01,889.579 1581.8,899.551 1595.52,908.196 1610.03,915.429 1625.19,921.177 1640.85,925.383 1656.85,928.005 1673.03,929.016 1689.23,928.406 1705.3,926.182 \n",
       "  1721.05,922.365 1736.35,916.994 1751.04,910.123 1764.97,901.82 1777.99,892.168 1789.99,881.264 1800.85,869.216 1810.44,856.145 1818.68,842.181 1825.49,827.464 \n",
       "  1830.79,812.142 1834.54,796.366 1836.69,780.296 1837.25,766.818 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip212)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:0.5; fill:none\" points=\"\n",
       "  1038.09,1085.29 1500.94,853.869 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip212)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:0.5; fill:none\" points=\"\n",
       "  1038.09,448.343 1500.94,679.766 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip212)\" style=\"stroke:#000000; stroke-linecap:butt; stroke-linejoin:round; stroke-width:4; stroke-opacity:0.5; fill:none\" points=\"\n",
       "  1869.69,766.818 2291.44,766.818 \n",
       "  \"/>\n",
       "</svg>\n"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "draw_network([2, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each link between circles above represents a **weight** $w$ that can be modified to allow the neuron to learn, so in this case the neuron has two weights, $w_1$ and $w_2$.\n",
    "\n",
    "The neuron also has a single bias $b$, and an **activation function**, which we will take to be the $\\sigma$ sigmoidal function that we have been using. (Note that other activation functions can be used!)\n",
    "\n",
    "Let's call our neuron $f_{w_1,w_2, b}(x_1, x_2)$, where\n",
    "\n",
    "$$f_{w_1,w_2, b}(x_1, x_2) := \\sigma(w_1 x_1 + w_2 x_2 + b).$$\n",
    "\n",
    "Note that $f_{w_1,w_2, b}(x_1, x_2)$ has 3 parameters: two weights and a bias.\n",
    "\n",
    "To simplify the notation, and to prepare for more complicated scenarios later, we put the two weights into a vector, and the two data values into another vector:\n",
    "\n",
    "$$\n",
    "\\mathbf{w} = \\begin{pmatrix} w_1 \\\\ w_2 \\end{pmatrix};\n",
    "\\qquad\n",
    "\\mathbf{x} = \\begin{pmatrix} x_1 \\\\ x_2 \\end{pmatrix}.\n",
    "$$\n",
    "\n",
    "We thus have\n",
    "\n",
    "$$f_{\\mathbf{w}, b}(\\mathbf{x}) = \\sigma(\\mathbf{w} \\cdot \\mathbf{x} + b),$$\n",
    "\n",
    "where the dot product $\\mathbf{w} \\cdot \\mathbf{x}$ is an abbreviated syntax for $w_1 x_1 + w_2 x_2$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 1\n",
    "\n",
    "Declare the function `f(x, w, b)` in Julia. `f` should take vectors `x` and `w` as vectors and `b` as a scalar. Furthermore `f` should call\n",
    "\n",
    "```julia\n",
    "σ(x) = 1 / (1 + exp(-x))\n",
    "```\n",
    "\n",
    "What output do you get for\n",
    "\n",
    "```julia\n",
    "f(3, 4, 5)\n",
    "```\n",
    "?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Solution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "f (generic function with 1 method)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "σ(x) = 1 / (1 + exp(-x))\n",
    "\n",
    "f(x, w, b) = σ(w'x + b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Tests**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "@assert f(3, 4, 5) ≈ 0.99999996"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 1.6.0",
   "language": "julia",
   "name": "julia-1.6"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}