H20.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "H20.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyMXmfn/iEgsOjmdTpguKaLY",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/sidharth178/AutoML/blob/master/H20.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nobXHqiXDcu3"
      },
      "source": [
        "\n",
        "<h1 align=\"center\"><b> H2O - AutoML</h1>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BAlZpb0Tytda"
      },
      "source": [
        "- H2O is a fully open-source, distributed in-memory machine learning platform with linear scalability.\n",
        "- H2O’s AutoML can be used for automating the machine learning workflow, which includes automatic training and tuning of many models within a user-specified time-limit.\n",
        "- H2O only supports Java SE Runtime Environment Versions 8–11\n",
        "\n",
        "### **H2O AutoML functionalities**\n",
        "H2O’s AutoML is equipped with the following functionalities:\n",
        "\n",
        "- Necessary data pre-processing capabilities( as in all H2O algorithms ).\n",
        "- Trains a Random grid of algorithms like GBMs, DNNs, GLMs, etc. using a carefully chosen hyper-parameter space.\n",
        "- Individual models are tuned using cross-validation.\n",
        "- Two Stacked Ensembles are trained. One ensemble contains all the models (optimized for model performance), and the other ensemble provides just the best performing model from each algorithm class/family (optimized for production use).\n",
        "- Returns a sorted “Leaderboard” of all models.\n",
        "- All models can be easily exported to production.\n",
        "\n",
        "### **Context**\n",
        "  1. **Important Optimal Parameters**\n",
        "  2. **Install H2O**\n",
        "  3. **Start H2O**\n",
        "  4. **Loading data into H2O**\n",
        "  5. **Split the dataset**\n",
        "  6. **H2O Model Building**\n",
        "  7. **Train H2O Model**\n",
        "  8. **Show H20 Leaderboard**\n",
        "  9. **Get Best Model**\n",
        "  10. **Find Prediction**\n",
        "  11. **Show Model Performance**\n",
        "  12. **Show All Trained Model**\n",
        "  13. **Train Model Uning a Specific Algo**\n",
        "  14. **Retrive Model Performance**\n",
        "  15. **Show Variable Importance Plot**\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KpYLW8kafdk8"
      },
      "source": [
        "### **1. Important Optional Parameters**\n",
        "\n",
        "**max_runtime_secs:**[0] This specifies the maximum time that the AutoML process will run for. The default is 0 (no limit)\n",
        "\n",
        "**max_models:**[NULL] Specify the maximum number of models to build in an AutoML run, excluding the Stacked Ensemble models.\n",
        "\n",
        "**max_runtime_secs_per_model:**[0] Specify the max amount of time dedicated to the training of each individual model.\n",
        "\n",
        "**stopping_metric:**[auto] Specify the metric to use for early stopping. Defaults to AUTO. The available options are:\n",
        "AUTO: This defaults to logloss for classification and deviance for regression. Ex:[ deviance (mean residual deviance), logloss, MSE, RMSE, MAE, RMSLE, AUC (area under the ROC curve), AUCPR (area under the Precision-Recall curve), lift_top_group, misclassification, mean_per_class_error ]\n",
        "\n",
        "**sort_metric:** Specifies the metric used to sort the Leaderboard by at the end of an AutoML run. Available options include:\n",
        "AUTO: This defaults to AUC for binary classification, mean_per_class_error for multinomial classification, and deviance for regression. Ex:[ deviance (mean residual deviance), logloss, MSE, RMSE, MAE, RMSLE, AUC (area under the ROC curve), AUCPR (area under the Precision-Recall curve), mean_per_class_error ]\n",
        "\n",
        "**exclude_algos:** A list/vector of character strings naming the algorithms to skip during the model-building phase.\n",
        "\n",
        "**include_algos:** A list/vector of character strings naming the algorithms to include during the model-building phase.\n",
        "\n",
        "**nfolds:** Number of folds to be used for cross validation. For more on this, check out the H2O explanation of cross validation.\n",
        "\n",
        "**balance_classes:**[False] When we have a class imbalance in our target feature, we may want to resample our data, either by over-sampling the minority class or under-sampling the majority class when creating the new distribution.\n",
        "\n",
        "**ignored_columns:** To ignore a column"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R7woYzkl0xgM"
      },
      "source": [
        "### **2. Install H2O**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vqoz4SuucHEK",
        "outputId": "846b5073-9859-44ae-95e1-c1577fb3d2ae"
      },
      "source": [
        "!pip install h2o"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting h2o\n",
            "  Downloading h2o-3.34.0.3.tar.gz (175.8 MB)\n",
            "\u001b[K     |████████████████████████████████| 175.8 MB 22 kB/s \n",
            "\u001b[?25hRequirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from h2o) (2.23.0)\n",
            "Requirement already satisfied: tabulate in /usr/local/lib/python3.7/dist-packages (from h2o) (0.8.9)\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.7/dist-packages (from h2o) (0.16.0)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->h2o) (3.0.4)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->h2o) (2.10)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->h2o) (2021.10.8)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->h2o) (1.24.3)\n",
            "Building wheels for collected packages: h2o\n",
            "  Building wheel for h2o (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for h2o: filename=h2o-3.34.0.3-py2.py3-none-any.whl size=175832480 sha256=4fd7ecf60ae160f36795828b9da6fdfee1812b188a1e880105924efd9ebd0106\n",
            "  Stored in directory: /root/.cache/pip/wheels/2a/52/cf/01f6938288f0731963463ff72510db67f9f3076f801038b2f7\n",
            "Successfully built h2o\n",
            "Installing collected packages: h2o\n",
            "Successfully installed h2o-3.34.0.3\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5w4dxwv1Dbsl"
      },
      "source": [
        "### **3. Start H2O**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C8jbWP4QcHIa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 533
        },
        "outputId": "76200c7f-3dd4-4d9e-95ed-a4f4a7205920"
      },
      "source": [
        "# initializing a local H2O cluster\n",
        "import h2o\n",
        "from h2o.automl import H2OAutoML\n",
        "h2o.init() # It'll makes sure that no prior instance of H2O is running"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Checking whether there is an H2O instance running at http://localhost:54321 ..... not found.\n",
            "Attempting to start a local H2O server...\n",
            "  Java Version: openjdk version \"11.0.11\" 2021-04-20; OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.18.04); OpenJDK 64-Bit Server VM (build 11.0.11+9-Ubuntu-0ubuntu2.18.04, mixed mode, sharing)\n",
            "  Starting server from /usr/local/lib/python3.7/dist-packages/h2o/backend/bin/h2o.jar\n",
            "  Ice root: /tmp/tmp7n47thu6\n",
            "  JVM stdout: /tmp/tmp7n47thu6/h2o_unknownUser_started_from_python.out\n",
            "  JVM stderr: /tmp/tmp7n47thu6/h2o_unknownUser_started_from_python.err\n",
            "  Server is running at http://127.0.0.1:54321\n",
            "Connecting to H2O server at http://127.0.0.1:54321 ... successful.\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td>H2O_cluster_uptime:</td>\n",
              "<td>02 secs</td></tr>\n",
              "<tr><td>H2O_cluster_timezone:</td>\n",
              "<td>Etc/UTC</td></tr>\n",
              "<tr><td>H2O_data_parsing_timezone:</td>\n",
              "<td>UTC</td></tr>\n",
              "<tr><td>H2O_cluster_version:</td>\n",
              "<td>3.34.0.3</td></tr>\n",
              "<tr><td>H2O_cluster_version_age:</td>\n",
              "<td>1 month and 7 days </td></tr>\n",
              "<tr><td>H2O_cluster_name:</td>\n",
              "<td>H2O_from_python_unknownUser_zedsg2</td></tr>\n",
              "<tr><td>H2O_cluster_total_nodes:</td>\n",
              "<td>1</td></tr>\n",
              "<tr><td>H2O_cluster_free_memory:</td>\n",
              "<td>3.172 Gb</td></tr>\n",
              "<tr><td>H2O_cluster_total_cores:</td>\n",
              "<td>2</td></tr>\n",
              "<tr><td>H2O_cluster_allowed_cores:</td>\n",
              "<td>2</td></tr>\n",
              "<tr><td>H2O_cluster_status:</td>\n",
              "<td>locked, healthy</td></tr>\n",
              "<tr><td>H2O_connection_url:</td>\n",
              "<td>http://127.0.0.1:54321</td></tr>\n",
              "<tr><td>H2O_connection_proxy:</td>\n",
              "<td>{\"http\": null, \"https\": null}</td></tr>\n",
              "<tr><td>H2O_internal_security:</td>\n",
              "<td>False</td></tr>\n",
              "<tr><td>H2O_API_Extensions:</td>\n",
              "<td>Amazon S3, XGBoost, Algos, AutoML, Core V3, TargetEncoder, Core V4</td></tr>\n",
              "<tr><td>Python_version:</td>\n",
              "<td>3.7.12 final</td></tr></table></div>"
            ],
            "text/plain": [
              "--------------------------  ------------------------------------------------------------------\n",
              "H2O_cluster_uptime:         02 secs\n",
              "H2O_cluster_timezone:       Etc/UTC\n",
              "H2O_data_parsing_timezone:  UTC\n",
              "H2O_cluster_version:        3.34.0.3\n",
              "H2O_cluster_version_age:    1 month and 7 days\n",
              "H2O_cluster_name:           H2O_from_python_unknownUser_zedsg2\n",
              "H2O_cluster_total_nodes:    1\n",
              "H2O_cluster_free_memory:    3.172 Gb\n",
              "H2O_cluster_total_cores:    2\n",
              "H2O_cluster_allowed_cores:  2\n",
              "H2O_cluster_status:         locked, healthy\n",
              "H2O_connection_url:         http://127.0.0.1:54321\n",
              "H2O_connection_proxy:       {\"http\": null, \"https\": null}\n",
              "H2O_internal_security:      False\n",
              "H2O_API_Extensions:         Amazon S3, XGBoost, Algos, AutoML, Core V3, TargetEncoder, Core V4\n",
              "Python_version:             3.7.12 final\n",
              "--------------------------  ------------------------------------------------------------------"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t7RA3uqs6R7J"
      },
      "source": [
        "### **4. Loading data into H2O**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CG5-nzHQcHKS",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 260
        },
        "outputId": "bbea79cd-3ce0-4dec-9c05-e356b9dc3a8d"
      },
      "source": [
        "data_path = \"/content/churn_data_st.csv\"\n",
        "\n",
        "# Load data into H2O\n",
        "df = h2o.import_file(data_path)\n",
        "df.head()"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Parse progress: |████████████████████████████████████████████████████████████████| (done) 100%\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table>\n",
              "<thead>\n",
              "<tr><th>customerID  </th><th>gender  </th><th style=\"text-align: right;\">  SeniorCitizen</th><th style=\"text-align: right;\">  tenure</th><th style=\"text-align: right;\">  ServiceCount</th><th>Contract      </th><th>PaperlessBilling  </th><th style=\"text-align: right;\">  MonthlyCharges</th><th style=\"text-align: right;\">  TotalCharges</th><th>Churn  </th></tr>\n",
              "</thead>\n",
              "<tbody>\n",
              "<tr><td>7590-VHVEG  </td><td>Female  </td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">       1</td><td style=\"text-align: right;\">             2</td><td>Month-to-month</td><td>Yes               </td><td style=\"text-align: right;\">           29.85</td><td style=\"text-align: right;\">         29.85</td><td>No     </td></tr>\n",
              "<tr><td>5575-GNVDE  </td><td>Male    </td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">      34</td><td style=\"text-align: right;\">             4</td><td>One year      </td><td>No                </td><td style=\"text-align: right;\">           56.95</td><td style=\"text-align: right;\">       1889.5 </td><td>No     </td></tr>\n",
              "<tr><td>3668-QPYBK  </td><td>Male    </td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">       2</td><td style=\"text-align: right;\">             4</td><td>Month-to-month</td><td>Yes               </td><td style=\"text-align: right;\">           53.85</td><td style=\"text-align: right;\">        108.15</td><td>Yes    </td></tr>\n",
              "<tr><td>7795-CFOCW  </td><td>Male    </td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">      45</td><td style=\"text-align: right;\">             4</td><td>One year      </td><td>No                </td><td style=\"text-align: right;\">           42.3 </td><td style=\"text-align: right;\">       1840.75</td><td>No     </td></tr>\n",
              "<tr><td>9237-HQITU  </td><td>Female  </td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">       2</td><td style=\"text-align: right;\">             2</td><td>Month-to-month</td><td>Yes               </td><td style=\"text-align: right;\">           70.7 </td><td style=\"text-align: right;\">        151.65</td><td>Yes    </td></tr>\n",
              "<tr><td>9305-CDSKC  </td><td>Female  </td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">       8</td><td style=\"text-align: right;\">             6</td><td>Month-to-month</td><td>Yes               </td><td style=\"text-align: right;\">           99.65</td><td style=\"text-align: right;\">        820.5 </td><td>Yes    </td></tr>\n",
              "<tr><td>1452-KIOVK  </td><td>Male    </td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">      22</td><td style=\"text-align: right;\">             5</td><td>Month-to-month</td><td>Yes               </td><td style=\"text-align: right;\">           89.1 </td><td style=\"text-align: right;\">       1949.4 </td><td>No     </td></tr>\n",
              "<tr><td>6713-OKOMC  </td><td>Female  </td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">      10</td><td style=\"text-align: right;\">             2</td><td>Month-to-month</td><td>No                </td><td style=\"text-align: right;\">           29.75</td><td style=\"text-align: right;\">        301.9 </td><td>No     </td></tr>\n",
              "<tr><td>7892-POOKP  </td><td>Female  </td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">      28</td><td style=\"text-align: right;\">             7</td><td>Month-to-month</td><td>Yes               </td><td style=\"text-align: right;\">          104.8 </td><td style=\"text-align: right;\">       3046.05</td><td>Yes    </td></tr>\n",
              "<tr><td>6388-TABGU  </td><td>Male    </td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">      62</td><td style=\"text-align: right;\">             4</td><td>One year      </td><td>No                </td><td style=\"text-align: right;\">           56.15</td><td style=\"text-align: right;\">       3487.95</td><td>No     </td></tr>\n",
              "</tbody>\n",
              "</table>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              ""
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AhNw2o8DQJfX"
      },
      "source": [
        "### **5. Split the dataset**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kmJuHPk16fyc"
      },
      "source": [
        "train, test, valid = df.split_frame(ratios=[0.7, 0.15], seed=1)"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "80swXTyT6f0h"
      },
      "source": [
        "y = \"Churn\"\n",
        "x = df.columns\n",
        "x.remove(y)\n",
        "x.remove(\"customerID\")"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cY0t3t-4cvtS"
      },
      "source": [
        "### **6. H2O Model Building**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "az4yh1166f5L"
      },
      "source": [
        "h2o_model = H2OAutoML(max_models = 10,\n",
        "                      seed = 10,\n",
        "                      exclude_algos = [\"StackedEnsemble\", \"DeepLearning\"],\n",
        "                      verbosity=\"info\",\n",
        "                      nfolds=0,\n",
        "                      max_runtime_secs=120,\n",
        "                      # max_runtime_secs_per_model=30,\n",
        "                      stopping_metric = 'rmse',\n",
        "                      sort_metric = 'rmse',\n",
        "                      balance_classes = True\n",
        "                      )"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4crRHdTfc6l9"
      },
      "source": [
        "### **7. Train H2O Model**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "UhWge8C8NVQO",
        "outputId": "3683354b-4840-4afd-92eb-c7a08cb000c1"
      },
      "source": [
        "h2o_model.train(x = x, y = y, training_frame = train, validation_frame = valid)"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "AutoML progress: |\n",
            "15:23:26.855: Project: AutoML_1_20211114_152326\n",
            "15:23:26.856: Cross-validation disabled by user: no fold column nor nfolds > 1.\n",
            "15:23:26.864: Setting stopping tolerance adaptively based on the training frame: 0.014219127353430849\n",
            "15:23:26.865: Build control seed: 10\n",
            "15:23:26.866: Since cross-validation is disabled, and no leaderboard frame was provided, automatically split the training data into training and leaderboard frames in the ratio 90/10\n",
            "15:23:26.937: training frame: Frame key: AutoML_1_20211114_152326_training_py_3_sid_bf21    cols: 10    rows: 4467  chunks: 8    size: 172858  checksum: -4432760674848075498\n",
            "15:23:26.938: validation frame: Frame key: py_5_sid_bf21    cols: 10    rows: 1067  chunks: 8    size: 45155  checksum: -6380701859518444760\n",
            "15:23:26.941: leaderboard frame: Frame key: AutoML_1_20211114_152326_leaderboard_py_3_sid_bf21    cols: 10    rows: 479  chunks: 8    size: 35286  checksum: 8773663619283614381\n",
            "15:23:26.942: blending frame: NULL\n",
            "15:23:26.942: response column: Churn\n",
            "15:23:26.942: fold column: null\n",
            "15:23:26.942: weights column: null\n",
            "15:23:26.968: Loading execution steps: [{XGBoost : [def_2 (1g, 10w), def_1 (2g, 10w), def_3 (3g, 10w), grid_1 (4g, 90w), lr_search (6g, 30w)]}, {GLM : [def_1 (1g, 10w)]}, {DRF : [def_1 (2g, 10w), XRT (3g, 10w)]}, {GBM : [def_5 (1g, 10w), def_2 (2g, 10w), def_3 (2g, 10w), def_4 (2g, 10w), def_1 (3g, 10w), grid_1 (4g, 60w), lr_annealing (6g, 10w)]}, {DeepLearning : [def_1 (3g, 10w), grid_1 (4g, 30w), grid_2 (5g, 30w), grid_3 (5g, 30w)]}, {completion : [resume_best_grids (10g, 60w)]}, {StackedEnsemble : [best_of_family_1 (1g, 5w), best_of_family_2 (2g, 5w), best_of_family_3 (3g, 5w), best_of_family_4 (4g, 5w), best_of_family_5 (5g, 5w), all_2 (2g, 10w), all_3 (3g, 10w), all_4 (4g, 10w), all_5 (5g, 10w), monotonic (6g, 10w), best_of_family_xgboost (6g, 10w), best_of_family_gbm (6g, 10w), all_xgboost (7g, 10w), all_gbm (7g, 10w), best_of_family_xglm (8g, 10w), all_xglm (8g, 10w), best_of_family (10g, 10w), best_N (10g, 10w)]}]\n",
            "15:23:27.10: Disabling Algo: StackedEnsemble as requested by the user.\n",
            "15:23:27.10: Disabling Algo: DeepLearning as requested by the user.\n",
            "15:23:27.11: AutoML job created: 2021.11.14 15:23:26.785\n",
            "15:23:27.16: AutoML build started: 2021.11.14 15:23:27.15\n",
            "15:23:27.27: AutoML: starting XGBoost_1_AutoML_1_20211114_152326 model training\n",
            "\n",
            "█\n",
            "15:23:30.124: New leader: XGBoost_1_AutoML_1_20211114_152326, rmse: 0.4004339426975205\n",
            "15:23:30.129: AutoML: starting GLM_1_AutoML_1_20211114_152326 model training\n",
            "\n",
            "██\n",
            "15:23:32.169: New leader: GLM_1_AutoML_1_20211114_152326, rmse: 0.3952886001459375\n",
            "15:23:32.173: AutoML: starting GBM_1_AutoML_1_20211114_152326 model training\n",
            "\n",
            "██\n",
            "15:23:36.279: New leader: GBM_1_AutoML_1_20211114_152326, rmse: 0.39242482178020754\n",
            "15:23:36.280: Skipping StackedEnsemble 'best_of_family_1' due to the exclude_algos option or it is already trained.\n",
            "15:23:36.280: AutoML: starting XGBoost_2_AutoML_1_20211114_152326 model training\n",
            "\n",
            "█\n",
            "15:23:38.336: AutoML: starting DRF_1_AutoML_1_20211114_152326 model training\n",
            "\n",
            "██\n",
            "15:23:43.398: AutoML: starting GBM_2_AutoML_1_20211114_152326 model training\n",
            "\n",
            "█\n",
            "15:23:45.451: AutoML: starting GBM_3_AutoML_1_20211114_152326 model training\n",
            "\n",
            "█\n",
            "15:23:47.474: AutoML: starting GBM_4_AutoML_1_20211114_152326 model training\n",
            "\n",
            "█\n",
            "15:23:49.517: Skipping StackedEnsemble 'best_of_family_2' due to the exclude_algos option or it is already trained.\n",
            "15:23:49.523: Skipping StackedEnsemble 'all_2' due to the exclude_algos option or it is already trained.\n",
            "15:23:49.523: AutoML: starting XGBoost_3_AutoML_1_20211114_152326 model training\n",
            "\n",
            "█\n",
            "15:23:50.542: New leader: XGBoost_3_AutoML_1_20211114_152326, rmse: 0.39156365199507914\n",
            "15:23:50.542: AutoML: starting XRT_1_AutoML_1_20211114_152326 model training\n",
            "\n",
            "██\n",
            "15:23:53.620: Skipping StackedEnsemble 'best_of_family_3' due to the exclude_algos option or it is already trained.\n",
            "15:23:53.621: Skipping StackedEnsemble 'all_3' due to the exclude_algos option or it is already trained.\n",
            "15:23:53.621: Skipping StackedEnsemble 'best_of_family_4' due to the exclude_algos option or it is already trained.\n",
            "15:23:53.621: Skipping StackedEnsemble 'all_4' due to the exclude_algos option or it is already trained.\n",
            "15:23:53.621: Skipping StackedEnsemble 'best_of_family_5' due to the exclude_algos option or it is already trained.\n",
            "15:23:53.622: Skipping StackedEnsemble 'all_5' due to the exclude_algos option or it is already trained.\n",
            "15:23:53.623: Skipping StackedEnsemble 'monotonic' due to the exclude_algos option or it is already trained.\n",
            "15:23:53.623: Skipping StackedEnsemble 'best_of_family_xgboost' due to the exclude_algos option or it is already trained.\n",
            "15:23:53.624: Skipping StackedEnsemble 'best_of_family_gbm' due to the exclude_algos option or it is already trained.\n",
            "15:23:53.627: Skipping StackedEnsemble 'all_xgboost' due to the exclude_algos option or it is already trained.\n",
            "15:23:53.627: Skipping StackedEnsemble 'all_gbm' due to the exclude_algos option or it is already trained.\n",
            "15:23:53.627: Skipping StackedEnsemble 'best_of_family_xglm' due to the exclude_algos option or it is already trained.\n",
            "15:23:53.627: Skipping StackedEnsemble 'all_xglm' due to the exclude_algos option or it is already trained.\n",
            "15:23:53.628: Skipping StackedEnsemble 'best_of_family' due to the exclude_algos option or it is already trained.\n",
            "15:23:53.636: Skipping StackedEnsemble 'best_N' due to the exclude_algos option or it is already trained.\n",
            "15:23:53.636: Actual modeling steps: [{XGBoost : [def_2 (1g, 10w)]}, {GLM : [def_1 (1g, 10w)]}, {GBM : [def_5 (1g, 10w)]}, {XGBoost : [def_1 (2g, 10w)]}, {DRF : [def_1 (2g, 10w)]}, {GBM : [def_2 (2g, 10w), def_3 (2g, 10w), def_4 (2g, 10w)]}, {XGBoost : [def_3 (3g, 10w)]}, {DRF : [XRT (3g, 10w)]}]\n",
            "15:23:53.636: AutoML build stopped: 2021.11.14 15:23:53.636\n",
            "15:23:53.636: AutoML build done: built 10 models\n",
            "15:23:53.636: AutoML duration: 26.621 sec\n",
            "\n",
            "█████████████████████████████████████████████████| (done) 100%\n",
            "Model Details\n",
            "=============\n",
            "H2OXGBoostEstimator :  XGBoost\n",
            "Model Key:  XGBoost_3_AutoML_1_20211114_152326\n",
            "\n",
            "\n",
            "Model Summary: \n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th>number_of_trees</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td></td>\n",
              "      <td>30.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "     number_of_trees\n",
              "0               30.0"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "\n",
            "ModelMetricsBinomial: xgboost\n",
            "** Reported on train data. **\n",
            "\n",
            "MSE: 0.11133195820860814\n",
            "RMSE: 0.3336644395326061\n",
            "LogLoss: 0.34625350893801693\n",
            "Mean Per-Class Error: 0.18452457278767354\n",
            "AUC: 0.8994898011943848\n",
            "AUCPR: 0.7645364251698769\n",
            "Gini: 0.7989796023887696\n",
            "\n",
            "Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.37355014085769656: \n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th>No</th>\n",
              "      <th>Yes</th>\n",
              "      <th>Error</th>\n",
              "      <th>Rate</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>No</td>\n",
              "      <td>2803.0</td>\n",
              "      <td>489.0</td>\n",
              "      <td>0.1485</td>\n",
              "      <td>(489.0/3292.0)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Yes</td>\n",
              "      <td>285.0</td>\n",
              "      <td>890.0</td>\n",
              "      <td>0.2426</td>\n",
              "      <td>(285.0/1175.0)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Total</td>\n",
              "      <td>3088.0</td>\n",
              "      <td>1379.0</td>\n",
              "      <td>0.1733</td>\n",
              "      <td>(774.0/4467.0)</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "              No     Yes   Error             Rate\n",
              "0     No  2803.0   489.0  0.1485   (489.0/3292.0)\n",
              "1    Yes   285.0   890.0  0.2426   (285.0/1175.0)\n",
              "2  Total  3088.0  1379.0  0.1733   (774.0/4467.0)"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Maximum Metrics: Maximum metrics at their respective thresholds\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>metric</th>\n",
              "      <th>threshold</th>\n",
              "      <th>value</th>\n",
              "      <th>idx</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>max f1</td>\n",
              "      <td>0.373550</td>\n",
              "      <td>0.696946</td>\n",
              "      <td>202.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>max f2</td>\n",
              "      <td>0.169555</td>\n",
              "      <td>0.798156</td>\n",
              "      <td>291.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>max f0point5</td>\n",
              "      <td>0.496359</td>\n",
              "      <td>0.718670</td>\n",
              "      <td>152.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>max accuracy</td>\n",
              "      <td>0.496359</td>\n",
              "      <td>0.845086</td>\n",
              "      <td>152.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>max precision</td>\n",
              "      <td>0.961415</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>max recall</td>\n",
              "      <td>0.010989</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>387.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>max specificity</td>\n",
              "      <td>0.961415</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>max absolute_mcc</td>\n",
              "      <td>0.463329</td>\n",
              "      <td>0.585210</td>\n",
              "      <td>166.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>max min_per_class_accuracy</td>\n",
              "      <td>0.321204</td>\n",
              "      <td>0.808323</td>\n",
              "      <td>224.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>max mean_per_class_accuracy</td>\n",
              "      <td>0.294099</td>\n",
              "      <td>0.815475</td>\n",
              "      <td>235.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>max tns</td>\n",
              "      <td>0.961415</td>\n",
              "      <td>3292.000000</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>max fns</td>\n",
              "      <td>0.961415</td>\n",
              "      <td>1171.000000</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>max fps</td>\n",
              "      <td>0.001868</td>\n",
              "      <td>3292.000000</td>\n",
              "      <td>399.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>max tps</td>\n",
              "      <td>0.010989</td>\n",
              "      <td>1175.000000</td>\n",
              "      <td>387.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>max tnr</td>\n",
              "      <td>0.961415</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>max fnr</td>\n",
              "      <td>0.961415</td>\n",
              "      <td>0.996596</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>max fpr</td>\n",
              "      <td>0.001868</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>399.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>max tpr</td>\n",
              "      <td>0.010989</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>387.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                         metric  threshold        value    idx\n",
              "0                        max f1   0.373550     0.696946  202.0\n",
              "1                        max f2   0.169555     0.798156  291.0\n",
              "2                  max f0point5   0.496359     0.718670  152.0\n",
              "3                  max accuracy   0.496359     0.845086  152.0\n",
              "4                 max precision   0.961415     1.000000    0.0\n",
              "5                    max recall   0.010989     1.000000  387.0\n",
              "6               max specificity   0.961415     1.000000    0.0\n",
              "7              max absolute_mcc   0.463329     0.585210  166.0\n",
              "8    max min_per_class_accuracy   0.321204     0.808323  224.0\n",
              "9   max mean_per_class_accuracy   0.294099     0.815475  235.0\n",
              "10                      max tns   0.961415  3292.000000    0.0\n",
              "11                      max fns   0.961415  1171.000000    0.0\n",
              "12                      max fps   0.001868  3292.000000  399.0\n",
              "13                      max tps   0.010989  1175.000000  387.0\n",
              "14                      max tnr   0.961415     1.000000    0.0\n",
              "15                      max fnr   0.961415     0.996596    0.0\n",
              "16                      max fpr   0.001868     1.000000  399.0\n",
              "17                      max tpr   0.010989     1.000000  387.0"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Gains/Lift Table: Avg response rate: 26.30 %, avg score: 26.24 %\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>group</th>\n",
              "      <th>cumulative_data_fraction</th>\n",
              "      <th>lower_threshold</th>\n",
              "      <th>lift</th>\n",
              "      <th>cumulative_lift</th>\n",
              "      <th>response_rate</th>\n",
              "      <th>score</th>\n",
              "      <th>cumulative_response_rate</th>\n",
              "      <th>cumulative_score</th>\n",
              "      <th>capture_rate</th>\n",
              "      <th>cumulative_capture_rate</th>\n",
              "      <th>gain</th>\n",
              "      <th>cumulative_gain</th>\n",
              "      <th>kolmogorov_smirnov</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>0.010074</td>\n",
              "      <td>0.909985</td>\n",
              "      <td>3.717220</td>\n",
              "      <td>3.717220</td>\n",
              "      <td>0.977778</td>\n",
              "      <td>0.933290</td>\n",
              "      <td>0.977778</td>\n",
              "      <td>0.933290</td>\n",
              "      <td>0.037447</td>\n",
              "      <td>0.037447</td>\n",
              "      <td>271.721986</td>\n",
              "      <td>271.721986</td>\n",
              "      <td>0.037143</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>0.020372</td>\n",
              "      <td>0.873553</td>\n",
              "      <td>3.553765</td>\n",
              "      <td>3.634594</td>\n",
              "      <td>0.934783</td>\n",
              "      <td>0.893587</td>\n",
              "      <td>0.956044</td>\n",
              "      <td>0.913220</td>\n",
              "      <td>0.036596</td>\n",
              "      <td>0.074043</td>\n",
              "      <td>255.376503</td>\n",
              "      <td>263.459434</td>\n",
              "      <td>0.072827</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>0.029998</td>\n",
              "      <td>0.840112</td>\n",
              "      <td>3.713290</td>\n",
              "      <td>3.659848</td>\n",
              "      <td>0.976744</td>\n",
              "      <td>0.858049</td>\n",
              "      <td>0.962687</td>\n",
              "      <td>0.895516</td>\n",
              "      <td>0.035745</td>\n",
              "      <td>0.109787</td>\n",
              "      <td>271.329045</td>\n",
              "      <td>265.984757</td>\n",
              "      <td>0.108268</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>0.040072</td>\n",
              "      <td>0.811351</td>\n",
              "      <td>3.463773</td>\n",
              "      <td>3.610555</td>\n",
              "      <td>0.911111</td>\n",
              "      <td>0.826173</td>\n",
              "      <td>0.949721</td>\n",
              "      <td>0.878083</td>\n",
              "      <td>0.034894</td>\n",
              "      <td>0.144681</td>\n",
              "      <td>246.377305</td>\n",
              "      <td>261.055509</td>\n",
              "      <td>0.141947</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>0.050146</td>\n",
              "      <td>0.786985</td>\n",
              "      <td>3.125844</td>\n",
              "      <td>3.513180</td>\n",
              "      <td>0.822222</td>\n",
              "      <td>0.799521</td>\n",
              "      <td>0.924107</td>\n",
              "      <td>0.862301</td>\n",
              "      <td>0.031489</td>\n",
              "      <td>0.176170</td>\n",
              "      <td>212.584397</td>\n",
              "      <td>251.318009</td>\n",
              "      <td>0.171006</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>6</td>\n",
              "      <td>0.100067</td>\n",
              "      <td>0.684817</td>\n",
              "      <td>3.017495</td>\n",
              "      <td>3.265892</td>\n",
              "      <td>0.793722</td>\n",
              "      <td>0.737285</td>\n",
              "      <td>0.859060</td>\n",
              "      <td>0.799933</td>\n",
              "      <td>0.150638</td>\n",
              "      <td>0.326809</td>\n",
              "      <td>201.749451</td>\n",
              "      <td>226.589176</td>\n",
              "      <td>0.307671</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>7</td>\n",
              "      <td>0.149989</td>\n",
              "      <td>0.592277</td>\n",
              "      <td>2.642439</td>\n",
              "      <td>3.058384</td>\n",
              "      <td>0.695067</td>\n",
              "      <td>0.635696</td>\n",
              "      <td>0.804478</td>\n",
              "      <td>0.745269</td>\n",
              "      <td>0.131915</td>\n",
              "      <td>0.458723</td>\n",
              "      <td>164.243870</td>\n",
              "      <td>205.838425</td>\n",
              "      <td>0.418930</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>8</td>\n",
              "      <td>0.200358</td>\n",
              "      <td>0.517372</td>\n",
              "      <td>2.399296</td>\n",
              "      <td>2.892692</td>\n",
              "      <td>0.631111</td>\n",
              "      <td>0.552632</td>\n",
              "      <td>0.760894</td>\n",
              "      <td>0.696841</td>\n",
              "      <td>0.120851</td>\n",
              "      <td>0.579574</td>\n",
              "      <td>139.929645</td>\n",
              "      <td>189.269179</td>\n",
              "      <td>0.514568</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>9</td>\n",
              "      <td>0.299978</td>\n",
              "      <td>0.385510</td>\n",
              "      <td>1.631742</td>\n",
              "      <td>2.473943</td>\n",
              "      <td>0.429213</td>\n",
              "      <td>0.448758</td>\n",
              "      <td>0.650746</td>\n",
              "      <td>0.614455</td>\n",
              "      <td>0.162553</td>\n",
              "      <td>0.742128</td>\n",
              "      <td>63.174181</td>\n",
              "      <td>147.394347</td>\n",
              "      <td>0.599965</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>10</td>\n",
              "      <td>0.400045</td>\n",
              "      <td>0.270227</td>\n",
              "      <td>1.165175</td>\n",
              "      <td>2.146568</td>\n",
              "      <td>0.306488</td>\n",
              "      <td>0.326354</td>\n",
              "      <td>0.564633</td>\n",
              "      <td>0.542389</td>\n",
              "      <td>0.116596</td>\n",
              "      <td>0.858723</td>\n",
              "      <td>16.517493</td>\n",
              "      <td>114.656824</td>\n",
              "      <td>0.622393</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>11</td>\n",
              "      <td>0.500112</td>\n",
              "      <td>0.169738</td>\n",
              "      <td>0.833483</td>\n",
              "      <td>1.883834</td>\n",
              "      <td>0.219239</td>\n",
              "      <td>0.219148</td>\n",
              "      <td>0.495524</td>\n",
              "      <td>0.477712</td>\n",
              "      <td>0.083404</td>\n",
              "      <td>0.942128</td>\n",
              "      <td>-16.651721</td>\n",
              "      <td>88.383360</td>\n",
              "      <td>0.599783</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>12</td>\n",
              "      <td>0.600179</td>\n",
              "      <td>0.089698</td>\n",
              "      <td>0.365712</td>\n",
              "      <td>1.630719</td>\n",
              "      <td>0.096197</td>\n",
              "      <td>0.125703</td>\n",
              "      <td>0.428944</td>\n",
              "      <td>0.419022</td>\n",
              "      <td>0.036596</td>\n",
              "      <td>0.978723</td>\n",
              "      <td>-63.428816</td>\n",
              "      <td>63.071893</td>\n",
              "      <td>0.513657</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>13</td>\n",
              "      <td>0.700022</td>\n",
              "      <td>0.042011</td>\n",
              "      <td>0.127860</td>\n",
              "      <td>1.416368</td>\n",
              "      <td>0.033632</td>\n",
              "      <td>0.063288</td>\n",
              "      <td>0.372562</td>\n",
              "      <td>0.368284</td>\n",
              "      <td>0.012766</td>\n",
              "      <td>0.991489</td>\n",
              "      <td>-87.214006</td>\n",
              "      <td>41.636808</td>\n",
              "      <td>0.395499</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>14</td>\n",
              "      <td>0.800090</td>\n",
              "      <td>0.018240</td>\n",
              "      <td>0.068039</td>\n",
              "      <td>1.247733</td>\n",
              "      <td>0.017897</td>\n",
              "      <td>0.028007</td>\n",
              "      <td>0.328204</td>\n",
              "      <td>0.325726</td>\n",
              "      <td>0.006809</td>\n",
              "      <td>0.998298</td>\n",
              "      <td>-93.196059</td>\n",
              "      <td>24.773268</td>\n",
              "      <td>0.268954</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>15</td>\n",
              "      <td>0.900157</td>\n",
              "      <td>0.009152</td>\n",
              "      <td>0.017010</td>\n",
              "      <td>1.110918</td>\n",
              "      <td>0.004474</td>\n",
              "      <td>0.012766</td>\n",
              "      <td>0.292216</td>\n",
              "      <td>0.290935</td>\n",
              "      <td>0.001702</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>-98.299015</td>\n",
              "      <td>11.091768</td>\n",
              "      <td>0.135480</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>16</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.001484</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.005346</td>\n",
              "      <td>0.263040</td>\n",
              "      <td>0.262421</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>-100.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    group  cumulative_data_fraction  ...  cumulative_gain  kolmogorov_smirnov\n",
              "0       1                  0.010074  ...       271.721986            0.037143\n",
              "1       2                  0.020372  ...       263.459434            0.072827\n",
              "2       3                  0.029998  ...       265.984757            0.108268\n",
              "3       4                  0.040072  ...       261.055509            0.141947\n",
              "4       5                  0.050146  ...       251.318009            0.171006\n",
              "5       6                  0.100067  ...       226.589176            0.307671\n",
              "6       7                  0.149989  ...       205.838425            0.418930\n",
              "7       8                  0.200358  ...       189.269179            0.514568\n",
              "8       9                  0.299978  ...       147.394347            0.599965\n",
              "9      10                  0.400045  ...       114.656824            0.622393\n",
              "10     11                  0.500112  ...        88.383360            0.599783\n",
              "11     12                  0.600179  ...        63.071893            0.513657\n",
              "12     13                  0.700022  ...        41.636808            0.395499\n",
              "13     14                  0.800090  ...        24.773268            0.268954\n",
              "14     15                  0.900157  ...        11.091768            0.135480\n",
              "15     16                  1.000000  ...         0.000000            0.000000\n",
              "\n",
              "[16 rows x 14 columns]"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "\n",
            "ModelMetricsBinomial: xgboost\n",
            "** Reported on validation data. **\n",
            "\n",
            "MSE: 0.13968665068245656\n",
            "RMSE: 0.3737467734743091\n",
            "LogLoss: 0.43910824060153614\n",
            "Mean Per-Class Error: 0.22457912457912454\n",
            "AUC: 0.8369583278674187\n",
            "AUCPR: 0.6774149713007829\n",
            "Gini: 0.6739166557348375\n",
            "\n",
            "Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.3147416114807129: \n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th>No</th>\n",
              "      <th>Yes</th>\n",
              "      <th>Error</th>\n",
              "      <th>Rate</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>No</td>\n",
              "      <td>616.0</td>\n",
              "      <td>154.0</td>\n",
              "      <td>0.2</td>\n",
              "      <td>(154.0/770.0)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Yes</td>\n",
              "      <td>74.0</td>\n",
              "      <td>223.0</td>\n",
              "      <td>0.2492</td>\n",
              "      <td>(74.0/297.0)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Total</td>\n",
              "      <td>690.0</td>\n",
              "      <td>377.0</td>\n",
              "      <td>0.2137</td>\n",
              "      <td>(228.0/1067.0)</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "             No    Yes   Error             Rate\n",
              "0     No  616.0  154.0     0.2    (154.0/770.0)\n",
              "1    Yes   74.0  223.0  0.2492     (74.0/297.0)\n",
              "2  Total  690.0  377.0  0.2137   (228.0/1067.0)"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Maximum Metrics: Maximum metrics at their respective thresholds\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>metric</th>\n",
              "      <th>threshold</th>\n",
              "      <th>value</th>\n",
              "      <th>idx</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>max f1</td>\n",
              "      <td>0.314742</td>\n",
              "      <td>0.661721</td>\n",
              "      <td>206.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>max f2</td>\n",
              "      <td>0.055472</td>\n",
              "      <td>0.738518</td>\n",
              "      <td>348.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>max f0point5</td>\n",
              "      <td>0.562409</td>\n",
              "      <td>0.645463</td>\n",
              "      <td>110.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>max accuracy</td>\n",
              "      <td>0.429109</td>\n",
              "      <td>0.802249</td>\n",
              "      <td>159.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>max precision</td>\n",
              "      <td>0.961857</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>max recall</td>\n",
              "      <td>0.005568</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>394.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>max specificity</td>\n",
              "      <td>0.961857</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>max absolute_mcc</td>\n",
              "      <td>0.351242</td>\n",
              "      <td>0.518205</td>\n",
              "      <td>189.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>max min_per_class_accuracy</td>\n",
              "      <td>0.287675</td>\n",
              "      <td>0.768831</td>\n",
              "      <td>219.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>max mean_per_class_accuracy</td>\n",
              "      <td>0.314742</td>\n",
              "      <td>0.775421</td>\n",
              "      <td>206.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>max tns</td>\n",
              "      <td>0.961857</td>\n",
              "      <td>770.000000</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>max fns</td>\n",
              "      <td>0.961857</td>\n",
              "      <td>296.000000</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>max fps</td>\n",
              "      <td>0.002040</td>\n",
              "      <td>770.000000</td>\n",
              "      <td>399.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>max tps</td>\n",
              "      <td>0.005568</td>\n",
              "      <td>297.000000</td>\n",
              "      <td>394.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>max tnr</td>\n",
              "      <td>0.961857</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>max fnr</td>\n",
              "      <td>0.961857</td>\n",
              "      <td>0.996633</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>max fpr</td>\n",
              "      <td>0.002040</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>399.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>max tpr</td>\n",
              "      <td>0.005568</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>394.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                         metric  threshold       value    idx\n",
              "0                        max f1   0.314742    0.661721  206.0\n",
              "1                        max f2   0.055472    0.738518  348.0\n",
              "2                  max f0point5   0.562409    0.645463  110.0\n",
              "3                  max accuracy   0.429109    0.802249  159.0\n",
              "4                 max precision   0.961857    1.000000    0.0\n",
              "5                    max recall   0.005568    1.000000  394.0\n",
              "6               max specificity   0.961857    1.000000    0.0\n",
              "7              max absolute_mcc   0.351242    0.518205  189.0\n",
              "8    max min_per_class_accuracy   0.287675    0.768831  219.0\n",
              "9   max mean_per_class_accuracy   0.314742    0.775421  206.0\n",
              "10                      max tns   0.961857  770.000000    0.0\n",
              "11                      max fns   0.961857  296.000000    0.0\n",
              "12                      max fps   0.002040  770.000000  399.0\n",
              "13                      max tps   0.005568  297.000000  394.0\n",
              "14                      max tnr   0.961857    1.000000    0.0\n",
              "15                      max fnr   0.961857    0.996633    0.0\n",
              "16                      max fpr   0.002040    1.000000  399.0\n",
              "17                      max tpr   0.005568    1.000000  394.0"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Gains/Lift Table: Avg response rate: 27.84 %, avg score: 26.17 %\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>group</th>\n",
              "      <th>cumulative_data_fraction</th>\n",
              "      <th>lower_threshold</th>\n",
              "      <th>lift</th>\n",
              "      <th>cumulative_lift</th>\n",
              "      <th>response_rate</th>\n",
              "      <th>score</th>\n",
              "      <th>cumulative_response_rate</th>\n",
              "      <th>cumulative_score</th>\n",
              "      <th>capture_rate</th>\n",
              "      <th>cumulative_capture_rate</th>\n",
              "      <th>gain</th>\n",
              "      <th>cumulative_gain</th>\n",
              "      <th>kolmogorov_smirnov</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>0.010309</td>\n",
              "      <td>0.887744</td>\n",
              "      <td>3.592593</td>\n",
              "      <td>3.592593</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.917639</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.917639</td>\n",
              "      <td>0.037037</td>\n",
              "      <td>0.037037</td>\n",
              "      <td>259.259259</td>\n",
              "      <td>259.259259</td>\n",
              "      <td>0.037037</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>0.020619</td>\n",
              "      <td>0.857889</td>\n",
              "      <td>3.265993</td>\n",
              "      <td>3.429293</td>\n",
              "      <td>0.909091</td>\n",
              "      <td>0.873083</td>\n",
              "      <td>0.954545</td>\n",
              "      <td>0.895361</td>\n",
              "      <td>0.033670</td>\n",
              "      <td>0.070707</td>\n",
              "      <td>226.599327</td>\n",
              "      <td>242.929293</td>\n",
              "      <td>0.069408</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>0.029991</td>\n",
              "      <td>0.821435</td>\n",
              "      <td>2.514815</td>\n",
              "      <td>3.143519</td>\n",
              "      <td>0.700000</td>\n",
              "      <td>0.839621</td>\n",
              "      <td>0.875000</td>\n",
              "      <td>0.877942</td>\n",
              "      <td>0.023569</td>\n",
              "      <td>0.094276</td>\n",
              "      <td>151.481481</td>\n",
              "      <td>214.351852</td>\n",
              "      <td>0.089081</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>0.040300</td>\n",
              "      <td>0.792897</td>\n",
              "      <td>2.612795</td>\n",
              "      <td>3.007752</td>\n",
              "      <td>0.727273</td>\n",
              "      <td>0.811202</td>\n",
              "      <td>0.837209</td>\n",
              "      <td>0.860869</td>\n",
              "      <td>0.026936</td>\n",
              "      <td>0.121212</td>\n",
              "      <td>161.279461</td>\n",
              "      <td>200.775194</td>\n",
              "      <td>0.112121</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>0.050609</td>\n",
              "      <td>0.774703</td>\n",
              "      <td>3.265993</td>\n",
              "      <td>3.060357</td>\n",
              "      <td>0.909091</td>\n",
              "      <td>0.783467</td>\n",
              "      <td>0.851852</td>\n",
              "      <td>0.845102</td>\n",
              "      <td>0.033670</td>\n",
              "      <td>0.154882</td>\n",
              "      <td>226.599327</td>\n",
              "      <td>206.035665</td>\n",
              "      <td>0.144493</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>6</td>\n",
              "      <td>0.100281</td>\n",
              "      <td>0.678362</td>\n",
              "      <td>2.508036</td>\n",
              "      <td>2.786777</td>\n",
              "      <td>0.698113</td>\n",
              "      <td>0.723228</td>\n",
              "      <td>0.775701</td>\n",
              "      <td>0.784735</td>\n",
              "      <td>0.124579</td>\n",
              "      <td>0.279461</td>\n",
              "      <td>150.803634</td>\n",
              "      <td>178.677743</td>\n",
              "      <td>0.248292</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>7</td>\n",
              "      <td>0.149953</td>\n",
              "      <td>0.591876</td>\n",
              "      <td>2.236897</td>\n",
              "      <td>2.604630</td>\n",
              "      <td>0.622642</td>\n",
              "      <td>0.632889</td>\n",
              "      <td>0.725000</td>\n",
              "      <td>0.734436</td>\n",
              "      <td>0.111111</td>\n",
              "      <td>0.390572</td>\n",
              "      <td>123.689727</td>\n",
              "      <td>160.462963</td>\n",
              "      <td>0.333430</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>8</td>\n",
              "      <td>0.200562</td>\n",
              "      <td>0.532071</td>\n",
              "      <td>1.862826</td>\n",
              "      <td>2.417445</td>\n",
              "      <td>0.518519</td>\n",
              "      <td>0.562693</td>\n",
              "      <td>0.672897</td>\n",
              "      <td>0.691099</td>\n",
              "      <td>0.094276</td>\n",
              "      <td>0.484848</td>\n",
              "      <td>86.282579</td>\n",
              "      <td>141.744548</td>\n",
              "      <td>0.393939</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>9</td>\n",
              "      <td>0.299906</td>\n",
              "      <td>0.374171</td>\n",
              "      <td>1.897973</td>\n",
              "      <td>2.245370</td>\n",
              "      <td>0.528302</td>\n",
              "      <td>0.455597</td>\n",
              "      <td>0.625000</td>\n",
              "      <td>0.613089</td>\n",
              "      <td>0.188552</td>\n",
              "      <td>0.673401</td>\n",
              "      <td>89.797345</td>\n",
              "      <td>124.537037</td>\n",
              "      <td>0.517557</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>10</td>\n",
              "      <td>0.400187</td>\n",
              "      <td>0.264581</td>\n",
              "      <td>1.107996</td>\n",
              "      <td>1.960361</td>\n",
              "      <td>0.308411</td>\n",
              "      <td>0.316410</td>\n",
              "      <td>0.545667</td>\n",
              "      <td>0.538745</td>\n",
              "      <td>0.111111</td>\n",
              "      <td>0.784512</td>\n",
              "      <td>10.799585</td>\n",
              "      <td>96.036083</td>\n",
              "      <td>0.532564</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>11</td>\n",
              "      <td>0.500469</td>\n",
              "      <td>0.163544</td>\n",
              "      <td>0.537210</td>\n",
              "      <td>1.675198</td>\n",
              "      <td>0.149533</td>\n",
              "      <td>0.216397</td>\n",
              "      <td>0.466292</td>\n",
              "      <td>0.474155</td>\n",
              "      <td>0.053872</td>\n",
              "      <td>0.838384</td>\n",
              "      <td>-46.278989</td>\n",
              "      <td>67.519767</td>\n",
              "      <td>0.468254</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>12</td>\n",
              "      <td>0.599813</td>\n",
              "      <td>0.093678</td>\n",
              "      <td>0.610063</td>\n",
              "      <td>1.498785</td>\n",
              "      <td>0.169811</td>\n",
              "      <td>0.128288</td>\n",
              "      <td>0.417187</td>\n",
              "      <td>0.416871</td>\n",
              "      <td>0.060606</td>\n",
              "      <td>0.898990</td>\n",
              "      <td>-38.993711</td>\n",
              "      <td>49.878472</td>\n",
              "      <td>0.414574</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>13</td>\n",
              "      <td>0.700094</td>\n",
              "      <td>0.043535</td>\n",
              "      <td>0.570786</td>\n",
              "      <td>1.365858</td>\n",
              "      <td>0.158879</td>\n",
              "      <td>0.067892</td>\n",
              "      <td>0.380187</td>\n",
              "      <td>0.366883</td>\n",
              "      <td>0.057239</td>\n",
              "      <td>0.956229</td>\n",
              "      <td>-42.921426</td>\n",
              "      <td>36.585850</td>\n",
              "      <td>0.354930</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>14</td>\n",
              "      <td>0.799438</td>\n",
              "      <td>0.019445</td>\n",
              "      <td>0.271139</td>\n",
              "      <td>1.229821</td>\n",
              "      <td>0.075472</td>\n",
              "      <td>0.029103</td>\n",
              "      <td>0.342321</td>\n",
              "      <td>0.324908</td>\n",
              "      <td>0.026936</td>\n",
              "      <td>0.983165</td>\n",
              "      <td>-72.886094</td>\n",
              "      <td>22.982068</td>\n",
              "      <td>0.254594</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>15</td>\n",
              "      <td>0.900656</td>\n",
              "      <td>0.009703</td>\n",
              "      <td>0.066529</td>\n",
              "      <td>1.099087</td>\n",
              "      <td>0.018519</td>\n",
              "      <td>0.013619</td>\n",
              "      <td>0.305931</td>\n",
              "      <td>0.289925</td>\n",
              "      <td>0.006734</td>\n",
              "      <td>0.989899</td>\n",
              "      <td>-93.347051</td>\n",
              "      <td>9.908660</td>\n",
              "      <td>0.123665</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>16</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.001368</td>\n",
              "      <td>0.101677</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.028302</td>\n",
              "      <td>0.005898</td>\n",
              "      <td>0.278351</td>\n",
              "      <td>0.261708</td>\n",
              "      <td>0.010101</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>-89.832285</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    group  cumulative_data_fraction  ...  cumulative_gain  kolmogorov_smirnov\n",
              "0       1                  0.010309  ...       259.259259            0.037037\n",
              "1       2                  0.020619  ...       242.929293            0.069408\n",
              "2       3                  0.029991  ...       214.351852            0.089081\n",
              "3       4                  0.040300  ...       200.775194            0.112121\n",
              "4       5                  0.050609  ...       206.035665            0.144493\n",
              "5       6                  0.100281  ...       178.677743            0.248292\n",
              "6       7                  0.149953  ...       160.462963            0.333430\n",
              "7       8                  0.200562  ...       141.744548            0.393939\n",
              "8       9                  0.299906  ...       124.537037            0.517557\n",
              "9      10                  0.400187  ...        96.036083            0.532564\n",
              "10     11                  0.500469  ...        67.519767            0.468254\n",
              "11     12                  0.599813  ...        49.878472            0.414574\n",
              "12     13                  0.700094  ...        36.585850            0.354930\n",
              "13     14                  0.799438  ...        22.982068            0.254594\n",
              "14     15                  0.900656  ...         9.908660            0.123665\n",
              "15     16                  1.000000  ...         0.000000            0.000000\n",
              "\n",
              "[16 rows x 14 columns]"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "\n",
            "Scoring History: \n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th>timestamp</th>\n",
              "      <th>duration</th>\n",
              "      <th>number_of_trees</th>\n",
              "      <th>training_rmse</th>\n",
              "      <th>training_logloss</th>\n",
              "      <th>training_auc</th>\n",
              "      <th>training_pr_auc</th>\n",
              "      <th>training_lift</th>\n",
              "      <th>training_classification_error</th>\n",
              "      <th>validation_rmse</th>\n",
              "      <th>validation_logloss</th>\n",
              "      <th>validation_auc</th>\n",
              "      <th>validation_pr_auc</th>\n",
              "      <th>validation_lift</th>\n",
              "      <th>validation_classification_error</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td></td>\n",
              "      <td>2021-11-14 15:23:49</td>\n",
              "      <td>0.004 sec</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.500000</td>\n",
              "      <td>0.693147</td>\n",
              "      <td>0.500000</td>\n",
              "      <td>0.263040</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.736960</td>\n",
              "      <td>0.500000</td>\n",
              "      <td>0.693147</td>\n",
              "      <td>0.500000</td>\n",
              "      <td>0.278351</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.721649</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td></td>\n",
              "      <td>2021-11-14 15:23:49</td>\n",
              "      <td>0.099 sec</td>\n",
              "      <td>5.0</td>\n",
              "      <td>0.365118</td>\n",
              "      <td>0.424825</td>\n",
              "      <td>0.865244</td>\n",
              "      <td>0.691459</td>\n",
              "      <td>3.386971</td>\n",
              "      <td>0.213342</td>\n",
              "      <td>0.378944</td>\n",
              "      <td>0.451415</td>\n",
              "      <td>0.834617</td>\n",
              "      <td>0.672088</td>\n",
              "      <td>3.113580</td>\n",
              "      <td>0.215558</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td></td>\n",
              "      <td>2021-11-14 15:23:49</td>\n",
              "      <td>0.232 sec</td>\n",
              "      <td>10.0</td>\n",
              "      <td>0.352809</td>\n",
              "      <td>0.386994</td>\n",
              "      <td>0.874015</td>\n",
              "      <td>0.711082</td>\n",
              "      <td>3.553765</td>\n",
              "      <td>0.208865</td>\n",
              "      <td>0.373768</td>\n",
              "      <td>0.434617</td>\n",
              "      <td>0.836993</td>\n",
              "      <td>0.667551</td>\n",
              "      <td>3.265993</td>\n",
              "      <td>0.212746</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td></td>\n",
              "      <td>2021-11-14 15:23:49</td>\n",
              "      <td>0.348 sec</td>\n",
              "      <td>15.0</td>\n",
              "      <td>0.347268</td>\n",
              "      <td>0.372678</td>\n",
              "      <td>0.881253</td>\n",
              "      <td>0.725417</td>\n",
              "      <td>3.553765</td>\n",
              "      <td>0.203492</td>\n",
              "      <td>0.371998</td>\n",
              "      <td>0.431782</td>\n",
              "      <td>0.839473</td>\n",
              "      <td>0.675680</td>\n",
              "      <td>3.265993</td>\n",
              "      <td>0.193065</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td></td>\n",
              "      <td>2021-11-14 15:23:50</td>\n",
              "      <td>0.480 sec</td>\n",
              "      <td>20.0</td>\n",
              "      <td>0.340930</td>\n",
              "      <td>0.360452</td>\n",
              "      <td>0.890702</td>\n",
              "      <td>0.743700</td>\n",
              "      <td>3.801702</td>\n",
              "      <td>0.174614</td>\n",
              "      <td>0.371194</td>\n",
              "      <td>0.433142</td>\n",
              "      <td>0.839617</td>\n",
              "      <td>0.679110</td>\n",
              "      <td>3.265993</td>\n",
              "      <td>0.189316</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td></td>\n",
              "      <td>2021-11-14 15:23:50</td>\n",
              "      <td>0.585 sec</td>\n",
              "      <td>25.0</td>\n",
              "      <td>0.336805</td>\n",
              "      <td>0.352489</td>\n",
              "      <td>0.896043</td>\n",
              "      <td>0.756353</td>\n",
              "      <td>3.717220</td>\n",
              "      <td>0.177748</td>\n",
              "      <td>0.373015</td>\n",
              "      <td>0.436319</td>\n",
              "      <td>0.837813</td>\n",
              "      <td>0.678109</td>\n",
              "      <td>3.265993</td>\n",
              "      <td>0.197751</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td></td>\n",
              "      <td>2021-11-14 15:23:50</td>\n",
              "      <td>0.761 sec</td>\n",
              "      <td>30.0</td>\n",
              "      <td>0.333664</td>\n",
              "      <td>0.346254</td>\n",
              "      <td>0.899490</td>\n",
              "      <td>0.764536</td>\n",
              "      <td>3.717220</td>\n",
              "      <td>0.173271</td>\n",
              "      <td>0.373747</td>\n",
              "      <td>0.439108</td>\n",
              "      <td>0.836958</td>\n",
              "      <td>0.677415</td>\n",
              "      <td>3.592593</td>\n",
              "      <td>0.213683</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "               timestamp  ... validation_lift  validation_classification_error\n",
              "0    2021-11-14 15:23:49  ...        1.000000                         0.721649\n",
              "1    2021-11-14 15:23:49  ...        3.113580                         0.215558\n",
              "2    2021-11-14 15:23:49  ...        3.265993                         0.212746\n",
              "3    2021-11-14 15:23:49  ...        3.265993                         0.193065\n",
              "4    2021-11-14 15:23:50  ...        3.265993                         0.189316\n",
              "5    2021-11-14 15:23:50  ...        3.265993                         0.197751\n",
              "6    2021-11-14 15:23:50  ...        3.592593                         0.213683\n",
              "\n",
              "[7 rows x 16 columns]"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Variable Importances: \n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>variable</th>\n",
              "      <th>relative_importance</th>\n",
              "      <th>scaled_importance</th>\n",
              "      <th>percentage</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>MonthlyCharges</td>\n",
              "      <td>843.539856</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.246661</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>tenure</td>\n",
              "      <td>745.628540</td>\n",
              "      <td>0.883928</td>\n",
              "      <td>0.218031</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Contract.Month-to-month</td>\n",
              "      <td>567.875000</td>\n",
              "      <td>0.673205</td>\n",
              "      <td>0.166053</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>TotalCharges</td>\n",
              "      <td>492.928711</td>\n",
              "      <td>0.584357</td>\n",
              "      <td>0.144138</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Contract.Two year</td>\n",
              "      <td>252.040436</td>\n",
              "      <td>0.298789</td>\n",
              "      <td>0.073700</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>ServiceCount</td>\n",
              "      <td>218.434998</td>\n",
              "      <td>0.258950</td>\n",
              "      <td>0.063873</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>Contract.One year</td>\n",
              "      <td>111.063660</td>\n",
              "      <td>0.131664</td>\n",
              "      <td>0.032476</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>PaperlessBilling.No</td>\n",
              "      <td>87.808418</td>\n",
              "      <td>0.104095</td>\n",
              "      <td>0.025676</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>SeniorCitizen</td>\n",
              "      <td>49.417854</td>\n",
              "      <td>0.058584</td>\n",
              "      <td>0.014450</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>gender.Female</td>\n",
              "      <td>42.659378</td>\n",
              "      <td>0.050572</td>\n",
              "      <td>0.012474</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>PaperlessBilling.Yes</td>\n",
              "      <td>5.267341</td>\n",
              "      <td>0.006244</td>\n",
              "      <td>0.001540</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>gender.Male</td>\n",
              "      <td>3.169271</td>\n",
              "      <td>0.003757</td>\n",
              "      <td>0.000927</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                   variable  relative_importance  scaled_importance  percentage\n",
              "0            MonthlyCharges           843.539856           1.000000    0.246661\n",
              "1                    tenure           745.628540           0.883928    0.218031\n",
              "2   Contract.Month-to-month           567.875000           0.673205    0.166053\n",
              "3              TotalCharges           492.928711           0.584357    0.144138\n",
              "4         Contract.Two year           252.040436           0.298789    0.073700\n",
              "5              ServiceCount           218.434998           0.258950    0.063873\n",
              "6         Contract.One year           111.063660           0.131664    0.032476\n",
              "7       PaperlessBilling.No            87.808418           0.104095    0.025676\n",
              "8             SeniorCitizen            49.417854           0.058584    0.014450\n",
              "9             gender.Female            42.659378           0.050572    0.012474\n",
              "10     PaperlessBilling.Yes             5.267341           0.006244    0.001540\n",
              "11              gender.Male             3.169271           0.003757    0.000927"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              ""
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G0NX3_vvdErW"
      },
      "source": [
        "### **8. Show H2O Leaderboard**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 243
        },
        "id": "eqRwRtJeNVIN",
        "outputId": "c08e9def-a356-4f16-c7ff-7dc16c72d8b8"
      },
      "source": [
        "lb = h2o_model.leaderboard\n",
        "lb.head()"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table>\n",
              "<thead>\n",
              "<tr><th>model_id                          </th><th style=\"text-align: right;\">    rmse</th><th style=\"text-align: right;\">     auc</th><th style=\"text-align: right;\">  logloss</th><th style=\"text-align: right;\">   aucpr</th><th style=\"text-align: right;\">  mean_per_class_error</th><th style=\"text-align: right;\">     mse</th></tr>\n",
              "</thead>\n",
              "<tbody>\n",
              "<tr><td>XGBoost_3_AutoML_1_20211114_152326</td><td style=\"text-align: right;\">0.391564</td><td style=\"text-align: right;\">0.812519</td><td style=\"text-align: right;\"> 0.469925</td><td style=\"text-align: right;\">0.62498 </td><td style=\"text-align: right;\">              0.256432</td><td style=\"text-align: right;\">0.153322</td></tr>\n",
              "<tr><td>GBM_1_AutoML_1_20211114_152326    </td><td style=\"text-align: right;\">0.392425</td><td style=\"text-align: right;\">0.812074</td><td style=\"text-align: right;\"> 0.463099</td><td style=\"text-align: right;\">0.617066</td><td style=\"text-align: right;\">              0.25439 </td><td style=\"text-align: right;\">0.153997</td></tr>\n",
              "<tr><td>GBM_2_AutoML_1_20211114_152326    </td><td style=\"text-align: right;\">0.393977</td><td style=\"text-align: right;\">0.812552</td><td style=\"text-align: right;\"> 0.472863</td><td style=\"text-align: right;\">0.610996</td><td style=\"text-align: right;\">              0.246871</td><td style=\"text-align: right;\">0.155218</td></tr>\n",
              "<tr><td>XRT_1_AutoML_1_20211114_152326    </td><td style=\"text-align: right;\">0.395152</td><td style=\"text-align: right;\">0.807847</td><td style=\"text-align: right;\"> 0.469581</td><td style=\"text-align: right;\">0.604919</td><td style=\"text-align: right;\">              0.272057</td><td style=\"text-align: right;\">0.156145</td></tr>\n",
              "<tr><td>GLM_1_AutoML_1_20211114_152326    </td><td style=\"text-align: right;\">0.395289</td><td style=\"text-align: right;\">0.805174</td><td style=\"text-align: right;\"> 0.469605</td><td style=\"text-align: right;\">0.607282</td><td style=\"text-align: right;\">              0.261311</td><td style=\"text-align: right;\">0.156253</td></tr>\n",
              "<tr><td>GBM_3_AutoML_1_20211114_152326    </td><td style=\"text-align: right;\">0.397489</td><td style=\"text-align: right;\">0.806619</td><td style=\"text-align: right;\"> 0.480031</td><td style=\"text-align: right;\">0.614809</td><td style=\"text-align: right;\">              0.261343</td><td style=\"text-align: right;\">0.157998</td></tr>\n",
              "<tr><td>XGBoost_2_AutoML_1_20211114_152326</td><td style=\"text-align: right;\">0.398739</td><td style=\"text-align: right;\">0.811107</td><td style=\"text-align: right;\"> 0.479934</td><td style=\"text-align: right;\">0.60207 </td><td style=\"text-align: right;\">              0.254976</td><td style=\"text-align: right;\">0.158993</td></tr>\n",
              "<tr><td>XGBoost_1_AutoML_1_20211114_152326</td><td style=\"text-align: right;\">0.400434</td><td style=\"text-align: right;\">0.803468</td><td style=\"text-align: right;\"> 0.483027</td><td style=\"text-align: right;\">0.605782</td><td style=\"text-align: right;\">              0.258138</td><td style=\"text-align: right;\">0.160347</td></tr>\n",
              "<tr><td>GBM_4_AutoML_1_20211114_152326    </td><td style=\"text-align: right;\">0.401252</td><td style=\"text-align: right;\">0.803903</td><td style=\"text-align: right;\"> 0.495842</td><td style=\"text-align: right;\">0.615237</td><td style=\"text-align: right;\">              0.257258</td><td style=\"text-align: right;\">0.161003</td></tr>\n",
              "<tr><td>DRF_1_AutoML_1_20211114_152326    </td><td style=\"text-align: right;\">0.413038</td><td style=\"text-align: right;\">0.785312</td><td style=\"text-align: right;\"> 0.667094</td><td style=\"text-align: right;\">0.57144 </td><td style=\"text-align: right;\">              0.282466</td><td style=\"text-align: right;\">0.1706  </td></tr>\n",
              "</tbody>\n",
              "</table>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              ""
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q8BgD2B7elA5"
      },
      "source": [
        "### **9. Get Best Model**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SxiUNsqHandH"
      },
      "source": [
        "# get best model\n",
        "m = h2o_model.get_best_model()\n",
        "\n",
        "# Get the best model using a non-default metric\n",
        "m = h2o_model.get_best_model(criterion=\"logloss\")\n",
        "\n",
        "# Get the best XGBoost model using default sort metric\n",
        "xgb = h2o_model.get_best_model(algorithm=\"xgboost\")\n",
        "\n",
        "# Get the best XGBoost model, ranked by logloss\n",
        "xgb = h2o_model.get_best_model(algorithm=\"xgboost\", criterion=\"logloss\")"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5lhvhiS0da7q"
      },
      "source": [
        "### **9. Find Prediction**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XAh1JRRkNVAp",
        "outputId": "33aa4a89-8477-4ddb-80cb-c141e7d74def"
      },
      "source": [
        "pred = h2o_model.leader.predict(test)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "xgboost prediction progress: |███████████████████████████████████████████████████| (done) 100%\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 203
        },
        "id": "cN0_9pMyNU9w",
        "outputId": "4f64df62-aa03-4a49-afcb-7d5ec56f399a"
      },
      "source": [
        "# convert it to dataframe\n",
        "pred_df = pred.as_data_frame()\n",
        "pred_df.head()"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>predict</th>\n",
              "      <th>No</th>\n",
              "      <th>Yes</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Yes</td>\n",
              "      <td>0.344564</td>\n",
              "      <td>0.655436</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Yes</td>\n",
              "      <td>0.425146</td>\n",
              "      <td>0.574854</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Yes</td>\n",
              "      <td>0.595434</td>\n",
              "      <td>0.404566</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Yes</td>\n",
              "      <td>0.675803</td>\n",
              "      <td>0.324197</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>No</td>\n",
              "      <td>0.902871</td>\n",
              "      <td>0.097129</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  predict        No       Yes\n",
              "0     Yes  0.344564  0.655436\n",
              "1     Yes  0.425146  0.574854\n",
              "2     Yes  0.595434  0.404566\n",
              "3     Yes  0.675803  0.324197\n",
              "4      No  0.902871  0.097129"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "beY9Je4ddzpK"
      },
      "source": [
        "### **10. Show Model Performance**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PYEYrw6X6f8F",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "00dd28e9-82a8-4372-ac65-d3580abf2e35"
      },
      "source": [
        "h2o_model.leader.model_performance(test)"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "ModelMetricsBinomial: xgboost\n",
            "** Reported on test data. **\n",
            "\n",
            "MSE: 0.13877613619505524\n",
            "RMSE: 0.3725266919229483\n",
            "LogLoss: 0.431639855739024\n",
            "Mean Per-Class Error: 0.23922976501305482\n",
            "AUC: 0.8304454466334362\n",
            "AUCPR: 0.6248599460202824\n",
            "Gini: 0.6608908932668724\n",
            "\n",
            "Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.3088730722665787: \n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th>No</th>\n",
              "      <th>Yes</th>\n",
              "      <th>Error</th>\n",
              "      <th>Rate</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>No</td>\n",
              "      <td>591.0</td>\n",
              "      <td>175.0</td>\n",
              "      <td>0.2285</td>\n",
              "      <td>(175.0/766.0)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Yes</td>\n",
              "      <td>66.0</td>\n",
              "      <td>198.0</td>\n",
              "      <td>0.25</td>\n",
              "      <td>(66.0/264.0)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Total</td>\n",
              "      <td>657.0</td>\n",
              "      <td>373.0</td>\n",
              "      <td>0.234</td>\n",
              "      <td>(241.0/1030.0)</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "             No    Yes   Error             Rate\n",
              "0     No  591.0  175.0  0.2285    (175.0/766.0)\n",
              "1    Yes   66.0  198.0    0.25     (66.0/264.0)\n",
              "2  Total  657.0  373.0   0.234   (241.0/1030.0)"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Maximum Metrics: Maximum metrics at their respective thresholds\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>metric</th>\n",
              "      <th>threshold</th>\n",
              "      <th>value</th>\n",
              "      <th>idx</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>max f1</td>\n",
              "      <td>0.308873</td>\n",
              "      <td>0.621664</td>\n",
              "      <td>206.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>max f2</td>\n",
              "      <td>0.092757</td>\n",
              "      <td>0.727981</td>\n",
              "      <td>315.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>max f0point5</td>\n",
              "      <td>0.537425</td>\n",
              "      <td>0.605315</td>\n",
              "      <td>115.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>max accuracy</td>\n",
              "      <td>0.602899</td>\n",
              "      <td>0.800971</td>\n",
              "      <td>92.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>max precision</td>\n",
              "      <td>0.933198</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>max recall</td>\n",
              "      <td>0.004434</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>396.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>max specificity</td>\n",
              "      <td>0.933198</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>max absolute_mcc</td>\n",
              "      <td>0.383218</td>\n",
              "      <td>0.476269</td>\n",
              "      <td>175.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>max min_per_class_accuracy</td>\n",
              "      <td>0.297757</td>\n",
              "      <td>0.755875</td>\n",
              "      <td>212.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>max mean_per_class_accuracy</td>\n",
              "      <td>0.308873</td>\n",
              "      <td>0.760770</td>\n",
              "      <td>206.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>max tns</td>\n",
              "      <td>0.933198</td>\n",
              "      <td>766.000000</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>max fns</td>\n",
              "      <td>0.933198</td>\n",
              "      <td>263.000000</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>max fps</td>\n",
              "      <td>0.002010</td>\n",
              "      <td>766.000000</td>\n",
              "      <td>399.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>max tps</td>\n",
              "      <td>0.004434</td>\n",
              "      <td>264.000000</td>\n",
              "      <td>396.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>max tnr</td>\n",
              "      <td>0.933198</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>max fnr</td>\n",
              "      <td>0.933198</td>\n",
              "      <td>0.996212</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>max fpr</td>\n",
              "      <td>0.002010</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>399.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>max tpr</td>\n",
              "      <td>0.004434</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>396.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                         metric  threshold       value    idx\n",
              "0                        max f1   0.308873    0.621664  206.0\n",
              "1                        max f2   0.092757    0.727981  315.0\n",
              "2                  max f0point5   0.537425    0.605315  115.0\n",
              "3                  max accuracy   0.602899    0.800971   92.0\n",
              "4                 max precision   0.933198    1.000000    0.0\n",
              "5                    max recall   0.004434    1.000000  396.0\n",
              "6               max specificity   0.933198    1.000000    0.0\n",
              "7              max absolute_mcc   0.383218    0.476269  175.0\n",
              "8    max min_per_class_accuracy   0.297757    0.755875  212.0\n",
              "9   max mean_per_class_accuracy   0.308873    0.760770  206.0\n",
              "10                      max tns   0.933198  766.000000    0.0\n",
              "11                      max fns   0.933198  263.000000    0.0\n",
              "12                      max fps   0.002010  766.000000  399.0\n",
              "13                      max tps   0.004434  264.000000  396.0\n",
              "14                      max tnr   0.933198    1.000000    0.0\n",
              "15                      max fnr   0.933198    0.996212    0.0\n",
              "16                      max fpr   0.002010    1.000000  399.0\n",
              "17                      max tpr   0.004434    1.000000  396.0"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Gains/Lift Table: Avg response rate: 25.63 %, avg score: 25.80 %\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>group</th>\n",
              "      <th>cumulative_data_fraction</th>\n",
              "      <th>lower_threshold</th>\n",
              "      <th>lift</th>\n",
              "      <th>cumulative_lift</th>\n",
              "      <th>response_rate</th>\n",
              "      <th>score</th>\n",
              "      <th>cumulative_response_rate</th>\n",
              "      <th>cumulative_score</th>\n",
              "      <th>capture_rate</th>\n",
              "      <th>cumulative_capture_rate</th>\n",
              "      <th>gain</th>\n",
              "      <th>cumulative_gain</th>\n",
              "      <th>kolmogorov_smirnov</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>0.010680</td>\n",
              "      <td>0.899905</td>\n",
              "      <td>3.192149</td>\n",
              "      <td>3.192149</td>\n",
              "      <td>0.818182</td>\n",
              "      <td>0.912622</td>\n",
              "      <td>0.818182</td>\n",
              "      <td>0.912622</td>\n",
              "      <td>0.034091</td>\n",
              "      <td>0.034091</td>\n",
              "      <td>219.214876</td>\n",
              "      <td>219.214876</td>\n",
              "      <td>0.031480</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>0.020388</td>\n",
              "      <td>0.857270</td>\n",
              "      <td>3.121212</td>\n",
              "      <td>3.158369</td>\n",
              "      <td>0.800000</td>\n",
              "      <td>0.876297</td>\n",
              "      <td>0.809524</td>\n",
              "      <td>0.895324</td>\n",
              "      <td>0.030303</td>\n",
              "      <td>0.064394</td>\n",
              "      <td>212.121212</td>\n",
              "      <td>215.836941</td>\n",
              "      <td>0.059172</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>0.030097</td>\n",
              "      <td>0.819639</td>\n",
              "      <td>3.121212</td>\n",
              "      <td>3.146383</td>\n",
              "      <td>0.800000</td>\n",
              "      <td>0.834835</td>\n",
              "      <td>0.806452</td>\n",
              "      <td>0.875811</td>\n",
              "      <td>0.030303</td>\n",
              "      <td>0.094697</td>\n",
              "      <td>212.121212</td>\n",
              "      <td>214.638319</td>\n",
              "      <td>0.086864</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>0.041748</td>\n",
              "      <td>0.786931</td>\n",
              "      <td>3.251263</td>\n",
              "      <td>3.175652</td>\n",
              "      <td>0.833333</td>\n",
              "      <td>0.802767</td>\n",
              "      <td>0.813953</td>\n",
              "      <td>0.855427</td>\n",
              "      <td>0.037879</td>\n",
              "      <td>0.132576</td>\n",
              "      <td>225.126263</td>\n",
              "      <td>217.565187</td>\n",
              "      <td>0.122132</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>0.050485</td>\n",
              "      <td>0.763394</td>\n",
              "      <td>3.034512</td>\n",
              "      <td>3.151224</td>\n",
              "      <td>0.777778</td>\n",
              "      <td>0.776923</td>\n",
              "      <td>0.807692</td>\n",
              "      <td>0.841840</td>\n",
              "      <td>0.026515</td>\n",
              "      <td>0.159091</td>\n",
              "      <td>203.451178</td>\n",
              "      <td>215.122378</td>\n",
              "      <td>0.146036</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>6</td>\n",
              "      <td>0.100000</td>\n",
              "      <td>0.665808</td>\n",
              "      <td>2.448010</td>\n",
              "      <td>2.803030</td>\n",
              "      <td>0.627451</td>\n",
              "      <td>0.717655</td>\n",
              "      <td>0.718447</td>\n",
              "      <td>0.780350</td>\n",
              "      <td>0.121212</td>\n",
              "      <td>0.280303</td>\n",
              "      <td>144.800951</td>\n",
              "      <td>180.303030</td>\n",
              "      <td>0.242444</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>7</td>\n",
              "      <td>0.150485</td>\n",
              "      <td>0.585503</td>\n",
              "      <td>2.325903</td>\n",
              "      <td>2.642962</td>\n",
              "      <td>0.596154</td>\n",
              "      <td>0.621428</td>\n",
              "      <td>0.677419</td>\n",
              "      <td>0.727034</td>\n",
              "      <td>0.117424</td>\n",
              "      <td>0.397727</td>\n",
              "      <td>132.590326</td>\n",
              "      <td>164.296188</td>\n",
              "      <td>0.332453</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>8</td>\n",
              "      <td>0.200000</td>\n",
              "      <td>0.510170</td>\n",
              "      <td>1.912507</td>\n",
              "      <td>2.462121</td>\n",
              "      <td>0.490196</td>\n",
              "      <td>0.547033</td>\n",
              "      <td>0.631068</td>\n",
              "      <td>0.682471</td>\n",
              "      <td>0.094697</td>\n",
              "      <td>0.492424</td>\n",
              "      <td>91.250743</td>\n",
              "      <td>146.212121</td>\n",
              "      <td>0.393208</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>9</td>\n",
              "      <td>0.300000</td>\n",
              "      <td>0.377608</td>\n",
              "      <td>1.780303</td>\n",
              "      <td>2.234848</td>\n",
              "      <td>0.456311</td>\n",
              "      <td>0.443563</td>\n",
              "      <td>0.572816</td>\n",
              "      <td>0.602835</td>\n",
              "      <td>0.178030</td>\n",
              "      <td>0.670455</td>\n",
              "      <td>78.030303</td>\n",
              "      <td>123.484848</td>\n",
              "      <td>0.498131</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>10</td>\n",
              "      <td>0.400000</td>\n",
              "      <td>0.265131</td>\n",
              "      <td>0.984848</td>\n",
              "      <td>1.922348</td>\n",
              "      <td>0.252427</td>\n",
              "      <td>0.322940</td>\n",
              "      <td>0.492718</td>\n",
              "      <td>0.532861</td>\n",
              "      <td>0.098485</td>\n",
              "      <td>0.768939</td>\n",
              "      <td>-1.515152</td>\n",
              "      <td>92.234848</td>\n",
              "      <td>0.496093</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>11</td>\n",
              "      <td>0.500000</td>\n",
              "      <td>0.165903</td>\n",
              "      <td>0.909091</td>\n",
              "      <td>1.719697</td>\n",
              "      <td>0.233010</td>\n",
              "      <td>0.216937</td>\n",
              "      <td>0.440777</td>\n",
              "      <td>0.469676</td>\n",
              "      <td>0.090909</td>\n",
              "      <td>0.859848</td>\n",
              "      <td>-9.090909</td>\n",
              "      <td>71.969697</td>\n",
              "      <td>0.483869</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>12</td>\n",
              "      <td>0.600000</td>\n",
              "      <td>0.088817</td>\n",
              "      <td>0.606061</td>\n",
              "      <td>1.534091</td>\n",
              "      <td>0.155340</td>\n",
              "      <td>0.123476</td>\n",
              "      <td>0.393204</td>\n",
              "      <td>0.411976</td>\n",
              "      <td>0.060606</td>\n",
              "      <td>0.920455</td>\n",
              "      <td>-39.393939</td>\n",
              "      <td>53.409091</td>\n",
              "      <td>0.430898</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>13</td>\n",
              "      <td>0.700000</td>\n",
              "      <td>0.041408</td>\n",
              "      <td>0.416667</td>\n",
              "      <td>1.374459</td>\n",
              "      <td>0.106796</td>\n",
              "      <td>0.061084</td>\n",
              "      <td>0.352288</td>\n",
              "      <td>0.361849</td>\n",
              "      <td>0.041667</td>\n",
              "      <td>0.962121</td>\n",
              "      <td>-58.333333</td>\n",
              "      <td>37.445887</td>\n",
              "      <td>0.352461</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>14</td>\n",
              "      <td>0.800000</td>\n",
              "      <td>0.017759</td>\n",
              "      <td>0.189394</td>\n",
              "      <td>1.226326</td>\n",
              "      <td>0.048544</td>\n",
              "      <td>0.027491</td>\n",
              "      <td>0.314320</td>\n",
              "      <td>0.320054</td>\n",
              "      <td>0.018939</td>\n",
              "      <td>0.981061</td>\n",
              "      <td>-81.060606</td>\n",
              "      <td>22.632576</td>\n",
              "      <td>0.243463</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>15</td>\n",
              "      <td>0.900000</td>\n",
              "      <td>0.009613</td>\n",
              "      <td>0.113636</td>\n",
              "      <td>1.102694</td>\n",
              "      <td>0.029126</td>\n",
              "      <td>0.013536</td>\n",
              "      <td>0.282632</td>\n",
              "      <td>0.285996</td>\n",
              "      <td>0.011364</td>\n",
              "      <td>0.992424</td>\n",
              "      <td>-88.636364</td>\n",
              "      <td>10.269360</td>\n",
              "      <td>0.124278</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>16</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.001958</td>\n",
              "      <td>0.075758</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.019417</td>\n",
              "      <td>0.006114</td>\n",
              "      <td>0.256311</td>\n",
              "      <td>0.258008</td>\n",
              "      <td>0.007576</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>-92.424242</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    group  cumulative_data_fraction  ...  cumulative_gain  kolmogorov_smirnov\n",
              "0       1                  0.010680  ...       219.214876            0.031480\n",
              "1       2                  0.020388  ...       215.836941            0.059172\n",
              "2       3                  0.030097  ...       214.638319            0.086864\n",
              "3       4                  0.041748  ...       217.565187            0.122132\n",
              "4       5                  0.050485  ...       215.122378            0.146036\n",
              "5       6                  0.100000  ...       180.303030            0.242444\n",
              "6       7                  0.150485  ...       164.296188            0.332453\n",
              "7       8                  0.200000  ...       146.212121            0.393208\n",
              "8       9                  0.300000  ...       123.484848            0.498131\n",
              "9      10                  0.400000  ...        92.234848            0.496093\n",
              "10     11                  0.500000  ...        71.969697            0.483869\n",
              "11     12                  0.600000  ...        53.409091            0.430898\n",
              "12     13                  0.700000  ...        37.445887            0.352461\n",
              "13     14                  0.800000  ...        22.632576            0.243463\n",
              "14     15                  0.900000  ...        10.269360            0.124278\n",
              "15     16                  1.000000  ...         0.000000            0.000000\n",
              "\n",
              "[16 rows x 14 columns]"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              ""
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EXj5DU-UfPkJ"
      },
      "source": [
        "### **12. Show All Trained Model**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Jv3W5F7LcHT6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "23459954-c809-4844-91e6-2edd0fe1defc"
      },
      "source": [
        "model_ids = list(h2o_model.leaderboard['model_id'].as_data_frame().iloc[:,0])\n",
        "model_ids"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['XGBoost_3_AutoML_1_20211114_152326',\n",
              " 'GBM_1_AutoML_1_20211114_152326',\n",
              " 'GBM_2_AutoML_1_20211114_152326',\n",
              " 'XRT_1_AutoML_1_20211114_152326',\n",
              " 'GLM_1_AutoML_1_20211114_152326',\n",
              " 'GBM_3_AutoML_1_20211114_152326',\n",
              " 'XGBoost_2_AutoML_1_20211114_152326',\n",
              " 'XGBoost_1_AutoML_1_20211114_152326',\n",
              " 'GBM_4_AutoML_1_20211114_152326',\n",
              " 'DRF_1_AutoML_1_20211114_152326']"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EFbJOTLN-phE"
      },
      "source": [
        "### **13. Train Model Using a Specific Algo**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "7D9WB9VHX-Nh",
        "outputId": "3bb73135-a89f-41f8-e804-edd27ddeba6f"
      },
      "source": [
        "xgb_model = h2o.get_model('XGBoost_3_AutoML_1_20211114_152326')\n",
        "xgb_model"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model Details\n",
            "=============\n",
            "H2OXGBoostEstimator :  XGBoost\n",
            "Model Key:  XGBoost_3_AutoML_1_20211114_152326\n",
            "\n",
            "\n",
            "Model Summary: \n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th>number_of_trees</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td></td>\n",
              "      <td>30.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "     number_of_trees\n",
              "0               30.0"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "\n",
            "ModelMetricsBinomial: xgboost\n",
            "** Reported on train data. **\n",
            "\n",
            "MSE: 0.11133195820860814\n",
            "RMSE: 0.3336644395326061\n",
            "LogLoss: 0.34625350893801693\n",
            "Mean Per-Class Error: 0.18452457278767354\n",
            "AUC: 0.8994898011943848\n",
            "AUCPR: 0.7645364251698769\n",
            "Gini: 0.7989796023887696\n",
            "\n",
            "Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.37355014085769656: \n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th>No</th>\n",
              "      <th>Yes</th>\n",
              "      <th>Error</th>\n",
              "      <th>Rate</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>No</td>\n",
              "      <td>2803.0</td>\n",
              "      <td>489.0</td>\n",
              "      <td>0.1485</td>\n",
              "      <td>(489.0/3292.0)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Yes</td>\n",
              "      <td>285.0</td>\n",
              "      <td>890.0</td>\n",
              "      <td>0.2426</td>\n",
              "      <td>(285.0/1175.0)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Total</td>\n",
              "      <td>3088.0</td>\n",
              "      <td>1379.0</td>\n",
              "      <td>0.1733</td>\n",
              "      <td>(774.0/4467.0)</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "              No     Yes   Error             Rate\n",
              "0     No  2803.0   489.0  0.1485   (489.0/3292.0)\n",
              "1    Yes   285.0   890.0  0.2426   (285.0/1175.0)\n",
              "2  Total  3088.0  1379.0  0.1733   (774.0/4467.0)"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Maximum Metrics: Maximum metrics at their respective thresholds\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>metric</th>\n",
              "      <th>threshold</th>\n",
              "      <th>value</th>\n",
              "      <th>idx</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>max f1</td>\n",
              "      <td>0.373550</td>\n",
              "      <td>0.696946</td>\n",
              "      <td>202.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>max f2</td>\n",
              "      <td>0.169555</td>\n",
              "      <td>0.798156</td>\n",
              "      <td>291.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>max f0point5</td>\n",
              "      <td>0.496359</td>\n",
              "      <td>0.718670</td>\n",
              "      <td>152.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>max accuracy</td>\n",
              "      <td>0.496359</td>\n",
              "      <td>0.845086</td>\n",
              "      <td>152.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>max precision</td>\n",
              "      <td>0.961415</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>max recall</td>\n",
              "      <td>0.010989</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>387.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>max specificity</td>\n",
              "      <td>0.961415</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>max absolute_mcc</td>\n",
              "      <td>0.463329</td>\n",
              "      <td>0.585210</td>\n",
              "      <td>166.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>max min_per_class_accuracy</td>\n",
              "      <td>0.321204</td>\n",
              "      <td>0.808323</td>\n",
              "      <td>224.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>max mean_per_class_accuracy</td>\n",
              "      <td>0.294099</td>\n",
              "      <td>0.815475</td>\n",
              "      <td>235.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>max tns</td>\n",
              "      <td>0.961415</td>\n",
              "      <td>3292.000000</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>max fns</td>\n",
              "      <td>0.961415</td>\n",
              "      <td>1171.000000</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>max fps</td>\n",
              "      <td>0.001868</td>\n",
              "      <td>3292.000000</td>\n",
              "      <td>399.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>max tps</td>\n",
              "      <td>0.010989</td>\n",
              "      <td>1175.000000</td>\n",
              "      <td>387.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>max tnr</td>\n",
              "      <td>0.961415</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>max fnr</td>\n",
              "      <td>0.961415</td>\n",
              "      <td>0.996596</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>max fpr</td>\n",
              "      <td>0.001868</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>399.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>max tpr</td>\n",
              "      <td>0.010989</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>387.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                         metric  threshold        value    idx\n",
              "0                        max f1   0.373550     0.696946  202.0\n",
              "1                        max f2   0.169555     0.798156  291.0\n",
              "2                  max f0point5   0.496359     0.718670  152.0\n",
              "3                  max accuracy   0.496359     0.845086  152.0\n",
              "4                 max precision   0.961415     1.000000    0.0\n",
              "5                    max recall   0.010989     1.000000  387.0\n",
              "6               max specificity   0.961415     1.000000    0.0\n",
              "7              max absolute_mcc   0.463329     0.585210  166.0\n",
              "8    max min_per_class_accuracy   0.321204     0.808323  224.0\n",
              "9   max mean_per_class_accuracy   0.294099     0.815475  235.0\n",
              "10                      max tns   0.961415  3292.000000    0.0\n",
              "11                      max fns   0.961415  1171.000000    0.0\n",
              "12                      max fps   0.001868  3292.000000  399.0\n",
              "13                      max tps   0.010989  1175.000000  387.0\n",
              "14                      max tnr   0.961415     1.000000    0.0\n",
              "15                      max fnr   0.961415     0.996596    0.0\n",
              "16                      max fpr   0.001868     1.000000  399.0\n",
              "17                      max tpr   0.010989     1.000000  387.0"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Gains/Lift Table: Avg response rate: 26.30 %, avg score: 26.24 %\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>group</th>\n",
              "      <th>cumulative_data_fraction</th>\n",
              "      <th>lower_threshold</th>\n",
              "      <th>lift</th>\n",
              "      <th>cumulative_lift</th>\n",
              "      <th>response_rate</th>\n",
              "      <th>score</th>\n",
              "      <th>cumulative_response_rate</th>\n",
              "      <th>cumulative_score</th>\n",
              "      <th>capture_rate</th>\n",
              "      <th>cumulative_capture_rate</th>\n",
              "      <th>gain</th>\n",
              "      <th>cumulative_gain</th>\n",
              "      <th>kolmogorov_smirnov</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>0.010074</td>\n",
              "      <td>0.909985</td>\n",
              "      <td>3.717220</td>\n",
              "      <td>3.717220</td>\n",
              "      <td>0.977778</td>\n",
              "      <td>0.933290</td>\n",
              "      <td>0.977778</td>\n",
              "      <td>0.933290</td>\n",
              "      <td>0.037447</td>\n",
              "      <td>0.037447</td>\n",
              "      <td>271.721986</td>\n",
              "      <td>271.721986</td>\n",
              "      <td>0.037143</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>0.020372</td>\n",
              "      <td>0.873553</td>\n",
              "      <td>3.553765</td>\n",
              "      <td>3.634594</td>\n",
              "      <td>0.934783</td>\n",
              "      <td>0.893587</td>\n",
              "      <td>0.956044</td>\n",
              "      <td>0.913220</td>\n",
              "      <td>0.036596</td>\n",
              "      <td>0.074043</td>\n",
              "      <td>255.376503</td>\n",
              "      <td>263.459434</td>\n",
              "      <td>0.072827</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>0.029998</td>\n",
              "      <td>0.840112</td>\n",
              "      <td>3.713290</td>\n",
              "      <td>3.659848</td>\n",
              "      <td>0.976744</td>\n",
              "      <td>0.858049</td>\n",
              "      <td>0.962687</td>\n",
              "      <td>0.895516</td>\n",
              "      <td>0.035745</td>\n",
              "      <td>0.109787</td>\n",
              "      <td>271.329045</td>\n",
              "      <td>265.984757</td>\n",
              "      <td>0.108268</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>0.040072</td>\n",
              "      <td>0.811351</td>\n",
              "      <td>3.463773</td>\n",
              "      <td>3.610555</td>\n",
              "      <td>0.911111</td>\n",
              "      <td>0.826173</td>\n",
              "      <td>0.949721</td>\n",
              "      <td>0.878083</td>\n",
              "      <td>0.034894</td>\n",
              "      <td>0.144681</td>\n",
              "      <td>246.377305</td>\n",
              "      <td>261.055509</td>\n",
              "      <td>0.141947</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>0.050146</td>\n",
              "      <td>0.786985</td>\n",
              "      <td>3.125844</td>\n",
              "      <td>3.513180</td>\n",
              "      <td>0.822222</td>\n",
              "      <td>0.799521</td>\n",
              "      <td>0.924107</td>\n",
              "      <td>0.862301</td>\n",
              "      <td>0.031489</td>\n",
              "      <td>0.176170</td>\n",
              "      <td>212.584397</td>\n",
              "      <td>251.318009</td>\n",
              "      <td>0.171006</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>6</td>\n",
              "      <td>0.100067</td>\n",
              "      <td>0.684817</td>\n",
              "      <td>3.017495</td>\n",
              "      <td>3.265892</td>\n",
              "      <td>0.793722</td>\n",
              "      <td>0.737285</td>\n",
              "      <td>0.859060</td>\n",
              "      <td>0.799933</td>\n",
              "      <td>0.150638</td>\n",
              "      <td>0.326809</td>\n",
              "      <td>201.749451</td>\n",
              "      <td>226.589176</td>\n",
              "      <td>0.307671</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>7</td>\n",
              "      <td>0.149989</td>\n",
              "      <td>0.592277</td>\n",
              "      <td>2.642439</td>\n",
              "      <td>3.058384</td>\n",
              "      <td>0.695067</td>\n",
              "      <td>0.635696</td>\n",
              "      <td>0.804478</td>\n",
              "      <td>0.745269</td>\n",
              "      <td>0.131915</td>\n",
              "      <td>0.458723</td>\n",
              "      <td>164.243870</td>\n",
              "      <td>205.838425</td>\n",
              "      <td>0.418930</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>8</td>\n",
              "      <td>0.200358</td>\n",
              "      <td>0.517372</td>\n",
              "      <td>2.399296</td>\n",
              "      <td>2.892692</td>\n",
              "      <td>0.631111</td>\n",
              "      <td>0.552632</td>\n",
              "      <td>0.760894</td>\n",
              "      <td>0.696841</td>\n",
              "      <td>0.120851</td>\n",
              "      <td>0.579574</td>\n",
              "      <td>139.929645</td>\n",
              "      <td>189.269179</td>\n",
              "      <td>0.514568</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>9</td>\n",
              "      <td>0.299978</td>\n",
              "      <td>0.385510</td>\n",
              "      <td>1.631742</td>\n",
              "      <td>2.473943</td>\n",
              "      <td>0.429213</td>\n",
              "      <td>0.448758</td>\n",
              "      <td>0.650746</td>\n",
              "      <td>0.614455</td>\n",
              "      <td>0.162553</td>\n",
              "      <td>0.742128</td>\n",
              "      <td>63.174181</td>\n",
              "      <td>147.394347</td>\n",
              "      <td>0.599965</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>10</td>\n",
              "      <td>0.400045</td>\n",
              "      <td>0.270227</td>\n",
              "      <td>1.165175</td>\n",
              "      <td>2.146568</td>\n",
              "      <td>0.306488</td>\n",
              "      <td>0.326354</td>\n",
              "      <td>0.564633</td>\n",
              "      <td>0.542389</td>\n",
              "      <td>0.116596</td>\n",
              "      <td>0.858723</td>\n",
              "      <td>16.517493</td>\n",
              "      <td>114.656824</td>\n",
              "      <td>0.622393</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>11</td>\n",
              "      <td>0.500112</td>\n",
              "      <td>0.169738</td>\n",
              "      <td>0.833483</td>\n",
              "      <td>1.883834</td>\n",
              "      <td>0.219239</td>\n",
              "      <td>0.219148</td>\n",
              "      <td>0.495524</td>\n",
              "      <td>0.477712</td>\n",
              "      <td>0.083404</td>\n",
              "      <td>0.942128</td>\n",
              "      <td>-16.651721</td>\n",
              "      <td>88.383360</td>\n",
              "      <td>0.599783</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>12</td>\n",
              "      <td>0.600179</td>\n",
              "      <td>0.089698</td>\n",
              "      <td>0.365712</td>\n",
              "      <td>1.630719</td>\n",
              "      <td>0.096197</td>\n",
              "      <td>0.125703</td>\n",
              "      <td>0.428944</td>\n",
              "      <td>0.419022</td>\n",
              "      <td>0.036596</td>\n",
              "      <td>0.978723</td>\n",
              "      <td>-63.428816</td>\n",
              "      <td>63.071893</td>\n",
              "      <td>0.513657</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>13</td>\n",
              "      <td>0.700022</td>\n",
              "      <td>0.042011</td>\n",
              "      <td>0.127860</td>\n",
              "      <td>1.416368</td>\n",
              "      <td>0.033632</td>\n",
              "      <td>0.063288</td>\n",
              "      <td>0.372562</td>\n",
              "      <td>0.368284</td>\n",
              "      <td>0.012766</td>\n",
              "      <td>0.991489</td>\n",
              "      <td>-87.214006</td>\n",
              "      <td>41.636808</td>\n",
              "      <td>0.395499</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>14</td>\n",
              "      <td>0.800090</td>\n",
              "      <td>0.018240</td>\n",
              "      <td>0.068039</td>\n",
              "      <td>1.247733</td>\n",
              "      <td>0.017897</td>\n",
              "      <td>0.028007</td>\n",
              "      <td>0.328204</td>\n",
              "      <td>0.325726</td>\n",
              "      <td>0.006809</td>\n",
              "      <td>0.998298</td>\n",
              "      <td>-93.196059</td>\n",
              "      <td>24.773268</td>\n",
              "      <td>0.268954</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>15</td>\n",
              "      <td>0.900157</td>\n",
              "      <td>0.009152</td>\n",
              "      <td>0.017010</td>\n",
              "      <td>1.110918</td>\n",
              "      <td>0.004474</td>\n",
              "      <td>0.012766</td>\n",
              "      <td>0.292216</td>\n",
              "      <td>0.290935</td>\n",
              "      <td>0.001702</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>-98.299015</td>\n",
              "      <td>11.091768</td>\n",
              "      <td>0.135480</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>16</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.001484</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.005346</td>\n",
              "      <td>0.263040</td>\n",
              "      <td>0.262421</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>-100.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    group  cumulative_data_fraction  ...  cumulative_gain  kolmogorov_smirnov\n",
              "0       1                  0.010074  ...       271.721986            0.037143\n",
              "1       2                  0.020372  ...       263.459434            0.072827\n",
              "2       3                  0.029998  ...       265.984757            0.108268\n",
              "3       4                  0.040072  ...       261.055509            0.141947\n",
              "4       5                  0.050146  ...       251.318009            0.171006\n",
              "5       6                  0.100067  ...       226.589176            0.307671\n",
              "6       7                  0.149989  ...       205.838425            0.418930\n",
              "7       8                  0.200358  ...       189.269179            0.514568\n",
              "8       9                  0.299978  ...       147.394347            0.599965\n",
              "9      10                  0.400045  ...       114.656824            0.622393\n",
              "10     11                  0.500112  ...        88.383360            0.599783\n",
              "11     12                  0.600179  ...        63.071893            0.513657\n",
              "12     13                  0.700022  ...        41.636808            0.395499\n",
              "13     14                  0.800090  ...        24.773268            0.268954\n",
              "14     15                  0.900157  ...        11.091768            0.135480\n",
              "15     16                  1.000000  ...         0.000000            0.000000\n",
              "\n",
              "[16 rows x 14 columns]"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "\n",
            "ModelMetricsBinomial: xgboost\n",
            "** Reported on validation data. **\n",
            "\n",
            "MSE: 0.13968665068245656\n",
            "RMSE: 0.3737467734743091\n",
            "LogLoss: 0.43910824060153614\n",
            "Mean Per-Class Error: 0.22457912457912454\n",
            "AUC: 0.8369583278674187\n",
            "AUCPR: 0.6774149713007829\n",
            "Gini: 0.6739166557348375\n",
            "\n",
            "Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.3147416114807129: \n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th>No</th>\n",
              "      <th>Yes</th>\n",
              "      <th>Error</th>\n",
              "      <th>Rate</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>No</td>\n",
              "      <td>616.0</td>\n",
              "      <td>154.0</td>\n",
              "      <td>0.2</td>\n",
              "      <td>(154.0/770.0)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Yes</td>\n",
              "      <td>74.0</td>\n",
              "      <td>223.0</td>\n",
              "      <td>0.2492</td>\n",
              "      <td>(74.0/297.0)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Total</td>\n",
              "      <td>690.0</td>\n",
              "      <td>377.0</td>\n",
              "      <td>0.2137</td>\n",
              "      <td>(228.0/1067.0)</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "             No    Yes   Error             Rate\n",
              "0     No  616.0  154.0     0.2    (154.0/770.0)\n",
              "1    Yes   74.0  223.0  0.2492     (74.0/297.0)\n",
              "2  Total  690.0  377.0  0.2137   (228.0/1067.0)"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Maximum Metrics: Maximum metrics at their respective thresholds\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>metric</th>\n",
              "      <th>threshold</th>\n",
              "      <th>value</th>\n",
              "      <th>idx</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>max f1</td>\n",
              "      <td>0.314742</td>\n",
              "      <td>0.661721</td>\n",
              "      <td>206.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>max f2</td>\n",
              "      <td>0.055472</td>\n",
              "      <td>0.738518</td>\n",
              "      <td>348.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>max f0point5</td>\n",
              "      <td>0.562409</td>\n",
              "      <td>0.645463</td>\n",
              "      <td>110.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>max accuracy</td>\n",
              "      <td>0.429109</td>\n",
              "      <td>0.802249</td>\n",
              "      <td>159.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>max precision</td>\n",
              "      <td>0.961857</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>max recall</td>\n",
              "      <td>0.005568</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>394.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>max specificity</td>\n",
              "      <td>0.961857</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>max absolute_mcc</td>\n",
              "      <td>0.351242</td>\n",
              "      <td>0.518205</td>\n",
              "      <td>189.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>max min_per_class_accuracy</td>\n",
              "      <td>0.287675</td>\n",
              "      <td>0.768831</td>\n",
              "      <td>219.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>max mean_per_class_accuracy</td>\n",
              "      <td>0.314742</td>\n",
              "      <td>0.775421</td>\n",
              "      <td>206.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>max tns</td>\n",
              "      <td>0.961857</td>\n",
              "      <td>770.000000</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>max fns</td>\n",
              "      <td>0.961857</td>\n",
              "      <td>296.000000</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>max fps</td>\n",
              "      <td>0.002040</td>\n",
              "      <td>770.000000</td>\n",
              "      <td>399.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>max tps</td>\n",
              "      <td>0.005568</td>\n",
              "      <td>297.000000</td>\n",
              "      <td>394.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>max tnr</td>\n",
              "      <td>0.961857</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>max fnr</td>\n",
              "      <td>0.961857</td>\n",
              "      <td>0.996633</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>max fpr</td>\n",
              "      <td>0.002040</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>399.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>max tpr</td>\n",
              "      <td>0.005568</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>394.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                         metric  threshold       value    idx\n",
              "0                        max f1   0.314742    0.661721  206.0\n",
              "1                        max f2   0.055472    0.738518  348.0\n",
              "2                  max f0point5   0.562409    0.645463  110.0\n",
              "3                  max accuracy   0.429109    0.802249  159.0\n",
              "4                 max precision   0.961857    1.000000    0.0\n",
              "5                    max recall   0.005568    1.000000  394.0\n",
              "6               max specificity   0.961857    1.000000    0.0\n",
              "7              max absolute_mcc   0.351242    0.518205  189.0\n",
              "8    max min_per_class_accuracy   0.287675    0.768831  219.0\n",
              "9   max mean_per_class_accuracy   0.314742    0.775421  206.0\n",
              "10                      max tns   0.961857  770.000000    0.0\n",
              "11                      max fns   0.961857  296.000000    0.0\n",
              "12                      max fps   0.002040  770.000000  399.0\n",
              "13                      max tps   0.005568  297.000000  394.0\n",
              "14                      max tnr   0.961857    1.000000    0.0\n",
              "15                      max fnr   0.961857    0.996633    0.0\n",
              "16                      max fpr   0.002040    1.000000  399.0\n",
              "17                      max tpr   0.005568    1.000000  394.0"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Gains/Lift Table: Avg response rate: 27.84 %, avg score: 26.17 %\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>group</th>\n",
              "      <th>cumulative_data_fraction</th>\n",
              "      <th>lower_threshold</th>\n",
              "      <th>lift</th>\n",
              "      <th>cumulative_lift</th>\n",
              "      <th>response_rate</th>\n",
              "      <th>score</th>\n",
              "      <th>cumulative_response_rate</th>\n",
              "      <th>cumulative_score</th>\n",
              "      <th>capture_rate</th>\n",
              "      <th>cumulative_capture_rate</th>\n",
              "      <th>gain</th>\n",
              "      <th>cumulative_gain</th>\n",
              "      <th>kolmogorov_smirnov</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>0.010309</td>\n",
              "      <td>0.887744</td>\n",
              "      <td>3.592593</td>\n",
              "      <td>3.592593</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.917639</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.917639</td>\n",
              "      <td>0.037037</td>\n",
              "      <td>0.037037</td>\n",
              "      <td>259.259259</td>\n",
              "      <td>259.259259</td>\n",
              "      <td>0.037037</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>0.020619</td>\n",
              "      <td>0.857889</td>\n",
              "      <td>3.265993</td>\n",
              "      <td>3.429293</td>\n",
              "      <td>0.909091</td>\n",
              "      <td>0.873083</td>\n",
              "      <td>0.954545</td>\n",
              "      <td>0.895361</td>\n",
              "      <td>0.033670</td>\n",
              "      <td>0.070707</td>\n",
              "      <td>226.599327</td>\n",
              "      <td>242.929293</td>\n",
              "      <td>0.069408</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>0.029991</td>\n",
              "      <td>0.821435</td>\n",
              "      <td>2.514815</td>\n",
              "      <td>3.143519</td>\n",
              "      <td>0.700000</td>\n",
              "      <td>0.839621</td>\n",
              "      <td>0.875000</td>\n",
              "      <td>0.877942</td>\n",
              "      <td>0.023569</td>\n",
              "      <td>0.094276</td>\n",
              "      <td>151.481481</td>\n",
              "      <td>214.351852</td>\n",
              "      <td>0.089081</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>0.040300</td>\n",
              "      <td>0.792897</td>\n",
              "      <td>2.612795</td>\n",
              "      <td>3.007752</td>\n",
              "      <td>0.727273</td>\n",
              "      <td>0.811202</td>\n",
              "      <td>0.837209</td>\n",
              "      <td>0.860869</td>\n",
              "      <td>0.026936</td>\n",
              "      <td>0.121212</td>\n",
              "      <td>161.279461</td>\n",
              "      <td>200.775194</td>\n",
              "      <td>0.112121</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>0.050609</td>\n",
              "      <td>0.774703</td>\n",
              "      <td>3.265993</td>\n",
              "      <td>3.060357</td>\n",
              "      <td>0.909091</td>\n",
              "      <td>0.783467</td>\n",
              "      <td>0.851852</td>\n",
              "      <td>0.845102</td>\n",
              "      <td>0.033670</td>\n",
              "      <td>0.154882</td>\n",
              "      <td>226.599327</td>\n",
              "      <td>206.035665</td>\n",
              "      <td>0.144493</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>6</td>\n",
              "      <td>0.100281</td>\n",
              "      <td>0.678362</td>\n",
              "      <td>2.508036</td>\n",
              "      <td>2.786777</td>\n",
              "      <td>0.698113</td>\n",
              "      <td>0.723228</td>\n",
              "      <td>0.775701</td>\n",
              "      <td>0.784735</td>\n",
              "      <td>0.124579</td>\n",
              "      <td>0.279461</td>\n",
              "      <td>150.803634</td>\n",
              "      <td>178.677743</td>\n",
              "      <td>0.248292</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>7</td>\n",
              "      <td>0.149953</td>\n",
              "      <td>0.591876</td>\n",
              "      <td>2.236897</td>\n",
              "      <td>2.604630</td>\n",
              "      <td>0.622642</td>\n",
              "      <td>0.632889</td>\n",
              "      <td>0.725000</td>\n",
              "      <td>0.734436</td>\n",
              "      <td>0.111111</td>\n",
              "      <td>0.390572</td>\n",
              "      <td>123.689727</td>\n",
              "      <td>160.462963</td>\n",
              "      <td>0.333430</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>8</td>\n",
              "      <td>0.200562</td>\n",
              "      <td>0.532071</td>\n",
              "      <td>1.862826</td>\n",
              "      <td>2.417445</td>\n",
              "      <td>0.518519</td>\n",
              "      <td>0.562693</td>\n",
              "      <td>0.672897</td>\n",
              "      <td>0.691099</td>\n",
              "      <td>0.094276</td>\n",
              "      <td>0.484848</td>\n",
              "      <td>86.282579</td>\n",
              "      <td>141.744548</td>\n",
              "      <td>0.393939</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>9</td>\n",
              "      <td>0.299906</td>\n",
              "      <td>0.374171</td>\n",
              "      <td>1.897973</td>\n",
              "      <td>2.245370</td>\n",
              "      <td>0.528302</td>\n",
              "      <td>0.455597</td>\n",
              "      <td>0.625000</td>\n",
              "      <td>0.613089</td>\n",
              "      <td>0.188552</td>\n",
              "      <td>0.673401</td>\n",
              "      <td>89.797345</td>\n",
              "      <td>124.537037</td>\n",
              "      <td>0.517557</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>10</td>\n",
              "      <td>0.400187</td>\n",
              "      <td>0.264581</td>\n",
              "      <td>1.107996</td>\n",
              "      <td>1.960361</td>\n",
              "      <td>0.308411</td>\n",
              "      <td>0.316410</td>\n",
              "      <td>0.545667</td>\n",
              "      <td>0.538745</td>\n",
              "      <td>0.111111</td>\n",
              "      <td>0.784512</td>\n",
              "      <td>10.799585</td>\n",
              "      <td>96.036083</td>\n",
              "      <td>0.532564</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>11</td>\n",
              "      <td>0.500469</td>\n",
              "      <td>0.163544</td>\n",
              "      <td>0.537210</td>\n",
              "      <td>1.675198</td>\n",
              "      <td>0.149533</td>\n",
              "      <td>0.216397</td>\n",
              "      <td>0.466292</td>\n",
              "      <td>0.474155</td>\n",
              "      <td>0.053872</td>\n",
              "      <td>0.838384</td>\n",
              "      <td>-46.278989</td>\n",
              "      <td>67.519767</td>\n",
              "      <td>0.468254</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>12</td>\n",
              "      <td>0.599813</td>\n",
              "      <td>0.093678</td>\n",
              "      <td>0.610063</td>\n",
              "      <td>1.498785</td>\n",
              "      <td>0.169811</td>\n",
              "      <td>0.128288</td>\n",
              "      <td>0.417187</td>\n",
              "      <td>0.416871</td>\n",
              "      <td>0.060606</td>\n",
              "      <td>0.898990</td>\n",
              "      <td>-38.993711</td>\n",
              "      <td>49.878472</td>\n",
              "      <td>0.414574</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>13</td>\n",
              "      <td>0.700094</td>\n",
              "      <td>0.043535</td>\n",
              "      <td>0.570786</td>\n",
              "      <td>1.365858</td>\n",
              "      <td>0.158879</td>\n",
              "      <td>0.067892</td>\n",
              "      <td>0.380187</td>\n",
              "      <td>0.366883</td>\n",
              "      <td>0.057239</td>\n",
              "      <td>0.956229</td>\n",
              "      <td>-42.921426</td>\n",
              "      <td>36.585850</td>\n",
              "      <td>0.354930</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>14</td>\n",
              "      <td>0.799438</td>\n",
              "      <td>0.019445</td>\n",
              "      <td>0.271139</td>\n",
              "      <td>1.229821</td>\n",
              "      <td>0.075472</td>\n",
              "      <td>0.029103</td>\n",
              "      <td>0.342321</td>\n",
              "      <td>0.324908</td>\n",
              "      <td>0.026936</td>\n",
              "      <td>0.983165</td>\n",
              "      <td>-72.886094</td>\n",
              "      <td>22.982068</td>\n",
              "      <td>0.254594</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>15</td>\n",
              "      <td>0.900656</td>\n",
              "      <td>0.009703</td>\n",
              "      <td>0.066529</td>\n",
              "      <td>1.099087</td>\n",
              "      <td>0.018519</td>\n",
              "      <td>0.013619</td>\n",
              "      <td>0.305931</td>\n",
              "      <td>0.289925</td>\n",
              "      <td>0.006734</td>\n",
              "      <td>0.989899</td>\n",
              "      <td>-93.347051</td>\n",
              "      <td>9.908660</td>\n",
              "      <td>0.123665</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>16</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.001368</td>\n",
              "      <td>0.101677</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.028302</td>\n",
              "      <td>0.005898</td>\n",
              "      <td>0.278351</td>\n",
              "      <td>0.261708</td>\n",
              "      <td>0.010101</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>-89.832285</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    group  cumulative_data_fraction  ...  cumulative_gain  kolmogorov_smirnov\n",
              "0       1                  0.010309  ...       259.259259            0.037037\n",
              "1       2                  0.020619  ...       242.929293            0.069408\n",
              "2       3                  0.029991  ...       214.351852            0.089081\n",
              "3       4                  0.040300  ...       200.775194            0.112121\n",
              "4       5                  0.050609  ...       206.035665            0.144493\n",
              "5       6                  0.100281  ...       178.677743            0.248292\n",
              "6       7                  0.149953  ...       160.462963            0.333430\n",
              "7       8                  0.200562  ...       141.744548            0.393939\n",
              "8       9                  0.299906  ...       124.537037            0.517557\n",
              "9      10                  0.400187  ...        96.036083            0.532564\n",
              "10     11                  0.500469  ...        67.519767            0.468254\n",
              "11     12                  0.599813  ...        49.878472            0.414574\n",
              "12     13                  0.700094  ...        36.585850            0.354930\n",
              "13     14                  0.799438  ...        22.982068            0.254594\n",
              "14     15                  0.900656  ...         9.908660            0.123665\n",
              "15     16                  1.000000  ...         0.000000            0.000000\n",
              "\n",
              "[16 rows x 14 columns]"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "\n",
            "Scoring History: \n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th>timestamp</th>\n",
              "      <th>duration</th>\n",
              "      <th>number_of_trees</th>\n",
              "      <th>training_rmse</th>\n",
              "      <th>training_logloss</th>\n",
              "      <th>training_auc</th>\n",
              "      <th>training_pr_auc</th>\n",
              "      <th>training_lift</th>\n",
              "      <th>training_classification_error</th>\n",
              "      <th>validation_rmse</th>\n",
              "      <th>validation_logloss</th>\n",
              "      <th>validation_auc</th>\n",
              "      <th>validation_pr_auc</th>\n",
              "      <th>validation_lift</th>\n",
              "      <th>validation_classification_error</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td></td>\n",
              "      <td>2021-11-14 15:23:49</td>\n",
              "      <td>0.004 sec</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.500000</td>\n",
              "      <td>0.693147</td>\n",
              "      <td>0.500000</td>\n",
              "      <td>0.263040</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.736960</td>\n",
              "      <td>0.500000</td>\n",
              "      <td>0.693147</td>\n",
              "      <td>0.500000</td>\n",
              "      <td>0.278351</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.721649</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td></td>\n",
              "      <td>2021-11-14 15:23:49</td>\n",
              "      <td>0.099 sec</td>\n",
              "      <td>5.0</td>\n",
              "      <td>0.365118</td>\n",
              "      <td>0.424825</td>\n",
              "      <td>0.865244</td>\n",
              "      <td>0.691459</td>\n",
              "      <td>3.386971</td>\n",
              "      <td>0.213342</td>\n",
              "      <td>0.378944</td>\n",
              "      <td>0.451415</td>\n",
              "      <td>0.834617</td>\n",
              "      <td>0.672088</td>\n",
              "      <td>3.113580</td>\n",
              "      <td>0.215558</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td></td>\n",
              "      <td>2021-11-14 15:23:49</td>\n",
              "      <td>0.232 sec</td>\n",
              "      <td>10.0</td>\n",
              "      <td>0.352809</td>\n",
              "      <td>0.386994</td>\n",
              "      <td>0.874015</td>\n",
              "      <td>0.711082</td>\n",
              "      <td>3.553765</td>\n",
              "      <td>0.208865</td>\n",
              "      <td>0.373768</td>\n",
              "      <td>0.434617</td>\n",
              "      <td>0.836993</td>\n",
              "      <td>0.667551</td>\n",
              "      <td>3.265993</td>\n",
              "      <td>0.212746</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td></td>\n",
              "      <td>2021-11-14 15:23:49</td>\n",
              "      <td>0.348 sec</td>\n",
              "      <td>15.0</td>\n",
              "      <td>0.347268</td>\n",
              "      <td>0.372678</td>\n",
              "      <td>0.881253</td>\n",
              "      <td>0.725417</td>\n",
              "      <td>3.553765</td>\n",
              "      <td>0.203492</td>\n",
              "      <td>0.371998</td>\n",
              "      <td>0.431782</td>\n",
              "      <td>0.839473</td>\n",
              "      <td>0.675680</td>\n",
              "      <td>3.265993</td>\n",
              "      <td>0.193065</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td></td>\n",
              "      <td>2021-11-14 15:23:50</td>\n",
              "      <td>0.480 sec</td>\n",
              "      <td>20.0</td>\n",
              "      <td>0.340930</td>\n",
              "      <td>0.360452</td>\n",
              "      <td>0.890702</td>\n",
              "      <td>0.743700</td>\n",
              "      <td>3.801702</td>\n",
              "      <td>0.174614</td>\n",
              "      <td>0.371194</td>\n",
              "      <td>0.433142</td>\n",
              "      <td>0.839617</td>\n",
              "      <td>0.679110</td>\n",
              "      <td>3.265993</td>\n",
              "      <td>0.189316</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td></td>\n",
              "      <td>2021-11-14 15:23:50</td>\n",
              "      <td>0.585 sec</td>\n",
              "      <td>25.0</td>\n",
              "      <td>0.336805</td>\n",
              "      <td>0.352489</td>\n",
              "      <td>0.896043</td>\n",
              "      <td>0.756353</td>\n",
              "      <td>3.717220</td>\n",
              "      <td>0.177748</td>\n",
              "      <td>0.373015</td>\n",
              "      <td>0.436319</td>\n",
              "      <td>0.837813</td>\n",
              "      <td>0.678109</td>\n",
              "      <td>3.265993</td>\n",
              "      <td>0.197751</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td></td>\n",
              "      <td>2021-11-14 15:23:50</td>\n",
              "      <td>0.761 sec</td>\n",
              "      <td>30.0</td>\n",
              "      <td>0.333664</td>\n",
              "      <td>0.346254</td>\n",
              "      <td>0.899490</td>\n",
              "      <td>0.764536</td>\n",
              "      <td>3.717220</td>\n",
              "      <td>0.173271</td>\n",
              "      <td>0.373747</td>\n",
              "      <td>0.439108</td>\n",
              "      <td>0.836958</td>\n",
              "      <td>0.677415</td>\n",
              "      <td>3.592593</td>\n",
              "      <td>0.213683</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "               timestamp  ... validation_lift  validation_classification_error\n",
              "0    2021-11-14 15:23:49  ...        1.000000                         0.721649\n",
              "1    2021-11-14 15:23:49  ...        3.113580                         0.215558\n",
              "2    2021-11-14 15:23:49  ...        3.265993                         0.212746\n",
              "3    2021-11-14 15:23:49  ...        3.265993                         0.193065\n",
              "4    2021-11-14 15:23:50  ...        3.265993                         0.189316\n",
              "5    2021-11-14 15:23:50  ...        3.265993                         0.197751\n",
              "6    2021-11-14 15:23:50  ...        3.592593                         0.213683\n",
              "\n",
              "[7 rows x 16 columns]"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Variable Importances: \n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>variable</th>\n",
              "      <th>relative_importance</th>\n",
              "      <th>scaled_importance</th>\n",
              "      <th>percentage</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>MonthlyCharges</td>\n",
              "      <td>843.539856</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.246661</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>tenure</td>\n",
              "      <td>745.628540</td>\n",
              "      <td>0.883928</td>\n",
              "      <td>0.218031</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Contract.Month-to-month</td>\n",
              "      <td>567.875000</td>\n",
              "      <td>0.673205</td>\n",
              "      <td>0.166053</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>TotalCharges</td>\n",
              "      <td>492.928711</td>\n",
              "      <td>0.584357</td>\n",
              "      <td>0.144138</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Contract.Two year</td>\n",
              "      <td>252.040436</td>\n",
              "      <td>0.298789</td>\n",
              "      <td>0.073700</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>ServiceCount</td>\n",
              "      <td>218.434998</td>\n",
              "      <td>0.258950</td>\n",
              "      <td>0.063873</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>Contract.One year</td>\n",
              "      <td>111.063660</td>\n",
              "      <td>0.131664</td>\n",
              "      <td>0.032476</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>PaperlessBilling.No</td>\n",
              "      <td>87.808418</td>\n",
              "      <td>0.104095</td>\n",
              "      <td>0.025676</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>SeniorCitizen</td>\n",
              "      <td>49.417854</td>\n",
              "      <td>0.058584</td>\n",
              "      <td>0.014450</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>gender.Female</td>\n",
              "      <td>42.659378</td>\n",
              "      <td>0.050572</td>\n",
              "      <td>0.012474</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>PaperlessBilling.Yes</td>\n",
              "      <td>5.267341</td>\n",
              "      <td>0.006244</td>\n",
              "      <td>0.001540</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>gender.Male</td>\n",
              "      <td>3.169271</td>\n",
              "      <td>0.003757</td>\n",
              "      <td>0.000927</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                   variable  relative_importance  scaled_importance  percentage\n",
              "0            MonthlyCharges           843.539856           1.000000    0.246661\n",
              "1                    tenure           745.628540           0.883928    0.218031\n",
              "2   Contract.Month-to-month           567.875000           0.673205    0.166053\n",
              "3              TotalCharges           492.928711           0.584357    0.144138\n",
              "4         Contract.Two year           252.040436           0.298789    0.073700\n",
              "5              ServiceCount           218.434998           0.258950    0.063873\n",
              "6         Contract.One year           111.063660           0.131664    0.032476\n",
              "7       PaperlessBilling.No            87.808418           0.104095    0.025676\n",
              "8             SeniorCitizen            49.417854           0.058584    0.014450\n",
              "9             gender.Female            42.659378           0.050572    0.012474\n",
              "10     PaperlessBilling.Yes             5.267341           0.006244    0.001540\n",
              "11              gender.Male             3.169271           0.003757    0.000927"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              ""
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3HirjVDR_Gsw"
      },
      "source": [
        "### **14. Retrive Model Parameters**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oG6DaafcX-Ks",
        "outputId": "db4b4d0f-fb20-49e2-9856-6d9febe21268"
      },
      "source": [
        "xgb_model.params"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'auc_type': {'actual': 'AUTO', 'default': 'AUTO', 'input': 'AUTO'},\n",
              " 'backend': {'actual': 'cpu', 'default': 'auto', 'input': 'auto'},\n",
              " 'booster': {'actual': 'gbtree', 'default': 'gbtree', 'input': 'gbtree'},\n",
              " 'build_tree_one_node': {'actual': False, 'default': False, 'input': False},\n",
              " 'calibrate_model': {'actual': False, 'default': False, 'input': False},\n",
              " 'calibration_frame': {'actual': None, 'default': None, 'input': None},\n",
              " 'categorical_encoding': {'actual': 'OneHotInternal',\n",
              "  'default': 'AUTO',\n",
              "  'input': 'AUTO'},\n",
              " 'checkpoint': {'actual': None, 'default': None, 'input': None},\n",
              " 'col_sample_rate': {'actual': 0.8, 'default': 1.0, 'input': 0.8},\n",
              " 'col_sample_rate_per_tree': {'actual': 0.8, 'default': 1.0, 'input': 0.8},\n",
              " 'colsample_bylevel': {'actual': 0.8, 'default': 1.0, 'input': 1.0},\n",
              " 'colsample_bynode': {'actual': 1.0, 'default': 1.0, 'input': 1.0},\n",
              " 'colsample_bytree': {'actual': 0.8, 'default': 1.0, 'input': 1.0},\n",
              " 'distribution': {'actual': 'bernoulli',\n",
              "  'default': 'AUTO',\n",
              "  'input': 'bernoulli'},\n",
              " 'dmatrix_type': {'actual': 'dense', 'default': 'auto', 'input': 'auto'},\n",
              " 'eta': {'actual': 0.3, 'default': 0.3, 'input': 0.3},\n",
              " 'export_checkpoints_dir': {'actual': None, 'default': None, 'input': None},\n",
              " 'fold_assignment': {'actual': None, 'default': 'AUTO', 'input': 'AUTO'},\n",
              " 'fold_column': {'actual': None, 'default': None, 'input': None},\n",
              " 'gainslift_bins': {'actual': -1, 'default': -1, 'input': -1},\n",
              " 'gamma': {'actual': 0.0, 'default': 0.0, 'input': 0.0},\n",
              " 'gpu_id': {'actual': None, 'default': None, 'input': None},\n",
              " 'grow_policy': {'actual': 'depthwise',\n",
              "  'default': 'depthwise',\n",
              "  'input': 'depthwise'},\n",
              " 'ignore_const_cols': {'actual': True, 'default': True, 'input': True},\n",
              " 'ignored_columns': {'actual': ['customerID'],\n",
              "  'default': None,\n",
              "  'input': ['customerID']},\n",
              " 'interaction_constraints': {'actual': None, 'default': None, 'input': None},\n",
              " 'keep_cross_validation_fold_assignment': {'actual': False,\n",
              "  'default': False,\n",
              "  'input': False},\n",
              " 'keep_cross_validation_models': {'actual': False,\n",
              "  'default': True,\n",
              "  'input': False},\n",
              " 'keep_cross_validation_predictions': {'actual': True,\n",
              "  'default': False,\n",
              "  'input': True},\n",
              " 'learn_rate': {'actual': 0.3, 'default': 0.3, 'input': 0.3},\n",
              " 'max_abs_leafnode_pred': {'actual': 0.0, 'default': 0.0, 'input': 0.0},\n",
              " 'max_bins': {'actual': 256, 'default': 256, 'input': 256},\n",
              " 'max_delta_step': {'actual': 0.0, 'default': 0.0, 'input': 0.0},\n",
              " 'max_depth': {'actual': 5, 'default': 6, 'input': 5},\n",
              " 'max_leaves': {'actual': 0, 'default': 0, 'input': 0},\n",
              " 'max_runtime_secs': {'actual': 32.497333984375,\n",
              "  'default': 0.0,\n",
              "  'input': 32.497333984375},\n",
              " 'min_child_weight': {'actual': 3.0, 'default': 1.0, 'input': 1.0},\n",
              " 'min_rows': {'actual': 3.0, 'default': 1.0, 'input': 3.0},\n",
              " 'min_split_improvement': {'actual': 0.0, 'default': 0.0, 'input': 0.0},\n",
              " 'model_id': {'actual': {'URL': '/3/Models/XGBoost_3_AutoML_1_20211114_152326',\n",
              "   '__meta': {'schema_name': 'ModelKeyV3',\n",
              "    'schema_type': 'Key<Model>',\n",
              "    'schema_version': 3},\n",
              "   'name': 'XGBoost_3_AutoML_1_20211114_152326',\n",
              "   'type': 'Key<Model>'},\n",
              "  'default': None,\n",
              "  'input': None},\n",
              " 'monotone_constraints': {'actual': None, 'default': None, 'input': None},\n",
              " 'nfolds': {'actual': 0, 'default': 0, 'input': 0},\n",
              " 'normalize_type': {'actual': 'tree', 'default': 'tree', 'input': 'tree'},\n",
              " 'nthread': {'actual': -1, 'default': -1, 'input': -1},\n",
              " 'ntrees': {'actual': 10000, 'default': 50, 'input': 10000},\n",
              " 'offset_column': {'actual': None, 'default': None, 'input': None},\n",
              " 'one_drop': {'actual': False, 'default': False, 'input': False},\n",
              " 'quiet_mode': {'actual': True, 'default': True, 'input': True},\n",
              " 'rate_drop': {'actual': 0.0, 'default': 0.0, 'input': 0.0},\n",
              " 'reg_alpha': {'actual': 0.0, 'default': 0.0, 'input': 0.0},\n",
              " 'reg_lambda': {'actual': 1.0, 'default': 1.0, 'input': 1.0},\n",
              " 'response_column': {'actual': {'__meta': {'schema_name': 'ColSpecifierV3',\n",
              "    'schema_type': 'VecSpecifier',\n",
              "    'schema_version': 3},\n",
              "   'column_name': 'Churn',\n",
              "   'is_member_of_frames': None},\n",
              "  'default': None,\n",
              "  'input': {'__meta': {'schema_name': 'ColSpecifierV3',\n",
              "    'schema_type': 'VecSpecifier',\n",
              "    'schema_version': 3},\n",
              "   'column_name': 'Churn',\n",
              "   'is_member_of_frames': None}},\n",
              " 'sample_rate': {'actual': 0.8, 'default': 1.0, 'input': 0.8},\n",
              " 'sample_type': {'actual': 'uniform',\n",
              "  'default': 'uniform',\n",
              "  'input': 'uniform'},\n",
              " 'save_matrix_directory': {'actual': None, 'default': None, 'input': None},\n",
              " 'scale_pos_weight': {'actual': 1.0, 'default': 1.0, 'input': 1.0},\n",
              " 'score_each_iteration': {'actual': False, 'default': False, 'input': False},\n",
              " 'score_tree_interval': {'actual': 5, 'default': 0, 'input': 5},\n",
              " 'seed': {'actual': 18, 'default': -1, 'input': 18},\n",
              " 'skip_drop': {'actual': 0.0, 'default': 0.0, 'input': 0.0},\n",
              " 'stopping_metric': {'actual': 'RMSE', 'default': 'AUTO', 'input': 'RMSE'},\n",
              " 'stopping_rounds': {'actual': 3, 'default': 0, 'input': 3},\n",
              " 'stopping_tolerance': {'actual': 0.014219127353430849,\n",
              "  'default': 0.001,\n",
              "  'input': 0.014219127353430849},\n",
              " 'subsample': {'actual': 0.8, 'default': 1.0, 'input': 1.0},\n",
              " 'training_frame': {'actual': {'URL': '/3/Frames/AutoML_1_20211114_152326_training_py_3_sid_bf21',\n",
              "   '__meta': {'schema_name': 'FrameKeyV3',\n",
              "    'schema_type': 'Key<Frame>',\n",
              "    'schema_version': 3},\n",
              "   'name': 'AutoML_1_20211114_152326_training_py_3_sid_bf21',\n",
              "   'type': 'Key<Frame>'},\n",
              "  'default': None,\n",
              "  'input': {'URL': '/3/Frames/AutoML_1_20211114_152326_training_py_3_sid_bf21',\n",
              "   '__meta': {'schema_name': 'FrameKeyV3',\n",
              "    'schema_type': 'Key<Frame>',\n",
              "    'schema_version': 3},\n",
              "   'name': 'AutoML_1_20211114_152326_training_py_3_sid_bf21',\n",
              "   'type': 'Key<Frame>'}},\n",
              " 'tree_method': {'actual': 'exact', 'default': 'auto', 'input': 'auto'},\n",
              " 'tweedie_power': {'actual': 1.5, 'default': 1.5, 'input': 1.5},\n",
              " 'validation_frame': {'actual': {'URL': '/3/Frames/py_5_sid_bf21',\n",
              "   '__meta': {'schema_name': 'FrameKeyV3',\n",
              "    'schema_type': 'Key<Frame>',\n",
              "    'schema_version': 3},\n",
              "   'name': 'py_5_sid_bf21',\n",
              "   'type': 'Key<Frame>'},\n",
              "  'default': None,\n",
              "  'input': {'URL': '/3/Frames/py_5_sid_bf21',\n",
              "   '__meta': {'schema_name': 'FrameKeyV3',\n",
              "    'schema_type': 'Key<Frame>',\n",
              "    'schema_version': 3},\n",
              "   'name': 'py_5_sid_bf21',\n",
              "   'type': 'Key<Frame>'}},\n",
              " 'weights_column': {'actual': None, 'default': None, 'input': None}}"
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TZ0WG9x-X-Bn",
        "outputId": "69ae1807-a2b7-4a76-8a7b-528e05fb4c3a"
      },
      "source": [
        "xgb_model.convert_H2OXGBoostParams_2_XGBoostParams()"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "({'alpha': 0.0,\n",
              "  'booster': 'gbtree',\n",
              "  'colsample_bylevel': 0.8,\n",
              "  'colsample_bytree': 0.8,\n",
              "  'eta': 0.3,\n",
              "  'gamma': 0.0,\n",
              "  'grow_policy': 'depthwise',\n",
              "  'lambda': 1.0,\n",
              "  'max_delta_step': 0.0,\n",
              "  'max_depth': 5,\n",
              "  'min_child_weight': 3.0,\n",
              "  'nround': 10000,\n",
              "  'nthread': 2,\n",
              "  'objective': 'binary:logistic',\n",
              "  'seed': 18,\n",
              "  'silent': True,\n",
              "  'subsample': 0.8,\n",
              "  'tree_method': 'exact'},\n",
              " 10000)"
            ]
          },
          "metadata": {},
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 175
        },
        "id": "q7Y0YmVmX9-8",
        "outputId": "5bdc0277-eaa9-48d5-d049-e151b2badd05"
      },
      "source": [
        "xgb_model.confusion_matrix()"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.37355014085769656: \n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th>No</th>\n",
              "      <th>Yes</th>\n",
              "      <th>Error</th>\n",
              "      <th>Rate</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>No</td>\n",
              "      <td>2803.0</td>\n",
              "      <td>489.0</td>\n",
              "      <td>0.1485</td>\n",
              "      <td>(489.0/3292.0)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Yes</td>\n",
              "      <td>285.0</td>\n",
              "      <td>890.0</td>\n",
              "      <td>0.2426</td>\n",
              "      <td>(285.0/1175.0)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Total</td>\n",
              "      <td>3088.0</td>\n",
              "      <td>1379.0</td>\n",
              "      <td>0.1733</td>\n",
              "      <td>(774.0/4467.0)</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "              No     Yes   Error             Rate\n",
              "0     No  2803.0   489.0  0.1485   (489.0/3292.0)\n",
              "1    Yes   285.0   890.0  0.2426   (285.0/1175.0)\n",
              "2  Total  3088.0  1379.0  0.1733   (774.0/4467.0)"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              ""
            ]
          },
          "metadata": {},
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-inKhxym_VYs"
      },
      "source": [
        "### **15. Show Variable Importance Plot**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 612
        },
        "id": "dmzkojWOX92e",
        "outputId": "34d2d1fb-43e7-4dd9-e297-b611f90b16d1"
      },
      "source": [
        "xgb_model.varimp_plot()"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAA6EAAAJTCAYAAAD5dhCMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdebhkVX3v//dHGpkFRwQcWlE0MgoNXlQmgya5TRJMyA/RqKgRhVwVk2tCQqKYxEBCDA44oRfaWaNoJKKiAVpRZGigoSHiQGyjOIFKyyCIzff3x15lF9VV55we2KcPvF/PU8+p2nvttdbeVec89Tlr7b1TVUiSJEmS1If7zXYHJEmSJEn3HYZQSZIkSVJvDKGSJEmSpN4YQiVJkiRJvTGESpIkSZJ6YwiVJEmSJPXGECpJupskByapJCesYz1HtnqOXINtFrVt5q9L25IkacNlCJWkDUSSD7YAdswMyn6+lX12H327txgK2Itnuy/3tLX5J8B9QZLF7bgcOEWZRaPHLp3fTvLWJEuT/CzJ7Um+nuRNSbadpt2dkrwtybVJbklya9v27UmesIb7sHNr+/okD5pQ5i/aPvzbFP351ySXJ/lpkjvbz4uT/EuSvcZsc0Krc/hxV5IVSS5M8qdJ5q3JvvTtvvL7L23oDKGStOF4d/v5J1MVaqOEBwM/AP7jHujHJcBvAKfeA3VLc9UmwGeBo4AbgP8HvAO4HXgVcGWSx4/bMMkrgf8Cjqb7vX0n8HbgeuDlwDWtzIxU1TXAXwPbt7pG29sd+Hvg+63+4XVJ8jrga8CrgQI+Cvwz8AHgF8ArgCVJ/nRCF74IvL49/hH4d1b9zXjvTPdD0n3XBv3fKkm6L6mqxUm+ATw5yZ5VdfmEoi8BApxRVb+6B/pxG3Dt+q5XmuNWAn8DvL2qfjZYmOR+dIHyZcC/Ar87vFGSFwBvBn4KPLuqvjSyfj+6EPfmJD+rqvfPsD+nAIcAf5Tk+YPtkmwKfBDYGHhRVf10ZLvXAicA3wWOqKqvjFac5GHAscDWE9peXFUnjGzzKLqg/dwkx1fV8hnuh6T7IEdCJWnDMhgNfem4lUk2Al5EN3rxnrbs0CQfSPKNNsXv1iSXJXll+4I8WsdgquFjk7wiyVVJfjGYojbpnNAkeyV5c5Ir27S925N8M8kbkzxwqp1KsrBN17u1TWP8+KRRoynqeErb7odJfpnku0nelWT7NalnQt2/nrqa5JlJLmhTJm9IckaSbVq5Jyf5dNuHW5KclTHnr2bVlM9NkvxDkm8nuSPJdUlel+T+E/rxm0k+147vHe09PSnJamFgqI37J3ltm9p5R3t/FwNntKJnjEyfnN+2375t95WhY/r9JB9K8qQx7c1v2y9qzz+S5Mb2OViS5JApju/hSc4d+twsT/LhJAvGlD0iyflJbmplv5bkb5JsMqn+PlTVnVX1huEA2pbfBfxde3ng8LokWwFvai+fOxpA2/YXAM9rL9/UtplJfwo4ElgBnNpCIMCJwM7AqVX1+ZH+PJYuSP8S+J1xAbTV/eOq+mu60dEZqar/Ab7eXj50dH37+3Fmkh+3z+l30k1F3m5cfUm2Szd9eXn7bN6Q5BMZP034/u3v3eXtd/O2tt2nkhzcyhyZpNomB4z8Tpww0/2UtH4YQiVpw/Jeui+IRyTZfMz63wF2AP6zqr7dlp0E7AlcDLwVeB+wJd3oy1RT495MN2VvWXs+9gvpkJcCz6H7onkG3VTEHwB/Bnxlii/Pf0A30vO91s5XgT8ELsoMz4VL8uLWv98Bzqf7Yr+EburykqEv4Ovq94Cz6aZbvhP4Jt0X/U8m+V/Al+lmEf2/1p/fBT6dMWG/+TfgxXTTpk+l++fBCcCZSTKyjy8DvgA8je54nUI3evaXwIWDIDzGmcAxwIV0x2UZsAj4VFv/KVZNnXw9cFNbvj9wXHt9ZmvvIuAw4JJ0UzrHeTTdlO35wPvppnLuAnwqyUEj+5Qki4CPALsBn2jtXADsRzeSN1z+dOBDwONan97WjsHfA5/LyPmGWXWO4gkT+tqXO9vP0ZkJhwEPBC6pqnMmbVxVnwMuBR7UtpmRFvxeATwAeG+SZ9JNDf4a3edm1IvoPr8fb1N6p6t/xjMtkjwSeAJwM6vC6GDdIXSfz98F/pNuxPjrdNOTlyR5zEj5x9D9fh8DXAe8ETgHWEj3uzD6D49FdH9bNqb7+/cW4EvArsBvtzJL6T7/AN/h7r8Ti2e6n5LWk6ry4cOHDx8b0IPuS30BR45Z96m27rChZTuOKXc/ugBawFNG1i1qy68HHjNm2wPb+hNGlj8a2GhM+Ze08n85svzItryAQ0bWvaotP3dC3+YPLduJLph/C9hhpPxv0k2T/OQMj+1g3xZP6OuvgANGjuMX2rqfAs8b2e7/tXW/P7J8cVv+DeCBQ8s3pQvhBTx/5NjeAfwceOJIXW9v5U+b0MZVwEPG7Otgn1b7HLX1DwO2GrN8d+AW4LMjy+cPvZ+vG1n3W235Z0aWH9WWXwJsPbJuI2C7Mf39BLDZSNkT2rpXTVh+wrh9nLDfg+O2qG0/7rF0qmM3ps6/bOU/POHz8YYZ1PGGVvY9M92XoW0/1ra9ne53Zc8J5c5r5V6ypm2MHO/FQ8fqH+j+KfWT9viDkW22bMtXAvtNOG6fH1l+Tlt+/Mjyp9L9jv4E2LIt2xq4iy60jvv79OCR16v9/vvw4aP/x6x3wIcPHz583P1BF6wK+PLI8u3oRlx+BGw8g3r2bPW8dmT5onFf6IfWH7gmX+zpzk9dAZw3svxIxgTNtm4julBZwKPH9G3+0LJT2rKFE9r/ZPtiulqgmmLfFk/o6/vHbPOCtu5LY9YdwPhQtpiRoDmmD+cPLTu+LfvHMeUfSBdOfwFsMqaN3x/dZmSfjlyLz+BZdIFm46Fl81t9yyd82f8OcOPIsmVtmyfPoM0r2ud7mwmflxvpRhSHlz8EeCJjQvgU7QyO20we0x47YG/gtvYe7Tiy7jOtnpfPoJ6XMybIz3CfHjPU55OmKPdfrcxvj1k3n9XD+LEjZU6Y4ljdSTc74hEj2zyvrf/QmDbnAd9u6x/Vlj2ivf4OY/7O0Y2+F/CC9voB7fVXgMzgWBlCffjYAB5emEiSNjzn0U1Be1qS36iqr7Xlg6l0i6pqMP2PJA8GXgP8b+CxwBYj9e0woZ1L1qRTSTamu/jKc4An0Y1ADE9DndTOF0cXVNXKJF8GdgSeTPeFc5J9288Dkuw9Zv3D6ELKTsBlU+3DDCwZs+z77ee4uq9vPx8xob7V9p1uSu9Kuv0e2LP9PG+0cFX9LMkVdNNnnwhcOVJkjd7HYUkW0oWfBXSBbvR7wUPoplwPW1pVK8dU911WvVck2YJumu6PquqKafqxOd0I7I3AsSMzlQfuoLsC669V1Y1tm7VxUFUtntCfRcALp6sgyU50U603Bp5TVdetZV/W1QlDzw9J8rqqumMN65gPvG5k2XdYdU7rsNdXuzBRm4q+HXAo3bTZQ5PsU1XfbWWn+mz/KsmXWttPBv6HVb8XFwz/nRtyHvDHrdz7qurnSf6Dbqrv0iRn0k33vri6i6xJ2gAZQiVpA1NVleQ9dBcY+RPgz9v5g4Npr4OLF9HOE7yUbiTkErrzoX5KNzK4Dd2010kXdPnhGnbto8Czgf+mmxb8Q7pgAN2VNCe186Np2p90Bc6BB7efr5mm3JbTrJ+JFWOW/WoG6zaeUN9q+96+eN9IF54HBsdgNPAxsnzceaFr+j4CkORVdAHjZ3RTjv+HbkSv6ALF7ox/T28aswy6YzH8T4lBX68fU3bUA+lG1B/K6kFog9QC6Pl053E+p6rOGlNs8N48cgZVDsp8f8pSq/fjD+lG6y+im0b8crrbpvz5hP78Bt2tXe6mBfK0Ouex6jzXKVV3Yabrgbe1iwwdT3fxo5e1Imv62V6b34XD6ab2PpdV533enuTjwP+tqkl/gyTNEkOoJG2YzqC74uYLkvwV3UVcHks35fVbQ+X+hC6A/npkYiDJvnQhdJKaaWfaVUyfTXdRkd+poQuWtJGQv5hi820nLH94+zku3A0brN+6qn4+g+5uSLalC3e/1r7gP4Ru+ubAYB8fDoy7YMx2I+V+rapm/D6O9OEEulCyZ1X9YGT9vuO2W0ODsDpphHzYYL+uqKo9pyy5AUjyG8C5dP8g+aOq+tSEol+mm8FwMF04m8rB7ed0Fwgb7sd2wLuAW4Hn0wXY36QbTf5UrX413q8AB7Uyp8+0nTVwcfu5z9Cy4c/2OKOf7TUtT1X9gjaFuF0gaX+66eh/TDfKut+0PZfUK6+OK0kboPaf+7PowsqhdGET4LSRoo9rP88cU80B67FLg3bOqtWvmLkPsNkU267Wj3S3mnl6eznlVE26ER6Ym18kx70HT6ebPjy834PnB44WbqPde9Cdo/m10fVTGEyZ3WjMuofQjSRdOCaAbsmqKZRrrapuBa4Gtk3y5GnK3kIXvndO8qB1bfuelGRXuvNKH0R3EZ5JARTg43RhfJ925dpJdT6T7vfop22bmTqdLgj/eVV9q00/HZzDvGjMFasX0Y1YH9aC9Po2uFXT8PfLqT7b81j1e335SPmnj14NuTlopPzdVNV3q+qDdBfL+lar58FDRe5i/O+EpB4ZQiVpwzWYdvvndKOQN9JdhGfY8vbzwOGF7Uv/X63Hvkxq52F0t9GYyjPG3FLh/9CdD3p+VU11Pih0tza5EzilTYG8m3aPwA01oP5thu6hmmRTumnWsOo+ngAfoNvHVyR5HHf393QXX/nAGp7n95P2c9zta35MN/V2rxY6B/3bmO5WFw9Zg3am8pb2810ZuddpkvuN3CPyX4H7A6ePux1Nkgcm2XNk2UOSPDHJ+urvlJLsQTcFdyu6C0KdPVX5NnI/mBb7oSRPG1PnU+luSwPw6qq6eYZ9OYbu9iNnV9W7htq8CPgnuhkSp4z05zq6q9neH/hsa3ucSbcDmqo/m9DdUgXufsuTf6cL10e02xwNO7b18z+ru90MVfU9uunh89v64TaeQjfl9me0v4VJHtr+MTBqC7op+r+iu2LwwE+Y2fRoSfcgp+NK0obr83ThbzC17dSq+uVImffRnSv5pnaPxm8Cj6e7/+In6M6VWh8upZvK9wdJLqSbZrgt3X07v87U57H9B919Nj9JNzKxR9vup6z60jpRVV3b7hN6OnBNks/R3fpkY7qAtR/dfT2fuHa7do/6Gl2fP04XMn+fLnyfTXeVTwCqanmSY+kC/eVJ/o1unw6gu9jPtYy/7+NUvkoXNI9tI0GD8xPfWlUrkryF7j6hy5J8ii6YHEQ3wnc+q0ac1sV76N6f5wPfbO3cQHdO4jPo3tMTAKrq9CR70e4NmeQcuqnMD6ILKvvTBfeXD9X/f+jOIX09d784z3rX/plwbuvPucC+E6Ytv6mqfn3ebNuvbYB/Bi5IspjuIlcF7EV3nO+iuxLt+2bYl52Ak+n+MfWSMUVOoLtQ2UuS/HtVfXpo3d/Rnfv5t3T3972M7nzyn9KFz/msmho8Op134MChe7OGbors79BdoOu/6W43M9j/W9rv78eALyb5GN37uhfwLLrP5cu4u5fT/b05Ocmz6C4Y9kjgj+iO1YuGwvoOwBVJltHdrui7dP+0OYRuSu9bRoL9ucBz2sWMLqf7vfzSmKnLku5Js315Xh8+fPjwMfnBqlt3FPCECWWeRDd198d054ZdRjd9d37bbtFI+UWM3AZlZP2BjL9P6IPo7lm5nG5q6HV0F0DZvC1bPlL+yFbPkXRfCL/a+ncT3fThnca0PbFvdDeeX0R3xc476L40X013TtwzZng8B/u2eFJfZ3o82rpJx3hxW74J3cjTt1uf/5suNG0yoX/Povvnw89a+W/RhZdxty1ZTDsldIr9/e123G8Z+hzNb+vmAX9Gd9uOX9CFgffT3bN0tfdh0r7OpD90t+n4It15fLe34/FBxtzPsn1WPk33ef5l69cl7TiO3kP1hEnvzRTHZPDeHDhFmcH+Hzm0bLD/0z0m/V49ke4WJl+n++fAbXT/THnH6H5N0/957XgU8Owpyu3SjvUPGH8f2SfQjZQupfudvJPud+rStnzcezM43qOPW+mu2vwP4z6rbdu96UYvb2jv6/+0fd9+Qvkd2vrvtPI30o2q7j1SbhvgtXRXzb2e7vfmB+19PoKR27bQXRDsQ3QXDVu5pp8fHz58rJ9Hqtb4egaSJGkKbbTrgKoae68RSZLuyzwnVJIkSZLUG0OoJEmSJKk3hlBJkiRJUm88J1SSJEmS1Btv0aI19t73vrde+MIXznY3JEmSJG24Jl6cz+m4WmO33nrrbHdBkiRJ0hxlCJUkSZIk9cYQKkmSJEnqjSFUkiRJktQbQ6gkSZIkqTeGUEmSJElSbwyhkiRJkqTeGEIlSZIkSb0xhEqSJEmSemMIlSRJkiT1xhAqSZIkSeqNIVSSJEmS1BtDqCRJkiSpN4ZQSZIkSVJvDKGSJEmSpN4YQiVJkiRJvTGESpIkSZJ6YwiVJEmSJPXGECpJkiRJ6o0hVJIkSZLUG0OoJEmSJKk3hlBJkiRJUm8MoZIkSZKk3hhCJUmSJEm9MYRKkiRJknpjCJUkSZIk9cYQKkmSJEnqjSFUkiRJktQbQ6gkSZIkqTfzZrsDmnuWXb+C+cedPdvdkCRJkgQsP2nhbHdhjTgSKkmSJEnqjSFUkiRJktQbQ6gkSZIkqTeGUEmSJElSbwyhkiRJkqTeGEIlSZIkSb0xhEqSJEmSemMIlSRJkiT1xhAqSZIkSeqNIVSSJEmS1BtDqCRJkiSpN4ZQSZIkSVJvDKGSJEmSpN4YQiVJkiRJvTGESpIkSZJ6YwiVJEmSJPXmXhtCk1SSDwy9npfkhiSfXsv6tklyzNDrAyfVlWRxkgXT1PfwJB9Jcl2Sy5J8JslOU9UrSZIkSXPdvTaEArcCuyTZrL1+JnD9OtS3DXDMtKVmIEmATwKLq2rHqtoL+Ctg2/VQ97x1rUOSJEmS7in35hAK8BlgYXt+BPDhwYokD0ry70muSnJRkt3a8hOSnN5GM/87ySvbJicBOyZZmuTktmzLJB9Pcm2SD7ZwyVAbL07ypqHXL01yCnAQcGdVvXOwrqqurKoLpqo3yWuTXJrk6iSnDS1fnORNSZYAr0qyd9uvpUlOTnJ1K7dRe31pW/+ytny7JF9q5a9Ost+6H3pJkiRJWt29PYR+BHhOkk2B3YCLh9a9HriiqnYD/hp439C6JwK/BewDvC7JxsBxwHVVtUdVvaaVezJwLPAk4LHA00ba/zfgd9v2AC8CTgd2AS6bot+T6j21qvauql2AzYBDhra5f1UtqKo3AmcAL6uqPYCVQ2VeAqyoqr2BvYGXJnkM8FzgnFZ+d2DpaIeSHJVkSZIlK29bMUXXJUmSJGmye3UIraqrgPl0o6CfGVn9dOD9rdx5wIOTPKCtO7uq7qiqG4EfM3ma7CVV9b2quosuuM0faf8W4DzgkCRPBDauqmUz6Pqkeg9KcnGSZcAzgJ2HtvkodOeuAltV1Vfb8g8NlXkW8IIkS+kC+YOBxwOXAi9KcgKwa1XdPNqhqjqthdwFG22+9Qx2QZIkSZJWd184f/As4F+AA+lC10zcMfR8JZOP00zKvYdupPVauhFKgGuAw9ak/Taa+3ZgQVV9twXGTYfK3TpFfQMBXlFV56y2ItmfburyoiT/WlXvW21rSZIkSVpH9+qR0OZ04PVjRiAvAJ4H3ZVugRur6udT1HMzsNWaNl5VFwOPpJvyOjgn9TxgkyRHDcol2W2aczEHgfPGJFsyIcRW1U3AzUme0hY9Z2j1OcDRg+nB7Wq8WyR5NPCjqno3XWjec412UpIkSZJm6F4/ElpV3wPeMmbVCcDpSa4CbgNeOE09P0nylXaRn88CZ69BN/4N2KOqftbqqiTPBt6U5C+B24HldOeB7jCh/ZuSvBu4Gvgh3RTaSV4CvDvJXcAXgcFJnO+hm9p7ebuo0Q3AoXSjxK9JcidwC/CCNdg3SZIkSZqxVNVs9+Fer93385SqOren9rZs56OS5Dhgu6p61fqq/+jjT6zPrtxtfVUnSZIkaR0sP2nh9IX6l0kr7gvTcWdNkm2SfAP4RV8BtFk4uN0KsB/wDz22LUmSJEkT3eun486mdn7mTrPQ7kdpV8uVJEmSpA2JI6GSJEmSpN4YQiVJkiRJvTGESpIkSZJ6YwiVJEmSJPXGECpJkiRJ6o0hVJIkSZLUG0OoJEmSJKk3hlBJkiRJUm8MoZIkSZKk3hhCJUmSJEm9MYRKkiRJknozb7Y7oLln1x225h3HLJztbkiSJEmagxwJlSRJkiT1xhAqSZIkSeqNIVSSJEmS1BtDqCRJkiSpN4ZQSZIkSVJvDKGSJEmSpN4YQiVJkiRJvTGESpIkSZJ6YwiVJEmSJPVm3mx3QHPPsutXMP+4s2e7G5IkSZpjlp+0cLa7oA2AI6GSJEmSpN4YQiVJkiRJvTGESpIkSZJ6YwiVJEmSJPXGECpJkiRJ6o0hVJIkSZLUG0OoJEmSJKk3hlBJkiRJUm8MoZIkSZKk3hhCJUmSJEm9MYRKkiRJknpjCJUkSZIk9cYQKkmSJEnqjSFUkiRJktQbQ6gkSZIkqTeGUEmSJElSbwyhG6Ak2yQ5Zrb7IUmSJEnrmyF0w7QNcI+G0CTz7sn6JUmSJGkcQ+iG6SRgxyRLk5yc5DVJLk1yVZLXAySZn+RrSd6d5Jokn0+yWVu3OMmC9vwhSZa350cmOSvJecC5SbZIcnqSS5JckeT3Z2l/JUmSJN1HGEI3TMcB11XVHsAXgMcD+wB7AHsl2b+VezzwtqraGbgJ+MMZ1L0ncFhVHQAcD5xXVfsABwEnJ9li3EZJjkqyJMmSlbetWJd9kyRJknQfZgjd8D2rPa4ALgeeSBc+Ab5dVUvb88uA+TOo7wtV9dOhuo9LshRYDGwKPGrcRlV1WlUtqKoFG22+9drshyRJkiTheYEbvgAnVtW77rYwmQ/cMbRoJbBZe/4rVv2DYdOR+m4dqfsPq+rr66uzkiRJkjQVR0I3TDcDW7Xn5wAvTrIlQJIdkjxsmu2XA3u154dNUe4c4BVJ0up+8lr3WJIkSZJmwBC6AaqqnwBfSXI18EzgQ8BXkywDPs6qgDrJvwBHJ7kCeMgU5f4e2Bi4Ksk17bUkSZIk3WNSVbPdB80xRx9/Yn125W6z3Q1JkiTNMctPWjjbXVB/MmmFI6GSJEmSpN4YQiVJkiRJvTGESpIkSZJ6YwiVJEmSJPXGECpJkiRJ6o0hVJIkSZLUG0OoJEmSJKk3hlBJkiRJUm8MoZIkSZKk3hhCJUmSJEm9MYRKkiRJknpjCJUkSZIk9cYQKkmSJEnqjSFUkiRJktSbebPdAc09u+6wNe84ZuFsd0OSJEnSHORIqCRJkiSpN4ZQSZIkSVJvDKGSJEmSpN4YQiVJkiRJvTGESpIkSZJ6YwiVJEmSJPXGECpJkiRJ6o0hVJIkSZLUG0OoJEmSJKk382a7A5p7ll2/gvnHnT3b3ZAkSeto+UkLZ7sLku6DHAmVJEmSJPXGECpJkiRJ6o0hVJIkSZLUG0OoJEmSJKk3hlBJkiRJUm8MoZIkSZKk3hhCJUmSJEm9MYRKkiRJknpjCJUkSZIk9cYQKkmSJEnqjSFUkiRJktQbQ6gkSZIkqTeGUEmSJElSbwyhkiRJkqTeGEIlSZIkSb2ZUQhN8vAkH0lyXZLLknwmyU5r2liSY5NsvubdHFvX/CTPnWJ9JfnA0Ot5SW5I8um1bG+bJMcMvT5wJnVN18++JTk0yZOGXi9OsmA2+yRJkiTpvmPaEJokwCeBxVW1Y1XtBfwVsO1atHcsMDaEJtloDeuaD0wV7m4FdkmyWXv9TOD6NWxj2DbAMdOWWt18pu5n3w4FnjRtKUmSJEm6B8xkJPQg4M6qeudgQVVdCXw5yclJrk6yLMnh8OsRwsVJPp7k2iQfTOeVwPbA+UnOb2VvSfLGJFcC+yZ5bZJLW52ntQBMkscl+c8kVya5PMmOwEnAfkmWJnn1hL5/BljYnh8BfHiwIsmDkvx7kquSXJRkt7b8hCSnt33479ZvWns7tvZObsu2HN3PMX24Wz+TbJrkjHbMrkhy0LiOJ1mU5B2tb//djuvpSb6WZNFQuSNaXVcn+aeh5bckeUM7Zhcl2TbJU4HfA05u/dmxFf+jJJck+UaS/SYcS0mSJElaZzMJobsAl41Z/gfAHsDuwMF0wWa7tu7JdKOeTwIeCzytqt4CfB84qKoGwWsL4OKq2r2qvgycWlV7V9UuwGbAIa3cB4G3VdXuwFOBHwDHARdU1R5VdcqEvn8EeE6STYHdgIuH1r0euKKqdgP+Gnjf0LonAr8F7AO8LsnGrb3rWnuvmbSfY/ow2s8/BaqqdqULxu9t/RvngcC+wKuBs4BTgJ2BXZPskWR74J+AZ9C9F3snObRtuwVwUTtmXwJeWlUXtnpe0/pzXSs7r6r2afvyunEdSXJUkiVJlqy8bcWE7kqSJEnS1NblwkRPBz5cVSur6kfAF4G927pLqup7VXUXsJRuSuo4K4Ezh14flOTiJMvogtXOSbYCdqiqTwJU1e1VddtMOlhVV7W2j6AbFR3t//tbufOAByd5QFt3dlXdUVU3Aj9m8tTjme7naLsfaO1eC3wHmHR+7X9UVQHLgB9V1bLW1jWtrb3ppknfUFW/ogvr+7dtfwkMzlm9bJq+fWK6clV1WlUtqKoFG22+9XT7KEmSJEljzSSEXgPstYb13jH0fCUwb0K526tqJUAbDXw7cFgbJXw3MGmEcE2cBfwLQ1NxZ2Cm/V+tXJKntKmuS5P83kwbbFNnlyZZOqb+u0baumuKPg3c2QLsr/s2Rdk7ZlhOkiRJktbJTELoecAmSY4aLGjnT94EHJ5koyQPpRuBu2Saum4GtpqwbhA4b0yyJXAYQFXdDHxvMM00ySbprrA7VV3DTgdeX1XLRpZfADyv1XkgcGNV/Xwt+/5rVXVxm+q6R1WdNWa74XZ3Ah4FfL2qjh9sN4N9GrgEOCDJQ9qFnY6gG5GeykyPmyRJkiStd9OG0Daa9mzg4HS3aLkGOBH4EHAVcNYEHA0AACAASURBVCVdUP2LqvrhNNWdBnxucGGikXZuohv9vBo4B7h0aPXzgVcmuQq4EHh4a3tlu/DOq5Nsn2R0yi1tuuxbxvTlBGCvVudJwAun6nhV/QT4SrsA0MlTlR1xt37Sjfber005/ihwZFXdMWUNk/s0ODf2fLr34bKq+tQ0m30EeE27KNKO05SVJEmSpPUqq2ZsSjNz9PEn1mdX7jbb3ZAkSeto+UkLpy8kSWtn3J1DgHW7MJEkSZIkSWvEECpJkiRJ6o0hVJIkSZLUG0OoJEmSJKk3hlBJkiRJUm8MoZIkSZKk3hhCJUmSJEm9MYRKkiRJknpjCJUkSZIk9cYQKkmSJEnqjSFUkiRJktQbQ6gkSZIkqTeGUEmSJElSbwyhkiRJkqTezJvtDmju2XWHrXnHMQtnuxuSJEmS5iBHQiVJkiRJvTGESpIkSZJ6YwiVJEmSJPXGECpJkiRJ6o0hVJIkSZLUG0OoJEmSJKk3hlBJkiRJUm8MoZIkSZKk3hhCJUmSJEm9mTfbHdDcs+z6Fcw/7uzZ7oYkSWtt+UkLZ7sLknSf5UioJEmSJKk3hlBJkiRJUm8MoZIkSZKk3hhCJUmSJEm9MYRKkiRJknpjCJUkSZIk9cYQKkmSJEnqjSFUkiRJktQbQ6gkSZIkqTeGUEmSJElSbwyhkiRJkqTeGEIlSZIkSb0xhEqSJEmSemMIlSRJkiT1xhAqSZIkSeqNIVSSJEmS1BtD6ARJHpxkaXv8MMn1Q6/vP1L22CSbz6DOxUkWtOdbJnlXkuuSXNbWPSXJ/CRX31P7JUmSJEmzad5sd2BDVVU/AfYASHICcEtV/cuE4scCHwBuW4Mm3gN8G3h8Vd2V5DHAk4AfrXWnu77Oq6pfrUsdkiRJknRPcSR0DST5zSRXJFmW5PQkmyR5JbA9cH6S81u5dyRZkuSaJK8fU8+OwFOAv6mquwCq6ttVdXYrslGSd7ftP59ks7bdS5NcmuTKJGcORl+TLEryziQXA/+cZMckF7V+/kOSW4bafk2r46pB35JskeTsVu/VSQ6/546iJEmSpPsyQ+jMbQosAg6vql3pRpGPrqq3AN8HDqqqg1rZ46tqAbAbcECS3Ubq2hlYWlUrJ7T1eOBtVbUzcBPwh235J6pq76raHfga8JKhbR4BPLWq/gx4M/Dm1s/vDQokeVarex+6Ud69kuwP/Dbw/aravap2AT432qEkR7VgvWTlbSumOVSSJEmSNJ4hdOY2Ar5dVd9or98L7D+h7P+X5HLgCrrA+aQ1bOvbVbW0Pb8MmN+e75LkgiTLgOe1ugc+NhRq9wU+1p5/aKjMs9rjCuBy4Il0oXQZ8Mwk/5Rkv6paLWVW1WlVtaCqFmy0+dZruDuSJEmS1PGc0PWsndv5f4G9q+pnSRbRjaIOuwbYPclGE0ZD7xh6vhLYrD1fBBxaVVcmORI4cKjcrTPpHnBiVb1rTL/3BP438A9Jzq2qv5tBfZIkSZK0RhwJnbmVwPwkj2uvnw98sT2/GdiqPX8AXSBckWRb4HdGK6qq64AlwOuTBKBdFXfhNH3YCvhBko3pRkInuYhVU3ifM7T8HODFSbZsbe6Q5GFJtgduq6oPACcDe07TD0mSJElaK46EztztwIuAjyWZB1wKvLOtOw34XJLvV9VBSa4ArgW+C3xlQn1/ArwR+FaSXwA3Aq+Zpg9/C1wM3NB+bjWh3LHAB5IcT3d+5wqAqvp8kt8Avtqy7y3AHwOPA05OchdwJ3D0NP2QJEmSpLWSqprtPmg9a1fN/UVVVZLnAEdU1e+vr/qPPv7E+uzK0WstSZI0dyw/abrJR5KkdZRJKxwJvXfaCzi1TfW9CXjxLPdHkiRJkgBD6L1SVV0A7D7b/ZAkSZKkUV6YSJIkSZLUG0OoJEmSJKk3hlBJkiRJUm8MoZIkSZKk3hhCJUmSJEm9MYRKkiRJknpjCJUkSZIk9cYQKkmSJEnqjSFUkiRJktQbQ6gkSZIkqTeGUEmSJElSbwyhkiRJkqTezJvtDmju2XWHrXnHMQtnuxuSJEmS5iBHQiVJkiRJvTGESpIkSZJ6YwiVJEmSJPXGECpJkiRJ6o0hVJIkSZLUG0OoJEmSJKk3hlBJkiRJUm8MoZIkSZKk3hhCJUmSJEm9mTfbHdDcs+z6Fcw/7uzZ7oZ0n7L8pIWz3QVJkqT1wpFQSZIkSVJvDKGSJEmSpN4YQiVJkiRJvTGESpIkSZJ6YwiVJEmSJPXGECpJkiRJ6o0hVJIkSZLUG0OoJEmSJKk3hlBJkiRJUm8MoZIkSZKk3hhCJUmSJEm9MYRKkiRJknpjCJUkSZIk9cYQKkmSJEnqjSFUkiRJktSbORlCkzw8yUeSXJfksiSfSbLTWtZ1bJLN11O/5id57pjlD06ytD1+mOT6odf3Xx9tS5IkSdJcMOdCaJIAnwQWV9WOVbUX8FfAtmtZ5bHA2BCaZKM1rGs+sFoIraqfVNUeVbUH8E7glMHrqvrlmnb4npZk3mz3QZIkSdK905wLocBBwJ1V9c7Bgqq6sqouSOfkJFcnWZbkcIAkByZZnOTjSa5N8sFW9pXA9sD5Sc5vZW9J8sYkVwL7Jnltkktbnae1EEySxyX5zyRXJrk8yY7AScB+bYTz1VPtRJKHJbmsPd89SSV5VHt9XZLN28jqeUmuSnLuYP1QHfdL8s0kDx16/a0kD22PM1vfL03ytFZmnyRfTXJFkguTPKEtPzLJWUnOA85d53dJkiRJksaYiyF0F+CyCev+ANgD2B04GDg5yXZt3ZPpRj2fBDwWeFpVvQX4PnBQVR3Uym0BXFxVu1fVl4FTq2rvqtoF2Aw4pJX7IPC2qtodeCrwA+A44II2wnnKVDtRVT8GNk3yAGA/YAldgH008OOqug14K/DeqtqttfeWkTruAj4APK8tOhi4sqpuAN5MN+K6N/CHwHtamWuB/arqycBrgX8cqnJP4LCqOmC0v0mOSrIkyZKVt62YatckSZIkaaK5GEKn8nTgw1W1sqp+BHwR2Lutu6SqvteC21K6qbPjrATOHHp9UJKLkywDngHsnGQrYIeq+iRAVd3eQuOauhB4GrA/XRjcny6QXtDW7wt8qD1/f9u/UacDL2jPXwyc0Z4fDJyaZClwFvCAJFsCWwMfS3I1cAqw81BdX6iqn47raFWdVlULqmrBRptvvcY7KkmSJEkAc/Hcv2uAw9ZiuzuGnq9k8r7fXlUrAZJsCrwdWFBV301yArDpWrQ9yZfoQuejgU8BfwkUcPZMK2j9+lGSZwD7sGpU9H7A/6qq24fLJzkVOL+qnp1kPrB4aPWta7cbkiRJkjQzc3Ek9DxgkyRHDRYk2S3JYATx8CQbtfMk9wcumaa+m4GtJqwbBM4b2yjiYQBVdTPwvSSHtvY3aVfYnaqucS4A/hj4Zhuh/Snwv4Evt/UXAs9pz5/HqhHSUe+hm5b7sUGABj4PvGJQIMke7enWwPXt+ZFr0FdJkiRJWmdzLoRWVQHPBg5uF/C5BjgR+CHdVXOvAq6kC6t/UVU/nKbK04DPDS5MNNLWTcC7gauBc4BLh1Y/H3hlkqvowuLDW9sr28WKXp1k+ySfmWJflgOhGxGFLnzeVFU/a69fAbyotfF84FUTqjoL2JJVU3EBXgksaBc1+i/g5W35PwMnJrmCuTkSLkmSJGkOS5fpNJclWUB3EaL9+mjv6ONPrM+u3K2PpiQ1y09aONtdkCRJWhOZtMKRsDkuyXHA0aw6F1SSJEmSNlhzbjqu7q6qTqqqR7fbyUiSJEnSBs0QKkmSJEnqjSFUkiRJktQbQ6gkSZIkqTeGUEmSJElSbwyhkiRJkqTeGEIlSZIkSb0xhEqSJEmSemMIlSRJkiT1xhAqSZIkSeqNIVSSJEmS1BtDqCRJkiSpN4ZQSZIkSVJv5s12BzT37LrD1rzjmIWz3Q1JkiRJc5AjoZIkSZKk3hhCJUmSJEm9MYRKkiRJknpjCJUkSZIk9cYQKkmSJEnqjSFUkiRJktQbQ6gkSZIkqTeGUEmSJElSbwyhkiRJkqTezJvtDmjuWXb9CuYfd/Zsd0OaNctPWjjbXZAkSZqzHAmVJEmSJPXGECpJkiRJ6o0hVJIkSZLUG0OoJEmSJKk3hlBJkiRJUm8MoZIkSZKk3hhCJUmSJEm9MYRKkiRJknpjCJUkSZIk9cYQKkmSJEnqjSFUkiRJktQbQ6gkSZIkqTeGUEmSJElSbwyhkiRJkqTeGEIlSZIkSb0xhEqSJEmSemMIHSPJ8UmuSXJVkqVJnrIe6vy7JAev5bY7JflMkm8muTzJvyXZdl37NNLGoUmetD7rlCRJkqRR82a7AxuaJPsChwB7VtUdSR4C3H+G286rql+NW1dVr13L/mwKnA38WVX9R1t2IPBQ4EdrU+cEhwKfBv5rPdYpSZIkSXfjSOjqtgNurKo7AKrqxqr6fpK9knwxyWVJzkmyHUCSxUnelGQJcHyS7yS5X1u3RZLvJtk4yaIkh7Xleye5MMmVSS5JslWSjZKcnOTSNgL7staf5wJfHQTQ1qfFVXV1kk2TnJFkWZIrkhzU6j8yyamD8kk+3YIrSW5J8obW9kVJtk3yVOD3gJPbyO+O9/AxliRJknQfZQhd3eeBRyb5RpK3JzkgycbAW4HDqmov4HTgDUPb3L+qFlTV64GlwAFt+SHAOVV156BgkvsDHwVeVVW7AwcDvwBeAqyoqr2BvYGXJnkMsAtw2YS+/ilQVbUrcATw3jZyOpUtgIta218CXlpVFwJnAa+pqj2q6rrRjZIclWRJkiUrb1sxTROSJEmSNJ4hdERV3QLsBRwF3EAXGF9GFwa/kGQp8DfAI4Y2++jI88Pb8+eMrAN4AvCDqrq0tffzNoX3WcALWv0XAw8GHj9Nd58OfKDVcy3wHWCnabb5Jd20W+jC7fxpytPqP60F7QUbbb71TDaRJEmSpNV4TugYVbUSWAwsTrKMbsTxmqrad8Imtw49Pwv4xyQPoguz582w2QCvqKpz7rYweSSrRlZn6lfc/R8Mw6Ojd1ZVtecr8TMgSZIkqUeOhI5I8oQkwyOQewBfAx7aLlpEO8dz53Hbt5HUS4E3A59ugXbY14Htkuzd6toqyTzgHODoNvV3cEXcLYAPAU9NsnCoj/sn2QW4AHjeoDzwqFb/cmCPJPdrIXafGez6zcBWMygnSZIkSWvNUbDVbQm8Nck2dCOK36Kbmnsa8JYkW9MdtzcB10yo46PAx4ADR1dU1S+THN7a2IzufNCDgffQTY29PEnopgIfWlUrkhwCvCnJm4A7gauAVwFvB97RRmt/BRzZruj7FeDbdFe6/Rpw+Qz2+yPAu5O8ku7c19XOC5UkSZKkdZVVMzOlmTn6+BPrsyt3m+1uSLNm+UkLpy8kSZJ035ZJK5yOK0mSJEnqjSFUkiRJktQbQ6gkSZIkqTeGUEmSJElSbwyhkiRJkqTeGEIlSZIkSb0xhEqSJEmSemMIlSRJkiT1xhAqSZIkSeqNIVSSJEmS1BtDqCRJkiSpN4ZQSZIkSVJvDKGSJEmSpN7Mm+0OaO7ZdYeteccxC2e7G5IkSZLmIEdCJUmSJEm9MYRKkiRJknpjCJUkSZIk9cYQKkmSJEnqjSFUkiRJktQbQ6gkSZIkqTeGUEmSJElSbwyhkiRJkqTeGEIlSZIkSb2ZN9sd0Nyz7PoVzD/u7Nnuxn3e8pMWznYXJEmSpDXmSKgkSZIkqTeGUEmSJElSbwyhkiRJkqTeGEIlSZIkSb0xhEqSJEmSemMIlSRJkiT1xhAqSZIkSeqNIVSSJEmS1BtDqCRJkiSpN4ZQSZIkSVJvDKGSJEmSpN4YQiVJkiRJvTGESpIkSZJ6YwiVJEmSJPXGECpJkiRJ6o0hVJIkSZLUmzkVQpM8PMlHklyX5LIkn0my01rWdWySzddTv+Ynee4U6x+R5FNJvtn6/uYk918fbUuSJEnSXDJnQmiSAJ8EFlfVjlW1F/BXwLZrWeWxwNgQmmSjNaxrPjA2hLZ+fwL496p6PLATsCXwhjVsozdJ5s12HyRJkiTdO82ZEAocBNxZVe8cLKiqK6vqgnROTnJ1kmVJDgdIcmCSxUk+nuTaJB9sZV8JbA+cn+T8VvaWJG9MciWwb5LXJrm01XlaC5MkeVyS/0xyZZLLk+wInATsl2RpkleP9PsZwO1VdUbr80rg1cCLk2ye5Mgkn0jyuTZS+s+DDZM8K8lXWzsfS7LlcMVJdkxy+dDrxw9eJ9kryRfbiPE5SbZry1/a9uvKJGcORoOTLEryziQXA/+MJEmSJN0D5lII3QW4bMK6PwD2AHYHDgZOHoQu4Ml0o55PAh4LPK2q3gJ8Hzioqg5q5bYALq6q3avqy8CpVbV3Ve0CbAYc0sp9EHhbVe0OPBX4AXAccEFV7VFVp4z0befRflfVz4H/AR7XFu0BHA7sChye5JFJHgL8DXBwVe0JLAH+bKSe64AVSfZoi14EnJFkY+CtwGFtxPh0Vo28fqLt1+7A14CXDFX5COCpVXW3dgCSHJVkSZIlK29bMbpakiRJkmZkLoXQqTwd+HBVrayqHwFfBPZu6y6pqu9V1V3AUrqps+OsBM4cen1QkouTLKMbzdw5yVbADlX1SYCqur2qblsP/T+3qlZU1e3AfwGPBv4XXXD+SpKlwAvb8lHvAV7UphAfDnwIeAJdaP9C2/Zv6AImwC5JLmj79Ty6kDzwsTZSu5qqOq2qFlTVgo0233pd91eSJEnSfdRcOvfvGuCwtdjujqHnK5m8z7cPAliSTYG3Awuq6rtJTgA2XYu2oQuVd+t3kgcAjwK+Bew5oY8BvlBVR0xT/5nA64DzgMuq6idJtgeuqap9x5RfBBxaVVcmORI4cGjdrTPcJ0mSJElaK3NpJPQ8YJMkRw0WJNktyX7ABXTTWDdK8lBgf+CSaeq7GdhqwrpB4LyxnYd5GEBV3Qx8L8mhrf1N2jmVU9V1LrB5khe0bTYC3ggsmmYU9SLgaUke17bbYtyVgNvo6TnAO4Az2uKvAw9Nsm/bduMkgxHPrYAftCm7z5uifUmSJEla7+ZMCK2qAp4NHNxuc3INcCLwQ7qr5l4FXEkXVv+iqn44TZWnAZ8bXJhopK2bgHcDV9MFvEuHVj8feGWSq4ALgYe3tle2i/28Osn2ST4z0u8/SvJN4BvA7cBfT7O/NwBHAh9ubX0VeOKE4h8E7gI+37b9JV1w/qd2oaWldOevAvwtcDHwFeDaqfogSZIkSetbuoykuSzJ/wW2rqr/v727j7b0qusD/v2SCSIODUuiNEYkChGEEEISkQgiFFYKDIJKLFBYgGaJguJbSYkNalRsQpXWxasGxABViKEVo4hBgQhCUAYTExKL8hIpYBUqRMJLkMnuH+eJPYz3ztwJN8+Zk/l81rprzn2e/ez9O2f2ujPf2c9+5qfmGO9pZ50z3rDn+DmGYh+uOXfXqksAAIDNdLMT67QnlA20/e0kd8ni4UkAAAAHNSF0zY0xvmvVNQAAAGzV2uwJBQAAYP0JoQAAAMxGCAUAAGA2QigAAACzEUIBAACYjRAKAADAbIRQAAAAZiOEAgAAMBshFAAAgNkIoQAAAMxGCAUAAGA2QigAAACz2bHqAlg/9zr6iLzk6btWXQYAALCGrIQCAAAwGyEUAACA2QihAAAAzEYIBQAAYDZCKAAAALMRQgEAAJiNEAoAAMBshFAAAABmI4QCAAAwmx2rLoD1c+VHrs0xZ75+1WWslWvO3bXqEgAA4KBgJRQAAIDZCKEAAADMRggFAABgNkIoAAAAsxFCAQAAmI0QCgAAwGyEUAAAAGYjhAIAADAbIRQAAIDZCKEAAADMRggFAABgNkIoAAAAsxFCAQAAmI0QCgAAwGyEUAAAAGYjhAIAADCbgzqEtt3T9vK272l7Ydvb3szjXdP2yG3q6yltPzbVf1Xb195Yf9sfbPuk6fX5bU+bXl/S9uTp9e+3vf021TLaPm/p+2e2PXs7+gYAADgQB3UITfLZMcYJY4zjknw+yQ/eHIN04eb4LC6Y6r9nFvU/NknGGL8yxnjlvi4cYzxijPHJbarj+iTfvV0BGwAA4KY62EPosrcluWvb72j7p20va/tHbe+YJG3Pbvuqtpe2/eu233/jhW3PaPuutle0/dnp2DFt39v2lUnek+ROy4O1fWLbP5tWMn+17WHT1/nTyuyVbX98avsjba+e+n/N3oW33ZHkK5J8YqnWZ+7rzd64KjvV+ZdtXzqtqL6x7ZdPbb55GvPytr/Y9j2bdPeFJOcl+fENxjmm7Zunft7U9uv2VRcAAMCXYi1C6BTiHp7kyiR/kuR+Y4z7JHlNkv+41PT4JP8mySlJfrrt17Q9NcmxSe6b5IQkJ7V94NT+2CQvHmPcc4zxN0vjfVMWq5b3H2OckGRPkidM1x89xjhujHGvJL8+XXJmkvuMMY7PF6/WPrbt5Uk+kuQrk/zuTfwIjk3yomlF9ZNJHjMd//UkP7BU4768KMkT2h6x1/EXJHnFVPtvJHn+Rhe3fWrb3W137/nMtTfxbQAAAIe6gz2EfvkU4nYn+VCSX0vytUkubntlkjOS3HOp/e+MMT47xvh4krdkETxPnb4uS/LnSe6eRahLkr8ZY7xzg3EfkuSkJO+axn9Ikm9I8oEk39D2BW0fluQfp/ZXJPmNtk/MYtXxRhdMAfFfZxGgz7iJn8MHxxiXT6/fneSYab/o7cYYl07Hf3NfHYwx/jHJK5P8yF6nTlm69lVJHrDJ9eeNMU4eY5x82G33zrEAAABbc7CH0Bv3hJ4wxnjGGOPzWazcvXBaifyBJLdZaj/2un4kaZJzlvq56xjj16bzn95k3GaxOnjjNXcbY5w9xvhEknsnuSSLFc+XTe13ZbHSeGIWwXXHFxUxxshiFfSBuWmuX3q9J8mOzRruxy8nOT2LW4MBAABmd7CH0I0ckcXtrUny5L3OPbrtbdreIcmDkrwrycVJvq/tziRpe3Tbr97PGG9KctqN7dp+Zds7Tw/2udUY438keXaSE6cHGt1pjPGWJM+a6tu5QZ8PSPL+A3yvm5oeWvSptt8yHXrcFq75hyS/lUUQvdE7lq59QhZ7bwEAAG4WN3VFbZXOTnJh208keXOSr186d0UWt+EemeTnxxgfTfLRaY/npW2T5LokT8w+9lCOMa5u++wkb5xC5j8l+aEkn03y60tP0v3JJIcl+e/TXssmef4Y45PTWI9t+4Aswv6HkzzlS3/7X+T0JC9te0OSP05ybZK0/ZokLxtjPGKDa56X5IeXvn/G9J7OSPKxJN+7zTUCAAD8sy7uFF1/0/97ed0Y45dWXctc2u4cY1w3vT4zyVFjjB+9ucd92lnnjDfsOf7mHuYW5Zpzd626BAAAmFM3O7GOK6H8f7va/mQWv49/k+1faQUAANhWt5gQOsY4e9U1zG2McUGSC1ZdBwAAwFat44OJAAAAWFNCKAAAALMRQgEAAJiNEAoAAMBshFAAAABmI4QCAAAwGyEUAACA2QihAAAAzEYIBQAAYDZCKAAAALMRQgEAAJjNjlUXwPq519FH5CVP37XqMgAAgDVkJRQAAIDZCKEAAADMRggFAABgNkIoAAAAsxFCAQAAmI0QCgAAwGyEUAAAAGYjhAIAADAbIRQAAIDZ7Fh1AayfKz9ybY458/WrLmPLrjl316pLAAAAJlZCAQAAmI0QCgAAwGyEUAAAAGYjhAIAADAbIRQAAIDZCKEAAADMRggFAABgNkIoAAAAsxFCAQAAmI0QCgAAwGyEUAAAAGYjhAIAADAbIRQAAIDZCKEAAADMRggFAABgNkIoAAAAsxFCN9H2rLZXtb2i7eVtv+Um9HFy2+ffhOvu2/atbd/b9rK2L2t727aPanvm1OY7295j6Zqfa/vQAx0LAABgTjtWXcDBqO0pSR6Z5MQxxvVtj0xy6wPtZ4yxO8nuAxh3R5I7JLkwyePGGJdOx09LcrsxxkVJLpqaf2eS30ty9TTWTx9ofQAAAHOzErqxo5J8fIxxfZKMMT4+xvho25Pa/nHbd7e9uO1RSdL2krbPbftnbf+q7bdNxx/U9vem11/Z9nXTyuo72x4/HT+77avavj3Jq5L8UJJX3BhAp/FfO8b4u7ZPafvCtt+a5FFJfnFapb1L2/Pbnjatvl4+fV3Zdkzj3KXtH0y1v63t3afj57d9ftt3tP3AFHgBAABuFkLoxt6Y5E5ToHxx229ve3iSFyQ5bYxxUpKXJ/mFpWt2jDHum+THkvzMBn3+bJLLxhjHJ/lPSV65dO4eSR46xnh8kuOSvHtfxY0x3pHFiugZY4wTxhjvXzq3ezp2QpI/SPJL06nzkjxjqv2ZSV681OVRSR6QxervuRuN2fapbXe33b3nM9fuqzwAAIBNuR13A2OM69qelOTbkjw4yQVJnpNFQPzDtklyWJK/Xbrsf06/vjvJMRt0+4Akj5n6f3PbO7T9V9O5i8YYn93O99D2sUlOTHJq251JvjXJhVPtSfJlS81fN8a4IcnVbe+4UX9jjPOyCLJ52lnnjOzZzmoBAIBDhRC6iTHGniSXJLmk7ZVZ3CZ71RjjlE0uuX76dU8O/HP99NLrq5KclOR3DrCPf9b2uCRnJ3ngGGNP21sl+eS0OrqR65ded5M2AAAAXzK3426g7d3aHrt06IQkf5nkq6aHFqXt4W3veQDdvi3JE6ZrH5TFntN/3KDdC5M8eflpvG2/e4MVyk8lud0Gtd8+yauTPGmM8bEkmcb5YNvvmdq07b0PoHYAAIBtIYRubGeSV7S9uu0VWezZ/OkkpyV5btu/SHJ5Fre4btXZSU6a+js3yZM3ajTG+Lskj0vyS9N/0fKXSf5tFqFz2WuSnDH9Fy53WTr+6CR3TvLSGx9QNB1/QpLTp9qvmtoBAADMqmOMVdfAmnnaWeeMN+w5ftVlbNk15+5adQkAAHCo2XSbn5VQAAAAvGlAjwAACllJREFUZiOEAgAAMBshFAAAgNkIoQAAAMxGCAUAAGA2QigAAACzEUIBAACYjRAKAADAbIRQAAAAZiOEAgAAMBshFAAAgNkIoQAAAMxGCAUAAGA2QigAAACz2bHqAlg/9zr6iLzk6btWXQYAALCGrIQCAAAwGyEUAACA2QihAAAAzEYIBQAAYDZCKAAAALMRQgEAAJiNEAoAAMBshFAAAABmI4QCAAAwmx2rLoD1c+VHrs0xZ75+1WX8C9ecu2vVJQAAAPthJRQAAIDZCKEAAADMRggFAABgNkIoAAAAsxFCAQAAmI0QCgAAwGyEUAAAAGYjhAIAADAbIRQAAIDZCKEAAADMRggFAABgNkIoAAAAsxFCAQAAmI0QCgAAwGyEUAAAAGYjhG6Ttue3Pe0mXntM28+2vXzp69bbXePSeNfdXH0DAADsy45VF3CoartjjPGFpUPvH2OcsLKCAAAAZnBIroS2/am27237J21f3faZbe/S9g/avrvt29refWp7ftvnt31H2w/cuNrZhRdO/fxRkq9e6v+ktn889XVx26Om45e0/eW2u5P86BbqPLXtpW3/vO2FbXdOx69pe860Yrq77YnTOO9v+4NTm51t3zRde2XbR28yxhlt39X2irY/+6V+tgAAAPtyyIXQtt+c5DFJ7p3k4UlOnk6dl+QZY4yTkjwzyYuXLjsqyQOSPDLJudOx70pytyT3SPKkJN869X94khckOW3q6+VJfmGpr1uPMU4eYzxvr9LusnQr7ovaHpnk2UkeOsY4McnuJD+x1P5D08rp25Kcn+S0JPdLcmOQ/FyS75qufXCS57XtXp/FqUmOTXLfJCckOantAzf53J46Bd7dez5z7UZNAAAA9utQvB33/kl+Z4zxuSSfa/u7SW6TRYi8cCmnfdnSNa8bY9yQ5Oq2d5yOPTDJq8cYe5J8tO2bp+N3S3Jckj+c+josyd8u9XXBJnV90e24bR+ZRcB9+9TPrZNcutT+ounXK5PsHGN8Ksmn2l7f9vZJPp3kP0+h8oYkRye5Y5L/s9THqdPXZdP3O7MIpW/du7gxxnlZBPU87axzRvZs8i4AAAD24VAMoRu5VZJP7mNP5vVLr7tJm+XzV40xTtnk/Ke3WFOT/OEY4/H7qemGveq7IYvf1yck+aokJ40x/qntNVmE7b3HOGeM8atbrAkAAOBLcsjdjpvk7Um+o+1tpj2Wj0zymSQfbPs9yT/v97z3fvp5a5LHtj1s2vP54On4e5N8VdtTpr4Ob3vPm1DnO5Pcv+1dp36+ou03HsD1RyT5+ymAPjjJnTdoc3GS71vaa3p026/eoB0AAMC2OORC6BjjXVncynpFkjdkcTvrtVmsHJ7e9i+SXJVkwwf5LPntJH+d5Ookr8x0q+wY4/NZ7M987tTX5Zn2iy5re3Lbl+2jzo8leUqSV7e9Yur/7lt+o8lvJDm57ZVZ7Fn9XxuM8cYkv5nk0qnda5Pc7gDGAAAAOCAdY6y6htm13TnGuK7tbbNY0XzqGOPPV13XunjaWeeMN+w5ftVl/AvXnLtr1SUAAAALm25jPFT3hJ7X9h5Z7JF8hQAKAAAwj0MyhI4x/v2qawAAADgUHXJ7QgEAAFgdIRQAAIDZCKEAAADMRggFAABgNkIoAAAAsxFCAQAAmI0QCgAAwGyEUAAAAGYjhAIAADAbIRQAAIDZCKEAAADMRggFAABgNjtWXQDr515HH5GXPH3XqssAAADWkJVQAAAAZiOEAgAAMBshFAAAgNkIoQAAAMxGCAUAAGA2QigAAACzEUIBAACYjRAKAADAbIRQAAAAZiOEAgAAMBshFAAAgNkIoQAAAMxGCAUAAGA2QigAAACzEUIBAACYjRAKAADAbIRQAAAAZiOEAgAAMBshFAAAgNkIoQAAAMxGCAUAAGA2QigAAACzEUIBAACYjRAKAADAbIRQAAAAZiOEAgAAMBshFAAAgNkIoQAAAMxGCAUAAGA2HWOsugbWzLOe9axPHX744e9ddR3cclx33XVH7ty58+OrroNbDnOK7WQ+sd3MKbbbQTqnPv6c5zznYRudEEI5YG13jzFOXnUd3HKYU2w3c4rtZD6x3cwpttu6zSm34wIAADAbIRQAAIDZCKHcFOetugBuccwptps5xXYyn9hu5hTbba3mlD2hAAAAzMZKKAAAALMRQgEAAJiNEMqm2j6s7Xvbvq/tmRuc/7K2F0zn/7TtMfNXybrYwnz6ibZXt72i7Zva3nkVdbI+9jenlto9pu1ouzaPrmc1tjKn2v676WfVVW1/c+4aWS9b+LPv69q+pe1l059/j1hFnayHti9v+/dt37PJ+bZ9/jTfrmh74tw1bpUQyobaHpbkRUkenuQeSR7f9h57NTs9ySfGGHdN8t+SPHfeKlkXW5xPlyU5eYxxfJLXJvkv81bJOtninErb2yX50SR/Om+FrJutzKm2xyb5yST3H2PcM8mPzV4oa2OLP6eeneS3xhj3SfK4JC+et0rWzPlJHraP8w9Pcuz09dQkL5mhpptECGUz903yvjHGB8YYn0/ymiSP3qvNo5O8Ynr92iQPadsZa2R97Hc+jTHeMsb4zPTtO5N87cw1sl628jMqSX4+i38g+9ycxbGWtjKnvj/Ji8YYn0iSMcbfz1wj62Urc2ok+VfT6yOSfHTG+lgzY4y3JvmHfTR5dJJXjoV3Jrl926Pmqe7ACKFs5ugk/3vp+w9PxzZsM8b4QpJrk9xhlupYN1uZT8tOT/KGm7Ui1t1+59R0G9Kdxhivn7Mw1tZWfk59Y5JvbPv2tu9su68VCdjKnDo7yRPbfjjJ7yd5xjylcQt1oH/fWpkdqy4AYFnbJyY5Ocm3r7oW1lfbWyX5r0mesuJSuGXZkcVtbg/K4m6Nt7a91xjjkyutinX2+CTnjzGe1/aUJK9qe9wY44ZVFwY3JyuhbOYjSe609P3XTsc2bNN2Rxa3kfzfWapj3WxlPqXtQ5OcleRRY4zrZ6qN9bS/OXW7JMcluaTtNUnul+QiDydiH7byc+rDSS4aY/zTGOODSf4qi1AKG9nKnDo9yW8lyRjj0iS3SXLkLNVxS7Slv28dDIRQNvOuJMe2/fq2t85is/xFe7W5KMmTp9enJXnzGGPMWCPrY7/zqe19kvxqFgHUPiv2Z59zaoxx7RjjyDHGMWOMY7LYZ/yoMcbu1ZTLGtjKn3uvy2IVNG2PzOL23A/MWSRrZStz6kNJHpIkbb8pixD6sVmr5JbkoiRPmp6Se78k144x/nbVRW3E7bhsaIzxhbY/nOTiJIclefkY46q2P5dk9xjjoiS/lsVtI+/LYpP041ZXMQezLc6nX0yyM8mF0/OtPjTGeNTKiuagtsU5BVu2xTl1cZJT216dZE+SM8YY7gBiQ1ucU/8hyUvb/ngWDyl6in/QZzNtX53FP4QdOe0j/pkkhyfJGONXsthX/Igk70vymSTfu5pK96/mOQAAAHNxOy4AAACzEUIBAACYjRAKAADAbIRQAAAAZiOEAgAAMBshFAAAgNkIoQAAAMzm/wGfyre6E0RdzgAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 1008x720 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CNYezHMfX9x_"
      },
      "source": [
        ""
      ],
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "A_ij6RxbX9um"
      },
      "source": [
        ""
      ],
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "75w5AMbnX9sE"
      },
      "source": [
        ""
      ],
      "execution_count": 19,
      "outputs": []
    }
  ]
}