examples/03_evaluate/als_movielens_diversity_metrics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i>Copyright (c) Microsoft Corporation. All rights reserved.</i>\n",
    "\n",
    "<i>Licensed under the MIT License.</i>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Apply Diversity Metrics  \n",
    "## -- Compare ALS and Random Recommenders on MovieLens (PySpark)\n",
    "\n",
    "We demonstrate how to evaluate a recommender using diversity metrics in addition to commonly used rating/ranking metrics.\n",
    "\n",
    "Diversity metrics include:\n",
    "- Coverage - The proportion of items that can be recommended. It includes two metrics: \n",
    "    - (1) catalog_coverage, which measures the proportion of items that get recommended from the item catalog; \n",
    "    - (2) distributional_coverage, which measures how unequally different items are recommended in the recommendations to all users.\n",
    "- Novelty - A more novel item indicates it is less popular, i.e., it gets recommended less frequently.\n",
    "- Diversity - The dissimilarity of items being recommended.\n",
    "- Serendipity - The \"unusualness\" or \"surprise\" of recommendations to a user.\n",
    "\n",
    "We compare the performance of two algorithms: ALS recommender and a random recommender. \n",
    " - Matrix factorization by [ALS](https://spark.apache.org/docs/latest/api/python/_modules/pyspark/ml/recommendation.html#ALS) (Alternating Least Squares) is a well known collaborative filtering algorithm.\n",
    " - We also define a random recommender which randomly recommends unseen items to each user. \n",
    " \n",
    "The comparision results show that ALS recommender outperforms random recommender on ranking metrics (Precision@k, Recall@k, NDCG@k, and\tMean average precision), while random recommender outperforms ALS recommender on diversity metrics. Why ALS performs better on ranking metrics while worse on diversity metrics? ALS is optimized for estimating the item rating as accurate as possible, therefore it performs well on accuracy metrics including precision, recall, etc. Ranking metrics are built upoin these accuracy metrics. As a side effect, the items being recommended tend to be popular items, which are the items mostly sold or viewed. It leaves the long-tail less popular items having less chance to get introduced to the users. This is the reason why ALS is not as well performing as a random recommender on diversity metrics. \n",
    "\n",
    "We understand that there is usually a trade-off between one metric and the other. We should decide which set of metrics to optimize based on business scenarios."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: This notebook requires a PySpark environment to run properly. Please follow the steps in [SETUP.md](https://github.com/Microsoft/Recommenders/blob/master/SETUP.md#dependencies-setup) to install the PySpark environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "System version: 3.6.11 | packaged by conda-forge | (default, Nov 27 2020, 18:57:37) \n",
      "[GCC 9.3.0]\n",
      "Spark version: 2.4.5\n"
     ]
    }
   ],
   "source": [
    "# set the environment path to find Recommenders\n",
    "import sys\n",
    "import pyspark\n",
    "from pyspark.ml.recommendation import ALS\n",
    "import pyspark.sql.functions as F\n",
    "from pyspark.sql import SparkSession\n",
    "from pyspark.sql.types import StructType, StructField\n",
    "from pyspark.sql.types import StringType, FloatType, IntegerType, LongType\n",
    "\n",
    "from reco_utils.common.timer import Timer\n",
    "from reco_utils.dataset import movielens\n",
    "from reco_utils.common.notebook_utils import is_jupyter\n",
    "from reco_utils.dataset.spark_splitters import spark_random_split\n",
    "from reco_utils.evaluation.spark_evaluation import SparkRatingEvaluation, SparkRankingEvaluation\n",
    "from reco_utils.common.spark_utils import start_or_get_spark\n",
    "\n",
    "from reco_utils.evaluation.spark_diversity_evaluation import DiversityEvaluation\n",
    "from pyspark.sql.window import Window\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "print(\"System version: {}\".format(sys.version))\n",
    "print(\"Spark version: {}\".format(pyspark.__version__))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Set the default parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "# top k items to recommend\n",
    "TOP_K = 10\n",
    "\n",
    "# Select MovieLens data size: 100k, 1m, 10m, or 20m\n",
    "MOVIELENS_DATA_SIZE = '100k'\n",
    "\n",
    "# user, item column names\n",
    "COL_USER=\"UserId\"\n",
    "COL_ITEM=\"MovieId\"\n",
    "COL_RATING=\"Rating\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set up Spark context\n",
    "\n",
    "The following settings work well for debugging locally on VM - change when running on a cluster. We set up a giant single executor with many threads and specify memory cap. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the following settings work well for debugging locally on VM - change when running on a cluster\n",
    "# set up a giant single executor with many threads and specify memory cap\n",
    "\n",
    "spark = start_or_get_spark(\"ALS PySpark\", memory=\"16g\")\n",
    "\n",
    "spark.conf.set(\"spark.sql.crossJoin.enabled\", \"true\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download the MovieLens dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4.81k/4.81k [00:00<00:00, 17.5kKB/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+-------+------+---------+\n",
      "|UserId|MovieId|Rating|Timestamp|\n",
      "+------+-------+------+---------+\n",
      "|   196|    242|   3.0|881250949|\n",
      "|   186|    302|   3.0|891717742|\n",
      "|    22|    377|   1.0|878887116|\n",
      "|   244|     51|   2.0|880606923|\n",
      "|   166|    346|   1.0|886397596|\n",
      "|   298|    474|   4.0|884182806|\n",
      "|   115|    265|   2.0|881171488|\n",
      "|   253|    465|   5.0|891628467|\n",
      "|   305|    451|   3.0|886324817|\n",
      "|     6|     86|   3.0|883603013|\n",
      "|    62|    257|   2.0|879372434|\n",
      "|   286|   1014|   5.0|879781125|\n",
      "|   200|    222|   5.0|876042340|\n",
      "|   210|     40|   3.0|891035994|\n",
      "|   224|     29|   3.0|888104457|\n",
      "|   303|    785|   3.0|879485318|\n",
      "|   122|    387|   5.0|879270459|\n",
      "|   194|    274|   2.0|879539794|\n",
      "|   291|   1042|   4.0|874834944|\n",
      "|   234|   1184|   2.0|892079237|\n",
      "+------+-------+------+---------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Note: The DataFrame-based API for ALS currently only supports integers for user and item ids.\n",
    "schema = StructType(\n",
    "    (\n",
    "        StructField(COL_USER, IntegerType()),\n",
    "        StructField(COL_ITEM, IntegerType()),\n",
    "        StructField(COL_RATING, FloatType()),\n",
    "        StructField(\"Timestamp\", LongType()),\n",
    "    )\n",
    ")\n",
    "\n",
    "data = movielens.load_spark_df(spark, size=MOVIELENS_DATA_SIZE, schema=schema)\n",
    "data.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split the data using the Spark random splitter provided in utilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "N train 75193\n",
      "N test 24807\n"
     ]
    }
   ],
   "source": [
    "train, test = spark_random_split(data, ratio=0.75, seed=123)\n",
    "print (\"N train\", train.cache().count())\n",
    "print (\"N test\", test.cache().count())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get all possible user-item pairs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: We have the assumption that training data contains all users and all catalog items. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "users = train.select(COL_USER).distinct()\n",
    "items = train.select(COL_ITEM).distinct()\n",
    "user_item = users.crossJoin(items)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train the ALS model on the training data, and get the top-k recommendations for our testing data\n",
    "\n",
    "To predict movie ratings, we use the rating data in the training set as users' explicit feedback. The hyperparameters used in building the model are referenced from [here](http://mymedialite.net/examples/datasets.html). We do not constrain the latent factors (`nonnegative = False`) in order to allow for both positive and negative preferences towards movies.\n",
    "Timing will vary depending on the machine being used to train."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "header = {\n",
    "    \"userCol\": COL_USER,\n",
    "    \"itemCol\": COL_ITEM,\n",
    "    \"ratingCol\": COL_RATING,\n",
    "}\n",
    "\n",
    "\n",
    "als = ALS(\n",
    "    rank=10,\n",
    "    maxIter=15,\n",
    "    implicitPrefs=False,\n",
    "    regParam=0.05,\n",
    "    coldStartStrategy='drop',\n",
    "    nonnegative=False,\n",
    "    seed=42,\n",
    "    **header\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took 3.2652852770006575 seconds for training.\n"
     ]
    }
   ],
   "source": [
    "with Timer() as train_time:\n",
    "    model = als.fit(train)\n",
    "\n",
    "print(\"Took {} seconds for training.\".format(train_time.interval))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the movie recommendation use case, recommending movies that have been rated by the users does not make sense. Therefore, the rated movies are removed from the recommended items.\n",
    "\n",
    "In order to achieve this, we recommend all movies to all users, and then remove the user-movie pairs that exist in the training dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1477928\n",
      "9430\n"
     ]
    }
   ],
   "source": [
    "# Score all user-item pairs\n",
    "dfs_pred = model.transform(user_item)\n",
    "\n",
    "# Remove seen items.\n",
    "dfs_pred_exclude_train = dfs_pred.alias(\"pred\").join(\n",
    "    train.alias(\"train\"),\n",
    "    (dfs_pred[COL_USER] == train[COL_USER]) & (dfs_pred[COL_ITEM] == train[COL_ITEM]),\n",
    "    how='outer'\n",
    ")\n",
    "\n",
    "top_all = dfs_pred_exclude_train.filter(dfs_pred_exclude_train[\"train.Rating\"].isNull()) \\\n",
    "    .select('pred.' + COL_USER, 'pred.' + COL_ITEM, 'pred.' + \"prediction\")\n",
    "\n",
    "print(top_all.count())\n",
    "    \n",
    "window = Window.partitionBy(COL_USER).orderBy(F.col(\"prediction\").desc())    \n",
    "top_k_reco = top_all.select(\"*\", F.row_number().over(window).alias(\"rank\")).filter(F.col(\"rank\") <= 10).drop(\"rank\")\n",
    " \n",
    "print(top_k_reco.count())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Random Recommender\n",
    "\n",
    "We define a random recommender which randomly recommends unseen items to each user. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = train.select(COL_USER, COL_ITEM, COL_RATING)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# random recommender\n",
    "window = Window.partitionBy(COL_USER).orderBy(F.rand())\n",
    "\n",
    "# randomly generated recommendations for each user\n",
    "pred_df = (\n",
    "  train_df\n",
    "  # join training data with all possible user-item pairs (seen in training)\n",
    "  .join(user_item,\n",
    "        on=[COL_USER, COL_ITEM],\n",
    "        how=\"right\"\n",
    "  )\n",
    "  # get user-item pairs that were not seen in the training data\n",
    "  .filter(F.col(COL_RATING).isNull())\n",
    "  # count items for each user (randomly sorting them)\n",
    "  .withColumn(\"score\", F.row_number().over(window))\n",
    "  # get the top k items per user\n",
    "  .filter(F.col(\"score\") <= TOP_K)\n",
    "  .drop(COL_RATING)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. ALS vs Random Recommenders Performance Comparison"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_ranking_results(ranking_eval):\n",
    "    metrics = {\n",
    "        \"Precision@k\": ranking_eval.precision_at_k(),\n",
    "        \"Recall@k\": ranking_eval.recall_at_k(),\n",
    "        \"NDCG@k\": ranking_eval.ndcg_at_k(),\n",
    "        \"Mean average precision\": ranking_eval.map_at_k()\n",
    "      \n",
    "    }\n",
    "    return metrics   \n",
    "\n",
    "def get_diversity_results(diversity_eval):\n",
    "    metrics = {\n",
    "        \"catalog_coverage\":diversity_eval.catalog_coverage(),\n",
    "        \"distributional_coverage\":diversity_eval.distributional_coverage(), \n",
    "        \"novelty\": diversity_eval.novelty().first()[0], \n",
    "        \"diversity\": diversity_eval.diversity().first()[0], \n",
    "        \"serendipity\": diversity_eval.serendipity().first()[0]\n",
    "    }\n",
    "    return metrics "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_summary(data, algo, k, ranking_metrics, diversity_metrics):\n",
    "    summary = {\"Data\": data, \"Algo\": algo, \"K\": k}\n",
    "\n",
    "    if ranking_metrics is None:\n",
    "        ranking_metrics = {           \n",
    "            \"Precision@k\": np.nan,\n",
    "            \"Recall@k\": np.nan,            \n",
    "            \"nDCG@k\": np.nan,\n",
    "            \"MAP\": np.nan,\n",
    "        }\n",
    "    summary.update(ranking_metrics)\n",
    "    summary.update(diversity_metrics)\n",
    "    return summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ALS Recommender Performance Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "als_ranking_eval = SparkRankingEvaluation(\n",
    "    test, \n",
    "    top_all, \n",
    "    k = TOP_K, \n",
    "    col_user=\"UserId\", \n",
    "    col_item=\"MovieId\",\n",
    "    col_rating=\"Rating\", \n",
    "    col_prediction=\"prediction\",\n",
    "    relevancy_method=\"top_k\"\n",
    ")\n",
    "\n",
    "als_ranking_metrics = get_ranking_results(als_ranking_eval)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "als_diversity_eval = DiversityEvaluation(\n",
    "    train_df = train_df, \n",
    "    reco_df = top_k_reco,\n",
    "    col_user=\"UserId\", \n",
    "    col_item=\"MovieId\"\n",
    ")\n",
    "\n",
    "als_diversity_metrics = get_diversity_results(als_diversity_eval)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "als_results = generate_summary(MOVIELENS_DATA_SIZE, \"als\", TOP_K, als_ranking_metrics, als_diversity_metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Random Recommender Performance Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "random_ranking_eval = SparkRankingEvaluation(\n",
    "    test,\n",
    "    pred_df,\n",
    "    col_user=COL_USER,\n",
    "    col_item=COL_ITEM,\n",
    "    col_rating=COL_RATING,\n",
    "    col_prediction=\"score\",\n",
    "    k=TOP_K,\n",
    ")\n",
    "\n",
    "random_ranking_metrics = get_ranking_results(random_ranking_eval)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "random_diversity_eval = DiversityEvaluation(\n",
    "    train_df = train_df, \n",
    "    reco_df = pred_df, \n",
    "    col_user=COL_USER, \n",
    "    col_item=COL_ITEM\n",
    ")\n",
    "  \n",
    "random_diversity_metrics = get_diversity_results(random_diversity_eval)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "random_results = generate_summary(MOVIELENS_DATA_SIZE, \"random\", TOP_K, random_ranking_metrics, random_diversity_metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Result Comparison"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "cols = [\"Data\", \"Algo\", \"K\", \"Precision@k\", \"Recall@k\", \"NDCG@k\", \"Mean average precision\",\"catalog_coverage\", \"distributional_coverage\",\"novelty\", \"diversity\", \"serendipity\" ]\n",
    "df_results = pd.DataFrame(columns=cols)\n",
    "\n",
    "df_results.loc[1] = als_results \n",
    "df_results.loc[2] = random_results "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Data</th>\n",
       "      <th>Algo</th>\n",
       "      <th>K</th>\n",
       "      <th>Precision@k</th>\n",
       "      <th>Recall@k</th>\n",
       "      <th>NDCG@k</th>\n",
       "      <th>Mean average precision</th>\n",
       "      <th>catalog_coverage</th>\n",
       "      <th>distributional_coverage</th>\n",
       "      <th>novelty</th>\n",
       "      <th>diversity</th>\n",
       "      <th>serendipity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>100k</td>\n",
       "      <td>als</td>\n",
       "      <td>10</td>\n",
       "      <td>0.051911</td>\n",
       "      <td>0.017514</td>\n",
       "      <td>0.047460</td>\n",
       "      <td>0.005734</td>\n",
       "      <td>0.381906</td>\n",
       "      <td>0.009751</td>\n",
       "      <td>4.605485</td>\n",
       "      <td>0.892449</td>\n",
       "      <td>0.880236</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>100k</td>\n",
       "      <td>random</td>\n",
       "      <td>10</td>\n",
       "      <td>0.016773</td>\n",
       "      <td>0.006561</td>\n",
       "      <td>0.017131</td>\n",
       "      <td>0.002134</td>\n",
       "      <td>0.996964</td>\n",
       "      <td>0.012811</td>\n",
       "      <td>7.160845</td>\n",
       "      <td>0.923641</td>\n",
       "      <td>0.894481</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Data    Algo   K  Precision@k  Recall@k    NDCG@k  Mean average precision  \\\n",
       "1  100k     als  10     0.051911  0.017514  0.047460                0.005734   \n",
       "2  100k  random  10     0.016773  0.006561  0.017131                0.002134   \n",
       "\n",
       "   catalog_coverage  distributional_coverage   novelty  diversity  serendipity  \n",
       "1          0.381906                 0.009751  4.605485   0.892449     0.880236  \n",
       "2          0.996964                 0.012811  7.160845   0.923641     0.894481  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reference\n",
    "The metric definitions/formulations are based on following reference with modification:\n",
    "- G. Shani and A. Gunawardana, Evaluating Recommendation Systems, Recommender Systems Handbook pp. 257-297, 2010.\n",
    "- Y.C. Zhang, D.Ó. Séaghdha, D. Quercia and T. Jambor, Auralist: introducing serendipity into music recommendation, WSDM 2012\n",
    "- P. Castells, S. Vargas, and J. Wang, Novelty and diversity metrics for recommender systems: choice, discovery and relevance, ECIR 2011\n",
    "- Eugene Yan, Serendipity: Accuracy’s unpopular best friend in Recommender Systems, towards data science, April 2020\n",
    "- N. Hurley and M. Zhang, Novelty and diversity in top-n recommendation--analysis and evaluation, ACM Transactions, 2011"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# cleanup spark instance\n",
    "spark.stop()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (reco_pyspark)",
   "language": "python",
   "name": "reco_pyspark"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}