Text_analytics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 文本分析的基本概念\n",
    "\n",
    "近些年，随着算力和数据的逐渐积累，人们越来越多的关注文本，比如新闻、上市公司公告、国家机构公告等，这些文本中包含了大量的信息可以被挖掘。在经济学、金融学、会计学等领域，也有很多文献开始从文本数据中提取数据，与传统的统计、计量经济学等工具相结合，解决经济学中的问题。比如，Gentzkow, Kelly和Taddy（2017）就提供了一个使用文本作为数据的一个综述。使用文本分析的研究课题也越来越广，比如：\n",
    "\n",
    "* 政策不确定性：Baker, Blom和Davis（2016）；Benguria等（2022）\n",
    "* 政治/媒体倾向：Piotroski, Wong和Zhang（2017）；Gentzkow和Shapiro（2010）\n",
    "* 宏观经济：林建浩等（2021）\n",
    "* 公司金融、公司治理：罗进辉（2018）；汝毅、薛健和张乾（2019）；王雄元和高曦（2018）\n",
    "* 金融市场：阮睿等（2021）；Born, Ehrmann和Fratzscher（2014）\n",
    "* 股票价格、波动率\n",
    "\n",
    "而文本分析的方法也多种多样，比如：\n",
    "\n",
    "* 直接使用Baidu、Google的搜索指数，比如罗进辉（2018）\n",
    "* 从文本中找到相关词汇，比如山立威、甘犁和郑涛（2008）\n",
    "* 使用词频（或者TF-IDF），比如Piotroski, Wong和Zhang（2017）\n",
    "* 使用字符串长度，比如王雄元和高曦（2018）\n",
    "* 基于词典，比如阮睿等（2021）；Gentzkow和Shapiro（2010）;姚加权等（2021）\n",
    "* 情感分析，比如汝毅、薛健和张乾（2019）\n",
    "* 机器学，比如林建浩等（2021）\n",
    "\n",
    "与我们之前介绍的分析不同，文本数据是典型的非结构化数据，而且本身具有非常复杂的结构，相对之前的数据分析相比有其特有的分析技巧。这里我们将简单介绍文本分析的基本流程以及一些基础的工具。\n",
    "\n",
    "虽然差别很大，但是基本流程与之前的分析还是想通的，一般都需要如下步骤：\n",
    "\n",
    "1. 准备数据，在这一步除了要准备需要进行分析的数据之外，可能还需要准备额外的语料库（corpus）。\n",
    "2. 文本规范化处理，也就是我们之前清洗数据的步骤，比如分词、去除停用词、去除特殊符号等无意义字符、同义词转换、缩写转换等等。\n",
    "3. 特征工程，从已经清洗好的数据中提取特征。由于计算机只能处理数值型的变量，因而在这一步有一个比较关键的步骤是将文字转换为计算机可以理解的向量等数值型变量。\n",
    "4. 训练模型，经过这些步骤后，针对不同的目的，模型训练可能与之前的算法比较类似，但是也有针对文本数据特有的模型。\n",
    "5. 模型评价，评价模型的性能，重复以上步骤，改进模型。这一步有时涉及到人工评价模型，比如使用词典的方法就非常需要人口去查验效果。这一步非常关键，比如Baker, Blom和Davis（2016）就提供了人工查验数据的一个很好范例。\n",
    "\n",
    "针对文本数据，除了其他数据同样可以进行的相关性计算、聚类、分类等模型之外，还有一些任务是文本数据特有的，比如：\n",
    "\n",
    "* 分词\n",
    "* 词性标注\n",
    "* 词嵌入\n",
    "* 摘要和主题建模\n",
    "* 实体识别\n",
    "* 知识图谱\n",
    "* 语义分析\n",
    "* ......\n",
    "\n",
    "其中有的模型结果是其他模型的基础，比如分词、词性标注等是很多其他模型的基础。\n",
    "\n",
    "在本节，我们将主要使用Python中的**NLTK**（http://www.nltk.org ）、**Scikit-Learn**、**Jieba**（https://github.com/fxsjy/jieba ）、**Gensim**（https://github.com/RaRe-Technologies/gensim ）等工具介绍文本分析的基本原理。不过与此同时，自然语言处理，包括中文的自然语言处理正在蓬勃发展，很多新的工具可以使用，比如对标NLTK并且号称有更好性能的**spaCy**（ https://spacy.io ），以及已经经过预训练可以直接拿来用的模型比如最近如火如荼的**BERT**、**HanLP**（https://github.com/hankcs/HanLP ）、**Stanford CoreNLP**（https://github.com/stanfordnlp/CoreNLP ）等等，学习基本原理后可以直接使用这些包进行自己的研究。\n",
    "\n",
    "在这里我们从文本的规范化处理开始，介绍文本分析的基本原理和方法。\n",
    "\n",
    "此外，在这里可以提供一些文本分析、自然语言处理的比较好的学习资料：\n",
    "\n",
    "1. Anand, V., Bochkay, K., Chychyla, R., & Leone, A. J. (2020). Using Python for text analysis in accounting research. Vic Anand, Khrystyna Bochkay, Roman Chychyla and Andrew Leone (2020),\" Using Python for Text Analysis in Accounting Research\", Foundations and Trends®in Accounting, 14(3–4), 128–359.\n",
    "2. Sarkar, D. (2016). Text Analytics with python. Springer. （中文版：《Python文本分析》）\n",
    "\n",
    "**参考文献**\n",
    "1. Anand, V., Bochkay, K., Chychyla, R., & Leone, A. J. (2020). Using Python for text analysis in accounting research (Vol. 14, Issues 3–4).\n",
    "2. Baker, S. R., Bloom, N., & Davis, S. J. (2016). Measuring economic policy uncertainty. The Quarterly Journal of Economics, 131(4), 1593–1636.\n",
    "3. Benguria, F., Choi, J., Swenson, D. L., & Xu, M. J. (2022). Anxiety or pain? The impact of tariffs and uncertainty on Chinese firms in the trade war. Journal of International Economics, 103608.\n",
    "4. Born, B., Ehrmann, M., & Fratzscher, M. (2014). Central bank communication on financial stability. The Economic Journal, 124(577), 701–734.\n",
    "5. Gentzkow, M., Kelly, B., & Taddy, M. (2019). Text as data. Journal of Economic Literature, 57(3), 535–574.\n",
    "6. Gentzkow, M., & Shapiro, J. M. (2010). What drives media slant? Evidence from US daily newspapers. Econometrica, 78(1), 35–71.\n",
    "7. Piotroski, J. D., Wong, T. J., & Zhang, T. (2017). Political bias in corporate news: The role of conglomeration reform in China. The Journal of Law and Economics, 60(1), 173–207.\n",
    "8. 林建浩,陈良源,罗子豪,张一帆.央行沟通有助于改善宏观经济预测吗?——基于文本数据的高维稀疏建模[J].经济研究,2021,56(03):48-64.\n",
    "9. 罗进辉. 媒体报道与高管薪酬契约有效性[J]. 金融研究, 2018, 453(3): 190-206.\n",
    "10. 阮睿, 孙宇辰, 唐悦, 等. 资本市场开放能否提高企业信息披露质量?——基于 “沪港通” 和年报文本挖掘的分析[J]. 金融研究, 2021, 488(2): 188-206.\n",
    "11. 汝毅, 薛健, 张乾. 媒体新闻报道的声誉溢出效应[J]. 金融研究, 2019, 470(8): 189-206.\n",
    "12. 山立威, 甘犁, 郑涛. 公司捐款与经济动机[J]. 经济研究, 2008, 11: 51-60.\n",
    "13. 王雄元, 高曦. 年报风险披露与权益资本成本[J]. 金融研究, 2018, 451(1): 174-190.\n",
    "14. 姚加权, 冯绪, 王赞钧, 纪荣嵘, & 张维. (2021). 题名: 语调、情绪及市场影响:基于金融情绪词典. 管理科学学报, 24(5), 26–46.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 文本规范化\n",
    "\n",
    "文本是典型的非结构化数据，我们需要将文本转换为高度结构化的数据，首先要对文本进行有意义的划分，一般涉及到句子的**切分**（**tokenization**）以及其他清洗步骤。\n",
    "\n",
    "**标识**（**token**）是文本的有意义的最小成分，文本处理的最简单操作即将文本切成一个个的token，通常包括句子切分和词语切分。接下来我们很少有研究句子的成分和语义，更多时候是针对词语的分析，因而接下来主要介绍句子的切分方法。\n",
    "\n",
    "## 英文切分\n",
    "\n",
    "英文的词语切分一般比较简单，主要原因是因为英文的单词之间都有空格进行分割，而中文的切分就复杂很多。常见的自然语言处理包比如NKTL以及spaCy都肯定包含了切分的函数，比如在NLTK中："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'I am working very hard to help New York City & State. Dealing with both Mayor & Governor and producing tremendously for them, including four new medical centers and four new hospitals. Fake News that I won’t help them because I don’t like Cuomo (I do). Just sent 4000 ventilators!'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence = \"I am working very hard to help New York City & State. Dealing with both Mayor & Governor and producing tremendously for them, including four new medical centers and four new hospitals. Fake News that I won’t help them because I don’t like Cuomo (I do). Just sent 4000 ventilators!\"\n",
    "sentence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "from nltk import word_tokenize\n",
    "\n",
    "words = word_tokenize(sentence)\n",
    "print(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "注意运行以上命令可能先要下载相应的包：在Python解释器中运行：nltk.download('punkt') ，如果提示错误，可以参考：https://www.cnblogs.com/sddai/p/10543359.html\n",
    "\n",
    "当然NLTK不止支持这一种切分方法，比如我们可以使用正则表达式切分："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['I', 'am', 'working', 'very', 'hard', 'to', 'help', 'New', 'York', 'City', 'State', 'Dealing', 'with', 'both', 'Mayor', 'Governor', 'and', 'producing', 'tremendously', 'for', 'them', 'including', 'four', 'new', 'medical', 'centers', 'and', 'four', 'new', 'hospitals', 'Fake', 'News', 'that', 'I', 'won’t', 'help', 'them', 'because', 'I', 'don’t', 'like', 'Cuomo', 'I', 'do', 'Just', 'sent', '4000', 'ventilators']\n"
     ]
    }
   ],
   "source": [
    "from nltk import RegexpTokenizer\n",
    "\n",
    "Tokenizer = RegexpTokenizer(pattern=r\"[\\w\\-’']+\")\n",
    "words = Tokenizer.tokenize(sentence)\n",
    "print(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此外还可以使用空白字符（空格、缩进、换行）等进行切分："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['I', 'am', 'working', 'very', 'hard', 'to', 'help', 'New', 'York', 'City', '&', 'State.', 'Dealing', 'with', 'both', 'Mayor', '&', 'Governor', 'and', 'producing', 'tremendously', 'for', 'them,', 'including', 'four', 'new', 'medical', 'centers', 'and', 'four', 'new', 'hospitals.', 'Fake', 'News', 'that', 'I', 'won’t', 'help', 'them', 'because', 'I', 'don’t', 'like', 'Cuomo', '(I', 'do).', 'Just', 'sent', '4000', 'ventilators!']\n"
     ]
    }
   ],
   "source": [
    "from nltk import WhitespaceTokenizer\n",
    "\n",
    "Tokenizer = WhitespaceTokenizer()\n",
    "words = Tokenizer.tokenize(sentence)\n",
    "print(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "英文分词虽然原理简单，但是还是有很多细节的坑，比如上面Trump先生的「don’t」和「don't」，如果在切分或者其他清洗步骤中予以重视，计算机会认为这是两个词。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 中文分词\n",
    "\n",
    "中文文本分析与英文的文本分析一个最重要的区别在于，英文使用空格分割每个单词，但是中文没有分割单词的概念。\n",
    "\n",
    "为了克服这个问题，分词就应运而生了。结合字典和算法，分词软件可以帮助我们将中文的文章、句子分解为一个个的中文单词。\n",
    "\n",
    "目前已经有很多成熟的分词工具，比如中科院的NLPIR汉语分词系统、结巴分词以及腾讯、阿里、百度的分词系统等等。在这里我们以开源的结巴分词为例，介绍分词工具的用法。\n",
    "\n",
    "为了使用结巴分词，首先需要安装。在terminal中输入：\n",
    "```shell\n",
    "pip install jieba\n",
    "```\n",
    "\n",
    "就可以进行安装了。安装好之后，可以将jieba模块导入到Python程序中，就可以正常使用了：\n",
    "```shell\n",
    "import jieba\n",
    "```\n",
    "\n",
    "比如，最简单的用法："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Building prefix dict from the default dictionary ...\n",
      "Loading model from cache /tmp/jieba.cache\n",
      "Loading model cost 0.298 seconds.\n",
      "Prefix dict has been built successfully.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['今年以来', '，', '我国', '持续', '推进', '减税', '降费', '、', '提高', '最低工资', '标准', '、', '促进', '就业', '，', '特别', '是', '年初', '开始', '实施', '的', '个人所得税', '改革', '以及', '专项', '附加', '扣除', '方案', '，', '有效', '增加', '了', '居民', '可', '支配', '收入', '。', '与此同时', '，', '不断', '消除', '居民消费', '的', '后顾之忧', '。', '消费', '需求', '进一步', '释放', '，', '消费市场', '亮点', '纷呈']\n"
     ]
    }
   ],
   "source": [
    "import jieba\n",
    "\n",
    "line = \"今年以来，我国持续推进减税降费、提高最低工资标准、促进就业，特别是年初开始实施的个人所得税改革以及专项附加扣除方案，有效增加了居民可支配收入。与此同时，不断消除居民消费的后顾之忧。消费需求进一步释放，消费市场亮点纷呈\"\n",
    "wlist = jieba.cut(line)\n",
    "print(list(wlist))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "cut()函数有三个参数：必须要提供的是需要进行分词的字符串；此外，cut_all参数控制是否采用全模式；HMM参数用来控制是否使用HMM模型。其区别是：\n",
    "\n",
    "* cut_all=True， 代表使用全模式，全模式可以切出混合不同粒度的词\n",
    "* HMM=True，代表使用HMM模型，用于推断字典中没有的词\n",
    "\n",
    "比如："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['今年', '今年以来', '以来', '，', '我国', '持续', '推进', '减税', '降', '费', '、', '提高', '最低', '最低工资', '低工资', '工资', '工资标准', '标准', '、', '促进', '就业', '，', '特别', '是', '年初', '开始', '实施', '的', '个人', '个人所得', '个人所得税', '所得', '所得税', '税改', '改革', '以及', '专项', '附加', '扣除', '方案', '，', '有效', '增加', '了', '居民', '可支配', '支配', '收入', '。', '与此', '与此同时', '同时', '，', '不断', '消除', '居民', '居民消费', '消费', '的', '后顾之忧', '。', '消费', '需求', '求进', '进一步', '一步', '释放', '，', '消费', '消费市场', '市场', '亮点', '纷呈']\n"
     ]
    }
   ],
   "source": [
    "wlist = jieba.cut(line, cut_all=True)\n",
    "print(list(wlist))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['今年以来', '，', '我国', '持续', '推进', '减税', '降费', '、', '提高', '最低工资', '标准', '、', '促进', '就业', '，', '特别', '是', '年初', '开始', '实施', '的', '个人所得税', '改革', '以及', '专项', '附加', '扣除', '方案', '，', '有效', '增加', '了', '居民', '可', '支配', '收入', '。', '与此同时', '，', '不断', '消除', '居民消费', '的', '后顾之忧', '。', '消费', '需求', '进一步', '释放', '，', '消费市场', '亮点', '纷呈']\n"
     ]
    }
   ],
   "source": [
    "wlist = jieba.cut(line, HMM=True)\n",
    "print(list(wlist))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['今年', '今年以来', '以来', '，', '我国', '持续', '推进', '减税', '降', '费', '、', '提高', '最低', '最低工资', '低工资', '工资', '工资标准', '标准', '、', '促进', '就业', '，', '特别', '是', '年初', '开始', '实施', '的', '个人', '个人所得', '个人所得税', '所得', '所得税', '税改', '改革', '以及', '专项', '附加', '扣除', '方案', '，', '有效', '增加', '了', '居民', '可支配', '支配', '收入', '。', '与此', '与此同时', '同时', '，', '不断', '消除', '居民', '居民消费', '消费', '的', '后顾之忧', '。', '消费', '需求', '求进', '进一步', '一步', '释放', '，', '消费', '消费市场', '市场', '亮点', '纷呈']\n"
     ]
    }
   ],
   "source": [
    "wlist = jieba.cut(line, HMM=True, cut_all=True)\n",
    "print(list(wlist))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到，上面的分词结果......一言难尽。实际上，任何分词算法都不可避免的不能跟上时代的潮流，特别是网络时代，新的词语层出不穷，而在一些专业领域中，一些专有名词往往普通词典无法完全覆盖。比如“减费降税”、“可支配收入”这些专有名词，都没有被正确分出来。\n",
    "\n",
    "为了克服这个问题，往往需要用户自己添加字典。比如，我们可以把“减费降税”、“可支配收入”这些名词放在一个文本文件中，每个新词写成一行，然后使用load_userdict()函数给定这个文件，就可以添加自己的新词列表："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['今年以来', '，', '我国', '持续', '推进', '减税降费', '、', '提高', '最低工资标准', '、', '促进', '就业', '，', '特别', '是', '年初', '开始', '实施', '的', '个人所得税', '改革', '以及', '专项附加扣除', '方案', '，', '有效', '增加', '了', '居民', '可支配收入', '。', '与此同时', '，', '不断', '消除', '居民消费', '的', '后顾之忧', '。', '消费需求', '进一步', '释放', '，', '消费市场', '亮点', '纷呈']\n"
     ]
    }
   ],
   "source": [
    "with open('user_dict.txt', 'wt') as f:\n",
    "    f.write(\"减税降费\\n可支配收入\\n最低工资\\n最低工资标准\\n专项附加扣除\\n消费需求\")\n",
    "jieba.load_userdict('user_dict.txt')\n",
    "wlist = jieba.cut(line)\n",
    "print(list(wlist))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "载入了用户字典后，新的分词更加准确了。在实际应用的时候，无论使用什么分词工具，用户字典的构建往往是非常关键的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 大小写转换\n",
    "\n",
    "主要针对英文等字母文字，一般的做法是统一转换为小写字母，避免出现「FAKE NEWS$\\neq$fake news」的情况出现。可以使用字符串的.lower()方法很容易的完成大小写转换："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['i', 'am', 'working', 'very', 'hard', 'to', 'help', 'new', 'york', 'city', '&', 'state.', 'dealing', 'with', 'both', 'mayor', '&', 'governor', 'and', 'producing', 'tremendously', 'for', 'them,', 'including', 'four', 'new', 'medical', 'centers', 'and', 'four', 'new', 'hospitals.', 'fake', 'news', 'that', 'i', 'won’t', 'help', 'them', 'because', 'i', 'don’t', 'like', 'cuomo', '(i', 'do).', 'just', 'sent', '4000', 'ventilators!']\n"
     ]
    }
   ],
   "source": [
    "words = [w.lower() for w in words]\n",
    "print(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 删除停用词和特殊字符\n",
    "\n",
    "分词之后，在进一步进行处理之前，消除停用词和特殊符号往往是非常关键的。比如在上面的分词结果中，“的”、“是”、“与此同时”等，含义并不是非常明显，对其分析的意义不大，留着这些词只会空占内存，并且可能对分析结果产生巨大影响。一个常用的做法是，使用一个停用词列表，分词结束之后，把停用词列表中的词全都剔除出去。\n",
    "\n",
    "比如，在文本文档“Chinese/stopword.txt”中，我们列举出了一些常用的停用词以及特殊字符，我们可以使用如下方法消除停用词："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['今年以来', '，', '我国', '持续', '推进', '减税降费', '、', '提高', '最低工资标准', '、', '促进', '就业', '，', '特别', '是', '年初', '开始', '实施', '的', '个人所得税', '改革', '以及', '专项附加扣除', '方案', '，', '有效', '增加', '了', '居民', '可支配收入', '。', '与此同时', '，', '不断', '消除', '居民消费', '的', '后顾之忧', '。', '消费需求', '进一步', '释放', '，', '消费市场', '亮点', '纷呈']\n"
     ]
    }
   ],
   "source": [
    "with open('Chinese/stopword.txt', 'rt') as f:\n",
    "    stoplist = f.readlines()\n",
    "    stoplist = set([w.lower() for w in stoplist])\n",
    "wlist = jieba.cut(line)\n",
    "wlist = [w.lower() for w in wlist if w.lower() not in stoplist]\n",
    "print(wlist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "注意上面的函数里面我们还同时对停用词和词语转换为了小写。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 扩展缩写词、同义词转换\n",
    "\n",
    "缩写词即诸如：「isn't==is not」之类的缩写，一般而言也需要特殊处理。一般来说我们可以通过定义一个映射关系来处理这种情况，比如："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['今年以来', '，', '我国', '持续', '推进', '减税降费', '、', '提高', '最低工资标准', '、', '促进', '就业', '，', '特别', '是', '年初', '开始', '实施', '的', '个税', '改革', '以及', '专项附加扣除', '方案', '，', '有效', '增加', '了', '居民', '可支配收入', '。', '与此同时', '，', '不断', '消除', '居民消费', '的', '后顾之忧', '。', '消费需求', '进一步', '释放', '，', '消费市场', '亮点', '纷呈']\n"
     ]
    }
   ],
   "source": [
    "syno = {\n",
    "    \"isn't\": \"is not\",\n",
    "    \"aren't\": \"are not\",\n",
    "    \"i'll\": \"i will\",\n",
    "    \"个人所得税\": \"个税\"\n",
    "}\n",
    "wlist = [syno[w] if w in syno.keys() else w for w in wlist]\n",
    "print(wlist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此外，还有一些更加细致的工作，比如检查拼写错误、矫正重复字符等等，这些都需要大量细致的工作，特别是针对不同的应用场景进行特定的优化是非常有必要的。\n",
    "\n",
    "而对于英文，由于英文存在时态、单复数的问题，一个常见的操作是使用stemmer算法将其转化为词根，比如economics, economically, economic等都代表同一个词，所以可以使用相同的词根来表示。对于英文，比较常见的算法是所谓的Porter Stemmer，在NLTK中可以直接调用："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "econom\n",
      "econom\n",
      "econom\n"
     ]
    }
   ],
   "source": [
    "from nltk.stem import PorterStemmer\n",
    "\n",
    "porter = PorterStemmer()\n",
    "print(porter.stem('economically'))\n",
    "print(porter.stem('economics'))\n",
    "print(porter.stem('economic'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如此我们就有了三个词的共同词根：econom。使用stemmer可以极大的减少词的个数，从而达到降维的目的。比如对于以上英文，我们可以先分词、转化大小写再进行stem操作："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['i', 'am', 'work', 'veri', 'hard', 'to', 'help', 'new', 'york', 'citi', '&', 'state.', 'deal', 'with', 'both', 'mayor', '&', 'governor', 'and', 'produc', 'tremend', 'for', 'them,', 'includ', 'four', 'new', 'medic', 'center', 'and', 'four', 'new', 'hospitals.', 'fake', 'new', 'that', 'i', 'won’t', 'help', 'them', 'becaus', 'i', 'don’t', 'like', 'cuomo', '(i', 'do).', 'just', 'sent', '4000', 'ventilators!']\n"
     ]
    }
   ],
   "source": [
    "words = [porter.stem(w).lower() for w in Tokenizer.tokenize(sentence)]\n",
    "print(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "而对于中文，同样也可以进行同义词的转换步骤，如果需要，https://github.com/fighting41love/funNLP/tree/master/data 中提供了一个可用的近义词表。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 一个简单的例子：中文词频统计\n",
    "\n",
    "以下实现了对《越女剑》的词频统计："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>word</th>\n",
       "      <th>frequency</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>范蠡</td>\n",
       "      <td>118</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>道</td>\n",
       "      <td>113</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>剑士</td>\n",
       "      <td>105</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>青衣</td>\n",
       "      <td>47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>阿青</td>\n",
       "      <td>47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>勾践</td>\n",
       "      <td>44</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>锦衫</td>\n",
       "      <td>35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>长剑</td>\n",
       "      <td>34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>说道</td>\n",
       "      <td>31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>吴国</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  word  frequency\n",
       "0   范蠡        118\n",
       "1    道        113\n",
       "2   剑士        105\n",
       "3   青衣         47\n",
       "4   阿青         47\n",
       "5   勾践         44\n",
       "6   锦衫         35\n",
       "7   长剑         34\n",
       "8   说道         31\n",
       "9   吴国         30"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import jieba\n",
    "\n",
    "## 停用词\n",
    "with open('Chinese/stopword.txt', 'rt') as f:\n",
    "    stoplist = f.readlines()\n",
    "    stoplist = [w.lower().strip() for w in stoplist]\n",
    "## 读入小说\n",
    "wordlist = []\n",
    "with open('Chinese/越女剑.txt', 'rt') as f:\n",
    "    for l in f:\n",
    "        line_cut = jieba.cut(l)\n",
    "        line = [w.strip().lower() for w in line_cut]\n",
    "        wordlist.extend(line)\n",
    "wordlist = [w.lower() for w in wordlist if w.lower() not in stoplist]\n",
    "## 统计\n",
    "text_dict = dict()\n",
    "for l in wordlist:\n",
    "    if l not in text_dict:\n",
    "        text_dict[l] = 1\n",
    "    else:\n",
    "        text_dict[l] += 1\n",
    "text_freq = []\n",
    "for k in text_dict:\n",
    "    text_freq.append((k, text_dict[k]))\n",
    "## 排序\n",
    "text_freq.sort(key=lambda x: x[1], reverse=True)\n",
    "## 用pandas显示，更好看\n",
    "import pandas as pd\n",
    "\n",
    "freq = pd.DataFrame(text_freq, columns=['word', 'frequency'])\n",
    "freq.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 词典与词频的结合\n",
    "\n",
    "我们可以将词典与词频相结合，构造一些需要的指标。比如Benguria等（2022）就使用了类似的方法研究贸易政策不确定性。\n",
    "\n",
    "这里，作为示例，我们仿照Benguria等（2022）的方法，使用（部分）金融新闻的标题数据，构造一个宏观政策不确定性的指数。\n",
    "\n",
    "在Benguria等（2022），他们使用了两个不同的词典：\n",
    "* 贸易政策词典：国际贸易、出口、进口、关税、壁垒、反倾销、外包、保护主义、单边主义\n",
    "* 不确定性词典：不确定、不明确、不明朗、未明、（不明）、难料、难以估计、难以预计、难以预测、难以预料、风险、危险、危机、威胁、未知\n",
    "\n",
    "我们仿照他们的做法，不过选择一个与宏观经济政策有关的词典：\n",
    "* 宏观政策词典：政策、宏观、货币、财政、土地\n",
    "\n",
    "Benguria等（2022）使用了完整的报告，我们这里暂时只用标题。\n",
    "\n",
    "首先导入数据："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>date</th>\n",
       "      <th>title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>59021767</td>\n",
       "      <td>2011-02-19</td>\n",
       "      <td>沪争取增值税扩围改革试点</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>59021769</td>\n",
       "      <td>2011-02-19</td>\n",
       "      <td>周小川：外部施压不会影响人民币升值步伐</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>59021771</td>\n",
       "      <td>2011-02-19</td>\n",
       "      <td>准备金率再上调 达到19.5％创新高</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>59021774</td>\n",
       "      <td>2011-02-19</td>\n",
       "      <td>定基价格指数若涨20% 政府或出手调控</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>59021776</td>\n",
       "      <td>2011-02-19</td>\n",
       "      <td>政策倾斜和加大投入 西藏将做大做强藏药产业</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         id       date                  title\n",
       "0  59021767 2011-02-19           沪争取增值税扩围改革试点\n",
       "1  59021769 2011-02-19    周小川：外部施压不会影响人民币升值步伐\n",
       "2  59021771 2011-02-19     准备金率再上调 达到19.5％创新高\n",
       "4  59021774 2011-02-19    定基价格指数若涨20% 政府或出手调控\n",
       "5  59021776 2011-02-19  政策倾斜和加大投入 西藏将做大做强藏药产业"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "StockNews = pd.read_csv(\"csv/stocknews1.csv\")\n",
    "StockNews['date'] = pd.to_datetime(StockNews['date'])\n",
    "jianbao = StockNews['title'].str.match(\n",
    "    r'(.+\\d{4}年\\d{2}月\\d{2}日.+简报|.*二级市场.*简报|.*业绩公报.*|.*登记日期.*|.*交易安排表.*)')\n",
    "StockNews = StockNews[~jianbao]\n",
    "StockNews.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来定义模式，如果在句子中同时存在不确定性词典的词，以及宏观政策词典的词，那么就认为这是一条关于宏观政策不确定新的标题，具体做法可以使用正则表达式："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(.*不确定.*|.*不明确.*|.*不明朗.*|.*未明.*|.*不明.*|.*难料.*|.*难以估计.*|.*难以预计.*|.*难以预测.*|.*难以预料.*|.*风险.*|.*危险.*|.*危机.*|.*威胁.*|.*未知.*)\n",
      "(.*政策.*|.*宏观.*|.*货币.*|.*财政.*|.*土地.*|.*经济.*|.*改革.*|.*中国.*)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "2612"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "uncertainty_dict = [\n",
    "    \"不确定\", \"不明确\", \"不明朗\", \"未明\", \"不明\", \"难料\", \"难以估计\", \"难以预计\", \"难以预测\", \"难以预料\",\n",
    "    \"风险\", \"危险\", \"危机\", \"威胁\", \"未知\"\n",
    "]\n",
    "macro_dict = [\"政策\", \"宏观\", \"货币\", \"财政\", \"土地\", \"经济\", \"改革\", \"中国\"]\n",
    "\n",
    "uncertainty_dict=['.*'+d+'.*' for d in uncertainty_dict]\n",
    "macro_dict=['.*'+d+'.*' for d in macro_dict]\n",
    "uncertainty_re = \"(\" + '|'.join(uncertainty_dict) + \")\"\n",
    "macro_re = \"(\" + '|'.join(macro_dict) + \")\"\n",
    "print(uncertainty_re)\n",
    "print(macro_re)\n",
    "\n",
    "StockNews['uncertainty'] = StockNews['title'].str.match(uncertainty_re)\n",
    "StockNews['macro'] = StockNews['title'].str.match(macro_re)\n",
    "StockNews['macro_uncertainty'] = StockNews['uncertainty'] & StockNews['macro']\n",
    "StockNews['macro_uncertainty'].sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来我们根据年份对不确定性进行一个加总："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>macro_uncertainty</th>\n",
       "      <th>count</th>\n",
       "      <th>ratio</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>year</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1900</th>\n",
       "      <td>0</td>\n",
       "      <td>41</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1986</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1992</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1993</th>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1994</th>\n",
       "      <td>0</td>\n",
       "      <td>16</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1995</th>\n",
       "      <td>0</td>\n",
       "      <td>33</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1996</th>\n",
       "      <td>0</td>\n",
       "      <td>38</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1997</th>\n",
       "      <td>0</td>\n",
       "      <td>44</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1998</th>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1999</th>\n",
       "      <td>1</td>\n",
       "      <td>109</td>\n",
       "      <td>4.784689</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000</th>\n",
       "      <td>19</td>\n",
       "      <td>4489</td>\n",
       "      <td>4.140336</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2001</th>\n",
       "      <td>68</td>\n",
       "      <td>16542</td>\n",
       "      <td>4.086047</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2002</th>\n",
       "      <td>80</td>\n",
       "      <td>35024</td>\n",
       "      <td>2.277645</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2003</th>\n",
       "      <td>136</td>\n",
       "      <td>63149</td>\n",
       "      <td>2.150232</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004</th>\n",
       "      <td>95</td>\n",
       "      <td>40383</td>\n",
       "      <td>2.346664</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2005</th>\n",
       "      <td>70</td>\n",
       "      <td>35759</td>\n",
       "      <td>1.952090</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006</th>\n",
       "      <td>127</td>\n",
       "      <td>87728</td>\n",
       "      <td>1.446008</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2007</th>\n",
       "      <td>151</td>\n",
       "      <td>49529</td>\n",
       "      <td>3.042576</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2008</th>\n",
       "      <td>934</td>\n",
       "      <td>211787</td>\n",
       "      <td>4.408010</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2009</th>\n",
       "      <td>277</td>\n",
       "      <td>63917</td>\n",
       "      <td>4.326976</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010</th>\n",
       "      <td>227</td>\n",
       "      <td>69334</td>\n",
       "      <td>3.269292</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2011</th>\n",
       "      <td>87</td>\n",
       "      <td>45786</td>\n",
       "      <td>1.896003</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2012</th>\n",
       "      <td>136</td>\n",
       "      <td>43196</td>\n",
       "      <td>3.141168</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013</th>\n",
       "      <td>153</td>\n",
       "      <td>53787</td>\n",
       "      <td>2.839275</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014</th>\n",
       "      <td>12</td>\n",
       "      <td>6154</td>\n",
       "      <td>1.918772</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015</th>\n",
       "      <td>39</td>\n",
       "      <td>23263</td>\n",
       "      <td>1.669306</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      macro_uncertainty   count     ratio\n",
       "year                                     \n",
       "1900                  0      41  0.000000\n",
       "1986                  0       1  0.000000\n",
       "1992                  0       1  0.000000\n",
       "1993                  0      13  0.000000\n",
       "1994                  0      16  0.000000\n",
       "1995                  0      33  0.000000\n",
       "1996                  0      38  0.000000\n",
       "1997                  0      44  0.000000\n",
       "1998                  0      30  0.000000\n",
       "1999                  1     109  4.784689\n",
       "2000                 19    4489  4.140336\n",
       "2001                 68   16542  4.086047\n",
       "2002                 80   35024  2.277645\n",
       "2003                136   63149  2.150232\n",
       "2004                 95   40383  2.346664\n",
       "2005                 70   35759  1.952090\n",
       "2006                127   87728  1.446008\n",
       "2007                151   49529  3.042576\n",
       "2008                934  211787  4.408010\n",
       "2009                277   63917  4.326976\n",
       "2010                227   69334  3.269292\n",
       "2011                 87   45786  1.896003\n",
       "2012                136   43196  3.141168\n",
       "2013                153   53787  2.839275\n",
       "2014                 12    6154  1.918772\n",
       "2015                 39   23263  1.669306"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "StockNews['year'] = pd.DatetimeIndex(StockNews['date']).year\n",
    "aggregate_index = StockNews[['year',\n",
    "                             'macro_uncertainty']].groupby('year').sum()\n",
    "aggregate_index['count'] = StockNews[['year', 'macro_uncertainty'\n",
    "                                      ]].groupby('year').count()\n",
    "aggregate_index['ratio'] = aggregate_index['macro_uncertainty'] / (\n",
    "    100 + aggregate_index['count']) * 1000\n",
    "aggregate_index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 提取特征\n",
    "\n",
    "由于计算机只能处理数值变量进行运算，在文本数据清洗完毕后，接下来通常需要将其转换为计算机可以识别的向量。目前有多重方法可以完成这项任务，我们在这里主要介绍其中常见的三种：词袋、TF-IDF以及词嵌入三种模型。\n",
    "\n",
    "## 词袋模型\n",
    "\n",
    "**词袋**（**bag of words**）模型是最基础的一种将文本数据结构化为向量的一种方法。实际上词袋可以简单理解为每个文本的词频统计。比如，我们接下来使用一些上市公司的标题数据，并使用上面介绍的方法将每个标题转换为向量："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>date</th>\n",
       "      <th>title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>59021767</td>\n",
       "      <td>2011-02-19</td>\n",
       "      <td>沪争取增值税扩围改革试点</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>59021769</td>\n",
       "      <td>2011-02-19</td>\n",
       "      <td>周小川：外部施压不会影响人民币升值步伐</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>59021771</td>\n",
       "      <td>2011-02-19</td>\n",
       "      <td>准备金率再上调 达到19.5％创新高</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>59021772</td>\n",
       "      <td>2011-02-19</td>\n",
       "      <td>浙江亚太药业股份有限公司股票2011年02月14日2011年02月18日二级市场表现周简报</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>59021774</td>\n",
       "      <td>2011-02-19</td>\n",
       "      <td>定基价格指数若涨20% 政府或出手调控</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>59021776</td>\n",
       "      <td>2011-02-19</td>\n",
       "      <td>政策倾斜和加大投入 西藏将做大做强藏药产业</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>59021778</td>\n",
       "      <td>2011-02-19</td>\n",
       "      <td>新疆13个地州探矿权年 590个项目涉嫌“圈而不探”</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>59021779</td>\n",
       "      <td>2011-02-19</td>\n",
       "      <td>沪士电子股份有限公司股票2011年02月14日2011年02月18日二级市场表现周简报</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>59021783</td>\n",
       "      <td>2011-02-19</td>\n",
       "      <td>北京：楼市限购首日成交环比降9成</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>59021787</td>\n",
       "      <td>2011-02-19</td>\n",
       "      <td>上海：房管局发布“沪九条”限购执行细则</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         id       date                                          title\n",
       "0  59021767 2011-02-19                                   沪争取增值税扩围改革试点\n",
       "1  59021769 2011-02-19                            周小川：外部施压不会影响人民币升值步伐\n",
       "2  59021771 2011-02-19                             准备金率再上调 达到19.5％创新高\n",
       "3  59021772 2011-02-19  浙江亚太药业股份有限公司股票2011年02月14日2011年02月18日二级市场表现周简报\n",
       "4  59021774 2011-02-19                            定基价格指数若涨20% 政府或出手调控\n",
       "5  59021776 2011-02-19                          政策倾斜和加大投入 西藏将做大做强藏药产业\n",
       "6  59021778 2011-02-19                     新疆13个地州探矿权年 590个项目涉嫌“圈而不探”\n",
       "7  59021779 2011-02-19    沪士电子股份有限公司股票2011年02月14日2011年02月18日二级市场表现周简报\n",
       "8  59021783 2011-02-19                               北京：楼市限购首日成交环比降9成\n",
       "9  59021787 2011-02-19                            上海：房管局发布“沪九条”限购执行细则"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "RAW = pd.read_csv(\"csv/stocknews1.csv\")\n",
    "RAW['date'] = pd.to_datetime(RAW['date'])\n",
    "RAW.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里我们额外进行一些处理，注意到其中的类似「2011年02月14日2011年02月18日二级市场表现周简报」之类的title很没有营养，我们打算去除他，可以使用正则表达式方便的达到目的："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>date</th>\n",
       "      <th>title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>59021767</td>\n",
       "      <td>2011-02-19</td>\n",
       "      <td>沪争取增值税扩围改革试点</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>59021769</td>\n",
       "      <td>2011-02-19</td>\n",
       "      <td>周小川：外部施压不会影响人民币升值步伐</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>59021771</td>\n",
       "      <td>2011-02-19</td>\n",
       "      <td>准备金率再上调 达到19.5％创新高</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>59021774</td>\n",
       "      <td>2011-02-19</td>\n",
       "      <td>定基价格指数若涨20% 政府或出手调控</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>59021776</td>\n",
       "      <td>2011-02-19</td>\n",
       "      <td>政策倾斜和加大投入 西藏将做大做强藏药产业</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1134769</th>\n",
       "      <td>19839999</td>\n",
       "      <td>2006-12-27</td>\n",
       "      <td>定价基准不同 新利率基准将冲击旧浮动利率债</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1134770</th>\n",
       "      <td>19840000</td>\n",
       "      <td>2006-12-27</td>\n",
       "      <td>债市延续调整格局 投资者宜缩短投资久期</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1134771</th>\n",
       "      <td>19840023</td>\n",
       "      <td>2006-12-27</td>\n",
       "      <td>曾培炎:利用外汇储备优势</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1134772</th>\n",
       "      <td>19840026</td>\n",
       "      <td>2006-12-27</td>\n",
       "      <td>广州小时最低工资标准7.5元</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1134773</th>\n",
       "      <td>19840032</td>\n",
       "      <td>2006-12-27</td>\n",
       "      <td>左小蕾:警惕股市系统性风险</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>850153 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "               id       date                  title\n",
       "0        59021767 2011-02-19           沪争取增值税扩围改革试点\n",
       "1        59021769 2011-02-19    周小川：外部施压不会影响人民币升值步伐\n",
       "2        59021771 2011-02-19     准备金率再上调 达到19.5％创新高\n",
       "4        59021774 2011-02-19    定基价格指数若涨20% 政府或出手调控\n",
       "5        59021776 2011-02-19  政策倾斜和加大投入 西藏将做大做强藏药产业\n",
       "...           ...        ...                    ...\n",
       "1134769  19839999 2006-12-27  定价基准不同 新利率基准将冲击旧浮动利率债\n",
       "1134770  19840000 2006-12-27    债市延续调整格局 投资者宜缩短投资久期\n",
       "1134771  19840023 2006-12-27           曾培炎:利用外汇储备优势\n",
       "1134772  19840026 2006-12-27         广州小时最低工资标准7.5元\n",
       "1134773  19840032 2006-12-27         左小蕾:警惕股市系统性风险 \n",
       "\n",
       "[850153 rows x 3 columns]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jianbao = RAW['title'].str.match(r'(.+\\d{4}年\\d{2}月\\d{2}日.+简报|.*二级市场.*简报|.*业绩公报.*|.*登记日期.*|.*交易安排表.*)')\n",
    "RAW1 = RAW.iloc[list(~jianbao), :]\n",
    "RAW1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了方便起见，我们选取其中的前十条先进行分析："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>date</th>\n",
       "      <th>title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>59021767</td>\n",
       "      <td>2011-02-19</td>\n",
       "      <td>沪争取增值税扩围改革试点</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>59021769</td>\n",
       "      <td>2011-02-19</td>\n",
       "      <td>周小川：外部施压不会影响人民币升值步伐</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>59021771</td>\n",
       "      <td>2011-02-19</td>\n",
       "      <td>准备金率再上调 达到19.5％创新高</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>59021774</td>\n",
       "      <td>2011-02-19</td>\n",
       "      <td>定基价格指数若涨20% 政府或出手调控</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>59021776</td>\n",
       "      <td>2011-02-19</td>\n",
       "      <td>政策倾斜和加大投入 西藏将做大做强藏药产业</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>59021778</td>\n",
       "      <td>2011-02-19</td>\n",
       "      <td>新疆13个地州探矿权年 590个项目涉嫌“圈而不探”</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>59021783</td>\n",
       "      <td>2011-02-19</td>\n",
       "      <td>北京：楼市限购首日成交环比降9成</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>59021787</td>\n",
       "      <td>2011-02-19</td>\n",
       "      <td>上海：房管局发布“沪九条”限购执行细则</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>59021789</td>\n",
       "      <td>2011-02-19</td>\n",
       "      <td>6.5781！人民币升值容忍度继续提高？</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>59021791</td>\n",
       "      <td>2011-02-19</td>\n",
       "      <td>2010年上海商品住宅销售降四成</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          id       date                       title\n",
       "0   59021767 2011-02-19                沪争取增值税扩围改革试点\n",
       "1   59021769 2011-02-19         周小川：外部施压不会影响人民币升值步伐\n",
       "2   59021771 2011-02-19          准备金率再上调 达到19.5％创新高\n",
       "4   59021774 2011-02-19         定基价格指数若涨20% 政府或出手调控\n",
       "5   59021776 2011-02-19       政策倾斜和加大投入 西藏将做大做强藏药产业\n",
       "6   59021778 2011-02-19  新疆13个地州探矿权年 590个项目涉嫌“圈而不探”\n",
       "8   59021783 2011-02-19            北京：楼市限购首日成交环比降9成\n",
       "9   59021787 2011-02-19         上海：房管局发布“沪九条”限购执行细则\n",
       "10  59021789 2011-02-19        6.5781！人民币升值容忍度继续提高？\n",
       "11  59021791 2011-02-19            2010年上海商品住宅销售降四成"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = RAW1.iloc[:10, :]\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['沪', '增值税', '扩围', '改革', '试点'],\n",
       " ['周小川', '外部', '施压', '影响', '人民币', '升值', '步伐'],\n",
       " ['准备金率', '上调', '19.5', '创新', '高'],\n",
       " ['定基', '价格指数', '若涨', '20%', '政府', '出手', '调控'],\n",
       " ['政策', '倾斜', '加大', '投入', '西藏', '做', '做', '强', '藏药', '产业'],\n",
       " ['新疆', '地州', '探矿权', '590', '项目', '涉嫌', '圈', '不探'],\n",
       " ['北京', '楼市', '限购', '首日', '成交', '环', '比降', '成'],\n",
       " ['上海', '房管局', '发布', '沪', '九条', '限购', '执行', '细则'],\n",
       " ['6.5781', '人民币', '升值', '容忍度', '提高'],\n",
       " ['2010', '上海', '商品住宅', '销售', '降', '四成']]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## 首先进行分词、去除停用词等\n",
    "import jieba\n",
    "\n",
    "with open('Chinese/stopword.txt', 'rt') as f:\n",
    "    stoplist = f.readlines()\n",
    "    stoplist = [w.replace('\\n', '') for w in stoplist]\n",
    "\n",
    "\n",
    "def tokenize(w):\n",
    "    cut_w = jieba.cut(w)\n",
    "    ## 去除停用词\n",
    "    cut_w = [\n",
    "        w.strip().lower() for w in cut_w\n",
    "        if w.strip().lower() not in stoplist and len(w.strip()) > 0\n",
    "    ]\n",
    "    return cut_w\n",
    "\n",
    "\n",
    "tokenized_data = map(tokenize, data['title'])\n",
    "tokenized_data = list(tokenized_data)\n",
    "tokenized_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>沪</th>\n",
       "      <th>增值税</th>\n",
       "      <th>扩围</th>\n",
       "      <th>改革</th>\n",
       "      <th>试点</th>\n",
       "      <th>周小川</th>\n",
       "      <th>外部</th>\n",
       "      <th>施压</th>\n",
       "      <th>影响</th>\n",
       "      <th>人民币</th>\n",
       "      <th>...</th>\n",
       "      <th>执行</th>\n",
       "      <th>细则</th>\n",
       "      <th>6.5781</th>\n",
       "      <th>容忍度</th>\n",
       "      <th>提高</th>\n",
       "      <th>2010</th>\n",
       "      <th>商品住宅</th>\n",
       "      <th>销售</th>\n",
       "      <th>降</th>\n",
       "      <th>四成</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 63 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     沪  增值税   扩围   改革   试点  周小川   外部   施压   影响  人民币  ...   执行   细则  6.5781  \\\n",
       "0  1.0  1.0  1.0  1.0  1.0  0.0  0.0  0.0  0.0  0.0  ...  0.0  0.0     0.0   \n",
       "1  0.0  0.0  0.0  0.0  0.0  1.0  1.0  1.0  1.0  1.0  ...  0.0  0.0     0.0   \n",
       "2  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  ...  0.0  0.0     0.0   \n",
       "3  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  ...  0.0  0.0     0.0   \n",
       "4  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  ...  0.0  0.0     0.0   \n",
       "5  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  ...  0.0  0.0     0.0   \n",
       "6  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  ...  0.0  0.0     0.0   \n",
       "7  1.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  ...  1.0  1.0     0.0   \n",
       "8  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  1.0  ...  0.0  0.0     1.0   \n",
       "9  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  ...  0.0  0.0     0.0   \n",
       "\n",
       "   容忍度   提高  2010  商品住宅   销售    降   四成  \n",
       "0  0.0  0.0   0.0   0.0  0.0  0.0  0.0  \n",
       "1  0.0  0.0   0.0   0.0  0.0  0.0  0.0  \n",
       "2  0.0  0.0   0.0   0.0  0.0  0.0  0.0  \n",
       "3  0.0  0.0   0.0   0.0  0.0  0.0  0.0  \n",
       "4  0.0  0.0   0.0   0.0  0.0  0.0  0.0  \n",
       "5  0.0  0.0   0.0   0.0  0.0  0.0  0.0  \n",
       "6  0.0  0.0   0.0   0.0  0.0  0.0  0.0  \n",
       "7  0.0  0.0   0.0   0.0  0.0  0.0  0.0  \n",
       "8  1.0  1.0   0.0   0.0  0.0  0.0  0.0  \n",
       "9  0.0  0.0   1.0   1.0  1.0  1.0  1.0  \n",
       "\n",
       "[10 rows x 63 columns]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## 接下来进行词频统计\n",
    "\n",
    "\n",
    "def word_freq(wlist):\n",
    "    freq = {}\n",
    "    for w in wlist:\n",
    "        if w in freq:\n",
    "            freq[w] += 1\n",
    "        else:\n",
    "            freq[w] = 1\n",
    "\n",
    "    return freq\n",
    "\n",
    "\n",
    "freqs = map(word_freq, tokenized_data)\n",
    "## 放在pandas里\n",
    "pd_freqs = pd.DataFrame(freqs)\n",
    "## 把NaN换成0\n",
    "pd_freqs = pd_freqs.fillna(0)\n",
    "pd_freqs.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如此，一个简单的词袋模型就完成了。\n",
    "\n",
    "不过，可以看到虽然我们只有10条新闻，这个矩阵已经很大了。如果10万条新闻一起来，不仅词（列）多，而且行也多，最终这个矩阵的规模会变的非常巨大，甚至可能很轻易的会占满内存。\n",
    "\n",
    "然而注意到，这个矩阵里面多数的值都是0，这种类型的矩阵我们成为**稀疏矩阵**（**sparse matrix**），在SciPy和Pandas里面都提供了稀疏矩阵的存储结构和运算，所以更好的办法是使用稀疏矩阵进行存储："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>沪</th>\n",
       "      <th>增值税</th>\n",
       "      <th>扩围</th>\n",
       "      <th>改革</th>\n",
       "      <th>试点</th>\n",
       "      <th>周小川</th>\n",
       "      <th>外部</th>\n",
       "      <th>施压</th>\n",
       "      <th>影响</th>\n",
       "      <th>人民币</th>\n",
       "      <th>...</th>\n",
       "      <th>执行</th>\n",
       "      <th>细则</th>\n",
       "      <th>6.5781</th>\n",
       "      <th>容忍度</th>\n",
       "      <th>提高</th>\n",
       "      <th>2010</th>\n",
       "      <th>商品住宅</th>\n",
       "      <th>销售</th>\n",
       "      <th>降</th>\n",
       "      <th>四成</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 63 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     沪  增值税   扩围   改革   试点  周小川   外部   施压   影响  人民币  ...   执行   细则  6.5781  \\\n",
       "0  1.0  1.0  1.0  1.0  1.0    0    0    0    0    0  ...    0    0       0   \n",
       "1    0    0    0    0    0  1.0  1.0  1.0  1.0  1.0  ...    0    0       0   \n",
       "2    0    0    0    0    0    0    0    0    0    0  ...    0    0       0   \n",
       "3    0    0    0    0    0    0    0    0    0    0  ...    0    0       0   \n",
       "4    0    0    0    0    0    0    0    0    0    0  ...    0    0       0   \n",
       "5    0    0    0    0    0    0    0    0    0    0  ...    0    0       0   \n",
       "6    0    0    0    0    0    0    0    0    0    0  ...    0    0       0   \n",
       "7  1.0    0    0    0    0    0    0    0    0    0  ...  1.0  1.0       0   \n",
       "8    0    0    0    0    0    0    0    0    0  1.0  ...    0    0     1.0   \n",
       "9    0    0    0    0    0    0    0    0    0    0  ...    0    0       0   \n",
       "\n",
       "   容忍度   提高  2010  商品住宅   销售    降   四成  \n",
       "0    0    0     0     0    0    0    0  \n",
       "1    0    0     0     0    0    0    0  \n",
       "2    0    0     0     0    0    0    0  \n",
       "3    0    0     0     0    0    0    0  \n",
       "4    0    0     0     0    0    0    0  \n",
       "5    0    0     0     0    0    0    0  \n",
       "6    0    0     0     0    0    0    0  \n",
       "7    0    0     0     0    0    0    0  \n",
       "8  1.0  1.0     0     0    0    0    0  \n",
       "9    0    0   1.0   1.0  1.0  1.0  1.0  \n",
       "\n",
       "[10 rows x 63 columns]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sparse_freqs = pd_freqs.astype(pd.SparseDtype(\"float\", 0))\n",
    "sparse_freqs.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上面的意思是通过类型转换，把数据框转换为稀疏类型，其中的「0」就不存储了，虽然看起来没有变化，但是如果我们查看类型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "沪       Sparse[float64, 0]\n",
       "增值税     Sparse[float64, 0]\n",
       "扩围      Sparse[float64, 0]\n",
       "改革      Sparse[float64, 0]\n",
       "试点      Sparse[float64, 0]\n",
       "               ...        \n",
       "2010    Sparse[float64, 0]\n",
       "商品住宅    Sparse[float64, 0]\n",
       "销售      Sparse[float64, 0]\n",
       "降       Sparse[float64, 0]\n",
       "四成      Sparse[float64, 0]\n",
       "Length: 63, dtype: object"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sparse_freqs.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "会发现都变成了Sparse的float64类型，且0不存储。\n",
    "\n",
    "不过，如果使用稀疏类型，那么隐式索引将不能使用："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "沪       1.0\n",
      "增值税     1.0\n",
      "扩围      1.0\n",
      "改革      1.0\n",
      "试点      1.0\n",
      "       ... \n",
      "2010      0\n",
      "商品住宅      0\n",
      "销售        0\n",
      "降         0\n",
      "四成        0\n",
      "Name: 0, Length: 63, dtype: Sparse[float64, 0]\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    print(sparse_freqs.iloc[0,:])\n",
    "except Exception as e:\n",
    "    print(\"错误：\",e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果需要使用隐式索引，需要使用："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "沪       1.0\n",
      "增值税     1.0\n",
      "扩围      1.0\n",
      "改革      1.0\n",
      "试点      1.0\n",
      "       ... \n",
      "2010    0.0\n",
      "商品住宅    0.0\n",
      "销售      0.0\n",
      "降       0.0\n",
      "四成      0.0\n",
      "Name: 0, Length: 63, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "print(sparse_freqs.sparse.to_dense().iloc[0,:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "当然，以上是自己手写的词袋生成步骤，实际上很多自然语言处理包已经有比较成熟的词袋处理机制。比如在Scikit-Learn中已经准备好了词袋的提取函数："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/Anaconda/lib/python3.11/site-packages/sklearn/feature_extraction/text.py:521: UserWarning: The parameter 'token_pattern' will not be used since 'tokenizer' is not None'\n",
      "  warnings.warn(\n",
      "/opt/Anaconda/lib/python3.11/site-packages/sklearn/feature_extraction/text.py:406: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['##', 'a', 'ain', 'aren', 'c', 'couldn', 'd', 'didn', 'doesn', 'don', 'hadn', 'hasn', 'haven', 'i', 'isn', 'lex', 'll', 'm', 'mon', 'null', 's', 'shouldn', 't', 've', 'wasn', 'weren', 'won', 'wouldn', '±', '÷', 'β', 'δ', 'λ', 'ξ', 'ψ', 'в', '′', '″', 'ⅲ', '∈', '∧', '∪', '─', '☆', '一会', '一关', '一城', '一堆', '一对', '一批', '一方', '一期', '一村', '一根', '一派', '一班', '一百', '一眼', '一科', '一群', '一遍', '一道', '一部', '一集', '一页', '一颗', '三鲜', '为什', '什', '倒', '傥', '元', '元素', '先', '关', '兼', '前', '单元', '吨', '唷', '啪', '啷', '喔', '喜欢', '外', '多年', '大节', '大道', '大面儿', '天', '始', '子弹', '後', '抗拒', '敞开', '新', '昉', '更远', '有意', '有趣', '末', '次', '毫无保留', '波', '漫', '特', '特别', '理', '皆', '目前为止', '笑', '第三', '第五', '第四', '策略', '讲', '设', '话', '说', '赶早', '赶晚', '达', '钱', '限', '非', '面', '题', '麽', 'ａ', 'ｂ', 'ｃ', 'ｄ', 'ｅ', 'ｆ', 'ｇ', 'ｈ', 'ｉ', 'ｊ', 'ｌ', 'ｎ', 'ｏ', 'ｒ', 'ｔ', 'ｘ', 'ｚ'] not in stop_words.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<10x64 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 72 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import jieba\n",
    "\n",
    "with open('Chinese/stopword.txt', 'rt') as f:\n",
    "    stoplist = f.readlines()\n",
    "    stoplist = [w.replace('\\n', '') for w in stoplist]\n",
    "\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "count_vect = CountVectorizer(tokenizer=jieba.cut,\n",
    "                             stop_words=stoplist,\n",
    "                             min_df=1)\n",
    "bag_words = count_vect.fit_transform(data['title'])\n",
    "words_names = count_vect.get_feature_names_out()\n",
    "bag_words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "注意由于Scikit-Learn中一般只支持英文的自动分词（tokenize），为了让他能够处理中文的分词，我们把jieba.cut函数提交给了CountVectorizer，此外还额外提供了停用词列表。min_df选项设置了如果在所有文本中某个词出现的频率下线，如果出现太少则会被忽略，适当提高这个选项可以降低维数。\n",
    "\n",
    "上面的词袋结果是一个sparse matrix，实际上是SciPy中的系数矩阵形式，实际分析时已经可以使用。不过为了查看方便，我们不妨将其转换为Pandas的数据框（稀疏存储）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>19.5</th>\n",
       "      <th>20%</th>\n",
       "      <th>2010</th>\n",
       "      <th>590</th>\n",
       "      <th>6.5781</th>\n",
       "      <th>上海</th>\n",
       "      <th>上调</th>\n",
       "      <th>不探</th>\n",
       "      <th>九条</th>\n",
       "      <th>...</th>\n",
       "      <th>藏药</th>\n",
       "      <th>西藏</th>\n",
       "      <th>试点</th>\n",
       "      <th>调控</th>\n",
       "      <th>销售</th>\n",
       "      <th>降</th>\n",
       "      <th>限购</th>\n",
       "      <th>项目</th>\n",
       "      <th>首日</th>\n",
       "      <th>高</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 64 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      19.5  20%  2010  590  6.5781  上海  上调  不探  九条  ...  藏药  西藏  试点  调控  销售  \\\n",
       "0  0     0    0     0    0       0   0   0   0   0  ...   0   0   1   0   0   \n",
       "1  0     0    0     0    0       0   0   0   0   0  ...   0   0   0   0   0   \n",
       "2  1     1    0     0    0       0   0   1   0   0  ...   0   0   0   0   0   \n",
       "3  1     0    1     0    0       0   0   0   0   0  ...   0   0   0   1   0   \n",
       "4  1     0    0     0    0       0   0   0   0   0  ...   1   1   0   0   0   \n",
       "5  1     0    0     0    1       0   0   0   1   0  ...   0   0   0   0   0   \n",
       "6  0     0    0     0    0       0   0   0   0   0  ...   0   0   0   0   0   \n",
       "7  0     0    0     0    0       0   1   0   0   1  ...   0   0   0   0   0   \n",
       "8  0     0    0     0    0       1   0   0   0   0  ...   0   0   0   0   0   \n",
       "9  0     0    0     1    0       0   1   0   0   0  ...   0   0   0   0   1   \n",
       "\n",
       "   降  限购  项目  首日  高  \n",
       "0  0   0   0   0  0  \n",
       "1  0   0   0   0  0  \n",
       "2  0   0   0   0  1  \n",
       "3  0   0   0   0  0  \n",
       "4  0   0   0   0  0  \n",
       "5  0   0   1   0  0  \n",
       "6  0   1   0   1  0  \n",
       "7  0   1   0   0  0  \n",
       "8  0   0   0   0  0  \n",
       "9  1   0   0   0  0  \n",
       "\n",
       "[10 rows x 64 columns]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bag_words_df = pd.DataFrame.sparse.from_spmatrix(bag_words,\n",
    "                                                 columns=words_names)\n",
    "bag_words_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "也许你会觉着不舒服的是，分词和去除停用词等操作应该是第二步清洗数据完成的，现在如果都放到向量化的对象CountVectorizer中来，非常不灵活。\n",
    "\n",
    "比如显然上面的结果中，「13，19.5，20%」等都不是我们想要的，然而用停用词根本不可能将这些数字去除。\n",
    "\n",
    "那么如何将两者（手工清洗+自动计算词袋）结合起来呢？其实很简单，用空格将他们join起来就好了。比如："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['沪 增值税 扩围 改革 试点',\n",
       " '周小川 外部 施压 影响 人民币 升值 步伐',\n",
       " '准备金率 上调 创新 高',\n",
       " '定基 价格指数 若涨 政府 出手 调控',\n",
       " '政策 倾斜 加大 投入 西藏 做 做 强 藏药 产业',\n",
       " '新疆 地州 探矿权 项目 涉嫌 圈 不探',\n",
       " '北京 楼市 限购 首日 成交 环 比降 成',\n",
       " '上海 房管局 发布 沪 九条 限购 执行 细则',\n",
       " '人民币 升值 容忍度 提高',\n",
       " '上海 商品住宅 销售 降 四成']"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import jieba\n",
    "import re\n",
    "\n",
    "with open('Chinese/stopword.txt', 'rt') as f:\n",
    "    stoplist = f.readlines()\n",
    "    stoplist = [w.replace('\\n', '') for w in stoplist]\n",
    "\n",
    "\n",
    "def not_digit(w):\n",
    "    w = w.replace(',', '')\n",
    "    if re.match(r'\\d+', w) != None or re.match(r'\\d%', w) != None or re.match(\n",
    "            r'\\d*\\.\\d+', w) != None:\n",
    "        return False\n",
    "    else:\n",
    "        return True\n",
    "\n",
    "\n",
    "def tokenize(w):\n",
    "    cut_w = jieba.cut(w)\n",
    "    ## 去除停用词\n",
    "    cut_w = [\n",
    "        w.strip().lower() for w in cut_w\n",
    "        if ((w.strip().lower() not in stoplist) and not_digit(w) and len(w.strip()) > 0)\n",
    "    ]\n",
    "    return cut_w\n",
    "\n",
    "\n",
    "tokenized_data = map(tokenize, data['title'])\n",
    "tokenized_data = [' '.join(t) for t in tokenized_data]\n",
    "tokenized_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>上海</th>\n",
       "      <th>上调</th>\n",
       "      <th>不探</th>\n",
       "      <th>九条</th>\n",
       "      <th>产业</th>\n",
       "      <th>人民币</th>\n",
       "      <th>价格指数</th>\n",
       "      <th>倾斜</th>\n",
       "      <th>准备金率</th>\n",
       "      <th>出手</th>\n",
       "      <th>...</th>\n",
       "      <th>细则</th>\n",
       "      <th>若涨</th>\n",
       "      <th>藏药</th>\n",
       "      <th>西藏</th>\n",
       "      <th>试点</th>\n",
       "      <th>调控</th>\n",
       "      <th>销售</th>\n",
       "      <th>限购</th>\n",
       "      <th>项目</th>\n",
       "      <th>首日</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 50 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   上海  上调  不探  九条  产业  人民币  价格指数  倾斜  准备金率  出手  ...  细则  若涨  藏药  西藏  试点  调控  \\\n",
       "0   0   0   0   0   0    0     0   0     0   0  ...   0   0   0   0   1   0   \n",
       "1   0   0   0   0   0    1     0   0     0   0  ...   0   0   0   0   0   0   \n",
       "2   0   1   0   0   0    0     0   0     1   0  ...   0   0   0   0   0   0   \n",
       "3   0   0   0   0   0    0     1   0     0   1  ...   0   1   0   0   0   1   \n",
       "4   0   0   0   0   1    0     0   1     0   0  ...   0   0   1   1   0   0   \n",
       "5   0   0   1   0   0    0     0   0     0   0  ...   0   0   0   0   0   0   \n",
       "6   0   0   0   0   0    0     0   0     0   0  ...   0   0   0   0   0   0   \n",
       "7   1   0   0   1   0    0     0   0     0   0  ...   1   0   0   0   0   0   \n",
       "8   0   0   0   0   0    1     0   0     0   0  ...   0   0   0   0   0   0   \n",
       "9   1   0   0   0   0    0     0   0     0   0  ...   0   0   0   0   0   0   \n",
       "\n",
       "   销售  限购  项目  首日  \n",
       "0   0   0   0   0  \n",
       "1   0   0   0   0  \n",
       "2   0   0   0   0  \n",
       "3   0   0   0   0  \n",
       "4   0   0   0   0  \n",
       "5   0   0   1   0  \n",
       "6   0   1   0   1  \n",
       "7   0   1   0   0  \n",
       "8   0   0   0   0  \n",
       "9   1   0   0   0  \n",
       "\n",
       "[10 rows x 50 columns]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "count_vect = CountVectorizer()\n",
    "bag_words = count_vect.fit_transform(tokenized_data)\n",
    "words_names = count_vect.get_feature_names_out()\n",
    "bag_words_df = pd.DataFrame.sparse.from_spmatrix(bag_words,\n",
    "                                                 columns=words_names)\n",
    "bag_words_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "当然，更优雅的方法是直接将手写的tokenize函数调入即可："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/Anaconda/lib/python3.11/site-packages/sklearn/feature_extraction/text.py:521: UserWarning: The parameter 'token_pattern' will not be used since 'tokenizer' is not None'\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>上海</th>\n",
       "      <th>上调</th>\n",
       "      <th>不探</th>\n",
       "      <th>九条</th>\n",
       "      <th>产业</th>\n",
       "      <th>人民币</th>\n",
       "      <th>价格指数</th>\n",
       "      <th>倾斜</th>\n",
       "      <th>做</th>\n",
       "      <th>准备金率</th>\n",
       "      <th>...</th>\n",
       "      <th>藏药</th>\n",
       "      <th>西藏</th>\n",
       "      <th>试点</th>\n",
       "      <th>调控</th>\n",
       "      <th>销售</th>\n",
       "      <th>降</th>\n",
       "      <th>限购</th>\n",
       "      <th>项目</th>\n",
       "      <th>首日</th>\n",
       "      <th>高</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 58 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   上海  上调  不探  九条  产业  人民币  价格指数  倾斜  做  准备金率  ...  藏药  西藏  试点  调控  销售  降  限购  \\\n",
       "0   0   0   0   0   0    0     0   0  0     0  ...   0   0   1   0   0  0   0   \n",
       "1   0   0   0   0   0    1     0   0  0     0  ...   0   0   0   0   0  0   0   \n",
       "2   0   1   0   0   0    0     0   0  0     1  ...   0   0   0   0   0  0   0   \n",
       "3   0   0   0   0   0    0     1   0  0     0  ...   0   0   0   1   0  0   0   \n",
       "4   0   0   0   0   1    0     0   1  2     0  ...   1   1   0   0   0  0   0   \n",
       "5   0   0   1   0   0    0     0   0  0     0  ...   0   0   0   0   0  0   0   \n",
       "6   0   0   0   0   0    0     0   0  0     0  ...   0   0   0   0   0  0   1   \n",
       "7   1   0   0   1   0    0     0   0  0     0  ...   0   0   0   0   0  0   1   \n",
       "8   0   0   0   0   0    1     0   0  0     0  ...   0   0   0   0   0  0   0   \n",
       "9   1   0   0   0   0    0     0   0  0     0  ...   0   0   0   0   1  1   0   \n",
       "\n",
       "   项目  首日  高  \n",
       "0   0   0  0  \n",
       "1   0   0  0  \n",
       "2   0   0  1  \n",
       "3   0   0  0  \n",
       "4   0   0  0  \n",
       "5   1   0  0  \n",
       "6   0   1  0  \n",
       "7   0   0  0  \n",
       "8   0   0  0  \n",
       "9   0   0  0  \n",
       "\n",
       "[10 rows x 58 columns]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "count_vect = CountVectorizer(tokenizer=tokenize)\n",
    "bag_words = count_vect.fit_transform(data['title'])\n",
    "words_names = count_vect.get_feature_names_out()\n",
    "bag_words_df = pd.DataFrame.sparse.from_spmatrix(bag_words,\n",
    "                                                 columns=words_names)\n",
    "bag_words_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "实际上以上的词袋又被称为1元词袋，是N元（N-gram）词袋的一种特例。我们当然可以做成二元词袋："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/Anaconda/lib/python3.11/site-packages/sklearn/feature_extraction/text.py:521: UserWarning: The parameter 'token_pattern' will not be used since 'tokenizer' is not None'\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>上海 商品住宅</th>\n",
       "      <th>上海 房管局</th>\n",
       "      <th>上调 创新</th>\n",
       "      <th>九条 限购</th>\n",
       "      <th>人民币 升值</th>\n",
       "      <th>价格指数 若涨</th>\n",
       "      <th>倾斜 加大</th>\n",
       "      <th>做 做</th>\n",
       "      <th>做 强</th>\n",
       "      <th>准备金率 上调</th>\n",
       "      <th>...</th>\n",
       "      <th>环 比降</th>\n",
       "      <th>若涨 政府</th>\n",
       "      <th>藏药 产业</th>\n",
       "      <th>西藏 做</th>\n",
       "      <th>销售 降</th>\n",
       "      <th>降 四成</th>\n",
       "      <th>限购 执行</th>\n",
       "      <th>限购 首日</th>\n",
       "      <th>项目 涉嫌</th>\n",
       "      <th>首日 成交</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 53 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   上海 商品住宅  上海 房管局  上调 创新  九条 限购  人民币 升值  价格指数 若涨  倾斜 加大  做 做  做 强  准备金率 上调  \\\n",
       "0        0       0      0      0       0        0      0    0    0        0   \n",
       "1        0       0      0      0       1        0      0    0    0        0   \n",
       "2        0       0      1      0       0        0      0    0    0        1   \n",
       "3        0       0      0      0       0        1      0    0    0        0   \n",
       "4        0       0      0      0       0        0      1    1    1        0   \n",
       "5        0       0      0      0       0        0      0    0    0        0   \n",
       "6        0       0      0      0       0        0      0    0    0        0   \n",
       "7        0       1      0      1       0        0      0    0    0        0   \n",
       "8        0       0      0      0       1        0      0    0    0        0   \n",
       "9        1       0      0      0       0        0      0    0    0        0   \n",
       "\n",
       "   ...  环 比降  若涨 政府  藏药 产业  西藏 做  销售 降  降 四成  限购 执行  限购 首日  项目 涉嫌  首日 成交  \n",
       "0  ...     0      0      0     0     0     0      0      0      0      0  \n",
       "1  ...     0      0      0     0     0     0      0      0      0      0  \n",
       "2  ...     0      0      0     0     0     0      0      0      0      0  \n",
       "3  ...     0      1      0     0     0     0      0      0      0      0  \n",
       "4  ...     0      0      1     1     0     0      0      0      0      0  \n",
       "5  ...     0      0      0     0     0     0      0      0      1      0  \n",
       "6  ...     1      0      0     0     0     0      0      1      0      1  \n",
       "7  ...     0      0      0     0     0     0      1      0      0      0  \n",
       "8  ...     0      0      0     0     0     0      0      0      0      0  \n",
       "9  ...     0      0      0     0     1     1      0      0      0      0  \n",
       "\n",
       "[10 rows x 53 columns]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "count_vect = CountVectorizer(tokenizer=tokenize, ngram_range=(2, 2))\n",
    "bag_words = count_vect.fit_transform(data['title'])\n",
    "words_names = count_vect.get_feature_names_out()\n",
    "bag_words_df = pd.DataFrame.sparse.from_spmatrix(bag_words,\n",
    "                                                 columns=words_names)\n",
    "bag_words_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "其中ngram_range选项给出了一个区间，如果是(1,2)那么就是1元、2元词袋同时存在，而(2,2)代表仅使用2元词袋，(1,1)为默认，即只使用一元词袋。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TF-IDF模型\n",
    "\n",
    "词袋模型简单有效，但是有一个缺点，即只考虑了词的绝对频率，而没有考虑相对频率。比如有的词天然的出现频率更高，那么在每个文档里面，其重要性应该是更低的。\n",
    "\n",
    "而TF-IDF模型就是在词袋的基础上修正这一点。TF-IDF模型是两个度量的乘积：$$TFIDF=TF\\times IDF$$其中TF即词频，而IDF为逆文档频率，IDF的定义为：$$IDF\\left(w\\right)=1+\\ln\\left(\\frac{N}{1+df\\left(w\\right)}\\right)$$其中$N$为文档总数量，而$df\\left(w\\right)$为包含单词$w$的文档个数。\n",
    "\n",
    "可以看到根据上面的定义，一个单词$w$如果出现的文档越多，那么其$IDF\\left(w\\right)$值就越小，或者说权重就越小。\n",
    "\n",
    "最终，我们将上面词袋中每个文档每个词的词频（$TF_i\\left(w\\right),i=1,...,N$）乘以权重就得到了TF-IDF值：$$TFIDF_i\\left(w\\right)=TF_i\\left(w\\right)\\times IDF\\left(w\\right),i=1,...,N$$\n",
    "\n",
    "最后，对每个文档，还需要将以上得到的向量进行标准化，一般使用$L2$范数进行标准化（从而每个向量都在$M$维单位球上，$M$为词的个数）：$$NormalizedTFIDF_i\\left(w\\right)=\\frac{TFIDF_i\\left(w\\right)}{\\sqrt{\\sum_{k=1}^{M}\\left[TFIDF_i\\left(w\\right)\\right]^2}},i=1,...,N$$\n",
    "\n",
    "比如，对于之前计算的词袋："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/Anaconda/lib/python3.11/site-packages/sklearn/feature_extraction/text.py:521: UserWarning: The parameter 'token_pattern' will not be used since 'tokenizer' is not None'\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>上海</th>\n",
       "      <th>上调</th>\n",
       "      <th>不探</th>\n",
       "      <th>九条</th>\n",
       "      <th>产业</th>\n",
       "      <th>人民币</th>\n",
       "      <th>价格指数</th>\n",
       "      <th>倾斜</th>\n",
       "      <th>做</th>\n",
       "      <th>准备金率</th>\n",
       "      <th>...</th>\n",
       "      <th>藏药</th>\n",
       "      <th>西藏</th>\n",
       "      <th>试点</th>\n",
       "      <th>调控</th>\n",
       "      <th>销售</th>\n",
       "      <th>降</th>\n",
       "      <th>限购</th>\n",
       "      <th>项目</th>\n",
       "      <th>首日</th>\n",
       "      <th>高</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 58 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   上海  上调  不探  九条  产业  人民币  价格指数  倾斜  做  准备金率  ...  藏药  西藏  试点  调控  销售  降  限购  \\\n",
       "0   0   0   0   0   0    0     0   0  0     0  ...   0   0   1   0   0  0   0   \n",
       "1   0   0   0   0   0    1     0   0  0     0  ...   0   0   0   0   0  0   0   \n",
       "2   0   1   0   0   0    0     0   0  0     1  ...   0   0   0   0   0  0   0   \n",
       "3   0   0   0   0   0    0     1   0  0     0  ...   0   0   0   1   0  0   0   \n",
       "4   0   0   0   0   1    0     0   1  2     0  ...   1   1   0   0   0  0   0   \n",
       "5   0   0   1   0   0    0     0   0  0     0  ...   0   0   0   0   0  0   0   \n",
       "6   0   0   0   0   0    0     0   0  0     0  ...   0   0   0   0   0  0   1   \n",
       "7   1   0   0   1   0    0     0   0  0     0  ...   0   0   0   0   0  0   1   \n",
       "8   0   0   0   0   0    1     0   0  0     0  ...   0   0   0   0   0  0   0   \n",
       "9   1   0   0   0   0    0     0   0  0     0  ...   0   0   0   0   1  1   0   \n",
       "\n",
       "   项目  首日  高  \n",
       "0   0   0  0  \n",
       "1   0   0  0  \n",
       "2   0   0  1  \n",
       "3   0   0  0  \n",
       "4   0   0  0  \n",
       "5   1   0  0  \n",
       "6   0   1  0  \n",
       "7   0   0  0  \n",
       "8   0   0  0  \n",
       "9   0   0  0  \n",
       "\n",
       "[10 rows x 58 columns]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "count_vect = CountVectorizer(tokenizer=tokenize)\n",
    "bag_words = count_vect.fit_transform(tokenized_data)\n",
    "words_names = count_vect.get_feature_names_out()\n",
    "bag_words_df = pd.DataFrame.sparse.from_spmatrix(bag_words,\n",
    "                                                 columns=words_names)\n",
    "bag_words_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以计算「上海」这个词在两个文档中出现，从而其$IDF=1+\\ln\\left(10\\right)-\\ln\\left(1+2\\right)=2.20397$，而「做」这个词只有一个文档出现，从而其$IDF=1+\\ln\\left(10\\right)-\\ln\\left(1+1\\right)=2.60944$。\n",
    "\n",
    "以上计算略显复杂，不过Scikit-Learn中也给出了方便的计算函数："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>上海</th>\n",
       "      <th>上调</th>\n",
       "      <th>不探</th>\n",
       "      <th>九条</th>\n",
       "      <th>产业</th>\n",
       "      <th>人民币</th>\n",
       "      <th>价格指数</th>\n",
       "      <th>倾斜</th>\n",
       "      <th>做</th>\n",
       "      <th>准备金率</th>\n",
       "      <th>...</th>\n",
       "      <th>藏药</th>\n",
       "      <th>西藏</th>\n",
       "      <th>试点</th>\n",
       "      <th>调控</th>\n",
       "      <th>销售</th>\n",
       "      <th>降</th>\n",
       "      <th>限购</th>\n",
       "      <th>项目</th>\n",
       "      <th>首日</th>\n",
       "      <th>高</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.460158</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.334845</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.408248</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.408248</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.288675</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.288675</td>\n",
       "      <td>0.57735</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.288675</td>\n",
       "      <td>0.288675</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.377964</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.377964</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.305902</td>\n",
       "      <td>0</td>\n",
       "      <td>0.359846</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.317517</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.37351</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.317517</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.457985</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.391176</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.460158</td>\n",
       "      <td>0.460158</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 58 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         上海   上调        不探       九条        产业       人民币      价格指数        倾斜  \\\n",
       "0         0    0         0        0         0         0         0         0   \n",
       "1         0    0         0        0         0  0.334845         0         0   \n",
       "2         0  0.5         0        0         0         0         0         0   \n",
       "3         0    0         0        0         0         0  0.408248         0   \n",
       "4         0    0         0        0  0.288675         0         0  0.288675   \n",
       "5         0    0  0.377964        0         0         0         0         0   \n",
       "6         0    0         0        0         0         0         0         0   \n",
       "7  0.317517    0         0  0.37351         0         0         0         0   \n",
       "8         0    0         0        0         0  0.457985         0         0   \n",
       "9  0.391176    0         0        0         0         0         0         0   \n",
       "\n",
       "         做  准备金率  ...        藏药        西藏        试点        调控        销售  \\\n",
       "0        0     0  ...         0         0  0.460158         0         0   \n",
       "1        0     0  ...         0         0         0         0         0   \n",
       "2        0   0.5  ...         0         0         0         0         0   \n",
       "3        0     0  ...         0         0         0  0.408248         0   \n",
       "4  0.57735     0  ...  0.288675  0.288675         0         0         0   \n",
       "5        0     0  ...         0         0         0         0         0   \n",
       "6        0     0  ...         0         0         0         0         0   \n",
       "7        0     0  ...         0         0         0         0         0   \n",
       "8        0     0  ...         0         0         0         0         0   \n",
       "9        0     0  ...         0         0         0         0  0.460158   \n",
       "\n",
       "          降        限购        项目        首日    高  \n",
       "0         0         0         0         0    0  \n",
       "1         0         0         0         0    0  \n",
       "2         0         0         0         0  0.5  \n",
       "3         0         0         0         0    0  \n",
       "4         0         0         0         0    0  \n",
       "5         0         0  0.377964         0    0  \n",
       "6         0  0.305902         0  0.359846    0  \n",
       "7         0  0.317517         0         0    0  \n",
       "8         0         0         0         0    0  \n",
       "9  0.460158         0         0         0    0  \n",
       "\n",
       "[10 rows x 58 columns]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfTransformer\n",
    "\n",
    "tfidf_transformer = TfidfTransformer(norm='l2').fit(\n",
    "    bag_words)  ##使用L2范数，并fit模型（如计算IDF等）\n",
    "tfidf_words = tfidf_transformer.transform(bag_words)  ##变换数据\n",
    "tfidf_words_df = pd.DataFrame.sparse.from_spmatrix(tfidf_words,\n",
    "                                                   columns=words_names)\n",
    "tfidf_words_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "关于Scikit-Learn中词袋、TF-IDF计算的具体文档，可以查看：https://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html\n",
    "\n",
    "除了将句子转换为向量，TF-IDF还有很多其他用法，比如：\n",
    "\n",
    "* 使用TF-IDF降维：删掉TF-IDF比较小的词：林建浩等（2021）\n",
    "* 直接使用TF-IDF作为一个指标：Piotroski, Wong和Zhang（2017）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 词嵌入\n",
    "\n",
    "上面的词袋模型和TF-IDF模型都非常直观，但是有一个缺点，就是维数非常高。我们仅仅使用了10条新闻标题，就得到了59列特征，虽然在存储和计算上我们可以使用稀疏矩阵，但是在分析中，维数太高的模型分析起来总是比较困难的。而**词嵌入**（**word embedding**）的出现很大程度上缓解了这个问题。\n",
    "\n",
    "所谓词嵌入，实际上是把高维空间的向量向低维空间映射的过程。一个经典的算法是谷歌提出的word2vec算法，关于该算法的原理我们再次不再赘述，我们这里主要通过例子来展示如何使用该方法。\n",
    "\n",
    "Python中可以使用Gensim包实现word2vec算法，在使用之前需要先安装：\n",
    "\n",
    "```shell\n",
    "sudo pip3 install gensim\n",
    "```\n",
    "\n",
    "该算法需要提供如下几个信息：\n",
    "\n",
    "* 语料库，对于中文可以提交已经分好词的语料库\n",
    "* window，窗宽，上下文可以联系起来的单词个数\n",
    "* size，输出的词向量的维度，几十到几千都可以\n",
    "* min_count，只有当某个词出现次数大于该数值时才会被加入到模型中。\n",
    "\n",
    "比如，使用以上新闻标题数据，可以训练如下模型：\n",
    "\n",
    "```python\n",
    "import gensim\n",
    "\n",
    "CORPUS=map(tokenize,RAW1['title'])\n",
    "w2v_model=gensim.models.Word2Vec(CORPUS, window=5, size=10, min_count=5)\n",
    "w2v_model.save('word2vec')##后面节省时间，先把模型保存下来\n",
    "```\n",
    "\n",
    "当然在现实中，有的时候我们也会使用其他人已经预训练的模型。其实在本例中，只使用标题信息训练出的模型精度并不好（比如标题中使用了沪就不会使用上海，因而很难侦测到这种相关性）。\n",
    "\n",
    "我们使用1992年到2016年的CSMAR上式公司新闻全文数据训练了不同维度的词向量，保存在“Chinese/word2vec/”文件夹中（由于文件很大，我们在这个文件夹汇总值保留了30维模型的训练结果，更多维数的结果请从百度网盘：https://pan.baidu.com/s/1GkxOSFXlzpMzJ6StYN23Sg \n",
    "提取码：ly9c 下载），其中\"word2vec.py\"为训练代码，由于数据量太大（共有74885004个词汇参与训练），我们不提供该原始数据，如有需要可以从CSMAR下载。\"word2vec**\"位已经训练好的词向量模型，其中\\*\\*为词向量的维度，我们训练了30、100、300、500等不同维度。如果需要导入预训练的模型，可以直接使用load函数："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gensim\n",
    "\n",
    "w2v_model = gensim.models.Word2Vec.load(\n",
    "    './Chinese/word2vec/word2vec30')  ##节省时间，直接导入预训练的模型\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "训练模型的具体语法和解释可以查看：https://radimrehurek.com/gensim/models/word2vec.html#module-gensim.models.word2vec\n",
    "\n",
    "有了模型后，我们可以查看每个词对应的向量："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "贵州茅台：\n",
      " [-2.6204739  -0.23999904 -0.4501495  -4.7471104  -2.6559854   6.1215234\n",
      " -1.19105     0.08704755  4.4575567  -5.9783154   1.6476333  -2.9682255\n",
      "  0.05026169 -2.8163173   3.1344664  -2.513116   -2.3200655  -6.645488\n",
      "  1.7873464  -0.28822428 -0.41796866  5.3326087  -1.7870761   0.31191105\n",
      " -2.4249952  -8.8343725   0.67447054  1.6748495  -2.1062326  -5.243416  ]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(30,)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"贵州茅台：\\n\", w2v_model.wv['贵州茅台'])\n",
    "w2v_model.wv['贵州茅台'].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "也可以查看跟某些词最相关的词："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "上海： [('广州', 0.8821085691452026), ('北京', 0.8780174255371094), ('深圳', 0.8698581457138062), ('天津', 0.8337386250495911), ('上海浦东', 0.8273128271102905), ('浦东', 0.7818915843963623), ('成都', 0.7796351909637451), ('虹口', 0.7682737708091736), ('重庆', 0.7635040283203125), ('穗三大', 0.7591032385826111)]\n",
      "贵州茅台： [('洋河股份', 0.9588034152984619), ('山西汾酒', 0.9121654629707336), ('张裕a', 0.9066106081008911), ('g茅台', 0.9013229012489319), ('古井贡酒', 0.8815901875495911), ('伊利股份', 0.8797950744628906), ('金种子酒', 0.8770318627357483), ('老白干酒', 0.8755485415458679), ('泸州老窖', 0.8753379583358765), ('g五粮液', 0.8694736361503601)]\n"
     ]
    }
   ],
   "source": [
    "print(\"上海：\", w2v_model.wv.most_similar(positive=['上海']))\n",
    "print(\"贵州茅台：\", w2v_model.wv.most_similar(positive=['贵州茅台']))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "或者跟某个词的向量的相反数最相关的（与$-1\\times$上海 最相关的）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "上海： [('仅当', 0.6410440802574158), ('恕', 0.62154620885849), ('杜晓锋', 0.6094518303871155), ('幸敬华', 0.6002947092056274), ('毛惟德', 0.5936870574951172), ('蓝保湾', 0.5931683778762817), ('五年制', 0.592823326587677), ('资本保全', 0.585964560508728), ('杜干', 0.5858868360519409), ('质量规划', 0.5837689638137817)]\n",
      "贵州茅台： [('先筑底', 0.7058586478233337), ('法定清算', 0.6609489321708679), ('经援', 0.6546811461448669), ('下除', 0.652676522731781), ('动产担保', 0.6491249203681946), ('外部边界', 0.6484469771385193), ('混合贷款', 0.6440901160240173), ('国际金融组织贷款', 0.6322882771492004), ('所在国', 0.629910409450531), ('区域信用', 0.6297281384468079)]\n"
     ]
    }
   ],
   "source": [
    "print(\"上海：\", w2v_model.wv.most_similar(negative=['上海']))\n",
    "print(\"贵州茅台：\", w2v_model.wv.most_similar(negative=['贵州茅台']))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "或者查看：上海+杭州-北京=？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "上海+杭州-北京=\n",
      "[('宁波', 0.8586423993110657), ('广州', 0.8504917621612549), ('东莞', 0.822361409664154), ('南京', 0.8192954063415527), ('厦门', 0.8146063089370728), ('佛山', 0.8060692548751831), ('南一', 0.7999212741851807), ('溧阳', 0.7996323108673096), ('成都', 0.7981356382369995), ('浙江', 0.7962439656257629)]\n",
      "上海+北京-杭州=\n",
      "[('上海浦东', 0.7623593807220459), ('上海综合保税区', 0.7335734367370605), ('深圳', 0.7314644455909729), ('e-cbd', 0.7249362468719482), ('京', 0.7226858139038086), ('辐射式', 0.7162173986434937), ('浦东', 0.7120426893234253), ('三地', 0.7035841345787048), ('徐雯', 0.7003939747810364), ('盛汇', 0.6990734338760376)]\n"
     ]
    }
   ],
   "source": [
    "print(\"上海+杭州-北京=\\n%s\" %\n",
    "      w2v_model.wv.most_similar(positive=['上海', '杭州'], negative=['北京']))\n",
    "print(\"上海+北京-杭州=\\n%s\" %\n",
    "      w2v_model.wv.most_similar(positive=['上海', '北京'], negative=['杭州']))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "当然，直接看两个词的相似度（词向量的相关系数）也是可以的："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "上海,北京的相似度= 0.8780\n",
      "上海,日本的相似度= -0.0521\n",
      "北京,日本的相似度= 0.5247\n",
      "上海,人民币的相似度= 0.1460\n",
      "人民币,北京的相似度= 0.0441\n",
      "贵州茅台,比亚迪的相似度= 0.3611\n",
      "吉利汽车,沃尔沃的相似度= 0.8491\n",
      "华为,中兴的相似度= 0.8539\n"
     ]
    }
   ],
   "source": [
    "print(\"%s,%s的相似度= %.4f\" % (\"上海\", \"北京\", w2v_model.wv.similarity(\"上海\", \"北京\")))\n",
    "print(\"%s,%s的相似度= %.4f\" % (\"上海\", \"日本\", w2v_model.wv.similarity(\"上海\", \"日本\")))\n",
    "print(\"%s,%s的相似度= %.4f\" % (\"北京\", \"日本\", w2v_model.wv.similarity(\"医药\", \"医疗\")))\n",
    "print(\"%s,%s的相似度= %.4f\" % (\"上海\", \"人民币\", w2v_model.wv.similarity(\"上海\", \"人民币\")))\n",
    "print(\"%s,%s的相似度= %.4f\" % (\"人民币\", \"北京\", w2v_model.wv.similarity(\"人民币\", \"北京\")))\n",
    "print(\"%s,%s的相似度= %.4f\" %\n",
    "      (\"贵州茅台\", \"比亚迪\", w2v_model.wv.similarity(\"贵州茅台\", \"比亚迪\")))\n",
    "print(\"%s,%s的相似度= %.4f\" %\n",
    "      (\"吉利汽车\", \"沃尔沃\", w2v_model.wv.similarity(\"吉利汽车\", \"沃尔沃\")))\n",
    "print(\"%s,%s的相似度= %.4f\" % (\"华为\", \"中兴\", w2v_model.wv.similarity(\"华为\", \"中兴\")))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然而，以上算法仅仅计算了每个词的向量，我们数据中的却是文档，因而需要将词向量加总成文档向量。为此，我们可以通过平均的方式求文档的向量："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['沪', '增值税', '扩围', '改革', '试点'],\n",
       " ['周小川', '外部', '施压', '影响', '人民币', '升值', '步伐'],\n",
       " ['准备金率', '上调', '创新', '高'],\n",
       " ['定基', '价格指数', '若涨', '政府', '出手', '调控'],\n",
       " ['政策', '倾斜', '加大', '投入', '西藏', '做', '做', '强', '藏药', '产业'],\n",
       " ['新疆', '地州', '探矿权', '项目', '涉嫌', '圈', '不探'],\n",
       " ['北京', '楼市', '限购', '首日', '成交', '环', '比降', '成'],\n",
       " ['上海', '房管局', '发布', '沪', '九条', '限购', '执行', '细则'],\n",
       " ['人民币', '升值', '容忍度', '提高'],\n",
       " ['上海', '商品住宅', '销售', '降', '四成']]"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenized_data = map(tokenize, data['title'])\n",
    "tokenized_data = list(tokenized_data)\n",
    "tokenized_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "      <th>24</th>\n",
       "      <th>25</th>\n",
       "      <th>26</th>\n",
       "      <th>27</th>\n",
       "      <th>28</th>\n",
       "      <th>29</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-2.297971</td>\n",
       "      <td>-3.161334</td>\n",
       "      <td>2.866893</td>\n",
       "      <td>-0.996126</td>\n",
       "      <td>0.284920</td>\n",
       "      <td>1.531972</td>\n",
       "      <td>0.906654</td>\n",
       "      <td>-4.272559</td>\n",
       "      <td>1.360739</td>\n",
       "      <td>-7.145058</td>\n",
       "      <td>...</td>\n",
       "      <td>0.618608</td>\n",
       "      <td>-2.027417</td>\n",
       "      <td>0.354830</td>\n",
       "      <td>-5.920053</td>\n",
       "      <td>2.606006</td>\n",
       "      <td>0.943722</td>\n",
       "      <td>0.710307</td>\n",
       "      <td>3.250579</td>\n",
       "      <td>4.710419</td>\n",
       "      <td>-0.751039</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.090696</td>\n",
       "      <td>-2.051241</td>\n",
       "      <td>-0.290815</td>\n",
       "      <td>-1.770225</td>\n",
       "      <td>-0.407113</td>\n",
       "      <td>-4.449056</td>\n",
       "      <td>0.354794</td>\n",
       "      <td>-1.708066</td>\n",
       "      <td>-2.584469</td>\n",
       "      <td>-3.557123</td>\n",
       "      <td>...</td>\n",
       "      <td>-2.584700</td>\n",
       "      <td>-3.989273</td>\n",
       "      <td>3.316375</td>\n",
       "      <td>0.735364</td>\n",
       "      <td>4.073279</td>\n",
       "      <td>1.923687</td>\n",
       "      <td>0.489768</td>\n",
       "      <td>2.243266</td>\n",
       "      <td>-2.430693</td>\n",
       "      <td>-3.075062</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-2.476211</td>\n",
       "      <td>0.077883</td>\n",
       "      <td>2.481291</td>\n",
       "      <td>-3.441597</td>\n",
       "      <td>0.833566</td>\n",
       "      <td>-4.324602</td>\n",
       "      <td>-1.038114</td>\n",
       "      <td>1.693192</td>\n",
       "      <td>1.889202</td>\n",
       "      <td>-8.951756</td>\n",
       "      <td>...</td>\n",
       "      <td>-1.262578</td>\n",
       "      <td>-2.274254</td>\n",
       "      <td>2.063563</td>\n",
       "      <td>1.068579</td>\n",
       "      <td>3.879284</td>\n",
       "      <td>0.288383</td>\n",
       "      <td>2.252155</td>\n",
       "      <td>2.311087</td>\n",
       "      <td>-0.089803</td>\n",
       "      <td>-3.682970</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-1.604671</td>\n",
       "      <td>0.437567</td>\n",
       "      <td>0.390226</td>\n",
       "      <td>-1.791509</td>\n",
       "      <td>-0.129578</td>\n",
       "      <td>0.469662</td>\n",
       "      <td>-1.043747</td>\n",
       "      <td>-0.333066</td>\n",
       "      <td>0.313937</td>\n",
       "      <td>-4.538773</td>\n",
       "      <td>...</td>\n",
       "      <td>-1.362519</td>\n",
       "      <td>-0.549756</td>\n",
       "      <td>-1.546997</td>\n",
       "      <td>0.896517</td>\n",
       "      <td>5.921604</td>\n",
       "      <td>0.422985</td>\n",
       "      <td>0.351355</td>\n",
       "      <td>-0.108700</td>\n",
       "      <td>-1.117601</td>\n",
       "      <td>0.397091</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-2.415386</td>\n",
       "      <td>-1.761841</td>\n",
       "      <td>-0.055961</td>\n",
       "      <td>-1.782646</td>\n",
       "      <td>-2.699580</td>\n",
       "      <td>-1.282648</td>\n",
       "      <td>-0.357748</td>\n",
       "      <td>0.125751</td>\n",
       "      <td>5.077182</td>\n",
       "      <td>-2.138223</td>\n",
       "      <td>...</td>\n",
       "      <td>0.570974</td>\n",
       "      <td>1.046115</td>\n",
       "      <td>-0.687799</td>\n",
       "      <td>1.032049</td>\n",
       "      <td>4.162136</td>\n",
       "      <td>3.645957</td>\n",
       "      <td>2.473218</td>\n",
       "      <td>-0.688791</td>\n",
       "      <td>-0.590855</td>\n",
       "      <td>-1.642059</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>-2.042580</td>\n",
       "      <td>-0.545288</td>\n",
       "      <td>-2.933997</td>\n",
       "      <td>0.435380</td>\n",
       "      <td>-1.932752</td>\n",
       "      <td>2.237168</td>\n",
       "      <td>-0.882077</td>\n",
       "      <td>-1.852991</td>\n",
       "      <td>0.445002</td>\n",
       "      <td>1.514655</td>\n",
       "      <td>...</td>\n",
       "      <td>0.583058</td>\n",
       "      <td>0.847895</td>\n",
       "      <td>-0.823864</td>\n",
       "      <td>0.188446</td>\n",
       "      <td>0.955781</td>\n",
       "      <td>-0.026733</td>\n",
       "      <td>-2.018269</td>\n",
       "      <td>-0.606821</td>\n",
       "      <td>2.170595</td>\n",
       "      <td>2.485991</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>-0.136633</td>\n",
       "      <td>-1.237760</td>\n",
       "      <td>-1.183110</td>\n",
       "      <td>-0.815889</td>\n",
       "      <td>1.367206</td>\n",
       "      <td>1.674132</td>\n",
       "      <td>-6.512691</td>\n",
       "      <td>-1.017015</td>\n",
       "      <td>0.348660</td>\n",
       "      <td>-4.619732</td>\n",
       "      <td>...</td>\n",
       "      <td>-2.668415</td>\n",
       "      <td>1.663260</td>\n",
       "      <td>-3.336064</td>\n",
       "      <td>-0.116358</td>\n",
       "      <td>3.205699</td>\n",
       "      <td>-4.671846</td>\n",
       "      <td>2.095880</td>\n",
       "      <td>3.111485</td>\n",
       "      <td>-1.464048</td>\n",
       "      <td>0.708383</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.767114</td>\n",
       "      <td>2.096215</td>\n",
       "      <td>0.746133</td>\n",
       "      <td>0.287825</td>\n",
       "      <td>2.045303</td>\n",
       "      <td>2.857263</td>\n",
       "      <td>-2.008423</td>\n",
       "      <td>-2.189678</td>\n",
       "      <td>0.697154</td>\n",
       "      <td>-3.994444</td>\n",
       "      <td>...</td>\n",
       "      <td>0.495719</td>\n",
       "      <td>-0.121336</td>\n",
       "      <td>-2.238388</td>\n",
       "      <td>-3.878916</td>\n",
       "      <td>4.265755</td>\n",
       "      <td>-1.114963</td>\n",
       "      <td>2.158192</td>\n",
       "      <td>2.761949</td>\n",
       "      <td>1.457491</td>\n",
       "      <td>2.958742</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>-3.401513</td>\n",
       "      <td>-0.506430</td>\n",
       "      <td>0.380262</td>\n",
       "      <td>-2.048207</td>\n",
       "      <td>1.514004</td>\n",
       "      <td>-5.812137</td>\n",
       "      <td>-1.194896</td>\n",
       "      <td>-0.759125</td>\n",
       "      <td>-1.582029</td>\n",
       "      <td>-5.668441</td>\n",
       "      <td>...</td>\n",
       "      <td>-2.411451</td>\n",
       "      <td>-7.510658</td>\n",
       "      <td>2.072422</td>\n",
       "      <td>2.958090</td>\n",
       "      <td>2.926072</td>\n",
       "      <td>1.766376</td>\n",
       "      <td>0.580327</td>\n",
       "      <td>0.644158</td>\n",
       "      <td>-1.897927</td>\n",
       "      <td>-4.794060</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>-1.284938</td>\n",
       "      <td>0.939088</td>\n",
       "      <td>-3.202075</td>\n",
       "      <td>0.634389</td>\n",
       "      <td>2.169413</td>\n",
       "      <td>3.406785</td>\n",
       "      <td>-5.344977</td>\n",
       "      <td>-0.291755</td>\n",
       "      <td>0.007644</td>\n",
       "      <td>-3.117552</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.304967</td>\n",
       "      <td>-0.849428</td>\n",
       "      <td>-5.568500</td>\n",
       "      <td>2.012764</td>\n",
       "      <td>2.936881</td>\n",
       "      <td>-1.867398</td>\n",
       "      <td>2.681815</td>\n",
       "      <td>1.888706</td>\n",
       "      <td>-1.198099</td>\n",
       "      <td>-2.365074</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 30 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         0         1         2         3         4         5         6   \\\n",
       "0 -2.297971 -3.161334  2.866893 -0.996126  0.284920  1.531972  0.906654   \n",
       "1  0.090696 -2.051241 -0.290815 -1.770225 -0.407113 -4.449056  0.354794   \n",
       "2 -2.476211  0.077883  2.481291 -3.441597  0.833566 -4.324602 -1.038114   \n",
       "3 -1.604671  0.437567  0.390226 -1.791509 -0.129578  0.469662 -1.043747   \n",
       "4 -2.415386 -1.761841 -0.055961 -1.782646 -2.699580 -1.282648 -0.357748   \n",
       "5 -2.042580 -0.545288 -2.933997  0.435380 -1.932752  2.237168 -0.882077   \n",
       "6 -0.136633 -1.237760 -1.183110 -0.815889  1.367206  1.674132 -6.512691   \n",
       "7  0.767114  2.096215  0.746133  0.287825  2.045303  2.857263 -2.008423   \n",
       "8 -3.401513 -0.506430  0.380262 -2.048207  1.514004 -5.812137 -1.194896   \n",
       "9 -1.284938  0.939088 -3.202075  0.634389  2.169413  3.406785 -5.344977   \n",
       "\n",
       "         7         8         9   ...        20        21        22        23  \\\n",
       "0 -4.272559  1.360739 -7.145058  ...  0.618608 -2.027417  0.354830 -5.920053   \n",
       "1 -1.708066 -2.584469 -3.557123  ... -2.584700 -3.989273  3.316375  0.735364   \n",
       "2  1.693192  1.889202 -8.951756  ... -1.262578 -2.274254  2.063563  1.068579   \n",
       "3 -0.333066  0.313937 -4.538773  ... -1.362519 -0.549756 -1.546997  0.896517   \n",
       "4  0.125751  5.077182 -2.138223  ...  0.570974  1.046115 -0.687799  1.032049   \n",
       "5 -1.852991  0.445002  1.514655  ...  0.583058  0.847895 -0.823864  0.188446   \n",
       "6 -1.017015  0.348660 -4.619732  ... -2.668415  1.663260 -3.336064 -0.116358   \n",
       "7 -2.189678  0.697154 -3.994444  ...  0.495719 -0.121336 -2.238388 -3.878916   \n",
       "8 -0.759125 -1.582029 -5.668441  ... -2.411451 -7.510658  2.072422  2.958090   \n",
       "9 -0.291755  0.007644 -3.117552  ... -0.304967 -0.849428 -5.568500  2.012764   \n",
       "\n",
       "         24        25        26        27        28        29  \n",
       "0  2.606006  0.943722  0.710307  3.250579  4.710419 -0.751039  \n",
       "1  4.073279  1.923687  0.489768  2.243266 -2.430693 -3.075062  \n",
       "2  3.879284  0.288383  2.252155  2.311087 -0.089803 -3.682970  \n",
       "3  5.921604  0.422985  0.351355 -0.108700 -1.117601  0.397091  \n",
       "4  4.162136  3.645957  2.473218 -0.688791 -0.590855 -1.642059  \n",
       "5  0.955781 -0.026733 -2.018269 -0.606821  2.170595  2.485991  \n",
       "6  3.205699 -4.671846  2.095880  3.111485 -1.464048  0.708383  \n",
       "7  4.265755 -1.114963  2.158192  2.761949  1.457491  2.958742  \n",
       "8  2.926072  1.766376  0.580327  0.644158 -1.897927 -4.794060  \n",
       "9  2.936881 -1.867398  2.681815  1.888706 -1.198099 -2.365074  \n",
       "\n",
       "[10 rows x 30 columns]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## 知识提要：闭包\n",
    "def mean_vector(model):\n",
    "\n",
    "    def mean_vector_compute(sentence):\n",
    "        n_w = 0\n",
    "        for w in sentence:\n",
    "            if w in model.wv:\n",
    "                try:\n",
    "                    mv += model.wv[w]\n",
    "                except:\n",
    "                    mv = model.wv[w].copy()\n",
    "                n_w += 1\n",
    "        mv /= n_w\n",
    "        return mv\n",
    "\n",
    "    return mean_vector_compute\n",
    "\n",
    "\n",
    "mv_compute = mean_vector(w2v_model)\n",
    "mean_vecs = map(mv_compute, tokenized_data)\n",
    "mean_vecs = pd.DataFrame(mean_vecs)\n",
    "mean_vecs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "或者我们可以使用TF-IDF进行加权："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/Anaconda/lib/python3.11/site-packages/sklearn/feature_extraction/text.py:521: UserWarning: The parameter 'token_pattern' will not be used since 'tokenizer' is not None'\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "      <th>24</th>\n",
       "      <th>25</th>\n",
       "      <th>26</th>\n",
       "      <th>27</th>\n",
       "      <th>28</th>\n",
       "      <th>29</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.148581</td>\n",
       "      <td>-0.036319</td>\n",
       "      <td>-0.018068</td>\n",
       "      <td>0.009148</td>\n",
       "      <td>0.061901</td>\n",
       "      <td>0.120478</td>\n",
       "      <td>0.000085</td>\n",
       "      <td>-0.216511</td>\n",
       "      <td>0.146551</td>\n",
       "      <td>-0.333161</td>\n",
       "      <td>...</td>\n",
       "      <td>0.046125</td>\n",
       "      <td>-0.019234</td>\n",
       "      <td>0.038737</td>\n",
       "      <td>-0.305739</td>\n",
       "      <td>0.201816</td>\n",
       "      <td>0.137147</td>\n",
       "      <td>0.045861</td>\n",
       "      <td>0.066238</td>\n",
       "      <td>0.124265</td>\n",
       "      <td>0.169765</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.095334</td>\n",
       "      <td>-0.339919</td>\n",
       "      <td>-0.069402</td>\n",
       "      <td>0.052371</td>\n",
       "      <td>-0.282315</td>\n",
       "      <td>0.030940</td>\n",
       "      <td>-0.090470</td>\n",
       "      <td>-0.162104</td>\n",
       "      <td>0.081100</td>\n",
       "      <td>-0.100262</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.036592</td>\n",
       "      <td>-0.020217</td>\n",
       "      <td>0.274612</td>\n",
       "      <td>0.158418</td>\n",
       "      <td>0.227180</td>\n",
       "      <td>0.140554</td>\n",
       "      <td>-0.001166</td>\n",
       "      <td>0.191003</td>\n",
       "      <td>-0.091173</td>\n",
       "      <td>-0.195360</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.101324</td>\n",
       "      <td>-0.059106</td>\n",
       "      <td>0.015652</td>\n",
       "      <td>0.015128</td>\n",
       "      <td>0.030852</td>\n",
       "      <td>-0.377617</td>\n",
       "      <td>-0.230746</td>\n",
       "      <td>0.177777</td>\n",
       "      <td>0.244334</td>\n",
       "      <td>-0.295227</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.030645</td>\n",
       "      <td>-0.046572</td>\n",
       "      <td>-0.194047</td>\n",
       "      <td>0.151977</td>\n",
       "      <td>0.000971</td>\n",
       "      <td>0.041661</td>\n",
       "      <td>0.033858</td>\n",
       "      <td>-0.158417</td>\n",
       "      <td>-0.074691</td>\n",
       "      <td>-0.223420</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.034708</td>\n",
       "      <td>0.109602</td>\n",
       "      <td>0.150284</td>\n",
       "      <td>-0.061393</td>\n",
       "      <td>-0.019534</td>\n",
       "      <td>-0.011336</td>\n",
       "      <td>-0.143899</td>\n",
       "      <td>-0.011558</td>\n",
       "      <td>0.081243</td>\n",
       "      <td>-0.252050</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.078418</td>\n",
       "      <td>0.018511</td>\n",
       "      <td>-0.043188</td>\n",
       "      <td>0.082479</td>\n",
       "      <td>0.605613</td>\n",
       "      <td>0.021741</td>\n",
       "      <td>0.265413</td>\n",
       "      <td>0.100253</td>\n",
       "      <td>0.098343</td>\n",
       "      <td>0.056790</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.004909</td>\n",
       "      <td>-0.309575</td>\n",
       "      <td>0.045406</td>\n",
       "      <td>0.192595</td>\n",
       "      <td>-0.310284</td>\n",
       "      <td>0.184387</td>\n",
       "      <td>-0.032352</td>\n",
       "      <td>-0.138226</td>\n",
       "      <td>0.227460</td>\n",
       "      <td>-0.116081</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.123986</td>\n",
       "      <td>0.069369</td>\n",
       "      <td>0.011410</td>\n",
       "      <td>0.055064</td>\n",
       "      <td>0.134197</td>\n",
       "      <td>0.362929</td>\n",
       "      <td>0.153390</td>\n",
       "      <td>-0.150784</td>\n",
       "      <td>0.022146</td>\n",
       "      <td>-0.203349</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>-0.217748</td>\n",
       "      <td>-0.081696</td>\n",
       "      <td>-0.159856</td>\n",
       "      <td>0.023670</td>\n",
       "      <td>-0.139118</td>\n",
       "      <td>0.087813</td>\n",
       "      <td>0.286302</td>\n",
       "      <td>-0.192930</td>\n",
       "      <td>0.070609</td>\n",
       "      <td>0.171712</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.153864</td>\n",
       "      <td>0.171508</td>\n",
       "      <td>-0.301370</td>\n",
       "      <td>-0.072271</td>\n",
       "      <td>0.378945</td>\n",
       "      <td>0.070726</td>\n",
       "      <td>-0.184511</td>\n",
       "      <td>-0.074120</td>\n",
       "      <td>0.327844</td>\n",
       "      <td>0.325883</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>-0.188053</td>\n",
       "      <td>-0.365733</td>\n",
       "      <td>0.128107</td>\n",
       "      <td>-0.058815</td>\n",
       "      <td>0.112551</td>\n",
       "      <td>0.065304</td>\n",
       "      <td>-0.214734</td>\n",
       "      <td>0.087087</td>\n",
       "      <td>0.064707</td>\n",
       "      <td>-0.258617</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.026033</td>\n",
       "      <td>0.340718</td>\n",
       "      <td>-0.110482</td>\n",
       "      <td>0.069005</td>\n",
       "      <td>0.029456</td>\n",
       "      <td>-0.129096</td>\n",
       "      <td>-0.312034</td>\n",
       "      <td>-0.101793</td>\n",
       "      <td>-0.217690</td>\n",
       "      <td>-0.227760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.113986</td>\n",
       "      <td>0.248894</td>\n",
       "      <td>0.027087</td>\n",
       "      <td>0.231291</td>\n",
       "      <td>-0.032114</td>\n",
       "      <td>0.048062</td>\n",
       "      <td>-0.143418</td>\n",
       "      <td>0.060145</td>\n",
       "      <td>0.016227</td>\n",
       "      <td>-0.305819</td>\n",
       "      <td>...</td>\n",
       "      <td>0.223890</td>\n",
       "      <td>0.143665</td>\n",
       "      <td>0.066184</td>\n",
       "      <td>-0.426862</td>\n",
       "      <td>0.333553</td>\n",
       "      <td>-0.009735</td>\n",
       "      <td>-0.001862</td>\n",
       "      <td>0.181096</td>\n",
       "      <td>0.035173</td>\n",
       "      <td>0.060070</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>-0.101069</td>\n",
       "      <td>0.166036</td>\n",
       "      <td>0.097794</td>\n",
       "      <td>-0.053011</td>\n",
       "      <td>0.014484</td>\n",
       "      <td>-0.277415</td>\n",
       "      <td>-0.094133</td>\n",
       "      <td>0.043053</td>\n",
       "      <td>0.205265</td>\n",
       "      <td>-0.320228</td>\n",
       "      <td>...</td>\n",
       "      <td>0.092974</td>\n",
       "      <td>-0.387319</td>\n",
       "      <td>-0.145867</td>\n",
       "      <td>0.229892</td>\n",
       "      <td>0.168001</td>\n",
       "      <td>0.228359</td>\n",
       "      <td>0.145979</td>\n",
       "      <td>-0.175168</td>\n",
       "      <td>-0.121800</td>\n",
       "      <td>-0.119020</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>-0.175843</td>\n",
       "      <td>-0.196618</td>\n",
       "      <td>-0.361817</td>\n",
       "      <td>0.185863</td>\n",
       "      <td>0.014346</td>\n",
       "      <td>0.061425</td>\n",
       "      <td>-0.123399</td>\n",
       "      <td>0.137843</td>\n",
       "      <td>-0.129746</td>\n",
       "      <td>-0.283890</td>\n",
       "      <td>...</td>\n",
       "      <td>0.174996</td>\n",
       "      <td>0.088702</td>\n",
       "      <td>-0.223863</td>\n",
       "      <td>0.004097</td>\n",
       "      <td>-0.065761</td>\n",
       "      <td>-0.478842</td>\n",
       "      <td>0.188946</td>\n",
       "      <td>0.146038</td>\n",
       "      <td>-0.237623</td>\n",
       "      <td>-0.299823</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 30 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         0         1         2         3         4         5         6   \\\n",
       "0 -0.148581 -0.036319 -0.018068  0.009148  0.061901  0.120478  0.000085   \n",
       "1  0.095334 -0.339919 -0.069402  0.052371 -0.282315  0.030940 -0.090470   \n",
       "2 -0.101324 -0.059106  0.015652  0.015128  0.030852 -0.377617 -0.230746   \n",
       "3 -0.034708  0.109602  0.150284 -0.061393 -0.019534 -0.011336 -0.143899   \n",
       "4  0.004909 -0.309575  0.045406  0.192595 -0.310284  0.184387 -0.032352   \n",
       "5 -0.217748 -0.081696 -0.159856  0.023670 -0.139118  0.087813  0.286302   \n",
       "6 -0.188053 -0.365733  0.128107 -0.058815  0.112551  0.065304 -0.214734   \n",
       "7  0.113986  0.248894  0.027087  0.231291 -0.032114  0.048062 -0.143418   \n",
       "8 -0.101069  0.166036  0.097794 -0.053011  0.014484 -0.277415 -0.094133   \n",
       "9 -0.175843 -0.196618 -0.361817  0.185863  0.014346  0.061425 -0.123399   \n",
       "\n",
       "         7         8         9   ...        20        21        22        23  \\\n",
       "0 -0.216511  0.146551 -0.333161  ...  0.046125 -0.019234  0.038737 -0.305739   \n",
       "1 -0.162104  0.081100 -0.100262  ... -0.036592 -0.020217  0.274612  0.158418   \n",
       "2  0.177777  0.244334 -0.295227  ... -0.030645 -0.046572 -0.194047  0.151977   \n",
       "3 -0.011558  0.081243 -0.252050  ... -0.078418  0.018511 -0.043188  0.082479   \n",
       "4 -0.138226  0.227460 -0.116081  ... -0.123986  0.069369  0.011410  0.055064   \n",
       "5 -0.192930  0.070609  0.171712  ... -0.153864  0.171508 -0.301370 -0.072271   \n",
       "6  0.087087  0.064707 -0.258617  ... -0.026033  0.340718 -0.110482  0.069005   \n",
       "7  0.060145  0.016227 -0.305819  ...  0.223890  0.143665  0.066184 -0.426862   \n",
       "8  0.043053  0.205265 -0.320228  ...  0.092974 -0.387319 -0.145867  0.229892   \n",
       "9  0.137843 -0.129746 -0.283890  ...  0.174996  0.088702 -0.223863  0.004097   \n",
       "\n",
       "         24        25        26        27        28        29  \n",
       "0  0.201816  0.137147  0.045861  0.066238  0.124265  0.169765  \n",
       "1  0.227180  0.140554 -0.001166  0.191003 -0.091173 -0.195360  \n",
       "2  0.000971  0.041661  0.033858 -0.158417 -0.074691 -0.223420  \n",
       "3  0.605613  0.021741  0.265413  0.100253  0.098343  0.056790  \n",
       "4  0.134197  0.362929  0.153390 -0.150784  0.022146 -0.203349  \n",
       "5  0.378945  0.070726 -0.184511 -0.074120  0.327844  0.325883  \n",
       "6  0.029456 -0.129096 -0.312034 -0.101793 -0.217690 -0.227760  \n",
       "7  0.333553 -0.009735 -0.001862  0.181096  0.035173  0.060070  \n",
       "8  0.168001  0.228359  0.145979 -0.175168 -0.121800 -0.119020  \n",
       "9 -0.065761 -0.478842  0.188946  0.146038 -0.237623 -0.299823  \n",
       "\n",
       "[10 rows x 30 columns]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def mean_vector_tfidf_weight(model):\n",
    "\n",
    "    def mean_vector_compute(sentence, tfidf):\n",
    "        for w in sentence:\n",
    "            if w in model.wv:\n",
    "                try:\n",
    "                    mv += model.wv[w] * tfidf[w][0]\n",
    "                except:\n",
    "                    mv = model.wv[w].copy()\n",
    "        ## L2规范化\n",
    "        import numpy as np\n",
    "        mv /= np.linalg.norm(mv)\n",
    "        return mv\n",
    "\n",
    "    return mean_vector_compute\n",
    "\n",
    "\n",
    "tokenized_data = map(tokenize, data['title'])\n",
    "##首先计算词袋\n",
    "count_vect = CountVectorizer(tokenizer=tokenize)\n",
    "bag_words = count_vect.fit_transform(data['title'])\n",
    "words_names = count_vect.get_feature_names_out()\n",
    "##计算TF-IDF\n",
    "from sklearn.feature_extraction.text import TfidfTransformer\n",
    "\n",
    "tfidf_transformer = TfidfTransformer(norm='l2').fit(bag_words)\n",
    "tfidf_words = tfidf_transformer.transform(bag_words)\n",
    "tfidf_words_df = pd.DataFrame.sparse.from_spmatrix(tfidf_words,\n",
    "                                                   columns=words_names)\n",
    "##带入模型\n",
    "tfidf_weight_mean_vec = mean_vector_tfidf_weight(w2v_model)\n",
    "mean_vecs = []\n",
    "for i, sentence in enumerate(tokenized_data):\n",
    "    mean_vecs.append(tfidf_weight_mean_vec(sentence,\n",
    "                                           tfidf_words_df.sparse.to_dense().iloc[i, :]))\n",
    "mean_vecs = pd.DataFrame(mean_vecs)\n",
    "mean_vecs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 文本距离与相似度\n",
    "\n",
    "距离和相似度在文本的概念上都是文本之间某种相似程度的度量，只不过距离（distance）通常用于比较短小的词汇、句子上的差异性有多大，而相似度则主要针对更长的文档等。\n",
    "\n",
    "## 编辑距离\n",
    "\n",
    "经过适当的向量化之后，我们可以轻松地使用向量进行向量空间的任何距离操作，距离越小代表两个字符串之间越相似。在这里我们额外介绍一种常用的距离，即基于编辑距离的Levenshtein距离，该距离度量了从一个字符串str1需要经过多少步的编辑（替换一个字符、插入一个字符、删除一个字符）才能变成另外一个字符串str2。这个步数的计算可以通过动态规划（dynamic programming）来完成。\n",
    "\n",
    "Python中可以安装Levenshtein包：\n",
    "```shell\n",
    "sudo pip3 install python-Levenshtein\n",
    "```\n",
    "\n",
    "计算Levenshtein距离："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import Levenshtein\n",
    "\n",
    "Levenshtein.distance('色即是空，空即是色', '色不异空，空不异色')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "以上给出了最小修改次数，不过最好是将其规范化：$$ratio=\\frac{len\\left(str1\\right)+len\\left(str2\\right)-distance}{len\\left(str1\\right)+len\\left(str2\\right)}$$可以使用Levenstein.ratio()计算该比例："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5555555555555556"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import Levenshtein\n",
    "\n",
    "Levenshtein.ratio('色即是空，空即是色', '色不异空，空不异色')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "编辑距离有很多用处，比如比照不同版本、侦测细微的字符串差异（比如可能存在的地址输入差异「上海松江文汇路」和「上海市松江区文汇路」）、检查抄袭等。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.627906976744186"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import Levenshtein\n",
    "\n",
    "xj_jmls = \"观世音菩萨，行深般若波罗蜜时，照见五阴空，度一切苦厄。舍利弗，色空故，无恼坏相，受空故，无受相，想空故，无知相，行空故，无作相，识空故，无觉相。何以故？舍利弗，非色异空，非空异色，色即是空，空即是色，受想行识，亦复如是。\"\n",
    "xj_xz = \"观自在菩萨，行深般若波罗蜜多时，照见五蕴皆空，度一切苦厄。舍利子，色不异空，空不异色，色即是空，空即是色，受想行识亦复如是。\"\n",
    "Levenshtein.ratio(xj_jmls, xj_xz)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 文本相似度\n",
    "\n",
    "不管是使用词袋模型，还是TF-IDF模型，或者使用词嵌入方法通过平均、加权平均的方式，都可以讲一个文本向量化。\n",
    "\n",
    "将文本向量化之后，计算文本之间的相似度就非常简单了：只要将两个文本的向量之间的相似度计算出来即可，常用的度量是余弦相似度（即两个向量的夹角）：$$cs\\left(u,v\\right)=\\frac{u\\cdot v}{\\left\\Vert u\\right\\Vert \\cdot \\left\\Vert v\\right\\Vert }$$其中分子为内积，分母上位L2范数的乘积。我们可以使用NumPy很快的计算出该值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "cos_similarity = lambda u, v: np.inner(u, v) / (np.linalg.norm(u) * np.linalg.\n",
    "                                                norm(v))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "比如如果我们使用词袋模型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/Anaconda/lib/python3.11/site-packages/sklearn/feature_extraction/text.py:521: UserWarning: The parameter 'token_pattern' will not be used since 'tokenizer' is not None'\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "##计算词袋\n",
    "count_vect = CountVectorizer(tokenizer=tokenize)\n",
    "bag_words = count_vect.fit_transform(data['title'])\n",
    "words_names = count_vect.get_feature_names_out()\n",
    "bag_words_df = pd.DataFrame.sparse.from_spmatrix(bag_words,\n",
    "                                                 columns=words_names)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.15811388300841897"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cos_similarity(bag_words_df.sparse.to_dense().iloc[7, :], bag_words_df.sparse.to_dense().iloc[9, :])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "当然也可以找出于某一个标题最为相似的，内存限制只算前50000个（如果要算所有的可以分开来算）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/Anaconda/lib/python3.11/site-packages/sklearn/feature_extraction/text.py:521: UserWarning: The parameter 'token_pattern' will not be used since 'tokenizer' is not None'\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "沪争取增值税扩围改革试点\n",
      "增值税扩围和资源税改革今年有望取得突破\n"
     ]
    }
   ],
   "source": [
    "##计算词袋\n",
    "count_vect = CountVectorizer(tokenizer=tokenize)\n",
    "bag_words = count_vect.fit_transform(RAW1['title'][:50000])\n",
    "words_names = count_vect.get_feature_names_out()\n",
    "bag_words_df = pd.DataFrame.sparse.from_spmatrix(bag_words,\n",
    "                                                 columns=words_names)\n",
    "##计算相关系数（自己用循环试一下，奇慢无比，如果不向量化计算，估计要跑的时间按天算）\n",
    "ip = np.array(np.dot(bag_words_df, bag_words_df.sparse.to_dense().iloc[0, :]))  ## 这里用到了广播\n",
    "norm1 = np.array(np.linalg.norm(bag_words_df, axis=1))  ## 第一行的norm都一样，所以不用除\n",
    "corr = ip / norm1\n",
    "corr = corr[1:]\n",
    "##找最大值\n",
    "argi = np.argmax(corr)\n",
    "print(RAW1['title'].iloc[0])\n",
    "print(RAW1['title'].iloc[argi + 1])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在其他向量化方法下同理，在此不再赘述。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 文本分类：情感分析\n",
    "\n",
    "情感分析（sentiment analysis）最初指对文本的情感，比如褒义还是贬义，以及文章中的情感倾向进行分析，实际上是文本分类的一种。实际上，我们之前学到过的所有的监督学习方法都可以使用在文本分类中。然而，监督学习需要大量的带有标签的词典，有时这种方法是行不通的，所以也会有根据特定模式对文本进行分类的方法，或者使用预训练模型的方法。比如，一个经常使用的方法是使用情感词典。我们将分别介绍使用情感词典的方法和使用监督学习的方法。\n",
    "\n",
    "## 基于词典\n",
    "\n",
    "情感词典即标记了情感得分的一个词典，这个词典可以看作是一个预训练的模型，模型的训练结果是情感得分，而我们只需要使用这些情感得分就可以得到情感的具体取值。\n",
    "\n",
    "情感词典方法的好处是非常的简单：只需要简单计算得分即可。然而缺点也是非常突出的：情感词典通常不对特殊问题进行优化，此外其标签是固定的。比如，一般情感词典也许会标记正面负面，但是当我们将其用在金融领域时，会发现“降准”这个正向词汇甚至不会出现在情感词典中，虽然这个词在金融领域应该对股票市场是一个正向词汇。\n",
    "\n",
    "此外，不同情境下也许同一个词也有不同的情感倾向，比如如果我们讨论股票，“降准”也许是一个正向词汇，然而如果我们讨论的是债券，“降准”就不见得是什么好词了。\n",
    "\n",
    "当然，情感词典在一般的领域中应用也许也可以达到比较高的精准度。在这里我们先介绍情感词典的使用方法。\n",
    "\n",
    "使用情感词典的第一步是获得情感词典，我们在这里列举了几个比较常用的情感词典：\n",
    "\n",
    "* 清华大学李军中文褒贬义词典（ http://nlp.csai.tsinghua.edu.cn/site2/index.php/13-sms ）（./Chinese/BosonNLP/）\n",
    "* 知网HowNet情感词典（./Chinese/HowNet/）\n",
    "* 玻森公司是情感词典（ http://static.bosonnlp.com/dev/resource ）（./Chinese/Tsinghua/）\n",
    "* 金融领域中文情绪词典（姚加权等，2021）（./Chinese/FinanceSenti）\n",
    "\n",
    "以上词典都可以在括号中的地址，或者括号中的路径里面找到。\n",
    "\n",
    "比如，对于一个句子，我们可以先将其使用词袋模型将其整理为词袋，然后对于每个词都是用其情感值对其进行打分，并处理否定词，最后根据每个词的打分情况汇总为这个句子的情感。\n",
    "\n",
    "比如，以HowNet的情感词典为例，其情感词典的内容大约为："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"Chinese/HowNet/正面情感词语（中文）.txt\", encoding='gb18030') as f:\n",
    "    posilist = f.readlines()\n",
    "del posilist[0]\n",
    "SentDict = {}\n",
    "for w in posilist:\n",
    "    SentDict[w.strip()] = 1\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "以上代码我们将所有的正面词汇给以一个数值1，代表正面，后面我们还将加载负面词汇，用-1代表负面。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "SentDict = {}\n",
    "Files = ['面情感词语（中文）.txt', '面情感词语（英文）.txt', '面评价词语（中文）.txt', '面评价词语（英文）.txt']\n",
    "for p in ['正', '负']:\n",
    "    v = (1 if p == '正' else -1)\n",
    "    for f in Files:\n",
    "        with open(\"Chinese/HowNet/\" + p + f, encoding='gb18030') as f:\n",
    "            posilist = f.readlines()\n",
    "        del posilist[0]\n",
    "        for w in posilist:\n",
    "            if w not in SentDict:\n",
    "                SentDict[w.strip()] = v\n",
    "            else:\n",
    "                SentDict[w.strip()] += v\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来导入否定词："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['no', '不', '没', '无', '非', '莫', '弗', '毋', '未', '否', '别', '无', '不够', '不是', '不可', '不曾', '未必', '没有', '不要', '难以', '未曾', '否认', '取消', '撤回']\n"
     ]
    }
   ],
   "source": [
    "NegaList = []\n",
    "with open(\"Chinese/negative.txt\") as f:\n",
    "    for w in f:\n",
    "        NegaList.append(w.strip())\n",
    "print(NegaList)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们以大众点评的数据作为例子展示情感词典的用法，首先读入数据："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cus_id</th>\n",
       "      <th>comment_time</th>\n",
       "      <th>comment_star</th>\n",
       "      <th>cus_comment</th>\n",
       "      <th>kouwei</th>\n",
       "      <th>huanjing</th>\n",
       "      <th>fuwu</th>\n",
       "      <th>shopID</th>\n",
       "      <th>stars</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>weekday</th>\n",
       "      <th>hour</th>\n",
       "      <th>comment_len</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>迷糊泰迪</td>\n",
       "      <td>2018-09-20 06:48:00</td>\n",
       "      <td>sml-str40</td>\n",
       "      <td>南信 算是 广州 著名 甜品店 吧 好几个 时间段 路过 都 是 座无虚席 看着 餐单 上 ...</td>\n",
       "      <td>非常好</td>\n",
       "      <td>好</td>\n",
       "      <td>好</td>\n",
       "      <td>518986</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2018</td>\n",
       "      <td>9</td>\n",
       "      <td>3</td>\n",
       "      <td>6</td>\n",
       "      <td>184</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>稱霸幼稚園</td>\n",
       "      <td>2018-09-22 21:49:00</td>\n",
       "      <td>sml-str40</td>\n",
       "      <td>中午 吃 完 了 所谓 的 早茶 回去 放下 行李 休息 了 会 就 来 吃 下午茶 了 服...</td>\n",
       "      <td>很好</td>\n",
       "      <td>很好</td>\n",
       "      <td>很好</td>\n",
       "      <td>518986</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2018</td>\n",
       "      <td>9</td>\n",
       "      <td>5</td>\n",
       "      <td>21</td>\n",
       "      <td>266</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>爱吃的美美侠</td>\n",
       "      <td>2018-09-22 22:16:00</td>\n",
       "      <td>sml-str40</td>\n",
       "      <td>冲刺 王者 战队 吃遍 蓉城 战队 有 特权 五月份 和 好 朋友 毕业 旅行 来 了 广州...</td>\n",
       "      <td>很好</td>\n",
       "      <td>很好</td>\n",
       "      <td>很好</td>\n",
       "      <td>518986</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2018</td>\n",
       "      <td>9</td>\n",
       "      <td>5</td>\n",
       "      <td>22</td>\n",
       "      <td>341</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>姜姜会吃胖</td>\n",
       "      <td>2018-09-19 06:36:00</td>\n",
       "      <td>sml-str40</td>\n",
       "      <td>都 说来 广州 吃 糖水 就要 来南信 招牌 姜撞奶 红豆 双皮奶 牛 三星 云吞面 一楼 ...</td>\n",
       "      <td>非常好</td>\n",
       "      <td>很好</td>\n",
       "      <td>很好</td>\n",
       "      <td>518986</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2018</td>\n",
       "      <td>9</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>197</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>forevercage</td>\n",
       "      <td>2018-08-24 17:58:00</td>\n",
       "      <td>sml-str50</td>\n",
       "      <td>一直 很 期待 也 最 爱 吃 甜品 广州 的 甜品 很 丰富 很 多样 来 之前 就 一直...</td>\n",
       "      <td>非常好</td>\n",
       "      <td>很好</td>\n",
       "      <td>很好</td>\n",
       "      <td>518986</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2018</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>17</td>\n",
       "      <td>261</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        cus_id         comment_time comment_star  \\\n",
       "0         迷糊泰迪  2018-09-20 06:48:00    sml-str40   \n",
       "1        稱霸幼稚園  2018-09-22 21:49:00    sml-str40   \n",
       "2       爱吃的美美侠  2018-09-22 22:16:00    sml-str40   \n",
       "3        姜姜会吃胖  2018-09-19 06:36:00    sml-str40   \n",
       "4  forevercage  2018-08-24 17:58:00    sml-str50   \n",
       "\n",
       "                                         cus_comment kouwei huanjing fuwu  \\\n",
       "0  南信 算是 广州 著名 甜品店 吧 好几个 时间段 路过 都 是 座无虚席 看着 餐单 上 ...    非常好        好    好   \n",
       "1  中午 吃 完 了 所谓 的 早茶 回去 放下 行李 休息 了 会 就 来 吃 下午茶 了 服...     很好       很好   很好   \n",
       "2  冲刺 王者 战队 吃遍 蓉城 战队 有 特权 五月份 和 好 朋友 毕业 旅行 来 了 广州...     很好       很好   很好   \n",
       "3  都 说来 广州 吃 糖水 就要 来南信 招牌 姜撞奶 红豆 双皮奶 牛 三星 云吞面 一楼 ...    非常好       很好   很好   \n",
       "4  一直 很 期待 也 最 爱 吃 甜品 广州 的 甜品 很 丰富 很 多样 来 之前 就 一直...    非常好       很好   很好   \n",
       "\n",
       "   shopID  stars  year  month  weekday  hour  comment_len  \n",
       "0  518986    4.0  2018      9        3     6          184  \n",
       "1  518986    4.0  2018      9        5    21          266  \n",
       "2  518986    4.0  2018      9        5    22          341  \n",
       "3  518986    4.0  2018      9        2     6          197  \n",
       "4  518986    5.0  2018      8        4    17          261  "
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "dianping = pd.read_csv(\"csv/dianping.csv\")\n",
    "dianping.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到，cus_comment里面是已经分好词的句子，此外还有详细的评分数据。我们可以写一个评分函数，然后使用map()方法进行评分："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "\n",
    "def score(s):\n",
    "    w_list = str(s).split(' ')\n",
    "    multiplier = 1\n",
    "    senti = 0\n",
    "    for w in w_list:\n",
    "        if w in NegaList:\n",
    "            multiplier *= (-1)\n",
    "        if w in SentDict and w not in NegaList:\n",
    "            senti += (multiplier * SentDict[w])\n",
    "            multiplier = 1\n",
    "    return np.sign(senti)\n",
    "\n",
    "\n",
    "dianping['score'] = dianping['cus_comment'].map(score)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了查看分类效果，简单的可以分类求均值："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"8\" halign=\"left\">score</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>stars</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1.0</th>\n",
       "      <td>797.0</td>\n",
       "      <td>0.225847</td>\n",
       "      <td>0.866277</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>-1.00</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2.0</th>\n",
       "      <td>982.0</td>\n",
       "      <td>0.423625</td>\n",
       "      <td>0.817602</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3.0</th>\n",
       "      <td>5152.0</td>\n",
       "      <td>0.629076</td>\n",
       "      <td>0.689405</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>0.75</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4.0</th>\n",
       "      <td>10849.0</td>\n",
       "      <td>0.787169</td>\n",
       "      <td>0.534707</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5.0</th>\n",
       "      <td>9067.0</td>\n",
       "      <td>0.835999</td>\n",
       "      <td>0.465336</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         score                                              \n",
       "         count      mean       std  min   25%  50%  75%  max\n",
       "stars                                                       \n",
       "1.0      797.0  0.225847  0.866277 -1.0 -1.00  1.0  1.0  1.0\n",
       "2.0      982.0  0.423625  0.817602 -1.0  0.00  1.0  1.0  1.0\n",
       "3.0     5152.0  0.629076  0.689405 -1.0  0.75  1.0  1.0  1.0\n",
       "4.0    10849.0  0.787169  0.534707 -1.0  1.00  1.0  1.0  1.0\n",
       "5.0     9067.0  0.835999  0.465336 -1.0  1.00  1.0  1.0  1.0"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dianping[['stars', 'score']].groupby('stars').describe()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到评分越高，用户的评星也越高，结果可以接受。不过即使是1星，最终的评分平均也是大于0的，所以在使用的时候也可能会出现问题，即很多评分为正的实际上却是负面评价，比如如果我们根据评分计算："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"8\" halign=\"left\">stars</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>score</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>-1</th>\n",
       "      <td>2062.0</td>\n",
       "      <td>3.337536</td>\n",
       "      <td>1.200654</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2766.0</td>\n",
       "      <td>3.755965</td>\n",
       "      <td>1.092184</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>22019.0</td>\n",
       "      <td>4.072710</td>\n",
       "      <td>0.900757</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         stars                                             \n",
       "         count      mean       std  min  25%  50%  75%  max\n",
       "score                                                      \n",
       "-1      2062.0  3.337536  1.200654  1.0  3.0  3.0  4.0  5.0\n",
       " 0      2766.0  3.755965  1.092184  1.0  3.0  4.0  5.0  5.0\n",
       " 1     22019.0  4.072710  0.900757  1.0  4.0  4.0  5.0  5.0"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dianping[['stars', 'score']].groupby('score').describe()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 基于机器学习\n",
    "\n",
    "如果数据中存在标签，我们可以使用之前学过的机器学习方法，以标签作为输出，以我们上面介绍的提取特征的做法作为输入，通过机器学习算法进行文本分类。\n",
    "\n",
    "但是，由于文本数据的高度复杂性和非线性性、高维性，机器学习方法的选择对结果往往有较大影响。一般可以使用结合Lasso等方法的回归（linear/logistic/ordered/count/...），或者使用支持向量机、贝叶斯方法、决策树、随机森林等非线性、非参数的方法。随着神经网络的兴起，也可以通过构建神经网络的方法，结合LSTM、RNN等网络设计方法进行分类。此外，我们还可以使用BERT等预训练模型进行fine-tuning，这种预训练模型允许我们在小样本的条件下达到更好的预测效果。\n",
    "\n",
    "当然，模型是固定的，模型的好坏也很大程度上取决于特征提取的方式以及参数设定。\n",
    "\n",
    "接下来我们对以上豆瓣评分的数据进行一个简单的有监督学习。Gentzkow、Kelly和Taddy（2017）建议在样本量不是特别大的情况下，尽量少引入决策树、支持向量机等非线性方法，所以这里我们选择使用Logistic回归的方法，将评分在4星及以下的设为0，5星设为1，进行一个简单的二分类预测。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "dianping = pd.read_csv(\"csv/dianping.csv\")\n",
    "dianping['y'] = dianping['stars'] > 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来进行一些简单的处理。虽然数据中已经帮我们做好分词，不过我们这里还是重新进行分词，并进行祛除停用词等的操作："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cus_id</th>\n",
       "      <th>comment_time</th>\n",
       "      <th>comment_star</th>\n",
       "      <th>cus_comment</th>\n",
       "      <th>kouwei</th>\n",
       "      <th>huanjing</th>\n",
       "      <th>fuwu</th>\n",
       "      <th>shopID</th>\n",
       "      <th>stars</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>weekday</th>\n",
       "      <th>hour</th>\n",
       "      <th>comment_len</th>\n",
       "      <th>y</th>\n",
       "      <th>len_comment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>迷糊泰迪</td>\n",
       "      <td>2018-09-20 06:48:00</td>\n",
       "      <td>sml-str40</td>\n",
       "      <td>南信算是广州著名甜品店吧好几个时间段路过都是座无虚席看着餐单上密密麻麻满满当当好吃的容易范选...</td>\n",
       "      <td>非常好</td>\n",
       "      <td>好</td>\n",
       "      <td>好</td>\n",
       "      <td>518986</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2018</td>\n",
       "      <td>9</td>\n",
       "      <td>3</td>\n",
       "      <td>6</td>\n",
       "      <td>184</td>\n",
       "      <td>False</td>\n",
       "      <td>97.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>稱霸幼稚園</td>\n",
       "      <td>2018-09-22 21:49:00</td>\n",
       "      <td>sml-str40</td>\n",
       "      <td>中午吃完了所谓的早茶回去放下行李休息了会就来吃下午茶了服务两层楼楼下只能收现金楼上可以微信支...</td>\n",
       "      <td>很好</td>\n",
       "      <td>很好</td>\n",
       "      <td>很好</td>\n",
       "      <td>518986</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2018</td>\n",
       "      <td>9</td>\n",
       "      <td>5</td>\n",
       "      <td>21</td>\n",
       "      <td>266</td>\n",
       "      <td>False</td>\n",
       "      <td>175.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>爱吃的美美侠</td>\n",
       "      <td>2018-09-22 22:16:00</td>\n",
       "      <td>sml-str40</td>\n",
       "      <td>冲刺王者战队吃遍蓉城战队有特权五月份和好朋友毕业旅行来了广州我们都是双皮奶爱好者搜到啦最火的...</td>\n",
       "      <td>很好</td>\n",
       "      <td>很好</td>\n",
       "      <td>很好</td>\n",
       "      <td>518986</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2018</td>\n",
       "      <td>9</td>\n",
       "      <td>5</td>\n",
       "      <td>22</td>\n",
       "      <td>341</td>\n",
       "      <td>False</td>\n",
       "      <td>238.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>姜姜会吃胖</td>\n",
       "      <td>2018-09-19 06:36:00</td>\n",
       "      <td>sml-str40</td>\n",
       "      <td>都说来广州吃糖水就要来南信招牌姜撞奶红豆双皮奶牛三星云吞面一楼现金二楼微信支付宝位置不少但是...</td>\n",
       "      <td>非常好</td>\n",
       "      <td>很好</td>\n",
       "      <td>很好</td>\n",
       "      <td>518986</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2018</td>\n",
       "      <td>9</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>197</td>\n",
       "      <td>False</td>\n",
       "      <td>122.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>forevercage</td>\n",
       "      <td>2018-08-24 17:58:00</td>\n",
       "      <td>sml-str50</td>\n",
       "      <td>一直很期待也最爱吃甜品广州的甜品很丰富很多样来之前就一直想着一定要过来吃到腻今天总算实现了先...</td>\n",
       "      <td>非常好</td>\n",
       "      <td>很好</td>\n",
       "      <td>很好</td>\n",
       "      <td>518986</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2018</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>17</td>\n",
       "      <td>261</td>\n",
       "      <td>True</td>\n",
       "      <td>176.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32478</th>\n",
       "      <td>darayan</td>\n",
       "      <td>2007-04-28 16:57:00</td>\n",
       "      <td>NAN</td>\n",
       "      <td>我觉得姜撞奶一般咯有苦公司岩搬过来东风西呢边下午茶仲会时不时定下甜品不过就无乜惊喜咯都系中意...</td>\n",
       "      <td>一般</td>\n",
       "      <td>一般</td>\n",
       "      <td>一般</td>\n",
       "      <td>521698</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2007</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>16</td>\n",
       "      <td>68</td>\n",
       "      <td>False</td>\n",
       "      <td>55.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32479</th>\n",
       "      <td>chenjj920</td>\n",
       "      <td>2006-10-27 09:12:00</td>\n",
       "      <td>NAN</td>\n",
       "      <td>味道啦还算正宗值得一试</td>\n",
       "      <td>好</td>\n",
       "      <td>一般</td>\n",
       "      <td>一般</td>\n",
       "      <td>521698</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2006</td>\n",
       "      <td>10</td>\n",
       "      <td>4</td>\n",
       "      <td>9</td>\n",
       "      <td>16</td>\n",
       "      <td>False</td>\n",
       "      <td>11.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32480</th>\n",
       "      <td>winny311111</td>\n",
       "      <td>2018-05-14 03:09:00</td>\n",
       "      <td>sml-str50</td>\n",
       "      <td>第二次来好吃第二次来好吃第二次来好吃</td>\n",
       "      <td>非常好</td>\n",
       "      <td>非常好</td>\n",
       "      <td>非常好</td>\n",
       "      <td>521698</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2018</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>24</td>\n",
       "      <td>True</td>\n",
       "      <td>18.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32481</th>\n",
       "      <td>Polykat_年嫿</td>\n",
       "      <td>2018-04-28 02:33:00</td>\n",
       "      <td>sml-str50</td>\n",
       "      <td>兒時的味道兒時的味道兒時的味道</td>\n",
       "      <td>非常好</td>\n",
       "      <td>很好</td>\n",
       "      <td>好</td>\n",
       "      <td>521698</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2018</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>15</td>\n",
       "      <td>True</td>\n",
       "      <td>15.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32482</th>\n",
       "      <td>不瘦到120斤不换ID</td>\n",
       "      <td>2018-04-09 01:22:00</td>\n",
       "      <td>sml-str50</td>\n",
       "      <td>好喝好喝好喝好喝好喝好喝好喝好喝好喝好喝好喝好喝好喝好喝好喝好喝好喝好喝好喝</td>\n",
       "      <td>非常好</td>\n",
       "      <td>非常好</td>\n",
       "      <td>非常好</td>\n",
       "      <td>521698</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2018</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>38</td>\n",
       "      <td>True</td>\n",
       "      <td>38.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>32474 rows × 16 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            cus_id         comment_time comment_star  \\\n",
       "0             迷糊泰迪  2018-09-20 06:48:00    sml-str40   \n",
       "1            稱霸幼稚園  2018-09-22 21:49:00    sml-str40   \n",
       "2           爱吃的美美侠  2018-09-22 22:16:00    sml-str40   \n",
       "3            姜姜会吃胖  2018-09-19 06:36:00    sml-str40   \n",
       "4      forevercage  2018-08-24 17:58:00    sml-str50   \n",
       "...            ...                  ...          ...   \n",
       "32478      darayan  2007-04-28 16:57:00          NAN   \n",
       "32479    chenjj920  2006-10-27 09:12:00          NAN   \n",
       "32480  winny311111  2018-05-14 03:09:00    sml-str50   \n",
       "32481   Polykat_年嫿  2018-04-28 02:33:00    sml-str50   \n",
       "32482  不瘦到120斤不换ID  2018-04-09 01:22:00    sml-str50   \n",
       "\n",
       "                                             cus_comment kouwei huanjing fuwu  \\\n",
       "0      南信算是广州著名甜品店吧好几个时间段路过都是座无虚席看着餐单上密密麻麻满满当当好吃的容易范选...    非常好        好    好   \n",
       "1      中午吃完了所谓的早茶回去放下行李休息了会就来吃下午茶了服务两层楼楼下只能收现金楼上可以微信支...     很好       很好   很好   \n",
       "2      冲刺王者战队吃遍蓉城战队有特权五月份和好朋友毕业旅行来了广州我们都是双皮奶爱好者搜到啦最火的...     很好       很好   很好   \n",
       "3      都说来广州吃糖水就要来南信招牌姜撞奶红豆双皮奶牛三星云吞面一楼现金二楼微信支付宝位置不少但是...    非常好       很好   很好   \n",
       "4      一直很期待也最爱吃甜品广州的甜品很丰富很多样来之前就一直想着一定要过来吃到腻今天总算实现了先...    非常好       很好   很好   \n",
       "...                                                  ...    ...      ...  ...   \n",
       "32478  我觉得姜撞奶一般咯有苦公司岩搬过来东风西呢边下午茶仲会时不时定下甜品不过就无乜惊喜咯都系中意...     一般       一般   一般   \n",
       "32479                                        味道啦还算正宗值得一试      好       一般   一般   \n",
       "32480                                 第二次来好吃第二次来好吃第二次来好吃    非常好      非常好  非常好   \n",
       "32481                                    兒時的味道兒時的味道兒時的味道    非常好       很好    好   \n",
       "32482             好喝好喝好喝好喝好喝好喝好喝好喝好喝好喝好喝好喝好喝好喝好喝好喝好喝好喝好喝    非常好      非常好  非常好   \n",
       "\n",
       "       shopID  stars  year  month  weekday  hour  comment_len      y  \\\n",
       "0      518986    4.0  2018      9        3     6          184  False   \n",
       "1      518986    4.0  2018      9        5    21          266  False   \n",
       "2      518986    4.0  2018      9        5    22          341  False   \n",
       "3      518986    4.0  2018      9        2     6          197  False   \n",
       "4      518986    5.0  2018      8        4    17          261   True   \n",
       "...       ...    ...   ...    ...      ...   ...          ...    ...   \n",
       "32478  521698    NaN  2007      4        5    16           68  False   \n",
       "32479  521698    NaN  2006     10        4     9           16  False   \n",
       "32480  521698    5.0  2018      5        0     3           24   True   \n",
       "32481  521698    5.0  2018      4        5     2           15   True   \n",
       "32482  521698    5.0  2018      4        0     1           38   True   \n",
       "\n",
       "       len_comment  \n",
       "0             97.0  \n",
       "1            175.0  \n",
       "2            238.0  \n",
       "3            122.0  \n",
       "4            176.0  \n",
       "...            ...  \n",
       "32478         55.0  \n",
       "32479         11.0  \n",
       "32480         18.0  \n",
       "32481         15.0  \n",
       "32482         38.0  \n",
       "\n",
       "[32474 rows x 16 columns]"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dianping['cus_comment'] = dianping['cus_comment'].str.replace(' ', '')\n",
    "dianping['len_comment'] = dianping['cus_comment'].str.len()\n",
    "dianping = dianping[dianping['len_comment'] > 1]\n",
    "dianping\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/Anaconda/lib/python3.11/site-packages/sklearn/feature_extraction/text.py:521: UserWarning: The parameter 'token_pattern' will not be used since 'tokenizer' is not None'\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "##接下来处理文本数据，我们使用简单的词袋模型作为预测特征\n",
    "import jieba\n",
    "import re\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "with open('Chinese/stopword.txt', 'rt') as f:\n",
    "    stoplist = f.readlines()\n",
    "    stoplist = [w.replace('\\n', '') for w in stoplist]\n",
    "\n",
    "\n",
    "def not_digit(w):\n",
    "    w = w.replace(',', '')\n",
    "    if re.match(r'\\d+', w) != None or re.match(r'\\d%', w) != None or re.match(\n",
    "            r'\\d*\\.\\d+', w) != None:\n",
    "        return False\n",
    "    else:\n",
    "        return True\n",
    "\n",
    "\n",
    "def tokenize(w):\n",
    "    cut_w = jieba.cut(w)\n",
    "    ## 去除停用词\n",
    "    cut_w = [\n",
    "        w.strip().lower() for w in cut_w\n",
    "        if ((w not in stoplist) and not_digit(w) and len(w.strip()) > 0)\n",
    "    ]\n",
    "    return cut_w\n",
    "\n",
    "\n",
    "count_vect = CountVectorizer(tokenizer=tokenize, min_df=5)\n",
    "bag_words = count_vect.fit_transform(dianping['cus_comment'])\n",
    "words_names = count_vect.get_feature_names_out()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来我们做一个简单降维操作（由于我们使用了Lasso降维，这一步也许可以不做，不过做的话可以降低运行时间）。我们将TF-IDF值比较低的特征剔除掉，从而达到降维。\n",
    "\n",
    "或者，这一步也可以考虑其他的降维方法，比如Fan(2007)的screening方法等。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>一丁点</th>\n",
       "      <th>一万个</th>\n",
       "      <th>一上</th>\n",
       "      <th>一下下</th>\n",
       "      <th>一下子</th>\n",
       "      <th>一不小心</th>\n",
       "      <th>一丝</th>\n",
       "      <th>一丝丝</th>\n",
       "      <th>一两个</th>\n",
       "      <th>一两口</th>\n",
       "      <th>...</th>\n",
       "      <th>齁</th>\n",
       "      <th>齐全</th>\n",
       "      <th>齐名</th>\n",
       "      <th>齿</th>\n",
       "      <th>龍</th>\n",
       "      <th>龙</th>\n",
       "      <th>龙津</th>\n",
       "      <th>龜苓</th>\n",
       "      <th>龟</th>\n",
       "      <th>龟苓膏</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32469</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32470</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32471</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32472</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32473</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>32474 rows × 11158 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       一丁点  一万个  一上  一下下  一下子  一不小心  一丝  一丝丝  一两个  一两口  ...  齁  齐全  齐名  齿  龍  \\\n",
       "0        0    0   0    0    0     0   0    0    0    0  ...  0   0   0  0  0   \n",
       "1        0    0   0    0    0     0   0    0    0    0  ...  0   0   0  0  0   \n",
       "2        0    0   0    0    0     0   0    0    0    0  ...  0   0   0  0  0   \n",
       "3        0    0   0    0    0     0   0    0    0    0  ...  0   0   0  0  0   \n",
       "4        0    0   0    0    0     0   0    0    0    0  ...  0   0   0  0  0   \n",
       "...    ...  ...  ..  ...  ...   ...  ..  ...  ...  ...  ... ..  ..  .. .. ..   \n",
       "32469    0    0   0    0    0     0   0    0    0    0  ...  0   0   0  0  0   \n",
       "32470    0    0   0    0    0     0   0    0    0    0  ...  0   0   0  0  0   \n",
       "32471    0    0   0    0    0     0   0    0    0    0  ...  0   0   0  0  0   \n",
       "32472    0    0   0    0    0     0   0    0    0    0  ...  0   0   0  0  0   \n",
       "32473    0    0   0    0    0     0   0    0    0    0  ...  0   0   0  0  0   \n",
       "\n",
       "       龙  龙津  龜苓  龟  龟苓膏  \n",
       "0      0   0   0  0    0  \n",
       "1      0   0   0  0    0  \n",
       "2      0   0   0  0    0  \n",
       "3      0   0   0  0    0  \n",
       "4      0   0   0  0    0  \n",
       "...   ..  ..  .. ..  ...  \n",
       "32469  0   0   0  0    0  \n",
       "32470  0   0   0  0    0  \n",
       "32471  0   0   0  0    0  \n",
       "32472  0   0   0  0    0  \n",
       "32473  0   0   0  0    0  \n",
       "\n",
       "[32474 rows x 11158 columns]"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfTransformer\n",
    "\n",
    "tfidf_transformer = TfidfTransformer().fit(bag_words)  ##使用L2范数，并fit模型（如计算IDF等）\n",
    "tfidf_words = tfidf_transformer.transform(bag_words)  ##变换数据\n",
    "tfidf_words_df = pd.DataFrame.sparse.from_spmatrix(tfidf_words,\n",
    "                                                   columns=words_names)\n",
    "tfidf_words_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.9101416620357718\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>一丁点</th>\n",
       "      <th>一下下</th>\n",
       "      <th>一下子</th>\n",
       "      <th>一不小心</th>\n",
       "      <th>一丝</th>\n",
       "      <th>一两个</th>\n",
       "      <th>一两次</th>\n",
       "      <th>一两碗</th>\n",
       "      <th>一个个</th>\n",
       "      <th>一个多</th>\n",
       "      <th>...</th>\n",
       "      <th>齁</th>\n",
       "      <th>齐全</th>\n",
       "      <th>齐名</th>\n",
       "      <th>齿</th>\n",
       "      <th>龍</th>\n",
       "      <th>龙</th>\n",
       "      <th>龙津</th>\n",
       "      <th>龜苓</th>\n",
       "      <th>龟</th>\n",
       "      <th>龟苓膏</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32469</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32470</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32471</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32472</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32473</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>32474 rows × 7810 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       一丁点  一下下  一下子  一不小心  一丝  一两个  一两次  一两碗  一个个  一个多  ...  齁  齐全  齐名  齿  龍  \\\n",
       "0        0    0    0     0   0    0    0    0    0    0  ...  0   0   0  0  0   \n",
       "1        0    0    0     0   0    0    0    0    0    0  ...  0   0   0  0  0   \n",
       "2        0    0    0     0   0    0    0    0    0    0  ...  0   0   0  0  0   \n",
       "3        0    0    0     0   0    0    0    0    0    0  ...  0   0   0  0  0   \n",
       "4        0    0    0     0   0    0    0    0    0    0  ...  0   0   0  0  0   \n",
       "...    ...  ...  ...   ...  ..  ...  ...  ...  ...  ...  ... ..  ..  .. .. ..   \n",
       "32469    0    0    0     0   0    0    0    0    0    0  ...  0   0   0  0  0   \n",
       "32470    0    0    0     0   0    0    0    0    0    0  ...  0   0   0  0  0   \n",
       "32471    0    0    0     0   0    0    0    0    0    0  ...  0   0   0  0  0   \n",
       "32472    0    0    0     0   0    0    0    0    0    0  ...  0   0   0  0  0   \n",
       "32473    0    0    0     0   0    0    0    0    0    0  ...  0   0   0  0  0   \n",
       "\n",
       "       龙  龙津  龜苓  龟  龟苓膏  \n",
       "0      0   0   0  0    0  \n",
       "1      0   0   0  0    0  \n",
       "2      0   0   0  0    0  \n",
       "3      0   0   0  0    0  \n",
       "4      0   0   0  0    0  \n",
       "...   ..  ..  .. ..  ...  \n",
       "32469  0   0   0  0    0  \n",
       "32470  0   0   0  0    0  \n",
       "32471  0   0   0  0    0  \n",
       "32472  0   0   0  0    0  \n",
       "32473  0   0   0  0    0  \n",
       "\n",
       "[32474 rows x 7810 columns]"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf_sum = tfidf_words_df.sum(axis=0)\n",
    "quantile_tfidf = tfidf_sum.quantile(0.3)\n",
    "print(quantile_tfidf)\n",
    "tfidf_sub = tfidf_sum[tfidf_sum > quantile_tfidf]\n",
    "tfidf_words_df = tfidf_words_df.loc[:, tfidf_sub.index]\n",
    "tfidf_words_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来我们训练模型，选取前30000条作为训练集，剩下的作为测试集："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "开始训练模型......\n",
      "模型训练完成......\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAArEAAAKeCAYAAACyIEPOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAAB1u0lEQVR4nO3deXxU5dn/8e+ZNZOEhEUCKkZEBJdWEUVAEKlLBaG1rRuttIhWaUtR0ar0aX8oWiui1gWtu7jQavu0T61FQdGqKNSK4l7rhqKChD0Jyezn/v0xZExIAplkkjMz5/N+vfpq5iRnuOLJJN/cuc51W8YYIwAAACCPeJwuAAAAAMgUIRYAAAB5hxALAACAvEOIBQAAQN4hxAIAACDvEGIBAACQdwixAAAAyDuEWAAAAOQdQiwAAADyTsYhds2aNZo9e7ZOOeUUHXzwwZo4cWKbzjPG6O6779bYsWN16KGH6swzz9Qbb7yR6T8PAAAAZB5iP/zwQ73wwgvad999tf/++7f5vHvuuUe33nqrzj77bN11113q3bu3zjnnHH3++eeZlgAAAACXs4wxJpMTbNuWx5PKvrNmzdI777yjRYsW7fKcaDSqo48+WmeddZYuvvhiSVIsFtO4ceM0ZswYXXnlle2rHgAAAK6U8UpsQ4DNxKpVq7R9+3aNHz8+fSwQCOjEE0/UsmXLMn4+AAAAuFuX3Ni1evVqSdKAAQOaHN9///21bt06RSKRrigDAAAABaJLQmxNTY0CgYCCwWCT42VlZTLGqLq6uivKAAAAQIHwOV1ARxhjZFmW02UAAAAUJGOMttREFE/YGZ/rU1K9uhfL8nZO3OySEFtWVqZYLKZoNNpkNbampkaWZam8vLxdz2tZlmpqwkomM/8Pi/zi9XpUVhbiersE19tduN7uwvXOXcYYxeJNr0k0ntTPb2r9/qXKPqX69ZQjZan5omJCUjIRly+fQ2xDL+wnn3yiAw88MH189erV2muvvVRUVNTu504mbSXa8dsB8hPX21243u7C9XYXrnd2tBQ82/U8Mpq7cJU+27C91Y+5ecZoBf3eJscCfk+Tv4qbREwmXCNPtz0kSZavaStpNnVJiB06dKhKS0u1ePHidIiNx+N6+umnNWbMmK4oAQAAIO/sKqS2JXhmy8B+5epW7N9lG6dJxBR++lbZW9ep+Fuz5Cmr6NSaMg6x4XBYL7zwgiRp7dq12r59u5YsWSJJOuqoo9SzZ09NmTJF69at09KlSyVJwWBQ06ZN0/z589WzZ08NGjRIjzzyiLZt26Zzzz03i58OAACA87KxQtqVIbVBZUWpZk0e2qw9YOcV152ZREzhp25Rcu27ki8ou35b7oXYzZs368ILL2xyrOHxQw89pOHDh8u2bSWTySYfc95558kYo/vvv19btmzRQQcdpPvuu0/77LNPB8oHAADIHcYYRePJLg2frQXP9thdWG2JSUQVfurWdIANjb9Yvr6DOlzL7mS8Y1eu2bq1brc9NbadbBaqkV+8Xkvl5cWqrq5XMpnXX7Jt4vV65fF4d/+BBcrn86hHj5I2vb6R/7je7lIo17ulldbOWjndXUhtT/DMFpOIKrzkZiXXvSf5i5oF2J49S+T1ds5E17wesbU7xhjV1GxROFwnqfCDT6HbtMkj287fb3iZsRQKlaisrCdj5AAgxxhjdO3CVfpo7a7n3GdrhdTJkLorJh5V+KnGAfYS+foe0GX/fkGH2HC4TuHwdpWWdlcwWCRlYZkdzvF6LVeswkpG0WhE27dvk98fVHFxqdMFAQD01eprNJ7cZYBtCK9Bvzcnw2fWJOMy0TrJX6Ti8ZfI24UBVirgEGuM0fbt21RUVKLS0vbNoUVu8fk8ef2np0z4/UElEnFt375NoVBJYX8TBIA8YBujqxasbNYq0JaxU4XKKipV8YTLZNdulLf3fl3+7xdsiLVtW7adVFFRsdOlAO1SVFSsSKROtm3L63VvfywAdKXWel3nLFipqq3hJsfbMnaq0Jh4RInP35Z/wDBJqSDrLXLmL4YFHGJTN3K5+eYY5LeGr13bThJiASDL2ntjVp8eIV0xdZgsWa5ZcW1gYmGFl9yk5PoPZI45W4GDxjpaT8GG2AZu+uJCYeFrFwA6R2utAbtTWVGq2VOHyePC788mFlZ48e+UrPpQCoTk7VXpdEmFH2IBAAAa2MboV3e/3Kw1oLH2DvwvVCYWVv3iG2VXfSQFilU84VJHemB3RojNA/fdd5cWLLgn/bisrFz77ttfP/rRVI0cObrZx9fU1OjhhxfohRf+qY0bN6i0tJuGDj1SU6eep/79m3/R1dfX69FHF+q5557RunVrZVmW9ttvgI4//pv67ndPVzDYvn2Pp0z5vj7++EPdfvs9Ouyww5u8b9WqV3XBBT/Rvfc+pAMPPLjJ+z788H1NnXqWbr31Tg0demSTOhcufCjrdUrSmjWf6qab5umdd95ScXGJxo07Weed9zP5/f5Wz2n4HFpSWbmv/vjHvzY7btu2fvzjH+mDD/6rq6+eq29844R21wwAaLuGTQga97Y2bg1ozK1htSVNAmywRMUnXypv7/5OlyWJEJs3gsGgbrnlTknS5s0b9dBDC3T55Rfr9tvv0de/flj64zZv3qTp08/X9u01+tGPztEBBwzWxo0b9MgjD+u8836k66+/RUOGDE1//LZt23TBBdO0YUOVTj/9+zr00CGSpHfffVsLFz4oj8erM874fsb1rl79sT7++ENJ0tKlS5qF2Ext27ZNF174E1VVrc9qnVIq9F9wwU+0zz6Vuuaa67Vx4wbddttNikQiuvjiy1s9b/DgA3XnnQuaHKuvr9MvfnGBRow4usVz/v73/9OmTRvbVScAoG127ndtqde1T4+Qrjl/hCtbA9rKJOOqf/IG2Rs+TgXYCZfKu0d/p8tKI8TmCY/Ho6997evpxwcf/DV973sTtHjxoiYh9sYbr9OGDeu1YMEfte++/dPHjzlmrM4770eaM+fXevTRv6VXLW+8ca7WrVuru+9+QAMGDEx//LBhw/W9752uNWs+bVe9S5cukcfj0ZAhQ/Xcc8/ooosulc/X/i+3G2+cq7Vrv8h6nZL097//VfX1dfrtb69XWVlqHFsymdTvfnedfvSjc7THHr1bPK+kpLTJNZGkJ5/8h2zb1oknjmv28du2bdM999yh6dMv1LXXXtXuegHA7Vq6KSv9vjbcnOXm3tZMWF6/fPt8XbHq9SqecJm8e+zrdElNEGLzVO/eFerevYeqqqrSx9av/1Ivvvi8Jk78TpMAK0mhUEg/+tE5uuqq/6fnnntG48ZN0Pr1X+r555/VGWf8oEkwbFBWVt4kILeVMUbPPPOUhg49Uqef/n1dfvlMvfzyCo0ePSbj52r4vJ5//llNmnRWVuts8PLLK3TkkUelA6wkHXfcibrhhmv1yisv6+STv9Xm51q6dIn69avUQQcd0ux9d911m4YOPaJJiwQAYNfasqraVq7ZhCCLgkd8R/6DxspT3N3pUppxXYjd1W9vXSFbfTb19fWqqanWnnvulT72xhurZIzRqFHHtHhOw/E333xd48ZN0Jtvvi5jjEaMGNnhehp7++039eWX63T22T/W8OEjVV5erqVLl7Q7xH5VZ8t/ot9ZIpHY7cd4vV99A1uz5lNNmPDtJu/v1q2bevXaI6MV3i1bNmvVqlf1ox+d0+x9//nPO1q6dIkefvjPbX4+AHCzhh7WjgbWxv2u9LrunonWKbry/xQcfrosf5Ek5WSAlVwWYtu613FnGtivXL88a2i7XkQN4WzTpk26445bVFxcojPOmJR+/8aNqV7LPn36tnh+SUmpSku7acOGDTs+fsMuP769li59SoFAUMcee5x8Pp/Gjj1eTz31pOrr61VcnPnmEw119u27+zq//HKdTj/927v9uP/5nyvSK6y1tTUqLe3W7GO6deummpqaNtf57LNLlUwmm7US2Lat3/1uniZNmqw999xLX365rs3PCQCFrr3zWlubINCAwJo5E61T/RPXy970qUy4WqETf+50SbvkqhArSa18ree8cDissWNHpB97vV5de+2Nqqzsn4Vnz95/lEQioeeee0YjRx6t0tLUDh4nnjhOf//7/2nZsuc0btyEdj93W74Z7bFHb91770O7/bjGK9jZ8vTTizV48EGqrGzaM/SPfzymLVs2a/Lks7P+bwJAPjHGKBpLKhJNKBpLKp5o20orq6qdz0S2q/7J62VvWiOrqJsCQ09xuqTdclWItSxLvzxraF62EwSDQd1++z2ybVtffPG57rzzNv3mN1fooYf+pD322EOS1Lt36gakqqr1OuCAQc2eo76+Ttu316qiomLHx1ekP37n4NVeK1e+rG3btmrUqDGqra2VJA0YMFC9eu2hpUufSodYrzf1pZdMNr8WDccabgRrqHP9+vXaa699dvnv+/1+DRzY/HPfWeMdsLp1K1NdXfNvoLW1tSorK9vtc0nS2rVf6L333tWMGTObHK+vr9fdd9+u8877meLxuOLxuOrq6iRJkUhEdXXbVVLizHZ9ANCV2vPXUHpYu4aJbE+twG5OBdjQxMvl7dnP6bJ2y1UhVkoF2WAg/7bw9Hg86XmqBx/8NVVW7qvzzz9bDzxwj37xi19KkoYMSbUp/OtfL7XYf7p8+YuSlB53ddhhqY9/5ZWXNWzY8KzUuXTpU5Kk3/52jqQ5Td63bdtWbd26RT169FSPHt0lSVu2bGr2HA0jqHr06Nmkzn//e4WGDh22y3+/Pe0E++7bv1nv6/bt27V586ZmN8i1pmEaw/HHn9TkeHX1NlVXV+uGG67VDTdc2+R911xzpe64o5cef/ypNv0bAJBPdm4TiMaTrQZYNhdwTirAzpO9+TNZoTKFJlwub8+9nS6rTVwXYgvFgQcerBNOOElPPvkPTZ16nnr12kN9++6pY44Zq8WLn9CZZ57VZHU1EonooYfuV0VFn/SA/b59+2rs2OP12GN/0cknf0v77Tegyb9RW1urNWs+0de+dmibaopEInrxxRd0zDFjdfrpk5q8b8uWzbryyl/p2Wef1mmnTVK/fpXq1WuP9Mc39uKLz6tXrz3Ur98+Ter8v//7i8aN23Wd7WknGDHiaD300ALV1taqW7dUb+xzzz0jj8ejo44a0dpTNPHMM0/p8MOPSK+KN+jZs5duvfXOFv9bnHPO+Vn75QEAcsnuVl0fvnKcIvVRJRKpkEtYdU742Tu+CrATL5e3R34EWIkQm9fOPvtcPfvs0/rznx/RT386Q5J0ySWXa/r08zV9+nn64Q+natCghs0OFmr9+i91/fW3NNnZ6pJLZmnGjPP105+eqzPP/EF6VNV//vOO/vrXP+mss85Oh9jRo4/U+PET9atfXdliPS+++LzC4XqdfvqkFsdI/fGPD2np0qd02mmT5PF4dO650zRv3jXyer0aPfpYSdJLL72gJ554XJdf/usm39AuuWSWLrhg2m7r9Pv9zXYA251TTjlVf/nLn/TLX16iH/3oHG3cuEG3336LTjnle01mxF544U+1fv2X+tOfHmty/gcf/FeffvqJJk06q9lzB4PBZv8tGm7s2m+/AR0aDQYAuSoWt1sNsAf0K1d5aUAmkZDXQ3B1WnDEmYr88y4VnfCzvAqwEiE2r1VW9tfxx39Tjz32F/3wh1NVWlqqXr320N13P6CHH16g//3fR7VxY5VKS7vpiCOO1OzZVzfbdrZ79+66664FevTRP+jZZ5fq4YcfkMeT2s71rLOm6JRTvicpdWOZlFpZbM3SpU+pT5++OvzwI1p8/7hxE3XrrTdq7dovtPfe/fTtb39XxcXF+tOf/qCnn14sSdpvv/11xRW/aXaHf/fu3XXPPQ/oD394eJd1tkdZWZluueUO3XTT9frlLy9RcXGJvvWt7+j883/W5OOSyaSSyWSLn3cgENCxxx7f7hoAIB+0dUxlNP7V98qbZ4xW0P9VG19xyMeqq8OMMelr4O1VqeLTrpZleRyuKnOWMcY4XURHbN1al/5zRGPxeEybN3+pXr32lN8fcKCywvLqq6/o0ksv1J/+9JgqKvo4UoPP52nxWhcqN38N+3we9ehR0urrG4WF652bsrXJwB0XH9vkXhSut7PscI3CT9+q4PAz5Ou7+xuhO6pnzxJ5vZ0TkFmJRZu8/fabGjduomMBFgDQdbI1V31gv3IF/Pm3wleo7HCNwouuk711rSIv3K+S06+R5cm/m90bEGLRJlOnnud0CQCATtaw+tqeSQIt4Yat3GHXVyv8xHWyt66TVdxdxSddlNcBViLEAgDgKq31tbbWLrBzTyvBNP/Y9dsUXjRP9rZ1skp6qHji5fKUZ3e3TicQYgEAcAFjjKLxtu2Q1WBgv3J1K/YTWvNYKsBeJ3vblzsC7Cx5ygujNZAQCwBAAcs0vDZuF2DVNf/F3nhiR4DtuWMFtjACrOSCEJvnwxfgYnztAugo2xhdtWBls/C6q75WgmthCQ4/Q0rEFBgyQZ6yCqfLyaqCDbFeb6p/JxaLKhAI7uajgdwTi0UlSV5vwb5MAWRRSyOx5ixYqaqt4fSxhvAa9HsJqgXMROukQLEsy5Ll9atozFSnS+oUBfvT0ePxKhQq1fbtWyVJgUCQF2yes21LyWThr04aYxSLRbV9+1aFQqXyeBhPA2DXWltxbdCnR0hXTB1GeHUBu26r6hfNlW+fQxUc+YOCvt4FG2IlqayspySlgyzym8fjkW27ZzB2KFSa/hoGgNaY3QTYyopSzZ46TJ4CDjNIsbdvUf2i62RqqpSwX1dg6LdlFXVzuqxOU9Ah1rIslZf3UrduPZRMJpwuBx3g9VoqLy9WdXW9K1ZjvV4fK7AAdssYo9r6eDrANqy4Nu51pcfVHeztm3cE2A2yuvVO3cRVwAFWKvAQ28Dj8cjjcde2nYXG5/OoqKhI4XCSbQoBQC23EFwxdZiKAq740Y5G7O2bVf+PuTK1G1MB9luz5Cnt5XRZnY6vdAAA8oxtjH5198tNbtoa2K+8yaYEcAe3BliJEAsAQM5rPHlg56kD3LTlbsmNn8hs3ySrrCLVQuCSACsRYgEAyFm726igT4+Qrjl/BDdtuZh/vyOlE6bL23uAPKXuuhmYEAsAQA7a3dgspg64l12zUfL65CnpIWlHkHUhQiwAADmmpbFZO++yxdQBd7JrNqh+0XWS16fiibPSQdaNCLEAAOSYaDzZbGwWPa+wazakbuKq2yJPeV+ny3EcIRYAgBxijNHchavSjxmbBUmyq6tSc2DrtsjTfU+FJl4uT3F3p8tyFK8KAAByxM6bF1RWlDI2CzsC7FyZuq3ydN9LoYmXuT7ASoRYAAAc1TA+y8g0m0Iwa/JQWghczq5en2ohqN8mT4+9FJpwuTzF5U6XlRMIsQAAdKGdZ762Nj6LzQsgSfIFJX9Qnh57KzThMgJsI4RYAAC6iDFG1y5cpY/WVrf6MQ1TCLiRC5LkKemh4omzJI9XnlCZ0+XkFEIsAABdJBpPthhgG4/PYnQWktvWyd7yhfwDjpIkV4/R2hVCLAAAXWDnqQM3zxidbhcguKJBcus6hRfNlYnUyvL65dv3cKdLylmEWAAAOknj/tfGs18rK0rVrdhPcEUTya1rFV50nUy4Rp6e+8jTZ6DTJeU0QiwAAFlmjFE0nmz1pi2mDmBnyS1rFX5iR4DttY+KJ1wuq6jU6bJyGiEWAIB2aLzK2uT4LiYOSEwdQHPJLV+kVmAjtfL0qlTxhMsIsG1AiAUAoBXtDaqNNb5pS6L/FU3ZdVsbBdh9VTzhUgJsGxFiAQBoZFebD2SCUVloC6u4u3z7D1ey6kMVn0yAzQQhFgDgeu0NrjuvsjbGiivawrIsBY8+S0pEZfmLnC4nrxBiAQCuZhujqxasbDW4ElSRbcnNnyn29lMqOuZsWd4dUyoIsBkjxAIAXKlhgsCcBStVtTXc5H1sPoDOkty0RvVPzJOidYoVd1fwqNOdLilvEWIBAK7T0uprnx4hXTF1GMEVnSa56VPVP3G9FK2Tp2KAAkMmOF1SXiPEAgBco7XV18qKUs2eOkwegis6SXLjp6p/siHA7q/iky+RFSh2uqy8RogFALjCrlZfmSCAzpTc+ElqBTZWL0+fgSoef4msQMjpsvIeIRYAUPBMCwGW1Vd0BZOMK/z0fAJsJyDEAgAKXixupwMsq6/oSpbXr6Ljf6LY6/9Q6PifEWCziBALAHCVK6YOU1GAH3/oXMZOyPKkvs58fQfJN/4ShysqPB6nCwAAoLMZmfTbLc17BbIpWfWR6v70SyU3f+Z0KQWNEAsAKEjGGEVjSUViCc1ZsNLpcuASyaqPVP/kDTK1GxV77e9Ol1PQ+HsKAKBg7G772MqKUgX8rN+gcyTXf6j6xTdK8Yi8ex6oom+c73RJBY0QCwDIa7sLrg0aphFwMxc6Q2L9hwo3BNi9DlLopItk+YNOl1XQCLEAgLzUsHHB7oIr28eisyXWf6Dw4t99FWDHXSTLR4DtbIRYAEBeaFhxlbTLVVeCK7pa7I0nUgF270MUOukCAmwXIcQCAHJeS7ttNUZwhZNCx/9UsVWPK3DEd2T5Ak6X4xqEWABATrON0a/ufllVW8PN3tcQXtm4AF3Nrt0kT7c9JEmWv0jB4Wc4XJH7EGIBADmrYbvYhgDbsNtWw6xXVl3hhMS69xRefJMCh09QcOgpTpfjWoRYAEDOisaTTbaLveb8EfIQWuGgxNr/KLzkZikZU7LqYxk7KcvjdbosVyLEAgBykm1Mk00Krpg6jAALR6UC7E1SMi5v5WEKnfhzAqyDCLEAgJyzcxtBZUWpgn7CApyT+OJdhZ+6uWmA9fqdLsvVCLEAgJzQsE2s1LyNgE0K4KTEF+8o/NQtOwLsEIVOnE6AzQGEWACA42zb6P/d+299VtV8hBZtBHCaXbNBSsbl2/dwFZ0wXZaX+JQLuAoAAEcZYzTzphdaDLAD+5XTRgDHBQ4+Tp6SnvL2+xoBNodwJQAAjmjYgStpjFavq5bECC3kjsS69+TtuY+solJJkm/fIc4WhGYIsQCALmWMUTSebHHb2CumDlNRgB9NcFbiszcUfvo2eXrsreKJl8kKljhdElrAdwoAQKdrWHU1Mi2GV0k6gNYB5IDEmtcVXnqbZCflKest+YNOl4RWEGIBAJ2iLcG1sqJUvz77SPXoXqL6uoiSSeNApUBK4tPXFX4mFWB9+x2pouN/IstDVMpVXBkAQNa0JbhKqfA6a/JQBf1e+f1eFQV9CtdbkgixcEb801WKPHN7KsAOOEpFx51PgM1xXB0AQFbYOzYo2F1wtWRxwxZySmLNG4osvV0yDQF2Gjtx5QFCLACgQxpu1JrTaIetBgRX5ANP9z1lhbrJu+dgFX3jfAJsniDEAgDarKFdIP24hbaBxmOyCK7IB57yPir+zmxZxeUE2DxCiAUANAunLX7MbvpcpdTK62x22EIeiK9eKcsXkK/yMEmSp7SnwxUhU4RYAHA5Y4yuXbhKH62tbvdzNL5Ri5VX5Lr46lcUefZOyfKo+Dv/T9499nW6JLQDIRYAXGbnVddoPJlRgG3c59qAtgHki/jHryjyzzslY8s3cKQ8PfdxuiS0EyEWAFxiVztlNbh5xujdbjhAYEW+in/0siLP3Z0KsINGq2jMObI8HqfLQjsRYgHABdrSMjCwX7m6FfsJqChIqQB7l2SMfIOOUdGYqQTYPEeIBQAXiMXtJgGWlgC4SWL9h+kA6x88RsExZ8uyCLD5jhALAC5z84zRrLjCVbwV+8u3/whZPr+CxxBgCwUhFgBcwDTazpUJAnAby+NR0djzJEsE2ALClQSAAmcbozkLVjpdBtCl4h+8pPBz98jYqUkclsdDgC0wrMQCQAGzjdGv7n45vR1sZUWpAn5+kKOwxd9/UZEX7pdklNj7YPkHjXK6JHQCQiwAFKCGcVpzFqxMB9g+PUKaPXUYrQQoaPH/LlNk2QJJRv6Dj5PvgKOdLgmdhBALAAWg8QYGLW0P26dHSNecP4LtYFHQYv99QdFlD0gy8h9yvIJHT+aXtgJGiAWAPLe7GbCVFaWaPXUYARYFLfbe84q++IAkyX/ICQoefRYBtsARYgEgz7W2bWzDLFimEaDQ2XVbFV3xB0mS/2snKjjyB3zNuwAhFgDyVOO+1waNt41l8wK4haekh0In/lyJdf9VcPgZfN27BCEWAPJQSy0ElRWlbGIAVzGxsKxASJLkqzxMvsrDHK4IXYk5KwCQh1raRpbJA3CT2LvPqO5/fyW7usrpUuAQVmIBIA813oGLbWThNrF3nlF0xUJJUnz1SgUPn+hwRXACIRYA8owxqRFaDbhxC24Se2dp+iauwGEnKzBkgsMVwSmEWADII8YY1dbH0zNg2YELbhJ7+2lF//VHSVJgyAQFhp3GL3AuRogFgDzR0s1csyYP5Yc4XCH21lOKvvyIJCkwZKICw07la9/lCLEAkAcaVmAbB9iB/crT47SAQmaSCcU/WiFJChz+LQWO/B4BFoRYAMh1La3AcjMX3MTy+lR88qWKr35F/oO+wdc9JDFiCwBykjFG0VhS0ViyxRVYAizcILlpTfptq6hUgYOP4+seaazEAkCOaWnltQErsHCL6BtPKPbK/yo46ocKHHK80+UgBxFiASDHROPJFgMsK7Bwi+jrixRb+RdJkolud7ga5CpCLADkAGOMYnFbRkZzFqxMH795xuj0zVsBv4cAi4IXXfW4Yq/+nyQpcOT3FBz6bYcrQq4ixAKAw1prH6isKGXlFa4SXfV3xV79myQpMOxUBQ//lsMVIZcRYgHAIQ2rry21D1RWlGr21GEEWLhG9LXHFHvtMUlSYNhpbCWL3co4xH788cf6zW9+o9dff10lJSU65ZRTdNFFFykQCOzyvK1bt+qmm27SsmXLtG3bNvXr109nnXWWvv/977e7eADIV62tvja0D9A6APdJfb0HjjpdQbaSRRtkFGKrq6s1ZcoU9e/fX/Pnz1dVVZXmzp2rSCSi2bNn7/LcCy+8UKtXr9bFF1+sPffcU8uWLdOVV14pr9erM844o0OfBADkqobV1p21tPrKjVtws+ARp8i798Hy9T3A6VKQJzIKsY8++qjq6up02223qXv37pKkZDKpOXPmaNq0aerTp0+L523cuFH//ve/de211+p73/ueJGnkyJF6++239cQTTxBiARQk2xhdtWClPtuw67urWX2FGxljFPvvC/LvP1yWv0iSCLDISEabHSxbtkwjR45MB1hJGj9+vGzb1vLly1s9L5FISJK6devW5HhpaamMMZmUAAB5wTZGv7r75d0G2IbV12DAS4CFaxhjFHnlr4ouW6Dwkptk7KTTJSEPZbQSu3r1ap166qlNjpWVlal3795avXp1q+ftueeeGj16tO68807tt99+6tu3r5YtW6bly5frhhtuaF/lAJCjzI4V2KqtYUlSnx4hXTF1mCw1D6msvsJtjDHa+vwfFXntcUmSb9+hsjxeh6tCPsooxNbU1KisrKzZ8fLyclVXNx/M3dj8+fM1c+ZMTZiQatb2er369a9/rZNOOimTEprxetk51w0arjPX2x3y8Xo37n2NxpPpFdg+PYt13U9HykNQbVU+Xm+0jzFG0Vf+ovBr/5AkhUadpaLDOpYDkNs681tfl4zYMsbol7/8pT799FPdeOON6t27t1asWKHf/va3Ki8vTwfb9igrC2WxUuQ6rre75MP1NsYoEktq1m0vafW65r/Mz//FNxQKMs2wLfLheqP9jDHa8tzCdIDt9c1zVT7sZIerQj7L6DtrWVmZamtrmx2vrq5WeXl5q+c9//zzWrJkiR5//HENHjxYkjR8+HBt3rxZc+fO7VCIrakJK5lsfucvCovX61FZWYjr7RL5cr2NMfrNg6/qwy9a/kvUAf3KFa6LKFLPKuyu5Mv1RseEVz6myMrHJKUCrDX4G9q6tc7ZotDpystD8ng6568sGYXYAQMGNOt9ra2t1caNGzVgwIBWz/voo4/k9Xo1aNCgJscPOugg/e///q/C4bBCofb9Bp5M2kok+KbnFlxvd8nl622MUW19vEmArawo1azJQ9O9rwG/R8mkkcQNrG2Ry9cbHeepHCLrradVdNT3VD7sZG3dWsf1doHOvH8/oxA7ZswY3XnnnU16Y5csWSKPx6NRo0a1et7ee++tZDKp999/XwceeGD6+LvvvqtevXq1O8ACQFczxigaT2ruwlVNJg/cPGM0M16BXfD2qlTJmXPlL21+bw3QHhmF2EmTJunhhx/W9OnTNW3aNFVVVWnevHmaNGlSkxmxU6ZM0bp167R06VJJqfC711576YILLtD06dNVUVGhl156SX/72980Y8aM7H5GANBJWpv7yiYFQHPGGMVW/kXeysPk65v6S6xVVOpwVSgkGYXY8vJyPfjgg7r66qs1ffp0lZSU6LTTTtPMmTObfJxt20omv5r5VlpaqgceeEA33XSTbrjhBtXW1qpfv36aNWuWJk+enJ3PBAA6kWkhwDa0DwT9zHgFGjPGKPqvPyr+zlLp3WdVMmmePCFWYJFdlsnz3QboqXEHn8+jHj1KuN4ukWvXu6H/9aL5L0n6au4r4TU7cu16o2OMMYqu+IPi7z4jSQqOmarAgcem38/1dpeePUs6bXwec18AYBdaaiG4YuowFQX49gnszBij6PKFiv/nWUmWgmPObhJggWziuzAAtKKlFoKB/coV9LO7ELAzY+wdAfafkiwVjZkq/4FjnC4LBYwQCwCtiMXtr3beooUA2KX4ey98FWCPPUf+wcc4XRIKHCEWABrZefvYBrQQALvmHzxayc/fkm+/I+QfNNrpcuACfEcGALU+/7VBwwYGAL5ijC3JkmVZsrx+FX3zAv5SgS5DiAXgesYYXbtwlT5a2/L2sQP7lSvg75y7a4F8ZYyt6IsPSL6ggiN/kAqyBFh0IUIsANeLxpNNAmxL28fywxn4ijG2ossWKP7+i5JlyX/A0fL23s/psuAyhFgArtG43zV9TEZzFqxMP2b7WGDXjG0rsux+JT54SbIsFX3jfAIsHEGIBeAKu2sZkFIrsARYoHWpAHufEh8s3xFgp8k/cITTZcGlCLEAXGHnloGdVVaUavbUYQRYoBXGthV54T4lPlwuWR4VHTdN/v2HO10WXIwQC6Dg2aZ5y8DOGxbQ9wrsmr3hYyU+WrEjwP5E/v2PcrokuBwhFkBBauh/beh5rdoalkTLANBe3r4HqOgb50ser/wDCLBwHiEWQMFprf+1T48QLQNABoydlInWyRMqkyT5B450uCLgKww+BFBwYnG7WYCtrCjVNeePkIcAC7SJsZOKPHe36h+/RnbdVqfLAZphJRZAQWvof6XnFWg7YycV+eddSqx+RfJ4ZW/5XJ6SHk6XBTRBiAVQcIxM+u2g36tgwLuLjwbQmLETOwLsSsnjVeiEn8u3z6FOlwU0Q4gFUDCMMYrGk00mEQBoO2MnFHn2TiU+eTUVYE/8uXz7Hu50WUCLCLEACoJtjK5asFKfbdiePlZZUaqAn9Z/oC2aBljfjgA7xOmygFYRYgHkpcZbyO48Rkti8wIgUyZar+SWL1IB9ps/l69yiNMlAbtEiAWQd3a1hWyfHiFdMXWYgn4vARbIgCdUpuKJl8veuk6+foc4XQ6wW4RYAHmntS1kG1ZfGaMFtI1JJpSs+lC+vQ6SJHlKejCFAHmDEAsgrxhjNHfhqvTjxlvIMkYLaDuTjCu89HYlP39TRd+YJv/AEU6XBGSEEAsgr8TidvrmLbaQBdonFWBvU/KzNyWvX1ZRqdMlARkjxALICw03ckXjyfSxWZOHEmCBDO0cYEMnXUQPLPISIRZATmuY/Tp34aom47MkyRIBFsiEScRSAfbztyRvQKFxF8m398FOlwW0CyEWQM7a1RSCgf3KmQELZMAkEwovna/k528TYFEQCLEAclYsbjcJsJUVpakWAlncxAVkyuOVp7yvkl++r9C4memJBEC+IsQCyDnGGEWiiSb9rzfPGM1NXEAHWJal4MgfKHDw8fJ07+t0OUCHEWIB5BRjjH7z4Gv68IumLQRsXgBkziSiir25WIEhE2R5U78EWgRYFAhCLICc0Hj6wM4Blv5XIHMmEVX4qVuUXPsf2dvWK3T8T5wuCcgqQiwARzSEVkkyMi1OH7ht5hh5LfpfgUyZRFThJTcrue49yV8k/yHHOV0SkHWEWABdbldTBxoc1L+nuhX7lUyaLqwMyH8mHlX4qa8CbGj8JfL1PcDpsoCsI8QC6HLReLLFANswfcDv86pPRTdt21YviRALtJWJRxVecpOSX/5X8hepePwl8hJgUaAIsQC6lDGp1oEGN88YraDfK0nptgGfj/YBoD3Cz/7+qwB78i/k7TPQ6ZKATkOIBdClYnE73ftaWVHK2CwgiwJDJiiy+TOFTphOgEXBI8QC6BKNpw80mDV5KAEWyCJf30EqOfM6Wb6A06UAnY4QC6BTtGX6gCUCLNARJhZW5Lm7FTjiO/Lusa8kEWDhGoRYAFnXlukDzH4FOsbEwqpffKPsqo+U3LpWJWdcK8vjdbosoMsQYgFkTeOWgV1NH7DE7FegIxoHWAWKFTr+pwRYuA4hFkBWtLb62tL0AQDtZ2L1qn/yRtkbPpaCJSqecKm8e/R3uiygyxFiAWRFLG43C7AD+5UzfQDIIhOtSwXYjasJsHA9QiyArDCNNiVoWH1l5RXIruirf2sUYC9L38wFuBEhFkCH7byBQdDvVTBAfx6QbcGjTpOp26rA0G8TYOF6hFgAHbbzBgZMHQCyxyRi6bFZlr9IoW/OcLgiIDfwkwZAVrGBAZA9Jlqn+sd/q+hrf3e6FCDnEGIBdFjjflg2MACyw0S2q/6JebI3far4u8/IDtc4XRKQU2gnANAhO/fDAui4dIDd/Jmsom4KTbxcnlCZ02UBOYUQC6BdGm9sQD8skD1NAmyoTKEJl8vbc2+nywJyDiEWQMZsY3TVgpXp8NqAfligY+xIrcJPzJO9+fNUgJ14ubw9CLBASwixANrMGKNoPKk5C1aqamu4yfsG9itP78wFoH2SX7zbKMDOkrfHXk6XBOQsQiyANmlpW9k+PUK6YuowWbLY2ADIAv/AETKJqLx9D5C3OwEW2BVCLIA2icaTTQJsZUWpZk8dJg/BFegQO1wjy/LIKiqVJAUOPNbhioD8QIgFsFs7TyC4ecZodSv2s/IKdJBdX63wE9dJHr+KJ1yaDrIAdo/biAHs1s47chFggY6z67cpvOg62VvXyURqZGL1TpcE5BVWYgHsVuPNDJhAAHRcOsBu+1JWSU8VT7xcnrIKp8sC8gohFsAu7dxKwI5cQMc0C7DfmkWABdqBEAtgl3ZuJWAzA6D97PptCv9jruzq9bJKe7ECC3QAIRZAMw27cUmpqQQNaCUAOigRk0lEdwTYWfKU9Xa6IiBvEWIBNNHablwSrQRAR3nKKlQ8cZbk8cjTjQALdAQhFkCabYx+dffLzXbjklI7ctFKAGTO3r5F9rZ18vX7miTJU97H4YqAwkCIBdDidrKNd+OSxI5cQDvY2zer/h9zZeq2KjTuonSQBdBxhFjA5VrbTvaa80ewGxfQAekAW7tRVrfe8nTf0+mSgIJCiAVcju1kgeyzazepftF16QBb/K1Z8pT2crosoKAQYgEXs43RnAUr04/ZThboOLt2444Au0lWWUVqjBYBFsg6QizgUmbHFIKGHli2kwU6zq7flmoh2L5ZVlmfHQG2p9NlAQWJW40Bl4rGk+kxWn16hDR76jACLNBBVlGZvHsOllXeZ0cLAQEW6CysxAIutPNWslfQAwtkheXxqOjYH8vE6uQp6uZ0OUBBYyUWcKGdt5IN+r0OVwTkL7tmgyIvPypjp3a5szweAizQBViJBVzIyKTfZitZoP3smg075sBukeXxKXjUaU6XBLgGIRZwmZ1bCdhKFmgfu7pK9YtSGxl4uu8p/9dOcLokwFUIsYDL7NxKwFayQObs6vWpFdj6bfJ030uhiZfLU1zudFmAqxBiAZcwxigWtxWNJ9PHaCUAMmdvW59aga3fJk+PvRSaQIAFnECIBVygpa1lJVoJgEyZZEL1i2/cEWD3Tq3AhsqcLgtwJf6OCLjAzlvLStLAfuW0EgAZsrw+FY2aLE/v/QiwgMNYiQUK3M43ct08Y7SCfq8Cfg+tBEAbGWPSrxdf5WHy7vN1WRa/BAJO4hUIFLidb+TqVuxXMOAlwAJtlNy6VvV/myO7uip9jAALOI9XIeAi3MgFZCa5da3Ci66TvelTRf71R6fLAdAI7QRAATPGNJlGwI1cQNslt3yh8KLrZCK18vSqVGjseU6XBKARQixQgBrC69yFq9KtBADarmmA3VfFEy6VVVTqdFkAGiHEAgXGNkZXLVjZLLwyjQBom+SWzxVeNC8VYPfYV8UnE2CBXESIBQpAw0YGRkZzFqxU1dZw+n2VFaWaNXmogn5u5gLaIvryn3YE2P6pFdhgidMlAWgBIRbIc61tZNCnR0hXTB1GeAUyFDruJ4r++08KjphEgAVyGCEWyHMtbWRQWVGq2VOHyUN4BdrERLanWwasolIVHXuuwxUB2B1CLJDH2MgA6Ljkpk9V/8T1Ch75XQUOOcHpcgC0EXd5AHnKGKPa+jgbGQAd0BBgFa1T/KOXZezk7k8CkBNYiQXyUEt9sGxkAGQmufFT1T8xT4rVy9NnoIrHXyLL43W6LABtRIgF8lAsbjcJsAP7lSvo54cv0FbJDatV/+QNUqxe3j4HKDT+YlmBkNNlAcgAIRbIczfPGK1uxX5WYYE2SgXY66VYWN6+gxQaN5MAC+QhQiyQ4xpmwDbWeCtZRmgBmUl++V8CLFAACLFADmtt9y0A7Rc47GRZoTL59jtSlr/I6XIAtBPTCYAcZRujX9398i4DLFvJAm2T3LRGJh5JP/YPGk2ABfIcK7FADmoIsA3bxzbsvmWpadsA82CB3Uus/1DhxTfK26sydQMX4RUoCIRYIIcYYxSNJzVnwcomAfaa80ew+xbQDon1Hyi8+HdSPCJ5vBKvI6BgEGKBHNHS7FcCLNB+iS/fTwXYRFTevQ5SaNxFsnxBp8sCkCWEWCAHNOy+1TjAVlaUavbUYQRYoB2aBNi9D1HopAsIsECBIcQCDmtpBZbZr0D7pQLsjVIitiPAXijLF3C6LABZRogFHNbS7lsEWKD9rEBI8vpTc2C/eQEBFihQhFjAYUYm/TYrsEDHeXtVqviUX8lTugcBFihghFjAAQ27cBkZzVmwMn2c3beA9kms/Y/k8cq352BJkrf7Xg5XBKCzEWKBLtbaLlyVFaVsXAC0Q2LtfxRecpNkeVR8yq/k7VXpdEkAugA/MYEuZHYRYGdPHcYqLJChxBfvpgJsMi7vXgfK031Pp0sC0EVYiQW6UCxupwNs41242HkLyFzii3cUfuqWVICtHKLQidNlef1OlwWgixBigS7Q0AMbjSfTx66YOkxFAV6CQHskPn9b4advkZIJ+fY9XEUn/IwAC7gMP0GBTtbSHFhJssTKK9AeyQ2rdwqw02V5+XEGuA2veqCTRePJZgF2YL9ybuIC2snTax959z5ElserouN/RoAFXIpXPtBJjDGKxpNNRmjdPGO0gn4vPbBAB1hev0In/lySRYAFXIxXP5BlDeF17sJVTaYQVFaUspEB0E6JNW8ose49BUdMkmVZ9L8CIMQC2dRa/ysjtID2S6x5XeGlt0l2Ut5e+8g/aLTTJQHIAYRYIIticbtJgK2sKNWsyUPZiQtop/inqxR55nbJTso34Cj5Bo5wuiQAOYIQC2SRkUm/ffOM0bQPAB0Q//Q1RZ75fSrA7j9cRd84X5bH63RZAHJExrdHf/zxx5o6daqGDBmiUaNGad68eYrFYm06t6qqSpdffrlGjBihQw89VOPHj9fjjz+ecdFALjLGaO7CVenHrL4C7Rf/5DVFljYE2BEEWADNZLQSW11drSlTpqh///6aP3++qqqqNHfuXEUiEc2ePXuX527YsEFnnnmm9ttvP1199dUqLS3Vhx9+2OYADOQyY4xq6+PpG7kqK0oZoQW0k12/TZF/3iGZpHwDR6ho7HkEWADNZBRiH330UdXV1em2225T9+7dJUnJZFJz5szRtGnT1KdPn1bPvf7669W3b1/de++98npT34xGjhzZ/sqBHNHSzVyzJg9lFRZoJ09xdxUd+2MlvnhbRWPOleXhF0IAzWX0nWHZsmUaOXJkOsBK0vjx42XbtpYvX97qedu3b9fixYv1gx/8IB1ggULQsALbOMAO7FeuoJ+vcyBTJplIv+0fOEKhsecRYAG0KqOV2NWrV+vUU09tcqysrEy9e/fW6tWrWz3v3XffVTwel8/n0+TJk/X666+re/fu+s53vqOLLrpIfj/z/pB/bGN01YKVTWbBcjMX0D7b/7NcNc8/qtDJl8pT0sPpcgDkgYxCbE1NjcrKypodLy8vV3V1dQtnpGzatEmS9Otf/1pnnHGGfv7zn+utt97SrbfeKo/Ho0suuSTDsr/i9fJbuhs0XOdcuN4Nmxn8v3tfUdWW+vTxA/qVq0dZkACbBbl0vdH5Eh//W1ufvkMytpLv/VOBEac7XRI6Ea9vd+nMH4ldMmLLtm1J0tFHH61Zs2ZJkkaMGKG6ujrdf//9mj59uoqKitr13GVloazVidzn9PU2xujy217Se59uSR/ba48S3XzxWBUFmEaQbU5fb3S+7e+8mA6wpYcep94n/ZCbuFyC1zc6KqMQW1ZWptra2mbHq6urVV5evsvzpFRwbWzkyJG68847tWbNGg0ePDiTUtJqasJKJu12nYv84fV6VFYWcvx6R2KJJgG2sk+prvrxcEXqo4rU7+JEZCRXrjc6V/SDFap/9i7JGHU77DgFjjlb26ojTpeFTsbr213Ky0PydFJve0YhdsCAAc16X2tra7Vx40YNGDCg1fMGDhy4y+eNRqOZlNFEMmkrkeBF4BZOXm9jjH7zwKvpxw39r3bSyG60yQGyh9d34Yp/uEKR5++RjFHgoGO1x4Sfatu2MNfbRXh9u4PpxB+PGUXjMWPGaMWKFaqpqUkfW7JkiTwej0aNGtXqeXvvvbcGDRqkFStWNDm+YsUKFRUV7TbkArkgGk82mQPLDVxA+5hkQrHXF0nGyH/gsSoeO1WWRX8kgMxk9F1j0qRJKikp0fTp0/XSSy/pr3/9q+bNm6dJkyY1mRE7ZcoUnXjiiU3OnTlzpv75z3/qmmuu0fLly3XnnXfq/vvv19lnn63i4uLsfDZAJ7GN0ZwFK9OPmQMLtJ/l9Sk04VIFjviugsdMIcACaJeM2gnKy8v14IMP6uqrr9b06dNVUlKi0047TTNnzmzycbZtK5lMNjl23HHH6Xe/+51+//vf65FHHlFFRYVmzJih888/v+OfBdAJjDGKxW0ZpQJs1dawpNQqLHNggczZtRvl6dZbkuQp6aHgEac4XBGAfGYZ05ndCp1v69Y6empcwOfzqEePki673i3twiVJfXqEdM35I+RhFbZTdfX1RueL/3eZIi8+qKJv/Fj+gU13a+R6uwvX21169izptHFq/A0HaEE0nmwWYCsrSgmwQDvE/vuCIsvul0xSyQ2tb4wDAJnokjmxQD4xxmjuwlXpxzfPGK2g36uA30MfLJCh2HvPK/riA5Ik/yEnKDjyB84WBKBgEGKBnTCFAMiOJgH2aycqOPIHvJYAZA0hFmhk51VYphAA7RP7zz8VfekhSZL/a99UcOT3eS0ByCpCLNDIzquwTCEA2seurpIk+b9+koIjJhFgAWQdIRbYgVVYIHuCIybJu+dg+fY9nNcRgE7BdAJgh1jcZhUW6IDEmtdlknFJkmVZ8vfnF0EAnYcQC7SAVVggM7F3lir81C0KL71Nxk7u/gQA6CDaCQClWgmi8a9+8FoiwAJtFXv7KUX/9YgkydtzH4ltZAF0AUIsXM82RlctWJluJQDQdrG3nlL05VSADQyZqMCwU/krBoAuQYiF6xhjFIuntjo0MpqzYKWqtobT7x/Yr1wBPytJwO7E3lqi6MuPSpICh39LgSO/R4AF0GUIsXAVY4yuXbiq2ZayktSnR0hXTB2moN/LD2JgN2JvP/1VgB16igJHfIfXDYAuRYiFq0TjyRYDbGVFqWZPHSYPP4SBNvH07i/5ggocOk7BI7/rdDkAXIgQC9fYeQ7szTNGp8doBfweVpGADPj6DlLJGb+Vp7SX06UAcCka/+AaO8+B7VbsVzDgVTBA+wDQFrG3nlJy05r0YwIsACcRYuEaRib9NnNggcxEVz2u6MuPqP6JebLDNU6XAwCEWLiDbVJTCBowBxZou+iqxxV79f8kSYFDx8kTKnO4IgCgJxYFrmETg8ZjtCorShmhBbRR9LW/K/ba3yRJgaNOU3DIRIcrAoAUQiwKVkubGPTpEdLsqcNoJQDaIPrq3xRb9XdJUuCo0xUcMsHhigDgK4RYFCTbGP3q7pebbGLAGC2g7eLvv5gOsMHhZyhw2MkOVwQATRFiUXB2DrBsYgBkzjfgKHk/eEm+fYcocOh4p8sBgGYIsSgoLQXYa84fweor0AbGmPQvepY/qNCEy2R5vA5XBQAt4+4WFAyzoweWAAtkzhij2Mq/KvraY+ljBFgAuYyVWBSMxpsZEGCBtksF2L8o9sYTkiTfPl+Xt2J/h6sCgF0jxCJvGWMUi9vpx9F4Mv32FdzABbSJMUaxV/5XsTeflCQFjz6LAAsgLxBikZdaGp/VGJsZALtnjFH0339W/K3FkqTg0ZMV+NoJDlcFAG1DiEXeaWl8VmMD+5WzmQGwG6kA+yfF31oiSQqOmqzAIQRYAPmDEIu80tLNW1dMHdZk5TXg9zBKC9iNZNVHjQLsDxU45HiHKwKAzBBikVe4eQvIDl/fAxQcNVmyPAocfJzT5QBAxgixyFvcvAVkxhgjxSOyAiFJon0AQF6jcRB5i5u3gLYzxii64g+q/8e1MpGWb4gEgHxCiAWAAmeMUXT5QsXffUb25s+VWPee0yUBQIfRToC8YmScLgHIK6kA+7Di//mnJEtFY6bKP2CY02UBQIcRYpEXjDEKRxP6f/e+4nQpQN4wxk6twDYE2GPPkX/wMU6XBQBZQYhFzrON0ZX3vqLPqr7q46usKGUWLLALxtiKvvSQ4u89L8lS0dhz5R802umyACBrCLHIaaaFnbkqK0o1e+owZsECu2DCNUp89pZSAfbH8g8a5XRJAJBVhFjktMZzYffao0RXnjNMXssiwAK74SnuruKJlyu5aY38+x/ldDkAkHX8PRY5yxijaDyZfnzzxWNVFPARYIFWGGMruWlN+rGnvA8BFkDBYiUWOckYo2sXrtJHa6vTx4iuQOuMbSuy7H4lPnpZoW/OkK/yMKdLAoBOxUosco4xRrX18SYB9oB+5QoGvA5WBeSuVIC9T4kPXpKMLROPOF0SAHQ6VmKRMxraB+YuXNXkRq6bZ4xWj7IgbQRAC4xtK/LCfUp8uFyyPCo67ie0EABwBUIsckJL7QOSNLBfuboV+wmwQAtSAfZeJT5ckQqwx/9E/gEEWADuQIhFTojF7SYBtrKiVLMmD1XQ7yXAAi0wtq3I8/co8dG/JMu7I8CyExcA9yDEIic03k725hmjWX0F2sKyUgH2hJ/Kv9+RTlcDAF2KEAvHGWM0d+Gq9GNWX4HdszweFR37Y9mHnCBvxQCnywGALsd0Ajiu8YYGbCcLtM7YScXefVbGtiWlgiwBFoBbsRILxzVuJZg1eSirsEALjJ1Q5J93KbF6pexNn6ro2HOdLgkAHEWIhaN2biWw2NIAaMbYCUWevVOJT16VPF75+g91uiQAcBwhFo5p2NSAVgKgdcZOKPLMHUp8+prk8Sl04s/l23eI02UBgOMIsXBES3NhaSUAmjLJhCLPNgqwbCcLAGmEWDhi57mwA/uVK+hnW1mgscjz96QCrNen0IkXyFd5qNMlAUDOIMTCccyFBVrmP2CUEp+/pdDxP5VvHwIsADRGiIUjGk8kYC4s0DJf5aEq/f4NsoIlTpcCADmHu2jQ5XaeSAAgxSTjirxwv+zq9eljBFgAaBkhFl2KiQRAy0wipvDT8xV/f5nqF98kYyecLgkAchrtBOgyTCQAWmYSMYWX3qbk529J3oCKjpkiy8O3ZwDYFb5Loks0rMAykQBoKhVg5yv5+duSN6DQuIvk2/tgp8sCgJxHiEWna2kFlokEQEMLwa1KfvFOKsCOnynfXgc5XRYA5AVCLDpdSzNhCbCAFH3lL6kA6wsoNI4ACwCZIMSiS7ECC3wlOPTbsjd9qsCR35NvrwOdLgcA8gohFp3KGKNoPJl+zExYuJ2xbVme1EQOq6hUoW/9ktcEALQDIRadoiG8zl24Kj1OC3A7k4gq/NQt8u07VIGvnSBJBFgAaCdCLLKupRu5pFQvLDNh4VYmHlX4qZuVXPeekhtWyzdgmDzF5U6XBQB5ixCLrNv5Rq7KilLNmjyUVgK4lolHFV5yk5Jf/lfyF6l4/CUEWADoIEIsssYYo1jcbtIDy41ccLtUgP2dkl++nwqwJ/9C3j4DnS4LAPIeIRZZYRujqxasbNb/yuor3MzEIztWYN+X/CEVn3wJARYAsoQQiw5puIFrzoKVqtoabvI+emDhdolPXvsqwE74hbwV+ztdEgAUDEIs2q2lG7j69AjpiqnDZMlSwO9hFRau5h80SnZ9tXx7HShvxQCnywGAgkKIRbu1dAPX7KnD5CG4wsVMLPUXCSsQkiQFh5zsZDkAULAIscgKbuACUgG2fvGNsiyPQuMvluUvcrokAChYNCyi3YxM+m1u4ILbmVi96p+8QXbVR0puXSu7drPTJQFAQWMlFhlrfDMXgEYBdsNqKVii4gmXydtzb6fLAoCCRohFRlq6mauyopQpBHAtE61T/ZM3yt7YKMDusa/TZQFAwSPEIiPReLLFm7loJYAbpQLsDbI3fiIrWKrQxMvk7VXpdFkA4AqEWLSZMUZzF65KP+ZmLridXb9NpmajrKJuCk24TN5e+zhdEgC4BiEWbRaL2+kduSorSgmwcD1vj70VmniZJIsACwBdjBCLdpk1eSgBFq5kIttl11Sld9+ifQAAnMHdOGiThokEDSwRYOE+JrJd9U/MU/2ieUqs/9DpcgDA1ViJxW7ZxuiqBSvTrQSAG9mRWoWfmCd78+eyQmWygsVOlwQArkaIxS6ZFgLswH7ljNSCq9iRWoUXzZO9JRVgQxMvl7cHc2ABwEmEWLTKGKPa+ng6wPbpEdIVU4exOxdcxQ7XpFZgt3whK1Su0Lcul7f7Xk6XBQCuR4hFi1pqIbhi6jAVBfiSgXuYyPbUCuzWL2QVd1fxxMvl6b6n02UBAESIRQtaayEI+r0OVgU4wF8kq9sesqLbVTxxljzd+zpdEQBgB0Ismmk8D5YWAriZ5fUpdOJ0mfpqebrt4XQ5AIBGuDsHu9TQQkCAhVvY9dWKvrFIxhhJkuX1E2ABIAexEosmmAcLN7Prtym86DrZ276UkgkFj/iO0yUBAFpBiEUa82DhZnb9NoX/MVd29XpZJT3lP+Bop0sCAOwCIRaSmAcLd7Prtqp+0XUyOwJs8bdmyVNW4XRZAIBdIMRCEjdzwb1SAXauTHWVrNJeqTFaBFgAyHmEWEiSjEz6bebBwi1MMqHwE/MaBdhZ8pT1drosAEAb8LdiyBijuQtXpR9zMxfcwvL6FBh6iqyyih0tBARYAMgXLLdB0Xgy3UpQWVFKHyxcxT9whHz7HSHL63e6FABABkgrLrfzKuysyUPpg0VBs7dvVv2TN8iu25o+RoAFgPxDiHW5xjd0VVaUsrUsCppdu0n1/5ir5BfvKLLsfqfLAQB0AO0ELmWMUSxuN9nYgFVYFDK7dlNqCkHtJlllFSo65mynSwIAdAAh1mUaduSau3BVs00NuKELhcqu3ZiaA1u7SVZZn9QYrdKeTpcFAOgAQqyLGGN07cJV+mhtdbP3sbEBCpVdszG1Art9s6zyPqkxWiU9nC4LANBBhFgXicXtJgG2sqI01UIgSwG/h1YCFKTIiw/sCLB9UyuwBFgAKAiEWBdpvKHBzTNGq1uxn+CKglc09seKvPigio6ZQoAFgAJCiHWJnUdpsaUsCplJxGT5ApIkT0kPFY+7yNmCAABZRxOkS+w8Sov+VxQqu7pKdX/+peIfrnC6FABAJyLJuBCjtFCo7Or16Zu4Ym8+KWMnnC4JANBJaCdwIUZpoRDZ23YE2Ppt8vTYS6GTL5Xl4VscABQqvsO7QMNsWKBQ2du+TM2Brd8mT4+9FZp4uTyhMqfLAgB0IkJsgdvVbFigECS3rVN40bxUgO3ZT6EJlxFgAcAF6IktcDvPhmVTAxSaxMevEGABwIVYiS1wzIZFoQsMPUWWLyjf4NHyFHVzuhwAQBdhSa6AMRsWhcqu2SCTjEuSLMtS4LDxBFgAcBlCbAFjNiwKUXLLF6p/7GqFl96WDrIAAPch1bgEs2FRCJJbvlB40XUykVqZum1SIuZ0SQAAh9AT6xLMhkW+S275PDWFIFIrzx77qvjkS2UFS5wuCwDgkIxXYj/++GNNnTpVQ4YM0ahRozRv3jzFYpmthjzwwAMaPHiwpk2bluk/D8CFkps/U/gf1+0IsP1VPOEyWUWlTpcFAHBQRiux1dXVmjJlivr376/58+erqqpKc+fOVSQS0ezZs9v0HBs3btTtt9+uXr16tatgtF3jyQRAvkpu/iy1AhvdLk/v/VR88i9YgQUAZBZiH330UdXV1em2225T9+7dJUnJZFJz5szRtGnT1KdPn90+x/XXX6/jjjtO69ata1fBaBvbGM1ZsNLpMoCOi0dl7AQBFgDQREbtBMuWLdPIkSPTAVaSxo8fL9u2tXz58t2e/+qrr+qZZ57RJZdcknGhaDtjjK5asFJVW8OSmEyA/Obte4CKJ15OgAUANJHRSuzq1at16qmnNjlWVlam3r17a/Xq1bs8N5lM6uqrr9ZPfvITVVRUZF5pK7xewlljxhjV1ifSo7X69CzWVecNlyfPJxM0XGeutzuYzWsUjYbkLe4rSfLtub/DFaEz8fp2F663u3Rm/MgoxNbU1KisrPmWjuXl5aqurm7hjK/88Y9/VDgc1tlnn51RgbtTVhbK6vPlM2OMLr/tJb336Zb0sfm/+IZCwcIZQsH1LnzRdR/py7/PVY1laa8fXq1A70qnS0IX4fXtLlxvdFSXpJvNmzfr1ltv1XXXXadAIJDV566pCSuZtLP6nPkqGks2CbAH9CtXuC6iSH1+r8JKqd/Yy8pCXO8Cl6hare2L5slE6xXsN1j1VqnqttY5XRY6Ga9vd+F6u0t5eUgeT+esumcUYsvKylRbW9vseHV1tcrLy1s975ZbbtHgwYN15JFHqqamRpKUSCSUSCRUU1Oj4uJi+Xzty9PJpK1EgheBpCb/HW6eMVrdiv1KJo1UQFMKuN6FK7lhteqfvF6KheXte4D2nPT/VF3P9XYTXt/uwvV2B9OJESSj5DhgwIBmva+1tbXauHGjBgwY0Op5n3zyiVauXKlhw4Y1e9+wYcN0zz33aMyYMZmUgt0I+r3s0IW8kdzwseqfuEGKh+XtO0jdJl4iTzAk1bMKCwBoWUYhdsyYMbrzzjub9MYuWbJEHo9Ho0aNavW8//mf/0mvwDb47W9/q6KiIl188cUaPHhwO0p3N2OMYvGmv8FG40mHqgHaL7n5M9U/cb0Uj8i752CFxs2UFaBXDgCwaxmF2EmTJunhhx/W9OnTNW3aNFVVVWnevHmaNGlSkxmxU6ZM0bp167R06VJJ0kEHHdTsucrKylRcXKzhw4d38FNwj4bgamQ0d+Gq9AQCIJ95yvrIu8e+kpQKsP4ihysCAOSDjEJseXm5HnzwQV199dWaPn26SkpKdNppp2nmzJlNPs62bSWTrApmkzFG1y5cpY/W7noKhCQN7FfOXFjkDcsfVGjcTEmWLH/Q6XIAAHnCMqYzW24739atda5oDI/Gkvrp715ocqyyolSzJg+Vpaa9rwG/p+D6YX0+j3r0KHHN9S50ifUfKvnl+woePrHF93O93YXr7S5cb3fp2bOk02YCF84AURe5ecZoBf3eggyrKHyJL99XePHvpERUnpIe8g9qvZ8eAIDWEGLzgDGmyU1bQb9XwYDXwYqA9mkcYL17HyLfgCOdLgkAkKcIsTnONkZXLVjJTVzIe4l1/1V4yU3pABs66UJZvuxufgIAcA9CbA4zLQRYbtpCPkqse29HgI3J2+9rCn3zAgIsAKBDCLE5LBpPpgNsnx4hXTF1GJsYIO/Y9dsUXnJzKsDu83WFTpxBgAUAdBghNkfZxmjOgpXpx1dMHaaiAJcL+cdT3F3B4Wco8dmbCp34cwIsACArSEU5qKGNoGprWFJqlFbQz41cyC/GmPRfDQKHHC//wd+QZdEKAwDIDn6i5KBY3G7SRjB76jBaCJBXEl+8o/rHr5GJfNXPTYAFAGQTP1Vy3BVTh8lDgEUeSXzxjsJP3SK76iNF31jkdDkAgAJFO0GO23k3LiCXJT5/W+Gnb5GSCXkrhyg47FSnSwIAFChCLICsSHz+lsJP3yolE/Lte7iKTpguy8u3GABA5+AnDIAOS3z2psJPz5fshHz9h6ro+J8RYAEAnYqfMjnIyDhdAtBmxk4osuKPOwLsESo64aeyPHxrAQB0Ln7S5BhjjOYuXOV0GUCbWR6fik++RLG3nlLw6O8TYAEAXYLpBDmm8XityopStphFzrIjtem3PWUVKhr9QwIsAKDLkJByTONWglmThzIfFjkp/ukq1T1yqRJr3nC6FACASxFic8jOrQSM10Iuin/6miJLb5fiEcVXv+J0OQAAl+JvfzmEVgLkuvgnrynyzO8lk5Rv/+EqOvZcp0sCALgUITaH0EqAXBZfvVKRZ+/cEWBHqOgb58nyeJ0uCwDgUoTYHEErAXJZKsDeIRlbvoEjVTT2xwRYAICjCLE5IhpP0kqAnJX8/O1GAfY8WR6+PgEAziLE5oCdV2FpJUCuCR5ztjx99pd/0DEEWABATuCnUQ7Y+YauoJ8/08J5ifUfyti2JMnyeBQ48FgCLAAgZ/ATKQdwQxdyTfyjfyn8j98q8vw96SALAEAuoZ3AYbYxmrNgZfoxN3TBafEPVyjy/D2SMbK8fvElCQDIRYRYBxljdNWClaraGpbEDV1wXuMA6z9wjILHnC3L4msSAJB7CLEOatwL26dHSLOnDqOVAI6Jf7BckefvlWTkP3Csgsf8iAALAMhZhNgcccXUYfIQYOGQJgH2oLEKjibAAgByGyE2R9ALCydZRd0kjzfVQjBqMgEWAJDzCLEA5Ks8VMXfu0KeHv1oaQEA5AWWWwCXin+4Qnb1+vRjb899CLAAgLxBiHVQ4/mwQFeKvfe8Is/drfp/zJVdX+10OQAAZIwQ65Cd58MCXSX23vOKvviAJMk3YJisUJmzBQEA0A70xDqA+bBwSuw//1T0pYckSf6vn6TgiEm0EAAA8hIh1gHMh4UTYu8+q+jyhyURYAEA+Y8Q6zDmw6IrxD96+asAe+g4BYefSYAFAOQ1QqwDGt/QxXxYdAVfv6/J02tf+fodosBRpxNgAQB5jxDbxbihC06wikpV/O3/kXwBAiwAoCAQYrsQN3ShK8XeflqyLAW+dqIkyfIHHa4IAIDsIcR2IW7oQleJvfWUoi8/Ikny9t5P3j4DHa4IAIDsIsR2oca9sNzQhc4Se2uxoi//SZIUOPxb8lTs73BFAABkHyG2ixhjNHfhqvRjbuhCZ4i9uVjRf+8IsEO/rcAR32W1HwBQkAixXaRxKwG9sOgM0TeeVOyVP0uSAkNPUfDI7zpcEQAAnYcQ20UatxLMmjyU1TFkVbLqo68C7BHfUfCI7zhbEAAAnYwQ2wVoJUBn8/YZqMCR35NkFBx6itPlAADQ6QixXYBWAnQWk0zI8qZexsGh33a4GgAAug5pqovRSoBsia56XOEn5snEI06XAgBAlyPEdgG2mUW2RV/7u2Kv/p+S6z9Q4tNVuz8BAIACQztBJ2ObWWRb9LXHFHvtMUlS4KjT5T/gaGcLAgDAAYTYTsQ2s8gmY4xirz2m2Kq/S5ICR52h4JCTHa4KAABnEGI7EdvMIltSAfZviq16XJIUHH6mAoeNd7gqAACcQ4jtImwzi44w4WrF//OcJCk4YpICh45zuCIAAJxFiO0i3NCFjvAUd1dowmVKrv9AgUOOd7ocAAAcR4gFcpQxRqZ2kzxlvSVJ3l77yNtrH4erAgAgN3CXEZCDjDGKvfK/qvvLr5X48n2nywEAIOcQYoEcY4xR9N9/VuzNJ6VEVPbWtU6XBABAzqGdoJMYYxSNJ50uA3kmFWD/pPhbSyRJwVE/VODg4xyuCgCA3EOI7QTGGF27cJU+WlvtdCnII8YYRV9+VPG3n5IkBUf/iAALAEArCLGdIBa3mwTYgf3K2eQAu2SMUfRfjyj+ztOSpODoKQoc/A2HqwIAIHcRYjvZzTNGq1uxn00OsGsmKVO7UZIUPOZsBQ4a62w9AADkOEJsJwv6vQRY7Jbl8anohOlKrvuPfPsc6nQ5AADkPP7GDTjEGKP46ldkjJEkWV4fARYAgDYixAIOMMZWdPnDijzze0X/9UenywEAIO/QTtAJjIzTJSCHGWMr+tLDir/3nCRL3j32dbokAADyDiE2ixpmw85ZsNLpUpCjUgH2IcXfe16SpaKxP5Z/0CinywIAIO8QYrOgIbzOXbhKn23Ynj5eWVHKaC2kGWMr+uKDiv/3BcmyVDT2PPkPONrpsgAAyEuE2A5qbWODyopSzZ46jMkESIu+9DABFgCALCHEdtDOGxtUVpRq1uShjNZCM949Byv+/osqGnuu/ANHOl0OAAB5jRCbRWxsgF3xDxwhb98D5Cnt5XQpAADkPRo2s4jVVzRmbFvRV/4ie/uW9DECLAAA2UGIBTqBsW1FXrhXsTcWKfzkDTJ2wumSAAAoKLQTAFlmbFuR5+9R4qN/SZZHgSO/I8vDSw0AgGziJyuQRcZO7giwL0uWR0XH/1T+AcOcLgsAgIJDiAWyxNhJRZ67R4mPX5Ysr4qO/wkBFgCATkKIBbIktvKvXwXYE34m/35HOF0SAAAFixDbQUbG6RKQI/xf/6YSn72lwLDvyt+fAAsAQGcixHaAbYzmLFjpdBlwkDEmPVbNU9xdxafOkeXxOlwVAACFjxFb7WSM0VULVqpqa1hSaqeugJ//nG5i7IQiz96h+Icr0scIsAAAdA1WYtspGk/qsw3bJUl9eoQ0e+owNjpwkVSAvVOJT15VYs0b8u59sDzF3Z0uCwAA1yDEtsPObQRXTB0mDwHWNYydUOSZO5T49DXJ41PoxJ8RYAEA6GKE2Ay11EYQ9PMnZLcwyVQLQeLT1ySvT6ETL5Cv8lCnywIAwHUIsRmijcC9TDKhyDO3K7Hm9VSA/eYF8u1DgAUAwAmE2AwYYzR34ar0Y9oI3CXx0b8aBdgL5dvn606XBACAaxFi28gYo9r6eHoVljYC9/ENGq3Ati/l3ftg+fp9zelyAABwNUJsGxhjdO3CVfpobXX62KzJQ2kjcAGTjEuSLK9flmUpOPwMhysCAAASIbZNovFkkwA7sF85q7AuYBIxhZfeJkkKfXOGLK/f4YoAAEADQuxu7NwHe/OM0epW7GcVtsClAux8JT9/W/IGZG9ZK2/v/k6XBQAAdiDE7kYsbjfpgyXAFj6TiCn89K1KfvGO5A0oNH4mARYAgBxDiN0NI5N+mz7YwtckwPoCCo2bKd9eBzldFgAA2Akhdhd2biWwRIAtZCYRU/ipW5Rc+67kCyo0/mL59hzsdFkAAKAFHqcLyGU7txIE/PznKmR29XolN3xMgAUAIA+wEttGtBIUPm+vSoXGXyIZmwALAECOI8S2Ea0EhcnEo7K3b5K3x96SJF/fAxyuCAAAtAV/H4drmXhU4SU3qf7x3yq5aY3T5QAAgAwQYneh8WQCFBYTjyi85HdKfvlfyU5KO3bmAgAA+YF2glbsPJkAhcPEIwov/p2S6z+Q/CEVn3yJvH0GOl0WAADIACG2FUwmKEwmFlZ4yU1fBdgJv5C3Yn+nywIAABkixLYBkwkKg4mFUyuwVR9KgZCKT75U3ooBTpcFAADagRDbBkwmKBCWR/J4pUCxiidcKm/v/ZyuCAAAtBMhFq5h+YMKjZspu3ajvD37OV0OAADoABo9W8FkgsJgYvWK/feF9GPLHyTAAgBQAFiJbQGTCQqDidap/skbZW9cLROpU3DIyU6XBAAAsoQQ2wImE+S/VIC9QfbGT6RgiXz9DnG6JAAAkEWE2N1gMkH+aRxgrWCpQhMvk7dXpdNlAQCALCLEtqBxPyyTCfKLidap/onrZW/6VFZRN4UmXCZvr32cLgsAAGQZIXYn9MPmL2MnmgbYiZfJ25MACwBAIaLZcyf0w+Yvy+OT/8AxskJlCk28nAALAEABYyV2F+iHzT+Bg4+Tf//hsoIlTpcCAAA6EcuMjRhjFI0n04/ph819dqRW4X/eJRPZnj5GgAUAoPCxEruDMUbXLlylj9ZWO10K2siO1Cq8aJ7sLZ8rHN2u4vGXOF0SAADoIhmH2I8//li/+c1v9Prrr6ukpESnnHKKLrroIgUCgVbP2bBhgx544AEtX75cn332mbp166Zhw4bp4osv1t57792hTyBbYnG7SYAd2K+cftgcZodrFH5inuwtX8gKlSs48vtOlwQAALpQRiG2urpaU6ZMUf/+/TV//nxVVVVp7ty5ikQimj17dqvnvfvuu1q6dKlOPfVUHXbYYdq6davuuOMOnX766Vq0aJF69uzZ4U8km26eMVrdiv30w+YoO1yTWoHdmgqwoW9dLm/3vZwuCwAAdKGMQuyjjz6quro63XbbberevbskKZlMas6cOZo2bZr69OnT4nlHHHGEFi9eLJ/vq39u6NChGjt2rB577DGdc8457f8MsqTxbNig30uAzVF2fY3Ci66TvXWtrOLuKp54uTzd93S6LAAA0MUy+nv5smXLNHLkyHSAlaTx48fLtm0tX7681fPKysqaBFhJ6tu3r3r27KkNGzZkVnEnYDZs/qj7592NAuwsAiwAAC6VUYhdvXq1BgwY0ORYWVmZevfurdWrV2f0D3/yySfavHmz9t9//4zO6wzMhs0fxcf8UJ6K/VX8rVnydO/rdDkAAMAhGbUT1NTUqKysrNnx8vJyVVe3/a5+Y4x+85vfqKKiQhMmTMikhGa83o4HzqT9VSvBr88+Un6/t8PPiewxtp2+zoGee6rs1Nm0exS4huudjdc3ch/X21243u7SmT+uHRmxNX/+fL388su69957VVxc3KHnKisLdbieSDSRfrtH9xIVBZk8lisStVu1/tGr1WPsD6SyI7NyvZE/uN7uwvV2F643OiqjtFZWVqba2tpmx6urq1VeXt6m5/jzn/+s22+/Xddcc41GjhyZyT/fopqasJJJu0PPEY19tcHBtm31CgZYic0Fdt1W1f79Wtnb1mvjU/ereMAQ1dbFO3y9kfu8Xo/KykJZeX0j93G93YXr7S7l5SF5PJ2z6p5RiB0wYECz3tfa2lpt3LixWa9sS5YuXaorr7xSF1xwgU477bTMKm1FMmkrkejYi6Dx+YmELa+HP1U7za7bqvpFc2Wqq2SV9lLpty6T5fUpmYx2+Hojf2Tj9Y38wfV2F663Oxiz+49pr4yi8ZgxY7RixQrV1NSkjy1ZskQej0ejRo3a5bn//ve/dfHFF+v000/X9OnT21ctXGHnAFs8cZa8Zb2dLgsAAOSQjELspEmTVFJSounTp+ull17SX//6V82bN0+TJk1qMiN2ypQpOvHEE9OPP/74Y02fPl39+/fXKaecojfeeCP9v88++yx7n007NZ4RC2fZ27eo/h87Amy3PVJTCAiwAABgJxm1E5SXl+vBBx/U1VdfrenTp6ukpESnnXaaZs6c2eTjbNtWMvlVn+mbb76p2tpa1dbW6vvfb7o96He/+13NnTu3A59CxzAjNrfE//NPmZoqWd16pzYy6LaH0yUBAIAcZBnTmd0KnW/r1roO9dREY0n99HcvSErNiL1i6jDGNznI2LZiK/8i/yHHy1PaK33c5/OoR4+SDl9v5Aeut7twvd2F6+0uPXuWdNo4NWZJNTJr8lACrAPs+m2yirrJ8nhleTwKDj/D6ZIAAECOY9JwI5YIsF3Nrt2o+r//RpHn75Gx+Y0cAAC0DSuxcIxduzF1E9f2zUpaHpnodlmh5jvCAQAA7IwQC0fYNRtTY7S2b5ZV3kfFE2fJQ4AFAABtRIhFl7NrNqh+0XVNA2xJD6fLAgAAeYQQiy5l12xItRDUbZFV3jc1RosACwAAMkSIRZeyazbKRGrkKe+r0LdmyVPc3emSAABAHnJ9iGW3rq7l63eIQuMulqfHXgRYAADQbq4OsezW1TXs6vWSJE95X0mSb++DnSwHAAAUAFfPiY3FbX22Ybuk1G5dAb+r/3N0CnvbetX/Y67q/zFXdnWV0+UAAIACQWrbgd26ss/e9mVqjFb9NlnBUikQcrokAABQIFzdTtAYu3VlV3LbOoX/cZ1MuFqenv0UmnAZc2ABAEDWuDbEGmMUjSedLqMgJbeuU3hRQ4DdR6EJlxJgAQBAVrkyxBpjdO3CVfpobbXTpRQce9uXCi+aKxOukafXPqkV2KJuTpcFAAAKjCtDbCxuNwmwA/uVc1NXllihMlklPWQVd1fxhMtkFZU6XRIAAChArgyxjWfD3jxjtLoV+7mpK0usYImKT7409TYBFgAAdBLXLT/axmjOgpXpx0G/lwDbQcktnyv23vPpx1ZRKQEWAAB0KletxBpjdNWClaraGpbEbNhsSG7+XOEn5slEamX5i+QfOMLpkgAAgAu4KsQ23tygT4+QZk8dxipsByQ3f6bwonky0e3y9N5Pvn2+7nRJAADAJVwVYhu7YuoweQiw7ZbctEbhJ65PB9jik38hK1jidFkAAMAlXBti2dyg/ZKb1qj+iXlStE6e3gNUfPIlBFgAANClXBti0T52ffVXAbZiQGoFNlDsdFkAAMBlXHVXU+PRWmgfT3G5Al8/SZ6K/VV88qUEWAAA4AjXrMQaYzR34SqnyygIwaHfVuCw8bK8fqdLAQAALuWaldjGkwkYrZWZ5IbVql9yk0wsnD5GgAUAAE5yZZKbNXkoo7XaKLlhteqfvF7Jz95U9NW/OV0OAACAJDe1EzTqh2UyQdskN3ys+idukOJhefsOUvDI7zpdEgAAgCSXhFj6YTOXrPpI9U/emA6wofEXy/IXOV0WAACAJJeEWPphM5MKsDdI8Yi8ew5WaNxMAiwAAMgprgixjdEPu2vGTij8z7t2BNgDdwTYoNNlAQAANOG6JUn6YXfN8vgU+uYF8g0YptB4AiwAAMhNrguxaJlJRNNve3vto9AJ02X5CLAAACA3EWKhxJfvq+6RS5X48n2nSwEAAGgTQqzLJdb9V+HFv5MJ1yj+9tNOlwMAANAmrruxC19JrHtP4SU3SYmYvP2+pqLjpjldEgAAQJsQYl0qse49hRffJCVTATb0zQtk+QJOlwUAANAmhFgXSqz9j8JLbk4F2H2+rtCJMwiwAAAgr7gixDbechZS/L/LdgTYQxU68ecEWAAAkHcKPsSy5WxzRWN/rFjPfgocepIsr9/pcgAAADJW8NMJ2HI2Jbl1rYxJrUhbXp+Ch08kwAIAgLxV8ImucSuBW7ecTXz+tur/7wpFly9MB1kAAIB8VtAhdudWAjduOZv4/C2Fn75FSiZk6rZIJul0SQAAAB1W0D2xbm8lSHz2lsJLb5WSCfn6D1XR8T+T5SnoSw4AAFyioBONm1sJEp+9qfDT8yU7IV//I1R0wk8JsAAAoGAUbKpxcytBYs0bCi+9LRVg9ztSRcf/hAALAAAKSsH+fd3NrQQmEZVMkgALAAAKVsGmGze3Evj3Hy6ruLu8ffYnwAIAgIJUkMuTbmwlSHz2luztW9KPfXsOJsACAICCVZAh1m2tBPFPXlP4qVtUv+g62eEap8sBAADodAW5VOemVoL46pWKPHunZJLyVuwnK1jidEkAAACdruBCrJtaCVIB9g7J2PINHKmisefJ8hT2qjMAAIBUgCHWLa0E8dWv7FiBteU74GgVHftjAiwAAHCNgguxjRVqK0FizetfBdhBo1Q05lwCLAAAcJWCDrGF2krg6b2frLIKefsMVNGYcwiwAADAdQouxDa+qatQeYq7q+SUX0vBYlkWARYAALhPQYXYnW/qKiTxD1dIkvwHHC1JsopKnSwHAADAUQUVYgv1pq74B8sVeeFeSZJnRxsBAACAmxVGymtBodzUFf9guSLP3ysZI//gMfJUDHC6JAAAAMcV1EpsY4VwU1f8g5cUef4+SUb+g8YqOPpH9MACAACogENsvou//6IiL9wvych/8HEKjppMgAUAANiBEJuDkhs+bhRgj98RYPN/ZRkAACBbCLE5yNN7gPxfO1EySQWPJsACAADsrKBCbL7PiDXGyLIsWZal4MjvSxIBFgAAoAUF02SZ7zNiY+89r/BTN8sk45KUDrMAAABormBCbD7PiI395zlFX3xAyc/eTG9qAAAAgNYVVDtBg3yaERv7zz8VfekhSZL/6yfJP3iMwxUBAADkvoIMsfkyIzb27jOKLl8oSfIfOk7B4WfmTfgGAABwUsGE2Hy7qSv2zlJFV/xBkuQ/dLyCw88gwAIAALRRQYTYfLupyw7XKLryr5KkwGEnK3DU6QRYAACADBREiM23m7o8oTIVj79EiS/eUeCI7xBgAQAAMlQQIbaxXL6py47UylPUTZLk7XuAvH0PcLgiAACA/JTbS5btkKs3dcXeWqK6P81SctMap0sBAADIewUXYnNR7M3Fir78qBStU+Lzt50uBwAAIO8VRDtBLk8miL35pKL//rMkKTD0FAUPn+hwRQAAAPkv70NsLk8miL7xhGKv/K8kKXDEdxQ84jvOFgQAAFAg8j7E5upkgujrixRb+RdJUuCI7yp4xCkOVwQAAFA48j7ENpYrkwmMnVTyi3ckSYEjv6fg0G87XBEAAEBhKagQmyuTCSyPV6FxM5X45FX5B41yuhwAAICCkxt/ey8QiXX/Tb9t+YMEWAAAgE5CiM2S6GuPKbxorqKv/s3pUgAAAApeQbUTOMEYo9hrjym26u+SJMtf5HBFAAAAhY8Q2wGpAPs3xVY9LkkKjjhTgUPHO1wVAABA4SPEtpMxRrFX/0+x1/8hSQqOmKTAoeMcrgoAAMAdCLHt1DTAfl+BQ09yuCIAAAD3IMS2k1XSQ5IUHPkDBb7+TYerAQAAcBdCbDsFDj5O3r6D5O3Zz+lSAAAAXIcRW21kjFHs7adkR2rTxwiwAAAAzsj7EGtkOv/fMEbRlx9V9F+PKPzE9TLJRKf/mwAAAGhdXodYY4x+8+Crnf5vRF9+VPG3n5Ik+Q8aK8tLFwYAAICT8jqNGUmfVW2XJFVWlCrgz24mN8Yo+q8/Kv7OUklScPQUBQ7+Rlb/DQAAAGQur0NsY7MmD5VlWVl7vmYB9pizFThobNaeHwAAAO1XMCHWUvYCrCTFVv39qwA7ZqoCBx6b1ecHAABA++V1T2xn8h9wtKzSXgRYAACAHFQwK7HZ5imrUMnpv5XlDzpdCgAAAHbCSuwOxtiKrPiDEmveSB8jwAIAAOQmQqxSATb60kOKv7NU4Wdul12/zemSAAAAsAt53U6wpTrS4ecwxlb0xQcV/+8LkiwVHXO2PMXdO/y8AAAA6Dx5HWITSVtS+2fEGmMruuwBxd9fJlmWisaeJ/8BR2e7TAAAAGRZXofYBu2ZEWuMrcgLC5T44MVUgP3G+fIPHNlJFQIAACCbCiLEtmdGbOKD5Y0C7DT5B47ohMoAAADQGQoixLaH74BR8q//QN69DyHAAgAA5BlXhVhj25KMLI9XlsejomPPdbokAAAAtINrRmwZ21bkhXsVef4eGTvpdDkAAADoAFeEWGPbijx/jxIfrlDi45WyN37idEkAAADogIJvJzB2MhVgP3pZsrwqOuGn8vYZ6HRZAAAA6ICCDrHGTiry3N1KfPzvHQH2Z/Lvd4TTZQEAAKCDCjbEGjupyD/vUmL1K5LHq6ITpsvff6jTZQEAACALCjbE2ls+V2LNKsnjVeiEn8vX/3CnSwIAAECWFGyI9e7RX6FvXiDZSfn2JcACAAAUkoIKscZOyNRXy1PaS5Lk2+dQhysCAABAZ8h4xNbHH3+sqVOnasiQIRo1apTmzZunWCy22/OMMbr77rs1duxYHXrooTrzzDP1xhtvtKfmlp8/mVDkmTtU/9jVsretz9rzAgAAIPdkFGKrq6s1ZcoUxeNxzZ8/XzNnztSf//xnzZ07d7fn3nPPPbr11lt19tln66677lLv3r11zjnn6PPPP2938ZJU2adUfo+tyLO/V+LT12Qi22XXbuzQcwIAACC3ZdRO8Oijj6qurk633XabunfvLklKJpOaM2eOpk2bpj59+rR4XjQa1V133aVzzjlHZ599tiTpiCOO0Lhx43TffffpyiuvbPcn8KvJQxR99vdKrHld8voU+uYF8u3z9XY/HwAAAHJfRiuxy5Yt08iRI9MBVpLGjx8v27a1fPnyVs9btWqVtm/frvHjx6ePBQIBnXjiiVq2bFnmVTcSbxJgL6QPFgAAwAUyCrGrV6/WgAEDmhwrKytT7969tXr16l2eJ6nZufvvv7/WrVunSCSSSRlp3aywkp+9IXn9Cp10ESuwAAAALpFRO0FNTY3KysqaHS8vL1d1dfUuzwsEAgoGg02Ol5WVyRij6upqFRUVZVKKJClU3kP7TP+9PKFyyRfI+HzkD8tK/X95eUjGOFsLOh/X21243u7C9XYXj8fqtOfO6xFbHq9Xnu4t9+GiMHk8GQ/UQB7jersL19tduN7oqIy+gsrKylRbW9vseHV1tcrLy3d5XiwWUzQabXK8pqZGlmXt8lwAAABgZxmF2AEDBjTrfa2trdXGjRub9bvufJ4kffLJJ02Or169WnvttVe7WgkAAADgXhmF2DFjxmjFihWqqalJH1uyZIk8Ho9GjRrV6nlDhw5VaWmpFi9enD4Wj8f19NNPa8yYMe0oGwAAAG6WUU/spEmT9PDDD2v69OmaNm2aqqqqNG/ePE2aNKnJjNgpU6Zo3bp1Wrp0qSQpGAxq2rRpmj9/vnr27KlBgwbpkUce0bZt23Tuuedm9zMCAABAwcsoxJaXl+vBBx/U1VdfrenTp6ukpESnnXaaZs6c2eTjbNtWMplscuy8886TMUb333+/tmzZooMOOkj33Xef9tlnn45/FgAAAHAVyxgGXAAAACC/MN8CAAAAeYcQCwAAgLxDiAUAAEDeIcQCAAAg7xBiAQAAkHcIsQAAAMg7ORliP/74Y02dOlVDhgzRqFGjNG/ePMVisd2eZ4zR3XffrbFjx+rQQw/VmWeeqTfeeKPzC0aHtOd6b9iwQfPmzdMpp5yiww8/XGPGjNEll1yitWvXdlHVaK/2vr4be+CBBzR48GBNmzatk6pEtnTkeldVVenyyy/XiBEjdOihh2r8+PF6/PHHO7lidER7r/fWrVs1e/ZsjR07VkOGDNHEiRP1yCOPdEHF6Ig1a9Zo9uzZOuWUU3TwwQdr4sSJbTovW3kto80OukJ1dbWmTJmi/v37a/78+aqqqtLcuXMViUQ0e/bsXZ57zz336NZbb9UvfvELDR48WH/4wx90zjnn6O9//zubKuSo9l7vd999V0uXLtWpp56qww47TFu3btUdd9yh008/XYsWLVLPnj278LNAW3Xk9d1g48aNuv3229WrV69OrhYd1ZHrvWHDBp155pnab7/9dPXVV6u0tFQffvhhxr/woOt05HpfeOGFWr16tS6++GLtueeeWrZsma688kp5vV6dccYZXfQZIFMffvihXnjhBR122GGybVtt3Xoga3nN5Jg777zTDBkyxGzdujV97NFHHzUHHXSQWb9+favnRSIRM3ToUHPjjTemj0WjUfONb3zDXHHFFZ1YMTqivde7urraxOPxJse+/PJLM3jwYHPfffd1VrnooPZe78YuvfRSc9lll5nJkyeb888/v5MqRTZ05Hr/4he/MGeeeaZJJBKdXCWypb3Xe8OGDWbQoEHmr3/9a5PjZ511lvnRj37UWeUiC5LJZPrtyy+/3EyYMGG352Qzr+VcO8GyZcs0cuRIde/ePX1s/Pjxsm1by5cvb/W8VatWafv27Ro/fnz6WCAQ0Iknnqhly5Z1ZsnogPZe77KyMvl8Tf+Q0LdvX/Xs2VMbNmzorHLRQe293g1effVVPfPMM7rkkks6sUpkS3uv9/bt27V48WL94Ac/kNfr7YJKkQ3tvd6JREKS1K1btybHS0tL27yyB2d4PJnHyGzmtZwLsatXr9aAAQOaHCsrK1Pv3r21evXqXZ4nqdm5+++/v9atW6dIJJL9YtFh7b3eLfnkk0+0efNm7b///tksEVnUkeudTCZ19dVX6yc/+YkqKio6s0xkSXuv97vvvqt4PC6fz6fJkyfrkEMO0ahRo3T99dcrHo93dtlop/Ze7z333FOjR4/WnXfeqY8++kjbt2/Xk08+qeXLl+uss87q7LLRxbKZ13KuJ7ampkZlZWXNjpeXl6u6unqX5wUCAQWDwSbHy8rKZIxRdXW1ioqKsl4vOqa913tnxhj95je/UUVFhSZMmJDNEpFFHbnef/zjHxUOh3X22Wd3UnXItvZe702bNkmSfv3rX+uMM87Qz3/+c7311lu69dZb5fF4WInPUR15fc+fP18zZ85Mf//2er369a9/rZNOOqlTaoVzspnXci7EAu0xf/58vfzyy7r33ntVXFzsdDnIss2bN+vWW2/Vddddp0Ag4HQ56GS2bUuSjj76aM2aNUuSNGLECNXV1en+++/X9OnTWZQoIMYY/fKXv9Snn36qG2+8Ub1799aKFSv029/+VuXl5SxMoFU5F2LLyspUW1vb7Hh1dbXKy8t3eV4sFlM0Gm2S7mtqamRZ1i7PhXPae70b+/Of/6zbb79d11xzjUaOHJntEpFF7b3et9xyiwYPHqwjjzxSNTU1klJ9dIlEQjU1NSouLm7WIw3ndeT7uZQKro2NHDlSd955p9asWaPBgwdnt1h0WHuv9/PPP68lS5bo8ccfT1/X4cOHa/PmzZo7dy4htsBkM6/lXE/sgAEDmvXO1NbWauPGjc36J3Y+T0r1RTa2evVq7bXXXvzWnqPae70bLF26VFdeeaUuuOACnXbaaZ1VJrKkvdf7k08+0cqVKzVs2LD0/1atWqWXXnpJw4YN04oVKzq7dLRDe6/3wIEDd/m80Wg0K/Uhu9p7vT/66CN5vV4NGjSoyfGDDjpIGzZsUDgc7pR64Yxs5rWcC7FjxozRihUr0qstkrRkyRJ5PB6NGjWq1fOGDh2q0tJSLV68OH0sHo/r6aef1pgxYzq1ZrRfe6+3JP373//WxRdfrNNPP13Tp0/v7FKRBe293v/zP/+jhx56qMn/DjzwQA0ZMkQPPfSQDj300K4oHxlq7/Xee++9NWjQoGa/nKxYsUJFRUW7DblwRkeudzKZ1Pvvv9/k+LvvvqtevXopFAp1Ws3oetnMazn397dJkybp4Ycf1vTp0zVt2jRVVVVp3rx5mjRpkvr06ZP+uClTpmjdunVaunSpJCkYDGratGmaP3++evbsqUGDBumRRx7Rtm3bdO655zr16WA32nu9P/74Y02fPl39+/fXKaec0mSnj549e6qysrKrPxW0QXuv90EHHdTsucrKylRcXKzhw4d3Wf3ITHuvtyTNnDlTP/vZz3TNNddo7Nixevvtt3X//ffr3HPPpe89R7X3eo8ZM0Z77bWXLrjgAk2fPl0VFRV66aWX9Le//U0zZsxw6tNBG4TDYb3wwguSpLVr12r79u1asmSJJOmoo45Sz549OzWv5VyILS8v14MPPqirr75a06dPV0lJiU477TTNnDmzycfZtq1kMtnk2HnnnSdjjO6//35t2bJFBx10kO677z5268ph7b3eb775pmpra1VbW6vvf//7TT72u9/9rubOndsl9SMzHXl9I/905Hofd9xx+t3vfqff//73euSRR1RRUaEZM2bo/PPP78pPARlo7/UuLS3VAw88oJtuukk33HCDamtr1a9fP82aNUuTJ0/u6k8DGdi8ebMuvPDCJscaHj/00EMaPnx4p+Y1yzBJGAAAAHkm53piAQAAgN0hxAIAACDvEGIBAACQdwixAAAAyDuEWAAAAOQdQiwAAADyDiEWAAAAeYcQCwAAgLxDiAUAAEDeIcQCAAAg7xBiAQAAkHcIsQAAAMg7/x+LpxFevU3oGAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 800x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "##训练模型\n",
    "print(\"开始训练模型......\")\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "LR = LogisticRegression()\n",
    "LR.fit(tfidf_words_df.iloc[:30000,:], dianping['y'].iloc[:30000])  ## 训练模型\n",
    "LR.coef_\n",
    "print(\"模型训练完成......\")\n",
    "## 预测及概率\n",
    "dianping['prob']=LR.predict_proba(tfidf_words_df)[:,1]\n",
    "dianping['pred']=dianping['prob']>=0.5\n",
    "# 画图\n",
    "from sklearn.metrics import roc_curve, auc\n",
    "fpr, tpr, threshold = roc_curve(dianping['y'][30000:], dianping['prob'][30000:])\n",
    "roc_auc = auc(fpr, tpr)\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sb\n",
    "sb.set()\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = (8.0, 8.0)\n",
    "plt.plot(fpr,tpr,label='ROC, AUC=%.2f' % roc_auc)\n",
    "plt.legend(loc='upper left', frameon=True)\n",
    "plt.plot([0, 1], [0, 1], linestyle='--')\n",
    "plt.xlim([0,1])\n",
    "plt.ylim([0,1])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "当然，以上模型没有进行调参，模型仍有改进的空间。"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
  },
  "kernelspec": {
   "display_name": "Anaconda",
   "language": "python",
   "name": "anaconda"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}