XGBoost/XGBoost.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "资料:  \n",
    "[XGBoost原始论文](https://arxiv.org/pdf/1603.02754v1.pdf)  \n",
    "[Xgboost Slides ](https://homes.cs.washington.edu/~tqchen/pdf/BoostedTree.pdf)  \n",
    "(推荐)[XGBoost 与 Boosted Tree](http://www.52cs.org/?p=429)  \n",
    "(推荐)[xgboost入门与实战（原理篇）](http://blog.csdn.net/sb19931201/article/details/52557382)  \n",
    "[XGBoost浅入浅出](http://wepon.me/2016/05/07/XGBoost%E6%B5%85%E5%85%A5%E6%B5%85%E5%87%BA/)  \n",
    "[xgboost调参演示](https://github.com/lytforgood/MachineLearningTrick/blob/master/xgboost%E8%B0%83%E5%8F%82%E6%BC%94%E7%A4%BA.md)  \n",
    "[xgboost官网](http://xgboost.readthedocs.io/en/latest/python/python_api.html#module-xgboost.sklearn)  \n",
    "[xgboost算法原理与实战](https://blog.csdn.net/jasonzhangoo/article/details/73061060)    \n",
    "[利用GBDT模型构造新特征](https://breezedeus.github.io/2014/11/19/breezedeus-feature-mining-gbdt.html#fn:fbgbdt)  \n",
    "[XGBoost Plotting API以及GBDT组合特征实践](https://blog.csdn.net/sb19931201/article/details/65445514)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## XGBoost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "XGBoost(eXtreme Gradient Boosting)是GBDT是的一种实现,正如其名,它在原有GBDT的基础做了很多改进,在训练速度和精度上面都特别优秀.也是目前数据比赛中特别常用的算法.xgboost算法比较复杂，针对传统GBDT算法做了很多细节改进，包括损失函数、正则化、切分点查找算法优化、稀疏感知算法、并行化算法设计等等。本文主要介绍xgboost基本原理以及与传统gbdt算法对比总结，后续会基于python版本做了一些实战调参试验。想详细学习xgboost算法原理建议通读作者原始论文与slide讲解。 相关笔记 [Adaboost](https://github.com/linzhenpeng/machinelearning/blob/master/Adaboost/Adaboost.ipynb),[GBDT ](https://github.com/linzhenpeng/machinelearning/blob/master/GBDT/GBDT.ipynb) ,[决策树 ](https://github.com/linzhenpeng/machinelearning/blob/master/decisionTree/decisionTree.ipynb)  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### xgboost特点（与gbdt对比）来自[xgboost入门与实战（原理篇）](https://blog.csdn.net/sb19931201/article/details/52557382)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. 传统GBDT以CART作为基分类器，xgboost还支持线性分类器，这个时候xgboost相当于带L1和L2正则化项的逻辑斯蒂回归（分类问题）或者线性回归（回归问题）。 —**可以通过booster [default=gbtree]设置参数:gbtree: tree-based models/gblinear: linear models**    \n",
    "2. 传统GBDT在优化时只用到一阶导数信息，xgboost则对代价函数进行了二阶泰勒展开，同时用到了一阶和二阶导数。顺便提一下，xgboost工具支持自定义代价函数，只要函数可一阶和二阶求导。     \n",
    "3. xgboost在代价函数里加入了正则项，用于控制模型的复杂度。正则项里包含了树的叶子节点个数、每个叶子节点上输出的score的L2模的平方和。从Bias-variance tradeoff角度来讲，正则项降低了模型variance，使学习出来的模型更加简单，防止过拟合，这也是xgboost优于传统GBDT的一个特性  **—正则化包括了两个部分，都是为了防止过拟合，剪枝是都有的，叶子结点输出L2平滑是新增的。**    \n",
    "4. shrinkage and column subsampling —还是为了防止过拟合，论文2.3节有介绍，这里答主已概括的非常到位      \n",
    "    （1）shrinkage缩减类似于学习速率，在每一步tree boosting之后增加了一个参数n（权重），通过这种方式来减小每棵树的影响力，给后面的树提供空间去优化模型。   \n",
    "    （2）column subsampling列(特征)抽样，说是从随机森林那边学习来的，防止过拟合的效果比传统的行抽样还好（行抽样功能也有），并且有利于后面提到的并行化处理算法。    \n",
    "    \n",
    "5. split finding algorithms(划分点查找算法)：   \n",
    "    （1）exact greedy algorithm—贪心算法获取最优切分点   \n",
    "    （2）approximate algorithm— 近似算法，提出了候选分割点概念，先通过直方图算法获得候选分割点的分布情况，然后根据候选分割点将连续的特征信息映射到不同的buckets中，并统计汇总信息。详细见论文3.3节   \n",
    "    (3）Weighted Quantile Sketch—分布式加权直方图算法，论文3.4节   \n",
    "    这里的算法（2）、（3）是为了解决数据无法一次载入内存或者在分布式情况下算法（1）效率低的问题，以下引用的还是wepon大神的总结：  \n",
    "    可并行的近似直方图算法。树节点在进行分裂时，我们需要计算每个特征的每个分割点对应的增益，即用贪心法枚举所有可能的分割点。当数据无法一次载入内存或者在分布式情况下，贪心算法效率就会变得很低，所以xgboost还提出了一种可并行的近似直方图算法，用于高效地生成候选的分割点。**xgboost里面分割特征使用的不是 gini系数也不是信息增益**  \n",
    "6. 对缺失值的处理。  \n",
    "    对于特征的值有缺失的样本，xgboost可以自动学习出它的分裂方向。 —稀疏感知算法，论文3.4节，Algorithm 3: Sparsity-aware Split Finding  \n",
    "7. Built-in Cross-Validation（内置交叉验证)\n",
    "\n",
    "    XGBoost允许在每一轮boosting迭代中使用交叉验证。因此，可以方便地获得最优boosting迭代次数。 \n",
    "     而GBM使用网格搜索，只能检测有限个值。\n",
    "\n",
    "8.  continue on Existing Model（接着已有模型学习）\n",
    "\n",
    "    XGBoost可以在上一轮的结果上继续训练。 \n",
    "    sklearn中的GBM的实现也有这个功能，两种算法在这一点上是一致的\n",
    "    \n",
    "9. High Flexibility（高灵活性）  \n",
    "    XGBoost 允许用户定义自定义优化目标和评价标准 。 \n",
    "    \n",
    "10. 并行化处理 —系统设计模块,块结构设计等\n",
    "    xgboost工具支持并行。boosting不是一种串行的结构吗?怎么并行的？注意xgboost的并行不是tree粒度的并行，xgboost也是一次迭代完才能进行下一次迭代的（第t次迭代的代价函数里包含了前面t-1次迭代的预测值）。xgboost的并行是在特征粒度上的。我们知道，决策树的学习最耗时的一个步骤就是对特征的值进行排序（因为要确定最佳分割点），xgboost在训练之前，预先对数据进行了排序，然后保存为block结构，后面的迭代中重复地使用这个结构，大大减小计算量。这个block结构也使得并行成为了可能，在进行节点的分裂时，需要计算每个特征的增益，最终选增益最大的那个特征去做分裂，那么各个特征的增益计算就可以开多线程进行。\n",
    "    \n",
    "此外xgboost还设计了高速缓存压缩感知算法，这是系统设计模块的效率提升。 \n",
    "当梯度统计不适合于处理器高速缓存和高速缓存丢失时，会大大减慢切分点查找算法的速度。\n",
    "xgboost 还考虑了当数据量比较大，内存不够时怎么有效的使用磁盘，主要是结合多线程、数据压缩、分片的方法，尽可能的提高算法的效率。\n",
    "（1）针对 exact greedy algorithm采用缓存感知预取算法 \n",
    "（2）针对 approximate algorithms选择合适的块大小"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### XGBoost原理"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "截图来自[XGBoost 与 Boosted Tree](http://www.52cs.org/?p=429)  红色部分文字为自己的笔记(github不支持使用html标签??)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "![](images/xgboost1.png)\n",
    "![](images/xgboost2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<font color=#DC143C size=4>\n",
    "这里使用的是加法模型$\\hat{y_i} =\\sum_{k=1}^{K}f_k(x_i)$  我们要预测的结果由K个弱分类器累加而成  \n",
    "$$Obj(\\Theta)=\\sum_{i}^{n}l(y_i,\\hat{y_i})+\\sum_{k=1}^{K}\\Omega(f_k)$$\n",
    "$\\sum_{i}^{n}l(y_i,\\hat{y_i})$为误差函数:我们的模型有多拟合数据   \n",
    "$\\sum_{k=1}^{K}\\Omega(f_k)$正则化项:惩罚复杂模型\n",
    "</font>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/xgboost3.png)\n",
    "![](images/xgboost4.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<font color=#DC143C size=4>\n",
    "看到这里特别佩服陈天奇大神,这里没有使用特定的损失函数而是进行一般形式的推导,使用泰勒二阶展开式.这在我们使用xgboost工具时就可以自己定义损失函数,只要一阶和二阶可导就行,增加了灵活性.上面的推导中比较难理解的是   \n",
    "$g_i$和$h_i$ 其实这两个分别是 损失函数的一阶导数和二阶导数  \n",
    "$l(y_i,\\hat{y_{i-1}})$之所以被移除是因为这两个是已知量\n",
    "</font>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/xgboost5.png)\n",
    "![](images/xgboost6.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<font color=#DC143C size=4>\n",
    "这里定义正则函数 $\\Omega(f_t)$   \n",
    "正则函数不仅关注了  叶子的总个数$T$ 还关注叶子的总分数 $w$的平方 使得模型更加平滑  \n",
    "公式的中的$\\gamma$和$\\lambda$ 是超参数 对应gamma 和L2(reg_lambda)的正则化系数??\n",
    "</font>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/xgboost7.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<font color=#DC143C size=4>\n",
    "这里有点难理解,最终优化的结果可以看成是一个一元二次方程的求解对于$y=ax_2+bx+c$的解在x取值为 $-\\frac{b}{2a}$时 y为 $c-\\frac{b_2}{4a}$\n",
    "</font>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/xgboost8.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<font color=#DC143C size=4>\n",
    "这里重新定义了树结构的打分方式,减少了传统决策树的很多启发式方法,使结果更加一般.\n",
    "传统的特征分割,决策树剪枝,树的最大深度,叶子树的平滑值,这些通常都是采用启发式的方法(来自xgboost slides)\n",
    "</font>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/xgboost9.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<font color=#DC143C size=4>\n",
    "使用的新的方式分割特征,在寻找最佳分割点的时候，可以引入并行计算.\n",
    "</font>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## XGboost实例"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 参数说明"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "sklearn中没有xgboost需要另外安装.    \n",
    "xgboost的一些参数说明更详细看[官网API](http://xgboost.readthedocs.io/en/latest/python/python_api.html#module-xgboost.sklearn):    \n",
    "  \n",
    "**General Parameters（常规参数）**    \n",
    "\n",
    "1.**booster [default=gbtree]**：选择基分类器，gbtree: tree-based models/gblinear: linear models   \n",
    "2.**silent [default=0]**:设置成1则没有运行信息输出，最好是设置为0.   \n",
    "3.**nthread **[default to maximum number of threads available if not set]：线程数  \n",
    "  \n",
    "**Booster Parameters（模型参数）**     \n",
    "1.**eta [default=0.3]**:shrinkage参数，用于更新叶子节点权重时，乘以该系数，避免步长过大。参数值越大，越可能无法收敛。把学习率 eta 设置的小一些，小学习率可以使得后面的学习更加仔细。   \n",
    "2.**min_child_weight [default=1]**:这个参数默认是 1，是每个叶子里面 h 的和至少是多少，对正负样本不均衡时的 0-1 分类而言，假设 h 在 0.01 附近，min_child_weight 为 1 意味着叶子节点中最少需要包含 100 个样本。这个参数非常影响结果，控制叶子节点中二阶导的和的最小值，该参数值越小，越容易 overfitting。  \n",
    "3.**max_depth [default=6]**: 每颗树的最大深度，树高越深，越容易过拟合。   \n",
    "4.**max_leaf_nodes**:最大叶结点数，与max_depth作用有点重合。   \n",
    "5.**gamma [default=0]**：后剪枝时，用于控制是否后剪枝的参数。   \n",
    "6.**max_delta_step [default=0]**：这个参数在更新步骤中起作用，如果取0表示没有约束，如果取正值则使得更新步骤更加保守。可以防止做太大的更新步子，使更新更加平缓。   \n",
    "7.**subsample [default=1]**：样本随机采样，较低的值使得算法更加保守，防止过拟合，但是太小的值也会造成欠拟合。   \n",
    "8.**colsample_bytree [default=1]**：列采样，对每棵树的生成用的特征进行列采样.一般设置为： 0.5-1   \n",
    "9.**lambda [default=1]**：控制模型复杂度的权重值的L2正则化项参数，参数越大，模型越不容易过拟合。   \n",
    "10.**alpha [default=0]**:控制模型复杂程度的权重值的 L1 正则项参数，参数值越大，模型越不容易过拟合。   \n",
    "11.scale_pos_weight [default=1]：如果取值大于0的话，在类别样本不平衡的情况下有助于快速收敛。  \n",
    "\n",
    "**Learning Task Parameters（学习任务参数） **  \n",
    "1.**objective [default=reg:linear]**：定义最小化损失函数类型，常用参数： \n",
    "binary:logistic –logistic regression for binary classification, returns predicted probability (not class)    \n",
    "multi:softmax –multiclass classification using the softmax objective, returns predicted class (not probabilities)\n",
    "you also need to set an additional num_class (number of classes) parameter defining the number of unique classes   \n",
    "multi:softprob –same as softmax, but returns predicted probability of each data point belonging to each class.   \n",
    "2.**eval_metric [ default according to objective ]**： \n",
    "The metric to be used for validation data. \n",
    "The default values are rmse for regression and error for classification. \n",
    "Typical values are:   \n",
    "rmse – root mean square error   \n",
    "mae – mean absolute error   \n",
    "logloss – negative log-likelihood   \n",
    "error – Binary classification error rate (0.5 threshold)   \n",
    "merror – Multiclass classification error rate   \n",
    "mlogloss – Multiclass logloss   \n",
    "auc: Area under the curve   \n",
    "3.**seed [default=0]**：   \n",
    "The random number seed. 随机种子，用于产生可复现的结果 \n",
    "Can be used for generating reproducible results and also for parameter tuning.\n",
    "\n",
    "\n",
    "注意: python sklearn style参数名会有所变化   \n",
    "eta –> learning_rate   \n",
    "lambda –> reg_lambda   \n",
    "alpha –> reg_alpha  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 实例测试"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "测试实例,先导入要使用到的package,测试xgboost 在make_hastie_10_2数据集上二分类的表现,同时打印树和特征重要性,与sklearn的gbdt实现GradientBoostingClassifier做对比,同时做GDBT特征组合后比较效果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_hastie_10_2\n",
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "from xgboost import plot_importance\n",
    "from xgboost import plot_tree\n",
    "from xgboost.sklearn import XGBClassifier\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from graphviz import Digraph\n",
    "import pydot\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X,y=make_hastie_10_2(random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x_train,x_test,y_train,y_test=train_test_split(X,y,test_size=0.2,random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用默认参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "              learning_rate=0.1, loss='deviance', max_depth=3,\n",
       "              max_features=None, max_leaf_nodes=None,\n",
       "              min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "              min_samples_leaf=1, min_samples_split=2,\n",
       "              min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "              presort='auto', random_state=None, subsample=1.0, verbose=0,\n",
       "              warm_start=False)"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gbc=GradientBoostingClassifier()\n",
    "gbc.fit(x_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.91500000000000004"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gbc.score(x_test,y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "xgb=XGBClassifier()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n",
       "       colsample_bytree=1, gamma=0, learning_rate=0.1, max_delta_step=0,\n",
       "       max_depth=3, min_child_weight=1, missing=None, n_estimators=100,\n",
       "       n_jobs=1, nthread=None, objective='binary:logistic', random_state=0,\n",
       "       reg_alpha=0, reg_lambda=1, scale_pos_weight=1, seed=None,\n",
       "       silent=True, subsample=1)"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgb.fit(x_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.91041666666666665"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgb.score(x_test,y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看出使用默认参数的 xgb和sklearn的gdbt差不多,xgb要差一点"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们使用网格搜索来调参"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best -0.091874 using {'colsample_bytree': 0.6, 'gamma': 0.4, 'max_depth': 3, 'n_estimators': 2000, 'subsample': 0.6} \n"
     ]
    }
   ],
   "source": [
    "xgb=XGBClassifier()\n",
    "max_depth=[2,3,4]\n",
    "n_estimators=[500,1000,1500,2000]\n",
    "gamma=[0.2,0.3,0.4]\n",
    "subsample=[0.6,0.7,0.8]\n",
    "colsample_bytree=[0.6,0.7,0.8]\n",
    "param_grid=dict(gamma=gamma,\n",
    "                subsample=subsample,\n",
    "                colsample_bytree=colsample_bytree,\n",
    "                max_depth=max_depth,\n",
    "                n_estimators=n_estimators\n",
    "               )\n",
    "kfold=StratifiedKFold(n_splits=10,shuffle=True,random_state=0)\n",
    "grid_search=GridSearchCV(xgb,param_grid,scoring=\"neg_log_loss\",n_jobs=2,cv=kfold)\n",
    "grid_result=grid_search.fit(x_train,y_train)\n",
    "print(\"Best %f using %s \"%(grid_result.best_score_,grid_result.best_params_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "把上面搜索到的最好参数组合看下效果,上面的网格搜索特别耗时"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.96208333333333329"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgb=XGBClassifier(learning_rate=0.1,max_depth=3,n_estimators=2000,gamma=0.4,min_child_weight=1,subsample=0.6,colsample_bytree=0.6,silent=0,n_jobs=-1)\n",
    "xgb.fit(x_train,y_train)\n",
    "xgb.score(x_test,y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "比原来增加了0.05 可以说效果非常显著"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "xgb特别好的地方就是,可以画出特征的重要性"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAEWCAYAAACOv5f1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XmYFPW1//H3gRFEQBQRRAWGkV2W\nQYnCo5FBBUE0CYHrmqAYs7pgrpiYq+HHVSPEJWokcV8w7qgRNS4YYZQkKoJhVQcMDBcEAZVFRhRG\nzu+PqhmbYZZmqe5q6vN6nn6mu7qq65xhOF39rervMXdHRESSpV62AxARkcxT8RcRSSAVfxGRBFLx\nFxFJIBV/EZEEUvEXEUkgFX+RKszsTjP7bbbjEImS6Tp/2VPMrBRoBXydsriTu6/cjdcsAh5298N3\nL7rcZGYPAivc/epsxyJ7Fx35y552urs3SbntcuHfE8wsL5v73x1mVj/bMcjeS8VfMsLM+prZv8xs\nvZnNDY/oK54bZWbvm9nnZrbEzH4aLm8MvAQcamabwtuhZvagmV2Xsn2Rma1IeVxqZr82s3lAmZnl\nhds9bWZrzWypmV1aS6yVr1/x2mb2KzNbY2arzOx7ZnaqmS0ys8/M7H9Sth1nZk+Z2RNhPu+aWa+U\n57uaWXH4e1hoZt+pst87zOxFMysDfgScC/wqzP35cL0rzew/4eu/Z2bDUl7jfDP7h5ndZGbrwlyH\npDzf3MweMLOV4fPPpjx3mpnNCWP7l5n1TPsfWHKOir9EzswOA/4GXAc0B8YAT5vZweEqa4DTgP2B\nUcAtZnaUu5cBQ4CVu/BJ4mxgKHAAsA14HpgLHAacBFxmZqek+VqHAPuG244F7gF+ABwNfBsYa2YF\nKet/F5gc5voo8KyZ7WNm+4RxTAVaApcAj5hZ55RtzwF+BzQFHgIeAW4Icz89XOc/4X6bAf8LPGxm\nrVNe41igBGgB3ADcZ2YWPvcXYD/gyDCGWwDM7CjgfuCnwEHAXcBzZtYwzd+R5BgVf9nTng2PHNen\nHFX+AHjR3V90923u/iowCzgVwN3/5u7/8cDrBMXx27sZxx/dfbm7bwa+BRzs7te4+xZ3X0JQwM9K\n87W2Ar9z963A4wRF9TZ3/9zdFwILgdSj5Nnu/lS4/h8I3jj6hrcmwIQwjmnACwRvVBWmuPs/w9/T\nl9UF4+6T3X1luM4TwGLgmJRVlrn7Pe7+NTAJaA20Ct8ghgA/c/d17r41/H0D/Bi4y93fdvev3X0S\n8FUYs+yFcnY8VGLre+7+9yrL2gH/ZWanpyzbB5gOEA5L/D+gE8EByX7A/N2MY3mV/R9qZutTltUH\nZqT5Wp+GhRRgc/hzdcrzmwmK+g77dvdt4ZDUoRXPufu2lHWXEXyiqC7uapnZSOC/gfxwUROCN6QK\nH6fs/4vwoL8JwSeRz9x9XTUv2w44z8wuSVnWICVu2cuo+EsmLAf+4u4/rvpEOKzwNDCS4Kh3a/iJ\noWKYorrL0coI3iAqHFLNOqnbLQeWunvHXQl+F7SpuGNm9YDDgYrhqjZmVi/lDaAtsChl26r5bvfY\nzNoRfGo5CXjT3b82szl88/uqzXKguZkd4O7rq3nud+7+uzReR/YCGvaRTHgYON3MTjGz+ma2b3gi\n9XCCo8uGwFqgPPwUMChl29XAQWbWLGXZHODU8OTlIcBldex/JrAxPAncKIyhu5l9a49luL2jzez7\n4ZVGlxEMn7wFvE3wxvWr8BxAEXA6wVBSTVYDqecTGhO8IayF4GQ50D2doNx9FcEJ9D+b2YFhDCeE\nT98D/MzMjrVAYzMbamZN08xZcoyKv0TO3ZcTnAT9H4KitRy4Aqjn7p8DlwJPAusITng+l7LtB8Bj\nwJLwPMKhBCct5wKlBOcHnqhj/18TFNlCYCnwCXAvwQnTKEwBziTI54fA98Px9S3AdwjG3T8B/gyM\nDHOsyX1At4pzKO7+HnAz8CbBG0MP4J87EdsPCc5hfEBwov0yAHefRTDuPzGM+0Pg/J14Xckx+pKX\nyB5kZuOADu7+g2zHIlIbHfmLiCSQir+ISAJp2EdEJIF05C8ikkCxvc7/gAMO8A4dOmQ7jN1WVlZG\n48aNsx3GblMe8aI84iVOecyePfsTdz+4rvViW/xbtWrFrFmzsh3GbisuLqaoqCjbYew25REvyiNe\n4pSHmS1LZz0N+4iIJJCKv4hIAqn4i4gkkIq/iEgCqfiLiCSQir+ISAKp+IuIJJCKv4hIAqn4i4gk\nkIq/iEgCqfiLiCSQir+ISAKp+IuIJJCKv4hIAqn4i4hEaP369YwYMYIuXbrQtWtX3nzzTT777DMG\nDhxIx44dGThwIOvWrQNg3bp1DBs2jJ49e3LMMcewYMGCyOKKtPib2aVm9r6ZPRI+/paZfW1mI6Lc\nr4hIXIwePZrBgwfzwQcfMHfuXLp27cqECRM46aSTWLx4MSeddBITJkwA4Prrr6ewsJB58+bx0EMP\nMXr06MjiirSHr5l9AAxx96VmVh94FfgSuN/dn6pt27YFHbzeGbdFFlumXN6jnJvnx7ZnTtqUR7wo\nj3ipLo/SCUPZuHEjvXr1YsmSJZhZ5XOdO3emuLiY1q1bs2rVKoqKiigpKWHo0KH85je/4fjjjwfg\niCOO4F//+hetWrVKOxYzm+3ufepaL7IjfzO7EygAnjOzXwKXAE8Da6Lap4hInCxZsoSDDz6YUaNG\n0bt3by688ELKyspYvXo1rVu3BqB169asWROUxV69evHMM88AMHPmTJYtW8aKFSsiiS2yt1x3/5mZ\nDQYGAA2BR4ETgW/VtI2Z/QT4CUCLFgcztkd5VOFlTKtGwVFBrlMe8aI84qW6PIqLiykpKWH27Nmc\nf/75nH/++dx+++38/Oc/p7y8nOLi4sp1Kx4fd9xxTJw4kQ4dOlBQUECHDh3497//zeeff77HY456\n2KcU6APcAdzs7m+Z2YPACxr2yS3KI16UR7zUNOzz8ccf07dvX0pLSwGYMWMGEyZM4MMPP6x22CeV\nu9O+fXvmzZvH/vvvn3Ys6Q77ZOq33gd4PBzzagGcambl7v5sTRs02qc+JROGZii86BQXF1N6blG2\nw9htyiNelEe81JTHIYccQps2bSgpKaFz58689tprdOvWjW7dujFp0iSuvPJKJk2axHe/+10guDJo\nv/32o0GDBtx7772ccMIJO1X4d0ZGir+7t6+4n3LkX2PhFxHZW9x+++2ce+65bNmyhYKCAh544AG2\nbdvGGWecwX333Ufbtm2ZPHkyAO+//z4jR46kfv36dOvWjfvuuy+yuHL/85aISIwVFhYya9asHZa/\n9tprOyzr168fixcvzkRY0RZ/d8+vZtn5Ue5TRETqpm/4iogkkIq/iEgCqfiLiCSQir+ISAKp+IuI\nJJCKv4hIAqn4i4gkkIq/iEgCqfiLiCSQir+ISAKp+IuI7KazzjqLHj16UFhYSJ8+wWzKc+fOpV+/\nfvTo0YPTTz+djRs3AlBaWkqjRo0oLCyksLCQn/3sZ1mJObK5fczsUuDnwHvAocBRwFXuflNU+xQR\nyZbp06fTokWLyscXXnghN910E/379+f+++/nxhtv5NprrwWC9oxz5szJVqhAhM1cKvr3AmVAO+B7\nwLp0i7+aucSL8ogX5REPpWHPkUMOOYQFCxZsV/z3339/NmzYgJmxfPlyTjnlFN577z1KS0s57bTT\nWLBgQSQxZbWHb2r/XuBcd38H2BrFvkREss3MGDRoEEcffTR33303AN27d+e5554DYPLkySxfvrxy\n/aVLl9K7d2/69+/PjBkzshNzhEf+pUAfd/8kfDwO2FTbkX+VHr5Hj731nkhiy6RWjWD15mxHsfuU\nR7woj3jocVgzAJYtW0a7du1Yt24dY8aM4dJLL+XAAw/k9ttvZ8OGDRx33HE888wzTJkyhS1btrB5\n82aaNWtGSUkJv/3tb3nggQdo3LjxHolpwIABsWrjmBZ3vxu4G4Jhn1z+OFgh1z/WVlAe8aI84qGi\ndWNxcTFFRcH9uXPnsnXrVkaOHMnIkSMBWLRoEQsXLqxcp0JRURGPPfYYrVq1qjxRnCmx/a2rh2+8\nKI94UR7xUVZWxhdffFF5f+rUqYwdO5Y1a9bQsmVLtm3bxnXXXVd5Vc/atWtp3rw59evXZ8mSJSxe\nvJiCgoKMx61LPUVEdsPq1au55JJL6NWrF8cccwxDhw5l8ODBPPbYY3Tq1IkuXbpw6KGHMmrUKADe\neOMNevbsSa9evRgxYgR33nknzZs3z3jckR/5m9khwCxgf2CbmV0GdHP3jVHvW0QkagUFBdx33307\nDOmMHj2a0aNH77D+8OHDGT58eIaiq1lkxb9K/97Do9qPiIjsPA37iIgkkIq/iEgCqfiLiCSQir+I\nSAKp+IuIJJCKv4hIAqn4i4gkkIq/iEgCqfiLiCSQir+ISAKp+IuI7IT8/Pwd+vV++OGH9O3bt3LZ\nzJkzgWDW0mbNmlX2673mmmuyGfp2Ip3YLaWP7yHAcmAbUA5c5u7/iHLfIiJRqdqv96677uKaa65h\nyJAhvPjii/zqV7+iuLgYgG9/+9u88MILWYq0ZlHP6vkLgj6+a4Eyd3cz6wk8CXSpbcPNW78m/8q/\nRRxe9C7vUc75yiM2lEe85FIepXX0F9m4MZioeMOGDRx66KGZCGm3RDbsU6WP74/9m36RjYFoekeK\niESsun69F198MVdccQVt2rRhzJgxjB8/vnL9N998k169ejFkyBAWLlyYrbB3EFkPX9i+j6+ZDQPG\nAy2Boe7+ZjXrq4dvTCmPeFEemVfRr/eTTz6hRYsW2/Xr/fvf/06fPn3o378/06dP54UXXuDmm2+m\nrKyMevXq0ahRI9566y0mTpzIww8/HGmc6fbwzVjxT1l2AjDW3U+ubdu2BR283hm3RRZbpuR6j9IK\nyiNelEfmVTfsM27cOJo0acK4ceP4/PPPMTPcnWbNmlUOA6XKz89n1qxZ250v2NPMLJ4N3N39DTM7\nwsxapL4pVKUevvGiPOJFeWRHWVkZ27Zto2nTptv16z3ooIN4/fXXKSoqYtq0aXTs2BGAjz/+mFat\nWmFmzJw5k23btnHQQQdlOYtARoq/mXUA/hOe8D0KaAB8mol9i4jsKatXr2bYsGEAlJeXc8455zB4\n8GDGjBnD5ZdfTnl5Ofvuu2/luYCnnnqKO+64g7y8PBo1asTjjz+OmWUzhUqZOvIfDow0s63AZuBM\nj3K8SUQkAgUFBcydO3eH5T169GD27Nk7LL/44ou5+OKLMxHaTou0+Kf08f19eBMRkRjQN3xFRBJI\nxV9EJIFU/EVEEkjFX0QkgVT8RUQSSMVfRCSBVPxFRBJIxV9EJIFU/EVEEig3ptMTEcmw/Px8mjZt\nSv369cnLy2PWrFmceeaZlJSUALB+/XoOOOAA5syZw/vvv89ll10GgLszbty4yjmA4ipTbRzfJZjI\n7VTgC+B8d383yn2LiOyuqu0an3jiicr7l19+Oc2aBXP8t2/fnlmzZpGXl8eqVavo1asXp59+Onl5\n8T2+zlQbx67AJUBH4FjgjvBnjdTGMV6UR7woj+jU1a4RgqP7J598kmnTpgGw7777Vhb6L7/8MjYz\nd9YmU20c/wo85IG3gAPMrHVU+xYR2V3VtWusMGPGDFq1alU5bz/A22+/zZFHHkmPHj248847Y33U\nDxnq5AU8CExw93+Ey18Dfu3us6qsrzaOMaU84kV5RKe2do29evUC4JZbbuGwww7jjDPOAGDTpk00\nadIEgGXLljFhwgRuu+02GjRokPH4023jmKm3puo+A+3wruPudwN3Q9DGMVfau9Uml9rU1UZ5xIvy\niE51ncXmzp3L1q1bKSoqory8nDPPPJPZs2dz+OGHA0FHsqKib7Z78MEHad68OX361FmDsyZTv/UV\nQJuUx4cDK2vbQG0c40V5xIvyiFZN7RoB/v73v9OlS5fKwg+watUqysvLycvLY9myZZSUlJCfn5+l\n6NOTqeL/HHCxmT1OcKJ3g7uvytC+RUR2Sk3tGgEef/xxzj777O3Wnz9/Ptdddx377LMP9erV489/\n/nOkTdr3hEwV/xcJLvP8kOBSz1EZ2q+IyE6rqV0jBEM6VQ0aNIjrr78+4qj2rEy1cQS4KMp9iYhI\n+jS9g4hIAqn4i4gkkIq/iEgCqfiLiCSQir+ISALtdPE3swPNrGcUwYiISGakVfzNrNjM9jez5sBc\n4AEz+0O0oYmISFTSPfJv5u4bge8DD7j70cDJ0YUlIiJRSrf454VTMJ8BvBBhPCIikgHpFv9rgFeA\n/7j7O2ZWACyOLiwREYlSWtM7uPtkYHLK4yXA8KiCEhHJpp3p3/vqq69y0UUX0bBhQxo0aMCNN97I\niSeemOUM6pZW8TezTgStF1u5e/fwap/vuPt1tWyT2r/3HuBWYB/gE3fvv9uRi4hEKN3+vS1atOD6\n669nxIgRLFiwgFNOOYWPPvoo4/HurLQ6eZnZ68AVwF3u3jtctsDdu9eyzQcE/XvXAf8CBrv7/5lZ\nS3dfU9c+2xZ08Hpn3JZmGvEVx2YVu0J5xIvyiEZF/978/HxmzZpV7bTM7k7btm2ZNm1aZRvHimYu\n7k6LFi1YuXIlDRs2zGjsFcwsrU5e6Y757+fuM6ssK69l56n9ey8CnnH3/wNIp/CLiGTTzvbvrfD0\n00/Tu3fvrBX+nZHukf9LwMXAZHc/ysxGAD9y9yG1bFNK0L/3aoLhniOBpsBt7v5QDduoh29MKY94\nUR7R2JX+vRD08F27di1XX301N9xwA4cddlhW4oc938P3IoLeul3M7CNgKXBumtvmAUcDJwGNgDfN\n7C13X1R1RfXwjS/lES/KIxq70r8XYPLkyYwfP54nn3yS4447LoMR7wZ3r/VGMDR0Rni/MdC0rm3C\ndUuBFsCVwLiU5fcB/1XX9p06dfK9wfTp07Mdwh6hPOJFeURn06ZNvnHjxsr7/fr185deesnd3V96\n6SU/4YQTtlt/3bp1XlBQ4E899VTGY60OMMvTqNF1jvm7+zaCIR/cvczdP9/J95cpwLfNLM/M9iPo\n4fv+Tr6GiEhGrF69muOPP55evXpxzDHHMHTo0Fr7906cOJGVK1dy7bXXUlhYSGFhIWvWxP/UZrqf\nt141szHAE0BZxUJ3/6yuDd39fTN7GZgHbAPudfcFuxKsiEjUdrZ/79VXX83xxx9PUVFRtIHtYekW\n/wvCn6l9eJ3gip5qeUr/Xne/EbhxZ4MTEZFopPsN3/ZRByIiIpmT7jd8R1a33Gu4ZFNEROIt3WGf\nb6Xc35fgss13ARV/EZEclO6wzyWpj82sGfCXSCISEZHI7WoP3y+AHb/bLCIiOSHdMf/nCa7ugeAN\noxspUzyLiEhuSXfM/6aU++XAMndfEUE8IiKSAekO+5zq7q+Ht3+6+woz+32kkYmISGTSLf4Dq1lW\n44yeIiISb7UO+5jZz4FfAAVmNi/lqabAP6MMTEREolPXmP+jwEvAeILZOSt8ns68PiIiuaK6vr0A\nt99+OxMnTiQvL4+hQ4dyww03UFpaSteuXencuTMAbdu23bvm9nH3DcAG4GwAM2tJ8CWvJmbWxMPu\nXNVJ6eG7BNgCHAF8CVygid1EJI6q9u2dPn06U6ZMYd68eTRs2HC72TqPOOII5syZAwRtHHNNupd6\nng78ATgUWAO0I5iW+chaNvsFwXmBXwCb3H2YmXUB/kTwDeFabd76NflX/i2d8GLt8h7lnK88YkN5\nxEsc8qjo21udO+64gyuvvLKyLWPLli0zFVbk0j3hex3QF1gUTvJ2ErWM+Vfp4fsL4DUAd/8AyDez\nVrsTtIjInlZd395FixYxY8YMjj32WPr3788777xTuf7SpUvp3bs3/fv3Z968eTW9bGyle53/Vnf/\n1MzqmVk9d59e26We7v4zMxsMDAD+G/g+8A8zO4bgU8PhwOqq21Xp4cvYHjX2iM8ZrRoFRze5TnnE\ni/LYcyqGbG688cbt+vZu3ryZDRs2MH/+fCZMmMAHH3zAd77zHR599FG2bt3Ko48+SrNmzSgpKeHq\nq6/miCOOoHHjxlnNZWekW/zXm1kTYAbwiJmtIfiyVzomALeZ2RxgPvDvmrZ19fCNLeURL8pjz6mt\nb2/nzp259NJLKSoqYsCAAdx00010796dgw8+uHLdoqIi7rjjDlq1akWfPnX2TY+NdH/r3wU2A5cR\nNG5vBlyTzobuvhEYBWBmRtD8fWld2zXapz4ltYzF5Yri4uJq/7hyjfKIF+WxZ5WVlbFt2zaaNm1K\nWVkZU6dOZezYsTRp0oRp06ZRVFTEokWL2LJlCy1atGDt2rU0b96c+vXrs2TJEj766CMKCmrsbRVL\n6c7qWWZm7YCO7j4p7MVbP51tzewA4At33wJcCLwRviGIiMTC6tWrGTZsGADl5eWcc845DB48mC1b\ntnDBBRfQvXt3GjRowKRJkzAz3njjDcaOHUteXh7169fnl7/8Jc2bN89yFjsn3at9fkwwFt+c4JLN\nw4A7SeOqHaAr8JCZfQ28B/xo10IVEYlGTX17GzRowMMPP7zD8uHDhzN8+PDKx3vtpZ4EvXuPAd4G\ncPfF4TX/NUrp4fsJmv5ZRCRW0r3U86tw2AYAM8vjmymeRUQkx6Rb/F83s/8BGpnZQIK5/J+PLiwR\nEYlSusX/SmAtwaWaPwVeBK6OKigREYlWXbN6tnX3/3P3bcA94U1ERHJcXUf+z1bcMbOnI45FREQy\npK7ibyn3c+sbDCIiUqO6ir/XcF9ERHJYXdf59zKzjQSfABqF9wkfu7vvH2l0IiISibqauaQ1hYOI\niOSWdC/1FBGRvYiKv4gkUn5+Pj169KCwsLByKuZx48Zx2GGHUVhYSGFhIS+++CIAW7du5bzzzqNH\njx507dqV8ePHZzP0PSKyibRTevh2IfhyGMAm4OfuvuMMSiIiGVa1Zy/AL3/5S8aMGbPdssmTJ/PV\nV18xf/58vvjiC7p168bZZ59Nfn5+BqPds6LsolDRw7c18L67rzOzIQTNWo6ta2P18I0X5REvymPX\n1daztyZmRllZGeXl5WzevJkGDRqw//65fb1LJMM+VXr4Huvu68Kn3iJo4SgiklXV9ewFmDhxIj17\n9uSCCy5g3bqgdI0YMYLGjRvTunVr2rZty5gxY3Ju/v6qzD2ay/fNrBTo4+6fpCwbA3Rx9wtr2Ca1\nh+/RY2/N/dkkWjWC1ZuzHcXuUx7xojx2XY/DmgHwySefbNez99JLL6VNmzY0a9YMM+P+++/n008/\n5de//jXz589nypQpXHnllXz++eeMHj2aCRMmcOihhwKwadMmmjRpktlEajBgwIDZ7l5nP8mMNc80\nswEEjVyOr2kd9fCNL+URL8pj19XWs/f73/9+5bKCggJOO+00ioqKmDx5Mueddx4nn3wyAM8//zx5\neXkUFQWvVVxcXHk/V2Tkt25mPYF7gSHu/mk626iHb7woj3hRHrunpp69q1atonXr1gD89a9/pXv3\n7gC0bduWadOm8YMf/IAvvviCt956i8suuyzjce9JkRd/M2sLPAP80N0XRb0/EZG61NSz94c//CFz\n5szBzMjPz+euu+4C4KKLLmLUqFF0794dd2fUqFH07Nkzmynstkwc+Y8FDgL+bGYA5emMR4mIRKWm\nnr1/+ctfql2/SZMmTJ48OeqwMiqy4p/Sw/fC8CYiIjGhb/iKiCSQir+ISAKp+IuIJJCKv4hIAqn4\ni4gkkIq/iEgCqfiLiCSQir+ISAKp+IuIJJCKv4gkws60bQQYP348HTp0oHPnzrzyyivZCjsymWjj\n2BZYnLK/rsDB7v5ZVPsWEalOum0b33vvPR5//HEWLlzIypUrOfnkk1m0aBH169fPZLiRivLI/xfA\nqe7e2N0L3b0Q+A3wugq/iMTZlClTOOuss2jYsCHt27enQ4cOzJw5M9th7VGRHPmntnE0s/vd/Zbw\nqbOBx9J5DfXwjRflES/KI30VPXsr2jaaGT/96U/5yU9+AgRtGx966CH69OnDzTffzIEHHshHH31E\n3759K1/j8MMP56OPPoo0zkzLWBtHM9sPWAF0qOnIX20c40t5xIvySN+utG289dZbOfLIIxk4cCAA\nN9xwA8ceeyz9+/evdh9q41i704F/1jbkozaO8aU84kV5pG9X2ja++eabAJWtGcePH8+gQYPo169f\ntfvIxTaOuHskN6AUaJHy+K/AOelu36lTJ98bTJ8+Pdsh7BHKI16Ux87ZtGmTb9y4sfJ+v379/KWX\nXvKVK1dWrvOHP/zBzzzzTHd3X7Bggffs2dO//PJLX7Jkibdv397Ly8trfP04/XsAszyNGpupHr7N\ngP7ADzKxPxGRVDvbtvHII4/kjDPOoFu3buTl5fGnP/1pr7rSBzI37DMMmOruZRnan4hIpZ1t2whw\n1VVXcdVVV0UZVlZloo0j7v4g8GBU+xIRkZ2jb/iKiCSQir+ISAKp+IuIJJCKv4hIAqn4i4gkkIq/\niEgCqfiLiCSQir+ISAKp+IuIJJCKv4hIAqn4i0jOq64/729/+1t69uxJYWEhgwYNYuXKlQB88MEH\n9OvXj4YNG3LTTTdlM+ysirT4m9mlZva+mf3VzJ43s7lmttDMRkW5XxFJnunTpzNnzhxmzZoFwBVX\nXMG8efOYM2cOp512Gtdccw0AzZs3549//OMOfXuTJupZPX8BDCFo39jM3U83s4OBEjN7xN231LSh\n2jjGi/KIF+URqGjRWJ3999+/8n5ZWRlmBkDLli1p2bIlf/tb7v/+dkdkR/6pfXwBB5pa8NtvAnwG\nlEe1bxFJlor+vEcffTR333135fKrrrqKNm3a8Mgjj1Qe+Usgsh6+8E0fX+ArgjeBLkBT4Ex33+Ft\nVz1840t5xIvyCNTWn7dXr16V6z3yyCNs2bKFUaO+GXF+8MEHadSoEWeeeeauBxBSD9+anQLMAU4E\njgBeNbMZ7r4xdSVXD9/YUh7xojwCtfXnTe2p2759e4YOHcqkSZMqlxUXF9OkSZM90ns3F3v4Zuqv\nZxQwIewv+aGZLSX4FDCzpg0a7VOfklrG83JFcXFxtX+guUZ5xIvy+EZZWRnbtm2jadOmlJWVMXXq\nVMaOHcvixYvp2LEjAM899xxSEfOFAAAJEklEQVRdunTZAxHvPTJV/P8POAmYYWatgM7AkgztW0T2\nYjX15x0+fDglJSXUq1ePdu3aceeddwLw8ccf06dPHzZu3Ei9evW49dZbee+997Y7QZwEmSr+1wIP\nmtl8wIBfu/snGdq3iOzFaurP+/TTT1e7/iGHHMKKFSuiDiv2Ii3+qX18gUFR7ktERNKnb/iKiCSQ\nir+ISAKp+IuIJJCKv4hIAqn4i4gkkIq/iEgCqfiLiCSQir+ISAKp+IuIJJCKv4hIAqn4i0hOqa5f\n7xVXXEGXLl3o2bMnw4YNY/369QB8+umnDBgwgCZNmnDxxRdnM+zYibKTV0X/3o/MbIOZzQlvY6Pa\np4gkQ9V+vQMHDmTBggXMmzePTp06MX78eAD23Xdfrr322kQ3aq9JlBO7VfTvbQeMcffTdmZj9fCN\nF+URL0nMo7Z+vYMGfTNvZN++fXnqqacAaNy4Mccffzwffvjh7gW6F4rkyL9K/97eUexDRJKppn69\nFe6//36GDBmShchyS2Q9fFP693YHngZWACsJPgUsrGEb9fCNKeURL0nMI51+vQ8//DAlJSVcc801\nmFnlti+//DIlJSWMHj16j+cA6uFbk3eBdu6+ycxOBZ4FOla3onr4xpfyiJck5lFXv95JkyaxcOFC\nXnvtNfbbb7/tty0tZdOmTZH12VUP32qkNml39xfN7M9m1qKuTl7q4RsvyiNekppHTf16X375ZX7/\n+9/z+uuv71D4pXqRF38zOwRY7e5uZscQnGf4NOr9isjep6Z+vR06dOCrr75i4MCBQHDSt6Jnb35+\nPhs3bmTLli08++yzTJ06lW7dumUth7jIxOfGEcDPzawc2Ayc5VGdaBCRvVpN/Xpru5qntLQ0wohy\nV2TFP6V/78TwJiIiMaFv+IqIJJCKv4hIAqn4i4gkkIq/iEgCqfiLiCSQir+ISAKp+IuIJJCKv4hI\nAqn4i4gkkIq/iEgCqfiLiCSQir+ISAKp+IuIJJCKv4hIAkXWw3d3mdnnQEm249gDWgC1di3LEcoj\nXpRHvMQpj3bufnBdK8W5CWhJOk2I487MZimP+FAe8aI8skfDPiIiCaTiLyKSQHEu/ndnO4A9RHnE\ni/KIF+WRJbE94SsiItGJ85G/iIhERMVfRCSBYln8zWywmZWY2YdmdmW246mNmd1vZmvMbEHKsuZm\n9qqZLQ5/HhguNzP7Y5jXPDM7KnuRf8PM2pjZdDN738wWmtnocHmu5bGvmc00s7lhHv8bLm9vZm+H\neTxhZg3C5Q3Dxx+Gz+dnM/6qzKy+mf3bzF4IH+dcHmZWambzzWyOmc0Kl+XU3xWAmR1gZk+Z2Qfh\n/5N+uZhHqtgVfzOrD/wJGAJ0A842s27ZjapWDwKDqyy7EnjN3TsCr4WPIcipY3j7CXBHhmKsSzlw\nubt3BfoCF4W/81zL4yvgRHfvBRQCg82sL/B74JYwj3XAj8L1fwSsc/cOwC3henEyGng/5XGu5jHA\n3QtTroPPtb8rgNuAl929C9CL4N8lF/P4hrvH6gb0A15Jefwb4DfZjquOmPOBBSmPS4DW4f3WBF9Y\nA7gLOLu69eJ0A6YAA3M5D2A/4F3gWIJvXuZV/fsCXgH6hffzwvUs27GH8RxOUFBOBF4ALEfzKAVa\nVFmWU39XwP7A0qq/01zLo+otdkf+wGHA8pTHK8JluaSVu68CCH+2DJfHPrdwyKA38DY5mEc4VDIH\nWAO8CvwHWO/u5eEqqbFW5hE+vwE4KLMR1+hW4FfAtvDxQeRmHg5MNbPZZvaTcFmu/V0VAGuBB8Jh\nuHvNrDG5l8d24lj8rZple8v1qLHOzcyaAE8Dl7n7xtpWrWZZLPJw96/dvZDgyPkYoGt1q4U/Y5mH\nmZ0GrHH32amLq1k11nmEjnP3owiGQi4ysxNqWTeueeQBRwF3uHtvoIxvhniqE9c8thPH4r8CaJPy\n+HBgZZZi2VWrzaw1QPhzTbg8trmZ2T4Ehf8Rd38mXJxzeVRw9/VAMcE5jAPMrGIeq9RYK/MIn28G\nfJbZSKt1HPAdMysFHicY+rmV3MsDd18Z/lwD/JXgDTnX/q5WACvc/e3w8VMEbwa5lsd24lj83wE6\nhlc2NADOAp7Lckw76zngvPD+eQRj6BXLR4ZXA/QFNlR8bMwmMzPgPuB9d/9DylO5lsfBZnZAeL8R\ncDLBibnpwIhwtap5VOQ3Apjm4SBtNrn7b9z9cHfPJ/j7n+bu55JjeZhZYzNrWnEfGAQsIMf+rtz9\nY2C5mXUOF50EvEeO5bGDbJ90qOEEy6nAIoLx2quyHU8dsT4GrAK2Erzj/4hgvPU1YHH4s3m4rhFc\nyfQfYD7QJ9vxh3EdT/CxdB4wJ7ydmoN59AT+HeaxABgbLi8AZgIfApOBhuHyfcPHH4bPF2Q7h2py\nKgJeyMU8wnjnhreFFf+Xc+3vKoytEJgV/m09CxyYi3mk3jS9g4hIAsVx2EdERCKm4i8ikkAq/iIi\nCaTiLyKSQCr+IiIJFOcG7iKRMLOvCS7Bq/A9dy/NUjgiWaFLPSVxzGyTuzfJ4P7y/Js5eURiQcM+\nIlWYWWszeyOcg36BmX07XD7YzN61oF/Aa+Gy5mb2bDhv+1tm1jNcPs7M7jazqcBD4YRzN5rZO+G6\nP81iiiIa9pFEahTO/Amw1N2HVXn+HILpkn8X9pfYz8wOBu4BTnD3pWbWPFz3f4F/u/v3zOxE4CGC\nb4MCHA0c7+6bwxktN7j7t8ysIfBPM5vq7kujTFSkJir+kkSbPZj5sybvAPeHk9096+5zzKwIeKOi\nWLt7xcRpxwPDw2XTzOwgM2sWPvecu28O7w8CeppZxdw8zQiafaj4S1ao+ItU4e5vhFMPDwX+YmY3\nAuupflre2qbvLauy3iXu/soeDVZkF2nMX6QKM2tHMJ/+PQSznR4FvAn0N7P24ToVwz5vAOeGy4qA\nT7z6XgivAD8PP01gZp3CmS5FskJH/iI7KgKuMLOtwCZgpLuvDcftnzGzegRztw8ExhF0eJoHfME3\nU/xWdS9Bu893wym01wLfizIJkdroUk8RkQTSsI+ISAKp+IuIJJCKv4hIAqn4i4gkkIq/iEgCqfiL\niCSQir+ISAL9f9qY+o0iT4ORAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f2bac053278>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_importance(xgb)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "画出来的树很模糊."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAADhCAYAAADGdn6kAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xl0FFXaP/BvJyF7TNhiTFjERBNA\nkEUYFmFYJEQRZByIsoqyiKCAMAi8cNheFwKHgYCMwAEUEAOiDAZ59Yc4wmjYg0AgBgiLLIGAkEAI\nIYTk+f3RVKc76U56qepbVf18zsnpdHXVvU/frn5SuVV1r4GIwBhjTL+8RAfAGGNMWZzoGWNM5zjR\nM8aYznGiZ4wxneNEzxhjOseJnjHGdI4TPWOM6RwnesYY0zlO9IwxpnOc6BljTOd8RAfwEI/DwBhj\njjPYsxIf0TNdefDgAeLj42EwGNCkSRP8/e9/R48ePfDII48gKCgIqampokNkzO0MKhnUTBVBMG1K\nTEzESy+9hKFDhzq0XY0aNVBSUqJQVIy5hV1H9JzomWYFBQWhsLDQ5XIuXLiAQ4cO4ZVXXpEhKsbc\nihM906fi4mKcPHkSzZs3l7XcRo0a4dy5c7KWyZjCuI+e6U90dDT8/PxkT/IAcO7cOSQlJcnyXwJj\nasJH9EwzAgMDcffuXbfU9eWXX2LgwIFuqYsxF3DXDdOPsLAw5Ofniw6DMbWxK9Gr5Tp6xmzy8fHB\ngwcP3F7v3bt3ERgY6PZ6GZMb99EzVdu0aZOQJA8Yu4oGDx4spG7G5MRdN0y1rl69ioiICNFhICYm\nBtnZ2aLDYMwavuqGadugQYNEhwAAnOSZ5vERPVOl+/fvw9fXV3QYJpGRkcjJyREdBmMV8RE90674\n+HjRIVjgJM+0jI/oGbMT99UzFeIjeqZNGRkZokOwqk+fPqJDYMwpfETPVMdgMEAl+yVjasdH9Eyb\ndu7cKToEm7p37y46BMYcxkf0jDmA/9tgKsNH9Eyfpk6ditWrV6NFixZWX8/JyUGHDh1Mz/v27YsB\nAwYAMCZqAPjss89Mz6Vl06ZNq7ZuqRzGtIQTPdOU7du349atWxg+fDiOHDkCAFi3bh3OnDljWicy\nMhJ5eXmm51u3bsXzzz+PSZMmwc/PDwDwxhtvAACIyDS94Ntvv11t/c8995xs74Uxd+FEzzSlV69e\n+PTTTy2WJScnIzo6usrtEhMTsXDhQty7d6/Sa+np6QCABg0aVFu/9IeCMS3hRM805fjx45g7d65F\nl0t6ejpWrVplWmfXrl0WR/i1atXCgQMH0LJlS2zbtg0A8NZbb+Ho0aMAgNmzZwMA/vGPf1Rb/4UL\nF+R6K4y5DZ+MZcwBfNMUUxk+Gcu0KTc3V3QINt2+fVt0CIw5jBM9U51WrVqJDsGmgwcPig6BMYdx\nomeq06RJE9EhWFVUVISGDRuKDoMxh3EfPWN24pulmApxHz3Tri+++EJ0CJVwkmdaxYmeqdLSpUtF\nh2CBT8IyLeNEz1Rp//79okOwMH/+fNEhMOY0TvRMtV5//XXRIQAAvv/+e3zwwQeiw2DMaXwylqla\nnTp18OeffwqrPz8/H2FhYcLqZ6wafDKWad+ff/6JwsJCYfUvW7ZMWN2MyYWP6BmzITAwEHfv3hUd\nBmNV4SN6ph/379/Hjh073Fafj48PJ3mmG5zomSb4+voiPj4eHTt2VLyuFY89hgcPHiheD2Puwome\naUpaWhqefHK6YuUfOXIEb125AixfrlgdjLkb99EzTdm7F2jfHti4cSPOnDmD6dPlSfpWhzcoLQW8\nvWUpnzGF2NVHz4meadqdO3fQs2dPpKWlObztmTNnMHbsWPzwww/WVyACDHZ9jxgThU/GMn2pVavy\nsuDgYKSlpeHatWsICwvDJ598UmUZN2/eREhICN59911ER0fbTvKAMcn7+7sYNWPi8RE904QWLYCH\nc4G7X/PmwLFjgipnrErcdcOYbAICgKIi0VEwVhF33TB92L1bdAQwJvn//V/RUTDmFD6iZ6r28cfA\ntGmio3jowQMgNxeIihIdCWMS7rph2vbLL0CnTqKjYEzVONEzpojffwcaNxYdBWMA99EzLWvVSnQE\nVWjcGOjSRXQUjNmNEz1TnXv3gMOHRUdRjV27gGqu2WdMLTjRM9UpLASCgoIAAGvXrsWIESMAAO3a\ntbMYm/7u3bvYt28fTp06hfT0dJSVlQEARo0ahcLCQowaNcq0rH79+mjWrBkAwN/fHwY57nh95x0g\nPh4AkJqaivz8fKSnp2PKlCmYNGmS6+UzJhPuo2eqUqMGUFJi/D09PR2tW7cGAJSVleH8+fNIS0vD\nkCFDKm1nPlZNQkICBg0ahA0bNpjufDUYDIiLi8Pvv/+Ovn37YuvWrfIFnZ0NxMSY6jl+/Diio6Ph\nz3fVMuVxHz3TllmzypM8APz+++8AgNq1a8PLywv9+/dHp06dUK9ePQDAtWvX8NNPP2H8+PEAjEf4\nixYtQteuXdGpUyd07drVtAwAsrKysHPnTnz77bfYsmWLfIHfvo1//etf+PDDDwEA69atw4IFC+Qr\nnzEX8RE9U6VGjRrh3Llzspc7btw4XL16FV999ZW8BbdpAxw8KG+ZjFWPL69k2jFhArB4segoXPT+\n+8D8+aKjYJ6FEz3ThrZtgQMHREchE39/42VDjLkHJ3qmDbob9v3SJeDheQTGFMYnY5n61a+vsyQP\ncJJnqsOJngnj6wtcvCg6CoWcPCk6AsZMONEzYe7fFx2BgmJjgbffFh0FYwA40TMHJSQkoH79+pg7\ndy6ys7MBAHl5edi+fTu6du2KOnXq4M6dOza3P3rUeFPUt9+6K2KBPv0U8PERHQVjfDKWVY2IEBER\ngdzcXIe3/cc//oH4+HjEPxwmALDsj1fHrucGnTsD//2vXasSERISErBjxw40b94csbGxyMvLwy+/\n/ILw8HD88MMPaNKkicIBMw3hq26Y83bv3o2UlBQsX75clvJat26N9PR0GAyAlxdQWipLsdoREWGc\ntMTK9+3jjz9Gamoq9u7da3dxFy9eRLdu3XD69Gk5o2Taw4meOadr1674+eefZS/Xx8cHgwc/wOef\ny160uiUmAps3G3+v8H1r27YtDrhwE8G3336LL774Apul8pmn4UTPHNemTRscVPBW/j/++APHjx9H\nr169FKtDld591zis8dChwNq18Pf3xz0Zb6w6cuQImjdvDi8vPu3mYTjRM/vdv38f//3vf/H888+7\npb7evXtj27ZtbqlLTcoaN4bXw8HalODn54fi4mLFymeqwzdMMfv5+vq6LckDwLZt2/C2h11+uH//\nfkWTPAAUFxeje/fuitbBtIcTPcOECROE1Pvpp5/i2LFjQup2t/Pnz6N58+Zuqeunn34yDeXMGMBd\nNx4vICAARUVFosNgCli3bh2GDh0qOgymLO66YVVr166dKpJ827ZtRYegqPXr1wupd+jQofjll1+E\n1M3UhRO9B/vuu+9EhwAALl1eqHaZmZlWpz50l99++01Y3Uw9ONF7qEmTJqFOnTqiwzCJiooSHYIi\nZJmE3AXjxo3DN998IzQGJh4neg81YsQI0SFYuHz5sugQZPfrr7+icePGosNAeHi46BCYYJzoPZQa\nElBFXbt2FR2CrFJSUkSHAADo1KmT6BCYYJzoPVDv3r1Fh2BVSEiI6BBktWzZMtEhmDRr1kx0CEwg\nTvQeqHbt2qJDsCo1NVV0CLIJCwsTHYKFkSNHig6BCcSJ3gN9ruJRxaZOnSo6BFmo7RzIuHHjRIfA\nBOIbppiqGAwGqGSf1J3ly5dj9OjRosNg8uIbpphzSktLMWTIEAQGBtpcx/xk7tq1azFmzBgAQFBQ\nkMV6BoMBmZmZ+P3337F27dpq665Zs6aTUWuL1MbWLr+seKJcatNDhw4BgGm7uLg4rF692u46t27d\n6kLETNOISA0/TIXS0tKIiCgsLKzSa3FxcRbP58yZQ3v37q20/Pr16zRx4kS76+zRo4cTkWpXQEAA\nFRQUWLSxeRv+85//rNSm0nbGry/RxYsX7arL2ufINM+uHMsTWjKrfvzxR/To0QOAsb+5qKgIAQEB\nVtctLCzEzJkzrb5Wt25dU1dMUFAQCgsLq6y3rKzMhai15ccff8Tdu3cB2G7j9957DytXrrS5XVxc\nHLKysuyqj8eq91zcR89UpX79+rh48aLoMDRh7ty5Nv/AWtOpUyce+0Z/eOIRpj16ORlbVlamuiPo\nDRs2YNCgQaLDYPLik7FMe+bPny86BFkMHz5cdAiVcJL3XJzoPVBCQoLoEGyaPHmy6BBkkZaWJjoE\nC3q5P4E5hxO9Bzp9+rToEKz6n//5H9EhyObw4cOiQ7CQnJwsOgQmECd6D3TmzBnRIVh17do10SHI\nJjg42K1z8FZHDRPMMHE40XsotUw6Ym7VqlWiQ5DVl19+KToEAOoaXI2JwYneQ/3555+iQ7BQo0YN\n0SHILjw8XBXJXk3/WTAxONF7qGHDhokOwUJxcbHoEBSRnZ0ttP5Lly4hNjZWaAxMPE70HkwtV7jU\nrFlTddecy2XmzJlC54ytarwi5jn4hikP16xZM2RkZAirX403FulFWUgIvAoKRIfBlMU3TLHqZWRk\n4OzZs8LqV+ONRUp477333Fqfn5+fMclHR7u1XqZOfETPhDEfOM0TFBcXw8/PT/F6QkNDcevWrfIF\nFy4ADRooXi8Tgo/omX3atTM+jho1ym11bt682aOSPABTkr98+bJidVRK8kB5krcx+ijTP070Hq5b\nN2DfPuPvK1euVPwyx5UrVyI/Px/9+/dXtB41i4qKQrdu3WQtc/z48QBQOcmbKyoCEhNlrZdpA3fd\neLBevYDt2ysv9/b2Rmlpqez1vfvuu1i6dKns5WpVeno6vLy80LJlS5fKcXjEzw4dgD17XKqTqQYP\nU8xsGzwY+OIL26+XlJQgIiICN27ccLmugoICrFu3DmPHjnW5LD3KyclBVFSUQ8l6zpw58PX1xbRp\n05yr9PZt4LPPgIf/CTDN4kTPrJs2Dfj4Y/vX9/b2RklJicOXQYaHhyMnJwc+PjyRmSM2b96MJUuW\n4NdffzUtq1GjBqZPn45Zs2bJW5mXF+BBs3rpECd6VtmqVcCIEc5vv2nTJrz22mto2LAhBg8ejAYN\nGiAvLw8HDhzAli1bMHnyZN2MKa8GiYnAV18pXMnUqcC8eQpXwhTCiZ5Z2rEDiI+XuVC3ZCLP5bbm\nTUwENm0CDHblDaYefHklK5edrUCSZ/rx1VfGJM999rrEid4DFBYCMTGio2CakJwMeHuLjoLJjBO9\nBzhyRHQETFNKSwEPGZrCU3Ci17lZs4COHUVHwTRn9WrgwQPg/n3RkTAZcKLXsYICYM4c0VEwzfLx\nAXx9gb/9TXQkzEWc6HXq11+BkBDRUTBd+Pe/q767jqkeJ3od8vcHnnvOuW0HDBhg1zKmbrJ/ZoMH\nA0lJ8pbJ3IYTvc7Uqwfcu2f8PTMzExcvXsSoUaPQs2dPq1PKGQwGGAwGHHl4xnbjxo2IjY1FYmKi\n6XHjxo0ICQnBhQsXAAA3b96Ewex6648++ggGgwGxsbG4fv268m/SwxgMBovP0dY6BoMBt2/fBlD+\nOQIwfY4XL17E5cuXTZ+jj4+PxedYrSlTjI95eaZFly9fxkcffYR27dphzpw5PG2hWhGRGn6YDLy8\nLJ9PnjyZsrOzyfgxE82cOZNSUlIs1gFAcXFxFs9nzpxp8VhxnVOnTpUX0L8/ERHFxcWZ6mHyedi8\nFp8jEdn9OaakpJg+x4omTJjgfGChoZXqT0pKstw3mDvYlWP5zlideP55YOdOy2ULFiywOi/s3bt3\n5ZtLlO+MVVRVzevq53jz5k3UqlXL6e2xezfw1786vz2TA98Z60kqJnnA9uTfPGG0Prj6ObqU5AFj\nkv/iCyA317VymOI40WvcvXv8PWMCDR4MPPoo0KqV6EhYFTjRa5y/v/F7xphQhw8DDRuKjoLZwAOF\na9hjjwFXrggMQLpiw2AA1HGuh4n0xx9AWBiQny86ElYBH9FrVK1agpM8UH57/COPiI1DpyZOBDZv\n1tjIwVKSz8wUGwezwFfdaNDixcCECaKjeIiP5hUlJXlNNnH9+sDFi6Kj0DueeESPbE3oLQwnekV5\nexsHk9SsZ54Bjh4VHYWe8eWVajF79mwEBASY7l40GAxo2rQpFixY4HBZIpP8nj17YDAYsGDBAty8\nedO48NIl0+vffPMN/Pz88OGHHwqKUB+WL1+ONm3awGAwoKzs/8FgMMDHxwfvv/++6NAcd/QocOiQ\n6Cg8Hh/RK6B3794YOHCgQ+ONPPvss3juueewePHiSq/16AH8/LPxWvkuXWQM1A5r1qxBUVERxo4d\n6/C2eXl56NKlC47yEV2VFi9ejAULFuDy5ct2b/Pzzz9jwoQJ2mrb//wH6NZN9mLXr1+PEydOIC8v\nD8888wxGjhyJGjVqyF6PSnHXjbu1adMGBw8edLmcsLAw5JtduSD103p7G4cId4ennnoK+/btc/2m\nmocCAgJQVFQkS1l60bRpU5w4ccLlcoYOHYp169bJEJEbtG8P7N3r9OZXrlzBM888g5MnT6JmzZrV\nrt+zZ08EBQVhy5YtTtepcpzo3cXf3x/3pJHEZBQdHY0zZ87AYADCw913Y9SJEyfQtGlTRcr+/PPP\nMWzYMEXK1oo+ffpg/fr1CA0NlbVcLy8vlJWVyVqmIm7eNF425oDw8HAcP34c4eHhTlermfZxDCd6\nd1B65/H1/T8UF7/otkvsLl++jKioKEXrCA0Nxa1btxStQ63atm2LAwcOKFb+xo0b8dprrylWvmyy\ns+2eyLhDhw7Ys2ePLNWuXLkS9erVw4svvihLeSrAiV5pRUVFCAgIULyexx9/HOfPn1e0jr59+2Lr\n1q2K1mHOE5O9r68v7rtpar6ysjJ4eWngWouAAKCoCHjhBeD77y1e2rdvH9q1a6dItX5+figuLlak\nbDfjq26U5o4kDwDnz5+36LOX27Bhw9ya5AHg1q1bCA4OdmudIu3atcttSR7QUDdFUREQEQH88IPF\n4pCQEMWSPAAUFxdj/PjxipWvNnxE74SOHTsiLS3N7fWGhISgoKDA7fUqyRNO0vr4+OCBu86im/ng\ngw8wY8YMt9frMKlf0ssLKC3Fyy+/jG+//dYtVWvmD6JtfESvhPHjxwtJ8gBQUFCAhIQEWctcsmSJ\nrOU5qqioCPv37xcag5KWL18uJMkDwIwZM/Cf//xHSN0OITL+lJWhoHlztyV5wNjF1aJFC7fVJwof\n0Tvgueeew6+//io6DCQnJ8vyb2d8fDx27NghQ0SuISLHprRjDgkMDMTdu3dFh1Gt9PR0tG7dWkjd\nzZo1Q0ZGhpC6XcQnY5ltmZmZaNKkiegwTER1byjpkUceMc3hyqqnmRPI6sJdN3KSu8vEVevXr3dp\n+yFDhsgUiTxKSkpEhyCr69evqyrJq/3E97x584QneT3fTcuJ3k7btm0THYKFevXqubR9enq6TJHI\nw2AwIDk5WXQYspk9e7boECzcuXMHKvnv3aqRI0eKDgElJSW4qNPRNjnR2+HatWuq+2vftWtXp7et\nXbu2jJHIR09X3yxbtkx0CJUEBQWJDsGqS5cuqWafnKCa8b/lxX30dnDnjS6OcPbSxOPHj+Ppp59W\nICLX/fbbb2jZsqXoMFzyt7/9Df/+979Fh6EZUVFRDg3oxixwH71c9rowCJOSxowZ49R2ak3ygPEe\nBa3T1IiSKpCVlSU6BAtauELJUZzoq1FUVCTskq/qLFy4UHQIsnvvvfdEh+AytSUucxs3bhQdQiUh\nISGiQ7Dg6vkvNeJEXw29jbQ4b9480SFUSQ+Tlvj6+ooOwSa97c9KaNWqlegQZMeJvhpff/11tet0\n69atyn7liq/VqVPH9PuuXbsAADEPR/JLTEw0vabETUSbN2+udp1u3bqhb9++VidOMb86QpotS/od\nsHyvdevWBWAcMO3KlStYvXo1PvvsM5fi14OpU6ciMjLS5uebk5ODDh06mJ6bt5t5exsMhkrrVkdt\n55rsOZfRuXNnq0f9OTk5Fs+lNlm/fj26VZjgZM2aNUhJSQEAvPrqq1XWp8s/hkSkhh/VMjaRbaNH\nj6bRo0cTEVGvXr2IiKhp06aV1pNeq1h2q1atqF+/fhQcHGyxfOLEieTn5+dK6FY58n5WrVpFROXv\nx9q2Tz/9NE2cONHiNem9Tps2rdL6oaGhzgWuE999952pfVNTU4mIaO3ataZlkri4OIvnoaGhNveJ\niutWpXPnzo6GrKgxY8ZU+ToAunPnDhGV71f+/v5VbhMdHU3z5s2zWJaTk0OXLl0ylVmV3NzcKl9X\nGbtyrOgEr/lEL9m2bRsREZ04cYI2bdpk9TVzCQkJ5O/vT8XFxdSjRw8iIurZs6dDdTrDkbJv375t\n8X7s2dbae3W0Xk8xa9YsIiJq1apVpdccSd5xcXHk7+9PMF69VuW6L774okMxKu3111+v8nXp/UiJ\n+4033qAHDx5Uu83Fixct2uTw4cOUnp5uUaYt0h8WjbArx/LlldUwGAyqvtHEUbVq1Sqf2Jt5nAYN\nGuDChQuiwzBZuHAhJk2aJDoMC2lpaVq6+osvr5RDw4YNRYcgq7/+9a+iQ2ACqe3Oz549e4oOoZLv\nK0yAogd8RF+Nr7/+Gv369RMdhmxOnjyJ2NhY0WHYpOSsQgwYNGgQNmzYIDoMVdPYf/F8RC8HNSd5\nZ67XVnOSB4Du3buLDsFlH330kegQbOIkX71PPvlEdAiy40Rvh2bNmokOwarGjRs7tZ2a58pcvHix\n6BBctmLFCtEhaMorr7wiOgQLY8eOFR2C7DjR20GtQwakpqY6td1TTz0lcyTyICJVjGLoqpMnT4oO\nwSq1jg5aWFgoOgQTvU5Yz330HujcuXNo1KiR6DAq8fb2RmlpqegwZFFaWgpvb2/RYViYMmUKkpKS\nRIeharGxsar9Q20D99HL6dFHHxUdggVX7ppt1KgR8vPzZYxGHnpJ8oD6uvuGDRum6iTv4+MjOgTM\nnj1ba0nebpzo7XTkyBHRIVhw9T+xBg0ayBSJPNR8AtMZmZmZOHv2rOgwTLp06SI6hCqpYRrJ8PBw\n0SEohhO9nR577DH07t1bdBgAgNDQUJfLUNM0dwAwefJk0SHITi39vWvWrNHE+C0ipxIcM2aM08N+\nawH30WvMK6+8gi1btshS1qpVqzBixAhZynKFHicGlzz66KPIzc0VHYZmBAUFqerkrAZwH70SRI7n\nTUSyJXkAGDFiBKZMmSJbec44cuSIbpM8AOTm5uLSpUvC6lfbpYvVKSwsdOt8uwUFBbhx44bb6hOF\nE72DXnvtNWETkSjxBUhKSrI6HLE7EJHQf9fdRVSiDwoKkvXAwF3cmehvLFyomvlqlcRdN05y97/k\nL7zwgqJjcKilG0evkpOTMX78eLfV5+XlhbKyMrfVp4QOHTpgz549ipV/48YNY5L38wNUfBNhNbjr\nRglr1hgfc3Nz3XIkXFBQAED5gZZGjBiBoKAgReuQ3Llzxy31qImU5A8ePKh4XU888YTmkzwA7Nmz\nR5EhO9555x0AKD+SLy4GZLjAQc040TvozTfLf09JScG5c+cUGwDp5s2b8Pf3V6RsawoLCxXvSpk8\neTJq6Oh6eUe1adNG0WQfHh6uqss6XXXy5El8+OGHsn3HmjZtan0sm1u3gMxMWepQI+66cUBZGWAr\nD8o94l1gYKCw2eiPHTuGJ554AsHBwbKW265dO+zbtw+4dw9w4x8wNdq8eTNeeuklBAQEyFJecHCw\n8T8lb29Ap39Is7OzMXfuXKxbt86h7e7cuYMWLVogOzu7+pWffBI4fdrJCIXgrhs55efbTvJA+Q1M\n7du3d3pij9OnT5uu1ReV5AGgefPmCA4OxrFjxxz+UlVUVlZmuot33759xoX+/oCTA7LpRf/+/REQ\nEICkpCS75vG1pqyszHSTj6k7rLTUuLPqUExMjGl/zM/PR2BgIPr06YPdu3cjLy8PAHD48GEMHz4c\nBoMBixYtAmD8I2hXkgeMSV6Hk4OLnkJQ9VMJFhYSZWU5t+2TTz5JzZs3p/Pnz1t9PSMjgyIjI2n6\n9OkuROgeBoOBFi1aZNe6x48fJ39/f7py5UrVKyYkyBCZfixZsoSCg4Np7969Vl+/du0ade7cmWrX\nrl11QXl5CkTnYeLjRUdgL55KUA779gGyzoNhMADqaHOnEAEjR47A2rVrUVZWhscffxx5eXnIy8tD\n165dsWnTJtStW9f+wvLygFq1lA1aAxYvBiZMkLHAvn2BrVtlLNADVdVXqx52dd1woq/CwoWA7NNZ\najzRy6601Niv7OG6dwd++kl0FKyS+fOB998XHUVVuI/eFX36KJDkNU6RS5q9vYFnnlGgYG1RJMnP\nmKFAoR7m/feB7dtFR+EyPqK3olEj4Nw5hQrX8BG9oqH37w84eVJS63x8AMVGgfjLX4D9+xUq3IOc\nPAmocxpOu47oxQ8CrTK+vsD9+6KjUCdFZ1j78ksFC1e3pUsVLHz/fuMO7eurYCUeQOPnkfiI3kxY\nmBuuTNPwEb3iFP1XyoP98QfQsKHoKLRv3z6gbVu1naDlPnpH9Oyp28uPZfHuu26o5Nw5wAMGmDK3\ncqUbKmnYkE94y6FdO7UlebvxET2AVasAt43npdEjeo2GrXpuvZH1+++BF15wU2U69sYbwGefiY5C\nwpdX2uP2beCRR9xYoUYzZloa0LGjmyo7cQJo2tRNlTHmhMcfB86fFx0FwF039rGW5A0Gg+nH2pR7\nBoPBNFl448aNTeutWLECt2/fNi2LiIiosu6rV69i/fr1srwPpXXsCIwaNQrbt283tU14eDgyMjIs\n1pPeu/lzaT2pnQwGA+Li4kzrlJSUoH379uWFNG1qtatBS+3lKPN2NRgMNtvV2n4nPTZu3NiiXQFY\ntitgHMsFqLSeK5PNq4U09MiBAwdgMBiwfft203fSmpo1a5p+X7FihcWjJCIiAi+//DJOnDhhufH5\n80CFSYgiIiLU24723kKr8I8Qp09bPvfz8yMiIgAUHx9PeXl51LZtW2rSpInFegDo8ccfJyKiyMhI\n03pERG3btqXIyEgCQFFRUUSpMf+zAAAJdElEQVRE9MMPP5hvTEREJSUlVK9ePRo/frwSb00RV69e\nJSLj+58xYwYFBgZSaGioRfvExsYSUP6RRkZGmtaT2sm8/UpKSmj16tX04MGDyhWaDaGgxfaqTp06\nxkdr+52tdrW230mPkZGRpteJjPud1XZdscKi/a9cuUIdO3ZU6F26V5cuXejevXv05ptv0u3bt4nI\n+J1MSUkhIqKioiIiIiooKCAAVFBQUGV5UVFRNHfuXNqzZ4/1Ffr1s1j36aefluFdOMSuHCs6wQtL\n9L6+lZclJydbXfedd95xup558+YREVHLli2NCyDk7brk7beNj/Pnz7f6uivtIzl79mzlhVOmuFyu\nmk2ebHxUcr+z2q6//eZ0uVqVkZHh9Lao7jvbpo3TZcuAx7qxRdEbVKqjwT76qCjg8mVBlT/7LHDo\nkKDKdczf3zhcNJPHsmUK32hiE/fRW9Orl8Akr1FC57E4dMg43ojO/Pij4ADu3eMT3nIaOxaIjhYd\nhU0edUR/9SpQzflR5WnoiJ7vs1FORIRxf2Q6M3u28cd9+IheMnKkcYAy4T0Afn7GR7Wema+gXj1j\nqKq418bXVyWBuM5gAG7eNP63L1ydOprZH6tiMBh/duwQHMjs2cYbI7p2FRyIJY84ojffj4W+3ZMn\ngbg4zRzRA8a2a9xY8HSa//pXef9nYSEQGCgwGNdJ+6Pw3SArq3ymL+HBuEY1bbp3L9Chg/F39wTD\nR/TmFi9WwU6gztHvqiV8zuQxY4wfXo0abr67TTnC90Wg/KDDYND8mNzS7iFc+/bl44WYXacvmu4T\n/YoVxrtfx48XHYk2JSWJjsDM/fu6mPh6507REVRQVgb885+io3CZat7CkiXGfVVFg2epJtErdUfZ\nW28BISH2r//VV18pEoeJDIdyisdoRsnJdfr37+/4Rm4+FFZiv+ze3bXtnWq36ijcru7YZ995R55y\nqmpfu99HjRqKtqmj+6VqEj1jjDFlcKJnjDGd40TPGGM6p8pEn5WVZfO1bt26WTxPTEy0eP7qq68C\nAPr06YMxY8YAAEaOHAkAmDVrFgDgk08+MfVxhTjSgW/mnSo6BDt37mzxfEaFSZo7SJdfPbRmzRqk\npKRYLFu/fr0p/qVOzjXnSoxSm0lycnJMcUvrjhkzBleuXAFQ3mcotbEzGkuX+lUwdepUi+cV2+sz\ns7HB69atCwDo27cvrly5gszMTAwYMAAAUKdOHdNr0jJHZGVl2YzR2n5pHqPUZtLolOYKCgqwa9cu\nAOWftdSO1ta3l63tli5dilOnTlVaXrFdpe9WZmamqV3ffXhFifl3SHo0/y62aNHC4XgdaV9bsQIw\ntaX0eZt/l6T9Wto/XGlfW98va+1r/v0y/y6Zq9ie5vt1TEyMUzGa2DsojsI/VgcOGj58OGVkZFC/\nfv3o2LFjBIBGjx5NWVlZlJWVRS+++CJV3Fb6PSoqiqZOnWp63rp1a9PrPXr0MI0YWNGmTZuqHkLI\nSoxERFMeDsAFgC5cuFApxoEDB5q2iYuLIyKi/fv3ExFRTk4OXbp0yaLc6Ohomjp1qsX6csQotaO1\nGKW2kh4HDhxoilGKw3zdnJwcGjBgAE2cONHqZ1CdfmYj/0nlFxUV0fPPP0+nTp2iTp060XfffWfx\nmf/xxx9W20sybdq0SstWrVpl1zJrKr6XuLg4SkpKMu2XnTp1otGjR1vEKJFirNi+FUc4HD16NLVq\n1cpUvvm+SkSUmppqsX7Fdqsu9meffZaIyvfRsLAwU6zm8VprV6kMqV07d+5MREQ9evSguXPn2v3Z\n27vPmrevtA9Ya1/zWM3bF4CpLSXSd8l8vzbnSPtaex/Vta+1eit+p5csWUIAKrWnFE9wcLDFMrN1\ntDV6ZVXJoVGjRqYPkYhMDSiN+mdtR7t+/brFNpINGzZYPJdG+ZM4kkTnzJlDRESlpaX0/fffm+q/\nceNGpRiTkpJM28XFxZG/v78pvsOHD1N6erppWUxMjOk16ScmJsapGCVSjFJ51mKsmOiTkpIs2jAu\nLs7muuYqtrEt1hJ9YGAgEREdOHDA4stgnuil9qqOFJs0XC0RUUJCQqVl9pRhHmNhYSERGfdLazFK\npBit7afmnzURUXFxcaX9VWrHWbNmWcTgSKKX9lEiMu2jcXFxVhO9tXa19vn27NmTiMg0DHd16xM5\nluil9rW1D1SM1bx9/f39qbi42LRNQkKCxfeIqPy7KD13pH0rvg972rdivdI65jnAWq6S9OjRo9Iy\nXST6s2fP0uLFi6lWrVp0/fp1Cg0NpZo1axIAOnr0qMUbHjJkCBERNWzY0KIB2rZtSzNnziQiov79\n+xMR0V/+8hciIgoKCqLU1FTas2cPAaAzZ86YyrN3hxw3bhzl5eVRrVq1TPW9/PLLVj+s6dOnW8RY\no0YNi9dXr15NK1eutFi2bNkyU/zOHtGPGzfO1I5SjFI7Vhej1GaSn3/+2RS3tO6JEyeIiKhFixb0\n5ptvElF5G9uj4hcKAJ0/f55GjRpFy5Yto+TkZNq9e7fFl4mocnuNGjWKjhw5QrNnzzaNV1+zZk0i\nIoqJiaGdO3cSEdHbD8dbrlmzpmlZdSrulwCoT58+pv0yOTmZRo8ebXW/XLlyJS1atIiIytvMmjff\nfJMGDx5MROWfdVXt6Eiiz8vLo4iICCIq30e9vLysri+1qxSz9N2KiYkxtevQoUOJyPgdIrL87KX1\nAwMDK+1j9uyzFdtX2gesta+tWPPy8kxtKX3e5t8lab+W9g9r7E30Ug6wp32lz//SpUsW3yVzACza\nU9qviYhmzJhRaV3pV3t+RCd4u47o3cmZo2V300KM9rA3YYmklv3SnBbarSIt7bOOdt2I4GiiV+XJ\nWMYYY/LhRM8YYzrHiZ4xxnRONcMUm89wL9K9e/eQb2Mwogjhs5YYaSFGc1dtzLDRv39//PLLL26O\nxjpbMaplvzSXn5+PezamAVTj5w8AS5YsqXTPi0RtMVfVvmFhYfD393dzRJXl5ubiYe626yYA1SR6\n0QEwxpgG2ZXofZSOwk7an+KGMcZUivvoGWNM5zjRM8aYznGiZ4wxneNEzxhjOseJnjHGdI4TPWOM\n6RwnesYY0zlO9IwxpnOc6BljTOc40TPGmM5xomeMMZ3jRM8YYzrHiZ4xxnSOEz1jjOkcJ3rGGNM5\nTvSMMaZznOgZY0znONEzxpjOcaJnjDGd40TPGGM6x4meMcZ0jhM9Y4zpHCd6xhjTuf8PaySRs2ri\nitcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f2bac236860>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax=plot_tree(xgb,num_trees=0)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 组合特征(生成新的特征)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用GBDT生成新的特征,这里只用了30颗树"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X,y=make_hastie_10_2(random_state=0)\n",
    "X=pd.DataFrame(X)\n",
    "y=pd.DataFrame(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y.columns={\"label\"}\n",
    "label={-1:0,1:1}\n",
    "y.label=y.label.map(label)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9600, 10)"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>843</th>\n",
       "      <td>-1.696838</td>\n",
       "      <td>-0.851699</td>\n",
       "      <td>-0.922525</td>\n",
       "      <td>0.923324</td>\n",
       "      <td>-0.395004</td>\n",
       "      <td>0.709935</td>\n",
       "      <td>-0.042934</td>\n",
       "      <td>-0.649640</td>\n",
       "      <td>-0.318538</td>\n",
       "      <td>-1.872091</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9450</th>\n",
       "      <td>0.235742</td>\n",
       "      <td>-0.134053</td>\n",
       "      <td>-0.105796</td>\n",
       "      <td>1.635922</td>\n",
       "      <td>0.388091</td>\n",
       "      <td>-0.337705</td>\n",
       "      <td>-0.424280</td>\n",
       "      <td>0.170233</td>\n",
       "      <td>1.450414</td>\n",
       "      <td>-0.211502</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7766</th>\n",
       "      <td>0.889241</td>\n",
       "      <td>-0.310546</td>\n",
       "      <td>-0.978214</td>\n",
       "      <td>0.986720</td>\n",
       "      <td>0.304275</td>\n",
       "      <td>-1.228938</td>\n",
       "      <td>1.245357</td>\n",
       "      <td>-0.908876</td>\n",
       "      <td>3.678521</td>\n",
       "      <td>-0.813448</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9802</th>\n",
       "      <td>-0.333380</td>\n",
       "      <td>0.015419</td>\n",
       "      <td>2.148760</td>\n",
       "      <td>-0.909960</td>\n",
       "      <td>1.298145</td>\n",
       "      <td>-0.460775</td>\n",
       "      <td>0.750866</td>\n",
       "      <td>-1.151144</td>\n",
       "      <td>0.685288</td>\n",
       "      <td>0.303376</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8555</th>\n",
       "      <td>1.825373</td>\n",
       "      <td>0.203503</td>\n",
       "      <td>-0.344880</td>\n",
       "      <td>0.848101</td>\n",
       "      <td>-0.449497</td>\n",
       "      <td>0.898452</td>\n",
       "      <td>-0.314597</td>\n",
       "      <td>0.110325</td>\n",
       "      <td>-0.221885</td>\n",
       "      <td>-2.015231</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             0         1         2         3         4         5         6  \\\n",
       "843  -1.696838 -0.851699 -0.922525  0.923324 -0.395004  0.709935 -0.042934   \n",
       "9450  0.235742 -0.134053 -0.105796  1.635922  0.388091 -0.337705 -0.424280   \n",
       "7766  0.889241 -0.310546 -0.978214  0.986720  0.304275 -1.228938  1.245357   \n",
       "9802 -0.333380  0.015419  2.148760 -0.909960  1.298145 -0.460775  0.750866   \n",
       "8555  1.825373  0.203503 -0.344880  0.848101 -0.449497  0.898452 -0.314597   \n",
       "\n",
       "             7         8         9  \n",
       "843  -0.649640 -0.318538 -1.872091  \n",
       "9450  0.170233  1.450414 -0.211502  \n",
       "7766 -0.908876  3.678521 -0.813448  \n",
       "9802 -1.151144  0.685288  0.303376  \n",
       "8555  0.110325 -0.221885 -2.015231  "
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clf = XGBClassifier(\n",
    "    n_estimators=30,#三十棵树\n",
    "    learning_rate =0.3,\n",
    "    max_depth=3,\n",
    "    min_child_weight=1,\n",
    "    gamma=0.3,\n",
    "    subsample=0.8,\n",
    "    colsample_bytree=0.8,\n",
    "    objective= 'binary:logistic',\n",
    "    nthread=12,\n",
    "    scale_pos_weight=1,\n",
    "    reg_lambda=1,\n",
    "    seed=27)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/python/lib/python3.6/site-packages/sklearn/preprocessing/label.py:95: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n",
      "/usr/local/python/lib/python3.6/site-packages/sklearn/preprocessing/label.py:128: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n",
       "       colsample_bytree=0.8, gamma=0.3, learning_rate=0.3,\n",
       "       max_delta_step=0, max_depth=3, min_child_weight=1, missing=None,\n",
       "       n_estimators=30, n_jobs=1, nthread=12, objective='binary:logistic',\n",
       "       random_state=0, reg_alpha=0, reg_lambda=1, scale_pos_weight=1,\n",
       "       seed=27, silent=True, subsample=0.8)"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.89000000000000001"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.score(X_test,y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_new_feature= clf.apply(X_train)#每个样本在每颗树叶子节点的索引值\n",
    "test_new_feature= clf.apply(X_test)\n",
    "train_new_feature2 = pd.DataFrame(train_new_feature)\n",
    "test_new_feature2 = pd.DataFrame(test_new_feature)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9600, 30)"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_new_feature2.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "      <th>24</th>\n",
       "      <th>25</th>\n",
       "      <th>26</th>\n",
       "      <th>27</th>\n",
       "      <th>28</th>\n",
       "      <th>29</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8</td>\n",
       "      <td>11</td>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>10</td>\n",
       "      <td>8</td>\n",
       "      <td>11</td>\n",
       "      <td>12</td>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>...</td>\n",
       "      <td>10</td>\n",
       "      <td>8</td>\n",
       "      <td>10</td>\n",
       "      <td>11</td>\n",
       "      <td>9</td>\n",
       "      <td>10</td>\n",
       "      <td>10</td>\n",
       "      <td>8</td>\n",
       "      <td>11</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10</td>\n",
       "      <td>11</td>\n",
       "      <td>9</td>\n",
       "      <td>11</td>\n",
       "      <td>11</td>\n",
       "      <td>9</td>\n",
       "      <td>11</td>\n",
       "      <td>12</td>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>...</td>\n",
       "      <td>10</td>\n",
       "      <td>9</td>\n",
       "      <td>11</td>\n",
       "      <td>11</td>\n",
       "      <td>9</td>\n",
       "      <td>10</td>\n",
       "      <td>10</td>\n",
       "      <td>8</td>\n",
       "      <td>11</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>10</td>\n",
       "      <td>11</td>\n",
       "      <td>9</td>\n",
       "      <td>12</td>\n",
       "      <td>10</td>\n",
       "      <td>9</td>\n",
       "      <td>11</td>\n",
       "      <td>12</td>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>...</td>\n",
       "      <td>10</td>\n",
       "      <td>9</td>\n",
       "      <td>14</td>\n",
       "      <td>11</td>\n",
       "      <td>10</td>\n",
       "      <td>13</td>\n",
       "      <td>10</td>\n",
       "      <td>8</td>\n",
       "      <td>11</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>10</td>\n",
       "      <td>11</td>\n",
       "      <td>9</td>\n",
       "      <td>10</td>\n",
       "      <td>10</td>\n",
       "      <td>9</td>\n",
       "      <td>11</td>\n",
       "      <td>14</td>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>...</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>11</td>\n",
       "      <td>12</td>\n",
       "      <td>9</td>\n",
       "      <td>10</td>\n",
       "      <td>10</td>\n",
       "      <td>8</td>\n",
       "      <td>13</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>12</td>\n",
       "      <td>11</td>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>10</td>\n",
       "      <td>7</td>\n",
       "      <td>11</td>\n",
       "      <td>12</td>\n",
       "      <td>9</td>\n",
       "      <td>10</td>\n",
       "      <td>...</td>\n",
       "      <td>10</td>\n",
       "      <td>8</td>\n",
       "      <td>13</td>\n",
       "      <td>11</td>\n",
       "      <td>9</td>\n",
       "      <td>10</td>\n",
       "      <td>10</td>\n",
       "      <td>8</td>\n",
       "      <td>12</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 30 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   0   1   2   3   4   5   6   7   8   9  ...  20  21  22  23  24  25  26  27  \\\n",
       "0   8  11   9   9  10   8  11  12   9   9 ...  10   8  10  11   9  10  10   8   \n",
       "1  10  11   9  11  11   9  11  12   9   9 ...  10   9  11  11   9  10  10   8   \n",
       "2  10  11   9  12  10   9  11  12   9   9 ...  10   9  14  11  10  13  10   8   \n",
       "3  10  11   9  10  10   9  11  14   9   9 ...   8   9  11  12   9  10  10   8   \n",
       "4  12  11   9   9  10   7  11  12   9  10 ...  10   8  13  11   9  10  10   8   \n",
       "\n",
       "   28  29  \n",
       "0  11  12  \n",
       "1  11  13  \n",
       "2  11  13  \n",
       "3  13  12  \n",
       "4  12  12  \n",
       "\n",
       "[5 rows x 30 columns]"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_new_feature2.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在每个样本就是100维的,每一维表示原样本在这颗树的叶子节点的索引,这样我们就得到了新的特征.下面使用新的特征训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/python/lib/python3.6/site-packages/sklearn/preprocessing/label.py:95: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n",
      "/usr/local/python/lib/python3.6/site-packages/sklearn/preprocessing/label.py:128: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n",
       "       colsample_bytree=0.8, gamma=0.3, learning_rate=0.3,\n",
       "       max_delta_step=0, max_depth=3, min_child_weight=1, missing=None,\n",
       "       n_estimators=30, n_jobs=1, nthread=12, objective='binary:logistic',\n",
       "       random_state=0, reg_alpha=0, reg_lambda=1, scale_pos_weight=1,\n",
       "       seed=27, silent=True, subsample=0.8)"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.fit(train_new_feature2, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.87541666666666662"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.score(test_new_feature2,y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 原特征和组合特征一起训练"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里只用新的特征,也可以有很好的预测效果,说明新的特征是有用的,\n",
    "现在我们把旧的特征和新的特征整合在一起训练.先写把特征合并的方法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn import metrics\n",
    "from  sklearn.datasets  import  make_hastie_10_2\n",
    "from xgboost.sklearn import XGBClassifier\n",
    "##载入示例数据 10维度\n",
    "X, y = make_hastie_10_2(random_state=0)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)##test_size测试集合所占比例\n",
    "##X_train_1用于生成模型  X_train_2用于和新特征组成新训练集合\n",
    "X_train_1, X_train_2, y_train_1, y_train_2 = train_test_split(X_train, y_train, test_size=0.6, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "def mergeToOne(X,X2):\n",
    "    X3=[]\n",
    "    for i in range(X.shape[0]):\n",
    "        tmp=np.array([list(X[i]),list(X2[i])])\n",
    "        X3.append(list(np.hstack(tmp)))\n",
    "    X3=np.array(X3)\n",
    "    return X3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clf = XGBClassifier(\n",
    " learning_rate =0.3, #默认0.3\n",
    " n_estimators=30, #树的个数\n",
    " max_depth=3,\n",
    " min_child_weight=1,\n",
    " gamma=0.5,\n",
    " subsample=0.6,\n",
    " colsample_bytree=0.6,\n",
    " objective= 'binary:logistic', #逻辑回归损失函数\n",
    " nthread=4,  #cpu线程数\n",
    " scale_pos_weight=1,\n",
    " reg_alpha=1e-05,\n",
    " reg_lambda=1,\n",
    " seed=27)  #随机种子"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clf.fit(X_train_1, y_train_1)\n",
    "new_feature= clf.apply(X_train_2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train_new2=mergeToOne(X_train_2,new_feature)\n",
    "new_feature_test= clf.apply(X_test)\n",
    "X_test_new=mergeToOne(X_test,new_feature_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = XGBClassifier(\n",
    " learning_rate =0.1,\n",
    " n_estimators=2000,\n",
    " max_depth=3,\n",
    " min_child_weight=1,\n",
    " gamma=0.4,\n",
    " subsample=0.6,\n",
    " colsample_bytree=0.6,\n",
    " objective= 'binary:logistic', \n",
    " nthread=4, \n",
    " scale_pos_weight=1,\n",
    " reg_alpha=1e-05,\n",
    " reg_lambda=1,\n",
    " seed=27)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AUC Score : 0.992485\n",
      "Accuracy : 0.9563\n"
     ]
    }
   ],
   "source": [
    "model.fit(X_train_new2, y_train_2)\n",
    "y_pre= model.predict(X_test_new)\n",
    "y_pro= model.predict_proba(X_test_new)[:,1] \n",
    "print(\"AUC Score : %f\" % metrics.roc_auc_score(y_test, y_pro) )\n",
    "print(\"Accuracy : %.4g\" % metrics.accuracy_score(y_test, y_pre) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AUC Score : 0.993615\n",
      "Accuracy : 0.9625\n"
     ]
    }
   ],
   "source": [
    "model = XGBClassifier(\n",
    " learning_rate =0.1,\n",
    " n_estimators=2000,\n",
    " max_depth=3,\n",
    " min_child_weight=1,\n",
    " gamma=0.6,\n",
    " subsample=0.7,\n",
    " colsample_bytree=0.8,\n",
    " objective= 'binary:logistic', \n",
    " nthread=4, \n",
    " scale_pos_weight=1,\n",
    " reg_alpha=1e-05,\n",
    " reg_lambda=1,\n",
    " seed=27)\n",
    "model.fit(X_train_new2, y_train_2)\n",
    "y_pre= model.predict(X_test_new)\n",
    "y_pro= model.predict_proba(X_test_new)[:,1] \n",
    "print(\"AUC Score : %f\" % metrics.roc_auc_score(y_test, y_pro) )\n",
    "print(\"Accuracy : %.4g\" % metrics.accuracy_score(y_test, y_pre) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "生成新的特征后再和原来的特征一起合并作为训练数据,这是目前最好的效果,这里还没有进行网格搜索,应该有更好的效果"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}