原理見：http://www.myexception.cn/operating-system/2084839.html
譯文轉自：http://blog.csdn.net/zc02051126/article/details/46771793
在Python中使用XGBoost
下面將介紹XGBoost的Python模塊，內容如下：
* 編譯及導入Python模塊
* 數據接口
* 參數設置
* 訓練模型l
* 提前終止程序
* 預測
A walk through python example for UCI Mushroom dataset is provided.
安裝
首先安裝XGBoost的C++版本，然後進入源文件的根目錄下的 wrappers文件夾執行如下腳本安裝Python模塊
<code class="language-shell hljs cmake has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">python setup.py <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">install</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
安裝完成後按照如下方式導入XGBoost的Python模塊
<code class="language-python hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> xgboost <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">as</span> xgb</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
=
數據接口
XGBoost可以加載libsvm格式的文本數據，加載的數據格式可以爲Numpy的二維數組和XGBoost的二進制的緩存文件。加載的數據存儲在對象DMatrix中。
- 加載libsvm格式的數據和二進制的緩存文件時可以使用如下方式
<code class="language-python hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">dtrain = xgb.DMatrix(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'train.svm.txt'</span>) dtest = xgb.DMatrix(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'test.svm.buffer'</span>)</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>
- 加載numpy的數組到
DMatrix對象時，可以用如下方式
<code class="language-python hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">data = np.random.rand(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>) <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># 5 entities, each contains 10 features</span> label = np.random.randint(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>, size=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>) <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># binary target</span> dtrain = xgb.DMatrix( data, label=label)</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li></ul>
- 將
scipy.sparse格式的數據轉化爲DMatrix格式時，可以使用如下方式
<code class="language-python hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">csr = scipy.sparse.csr_matrix( (dat, (row,col)) ) dtrain = xgb.DMatrix( csr )</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>
- 將
DMatrix格式的數據保存成XGBoost的二進制格式，在下次加載時可以提高加載速度，使用方式如下
<code class="language-python hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">dtrain = xgb.DMatrix(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'train.svm.txt'</span>) dtrain.save_binary(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"train.buffer"</span>)</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>
- 可以用如下方式處理
DMatrix中的缺失值：
<code class="language-python hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">dtrain = xgb.DMatrix( data, label=label, missing = -<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">999.0</span>)</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
- 當需要給樣本設置權重時，可以用如下方式
<code class="language-python hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">w = np.random.rand(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>) dtrain = xgb.DMatrix( data, label=label, missing = -<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">999.0</span>, weight=w)</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>
參數設置
XGBoost使用key-value格式保存參數. Eg
* Booster（基本學習器）參數
<code class="language-python hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">param = {<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'bst:max_depth'</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'bst:eta'</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'silent'</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'objective'</span>:<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'binary:logistic'</span> }
param[<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'nthread'</span>] = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span>
plst = param.items()
plst += [(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'eval_metric'</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'auc'</span>)] <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># Multiple evals can be handled in this way</span>
plst += [(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'eval_metric'</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'ams@0'</span>)]</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li></ul>
- 還可以定義驗證數據集，驗證算法的性能
<code class="language-python hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">evallist = [(dtest,<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'eval'</span>), (dtrain,<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'train'</span>)]</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
=
訓練模型
有了參數列表和數據就可以訓練模型了
* 訓練
<code class="language-python hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">num_round = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span> bst = xgb.train( plst, dtrain, num_round, evallist )</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>
- 保存模型
在訓練完成之後可以將模型保存下來，也可以查看模型內部的結構
<code class="language-python hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">bst.save_model(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'0001.model'</span>)</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
- Dump Model and Feature Map
You can dump the model to txt and review the meaning of model
<code class="language-python hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># dump model</span> bst.dump_model(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'dump.raw.txt'</span>) <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># dump model with feature map</span> bst.dump_model(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'dump.raw.txt'</span>,<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'featmap.txt'</span>)</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li></ul>
- 加載模型
通過如下方式可以加載模型
<code class="language-python hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">bst = xgb.Booster({<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'nthread'</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span>}) <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#init model</span>
bst.load_model(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"model.bin"</span>) <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># load data</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>
=
提前終止程序
如果有評價數據，可以提前終止程序，這樣可以找到最優的迭代次數。如果要提前終止程序必須至少有一個評價數據在參數evals中。 If there’s more than one, it will use the last.
train(..., evals=evals, early_stopping_rounds=10)
The model will train until the validation score stops improving. Validation error needs to decrease at least every early_stopping_rounds to
continue training.
If early stopping occurs, the model will have two additional fields: bst.best_score and bst.best_iteration.
Note that train() will return a model from the last iteration, not the best one.
This works with both metrics to minimize (RMSE, log loss, etc.) and to maximize (MAP, NDCG, AUC).
=
Prediction
After you training/loading a model and preparing the data, you can start to do prediction.
<code class="language-python hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">data = np.random.rand(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">7</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>) <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># 7 entities, each contains 10 features</span> dtest = xgb.DMatrix( data, missing = -<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">999.0</span> ) ypred = bst.predict( xgmat )</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li></ul>
If early stopping is enabled during training, you can predict with the best iteration.
<code class="language-python hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">ypred = bst.predict(xgmat,ntree_limit=bst.best_iteration)</code>
