關注微信公共號：小程在線
關注CSDN博客：程志偉的博客
評分卡之數據預處理：重複值、填補缺失值、異常值與數據不平衡
在銀行借貸場景中，評分卡是一種以分數形式來衡量一個客戶的信用風險大小的手段，它衡量向別人借錢的人（受信人，需要融資的公司）不能如期履行合同中的還本付息責任，並讓借錢給別人的人（授信人，銀行等金融機構）造成經濟損失的可能性。一般來說，評分卡打出的分數越高，客戶的信用越好，風險越小。
Python 3.7.3 (default, Apr 24 2019, 15:29:51) [MSC v.1915 64 bit (AMD64)]
Type "copyright", "credits" or "license" for more information.
IPython 7.6.1 -- An enhanced Interactive Python.
1.導入庫並獲取數據
%matplotlib inline
import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression as LR
data = pd.read_csv(r"H:\程志偉\python\rankingcard.csv",index_col=0)
2.數據探索與數據預處理
#觀察數據類型
data.head()
Out[3]:
SeriousDlqin2yrs ... NumberOfDependents
1 1 ... 2.0
2 0 ... 1.0
3 0 ... 0.0
4 0 ... 0.0
5 0 ... 0.0
[5 rows x 11 columns]
#觀察數據結構
data.shape
Out[4]: (150000, 11)
data.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 150000 entries, 1 to 150000
Data columns (total 11 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 SeriousDlqin2yrs 150000 non-null int64
1 RevolvingUtilizationOfUnsecuredLines 150000 non-null float64
2 age 150000 non-null int64
3 NumberOfTime30-59DaysPastDueNotWorse 150000 non-null int64
4 DebtRatio 150000 non-null float64
5 MonthlyIncome 120269 non-null float64
6 NumberOfOpenCreditLinesAndLoans 150000 non-null int64
7 NumberOfTimes90DaysLate 150000 non-null int64
8 NumberRealEstateLoansOrLines 150000 non-null int64
9 NumberOfTime60-89DaysPastDueNotWorse 150000 non-null int64
10 NumberOfDependents 146076 non-null float64
dtypes: float64(4), int64(7)
memory usage: 13.7 MB
數據字段描述
SeriousDlqin2yrs 出現 90 天或更長時間的逾期行爲（即定義好壞客戶）
RevolvingUtilizationOfUnsecuredLines 貸款以及信用卡可用額度與總額度比例
age 借款人借款年齡
NumberOfTime30-59DaysPastDueNotWorse 過去兩年內出現35-59天逾期但是沒有發展得更壞的次數
DebtRatio 每月償還債務，贍養費，生活費用除以月總收入
MonthlyIncome 月收入
NumberOfOpenCreditLinesAndLoans 開放式貸款和信貸數量
NumberOfTimes90DaysLate 過去兩年內出現90天逾期或更壞的次數
NumberRealEstateLoansOrLines 抵押貸款和房地產貸款數量，包括房屋淨值信貸額度
NumberOfTime60-89DaysPastDueNotWorse 過去兩年內出現60-89天逾期但是沒有發展得更壞的次數
NumberOfDependents 家庭中不包括自身的家屬人數（配偶，子女等）
2.1 去重數據
#去除重複值
data.drop_duplicates(inplace=True)
data.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 149391 entries, 1 to 150000
Data columns (total 11 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 SeriousDlqin2yrs 149391 non-null int64
1 RevolvingUtilizationOfUnsecuredLines 149391 non-null float64
2 age 149391 non-null int64
3 NumberOfTime30-59DaysPastDueNotWorse 149391 non-null int64
4 DebtRatio 149391 non-null float64
5 MonthlyIncome 120170 non-null float64
6 NumberOfOpenCreditLinesAndLoans 149391 non-null int64
7 NumberOfTimes90DaysLate 149391 non-null int64
8 NumberRealEstateLoansOrLines 149391 non-null int64
9 NumberOfTime60-89DaysPastDueNotWorse 149391 non-null int64
10 NumberOfDependents 145563 non-null float64
dtypes: float64(4), int64(7)
memory usage: 13.7 MB
#刪除之後千萬不要忘記，恢復索引，標紅部分發生變化
data.index = range(data.shape[0])
data.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 149391 entries, 0 to 149390
Data columns (total 11 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 SeriousDlqin2yrs 149391 non-null int64
1 RevolvingUtilizationOfUnsecuredLines 149391 non-null float64
2 age 149391 non-null int64
3 NumberOfTime30-59DaysPastDueNotWorse 149391 non-null int64
4 DebtRatio 149391 non-null float64
5 MonthlyIncome 120170 non-null float64
6 NumberOfOpenCreditLinesAndLoans 149391 non-null int64
7 NumberOfTimes90DaysLate 149391 non-null int64
8 NumberRealEstateLoansOrLines 149391 non-null int64
9 NumberOfTime60-89DaysPastDueNotWorse 149391 non-null int64
10 NumberOfDependents 145563 non-null float64
dtypes: float64(4), int64(7)
memory usage: 12.5 MB
2.2 填補缺失值
查看每列缺失值的情況
data.isnull().sum()
Out[10]:
SeriousDlqin2yrs 0
RevolvingUtilizationOfUnsecuredLines 0
age 0
NumberOfTime30-59DaysPastDueNotWorse 0
DebtRatio 0
MonthlyIncome 29221
NumberOfOpenCreditLinesAndLoans 0
NumberOfTimes90DaysLate 0
NumberRealEstateLoansOrLines 0
NumberOfTime60-89DaysPastDueNotWorse 0
NumberOfDependents 3828
dtype: int64
#查看缺失值的比例
data.isnull().sum()/data.shape[0]
Out[11]:
SeriousDlqin2yrs 0.000000
RevolvingUtilizationOfUnsecuredLines 0.000000
age 0.000000
NumberOfTime30-59DaysPastDueNotWorse 0.000000
DebtRatio 0.000000
MonthlyIncome 0.195601
NumberOfOpenCreditLinesAndLoans 0.000000
NumberOfTimes90DaysLate 0.000000
NumberRealEstateLoansOrLines 0.000000
NumberOfTime60-89DaysPastDueNotWorse 0.000000
NumberOfDependents 0.025624
dtype: float64
data.isnull().mean()
Out[12]:
SeriousDlqin2yrs 0.000000
RevolvingUtilizationOfUnsecuredLines 0.000000
age 0.000000
NumberOfTime30-59DaysPastDueNotWorse 0.000000
DebtRatio 0.000000
MonthlyIncome 0.195601
NumberOfOpenCreditLinesAndLoans 0.000000
NumberOfTimes90DaysLate 0.000000
NumberRealEstateLoansOrLines 0.000000
NumberOfTime60-89DaysPastDueNotWorse 0.000000
NumberOfDependents 0.025624
dtype: float64
使用均值填補“家屬人數”
data["NumberOfDependents"].fillna(int(data["NumberOfDependents"].mean()),inplace=True)
data.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 149391 entries, 0 to 149390
Data columns (total 11 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 SeriousDlqin2yrs 149391 non-null int64
1 RevolvingUtilizationOfUnsecuredLines 149391 non-null float64
2 age 149391 non-null int64
3 NumberOfTime30-59DaysPastDueNotWorse 149391 non-null int64
4 DebtRatio 149391 non-null float64
5 MonthlyIncome 120170 non-null float64
6 NumberOfOpenCreditLinesAndLoans 149391 non-null int64
7 NumberOfTimes90DaysLate 149391 non-null int64
8 NumberRealEstateLoansOrLines 149391 non-null int64
9 NumberOfTime60-89DaysPastDueNotWorse 149391 non-null int64
10 NumberOfDependents 149391 non-null float64
dtypes: float64(4), int64(7)
memory usage: 12.5 MB
#填補之後在觀測每列的數據缺失比例
data.isnull().sum()/data.shape[0]
Out[15]:
SeriousDlqin2yrs 0.000000
RevolvingUtilizationOfUnsecuredLines 0.000000
age 0.000000
NumberOfTime30-59DaysPastDueNotWorse 0.000000
DebtRatio 0.000000
MonthlyIncome 0.195601
NumberOfOpenCreditLinesAndLoans 0.000000
NumberOfTimes90DaysLate 0.000000
NumberRealEstateLoansOrLines 0.000000
NumberOfTime60-89DaysPastDueNotWorse 0.000000
NumberOfDependents 0.000000
dtype: float64
使用隨機森林填補一個特徵的缺失值的函數
def fill_missing_rf(X,y,to_fill):
"""
使用隨機森林填補一個特徵的缺失值的函數
參數：
X：要填補的特徵矩陣
y：完整的，沒有缺失值的標籤
to_fill：字符串，要填補的那一列的名稱
X_train 特徵T不缺失的值：
Y_train 特徵T缺失的值對應的其他n-1個特徵 + 本來的標籤：
X_test 特徵T缺失的值：未知，
我們需要預測的Y_test
"""
#構建我們的新特徵矩陣和新標籤
df = X.copy()
fill = df.loc[:,to_fill]
df = pd.concat([df.loc[:,df.columns != to_fill],pd.DataFrame(y)],axis=1)
#找出我們的訓練集和測試集
Ytrain = fill[fill.notnull()]
Ytest = fill[fill.isnull()]
Xtrain = df.iloc[Ytrain.index,:]
Xtest = df.iloc[Ytest.index,:]
#用隨機森林迴歸來填補缺失值
from sklearn.ensemble import RandomForestRegressor as rfr
rfr = rfr(n_estimators=100)
rfr = rfr.fit(Xtrain, Ytrain)
Ypredict = rfr.predict(Xtest)
return Ypredict
將參數導入函數，產出結果：
X = data.iloc[:,1:]
y = data["SeriousDlqin2yrs"]
X.shape
Out[20]: (149391, 10)
y_pred = fill_missing_rf(X,y,"MonthlyIncome")
y_pred.shape
Out[22]: (29221,)
data.loc[data.loc[:,"MonthlyIncome"].isnull(),"MonthlyIncome"].shape
Out[23]: (29221,)
將預測數據填回到數據集中
data.loc[data.loc[:,"MonthlyIncome"].isnull(),"MonthlyIncome"] = y_pred
再次查看數據的缺失情況
data.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 149391 entries, 0 to 149390
Data columns (total 11 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 SeriousDlqin2yrs 149391 non-null int64
1 RevolvingUtilizationOfUnsecuredLines 149391 non-null float64
2 age 149391 non-null int64
3 NumberOfTime30-59DaysPastDueNotWorse 149391 non-null int64
4 DebtRatio 149391 non-null float64
5 MonthlyIncome 149391 non-null float64
6 NumberOfOpenCreditLinesAndLoans 149391 non-null int64
7 NumberOfTimes90DaysLate 149391 non-null int64
8 NumberRealEstateLoansOrLines 149391 non-null int64
9 NumberOfTime60-89DaysPastDueNotWorse 149391 non-null int64
10 NumberOfDependents 149391 non-null float64
dtypes: float64(4), int64(7)
memory usage: 12.5 MB
2.3 描述性統計處理異常值
data.describe([0.01,0.1,0.25,.5,.75,.9,.99]).T
Out[26]:
count ... max
SeriousDlqin2yrs 149391.0 ... 1.0
RevolvingUtilizationOfUnsecuredLines 149391.0 ... 50708.0
age 149391.0 ... 109.0
NumberOfTime30-59DaysPastDueNotWorse 149391.0 ... 98.0
DebtRatio 149391.0 ... 329664.0
MonthlyIncome 149391.0 ... 3008750.0
NumberOfOpenCreditLinesAndLoans 149391.0 ... 58.0
NumberOfTimes90DaysLate 149391.0 ... 98.0
NumberRealEstateLoansOrLines 149391.0 ... 54.0
NumberOfTime60-89DaysPastDueNotWorse 149391.0 ... 98.0
NumberOfDependents 149391.0 ... 20.0
[11 rows x 12 columns]
#異常值也被我們觀察到，年齡的最小值居然有0，這不符合銀行的業務需求，即便是兒童賬戶也要至少8歲，我們可以
(data["age"] == 0).sum()
Out[27]: 1
#發現只有一個人年齡爲0，可以判斷這肯定是錄入失誤造成的，可以當成是缺失值來處理，直接刪除掉這個樣本
data = data[data["age"] != 0]
data.shape
Out[29]: (149390, 11)
"""
另外，有三個指標看起來很奇怪：
"NumberOfTime30-59DaysPastDueNotWorse"
"NumberOfTime60-89DaysPastDueNotWorse"
"NumberOfTimes90DaysLate"
這三個指標分別是“過去兩年內出現35-59天逾期但是沒有發展的更壞的次數”，“過去兩年內出現60-89天逾期但是沒
有發展的更壞的次數”,“過去兩年內出現90天逾期的次數”。這三個指標，在99%的分佈的時候依然是2，最大值卻是
98，看起來非常奇怪。一個人在過去兩年內逾期35~59天98次，一年6個60天，兩年內逾期98次這是怎麼算出來的？
我們可以去諮詢業務人員，請教他們這個逾期次數是如何計算的。如果這個指標是正常的，那這些兩年內逾期了98次的
客戶，應該都是壞客戶。在我們無法詢問他們情況下，我們查看一下有多少個樣本存在這種異常：
"""
data[data.loc[:,"NumberOfTimes90DaysLate"] > 90].count()
Out[30]:
SeriousDlqin2yrs 225
RevolvingUtilizationOfUnsecuredLines 225
age 225
NumberOfTime30-59DaysPastDueNotWorse 225
DebtRatio 225
MonthlyIncome 225
NumberOfOpenCreditLinesAndLoans 225
NumberOfTimes90DaysLate 225
NumberRealEstateLoansOrLines 225
NumberOfTime60-89DaysPastDueNotWorse 225
NumberOfDependents 225
dtype: int64
#有225個樣本存在這樣的情況，並且這些樣本，我們觀察一下，標籤並不都是1，他們並不都是壞客戶。因此，我們基
本可以判斷，這些樣本是某種異常，應該把它們刪除。
data = data[data.loc[:,"NumberOfTimes90DaysLate"] < 90]
data.index = range(data.shape[0])
data.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 149165 entries, 0 to 149164
Data columns (total 11 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 SeriousDlqin2yrs 149165 non-null int64
1 RevolvingUtilizationOfUnsecuredLines 149165 non-null float64
2 age 149165 non-null int64
3 NumberOfTime30-59DaysPastDueNotWorse 149165 non-null int64
4 DebtRatio 149165 non-null float64
5 MonthlyIncome 149165 non-null float64
6 NumberOfOpenCreditLinesAndLoans 149165 non-null int64
7 NumberOfTimes90DaysLate 149165 non-null int64
8 NumberRealEstateLoansOrLines 149165 non-null int64
9 NumberOfTime60-89DaysPastDueNotWorse 149165 non-null int64
10 NumberOfDependents 149165 non-null float64
dtypes: float64(4), int64(7)
memory usage: 12.5 MB
2.4 樣本不均衡問題
X = data.iloc[:,1:]
y = data.iloc[:,0]
y.value_counts()
Out[35]:
0 139292
1 9873
Name: SeriousDlqin2yrs, dtype: int64
可以看出，樣本嚴重不均衡
n_sample = X.shape[0]
n_1_sample = y.value_counts()[1]
n_0_sample = y.value_counts()[0]
print('樣本個數：{}; 1佔{:.2%}; 0佔{:.2%}'.format(n_sample,n_1_sample/n_sample,n_0_sample/n_sample))
樣本個數：149165; 1佔6.62%; 0佔93.38%
使用上採樣方法來平衡樣本
import imblearn
from imblearn.over_sampling import SMOTE
sm = SMOTE(random_state=42) #實例化
X,y = sm.fit_sample(X,y)
n_sample_ = X.shape[0]
pd.Series(y).value_counts()
Out[38]:
1 139292
0 139292
dtype: int64
n_1_sample = pd.Series(y).value_counts()[1]
n_0_sample = pd.Series(y).value_counts()[0]
print('樣本個數：{}; 1佔{:.2%}; 0佔{:.2%}'.format(n_sample_,n_1_sample/n_sample_,n_0_sample/n_sample_))
樣本個數：278584; 1佔50.00%; 0佔50.00%
我們就實現了樣本平衡，樣本量也增加了
2.5 分訓練集和測試集
from sklearn.model_selection import train_test_split
X = pd.DataFrame(X)
y = pd.DataFrame(y)
X_train, X_vali, Y_train, Y_vali = train_test_split(X,y,test_size=0.3,random_state=420)
model_data = pd.concat([Y_train, X_train], axis=1)
model_data.index = range(model_data.shape[0])
model_data.columns = data.columns
vali_data = pd.concat([Y_vali, X_vali], axis=1)
vali_data.index = range(vali_data.shape[0])
vali_data.columns = data.columns
model_data.to_csv(r"H:\程志偉\python\model_data.csv")
vali_data.to_csv(r"H:\程志偉\python\\vali_data.csv")
作爲評分卡的數據預處理完成。