\qquad S ( x ) S(x) S ( x )
S ( x ) = 1 1 + e − x \qquad\qquad S(x)=\dfrac{1}{1+e^{-x}} S ( x ) = 1 + e − x 1
\qquad [ S ( x ) ] ′ = S ( x ) [ 1 − S ( x ) ] [S(x) ]^{'}=S(x) [ 1-S(x) ] [ S ( x ) ] ′ = S ( x ) [ 1 − S ( x ) ]
Sigmoid函數的曲線在中心 ( x = 0 , y = 0.5 ) (x=0,y=0.5 ) ( x = 0 , y = 0 . 5 )
\qquad S ( x ) S(x) S ( x ) f ( x ) = w T x + b f(\boldsymbol x)=\boldsymbol{w}^T \boldsymbol x + b f ( x ) = w T x + b
y ( x ) = S [ f ( x ) ] = 1 1 + e − ( w T x + b ) \qquad\qquad y(\boldsymbol{x})=S [ f(\boldsymbol x) ]=\dfrac{1}{1+e^{-(\boldsymbol{w}^{T}\boldsymbol{x}+b)}} y ( x ) = S [ f ( x ) ] = 1 + e − ( w T x + b ) 1
\qquad x ∗ \boldsymbol{x^{\ast}} x ∗ y = y ( x ∗ ) y=y(\boldsymbol x^{\ast}) y = y ( x ∗ )
ln ( y 1 − y ) = w T x ∗ + b \qquad\qquad \ln\left( \dfrac{y}{1-y}\right)=\boldsymbol{w}^{T}\boldsymbol{x^{\ast}}+b ln ( 1 − y y ) = w T x ∗ + b
\qquad y y y x ∗ \boldsymbol{x^{\ast}} x ∗ 正例 的可能性（概率），將 1 − y 1-y 1 − y x ∗ \boldsymbol{x^{\ast}} x ∗ 負例 的可能性（概率），兩者的比率取對數 ln ( y 1 − y ) \ln\left( \dfrac{y}{1-y}\right) ln ( 1 − y y ) x ∗ \boldsymbol{x^{\ast}} x ∗ 進行“線性分類”的情況 （如下圖所示）：
1 ) \qquad1) 1 ) y = 0.5 y=0.5 y = 0 . 5 1 − y = 0.5 1-y=0.5 1 − y = 0 . 5 ln ( y 1 − y ) = 0 \ln\left( \dfrac{y}{1-y}\right)=0 ln ( 1 − y y ) = 0 w T x ∗ + b = 0 \boldsymbol{w}^{T}\boldsymbol{x^{\ast}}+b=0 w T x ∗ + b = 0
\qquad x ∗ \boldsymbol{x^{\ast}} x ∗
2 ) \qquad2) 2 ) y > 0.5 y>0.5 y > 0 . 5 1 − y < 0.5 1-y<0.5 1 − y < 0 . 5 ln ( y 1 − y ) > 0 \ln\left( \dfrac{y}{1-y}\right)>0 ln ( 1 − y y ) > 0 w T x ∗ + b > 0 \boldsymbol{w}^{T}\boldsymbol{x^{\ast}}+b>0 w T x ∗ + b > 0
\qquad x ∗ \boldsymbol{x^{\ast}} x ∗
3 ) \qquad3) 3 ) y < 0.5 y<0.5 y < 0 . 5 1 − y > 0.5 1-y>0.5 1 − y > 0 . 5 ln ( y 1 − y ) < 0 \ln\left( \dfrac{y}{1-y}\right)<0 ln ( 1 − y y ) < 0 w T x ∗ + b < 0 \boldsymbol{w}^{T}\boldsymbol{x^{\ast}}+b<0 w T x ∗ + b < 0
\qquad x ∗ \boldsymbol{x^{\ast}} x ∗
通過Sigmoid函數可以將線性模型 y = w T x ∗ + b y=\boldsymbol{w}^{T}\boldsymbol{x^{\ast}}+b y = w T x ∗ + b y y y [ 0 , 1 ] [0,1] [ 0 , 1 ] p p p 機率 （odds）定義爲 p 1 − p \dfrac{p}{1-p} 1 − p p 對數機率 （log odds）定義爲 ln ( p 1 − p ) \ln\left(\dfrac{p}{1-p}\right) ln ( 1 − p p )
\qquad c = 1 c=1 c = 1 R 1 \mathcal R_1 R 1 c = 0 c=0 c = 0 R 2 \mathcal R_2 R 2 y ( x ) y(\boldsymbol x) y ( x ) 類後驗概率 ，即：
p ( c = 1 ∣ x ) = y ( x ) = 1 1 + e − ( w T x + b ) \qquad\qquad p(c=1|\boldsymbol{x})=y(\boldsymbol{x})=\dfrac{1}{1+e^{-(\boldsymbol{w}^{T}\boldsymbol{x}+b)}} p ( c = 1 ∣ x ) = y ( x ) = 1 + e − ( w T x + b ) 1
\qquad p ( c = 1 ∣ x ) p(c=1|\boldsymbol{x}) p ( c = 1 ∣ x )
ln p ( c = 1 ∣ x ) p ( c = 0 ∣ x ) = w T x + b \qquad\qquad\ln \dfrac{p(c=1|\boldsymbol{x})}{p(c=0|\boldsymbol{x})}=\boldsymbol{w}^T\boldsymbol{x}+b ln p ( c = 0 ∣ x ) p ( c = 1 ∣ x ) = w T x + b
x \boldsymbol{x} x 正例 ( c = 1 ) (c=1) ( c = 1 )
p ( c = 1 ∣ x ) = 1 1 + e − ( w T x + b ) \qquad\qquad p(c=1|\boldsymbol{x})=\dfrac{1}{1+e^{-(\boldsymbol{w}^{T}\boldsymbol{x}+b)}} p ( c = 1 ∣ x ) = 1 + e − ( w T x + b ) 1
\qquad w T x + b \boldsymbol{w}^{T}\boldsymbol{x}+b w T x + b + ∞ +\infty + ∞ 1 1 1 \qquad w T x + b \boldsymbol{w}^{T}\boldsymbol{x}+b w T x + b − ∞ -\infty − ∞ 0 0 0
x \boldsymbol{x} x 負例 ( c = 0 ) (c=0) ( c = 0 )
p ( c = 0 ∣ x ) = 1 − p ( y = 1 ∣ x ) = e − ( w T x + b ) 1 + e − ( w T x + b ) = 1 1 + e ( w T x + b ) \qquad\qquad\begin{aligned} p(c=0|\boldsymbol{x})&=1-p(y=1|\boldsymbol{x})\\
&=\dfrac{e^{-(\boldsymbol{w}^{T}\boldsymbol{x}+b)}}{1+e^{-(\boldsymbol{w}^{T}\boldsymbol{x}+b)}}\\
&=\dfrac{1}{1+e^{(\boldsymbol{w}^{T}\boldsymbol{x}+b)}}\end{aligned} p ( c = 0 ∣ x ) = 1 − p ( y = 1 ∣ x ) = 1 + e − ( w T x + b ) e − ( w T x + b ) = 1 + e ( w T x + b ) 1
\qquad w T x + b \boldsymbol{w}^{T}\boldsymbol{x}+b w T x + b + ∞ +\infty + ∞ 0 0 0 \qquad w T x + b \boldsymbol{w}^{T}\boldsymbol{x}+b w T x + b − ∞ -\infty − ∞ 1 1 1
\qquad x ∗ \boldsymbol{x^{\ast}} x ∗ p ( y = 1 ∣ x ∗ ) > p ( y = 0 ∣ x ∗ ) p(y=1|\boldsymbol{x^{\ast}})>p(y=0|\boldsymbol{x^{\ast}}) p ( y = 1 ∣ x ∗ ) > p ( y = 0 ∣ x ∗ ) x ∗ \boldsymbol{x^{\ast}} x ∗ R 1 R_{1} R 1 p ( y = 1 ∣ x ∗ ) < p ( y = 0 ∣ x ∗ ) p(y=1|\boldsymbol{x^{\ast}})<p(y=0|\boldsymbol{x^{\ast}}) p ( y = 1 ∣ x ∗ ) < p ( y = 0 ∣ x ∗ ) x ∗ \boldsymbol{x^{\ast}} x ∗ R 2 R_{2} R 2 \qquad
\qquad { ( x i , c i ) } i = 1 N \{ ( \boldsymbol{x}_{i},c_{i})
\} _{i=1}^N { ( x i , c i ) } i = 1 N x i ∈ R n , c i ∈ { 0 , 1 } \boldsymbol{x}_{i}\in R^{n},c_{i}\in \{0,1\} x i ∈ R n , c i ∈ { 0 , 1 } ( w , b ) (\boldsymbol{w},b) ( w , b )
1 ) \qquad1) 1 ) y ( x ) = p ( c = 1 ∣ x ) y(\boldsymbol{x})=p(c=1|\boldsymbol{x}) y ( x ) = p ( c = 1 ∣ x )
L ( w , b ) = ∏ i = 1 N y ( x i ) c i [ 1 − y ( x i ) ] 1 − c i \qquad\qquad L(\boldsymbol{w},b)=\displaystyle\prod_{i=1}^N y(\boldsymbol{x}_{i})^{c_{i}}\left[ 1-y(\boldsymbol{x}_{i})\right]
^{1-c_{i}} L ( w , b ) = i = 1 ∏ N y ( x i ) c i [ 1 − y ( x i ) ] 1 − c i
2 ) \qquad2) 2 )
ln L ( w , b ) = ∑ i = 1 N { c i ln [ y ( x i ) ] + ( 1 − c i ) ln [ 1 − y ( x i ) ] } = ∑ i = 1 N { c i ln y ( x i ) 1 − y ( x i ) + ln [ 1 − y ( x i ) ] } = ∑ i = 1 N { c i ( w T x i + b ) − ln [ 1 + e ( w T x i + b ) ] } \qquad\qquad\begin{aligned} \ln L(\boldsymbol{w},b)&=\displaystyle\sum_{i=1}^N \{ c_{i}\ln\left[
y\left( \boldsymbol{x}_{i}\right) \right] +(1-c_{i})\ln\left[ 1-y\left(
\boldsymbol{x}_{i}\right) \right] \}\\
&=\displaystyle\sum_{i=1}^N\left\{ c_{i}\ln\dfrac{
y\left( \boldsymbol{x}_{i}\right)}{1-y\left(
\boldsymbol{x}_{i}\right)} +\ln\left[ 1-y\left(
\boldsymbol{x}_{i}\right) \right] \right\} \\
&=\displaystyle\sum_{i=1}^N\left\{ c_{i}\left(\boldsymbol{w}^{T}\boldsymbol{x}_{i}+b\right)-\ln\left[ 1+e^{\left(\boldsymbol{w}^{T}\boldsymbol{x}_{i}+b\right)} \right] \right\} \\
\end{aligned} ln L ( w , b ) = i = 1 ∑ N { c i ln [ y ( x i ) ] + ( 1 − c i ) ln [ 1 − y ( x i ) ] } = i = 1 ∑ N { c i ln 1 − y ( x i ) y ( x i ) + ln [ 1 − y ( x i ) ] } = i = 1 ∑ N { c i ( w T x i + b ) − ln [ 1 + e ( w T x i + b ) ] }
3 ) \qquad3) 3 ) β = [ w T , b ] T , x ^ i = [ x i T , 1 ] T \boldsymbol{\beta}=[\boldsymbol{w}^T,b]^{T},\ \hat{\boldsymbol{x}}_{i}=[\boldsymbol{x}_{i}^T,1]^{T} β = [ w T , b ] T , x ^ i = [ x i T , 1 ] T w T x i + b = β T x ^ i \boldsymbol{w}^{T}\boldsymbol{x}_{i}+b=\boldsymbol{\beta}^{T}\hat{\boldsymbol{x}}_{i} w T x i + b = β T x ^ i
ln L ( β ) = ∑ i = 1 N [ c i β T x ^ i − ln ( 1 + e β T x ^ i ) ] \qquad\qquad \ln L(\boldsymbol{\beta})=\displaystyle\sum_{i=1}^N \left[ c_{i}\boldsymbol{\beta}^{T}\hat{\boldsymbol{x}}_{i}-\ln\left( 1+e^{\boldsymbol{\beta}^{T}\hat{\boldsymbol{x}}_{i}} \right) \right] ln L ( β ) = i = 1 ∑ N [ c i β T x ^ i − ln ( 1 + e β T x ^ i ) ]
\qquad β = [ w T , b ] T \boldsymbol{\beta}=[\boldsymbol{w}^{T},b]^{T} β = [ w T , b ] T
\qquad l ( β ) = − ln L ( w , b ) = − ln L ( β ) l(\boldsymbol{\beta})=-\ln L(\boldsymbol{w},b)=-\ln L(\boldsymbol{\beta}) l ( β ) = − ln L ( w , b ) = − ln L ( β ) l ( β ) l(\boldsymbol{\beta}) l ( β )
l ( β ) = − ln L ( β ) = − ∑ i = 1 N [ c i β T x ^ i − ln ( 1 + e β T x ^ i ) ] \qquad\qquad l(\boldsymbol{\beta})=-\ln L(\boldsymbol{\beta})=-\displaystyle\sum_{i=1}^N \left[ c_{i}\boldsymbol{\beta}^{T}\hat{\boldsymbol{x}}_{i}-\ln\left( 1+e^{\boldsymbol{\beta}^{T}\hat{\boldsymbol{x}}_{i}} \right) \right] l ( β ) = − ln L ( β ) = − i = 1 ∑ N [ c i β T x ^ i − ln ( 1 + e β T x ^ i ) ]
\qquad l ( β ) l(\boldsymbol{\beta}) l ( β ) β \boldsymbol{\beta} β β \boldsymbol{\beta} β \qquad
\qquad
∂ l ( β ) ∂ β = − ∑ i = 1 N ( c i x ^ i − 1 1 + e β T x ^ i e β T x ^ i x ^ i ) = − ∑ i = 1 N ( c i − e β T x ^ i 1 + e β T x ^ i ) x ^ i = − ∑ i = 1 N [ c i − y ( x i ) ] x ^ i ( 1 ) \qquad\qquad \begin{aligned} \dfrac{\partial l(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}&=-\displaystyle\sum_{i=1}^N \left( c_{i}\hat{\boldsymbol{x}}_{i}-\dfrac{1}{1+e^{\boldsymbol{\beta}^{T}\hat{\boldsymbol{x}}_{i}}}e^{\boldsymbol{\beta}^{T}\hat{\boldsymbol{x}}_{i}}\hat{\boldsymbol{x}}_{i} \right)\\
&=-\displaystyle\sum_{i=1}^N \left( c_{i}-\dfrac{e^{\boldsymbol{\beta}^{T}\hat{\boldsymbol{x}}_{i}}}{1+e^{\boldsymbol{\beta}^{T}\hat{\boldsymbol{x}}_{i}}} \right)\hat{\boldsymbol{x}}_{i} \\
&=-\displaystyle\sum_{i=1}^N [ c_{i}-y(\boldsymbol{x}_{i}) ]\hat{\boldsymbol{x}}_{i} \qquad\qquad\qquad\ (1)\\
\end{aligned} ∂ β ∂ l ( β ) = − i = 1 ∑ N ( c i x ^ i − 1 + e β T x ^ i 1 e β T x ^ i x ^ i ) = − i = 1 ∑ N ( c i − 1 + e β T x ^ i e β T x ^ i ) x ^ i = − i = 1 ∑ N [ c i − y ( x i ) ] x ^ i ( 1 )
\qquad β = [ w T , b ] T = [ w 1 , ⋯ , w n , b ] T \boldsymbol{\beta}=[\boldsymbol{w}^T,b]^{T}=[w_1,\cdots,w_n,b]^T β = [ w T , b ] T = [ w 1 , ⋯ , w n , b ] T x ^ i = [ x i T , 1 ] T = [ x i ( 1 ) , ⋯ , x i ( n ) , 1 ] T \hat\boldsymbol{x}_{i}=[\boldsymbol{x}_{i}^T,1]^T=[x_i^{(1)},\cdots,x_i^{(n)},1]^T x ^ i = [ x i T , 1 ] T = [ x i ( 1 ) , ⋯ , x i ( n ) , 1 ] T
{ ∂ l ( β ) ∂ w = − ∑ i = 1 N [ c i − y ( x i ) ] x i ( 2 ) ∂ l ( β ) ∂ b = − ∑ i = 1 N [ c i − y ( x i ) ] ( 3 ) \qquad\qquad\begin{cases}\ \ \ \dfrac{\partial l(\boldsymbol{\beta})}{\partial \boldsymbol w}=-\displaystyle\sum_{i=1}^N [ c_{i}-y\left( \boldsymbol{x}_{i}\right) ]\boldsymbol{x}_{i}\qquad\qquad\ \ (2) \\
\\
\ \ \ \dfrac{\partial l(\boldsymbol{\beta})}{\partial b}=-\displaystyle\sum_{i=1}^N\left[ c_{i}-y\left( \boldsymbol{x}_{i}\right) \right] \qquad\qquad(3) \end{cases} ⎩ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎧ ∂ w ∂ l ( β ) = − i = 1 ∑ N [ c i − y ( x i ) ] x i ( 2 ) ∂ b ∂ l ( β ) = − i = 1 ∑ N [ c i − y ( x i ) ] ( 3 )
\qquad x i = [ x i ( 1 ) , ⋯ , x i ( n ) ] T \boldsymbol{x}_{i}=[x_i^{(1)},\cdots,x_i^{(n)}]^T x i = [ x i ( 1 ) , ⋯ , x i ( n ) ] T x i ( j ) x_{i}^{(j)} x i ( j )
∂ l ( β ) ∂ w j = − ∑ i = 1 N [ c i − y ( x i ) ] x i ( j ) \qquad\qquad \dfrac{\partial l(\boldsymbol{\beta})}{\partial w_{j}}=-\displaystyle\sum_{i=1}^N [ c_{i}-y\left( \boldsymbol{x}_{i}\right) ]x_{i}^{(j)} ∂ w j ∂ l ( β ) = − i = 1 ∑ N [ c i − y ( x i ) ] x i ( j )
\qquad
∂ l ( β ) ∂ β = [ ∂ l ( β ) ∂ w 1 , ⋯ , ∂ l ( β ) ∂ w n , ∂ l ( β ) ∂ b ] T \qquad\qquad \dfrac{\partial l(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=\left[\dfrac{\partial l(\boldsymbol{\beta})}{\partial w_1},\cdots,\dfrac{\partial l(\boldsymbol{\beta})}{\partial w_n},\dfrac{\partial l(\boldsymbol{\beta})}{\partial b} \right]^{T} ∂ β ∂ l ( β ) = [ ∂ w 1 ∂ l ( β ) , ⋯ , ∂ w n ∂ l ( β ) , ∂ b ∂ l ( β ) ] T
\qquad α \alpha α
β t + 1 = β t − α ∂ l ( β ) ∂ β \qquad\qquad \boldsymbol{\beta}^{t+1}=\boldsymbol{\beta}^{t}-\alpha\dfrac{\partial l(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} β t + 1 = β t − α ∂ β ∂ l ( β ) \qquad
\qquad 0 0 0 h e s s i a n hessian h e s s i a n
\qquad ∂ l ( β ) ∂ β = − ∑ i = 1 N [ c i − y ( x i ) ] x ^ i \ \dfrac{\partial l(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-\displaystyle\sum_{i=1}^N [ c_{i}-y(\boldsymbol{x}_{i}) ]\hat{\boldsymbol{x}}_{i} ∂ β ∂ l ( β ) = − i = 1 ∑ N [ c i − y ( x i ) ] x ^ i
\qquad h e s s i a n hessian h e s s i a n
∂ ∂ β T ( ∂ l ( β ) ∂ β ) = ∂ ∂ β T ( − ∑ i = 1 N [ c i − y ( x i ) ] x ^ i ) = ∂ ∂ β T ( ∑ i = 1 N y ( x i ) x ^ i ) = ∑ i = 1 N y ( x i ) [ 1 − y ( x i ) ] x ^ i ∂ ∂ β T ( β T x ^ i ) = ∑ i = 1 N y ( x i ) [ 1 − y ( x i ) ] x ^ i x ^ i T \qquad\qquad \begin{aligned} \dfrac{\partial}{\partial \boldsymbol\beta^T}\left(\dfrac{\partial l(\boldsymbol{\beta})}{\partial \boldsymbol\beta}\right)
&=\dfrac{\partial}{\partial \boldsymbol\beta^T}\left(-\displaystyle\sum_{i=1}^N [ c_{i}-y(\boldsymbol{x}_{i}) ]\hat{\boldsymbol{x}}_{i}\right)\\
&=\dfrac{\partial}{\partial \boldsymbol\beta^T}\left(\displaystyle\sum_{i=1}^Ny(\boldsymbol{x}_{i})\hat\boldsymbol{x}_i\right)\\
&=\displaystyle\sum_{i=1}^Ny(\boldsymbol{x}_{i})[1-y(\boldsymbol{x}_{i})]\hat\boldsymbol{x}_i\dfrac{\partial}{\partial \boldsymbol\beta^T}\left(\boldsymbol{\beta}^{T}\hat{\boldsymbol{x}}_{i}\right)\\
&=\displaystyle\sum_{i=1}^Ny(\boldsymbol{x}_{i})[1-y(\boldsymbol{x}_{i})]\hat\boldsymbol{x}_i\hat{\boldsymbol{x}}_{i}^T\\
\end{aligned} ∂ β T ∂ ( ∂ β ∂ l ( β ) ) = ∂ β T ∂ ( − i = 1 ∑ N [ c i − y ( x i ) ] x ^ i ) = ∂ β T ∂ ( i = 1 ∑ N y ( x i ) x ^ i ) = i = 1 ∑ N y ( x i ) [ 1 − y ( x i ) ] x ^ i ∂ β T ∂ ( β T x ^ i ) = i = 1 ∑ N y ( x i ) [ 1 − y ( x i ) ] x ^ i x ^ i T
\qquad
β t + 1 = β t − ( ∂ 2 l ( β ) ∂ β ∂ β T ) − 1 ∂ l ( β ) ∂ β \qquad\qquad \boldsymbol{\beta}^{t+1}=\boldsymbol{\beta}^{t}-\left(\dfrac{\partial^2 l(\boldsymbol\beta)}{\partial \boldsymbol\beta\partial \beta^T}\right)^{-1}\dfrac{\partial l(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} β t + 1 = β t − ( ∂ β ∂ β T ∂ 2 l ( β ) ) − 1 ∂ β ∂ l ( β )
\qquad
\qquad
1 ) \qquad1) 1 ) β = [ w T , b ] T \boldsymbol{\beta}=[\boldsymbol{w}^T,b]^{T} β = [ w T , b ] T β 0 \boldsymbol{\beta}^{0} β 0
2 ) \qquad2) 2 ) α \alpha α
∂ l ( β ) ∂ β = − ∑ i = 1 N [ c i − y ( x i ) ] x ^ i = − ∑ i = 1 N [ c i − y ( x i ) ] [ x i 1 ] \qquad\qquad \begin{aligned} \dfrac{\partial l(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}
&=-\displaystyle\sum_{i=1}^N [ c_{i}-y(\boldsymbol{x}_{i}) ]\hat{\boldsymbol{x}}_{i}\\
&=-\displaystyle\sum_{i=1}^N [ c_{i}-y(\boldsymbol{x}_{i}) ]\left[\begin{matrix}\boldsymbol{x}_{i}\\ 1\end{matrix}\right]\\
\end{aligned} ∂ β ∂ l ( β ) = − i = 1 ∑ N [ c i − y ( x i ) ] x ^ i = − i = 1 ∑ N [ c i − y ( x i ) ] [ x i 1 ]
β t + 1 = β t − α ∂ l ( β ) ∂ β \qquad\qquad \boldsymbol{\beta}^{t+1}=\boldsymbol{\beta}^{t}-\alpha\dfrac{\partial l(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} β t + 1 = β t − α ∂ β ∂ l ( β ) \qquad
1) 定義Sigmoid函數
def sigmoid ( x) :
'''Sigmoid function
'''
return 1.0 / ( 1 + np. exp( - x) )
2) 讀取訓練/測試集數據函數
假設在二維平面R 2 R^{2} R 2 ( x i , y i ) = ( x i ( 1 ) , x i ( 2 ) , y i ) , y i ∈ { 0 , 1 } (\boldsymbol{x}_{i},y_{i})=(x_{i}^{(1)},x_{i}^{(2)},y_{i}), \ y_{i}\in\{0,1\} ( x i , y i ) = ( x i ( 1 ) , x i ( 2 ) , y i ) , y i ∈ { 0 , 1 }
3.562302 , 25.329208 , 1.000000 − 24.268267 , 1.272092 , 1.000000 25.405790 , 8.463017 , 1.000000 − 6.908775 , 23.298889 , 1.000000 40.621010 , − 25.134052 , 0.000000 − 9.305521 , 14.983097 , 1.000000 20.041330 , − 25.381725 , 0.000000 37.298540 , − 26.767307 , 0.000000 35.856177 , − 31.080316 , 0.000000 − 17.976889 , 4.244106 , 1.000000 ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ 3.562302,25.329208,1.000000\newline
-24.268267,1.272092,1.000000\newline
25.405790,8.463017,1.000000\newline
-6.908775,23.298889,1.000000\newline
40.621010,-25.134052,0.000000\newline
-9.305521,14.983097,1.000000\newline
20.041330,-25.381725,0.000000\newline
37.298540,-26.767307,0.000000\newline
35.856177,-31.080316,0.000000\newline
-17.976889,4.244106,1.000000\newline
\cdot\cdot\cdot\cdot\cdot\cdot 3 . 5 6 2 3 0 2 , 2 5 . 3 2 9 2 0 8 , 1 . 0 0 0 0 0 0 − 2 4 . 2 6 8 2 6 7 , 1 . 2 7 2 0 9 2 , 1 . 0 0 0 0 0 0 2 5 . 4 0 5 7 9 0 , 8 . 4 6 3 0 1 7 , 1 . 0 0 0 0 0 0 − 6 . 9 0 8 7 7 5 , 2 3 . 2 9 8 8 8 9 , 1 . 0 0 0 0 0 0 4 0 . 6 2 1 0 1 0 , − 2 5 . 1 3 4 0 5 2 , 0 . 0 0 0 0 0 0 − 9 . 3 0 5 5 2 1 , 1 4 . 9 8 3 0 9 7 , 1 . 0 0 0 0 0 0 2 0 . 0 4 1 3 3 0 , − 2 5 . 3 8 1 7 2 5 , 0 . 0 0 0 0 0 0 3 7 . 2 9 8 5 4 0 , − 2 6 . 7 6 7 3 0 7 , 0 . 0 0 0 0 0 0 3 5 . 8 5 6 1 7 7 , − 3 1 . 0 8 0 3 1 6 , 0 . 0 0 0 0 0 0 − 1 7 . 9 7 6 8 8 9 , 4 . 2 4 4 1 0 6 , 1 . 0 0 0 0 0 0 ⋅ ⋅ ⋅ ⋅ ⋅ ⋅
生成具有兩個中心的二維高斯分佈的數據集：
def gen_gausssian ( mean1, mean2, cov1, cov2, num) :
'''generate 2-d gaussian dataset with 2 clusters
'''
data1 = np. random. multivariate_normal( mean1, cov1, num)
label1 = np. ones( ( 1 , num) ) . T
data_pos = np. append( data1, label1, axis= 1 )
data2 = np. random. multivariate_normal( mean2, cov2, num)
label2 = np. zeros( ( 1 , num) ) . T
data_neg = np. append( data2, label2, axis= 1 )
data = np. append( data_pos, data_neg, axis= 0 )
shuffle_data = np. random. permutation( data)
x1, y1 = data1. T
x2, y2 = data2. T
plt. scatter( x1, y1, c= 'r' , s= 3 )
plt. plot( mean1[ 0 ] , mean1[ 1 ] , 'ko' )
plt. scatter( x2, y2, c= 'b' , s= 3 )
plt. plot( mean2[ 0 ] , mean2[ 1 ] , 'ko' )
plt. axis( )
plt. title( "2-d gaussian dataset with 2 clusters" )
plt. xlabel( "x" )
plt. ylabel( "y" )
plt. show( )
np. savetxt( 'gaussdata.txt' , shuffle_data, fmt= '%f' , delimiter= ',' )
return shuffle_data, data_pos, data_neg
用散點圖表示：( x i , y i ) = ( x i ( 1 ) , x i ( 2 ) , y i ) , y i ∈ { 0 , 1 } (\boldsymbol{x}_{i},y_{i})=(x_{i}^{(1)},x_{i}^{(2)},y_{i}), \ y_{i}\in\{0,1\} ( x i , y i ) = ( x i ( 1 ) , x i ( 2 ) , y i ) , y i ∈ { 0 , 1 }
def load_data ( filename) :
'''Load data of training or testing set
'''
tdata = [ ]
with open ( filename) as f:
while True :
line = f. readline( )
if not line:
break
line = line. split( ',' )
tdata. append( [ float ( item) for item in line] )
f. close( )
return np. array( tdata)
3) 迭代計算公式 ( 2 ) (2) ( 2 )
def lr_train ( xhat, c, alpha, num) :
beta = np. random. rand( 3 , 1 )
for i in range ( num) :
yx = sigmoid( np. dot( xhat, beta) )
beta = beta + alpha * np. dot( xhat. T, ( c - yx) )
print ( '#' + str ( i) + ',training loss:' + str ( train_loss( c, yx) ) )
return beta
由公式 − ln L ( w , b ) = − ∑ i = 1 N { c i ln [ y ( x i ) ] + ( 1 − c i ) ln [ 1 − y ( x i ) ] } \ -\ln L(\boldsymbol{w},b)=-\displaystyle\sum_{i=1}^N \{ c_{i}\ln\left[
y\left( \boldsymbol{x}_{i}\right) \right] +(1-c_{i})\ln\left[ 1-y\left(
\boldsymbol{x}_{i}\right) \right] \} − ln L ( w , b ) = − i = 1 ∑ N { c i ln [ y ( x i ) ] + ( 1 − c i ) ln [ 1 − y ( x i ) ] }
計算損失函數（誤差值） ：
def train_loss ( c, yx) :
err = 0.0
for i in range ( len ( yx) ) :
if yx[ i, 0 ] > 0 and ( 1 - yx[ i, 0 ] ) > 0 :
err -= c[ i, 0 ] * np. log( yx[ i, 0 ] ) + ( 1 - c[ i, 0 ] ) * np. log( 1 - yx[ i, 0 ] )
return err
主程序：
mean1 = [ 3 , - 1 ]
cov1 = [ [ 5 , 0 ] , [ 0 , 10 ] ]
mean2 = [ - 5 , 7 ]
cov2 = [ [ 10 , 0 ] , [ 0 , 5 ] ]
data, data_pos, data_neg = gen_gausssian( mean1, mean2, cov1, cov2, 1100 )
training_data = data
tmp1 = training_data[ 0 : 2000 , 0 : 2 ]
tmp2 = np. ones( ( 2000 , 1 ) )
xhat = np. concatenate( ( tmp1, tmp2) , axis= 1 )
target = training_data[ 0 : 2000 , 2 : ]
beta = lr_train( xhat, target, 0.01 , 100 )
print ( 'beta:\n' , beta)
tmp1 = training_data[ 2000 : 2200 , 0 : 2 ]
tmp2 = np. ones( ( 200 , 1 ) )
testing_data = np. concatenate( ( tmp1, tmp2) , axis= 1 )
target = training_data[ 2000 : 2200 , 2 : ]
y1 = classification( testing_data, beta)
print ( np. abs ( y1- target) . T)
對一個大小爲2000的數據進行訓練，可得到輸出結果爲：
beta:
這裏的beta值就是用梯度下降法所求得β = [ w T , b ] T \boldsymbol{\beta}=[\boldsymbol{w}^{T},b]^{T} β = [ w T , b ] T
def classification ( testing_data, beta) :
y = sigmoid( np. dot( testing_data, beta) )
for i in range ( len ( y) ) :
if y[ i, 0 ] < 0.5 :
y[ i, 0 ] = 0.0
else :
y[ i, 0 ] = 1.0
return y
判別結果：200個測試數據，有2個數據被錯誤分類
