python 面向對象實現CNN（四）

基礎的理論知識參考：https://www.zybuluo.com/hanbingtao/note/485480

下面的代碼也是基於上面文章的實現：
整個算法分爲三個步驟：

前向計算每個神經元的輸出值 $a_{j}$ （ $j$ 表示網絡的第 $j$ 個神經元，以下同）；
反向計算每個神經元的誤差項 $δ_{j}$ ， $δ_{j}$ 在有的文獻中也叫做敏感度(sensitivity)。它實際上是 $E_{d}$ 網絡的損失函數對神經元 $n e t_{j}$ 加權輸入的偏導數，即 $δ_{j} = \frac{\partial E_{d}}{\partial n e t_{j}}$ ；
計算每個神經元連接權重 $w_{j i}$ 的梯度（ $w_{j i}$ 表示從神經元連接到神經元 $j$ 的權重），公式爲 $\frac{\partial E_{d}}{\partial w_{j i}} = a_{i} δ_{j}$ ，其中 $a_{i}$ ，表示神經元 $i$ 的輸出。

最後，根據梯度下降法則更新每個權重即可。

具體的細節參考上面的連接文章，這裏只貼出代碼實現：

首先是activators.py文件：

#!/usr/bin/env python
# -*- coding: UTF-8 -*-

import numpy as np


class ReluActivator(object):
    def forward(self, weighted_input):
        #return weighted_input
        return max(0, weighted_input)

    def backward(self, output):
        return 1 if output > 0 else 0


class IdentityActivator(object):
    def forward(self, weighted_input):
        return weighted_input

    def backward(self, output):
        return 1


class SigmoidActivator(object):
    def forward(self, weighted_input):
        return 1.0 / (1.0 + np.exp(-weighted_input))

    def backward(self, output):
        return output * (1 - output)


class TanhActivator(object):
    def forward(self, weighted_input):
        return 2.0 / (1.0 + np.exp(-2 * weighted_input)) - 1.0

    def backward(self, output):
        return 1 - output * output

是一些基本的激活函數的實現

下面的CNN.py文件實現cnn網絡主要的組件

#!/usr/bin/env python
# -*- coding: UTF-8 -*-

import numpy as np
from activators import ReluActivator, IdentityActivator


# 獲取卷積區域
def get_patch(input_array, i, j, filter_width,filter_height, stride):
    '''
    從輸入數組中獲取本次卷積的區域，
    自動適配輸入爲2D和3D的情況
    '''
    start_i = i * stride
    start_j = j * stride
    if input_array.ndim == 2:
        return input_array[ 
            start_i : start_i + filter_height, 
            start_j : start_j + filter_width]
    elif input_array.ndim == 3:
        return input_array[:, 
            start_i : start_i + filter_height, 
            start_j : start_j + filter_width]


# 獲取一個2D區域的最大值所在的索引
def get_max_index(array):
    max_i = 0
    max_j = 0
    max_value = array[0,0]
    for i in range(array.shape[0]):
        for j in range(array.shape[1]):
            if array[i,j] > max_value:
                max_value = array[i,j]
                max_i, max_j = i, j
    return max_i, max_j


# 計算卷積:conv函數實現了2維和3維數組的卷積
def conv(input_array,kernel_array,output_array,stride, bias):
    '''
    計算卷積，自動適配輸入爲2D和3D的情況,是在get_patch函數中判斷的
    '''
    #print 'shape 1:',np.shape(input_array)
    #print 'shape 2:',np.shape(kernel_array)
    #print 'shape 3:',np.shape(output_array)
    channel_number = input_array.ndim 
    output_width = output_array.shape[1]
    output_height = output_array.shape[0]
    kernel_width = kernel_array.shape[-1]
    kernel_height = kernel_array.shape[-2]
    for i in range(output_height):
        for j in range(output_width):
            # 這裏的*是np.array*np.array的對應元素相乘
            #print 'get_patch:\n',get_patch(input_array, i, j, kernel_width,kernel_height, stride)
            #print 'kernel_array:\n',kernel_array
            output_array[i][j] = (get_patch(input_array, i, j, kernel_width, 
                                  kernel_height, stride) * kernel_array).sum() + bias

# padding函數實現了zero padding操作
def padding(input_array, zp): 
    '''
    爲數組增加Zero padding，自動適配輸入爲2D和3D的情況
    '''
    if zp == 0:
        return input_array
    else:
        # 輸入爲3D時
        if input_array.ndim == 3:
            input_width = input_array.shape[2]
            input_height = input_array.shape[1]
            input_depth = input_array.shape[0]
            padded_array = np.zeros((input_depth, 
                                     input_height + 2 * zp,
                                     input_width + 2 * zp))
            padded_array[:,zp : zp + input_height,zp : zp + input_width] = input_array
            return padded_array
        # # 輸入爲2D時
        elif input_array.ndim == 2:
            input_width = input_array.shape[1]
            input_height = input_array.shape[0]
            padded_array = np.zeros((input_height + 2 * zp,input_width + 2 * zp))
            # 二維數組直接賦值
            padded_array[zp : zp + input_height,zp : zp + input_width] = input_array 
            return padded_array


# 對numpy數組進行element wise操作
# element_wise_op函數實現了對numpy數組進行按元素操作，並將返回值寫回到數組中
def element_wise_op(array, op):
    for i in np.nditer(array,op_flags=['readwrite']):
        i[...] = op(i)


# Filter類保存了卷積層的參數以及梯度，並且實現了用梯度下降算法來更新參數
class Filter(object):
    def __init__(self, width, height, depth):
        self.weights = np.random.uniform(-1e-4, 1e-4,(depth, height, width))
        self.bias = 0
        self.weights_grad = np.zeros(self.weights.shape)
        self.bias_grad = 0

    def __repr__(self):
        return 'filter weights:\n%s\nbias:\n%s' % (
            repr(self.weights), repr(self.bias))

    def get_weights(self):
        return self.weights

    def get_bias(self):
        return self.bias

    def update(self, learning_rate):
        self.weights -= learning_rate * self.weights_grad
        self.bias -= learning_rate * self.bias_grad

# 用ConvLayer類來實現一個卷積層
class ConvLayer(object):
    # 初始化
    def __init__(self, input_width, input_height, 
                 channel_number, filter_width, 
                 filter_height, filter_number, 
                 zero_padding, stride, activator,
                 learning_rate):
        self.input_width = input_width
        self.input_height = input_height
        self.channel_number = channel_number
        self.filter_width = filter_width
        self.filter_height = filter_height
        self.filter_number = filter_number
        self.zero_padding = zero_padding
        self.stride = stride
        self.activator = activator
        self.learning_rate = learning_rate
        # 卷積後的Feature Map的高度和寬度
        self.output_width = ConvLayer.calculate_output_size(
                            self.input_width, filter_width, zero_padding,stride)
        self.output_height = ConvLayer.calculate_output_size(
                            self.input_height, filter_height, zero_padding,stride)
        # 把輸出的feature map用列表存起來
        self.output_array = np.zeros((self.filter_number, 
                                      self.output_height, self.output_width))
        # filters的每個元素是過濾器對象
        self.filters = []
        for i in range(filter_number):
            self.filters.append(Filter(filter_width,filter_height, self.channel_number))

    # 用來確定卷積層輸出的大小
    @staticmethod
    def calculate_output_size(input_size,filter_size, zero_padding, stride):
        return (input_size - filter_size + 2 * zero_padding) / stride + 1

    # forward方法實現了卷積層的前向計算                     
    def forward(self, input_array):
        '''
        計算卷積層的輸出
        輸出結果保存在self.output_array
        '''
        self.input_array = input_array
        # 爲數組增加Zero padding
        self.padded_input_array = padding(input_array,self.zero_padding)
        for f in range(self.filter_number):
            filter = self.filters[f]
            #print 'shape of filter:',np.shape(filter.get_weights())
            conv(self.padded_input_array, filter.get_weights(), self.output_array[f],
                                                         self.stride, filter.get_bias())
        element_wise_op(self.output_array,self.activator.forward)

    def backward(self, input_array, sensitivity_array, activator):
        '''
        計算傳遞給前一層的誤差項，以及計算每個權重的梯度
        前一層的誤差項保存在:self.delta_array
        梯度保存在:Filter對象的weights_grad
        '''
        self.forward(input_array)
        self.bp_sensitivity_map(sensitivity_array,activator)
        self.bp_gradient(sensitivity_array)

    def update(self):
        '''
        按照梯度下降，更新權重
        '''
        for filter in self.filters:
            filter.update(self.learning_rate)

    def bp_sensitivity_map(self, sensitivity_array,activator):
        '''
        計算傳遞到上一層的sensitivity map
        sensitivity_array: 本層的sensitivity map
        activator: 上一層的激活函數
        '''
        # 處理卷積步長，對原始sensitivity map進行擴展
        expanded_array = self.expand_sensitivity_map(sensitivity_array)
        # full卷積，對sensitivitiy map進行zero padding
        # 雖然原始輸入的zero padding單元也會獲得殘差
        # 但這個殘差不需要繼續向上傳遞，因此就不計算了
        expanded_width = expanded_array.shape[2]
        zp = (self.input_width + self.filter_width - 1 - expanded_width) / 2
        #print 'zp:',zp
        # 對誤差圖進行擴展後再進行0填充
        padded_array = padding(expanded_array, zp)
        print 'padded_array:',np.shape(padded_array)
        # 初始化delta_array，用於保存傳遞到上一層的sensitivity map
        self.delta_array = self.create_delta_array()
        # 對於具有多個filter的卷積層來說，最終傳遞到上一層的
        # sensitivity map相當於所有的filter的sensitivity map之和
        for f in range(self.filter_number):
            filter = self.filters[f]
            # 將filter權重翻轉180度
            flipped_weights = np.array(map(lambda i: np.rot90(i, 2),filter.get_weights()))
            print 'flipped_weights:',np.shape(flipped_weights)
            # 計算與一個filter對應的delta_array
            delta_array = self.create_delta_array()
            for d in range(delta_array.shape[0]):
                # input_array,kernel_array,output_array,stride, bias
                conv(padded_array[f], flipped_weights[d],delta_array[d], 1, 0)
            self.delta_array += delta_array
        # 將計算結果與激活函數的偏導數做element-wise乘法操作
        derivative_array = np.array(self.input_array)
        element_wise_op(derivative_array,activator.backward)
        self.delta_array *= derivative_array

    def bp_gradient(self, sensitivity_array):
        # 處理卷積步長，對原始sensitivity map進行擴展
        expanded_array = self.expand_sensitivity_map(sensitivity_array)
        for f in range(self.filter_number):
            # 計算每個權重的梯度
            filter = self.filters[f]
            for d in range(filter.weights.shape[0]):
                conv(self.padded_input_array[d],expanded_array[f],
                     filter.weights_grad[d], 1, 0)
            # 計算偏置項的梯度
            filter.bias_grad = expanded_array[f].sum()

    def expand_sensitivity_map(self, sensitivity_array):
        print 'sensitivity_array:\n',sensitivity_array
        depth = sensitivity_array.shape[0]
        # 確定擴展後sensitivity map的大小,計算stride爲1時sensitivity map的大小
        expanded_width = (self.input_width - self.filter_width + 2 * self.zero_padding + 1)
        expanded_height = (self.input_height - self.filter_height + 2 * self.zero_padding + 1)
        # 構建新的sensitivity_map
        expand_array = np.zeros((depth, expanded_height, expanded_width))
        # 從原始sensitivity map拷貝誤差值
        for i in range(self.output_height):
            for j in range(self.output_width):
                i_pos = i * self.stride
                j_pos = j * self.stride
                expand_array[:,i_pos,j_pos] = sensitivity_array[:,i,j]
        print 'expand_array:\n',expand_array
        return expand_array

    def create_delta_array(self):
        return np.zeros((self.channel_number,self.input_height, self.input_width))

# 池化層
class MaxPoolingLayer(object):
    def __init__(self, input_width, input_height, 
                 channel_number, filter_width, 
                 filter_height, stride):
        self.input_width = input_width
        self.input_height = input_height
        self.channel_number = channel_number
        self.filter_width = filter_width
        self.filter_height = filter_height
        self.stride = stride
        self.output_width = (input_width - filter_width) / self.stride + 1
        self.output_height = (input_height -filter_height) / self.stride + 1
        self.output_array = np.zeros((self.channel_number,
                            self.output_height, self.output_width))

    def forward(self, input_array):
        for d in range(self.channel_number):
            for i in range(self.output_height):
                for j in range(self.output_width):
                    self.output_array[d,i,j] = ( get_patch(input_array[d], i, j,
                                self.filter_width, self.filter_height, self.stride).max())

    def backward(self, input_array, sensitivity_array):
        self.delta_array = np.zeros(input_array.shape)
        for d in range(self.channel_number):
            for i in range(self.output_height):
                for j in range(self.output_width):
                    patch_array = get_patch(
                        input_array[d], i, j,
                        self.filter_width, 
                        self.filter_height, 
                        self.stride)
                    k, l = get_max_index(patch_array)
                    self.delta_array[d, 
                        i * self.stride + k, 
                        j * self.stride + l] = \
                        sensitivity_array[d,i,j]

##.............................卷積層的一些測試.......................................
# 卷積層前向傳播數據初始化
def init_test():
    a = np.array(
        [[[0,1,1,0,2],
          [2,2,2,2,1],
          [1,0,0,2,0],
          [0,1,1,0,0],
          [1,2,0,0,2]],
         [[1,0,2,2,0],
          [0,0,0,2,0],
          [1,2,1,2,1],
          [1,0,0,0,0],
          [1,2,1,1,1]],
         [[2,1,2,0,0],
          [1,0,0,1,0],
          [0,2,1,0,1],
          [0,1,2,2,2],
          [2,1,0,0,1]]])
    # 假設誤差項矩陣已經算好                    
    b = np.array(
        [[[0,1,1],
          [2,2,2],
          [1,0,0]],
         [[1,0,2],
          [0,0,0],
          [1,2,1]]])
    # input_width, input_height, channel_number, filter_width, filter_height, 
    # filter_number, zero_padding, stride, activator,learning_rate                    
    cl = ConvLayer(5,5,3,3,3,  2,1,2,IdentityActivator(),0.001)
    cl.filters[0].weights = np.array(
        [[[-1,1,0],
          [0,1,0],
          [0,1,1]],
         [[-1,-1,0],
          [0,0,0],
          [0,-1,0]],
         [[0,0,-1],
          [0,1,0],
          [1,-1,-1]]], dtype=np.float64)
    cl.filters[0].bias=1
    cl.filters[1].weights = np.array(
        [[[1,1,-1],
          [-1,-1,1],
          [0,-1,1]],
         [[0,1,0],
         [-1,0,-1],
          [-1,1,0]],
         [[-1,0,0],
          [-1,0,1],
          [-1,0,0]]], dtype=np.float64)
    cl.filters[1].bias=1            
    return a, b, cl

# 卷積層前向傳播測試
def test():
    a, b, cl = init_test()
    cl.forward(a)
    print 'cl.output_array:\n',cl.output_array

# 卷積層的反向傳播測試
def test_bp():
    a, b, cl = init_test()
    cl.backward(a, b, IdentityActivator())
    cl.update()
    print 'cl.filters[0]:\n',cl.filters[0]
    print 'cl.filters[1]:\n',cl.filters[1]

#.............................池化層的一些測試.......................................
# 池化層測試數據初始化
def init_pool_test():
    a = np.array(
        [[[1,1,2,4],
          [5,6,7,8],
          [3,2,1,0],
          [1,2,3,4]],
         [[0,1,2,3],
          [4,5,6,7],
          [8,9,0,1],
          [3,4,5,6]]], dtype=np.float64)

    b = np.array(
        [[[1,2],
          [2,4]],
         [[3,5],
          [8,2]]], dtype=np.float64)
    # input_width, input_height, channel_number, filter_width, filter_height, stride                    
    mpl = MaxPoolingLayer(4,4,2,2,2,2)
    return a, b, mpl

# 池化層測試
def test_pool():
    a, b, mpl = init_pool_test()
    mpl.forward(a)
    print 'input array:\n%s\noutput array:\n%s' % (a,mpl.output_array)


def test_pool_bp():
    a, b, mpl = init_pool_test()
    mpl.backward(a, b)
    print 'input array:\n%s\nsensitivity array:\n%s\ndelta array:\n%s' % (
            a, b, mpl.delta_array)

if __name__ == '__main__':
    test()
    test_pool()
    test_bp()
    print '................................................'
    test_pool_bp()



    # 測試np.nditer
    '''
    a = np.arange(6).reshape(2, 3)  
    print a  
    for x in np.nditer(a, op_flags = ['readwrite']):  
        x[...] = 2*x   
    print a  
    '''

一些基本得運行結果：

cl.output_array:
[[[ 6.  7.  5.]
  [ 3. -1. -1.]
  [ 2. -1.  4.]]

 [[ 3. -4. -7.]
  [ 2. -3. -3.]
  [ 1. -4. -4.]]]
input array:
[[[ 1.  1.  2.  4.]
  [ 5.  6.  7.  8.]
  [ 3.  2.  1.  0.]
  [ 1.  2.  3.  4.]]

 [[ 0.  1.  2.  3.]
  [ 4.  5.  6.  7.]
  [ 8.  9.  0.  1.]
  [ 3.  4.  5.  6.]]]
output array:
[[[ 6.  8.]
  [ 3.  4.]]

 [[ 5.  7.]
  [ 9.  6.]]]
sensitivity_array:
[[[0 1 1]
  [2 2 2]
  [1 0 0]]

 [[1 0 2]
  [0 0 0]
  [1 2 1]]]
expand_array:
[[[ 0.  0.  1.  0.  1.]
  [ 0.  0.  0.  0.  0.]
  [ 2.  0.  2.  0.  2.]
  [ 0.  0.  0.  0.  0.]
  [ 1.  0.  0.  0.  0.]]

 [[ 1.  0.  0.  0.  2.]
  [ 0.  0.  0.  0.  0.]
  [ 0.  0.  0.  0.  0.]
  [ 0.  0.  0.  0.  0.]
  [ 1.  0.  2.  0.  1.]]]
padded_array: (2L, 7L, 7L)
flipped_weights: (3L, 3L, 3L)
flipped_weights: (3L, 3L, 3L)
sensitivity_array:
[[[0 1 1]
  [2 2 2]
  [1 0 0]]

 [[1 0 2]
  [0 0 0]
  [1 2 1]]]
expand_array:
[[[ 0.  0.  1.  0.  1.]
  [ 0.  0.  0.  0.  0.]
  [ 2.  0.  2.  0.  2.]
  [ 0.  0.  0.  0.  0.]
  [ 1.  0.  0.  0.  0.]]

 [[ 1.  0.  0.  0.  2.]
  [ 0.  0.  0.  0.  0.]
  [ 0.  0.  0.  0.  0.]
  [ 0.  0.  0.  0.  0.]
  [ 1.  0.  2.  0.  1.]]]
cl.filters[0]:
filter weights:
array([[[-1.008,  0.99 , -0.009],
        [-0.005,  0.994, -0.006],
        [-0.006,  0.995,  0.996]],

       [[-1.004, -1.001, -0.004],
        [-0.01 , -0.009, -0.012],
        [-0.002, -1.002, -0.002]],

       [[-0.002, -0.002, -1.003],
        [-0.005,  0.992, -0.005],
        [ 0.993, -1.008, -1.007]]])
bias:
0.99099999999999999
cl.filters[1]:
filter weights:
array([[[  9.98000000e-01,   9.98000000e-01,  -1.00100000e+00],
        [ -1.00400000e+00,  -1.00700000e+00,   9.97000000e-01],
        [ -4.00000000e-03,  -1.00400000e+00,   9.98000000e-01]],

       [[  0.00000000e+00,   9.99000000e-01,   0.00000000e+00],
        [ -1.00900000e+00,  -5.00000000e-03,  -1.00400000e+00],
        [ -1.00400000e+00,   1.00000000e+00,   0.00000000e+00]],

       [[ -1.00400000e+00,  -6.00000000e-03,  -5.00000000e-03],
        [ -1.00200000e+00,  -5.00000000e-03,   9.98000000e-01],
        [ -1.00200000e+00,  -1.00000000e-03,   0.00000000e+00]]])
bias:
0.99299999999999999
................................................
input array:
[[[ 1.  1.  2.  4.]
  [ 5.  6.  7.  8.]
  [ 3.  2.  1.  0.]
  [ 1.  2.  3.  4.]]

 [[ 0.  1.  2.  3.]
  [ 4.  5.  6.  7.]
  [ 8.  9.  0.  1.]
  [ 3.  4.  5.  6.]]]
sensitivity array:
[[[ 1.  2.]
  [ 2.  4.]]

 [[ 3.  5.]
  [ 8.  2.]]]
delta array:
[[[ 0.  0.  0.  0.]
  [ 0.  1.  0.  2.]
  [ 2.  0.  0.  0.]
  [ 0.  0.  0.  4.]]

 [[ 0.  0.  0.  0.]
  [ 0.  3.  0.  5.]
  [ 0.  8.  0.  0.]
  [ 0.  0.  0.  2.]]]

全連接層的實現和上一篇文章類似，在此就不再贅述了。至此，你已經擁有了實現了一個簡單的卷積神經網絡所需要的基本組件，並沒有完全實現一個CNN網絡。
對於卷積神經網絡，現在有很多優秀的開源實現，因此我們並不需要真的自己去實現一個。這裏貼出這些代碼能讓我們更深的理解卷積神經網絡的原理，僅供參考學習。

python 面向對象實現CNN（四）

《Python進階》學習筆記

Leetcode 3161. 物塊放置查詢

leetcode 60 排列序列

一個docker容器暴露多個端口

微服務實踐之使用 Visual Studio 2022 調試Dapr 應用程序

wpf附加屬性理解 WPF附加屬性

java 二叉樹的深度、平衡二叉樹、二叉樹的下一個結點

java 反轉鏈表、合併鏈表

java 對稱的二叉樹

java 調整數組順序使奇數位於偶數前面

java 鏈表中倒數第k個結點

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結