前言
網上有很多關於linux系統下,利用VGG16對圖片進行分類,但是運用到Windows下會出現很多問題,所以寫這篇博客做個總結。
關於配置caffe環境可以參考這篇博客win10+vs2013+caffe+gpu+python環境配置,在這裏就不在敘述了。下面將從以下5個步驟講起。
1.下載圖片,生成.txt標籤
2.把訓練圖片轉換成lmdb格式
3.計算圖片均值
4.修改網絡參數和網絡的圖片路徑等
5.訓練網絡
1 準備數據集
1.網上有很多數據集,可以下載這個數據集re.zip做爲參考,如果你要是訓練自己的數據集的話,可以模仿這個數據集製作,首先將你的數據集尺寸統一大小384*256(我有matlab代碼,想要的話可以私聊),然後以5:1比例分爲訓練集和測試集,訓練集命名爲train,測試集命名爲test,將數據拷貝到D:\caffe\caffe-master\data\re文件下(這是我caffe的路徑),如下圖所示。
2. 數據集準備好後,開始製作標籤。在D:/caffe/caffe-master/examples路徑下,新建文件夾myfile。
用python編寫代碼,直接運行即可
# -*- coding: utf-8 -*-
import os
data_path='D:/caffe/caffe-master/data/re/' #數據集路徑
my='D:/caffe/caffe-master/examples/myfile/' #文件輸出路徑
classes=[3,4,5,6,7]
def gen_txt(phase):
f=open(my+phase+'.txt','w')
for c in classes:
folder=str(c)
images=os.listdir(data_path+phase+'/')
for img in images:
f.write(phase+'/'+img+' '+folder+'\n')
gen_txt('train')
gen_txt('test')
程序運行後,會在你設置的路徑下,生成train.txt和test.txt文件,打開如下圖所示格式,表示你的標籤製作成功。
2. 將數據轉換成lmdb格式
這一步是最麻煩的,因爲網上很多教程都是基於linux系統的,要想運用到Windows下,在這我研究了好長時間。
- 新建txt文件,命名爲create_lmdb.sh,這裏的點是後綴,但是先彆着急把txt後綴去掉,因爲這樣寫代碼比較方便,然後寫入如下代碼
#!/usr/bin/env sh
MY=D:/caffe/caffe-master/examples/myfile #文件輸出路徑
echo "Create train lmdb.."
rm -rf $MY/img_train_lmdb
D:/caffe/caffe-master/Build/x64/Release/convert_imageset \ #caffe配置路徑
--shuffle \
--resize_height=256 \
--resize_width=256 \
D:/caffe/caffe-master/data/re/ \ #數據集路徑
$MY/train.txt \
$MY/img_train_lmdb
echo "Create test lmdb.."
rm -rf $MY/img_test_lmdb
D:/caffe/caffe-master/Build/x64/Release/convert_imageset \ #caffe配置路徑
--shuffle \
--resize_width=256 \
--resize_height=256 \
D:/caffe/caffe-master/data/re/ \ #數據集路徑
$MY/test.txt \ #是test,不是val
$MY/img_test_lmdb #是test,不是val
echo "All Done.."
注意以上修改4個路徑,改爲你電腦的路徑。改完後,把漢字註釋去掉,因爲會影響運行,然後保存,把txt後綴改成sh。如下圖所示
然後運行程序,就是這最麻煩,因爲sh是linux系統命令,所以在運行程序前,先使用電腦管家的軟件管理,輸入git然後下載安裝,一直點next就行,安裝結束後,在環境變量中添加C:\Program Files\Git\bin路徑。這裏就不在介紹如何添加了路徑了。
添加完後,在程序中打開cmd命令窗口,然後cd到D:\caffe\caffe-master\examples\myfile路徑下,輸入指令 sh create_lmdb.sh
如下圖所示
如果在你的路徑下生成如下兩個文件夾,表示圖像數據已經成功製作成lmdb格式了。到這可以鬆一口氣了,因爲最難得已經過去了,後面的就很簡單了。
3.計算圖片均值
圖片減去均值再訓練,會提高訓練速度和精度。因此,一般都會有這個操作。
在D:\caffe\caffe-master\examples\imagenet路徑下找到make_imagenet_mean.sh文件拷貝到D:\caffe\caffe-master\examples\myfile路徑下,如下圖所示,
然後用記事本打開文件,修改代碼
#!/usr/bin/env sh
# Compute the mean image from the imagenet training lmdb
# N.B. this is available in data/ilsvrc12
EXAMPLE=D:/caffe/caffe-master/examples/myfile
DATA=D:/caffe/caffe-master/examples/myfile
TOOLS=D:/caffe/caffe-master/Build/x64/Release
$TOOLS/compute_image_mean $EXAMPLE/img_train_lmdb \
$DATA/imagenet_mean.binaryproto
echo "Done."
EXAMPLE=D:/caffe/caffe-master/examples/myfile 爲lmdb文件目錄
DATA=D:/caffe/caffe-master/examples/myfile 生成文件所要存放的目錄
TOOLS=D:/caffe/caffe-master/Build/x64/Release caffe配置路徑
代碼修改後,在cmd命令窗口cd到D:\caffe\caffe-master\examples\myfile路徑下,然後輸入代碼 sh make_imagenet_mean.sh
如下圖所示
如果運行成功的話,會生成如下圖imagenet_mean.binaryproto文件
4.定義模型
在路徑D:\caffe\caffe-master\examples\myfile下新建vggnet_train_val.prototxt文件,用vs打開,複製以下代碼
name: "VGGNet"
layer {
name: "data"
type: "Data"
top: "data"
top: "label"
include {
phase: TRAIN
}
transform_param {
crop_size: 224
mean_value: 104
mean_value: 117
mean_value: 123
mirror: true
}
data_param {
source: "D:/caffe/caffe-master/examples/myfile/img_train_lmdb" #注意訓練集文件的路徑
batch_size: 20 #訓練批次大小根據自己的顯卡顯存而定
backend: LMDB
}
}
layer {
name: "data"
type: "Data"
top: "data"
top: "label"
include {
phase: TEST
}
transform_param {
crop_size: 224
mean_value: 104
mean_value: 117
mean_value: 123
mirror: false
}
data_param {
source: "D:/caffe/caffe-master/examples/myfile/img_test_lmdb" #注意驗證集文件的路徑
batch_size: 10
backend: LMDB
}
}
layer {
name: "conv1_1"
type: "Convolution"
bottom: "data"
top: "conv1_1"
param {
lr_mult: 1
}
param {
lr_mult: 1
}
convolution_param {
num_output: 64
kernel_size: 3
pad: 1
stride: 1
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "relu1_1"
type: "ReLU"
bottom: "conv1_1"
top: "conv1_1"
}
layer {
name: "conv1_2"
type: "Convolution"
bottom: "conv1_1"
top: "conv1_2"
param {
lr_mult: 1
}
param {
lr_mult: 1
}
convolution_param {
num_output: 64
kernel_size: 3
pad: 1
stride: 1
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "relu1_2"
type: "ReLU"
bottom: "conv1_2"
top: "conv1_2"
}
layer {
name: "pool1"
type: "Pooling"
bottom: "conv1_2"
top: "pool1"
pooling_param {
pool: MAX
kernel_size: 2
stride: 2
}
}
layer {
name: "conv2_1"
type: "Convolution"
bottom: "pool1"
top: "conv2_1"
param {
lr_mult: 1
}
param {
lr_mult: 1
}
convolution_param {
num_output: 128
kernel_size: 3
pad: 1
stride: 1
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "relu2_1"
type: "ReLU"
bottom: "conv2_1"
top: "conv2_1"
}
layer {
name: "conv2_2"
type: "Convolution"
bottom: "conv2_1"
top: "conv2_2"
param {
lr_mult: 1
}
param {
lr_mult: 1
}
convolution_param {
num_output: 128
kernel_size: 3
pad: 1
stride: 1
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "relu2_2"
type: "ReLU"
bottom: "conv2_2"
top: "conv2_2"
}
layer {
name: "pool2"
type: "Pooling"
bottom: "conv2_2"
top: "pool2"
pooling_param {
pool: MAX
kernel_size: 2
stride: 2
}
}
layer {
name: "conv3_1"
type: "Convolution"
bottom: "pool2"
top: "conv3_1"
param {
lr_mult: 1
}
param {
lr_mult: 1
}
convolution_param {
num_output: 256
kernel_size: 3
pad: 1
stride: 1
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "relu3_1"
type: "ReLU"
bottom: "conv3_1"
top: "conv3_1"
}
layer {
name: "conv3_2"
type: "Convolution"
bottom: "conv3_1"
top: "conv3_2"
param {
lr_mult: 1
}
param {
lr_mult: 1
}
convolution_param {
num_output: 256
kernel_size: 3
pad: 1
stride: 1
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "relu3_2"
type: "ReLU"
bottom: "conv3_2"
top: "conv3_2"
}
layer {
name: "conv3_3"
type: "Convolution"
bottom: "conv3_2"
top: "conv3_3"
param {
lr_mult: 1
}
param {
lr_mult: 1
}
convolution_param {
num_output: 256
kernel_size: 3
pad: 1
stride: 1
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "relu3_3"
type: "ReLU"
bottom: "conv3_3"
top: "conv3_3"
}
layer {
name: "pool3"
type: "Pooling"
bottom: "conv3_3"
top: "pool3"
pooling_param {
pool: MAX
kernel_size: 2
stride: 2
}
}
layer {
name: "conv4_1"
type: "Convolution"
bottom: "pool3"
top: "conv4_1"
param {
lr_mult: 1
}
param {
lr_mult: 1
}
convolution_param {
num_output: 512
kernel_size: 3
pad: 1
stride: 1
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "relu4_1"
type: "ReLU"
bottom: "conv4_1"
top: "conv4_1"
}
layer {
name: "conv4_2"
type: "Convolution"
bottom: "conv4_1"
top: "conv4_2"
param {
lr_mult: 1
}
param {
lr_mult: 1
}
convolution_param {
num_output: 512
kernel_size: 3
pad: 1
stride: 1
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "relu4_2"
type: "ReLU"
bottom: "conv4_2"
top: "conv4_2"
}
layer {
name: "conv4_3"
type: "Convolution"
bottom: "conv4_2"
top: "conv4_3"
param {
lr_mult: 1
}
param {
lr_mult: 1
}
convolution_param {
num_output: 512
kernel_size: 3
pad: 1
stride: 1
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "relu4_3"
type: "ReLU"
bottom: "conv4_3"
top: "conv4_3"
}
layer {
name: "pool4"
type: "Pooling"
bottom: "conv4_3"
top: "pool4"
pooling_param {
pool: MAX
kernel_size: 2
stride: 2
}
}
layer {
name: "conv5_1"
type: "Convolution"
bottom: "pool4"
top: "conv5_1"
param {
lr_mult: 1
}
param {
lr_mult: 1
}
convolution_param {
num_output: 512
kernel_size: 3
pad: 1
stride: 1
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "relu5_1"
type: "ReLU"
bottom: "conv5_1"
top: "conv5_1"
}
layer {
name: "conv5_2"
type: "Convolution"
bottom: "conv5_1"
top: "conv5_2"
param {
lr_mult: 1
}
param {
lr_mult: 1
}
convolution_param {
num_output: 512
kernel_size: 3
pad: 1
stride: 1
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "relu5_2"
type: "ReLU"
bottom: "conv5_2"
top: "conv5_2"
}
layer {
name: "conv5_3"
type: "Convolution"
bottom: "conv5_2"
top: "conv5_3"
param {
lr_mult: 1
}
param {
lr_mult: 1
}
convolution_param {
num_output: 512
kernel_size: 3
pad: 1
stride: 1
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "relu5_3"
type: "ReLU"
bottom: "conv5_3"
top: "conv5_3"
}
layer {
name: "pool5"
type: "Pooling"
bottom: "conv5_3"
top: "pool5"
pooling_param {
pool: MAX
kernel_size: 2
stride: 2
}
}
layer {
name: "fc6"
type: "InnerProduct"
bottom: "pool5"
top: "fc6"
param {
lr_mult: 1
}
param {
lr_mult: 1
}
inner_product_param {
num_output: 4096
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "relu6"
type: "ReLU"
bottom: "fc6"
top: "fc6"
}
layer {
name: "drop6"
type: "Dropout"
bottom: "fc6"
top: "fc6"
dropout_param {
dropout_ratio: 0.5
}
}
layer {
name: "fc7"
type: "InnerProduct"
bottom: "fc6"
top: "fc7"
param {
lr_mult: 1
}
param {
lr_mult: 1
}
inner_product_param {
num_output: 4096
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "relu7"
type: "ReLU"
bottom: "fc7"
top: "fc7"
}
layer {
name: "drop7"
type: "Dropout"
bottom: "fc7"
top: "fc7"
dropout_param {
dropout_ratio: 0.5
}
}
layer {
name: "fc8"
type: "InnerProduct"
bottom: "fc7"
top: "fc8"
param {
lr_mult: 1
}
param {
lr_mult: 2
}
inner_product_param {
num_output: 10 #注意將fc8層改成自己的圖像類別數目
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "accuracy"
type: "Accuracy"
bottom: "fc8"
bottom: "label"
top: "accuracy"
include {
phase: TEST
}
}
layer {
name: "loss"
type: "SoftmaxWithLoss"
bottom: "fc8"
bottom: "label"
top: "loss"
}
以上是VGG16模型,需改路徑和類別數後保存退出即可。
在當前路徑下新建文件vggnet_solver.prototxt,複製以下代碼,別忘了修改相對應的路徑
net: "D:/caffe/caffe-master/models/VGG16/vggnet_train_val.prototxt"
test_iter: 10
test_interval: 500 #每經過500次訓練,進行一次驗證查看accuracy
base_lr: 0.001
lr_policy: "step"
gamma: 0.1
stepsize: 1000
display: 20
max_iter: 2000 #只是做做練習,2000次迭代就夠了
momentum: 0.9
weight_decay: 0.0005
snapshot: 1000 #每經過1000次迭代訓練保存一次快照
snapshot_prefix: "D:/caffe/caffe-master/examples/myfile/caffenet_train" #在路徑下新建caffenet_train文件夾
solver_mode: GPU
5.訓練網絡
在當前路徑下新建train_vggnet.sh文件,同樣用記事本打開,複製以下代碼,同樣修改相對應的路徑
#!/usr/bin/env sh
set -e
D:/caffe/caffe-master/Build/x64/Release/caffe train \
--solver=D:/caffe/caffe-master/models/VGG16/vggnet_solver.prototxt $@
修改後保存退出,然後在cmd命令窗口cd到當前路徑,然後輸入命令 sh train_vggnet.sh,如下圖所示
如果運行沒出錯的話,那恭喜你訓練成功,然後就是等待訓練結果就行