博文參考standford UFLDL網頁教程線性解碼器。
1、線性解碼器
前面說過的稀疏自編碼器是一個三層的feed-forward神經網絡結構，包含輸入層、隱含層和輸出層，隱含層和輸出層採用的激活函數都是sigmoid函數，由於sigmoid函數的y值範圍在[0,1]，這就要求輸入也要在這個範圍內，MNIST數據是在這個範圍內的，但是對於有些數據，我們不知道用什麼辦法縮放到[0,1]才合適，所以就有線性解碼器。線性解碼器（linear decoders）其實就是輸出層採用線性激活函數，最簡單的線性激活函數就是恆等激活函數，就是a=f(z)=z。但是中間隱含層必須採用sigmoid函數或者tanh函數，這兩個都是對輸入的非線性變換，如果採用線性變換，一方面表達能力沒有那麼強，另一方面就是沒有必要採用三層結構了，直接隱含層當做輸出層也可以學到一樣的函數關係。有了線性解碼器之後，輸入層的單元就沒必要限制在[0,1]了。
線性解碼器的輸出層可以通過調整W2使得輸出數值可以大於1或者小於0。
對於線性解碼器，當輸出層的激活函數變爲恆等激活函數，輸出單元的誤差項變爲：
使用BP算法計算隱含層單元的誤差爲：
2、Learning color features with Sparse Autoencoders
關於實驗的一些說明：
- 實驗的數據集是STL-10數據，是RGB三通道圖，之前的實驗用的是MNIST數據集，MNIST是灰度圖；STL-10數據是把RGB組成一個長向量，這樣就跟MNIST數據一樣了。實驗數據patches的大小是192*100000，因爲RGB patches大小是8x8，把RGB組合起來就是192.
- 數據預處理是ZCAWhiten，ZCAWhiten並沒有對像PCAWhiten那樣對數據進行降維，ZCAWhiten可以得到儘量接近原始數據，但是數據維度之間沒有相關性，而且維度的方差一樣。
- 比較奇怪的是最後顯示學到的權重圖時，代碼是displayColorNetwork( (W*ZCAWhite)')，我的理解由於輸入一個樣本x時，得到的特徵是W*ZCAWhite*x，它要顯示的是W*ZCAWhite這個變換，如果是把ZCAWhite*x當做原始輸入，就跟以前的直接顯示W一樣了。
實驗結果：
最終學到的特徵圖爲：
Matlab代碼把sparseAutoencoderCost.m的代碼複製到sparseAutoencoderLinearCost.m並修改幾行即可.
function [cost,grad,features] = sparseAutoencoderLinearCost(theta, visibleSize, hiddenSize, ...
lambda, sparsityParam, beta, data)
% visibleSize: the number of input units (probably 64)
% hiddenSize: the number of hidden units (probably 25)
% lambda: weight decay parameter
% sparsityParam: The desired average activation for the hidden units (denoted in the lecture
% notes by the greek alphabet rho, which looks like a lower-case "p").
% beta: weight of sparsity penalty term
% data: Our 192x1000000 matrix containing the training data. So, data(:,i) is the i-th training example.
% The input theta is a vector (because minFunc expects the parameters to be a vector).
% We first convert theta to the (W1, W2, b1, b2) matrix/vector format, so that this
% follows the notation convention of the lecture notes.
W1 = reshape(theta(1:hiddenSize*visibleSize), hiddenSize, visibleSize);
W2 = reshape(theta(hiddenSize*visibleSize+1:2*hiddenSize*visibleSize), visibleSize, hiddenSize);
b1 = theta(2*hiddenSize*visibleSize+1:2*hiddenSize*visibleSize+hiddenSize);
b2 = theta(2*hiddenSize*visibleSize+hiddenSize+1:end);
% Cost and gradient variables (your code needs to compute these values).
% Here, we initialize them to zeros.
cost = 0;
W1grad = zeros(size(W1));
W2grad = zeros(size(W2));
b1grad = zeros(size(b1));
b2grad = zeros(size(b2));
%% ---------- YOUR CODE HERE --------------------------------------
% Instructions: Compute the cost/optimization objective J_sparse(W,b) for the Sparse Autoencoder,
% and the corresponding gradients W1grad, W2grad, b1grad, b2grad.
%
% W1grad, W2grad, b1grad and b2grad should be computed using backpropagation.
% Note that W1grad has the same dimensions as W1, b1grad has the same dimensions
% as b1, etc. Your code should set W1grad to be the partial derivative of J_sparse(W,b) with
% respect to W1. I.e., W1grad(i,j) should be the partial derivative of J_sparse(W,b)
% with respect to the input parameter W1(i,j). Thus, W1grad should be equal to the term
% [(1/m) \Delta W^{(1)} + \lambda W^{(1)}] in the last block of pseudo-code in Section 2.2
% of the lecture notes (and similarly for W2grad, b1grad, b2grad).
%
% Stated differently, if we were using batch gradient descent to optimize the parameters,
% the gradient descent update to W1 would be W1 := W1 - alpha * W1grad, and similarly for W2, b1, b2.
%
%矩陣向量化形式實現,速度比不用向量快得多
Jocst = 0; %平方誤差
Jweight = 0; %規則項懲罰
Jsparse = 0; %稀疏性懲罰
[n, m] = size(data); %m爲樣本數，這裏是1000000，n爲樣本維數，這裏是192
%feedforward前向算法計算隱含層和輸出層的每個節點的z值（線性組合值）和a值（激活值）
%data每一列是一個樣本，
z2 = W1*data + repmat(b1,1,m); %W1*data的每一列是每個樣本的經過權重W1到隱含層的線性組合值,repmat把列向量b1擴充成m列b1組成的矩陣
a2 = sigmoid(z2);
z3 = W2*a2 + repmat(b2,1,m);
%%%%對於線性解碼器，要修改下面一行%%%%
%a3 = sigmoid(z3);
a3 = z3;
%%%%%%%%%%%%%%%%%%%%
%計算預測結果與理想結果的平均誤差
Jcost = (0.5/m)*sum(sum((a3-data).^2));
%計算權重懲罰項
Jweight = (1/2)*(sum(sum(W1.^2))+sum(sum(W2.^2)));
%計算稀疏性懲罰項
rho_hat = (1/m)*sum(a2,2);
Jsparse = sum(sparsityParam.*log(sparsityParam./rho_hat)+(1-sparsityParam).*log((1-sparsityParam)./(1-rho_hat)));
%計算總損失函數
cost = Jcost + lambda*Jweight + beta*Jsparse;
%%%% 修改下面一行對a3的求導%%%%%
%反向傳播求誤差值
%delta3 = -(data-a3).*fprime(a3); %每一列是一個樣本對應的誤差
delta3 = -(data-a3);
%%%%%%%%%%%%%%%%%%%%%%
sterm = beta*(-sparsityParam./rho_hat+(1-sparsityParam)./(1-rho_hat));
delta2 = (W2'*delta3 + repmat(sterm,1,m)).*fprime(a2);
%計算梯度
W2grad = delta3*a2';
W1grad = delta2*data';
W2grad = W2grad/m + lambda*W2;
W1grad = W1grad/m + lambda*W1;
b2grad = sum(delta3,2)/m; %因爲對b的偏導是個向量，這裏要把delta3的每一列加起來
b1grad = sum(delta2,2)/m;
%%----------------------------------
% %對每個樣本進行計算, non-vectorial implementation
% [n m] = size(data);
% a2 = zeros(hiddenSize,m);
% a3 = zeros(visibleSize,m);
% Jcost = 0; %平方誤差項
% rho_hat = zeros(hiddenSize,1); %隱含層每個節點的平均激活度
% Jweight = 0; %權重衰減項
% Jsparse = 0; % 稀疏項代價
%
% for i=1:m
% %feedforward向前轉播
% z2(:,i) = W1*data(:,i)+b1;
% a2(:,i) = sigmoid(z2(:,i));
% z3(:,i) = W2*a2(:,i)+b2;
% %a3(:,i) = sigmoid(z3(:,i));
% a3(:,i) = z3(:,i);
% Jcost = Jcost+sum((a3(:,i)-data(:,i)).*(a3(:,i)-data(:,i)));
% rho_hat = rho_hat+a2(:,i); %累加樣本隱含層的激活度
% end
%
% rho_hat = rho_hat/m; %計算平均激活度
% Jsparse = sum(sparsityParam*log(sparsityParam./rho_hat) + (1-sparsityParam)*log((1-sparsityParam)./(1-rho_hat))); %計算稀疏代價
% Jweight = sum(W1(:).*W1(:))+sum(W2(:).*W2(:));%計算權重衰減項
% cost = Jcost/2/m + Jweight/2*lambda + beta*Jsparse; %計算總代價
%
% for i=1:m
% %backpropogation向後傳播
% %delta3 = -(data(:,i)-a3(:,i)).*fprime(a3(:,i));
% delta3 = -(data(:,i)-a3(:,i));
% delta2 = (W2'*delta3 +beta*(-sparsityParam./rho_hat+(1-sparsityParam)./(1-rho_hat))).*fprime(a2(:,i));
%
% W2grad = W2grad + delta3*a2(:,i)';
% W1grad = W1grad + delta2*data(:,i)';
% b2grad = b2grad + delta3;
% b1grad = b1grad + delta2;
% end
% %計算梯度
% W1grad = W1grad/m + lambda*W1;
% W2grad = W2grad/m + lambda*W2;
% b1grad = b1grad/m;
% b2grad = b2grad/m;
% -------------------------------------------------------------------
% After computing the cost and gradient, we will convert the gradients back
% to a vector format (suitable for minFunc). Specifically, we will unroll
% your gradient matrices into a vector.
grad = [W1grad(:) ; W2grad(:) ; b1grad(:) ; b2grad(:)];
end
%% Implementation of derivation of f(z)
% f(z) = sigmoid(z) = 1./(1+exp(-z))
% a = 1./(1+exp(-z))
% delta(f) = a.*(1-a)
function dz = fprime(a)
dz = a.*(1-a);
end
%%
%-------------------------------------------------------------------
% Here's an implementation of the sigmoid function, which you may find useful
% in your computation of the costs and the gradients. This inputs a (row or
% column) vector (say (z1, z2, z3)) and returns (f(z1), f(z2), f(z3)).
function sigm = sigmoid(x)
sigm = 1 ./ (1 + exp(-x));
end