用R解析Mahout用戶推薦協同過濾算法(UserCF)

前言
用R全面解析Mahout的基於用戶推薦協同過濾算法(UserCF)，改進的採用歐式距離，並用R語言實現，與Mahout的結果進行對比。
Mahout是Hadoop家族用於機器學習的一個框架，包括三個部分，推薦，聚類，分類！
我在這裏做的是推薦部分。推薦系統在現在的互聯網應用中很常見，比如，亞馬遜會推薦你買書，豆瓣會給你一個書評，影評。
目錄

1. Mahout的模型介紹
2. R語言模型實現
3. 算法實現的原理-矩陣變換
4. 算法總結
5. 參考資料
   ## 1.Mahout的模型介紹 ##
   
   Mahout版本

<dependency>
<groupId>org.apache.mahout</groupId>
<artifactId>mahout-core</artifactId>
<version>0.5</version>
</dependency>

Mahout程序寫法

public class UserBaseCFMain {

    final static int NEIGHBORHOOD_NUM = 2;
    final static int RECOMMENDER_NUM = 3;

    public static void main(String[] args) throws IOException, TasteException {
        String file = "metadata/data/testCF.csv";
        DataModel model = new FileDataModel(new File(file));
        UserSimilarity user = new EuclideanDistanceSimilarity(model);
        NearestNUserNeighborhood neighbor = new NearestNUserNeighborhood(NEIGHBORHOOD_NUM, user, model);
        Recommender r = new GenericUserBasedRecommender(model, neighbor, user);
        LongPrimitiveIterator iter = model.getUserIDs();

        while (iter.hasNext()) {
            long uid = iter.nextLong();
            List list = r.recommend(uid, RECOMMENDER_NUM);
            System.out.printf("uid:%s", uid);
            for (RecommendedItem ritem : list) {
                System.out.printf("(%s,%f)", ritem.getItemID(), ritem.getValue());
            }
            System.out.println();
        }
    }
} 

推薦結果：

uid:1(104,4.250000)(106,4.000000)
uid:2(105,3.956999)
uid:3(103,3.185407)(102,2.802432)
uid:4(102,3.000000)
uid:5 

2.R語言模型實現

1). 建立數據模型
2). 歐氏距離相似度算法
3). 最緊鄰算法
4). 推薦算法
5). 運行程序

1). 建立數據模型

FileDataModel<-function(file){
data<-read.csv(file,header=FALSE)
names(data)<-c("uid","iid","pref")

user <- unique(data$uid)
item <- unique(sort(data$iid))
uidx <- match(data$uid, user)
iidx <- match(data$iid, item)
M <- matrix(0, length(user), length(item))
i <- cbind(uidx, iidx, pref=data$pref)
for(n in 1:nrow(i)){
M[i[n,][1],i[n,][2]]<-i[n,][3]
}
dimnames(M)[[2]]<-item
M
}

2). 歐氏距離相似度算法

EuclideanDistanceSimilarity<-function(M){
row<-nrow(M)
s<-matrix(0, row, row)
for(z1 in 1:row){
for(z2 in 1:row){
if(z1<z2){< span="">
num<-intersect(which(M[z1,]!=0),which(M[z2,]!=0)) #可計算的列

sum<-0
for(z3 in num){
sum<-sum+(M[z1,][z3]-M[z2,][z3])^2
}

s[z2,z1]<-length(num)/(1+sqrt(sum))

if(s[z2,z1]>1) s[z2,z1]<-1 #標準化
if(s[z2,z1]< -1) s[z2,z1]<- -1 #標準化

#print(paste(z1,z2));print(num);print(sum)
}
}
}
#補全三角矩陣
ts<-t(s)
w<-which(upper.tri(ts))
s[w]<-ts[w]
s
}

3). 最緊鄰算法

NearestNUserNeighborhood<-function(S,n){
row<-nrow(S)
neighbor<-matrix(0, row, n)
for(z1 in 1:row){
for(z2 in 1:n){
m<-which.max(S[,z1])
#       print(paste(z1,z2,m,'\n'))
neighbor[z1,][z2]<-m
S[,z1][m]=0
}
}
neighbor
}

4).推薦算法

UserBasedRecommender<-function(uid,n,M,S,N){
row<-ncol(N)
col<-ncol(M)
r<-matrix(0, row, col)
N1<-N[uid,]
for(z1 in 1:length(N1)){
num<-intersect(which(M[uid,]==0),which(M[N1[z1],]!=0)) #可計算的列
#     print(num)

for(z2 in num){
#       print(paste("for:",z1,N1[z1],z2,M[N1[z1],z2],S[uid,N1[z1]]))
r[z1,z2]=M[N1[z1],z2]*S[uid,N1[z1]]
}
}

sum<-colSums(r)
s2<-matrix(0, 2, col)
for(z1 in 1:length(N1)){
num<-intersect(which(colSums(r)!=0),which(M[N1[z1],]!=0))
for(z2 in num){
s2[1,][z2]<-s2[1,][z2]+S[uid,N1[z1]]
s2[2,][z2]<-s2[2,][z2]+1
}
}

s2[,which(s2[2,]==1)]=10000
s2<-s2[-2,]

r2<-matrix(0, n, 2)
rr<-sum/s2
item <-dimnames(M)[[2]]
for(z1 in 1:n){
w<-which.max(rr)
if(rr[w]>0.5){
r2[z1,1]<-item[which.max(rr)]
r2[z1,2]<-as.double(rr[w])
rr[w]=0
}
}
r2
}

5).運行程序

FILE<-"testCF.csv"
NEIGHBORHOOD_NUM<-2
RECOMMENDER_NUM<-3

M<-FileDataModel(FILE)
S<-EuclideanDistanceSimilarity(M)
N<-NearestNUserNeighborhood(S,NEIGHBORHOOD_NUM)

R1<-UserBasedRecommender(1,RECOMMENDER_NUM,M,S,N);R1
##      [,1]  [,2]  
## [1,] "104" "4.25"
## [2,] "106" "4"   
## [3,] "0"   "0" 

R2<-UserBasedRecommender(2,RECOMMENDER_NUM,M,S,N);R2
##      [,1]  [,2]
## [1,] "105" "3.95699903407931"
## [2,] "0"   "0"
## [3,] "0"   "0"

R3<-UserBasedRecommender(3,RECOMMENDER_NUM,M,S,N);R3
##      [,1]  [,2]
## [1,] "103" "3.18540697329411"
## [2,] "102" "2.80243217111765"
## [3,] "0"   "0"

R4<-UserBasedRecommender(4,RECOMMENDER_NUM,M,S,N);R4
##      [,1]  [,2]
## [1,] "102" "3" 
## [2,] "0"   "0" 
## [3,] "0"   "0"

R5<-UserBasedRecommender(5,RECOMMENDER_NUM,M,S,N);R5
##      [,1] [,2]
## [1,]    0    0
## [2,]    0    0
## [3,]    0    0

3.算法實現的原理——矩陣變換

所謂協同過濾算法，其實就是矩陣變換的結果！請大家下面留意矩陣操作！
1).原始數據

 1,101,5.0
  1,102,3.0
  1,103,2.5
  2,101,2.0
  2,102,2.5
  2,103,5.0
  2,104,2.0
  3,101,2.5
  3,104,4.0
  3,105,4.5
  3,107,5.0
  4,101,5.0
  4,103,3.0
  4,104,4.5
  4,106,4.0
  5,101,4.0
  5,102,3.0
  5,103,2.0
  5,104,4.0
  5,105,3.5
  5,106,4.0 

2).矩陣變換

101 102 103 104 105 106 107
[1,] 5.0 3.0 2.5 0.0 0.0   0   0
[2,] 2.0 2.5 5.0 2.0 0.0   0   0
[3,] 2.5 0.0 0.0 4.0 4.5   0   5
[4,] 5.0 0.0 3.0 4.5 0.0   4   0
[5,] 4.0 3.0 2.0 4.0 3.5   4   0 

3).歐式相似矩陣轉換

[,1]      [,2]      [,3]      [,4]      [,5]
[1,] 0.0000000 0.6076560 0.2857143 1.0000000 1.0000000
[2,] 0.6076560 0.0000000 0.6532633 0.5568464 0.7761999
[3,] 0.2857143 0.6532633 0.0000000 0.5634581 1.0000000
[4,] 1.0000000 0.5568464 0.5634581 0.0000000 1.0000000
[5,] 1.0000000 0.7761999 1.0000000 1.0000000 0.0000000 

4).最近鄰矩陣

 top1 top2
[1,]    4    5
[2,]    5    3
[3,]    5    2
[4,]    1    5
[5,]    1    3 

5).以R1爲例的推薦矩陣

101  102  103  104  105  106  107
   4    0    0    0  4.5  0.0    4    0
   5    0    0    0  4.0  3.5    4    0 

6).以R1爲例的推薦結果

推薦物品  物品得分
[1,] "104"    "4.25"
[2,] "106"    "4" 

4.算法總結

我這裏只是用R語言實現了Mahout的基於“用戶的”，“歐式距離”，“最近鄰的”的協同過濾算法。實現過程中發現，Mahout做各種算法時，都有自己的優化。
比如，算歐式距離時，並不是標準的

similar = 1/(1+sqrt( (a-b)2 + (a-c)2 ))

而是改進的算法

similar = n/(1+sqrt( (a-b)2 + (a-c)2 ))

1. n爲b,c的個數
2. similar>1 => similar=1
3. similar<-1 => similar=-1
   從而更能優化結果！