數據集Data01.txt如下
Aaron,OperatingSystem,100
Aaron,Python,50
Aaron,ComputerNetwork,30
Aaron,Software,94
Abbott,DataBase,18
Abbott,Python,82
Abbott,ComputerNetwork,76
Abel,Algorithm,30
Abel,DataStructure,38
Abel,OperatingSystem,38
Abel,ComputerNetwork,92
Abraham,DataStructure,12
Abraham,ComputerNetwork,78
Abraham,Software,98
Adair,DataBase,20
Adair,Python,98
Adair,Software,88
Adam,Algorithm,18
Adam,ComputerNetwork,70
Adam,Software,80
Adolph,DataStructure,82
Adolph,CLanguage,100
Adolph,ComputerNetwork,70
Adolph,Software,18
Adonis,DataBase,86
Adonis,Algorithm,34
Adonis,DataStructure,52
Adonis,CLanguage,30
Adonis,Python,86
Alan,Algorithm,48
...........................................
1、導入數據文件
val dataFile = sc.textFile("file:///root/Data01.txt")
2、該系總共有多少學生；
dataFile.map(s => s.split(",").reverse(2)).distinct().count()
3、該系共開設來多少門課程；
dataFile.map(s => s.split(",").reverse(1)).distinct().count()
4、Tom同學的總成績平均分是多少；
dataFile.filter(line => line.contains("Tom")).map(s => s.split(",").reverse(0).toInt).sum() / dataFile.filter(line => line.contains("Tom")).count()
5、求每名同學的選修的課程門數；
dataFile.map(s => s.split(",").reverse(2)).map(s => (s,1)).reduceByKey(_+_).foreach(println)
6、該系DataBase課程共有多少人選修；
dataFile.map(s => s.split(",").reverse(1)).filter(l => l.contains("DataBase")).count()
7、各門課程的平均分是多少。
dataFile.map(s => (s.split(",")(1),s.split(",")(2).toInt)).mapValues(x => (x, 1)).reduceByKey((x, y) => (x._1 + y._1, x._2 + y._2)).mapValues(x => x._1.toDouble / x._2.toDouble).foreach(println)
