MLlib數據統計基本概念

備註：kimi.txt中的內容如下：
     1
     2
     3
     4
     5

一.求數據的均值和標準差
import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.stat.Statistics
import org.apache.spark.{SparkConf, SparkContext}
object testVector { def main(args: Array[String]): Unit = {
val conf = new SparkConf().setMaster("local")
.setAppName("testVector");
val sc = new SparkContext(conf);
var rdd = sc.textFile("kimi.txt")
.map(_.split(' ')
.map(_.toDouble))
.map(line => Vectors.dense(line));
var summary = Statistics.colStats(rdd);
println(summary.mean);//計算均值
println(summary.variance);//計算標準差
}
}
程序結果：[3.0][2.5]
二.距離計算
1.歐幾里得距離（normL1）:指在m維空間中兩個點之間的真實距離，或者向量的自然長度（即該點到原點的距離）。
2.曼哈段距離（normL2）：兩個點在標準座標系上的絕對軸距總和。

import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.stat.Statistics
import org.apache.spark.{SparkConf, SparkContext}
object testVector {
  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setMaster("local")
    .setAppName("testVector");
    val sc = new SparkContext(conf);
    var rdd = sc.textFile("kimi.txt")
    .map(_.split(' ')
    .map(_.toDouble))
    .map(line => Vectors.dense(line));
    var summary = Statistics.colStats(rdd);
    println(summary.normL1);
    println(summary.normL2);
  }
}

程序結果：
[15.0]
[7.416198487095663]
三.相關係數
x.txt，y.txt內容：
1 2 3 4 5
2 4 6 8 10

import org.apache.spark.mllib.stat.Statistics
import org.apache.spark.{SparkConf, SparkContext}
object testVector {
  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setMaster("local")
    .setAppName("testVector");
    val sc = new SparkContext(conf);
    var rddX = sc.textFile("x.txt")
    .flatMap(_.split(' ')
    .map(_.toDouble));
    var rddY = sc.textFile("y.txt")
      .flatMap(_.split(' ')
        .map(_.toDouble));
    var correlation: Double = Statistics.corr(rddX,rddY);//皮爾遜相關係數 1.0
    println(correlation);
    val correlation2: Double = Statistics.corr(rddX,rddY,"spearman");//斯皮爾曼相關係數 1.0000000000000009
    println(correlation2);
  }
}
單個數據集相關係數的計算

import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.stat.Statistics
import org.apache.spark.{SparkConf, SparkContext}
object testVector {
  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setMaster("local")
    .setAppName("testVector");
    val sc = new SparkContext(conf);
    var rdd = sc.textFile("x.txt")
    .map(_.split(' ')
    .map(_.toDouble))
    .map(line => Vectors.dense(line))
    println(Statistics.corr(rdd,"spearman"));
  }
}

1.0                 0.9999999999999998  0.9999999999999998  ... (5 total)
0.9999999999999998  1.0                 0.9999999999999998  ...
0.9999999999999998  0.9999999999999998  1.0                 ...
0.9999999999999998  0.9999999999999998  0.9999999999999998  ...
0.9999999999999998  0.9999999999999998  0.9999999999999998  ...