Spark Standalone安裝

Spark Standalone安裝

0.基本信息

	CentOS7
	jdk-8u181
	spark-2.4.4-bin-hadoop2.7.tgz
	主機
		192.168.1.101
		192.168.1.102

1.免密碼登錄配置

	0.登錄到主節點
	1.ssh-keygen -t rsa
	2.cd ~/.ssh/
	3.cat id_rsa.pub >> authorized_keys
	
	4.authorized_keys拷貝到子節點
	
	5.登錄子節點 cd ~/.ssh/
	6.cat id_rsa.pub >> authorized_keys
	7.authorized_keys拷貝到主機點

2.Spark配置spark-2.4.4-bin-hadoop2.7

conf/spark-env.sh
	SPARK_MASTER_HOST=192.168.1.102
	SPARK_MASTER_PORT=7077
conf/slaves
	192.168.1.101
	192.168.1.102

3.同步文件

spark-2.4.4-bin-hadoop2.7同步到從節點172.17.0.2

4.啓動

主節點/opt/spark/spark-2.4.4/sbin/start-all.sh

5.驗證

bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
--master spark://192.168.1.102:7077 \
--executor-memory 1G \
--total-executor-cores 2 \
./examples/jars/spark-examples_2.11-2.4.4.jar \
100

6.停止

主節點/opt/spark/spark-2.4.4/sbin/start-all.sh