大數據集羣之spark2.4.0高可用安裝配置

一、安裝準備

下載地址：https://archive.apache.org/dist/spark/

官方文檔：http://spark.apache.org/docs/latest/

 

二、解壓安裝

解壓縮文件

cd /usr/local/hadoop
tar zxpf spark-2.4.0-bin-hadoop2.7.tgz

2、創建軟鏈接

ln -s spark-2.4.0-bin-hadoop2.7.tgz spark

三、修改配置文件

slaves

hadoop001
hadoop002
hadoop003
hadoop004
hadoop005

spark-env.sh

export JAVA_HOME=/usr/java/jdk1.8
export SCALA_HOME=/usr/local/hadoop/scala
export MYSQL_HOME=/usr/local/mysql
export CLASSPATH=.:/usr/java/jdk1.8/lib/dt.jar:/usr/java/jdk1.8/lib/tools.jar
export SPARK_HOME=/usr/local/hadoop/spark
export HADOOP_HOME=/usr/local/hadoop/hadoop
export HBASE_HOME=/usr/local/hadoop/hbase
export GEOMESA_HBASE_HOME=/usr/local/hadoop/geomesa-hbase
export ZOO_HOME=/usr/local/hadoop/zookeeper

export SPARK_WORKING_MEMORY=4096m
# export SPARK_MASTER_IP=hadoop1
export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=hadoop001:2181,hadoop002:2181,hadoop003:2181 -Dspark.deploy.zookeeper.dir=/spark"
export HADOOP_CONF_DIR=/usr/local/hadoop/hadoop/etc/hadoop/
export YARN_CONF_DIR=/usr/local/hadoop/hadoop/etc/hadoop/

四、環境變量配置

編輯 /etc/profile 文件

vim /etc/profile

添加以下內容

export SPARK_HOME=/usr/local/hadoop/spark
export PATH=$PATH:$SPARK_HOME

五、啓動spark

在master上執行

$SPARK_HOME/sbin/start-all.sh

在備用master上執行

$SPARK_HOME/sbin/start-master.sh

六、驗證安裝

http://hadoop001:8080/

http://hadoop002:8080/