pyspark on yarn 出現 Cannot run program python

pyspark \
	--master yarn \
	--name shell \
	--driver-cores 1 \
	--driver-memory 512m \
	--num-executors 1 \
	--executor-cores 1 \
	--executor-memory 512m \
	--verbose 

使用pyspark運行在yarn上的時候，錯誤如下

Lost task 0.0 in stage 0.0 (TID 0, server, executor 1): java.io.IOException: Cannot run program "python": error=2, No such file or directory
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
        at org.apache.spark.api.python.PythonWorkerFactory.startDaemon(PythonWorkerFactory.scala:197)
        at org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:122)
        at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:95)
        at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:117)
        at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:108)
        at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:65)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
        at org.apache.spark.scheduler.Task.run(Task.scala:121)
        at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: error=2, No such file or directory
        at java.lang.UNIXProcess.forkAndExec(Native Method)
        at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
        at java.lang.ProcessImpl.start(ProcessImpl.java:134)
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
        ... 16 more

這裏就是因爲python的執行文件沒有找到造成的

解決方式，配置環境變量，新增

# 這裏需要些絕對路徑
export PYSPARK_PYTHON=$PYTHON_HOME/bin/python3

這是spark還不支持python3.7的版本，所以使用的python3.6

重啓環境變量source /etc/profile，重新執行pyspark，截圖如下