pyspark讀取hive數據實例

使用pyspark讀取hive中的數據，測試代碼：

vi test.py

#!-*- coding:utf-8 -*-

from pyspark import SparkConf, SparkContext
from pyspark.sql import HiveContext

conf = (SparkConf().setMaster("yarn").setAppName("My app").set("spark.executor.memory", "1g"))
sc = SparkContext(conf = conf)
sqlContext = HiveContext(sc)

my_dataframe = sqlContext.sql("Select * from database.table limit 10")
my_dataframe.show()

sc.stop()

報錯：
python pyspark_hive.py
Traceback (most recent call last):
  File "pyspark_hive.py", line 2, in <module>
    from pyspark.sql import HiveContext,SparkSession
ModuleNotFoundError: No module named 'pyspark'

環境變量設置：
vi ~/.profile
 

export SPARK_HOME=/usr/lib/spark-current
export PYTHONPATH=$SPARK_HOME/python/:$SPARK_HOME/python/lib/py4j-0.10.7-src.zip:$PYTHONPATH

參考：
https://www.cnblogs.com/tong775131501/p/7582258.html