Python鏈接Hive讀取數據的幾種方式

以下有幾種鏈接方式, 標題均包含了官方鏈接, 詳情大家可點擊查看

注意:

• Python鏈接Hive和Python版本和包版本有很大關係, 建議大家通過Anaconda進行管理, 歡迎閱讀Mac下Anaconda的安裝和使用
• thrift, sasl, thrift_sasl, thriftpy這幾個包均會用到, 請提前安裝好, 建議用安裝對應版本
  • thrift - 0.11.0
  • sasl - 0.2.1
  • thrift_sasl - 0.3.0
  • thriftpy - 0.3.9

筆主環境

• 版本

# 系統
macOS Mojave Version 10.14.2
# Python
Python 3.7.1

依賴包準備

PyHive

包安裝

pip install pyhive

鏈接方式

from pyhive import hive
conn = hive.Connection(host = '', port = 10000, username = '****', database = 'default')
#host主機ip,port：端口號，username:用戶名，database:使用的數據庫名稱
cursor = conn.cursor()
cursor.execute('SHOW DATABASES')
# 打印結果
for result in cursor.fetchall():
     print(result) 

impyla

包安裝

pip install impyla

錯誤處理

impyla pip成功安裝後, 但實際執行導入時候報錯

In [4]: from impala.dbapi import connect
Traceback (most recent call last):

  File "/anaconda3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3267, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-4-75666c8dbd41>", line 1, in <module>
    from impala.dbapi import connect

  File "/anaconda3/lib/python3.7/site-packages/impala/dbapi.py", line 28, in <module>
    import impala.hiveserver2 as hs2

  File "/anaconda3/lib/python3.7/site-packages/impala/hiveserver2.py", line 340
    async=True)
        ^
SyntaxError: invalid syntax

處理方式

conda install -c anaconda impyla

如果以上處理之後還有問題

/anaconda3/lib/python3.7/site-packages/thriftpy/protocol/binary.py in read_message_begin(inbuf, strict)
    176         if strict:
    177             raise TProtocolException(type=TProtocolException.BAD_VERSION,
--> 178                                      message='No protocol version header')
    179
    180         name = inbuf.read(sz).decode('utf-8')

TProtocolException: TProtocolException(type=4)

建議重新安裝相關依賴包, 版本一定要正確.

鏈接方式

conn = connect(host = '172.18.33.23', port = 10000, auth_mechanism = 'PLAIN')
cursor = conn.cursor()
cursor.execute('SHOW DATABASES')
# 打印結果
for result in cursor.fetchall():
     print(result)