1.Hive簡介
起源自facebook由Jeff Hammerbacher領導的團隊
構建在Hadoop上的數據倉庫框架
設計目的是讓SQL技能良好,但Java技能較弱的分析師可以查詢海量數據
2008年facebook把hive項目貢獻給Apache
Hive的組件與體系架構
用戶接口:shell, thrift, web等
Thrift服務器
元數據庫“Derby, Mysql等
解析器
Hadoop
Hive安裝模式
內嵌模式:元數據保持在內嵌的Derby模式,只允許一個會話連接(默認)
本地獨立模式:在本地安裝Mysql,把元數據放到Mysql內
遠程模式:元數據放置在遠程的Mysql數據庫
2.配置Mysql
1:copy mysql-connector-java-5.1.6-bin.jar到$HIVE_HOME/lib
- [jifeng@jifeng02 hadoop]$ ls
- 7287OS_Code hadoop-1.2.1.tar.gz hive-0.12.0-bin tmp
- hadoop-1.2.1 hadoop-2.4.1-src.tar.gz hive-0.12.0-bin.tar.gz
- hadoop-1.2.1-bin.tar.gz hadoop-2.4.1.tar.gz mysql-connector-java-5.1.6-bin.jar
- [jifeng@jifeng02 hadoop]$ cp mysql-connector-java-5.1.6-bin.jar hive-0.12.0-bin/lib
2:修改$HIVE_HOME/conf/hive-site.xml
- <property>
- <name>javax.jdo.option.ConnectionURL</name>
- <value>jdbc:mysql://jifengsql:3306/hive?createDatabaseIfNotExist=true</value>
- <description>JDBC connect string for a JDBC metastore</description>
- </property>
- <property>
- <name>javax.jdo.option.ConnectionDriverName</name>
- <value>com.mysql.jdbc.Driver</value>
- <description>Driver class name for a JDBC metastore</description>
- </property>
- <property>
- <name>javax.jdo.PersistenceManagerFactoryClass</name>
- <value>org.datanucleus.api.jdo.JDOPersistenceManagerFactory</value>
- <description>class implementing the jdo persistence</description>
- </property>
- <property>
- <name>javax.jdo.option.DetachAllOnCommit</name>
- <value>true</value>
- <description>detaches all objects from session so that they can be used after transaction is committed</
- description>
- </property>
- <property>
- <name>javax.jdo.option.NonTransactionalRead</name>
- <value>true</value>
- <description>reads outside of transactions</description>
- </property>
- <property>
- <name>javax.jdo.option.ConnectionUserName</name>
- <value>dss</value>
- <description>username to use against metastore database</description>
- </property>
- <property>
- <name>javax.jdo.option.ConnectionPassword</name>
- <value>jifeng</value>
- <description>password to use against metastore database</description>
- </property>
3:啓動hive
- [jifeng@jifeng02 hive-0.12.0-bin]$ hive
- Logging initialized using configuration in jar:file:/home/jifeng/hadoop/hive-0.12.0-bin/lib/hive-common-0.12.0.jar!/hive-log4j.properties
- hive> show tables;
- FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
- hive> quit;
把 mysql-connector-java-5.1.6-bin.jar 替換成mysql-connector-java-5.1.10-bin.jar還是不行。
檢查 mysql發現連接不上,換個虛擬機上mysql
- [dss@localhost ~]$ mysql -u root -p
- Enter password:
- --root 登陸mysql
- Welcome to the MySQL monitor. Commands end with ; or \g.
- Your MySQL connection id is 70
- Server version: 5.6.16 MySQL Community Server (GPL)
- Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved.
- Oracle is a registered trademark of Oracle Corporation and/or its
- affiliates. Other names may be trademarks of their respective
- owners.
- Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
- mysql> create database hive;
- Query OK, 1 row affected (0.01 sec) --創建hive庫
- mysql> GRANT all ON hive.* TO dss@'%' IDENTIFIED BY 'abc123';
- Query OK, 0 rows affected (0.03 sec)--給dss用戶對hive庫授權
- mysql> flush privileges;
- Query OK, 0 rows affected (0.02 sec)--刷新系統權限表
- mysql> set globalbinlog_format='MIXED';
- ERROR 1193 (HY000): Unknown system variable 'globalbinlog_format'
- mysql> alter database hive character set latin1 ;
- Query OK, 1 row affected (0.00 sec)--更新字符集
再次啓動
- [jifeng@jifeng02 hive-0.12.0-bin]$ hive
- Logging initialized using configuration in jar:file:/home/jifeng/hadoop/hive-0.12.0-bin/lib/hive-common-0.12.0.jar!/hive-log4j.properties
- hive> show tables;
- OK
- Time taken: 6.273 seconds
- hive>
沒有錯誤了。
4.Hive的運行模式即任務的執行環境
1啓動hive 命令行模式:
1:直接輸入#hive的執行程序,
2:或者輸入 #hive --service cli
l 分爲本地與集羣兩種
我們可以通過mapred.job.tracker 來指明
設置方式:
hive >
SET mapred.job.tracker=local
2.hive驗證啓動的方法
l 1、hive web界面的(端口號9999) 啓動方式
#hive --service hwi
- [jifeng@jifeng02 hive-0.12.0-bin]$ hive --service cli
- Logging initialized using configuration in jar:file:/home/jifeng/hadoop/hive-0.12.0-bin/lib/hive-common-0.12.0.jar!/hive-log4j.properties
- hive> quit;
- [jifeng@jifeng02 hive-0.12.0-bin]$ hive --service hwi
- 15/08/17 15:17:10 INFO hwi.HWIServer: HWI is starting up
- 15/08/17 15:17:10 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
- 15/08/17 15:17:10 INFO mortbay.log: jetty-6.1.26
- 15/08/17 15:17:10 INFO mortbay.log: Extract /home/jifeng/hadoop/hive-0.12.0-bin/lib/hive-hwi-0.12.0.war to /tmp/Jetty_0_0_0_0_9999_hive.hwi.0.12.0.war__hwi__ow27i/webapp
- 15/08/17 15:17:11 INFO mortbay.log: Started [email protected]:9999
用於通過瀏覽器來訪問hive
http://jifeng02:9999/hwi/
l 2、hive 遠程服務(端口號10000) 啓動方式
#hive --service hiveserver