其它項目組需要一套hive環境,原來想會很快搭好,沒想到還頗費了些周折。這裏記錄一下。
hadoop用現成的腳本安裝。
mysql也用現成的。
hive搭建的主要過程是按照這個教程做的:
http://www.cnblogs.com/zlslch/p/6700695.html,這個帖子雖然是安裝1.0的,但安裝2.0、3.0也好使。作者費心了。
修改hive-site.xml也參考了https://stackoverflow.com/questions/27099898/java-net-urisyntaxexception-when-starting-hive
也參考這篇文章做了初始化元數據:
schematool -dbType mysql -initSchema
schematool -dbType mysql -info
參考官方指南https://cwiki.apache.org/confluence/display/Hive/GettingStarted做了:
hadoop目錄初始化:
$ $HADOOP_HOME/bin/hadoop fs -mkdir /tmp $ $HADOOP_HOME/bin/hadoop fs -mkdir /user/hive/warehouse $ $HADOOP_HOME/bin/hadoop fs -chmod g+w /tmp $ $HADOOP_HOME/bin/hadoop fs -chmod g+w /user/hive/warehouse
建表、修改表、插入數據等也參照官方指南。
============================================================================
出現的問題:
啓用debug日誌: hive -hiveconf hive.root.logger=DEBUG,console)
運行簡單的show databases命令報錯:
2019-03-20T18:00:24,901 ERROR [08224acd-57c4-4e6f-bbf0-12c81aeef9c9 main] CliDriver: Failed with exception java.io.IOException:java.lang.RuntimeException: Error in configuring object
java.io.IOException: java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:602)
at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:509)
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146)
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2691)
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.RunJar.run(RunJar.java:234)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: java.lang.RuntimeException: Error in configuring object
at org.apache.hive.common.util.ReflectionUtil.setJobConf(ReflectionUtil.java:115)
at org.apache.hive.common.util.ReflectionUtil.setConf(ReflectionUtil.java:103)
at org.apache.hive.common.util.ReflectionUtil.newInstance(ReflectionUtil.java:87)
at org.apache.hadoop.hive.ql.exec.FetchOperator.getInputFormatFromCache(FetchOperator.java:221)
at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextSplits(FetchOperator.java:381)
at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:314)
at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:540)
... 16 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hive.common.util.ReflectionUtil.setJobConf(ReflectionUtil.java:112)
... 22 more
Caused by: java.lang.IllegalArgumentException: Compression codec org.apache.hadoop.io.compress.Bzip2Codec not found.
at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:139)
at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:179)
at org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:45)
... 27 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.io.compress.Bzip2Codec not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2122)
at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:132)
... 29 more
分析:
命令是正常執行了,但是結果不能正常顯示。
搜了一下,網上基本沒有報 Class org.apache.hadoop.io.compress.Bzip2Codec not found這個錯的,有報lzo錯的。
初步判斷是配置文件有錯。把core-site.xml中關於compress的配置全部去掉,問題照舊。
再判斷是本地庫編譯有問題,確實當初編譯的hadoop不支持Bzip2Codec。於是裝了個centos編譯源碼,再重裝hadoop,問題照舊。
再仔細分析錯誤,忽然發現Bzip2Codec這個類名與編譯出來的類名不一樣。編譯出來的是BZip2Codec。然後修改core-site.xml中的:
<property>
<name>io.compression.codecs</name> <value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
重起hadoop後,問題照舊。
再忽然想起hbase中還有一份core-site.xml,修改完重起,OK,問題解決。
真是個慘痛的教訓,一個字母大小寫耗了好幾天時間。