Pig初探

Pig環境安裝

pig的安裝非常簡單,將pig-0.14.0.tar.gz解壓到合適的目錄下

tar -zxvf pig-0.14.0.tar.gz

修改環境變量:

#pig  
export PIG_HOME=/usr/local/cloud/pig-0.11.1/pig-0.11.1
export PATH=.:$PIG_HOME/bin:$PATH
export PIG_CLASSPATH=$HADOOP_HOME/etc/hadoop   #爲了讓pig識別你的hadoop版本,如果只想用pig的本地模式,不用配置這項。
使環境變量修改生效:
source /etc/profile


本地模式啓動方式pig -x local

[root@leaf pig-0.11.1]# pig -x local
2014-11-24 07:50:19,622 [main] INFO  org.apache.pig.Main - Apache Pig version 0.11.1 (r1459641) compiled Mar 22 2013, 02:13:53
2014-11-24 07:50:19,622 [main] INFO  org.apache.pig.Main - Logging error messages to: /usr/local/cloud/pig-0.11.1/pig-0.11.1/logs/pig_1416844219621.log
2014-11-24 07:50:19,663 [main] INFO  org.apache.pig.impl.util.Utils - Default bootup file /root/.pigbootup not found
2014-11-24 07:50:19,901 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
2014-11-24 07:50:19,903 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2014-11-24 07:50:19,907 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-11-24 07:50:19,907 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2014-11-24 07:50:20,188 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2014-11-24 07:50:20,190 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
修改pig默認的日誌存放目錄
pig解壓後的目錄下pig-0.14.0/conf/pig.properties中,修改爲合適的目錄:

pig.logfile=/usr/local/cloud/pig-0.11.1/pig-0.11.1/logs

pig的mapreduce啓動方式(必須啓動hadoop):

[root@leaf pig-0.11.1]# pig
2014-11-24 07:57:16,370 [main] INFO  org.apache.pig.Main - Apache Pig version <span style="font-family: Arial, Helvetica, sans-serif;">0.11.1</span><span style="font-family: Arial, Helvetica, sans-serif;">(r1459641) compiled Mar 22 2013, 02:13:53</span>
2014-11-24 07:57:16,370 [main] INFO  org.apache.pig.Main - Logging error messages to: /usr/local/cloud/pig-0.11.1/pig-0.11.1/logs/pig_1416844636369.log
2014-11-24 07:57:16,410 [main] INFO  org.apache.pig.impl.util.Utils - Default bootup file /root/.pigbootup not found
2014-11-24 07:57:16,681 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://192.168.1.240:9000
2014-11-24 07:57:16,684 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2014-11-24 07:57:16,685 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-11-24 07:57:17,634 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
測試是否安裝成功:

grunt> ls hdfs:///
hdfs://192.168.1.240:9000/source	<dir>
hdfs://192.168.1.240:9000/testdata	<dir>
hdfs://192.168.1.240:9000/tmp	<dir>
hdfs://192.168.1.240:9000/user	<dir>
hdfs://192.168.1.240:9000/usr	<dir>
pig常見命令:

grunt> cd /user/root/output
grunt> ls
hdfs://192.168.1.240:9000/user/root/output/_policy<r 1>	194
hdfs://192.168.1.240:9000/user/root/output/clusteredPoints	<dir>
hdfs://192.168.1.240:9000/user/root/output/clusters-0	<dir>
hdfs://192.168.1.240:9000/user/root/output/clusters-1	<dir>
hdfs://192.168.1.240:9000/user/root/output/clusters-10-final	<dir>
hdfs://192.168.1.240:9000/user/root/output/clusters-2	<dir>
hdfs://192.168.1.240:9000/user/root/output/clusters-3	<dir>
hdfs://192.168.1.240:9000/user/root/output/clusters-4	<dir>
hdfs://192.168.1.240:9000/user/root/output/clusters-5	<dir>
hdfs://192.168.1.240:9000/user/root/output/clusters-6	<dir>
hdfs://192.168.1.240:9000/user/root/output/clusters-7	<dir>
hdfs://192.168.1.240:9000/user/root/output/clusters-8	<dir>
hdfs://192.168.1.240:9000/user/root/output/clusters-9	<dir>
hdfs://192.168.1.240:9000/user/root/output/data	<dir>
hdfs://192.168.1.240:9000/user/root/output/random-seeds	<dir>
grunt> pig -help
2014-11-24 08:04:11,969 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Encountered " <IDENTIFIER> "pig "" at line 1, column 1.
Was expecting one of:
    <EOF> 
    "cat" ...
    "clear" ...
    "fs" ...
    "sh" ...
    "cd" ...
    "cp" ...
    "copyFromLocal" ...
    "copyToLocal" ...
    "dump" ...
    "describe" ...
    "aliases" ...
    "explain" ...
    "help" ...
    "history" ...
    "kill" ...
    "ls" ...
    "mv" ...
    "mkdir" ...
    "pwd" ...
    "quit" ...
    "register" ...
    "rm" ...
    "rmf" ...
    "set" ...
    "illustrate" ...
    "run" ...
    "exec" ...
    "scriptDone" ...
    "" ...
    "" ...
    <EOL> ...
    ";" ...




發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章