部署Hbase + Phoenix，並通過DBeaver連接Phoenix

1.Hbase安裝步驟
https://blog.csdn.net/muyingmiao/article/details/103002598
2 Phoenix的安裝
2.1 Phoenix 的官網
http://phoenix.apache.org/
2.2 Phoenix 安裝文件地址
http://www.apache.org/dyn/closer.lua/phoenix/
https://mirrors.tuna.tsinghua.edu.cn/apache/phoenix/
如果不想自己編譯對應的版本，可以使用官網相應的版本，比如使用CDH5.16.1的，可以選擇如下版本：http://www.apache.org/dyn/closer.lua/phoenix/apache-phoenix-4.14.0-cdh5.14.2/bin/apache-phoenix-4.14.0-cdh5.14.2-bin.tar.gz
2.3 Phoenix安裝步驟

Download and expand the latest phoenix-[version]-bin.tar.
Add the phoenix-[version]-server.jar to the classpath of all HBase region server and ~~master~~ and remove any previous version. An easy way to do this is to copy it into the HBase lib directory (use phoenix-core-[version].jar for Phoenix 3.x)
Restart HBase.
Add the phoenix-[version]-client.jar to the classpath of any Phoenix client.（把clinet的jar包添加到海狸）

具體操作如下
2.3.1.下載（CDH環境：Hadoop Hbase + 4.14.0-cdh5.11.2（比如這個版本））
2.3.2.將phoenix的server的依賴包拷貝到hbase的lib下

[hadoop@hadoop002 phoenix]$ cp phoenix-4.14.0-cdh5.14.2-server.jar ../hbase-1.2.0-cdh5.15.1/lib/

2.3.3.修改hbase的hbase-site.xml配置文件，添加如下4個參數

	<property>
        <name>hbase.table.sanity.checks</name>
        <value>false</value>
    </property>
    <property>
        <!-- phoenix支持索引 -->
        <name>hbase.regionserver.wal.codec</name>
        <value>org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec</value>
    </property>
    <property>
        <name>phoenix.schema.isNamespaceMappingEnabled</name>
        <value>true</value>
    </property>
    <property>
        <name>phoenix.schema.mapSystemTablesToNamespace</name>
        <value>true</value>
    </property>

2.3.4.重啓Hbase

[hadoop@hadoop002 bin]$ ./stop-hbase.sh 
stopping hbase....................
[hadoop@hadoop002 bin]$ ./start-hbase.sh 
starting master, logging to /home/hadoop/app/hbase-1.2.0-cdh5.15.1//logs/hbase-hadoop-master-hadoop002.out
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
hadoop002: starting regionserver, logging to /home/hadoop/app/hbase-1.2.0-cdh5.15.1/bin/../logs/hbase-hadoop-regionserver-hadoop002.out
hadoop002: Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
hadoop002: Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
[hadoop@hadoop002 bin]$ jps
9874 SecondaryNameNode
21667 QuorumPeerMain
9398 NameNode
17094 HMaster
10151 NodeManager
9513 DataNode
17561 Jps
22780 nacos-server.jar
17404 HRegionServer
10030 ResourceManager
[hadoop@hadoop002 bin]$

2.3.5.拷貝Hbase的配置文件hbase-site.xml到phoenix的bin目錄下，替換之前phoenix的hbase-site.xml
建議做軟連接

[hadoop@hadoop002 bin]$ ln -s /home/hadoop/app/hbase-1.2.0-cdh5.15.1/conf/hbase-site.xml hbase-site.xml
[hadoop@hadoop002 bin]$ ll
total 144
drwxrwxr-x 2 hadoop hadoop    24 Jun  5  2018 argparse-1.4.0
drwxrwxr-x 4 hadoop hadoop    96 Jun  5  2018 config
-rw-rw-r-- 1 hadoop hadoop 32864 Jun  5  2018 daemon.py
-rwxrwxr-x 1 hadoop hadoop  1881 Jun  5  2018 end2endTest.py
-rw-rw-r-- 1 hadoop hadoop  1621 Jun  5  2018 hadoop-metrics2-hbase.properties
-rw-rw-r-- 1 hadoop hadoop  3056 Jun  5  2018 hadoop-metrics2-phoenix.properties
lrwxrwxrwx 1 hadoop hadoop    58 Mar 21 09:30 hbase-site.xml -> /home/hadoop/app/hbase-1.2.0-cdh5.15.1/conf/hbase-site.xml

如果是CDH的HDFS的HA環境的話，要把/etc/hdfs/conf/core-site.xml /etc/hdfs/conf/hdfs-site.xml 兩個文件拷貝到bin目錄下
2.3.6.phoenix初始化，注意python的版本要用默認的2.7.5

[hadoop@hadoop002 bin]$ ./sqlline.py hadoop002:2181
Setting property: [incremental, false]
Setting property: [isolation, TRANSACTION_READ_COMMITTED]
issuing: !connect jdbc:phoenix:hadoop002:2181 none none org.apache.phoenix.jdbc.PhoenixDriver
Connecting to jdbc:phoenix:hadoop002:2181
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/app/apache-phoenix-4.14.0-cdh5.14.2-bin/phoenix-4.14.0-cdh5.14.2-client.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/app/hadoop-2.6.0-cdh5.15.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
20/03/21 09:45:15 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Connected to: Phoenix (version 4.14)
Driver: PhoenixEmbeddedDriver (version 4.14)
Autocommit status: true
Transaction isolation: TRANSACTION_READ_COMMITTED
Building list of tables and columns for tab-completion (set fastconnect to true to skip)...
133/133 (100%) Done
Done
sqlline version 1.2.0
0: jdbc:phoenix:hadoop002:2181>

2.3.7 Phoenix Schema 映射及配置注意
http://phoenix.apache.org/namspace_mapping.html
在Phoenix4.8.0版本之前，Phoenix的schema和Hbase 的Namespace是沒有開啓映射參數的。

Property	Description	Default
phoenix.schema.isNamespaceMappingEnabled	If it is enabled, then the tables created with schema will be mapped to namespace.This needs to be set at client and server both. if set once, should not be rollback. Old client will not work after this property is enabled.	false
phoenix.schema.mapSystemTablesToNamespace	Enabling this property will take affect when phoenix.connection.isNamespaceMappingEnabled is also set to true. If it is enabled, SYSTEM tables if present will automatically migrated to SYSTEM namespace. And If set to false , then system tables will be created in default namespace only. This needs to be set at client and server both.	true

client端和server端都要設置（如下2個參數已經在上述的配置文件中添加），最好是clinet端使用server的配置文件的軟連接。

    <property>
        <name>phoenix.schema.isNamespaceMappingEnabled</name>
        <value>true</value>
    </property>
    <property>
        <name>phoenix.schema.mapSystemTablesToNamespace</name>
        <value>true</value>
    </property>

設置好後，在phoenix中創建schema後

0: jdbc:phoenix:hadoop002:2181> !tables
+------------+--------------+-------------+---------------+----------+------------+----------------------------+-----------------+--------------+-----------------+----+
| TABLE_CAT  | TABLE_SCHEM  | TABLE_NAME  |  TABLE_TYPE   | REMARKS  | TYPE_NAME  | SELF_REFERENCING_COL_NAME  | REF_GENERATION  | INDEX_STATE  | IMMUTABLE_ROWS  | SA |
+------------+--------------+-------------+---------------+----------+------------+----------------------------+-----------------+--------------+-----------------+----+
|            | TEST         | TEST_IDX    | INDEX         |          |            |                            |                 | ACTIVE       | false           | nu |
|            | SYSTEM       | CATALOG     | SYSTEM TABLE  |          |            |                            |                 |              | false           | nu |
|            | SYSTEM       | FUNCTION    | SYSTEM TABLE  |          |            |                            |                 |              | false           | nu |
|            | SYSTEM       | LOG         | SYSTEM TABLE  |          |            |                            |                 |              | true            | 32 |
|            | SYSTEM       | SEQUENCE    | SYSTEM TABLE  |          |            |                            |                 |              | false           | nu |
|            | SYSTEM       | STATS       | SYSTEM TABLE  |          |            |                            |                 |              | false           | nu |
|            | TEST         | TEST        | TABLE         |          |            |                            |                 |              | false           | nu |
+------------+--------------+-------------+---------------+----------+------------+----------------------------+-----------------+--------------+-----------------+----+
0: jdbc:phoenix:hadoop002:2181>

在Hbase 的namespace中也會展示相應的大寫的名字
phoenix創建schema

0: jdbc:phoenix:hadoop002:2181> create schema muyingmiao;

在Hbase的shell下查看

hbase(main):002:0> list_namespace
NAMESPACE                                                                                                                                                               
MUYINGMIAO                                                                                                                                                              
SYSTEM                                                                                                                                                                  
TEST                                                                                                                                                                    
default                                                                                                                                                                 
hbase                                                                                                                                                                   
ruozedata                                                                                                                                                               
6 row(s) in 0.0140 seconds

3.通過DBeaver連接Phoenix
3.1下載並安裝DBeaver
3.2 配置DBeaver的JDBC

3.3 將phoenix-4.14.0-cdh5.14.2-client.jar 下載到本地，在Dbserver中配置驅動設置

3.5 在C:\Windows\System32\drivers\etc\hosts文件中配置
ip hostname
3.6 測試連接

3.7 如果是雲服務器，要打開60000,60010,2181,60020端口
如果報錯
Unexpected version format: 11.0.5
需要在dbeave.ini文件中配置本地的JDK路徑，比如我自己的

-vm 
C:/app/Java/jdk1.8.0_221/bin
-startup
plugins/org.eclipse.equinox.launcher_1.5.600.v20191014-2022.jar
--launcher.library
plugins/org.eclipse.equinox.launcher.win32.win32.x86_64_1.1.1100.v20190907-0426
-vmargs
-XX:+IgnoreUnrecognizedVMOptions
--add-modules=ALL-SYSTEM
-Xms64m
-Xmx1024m

3.8 Phoenix 一些shell命令
!tables :查看所有的系統表和用戶表
4 Phoenix的數據類型
http://phoenix.apache.org/language/datatypes.html
MySQL中的INT對應phoenix中的INTEGER
MySQL中的CHAR/VARCHAR的長度在phoenix要翻倍，防止數據同步到phoenix出現長度不夠的問題
5.在phoenix中創建schema和table、index
5.1 創建schema，table和index

0: jdbc:phoenix:hadoop002:2181> create schema test;
No rows affected (0.056 seconds)
0: jdbc:phoenix:hadoop002:2181> CREATE TABLE test.test ( id BIGINT not null primary key, name varchar(200), age integer);
No rows affected (2.341 seconds)
0: jdbc:phoenix:hadoop002:2181> upsert into test.test values(1,'user01',18);
1 row affected (0.04 seconds)
0: jdbc:phoenix:hadoop002:2181> select * from test.test;
+-----+---------+------+
| ID  |  NAME   | AGE  |
+-----+---------+------+
| 1   | user01  | 18   |
+-----+---------+------+
1 row selected (0.033 seconds)
0: jdbc:phoenix:hadoop002:2181> CREATE INDEX test_idx ON test.test(name, age);
2 rows affected (6.315 seconds)
0: jdbc:phoenix:hadoop002:2181> !tables
+------------+--------------+-------------+---------------+----------+------------+----------------------------+-----------------+--------------+-----------------+----+
| TABLE_CAT  | TABLE_SCHEM  | TABLE_NAME  |  TABLE_TYPE   | REMARKS  | TYPE_NAME  | SELF_REFERENCING_COL_NAME  | REF_GENERATION  | INDEX_STATE  | IMMUTABLE_ROWS  | SA |
+------------+--------------+-------------+---------------+----------+------------+----------------------------+-----------------+--------------+-----------------+----+
|            | TEST         | TEST_IDX    | INDEX         |          |            |                            |                 | ACTIVE       | false           | nu |
|            | SYSTEM       | CATALOG     | SYSTEM TABLE  |          |            |                            |                 |              | false           | nu |
|            | SYSTEM       | FUNCTION    | SYSTEM TABLE  |          |            |                            |                 |              | false           | nu |
|            | SYSTEM       | LOG         | SYSTEM TABLE  |          |            |                            |                 |              | true            | 32 |
|            | SYSTEM       | SEQUENCE    | SYSTEM TABLE  |          |            |                            |                 |              | false           | nu |
|            | SYSTEM       | STATS       | SYSTEM TABLE  |          |            |                            |                 |              | false           | nu |
|            | TEST         | TEST        | TABLE         |          |            |                            |                 |              | false           | nu |
+------------+--------------+-------------+---------------+----------+------------+----------------------------+-----------------+--------------+-----------------+----+
0: jdbc:phoenix:hadoop002:2181>

5.2 Hbase查詢的底層實現是從左到右字典查詢

0: jdbc:phoenix:hadoop002:2181> explain select age from test.test where name = 'user01';
+-------------------------------------------------------------------------------------+-----------------+----------------+--------------+
|                                        PLAN                                         | EST_BYTES_READ  | EST_ROWS_READ  | EST_INFO_TS  |
+-------------------------------------------------------------------------------------+-----------------+----------------+--------------+
| CLIENT 1-CHUNK PARALLEL 1-WAY ROUND ROBIN RANGE SCAN OVER TEST:TEST_IDX ['user01']  | null            | null           | null         |
|     SERVER FILTER BY FIRST KEY ONLY                                                 | null            | null           | null         |
+-------------------------------------------------------------------------------------+-----------------+----------------+--------------+
2 rows selected (0.045 seconds)
0: jdbc:phoenix:hadoop002:2181> explain select name from test.test where age = 18;
+-------------------------------------------------------------------------+-----------------+----------------+--------------+
|                                  PLAN                                   | EST_BYTES_READ  | EST_ROWS_READ  | EST_INFO_TS  |
+-------------------------------------------------------------------------+-----------------+----------------+--------------+
| CLIENT 1-CHUNK PARALLEL 1-WAY ROUND ROBIN FULL SCAN OVER TEST:TEST_IDX  | null            | null           | null         |
|     SERVER FILTER BY FIRST KEY ONLY AND TO_INTEGER("AGE") = 18          | null            | null           | null         |
+-------------------------------------------------------------------------+-----------------+----------------+--------------+
2 rows selected (0.03 seconds)
0: jdbc:phoenix:hadoop002:2181>

在Hbase中查看

hbase(main):004:0> list_namespace
NAMESPACE                                                                                                                                                               
SYSTEM                                                                                                                                                                  
TEST                                                                                                                                                                    
default                                                                                                                                                                 
hbase                                                                                                                                                                   
ruozedata                                                                                                                                                               
5 row(s) in 0.0230 seconds

hbase(main):005:0> list_namespace_tables 'TEST'
TABLE                                                                                                                                                                   
TEST                                                                                                                                                                    
1 row(s) in 0.0280 seconds

hbase(main):006:0>

其他擴展知識：
Apache Hbase
1.https://hbase.apache.org/book.html#quickstart
2.Hbase的集羣配置文件hbase-site.xml

<configuration>
    <!--hbase.rootdir的前端與$HADOOP_HOME/conf/core-site.xml的fs.defaultFS一致 -->
    <property>
        <name>hbase.rootdir</name>
        <value>hdfs://mycluster/hbase</value>
    </property>
    <property>
        <name>hbase.cluster.distributed</name>
        <value>true</value>
    </property>

    <!--本地文件系統的臨時文件夾。可以修改到一個更爲持久的目錄上。(/tmp會在重啓時清除) -->
    <property>
        <name>hbase.tmp.dir</name>
        <value>/hadoop/hbase/tmp</value>
    </property>

    <!--如果只設置單個 Hmaster，那麼 hbase.master 屬性參數需要設置爲 master5:60000 (主機名:60000) -->
    <!--如果要設置多個 Hmaster，那麼我們只需要提供端口 60000，因爲選擇真正的 master 的事情會有 zookeeper 去處理 -->
    <property>
        <name>hbase.master</name>
        <value>60000</value>
    </property>

    <!--這個參數用戶設置 ZooKeeper 快照的存儲位置，默認值爲 /tmp，顯然在重啓的時候會清空。因爲筆者的 ZooKeeper 是獨立安裝的，所以這裏路徑是指向了 $ZOOKEEPER_HOME/conf/zoo.cfg 中 dataDir 所設定的位置 -->
    <property>
        <name>hbase.zookeeper.property.dataDir</name>
        <value>/hadoop/zookeeper/data</value>
    </property>

    <property>
        <name>hbase.zookeeper.quorum</name>
        <value>sht-sgmhadoopdn-01,sht-sgmhadoopdn-02,sht-sgmhadoopdn-03</value>
    </property>
    <!--表示客戶端連接 ZooKeeper 的端口 -->
    <property>
        <name>hbase.zookeeper.property.clientPort</name>
        <value>2181</value>
    </property>
    !--ZooKeeper 會話超時。Hbase 把這個值傳遞改 zk 集羣，向它推薦一個會話的最大超時時間 -->
    <property>
        <name>zookeeper.session.timeout</name>
        <value>120000</value>
    </property>

    <!--當 regionserver 遇到 ZooKeeper session expired ， regionserver 將選擇 restart 而不是 abort -->
    <property>
        <name>hbase.regionserver.restart.on.zk.expire</name>
        <value>true</value>
    </property>
</configuration>

3.單機部署配置文件

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
    <!--hbase.rootdir的前端與$HADOOP_HOME/conf/core-site.xml的fs.defaultFS一致 -->
    <property>
        <name>hbase.rootdir</name>
        <value>hdfs://hostname:9000/hbase</value>
    </property>
    <property>
        <name>hbase.cluster.distributed</name>
        <value>true</value>
    </property>
    
    <!--本地文件系統的臨時文件夾。可以修改到一個更爲持久的目錄上。(/tmp會在重啓時清除) -->
    <property>
        <name>hbase.tmp.dir</name>
        <value>/home/hadoop/tmp/hbase</value>
    </property>
    
    <!--如果只設置單個 Hmaster，那麼 hbase.master 屬性參數需要設置爲 master5:60000 (主機名:60000) -->
    <!--如果要設置多個 Hmaster，那麼我們只需要提供端口 60000，因爲選擇真正的 master 的事情會有 zookeeper 去處理 -->
    <property>
        <name>hbase.master</name>
        <value>60000</value>
    </property>

    <!--這個參數用戶設置 ZooKeeper 快照的存儲位置，默認值爲 /tmp，顯然在重啓的時候會清空。因爲筆者的 ZooKeeper 是獨立安裝的，所以這裏路徑是指向了 $ZOOKEEPER_HOME/conf/zoo.cfg 中 dataDir 所設定的位置 -->
    <property>
        <name>hbase.zookeeper.property.dataDir</name>
        <value>/home/hadoop/data/zookeeper</value>
    </property>

    <property>
        <name>hbase.zookeeper.quorum</name>
        <value>hostname</value>
    </property>
    <!--表示客戶端連接 ZooKeeper 的端口 -->
    <property>
        <name>hbase.zookeeper.property.clientPort</name>
        <value>2181</value>
    </property>
        <!--ZooKeeper 會話超時。Hbase 把這個值傳遞改 zk 集羣，向它推薦一個會話的最大超時時間 -->
    <property>
        <name>zookeeper.session.timeout</name>
        <value>120000</value>
    </property>

    <!--當 regionserver 遇到 ZooKeeper session expired ， regionserver 將選擇 restart 而不是 abort -->
    <property>
        <name>hbase.regionserver.restart.on.zk.expire</name>
        <value>true</value>
    </property>
</configuration>

部署Hbase + Phoenix，並通過DBeaver連接Phoenix

DAPPER 事務 TRANSACTION

Java中線程的創建方式

一鍵自動化博客發佈工具,chrome和firfox詳細配置

Linux命令（vi，權限，軟連接，系統命令，壓縮解壓，定時任務，後臺運行）

kudu的基本操作（查看集羣狀態，APT操作，impala操作，spark整合）

部署Hbase + Phoenix，並通過DBeaver連接Phoenix

一臺機器搭建Kafka集羣及Kafka的簡單操作（僞分佈式）

Hbase通過Phoenix創建二級索引

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結