Tinkerpop + JanusGraph + Hbase簡單實現
〇、機器環境說明
192.168.1.2 master 安裝配置好jdk1.8;擁有一套正常運行的Hadoop+Hbase+Zookeeper集羣. ... 集羣其他信息自定義.
注：文檔中的主機IP，主機名稱，安裝目錄僅供參考，請以實際環境進行調整。
Apache TinkerPop™是圖形數據庫（OLTP）和圖形分析系統（OLAP）的圖形計算框架。
JanusGraph是一個可擴展的圖形數據庫，專門用於存儲和查詢包含數千億個分佈在多機羣集中的頂點和邊緣的圖形。JanusGraph是一個事務數據庫，可以支持數千個併發用戶實時執行復雜的圖形遍歷。
一、Hadoop+Hbase+Zookeeper的集羣搭建
此過程雖不麻煩, 但一時講清楚, 還是很費時間的, 在這就省略了.
二、創建目錄, 下載Tinkerpop
根據JanusGraph官方文檔<<Table B.1. Version Compatibility Matrix>>提供的信息, 在此次搭建中, 採用Tinkerpop 3.2.7, JanusGraph 0.2.0, 相關地址http://docs.janusgraph.org/latest/version-compat.html.
1. 創建目錄
[root@master ~]# mkdir -p /usr/local/program [root@master ~]# cd /usr/local/program/ [root@master program]# pwd /usr/local/programe2. 下載Tinkerpop
[root@master program]# wget http://www-eu.apache.org/dist/tinkerpop/3.2.7/apache-tinkerpop-gremlin-console-3.2.7-bin.zip # console, 相關地址http://tinkerpop.apache.org/. [root@master program]# wget http://mirror.bit.edu.cn/apache/tinkerpop/3.2.7/apache-tinkerpop-gremlin-server-3.2.7-bin.zip # server,相關地址http://tinkerpop.apache.org/.3. 解壓Tinkerpop
[root@master program]# unzip apache-tinkerpop-gremlin-server-3.2.7-bin.zip [root@master program]# unzip apache-tinkerpop-gremlin-console-3.2.7-bin.zip [root@master program]# ls -1 apache-tinkerpop-gremlin-console-3.2.7 apache-tinkerpop-gremlin-server-3.2.7 apache-tinkerpop-gremlin-console-3.2.7-bin.zip apache-tinkerpop-gremlin-server-3.2.7-bin.zip
三、在Tinkerpop Server上安裝JanusGraph依賴
Note: 第四節有講述如何配置grapeConfig.xml的內容, 如果下載失敗, 可參考第四節.
1. 進入apache-tinkerpop-gremlin-server-3.2.7下載依賴
安裝方式: bin/gremlin-server.sh -i org.janusgraph janusgraph-all $VERSION, 在這裏下載JanusGraph的0.2.0版本.
[root@master apache-tinkerpop-gremlin-server-3.2.7]# bin/gremlin-server.sh -i org.janusgraph janusgraph-all 0.2.02. 重要提示(Important, 這個地方應該重點顯示, 爲了好看, 就這樣了)
下面引用JanusGraph官方文檔中7.4.2. Using TinkerPop Gremlin Server with JanusGraph的一句話:
The above command uses Groovy Grape and if it is not configured properly download errors may ensue. Please refer to this section of the TinkerPop documentation for more information around setting up ~/.groovy/grapeConfig.xml.
相關地址http://docs.janusgraph.org/latest/server.html.
大意就是:
以上命令使用Groovy Grape，如果沒有正確配置，可能會出現下載錯誤。有關設置的更多信息，請參閱TinkerPop文檔的這一部分~/.groovy/grapeConfig.xml。
上文提到的地址http://tinkerpop.apache.org/docs/3.2.6/reference/#gremlin-applications.
四、爲避免下載依賴失敗, 修改Groovy Grape
Note: Windows目錄: C:\Users\[User_Name]\.groovy; Linux目錄: ~/.groovy
修改內容如下:
1. 在上述目錄中創建grapeConfig.xml文件, 或目錄不存在, 先創建目錄.
2. 修改文件內容如下:
<ivysettings>
<settings defaultResolver="downloadGrapes"/>
<property name="m2-pattern" value="${user.home}/.m2/repository/org/apache/tinkerpop/[module]/[revision]/[module]-[revision](-[classifier]).[ext]" />
<property name="m2-pattern-ivy" value="${user.home}/.m2/repository/org/apache/tinkerpop/[module]/[revision]/[module]-[revision](-[classifier]).pom" />
<caches>
<cache name="nocache" useOrigin="true" />
</caches>
<resolvers>
<chain name="downloadGrapes">
<filesystem name="local-maven2" checkmodified="true" changingPattern=".*" changingMatcher="regexp" m2compatible="true" cache="nocache">
<artifact pattern="${m2-pattern}"/>
<ivy pattern="${m2-pattern-ivy}"/>
</filesystem>
<filesystem name="cachedGrapes">
<ivy pattern="${user.home}/.groovy/grapes/[organisation]/[module]/ivy-[revision].xml"/>
<artifact pattern="${user.home}/.groovy/grapes/[organisation]/[module]/[type]s/[artifact]-[revision].[ext]"/>
</filesystem>
<ibiblio name="ibiblio" m2compatible="true"/>
<ibiblio name="local" root="file:${user.home}/.m2/repository/" m2compatible="true"/>
<ibiblio name="oracle" root="http://download.oracle.com/maven" m2compatible="true"/>
</chain>
</resolvers>
</ivysettings>
3. 重新執行第三節的內容.
五、參數設置(master節點操作)
1. 在${TINKERPOP_HOME}/conf/目錄創建janusgraph-hbase-server.properties, 添加內容如下:
gremlin.graph=org.janusgraph.core.JanusGraphFactory storage.backend=hbase # 存儲方式爲Hbase. storage.hostname=localhost # Zookeeper地址, 需要根據實際地址修改, 多臺用','間隔. # 以下信息可參考, 不影響下面步驟. cache.db-cache = true cache.db-cache-clean-wait = 20 cache.db-cache-time = 180000 cache.db-cache-size = 0.5 # 以下信息可參考, 用於配置ElasticSearch索引信息, 不影響下面步驟. index.search.backend=elasticsearch # 索引方式. index.search.hostname=localhost # ElasticSearch主機地址, 多臺用','間隔. index.search.port=9200 # ElasticSearch通信端口. index.search.elasticsearch.client-only=false
2. 在${TINKERPOP_HOME}/conf/目錄創建janusgraph-gremlin-server.yaml, 添加內容如下:
host: 0.0.0.0
port: 8182
scriptEvaluationTimeout: 300000
channelizer: org.apache.tinkerpop.gremlin.server.channel.WebSocketChannelizer
graphs: {
graph: conf/janusgraph-hbase-server.properties
}
plugins:
- janusgraph.imports
scriptEngines: {
gremlin-groovy: {
imports: [java.lang.Math],
staticImports: [java.lang.Math.PI],
scripts: [scripts/empty-sample.groovy]}}
serializers:
- { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}
- { className: org.apache.tinkerpop.gremlin.driver.ser.GryoLiteMessageSerializerV1d0, config: {ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}
- { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0, config: { serializeResultToString: true }}
- { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV1d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistryV1d0] }}
- { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV2d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}
- { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistryV1d0] }}
processors:
- { className: org.apache.tinkerpop.gremlin.server.op.session.SessionOpProcessor, config: { sessionTimeout: 28800000 }}
- { className: org.apache.tinkerpop.gremlin.server.op.traversal.TraversalOpProcessor, config: { cacheExpirationTime: 600000, cacheMaxSize: 1000 }}
metrics: {
consoleReporter: {enabled: true, interval: 180000},
csvReporter: {enabled: true, interval: 180000, fileName: /tmp/gremlin-server-metrics.csv},
jmxReporter: {enabled: true},
slf4jReporter: {enabled: true, interval: 180000},
gangliaReporter: {enabled: false, interval: 180000, addressingMode: MULTICAST},
graphiteReporter: {enabled: false, interval: 180000}}
maxInitialLineLength: 4096
maxHeaderSize: 8192
maxChunkSize: 8192
maxContentLength: 65536
maxAccumulationBufferComponents: 1024
resultIterationBatchSize: 64
writeBufferLowWaterMark: 32768
writeBufferHighWaterMark: 65536
3. 在${TINKERPOP_HOME}/scripts/目錄創建empty-sample.groovy,(此文件默認存在, 可直接使用.) 添加內容如下:
def globals = [:]
globals << [g : graph.traversal()]
六、運行Tinkerpop Server
[root@master apache-tinkerpop-gremlin-server-3.2.7]# bin/gremlin-server.sh conf/janusgraph-gremlin-server.yaml
1. 當顯示如下內容時, 說明啓動成功:
...... INFO] GremlinServer - Executing start up LifeCycleHook [INFO] Logger$info - Executed once at startup of Gremlin Server. [INFO] AbstractChannelizer - Configured application/vnd.gremlin-v1.0+gryo with org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0 [WARN] AbstractChannelizer - The org.apache.tinkerpop.gremlin.driver.ser.GryoLiteMessageSerializerV1d0 serialization class is deprecated. [INFO] AbstractChannelizer - Configured application/vnd.gremlin-v1.0+gryo-lite with org.apache.tinkerpop.gremlin.driver.ser.GryoLiteMessageSerializerV1d0 [INFO] AbstractChannelizer - Configured application/vnd.gremlin-v1.0+gryo-stringd with org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0 [INFO] AbstractChannelizer - Configured application/vnd.gremlin-v1.0+json with org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV1d0 [INFO] AbstractChannelizer - Configured application/vnd.gremlin-v2.0+json with org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV2d0 [INFO] AbstractChannelizer - Configured application/json with org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0 [INFO] GremlinServer$1 - Gremlin Server configured with worker thread pool of 1, gremlin pool of 32 and boss thread pool of 1. [INFO] GremlinServer$1 - Channel started at port 8182.
2. 當提示以下內容時, 修改文件, 去除\t
[ERROR] GremlinServer - Configuration file at conf/janusgraph-gremlin-server.yaml could not be found or parsed properly. [while scanning for the next token found character '\t(TAB)' that cannot start any token. (Do not use \t(TAB) for indentation) in 'reader', line 6, column 1: graph: conf/janusgraph-hbase-se ... ^ ]
七、通過Tinkerpop Console測試
1. 進入apache-tinkerpop-gremlin-console-3.2.7目錄, 運行:
[root@master apache-tinkerpop-gremlin-console-3.2.7]# ./bin/gremlin.sh Feb 08, 2018 4:57:21 PM java.util.prefs.FileSystemPreferences$1 run INFO: Created user preferences directory. \,,,/ (o o) -----oOOo-(3)-oOOo----- plugin activated: tinkerpop.server plugin activated: tinkerpop.utilities plugin activated: tinkerpop.tinkergraph gremlin> :remote connect tinkerpop.server conf/remote.yaml ==>Configured localhost/127.0.0.1:8182 gremlin> :> g.V().count() ==> 0
運行無阻時, 說明部署完成. 可以繼續往下進行.