mycat介紹
mmycat是阿里開源的一個分佈式數據庫中間層。
- 作用:
實現數據庫的讀寫分離
支持讀負載均衡、後端mysql高可用
數據庫垂直拆分、水平拆分
- 應用場景:
需要讀寫分離
需要分庫分表
多租戶
數據統計系統
HBASE替代
同樣方式查詢多種數據庫
- 關鍵特性:
支持SQL92標準
支持mysql集羣
支持jdbc連接數據庫
支持nosql數據庫
支持自動故障切換、高可用性
支持全局表、全局序列號
支持一致性hash分片、基於ER關係的分片策略
mycat部署
- 環境:
192.168.1.231 mycat
192.168.1.232 mysql master
192.168.1.233 mysql slave
- 安裝java:
mkdir /software && cd /software #將安裝包放到該目錄下
tar xf jdk-8u231-linux-x64.tar.gz && mv jdk1.8.0_231/ /usr/local/jdk
- 下載mycat:
wget http://dl.mycat.io/1.6.7.4/Mycat-server-1.6.7.4-release/Mycat-server-1.6.7.4-release-20200105164103-linux.tar.gz
tar xf Mycat-server-1.6.7.4-release-20200105164103-linux.tar.gz
mv mycat/ /usr/local/
useradd mycat
chown -R mycat:mycat /usr/local/mycat
vim /etc/profile
JAVA_HOME=/usr/local/jdk
MYCAT_HOME=/usr/local/mycat
PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$MYCAT_HOME/bin
CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/jre/lib
export JAVA_HOME MYCAT_HOME PATH CLASSPATH
source !$
java -version
- 啓動mycat:
su - mycat
cd /usr/local/mycat/
startup_nowrap.sh
-bash: /usr/local/mycat/bin/startup_nowrap.sh: /bin/sh^M: 壞的解釋器: 沒有那個文件或目錄
sed -i 's/\r$//' bin/startup_nowrap.sh
startup_nowrap.sh
ps -aux |grep java |grep mycat
cat logs/console.log
MyCAT Server startup successfully. see logs in logs/mycat.log
- mycat配置說明:
schema.xml 用於配置邏輯庫表及數據節點
<schema><table></table></schema> 定義邏輯庫表
<dataNode></dataNode> 定義數據節點
<dataHost></dataHost> 定義數據節點的物理數據源
rule.xml 用於配置表的分片規則
<tableRule name=""></tableRule> 定義表使用的分片規則
<function name=""></function> 定義分片算法
server.xml 用於配置服務器權限
<system><property name=""></property></system> 定義系統配置
<user></user> 定義連接mycat的用戶
- 配置讀寫分離:
首先mysql機器配置好mysql主從複製,並在mysql master上創建user_db
庫及user_db.customer_login
表。
CREATE database user_db;
USE user_db;
CREATE TABLE customer_login(customer_id SMALLINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,login_name VARCHAR(40),password VARCHAR(40),user_stats TINYINT UNSIGNED);
vim conf/schema.xml #修改
<schema name="USERDB" checkSQLschema="false" sqlMaxLimit="100" dataNode="dn1"></schema>
<dataNode name="dn1" dataHost="node1" database="user_db" />
<dataHost name="node1" maxCon="1000" minCon="10" balance="1"
writeType="0" dbType="mysql" dbDriver="native" switchType="1" slaveThreshold="100">
<heartbeat>select user()</heartbeat>
<writeHost host="192.168.1.232" url="192.168.1.232:3306" user="root"
password="123456789">
<readHost host="192.168.1.233" url="192.168.1.233:3306" user="root"
password="123456789"/>
</writeHost>
<writeHost host="192.168.1.233" url="192.168.1.233:3306" user="root" password="123456789"/>
</dataHost>
vim conf/server.xml #修改,其餘配置不變
<system>
<property name="serverPort">3306</property> <property name="managerPort">9066</property>
</system>
<user name="root" defaultAccount="true">
<property name="password">123456</property>
<property name="schemas">USERDB</property>
<property name="defaultSchema">USERDB</property>
</user>
<user name="user">
<property name="password">123456</property>
<property name="schemas">USERDB</property>
<property name="readOnly">true</property>
<property name="defaultSchema">USERDB</property>
</user>
配置完成,重啓mycat,
mycat restart
- 測試讀寫分離:
vim /software/init_data.py
#! /usr/bin/env python
import MySQLdb
from faker import Faker
try:
conn = MySQLdb.connect(host='127.0.0.1', user='root', passwd='123456', db='USERDB', charset="utf8")
cursor= conn.cursor(MySQLdb.cursors.DictCursor)
except MySQLdb.Error, e:
print "Error %d: %s \n" % (e.args[0], e.args[1])
try:
faker=Faker()
#write data
for i in range(0,10):
InSQL="""
insert into customer_login(login_name, password, user_stats) VALUES('%s',md5('%s'),1);
"""%(faker.name(), faker.ean13())
print InSQL
cursor.execute(InSQL)
cursor.execute('commit')
#read data
for i in range(1,11):
SQLstr="""
select login_name from customer_login where customer_id = %d;
"""%(i)
cursor.execute(SQLstr)
result = cursor.fetchall()
print result
except MySQLdb.Error, e:
print "Error %d: %s \n" % (e.args[0], e.args[1])
finally:
cursor.close()
conn.close()
yum install -y python-pip mysql-devel python-devel
pip install --upgrade pip
pip install MySQL-python faker
python /software/init_data.py
執行結果,
insert into customer_login(login_name, password, user_stats) VALUES('Angela Hernandez',md5('4151032118150'),1);
insert into customer_login(login_name, password, user_stats) VALUES('Cathy Howell',md5('8122034728520'),1);
insert into customer_login(login_name, password, user_stats) VALUES('Gregory Taylor',md5('2661518375233'),1);
insert into customer_login(login_name, password, user_stats) VALUES('Edward Gilmore',md5('8421803476830'),1);
insert into customer_login(login_name, password, user_stats) VALUES('Jeremy Dawson',md5('2803656252087'),1);
insert into customer_login(login_name, password, user_stats) VALUES('Michael Walton',md5('2220300773833'),1);
insert into customer_login(login_name, password, user_stats) VALUES('Debra Williams',md5('0101458681817'),1);
insert into customer_login(login_name, password, user_stats) VALUES('Kimberly Cooper',md5('9468150847615'),1);
insert into customer_login(login_name, password, user_stats) VALUES('Andrew Taylor',md5('4304499804351'),1);
insert into customer_login(login_name, password, user_stats) VALUES('Teresa Edwards',md5('6814811117352'),1);
({'login_name': u'Angela Hernandez'},)
({'login_name': u'Cathy Howell'},)
({'login_name': u'Gregory Taylor'},)
({'login_name': u'Edward Gilmore'},)
({'login_name': u'Jeremy Dawson'},)
({'login_name': u'Michael Walton'},)
({'login_name': u'Debra Williams'},)
({'login_name': u'Kimberly Cooper'},)
({'login_name': u'Andrew Taylor'},)
({'login_name': u'Teresa Edwards'},)
在mycat主機上登錄mysql,故意插入一條錯誤數據,
mysql -uroot -p -h192.168.1.231 -P3306
use USERDB;
insert into customer_login(login_name, password, user_stats) VALUES('%s',md5('%s'));
ERROR 1136 (HY000): Column count doesn't match value count at row 1
tail -f logs/mycat.log
2020-01-20 17:27:53.000 WARN [$_NIOREACTOR-1-RW] (io.mycat.backend.mysql.nio.handler.SingleNodeHandler.backConnectionErr(SingleNodeHandler.java:284)) - execute sql err : errno:1136 Column count doesn't match value count at row 1 con:MySQLConnection@762087023 [id=7, lastTime=1579512472990, user=root, schema=user_db, old shema=user_db, borrowed=true, fromSlaveDB=false, threadId=63, charset=utf8, txIsolation=3, autocommit=true, attachment=dn1{insert into customer_login(login_name, password, user_stats) VALUES('%s',md5('%s'))}, respHandler=SingleNodeHandler [node=dn1{insert into customer_login(login_name, password, user_stats) VALUES('%s',md5('%s'))}, packetId=1], host=192.168.1.232, port=3306, statusSync=null, writeQueue=0, modifiedSQLExecuted=true] frontend host:192.168.1.231/45118/root
2020-01-20 17:27:53.003 ERROR [$_NIOREACTOR-1-RW] (io.mycat.net.FrontendConnection.writeErrMessage(FrontendConnection.java:210)) - ServerConnection [id=2, schema=USERDB, host=192.168.1.231, user=root,txIsolation=3, autocommit=true, schema=USERDB, executeSql=insert into customer_login(login_name, password, user_stats) VALUES('%s',md5('%s'))]Column count doesn't match value count at row 1java.lang.Thread .getStackTrace1559
io.mycat.net.FrontendConnection .getStack224
io.mycat.net.FrontendConnection .writeErrMessage210
io.mycat.backend.mysql.nio.handler.SingleNodeHandler .backConnectionErr311
io.mycat.backend.mysql.nio.handler.SingleNodeHandler .errorResponse272
io.mycat.backend.mysql.nio.MySQLConnectionHandler .handleErrorPacket168
io.mycat.backend.mysql.nio.MySQLConnectionHandler .handleData97
io.mycat.net.handler.BackendAsyncHandler .offerData36
io.mycat.backend.mysql.nio.MySQLConnectionHandler .handle79
io.mycat.net.AbstractConnection .handle269
io.mycat.net.AbstractConnection .onReadData327
io.mycat.net.NIOSocketWR .asynRead216
io.mycat.net.AbstractConnection .asynRead279
io.mycat.net.NIOReactor$RW .run113
java.lang.Thread .run748
write errorMsg:{} error
可以看到是在mysql master(192.168.1.232)上寫數據。接着使用mycat查看一條數據,
select * from customer_login where customer_id=7;
+-------------+----------------+----------------------------------+------------+
| customer_id | login_name | password | user_stats |
+-------------+----------------+----------------------------------+------------+
| 7 | Debra Williams | 00cf6d531b72b076370e21da78c49779 | 1 |
+-------------+----------------+----------------------------------+------------+
1 row in set (0.00 sec)
tail -f logs/mycat.log
2020-01-20 17:28:37.204 INFO [$_NIOREACTOR-1-RW] (io.mycat.backend.mysql.nio.handler.NewConnectionRespHandler.connectionAcquired(NewConnectionRespHandler.java:44)) - connectionAcquired MySQLConnection@28406540 [id=15, lastTime=1579512517204, user=root, schema=user_db, old shema=user_db, borrowed=true, fromSlaveDB=true, threadId=147, charset=utf8, txIsolation=3, autocommit=true, attachment=null, respHandler=null, host=192.168.1.233, port=3306, statusSync=null, writeQueue=0, modifiedSQLExecuted=false]
可以看到讀數據是在mysql slave(192.168.1.233)上操作的。這就是mycat完成的讀寫分離。
mycat管理
可以使用mysql客戶端對mycat進行管理。
- 登錄mycat:
mysql -uroot -p -h192.168.1.231 -P9066
- 常用命令:
show @@help; 查看管理命令
reload @@config; 動態加載配置
show @@datanode; 查看數據節點
show @@datasource; 查看物理節點
show @@backend; 查看連接情況
命令示例:
reload @@config;
Query OK, 1 row affected (0.08 sec)
Reload config success
show @@datanode;
+------+---------------+-------+-------+--------+------+------+---------+------------+----------+---------+---------------+
| NAME | DATHOST | INDEX | TYPE | ACTIVE | IDLE | SIZE | EXECUTE | TOTAL_TIME | MAX_TIME | MAX_SQL | RECOVERY_TIME |
+------+---------------+-------+-------+--------+------+------+---------+------------+----------+---------+---------------+
| dn1 | node1/user_db | 0 | mysql | 0 | 9 | 1000 | 129 | 0 | 0 | 0 | -1 |
+------+---------------+-------+-------+--------+------+------+---------+------------+----------+---------+---------------+
1 row in set (0.00 sec)
show @@datasource;
+----------+---------------+-------+---------------+------+------+--------+------+------+---------+-----------+------------+
| DATANODE | NAME | TYPE | HOST | PORT | W/R | ACTIVE | IDLE | SIZE | EXECUTE | READ_LOAD | WRITE_LOAD |
+----------+---------------+-------+---------------+------+------+--------+------+------+---------+-----------+------------+
| dn1 | 192.168.1.232 | mysql | 192.168.1.232 | 3306 | W | 0 | 9 | 1000 | 137 | 0 | 22 |
| dn1 | 192.168.1.233 | mysql | 192.168.1.233 | 3306 | W | 0 | 1 | 1000 | 107 | 2 | 0 |
| dn1 | 192.168.1.233 | mysql | 192.168.1.233 | 3306 | R | 0 | 7 | 1000 | 112 | 0 | 0 |
+----------+---------------+-------+---------------+------+------+--------+------+------+---------+-----------+------------+
3 rows in set (0.00 sec)
show @@backend;
+------------+------+---------+---------------+------+--------+--------+---------+------+--------+----------+------------+---------+---------+---------+------------+
| processor | id | mysqlId | host | port | l_port | net_in | net_out | life | closed | borrowed | SEND_QUEUE | schema | charset | txlevel | autocommit |
+------------+------+---------+---------------+------+--------+--------+---------+------+--------+----------+------------+---------+---------+---------+------------+
| Processor0 | 16 | 148 | 192.168.1.233 | 3306 | 54458 | 1337 | 358 | 921 | false | false | 0 | user_db | utf8:33 | 3 | true |
| Processor0 | 2 | 65 | 192.168.1.232 | 3306 | 37152 | 1308 | 387 | 1221 | false | false | 0 | user_db | utf8:33 | 3 | true |
| Processor0 | 19 | 151 | 192.168.1.233 | 3306 | 54466 | 89 | 70 | 21 | false | false | 0 | user_db | utf8:33 | 3 | true |
| Processor0 | 4 | 66 | 192.168.1.232 | 3306 | 37154 | 1308 | 387 | 1221 | false | false | 0 | user_db | utf8:33 | 3 | true |
| Processor0 | 6 | 64 | 192.168.1.232 | 3306 | 37150 | 1427 | 838 | 1221 | false | false | 0 | user_db | utf8:33 | 3 | true |
| Processor0 | 8 | 67 | 192.168.1.232 | 3306 | 37156 | 1308 | 387 | 1221 | false | false | 0 | user_db | utf8:33 | 3 | true |
| Processor0 | 10 | 62 | 192.168.1.232 | 3306 | 37146 | 1231 | 369 | 1221 | false | false | 0 | user_db | utf8:33 | 3 | true |
| Processor0 | 12 | 144 | 192.168.1.233 | 3306 | 54446 | 2117 | 538 | 1221 | false | false | 0 | user_db | utf8:33 | 3 | true |
| Processor0 | 14 | 146 | 192.168.1.233 | 3306 | 54450 | 10219 | 2353 | 1221 | false | false | 0 | user_db | utf8:33 | 3 | true |
| Processor1 | 1 | 68 | 192.168.1.232 | 3306 | 37158 | 1230 | 370 | 1221 | false | false | 0 | user_db | utf8:33 | 3 | true |
| Processor1 | 17 | 149 | 192.168.1.233 | 3306 | 54460 | 791 | 232 | 621 | false | false | 0 | user_db | utf8:33 | 3 | true |
| Processor1 | 18 | 150 | 192.168.1.233 | 3306 | 54464 | 401 | 142 | 321 | false | false | 0 | user_db | utf8:33 | 3 | true |
| Processor1 | 5 | 60 | 192.168.1.232 | 3306 | 37142 | 1308 | 387 | 1221 | false | false | 0 | user_db | utf8:33 | 3 | true |
| Processor1 | 7 | 63 | 192.168.1.232 | 3306 | 37148 | 1211 | 439 | 1221 | false | false | 0 | user_db | utf8:33 | 3 | true |
| Processor1 | 9 | 61 | 192.168.1.232 | 3306 | 37144 | 1485 | 1369 | 1221 | false | false | 0 | user_db | utf8:33 | 3 | true |
| Processor1 | 11 | 143 | 192.168.1.233 | 3306 | 54444 | 2195 | 556 | 1221 | false | false | 0 | user_db | utf8:33 | 3 | true |
| Processor1 | 13 | 145 | 192.168.1.233 | 3306 | 54448 | 2117 | 538 | 1221 | false | false | 0 | user_db | utf8:33 | 3 | true |
| Processor1 | 15 | 147 | 192.168.1.233 | 3306 | 54456 | 1259 | 340 | 921 | false | false | 0 | user_db | utf8:33 | 3 | true |
+------------+------+---------+---------------+------+--------+--------+---------+------+--------+----------+------------+---------+---------+---------+------------+
18 rows in set (0.00 sec)