MHA簡介:
MHA,即MasterHigh Availability Manager and Toolsfor MySQL,是日本的一位MySQL專家採用Perl語言編寫的一個腳本管理工具,該工具僅適用於MySQLReplication 環境,目的在於維持Master主庫的高可用性。
MHA(Master High Availability)是自動的master故障轉移和Slave提升的軟件包.它是基於標準的MySQL複製(異步/半同步).
MHA組成部分:
MHA由兩部分組成:
MHA Manager(管理節點)
MHA Node(數據節點)
MHA部署解讀:
MHA Manager可以單獨部署在一臺獨立機器上管理多個master-slave集羣,也可以部署在一臺slave上.MHA Manager探測集羣的node節點,當發現master出現故障的時候,它可以自動將具有最新數據的slave提升爲新的master,然後將所有其它的slave導向新的master上.整個故障轉移過程對應用程序是透明的。
MHA node運行在每臺MySQL服務器上(master/slave/manager),它通過監控具備解析和清理logs功能的腳本來加快故障轉移的。
MHA優缺點介紹:
優點:
1. 故障切換時,可以自行判斷哪個從庫與主庫的數據最接近,就切換到上面,可以減少數據的丟失,保證數據的一致性
2. 支持 binlog server,可提高 binlog 傳送效率,進一步減少數據丟失風險。
3. 可以配置 mysql 5.7 的增強半同步,來保證數據的時時同步
缺點:
1. 自動切換的腳本太簡單了,而且比較老化,建議後期逐漸完善。
2. 搭建 MHA 架構,需要開啓 linux 系統互信協議,所以對於系統安全性來說,是個不小的考驗。
原理介紹:
MHA的目的在於維持MySQL Replication中Master庫的高可用性,其最大特點是可以修復多個Slave之間的差異日誌,最終使所有Slave保持數據一致,然後從中選擇一個充當新的Master,並將其它Slave指向它。
當master出現故障時,可以通過對比slave之間I/O thread 讀取主庫binlog的position號,選取最接近的slave做爲備選主庫(備胎)。其它的從庫可以通過與備選主庫對比生成差異的中繼日誌。在備選主庫上應用從原來master保存的binlog,同時將備選主庫提升爲master。最後在其它slave上應用相應的差異中繼日誌並開始從新的master開始複製。
MHA工具包功能介紹:
Manager工具:
1 2 3 4 5 6 7 | #masterha_check_ssh : 檢查MHA的SSH配置。 #masterha_check_repl : 檢查MySQL複製。 #masterha_manager : 啓動MHA。 #masterha_check_status : 檢測當前MHA運行狀態。 #masterha_master_monitor : 監測master是否宕機。 #masterha_master_switch : 控制故障轉移(自動或手動)。 #masterha_conf_host : 添加或刪除配置的server信息。 |
2. Node工具:
1 2 3 4 | #save_binary_logs : 保存和複製master的二進制日誌。 #apply_diff_relay_logs : 識別差異的中繼日誌事件並應用於其它slave。 #filter_mysqlbinlog : 去除不必要的ROLLBACK事件(MHA已不再使用這個工具)。 #purge_relay_logs : 清除中繼日誌(不會阻塞SQL線程)。 |
實戰圖表展示:
實戰開始:
環境介紹:
1 2 3 4 | 192.168.56.100 master node 192.168.56.101 slave1 node 192.168.56.102 slave2(manager,node) vip 192.168.56.123 |
第一步操作:生成ssh無密鑰證書
主庫(100)執行生成密鑰操作:
1 2 3 4 | ssh -keygen -t dsa -P '' -f id_dsa Id_dsa.pub爲公鑰,id_dsa爲私鑰,緊接着將公鑰文件複製成authorized_keys文件, 這個步驟是必須的,過程如下: cat id_dsa.pub >> authorized_keys |
從庫(101)執行生成密鑰操作:
1 2 | ssh -keygen -t dsa -P '' -f id_dsa cat id_dsa.pub >> authorized_keys |
管理節點(102)生成密鑰過程:
1 2 | ssh -keygen -t dsa -P '' -f id_dsa cat id_dsa.pub >> authorized_keys |
主庫(100)執行接收密鑰的過程:
1 2 | scp 192.168.56.101: /root/ . ssh /id_dsa .pub . /id_dsa .pub.101 scp 192.168.56.102: /root/ . ssh /id_dsa .pub . /id_dsa .pub.102 |
1 2 3 | 合併密鑰: cat id_dsa.pub.101 >> authorized_keys cat id_dsa.pub.102 >> authorized_keys |
在主庫上傳送合成密鑰:
1 2 | scp authorized_keys 192.168.56.101: /root/ . ssh / scp authorized_keys 192.168.56.102: /root/ . ssh / |
在三臺服務器上,編輯/etc/hosts文件,分別加入三臺主機hostname
1 2 3 4 | vim /etc/hosts node1 192.168.56.100 node2 192.168.56.101 node3 192.168.56.102 |
驗證主機名登陸,密鑰驗證:
分別在三臺機器上執行:
1 2 3 4 5 6 7 8 9 | 在192.168.56.100執行 ssh node2 ssh node3 在192.168.56.101執行 ssh node1 ssh node3 在192.168.56.102執行 ssh node1 ssh node2 |
搭建主從環境(一主兩從架構,我這裏是mysql 5.7版本)
在所有節點上都要執行
1 2 3 | 創建主從賬號: grant replication slave on *.* to 'repl' @ '192.168.56.%' identified by 'repl' ; flush privileges; |
1 2 3 | 創建管理賬號: grant all privileges on *.* to 'zs' @ '192.168.56.%' identified by '123456' ; flush privileges; |
在Master(100)上面安裝數據節點:
首先要先安裝mysql依賴的perl環境
1 | yum install perl-DBD-MySQL |
解壓數據節點的包
1 | tar -zxvf mha4mysql-node-0.57. tar .gz |
安裝perl-cpan軟件包
1 2 3 4 | cd mha4mysql-node-0.57 yum -y install perl-CPAN* perl Makefile.PL make && make install |
在從庫(101)上面安裝數據節點:
同主庫一樣的安裝操作;
在manager管理節點(102)上面安裝管理節點:
首先先要安裝環境需要的介質包
1 2 3 4 5 | yum install -y perl-DBD-MySQL* rpm -ivh perl-Params-Validate-0.92-3.el6.x86_64.rpm rpm -ivh perl-Config-Tiny-2.12-1.el6.rfx.noarch.rpm rpm -ivh perl-Log-Dispatch-2.26-1.el6.rf.noarch.rpm rpm -ivh perl-Parallel-ForkManager-0.7.5-2.2.el6.rf.noarch.rpm |
再安裝數據節點:
1 2 3 4 5 | tar -zxvf mha4mysql-node-0.57. tar .gz cd mha4mysql-node-0.57 yum -y install perl-CPAN* perl Makefile.PL make && make install |
最後安裝管理節點:
1 2 3 4 | tar -zxvf mha4mysql-manager-0.57. tar .gz perl Makefile.PL make make install |
環境配置,基礎安裝操作完成。
下面來進行管理節點MHA配置:
創建mha家目錄,編輯啓動配置文件
1 2 | mkdir -p /usr/local/mha mkdir -p /etc/mhacd /etc/mha/ |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | 編輯MHA配置文件 vim /etc/mha/mha .conf [server default] user=zs password=123456 manager_workdir= /usr/local/mha manager_log= /usr/local/mha/manager .log remote_workdir= /usr/local/mha ssh_user=root repl_user=repl repl_password=repl ping_interval=1 master_ip_failover_script= /usr/local/scripts/master_ip_failover master_ip_online_change_script= /usr/local/scripts/master_ip_online_change |
1 2 3 4 5 6 | [server1] hostname =192.168.56.100 ssh_port=22 master_binlog_dir= /data/mysql candidate_master=1 port=3306 |
1 2 3 4 5 6 | [server2] hostname =192.168.56.101 ssh_port=22 master_binlog_dir= /data/mysql candidate_master=1 port=3306 |
1 2 3 4 5 6 | [server3] hostname =192.168.56.102 ssh_port=22 master_binlog_dir= /data/mysql no_master=1 port=3306 |
創建failover,online 腳本的目錄
1 | mkdir -p /usr/local/scripts |
編輯failover切換腳本:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 | vim master_ip_failover #!/usr/bin/env perl use strict; use warnings FATAL => 'all' ; use Getopt::Long; my ( $ command , $ssh_user, $orig_master_host, $orig_master_ip, $orig_master_port, $new_master_host, $new_master_ip, $new_master_port ); my $vip = '192.168.56.123/24' ; my $key = '0' ; my $ssh_start_vip = "/sbin/ifconfig eth0:$key $vip" ; my $ssh_stop_vip = "/sbin/ifconfig eth0:$key down" ; GetOptions( 'command=s' => \$ command , 'ssh_user=s' => \$ssh_user, 'orig_master_host=s' => \$orig_master_host, 'orig_master_ip=s' => \$orig_master_ip, 'orig_master_port=i' => \$orig_master_port, 'new_master_host=s' => \$new_master_host, 'new_master_ip=s' => \$new_master_ip, 'new_master_port=i' => \$new_master_port, ); exit &main(); |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 | sub main { print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n" ; if ( $ command eq "stop" || $ command eq "stopssh" ) { my $exit_code = 1; eval { print "Disabling the VIP on old master: $orig_master_host \n" ; &stop_vip(); $exit_code = 0; }; if ($@) { warn "Got Error: $@\n" ; exit $exit_code; } exit $exit_code; } elsif ( $ command eq "start" ) { my $exit_code = 10; eval { print "Enabling the VIP - $vip on the new master - $new_master_host \n" ; &start_vip(); $exit_code = 0; }; if ($@) { warn $@; exit $exit_code; } exit $exit_code; } elsif ( $ command eq "status" ) { print "Checking the Status of the script.. OK \n" ; exit 0; } else { &usage(); exit 1; } } |
1 2 3 4 5 6 7 8 9 10 11 12 13 | sub start_vip() { ` ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`; } sub stop_vip() { return 0 unless ($ssh_user); ` ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`; } sub usage { print "Usage: master_ip_failover -- command =start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n"; } |
編輯online_change的腳本:
cd /usr/local/scripts/
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 | vim master_ip_online_change #!/usr/bin/env perl use strict; use warnings FATAL => 'all' ; use Getopt::Long; my $vip = '192.168.1.123/24' ; # Virtual IP my $key = "0" ; my $ssh_start_vip = "/sbin/ifconfig eth0:$key $vip" ; my $ssh_stop_vip = "/sbin/ifconfig eth0:$key down" ; my $exit_code = 0; my ( $ command , $orig_master_is_new_slave, $orig_master_host, $orig_master_ip, $orig_master_port, $orig_master_user, $orig_master_password, $orig_master_ssh_user, $new_master_host, $new_master_ip, $new_master_port, $new_master_user, $new_master_password, $new_master_ssh_user, ); GetOptions( 'command=s' => \$ command , 'orig_master_is_new_slave' => \$orig_master_is_new_slave, 'orig_master_host=s' => \$orig_master_host, 'orig_master_ip=s' => \$orig_master_ip, 'orig_master_port=i' => \$orig_master_port, 'orig_master_user=s' => \$orig_master_user, 'orig_master_password=s' => \$orig_master_password, 'orig_master_ssh_user=s' => \$orig_master_ssh_user, 'new_master_host=s' => \$new_master_host, 'new_master_ip=s' => \$new_master_ip, 'new_master_port=i' => \$new_master_port, 'new_master_user=s' => \$new_master_user, 'new_master_password=s' => \$new_master_password, 'new_master_ssh_user=s' => \$new_master_ssh_user, ); exit &main(); |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 | sub main { #print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n"; if ( $ command eq "stop" || $ command eq "stopssh" ) { # $orig_master_host, $orig_master_ip, $orig_master_port are passed. # If you manage master ip address at global catalog database, # invalidate orig_master_ip here. my $exit_code = 1; eval { print "\n\n\n***************************************************************\n" ; print "Disabling the VIP - $vip on old master: $orig_master_host\n" ; print "***************************************************************\n\n\n\n" ; &stop_vip(); $exit_code = 0; }; if ($@) { warn "Got Error: $@\n" ; exit $exit_code; } exit $exit_code; } elsif ( $ command eq "start" ) { # all arguments are passed. # If you manage master ip address at global catalog database, # activate new_master_ip here. # You can also grant write access (create user, set read_only=0, etc) here. my $exit_code = 10; eval { print "\n\n\n***************************************************************\n" ; print "Enabling the VIP - $vip on new master: $new_master_host \n" ; print "***************************************************************\n\n\n\n" ; &start_vip(); $exit_code = 0; }; if ($@) { warn $@; exit $exit_code; } exit $exit_code; } |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | elsif ( $ command eq "status" ) { print "Checking the Status of the script.. OK \n" ; ` ssh $orig_master_ssh_user\@$orig_master_host \" $ssh_start_vip \"`; exit 0; } else { &usage(); exit 1; } } # A simple system call that enable the VIP on the new master sub start_vip() { ` ssh $new_master_ssh_user\@$new_master_host \" $ssh_start_vip \"`; } # A simple system call that disable the VIP on the old_master sub stop_vip() { ` ssh $orig_master_ssh_user\@$orig_master_host \" $ssh_stop_vip \"`; } sub usage { print "Usage: master_ip_failover – command =start|stop|stopssh|status –orig_master_host=host –orig_master_ip=ip –orig_master_port=po rt –new_master_host=host –new_master_ip=ip –new_master_port=port\n"; } |
創建完兩個腳本,記得賦予執行權限
利用mha工具檢測ssh
安裝需要的環境包:
1 | yum -y install perl-Time-HiRes |
執行檢測命令;
1 | /usr/local/bin/masterha_check_ssh --conf= /etc/mha/mha .conf |
檢測結果顯示:都爲OK,代表ssh檢測成功
結果展示:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 | Sun Jul 23 09:39:09 2017 - [warning] Global configuration file /etc/masterha_default .cnf not found. Skipping. Sun Jul 23 09:39:09 2017 - [info] Reading application default configuration from /etc/mha/mha .conf.. Sun Jul 23 09:39:09 2017 - [info] Reading server configuration from /etc/mha/mha .conf.. Sun Jul 23 09:39:09 2017 - [info] Starting SSH connection tests.. Sun Jul 23 09:39:11 2017 - [debug] Sun Jul 23 09:39:09 2017 - [debug] Connecting via SSH from [email protected](192.168.56.100:22) to [email protected](192.168.56.101:22).. Sun Jul 23 09:39:10 2017 - [debug] ok. Sun Jul 23 09:39:10 2017 - [debug] Connecting via SSH from [email protected](192.168.56.100:22) to [email protected](192.168.56.102:22).. Sun Jul 23 09:39:11 2017 - [debug] ok. Sun Jul 23 09:39:11 2017 - [debug] Sun Jul 23 09:39:10 2017 - [debug] Connecting via SSH from [email protected](192.168.56.101:22) to [email protected](192.168.56.100:22).. Sun Jul 23 09:39:10 2017 - [debug] ok. Sun Jul 23 09:39:10 2017 - [debug] Connecting via SSH from [email protected](192.168.56.101:22) to [email protected](192.168.56.102:22).. Sun Jul 23 09:39:11 2017 - [debug] ok. Sun Jul 23 09:39:11 2017 - [debug] Sun Jul 23 09:39:10 2017 - [debug] Connecting via SSH from [email protected](192.168.56.102:22) to [email protected](192.168.56.100:22).. Warning: Permanently added '192.168.56.102' (RSA) to the list of known hosts. Sun Jul 23 09:39:11 2017 - [debug] ok. Sun Jul 23 09:39:11 2017 - [debug] Connecting via SSH from [email protected](192.168.56.102:22) to [email protected](192.168.56.101:22).. Sun Jul 23 09:39:11 2017 - [debug] ok. Sun Jul 23 09:39:11 2017 - [info] All SSH connection tests passed successfully. 檢測masterha_check_repl --conf= /etc/mha/mha .conf Sun Jul 23 10:14:11 2017 - [warning] Global configuration file /etc/masterha_default .cnf not found. Skipping. Sun Jul 23 10:14:11 2017 - [info] Reading application default configuration from /etc/mha/mha .conf.. Sun Jul 23 10:14:11 2017 - [info] Reading server configuration from /etc/mha/mha .conf.. Sun Jul 23 10:14:11 2017 - [info] MHA::MasterMonitor version 0.57. Sun Jul 23 10:14:11 2017 - [info] Multi-master configuration is detected. Current primary(writable) master is 192.168.56.100(192.168.56.100:3306) Sun Jul 23 10:14:11 2017 - [info] Master configurations are as below: Master 192.168.56.100(192.168.56.100:3306), replicating from 192.168.56.101(192.168.56.101:3306) Master 192.168.56.101(192.168.56.101:3306), replicating from 192.168.56.100(192.168.56.100:3306), read -only Sun Jul 23 10:14:11 2017 - [info] GTID failover mode = 1 Sun Jul 23 10:14:11 2017 - [info] Dead Servers: Sun Jul 23 10:14:11 2017 - [info] Alive Servers: Sun Jul 23 10:14:11 2017 - [info] 192.168.56.100(192.168.56.100:3306) Sun Jul 23 10:14:11 2017 - [info] 192.168.56.101(192.168.56.101:3306) Sun Jul 23 10:14:11 2017 - [info] 192.168.56.102(192.168.56.102:3306) Sun Jul 23 10:14:11 2017 - [info] Alive Slaves: Sun Jul 23 10:14:11 2017 - [info] 192.168.56.101(192.168.56.101:3306) Version=5.7.14-log (oldest major version between slaves) log-bin:enabled Sun Jul 23 10:14:11 2017 - [info] GTID ON Sun Jul 23 10:14:11 2017 - [info] Replicating from 192.168.56.100(192.168.56.100:3306) Sun Jul 23 10:14:11 2017 - [info] Primary candidate for the new Master (candidate_master is set ) Sun Jul 23 10:14:11 2017 - [info] 192.168.56.102(192.168.56.102:3306) Version=5.7.14-log (oldest major version between slaves) log-bin:enabled Sun Jul 23 10:14:11 2017 - [info] GTID ON Sun Jul 23 10:14:11 2017 - [info] Replicating from 192.168.56.100(192.168.56.100:3306) Sun Jul 23 10:14:11 2017 - [info] Not candidate for the new Master (no_master is set ) Sun Jul 23 10:14:11 2017 - [info] Current Alive Master: 192.168.56.100(192.168.56.100:3306) Sun Jul 23 10:14:11 2017 - [info] Checking slave configurations.. Sun Jul 23 10:14:11 2017 - [info] Checking replication filtering settings.. Sun Jul 23 10:14:11 2017 - [info] binlog_do_db= , binlog_ignore_db= Sun Jul 23 10:14:11 2017 - [info] Replication filtering check ok. Sun Jul 23 10:14:11 2017 - [info] GTID (with auto-pos) is supported. Skipping all SSH and Node package checking. Sun Jul 23 10:14:11 2017 - [info] Checking SSH publickey authentication settings on the current master.. Sun Jul 23 10:14:11 2017 - [info] HealthCheck: SSH to 192.168.56.100 is reachable. Sun Jul 23 10:14:11 2017 - [info] 192.168.56.100(192.168.56.100:3306) (current master) +--192.168.56.101(192.168.56.101:3306) +--192.168.56.102(192.168.56.102:3306) Sun Jul 23 10:14:11 2017 - [info] Checking replication health on 192.168.56.101.. Sun Jul 23 10:14:11 2017 - [info] ok. Sun Jul 23 10:14:11 2017 - [info] Checking replication health on 192.168.56.102.. Sun Jul 23 10:14:11 2017 - [info] ok. Sun Jul 23 10:14:11 2017 - [info] Checking master_ip_failover_script status: Sun Jul 23 10:14:11 2017 - [info] /usr/local/scripts/master_ip_failover -- command =status --ssh_user=root --orig_master_host=192.168.56.100 --orig_master_ip=192.168.56.100 --orig_master_port=3306 IN SCRIPT TEST==== /sbin/ifconfig eth0:0 down== /sbin/ifconfig eth0:0 192.168.56.123 /24 === Checking the Status of the script.. OK Sun Jul 23 10:14:11 2017 - [info] OK. Sun Jul 23 10:14:11 2017 - [warning] shutdown_script is not defined. Sun Jul 23 10:14:11 2017 - [info] Got exit code 0 (Not master dead). MySQL Replication Health is OK. |
在主庫(100)執行添加vip的過程:(第一次手動添加)
1 | ip addr add 192.168.56.123 dev eth0 |
在管理節點(102)上,執行mha的啓動
1 | nohup masterha_manager --conf= /etc/mha/mha .conf > /tmp/mha_manager .log < /dev/null 2>&1 & |
驗證啓動成功的命令:查看顯示狀態
1 | masterha_check_status --conf= /etc/mha/mha .conf |
模擬主庫故障,查看是否自動切換:
在主庫(100)上面執行停掉mysql操作。
1 2 | mysqladmin -uroot -proot123 shutdown 從庫192.168.56.101自動獲取vip且轉換爲新的主庫,192.168.56.102自動指向新的主庫。 |
切換後,MHA進程會自動停止運行
在管理節點查看:
1 2 | masterha_check_status --conf= /etc/mha/mha .conf mha is stopped(2:NOT_RUNNING). |
恢復操作:
把宕掉的主庫192.168.56.100恢復起來;
1 | /usr/local/mysql/bin/mysqld_safe --defaults- file = /etc/my .cnf & |
驗證結果:重新指向現在的主庫192.168.56.101
主從狀態一切ok!