數據庫管理與高可用----MHA高可用配置及故障切換

數據庫管理與高可用----MHA高可用配置及故障切換

實驗思路:
1.MHA架構
1）數據庫安裝
2)一主兩從
3)MHA搭建
故障模擬
1）主庫失效
2）備選主庫成爲主庫
3)從庫2將備選主庫指向爲主庫
案例環境
1.本案例環境
服務器CentOS7.4(64位)MHA-manager/192.168.100.130
管理節點，安裝manager組件
服務器CentOS7.4(64位)Mysq11/192.168.100.150 Master 節點，安裝node組件
服務器CentOS7.4(64位)Mysql2/192.168.100.160 Slave 節點，安裝node組件
服務器CentOS7.4(64位)Mysql3/192.168.100.170 Slave節點，安裝node組件
這裏操作系統是CentOS7版本，所以這裏下載MHA版本是0.57版本。
2.案例需求
本案例要求通過MHA監控MySQL數據庫在故障時進行自動切換，不影響業務。
3.案例實現思路
1)安裝MySQL數據庫
2)配置MySQL一主兩從
3)安裝MHA軟件
4)配置無密碼認證
5)配置 MySQL MHA高可用
6)模擬master故障切換
在三臺MySQL 節點上分別安裝數據庫，MySQL版本請使用5.6.36，cmake版本請使
用2.8.6。下面只在Mysql1上面做演示，安裝過程如下。

1.安裝編譯依賴的環境
[root@Mysql1~]# yum install ncurses-devel gcc-c++ perl-Module-Install -y

2.安裝gmake編譯軟件
[root@Mysql1~ ]# tar zxvf cmake-2.8.6.tar.gz
[root@Mysql1 ~]# cd cmake-2.8.6
[root@Mysql1 cmake-2.8.6]# ./configure
[root@Mysql1 cmake-2.8.6]# gmake && gmake install

3.安裝MySQL數據庫
[root@Mysql1~ ]# tar zxvf mysql-5.6.36.tar.gz
[root@Mysql1 ~]# cd mysql-5.6.36

cmake -DCMAKE_INSTALL_PREFIX=/usr/local/mysql \
-DDEFAULT_CHARSET=utf8 \
-DDEFAULT_COLLATION=utf8_general_ci \
-DWITII_EXTRA_CHARSETS=all \
-DSYSCONFDTR=/etc

[root@Mysql1 mysql-5.6.36]# make && make install
[root@Mysql1 mysql-5.6.36]# cp support-files/my-default.cnf /etc/my.cnf
[root@Mysql1 mysql-5.6.36]# cp support-files/mysql.server /etc/rc.d/init.d/mysqld
[root@Mysql1 ~]# chmod +x /etc/rc.d/init.d/mysqld
[root@MysqI1 ~]# chkconfig --add mysqld
[root@Mysql1 ~]# echo  "PATH=$PATH:/usr/local/mysql/bin" >> /etc/profile
[root@Mysql1 ~]# source /etc/profile
[root@Mysql1 ~]# groupadd mysql
[root@Mysql1 ~]# useradd -M -s /sbin/nologin mysql -g mysql
[root@Mysql1 ~]# chown -R mysql.mysql /usr/local/mysql

source /etc/profile
groupadd mysql
useradd -M -s /sbin/nologin mysql -g mysql
chown -R mysql.mysql /usr/local/mysql

[root@Mysql1 ~]# mkdir -p /data/mysql
/usr/local/mysql/scripts/mysql_install_db \
--basedir=/usr/local/mysql \
--datadir=/usr/local/mysql/data \
--user=mysql
/usr/local/mysql/scripts/mysql_install_db \
--basedir=/usr/local/mysql \
--datadir=/usr/local/mysql/data \
--user=mysql

4.修改Master的主配置文件/etc/my.cnf文件，三臺服務器的server-id不能一樣
[root@Mysql1 ~]# cat /etc/my.cnf
[mysqld]
server-id = 1
log_bin = master-bin
log-slave-updates = true
配置從服務器:
在/etc/my.cnf中修改或者增加下面內容。
[root@Mysql2 ~]# vim /etc/my.cnf
server-id = 2
log_bin = master-bin
relay-log = relay-log-bin
relay-log-index = slave-relay-bin.index

5.Mysql1、Mysql2、Mysql3分別做兩個軟鏈接
[root@Mysql1 ~]# In -s /usr/local/mysql/bin/mysql /usr/sbin/
[root@Mysql1 ~]# In -s /usr/local/mysql/bin/mysqlbinlog /usr/sbin/
ln -s /usr/local/mysql/bin/mysql /usr/sbin/
ln -s /usr/local/mysql/bin/mysqlbinlog /usr/sbin/
6、Mysql1、Mysql2、Mysql3啓動MySQL。

systemctl stop firewalld
setenforce 0
/usr/local/mysql/bin/mysqld_safe --user=mysql &
netstat -ntap | grep 3306

7.配置MySQL一主兩從

1).MySQL主從配置相對比較簡單。需要注意的是授權。步驟如下:
在所有數據庫節點上授權兩個用戶，一個是從庫同步使用，另外一個是manager使用。
grant replication slave on*.* to 'myslave'@'192.168.100.%' identified by '123';
grant all privileges on *.* to 'mha'@'192.168.100.%' identified by 'manager';
flush privileges;
grant replication slave on *.* to 'myslave'@'192.168.100.%' identified by '123';
grant all privileges on *.* to 'mha'@'192.168.100.%' identified by 'manager';
flush privileges;

2).下面三條授權按理論是不用添加的，但是做案例實驗環境時候通過MHA檢查MySQL主從有報錯，
報兩個從庫通過主機名連接不上主庫，所以所有數據庫加上下面的授權。
grant all privileges on *.* to 'mha'@'Mysql1' identified by 'manager';
grant all privileges on *.* to 'mha'@'Mysql2' identified by 'manager';
grant all privileges on *.* to 'mha'@'Mysql3' identified by 'manager';
flush privileges;
3).在Mysq1主機上查看二進制文件和同步點
mysql> show master status;
+-------------------+----------+--------------+------------------+-------------------+
| File              | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+-------------------+----------+--------------+------------------+-------------------+
| master-bin.000001 |     1294 |              |                  |                   |
+-------------------+----------+--------------+------------------+-------------------+
1 row in set (0.01 sec)

4.接下來在Mysql2和Mysql3分別執行同步。
mysql> change master to master_host='192.168.100.150',master_user='myslave',master_password='123',master_log_file='master-bin.000010',master_log_pos=1294;
flush privileges;
start slave;
stop slave;
start slave;
5.查看IO和SQL線程都是yes代表同步是否正常。
mysql> show slave status\G;
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
mysql> show master status;
"查看二進制日誌路徑"
mysql> show variables like 'datadir';
+---------------+--------------+
| Variable_name | Value        |
+---------------+--------------+
| datadir       | /home/mysql/ |
+---------------+--------------+
必須設置兩個從庫爲只讀模式:
set global read_only=1;
flush privileges;
注意:設置完成直接驗證主從複製功能

--------------------------安裝MHA軟件----------------------------
1.所有服務器上都安裝MHA依賴的環境，首先安裝epel源。
[root@MHA-manager ~]# yum install epel-release --nogpgcheck -y

yum install -y perl-DBD-MySQL \
perl-Config-Tiny \
perl-Log-Dispatch \
perl-Parallel-ForkManager \
perl-ExtUtils-CBuilder \
perl-ExtUtils-MakeMaker \
perl-CPAN

2.MHA 軟件包對於每個操作系統版本不一樣，這裏CentOS7.4必須選擇0.57版本，
在<注意:所有服務器>上必須先安裝node組件，最後在MHA-manager節點上安裝manager組件，
因爲manager依賴node組件，下面都是在Mysql1上操作演示安裝node組件。
//上傳mha4mysql-node-0.57.tar.gz
[root@Mysql1~]# tar zxvf mha4mysql-node-0.57.tar.gz
[root@Mysql1 ~]# cd mha4mysql-node-0.57
[root@Mysql1 mha4mysql-node-o.57]# perl Makefile.PL 
[root@Mysql1 mha4mysql-node-o.57]# make
[root@Mysql1 mha4mysql-node-0.57]# make && make install

3.僅在MHA-manager上安裝manager組件(!注意:一定要先安裝node組件才能安裝manager組件)
[root@Mysql1 mha4mysql-node-0.57]# cd ~
[root@MHA-manager ~]# tar zxvf mha4mysql-manager-0.57.tar.gz
[root@MHA-manager~]# cd mha4mysql-manager-0.57
[root@MHA-manager mha4mysql-manager-0.57]# perl Makefile.PL
[root@MHA-manager mha4mysql-manager-0.57]# make
[root@MHA-manager mha4mysql-manager-o.57]# make && make install

manager安裝後在/usr/local/bin下面會生成幾個工具，主要包括以下幾個:
masterha_check_ssh檢查MHA的SSH配置狀況
masterha_check_repl檢查MySQL複製狀況
masterha_manger啓動manager的腳本
masterha_check_status檢測當前MHA運行狀態
masterha_master_monitor檢測master是否宕機
masterha_master_switch控制故障轉移（自動或者手動)
masterha_conf_host添加或刪除配置的server信息
masterha_stop關閉manager

4.node安裝後也會在/usr/local/bin下面會生成幾個腳本(這些工具通常由MHA
Manager的腳本觸發，無需人爲操作）主要如下:
save_binary_logs保存和複製master的二進制日誌
apply_diff_relay_logs 識別差異的中繼日誌事件並將其差異的事件應用於其他的slave
filter_mysqlbinlog去除不必要的ROLLBACK事件(MHA已不再使用這個工具)
purge_relay_logs清除中繼日誌(不會阻塞SQL線程)

5.配置無密碼認證
1).在manager上配置到所有數據庫節點的無密碼認證
[root@MHA-manager ~]# ssh-keygen -t rsa    //一路按回車鍵
[root@MHA-manager ~]# ssh-copy-id 192.168.100.150   //輸入yes 及150的root密碼
[root@MHA-manager ~]# ssh-copy-id 192.168.100.160
[root@MHA-manager ~]# ssh-copy-id 192.168.100.170
2).在Mysql1上配置到數據庫節點Mysql2和Mysql3的無密碼認證
[root@Mysql1 ~]# ssh-keygen -t rsa
[root@MysqI1 ~]# ssh-copy-id 192.168.100.160
[root@Mysql1 ~]# ssh-copy-id 192.168.100.170
3).在Mysql2上配置到數據庫節點MysqI1和Mysql3的無密碼認證
[root@Mysql2 ~]# ssh-keygen -t rsa
[root@Mysql2 ~]# ssh-copy-id 192.168.100.150
[root@Mysql2 ~]# ssh-copy-id 192.168.100.170
4).在Mysql3上配置到數據庫節點Mysql1和Mysql2的無密碼認證
[root@Mysql3 ~]# ssh-keygen -t rsa
[root@Mysql3 ~]# ssh-copy-id 192.168.100.150
[root@Mysql3 ~]# ssh-copy-id 192.168.100.160

6.配置MHA

1).在manager節點上覆制相關腳本到/usr/local/bin目錄。
[root@mha-manager~]# cp -ra /root/mha4mysql-manager-0.57/samples/scripts /usr/local/bin
//拷貝後會有四個執行文件
[root@MHA-manager ~]#ll/usr/local/bin/scripts/
總用量32
-rwxr-xr-x 1 mysql mysql 36485月312015 master_ip_failover#自動切換時VIP管理的腳本
-rwxr-xr-x 1 mysql mysql 98725月2509:07 master_ip_online_change #在線切換時vip的管理
-rwxr-xr-x 1 mysql mysql118675月312015 power_manager #故障發生後關閉主機的腳本
-rwxr-Xr-x 1 mysql mysql 13605月312015 send_report #因故障切換後發送報警的腳本

2).複製上述的自動切換時VIP管理的腳本到/usr/local/bin目錄，這裏使用腳本管理VIP,
[root@mha-manager~]# cp /usr/local/bin/scripts/master_ip_failover /usr/local/bin

3).修改內容如下:(刪除原有內容，直接複製)

[root@MHA-manager ~]# vim /usr/local/bin/master_ip_failover
#!/usr/bin/env perl
use strict;
use warnings FATAL => 'all';

use Getopt::Long;

my (
$command, $ssh_user, $orig_master_host, $orig_master_ip,
$orig_master_port, $new_master_host, $new_master_ip, $new_master_port
);
#############################添加內容部分#########################################
my $vip = '192.168.100.200';
my $brdc = '192.168.100.255';
my $ifdev = 'ens33';
my $key = '1';
my $ssh_start_vip = "/sbin/ifconfig ens33:$key $vip";
my $ssh_stop_vip = "/sbin/ifconfig ens33:$key down";
my $exit_code = 0;
#my $ssh_start_vip = "/usr/sbin/ip addr add $vip/24 brd $brdc dev $ifdev label $ifdev:$key;/usr/sbin/arping -q -A -c 1 -I $ifdev $vip;iptables -F;";
#my $ssh_stop_vip = "/usr/sbin/ip addr del $vip/24 dev $ifdev label $ifdev:$key";
##################################################################################
GetOptions(
'command=s' => \$command,
'ssh_user=s' => \$ssh_user,
'orig_master_host=s' => \$orig_master_host,
'orig_master_ip=s' => \$orig_master_ip,
'orig_master_port=i' => \$orig_master_port,
'new_master_host=s' => \$new_master_host,
'new_master_ip=s' => \$new_master_ip,
'new_master_port=i' => \$new_master_port,
);

exit &main();

sub main {

print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";

if ( $command eq "stop" || $command eq "stopssh" ) {

my $exit_code = 1;
eval {
print "Disabling the VIP on old master: $orig_master_host \n";
&stop_vip();
$exit_code = 0;
};
if ($@) {
warn "Got Error: $@\n";
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "start" ) {

my $exit_code = 10;
eval {
print "Enabling the VIP - $vip on the new master - $new_master_host \n";
&start_vip();
$exit_code = 0;
};
if ($@) {
warn $@;
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "status" ) {
print "Checking the Status of the script.. OK \n";
exit 0;
}
else {
&usage();
exit 1;
}
}
sub start_vip() {
`ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
}
# A simple system call that disable the VIP on the old_master
sub stop_vip() {
`ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}

sub usage {
print
"Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";
}

4).創建MHA軟件目錄並拷貝配置文件。
[root@MHA-manager ~]# cd /usr/local/bin/scripts/
[root@MHA-manager scripts]# cp master_ip_online_change /usr/local/bin/
[root@MHA-manager scripts]# cp send_report /usr/local/
[root@MHA-manager ~]# mkdir /etc/masterha
[root@MHA-manager ~]# cp /root/mha4mysql-manager-0.57/samples/conf/app1.cnf /etc/masterha
[root@MHA-manager ~]# vim /etc/masterha/app1.cnf

[server default]
manager_log=/var/log/masterha/app1/manager.log
manager_workdir=/var/log/masterha/app1
master_binlog_dir=/usr/local/mysql/data
master_ip_failover_script=/usr/local/bin/master_ip_failover
master_ip_online_change_script=/usr/local/bin/master_ip_online_change
password=manager
ping_interval=1
remote_workdir=/tmp
repl_password=123
repl_user=myslave
secondary_check_script=/usr/local/bin/masterha_secondary_check -s 192.168.100.160 -s 192.168.100.170
shutdown_script=""
ssh_user=root
user=mha

[server1]    #master節點
hostname=192.168.100.150
port=3306

[server2]    #salve1
hostname=192.168.100.160
port=3306
candidate_master=1
check_repl_delay=0

[server3]    #slave2
hostname=192.168.100.170
port=3306

################配置項解釋
[server default]
manager_log=/var/log/masterha/app1/manager.log  'manager日誌'
manager_workdir=/var/log/masterha/app1    'manager工作目錄'
master_binlog_dir=/usr/local/mysql/data    '#master保存binlog的位置，這裏的路徑要與master裏配置的binlog的路徑一直，以便mha能找到,這裏也對應編譯安裝數據庫的數據目錄'
master_ip_failover_script=/usr/local/bin/master_ip_failover  '設置自動failover時候的切換腳本，也就是上邊的那個腳本'
master_ip_online_change_script=/usr/local/bin/master_ip_online_change '設置手動切換時候的切換腳本'
password=manager  '#設置mysql中root用戶的密碼，這個密碼是前文中創建監控用戶的那個密碼'
ping_interval=1  '設置監控主庫，發送ping包的時間間隔，默認是3秒，嘗試三次沒有迴應的時候自動進行rail'
remote_workdir=/tmp  '設置遠端mysql在發生切換時binlog的保存位置'
repl_password=123  '設置複製用戶的密碼'
repl_user=myslave   '設置複製用戶的用戶'
secondary_check_script=/usr/local/bin/masterha_secondary_check -s 192.168.179.124 -s 192.168.179.125 '設置從服務器的地址'
shutdown_script="" '設置故障發生後關閉故障主機腳本（該腳本的主要作用是關閉主機放在發生腦裂,這裏沒有使用)'
ssh_user=root  '設置ssh的登錄用戶名'
user=mha

[server1]
hostname=192.168.179.123
port=3306

[server2]
candidate_master=1  '設置爲候選master'
hostname=192.168.179.124
check_repl_delay=0  '默認情況下如果一個slave落後master 100M的relay logs 的話，mha將不會選擇該slave作爲一個新的master'
port=3306

[server3]
hostname=192.168.179.125
port=3306

5).測試ssh無密碼認證，如果正常最後會輸出successfully，如下所示。
[root@mha-manager~]# masterha_check_ssh -conf=/etc/masterha/app1.cnf
Fri Aug 28 12:42:32 2020 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Fri Aug 28 12:42:32 2020 - [info] Reading application default configuration from /etc/masterha/app1.cnf..
Fri Aug 28 12:42:32 2020 - [info] Reading server configuration from /etc/masterha/app1.cnf..
Fri Aug 28 12:42:32 2020 - [info] Starting SSH connection tests..
Fri Aug 28 12:42:38 2020 - [debug] 
Fri Aug 28 12:42:32 2020 - [debug]  Connecting via SSH from [email protected](192.168.100.150:22) to [email protected](192.168.100.160:22)..
Fri Aug 28 12:42:33 2020 - [debug]   ok.
Fri Aug 28 12:42:33 2020 - [debug]  Connecting via SSH from [email protected](192.168.100.150:22) to [email protected](192.168.100.170:22)..
Fri Aug 28 12:42:38 2020 - [debug]   ok.
Fri Aug 28 12:42:39 2020 - [debug] 
Fri Aug 28 12:42:32 2020 - [debug]  Connecting via SSH from [email protected](192.168.100.160:22) to [email protected](192.168.100.150:22)..
Fri Aug 28 12:42:33 2020 - [debug]   ok.
Fri Aug 28 12:42:33 2020 - [debug]  Connecting via SSH from [email protected](192.168.100.160:22) to [email protected](192.168.100.170:22)..
Fri Aug 28 12:42:39 2020 - [debug]   ok.
Fri Aug 28 12:42:39 2020 - [debug] 
Fri Aug 28 12:42:33 2020 - [debug]  Connecting via SSH from [email protected](192.168.100.170:22) to [email protected](192.168.100.150:22)..
Fri Aug 28 12:42:34 2020 - [debug]   ok.
Fri Aug 28 12:42:34 2020 - [debug]  Connecting via SSH from [email protected](192.168.100.170:22) to [email protected](192.168.100.160:22)..
Fri Aug 28 12:42:39 2020 - [debug]   ok.
Fri Aug 28 12:42:39 2020 - [info] All SSH connection tests passed successfully.

//這一步有問題說明祕鑰分發問題

[root@mha-manager ~]# masterha_check_repl -conf=/etc/masterha/app1.cnf

Fri Aug 28 16:20:51 2020 - [info] Checking replication health on 192.168.100.160..
Fri Aug 28 16:20:51 2020 - [info]  ok.
Fri Aug 28 16:20:51 2020 - [info] Checking replication health on 192.168.100.170..
Fri Aug 28 16:20:51 2020 - [info]  ok.
Fri Aug 28 16:20:51 2020 - [info] Checking master_ip_failover_script status:
Fri Aug 28 16:20:51 2020 - [info]   /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.100.150 --orig_master_ip=192.168.100.150 --orig_master_port=3306 

IN SCRIPT TEST====/sbin/ifconfig ens33:1 down==/sbin/ifconfig ens33:1 192.168.100.200===

Checking the Status of the script.. OK 
Fri Aug 28 16:20:51 2020 - [info]  OK.
Fri Aug 28 16:20:51 2020 - [warning] shutdown_script is not defined.
Fri Aug 28 16:20:51 2020 - [info] Got exit code 0 (Not master dead).

MySQL Replication Health is OK.

//注意:第一次配置需要去master上手動開啓虛擬IP
[root@Mysql1 ~]# /sbin/ifconfig ens33:1 192.168.100.200/24

7.啓動MHA

[root@mha-manager ~]# nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/masterha/app1/manager.log 2>&1 &

--remove_dead_master_conf該參數代表當發生主從切換後，老的主庫的ip將會從配置文件中移除
--manger_log日誌存放位置
--ignore_last_failover在缺省情況下，如果MHA 檢測到連續發生宕機，且兩次宕機間隔不足8小時的話，則不會進行Failover，之所以這樣限制是爲了避免ping-pong效應。
該參數代表忽略上次MHA觸發切換產生的文件，默認情況下，MHA 發生切換後會在日誌記目錄，也就是上面設置的日誌app1.failover.complete文件，
下次再次切換的時候如果發現該目錄下存在該文件將不允許觸發切換，除非在第一次切換後收到刪除該文件，爲了方便，這裏設置爲-ignore_last_failover。
隔不足8小時的話，則不會進行Failover，之所以這樣限制是爲了避免ping-pong效應。該
參數代表忽略上次MHA觸發切換產生的文件，默認情況下，MHA 發生切換後會在日誌記
目錄，也就是上面設置的日誌app1.failover.complete文件，下次再次切換的時候如果發現
該目錄下存在該文件將不允許觸發切換，除非在第一次切換後收到刪除該文件，爲了方便，
這裏設置爲-ignore_last_failover.

8.查看MHA狀態，可以看到當前的master是Mysql1節點。
[root@mha-manager~]# masterha_check_status --conf=/etc/masterha/app1.cnf
app1 (pid:70465) is running(0:PING_OK), master:192.168.100.150

9.查看MHA日誌，也以看到當前的master是192.168.100.150，如下所示。
[root@mha-manager ~]# cat /var/log/masterha/app1/manager.log
.......
IN SCRIPT TEST====/sbin/ifconfig ens33:1 down==/sbin/ifconfig ens33:1 192.168.100.200===
.........

10.查看 Mysql1的VIP地址192.168.100.200是否存在?，這個VIР地址不會因爲
manager 節點停止MHA 服務而消失。
[root@mysql1 mha4mysql-node-0.57]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.100.150  netmask 255.255.255.0  broadcast 192.168.100.255
        inet6 fe80::7264:7578:de4e:190  prefixlen 64  scopeid 0x20<link>
        inet6 fe80::9433:6cdb:ee1d:7b3d  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:17:9c:75  txqueuelen 1000  (Ethernet)
        RX packets 101492  bytes 130137031 (124.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 43150  bytes 7281255 (6.9 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens33:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.100.200  netmask 255.255.255.0  broadcast 192.168.100.255
        ether 00:0c:29:17:9c:75  txqueuelen 1000  (Ethernet)

------------------------------------驗證------------------------------

[root@mha-manager ~]# tailf /var/log/masterha/app1/manager.log
//啓用監控觀察日誌記錄
[root@MysqI1~]# pkill -9 mysql /查看master變化
[root@mha-manager ~]# tailf /var/log/masterha/app1/manager.log
Started automated(non-interactive) failover.
Invalidated master IP address on 192.168.100.150(192.168.100.150:3306)
The latest slave 192.168.100.160(192.168.100.160:3306) has all relay logs for recovery.
Selected 192.168.100.160(192.168.100.160:3306) as a new master.
192.168.100.160(192.168.100.160:3306): OK: Applying all logs succeeded.
192.168.100.160(192.168.100.160:3306): OK: Activated master IP address.
192.168.100.170(192.168.100.170:3306): This host has the latest relay log events.
Generating relay diff files from the latest slave succeeded.
192.168.100.170(192.168.100.170:3306): OK: Applying all logs succeeded. Slave started, replicating from 192.168.100.160(192.168.100.160:3306)
192.168.100.160(192.168.100.160:3306): Resetting slave info succeeded.
Master failover to 192.168.100.160(192.168.100.160:3306) completed successfully.  'master已經成功切換到備用160上面'
[root@mha-manager bin]# masterha_check_status --conf=/etc/masterha/app1.cnf
app1 (pid:72487) is running(0:PING_OK), master:192.168.100.160
[root@mha-manager bin]# 

[root@mysql2 mha4mysql-node-0.57]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.100.160  netmask 255.255.255.0  broadcast 192.168.100.255
        inet6 fe80::1bbb:ba05:a579:42dd  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:9f:ee:79  txqueuelen 1000  (Ethernet)
        RX packets 100963  bytes 130156228 (124.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 40046  bytes 6979445 (6.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens33:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.100.200  netmask 255.255.255.0  broadcast 192.168.100.255
        ether 00:0c:29:9f:ee:79  txqueuelen 1000  (Ethernet)
虛擬地址也切換到mysql2
故障模擬:
在主庫上:
pkill mysqld
可以看到從庫的狀態，其中之一肯定有切換到主庫的
切換備選主庫的算法:
1.一般判斷從庫的是從(position/GTID）判斷優劣，數據有差異，最接近於master的slave，成爲備選主。
2.數據一致的情況下，按照配置文件順序，選擇備選主庫。
3.設定有權重(candidate_master=1)，按照權重強制指定備選主。
1）默認情況下如果一個slave落後master 100M的relay logs的話，即使有權重，也會失效。
2）如果check_repl_delay=0的話，即使落後很多日誌，也強制選擇其爲備選主。

故障修復步驟:
1.修復db
/etc/init.d/mysqld start
2修復主從
>change master to master_host='192.168.100.150',master_port=3306,master_auto_position=1,master_user='mha',master_passwd='manager',master_log_file='master-bin.000001'
change master to master_host='192.168.100.150',master_user='myslave',master_password='123',master_log_file='master-bin.000001',master_log_pos=1294;
>start slave;
3.修改配置文件(再把這個記錄添加進去，因爲它檢測掉失效時候會自動消失)
vi /etc/masterha/app1.cnf
[server1]
hostname=192.168.195.129
port=3306
4.啓動manager(在manager那臺機器上)
nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --gnore _last failover < /dev/null

dos2unix /usr/local/bin/master_ip_failover解決中英字不兼容報錯的問題