一、MHA
MHA是開源的MySQL的高可用程序，它爲MySQL的主從複製架構提供了主節點故障自動轉移的功能，它會監控master節點故障的時候，會提升其中的擁有最新數據的slave節點稱爲新的master節點，同時它還提供了master節點的在線切換的功能，按需切換master/slave節點。
MHA服務有兩種角色，MHA Manager和MHA Node：
MHA Manager（管理節點）：單獨部署在一臺獨立機器上管理多個master/slave主從複製集羣， 每個master/slave稱爲一個application |
| MHA Node（數據節點）：運行在每臺MySQL服務器上，通過腳本來加快故障轉移 |
二、MHA組件
Mnager管理節點： -- masterha_check_ssh : 檢查MHA的SSH配置。 |
Node節點： -- save_binary_logs : 保存和複製master的二進制日誌。 |
三、MHA實現主節點故障自動故障轉移過程：
1、準備工作
①測試環境centos6.6
| 主機名 | MHA角色 | MySQL主從架構角色 | ip地址 |
| manager | Manager | 172.16.16.8 | |
| node1 | Node | master | 172.16.16.2 |
| node2 | Node | slave1 | 172.16.16.3 |
| node3 | Node | slave2 | 172.16.16.4 |
②由於在每個節點上的很多操作都很近似，故使用ansible進行了！配置epel源安裝ansible
# yum install -y python-jinja2-2.2.1-2.el6_5.x86_64.rpm PyYAML-3.10-3.1.el6.x86_64.rpm # yum install -y ansible # vim /etc/ansible/hosts //定義ansible需要管理的節點 [mha] 172.16.16.8 //MHA的manager節點 [db] 172.16.16.2 //MySQL的master 172.16.16.3 //MySQL的slave1 172.16.16.4 //MySQL的slave2
③時間同步及主機互信等 ;基於manager做所有設置；
# vim /etc/hosts //主機名解析
172.16.16.8 manager
172.16.16.2 node1
172.16.16.3 node2
172.16.16.4 node3
# vim /etc/resolv.conf //設置名稱服務器
nameserver 172.16.0.1
# vim /etc/selinux/config //設置selinux
SELINUX=permissive
# ssh-keygen -t rsa -P '' //生成密鑰對
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
f1:e9:55:b5:98:94:40:1b:1e:b5:ef:d3:24:da:3b:cc root@manager
The key's randomart image is:
+--[ RSA 2048]----+
| .=oo. .|
| . =.+..|
| . o +.. |
| o . .. |
| S o . ...|
| . . o.o.|
| . .o.o.|
| E..|
| .. |
+-----------------+
[root@manager ~]# cat .ssh/id_rsa.pub >> .ssh/authorized_keys
//公鑰信息，4個節點使用同一個
[root@manager ~]# cd .ssh/
[root@manager .ssh]# chmod 600 ./* //設置權限
[root@manager ~]# ansible all -m ping //測試ansible是否可正常管理定義節點
172.16.16.8 | success >> {
"changed": false,
"ping": "pong"
}
172.16.16.2 | success >> {
"changed": false,
"ping": "pong"
}
172.16.16.3 | success >> {
"changed": false,
"ping": "pong"
}
172.16.16.4 | success >> {
"changed": false,
"ping": "pong"
}
//爲db的3個節點配置相關信息
# ansible all -m shell -a 'yum install -y libselinux-python'
#ansible db -m copy -a "src=/etc/hosts dest=/etc/hosts"
# ansible db -m copy -a "src=/etc/yum.repo.d/epel.repo dest=/etc/yum.repo.d/epel.repo"
# ansible db -m copy -a "src=/etc/resolv.conf dest=/etc/resolv.conf"
# ansible db -m copy -a "src=/etc/selinux/config dest=/etc/selinux/config"
# ansible db -m yum -a 'name=mysql-server state=present' //爲db節點安裝mysql2、配置MySQL的主從複製架構
master
# vim /etc/my.cnf innodb_file_per_table=1 # skip_name_resolve=1 //centos6.6自帶的mysql-server不支持？？ log_bin=master-bin relay_log=relay-log server_id=1 // 啓動mysqld時候報錯 [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.host' doesn't exist //解決方法：# mysql_install_db //先要初始化數據庫 [root@node1 ~]# service mysqld start Starting mysqld: [ OK ] mysql> show master status; //查看當前使用的二進制日誌的事件位置 +-------------------+----------+--------------+------------------+ | File | Position | Binlog_Do_DB | Binlog_Ignore_DB | +-------------------+----------+--------------+------------------+ | master-bin.000003 | 106 | | | +-------------------+----------+--------------+------------------+ 1 row in set (0.00 sec) mysql> grant replication slave, replication client on *.* to 'repluser'@'172.16.16.%' identified by 'replpass'; //授權複製用戶 Query OK, 0 rows affected (0.00 sec) mysql> flush privileges; Query OK, 0 rows affected (0.00 sec) mysql> grant all on *.* to 'mha'@'172.16.16.%' identified by 'mhapass'; Query OK, 0 rows affected (0.00 sec) //授權遠程登錄用戶 mysql> flush privileges; Query OK, 0 rows affected (0.00 sec)
slave1和slave2
# vim /etc/my.cnf innodb_file_per_table=1 //每表一個表空間 skip_name_resolve //跳過主機名反解 log_bin=master-bin //二進制日誌 relay_log=relay-log //中繼日誌 server_id=2 // 全局唯一id號； 注：slave2的設置爲server_id=3 read_only=1 relay_log_purge=0 mysql> change master to master_host='172.16.16.2',master_user='repluser',master_password='replpass',master_log_file='master-bin.000003',master_log_pos=106; Query OK, 0 rows affected (0.08 sec) mysql> start slave; Query OK, 0 rows affected (0.00 sec) mysql> show slave status\G; *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: 172.16.16.2 Master_User: repluser Master_Port: 3306 Connect_Retry: 60 Master_Log_File: master-bin.000003 Read_Master_Log_Pos: 572 Relay_Log_File: relay-log.000002 Relay_Log_Pos: 718 Relay_Master_Log_File: master-bin.000003 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 572 Relay_Log_Space: 867 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: 1 row in set (0.00 sec)
至此主從複製架構配置完成！！！接下來配置MHA~~
3、配置MHA Manager和MHA Node節點
//MHA Manager要安裝的所有包及相應依賴包 ； mha4mysql-manager-0.56-0.el6.noarch.rpm mha4mysql-node-0.56-0.el6.noarch.rpm perl-Config-Tiny-2.12-7.1.el6.noarch.rpm perl-Email-Date-Format-1.002-5.el6.noarch.rpm perl-MIME-Lite-3.027-2.el6.noarch.rpm perl-MIME-Types-1.28-2.el6.noarch.rpm //MHA Node只需安裝 mha4mysql-node-0.56-0.el6.noarch.rpm //安裝後生成的文件 [root@manager ~]# rpm -ql mha4mysql-manager /usr/bin/masterha_check_repl /usr/bin/masterha_check_ssh /usr/bin/masterha_check_status /usr/bin/masterha_conf_host /usr/bin/masterha_manager /usr/bin/masterha_master_monitor /usr/bin/masterha_master_switch /usr/bin/masterha_secondary_check /usr/bin/masterha_stop [root@manager ~]# rpm -ql mha4mysql-node /usr/bin/apply_diff_relay_logs /usr/bin/filter_mysqlbinlog /usr/bin/purge_relay_logs /usr/bin/save_binary_logs [server default] user=mha //遠程登錄MySQL數據庫的用戶名 password=mhapass //遠程登錄MySQL數據庫用戶的密碼 manager_workdir=/data/mha/app1 //MHA管理節點的工作目錄,自動創建 manager_log=/data/mha/app1/manager.log //日誌文件 remote_workdir=/data/mha/app1 //遠程節點的工作目錄 ssh_user=root //ssh遠程連接默認使用的用戶 repl_user=repluser //MySQL集羣複製數據庫的用戶名 repl_password=replpass //密碼 ping_interval=1 //ping主節點的時間間隔爲1s [server1] //定義MySQL集羣節點 hostname=172.16.16.2 candidate_master=1 //有可能會提升爲主節點 [server2] hostname=172.16.16.3 candidate_master=1 [server3] hostname=172.16.16.4 candidate_master=1 ~ //檢查各節點間ssh互信通信配置是否ok！ [root@manager ~]# masterha_check_ssh --conf=/etc/mha/app1.cnf Sat Oct 24 02:14:49 2015 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sat Oct 24 02:14:49 2015 - [info] Reading application default configuration from /etc/mha/app1.cnf.. Sat Oct 24 02:14:49 2015 - [info] Reading server configuration from /etc/mha/app1.cnf.. Sat Oct 24 02:14:49 2015 - [info] Starting SSH connection tests.. Sat Oct 24 02:14:52 2015 - [debug] Sat Oct 24 02:14:49 2015 - [debug] Connecting via SSH from [email protected](172.16.16.2:22) to [email protected](172.16.16.3:22).. Sat Oct 24 02:14:50 2015 - [debug] ok. Sat Oct 24 02:14:50 2015 - [debug] Connecting via SSH from [email protected](172.16.16.2:22) to [email protected](172.16.16.4:22).. Sat Oct 24 02:14:52 2015 - [debug] ok. Sat Oct 24 02:14:52 2015 - [debug] Sat Oct 24 02:14:50 2015 - [debug] Connecting via SSH from [email protected](172.16.16.3:22) to [email protected](172.16.16.2:22).. Sat Oct 24 02:14:51 2015 - [debug] ok. Sat Oct 24 02:14:51 2015 - [debug] Connecting via SSH from [email protected](172.16.16.3:22) to [email protected](172.16.16.4:22).. Sat Oct 24 02:14:52 2015 - [debug] ok. Sat Oct 24 02:14:53 2015 - [debug] Sat Oct 24 02:14:50 2015 - [debug] Connecting via SSH from [email protected](172.16.16.4:22) to [email protected](172.16.16.2:22).. Sat Oct 24 02:14:52 2015 - [debug] ok. Sat Oct 24 02:14:52 2015 - [debug] Connecting via SSH from [email protected](172.16.16.4:22) to [email protected](172.16.16.3:22).. Sat Oct 24 02:14:53 2015 - [debug] ok. Sat Oct 24 02:14:53 2015 - [info] All SSH connection tests passed successfully. //檢查管理的MySQL複製集羣的連接配置參數是否ok？ [root@manager ~]# masterha_check_repl --conf=/etc/mha/app1.cnf Sat Oct 24 02:14:55 2015 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sat Oct 24 02:14:55 2015 - [info] Reading application default configuration from /etc/mha/app1.cnf.. Sat Oct 24 02:14:55 2015 - [info] Reading server configuration from /etc/mha/app1.cnf.. Sat Oct 24 02:14:55 2015 - [info] MHA::MasterMonitor version 0.56. Sat Oct 24 02:14:55 2015 - [info] GTID failover mode = 0 Sat Oct 24 02:14:55 2015 - [info] Dead Servers: Sat Oct 24 02:14:55 2015 - [info] Alive Servers: Sat Oct 24 02:14:55 2015 - [info] 172.16.16.2(172.16.16.2:3306) Sat Oct 24 02:14:55 2015 - [info] 172.16.16.3(172.16.16.3:3306) Sat Oct 24 02:14:55 2015 - [info] 172.16.16.4(172.16.16.4:3306) Sat Oct 24 02:14:55 2015 - [info] Alive Slaves: Sat Oct 24 02:14:55 2015 - [info] 172.16.16.3(172.16.16.3:3306) Version=5.1.73-log (oldest major version between slaves) log-bin:enabled Sat Oct 24 02:14:55 2015 - [info] Replicating from 172.16.16.2(172.16.16.2:3306) Sat Oct 24 02:14:55 2015 - [info] Primary candidate for the new Master (candidate_master is set) Sat Oct 24 02:14:55 2015 - [info] 172.16.16.4(172.16.16.4:3306) Version=5.1.73-log (oldest major version between slaves) log-bin:enabled Sat Oct 24 02:14:55 2015 - [info] Replicating from 172.16.16.2(172.16.16.2:3306) Sat Oct 24 02:14:55 2015 - [info] Primary candidate for the new Master (candidate_master is set) Sat Oct 24 02:14:55 2015 - [info] Current Alive Master: 172.16.16.2(172.16.16.2:3306) Sat Oct 24 02:14:55 2015 - [info] Checking slave configurations.. Sat Oct 24 02:14:55 2015 - [info] Checking replication filtering settings.. Sat Oct 24 02:14:55 2015 - [info] binlog_do_db= , binlog_ignore_db= Sat Oct 24 02:14:55 2015 - [info] Replication filtering check ok. Sat Oct 24 02:14:55 2015 - [info] GTID (with auto-pos) is not supported Sat Oct 24 02:14:55 2015 - [info] Starting SSH connection tests.. Sat Oct 24 02:14:59 2015 - [info] All SSH connection tests passed successfully. Sat Oct 24 02:14:59 2015 - [info] Checking MHA Node version.. Sat Oct 24 02:15:00 2015 - [info] Version check ok. Sat Oct 24 02:15:00 2015 - [info] Checking SSH publickey authentication settings on the current master.. Sat Oct 24 02:15:01 2015 - [info] HealthCheck: SSH to 172.16.16.2 is reachable. Sat Oct 24 02:15:01 2015 - [info] Master MHA Node version is 0.56. Sat Oct 24 02:15:01 2015 - [info] Checking recovery script configurations on 172.16.16.2(172.16.16.2:3306).. Sat Oct 24 02:15:01 2015 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql,/var/log/mysql --output_file=/data/mha/app1/save_binary_logs_test --manager_version=0.56 --start_file=master-bin.000004 Sat Oct 24 02:15:01 2015 - [info] Connecting to [email protected](172.16.16.2:22).. Creating /data/mha/app1 if not exists.. ok. Checking output directory is accessible or not.. ok. Binlog found at /var/lib/mysql, up to master-bin.000004 Sat Oct 24 02:15:02 2015 - [info] Binlog setting check done. Sat Oct 24 02:15:02 2015 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers.. Sat Oct 24 02:15:02 2015 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='mha' --slave_host=172.16.16.3 --slave_ip=172.16.16.3 --slave_port=3306 --workdir=/data/mha/app1 --target_version=5.1.73-log --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/ --slave_pass=xxx Sat Oct 24 02:15:02 2015 - [info] Connecting to [email protected](172.16.16.3:22).. Checking slave recovery environment settings.. Opening /var/lib/mysql/relay-log.info ... ok. Relay log found at /var/lib/mysql, up to relay-log.000005 Temporary relay log file is /var/lib/mysql/relay-log.000005 Testing mysql connection and privileges.. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done. Sat Oct 24 02:15:03 2015 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='mha' --slave_host=172.16.16.4 --slave_ip=172.16.16.4 --slave_port=3306 --workdir=/data/mha/app1 --target_version=5.1.73-log --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/ --slave_pass=xxx Sat Oct 24 02:15:03 2015 - [info] Connecting to [email protected](172.16.16.4:22).. Creating directory /data/mha/app1.. done. Checking slave recovery environment settings.. Opening /var/lib/mysql/relay-log.info ... ok. Relay log found at /var/lib/mysql, up to relay-log.000006 Temporary relay log file is /var/lib/mysql/relay-log.000006 Testing mysql connection and privileges.. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done. Sat Oct 24 02:15:03 2015 - [info] Slaves settings check done. Sat Oct 24 02:15:03 2015 - [info] 172.16.16.2(172.16.16.2:3306) (current master) //看到這段日誌信息 +--172.16.16.3(172.16.16.3:3306) +--172.16.16.4(172.16.16.4:3306) Sat Oct 24 02:15:03 2015 - [info] Checking replication health on 172.16.16.3.. Sat Oct 24 02:15:03 2015 - [info] ok. Sat Oct 24 02:15:03 2015 - [info] Checking replication health on 172.16.16.4.. Sat Oct 24 02:15:03 2015 - [info] ok. Sat Oct 24 02:15:03 2015 - [warning] master_ip_failover_script is not defined.//沒有vip Sat Oct 24 02:15:03 2015 - [warning] shutdown_script is not defined. //沒有這個腳本 Sat Oct 24 02:15:03 2015 - [info] Got exit code 0 (Not master dead). MySQL Replication Health is OK. //看到這個信息說明ok //啓動 MHA [root@manager ~]# nohup masterha_manager --conf=/etc/mha/app1.cnf > /data/mha/app1/manager.log 2>&1 & [1] 4007 //查看監控狀態；主節點是172.16.16.2 [root@manager ~]# masterha_check_status --conf=/etc/mha/app1.cnf app1 (pid:4007) is running(0:PING_OK), master:172.16.16.2
4、測試故障轉移
（1）在master節點上關閉mysqld服務
[root@node1 ~]# killall -9 mysqld mysqld_safe
（2）查看manager節點的日誌
----- Failover Report ----- //故障轉移報告 app1: MySQL Master failover 172.16.16.2(172.16.16.2:3306) to 172.16.16.3(172.16.16.3:3306) succeeded Master 172.16.16.2(172.16.16.2:3306) is down! Check MHA Manager logs at manager:/data/mha/app1/manager.log for details. Started automated(non-interactive) failover. The latest slave 172.16.16.3(172.16.16.3:3306) has all relay logs for recovery. Selected 172.16.16.3(172.16.16.3:3306) as a new master. 172.16.16.3(172.16.16.3:3306): OK: Applying all logs succeeded. 172.16.16.4(172.16.16.4:3306): This host has the latest relay log events. Generating relay diff files from the latest slave succeeded. 172.16.16.4(172.16.16.4:3306): OK: Applying all logs succeeded. Slave started, replicating from 172.16.16.3(172.16.16.3:3306) 172.16.16.3(172.16.16.3:3306): Resetting slave info succeeded. Master failover to 172.16.16.3(172.16.16.3:3306) completed successfully.
5、還需改進的地方
需要提供下面的2個腳本；後續補充？！
master_ip_failover_script //主節點ip轉移，類似於keepalived的vip shutdown_script //爲避免資源爭用，類似於高可用集羣的stonith
至此，MHA高可用工具介紹完畢，基本實現了主節點的出現故障後的自動轉移，O(∩_∩)O~~