redis集羣實踐
- 1、創建集羣
[root@test04 7000]# redis-trib.rb create --replicas 1 10.9.251.104:7000 10.9.251.104:7001 10.9.251.104:7002 10.9.251.103:7000 10.9.251.103:7001 10.9.251.103:7002
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
10.9.251.104:7000
10.9.251.103:7000
10.9.251.104:7001
Adding replica 10.9.251.103:7001 to 10.9.251.104:7000
Adding replica 10.9.251.104:7002 to 10.9.251.103:7000
Adding replica 10.9.251.103:7002 to 10.9.251.104:7001
M: 23719618cb3996b776a5926061fce31be398d68b 10.9.251.104:7000
slots:0-5460 (5461 slots) master
M: bf01d4e7da106eaeaa52ccda12b310a59cf3c655 10.9.251.104:7001
slots:10923-16383 (5461 slots) master
S: 88645c07a73eb3df4b8598365b0bc07278c31929 10.9.251.104:7002
replicates 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2
M: 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2 10.9.251.103:7000
slots:5461-10922 (5462 slots) master
S: dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba 10.9.251.103:7001
replicates 23719618cb3996b776a5926061fce31be398d68b
S: 832cb8bf04047db470e67db4656d992d1df8c1b2 10.9.251.103:7002
replicates bf01d4e7da106eaeaa52ccda12b310a59cf3c655
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
29813:M 11 Jan 13:45:08.977 # configEpoch set to 1 via CLUSTER SET-CONFIG-EPOCH
29801:M 11 Jan 13:45:08.978 # configEpoch set to 2 via CLUSTER SET-CONFIG-EPOCH
29645:M 11 Jan 13:45:08.978 # configEpoch set to 3 via CLUSTER SET-CONFIG-EPOCH
>>> Sending CLUSTER MEET messages to join the cluster
29813:M 11 Jan 13:45:08.990 # IP address for this node updated to 10.9.251.104
29645:M 11 Jan 13:45:09.064 # IP address for this node updated to 10.9.251.104
29801:M 11 Jan 13:45:09.064 # IP address for this node updated to 10.9.251.104
Waiting for the cluster to join....29813:M 11 Jan 13:45:13.876 # Cluster state changed: ok
29801:M 11 Jan 13:45:13.973 # Cluster state changed: ok
29645:S 11 Jan 13:45:14.004 # Cluster state changed: ok
>>> Performing Cluster Check (using node 10.9.251.104:7000)
M: 23719618cb3996b776a5926061fce31be398d68b 10.9.251.104:7000
slots:0-5460 (5461 slots) master
1 additional replica(s)
M: 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2 10.9.251.103:7000
slots:5461-10922 (5462 slots) master
1 additional replica(s)
M: bf01d4e7da106eaeaa52ccda12b310a59cf3c655 10.9.251.104:7001
slots:10923-16383 (5461 slots) master
1 additional replica(s)
S: dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba 10.9.251.103:7001
slots: (0 slots) slave
replicates 23719618cb3996b776a5926061fce31be398d68b
S: 832cb8bf04047db470e67db4656d992d1df8c1b2 10.9.251.103:7002
slots: (0 slots) slave
replicates bf01d4e7da106eaeaa52ccda12b310a59cf3c655
S: 88645c07a73eb3df4b8598365b0bc07278c31929 10.9.251.104:7002
slots: (0 slots) slave
replicates 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
2、連接集羣，設置值
[root@test04 7000]# redis-cli -c -p 7000 -h 10.9.251.104
10.9.251.104:7000> get name
-> Redirected to slot [5798] located at 10.9.251.103:7000
"zhaojiwei"
10.9.251.103:7000> get age
-> Redirected to slot [741] located at 10.9.251.104:7000
"12"
10.9.251.104:7000> get key:001
-> Redirected to slot [13270] located at 10.9.251.104:7001
"val:001"
10.9.251.104:7001>
3、模擬集羣某一主機掛掉情況
以10.9.251.104:7000 掛掉，看數據是否異常。
[root@test04 7000]# redis-trib.rb check 10.9.251.104:7000
>>> Performing Cluster Check (using node 10.9.251.104:7000)
M: 23719618cb3996b776a5926061fce31be398d68b 10.9.251.104:7000
slots:0-5460 (5461 slots) master
1 additional replica(s)
M: 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2 10.9.251.103:7000
slots:5461-10922 (5462 slots) master
1 additional replica(s)
M: bf01d4e7da106eaeaa52ccda12b310a59cf3c655 10.9.251.104:7001
slots:10923-16383 (5461 slots) master
1 additional replica(s)
S: dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba 10.9.251.103:7001
slots: (0 slots) slave
replicates 23719618cb3996b776a5926061fce31be398d68b
S: 832cb8bf04047db470e67db4656d992d1df8c1b2 10.9.251.103:7002
slots: (0 slots) slave
replicates bf01d4e7da106eaeaa52ccda12b310a59cf3c655
S: 88645c07a73eb3df4b8598365b0bc07278c31929 10.9.251.104:7002
slots: (0 slots) slave
replicates 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@test04 7000]#ps -ef|grep redis
root 29645 28360 0 13:28 pts/1 00:00:04 redis-server 10.9.251.104:7002 [cluster]
root 29801 28360 0 13:44 pts/1 00:00:01 redis-server 10.9.251.104:7001 [cluster]
root 29813 28360 0 13:44 pts/1 00:00:03 redis-server 10.9.251.104:7000 [cluster]
root 30407 28360 0 14:52 pts/1 00:00:00 grep redis
[root@test04 7000]#kill 29813
29813:signal-handler (1484117582) Received SIGTERM scheduling shutdown...
[root@test04 7000]# 29813:M 11 Jan 14:53:02.322 # User requested shutdown...
29813:M 11 Jan 14:53:02.322 * Calling fsync() on the AOF file.
29813:M 11 Jan 14:53:02.322 * Saving the final RDB snapshot before exiting.
29813:M 11 Jan 14:53:02.338 * DB saved on disk
29813:M 11 Jan 14:53:02.338 * Removing the pid file.
29813:M 11 Jan 14:53:02.338 # Redis is now ready to exit, bye bye...
3.1現在查看集羣狀態:
可以看到原集羣10.9.251.104:7000的副節點10.9.251.103:7001變更爲主節點，頂替104:7000。集羣繼續運行。現在是3個主節點，2個副本節點如下：
[root@test03 7002]# redis-trib.rb check 10.9.251.103:7000
>>> Performing Cluster Check (using node 10.9.251.103:7000)
M: 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2 10.9.251.103:7000
slots:5461-10922 (5462 slots) master
1 additional replica(s)
M: dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba 10.9.251.103:7001
slots:0-5460 (5461 slots) master
0 additional replica(s)
S: 832cb8bf04047db470e67db4656d992d1df8c1b2 10.9.251.103:7002
slots: (0 slots) slave
replicates bf01d4e7da106eaeaa52ccda12b310a59cf3c655
S: 88645c07a73eb3df4b8598365b0bc07278c31929 10.9.251.104:7002
slots: (0 slots) slave
replicates 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2
M: bf01d4e7da106eaeaa52ccda12b310a59cf3c655 10.9.251.104:7001
slots:10923-16383 (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
3.2 查看此時集羣數據
[root@test03 7002]# redis-cli -c -p 7000 -h 10.9.251.103
10.9.251.103:7000> get name
"zhaojiwei"
10.9.251.103:7000> get age
-> Redirected to slot [741] located at 10.9.251.103:7001
"12"
10.9.251.103:7001> get addres
"\xe6\x9c\x9d\xe9\x98\xb3\xe5\x8c\xba"
10.9.251.103:7001> get key:001
-> Redirected to slot [13270] located at 10.9.251.104:7001
"val:001"
10.9.251.104:7001>
這與原集羣數據對比下：可以發現原來key爲age的數據位於104:7000上，現在位於103:7001上。數據並沒有丟失。
3.3
如果實在是太不幸，剛頂替的主節點10.9.251.103:7001也掛掉了，這時候集羣會怎麼樣？
too bad！集羣哈希槽16348個不能被完全覆蓋。集羣掛了，讀和寫都不可以。
也即是說集羣需要至少有3個主節點在運行。
[root@test03 7002]# ps -ef|grep redis
root 17012 16261 0 13:28 pts/3 00:00:02 redis-server 10.9.251.103:7000 [cluster]
root 17020 16261 0 13:29 pts/3 00:00:05 redis-server 10.9.251.103:7001 [cluster]
root 17076 16261 0 13:35 pts/3 00:00:08 redis-server 10.9.251.103:7002 [cluster]
root 17662 16261 0 15:07 pts/3 00:00:00 grep redis
[root@test03 7002]# kill 17020
17020:signal-handler (1484118485) Received SIGTERM scheduling shutdown...
[root@test03 7002]# 17020:M 11 Jan 15:08:05.200 # User requested shutdown...
17020:M 11 Jan 15:08:05.200 * Calling fsync() on the AOF file.
17020:M 11 Jan 15:08:05.200 * Saving the final RDB snapshot before exiting.
17020:M 11 Jan 15:08:05.219 * DB saved on disk
17020:M 11 Jan 15:08:05.219 * Removing the pid file.
17020:M 11 Jan 15:08:05.219 # Redis is now ready to exit, bye bye...
[root@test04 7000]#redis-trib.rb check 10.9.251.104:7001
>>> Performing Cluster Check (using node 10.9.251.104:7001)
M: bf01d4e7da106eaeaa52ccda12b310a59cf3c655 10.9.251.104:7001
slots:10923-16383 (5461 slots) master
1 additional replica(s)
S: 832cb8bf04047db470e67db4656d992d1df8c1b2 10.9.251.103:7002
slots: (0 slots) slave
replicates bf01d4e7da106eaeaa52ccda12b310a59cf3c655
S: 88645c07a73eb3df4b8598365b0bc07278c31929 10.9.251.104:7002
slots: (0 slots) slave
replicates 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2
M: 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2 10.9.251.103:7000
slots:5461-10922 (5462 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[ERR]Not all 16384 slots are covered by nodes.
[root@test04 7000]# redis-cli -c -p 7001 -h 10.9.251.104
10.9.251.104:7001> get name
(error) CLUSTERDOWN The cluster is down
10.9.251.104:7001> get age
(error) CLUSTERDOWN The cluster is down
10.9.251.104:7001> get addres
(error) CLUSTERDOWN The cluster is down
10.9.251.104:7001> get key:001
(error) CLUSTERDOWN The cluster is down
10.9.251.104:7001>set newkey newval
(error) CLUSTERDOWN The cluster is down
10.9.251.104:7001>
這種情況還能恢復嗎？答案是肯定的。啓動103:7001。
[root@test03 7001]#redis-server
redis.conf &
[1] 17783
[root@test03 7001]# 17783:M 11 Jan 15:22:07.368 * Node configuration loaded, I'm dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 3.2.4 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in cluster mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 7001
| `-._ `._ / _.-' | PID: 17783
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | http://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
17783:M 11 Jan 15:22:07.369 # Server started, Redis version 3.2.4
17783:M 11 Jan 15:22:07.369 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory
= 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
17783:M 11 Jan 15:22:07.369 * DB loaded from append only file: 0.000 seconds
17783:M 11 Jan 15:22:07.370 * The server is now ready to accept connections on port 7001
17783:M 11 Jan 15:22:09.370 # Cluster state changed: ok
通過啓動103:7001後，我們再看看集羣中該服務的狀態，103:7001處於master主節點，含有5461個哈希槽，但無副本。我們啓動104:7000服務後，它會自動變爲103:7001的副本。
[root@test03 7001]# redis-trib.rb check 10.9.251.103:7000
>>> Performing Cluster Check (using node 10.9.251.103:7000)
M: 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2 10.9.251.103:7000
slots:5461-10922 (5462 slots) master
1 additional replica(s)
M: dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba 10.9.251.103:7001
slots:0-5460 (5461 slots) master
0 additional replica(s)
S: 832cb8bf04047db470e67db4656d992d1df8c1b2 10.9.251.103:7002
slots: (0 slots) slave
replicates bf01d4e7da106eaeaa52ccda12b310a59cf3c655
S: 88645c07a73eb3df4b8598365b0bc07278c31929 10.9.251.104:7002
slots: (0 slots) slave
replicates 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2
M: bf01d4e7da106eaeaa52ccda12b310a59cf3c655 10.9.251.104:7001
slots:10923-16383 (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
啓動104:7000服務：
[root@test04 7000]# redis-server redis.conf &
...
[root@test03 7002]# redis-trib.rb check 10.9.251.103:7000
>>> Performing Cluster Check (using node 10.9.251.103:7000)
M: 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2 10.9.251.103:7000
slots:5461-10922 (5462 slots) master
1 additional replica(s)
M: dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba10.9.251.103:7001
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: 23719618cb3996b776a5926061fce31be398d68b 10.9.251.104:7000
slots: (0 slots) slave
replicates dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba
S: 832cb8bf04047db470e67db4656d992d1df8c1b2 10.9.251.103:7002
slots: (0 slots) slave
replicates bf01d4e7da106eaeaa52ccda12b310a59cf3c655
S: 88645c07a73eb3df4b8598365b0bc07278c31929 10.9.251.104:7002
slots: (0 slots) slave
replicates 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2
M: bf01d4e7da106eaeaa52ccda12b310a59cf3c655 10.9.251.104:7001
slots:10923-16383 (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
4、模擬新添加節點
我們在104服務器上同樣創建服務104:7003。
[root@test04 redis-cluster-test]#pwd
/usr/local/redis-cluster-test
[root@test04 redis-cluster-test]#ll
total 60
drwxr-xr-x. 2 root root 4096 Jan 11 15:29 7000
drwxr-xr-x. 2 root root 4096 Jan 11 14:43 7001
drwxr-xr-x. 2 root root 4096 Jan 11 14:41 7002
[root@test04 redis-cluster-test]#mkdir 7003&cp 7000/redis.conf 7003
[4] 30805
[4]+ Done mkdir 7003
[root@test04 redis-cluster-test]#ll
total 64
drwxr-xr-x. 2 root root 4096 Jan 11 15:29 7000
drwxr-xr-x. 2 root root 4096 Jan 11 14:43 7001
drwxr-xr-x. 2 root root 4096 Jan 11 14:41 7002
drwxr-xr-x. 2 root root 4096 Jan 11 15:37 7003
[root@test04 redis-cluster-test]#sed -i "s/7000/7003/g" 7003/redis.conf
[root@test04 redis-cluster-test]#cd 7003
[root@test04 7003]#redis-server redis.conf &
...
[root@test04 7003]# ps -ef|grep redis
root 29645 28360 0 13:28 pts/1 00:00:05 redis-server 10.9.251.104:7002 [cluster]
root 29801 28360 0 13:44 pts/1 00:00:02 redis-server 10.9.251.104:7001 [cluster]
root 30736 28360 0 15:29 pts/1 00:00:01 redis-server 10.9.251.104:7000 [cluster]
root 30883 28360 0 15:43 pts/1 00:00:00 redis-server 10.9.251.104:7003 [cluster]
root 30887 28360 0 15:43 pts/1 00:00:00 grep redis
[root@test04 7003]# redis-trib.rb check 10.9.251.104:7001
>>> Performing Cluster Check (using node 10.9.251.104:7001)
M: bf01d4e7da106eaeaa52ccda12b310a59cf3c655 10.9.251.104:7001
slots:10923-16383 (5461 slots) master
1 additional replica(s)
S: 23719618cb3996b776a5926061fce31be398d68b 10.9.251.104:7000
slots: (0 slots) slave
replicates dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba
S: 832cb8bf04047db470e67db4656d992d1df8c1b2 10.9.251.103:7002
slots: (0 slots) slave
replicates bf01d4e7da106eaeaa52ccda12b310a59cf3c655
M: dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba 10.9.251.103:7001
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: 88645c07a73eb3df4b8598365b0bc07278c31929 10.9.251.104:7002
slots: (0 slots) slave
replicates 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2
M: 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2 10.9.251.103:7000
slots:5461-10922 (5462 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
可以看104:7003已啓動，但是它現在還不是集羣的一部分，怎辦？
add-node 命令：添加新節點104:7003 到集羣。
[root@test04 7003]# redis-trib.rb add-node 10.9.251.104:7003 10.9.251.104:7000
>>> Adding node 10.9.251.104:7003 to cluster 10.9.251.104:7000
>>> Performing Cluster Check (using node 10.9.251.104:7000)
S: 23719618cb3996b776a5926061fce31be398d68b 10.9.251.104:7000
slots: (0 slots) slave
replicates dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba
S: 88645c07a73eb3df4b8598365b0bc07278c31929 10.9.251.104:7002
slots: (0 slots) slave
replicates 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2
M: dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba 10.9.251.103:7001
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: 832cb8bf04047db470e67db4656d992d1df8c1b2 10.9.251.103:7002
slots: (0 slots) slave
replicates bf01d4e7da106eaeaa52ccda12b310a59cf3c655
M: 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2 10.9.251.103:7000
slots:5461-10922 (5462 slots) master
1 additional replica(s)
M: bf01d4e7da106eaeaa52ccda12b310a59cf3c655 10.9.251.104:7001
slots:10923-16383 (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>>Send CLUSTER MEET to node 10.9.251.104:7003 to make it join the cluster.
[OK] New node added correctly.
[root@test04 7003]# 30883:M 11 Jan 15:50:28.187 # IP address for this node updated to 10.9.251.104
30883:M 11 Jan 15:50:33.212 # Cluster state changed: ok
[root@test04 7003]#redis-trib.rb check 10.9.251.104:7001
>>> Performing Cluster Check (using node 10.9.251.104:7001)
M: bf01d4e7da106eaeaa52ccda12b310a59cf3c655 10.9.251.104:7001
slots:10923-16383 (5461 slots) master
1 additional replica(s)
M: 624ef83bad5df4c7091c547afc336e6674a82f92 10.9.251.104:7003
slots: (0 slots) master
0 additional replica(s)
S: 23719618cb3996b776a5926061fce31be398d68b 10.9.251.104:7000
slots: (0 slots) slave
replicates dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba
S: 832cb8bf04047db470e67db4656d992d1df8c1b2 10.9.251.103:7002
slots: (0 slots) slave
replicates bf01d4e7da106eaeaa52ccda12b310a59cf3c655
M: dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba 10.9.251.103:7001
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: 88645c07a73eb3df4b8598365b0bc07278c31929 10.9.251.104:7002
slots: (0 slots) slave
replicates 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2
M: 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2 10.9.251.103:7000
slots:5461-10922 (5462 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
我們看下集羣狀態可知，現在新加的節點104:7003作爲master主節點，但其下並沒有分配哈希槽也沒有副本。好吧，讓我們先搞下哈希槽。
4.1 爲新增節點分配哈希槽
4.1.1 給節點分配指定大小哈希槽（reshard）
[root@test04 7003]# redis-trib.rb reshard 10.9.251.104:7000
>>> Performing Cluster Check (using node 10.9.251.104:7000)
S: 23719618cb3996b776a5926061fce31be398d68b 10.9.251.104:7000
slots: (0 slots) slave
replicates dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba
S: 88645c07a73eb3df4b8598365b0bc07278c31929 10.9.251.104:7002
slots: (0 slots) slave
replicates 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2
M: dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba 10.9.251.103:7001
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: 832cb8bf04047db470e67db4656d992d1df8c1b2 10.9.251.103:7002
slots: (0 slots) slave
replicates bf01d4e7da106eaeaa52ccda12b310a59cf3c655
M: 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2 10.9.251.103:7000
slots:5461-10922 (5462 slots) master
1 additional replica(s)
M: bf01d4e7da106eaeaa52ccda12b310a59cf3c655 10.9.251.104:7001
slots:10923-16383 (5461 slots) master
1 additional replica(s)
M:624ef83bad5df4c7091c547afc336e6674a82f9210.9.251.104:7003
slots: (0 slots) master
0 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
詢問劃分多大哈希槽給目標節點：
How many slots do you want to move (from 1 to 16384)? 4096
詢問哪個節點接收劃分的哈希槽：
What is the receiving node ID?624ef83bad5df4c7091c547afc336e6674a82f92
Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you entered all the source nodes IDs.
詢問劃分的哈希槽來源的節點：（即源節點）
Source node #1:all
...
Do you want to proceed with the proposed reshard plan (yes/no)? yes
...
ok，現在新節點有了哈希槽了，可用於接收存儲集羣的數據了。
[root@test04 7003]# redis-trib.rb check 10.9.251.104:7001
>>> Performing Cluster Check (using node 10.9.251.104:7001)
M: bf01d4e7da106eaeaa52ccda12b310a59cf3c655 10.9.251.104:7001
slots:12288-16383 (4096 slots) master
1 additional replica(s)
M: 624ef83bad5df4c7091c547afc336e6674a82f92 10.9.251.104:7003
slots:0-1364,5461-6826,10923-12287 (4096 slots) master
0 additional replica(s)
S: 23719618cb3996b776a5926061fce31be398d68b 10.9.251.104:7000
slots: (0 slots) slave
replicates dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba
S: 832cb8bf04047db470e67db4656d992d1df8c1b2 10.9.251.103:7002
slots: (0 slots) slave
replicates bf01d4e7da106eaeaa52ccda12b310a59cf3c655
M: dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba 10.9.251.103:7001
slots:1365-5460 (4096 slots) master
1 additional replica(s)
S:88645c07a73eb3df4b8598365b0bc07278c3192910.9.251.104:7002
slots: (0 slots) slave
replicates 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2
M: 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2 10.9.251.103:7000
slots:6827-10922 (4096 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
假如 新增的節點104:7003處理能力強些，可以將處理能力較弱的103:7000上的槽多分些給它。如下：
[root@test04 7003]# redis-trib.rb reshard 10.9.251.104:7000
>>> Performing Cluster Check (using node 10.9.251.104:7000)
S: 23719618cb3996b776a5926061fce31be398d68b 10.9.251.104:7000
slots: (0 slots) slave
replicates dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba
S: 88645c07a73eb3df4b8598365b0bc07278c31929 10.9.251.104:7002
slots: (0 slots) slave
replicates 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2
M: dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba 10.9.251.103:7001
slots:1365-5460 (4096 slots) master
1 additional replica(s)
S: 832cb8bf04047db470e67db4656d992d1df8c1b2 10.9.251.103:7002
slots: (0 slots) slave
replicates bf01d4e7da106eaeaa52ccda12b310a59cf3c655
M: 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2 10.9.251.103:7000
slots:6827-10922 (4096 slots) master
1 additional replica(s)
M: bf01d4e7da106eaeaa52ccda12b310a59cf3c655 10.9.251.104:7001
slots:12288-16383 (4096 slots) master
1 additional replica(s)
M: 624ef83bad5df4c7091c547afc336e6674a82f92 10.9.251.104:7003
slots:0-1364,5461-6826,10923-12287 (4096 slots) master
0 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)?5120
What is the receiving node ID? 624ef83bad5df4c7091c547afc336e6674a82f92
Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you entered all the source nodes IDs.
來源於103:7000節點id的槽：
Source node #1:4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2
Source node #2:done
...
Do you want to proceed with the proposed reshard plan (yes/no)? yes
...
[root@test04 7003]#redis-trib.rb check 10.9.251.104:7001
>>> Performing Cluster Check (using node 10.9.251.104:7001)
M: bf01d4e7da106eaeaa52ccda12b310a59cf3c655 10.9.251.104:7001
slots:12288-16383 (4096 slots) master
1 additional replica(s)
M: 624ef83bad5df4c7091c547afc336e6674a82f92 10.9.251.104:7003
slots:0-1364,5461-12287 (8192 slots) master
1 additional replica(s)
S: 23719618cb3996b776a5926061fce31be398d68b 10.9.251.104:7000
slots: (0 slots) slave
replicates dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba
S: 832cb8bf04047db470e67db4656d992d1df8c1b2 10.9.251.103:7002
slots: (0 slots) slave
replicates bf01d4e7da106eaeaa52ccda12b310a59cf3c655
M: dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba 10.9.251.103:7001
slots:1365-5460 (4096 slots) master
1 additional replica(s)
S:88645c07a73eb3df4b8598365b0bc07278c3192910.9.251.104:7002
slots: (0 slots) slave
replicates 624ef83bad5df4c7091c547afc336e6674a82f92
M: 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2 10.9.251.103:7000
slots: (0 slots) master
0 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
我擦， 有沒有發現103:7000上所有槽被移到 104:7003上！！ 看來從103:7000上移動5120個槽太多了，沒注意到它原有才4096個【slots:6827-10922 (4096 slots) master】。不過更有意外的是103:7000上也沒有副本了，其副本也被分配給了104:7003上了！！
可見當節點被剝奪哈希槽時，存在副本也沒意義了，副本會被一起剝奪！
4.1.2 均分哈希槽 rebalance
問題1：主節點哈希槽爲0時能不能用rebalance來均分集羣下的哈希槽呢？不能
拿上面的例子103:7000被我們剝奪了所有的槽。
[root@test04 7003]#redis-trib.rb rebalance 10.9.251.103:7000
>>> Performing Cluster Check (using node 10.9.251.103:7000)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Rebalancing across 3 nodes. Total weight = 3
Moving 1366 slots from 10.9.251.104:7003 to 10.9.251.103:7001
...
[root@test04 7003]# redis-trib.rb check 10.9.251.104:7001
>>> Performing Cluster Check (using node 10.9.251.104:7001)
M: bf01d4e7da106eaeaa52ccda12b310a59cf3c655 10.9.251.104:7001
slots:5462-6826,12288-16383 (5461 slots) master
1 additional replica(s)
M: 624ef83bad5df4c7091c547afc336e6674a82f92 10.9.251.104:7003
slots:6827-12287 (5461 slots) master
1 additional replica(s)
S: 23719618cb3996b776a5926061fce31be398d68b 10.9.251.104:7000
slots: (0 slots) slave
replicates dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba
S: 832cb8bf04047db470e67db4656d992d1df8c1b2 10.9.251.103:7002
slots: (0 slots) slave
replicates bf01d4e7da106eaeaa52ccda12b310a59cf3c655
M: dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba 10.9.251.103:7001
slots:0-5461 (5462 slots) master
1 additional replica(s)
S: 88645c07a73eb3df4b8598365b0bc07278c31929 10.9.251.104:7002
slots: (0 slots) slave
replicates 624ef83bad5df4c7091c547afc336e6674a82f92
M: 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2 10.9.251.103:7000
slots: (0 slots) master
0 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
吼吼，多麼悲催，103:7000上依舊沒有分配到哈希槽。
我們先用reshard分給2個槽給103:7000上，然後再rebalance呢？
[root@test04 7003]# redis-trib.rb reshard 10.9.251.104:7000
>>> Performing Cluster Check (using node 10.9.251.104:7000)
S: 23719618cb3996b776a5926061fce31be398d68b 10.9.251.104:7000
slots: (0 slots) slave
replicates dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba
S: 88645c07a73eb3df4b8598365b0bc07278c31929 10.9.251.104:7002
slots: (0 slots) slave
replicates 624ef83bad5df4c7091c547afc336e6674a82f92
M: dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba 10.9.251.103:7001
slots:0-5461 (5462 slots) master
1 additional replica(s)
S: 832cb8bf04047db470e67db4656d992d1df8c1b2 10.9.251.103:7002
slots: (0 slots) slave
replicates bf01d4e7da106eaeaa52ccda12b310a59cf3c655
M: 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2 10.9.251.103:7000
slots: (0 slots) master
0 additional replica(s)
M: bf01d4e7da106eaeaa52ccda12b310a59cf3c655 10.9.251.104:7001
slots:5462-6826,12288-16383 (5461 slots) master
1 additional replica(s)
M: 624ef83bad5df4c7091c547afc336e6674a82f92 10.9.251.104:7003
slots:6827-12287 (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 2
What is the receiving node ID? 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2
Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you entered all the source nodes IDs.
Source node #1:624ef83bad5df4c7091c547afc336e6674a82f92
Source node #2:done
Ready to move 2 slots.
Source nodes:
M: 624ef83bad5df4c7091c547afc336e6674a82f92 10.9.251.104:7003
slots:6827-12287 (5461 slots) master
1 additional replica(s)
Destination node:
M: 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2 10.9.251.103:7000
slots: (0 slots) master
0 additional replica(s)
Resharding plan:
Moving slot 6827 from 624ef83bad5df4c7091c547afc336e6674a82f92
Moving slot 6828 from 624ef83bad5df4c7091c547afc336e6674a82f92
Do you want to proceed with the proposed reshard plan (yes/no)? yes
Moving slot 6827 from 10.9.251.104:7003 to 10.9.251.103:7000:
Moving slot 6828 from 10.9.251.104:7003 to 10.9.251.103:7000:
[root@test04 7003]#redis-trib.rb rebalance 10.9.251.103:7000
>>> Performing Cluster Check (using node 10.9.251.103:7000)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Rebalancing across 4 nodes. Total weight = 4
Moving 1366 slots from 10.9.251.103:7001 to 10.9.251.103:7000
[root@test04 7003]#redis-trib.rb check 10.9.251.104:7001 再check下狀態
>>> Performing Cluster Check (using node 10.9.251.104:7001)
M: bf01d4e7da106eaeaa52ccda12b310a59cf3c655 10.9.251.104:7001
slots:12288-16383 (4096 slots) master
1 additional replica(s)
M: 624ef83bad5df4c7091c547afc336e6674a82f92 10.9.251.104:7003
slots:8192-12287 (4096 slots) master
1 additional replica(s)
S: 23719618cb3996b776a5926061fce31be398d68b 10.9.251.104:7000
slots: (0 slots) slave
replicates dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba
S: 832cb8bf04047db470e67db4656d992d1df8c1b2 10.9.251.103:7002
slots: (0 slots) slave
replicates bf01d4e7da106eaeaa52ccda12b310a59cf3c655
M: dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba 10.9.251.103:7001
slots:1366-5461 (4096 slots) master
1 additional replica(s)
S: 88645c07a73eb3df4b8598365b0bc07278c31929 10.9.251.104:7002
slots: (0 slots) slave
replicates 624ef83bad5df4c7091c547afc336e6674a82f92
M: 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2 10.9.251.103:7000
slots:0-1365,5462-8191 (4096 slots) master
0 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
吼吼，103:7000上有了均分的4096個槽了！！！
看來只有有槽（即能接收存儲數據的節點）節點纔有被rebalance的機會！！！
4.2 爲新增節點分配副本（或者說指定節點作爲）
103:7000被剝奪槽時其副本也被剝脫了！賠了夫人又折兵！現在爲止，爲其分配了4096個槽，但依舊沒有副本。
爲其添加副本節點：在103上創建服務節點7003，並將103:7003分給103:7000作爲副本（僅作爲測試用，爲數據安全實際應該是將slave與master分配到不同的服務器上）。
redis-trib.rb add-node --slave --master-id 目標主節點id 副本節點（ip:port） 集羣節點（ip:port）
[root@test03 7003]#redis-trib.rb add-node --slave --master-id 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2 10.9.251.103:7003
10.9.251.104:7000
>>> Adding node 10.9.251.103:7003 to cluster 10.9.251.104:7000
>>> Performing Cluster Check (using node 10.9.251.104:7000)
S: 23719618cb3996b776a5926061fce31be398d68b 10.9.251.104:7000
slots: (0 slots) slave
replicates dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba
S: 88645c07a73eb3df4b8598365b0bc07278c31929 10.9.251.104:7002
slots: (0 slots) slave
replicates 624ef83bad5df4c7091c547afc336e6674a82f92
M: dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba 10.9.251.103:7001
slots:1366-5461 (4096 slots) master
1 additional replica(s)
S: 832cb8bf04047db470e67db4656d992d1df8c1b2 10.9.251.103:7002
slots: (0 slots) slave
replicates bf01d4e7da106eaeaa52ccda12b310a59cf3c655
M: 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2 10.9.251.103:7000
slots:0-1365,5462-8191 (4096 slots) master
0 additional replica(s)
M: bf01d4e7da106eaeaa52ccda12b310a59cf3c655 10.9.251.104:7001
slots:12288-16383 (4096 slots) master
1 additional replica(s)
M: 624ef83bad5df4c7091c547afc336e6674a82f92 10.9.251.104:7003
slots:8192-12287 (4096 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 10.9.251.103:7003 to make it join the cluster.
Waiting for the cluster to join.18578:M 11 Jan 17:33:36.566 # IP address for this node updated to 10.9.251.103
>>> Configure node as replica of 10.9.251.103:7000.
18578:S 11 Jan 17:33:37.519 # Cluster state changed: ok
[OK] New node added correctly.
5、 刪除節點
5.1刪除副節點
[root@test04 7003]#redis-trib.rb check 10.9.251.104:7000
>>> Performing Cluster Check (using node 10.9.251.104:7000)
S: 23719618cb3996b776a5926061fce31be398d68b 10.9.251.104:7000
slots: (0 slots) slave
replicates dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba
M: bf01d4e7da106eaeaa52ccda12b310a59cf3c655 10.9.251.104:7001
slots:12288-16383 (4096 slots) master
1 additional replica(s)
S: 88645c07a73eb3df4b8598365b0bc07278c31929 10.9.251.104:7002
slots: (0 slots) slave
replicates 624ef83bad5df4c7091c547afc336e6674a82f92
M: 624ef83bad5df4c7091c547afc336e6674a82f92 10.9.251.104:7003
slots:8192-12287 (4096 slots) master
1 additional replica(s)
M: dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba 10.9.251.103:7001
slots:1366-5461 (4096 slots) master
1 additional replica(s)
S: 83d3de5543b7c79630b774001742e115ea5806bd 10.9.251.103:7003
slots: (0 slots) slave
replicates 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2
M: 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2 10.9.251.103:7000
slots:0-1365,5462-8191 (4096 slots) master
1 additional replica(s)
S: 832cb8bf04047db470e67db4656d992d1df8c1b2 10.9.251.103:7002
slots: (0 slots) slave
replicates bf01d4e7da106eaeaa52ccda12b310a59cf3c655
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@test04 7003]#redis-trib.rb del-node 10.9.251.104:7000 23719618cb3996b776a5926061fce31be398d68b
>>> Removing node 23719618cb3996b776a5926061fce31be398d68b from cluster 10.9.251.104:7000
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
刪除副節點，直接可以直接被集羣 forget掉。
恢復該節點：使用命令 redis-server redis.config &
即可將節點加入集羣。
5.1刪除master節點
[root@test04 7000]# redis-trib.rb check 10.9.251.104:7000
>>> Performing Cluster Check (using node 10.9.251.104:7000)
S: 23719618cb3996b776a5926061fce31be398d68b 10.9.251.104:7000
slots: (0 slots) slave
replicates dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba
S: 88645c07a73eb3df4b8598365b0bc07278c31929 10.9.251.104:7002
slots: (0 slots) slave
replicates 624ef83bad5df4c7091c547afc336e6674a82f92
M: 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2 10.9.251.103:7000
slots:0-1365,5462-8191 (4096 slots) master
1 additional replica(s)
S: 83d3de5543b7c79630b774001742e115ea5806bd 10.9.251.103:7003
slots: (0 slots) slave
replicates 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2
S: 832cb8bf04047db470e67db4656d992d1df8c1b2 10.9.251.103:7002
slots: (0 slots) slave
replicates bf01d4e7da106eaeaa52ccda12b310a59cf3c655
M: bf01d4e7da106eaeaa52ccda12b310a59cf3c655 10.9.251.104:7001
slots:12288-16383 (4096 slots) master
1 additional replica(s)
M: dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba 10.9.251.103:7001
slots:1366-5461 (4096 slots) master
1 additional replica(s)
M:624ef83bad5df4c7091c547afc336e6674a82f9210.9.251.104:7003
slots:8192-12287 (4096 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@test04 7000]# redis-trib.rb del-node 10.9.251.103:7000 624ef83bad5df4c7091c547afc336e6674a82f92
>>> Removing node 624ef83bad5df4c7091c547afc336e6674a82f92 from cluster 10.9.251.103:7000
[ERR] Node 10.9.251.104:7003 is not empty! Reshard data away and try again.
[root@test04 7000]#
提示該節點數據不爲空。需要reshard掉該節點哈希槽。 do it！ 將104:7003 上的哈希槽轉移到 103:7001上
[root@test04 7000]#redis-trib.rb reshard 10.9.251.103:7000
>>> Performing Cluster Check (using node 10.9.251.103:7000)
M: 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2 10.9.251.103:7000
slots:0-1365,5462-8191 (4096 slots) master
1 additional replica(s)
M:dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba10.9.251.103:7001
slots:1366-5461 (4096 slots) master
0 additional replica(s)
M:624ef83bad5df4c7091c547afc336e6674a82f9210.9.251.104:7003
slots:8192-12287 (4096 slots) master
1 additional replica(s)
S: 832cb8bf04047db470e67db4656d992d1df8c1b2 10.9.251.103:7002
slots: (0 slots) slave
replicates bf01d4e7da106eaeaa52ccda12b310a59cf3c655
M:bf01d4e7da106eaeaa52ccda12b310a59cf3c655
10.9.251.104:7001
slots:12288-16383 (4096 slots) master
1 additional replica(s)
S: 88645c07a73eb3df4b8598365b0bc07278c31929 10.9.251.104:7002
slots: (0 slots) slave
replicates 624ef83bad5df4c7091c547afc336e6674a82f92
S: 83d3de5543b7c79630b774001742e115ea5806bd 10.9.251.103:7003
slots: (0 slots) slave
replicates 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 4096
What is the receiving node ID?dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba
Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you entered all the source nodes IDs.
Source node #1:624ef83bad5df4c7091c547afc336e6674a82f92
Source node #2:done
...
Moving slot 9792 from 10.9.251.104:7003 to 10.9.251.103:7001:
...
[root@test04 7000]# 5236:M 12 Jan 16:08:58.792 # Connection with slave 10.9.251.104:7002 lost.
可以看到104:7003在移除哈希槽時也丟掉了其副本節點104:7002
[root@test04 7000]# redis-trib.rb check 10.9.251.104:7000
>>> Performing Cluster Check (using node 10.9.251.104:7000)
S: 23719618cb3996b776a5926061fce31be398d68b 10.9.251.104:7000
slots: (0 slots) slave
replicates dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba
S: 88645c07a73eb3df4b8598365b0bc07278c31929 10.9.251.104:7002
slots: (0 slots) slave
replicates dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba
M: 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2 10.9.251.103:7000
slots:0-1365,5462-8191 (4096 slots) master
1 additional replica(s)
S: 83d3de5543b7c79630b774001742e115ea5806bd 10.9.251.103:7003
slots: (0 slots) slave
replicates 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2
S: 832cb8bf04047db470e67db4656d992d1df8c1b2 10.9.251.103:7002
slots: (0 slots) slave
replicates bf01d4e7da106eaeaa52ccda12b310a59cf3c655
M: bf01d4e7da106eaeaa52ccda12b310a59cf3c655 10.9.251.104:7001
slots:12288-16383 (4096 slots) master
1 additional replica(s)
M: dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba 10.9.251.103:7001
slots:1366-5461,8192-12287 (8192slots)
master
2additional replica(s)
M: 624ef83bad5df4c7091c547afc336e6674a82f92 10.9.251.104:7003
slots: (0 slots) master
0 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@test04 7000]#redis-trib.rb del-node 10.9.251.103:7000 624ef83bad5df4c7091c547afc336e6674a82f92
>>> Removing node 624ef83bad5df4c7091c547afc336e6674a82f92 from cluster 10.9.251.103:7000
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
5236:M 12 Jan 16:14:34.170 # User requested shutdown...
5236:M 12 Jan 16:14:34.170 * Calling fsync() on the AOF file.
5236:M 12 Jan 16:14:34.170 * Saving the final RDB snapshot before exiting.
5236:M 12 Jan 16:14:34.191 * DB saved on disk
5236:M 12 Jan 16:14:34.191 * Removing the pid file.
5236:M 12 Jan 16:14:34.191 # Redis is now ready to exit, bye bye...
[1]- Done redis-server redis.conf (wd: /usr/local/redis-cluster-test/7003)
(wd now: /usr/local/redis-cluster-test/7000)
移除主節點後，查看集羣狀態：已不見104:7003
[root@test04 7000]# redis-trib.rb check 10.9.251.104:7000
[ERR]Sorry, can't connect to node 10.9.251.104:7003
>>> Performing Cluster Check (using node 10.9.251.104:7000)
S: 23719618cb3996b776a5926061fce31be398d68b 10.9.251.104:7000
slots: (0 slots) slave
replicates dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba
S: 88645c07a73eb3df4b8598365b0bc07278c31929 10.9.251.104:7002
slots: (0 slots) slave
replicates dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba
M: 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2 10.9.251.103:7000
slots:0-1365,5462-8191 (4096 slots) master
1 additional replica(s)
S: 83d3de5543b7c79630b774001742e115ea5806bd 10.9.251.103:7003
slots: (0 slots) slave
replicates 4b0a1103f4ede0b7c2b77fb0f25f1d95afd80fb2
S: 832cb8bf04047db470e67db4656d992d1df8c1b2 10.9.251.103:7002
slots: (0 slots) slave
replicates bf01d4e7da106eaeaa52ccda12b310a59cf3c655
M: bf01d4e7da106eaeaa52ccda12b310a59cf3c655 10.9.251.104:7001
slots:12288-16383 (4096 slots) master
1 additional replica(s)
M: dc663f990787a87ae12b97a0b1e3c9ee2e2e65ba 10.9.251.103:7001
slots:1366-5461,8192-12287 (8192 slots) master
2 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
如果此時再重啓（比如104:7003服務器修復完畢後重新上線），會如何？
[root@test04 7003]#redis-server redis.conf &
...
M: 624ef83bad5df4c7091c547afc336e6674a82f92 10.9.251.104:7003
slots: (0 slots) master
0 additional replica(s)
...