Redis3.0.7集羣安裝部署
Redis集羣沒有出來前，一直使用Codis集羣，現在部署Redis集羣看看效果如何。
一，架構
Centos6.5 64位
redis1 redis1:6379主 redis3:6380從
redis2 redis2:6379主 redis1:6380從
redis3 redis3:6379主 redis2:6380從
二，部署Redis實例
1，安裝依賴
yum -y install tcl-devel
2，下載
wget http://download.redis.io/releases/redis-3.0.7.tar.gz
3，編譯
cd /usr/local/src/
tar zxvf redis-3.0.7.tar.gz
cd redis-3.0.7
make
4，測試
make test
5，安裝二進制程序到/usr/local/bin/目錄
make install
ls /usr/local/bin/ -lh
6，安裝服務
sh ./utils/install_server.sh Welcome to the redis service installer This script will help you easily set up a running redis server Please select the redis port for this instance: [6379] Selecting default: 6379 Please select the redis config file name [/etc/redis/6379.conf] Selected default - /etc/redis/6379.conf Please select the redis log file name [/var/log/redis_6379.log] /data/redis/redis_6379.log Please select the data directory for this instance [/var/lib/redis/6379] /data/redis/6379 Please select the redis executable path [/usr/local/bin/redis-server] Selected config: Port : 6379 Config file : /etc/redis/6379.conf Log file : /data/redis/redis_6379.log Data dir : /data/redis/6379 Executable : /usr/local/bin/redis-server Cli Executable : /usr/local/bin/redis-cli Is this ok? Then press ENTER to go on or Ctrl-C to abort. Copied /tmp/6379.conf => /etc/init.d/redis_6379 Installing service... Successfully added to chkconfig! Successfully added to runlevels 345! Starting Redis server... Installation successful!
7，驗證服務
/etc/init.d/redis_6379 status
8，其他節點類似安裝
三，部署Cluster環境
1，集羣主從節點關係
redis1 redis1:6379主 redis3:6380從
redis2 redis2:6379主 redis1:6380從
redis3 redis3:6379主 redis2:6380從
2，集羣環境準備
這是源碼安裝
tar -xf ruby-2.2.3.tar.gz
cd ruby-2.2.3
./configure --disable-install-rdoc
make && make install
/usr/local/bin/ruby
/usr/local/bin/gem
YUM安裝
yum install ruby ruby-devel rubygems
設置ruby gem源爲淘寶,安裝gems更快。
# gem sources --add https://ruby.taobao.org/ --remove https://rubygems.org/
查看gem源
# gem sources -l
安裝ruby的redis模塊（依賴接口）
# gem install redis（注意這地方要選定版本排錯時會提到）
3，配置文件(其他實例相應修改端口）
cat /etc/redis/6379.conf |grep -Ev "^#|^$" daemonize yes pidfile /var/run/redis_6379.pid port 6379 timeout 0 logfile /data/redis/redis_6379.log dbfilename dump.rdb dir /data/redis/6379 appendonly no appendfilename "appendonly.aof" appendfsync everysec 取消如下注釋，讓Redis在集羣模式下運行 cluster-enabled yes 啓動cluster模式 cluster-config-file nodes-6379.conf 集羣信息文件名，由redis自己維護 cluster-node-timeout 5000 5秒中聯繫不到對方node，即認爲對方有故障可能
4，重啓redis (其他實例相應修改端口）
/etc/init.d/redis_6379 restart
會在/data/redis/6379/下面多出一個文件nodes-6379.conf
5，利用腳本工具建立集羣
首先查看幫助看redis-trib腳本功能
注意編譯安裝時redis-trib.rb沒有複製過來
cp ./src/redis-trib.rb /usr/local/bin/
ruby redis-trib.rb help
create：創建集羣
check：檢查集羣
info：查看集羣信息
fix：修復集羣
reshard：在線遷移slot
rebalance：平衡集羣節點slot數量
add-node：將新節點加入集羣
del-node：從集羣中刪除節點
set-timeout：設置集羣節點間心跳連接的超時時間
call：在集羣全部節點上執行命令
import：將外部redis數據導入集羣
開始創建集羣注意前三個爲主後三個爲從（但是不能指定主從關係，後面可以在Cluster裏面操作，目的是主從實例不能在統一節點服務器上）， 選項 --replicas 1 表示集羣中的每個主節點創建一個從節點。
/usr/local/bin/redis-trib.rb create --replicas 1 192.168.188.18:6379 192.168.188.28:6379 192.168.188.38:6379 192.168.188.18:6380 192.168.188.28:6380 192.168.188.38:6380 >>> Creating cluster >>> Performing hash slots allocation on 6 nodes... Using 3 masters: 192.168.188.38:6379 192.168.188.28:6379 192.168.188.18:6379 Adding replica 192.168.188.28:6380 to 192.168.188.38:6379 Adding replica 192.168.188.38:6380 to 192.168.188.28:6379 Adding replica 192.168.188.18:6380 to 192.168.188.18:6379 M: 53c202561155d141a8e24d42bb8eb2907998fce9 192.168.188.18:6379 slots:10923-16383 (5461 slots) master M: 654c0a7853724fe33683f54e8df665993ea704ed 192.168.188.28:6379 slots:5461-10922 (5462 slots) master M: b4cc74e0ac6ec9b634d93e86fdb376b4d89dc318 192.168.188.38:6379 slots:0-5460 (5461 slots) master S: ee3d5f85034e5a3c9d9369357b545d0a92751728 192.168.188.18:6380 replicates 53c202561155d141a8e24d42bb8eb2907998fce9 S: 10b558217d6618093b87f75a453ca39679bbcdf4 192.168.188.28:6380 replicates b4cc74e0ac6ec9b634d93e86fdb376b4d89dc318 S: 5dc55e90d3d3b56502a319ec9845e901102cfa6c 192.168.188.38:6380 replicates 654c0a7853724fe33683f54e8df665993ea704ed Can I set the above configuration? (type 'yes' to accept): yes >>> Nodes configuration updated >>> Assign a different config epoch to each node >>> Sending CLUSTER MEET messages to join the cluster Waiting for the cluster to join...... >>> Performing Cluster Check (using node 192.168.188.18:6379) M: 53c202561155d141a8e24d42bb8eb2907998fce9 192.168.188.18:6379 slots:10923-16383 (5461 slots) master M: 654c0a7853724fe33683f54e8df665993ea704ed 192.168.188.28:6379 slots:5461-10922 (5462 slots) master M: b4cc74e0ac6ec9b634d93e86fdb376b4d89dc318 192.168.188.38:6379 slots:0-5460 (5461 slots) master M: ee3d5f85034e5a3c9d9369357b545d0a92751728 192.168.188.18:6380 slots: (0 slots) master replicates 53c202561155d141a8e24d42bb8eb2907998fce9 M: 10b558217d6618093b87f75a453ca39679bbcdf4 192.168.188.28:6380 slots: (0 slots) master replicates b4cc74e0ac6ec9b634d93e86fdb376b4d89dc318 M: 5dc55e90d3d3b56502a319ec9845e901102cfa6c 192.168.188.38:6380 slots: (0 slots) master replicates 654c0a7853724fe33683f54e8df665993ea704ed [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered.
四，測試驗證
調整主從狀態
登錄redis集羣：
設置192.168.188.18:6380爲192.168.188.28:6379的從 redis-cli -c -p 6380 -h 192.168.188.18 192.168.188.18:6380> cluster replicate 654c0a7853724fe33683f54e8df665993ea704ed OK 設置192.168.188.38:6380爲192.168.188.18:6379的從 redis-cli -c -p 6380 -h 192.168.188.38 192.168.188.38:6380> cluster replicate 53c202561155d141a8e24d42bb8eb2907998fce9 OK 最後保存配置到硬盤 192.168.188.38:6380> cluster saveconfig OK
檢查集羣狀態：
redis-trib.rb check 192.168.188.18:6379 >>> Performing Cluster Check (using node 192.168.188.18:6379) M: 53c202561155d141a8e24d42bb8eb2907998fce9 192.168.188.18:6379 slots:10923-16383 (5461 slots) master 1 additional replica(s) S: 5dc55e90d3d3b56502a319ec9845e901102cfa6c 192.168.188.38:6380 slots: (0 slots) slave replicates 53c202561155d141a8e24d42bb8eb2907998fce9 M: 654c0a7853724fe33683f54e8df665993ea704ed 192.168.188.28:6379 slots:5461-10922 (5462 slots) master 1 additional replica(s) S: 10b558217d6618093b87f75a453ca39679bbcdf4 192.168.188.28:6380 slots: (0 slots) slave replicates b4cc74e0ac6ec9b634d93e86fdb376b4d89dc318 S: ee3d5f85034e5a3c9d9369357b545d0a92751728 192.168.188.18:6380 slots: (0 slots) slave replicates 654c0a7853724fe33683f54e8df665993ea704ed M: b4cc74e0ac6ec9b634d93e86fdb376b4d89dc318 192.168.188.38:6379 slots:0-5460 (5461 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered.
使用方法
集羣
CLUSTER INFO 打印集羣的信息
CLUSTER NODES 列出集羣當前已知的所有節點（node），以及這些節點的相關信息。
節點
CLUSTER MEET <ip> <port> 將 ip 和 port 所指定的節點添加到集羣當中，讓它成爲集羣的一份子。
CLUSTER FORGET <node_id> 從集羣中移除 node_id 指定的節點。
CLUSTER REPLICATE <node_id> 將當前節點設置爲 node_id 指定的節點的從節點。
CLUSTER SAVECONFIG 將節點的配置文件保存到硬盤裏面。
槽(slot)
CLUSTER ADDSLOTS <slot> [slot ...] 將一個或多個槽（slot）指派（assign）給當前節點。
CLUSTER DELSLOTS <slot> [slot ...] 移除一個或多個槽對當前節點的指派。
CLUSTER FLUSHSLOTS 移除指派給當前節點的所有槽，讓當前節點變成一個沒有指派任何槽的節點。
CLUSTER SETSLOT <slot> NODE <node_id> 將槽 slot 指派給 node_id 指定的節點，如果槽已經指派給另一個節點，那麼先讓另一個節點刪除該槽>，然後再進行指派。
CLUSTER SETSLOT <slot> MIGRATING <node_id> 將本節點的槽 slot 遷移到 node_id 指定的節點中。
CLUSTER SETSLOT <slot> IMPORTING <node_id> 從 node_id 指定的節點中導入槽 slot 到本節點。
CLUSTER SETSLOT <slot> STABLE 取消對槽 slot 的導入（import）或者遷移（migrate）。
鍵
CLUSTER KEYSLOT <key> 計算鍵 key 應該被放置在哪個槽上。
CLUSTER COUNTKEYSINSLOT <slot> 返回槽 slot 目前包含的鍵值對數量。
CLUSTER GETKEYSINSLOT <slot> <count> 返回 count 個 slot 槽中的鍵。
五，排錯優化
報錯
/usr/lib/ruby/gems/1.8/gems/redis-3.0.7/lib/redis/client.rb:97:in `call': ERR Slot 16011 is already busy (Redis::CommandError) from /usr/lib/ruby/gems/1.8/gems/redis-3.0.7/lib/redis.rb:2432:in `method_missing' from /usr/lib/ruby/gems/1.8/gems/redis-3.0.7/lib/redis.rb:37:in `synchronize' from /usr/lib/ruby/1.8/monitor.rb:242:in `mon_synchronize' from /usr/lib/ruby/gems/1.8/gems/redis-3.0.7/lib/redis.rb:37:in `synchronize' from /usr/lib/ruby/gems/1.8/gems/redis-3.0.7/lib/redis.rb:2431:in `method_missing' from /usr/local/bin/redis-trib.rb:212:in `flush_node_config' from /usr/local/bin/redis-trib.rb:775:in `flush_nodes_config' from /usr/local/bin/redis-trib.rb:774:in `each' from /usr/local/bin/redis-trib.rb:774:in `flush_nodes_config' from /usr/local/bin/redis-trib.rb:1295:in `create_cluster_cmd' from /usr/local/bin/redis-trib.rb:1695:in `send' from /usr/local/bin/redis-trib.rb:1695
處理
rm -rf /data/redis/6379/nodes-6379.conf rm -rf /data/redis/6380/nodes-6380.conf /etc/init.d/redis_6379 restart /etc/init.d/redis_6380 restart gem list gem uninstall redis --version 3.3.2 gem install redis --version 3.0.7 gem list
警告
WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
7997:M 08 Dec 15:04:59.760 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
優化
vm.overcommit_memory = 1
echo never > /sys/kernel/mm/transparent_hugepage/enabled