Redis with HA

1. 概述

Redis(http://redis.io/)一個高性能的key-value數據庫，可用於實時的計算和統計等。
Redis有多種HA方案，比如Master-Slave，Redis Cluster等方案。
本文采用Redis Sentinel來管理1 Master 2 Slave的3臺Centos 6.3 VM實例的方式，每臺VM實例有一個Redis進程和一個Sentinel進程。

對於小型的應用來說，此方案已經足夠。

2. 部署配置

假設

redis master	10.64.70.10
redis slave 1	10.64.70.20
redis slave 2	10.64.70.30

創建以下目錄結構

/data/redis/config	存放redis.conf sentinel.conf
/data/redis/backup/	存放定期持久化的backup db
/date/redis/log/	存放redis.log sentinel.log
/data/redis/logrotate/	存放redis logrotate config：redis.rotate

在3臺VM上安裝redis並拷貝配置文件

$ wget http://download.redis.io/releases/redis-3.0.5.tar.gz
$ tar xzf redis-3.0.5.tar.gz
$ cd redis-3.0.5
$ make
$ make install

拷貝redis-3.0.5.tar.gz包中原始的redis.conf sentinel.conf 到/data/redis/config/ 並修改3臺VM上的配置文件

Redis master redis.conf

# Master redis.conf

daemonize yes
pidfile /var/run/redis.pid
port 6379
bind 10.64.70.10
timeout 0
loglevel notice
logfile /data/redis/log/redis.log
databases 16
#save 900 1
#save 300 10
#save 60 10000
rdbcompression yes
dbfilename redis_dump.db
dir /data/redis/backup
#slaveof
appendonly no
slave-serve-stale-data yes
slave-read-only yes

Redis salve1 redis.conf

# Slave1 redis.conf

daemonize yes
pidfile /var/run/redis.pid
port 6379
bind 10.64.70.20
timeout 0
loglevel notice
logfile /data/redis/log/redis.log
databases 16
save 900 1
save 300 10
save 60 10000
rdbcompression yes
dbfilename redis_dump.db
dir /data/redis/backup
slaveof 10.64.70.10 6379
appendonly yes
slave-serve-stale-data yes
slave-read-only yes

Redis slave 2 redis.conf

# Slave2 redis.conf

daemonize yes
pidfile /var/run/redis.pid
port 6379
bind 10.64.70.30
timeout 0
loglevel notice
logfile /data/redis/log/redis.log
databases 16
save 900 1
save 300 10
save 60 10000
rdbcompression yes
dbfilename redis_dump.db
dir /data/redis/backup
slaveof 10.64.70.10 6379
appendonly yes
slave-serve-stale-data yes
slave-read-only yes

Every sentinel.conf

# all Sentinel sentinel.conf

daemonize yes
loglevel notice
logfile /data/redis/log/sentinel.log
port 26379
sentinel monitor redis1 10.64.70.10 6379 2
sentinel monitor redis2 10.64.70.20 6379 2
sentinel monitor redis3 10.64.70.30 6379 2
sentinel down-after-milliseconds redis1 5000
sentinel down-after-milliseconds redis2 5000
sentinel down-after-milliseconds redis3 5000
sentinel parallel-syncs redis1 1
sentinel parallel-syncs redis2 1
sentinel parallel-syncs redis3 1
sentinel failover-timeout redis1 900000
sentinel failover-timeout redis2 900000
sentinel failover-timeout redis3 900000

Logrotate: /data/redis/logrotate/redis.rotate

/data/redis/log/redis.log
/data/redis/log/sentinel.log
{
daily
rotate 30
copytruncate
notifempty
missingok
dateext
dateformat -%Y%m%d
}

Create cron job “00 00 * * * /usr/sbin/logrotate /data/redis/logrotate/redis.rotate -f” to execute the rotate.

3. 啓動關閉

啓動，在3臺上執行

redis-server /data/redis/config/redis.conf
redis-server /data/redis/config/sentinel.conf --sentinel

關閉redis，分別執行

redis-cli -h 10.64.70.10 shutdown
redis-cli -h 10.64.70.20 shutdown
redis-cli -h 10.64.70.30 shutdown

關閉sentinel，在3臺上執行

redis-cli -p 26379 shutdown

4. 監控

簡單的監控方式是判斷每臺上的redis和sentinel進程是否存在，不存在則發送alert出來。
判斷存在的腳本的例子，可以由zenoss定期執行根據返回結果向設定好的收件人發alert。

#!/bin/sh

LOCAL_IP=$(/sbin/ifconfig eth0 | /bin/awk '/inet addr/{print substr($2,6)}')
REDIS_PROCESS_NUM=$(/usr/bin/pgrep -l -f redis-server | /usr/bin/wc -l)

if [ "$REDIS_PROCESS_NUM" == "2" ]; then
    echo "OK: [$LOCAL_IP]Redis running"
    exit 0
else
    echo "ERROR: [$LOCAL_IP]Redis stopped"
    exit 1        
fi

實時監控方案1: zenoss + Redis Monitor ZenPack
http://wiki.zenoss.org/ZenPack:Redis_Monitor

實時監控方案2: Nagios + plugin check_redis.pl
http://exchange.nagios.org/directory/Plugins/Databases/check_redis-2Epl/details
https://github.com/willixix/WL-NagiosPlugins