Codis安裝部署2臺服務器沒有HA版本
項目地址：
https://github.com/wandoulabs/codis
中文說明文檔：
https://github.com/wandoulabs/codis/blob/master/doc/tutorial_zh.md
參考地址：
http://0xffff.me/blog/2014/11/11/codis-de-she-ji-yu-shi-xian-part-2/
http://www.cnblogs.com/xuanzhi201111/p/4425194.html
http://blog.csdn.net/freewebsys/article/details/44100919
個人：廣州-李惟忠
架構圖：
Codis 並不太適合 key 少，但是 value 特別大的應用， 而且你的 key 越少， value 越大，最後就會退化成單個 redis 的模型 （性能還不如 raw redis），所以 Codis 更適合海量 Key,
value比較小 （<= 1 MB) 的應用。
機器與應用列表：
操作系統：CentOS6.5
IP: 172.17.3.133 hostname: codis-133 apps: zookeeper, codis_proxy_1, codis_config, codis_server_master,slave
IP: 172.17.3.134 hostname: codis-134 apps: codis_server_master,slave
備註：由於是仿真測試，一臺機器跑多個應用，如生產環境，只需把應用分開部署即可，另外zookeeper需要3臺搭建集羣環境，還需要2臺搭建keepalived+haproxy。
一，安裝zookeeper
1，配置hosts文件 (所有機器上配置)
vim /etc/hosts
172.17.3.133 codis-133
172.17.3.134 codis-134
2，安裝java環境
ZooKeeper 要求 JAVA 的環境才能運行，並且需要 JAVA6 以上的版本，可以從 SUN 官網上下載，並對 JAVA 環境變量進行設置。
yum -y install java-1.7.0-openjdk-devel
java -version
java version "1.7.0_75"
OpenJDK Runtime Environment (rhel-2.5.4.0.el6_6-x86_64 u75-b13)
OpenJDK 64-Bit Server VM (build 24.75-b04, mixed mode)
3，安裝zookeeper
wget http://mirrors.cnnic.cn/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
tar zxvf zookeeper-3.4.6.tar.gz
mv zookeeper-3.4.6 /usr/local/zookeeper
mkdir -p /data/zookeeper/{data,logs}
配置zoo.cfg
vim /usr/local/zookeeper/conf/zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper/data
dataLogDir=/data/zookeeper/logs
clientPort=2181
4，啓動zookeeper
/usr/local/zookeeper/bin/zkServer.sh start
JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@localhost conf]# /usr/local/zookeeper/bin/zkServer.sh status
JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Mode: standalone
設置開機啓動
vim /etc/rc.local
/usr/local/zookeeper/bin/zkServer.sh start
二，安裝codis集羣
1，安裝go
設置環境變量
vim /etc/profile
export GOROOT=/usr/local/go
export PATH=$PATH:$GOROOT/bin
export GOPATH=/usr/local/codis
source /etc/profile
下載安裝go
cd /usr/local/
wget http://golangtc.com/static/go/go1.4.2.linux-amd64.tar.gz
tar -zxvf go1.4.2.linux-amd64.tar.gz
go version
go version go1.4.2 linux/amd64
2，安裝依賴環境
yum groupinstall "Development Tools"
3，安裝codis
yum install -y git
go get github.com/wandoulabs/codis #這個需要幾分鐘下載共30M文件
package github.com/wandoulabs/codis
imports github.com/wandoulabs/codis
imports github.com/wandoulabs/codis: no buildable Go source files in /usr/local/codis/src/github.com/wandoulabs/codis
cd $GOPATH/src/github.com/wandoulabs/codis
[root@localhost codis]# pwd
/usr/local/codis/src/github.com/wandoulabs/codis
#執行編譯測試腳本，編譯go和reids。
./bootstrap.sh #這個需要十幾分鍾共下載50M文件
make gotest
# 將編譯好後，把bin目錄和一些腳本複製過去/usr/local/codis目錄下：
mkdir -p /usr/local/codis/{logs,conf,scripts}
mkdir -p /data/codis_server/{logs,conf,data}
cp -rf bin /usr/local/codis/
cp sample/config.ini /usr/local/codis/conf/
cp sample/redis_conf/6381.conf /data/codis_server/conf/
cp -rf /usr/local/codis/src/github.com/wandoulabs/codis/sample/*.sh /usr/local/codis/scripts/
cp -rf /usr/local/codis/src/github.com/wandoulabs/codis/sample/usage.md /usr/local/codis/scripts/
cp /usr/local/codis/src/github.com/wandoulabs/codis/extern/redis-2.8.13/src/redis-cli /usr/local/codis/bin/redis-cli-2.8.13
cp /usr/local/codis/src/github.com/wandoulabs/codis/extern/redis-2.8.21/src/redis-cli /usr/local/codis/bin/redis-cli-2.8.21
ln -s /usr/local/codis/bin/redis-cli-2.8.21 /usr/local/codis/bin/redis-cli
4. 配置codis_proxy_1 ( codis-133 機器上配置)
cd /usr/local/codis
vim config.ini
zk=localhost:2181
product=bitauto-codis
proxy_id=proxy_1
net_timeout=5
dashboard_addr=localhost:18087
coordinator=zookeeper
5. 修改配置文件，啓動codis-server服務.
cd /data/codis_server/conf/
mv 6381.conf 6379.conf.bak
grep -Ev "^#|^$" 6379.conf.bak >6379.conf
vim 6379.conf
修改如下參數: (生產環境，參數適當進行調整)
daemonize yes
timeout 300
pidfile /var/run/redis_6379.pid
port 6379
logfile "/data/codis_server/logs/codis_6379.log"
save 900 1
save 300 10
save 60 10000
dbfilename 6379.rdb
dir /data/codis_server/data
appendfilename "6379_appendonly.aof"
appendfsync everysec
具體配置文件如下：
daemonize yes
pidfile /var/run/redis_6379.pid
port 6379
tcp-backlog 511
timeout 300
tcp-keepalive 0
loglevel notice
logfile "/data/codis_server/logs/redis_6379.log"
databases 16
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename 6379.rdb
dir /data/codis_server/data
slave-serve-stale-data yes
slave-read-only yes
repl-disable-tcp-nodelay no
slave-priority 100
maxclients 10000
maxmemory 2gb
maxmemory-policy allkeys-lru
appendonly no
appendfilename "6379_appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes
複製6380配置文件
cp 6379.conf 6380.conf
sed -i 's/6379/6380/g' 6380.conf
添加內核參數
echo "vm.overcommit_memory = 1" >> /etc/sysctl.conf
sysctl -p
內核參數說明如下：
overcommit_memory文件指定了內核針對內存分配的策略，其值可以是0、1、2。
0， 表示內核將檢查是否有足夠的可用內存供應用進程使用；如果有足夠的可用內存，內存申請允許；否則，內存申請失敗，並把錯誤返回給應用進程。
1， 表示內核允許分配所有的物理內存，而不管當前的內存狀態如何。
2， 表示內核允許分配超過所有物理內存和交換空間總和的內存
啓動codis-server服務
nohup /usr/local/codis/bin/codis-server /data/codis_server/conf/6379.conf &
nohup /usr/local/codis/bin/codis-server /data/codis_server/conf/6380.conf &
[root@localhost conf]#nohup /usr/local/codis/bin/codis-server /data/codis_server/conf/6379.conf &
[1] 15944
[root@localhost conf]# _._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 2.8.21 (575eed4b/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in stand alone mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
| `-._ `._ / _.-' | PID: 15944
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | http://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
[15944] 24 Jun 18:23:22.255 # Server started, Redis version 2.8.21
[15944] 24 Jun 18:23:22.256 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with
Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the
setting after a reboot. Redis must be restarted after THP is disabled.
[15944] 24 Jun 18:23:22.256 * The server is now ready to accept connections on port 6379
6. 查看一下啓動流程：
cat usage.md
0. start zookeeper
1. change config items in config.ini
2. ./start_dashboard.sh
3. ./start_redis.sh
4. ./add_group.sh
5. ./initslot.sh
之後可以通過瀏覽器給group分配solt，如果solt分配錯誤，可以通過瀏覽器的Migrate Slot重新分配，這個solt遷移需要一點時間，也可以Auto Rebalance平均分配，分片等操作一定要在開啓proxy之前。
6. ./start_proxy.sh
7. ./set_proxy_online.sh
8. open browser to http://localhost:18087/admin
這只是一個參考，有些順序不是必須的，但啓動dashboard前，必須啓動zookeeper服務，這是必須的，後面有很多操作，都可以在web頁面完成，例如添加/刪除組，添加/刪除redis實例等
7. 修改腳本，啓動 dashboard。( 只需在一臺機器上啓動即可。172.17.3.133上啓動 ，後續大部分操作都可以在面板上操作)
cat /usr/local/codis/scripts/start_dashboard.sh
#!/bin/sh
CODIS_HOME=/usr/local/codis
nohup $CODIS_HOME/bin/codis-config -c $CODIS_HOME/conf/config.ini -L $CODIS_HOME/logs/dashboard.log dashboard --addr=:18087 --http-log=$CODIS_HOME/logs/requests.log
&>/dev/null &
啓動dashboard
cd /usr/local/codis/scripts/
sh start_dashboard.sh
8. 啓動codis-server master , slave 實例 ，以上步驟已經啓動，不在描述(在第5步，此時還沒有分主從，只是分別在3.133和3.134上啓動2個實例6379和6380)。
9. 通過管理頁面添加組ID，爲組添加主從實例，一個組裏只能有一個redis-master：
http://172.17.3.133:18087（最好用Firefox瀏覽器或者谷歌瀏覽器）
登錄http://172.17.3.133:18087/admin/,添加2個組，組裏面有2個實例，一個主一個從，默認每個組裏面第一個實例是主
group_1
172.17.3.133:6379 master
172.17.3.134:6380 slave
group_2
172.17.3.134:6379 master
172.17.3.133:6380 slave
10. 修改腳本，初始化槽 ( 172.17.3.133 機器上配置 ，注意這個需要在添加group後操作，這裏的腳本是solt槽1024分成了2半，group1和group2個512個)
cat /usr/local/codis/scripts/initslot.sh
#!/bin/sh
CODIS_HOME=/usr/local/codis
echo "slots initializing..."
$CODIS_HOME/bin/codis-config -c $CODIS_HOME/conf/config.ini slot init -f
echo "done"
執行初始化槽腳本：
sh initslot.sh
slots initializing...
{
"msg": "OK",
"ret": 0
}
done
11.測試一下redis-master和redis-slave是否正常同步數據了：
這個需要用redis-cli測試：
[root@localhost conf]# /usr/local/codis/src/github.com/wandoulabs/codis/extern/redis-2.8.13/src/redis-cli -h 172.17.3.133 -p 6379
172.17.3.133:6379> ping
PONG
172.17.3.133:6379> set name foo
OK
172.17.3.133:6379> get name
"foo"
172.17.3.133:6379> quit
[root@localhost conf]# /usr/local/codis/src/github.com/wandoulabs/codis/extern/redis-2.8.13/src/redis-cli -h 172.17.3.134 -p 6380
172.17.3.134:6380> get name
"foo"
172.17.3.134:6380> set test 123
(error) READONLY You can't write against a read only slave.
172.17.3.134:6380> quit
[root@localhost conf]#
也可以通過日誌查看
root@localhost conf]# [15825] 24 Jun 18:59:42.717 * SLAVE OF 172.17.3.133:6379 enabled (user request)
[15825] 24 Jun 18:59:43.199 * Connecting to MASTER 172.17.3.133:6379
[15825] 24 Jun 18:59:43.199 * MASTER <-> SLAVE sync started
[15825] 24 Jun 18:59:43.200 * Non blocking connect for SYNC fired the event.
[15825] 24 Jun 18:59:43.202 * Master replied to PING, replication can continue...
[15825] 24 Jun 18:59:43.203 * Partial resynchronization not possible (no cached master)
[15825] 24 Jun 18:59:43.204 * Full resync from master: 65c0b5c071fab1b52892a5efe7fd422bbf84f391:1
[15825] 24 Jun 18:59:43.289 * MASTER <-> SLAVE sync: receiving 18 bytes from master
[15825] 24 Jun 18:59:43.289 * MASTER <-> SLAVE sync: Flushing old data
[15825] 24 Jun 18:59:43.290 * MASTER <-> SLAVE sync: Loading DB in memory
[15825] 24 Jun 18:59:43.290 * MASTER <-> SLAVE sync: Finished with success
12. 配置codis-ha服務，主從自動切換。( 隨便找個節點機器上配置即可,此環境中在172.17.3.134機器上配置 )
cd /usr/local
go get github.com/ngaut/codis-ha #ngaut大概3M左右
cd /usr/local/codis/src/github.com/ngaut
cp -r codis-ha /usr/local/
cd /usr/local/codis-ha
go build
創建啓動腳本，啓動codis-ha服務
vim start_codis_ha.sh
codis-ha --codis-config=172.17.3.133:18087 -log-level="info" --productName=bitauto-codis &> ./logs/codis-ha.log &
mkdir logs
sh start_codis_ha.sh
13. 修改start_proxy.sh，啓動codis-proxy服務 ( 在172.17.3.133上配置)
[root@codis-133 scripts]# cat start_proxy.sh
#!/bin/sh
CODIS_HOME=/usr/local/codis
echo "shut down codis_proxy_1..."
$CODIS_HOME/bin/codis-config -c $CODIS_HOME/conf/config.ini proxy offline codis_proxy_1
echo "done"
echo "start new codis_proxy_1..."
nohup $CODIS_HOME/bin/codis-proxy --log-level info -c $CODIS_HOME/conf/config.ini -L $CODIS_HOME/logs/codis_proxy_1.log --cpu=4 --addr=0.0.0.0:19000 --http-
addr=0.0.0.0:11000 &
echo "done"
echo "sleep 3s"
sleep 3
tail -n 30 $CODIS_HOME/logs/codis_proxy_1.log
cat set_proxy_online.sh
#!/bin/sh
CODIS_HOME=/usr/local/codis
echo "set codis_proxy_1 online"
$CODIS_HOME/bin/codis-config -c $CODIS_HOME/conf/config.ini proxy online codis_proxy_1
echo "done"
啓動codis-proxy
./start_proxy.sh
上線codis_proxy_1
./set_proxy_online.sh
14，最後設置開機啓動
172.17.3.133上配置如下
vim /etc/rc.local
/usr/local/zookeeper/bin/zkServer.sh start
/usr/local/codis/bin/codis-server /data/codis_server/conf/6379.conf
/usr/local/codis/bin/codis-server /data/codis_server/conf/6380.conf
/usr/local/codis/scripts/start_dashboard.sh
/usr/local/codis/scripts/initslot.sh(一次性的，開機不需要啓動)
/usr/local/codis/scripts/start_proxy.sh
/usr/local/codis/scripts/set_proxy_online.sh
172.17.3.134上配置如下
vim /etc/rc.local
/usr/local/zookeeper/bin/zkServer.sh start
/usr/local/codis/bin/codis-server /data/codis_server/conf/6379.conf
/usr/local/codis/bin/codis-server /data/codis_server/conf/6380.conf
/usr/local/codis-ha/start_codis_ha.sh
15，附加文件redis服務啓動腳本
#!/bin/sh
#Configurations injected by install_server below....
EXEC=/usr/local/codis/bin/codis-server
CLIEXEC=/usr/local/codis/bin/redis-cli
PIDFILE=/var/run/redis_6379.pid
CONF="/data/codis_server/conf/6379.conf"
REDISPORT="6379"
###############
# SysV Init Information
# chkconfig: - 58 74
# description: redis_6379 is the redis daemon.
### BEGIN INIT INFO
# Provides: redis_6379
# Required-Start: $network $local_fs $remote_fs
# Required-Stop: $network $local_fs $remote_fs
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Should-Start: $syslog $named
# Should-Stop: $syslog $named
# Short-Description: start and stop redis_6379
# Description: Redis daemon
### END INIT INFO
case "$1" in
start)
if [ -f $PIDFILE ]
then
echo "$PIDFILE exists, process is already running or crashed"
else
echo "Starting Redis server..."
$EXEC $CONF
fi
;;
stop)
if [ ! -f $PIDFILE ]
then
echo "$PIDFILE does not exist, process is not running"
else
PID=$(cat $PIDFILE)
echo "Stopping ..."
$CLIEXEC -p $REDISPORT shutdown
while [ -x /proc/${PID} ]
do
echo "Waiting for Redis to shutdown ..."
sleep 1
done
echo "Redis stopped"
fi
;;
status)
PID=$(cat $PIDFILE)
if [ ! -x /proc/${PID} ]
then
echo 'Redis is not running'
else
echo "Redis is running ($PID)"
fi
;;
restart)
$0 stop
$0 start
;;
*)
echo "Please use start, stop, restart or status as first argument"
;;
esac