【Redis】Codis集羣學習筆記

codis集羣介紹

---

一、CC Codis使用介紹

簡單一句話，在RedisCluster集羣方案沒出現時，業界只有twemproxy支持Sharding方案， 但是Twemproxy對集羣的擴縮容，resharding處理得不那麼優雅，Codis方案應運而生。當然不僅僅是上面的優缺點，還有更多大家自己發掘，本文是站在一個運維人員，也就是administrator的角度，針對codis集羣運維做以簡介，不討論業務層面相關知識。

二、Codis 3.0.x集羣介紹

2.1 參考

https://github.com/Littlegump/codis/tree/release3.0
https://github.com/Littlegump/codis/blob/release3.0/doc/tutorial_zh.md

2.2 codis環境初始化

自己搭建的環境，docker run 直接可用

• 基礎環境介紹：

FROM debian:8.9

WORKDIR /data

ADD . /data

RUN apt-get update && apt-get install -y python \
    supervisor \
    curl \
    vim \
    psmisc \
    wget \
    man-db \
    python-pip \
    git \
    gcc \
    make \
    net-tools \
    netcat \
    default-jdk \
    dirmngr \
    gnupg \
    && apt-get clean && rm -rf /var/lib/apt/lists/*

RUN bash initSoftware.sh ##內容參見附錄

EXPOSE 8080 18080 2181 11080 19000

CMD ["sleep","infinity"]

2.2 集羣架構&&組件

1. Codis-Server(n >=1, group >=1):
   基於2.8.21開發的redis實例，用於組建group（一個replication集羣，master-r1-r2）
2. Codis proxy(n>=1)
   客戶端入口，用於實現redis協議，由codis-dashboard保證狀態保證其狀態同步
3. Codis Dashboard（n = 0 || n = 1）:
   集羣管理page:
   • codis-proxy的增刪
   • codis-server的增刪
   • 數據遷移
   • 維護codis-proxy數據一致性[TODO]

4. Codis Admin(集羣管理的命令行工具):

   • 創建group
     codis_admin --create-group --gid $g
   • 將codis-server加入到group
     codis_admin --group-add --gid $g -x 127.0.0.1:${p}
   • 將codis-proxy加入到codis集羣dashboard管理範圍內
     codis_admin --create-proxy -x 127.0.0.1:${p1}
codis_admin --slot-action --interval=100 # 暫時未學習
     codis_admin --rebalance --confirm # 暫時未學習
5. Codis FE:
   展示頁面
6. Codis HA
   • 3.0版本中，codis-ha依賴codis-dashboard實例，可選組件，它自動獲取各個組件狀態
   • 根據集羣狀態自動生成主從切換策略（原理[TODO]）, 主從切換的執行人是codis-dashboard
7. Codis Storage
   不同集羣會按照配置各個組件的"product_name"配置來進行區分，也就是說，一個codis-dashboard可以管理多個codis集羣，對codis集羣做邏輯劃分

2.3 啓動

1. zookeeper方式
   附錄2 startEtcdCodis.sh
2. etcd方式
   附錄3 startZkCodis.sh

三、zookeeper

0. 參考
   中文教程 https://www.w3cschool.cn/zookeeper/zookeeper_overview.html
   集羣管理: https://zookeeper.apache.org/doc/current/zookeeperAdmin.html
   一個zoo集羣，由一個leader和n個followers組成
1. 安裝&&準備

   # 需要java環境
   ZKPKG=zookeeper-3.4.14
   wget -O $ZKPKG.tar.gz https://downloads.apache.org/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz
   tar zxf $ZKPKG.tar.gz -C /usr/local
   ln -s /usr/local/$ZKPKG /usr/local/zookeeper

2. 集羣搭建: 見DockerFile

   cat zoo1.cfg
   tickTime=2000  # hearbeat Detect
   initLimit=5   // leader和其他zoo成員通信相關的，暫未了解
   syncLimit=2   // 同上
   clientPort=2181  # zoo 客戶端端口
   dataDir=/data/zoo/01
   dataLogDir= /data/zoo/01
   server.1=127.0.0.1:2888:3888 // 下述三行列舉了本zookeeper集羣的所有成員，
   server.2=127.0.0.1:2889:3889 // server.x = [hostname]:[port1]:[port2]
   server.3=127.0.0.1:2890:3890 // x表示myid, port1用於L和F的通信，port2用於L選舉時的投票通信

3. 集羣管理tutorial點擊這裏

4. 常用命令

   zkCli.sh --server 127.0.0.1:2181
   [zk: 127.0.0.1:2182(CONNECTED) 8] help // 查看幫助
   [...] create /zk_test my_data # 創建一個znode，並與"my_data"關聯
   [...] ls /  # 用於查看當前zk集羣管理的znode
   [codis3, zookeeper, zk_test]
   [...] get /zk_test 驗證數據是否有與znode("/zk_test")關聯的字符串
   my_data
   cZxid = 0x100001c21
   ctime = Wed Apr 08 07:11:17 UTC 2020
   mZxid = 0x100001c21
   mtime = Wed Apr 08 07:11:17 UTC 2020
   pZxid = 0x100001c21
   cversion = 0
   dataVersion = 0
   aclVersion = 0
   ephemeralOwner = 0x0
   dataLength = 7
   numChildren = 0

四、etcd

2. 安裝

   ETCD_VERSION=${ETCD_VERSION:-v3.3.1}
   
   curl -L https://github.com/coreos/etcd/releases/download/$ETCD_VERSION/etcd-$ETCD_VERSION-linux-amd64.tar.gz -o etcd-$ETCD_VERSION-linux-amd64.tar.gz
   
   tar xzvf etcd-$ETCD_VERSION-linux-amd64.tar.gz
   rm etcd-$ETCD_VERSION-linux-amd64.tar.gz
   
   cd etcd-$ETCD_VERSION-linux-amd64
   cp etcd /usr/local/bin/
   cp etcdctl /usr/local/bin/
   
   rm -rf etcd-$ETCD_VERSION-linux-amd64
   etcdctl --version

3. 啓動
   nohup etcd --name="codisInstance" &> etcd.log &

五、附錄

• initSoftware.sh

#!/bin/bash
wget https://dl.google.com/go/go1.5.2.linux-amd64.tar.gz && tar -C /usr/local -xzf go1.5.2.linux-amd64.tar.gz
echo "export PATH=$PATH:/usr/local/go/bin:/root/go/bin" >> ~/.bashrc                                         |~
echo """export GOPATH="$HOME/go"""" >> ~/.bashrc
source ~/.bashrc
go get -u github.com/tools/godep
# install redis
wget http://download.redis.io/releases/redis-5.0.8.tar.gz && tar xzf redis-5.0.8.tar.gz && cd redis-5.0.8 && make
ln -sv /data/redis-5.0.8/src/redis* /usr/bin/
# install codis
mkdir -p $GOPATH/src/github.com/CodisLabs
cd $_ && git clone https://github.com/CodisLabs/codis.git -b release3.0
cd $GOPATH/src/github.com/CodisLabs/codis && make
# install etcd
ETCD_VERSION=${ETCD_VERSION:-v3.3.1}
curl -L https://github.com/coreos/etcd/releases/download/$ETCD_VERSION/etcd-$ETCD_VERSION-linux-amd64.tar.gz -o etcd-$ETCD_VERSION-linux-amd64.tar.gz
tar xzvf etcd-$ETCD_VERSION-linux-amd64.tar.gz
rm etcd-$ETCD_VERSION-linux-amd64.tar.gz
cd etcd-$ETCD_VERSION-linux-amd64
cp etcd /usr/local/bin/
cp etcdctl /usr/local/bin/
rm -rf etcd-$ETCD_VERSION-linux-amd64
etcdctl --version
# install zookeeper
ZKPKG=zookeeper-3.4.14
wget -O $ZKPKG.tar.gz https://downloads.apache.org/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz
tar zxf $ZKPKG.tar.gz -C /usr/local
ln -s /usr/local/$ZKPKG /usr/local/zookeeper

2. startZkCodis.sh

   #!/bin/bash
   PATH=$PATH:/root/go/src/github.com/CodisLabs/codis/bin
   # start ZooKeeper
   rm -rf /data/zookeeper/ && mkdir -p /data/zookeeper/{01,02,03}
   echo 1 > /data/zookeeper/01/myid
   echo 2 > /data/zookeeper/02/myid
   echo 3 > /data/zookeeper/03/myid
   cat > /data/zookeeper/zoo1.cfg <<EOF
   tickTime=2000
   dataDir=/data/zookeeper/01
   dataLogDir=/data/zookeeper/01
   clientPort=2181
   initLimit=5
   syncLimit=2
   server.1=0.0.0.0:2888:3888
   server.2=0.0.0.0:2889:3889
   server.3=0.0.0.0:2890:3890
   EOF
   cat > /data/zookeeper/zoo2.cfg <<EOF
   tickTime=2000
   dataDir=/data/zookeeper/02
   dataLogDir=/data/zookeeper/02
   clientPort=2182
   initLimit=5
   syncLimit=2
   server.1=0.0.0.0:2888:3888
   server.2=0.0.0.0:2889:3889
   server.3=0.0.0.0:2890:3890
   EOF
   cat > /data/zookeeper/zoo3.cfg <<EOF
   tickTime=2000
   dataDir=/data/zookeeper/03
   dataLogDir=/data/zookeeper/03
   clientPort=2183
   initLimit=5
   syncLimit=2
   server.1=0.0.0.0:2888:3888
   server.2=0.0.0.0:2889:3889
   server.3=0.0.0.0:2890:3890
   EOF
   pushd /usr/local/zookeeper
   ./bin/zkServer.sh start /data/zookeeper/zoo1.cfg
   ./bin/zkServer.sh start /data/zookeeper/zoo2.cfg
   ./bin/zkServer.sh start /data/zookeeper/zoo3.cfg
   rm -rf /data/tmp; mkdir -p /data/tmp && pushd /data/tmp
   cat > proxy1.toml <<EOF
   product_name="codisInstance"
   product_auth=""
   proto_type="tcp4"
   admin_addr="0.0.0.0:11080"
   proxy_addr="0.0.0.0:19000"
   EOF
   cat > proxy2.toml <<EOF
   product_name="codisInstance"
   product_auth=""
   proto_type="tcp4"
   admin_addr="0.0.0.0:11081"
   proxy_addr="0.0.0.0:19001"
   EOF
   cat > proxy3.toml <<EOF
   product_name="codisInstance"
   product_auth=""
   proto_type="tcp4"
   admin_addr="0.0.0.0:11082"
   proxy_addr="0.0.0.0:19002"
   EOF
   cat > redis1.conf <<EOF
   port 6379
   dbfilename dump1.rdb
   EOF
   cat > redis2.conf <<EOF
   port 6380
   dbfilename dump2.rdb
   EOF
   cat > redis3.conf <<EOF
   port 6381
   dbfilename dump3.rdb
   EOF
   cat > redis4.conf <<EOF
   port 6382
   dbfilename dump4.rdb
   EOF
   cat > redis5.conf <<EOF
   port 6383
   dbfilename dump5.rdb
   EOF
   cat > redis6.conf <<EOF
   port 6384
   dbfilename dump6.rdb
   EOF
   cat > codis.json <<EOF
   [
   {
   "name": "codisInstance",
   "dashboard": "127.0.0.1:18080"
   }
   ]
   EOF
   cat >dashboard.toml <<EOF
   coordinator_name = "zookeeper"
   coordinator_addr = "127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183"
   product_name = "codisInstance"
   product_auth = ""
   admin_addr = "0.0.0.0:18080"
   EOF
   nohup etcd --name="codisInstance" &> etcd.log &
   if [ $? -ne 0 ];then
   echo "codis-etcd start failed"
   exit 1
   fi
   nohup codis-server redis1.conf &> redis-6379.log &
   nohup codis-server redis2.conf &> redis-6380.log &
   nohup codis-server redis3.conf &> redis-6381.log &
   nohup codis-server redis4.conf &> redis-6382.log &
   nohup codis-server redis5.conf &> redis-6383.log &
   nohup codis-server redis6.conf &> redis-6384.log &
   if [ $? -ne 0 ];then
   echo "codis-server start failed"
   exit 1
   fi
   nohup codis-proxy -c proxy1.toml &>proxy1.log &
   nohup codis-proxy -c proxy2.toml &>proxy2.log &
   nohup codis-proxy -c proxy3.toml &>proxy3.log &
   if [ $? -ne 0 ];then
   echo "codis-proxy start failed"
   exit 1
   fi
   nohup codis-dashboard -c dashboard.toml &> dashboard.log &
   if [ $? -ne 0 ];then
   echo "codis-Dashboard start failed"
   exit 1
   fi
   nohup /root/go/src/github.com/CodisLabs/codis/bin/codis-fe -d codis.json --listen 0.0.0.0:8080 &> fe.log &
   if [ $? -ne 0 ];then
   echo "codis-FE start failed"
   exit 1
   fi
   nohup codis-ha --dashboard=127.0.0.1:18080 &> codis_ha.log &
   if [ $? -ne 0 ];then
   echo "codis-HA start failed"
   exit 1
   fi
   codis-admin --dashboard=127.0.0.1:18080 --create-group --gid 1
   codis-admin --dashboard=127.0.0.1:18080 --create-group --gid 2
   codis-admin --dashboard=127.0.0.1:18080 --create-group --gid 3
   codis-admin --dashboard=127.0.0.1:18080 --group-add --gid 1 -x 127.0.0.1:6379
   codis-admin --dashboard=127.0.0.1:18080 --group-add --gid 1 -x 127.0.0.1:6380
   codis-admin --dashboard=127.0.0.1:18080 --group-add --gid 2 -x 127.0.0.1:6381
   codis-admin --dashboard=127.0.0.1:18080 --group-add --gid 2 -x 127.0.0.1:6382
   codis-admin --dashboard=127.0.0.1:18080 --group-add --gid 3 -x 127.0.0.1:6383
   codis-admin --dashboard=127.0.0.1:18080 --group-add --gid 3 -x 127.0.0.1:6384
   codis-admin --dashboard=127.0.0.1:18080 --create-proxy -x 127.0.0.1:11080
   codis-admin --dashboard=127.0.0.1:18080 --create-proxy -x 127.0.0.1:11081
   codis-admin --dashboard=127.0.0.1:18080 --create-proxy -x 127.0.0.1:11082
   codis-admin --dashboard=127.0.0.1:18080 --slot-action --interval=100
   codis-admin --dashboard=127.0.0.1:18080 --rebalance --confirm
   while true; do
   date
   sleep 60
   done

3. startEtcdCodis.sh

   #!/bin/bash
   PATH=$PATH:/root/go/src/github.com/CodisLabs/codis/bin
   rm -rf tmp; mkdir -p tmp && pushd tmp
   cat > proxy1.toml <<EOF
   product_name="codisInstance"
   product_auth=""
   proto_type="tcp4"
   admin_addr="0.0.0.0:11080"
   proxy_addr="0.0.0.0:19000"
   EOF
   cat > proxy2.toml <<EOF
   product_name="codisInstance"
   product_auth=""
   proto_type="tcp4"
   admin_addr="0.0.0.0:11081"
   proxy_addr="0.0.0.0:19001"
   EOF
   cat > proxy3.toml <<EOF
   product_name="codisInstance"
   product_auth=""
   proto_type="tcp4"
   admin_addr="0.0.0.0:11082"
   proxy_addr="0.0.0.0:19002"
   EOF
   cat > redis1.conf <<EOF
   port 6379
   dbfilename dump1.rdb
   EOF
   cat > redis2.conf <<EOF
   port 6380
   dbfilename dump2.rdb
   EOF
   cat > redis3.conf <<EOF
   port 6381
   dbfilename dump3.rdb
   EOF
   cat > redis4.conf <<EOF
   port 6382
   dbfilename dump4.rdb
   EOF
   cat > redis5.conf <<EOF
   port 6383
   dbfilename dump5.rdb
   EOF
   cat > redis6.conf <<EOF
   port 6384
   dbfilename dump6.rdb
   EOF
   cat > codis.json <<EOF
   [
       {
           "name": "codisInstance",
           "dashboard": "127.0.0.1:18080"
       }
   ]
   EOF
   cat >dashboard.toml <<EOF
   coordinator_name = "etcd"
   coordinator_addr = "127.0.0.1:2379"
   product_name = "codisInstance"
   product_auth = ""
   admin_addr = "0.0.0.0:18080"
   EOF
   nohup etcd --name="codisInstance" &> etcd.log &
   nohup codis-server redis1.conf &> redis-6379.log &
   nohup codis-server redis2.conf &> redis-6380.log &
   nohup codis-server redis3.conf &> redis-6381.log &
   nohup codis-server redis4.conf &> redis-6382.log &
   nohup codis-server redis5.conf &> redis-6383.log &
   nohup codis-server redis6.conf &> redis-6384.log &
   nohup codis-proxy -c proxy1.toml &>proxy1.log &
   nohup codis-proxy -c proxy2.toml &>proxy2.log &
   nohup codis-proxy -c proxy3.toml &>proxy3.log &
   nohup codis-dashboard -c dashboard.toml &> dashboard.log &
   nohup /root/go/src/github.com/CodisLabs/codis/bin/codis-fe -d codis.json --listen 0.0.0.0:8080 &> fe.log &
   nohup codis-ha --dashboard=127.0.0.1:18080 &> codis_ha.log &
   codis-admin --dashboard=127.0.0.1:18080 --create-group --gid 1
   codis-admin --dashboard=127.0.0.1:18080 --create-group --gid 2
   codis-admin --dashboard=127.0.0.1:18080 --create-group --gid 3
   codis-admin --dashboard=127.0.0.1:18080 --group-add --gid 1 -x 127.0.0.1:6379
   codis-admin --dashboard=127.0.0.1:18080 --group-add --gid 1 -x 127.0.0.1:6380
   codis-admin --dashboard=127.0.0.1:18080 --group-add --gid 2 -x 127.0.0.1:6381
   codis-admin --dashboard=127.0.0.1:18080 --group-add --gid 2 -x 127.0.0.1:6382
   codis-admin --dashboard=127.0.0.1:18080 --group-add --gid 3 -x 127.0.0.1:6383
   codis-admin --dashboard=127.0.0.1:18080 --group-add --gid 3 -x 127.0.0.1:6384
   codis-admin --dashboard=127.0.0.1:18080 --create-proxy -x 127.0.0.1:11080
   codis-admin --dashboard=127.0.0.1:18080 --create-proxy -x 127.0.0.1:11081
   codis-admin --dashboard=127.0.0.1:18080 --create-proxy -x 127.0.0.1:11082
   codis-admin --dashboard=127.0.0.1:18080 --slot-action --interval=100
   codis-admin --dashboard=127.0.0.1:18080 --rebalance --confirm
   sleep 3
   while true; do
           date
           sleep 60
   done