MongoDB索引及複製集、shard雜記

一、索引

1、索引操作

1.1 插入數據

> use testdb
switched to db testdb
> for (i=1;i<=10000;i++) db.students.insert({name:"student"+i,age:(i%120),address:"#85 Wenhua Road,Zhengzhou,China"})

> db.students.find().count()
10000

1.2 創建索引

在name字段構建升序索引：
> db.students.ensureIndex({name: 1})
{
    "createdCollectionAutomatically" : false,
    "numIndexesBefore" : 1,
    "numIndexesAfter" : 2,
    "ok" : 1
}

查看索引：
> db.students.getIndexes()
[
    {
        "v" : 1,
        "key" : {
            "_id" : 1
        },
        "name" : "_id_",
        "ns" : "testdb.students"
    },
    {
        "v" : 1,
        "key" : {
            "name" : 1
        },
        "name" : "name_1",
        "ns" : "testdb.students"
    }
]

1.3 刪除索引

> db.students.dropIndex("name_1")
{ "nIndexesWas" : 2, "ok" : 1 }
> db.students.getIndexes()
[
    {
        "v" : 1,
        "key" : {
            "_id" : 1
        },
        "name" : "_id_",
        "ns" : "testdb.students"
    }
]
> 

1.4 創建唯一鍵索引

> db.students.ensureIndex({name: 1},{unique: true})
> db.students.getIndexes()
{
    "v" : 1,
    "unique" : true,
    "key" : {
        "name" : 1
    },
    "name" : "name_1",
    "ns" : "testdb.students"

插入同樣的值會有約束：
> db.students.insert({name: "student20",age: 20})
WriteResult({
    "nInserted" : 0,
    "writeError" : {
        "code" : 11000,
        "errmsg" : "E11000 duplicate key error index: testdb.students.$name_1 dup key: { : \"student20\" }"
    }
})

1.5 查看查找語句詳細執行過程

> db.students.find({name: "student5000"}).explain("executionStats")
{
    "queryPlanner" : {
        "plannerVersion" : 1,
        "namespace" : "testdb.students",
        "indexFilterSet" : false,
        "parsedQuery" : {
            "name" : {
                "$eq" : "student5000"
            }
        },
        "winningPlan" : {
            "stage" : "FETCH",
            "inputStage" : {
                "stage" : "IXSCAN",
                "keyPattern" : {
                    "name" : 1
                },
                "indexName" : "name_1",
                "isMultiKey" : false,
                "direction" : "forward",
                "indexBounds" : {
                    "name" : [
                        "[\"student5000\", \"student5000\"]"
                    ]
                }
            }

        },
        "rejectedPlans" : [ ]
    },
    "serverInfo" : {
        "host" : "master1.com",
        "port" : 27017,
        "version" : "3.0.0",
        "gitVersion" : "a841fd6394365954886924a35076691b4d149168"
    },
    "ok" : 1
}

查看大於5000的記錄執行過程
db.students.find({name: {$gt: "student5000"}}).explain("executionStats")

在15萬條記錄中查找name大於8萬的記錄，不做索引和做所用對比。
> for (i=1;i<=150000;i++) db.test.insert({name:"student"+i,age:(i%120),address:"#85 Wenhua Road,Zhengzhou,China"})
查找：
db.test.find({name: {$gt: "student80000"}}).explain("executionStats")

對比截圖，左邊是全表查找，右邊是添加索引後查找

二、MongoDB複製集

1、mongod複製集配置

1.1 雜項

主節點將數據修改操作保存至oplog中，從節點通過oplog複製到本地並應用。oplog一般存儲在local數據庫

> show dbs
local   0.078GB
testdb  0.078GB
> use local
switched to db local
> show collections
startup_log
system.indexes

只有啓動副本集，纔會產生相關的文件

1.2 準備三個節點

master1（主節點）,2,3

1.3 安裝MongoDB

master2,3安裝MongoDB
[root@master2 mongodb-3.0.0]# ls
mongodb-org-server-3.0.0-1.el7.x86_64.rpm
mongodb-org-shell-3.0.0-1.el7.x86_64.rpm
mongodb-org-tools-3.0.0-1.el7.x86_64.rpm
[root@master2 mongodb-3.0.0]# yum install *.rpm

master2配置：
[root@master2 ~]# mkdir -pv /mongodb/data
[root@master2 ~]# chown -R mongod.mongod /mongodb/

從master1拷貝配置到master2,並修改
[root@master1 ~]# scp /etc/mongod.conf root@master2:/etc/
[root@master1 ~]# scp /etc/mongod.conf root@master3:/etc/

啓動服務：
[root@master2 ~]# systemctl start mongod.service

master3同上配置，並啓動mongod服務。

1.4 主節點配置

先停止剛纔master1的mongod服務：
[root@master1 ~]# systemctl stop mongod.service

啓動主節點複製集功能
[root@master1 ~]# vim /etc/mongod.conf 

replSet=testSet     #複製集名稱
replIndexPrefetch=_id_only  

重新啓動服務：
[root@master1 ~]# systemctl start mongod.service

查看：
[root@master1 ~]# mongo

1.5 主節點（master1），複製集初始化

獲取複製命令相關幫助：
> rs.help()

主節點複製集初始化
> rs.initiate()

主節點rs狀態：
> rs.initiate()
{
    "info2" : "no configuration explicitly specified -- making one",
    "me" : "master1.com:27017",
    "ok" : 1
}
testSet:OTHER> 
testSet:PRIMARY> rs.status()
{
    "set" : "testSet",      #複製集名稱
    "date" : ISODate("2017-01-16T14:36:29.948Z"),
    "myState" : 1,
    "members" : [
        {
            "_id" : 0,      #節點標識
            "name" : "master1.com:27017",   #節點名稱
            "health" : 1,   #節點健康狀態
            "state" : 1,    #有沒有狀態信息
            "stateStr" : "PRIMARY", #節點角色
            "uptime" : 790, #運行時長
            "optime" : Timestamp(1484577363, 1),    #最後一次oplog時間戳
            "optimeDate" : ISODate("2017-01-16T14:36:03Z"), #最後一次oplog時間
            "electionTime" : Timestamp(1484577363, 2),  #選舉時間戳
            "electionDate" : ISODate("2017-01-16T14:36:03Z"),   #選舉時間
            "configVersion" : 1,
            "self" : true   #是不是當前節點
        }
    ],
    "ok" : 1
}

主節點rs配置：
testSet:PRIMARY> rs.conf()
{
    "_id" : "testSet",
    "version" : 1,
    "members" : [
        {
            "_id" : 0,
            "host" : "master1.com:27017",
            "arbiterOnly" : false,
            "buildIndexes" : true,
            "hidden" : false,
            "priority" : 1,
            "tags" : {

            },
            "slaveDelay" : 0,
            "votes" : 1
        }
    ],
    "settings" : {
        "chainingAllowed" : true,
        "heartbeatTimeoutSecs" : 10,
        "getLastErrorModes" : {

        },
        "getLastErrorDefaults" : {
            "w" : 1,
            "wtimeout" : 0
        }
    }
}

1.6 主節點添加從節點

testSet:PRIMARY> rs.add("10.201.106.132")
{ "ok" : 1 }

查看
testSet:PRIMARY> rs.status()
        {
            "_id" : 1,
            "name" : "10.201.106.132:27017",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 35,
            "optime" : Timestamp(1524666332, 1),
            "optimeDate" : ISODate("2018-04-25T14:25:32Z"),
            "lastHeartbeat" : ISODate("2018-04-25T14:26:07.009Z"),
            "lastHeartbeatRecv" : ISODate("2018-04-25T14:26:07.051Z"),
            "pingMs" : 0,
            "configVersion" : 2
        }
    ],

master2（從節點）查看：
[root@master2 ~]# mongo

遇到報錯：
Error: listDatabases failed:{ "note" : "from execCommand", "ok" : 0, "errmsg" : "not master" }

解決辦法
執行 rs.slaveOk()方法

查看：
testSet:SECONDARY> show dbs
local   2.077GB
testdb  0.078GB
testSet:SECONDARY> use testdb
switched to db testdb
testSet:SECONDARY> db.students.findOne()
{
    "_id" : ObjectId("587c9032fe3baa930c0f51d9"),
    "name" : "student1",
    "age" : 1,
    "address" : "#85 Wenhua Road,Zhengzhou,China"
}

查看誰是主節點：
testSet:SECONDARY> rs.isMaster()
{
    "setName" : "testSet",
    "setVersion" : 2,
    "ismaster" : false,
    "secondary" : true,
    "hosts" : [
        "master1.com:27017",
        "10.201.106.132:27017"
    ],
    "primary" : "master1.com:27017",    ###
    "me" : "10.201.106.132:27017",      ###
    "maxBsonObjectSize" : 16777216,
    "maxMessageSizeBytes" : 48000000,
    "maxWriteBatchSize" : 1000,
    "localTime" : ISODate("2018-04-25T14:43:35.956Z"),
    "maxWireVersion" : 3,
    "minWireVersion" : 0,
    "ok" : 1
}

主節點添加第三個節點（master3）
[root@master1 ~]# mongo
testSet:PRIMARY> rs.add("10.201.106.133")
{ "ok" : 1 }

從節點（master3）配置成可用節點：
[root@master3 ~]# mongo
testSet:SECONDARY> rs.slaveOk()

testSet:SECONDARY> use testdb
switched to db testdb
testSet:SECONDARY> db.students.findOne()
{
    "_id" : ObjectId("587c9032fe3baa930c0f51d9"),
    "name" : "student1",
    "age" : 1,
    "address" : "#85 Wenhua Road,Zhengzhou,China"
}

一旦添加從節點，從節點會自動克隆主節點的所有數據庫後，並開始複製主節點的oplog，並應用於本地併爲collection構建索引。

1.7 查看rs配置

testSet:SECONDARY> rs.conf()

1.8 主節點寫數據，測試同步

testSet:PRIMARY> db.classes.insert({class: "One",nostu: 40})

從節點查看：
testSet:SECONDARY> db.classes.findOne()
{
    "_id" : ObjectId("5ae09653f7aa5c90df36dc59"),
    "class" : "One",
    "nostu" : 40
}

從節點禁止插入數據：
testSet:SECONDARY> db.classes.insert({class: "Tow",nostu: 50})
WriteResult({ "writeError" : { "code" : undefined, "errmsg" : "not master" } })

1.9 主節點down機，測試切換

主節點手動down
testSet:PRIMARY> rs.stepDown()

重新查看狀態，master3已經成爲主節點：
testSet:SECONDARY> rs.status()

master3查看，狀態已經改變：
testSet:SECONDARY> 
testSet:PRIMARY> 

2、其他

2.1 查看oplog大小和同步時間

testSet:PRIMARY> db.printReplicationInfo()
configured oplog size:   1165.03515625MB
log length start to end: 390secs (0.11hrs)
oplog first event time:  Wed Apr 25 2018 22:46:37 GMT+0800 (CST)
oplog last event time:   Wed Apr 25 2018 22:53:07 GMT+0800 (CST)
now:                     Wed Apr 25 2018 23:32:35 GMT+0800 (CST)

2.2 修改master2的優先級，優先成爲主節點

rs.conf()對應的collection是local.system.replset
local.system.replset.members[n].priority

需要在主節點操作***
先將配置導入cfg變量
testSet:SECONDARY> cfg=rs.conf()
然後修改值（ID號默認從0開始）：
testSet:SECONDARY> cfg.members[1].priority=2
2
重新加載配置
testSet:SECONDARY> rs.reconfig(cfg)
{ "ok" : 1 }
重載後master2自動變成主節點，master3變回從節點

2.3 修改master3爲純粹的仲裁節點

需要在主節點配置***

需要先將master3的從節點角色移除：
testSet:PRIMARY> rs.remove("10.201.106.133:27017")
{ "ok" : 1 }

修改master3爲仲裁節點：
testSet:PRIMARY> rs.addArb("10.201.106.133")
{ "ok" : 1 }

testSet:PRIMARY> rs.status()
        {
            "_id" : 2,
            "name" : "10.201.106.133:27017",
            "health" : 1,
            "state" : 7,
            "stateStr" : "ARBITER",
            "uptime" : 21,
            "lastHeartbeat" : ISODate("2018-04-25T16:06:39.938Z"),
            "lastHeartbeatRecv" : ISODate("2018-04-25T16:06:39.930Z"),
            "pingMs" : 0,
            "syncingTo" : "master1.com:27017",
            "configVersion" : 6
        }

2.4 查看slave信息

testSet:PRIMARY> rs.printSlaveReplicationInfo()
source: master1.com:27017
    syncedTo: Thu Apr 26 2018 00:50:53 GMT+0800 (CST)
    0 secs (0 hrs) behind the primary 
source: 10.201.106.133:27017
    syncedTo: Wed Apr 25 2018 23:53:51 GMT+0800 (CST)
    3422 secs (0.95 hrs) behind the primary

三、MongoDB分片

生產環境，建議一對mongos服務器通過keepalived做高可用，config服務器至少3臺實現仲裁功能，多個shard節點

1、分片（master1：mongos，master2：config server，master3,4：shard），測試環境

1.1 環境準備

停止之前的服務：
[root@master1 ~]# systemctl stop mongod
[root@master2 ~]# systemctl stop mongod
[root@master3 ~]# systemctl stop mongod

刪除之前的數據：
[root@master1 ~]# rm -rf /mongodb/data/*
[root@master2 ~]# rm -rf /mongodb/data/*
[root@master3 ~]# rm -rf /mongodb/data/*

4個節點時間同步：
/usr/sbin/ntpdate ntp1.aliyun.com

master4安裝MongoDB
[root@master4 mongodb-3.0.0]# ls
mongodb-org-server-3.0.0-1.el7.x86_64.rpm  mongodb-org-tools-3.0.0-1.el7.x86_64.rpm
mongodb-org-shell-3.0.0-1.el7.x86_64.rpm
[root@master4 mongodb-3.0.0]# yum install -y *.rpm

[root@master4 ~]# mkdir -pv /mongodb/data
[root@master4 ~]# chown -R mongod:mongod /mongodb/

1.2 首先配置config-server（master2）

[root@master2 ~]# vim /etc/mongod.conf 

#註釋剛纔的複製集配置
#replSet=testSet
#replIndexPrefetch=_id_only

dbpath=/mongodb/data
#配置該節點爲config-server
configsvr=true

啓動mongod：
[root@master2 ~]# systemctl start mongod

監聽端口：
[root@master2 ~]# netstat -tanp  | grep mongod
tcp        0      0 0.0.0.0:27019           0.0.0.0:*               LISTEN      24036/mongod        
tcp        0      0 0.0.0.0:28019           0.0.0.0:*               LISTEN      24036/mongod  

1.3 mongos（master1）配置

安裝mongos包：
[root@master1 mongodb-3.0.0]# yum install mongodb-org-mongos-3.0.0-1.el7.x86_64.rpm

通過命令啓動，指向config服務器，並運行在後臺：
[root@master1 ~]# mongos --configdb=10.201.106.132 --fork --logpath=/var/log/mongodb/mongos.log

查看監聽端口：
[root@master1 ~]# netstat -tanp | grep mon
tcp        0      0 0.0.0.0:27017           0.0.0.0:*               LISTEN      27801/mongos        
tcp        0      0 10.201.106.131:60956    10.201.106.132:27019    ESTABLISHED 27801/mongos        
tcp        0      0 10.201.106.131:60955    10.201.106.132:27019    ESTABLISHED 27801/mongos        
tcp        0      0 10.201.106.131:60958    10.201.106.132:27019    ESTABLISHED 27801/mongos        
tcp        0      0 10.201.106.131:60957    10.201.106.132:27019    ESTABLISHED 27801/mongos 

連接：
[root@master1 ~]# mongo
查看當前shard狀態信息：
mongos> sh.status()
--- Sharding Status --- 
  sharding version: {
    "_id" : 1,
    "minCompatibleVersion" : 5,
    "currentVersion" : 6,
    "clusterId" : ObjectId("5ae16bddf4bf9c27f1816692")
}
  shards:
  balancer:
    Currently enabled:  yes
    Currently running:  no
    Failed balancer rounds in last 5 attempts:  0
    Migration Results for the last 24 hours: 
        No recent migrations
  databases:
    {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }

1.4 配置shard節點

master3配置：
[root@master3 ~]# vim /etc/mongod.conf 

#取消剛纔的複製集配置
#replSet=testSet
#replIndexPrefetch=_id_only

#其他配置不變
dbpath=/mongodb/data
#bind_ip=127.0.0.1

啓動服務：
[root@master3 ~]# systemctl start mongod

master4：
[root@master4 ~]# vim /etc/mongod.conf 

dbpath=/mongodb/data
#註釋127.0.0.1的配置，服務會自動監聽0.0.0.0地址
#bind_ip=127.0.0.1

啓動服務：
[root@master4 ~]# systemctl start mongod

1.5 在mongos（master1）上添加shard節點

[root@master1 ~]# mongo
添加第一個shard節點
mongos> sh.addShard("10.201.106.133")
{ "shardAdded" : "shard0000", "ok" : 1 }

查看狀態
mongos> sh.status()

添加第二個shard節點：
mongos> sh.addShard("10.201.106.134")
{ "shardAdded" : "shard0001", "ok" : 1 }

1.6 啓用shard

shard是collection級別的，不分片的collection放在主shard上。

testdb數據庫開啓shard功能：
mongos> sh.enableSharding("testdb")
{ "ok" : 1 }

查看狀態，testdb數據已經支持shard功能
mongos> sh .status()
--- Sharding Status --- 
  sharding version: {
    "_id" : 1,
    "minCompatibleVersion" : 5,
    "currentVersion" : 6,
    "clusterId" : ObjectId("5ae16bddf4bf9c27f1816692")
}
  shards:
    {  "_id" : "shard0000",  "host" : "10.201.106.133:27017" }
    {  "_id" : "shard0001",  "host" : "10.201.106.134:27017" }
  balancer:
    Currently enabled:  yes
    Currently running:  no
    Failed balancer rounds in last 5 attempts:  0
    Migration Results for the last 24 hours: 
        No recent migrations
  databases:
    {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
    {  "_id" : "test",  "partitioned" : false,  "primary" : "shard0000" }
    {  "_id" : "testdb",  "partitioned" : true,  "primary" : "shard0000" }  #主shard

1.7 collection啓用分片

students啓動分片，根據age索引進行分配：
mongos> sh.shardCollection("testdb.students",{"age": 1})
{ "collectionsharded" : "testdb.students", "ok" : 1 }

查看：
mongos> sh .status()

創建數據（需要等待一段時間，可另開窗口查看db.students.find().count()
）：
mongos> use testdb
switched to db testdb
mongos> for (i=1;i<=100000;i++) db.students.insert({name:"students"+i,age:(i%120),classes:"class"+(i%10),address:"www.magedu.com,MageEdu,#85 Wenhua Road,Zhenzhou,China"})

查看狀態，已經有5個分片，按照年齡段範圍進行分片：
mongos> sh.status()
--- Sharding Status --- 
  sharding version: {
    "_id" : 1,
    "minCompatibleVersion" : 5,
    "currentVersion" : 6,
    "clusterId" : ObjectId("5ae16bddf4bf9c27f1816692")
}
  shards:
    {  "_id" : "shard0000",  "host" : "10.201.106.133:27017" }
    {  "_id" : "shard0001",  "host" : "10.201.106.134:27017" }
  balancer:
    Currently enabled:  yes
    Currently running:  no
    Failed balancer rounds in last 5 attempts:  0
    Migration Results for the last 24 hours: 
        2 : Success
  databases:
    {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
    {  "_id" : "test",  "partitioned" : false,  "primary" : "shard0000" }
    {  "_id" : "testdb",  "partitioned" : true,  "primary" : "shard0000" }
        testdb.students
            shard key: { "age" : 1 }
            chunks:
                shard0000   3         ###
                shard0001   2       ###
            { "age" : { "$minKey" : 1 } } -->> { "age" : 2 } on : shard0001 Timestamp(2, 0) 
            { "age" : 2 } -->> { "age" : 6 } on : shard0001 Timestamp(3, 0) 
            { "age" : 6 } -->> { "age" : 54 } on : shard0000 Timestamp(3, 1) 
            { "age" : 54 } -->> { "age" : 119 } on : shard0000 Timestamp(2, 3) 
            { "age" : 119 } -->> { "age" : { "$maxKey" : 1 } } on : shard0000 Timestamp(2, 4) 

1.8 查看shard信息

列出有幾個shard
mongos> use admin
switched to db admin
mongos> db.runCommand("listShards")
{
    "shards" : [
        {
            "_id" : "shard0000",
            "host" : "10.201.106.133:27017"
        },
        {
            "_id" : "shard0001",
            "host" : "10.201.106.134:27017"
        }
    ],
    "ok" : 1
}

顯示集羣詳細信息：
mongos> db.printShardingStatus()

shard幫助：
mongos> sh.help()

1.9 查看均衡器

查看均衡器是否工作（需要重新均衡時系統纔會自動啓動，不用管它）：
mongos> sh.isBalancerRunning()
false

查看當前Balancer狀態：
mongos> sh.getBalancerState()
true

移動chunk（該操作會觸發config-server更新元數據，不到萬不得已建議不要手動移動chunk）：
mongos> sh.moveChunk("testdb.students",{Age: {$gt: 119}},"shard0000")