本教程基于Ubuntu 14.04.2 LTS \n \l和mongoDB 3.0, 配置了3个 Config Servers(在产品环境中被要求),2个Query Router查询分发(最少1个必要),和3个shard Server(最少2个)。
注意确保在所有上述Server上已经安装了MongoDB 3.0以上版本,否则首先安装MongoDB,要在Ubuntu上安装MongoDB可以按照这个教程,《升级MongoDB到3.0.2并启用wiredTiger存储引擎 基于ubuntu 14.04》,https://www.askmac.cn/archives/upgrade-to-mongodb-302-wiredtiger.html
以下是HOSTNAME 和 IP 列表,注意本文的网络拓扑按照如下配置,如果与之不同则需要相应修改
10.132.24.62 config0.dbdao.com 10.132.24.60 config1.dbdao.com 10.132.24.61 config2.dbdao.com 10.132.24.59 query0.dbdao.com 10.132.24.63 query1.dbdao.com 10.132.24.64 shard0.dbdao.com 10.132.24.65 shard1.dbdao.com 10.132.24.67 shard2.dbdao.com
第一步配置config server
需要在配置query router和shard server前配置3个 mongoDB config server:
以下需要在3个 mongoDB config server上均需以root用户执行
以下在三个config server上配置目录 mkdir -p /m01/mongo-metadata chown mongodb:mongodb /m01/mongo-metadata 在三个config server上分别执行,注意 IP要一一对应 config0: mongod --configsvr --dbpath /m01/mongo-metadata --bind_ip 10.132.24.62 --port 35001 config1: mongod --configsvr --dbpath /m01/mongo-metadata --bind_ip 10.132.24.60 --port 35001 config2: mongod --configsvr --dbpath /m01/mongo-metadata --bind_ip 10.132.24.61 --port 35001
第二步配置Query Router Instances
分别在2个Query Router服务器上执行下列的命令
确保2个Query Router服务器上的mongd服务已经关闭 su - root service mongod stop 启动mongos 在2个Query Router服务器上执行下列命令 su - root mongos --configdb config0.dbdao.com:35001,config1.dbdao.com:35001,config2.dbdao.com:35001
第三步 增加shard到集群
注意这里没有配置Replication Set RS ,如果需要配置replica-set可以关注maclean今后的文章。
以root用户在三个sharding server上执行下面的命令:
service mongod stop mkdir -p /m01/mongo-data chown mongodb:mongodb /m01/mongo-data 以下是针对不同server分别执行 shard0: mongod --dbpath /m01/mongo-data --bind_ip 10.132.24.64 --port 35001 --storageEngine wiredTiger shard1: mongod --dbpath /m01/mongo-data --bind_ip 10.132.24.65 --port 35001 --storageEngine wiredTiger shard2: mongod --dbpath /m01/mongo-data --bind_ip 10.132.24.67 --port 35001 --storageEngine wiredTiger
之后登陆query router来增加shard
sh.addShard( "shard0.dbdao.com:35001" ) sh.addShard( "shard1.dbdao.com:35001" ) sh.addShard( "shard2.dbdao.com:35001" )
之后可以在DB级别启用sharding
mongo query0.dbdao.com:27017 use test_db db show dbs sh.enableSharding("test_db") mongos> db.databases.find() { "_id" : "admin", "partitioned" : false, "primary" : "config" } { "_id" : "test_db", "partitioned" : true, "primary" : "shard0000" }
在collection级别启用sharding
mongo query0.dbdao.com:27017 use test_db db.test_collection.ensureIndex( { _id : "hashed" } ) mongos> db.test_collection.ensureIndex( { _id : "hashed" } ) { "raw" : { "shard0.dbdao.com:35001" : { "createdCollectionAutomatically" : true, "numIndexesBefore" : 1, "numIndexesAfter" : 2, "ok" : 1 } }, "ok" : 1 } mongos> sh.shardCollection("test_db.test_collection", { "_id": "hashed" } ) { "collectionsharded" : "test_db.test_collection", "ok" : 1 } mongos> mongos> use test_db switched to db test_db mongos> for (var i = 1; i <= 500; i++) db.test_collection.insert( { x : i } ) WriteResult({ "nInserted" : 1 }) mongos> db.test_collection.find(); { "_id" : ObjectId("554b296c160953211da4b523"), "x" : 2 } { "_id" : ObjectId("554b296c160953211da4b522"), "x" : 1 } { "_id" : ObjectId("554b296c160953211da4b524"), "x" : 3 } { "_id" : ObjectId("554b296c160953211da4b526"), "x" : 5 } { "_id" : ObjectId("554b296c160953211da4b529"), "x" : 8 } { "_id" : ObjectId("554b296c160953211da4b525"), "x" : 4 } { "_id" : ObjectId("554b296c160953211da4b52c"), "x" : 11 } { "_id" : ObjectId("554b296c160953211da4b52b"), "x" : 10 } { "_id" : ObjectId("554b296c160953211da4b527"), "x" : 6 } { "_id" : ObjectId("554b296c160953211da4b52d"), "x" : 12 } { "_id" : ObjectId("554b296c160953211da4b52f"), "x" : 14 } { "_id" : ObjectId("554b296c160953211da4b528"), "x" : 7 } { "_id" : ObjectId("554b296c160953211da4b52e"), "x" : 13 } { "_id" : ObjectId("554b296c160953211da4b530"), "x" : 15 } { "_id" : ObjectId("554b296c160953211da4b52a"), "x" : 9 } { "_id" : ObjectId("554b296c160953211da4b531"), "x" : 16 } { "_id" : ObjectId("554b296c160953211da4b532"), "x" : 17 } { "_id" : ObjectId("554b296c160953211da4b533"), "x" : 18 } { "_id" : ObjectId("554b296c160953211da4b53b"), "x" : 26 } { "_id" : ObjectId("554b296c160953211da4b534"), "x" : 19 } Type "it" for more mongos> db.test_collection.count(); 500 mongos> sh.status(); --- Sharding Status --- sharding version: { "_id" : 1, "minCompatibleVersion" : 5, "currentVersion" : 6, "clusterId" : ObjectId("554b241f4df23a46a60f6a9c") } shards: { "_id" : "shard0000", "host" : "shard0.dbdao.com:35001" } { "_id" : "shard0001", "host" : "shard1.dbdao.com:35001" } { "_id" : "shard0002", "host" : "shard2.dbdao.com:35001" } balancer: Currently enabled: yes Currently running: no Failed balancer rounds in last 5 attempts: 0 Migration Results for the last 24 hours: 2 : Success 1 : Failed with error 'could not acquire collection lock for test_db.test_collection to migrate chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for migrating chunk [{ : MinKey }, { : MaxKey }) in test_db.test_collection is taken.', from shard0000 to shard0002 databases: { "_id" : "admin", "partitioned" : false, "primary" : "config" } { "_id" : "test_db", "partitioned" : true, "primary" : "shard0000" } test_db.test_collection shard key: { "_id" : "hashed" } chunks: shard0000 2 shard0001 2 shard0002 2 { "_id" : { "$minKey" : 1 } } -->> { "_id" : NumberLong("-6148914691236517204") } on : shard0000 Timestamp(3, 2) { "_id" : NumberLong("-6148914691236517204") } -->> { "_id" : NumberLong("-3074457345618258602") } on : shard0000 Timestamp(3, 3) { "_id" : NumberLong("-3074457345618258602") } -->> { "_id" : NumberLong(0) } on : shard0001 Timestamp(3, 4) { "_id" : NumberLong(0) } -->> { "_id" : NumberLong("3074457345618258602") } on : shard0001 Timestamp(3, 5) { "_id" : NumberLong("3074457345618258602") } -->> { "_id" : NumberLong("6148914691236517204") } on : shard0002 Timestamp(3, 6) { "_id" : NumberLong("6148914691236517204") } -->> { "_id" : { "$maxKey" : 1 } } on : shard0002 Timestamp(3, 7) > use test_db; switched to db test_db > db.serverStatus(); { "host" : "shard1.dbdao.com:35001", > db.test_collection.count(); 171 > db.serverStatus(); { "host" : "shard0.dbdao.com:35001", > use test_db; switched to db test_db > > db.test_collection.count(); 169 > db.serverStatus(); { "host" : "shard2.dbdao.com:35001", > use test_db; switched to db test_db > db.test_collection.count(); 160
以上三个sharding server的数据分别为 171、169、160 , 正好为总数500.
vi /etc/hosts3 Config Servers (Required in production environments)2 Query Routers (Minimum of 1 necessary)4 Shard Servers (Minimum of 2 necessary) 10.132.24.62 config0.dbdao.com 10.132.24.60 config1.dbdao.com 10.132.24.61 config2.dbdao.com 10.132.24.59 query0.dbdao.com 10.132.24.63 query1.dbdao.com 10.132.24.64 shard0.dbdao.com 10.132.24.65 shard1.dbdao.com 10.132.24.67 shard2.dbdao.com 第一步配置config server,他们需要可用在query router和shards可配置前。 使用root 登陆,先创建目录 mkdir -p /m01/mongo-metadata chown mongodb:mongodb /m01/mongo-metadata mongod –configsvr –dbpath /m01/mongo-metadata –bind_ip 10.132.24.62 –port 35001 The server will start outputting information and will begin listening for connections from other components.Repeat this process exactly on the other two configuration servers. The port number should be the same across all three servers. config0: mongod –configsvr –dbpath /m01/mongo-metadata –bind_ip 10.132.24.62 –port 35001 config1: mongod –configsvr –dbpath /m01/mongo-metadata –bind_ip 10.132.24.60 –port 35001 config2: mongod –configsvr –dbpath /m01/mongo-metadata –bind_ip 10.132.24.61 –port 35001 升级MongoDB到3.0.2并启用wiredTiger存储引擎 基于ubuntu 14.04 第二步配置Query Router Instances service mongod stop 启动mongos mongos –configdb config0.dbdao.com:35001,config1.dbdao.com:35001,config2.dbdao.com:35001 保持上述语句在多个query上一样执行 第三步 增加shard到集群 这里没有用到configure replica sets service mongod stop mkdir -p /m01/mongo-data chown mongodb:mongodb /m01/mongo-data shard0: mongod –dbpath /m01/mongo-data –bind_ip 10.132.24.64 –port 35001 –storageEngine wiredTigershard1: mongod –dbpath /m01/mongo-data –bind_ip 10.132.24.65 –port 35001 –storageEngine wiredTigershard2: mongod –dbpath /m01/mongo-data –bind_ip 10.132.24.67 –port 35001 –storageEngine wiredTigermongo query0.dbdao.com:27017sh.addShard( “shard0.dbdao.com:35001” )sh.addShard( “shard1.dbdao.com:35001” )sh.addShard( “shard2.dbdao.com:35001” )为Collection启用Sharding mongo config0.dbdao.com:35001connecting to: query0.dbdao.com:27017/testServer has startup warnings: 2015-05-07T04:37:31.488-0400 I CONTROL ** WARNING: You are running this process as the root user, which is not recommended.2015-05-07T04:37:31.488-0400 I CONTROL mongos> mongos> use test_dbswitched to db test_dbmongos> dbtest_dbmongos> show dbsadmin (empty)config 0.016GBmongos> sh.enableSharding(“test_db”){ “ok” : 1 }mongos> use configswitched to db configmongos> db.databases.find(){ “_id” : “admin”, “partitioned” : false, “primary” : “config” }{ “_id” : “test_db”, “partitioned” : true, “primary” : “shard0000” }mongos> mongos> mongos> mongos> use test_dbswitched to db test_dbmongos> mongos> use test_dbswitched to db test_dbmongos> db.test_collection.ensureIndex( { _id : “hashed” } ){ “raw” : { “shard0.dbdao.com:35001” : { “createdCollectionAutomatically” : true, “numIndexesBefore” : 1, “numIndexesAfter” : 2, “ok” : 1 } }, “ok” : 1}mongos> sh.shardCollection(“test_db.test_collection”, { “_id”: “hashed” } ){ “collectionsharded” : “test_db.test_collection”, “ok” : 1 }mongos> mongos> use test_dbswitched to db test_dbmongos> for (var i = 1; i <= 500; i++) db.test_collection.insert( { x : i } )WriteResult({ “nInserted” : 1 })mongos> db.test_collection.find();{ “_id” : ObjectId(“554b296c160953211da4b523”), “x” : 2 }{ “_id” : ObjectId(“554b296c160953211da4b522”), “x” : 1 }{ “_id” : ObjectId(“554b296c160953211da4b524”), “x” : 3 }{ “_id” : ObjectId(“554b296c160953211da4b526”), “x” : 5 }{ “_id” : ObjectId(“554b296c160953211da4b529”), “x” : 8 }{ “_id” : ObjectId(“554b296c160953211da4b525”), “x” : 4 }{ “_id” : ObjectId(“554b296c160953211da4b52c”), “x” : 11 }{ “_id” : ObjectId(“554b296c160953211da4b52b”), “x” : 10 }{ “_id” : ObjectId(“554b296c160953211da4b527”), “x” : 6 }{ “_id” : ObjectId(“554b296c160953211da4b52d”), “x” : 12 }{ “_id” : ObjectId(“554b296c160953211da4b52f”), “x” : 14 }{ “_id” : ObjectId(“554b296c160953211da4b528”), “x” : 7 }{ “_id” : ObjectId(“554b296c160953211da4b52e”), “x” : 13 }{ “_id” : ObjectId(“554b296c160953211da4b530”), “x” : 15 }{ “_id” : ObjectId(“554b296c160953211da4b52a”), “x” : 9 }{ “_id” : ObjectId(“554b296c160953211da4b531”), “x” : 16 }{ “_id” : ObjectId(“554b296c160953211da4b532”), “x” : 17 }{ “_id” : ObjectId(“554b296c160953211da4b533”), “x” : 18 }{ “_id” : ObjectId(“554b296c160953211da4b53b”), “x” : 26 }{ “_id” : ObjectId(“554b296c160953211da4b534”), “x” : 19 }Type “it” for moremongos> db.test_collection.count();500mongos> db.test_collection.count();500mongos> db.test_collection.count();500mongos> db.test_collection.count();500mongos> sh.status();— Sharding Status — sharding version: { “_id” : 1, “minCompatibleVersion” : 5, “currentVersion” : 6, “clusterId” : ObjectId(“554b241f4df23a46a60f6a9c”)} shards: { “_id” : “shard0000”, “host” : “shard0.dbdao.com:35001” } { “_id” : “shard0001”, “host” : “shard1.dbdao.com:35001” } { “_id” : “shard0002”, “host” : “shard2.dbdao.com:35001” } balancer: Currently enabled: yes Currently running: no Failed balancer rounds in last 5 attempts: 0 Migration Results for the last 24 hours: 2 : Success 1 : Failed with error ‘could not acquire collection lock for test_db.test_collection to migrate chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for migrating chunk [{ : MinKey }, { : MaxKey }) in test_db.test_collection is taken.’, from shard0000 to shard0002 databases: { “_id” : “admin”, “partitioned” : false, “primary” : “config” } { “_id” : “test_db”, “partitioned” : true, “primary” : “shard0000” } test_db.test_collection shard key: { “_id” : “hashed” } chunks: shard0000 2 shard0001 2 shard0002 2 { “_id” : { “$minKey” : 1 } } –>> { “_id” : NumberLong(“-6148914691236517204”) } on : shard0000 Timestamp(3, 2) { “_id” : NumberLong(“-6148914691236517204”) } –>> { “_id” : NumberLong(“-3074457345618258602”) } on : shard0000 Timestamp(3, 3) { “_id” : NumberLong(“-3074457345618258602”) } –>> { “_id” : NumberLong(0) } on : shard0001 Timestamp(3, 4) { “_id” : NumberLong(0) } –>> { “_id” : NumberLong(“3074457345618258602”) } on : shard0001 Timestamp(3, 5) { “_id” : NumberLong(“3074457345618258602”) } –>> { “_id” : NumberLong(“6148914691236517204”) } on : shard0002 Timestamp(3, 6) { “_id” : NumberLong(“6148914691236517204”) } –>> { “_id” : { “$maxKey” : 1 } } on : shard0002 Timestamp(3, 7) use test_db; db.test_collection.count();