如何准备MongoDB 认证DBA考试 MongoDB certified DBA

2015/05/24 by Leave a Comment

MongoDB公司最近启动了其技术人员认证程序。这里分享一些准备MongoDB DBA认证考试的技巧。

为什么要获得MongoDB DBA认证？

按照MongoDB官方的介绍：“认证可以帮助用户建议技术上的可靠性和MongoDB的能力，进而对用户运行在该平台上的应用程序提供可靠、熟练的技术支持。”

MongoDB正发展为市场上最受青睐的NoSQL数据库。在支持海量数据以及易于横向扩展和管理的场景中十分灵活。不管是在云中还是非云的架构，许多公司都认为MongoDB将成为下一代的领头数据库。

随着mongodb用户的进一步增长，Mongodb的认证显然会让你在一段时间内有着求职优势。由于该认证考试仅仅在一年中的几个礼拜中可以考试，真正获得mongodb认证的开发人员和DBA数目并不会太多。 所以是时候开搞弄一个MongoDB认证了!

 

MongoDB公司提供的认证考试

MongoDB现在提供如下的认证：

  1. C100DBA: MongoDB Certified DBA Associate
  2. C100DEV: MongoDB Certified Developer Associate

不管是DBA还是DEV,mongoDB认证均会有三个级别：Associate、Professional和Master。目前为止MongoDB仅仅提供了其Associate专员级别(最低级别）的认证考试，其他的级别Professional和Master还在规划中（类似于OCP和OCM)。这里我们主要讨论MongoDB的DBA认证考试，其他的开发者Developer考试在今后的博客中讨论。

MongoDB DBA考试由哪几个部分组成？

按照MongoDB官方的说法：”一个MongoDB数据库管理员应当产品环境的mongodb数据库的运行时配置、进程、扩展、备份恢复、监控、和性能调优有着深入的理解。“

我的看法是该dba考试设计的初衷是考验参考者的MongoDB配置、管理和监控技巧的动手能力。我已经在AWS上管理mongoDB的分片和复制集集群有好几年了，这样我就很有信心去通过认证考试。大多数问题都需要MongoDB的动手经验。

认证考试题目具有普遍的综合性，而且被分成了多个部分，每一个部分分钟7~10个小题目。 题目一般都是选择题，分成多选和单选题。也有部分题目是需要在模拟的Mongodb终端里输入正确的答案。

整个考试时长为90分钟，题目的总数并不确定，应当是动态生成的。没有分值比重，每一道题目的分数应当是一样的。

MongoDB特性

这个部分是测试考生对NOSQL和MongoDB的基本概念了解的。大部分题目都是关于JSON格式、collection和document,复制和分片的原理。如果你对NOSQL概念，RDBMS和noSQL的区别， Document Store文档存放与Key-value键值存放之间的区别都很了解的话，则该环节并不太困难。

 CRUD操作

CRUD即create、read、update、delete4种操作，本环节讨论MongoDB中的所有CRUD操作。 主要集中在如何部分更新文档，更新mongodb中document的一系列知识 www.askmac.cn。该环节的问题直接了当，选项之间都是有着轻微的语法区别而已； 但如果完全没有动手经验仍很难取得佳绩。

Indexing 索引

此环节的问题主要测试用户对索引知识的深度； 大多数问题是从用户实际场景出发，如何为用户场景采用合适的索引是问题的关键。需要重点强调的是该环节更重视在适当的场景下配置合适的索引，而非关于语法正确与否。常规数据库和查询调优经验在本环节会有所帮助。

Aggregation聚集

Aggregation framework聚集框架在mongodb中算比较进阶的概念。此环节的问题很全面，有关于在使用聚集管道aggregation pipeline的场景。该场景为后续的多个题目提供背景。清楚的理解多个reducers操作 包括$match, $project, $group以及其语法在本环节十分重要。

Replication复制

本环节考察mongodb高可用特性replica set复制集的概念。问题主要围绕replica set的配置技术和最佳实践。 MongoDB的replica sets复制集，在复制集中的优先级，primary elections，arbiter’s角色等相关的配置是主要的考点。

Sharding分片

sharding是mongodb的重要特性之一。所以这个环节的题目较多也就不奇怪了，问题与sharding数据分片的概念密切相关，何时去分片一个collection，shard的配置，shared cluster中的进程和rebalancer的角色。对一个shard cluster中shared服务器间数据的迁移了解是考试的重点。

应用管理

该环节是关于MongoDB中的journaling，authentication和 authorization，对mongodb的journal日志了解很重要，很多题目围绕这个展开。

服务器管理

该环节较难，对profiler的理解、collection stats集合状态、解释查询都是主要考点。对管理命令例如 mongostat的输出内容的理解要十分清楚。问题都与具体场景相关， 会有给定的命令，基于这些命令的输出来判定数据库是否健康（www.askmac.cn)。

考试费用是150美元！

如何注册考试？

可以参考 https://www.askmac.cn/archives/%E9%A2%84%E7%BA%A6mongodb-university%E7%9A%84dba%E8%AE%A4%E8%AF%81%E8%80%83%E8%AF%95.html

  1. 注意考试是基于网络WEB的考试由Software Secure Inc提供考试技术，要求考生在一个安静的房间，在考试的间隙不能被打扰。系统要求如下：
  2. 操作系统可以是windows或mac OSX，目前Linux不受支持
  3. 需要有摄像头，摄像头用来确认你的身份证或其他身份证明，同时监控你的考试过程
  4. 需要有麦
  5. 网络带宽够快，（这个一开始会用webcam让你连到印度的考试中心验证你的身份，我考的时候遇到的是一个印度小姑娘监控 maclean按）

考试结果

考试结果会再考试完成后2~3周内公布，主要是发邮件给你。MongoDB公司会基于问题的难易度来均衡计算成绩。如果高于合格分数就会获得MongoDB certified DBA ,Associate Level的认证badge

mongodb-certified-dba-associate-level-300x160

我如何准备该MongoDB certified DBA考试？

最好的准备方法显然是先用上mongodb 1~2年咯。同时现在市面上也有好几们mongodb的书。

 

我们正在准备后续开国内mongodb的课程，敬请期待！！！ (askmac.cn)

 

本文翻译自：http://blog.cloudthat.in/mongodb-certified-dba-associate-exam/

Filed Under: MongoDB

C100DBA MongoDB 讲解-information can be extracted from rs.status() output?

2015/05/24 by Leave a Comment

C100DBA MongoDB 讲解-information can be extracted from rs.status() output?

what useful information can be extracted from rs.status() output?

 

  1. Data about the node against which rs.status() is run from the perspective of the nodes in the replica set
  2. Data about which server each secondary is syncing from(askmac.cn)
  3. Data about the state of other nodes in the replica set from the perspective of the node against which rs.status() is run

中文为：

从rs.status()返回的信息中可以获取以下哪些信息？

  1. Replica Set中运行rs.status()的节点的相关数据
  2. 每一个secondary服务器在基于那个服务器做同步
  3. Replica Set中运行rs.status()的节点意外服务器的信息

 

官方文档对rs.status()的解释是：

This output reflects the current status of the replica set, using data derived from the heartbeat packets sent by the other members of the replica set.

This method provides a wrapper around the replSetGetStatus command. See the documentation of the command for a complete description of the output.

http://docs.mongodb.org/manual/reference/method/rs.status/

官方的解释说明了数据是通过心跳包在replica set复制集的节点间传递的，其反应了replica set的当前状态，所以其不仅仅包括了replica set中本节点的信息(askmac.cn)，

 

实际rs.status()的样例， “syncingTo” : “rep0.dbdao.com:35001”,反应了secondary服务器在基于谁做同步：

rsdbao:PRIMARY> rs.status()
{
        "set" : "rsdbao",
        "date" : ISODate("2015-05-14T03:07:00.883Z"),
        "myState" : 1,
        "members" : [
                {
                        "_id" : 0,
                        "name" : "rep0.dbdao.com:35001",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 345,
                        "optime" : Timestamp(1431572809, 1),
                        "optimeDate" : ISODate("2015-05-14T03:06:49Z"),
                        "electionTime" : Timestamp(1431572753, 2),
                        "electionDate" : ISODate("2015-05-14T03:05:53Z"),
                        "configVersion" : 3,
                        "self" : true
                },
                {
                        "_id" : 1,
                        "name" : "rep1.dbdao.com:35001",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 22,
                        "optime" : Timestamp(1431572809, 1),
                        "optimeDate" : ISODate("2015-05-14T03:06:49Z"),
                        "lastHeartbeat" : ISODate("2015-05-14T03:07:00.011Z"),
                        "lastHeartbeatRecv" : ISODate("2015-05-14T03:07:00.650Z"),
                        "pingMs" : 1,
                        "syncingTo" : "rep0.dbdao.com:35001",
                        "configVersion" : 3
                },
                {
                        "_id" : 2,
                        "name" : "rep2.dbdao.com:35001",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 10,
                        "optime" : Timestamp(1431572809, 1),
                        "optimeDate" : ISODate("2015-05-14T03:06:49Z"),
                        "lastHeartbeat" : ISODate("2015-05-14T03:07:00.078Z"),
                        "lastHeartbeatRecv" : ISODate("2015-05-14T03:07:00.228Z"),
                        "pingMs" : 23,
                        "configVersion" : 3
                }
        ],
        "ok" : 1
}

 

同时早期版本存在rs.status()显示syncingTo信息不正确的问题。

This is a known issue. There is an open ticket about rs.status() showing the primary as syncingTo when run from a secondary if the current primary was a secondary in the past ( SERVER-9989 ). Fix verion is 2.5.1

Filed Under: MongoDB

MongoDB中修改密码和忘记密码的情况

2015/05/16 by Leave a Comment

MongoDB中修改密码和忘记密码的情况

对于忘记密码的情况，需要将mongod重启，并以auth=false的选项启动Mongod，之后就可以通过如下命令来修改密码：

 

db.changeUserPassword("用户名", "密码")

对于普通的修改密码，可以通过openssl生成随机密码并设置：

openssl rand -base64 48
db.changeUserPassword("reporting", "SOh3TbYhxuLiW8ypJPxmt1oOfL")
Filed Under: MongoDB

如何准备MongoDB 认证DBA考试 MongoDB certified DBA

2015/05/15 by Leave a Comment

MongoDB公司最近启动了其技术人员认证程序。这里分享一些准备MongoDB DBA认证考试的技巧。

为什么要获得MongoDB DBA认证？

按照MongoDB官方的介绍：“认证可以帮助用户建议技术上的可靠性和MongoDB的能力，进而对用户运行在该平台上的应用程序提供可靠、熟练的技术支持。”

MongoDB正发展为市场上最受青睐的NoSQL数据库。在支持海量数据以及易于横向扩展和管理的场景中十分灵活。不管是在云中还是非云的架构，许多公司都认为MongoDB将成为下一代的领头数据库。

随着mongodb用户的进一步增长，Mongodb的认证显然会让你在一段时间内有着求职优势。由于该认证考试仅仅在一年中的几个礼拜中可以考试，真正获得mongodb认证的开发人员和DBA数目并不会太多。 所以是时候开搞弄一个MongoDB认证了!

 

MongoDB公司提供的认证考试

MongoDB现在提供如下的认证：

  1. C100DBA: MongoDB Certified DBA Associate
  2. C100DEV: MongoDB Certified Developer Associate

不管是DBA还是DEV,mongoDB认证均会有三个级别：Associate、Professional和Master。目前为止MongoDB仅仅提供了其Associate专员级别(最低级别）的认证考试，其他的级别Professional和Master还在规划中（类似于OCP和OCM)。这里我们主要讨论MongoDB的DBA认证考试，其他的开发者Developer考试在今后的博客中讨论。

MongoDB DBA考试由哪几个部分组成？

按照MongoDB官方的说法：”一个MongoDB数据库管理员应当产品环境的mongodb数据库的运行时配置、进程、扩展、备份恢复、监控、和性能调优有着深入的理解。“

我的看法是该dba考试设计的初衷是考验参考者的MongoDB配置、管理和监控技巧的动手能力。我已经在AWS上管理mongoDB的分片和复制集集群有好几年了，这样我就很有信心去通过认证考试。大多数问题都需要MongoDB的动手经验。

认证考试题目具有普遍的综合性，而且被分成了多个部分，每一个部分分钟7~10个小题目。 题目一般都是选择题，分成多选和单选题。也有部分题目是需要在模拟的Mongodb终端里输入正确的答案。

整个考试时长为90分钟，题目的总数并不确定，应当是动态生成的。没有分值比重，每一道题目的分数应当是一样的。

MongoDB特性

这个部分是测试考生对NOSQL和MongoDB的基本概念了解的。大部分题目都是关于JSON格式、collection和document,复制和分片的原理。如果你对NOSQL概念，RDBMS和noSQL的区别， Document Store文档存放与Key-value键值存放之间的区别都很了解的话，则该环节并不太困难。

 CRUD操作

CRUD即create、read、update、delete4种操作，本环节讨论MongoDB中的所有CRUD操作。 主要集中在如何部分更新文档，更新mongodb中document的一系列知识 www.askmac.cn。该环节的问题直接了当，选项之间都是有着轻微的语法区别而已； 但如果完全没有动手经验仍很难取得佳绩。

Indexing 索引

此环节的问题主要测试用户对索引知识的深度； 大多数问题是从用户实际场景出发，如何为用户场景采用合适的索引是问题的关键。需要重点强调的是该环节更重视在适当的场景下配置合适的索引，而非关于语法正确与否。常规数据库和查询调优经验在本环节会有所帮助。

Aggregation聚集

Aggregation framework聚集框架在mongodb中算比较进阶的概念。此环节的问题很全面，有关于在使用聚集管道aggregation pipeline的场景。该场景为后续的多个题目提供背景。清楚的理解多个reducers操作 包括$match, $project, $group以及其语法在本环节十分重要。

Replication复制

本环节考察mongodb高可用特性replica set复制集的概念。问题主要围绕replica set的配置技术和最佳实践。 MongoDB的replica sets复制集，在复制集中的优先级，primary elections，arbiter’s角色等相关的配置是主要的考点。

Sharding分片

sharding是mongodb的重要特性之一。所以这个环节的题目较多也就不奇怪了，问题与sharding数据分片的概念密切相关，何时去分片一个collection，shard的配置，shared cluster中的进程和rebalancer的角色。对一个shard cluster中shared服务器间数据的迁移了解是考试的重点。

应用管理

该环节是关于MongoDB中的journaling，authentication和 authorization，对mongodb的journal日志了解很重要，很多题目围绕这个展开。

服务器管理

该环节较难，对profiler的理解、collection stats集合状态、解释查询都是主要考点。对管理命令例如 mongostat的输出内容的理解要十分清楚。问题都与具体场景相关， 会有给定的命令，基于这些命令的输出来判定数据库是否健康（www.askmac.cn)。

考试费用是150美元！

如何注册考试？

可以参考 https://www.askmac.cn/archives/%E9%A2%84%E7%BA%A6mongodb-university%E7%9A%84dba%E8%AE%A4%E8%AF%81%E8%80%83%E8%AF%95.html

  1. 注意考试是基于网络WEB的考试由Software Secure Inc提供考试技术，要求考生在一个安静的房间，在考试的间隙不能被打扰。系统要求如下：
  2. 操作系统可以是windows或mac OSX，目前Linux不受支持
  3. 需要有摄像头，摄像头用来确认你的身份证或其他身份证明，同时监控你的考试过程
  4. 需要有麦
  5. 网络带宽够快，（这个一开始会用webcam让你连到印度的考试中心验证你的身份，我考的时候遇到的是一个印度小姑娘监控 maclean按）

考试结果

考试结果会再考试完成后2~3周内公布，主要是发邮件给你。MongoDB公司会基于问题的难易度来均衡计算成绩。如果高于合格分数就会获得MongoDB certified DBA ,Associate Level的认证badge

mongodb-certified-dba-associate-level-300x160

我如何准备该MongoDB certified DBA考试？

最好的准备方法显然是先用上mongodb 1~2年咯。同时现在市面上也有好几们mongodb的书。

 

我们正在准备后续开国内mongodb的课程，敬请期待！！！ (askmac.cn)

 

本文翻译自：http://blog.cloudthat.in/mongodb-certified-dba-associate-exam/

 

Filed Under: MongoDB

快速部署有身份验证和授权的MongoDB Replica SET复制集

2015/05/14 by Leave a Comment

在启用身份验证后， MongoDB将强制所有客户端在访问数据库前先做身份验证authentication。 Authorization, 授权允许管理员限制用户使用的资源和允许做的操作。 对于产品环境中使用mongodb replica set 复制集，必须启用Authentication 和Authorization。

 

所有的MongoDB部署均支持身份验证。 默认情况下MongoDB不检测是否授权用户。对于现有部署和新部署都可以启用身份验证和授权。 但是要启用它们必须有停机时间，无法对在线的mongodb做修改。

以下我们将部署一个三节点的replica set复制集，其中的三个节点：

  1. rep0.dbdao.com    默认的primary
  2. rep1.dbdao.com    Secondary
  3. rep2.dbdao.com   Secondary

注意以下所有操作均在ubuntu 上以root用户执行，如果你要以其他用户执行，请注意权限。

ubuntu启用-root账号 可以参考： https://www.askmac.cn/archives/ubuntu%E5%90%AF%E7%94%A8-root%E8%B4%A6%E5%8F%B7.html

 

1、首先创建必要的目录

三个节点均运行

su – root                                                  ==》三个节点均运行

mkdir -p /m01/repdbdao                   ==》三个节点均运行

2、启动primary节点上的mongod

mongod –dbpath /m01/repdbdao –port 35001 –storageEngine wiredTiger

3、在无需身份验证的primary上创建用户

mongo rep0.dbdao.com:35001     ==》登陆 primary

 如下创建了siteUserAdmin和siteRootAdmin 2个超级用户

use admin
db.createUser( {
 user: "siteUserAdmin",
 pwd: "dbdao",
 roles: [ { role: "userAdminAnyDatabase", db: "admin" } ]
 });
db.createUser( {
 user: "siteRootAdmin",
 pwd: "dbdao",
 roles: [ { role: "root", db: "admin" } ]
 });

4、 ctrl+c 关掉步骤2启动的mongod

5、 在primary节点上生成一个为replica set 成员准备的SSL KEY文件

openssl rand -base64 741 > mongodb-keyfile
chmod 600 mongodb-keyfile

 

6、 将mongodb-keyfile 拷贝到其他2个节点上

 

scp mongodb-keyfile rep1.dbdao.com:~/
scp mongodb-keyfile rep2.dbdao.com:~/

 

7、 启动三个节点上的mongod

 rep0
  mongod --dbpath /m01/repdbdao --port 35001 --storageEngine wiredTiger  --replSet "rsdbao" --keyFile ~/mongodb-keyfile
  
 rep1 
  mongod --dbpath /m01/repdbdao --port 35001 --storageEngine wiredTiger  --replSet "rsdbao" --keyFile ~/mongodb-keyfile
  
 rep2 
  mongod --dbpath /m01/repdbdao --port 35001 --storageEngine wiredTiger  --replSet "rsdbao" --keyFile ~/mongodb-keyfile

8、连接到primary节点 开始Replica SET配置

mongo rep0:35001/admin -u siteRootAdmin -p

输入密码 dbdao

 

9、初始化replica set配置

执行

rs.initiate()

10、 将rep1和rep2节点加入replica set

rs.add(“rep1.dbdao.com:35001”)
rs.add(“rep2.dbdao.com:35001”)

 

11、查看replica set  状态

 

rsdbao:PRIMARY> rs.status()
{
        "set" : "rsdbao",
        "date" : ISODate("2015-05-14T03:07:00.883Z"),
        "myState" : 1,
        "members" : [
                {
                        "_id" : 0,
                        "name" : "rep0.dbdao.com:35001",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 345,
                        "optime" : Timestamp(1431572809, 1),
                        "optimeDate" : ISODate("2015-05-14T03:06:49Z"),
                        "electionTime" : Timestamp(1431572753, 2),
                        "electionDate" : ISODate("2015-05-14T03:05:53Z"),
                        "configVersion" : 3,
                        "self" : true
                },
                {
                        "_id" : 1,
                        "name" : "rep1.dbdao.com:35001",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 22,
                        "optime" : Timestamp(1431572809, 1),
                        "optimeDate" : ISODate("2015-05-14T03:06:49Z"),
                        "lastHeartbeat" : ISODate("2015-05-14T03:07:00.011Z"),
                        "lastHeartbeatRecv" : ISODate("2015-05-14T03:07:00.650Z"),
                        "pingMs" : 1,
                        "syncingTo" : "rep0.dbdao.com:35001",
                        "configVersion" : 3
                },
                {
                        "_id" : 2,
                        "name" : "rep2.dbdao.com:35001",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 10,
                        "optime" : Timestamp(1431572809, 1),
                        "optimeDate" : ISODate("2015-05-14T03:06:49Z"),
                        "lastHeartbeat" : ISODate("2015-05-14T03:07:00.078Z"),
                        "lastHeartbeatRecv" : ISODate("2015-05-14T03:07:00.228Z"),
                        "pingMs" : 23,
                        "configVersion" : 3
                }
        ],
        "ok" : 1
}

12、为了今后的管理可以为单个数据库创建管理者

 

use products
db.createUser(
  {
    user: "productsDBAdmin",
    pwd: "dbdao",
    roles:
    [
      {
        role: "dbOwner",
        db: "products"
      }
    ]
  }
)

 

以上基本完成了对有身份验证和授权的replica set的配置，可以使用如下命令登陆三个节点：

mongo rep1.dbdao.com:35001/admin -u siteRootAdmin -p dbdao
mongo rep0.dbdao.com:35001/admin -u siteRootAdmin -p dbdao
mongo rep2.dbdao.com:35001/admin -u siteRootAdmin -p dbdao

如上配置的replica set 会自动启用failover，但primary失败时会自动投票切换到secondary，无需人工接入。

 

 

 

Filed Under: MongoDB

MongoDB 用户角色授权与AUTH启用

2015/05/13 by Leave a Comment

MongoDB中几种常用用户角色：

dbAdmin 在db范围内包括下面的权限：

  • collStats
  • dbHash
  • dbStats
  • find
  • killCursors
  • listIndexes
  • listCollections
  • dropCollection 和 createCollection on system.profile only

userAdmin在db范围内包括如下权限：

  • changeCustomData
  • changePassword
  • createRole
  • createUser
  • dropRole
  • dropUser
  • grantRole
  • revokeRole
  • viewRole
  • viewUser

readAnyDatabase  对所有数据库中的collection可读，同时包含listDatabases权限

readWriteAnyDatabase  对所有数据库中的collection可读且可写，同时包含listDatabases权限

userAdminAnyDatabase 对所有数据库拥有userAdmin角色，同时包含listDatabases权限

dbAdminAnyDatabase 对所有数据库拥有dbAdmin角色，同时包含listDatabases权限

cluster相关的权限  clusterMonitor、hostManager、clusterManager、clusterAdmin

root权限， 包含 readWriteAnyDatabase, dbAdminAnyDatabase, userAdminAnyDatabase 和 clusterAdmin 等角色。 但不能访问system. 开头的collection(root does not include any access to collections that begin with the system. prefix.)

__system 超级角色

相关官方文档：
http://docs.mongodb.org/manual/reference/built-in-roles/#__system

__system包含下面这些权限：

 

> use admin
switched to db admin
> db.createUser(
...   {
...     user: "maclean_dbdao2",
...     pwd: "maclean_dbdao2",
...     roles: [ { role: "__system", db: "admin" } ]
...   }
... )
Successfully added user: {
	"user" : "maclean_dbdao2",
	"roles" : [
		{
			"role" : "__system",
			"db" : "admin"
		}
	]
}
> 
> 
> 
bye
10:~ maclean$ mongo localhost:35002/admin -u maclean_dbdao2  -p
MongoDB shell version: 3.0.2
Enter password: 
connecting to: localhost:35002/admin
> show roles
{
	"role" : "__system",
	"db" : "admin",
	"isBuiltin" : true,
	"roles" : [ ],
	"inheritedRoles" : [ ]
}
{
	"role" : "backup",
	"db" : "admin",
	"isBuiltin" : true,
	"roles" : [ ],
	"inheritedRoles" : [ ]
}
{
	"role" : "clusterAdmin",
	"db" : "admin",
	"isBuiltin" : true,
	"roles" : [ ],
	"inheritedRoles" : [ ]
}
{
	"role" : "clusterManager",
	"db" : "admin",
	"isBuiltin" : true,
	"roles" : [ ],
	"inheritedRoles" : [ ]
}
{
	"role" : "clusterMonitor",
	"db" : "admin",
	"isBuiltin" : true,
	"roles" : [ ],
	"inheritedRoles" : [ ]
}
{
	"role" : "dbAdmin",
	"db" : "admin",
	"isBuiltin" : true,
	"roles" : [ ],
	"inheritedRoles" : [ ]
}
{
	"role" : "dbAdminAnyDatabase",
	"db" : "admin",
	"isBuiltin" : true,
	"roles" : [ ],
	"inheritedRoles" : [ ]
}
{
	"role" : "dbOwner",
	"db" : "admin",
	"isBuiltin" : true,
	"roles" : [ ],
	"inheritedRoles" : [ ]
}
{
	"role" : "hostManager",
	"db" : "admin",
	"isBuiltin" : true,
	"roles" : [ ],
	"inheritedRoles" : [ ]
}
{
	"role" : "read",
	"db" : "admin",
	"isBuiltin" : true,
	"roles" : [ ],
	"inheritedRoles" : [ ]
}
{
	"role" : "readAnyDatabase",
	"db" : "admin",
	"isBuiltin" : true,
	"roles" : [ ],
	"inheritedRoles" : [ ]
}
{
	"role" : "readWrite",
	"db" : "admin",
	"isBuiltin" : true,
	"roles" : [ ],
	"inheritedRoles" : [ ]
}
{
	"role" : "readWriteAnyDatabase",
	"db" : "admin",
	"isBuiltin" : true,
	"roles" : [ ],
	"inheritedRoles" : [ ]
}
{
	"role" : "restore",
	"db" : "admin",
	"isBuiltin" : true,
	"roles" : [ ],
	"inheritedRoles" : [ ]
}
{
	"role" : "root",
	"db" : "admin",
	"isBuiltin" : true,
	"roles" : [ ],
	"inheritedRoles" : [ ]
}
{
	"role" : "userAdmin",
	"db" : "admin",
	"isBuiltin" : true,
	"roles" : [ ],
	"inheritedRoles" : [ ]
}
{
	"role" : "userAdminAnyDatabase",
	"db" : "admin",
	"isBuiltin" : true,
	"roles" : [ ],
	"inheritedRoles" : [ ]
}


mongodb 3.0中db.getUsers() 获得db中的用户信息


> db.getUsers();
[
	{
		"_id" : "admin.maclean",
		"user" : "maclean",
		"db" : "admin",
		"roles" : [
			{
				"role" : "userAdminAnyDatabase",
				"db" : "admin"
			}
		]
	},
	{
		"_id" : "admin.maclean1",
		"user" : "maclean1",
		"db" : "admin",
		"roles" : [
			{
				"role" : "__system",
				"db" : "admin"
			}
		]
	},
	{
		"_id" : "admin.maclean_dbdao2",
		"user" : "maclean_dbdao2",
		"db" : "admin",
		"roles" : [
			{
				"role" : "__system",
				"db" : "admin"
			}
		]
	}
]

 

 

启用mongodb授权认证的方法：

1、以–auth 启动mongod

2、在配置文件mongod.conf 中加入 auth = true

第一次启用–auth时会出现：

2015-05-13T11:20:22.296+0800 I ACCESS   [conn1] note: no users configured in admin.system.users, allowing localhost access

2015-05-13T11:20:22.297+0800 I ACCESS   [conn1] Unauthorized not authorized on admin to execute command { getLog: “startupWarnings” }

2015-05-13T12:07:08.680+0800 I INDEX    [conn1] build index on: admin.system.users properties: { v: 1, unique: true, key: { user: 1, db: 1 }, name: “user_1_db_1”, ns: “admin.system.users” }

即之前未定义过用户，所以mongod将允许本地直接访问

mongo 登陆后 创建一个合适的超级用户

 

use admin
db.createUser(
  {
    user: "maclean",
    pwd: "maclean",
    roles: [ { role: "__system", db: "admin" } ]
  }
)

http://docs.mongodb.org/manual/reference/method/db.createUser/

给一个用户授权 ：

use admin
db.grantRolesToUser(
  "macleanz",
  [
    { role: "readAnyDatabase", db:"admin" }
  ]
)


http://docs.mongodb.org/manual/tutorial/assign-role-to-user/

启用replica set 时需要做的授权：

use admin
db.createUser( {
    user: "siteUserAdmin",
    pwd: "",
    roles: [ { role: "userAdminAnyDatabase", db: "admin" } ]
  });
db.createUser( {
    user: "siteRootAdmin",
    pwd: "",
    roles: [ { role: "root", db: "admin" } ]
  });


http://docs.mongodb.org/manual/tutorial/deploy-replica-set-with-auth/
Filed Under: MongoDB

快速部署MongoDB Replica Set复制集

2015/05/12 by Leave a Comment

本文介绍如何快速部署MongoDB  Replica Set复制集，注意本文中的配置是不涉及授权验证的。

MongoDB版本为3.0.2 , 该部署全部在同一台服务器上完成，实际产品环境 Replica Set 中master和slave显然应当在不同的服务器上， 仅仅需要对本文中涉及IP的地方有所修改即可

我们假定用户已经成功安装mongoDB 3.0.2

 

启动 replica set的primary节点上的mongodb

sudo mkdir -p /m01/mongo1
sudo mkdir -p /m01/mongo2
sudo mkdir -p /m01/mongo3



mongod -dbpath /m01/mongo1 --replSet "rs_maclean" --port 35001 --storageEngine wiredTiger   &
mongod -dbpath /m01/mongo2 --replSet "rs_maclean" --port 35002 --storageEngine wiredTiger   &
mongod -dbpath /m01/mongo3 --replSet "rs_maclean" --port 35003 --storageEngine wiredTiger   &


登陆到已经启动的primary mongodb

mongo localhost:35001
MongoDB shell version: 3.0.2
connecting to: localhost:35001/test


初始化 replica set  成员

rs.initiate()

获得 replica set配置信息

rs.conf()

rs_maclean:PRIMARY> rs.conf();
{
	"_id" : "rs_maclean",
	"version" : 1,
	"members" : [
		{
			"_id" : 0,
			"host" : "shard0.dbdao.com:35001",
			"arbiterOnly" : false,
			"buildIndexes" : true,
			"hidden" : false,
			"priority" : 1,
			"tags" : {
				
			},
			"slaveDelay" : 0,
			"votes" : 1
		}
	],
	"settings" : {
		"chainingAllowed" : true,
		"heartbeatTimeoutSecs" : 10,
		"getLastErrorModes" : {
			
		},
		"getLastErrorDefaults" : {
			"w" : 1,
			"wtimeout" : 0
		}
	}
}



添加其他2个replica set member 

rs.add("ip2:35002")
rs.add("ip3:35003")


检查 Replica set 状态

rs_maclean:PRIMARY> rs.status()
{
	"set" : "rs_maclean",
	"date" : ISODate("2015-05-12T14:34:57.205Z"),
	"myState" : 1,
	"members" : [
		{
			"_id" : 0,
			"name" : "shard0.dbdao.com:35001",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
			"uptime" : 398,
			"optime" : Timestamp(1431441212, 1),
			"optimeDate" : ISODate("2015-05-12T14:33:32Z"),
			"electionTime" : Timestamp(1431440950, 2),
			"electionDate" : ISODate("2015-05-12T14:29:10Z"),
			"configVersion" : 2,
			"self" : true
		},
		{
			"_id" : 1,
			"name" : "shard1.dbdao.com:35002",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 85,
			"optime" : Timestamp(1431441212, 1),
			"optimeDate" : ISODate("2015-05-12T14:33:32Z"),
			"lastHeartbeat" : ISODate("2015-05-12T14:34:56.044Z"),
			"lastHeartbeatRecv" : ISODate("2015-05-12T14:34:56.061Z"),
			"pingMs" : 0,
			"configVersion" : 2
		}
.......
	],
	"ok" : 1
}

此时我们触发 primary的mongod 进程被kill

root@shard0:~# ps -ef|grep mongod
root     24591 24566  1 10:28 pts/1    00:00:12 mongod -dbpath /m01/mongo1 --replSet rs_maclean --port 35001 --storageEngine wiredTiger

root     24970 24566  0 10:41 pts/1    00:00:00 grep --color=auto mongod
root@shard0:~# 
root@shard0:~# kill -9 24591




此时触发 replSet possible election tie


2015-05-12T10:59:59.476-0400 E REPL     [rsBackgroundSync] sync producer problem: 10278 dbclient error communicating with server: shard0.dbdao.com:35001
2015-05-12T10:59:59.478-0400 I REPL     [ReplicationExecutor] could not find member to sync from
2015-05-12T11:00:00.382-0400 I NETWORK  [ReplExecNetThread-1] DBClientCursor::init call() failed
2015-05-12T11:00:00.382-0400 I REPL     [ReplicationExecutor] Standing for election
2015-05-12T11:00:00.384-0400 I REPL     [ReplicationExecutor] replSet possible election tie; sleeping 961ms until 2015-05-12T11:00:01.345-0400
2015-05-12T11:00:01.345-0400 I REPL     [ReplicationExecutor] Standing for election
2015-05-12T11:00:01.346-0400 I REPL     [ReplicationExecutor] replSet info electSelf
2015-05-12T11:00:01.348-0400 I REPL     [ReplicationExecutor] replSet election succeeded, assuming primary role
2015-05-12T11:00:01.348-0400 I REPL     [ReplicationExecutor] transition to PRIMARY
2015-05-12T11:00:01.605-0400 I REPL     [rsSync] transition to primary complete; database writes are now permitted
2015-05-12T11:00:04.853-0400 I NETWORK  [conn15] end connection 127.0.0.1:34149 (3 connections now open)
2015-05-12T11:00:05.735-0400 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:34160 #18 (4 connections now open)
2015-05-12T11:00:08.011-0400 I NETWORK  [conn18] end connection 127.0.0.1:34160 (3 connections now open)
2015-05-12T11:00:21.642-0400 I NETWORK  [conn17] end connection 104.236.21.67:44231 (2 connections now open)
2015-05-12T11:00:21.643-0400 I NETWORK  [initandlisten] connection accepted from 104.236.21.67:44264 #19 (3 connections now open)
2015-05-12T11:00:22.548-0400 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:34189 #20 (4 connections now open)
2015-05-12T11:00:23.932-0400 I NETWORK  [conn20] end connection 127.0.0.1:34189 (3 connections now open)
2015-05-12T11:00:51.671-0400 I NETWORK  [conn19] end connection 104.236.21.67:44264 (2 connections now open)
2015-05-12T11:00:51.671-0400 I NETWORK  [initandlisten] connection accepted from 104.236.21.67:44310 #21 (3 connections now open)
2015-05-12T11:01:21.700-0400 I NETWORK  [conn21] end connection 104.236.21.67:44310 (2 connections now open)
2015-05-12T11:01:21.701-0400 I NETWORK  [initandlisten] connection accepted from 104.236.21.67:44356 #22 (3 connections now open)


secondary 接管primary

当原primary节点重新启动会成为SECONDARY member

Filed Under: MongoDB

MongoDB BSONObj size is invalid错误一例

2015/05/12 by Leave a Comment

某个MongoDB数据库出现如下错误”assertion 10334 BSONObj size: 1684627202 (0x64695F02) is invalid. Size must be between 0 and 16793600(16MB) First element: EOO ns:local.oplog.$main query:{ query: {}, orderby: { $natural: -1 } }”一例：

 

 

2015-05-12T16:12:43.872+0800 I -        ***** SERVER RESTARTED *****
2015-05-12T16:12:43.883+0800 I -        [initandlisten] MongoDB starting : pid=1697 port=27019 dbpath=arb/mongodb master=1 64-bit host=ubuntu
2015-05-12T16:12:43.883+0800 I -        [initandlisten] 
2015-05-12T16:12:43.884+0800 I -        [initandlisten] ** NOTE: This is a development version (2.7.8) of MongoDB.
2015-05-12T16:12:43.884+0800 I -        [initandlisten] **       Not recommended for production.
2015-05-12T16:12:43.884+0800 I -        [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2015-05-12T16:12:43.884+0800 I -        [initandlisten] 
2015-05-12T16:12:43.884+0800 I -        [initandlisten] db version v2.7.8
2015-05-12T16:12:43.884+0800 I -        [initandlisten] git version: 0e9cb3b20498b9f167afaff7a5c4a4d1da7e06a2
2015-05-12T16:12:43.884+0800 I -        [initandlisten] build info: Linux build4.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2015-05-12T16:12:43.884+0800 I -        [initandlisten] allocator: tcmalloc
2015-05-12T16:12:43.884+0800 I -        [initandlisten] options: { config: "/etc/mongodb.conf", net: { bindIp: "0.0.0.0" }, processManagement: { fork: true, pidFilePath: "arb/mongodb/mongodb.pid" }, sharding: { clusterRole: "configsvr" }, storage: { dbPath: "arb/mongodb", preallocDataFiles: false }, systemLog: { destination: "file", logAppend: true, path: "ar/log/mongodb.log" } }
2015-05-12T16:12:43.887+0800 I JOURNAL  [initandlisten] journal dir=arb/mongodb/journal
2015-05-12T16:12:43.888+0800 I JOURNAL  [initandlisten] recover : no journal files present, no recovery needed
2015-05-12T16:12:43.914+0800 I -        [initandlisten] Assertion: 10334:BSONObj size: 1684627202 (0x64695F02) is invalid. Size must be between 0 and 16793600(16MB) First element: EOO
 mongod(main+0x134) [0x7bbf34]
 libc.so.6(__libc_start_main+0xF5) [0x7f13b661aec5]
 mongod(+0x3EEC01) [0x7eec01]
-----  END BACKTRACE  -----
2015-05-12T15:55:47.951+0800 I COMMANDS [initandlisten] assertion 10334 BSONObj size: 1684627202 (0x64695F02) is invalid. Size must be between 0 and 16793600(16MB) First element: EOO ns:local.oplog.$main query:{ query: {}, orderby: { $natural: -1 } }
2015-05-12T15:55:47.951+0800 I COMMANDS [initandlisten]  ntoskip:0 ntoreturn:1
2015-05-12T15:55:47.952+0800 I STORAGE  [initandlisten] exception in initAndListen: 13106 nextSafe(): { $err: "BSONObj size: 1684627202 (0x64695F02) is invalid. Size must be between 0 and 16793600(16MB) First element: EOO", code: 10334 }, terminating
2015-05-12T15:55:47.952+0800 I COMMANDS [initandlisten] now exiting
2015-05-12T15:55:47.952+0800 I NETWORK  [initandlisten] shutdown: going to close listening sockets...
2015-05-12T15:55:47.952+0800 I NETWORK  [initandlisten] removing socket file: /tmp/mongodb-27019.sock
2015-05-12T15:55:47.952+0800 I NETWORK  [initandlisten] shutdown: going to flush diaglog...
2015-05-12T15:55:47.952+0800 I NETWORK  [initandlisten] shutdown: going to close sockets...
2015-05-12T15:55:47.952+0800 I STORAGE  [initandlisten] shutdown: waiting for fs preallocator...
2015-05-12T15:55:47.952+0800 I STORAGE  [initandlisten] shutdown: final commit...
2015-05-12T15:55:47.953+0800 I STORAGE  [initandlisten] shutdown: closing all files...
2015-05-12T15:55:47.954+0800 I STORAGE  [initandlisten] closeAllFiles() finished
2015-05-12T15:55:47.954+0800 I JOURNAL  [initandlisten] journalCleanup...
2015-05-12T15:55:47.954+0800 I JOURNAL  [initandlisten] removeJournalFiles
2015-05-12T15:55:47.958+0800 I STORAGE  [initandlisten] shutdown: removing fs lock...
2015-05-12T15:55:47.958+0800 I COMMANDS [initandlisten] dbexit:  rc: 100
.

针对该 BSONObj size invalid，主要是mongodb读取到数据文件后发现存在BSON对象的大小超过16MB的非法对象所以报错，由于其认为数据库可能存在损坏，所以会强制关闭mongd实例

 

针对此类案例一般考虑：

1、repair database ,修复数据库

2、如果确实无法修复数据库，则考虑从备份中恢复数据库

 

一、repair database的例子：

 

mkdir  /m01/mongo-data02


mongodb数据文件位于 /m01/mongo-data01  ，使用repairpath指定修复后的文件的位置。

注意/m01/mongo-data01 下需要有journal 


root@shard0:~# 
root@shard0:~# mongod -dbpath /m01/mongo-data01 --repair --repairpath /m01/mongo-data02  --storageEngine wiredTiger|less
2015-05-12T04:39:41.234-0400 I CONTROL  [initandlisten] MongoDB starting : pid=19575 port=27017 dbpath=/m01/mongo-data01 64-bit host=shard0.dbdao.com
2015-05-12T04:39:41.234-0400 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2015-05-12T04:39:41.234-0400 I CONTROL  [initandlisten] 
2015-05-12T04:39:41.234-0400 I CONTROL  [initandlisten] 
2015-05-12T04:39:41.234-0400 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2015-05-12T04:39:41.234-0400 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2015-05-12T04:39:41.234-0400 I CONTROL  [initandlisten] 
2015-05-12T04:39:41.235-0400 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2015-05-12T04:39:41.235-0400 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2015-05-12T04:39:41.235-0400 I CONTROL  [initandlisten] 
2015-05-12T04:39:41.235-0400 I CONTROL  [initandlisten] db version v3.0.0
2015-05-12T04:39:41.235-0400 I CONTROL  [initandlisten] git version: a841fd6394365954886924a35076691b4d149168
2015-05-12T04:39:41.235-0400 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014
2015-05-12T04:39:41.235-0400 I CONTROL  [initandlisten] build info: Linux ip-10-179-177-12 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2015-05-12T04:39:41.235-0400 I CONTROL  [initandlisten] allocator: tcmalloc
2015-05-12T04:39:41.235-0400 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "/m01/mongo-data01", engine: "wiredTiger", repairPath: "/m01/mongo-data02" } }
2015-05-12T04:39:41.262-0400 W -        [initandlisten] Detected unclean shutdown - /m01/mongo-data01/mongod.lock is not empty.
2015-05-12T04:39:41.263-0400 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2015-05-12T04:39:41.263-0400 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),statistics=(fast),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2015-05-12T04:39:41.317-0400 I STORAGE  [initandlisten] Repairing size cache
2015-05-12T04:39:41.320-0400 I STORAGE  [initandlisten] WiredTiger progress session.verify 2
2015-05-12T04:39:41.320-0400 I STORAGE  [initandlisten] Verify succeeded on uri table:sizeStorer. Not salvaging.
2015-05-12T04:39:41.321-0400 I STORAGE  [initandlisten] Repairing catalog metadata
2015-05-12T04:39:41.323-0400 I STORAGE  [initandlisten] WiredTiger progress session.verify 2
2015-05-12T04:39:41.323-0400 I STORAGE  [initandlisten] Verify succeeded on uri table:_mdb_catalog. Not salvaging.
2015-05-12T04:39:41.329-0400 I STORAGE  [initandlisten] repairDatabase local
2015-05-12T04:39:41.329-0400 I STORAGE  [initandlisten] Repairing collection local.startup_log
2015-05-12T04:39:41.332-0400 I STORAGE  [initandlisten] WiredTiger progress session.verify 2
2015-05-12T04:39:41.332-0400 I STORAGE  [initandlisten] Verify succeeded on uri table:collection-0--8766758909797526255. Not salvaging.
2015-05-12T04:39:41.338-0400 I INDEX    [initandlisten] build index on: local.startup_log properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "local.startup_log" }
2015-05-12T04:39:41.338-0400 I INDEX    [initandlisten]          building index using bulk method
2015-05-12T04:39:41.343-0400 I STORAGE  [initandlisten] repairDatabase test
2015-05-12T04:39:41.343-0400 I STORAGE  [initandlisten] Repairing collection test. otherStuff
2015-05-12T04:39:41.349-0400 I STORAGE  [initandlisten] WiredTiger progress session.verify 2
2015-05-12T04:39:41.350-0400 I STORAGE  [initandlisten] Verify succeeded on uri table:collection-4--6404712422294369090. Not salvaging.
2015-05-12T04:39:41.355-0400 I INDEX    [initandlisten] build index on: test. otherStuff properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "test. otherStuff" }
2015-05-12T04:39:41.355-0400 I INDEX    [initandlisten]          building index using bulk method
2015-05-12T04:39:41.360-0400 I STORAGE  [initandlisten] Repairing collection test.alpha

 

关于repair 更多信息可以参考 http://docs.mongodb.org/manual/reference/command/repairDatabase/

 

二、通过原有备份的mongodump来恢复数据

Filed Under: MongoDB

MongoDB Aggregation聚集测试

2015/05/12 by Leave a Comment 

db.alpha.insert({_id:1, region:"NW1", leads:1 , email: "maclean@dbdao.com"});
db.alpha.insert({_id:2, region:"NW1", leads:1 , email: "maclean@dbdao.com"});
db.alpha.insert({_id:3, region:"NW1", leads:2 , email: "maclean@dbdao.com"});
db.alpha.insert({_id:4, region:"SE1", leads:8 , email: "maclean@dbdao.com"});
db.alpha.insert({_id:5, region:"SE2", leads:4 , email: "maclean@dbdao.com"});
db.alpha.insert({_id:6, region:"SE2", leads:2 , email: "maclean@dbdao.com"});
db.alpha.insert({_id:7, region:"SE2", leads:4 , email: "maclean@dbdao.com"});
db.alpha.insert({_id:8, region:"SW1", leads:1 , email: "maclean@dbdao.com"});
db.alpha.insert({_id:9, region:"SW1", leads:2 , email: "maclean@dbdao.com"});
db.alpha.insert({_id:10, region:"SW2", leads:2 , email: "maclean@dbdao.com"});
db.alpha.insert({_id:11, region:"SW2", leads:5, email: "maclean@dbdao.com"});


> db.alpha.find();
{ "_id" : 1, "region" : "NW1", "leads" : 1, "email" : "maclean@dbdao.com" }
{ "_id" : 2, "region" : "NW1", "leads" : 1, "email" : "maclean@dbdao.com" }
{ "_id" : 3, "region" : "NW1", "leads" : 2, "email" : "maclean@dbdao.com" }
{ "_id" : 4, "region" : "SE1", "leads" : 8, "email" : "maclean@dbdao.com" }
{ "_id" : 5, "region" : "SE2", "leads" : 4, "email" : "maclean@dbdao.com" }
{ "_id" : 6, "region" : "SE2", "leads" : 2, "email" : "maclean@dbdao.com" }
{ "_id" : 7, "region" : "SE2", "leads" : 4, "email" : "maclean@dbdao.com" }
{ "_id" : 8, "region" : "SW1", "leads" : 1, "email" : "maclean@dbdao.com" }
{ "_id" : 9, "region" : "SW1", "leads" : 2, "email" : "maclean@dbdao.com" }
{ "_id" : 10, "region" : "SW2", "leads" : 2, "email" : "maclean@dbdao.com" }
{ "_id" : 11, "region" : "SW2", "leads" : 5, "email" : "maclean@dbdao.com" }


> 
> db.alpha.aggregate ( [ { "$group" : { "_id" : "$region" , leads : { "$sum" :1 }}} , { "$match" : { "leads" : { "$gte" :3 }}} ])
{ "_id" : "SE2", "leads" : 3 }
{ "_id" : "NW1", "leads" : 3 }
>

 

db.alpha.aggregate ( [ { "$group" : { "_id" : "$region" , leads : { "$sum" :1 }}} , { "$match" : { "leads" : { "$gte" :3 }}} ])
{ "_id" : "SE2", "leads" : 3 }
{ "_id" : "NW1", "leads" : 3 }

其相当于SQL的

select region,count(*) from alpha group by region having count(*)>=3;

db.alpha.aggregate ( [ { "$group" : { "_id" : "$region" , "leads" : { "$sum" : "$leads" }}} , { "$match" : { "leads" : { "$gte" :2 }}} ]);

其相当于SQL的
select region,sum(leads) from alpha group by region having sum(leads)>=3;


db.collection.aggregate 的语法如下：


db.collection.aggregate()


 


New in version 2.2.


 




Definition


 




db.collection.aggregate(pipeline, options)


Calculates aggregate values for the data in a collection.


Type

Description

array


A sequence of data aggregation operations or stages. See theaggregation pipeline operators for details.


 


Changed in version 2.6: The method can still accept the pipeline stages as separate arguments instead of as elements in an array; however, if you do not specify the pipeline as an array, you cannot specify theoptions parameter.


document


Optional. Additional options that aggregate() passes to theaggregate command.


 


New in version 2.6: Available only if you specify the pipeline as an array.










Parameter




pipeline




options






 






db.stuff.insert ( { _id:1 , "x":1, "y" : [1,2,3,4]});
db.stuff.insert ( { _id:2 , "x":2, "y" : [1,2,3,4]});
db.stuff.insert ( { _id:3 , "x":3, "y" : [1,2,3,4]});
db.stuff.insert ( { _id:4 , "x":4, "y" : [1,2,3,4]});
db.stuff.insert ( { _id:5 , "x":5, "y" : [1,2,3,4]});



> db.stuff.aggregate([ { $unwind : "$y" }]);
{ "_id" : 1, "x" : 1, "y" : 1 }
{ "_id" : 1, "x" : 1, "y" : 2 }
{ "_id" : 1, "x" : 1, "y" : 3 }
{ "_id" : 1, "x" : 1, "y" : 4 }
{ "_id" : 2, "x" : 2, "y" : 1 }
{ "_id" : 2, "x" : 2, "y" : 2 }
{ "_id" : 2, "x" : 2, "y" : 3 }
{ "_id" : 2, "x" : 2, "y" : 4 }
{ "_id" : 3, "x" : 3, "y" : 1 }
{ "_id" : 3, "x" : 3, "y" : 2 }
{ "_id" : 3, "x" : 3, "y" : 3 }
{ "_id" : 3, "x" : 3, "y" : 4 }
{ "_id" : 4, "x" : 4, "y" : 1 }
{ "_id" : 4, "x" : 4, "y" : 2 }
{ "_id" : 4, "x" : 4, "y" : 3 }
{ "_id" : 4, "x" : 4, "y" : 4 }
{ "_id" : 5, "x" : 5, "y" : 1 }
{ "_id" : 5, "x" : 5, "y" : 2 }
{ "_id" : 5, "x" : 5, "y" : 3 }
{ "_id" : 5, "x" : 5, "y" : 4 }


> db.stuff.aggregate([ { $unwind : "$y" }  , { $project : { x:1, y:1, _id:0}}] );
{ "x" : 1, "y" : 1 }
{ "x" : 1, "y" : 2 }
{ "x" : 1, "y" : 3 }
{ "x" : 1, "y" : 4 }
{ "x" : 2, "y" : 1 }
{ "x" : 2, "y" : 2 }
{ "x" : 2, "y" : 3 }
{ "x" : 2, "y" : 4 }
{ "x" : 3, "y" : 1 }
{ "x" : 3, "y" : 2 }
{ "x" : 3, "y" : 3 }
{ "x" : 3, "y" : 4 }
{ "x" : 4, "y" : 1 }
{ "x" : 4, "y" : 2 }
{ "x" : 4, "y" : 3 }
{ "x" : 4, "y" : 4 }
{ "x" : 5, "y" : 1 }
{ "x" : 5, "y" : 2 }
{ "x" : 5, "y" : 3 }
{ "x" : 5, "y" : 4 }
> 

> db.stuff.aggregate([ { $unwind : "$y" }  , { $project : { x:1, y:1, _id:0}}, {$out:"otherStuff"}]);
> 
> db.otherStuff.find();
{ "_id" : ObjectId("5551b7566f1f9528e8ba7209"), "x" : 1, "y" : 1 }
{ "_id" : ObjectId("5551b7566f1f9528e8ba720a"), "x" : 1, "y" : 2 }
{ "_id" : ObjectId("5551b7566f1f9528e8ba720b"), "x" : 1, "y" : 3 }
{ "_id" : ObjectId("5551b7566f1f9528e8ba720c"), "x" : 1, "y" : 4 }
{ "_id" : ObjectId("5551b7566f1f9528e8ba720d"), "x" : 2, "y" : 1 }
{ "_id" : ObjectId("5551b7566f1f9528e8ba720e"), "x" : 2, "y" : 2 }
{ "_id" : ObjectId("5551b7566f1f9528e8ba720f"), "x" : 2, "y" : 3 }
{ "_id" : ObjectId("5551b7566f1f9528e8ba7210"), "x" : 2, "y" : 4 }
{ "_id" : ObjectId("5551b7566f1f9528e8ba7211"), "x" : 3, "y" : 1 }
{ "_id" : ObjectId("5551b7566f1f9528e8ba7212"), "x" : 3, "y" : 2 }
{ "_id" : ObjectId("5551b7566f1f9528e8ba7213"), "x" : 3, "y" : 3 }
{ "_id" : ObjectId("5551b7566f1f9528e8ba7214"), "x" : 3, "y" : 4 }
{ "_id" : ObjectId("5551b7566f1f9528e8ba7215"), "x" : 4, "y" : 1 }
{ "_id" : ObjectId("5551b7566f1f9528e8ba7216"), "x" : 4, "y" : 2 }
{ "_id" : ObjectId("5551b7566f1f9528e8ba7217"), "x" : 4, "y" : 3 }
{ "_id" : ObjectId("5551b7566f1f9528e8ba7218"), "x" : 4, "y" : 4 }
{ "_id" : ObjectId("5551b7566f1f9528e8ba7219"), "x" : 5, "y" : 1 }
{ "_id" : ObjectId("5551b7566f1f9528e8ba721a"), "x" : 5, "y" : 2 }
{ "_id" : ObjectId("5551b7566f1f9528e8ba721b"), "x" : 5, "y" : 3 }
{ "_id" : ObjectId("5551b7566f1f9528e8ba721c"), "x" : 5, "y" : 4 }




Filed Under: MongoDB 
MongoDB索引与排序


2015/05/12  by  Leave a Comment 
对于下面的例子而言，mongoDB会如何处理这个查询？




 


ongos> db.users.insert({user: { login:"maclean", desc:"maclean liu" , date : new Date("2015-01-01") }});

mongos> 
mongos> db.users.findOne();
{
        "_id" : ObjectId("555179f2315a40007db97d3c"),
        "user" : {
                "login" : "maclean",
                "desc" : "maclean liu",
                "date" : ISODate("2015-01-01T00:00:00Z")
        }
}


mongos> db.users.ensureIndex( { "user.login" :1 , "user.date" :-1} , "myIndex" );
{
        "raw" : {
                "shard0.dbdao.com:35001" : {
                        "createdCollectionAutomatically" : false,
                        "numIndexesBefore" : 1,
                        "numIndexesAfter" : 2,
                        "ok" : 1
                }
        },
        "ok" : 1
}


mongos> db.users.find ( { "user.login": /^mac.*/ }  , {"user":1,"_id":0}).sort({"user.date":1}).explain();
{
        "queryPlanner" : {
                "mongosPlannerVersion" : 1,
                "winningPlan" : {
                        "stage" : "SINGLE_SHARD",
                        "shards" : [
                                {
                                        "shardName" : "shard0000",
                                        "connectionString" : "shard0.dbdao.com:35001",
                                        "serverInfo" : {
                                                "host" : "shard0.dbdao.com",
                                                "port" : 35001,
                                                "version" : "3.0.0",
                                                "gitVersion" : "a841fd6394365954886924a35076691b4d149168"
                                        },
                                        "plannerVersion" : 1,
                                        "namespace" : "test_db.users",
                                        "indexFilterSet" : false,
                                        "parsedQuery" : {
                                                "user.login" : /^mac.*/
                                        },
                                        "winningPlan" : {
                                                "stage" : "PROJECTION",
                                                "transformBy" : {
                                                        "user" : 1,
                                                        "_id" : 0
                                                },
                                                "inputStage" : {
                                                        "stage" : "SORT",
                                                        "sortPattern" : {
                                                                "user.date" : 1
                                                        },
                                                        "inputStage" : {
                                                                "stage" : "FETCH",
                                                                "inputStage" : {
                                                                        "stage" : "IXSCAN",
                                                                        "filter" : {
                                                                                "user.login" : /^mac.*/
                                                                        },
                                                                        "keyPattern" : {
                                                                                "user.login" : 1,
                                                                                "user.date" : -1
                                                                        },
                                                                        "indexName" : "myIndex",
                                                                        "isMultiKey" : false,
                                                                        "direction" : "forward",
                                                                        "indexBounds" : {
                                                                                "user.login" : [
                                                                                        "[\"mac\", \"mad\")",
                                                                                        "[/^mac.*/, /^mac.*/]"
                                                                                ],
                                                                                "user.date" : [
                                                                                        "[MaxKey, MinKey]"
                                                                                ]
                                                                        }
                                                                }
                                                        }
                                                }
                                        },
                                        "rejectedPlans" : [ ]
                                }
                        ]
                }
        },
        "ok" : 1
}














mongos> db.users.find ( { "user.login": /^mac.*/ }  , {"user":1,"_id":0}).sort({"user.login":1}).explain();
{
        "queryPlanner" : {
                "mongosPlannerVersion" : 1,
                "winningPlan" : {
                        "stage" : "SINGLE_SHARD",
                        "shards" : [
                                {
                                        "shardName" : "shard0000",
                                        "connectionString" : "shard0.dbdao.com:35001",
                                        "serverInfo" : {
                                                "host" : "shard0.dbdao.com",
                                                "port" : 35001,
                                                "version" : "3.0.0",
                                                "gitVersion" : "a841fd6394365954886924a35076691b4d149168"
                                        },
                                        "plannerVersion" : 1,
                                        "namespace" : "test_db.users",
                                        "indexFilterSet" : false,
                                        "parsedQuery" : {
                                                "user.login" : /^mac.*/
                                        },
                                        "winningPlan" : {
                                                "stage" : "PROJECTION",
                                                "transformBy" : {
                                                        "user" : 1,
                                                        "_id" : 0
                                                },
                                                "inputStage" : {
                                                        "stage" : "FETCH",
                                                        "inputStage" : {
                                                                "stage" : "IXSCAN",
                                                                "filter" : {
                                                                        "user.login" : /^mac.*/
                                                                },
                                                                "keyPattern" : {
                                                                        "user.login" : 1,
                                                                        "user.date" : -1
                                                                },
                                                                "indexName" : "myIndex",
                                                                "isMultiKey" : false,
                                                                "direction" : "forward",
                                                                "indexBounds" : {
                                                                        "user.login" : [
                                                                                "[\"mac\", \"mad\")",
                                                                                "[/^mac.*/, /^mac.*/]"
                                                                        ],
                                                                        "user.date" : [
                                                                                "[MaxKey, MinKey]"
                                                                        ]
                                                                }
                                                        }
                                                }
                                        },
                                        "rejectedPlans" : [ ]
                                }
                        ]
                }
        },
        "ok" : 1
}
mongos> db.users.find ( { "user.login": /^mac.*/ }  , {"user":1,"_id":0}).sort({"user.date":-1}).explain();
{
        "queryPlanner" : {
                "mongosPlannerVersion" : 1,
                "winningPlan" : {
                        "stage" : "SINGLE_SHARD",
                        "shards" : [
                                {
                                        "shardName" : "shard0000",
                                        "connectionString" : "shard0.dbdao.com:35001",
                                        "serverInfo" : {
                                                "host" : "shard0.dbdao.com",
                                                "port" : 35001,
                                                "version" : "3.0.0",
                                                "gitVersion" : "a841fd6394365954886924a35076691b4d149168"
                                        },
                                        "plannerVersion" : 1,
                                        "namespace" : "test_db.users",
                                        "indexFilterSet" : false,
                                        "parsedQuery" : {
                                                "user.login" : /^mac.*/
                                        },
                                        "winningPlan" : {
                                                "stage" : "PROJECTION",
                                                "transformBy" : {
                                                        "user" : 1,
                                                        "_id" : 0
                                                },
                                                "inputStage" : {
                                                        "stage" : "SORT",
                                                        "sortPattern" : {
                                                                "user.date" : -1
                                                        },
                                                        "inputStage" : {
                                                                "stage" : "FETCH",
                                                                "inputStage" : {
                                                                        "stage" : "IXSCAN",
                                                                        "filter" : {
                                                                                "user.login" : /^mac.*/
                                                                        },
                                                                        "keyPattern" : {
                                                                                "user.login" : 1,
                                                                                "user.date" : -1
                                                                        },
                                                                        "indexName" : "myIndex",
                                                                        "isMultiKey" : false,
                                                                        "direction" : "forward",
                                                                        "indexBounds" : {
                                                                                "user.login" : [
                                                                                        "[\"mac\", \"mad\")",
                                                                                        "[/^mac.*/, /^mac.*/]"
                                                                                ],
                                                                                "user.date" : [
                                                                                        "[MaxKey, MinKey]"
                                                                                ]
                                                                        }
                                                                }
                                                        }
                                                }
                                        },
                                        "rejectedPlans" : [ ]
                                }
                        ]
                }
        },
        "ok" : 1
}


> db.users.find ( { "user.login": /^mac.*/ }  , {"user":1,"_id":0}).sort({"user.login":-1}).explain();
{
	"queryPlanner" : {
		"plannerVersion" : 1,
		"namespace" : "test.users",
		"indexFilterSet" : false,
		"parsedQuery" : {
			"user.login" : /^mac.*/
		},
		"winningPlan" : {
			"stage" : "PROJECTION",
			"transformBy" : {
				"user" : 1,
				"_id" : 0
			},
			"inputStage" : {
				"stage" : "FETCH",
				"inputStage" : {
					"stage" : "IXSCAN",
					"filter" : {
						"user.login" : /^mac.*/
					},
					"keyPattern" : {
						"user.login" : 1,
						"user.date" : -1
					},
					"indexName" : "myIndex",
					"isMultiKey" : false,
					"direction" : "backward",
					"indexBounds" : {
						"user.login" : [
							"[/^mac.*/, /^mac.*/]",
							"(\"mad\", \"mac\"]"
						],
						"user.date" : [
							"[MinKey, MaxKey]"
						]
					}
				}
			}
		},
		"rejectedPlans" : [ ]
	},
	"serverInfo" : {
		"host" : "ac.local",
		"port" : 27017,
		"version" : "3.0.2",
		"gitVersion" : "nogitversion"
	},
	"ok" : 1
}







从上述演示中可以看到：


对于索引 myIndex db.users.ensureIndex( { “user.login” :1 , “user.date” :-1} , “myIndex” );


db.users.find ( { “user.login”: /^mac.*/ } , {“user”:1,”_id”:0}).sort({“user.date”:1}).explain(); ==> 使用了索引myIndex ，但 出现 “stage” : “SORT”, 说明使用索引未避免排序，


 


db.users.find ( { “user.login”: /^mac.*/ }  , {“user”:1,”_id”:0}).sort({“user.login”:1}).explain(); ==》使用了索引myIndex，且未出现 “stage” : “SORT” ; 说明对于index prefix的排序可以利用到索引来避免排序


db.users.find ( { “user.login”: /^mac.*/ }  , {“user”:1,”_id”:0}).sort({“user.login”:-1}).explain();使用了索引myIndex，且未出现 “stage” : “SORT” ; 说明对于index prefix的排序可以利用到索引来避免排序


db.users.find ( { “user.login”: /^mac.*/ }  , {“user”:1,”_id”:0}).sort({“user.date”:-1}).explain();  使用了索引myIndex ，但 出现 “stage” : “SORT”, 说明使用索引未避免排序，


 


以上说明了：


    

  • 对于索引prefix的字段而言，不管是索引是正序还是倒序，排序是正序需求还是倒序需求，都可以使用到Index索引来避免排序
    • 

  • 对于非索引prefix的字段，无法利用其来避免排序，IXSCAN完还需要SORT。
    • 



 


Filed Under: MongoDB 







        
沪ICP备14014813号-2

	
沪公网安备 31010802001379号