Tag Archives: MongoDB

Sharded MongoDB config on Nutanix (1) : Deployment

So far I have posted on MongoDB deployments either as standalone or as part of a replica set. This is fine when you can size your VM memory to hold the entire database working set. However, if your VM’s RAM will not accommodate the working set in memory, you will need to shard to aggregate RAM from multiple replica sets and form a MongoDB cluster.

Having already discussed using clones of gold image VMs to create members for a replica set, then the most basic of MongoDB clusters requires at least two replica sets. On top of which we need a number of MongoDB “infrastructure” VMs that make MongoDB cluster operation possible. These entail a minimum of three (3) Configuration Databases (mongod –configsvr) per cluster and around one (1) Query Router (mongos) for every two shards. Here is the layout of a cluster deployment on my lab system:


In the above lab deployment, for availability considerations, I avoid co-locating any primary replica VM on the same physical host, and likewise any of the Query Router or ConfigDB VMs. One thing to bear in mind is that sharding is done on a per collection basis. Simply put, the idea behind sharding is that you split the collections across the replica sets and then by connecting to a mongos process you are routed to the appropriate shard holding the part of the collection that can serve your query. The following commands show the syntax to create one of the three required configdb’s (ran on three separate VMs, and need to be started first), and a Query Router, or mongos process (where we add the IP addresses of each configdb server VM) :

Config DB Servers – each ran as:
mongod --configsvr --dbpath /data/configdb --port 27019

Query Router - ran as:
mongos --configdb,,

- the above IP addresses in mongos command line are the addresses of each config DB.

This brings up an issue if you are not cloning replica VMs from “blank” gold VMs. By cloning a new replica set from a current working replica set, ie: so that you essentially have each replica set holding a full copy of all your databases and their collections. Then when you come to add such a replica set as a shard, you generate the error condition shown below.

Here’s the example of what can happen when you attempt to shard and your new replica set (rs02)  is simply cloned off a current running replica set (rs01):

mongos> sh.addShard("rs02/")
 "ok" : 0,
 "errmsg" : "can't add shard rs02/ because a local database 'ycsb' 
exists in another rs01:rs01/,,"

This is the successful workflow adding both shards (the primary of each replica set) via the mongos router VM:

$ mongo --host localhost --port 27017
MongoDB shell version: 3.0.3
connecting to: localhost:27017/test
mongos> sh.addShard("rs01/")
{ "shardAdded" : "rs01", "ok" : 1 }
mongos> sh.addShard("rs02/")
{ "shardAdded" : "rs02", "ok" : 1 }

We next need to enable sharding on the database and subsequently shard on the collection we want to distribute across the replica sets available. The choice of shard key is crucial to future MongoDB cluster performance. Issues such as read and write scaling, cardinality etc are covered here. For my test cluster I am using the _id field for demonstration purposes.

mongos> sh.enableSharding("ycsb")
{ "ok" : 1 }

mongos> sh.shardCollection("ycsb.usertable", { "_id": 1})
{ "collectionsharded" : "ycsb.usertable", "ok" : 1 }

The balancer process will run for the period of time needed to migrate data between the available shards. This can take anywhere from a number of hours to a number of days depending on the size of the collection, the number of shards, the current workload etc. Once complete however, this results in the following sharding status output. Notice  the “chunks” of the usertable collection held in the ycsb database are now shared across both shards (522 chunks in each shard) :

 mongos> sh.status()
--- Sharding Status ---
 sharding version: {
 "_id" : 1,
 "minCompatibleVersion" : 5,
 "currentVersion" : 6,
 "clusterId" : ObjectId("55f96e6c5dfc4a5c6490bea3")
 { "_id" : "rs01", "host" : "rs01/,," }
 { "_id" : "rs02", "host" : "rs02/,," }
 Currently enabled: yes
 Currently running: no
 Failed balancer rounds in last 5 attempts: 0
 Migration Results for the last 24 hours:
 No recent migrations
 { "_id" : "admin", "partitioned" : false, "primary" : "config" }
 { "_id" : "enron_mail", "partitioned" : false, "primary" : "rs01" }
 { "_id" : "mydocs", "partitioned" : false, "primary" : "rs01" }
 { "_id" : "sbtest", "partitioned" : false, "primary" : "rs01" }
 { "_id" : "ycsb", "partitioned" : true, "primary" : "rs01" }
 shard key: { "_id" : 1 }
 rs01 522
 rs02 522
 too many chunks to print, use verbose if you want to force print
 { "_id" : "test", "partitioned" : false, "primary" : "rs02" }

Additional Links:







Using Nutanix snapshots to backup MongoDB

In an earlier post I described how Nutanix clones can speed up deployment workflows for MongoDB replica sets. In this post we will cover the Nutanix VM-centric snapshot functionality to backup a MongoDB database instance. Even when the database is sharded, the ability to take a snapshot of a set of VMs at once (as part of a consistency group) ensures that a point in time (PIT) backup can easily be taken.

Let’s first consider the 3 member replica set as described in my last post. In order to take an OS level backup we would need to quiesce the I/O to one of the secondary members of the replica set. The command sequence below is for MMAPv1 storage engine only. The db.fsyncLock() flushes all pending writes and locks the instance for further writes until we run db.fsyncUnlock(). Even though we have journaling enabled, we have separated the I/O for data and journal files to different volumes (as part of our standard config). Hence the need to quiesce I/O to make sure we take a consistent snapshot.

rs01:SECONDARY> db.fsyncLock()
 "info" : "now locked against writes, use db.fsyncUnlock() to unlock",
 "seeAlso" : "http://dochub.mongodb.org/core/fsynccommand",
 "ok" : 1

Nutanix snapshots work at the VM level. This means we take a snapshot of all the vDisks in the VM hosting a MongoDB secondary at the same time. However, MongoDB can’t guarantee backup consistency if the journal and data are located in separate volumes. Hence, we still need db.fsyncLock()/db.fsyncUnlock().

<acropolis> vm.snapshot_create mongodb03
SnapshotCreate: complete

<acropolis> vm.snapshot_create mongodb03 snapshot_name_list=mongodb03-snap1
SnapshotCreate: complete

<acropolis> vm.snapshot_list mongodb03
Snapshot name Snapshot UUID
mongodb03-snap1 41c77ddc-8cc7-49bf-a250-23d52031b76e
mongodb03_2015-09-08T10:04:00.805780 7cb77f7f-1eb6-4103-8909-e8e50c318151

Above we have taken two snapshots (by way of example) and show the naming conventions returned for both a named snapshot and the default. See the rest of the workflow below…

<acropolis> vm.clone mongodb04 clone_from_snapshot=mongodb03-snap1
mongodb03-backup: complete

<acropolis> vm.restore mongodb03 mongodb03-snap1
VmRestore: complete

<acropolis> snapshot.delete mongodb03-snap1
Delete 1 snapshots? (yes/no) y
Please type 'yes' or 'no': yes
mongodb03-snap1: complete

We can carry out the same workflow via Nutanix Prism. I have taken a screenshot of the available commands (see below) in the VM Snapshots frame. The command sequence is issued via the  VM tab in the GUI. The desired CLI arguments such as clone and snapshot names for example, are entered in the resulting popups. For reasons of brevity I have not included all the required screenshots.


Having taken our snapshot, we can now unlock the replica – make sure you have kept the session where you called the db.fsyncLock() command open until this time.

rs01:SECONDARY> db.fsyncUnlock()
{ "ok" : 1, "info" : "unlock completed" }

Any newly created clones from the Nutanix VM snapshot can be used to seed a test environment with production strength data. Bearing in mind that the backup/clone was made from a VM that formed part of a replica set, we would like to be able to start the MongoDB instance in standalone mode. You can do this as follows ….

  • power off the newly created clone (mongodb04)
  • Login and create a sub-directory in the MongoDB data directory in order to save the local db files to…
[root@mongodb04 data]# mkdir local-old
[root@mongodb04 data]# mv local.* local-old/
  • edit /etc/mongod.conf : comment out the line naming the replica set and make sure the bind_ip reflects the IP address assigned to the VM (via DHCP in my case)
[mongod@mongodb04 ~]$ ip a | grep eth0
2: eth0: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000
 inet brd scope global eth0

[mongod@mongodb04 ~]$ egrep "bind_ip|replSet" /etc/mongod.conf
  • Start up the mongod (now as a standalone) instance with the newly edited configuration file
  • Drop into a mongo shell. You can then verify the consistency of the copied data (in my example I am using a synthetic database created by the YCSB benchmark software) …
[mongod@mongodb04 ~]$ mongo
MongoDB shell version: 3.0.3
connecting to: test
> show dbs
enron_mail 3.952GB
local 0.078GB
ycsb 207.853GB
> use ycsb
switched to db ycsb
> show collections
> db.usertable.findOne()
 "_id" : "user6284781860667377211",
 "field5" : BinData(0,"PTo4JCo4JiskPDU2PiIzPikmMycrNSQjJzo3NiEqJjImIyY0PSE5KS43ICM4Liw+MCclLyE1JCA3PDM3OCsoMz0nKD8rISYrKCw8Pik4JT0jIyYuKDkoOy4gJCkoPSMkOywnNA=="),
 "field4" : BinData(0,"PyI/NSIqITEhKzQxPTwlNyIzJy0zLz05IzUvLDkwLi0uNjQwKysnMSQ5Lj8hMDozPDo+JTg/IDw6NTI9PzAkNjIjLTw8OyYvOTsgLTM2IjcvPCk4Ij84Ny45LiYhJC0mKT0qKw=="),
 "field3" : BinData(0,"MTkwLj0sPTYtPTg7OTUmIz0xJSM3OzM+Kyw3PyckPDYnMyotMC84JDIjMzEnITktJi4mMz0rLTkzODkoPCUkMT4pJzEpKiM8JDUlKjUxLiY5PzIxIiM0OCgpLTUpLyQ1JCogLw=="),
 "field2" : BinData(0,"Pz4jNTcnLDgwKS04ISc7Oi01IDkrNDclPTsuMyElOSU/KywvMzY6PDU0Jyo3NyYjKig8NzMrOTs7LiQiMy0hKioiPiMkPTgyLDAyPSoxJS4tOiUyLzE9Pz8rJy0zPjo2PCszNA=="),
 "field9" : BinData(0,"KTknMSs2PyEiMjUiPz4qKiM9MS0iIj4vMyMyNCsjPSUkJyYsJz0nLD0/NSY3MyQ7KCo/OikiNyU3MjQ8MCArLj8wMyM0MiAgJiUzOCc5KjYiJTw8Izw+OzA7NCAiPzA5ISYxOg=="),
 "field8" : BinData(0,"KCgoLTQ4ICE0IDY0ODAyIS86LiYrKzQmIyIpJy0nKz0gICYpMTQsLT86PyAiKjs/Kzw3MSciJDYkJTssLiMsLiUtLzMtOiYpND8nMCQpOzopMyQ1OyYsNy8uPysxNyg1PCQ6MA=="),
 "field7" : BinData(0,"ITM1KiUmKCcnKjA8IT8xPCIjKzc4Ijs+IyYhICMlKCsjPy09MygxKTohPDgjLjIzMzwpPj45Oyg2JTgsLjYlJS0xPCc+IiolKDw0JiQkLyg4NiohID0xPjIyMic9KDgqKD0zKg=="),
 "field6" : BinData(0,"IjovMTotODUxPD0mPikqLyo4IzonKjc+LDwuLCEhICghOzYxJTw5PiAzIzktNjA9OzUqNiovMTszMDMqLDQmNjk8IDA0NjAvPz8mPygnITU+OSY8JCIzLDAgOSEmNyA1KCQ+Jw=="),
 "field1" : BinData(0,"ID0sMjg8JiAiKz4gITYjNCkxLSs+IC8/JTAiNCA8PignMTEjNyogKyU0Lz4lNz0mIzohNCQ8LSwhICw/OT82PCYyPjk2KSElIDo5MSU2Kig9NiUnOjwpLi0vOjIqJz0sKDExOw=="),
 "field0" : BinData(0,"OikgJykuOTchIjUlISghLCkwOis/Pyw4ISo7Jyw6Kz46NTwlODIsKDQqLT0uNi0rOTcsOzA4Lzo3PDcyMjozLiItKSY1JSsoLSIkPio7KC4xOzs7MzY7Oj0nOzU+IyEuJDMwJA==")

The VM can now be easily migrated anywhere across the Nutanix cluster using the features of the Nutanix Distributed filesystem (NDFS). If additional availability is required then we can clone “blank” gold image VMs and form a new replica set. To avoid confusion the naming convention of the new replica set(s) can reflect the intended use case perhaps : dev, test, QA etc.

Using Nutanix clones to deploy MongoDB replica set

In this post I am going to look at setting up a replica set to support high availability in a MongoDB environment. Replica sets contain a primary MongoDB database and a number of additional secondary replica databases. Any one of the allowed replicas can become primary in the event that the original primary fails for whatever reason. Replica set membership count is usually an odd number in order that new primary elections are not tied.

Building out an HA MongoDB setup on Nutanix is relatively easy to do. Each MongoDB instance is hosted in a separate, sandboxed environment. In our case a virtual machine (VM). Each VM is then located on a separate physical hypervisor host. I have a gold image VM that has a MongoDB instance installed along recommended best practice guidelines. This VM gets cloned as required when I need to build out a new MongoDB environment. So for a 3 member replica set I need 3 clones.


From a cluster CVM node type:

$ acli 

<acropolis> vm.clone mongodb01,mongodb02,mongodb03 clone_from_vm=mongodb30-gold
mongodb01: complete
mongodb02: complete
mongodb03: complete
<acropolis> vm.list
mongodb01: 2b9498c1-502e-454e-93c8-931a45a321b6
mongodb02: 9a445d26-caf9-4ddf-9d8e-296ea8b6e19e
mongodb03: 9a5512fa-3d19-4ddc-8cac-11721f999459

<acropolis> vm.on mongodb01,mongdb02,mongodb03
mongodb01: complete
mongodb01: complete
mongodb01: complete

After powering on the VMs, check that mongod starts correctly on default port 27017 on each VM. First thing to make sure is that the mongod process is listening on the correct address. I have set my VMs to use DHCP and this is the address that the service needs to listen on.

# ip a

2: eth0: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000
 link/ether 52:54:00:db:17:76 brd ff:ff:ff:ff:ff:ff
 inet brd scope global eth0

# cat /etc/mongod.conf | grep -i bind_ip

# service mongod restart
# service mongod status

Once all of the VMs are up and running on their respective address:port tuples, make sure that we enable firewall access via iptables. Each VM, that will form part of the replica set, needs to allow access to the other members via mongod port 27017. So for a replica set with members,,, then for each member, in this example, run…

# iptables -A INPUT -s -p tcp --destination-port 27017 -m state \
# iptables -A INPUT -s -p tcp --destination-port 27017 -m state \

# service iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables:[ OK ]
# service iptables reload

abridged iptables -L output after the above changes….

Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT tcp -- anywhere tcp dpt:27017 state NEW,ESTABLISHED
ACCEPT tcp -- anywhere tcp dpt:27017 state NEW,ESTABLISHED

Check access by performing a series of bi-directional tests between all the replica set members:

<>$ mongo --host --port 27017
MongoDB shell version: 3.0.3
connecting to:
> quit()

Should any of the connection tests fail then revisit the iptables entries. Usual troubleshooting applies with telnet or nc, netstat etc.

In order to create the replica set, connect via ssh to each VM and edit the mongod.conf to include the replSet functionality:

$ grep -i replSet /etc/mongod.conf

Restart the mongod process (sudo service mongod restart) and then start a mongo shell session, the first member of the set (primary) needs to run :

$ mongo
MongoDB shell version: 3.0.3
connecting to: test
> rs.initiate()
 "info2" : "no configuration explicitly specified -- making one",
 "me" : "",
 "ok" : 1

You can use the shell commands rs.conf() and rs.status() to check the replica set at any point. We’ll look at one of these outputs after completing the replica set creation. Next, from the same mongo shell session, add the other two replica nodes:

rs01:PRIMARY> rs.add("")
{ "ok" : 1 }

rs01:PRIMARY> rs.add("")
{ "ok" : 1 }

Potential error scenarios

  •  if you didn’t clone the VMs for the replica set from a blank gold image but rather from a VM already running a replicated mongodb configuration. Then the replication commands report errors similar to this :
 "info2" : "no configuration explicitly specified -- making one",
 "me" : "",
 "info" : "try querying local.system.replset to see current configuration",
 "ok" : 0,
 "errmsg" : "already initialized",
 "code" : 23

On the proviso that this is a greenfield install, delete the local db config files in the data directory and re-run the rs.initiate()

  • if the firewall rules are not set correctly then the following error message is thrown:
 "errmsg" : "Quorum check failed because not enough voting nodes responded; 
required 2 but only the following 1 voting nodes responded:; 
the following nodes did not respond affirmatively: failed with Failed attempt to connect to; 
couldn't connect to server (, 
connection attempt failed",

Ensure that the firewall rules allow proper access between the VM’s.

  • if replication is not enabled correctly in the mongod configuration files on each host of the replica set :
"errmsg" : "Quorum check failed because not enough voting nodes responded; 
required 2 but only the following 1 voting nodes responded:; 
the following nodes did not respond affirmatively: failed with not running with --replSet",

Once the replica set configuration is complete, check the setup by running rs.status() or rs.conf() to confirm :

rs01:PRIMARY> rs.conf()
 "_id" : "rs01",
 "version" : 3,
 "members" : [
 "_id" : 0,
 "host" : "",
 "arbiterOnly" : false,
 "buildIndexes" : true,
 "hidden" : false,
 "priority" : 1,
 "tags" : {

 "slaveDelay" : 0,
 "votes" : 1
 "_id" : 1,
 "host" : "",
 "arbiterOnly" : false,
 "buildIndexes" : true,
 "hidden" : false,
 "priority" : 1,
 "tags" : {

 "slaveDelay" : 0,
 "votes" : 1
 "_id" : 2,
 "host" : "",
 "arbiterOnly" : false,
 "buildIndexes" : true,
 "hidden" : false,
 "priority" : 1,
 "tags" : {

 "slaveDelay" : 0,
 "votes" : 1
 "settings" : {
 "chainingAllowed" : true,
 "heartbeatTimeoutSecs" : 10,
 "getLastErrorModes" : {

 "getLastErrorDefaults" : {
 "w" : 1,
 "wtimeout" : 0

From the output above we can see the full replica set membership, both the member function and status. Things like priority settings and whether or not the replica is hidden to user applications queries etc. Also, whether a replica is a full mongod instance or an arbiter (simply there to mitigate against primary election ties). Or, if any of the replicas have a delay enabled (used for backup/reporting duties).

In an earlier post I have shown the available mongo shell commands to calculate the working set for the database. For read intensive workloads, where your working set is sized to fit available RAM in the mongod server VMs; a replica set deployment can be used to run MongoDB and support high availability.

Getting started with MongoDB shell and pymongo

In my last blog article I described how to setup a MongoDB instance in a VM. In order to use that VM and run various diagnostic commands, we are going to need a some data to play with. The easiest way to get data is to use a data science archive . I was able to find the Enron Mail Corpus in mongodump format (credit for this must go to Bryan Nehl). This then becomes trivially easy to import the ~500,000 emails in the corpus in MongoDB document format. See below.

We uncompress and extract the downloaded tarfile to get the dump directory structure…

drwxr-xr-x. 3 mongod mongod 4096 Jan 18 2012 dump
-rw-r--r--. 1 mongod mongod 1459855360 Feb 2 2012 enron_mongo.tar

and using the mongorestore utility to load the database – we don’t specify additional cli options as mongorestore will look for the dump directory structure in the current directory by default:

$ mongorestore
2015-07-28T14:03:20.079+0100 using default 'dump' directory
2015-07-28T14:03:20.108+0100 building a list of dbs and collections to restore fr om dump dir
2015-07-28T14:03:20.131+0100 no metadata file; reading indexes from dump/enron_ma 
2015-07-28T14:03:20.140+0100 restoring enron_mail.messages from file dump/enron_m 
2015-07-28T14:03:23.124+0100 [##......................] enron_mail.messages 142.0 MB/1.4 GB (10.2%)
2015-07-28T14:03:26.124+0100 [#####...................] enron_mail.messages 337.5 MB/1.4 GB (24.2%)
2015-07-28T14:03:29.124+0100 [########................] enron_mail.messages 499.2 MB/1.4 GB (35.9%)
2015-07-28T14:03:32.124+0100 [###########.............] enron_mail.messages 645.9 MB/1.4 GB (46.4%)
2015-07-28T14:03:35.124+0100 [##############..........] enron_mail.messages 828.4 MB/1.4 GB (59.5%)
2015-07-28T14:03:38.124+0100 [#################.......] enron_mail.messages 1003.7 MB/1.4 GB 
2015-07-28T14:03:41.124+0100 [####################....] enron_mail.messages 1.1 GB/1.4 GB (83.5%)
2015-07-28T14:03:44.124+0100 [######################..] enron_mail.messages 1.3 GB/1.4 GB 
2015-07-28T14:03:45.326+0100 restoring indexes for collection enron_mail.messages from metadata
2015-07-28T14:03:45.372+0100 finished restoring enron_mail.messages
2015-07-28T14:03:45.372+0100 done

We can now see the database in a local mongo shell session :

> show dbs
enron_mail 1.435GB
local 0.000GB

To remove the database for any reason. For example, say you need to run subsequent benchmarks that reload a test database. Then run the following command to drop the current database prior to reloading it afresh.

from the mongo shell using sbtest database as an example ….

> use sbtest
switched to db sbtest
> db.runCommand( { dropDatabase: 1 } )
{ "dropped" : "sbtest", "ok" : 1 }

Configuration and sizing

The following commands can be used to size a database working set. This is useful in terms platform design and capacity planning. The db.serverStatus() command gives a great deal of information about the running instance. We will only concern ourselves with the memory component at this point. Note that it is imperative for good performance that the working set and associated indexes are always held in RAM. So, for pre 3.0 versions of MongoDB then :

"pagesInMemory" : 91521

Multiply working set pages by PAGESIZE to get size in bytes

# getconf PAGESIZE

The db.stats() command provides the size of the indexes in use


So for our example this can be calculated as follows :

(915211 * 4096) + 7131826688 ~ 6GB

As of MongoDB 3.0 the working set section is no longer available – the document now returns:

> db.serverStatus().mem
 "bits" : 64,
 "resident" : 20466,
 "virtual" : 148248,
 "supported" : true,
 "mapped" : 73725,
 "mappedWithJournal" : 147450

The above sizes (in bold) are in megabytes (MB), and correspond respectively to the virtual memory of the mongod process, the amount of mapped memory and the amount of mapped memory including the memory used for journaling. These numbers can be used in order to allocate sufficient RAM to your guest VM database host.

The following db.hostInfo() command reveals among other things, the instruction set supported by the VM, the various operating system limit settings and whether NUMA is disabled:

> db.hostInfo()
 "system" : {
 "currentTime" : ISODate("2015-07-28T13:45:36.093Z"),
 "hostname" : "mongowt01",
 "cpuAddrSize" : 64,
 "memSizeMB" : 64427,
 "numCores" : 8,
 "cpuArch" : "x86_64",
 "numaEnabled" : false
 "os" : {
 "type" : "Linux",
 "name" : "CentOS release 6.6 (Final)",
 "version" : "Kernel 2.6.32-504.el6.x86_64"
 "extra" : {
 "versionString" : "Linux version 2.6.32-504.el6.x86_64 
 (mockbuild@c6b9.bsys.dev.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-11) (GCC) ) #1 SMP Wed Oct 15 04:27:16 UTC 2014",
 "libcVersion" : "2.12",
 "kernelVersion" : "2.6.32-504.el6.x86_64",
 "cpuFrequencyMHz" : "2799.998",
 "cpuFeatures" : "fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb lm constant_tsc rep_good unfair_spinlock pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm xsaveopt fsgsbase smep erms",
 "pageSize" : NumberLong(4096),
 "numPages" : 16493566,
 "maxOpenFiles" : 65536
 "ok" : 1


In order to take a crash consistent backup then the following command sequence is required before and after the backup :

> db.fsyncLock()
 "info" : "now locked against writes, use db.fsyncUnlock() to unlock",
 "seeAlso" : "http://dochub.mongodb.org/core/fsynccommand",
 "ok" : 1

perform host level OS backup or better still, take VM-centric snapshot and then …

> db.fsyncUnlock()
{ "ok" : 1, "info" : "unlock completed" }

Delving into the database structure, show collections will list the document collections within a database (in this case, the previously loaded enron_mail db) and you can use that information to inspect individual documents:

> show collections

These next commands can be used to retrieve a document or set of documents. The document below has been edited to retain the privacy of the original sender.

> db.messages.findOne()
"_id" : ObjectId("4f16fc97d1e2d32371003e27"),
"body" : "the scrimmage is still up in the air...\n\n\nwebb said that they didnt want to scrimmage...\n\nthe aggies are scrimmaging each other... (the aggie teams practiced on \nSunday)\n\nwhen I called the aggie captains to see if we could use their field.... they \nsaid that it was tooo smalll for us to use...\n\n\nsounds like bullsh*t to me... but what can we do....\n\n\nanyway... we will have to do another practice Wed. night.... and I dont' \nknow where we can practice.... any suggestions...\n\n\nalso, we still need one more person...",
"subFolder" : "notes_inbox",
db.messages.findOne({_id: "4f16fc97d1e2d32371003e27" })

Just for the record – the database can be manually shutdown using:

>use admin

Performance issues

db.currentOp() is one of the commands available from the database profiler that allows admins to locate any queries or write operations that are running slow.

> db.currentOp()
 "inprog" : [
 "desc" : "conn374",
 "threadId" : "0x16574c340
 "connectionId" : 374,
 "opid" : 1032339378,
 "active" : true,
 "secs_running" : 0,
 "microsecs_running" : NumberLong(105738),
 "op" : "insert",
 "ns" : "sbtest.sbtest6",
 "insert" : {

A badly behaving database operation can be killed using :

> db.killOp(1032339378)
{ "info" : "attempting to kill op" }

In order to see the five most recent operations that took 100 milliseconds (the default) or more, you can enable profiling – see below (output shortened)

setProfilingLevel() arguments are 0 for no profiling, 1 for only slow operations, or 2 for all operations. You can add a second argument to change the threshold for what is considered a slow db operation, for example this can be reduced to 10 ms.

> db.setProfilingLevel(2)
{ "was" : 0, "slowms" : 100, "ok" : 1 }

> db.system.profile.find()
"op" : "insert", 
"ns" : "sbtest.sbtest6", 
"query" : { 
 "_id" : 2628714, 
 "k" : 4804469, 
 "c" : "42025084972-52016328459-02616906732-06037924356-25803606931-90180435635-33434735556-64942463775-51942983544-69579223058", 
 "pad" : "83483501744-16275794559-91512432879-42096600452- 97899816846" 
 "ninserted" : 1, 
 "keyUpdates" : 0, 
 "writeConflicts" : 0,
 "numYield" : 0,
 "locks" : {
 "Global" : {
 "acquireCount" : { 
 "w" : NumberLong(720) 
 "Database" : { 
 "acquireCount" : { 
 "w" : NumberLong(720)
 "Collection" : {
 "acquireCount" : {
 "w" : NumberLong(720)
"millis" : 0,
 "execStats" : { },
 "ts" : ISODate("2015-07-28T16:37:01.959Z"),
 "client" : "",
 "allUsers" : [ ],
 "user" : "" }

So far we have simply been working on a previously created database. If we wanted to generate a workload, we would need to use a well known synthetic workload generator such as sysbench or YCSB (more on these in a future post). One other alternative, is using the pymongo device driver to connect to a MongoDB instance. Then use standard Python idioms to call the MongoDB API. To install the pymongo driver, either install the pre-packaged version from the EPEL repo (for RHEL based Linux) or download the git repo and build the driver manually.

sudo yum -y install epel-release
sudo yum -y install python-pip
sudo pip install pymongo


git clone git://github.com/mongodb/mongo-python-driver.git
cd mongod-python-driver
python setup.py install

The following python interpreter session shows the basics of connecting to a MongoDB instance and how to load documents into a collection. This could be extended to do various read and write based workloads depending on what you are looking to test or characterise.

create a database client connection :

>>> from pymongo import MongoClient
>>> uri = 'mongodb://'
>>> conn = MongoClient(uri)

create a document collection object:

>>> collection = conn.mydocs.docs

Inserting documents:

>>> doc1 = {'author': 'Ray Hassan', 'title': 'My first doc'}
 >>> conn.mydocs.docs.insert_one(doc1)
<pymongo.results.InsertOneResult object at 0x11b9780>
>>> doc2 = {'author': 'Ray Hassan', 'title': 'My 2nd doc'}
>>> conn.mydocs.docs.insert_one(doc2)
<pymongo.results.InsertOneResult object at 0x11b90f0>

retrieving documents via a python list:

>>> cursor = collection.find()
>>> for doc in cursor: print doc
{u'_id': ObjectId('55b8ec5bd7cf7a74c8bdd3bf'), u'author': u'Ray Hassan', u'title': u'My first doc'}
{u'_id': ObjectId('55b8ec69d7cf7a74c8bdd3c0'), u'author': u'Ray Hassan', u'title': u'My 2nd doc'}

If we wanted to improve the performance of a particular query we can use the explain() command. First lets take a look at the explain() output from a query that uses a document without an index

>>> collection.find({'author': 'Ray Hassan'}).explain()
{u'executionStats': {u'executionTimeMillis': 0, u'nReturned': 4, u'totalKeysExamined': 0, u'allPlansExecution': [], u'executionSuccess': True, u'executionStages': {u'docsExamined': 4, u'restoreState': 0, u'direction': u'forward',u'saveState': 0, u'isEOF': 1, u'needFetch': 0, u'nReturned': 4, u'needTime': 1, u'filter': {u'author': {u'$eq': u'Ray 
Hassan'}}, u'executionTimeMillisEstimate': 0, u'invalidates': 0, u'works': 6, u'advanced': 4, u'stage': u'COLLSCAN'}, u'totalDocsExamined': 4}, u'queryPlanner': {u'parsedQuery': {u'author': {u'$eq': u'Ray Hassan'}}, u'rejectedPlans': [], u'namespace': u'mydocs.docs', u'winningPlan': {u'filter': {u'author': {u'$eq': u'Ray Hassan'}}, u'direction': u'forward', u'stage': u'COLLSCAN'}, u'indexFilterSet': False, u'plannerVersion': 1}, u'serverInfo': {u'host': u'mongowt01', u'version': u'3.0.3', u'port': 27017, u'gitVersion': u'b40106b36eecd1b4407eb1ad1af6bc60593c6105 modules: enterprise'}

Without an index the query above has to perform a full scan of the collection (COLLSCAN) and we get 4 documents returned (nReturned). In order to improve the performance of the query we could consider adding an index to one of the document fields.

>>>from pymongo import ASCENDING, DESCENDING
>>> collection.create_index([('author', ASCENDING)])

>>> collection.find({'author': 'Ray Hassan'}).explain()
{u'executionStats': {u'executionTimeMillis': 0, u'nReturned': 4, u'totalKeysExamined': 4, u'allPlansExecution': [], u'executionSuccess': True, u'executionStages': {u'restoreState': 0, u'docsExamined': 4, u'saveState': 0, u'isEOF': 1, u'inputStage': {u'matchTested': 0, u'restoreState': 0, u'direction': u'forward', u'saveState': 0, u'indexName': 
u'author_1', u'dupsTested': 0, u'isEOF': 1, u'needFetch': 0, u'nReturned': 4, u'needTime': 0, u'seenInvalidated': 0, u'dupsDropped': 0, u'keysExamined': 4, u'indexBounds': {u'author': [u'["Ray Hassan", "Ray Hassan"]']}, u'executionTimeMillisEstimate': 0, u'isMultiKey': False, u'keyPattern': {u'author': 1}, u'invalidates': 0, u'works': 4, u'advanced': 4, u'stage': u'IXSCAN'}, u'needFetch': 0, u'nReturned': 4, u'needTime': 0, 
u'executionTimeMillisEstimate': 0, u'alreadyHasObj': 0, u'invalidates': 0, u'works': 5, u'advanced': 4, u'stage': u'FETCH'}, u'totalDocsExamined': 4}, u'queryPlanner': {u'parsedQuery': {u'author': {u'$eq': u'Ray Hassan'}}, u'rejectedPlans': [], u'namespace': u'mydocs.docs', u'winningPlan': {u'inputStage': {u'direction': u'forward', u'indexName': u'author_1', u'indexBounds': {u'author': [u'["Ray Hassan", "Ray Hassan"]']}, u'isMultiKey': False, u'stage': u'IXSCAN', u'keyPattern': {u'author': 1}}, u'stage': u'FETCH'}, u'indexFilterSet': False, u'plannerVersion': 1}, 
u'serverInfo': {u'host': u'mongowt01', u'version': u'3.0.3', u'port': 27017, u'gitVersion': u'b40106b36eecd1b4407eb1ad1af6bc60593c6105 modules: enterprise'}}

In the above output we can see since adding an index that we now perform an Index scan (IXSCAN) – if appropriately chosen, this can reduce the number of documents returned in a query. In our case (a very trivial example) this has not been the case. Ordinarily for a larger (or perhaps better?) example this would tend to be more performant.

The above merely touches on what can be done based on NoSQL workload testing requirements. I do hope however, that  you find it a good place start.


Installing MongoDB on Nutanix XCP

As part of the recent MongoDB certification of Nutanix XCP as an Infrastructure as a Service  (IaaS) platform,  I thought I might collate some of the info I have collected while working to get the certification process completed. There’s a lot of great docs over at www.mongodb.com but I want to condense everything into a series of posts. This first post will deal with the initial install of a standalone MongoDB instance.

We saw in my previous post here how to create a Linux VM and add networking and vDisks. In this instance I have added 6 x 200GB vDisks for a data volume, and an additional 2 vDisks – one for the journal volume (50GB) and one volume to hold the log file (100GB). Here’s the output from /usr/bin/lsscsi showing the disks and their SCSI assignments :

[2:0:1:0] disk NUTANIX VDISK 0 /dev/sdj
[2:0:2:0] disk NUTANIX VDISK 0 /dev/sdk
[2:0:7:0] disk NUTANIX VDISK 0 /dev/sdb
[2:0:8:0] disk NUTANIX VDISK 0 /dev/sdc
[2:0:9:0] disk NUTANIX VDISK 0 /dev/sdd
[2:0:10:0] disk NUTANIX VDISK 0 /dev/sde
[2:0:11:0] disk NUTANIX VDISK 0 /dev/sdf
[2:0:12:0] disk NUTANIX VDISK 0 /dev/sdg
[2:0:13:0] disk NUTANIX VDISK 0 /dev/sdh
[2:0:14:0] disk NUTANIX VDISK 0 /dev/sdi

Create a user/group mongod that will own the MongoDB software :

# groupadd mongod 
# useradd mongod

To install the MongoDB Enterprise packages, create a new repo with the required information and then install as MongoDB user using yum :

# pwd
# cat mongodb-enterprise.repo
name=MongoDB Enterprise Repository
$ sudo yum install -y mongodb-enterprise

We use LVM to create a 6 column striped data volume. All Nutanix vDIsks are redundant (RF=2) so to create a RAID10 data volume just stripe the vDisks, and then create 2 further linear volumes. First create the underlying physical volumes :

# lsscsi | awk '{print $6}' | grep /dev/sd | grep -v sda | xargs pvcreate
 Physical volume "/dev/sdb" successfully created
 Physical volume "/dev/sdc" successfully created
 Physical volume "/dev/sdd" successfully created
 Physical volume "/dev/sde" successfully created
 Physical volume "/dev/sdf" successfully created
 Physical volume "/dev/sdg" successfully created
 Physical volume "/dev/sdh" successfully created
 Physical volume "/dev/sdi" successfully created
 Physical volume "/dev/sdj" successfully created

Then create both the volume groups and the required volumes

vgcreate mongodata /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg 
vgcreate mongojournal /dev/sdh 
vgcreate mongolog /dev/sdi
# lvcreate -i 6 -l 100%VG -n mongodata mongodata
# lvcreate -l 100%VG -n mongojournal mongojournal 
# lvcreate -l 100%VG -n mongolog mongolog

Create an XFS filesystem on each volume:

mkfs.xfs /dev/mapper/mongodata-mongodata
mkfs.xfs /dev/mapper/mongojournal-mongojournal
mkfs.xfs /dev/mapper/mongolog-mongolog

Create the required mountpoints:

mkdir -p /mongodb/data mongodb/journal /mongodb/log

Mount the filesystems – setting noatime option on the data volume

/dev/mapper/mongodata-mongodata /mongodb/data xfs defaults,auto,noatime,noexec 0 0
/dev/mapper/mongojournal-mongojournal /mongodb/journal xfs defaults,auto,noexec 0 0
/dev/mapper/mongolog-mongolog /mongodb/log xfs defaults,auto,noexec 0 0

Set up a  soft link to re-direct the journal I/O to a separate volume:

# ln -s /mongodb/journal /mongodb/data/journal
lrwxrwxrwx. 1 root root 21 Nov 21 14:13 journal -> /mongodb/journal

At this point set the filesystem ownership to the MongoDB user:

# chown -R mongod:mongod /mongodb/data mongodb/journal mongodb/log

Prior to starting MongoDB there are a few well known best practices that need to be adhered to. Firstly, we reduce the read ahead on the data volume in order to avoid filling RAM with unwanted pages of data. MongoDB documents are quite small and a large readahead figure will fill RAM with additional pages of data that will have to then be evicted to make room for other required pages. Filling virtual memory with this superfluous data can have an adverse effect on performance. Usual recommendation is to start with a setting of 16K (32 * 512M sectors) and then adjust upwards from there.

rwxrwxrwx. 1 root root 7 Feb 4 11:50 /dev/mapper/mongodata-mongodata -> ../dm-3 

# blockdev --setra 32 /dev/dm-3
# blockdev --getra /dev/dm-3

MongoDB recommends that you disable transparent huge pages, edit your startup files as follows :

 #disable THP at boot time
 if test -f /sys/kernel/mm/redhat_transparent_hugepage/enabled; then
 echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
 if test -f /sys/kernel/mm/redhat_transparent_hugepage/defrag; then
 echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag

Set swappiness = 1: MongoDB is a memory-based database; if the nodes are sized correctly, then we won’t need to swap. However, setting swappiness=0 could cause unexpected invocations of the OOM (Out of Memory) killer in certain Linux distros.

$ sudo sysctl vm.swappiness=1 (for current runtime)
$ sudo echo 'vm.swappiness=1' >> /etc/sysctl.conf (make permanent)

Disable NUMA, either in VM BIOS or, invoke mongod with NUMA disabled. All supported versions of MongoDB ship with an init script that automates this as follows:

numactl –interleave=all /usr/bin/mongod –f /etc/mongod.conf

Also ensure:

$ sudo cat /proc/sys/vm/zone_reclaim_mode

Finally, once you have configured the /etc/mongod.conf file (as root), you can start the mongod service –  see output from grep -v ^# /etc/mongod.conf below. Note, I have added the address for the primary NIC interface to the bind_ip in addition to the local loopback.

sudo service mongod start

Once the database has started then you can connect via the mongo shell and verify the database is up and running :

$ mongo
MongoDB shell version: 3.0.3
connecting to: test

Now that we have our mongodb instance installed, we can use it as a template to clone additional MongoDB hosts on demand. I will cover this in future posts when I create replica sets and shards etc. For now, we need to get some data loaded and perform a few CRUD operations and perform some additional testing. I’ll cover this in my next post.