Setting up and connecting to a remote MongoDB database

Image courtesy https://mongodb.com

At Ithaka quite a few of our microservices use MongoDB 3.6 for data persistence. If you have used MongoDB you probably already know that starting from version 3.4 MongoDB’s WiredTiger Storage engine takes up a considerable amount of RAM. To be specific it will take either 50% of (RAM minus 1GB) OR 256 MB, whichever is higher. You can read more about how and why in their manual.

This, along with the common understanding that it is never a good idea to run your database service on the same server instance as your production services, makes it clear that you need to establish remote MongoDB server/s which your services then connect to.

High level steps:

1. Setup a new EC2 instance
2. Setup database users and assign them roles
3. Enable MongoDB auth to protect access to the database
4. Open up the network port to connect to MongoDB
5. Connect to the remote server

Please note that even though at Ithaka we use a Linux distro on top of EC2 instances, the points mentioned in this article are generic enough to work on most cloud+OS combinations. With that out of the way, let’s get to setting things up.

1. Setup a new EC2 instance

Before doing this take into account your database size and number of indexes in order to decide an optimum disk space and RAM for your instance. A simple calculation that works out for us is that for every 16 GB of data in the DB we allocate 1GB of RAM for WiredTiger because of the way our data is structured. Which means for 16 GB of stored data your server needs to have atleast 2GB RAM. Once your EC2 instance is setup, ssh into the instance and install MongoDB.

2. Setup database users and assign them roles

Like all other databases, MongoDB has a built in support for access control. You can create users and assign them different roles and different levels of access to databases. For the scope of this article let us stick to two important roles — Admin and DB user. Using the Admin role you can create db users and assign them roles on individual databases.

Open up mongo shell

ubuntu:~$ mongo

Inside mongo shell access the admin database. Create a new admin user.

> use admin;
> db.createUser({
      user: "admin",
      pwd: "myadminpassword",
      roles: [
                { role: "userAdminAnyDatabase", db: "admin" },
                { role: "readWriteAnyDatabase", db: "admin" },
                { role: "dbAdminAnyDatabase",   db: "admin" }
             ]
  });

You can assign multiple roles to a single user in one go. These roles grant the admin user all privileges required to create users, modify users, read and write to any database.

Now create db users and grant them roles on their respective databases. If your system consists of multiple databases, is a good idea to have a separate user for each database.

> db.createUser({
      user: "user1",
      pwd: "user1password",
      roles: [
                { role: "userAdmin", db: "sampledb" },
                { role: "dbAdmin",   db: "sampledb" },
                { role: "readWrite", db: "sampledb" }
             ]
  });

Note how the roles have changed here. If you try to assign one of the admin roles like “userAdminAnyDatabase” to a user on any non admin database, mongo will throw an error indicating that such a role is not defined on that database. Admin roles work only on the admin database.

3. Enable MongoDB Auth

Once your admin is setup and database specific users have been created, it is now time to enable MongoDB to start using these access controls. To do so open up mongodb config file:

sudo vim /etc/mongod.conf

In this file add the following lines:

security:
    authorization: 'enabled'

This will tell mongodb that whenever it starts up next, it needs to enforce database access control using the roles we created in the previous step.

By default mongodb is configured to allow connections only from localhost. We need to allow remote connections. In the same config file, go to the network interfaces section and change the bindIp from 127.0.0.1 to 0.0.0.0 which means allow connections from all ip addresses.

# network interfaces
net:
    port: 27017
    bindIp: 0.0.0.0   #default value is 127.0.0.1

Now save and exit the config file and restart mongodb server.

ubuntu:~$ sudo service mongod restart

Now if you try to access the mongo shell by simply typing mongo in the terminal, you’ll get through but won’t be able to access any databases. You need to use your created users to access the databases.

# to access the admin database
ubuntu:~$ mongo -u admin -p myadminpassword 127.0.0.1/admin
# to access the other databases
ubuntu:~$ mongo -u user1 -p user1password 127.0.0.1/sampledb

However we are still accessing the mongo shell from within the EC2 instance. We haven’t tried connecting remotely. To be able to do that, change the network settings of your EC2 instance.

4. Open up network port on the EC2 instance

MongoDB uses port number 27017 for all connections by default. So let’s open up that port. You can go to the network settings of your AWS console and open up inbound and outbound traffic on port 27017. Use custom tcp as the protocol. More about that over here. If that doesn’t work, ssh into your instance and run the following commands

ubuntu:~$ sudo iptables -A INPUT -p tcp --destination-port 27017 -m state --state NEW,ESTABLISHED -j ACCEPT
ubuntu:~$ sudo iptables -A OUTPUT  -p tcp --source-port 27017 -m state --state ESTABLISHED -j ACCEPT

5. Test the remote connections

Exit your EC2 instance and try connecting to the database server from your local machine

ubuntu:~$ mongo -u user1 -p user1password <your_server_ip>/sampledb

Note: Despite following these instructions, you might still run into issues when connecting to the remote database within your code. And those issues are mostly specific to the library that you use in your code. So far we have solved issues with pymongo for python, mgo for golang and mongoose for node.js. If you are using any of these, ping me on twitter @MithileshSaid and I’ll be glad to help.