Install & run MongoDB in Amazon EC2 and run shell commands on EC2 from a lambda function

What is Amazon EC2 Instance?

• An EC2 instance is nothing but a virtual server in Amazon Web services terminology.
• It stands for Elastic Compute Cloud.
• It is a web service where an AWS subscriber can request and provision a compute server in AWS cloud.
• An on-demand EC2 instance is an offering from AWS where the subscriber/user can rent the virtual server per hour and use it to deploy his/her own applications.
• The instance will be charged per hour with different rates based on the type of the instance chosen. AWS provides multiple instance types for the respective business needs of the user.
• Thus, you can rent an instance based on your own CPU and memory requirements and use it as long as you want.
• You can terminate the instance when it’s no more used and save on costs. This is the most striking advantage of an on-demand instance- you can drastically save on your CAPEX.

Let us see in detail how to launch an on-demand EC2 instance in AWS Cloud.

Login and access to AWS services

Step 1: In this step,

• Login to your AWS account and go to the AWS Services tab at the top left corner.
• Here, you will see all of the AWS Services categorized as per their area viz. Compute, Storage, Database, etc. For creating an EC2 instance, we have to choose Compute EC2 as in the next step.

AWS Service console

• Open all the services and click on EC2 under Compute services. This will launch the dashboard of EC2.

Here is the EC2 dashboard. Here you will get all the information in gist about the AWS EC2 resources running.

Step 2: On the top right corner of the EC2 dashboard, choose the AWS Region in which you want to provision the EC2 server.

Here we are selecting US West(Oregon). AWS provides 16 Regions all over the globe.

Step 3: In this step

• Once your desired Region is selected, come back to the EC2 Dashboard.
• Click on ‘Launch Instance’ button in the section of Create Instance (as shown below).

• Instance creation wizard page will open as soon as you click ‘Launch Instance’.

Choose AMI

Step 1: In this step we will do,

• You will be asked to choose an AMI of your choice. (An AMI is an Amazon Machine Image. It is a template basically of an Operating System platform which you can use as a base to create your instance). Once you launch an EC2 instance from your preferred AMI, the instance will automatically be booted with the desired OS.
• Here we are choosing the default Amazon Linux (64 bit) AMI.

Choose EC2 Instance Types

Step 1: In the next step, you have to choose the type of instance you require based on your business needs.

• We will choose t2.micro instance type, which is a 1vCPU and 1GB memory server offered by AWS.
• Click on “Configure Instance Details” for further configurations.

• In the next step of the wizard, enter details like no. of instances you want to launch at a time.
• Here we are launching one instance.

Configure Instance

Step 1: Click on ‘Add Storage’ to add data volumes to your instance in next step.

Add Storage

Step 1: In this step, we do the following things,

• In the Add Storage step, you’ll see that the instance has been automatically provisioned a General Purpose SSD root volume of 8GB. ( Maximum volume size we can give to a General Purpose volume is 16GB)
• You can change your volume size, add new volumes, change the volume type, etc.
• AWS provides 3 types of EBS volumes- Magnetic, General Purpose SSD, Provisioned IOPs. You can choose a volume type based on your application’s IOPs needs.

Tag Instance

Step 1: In this step

• You can tag your instance with a key-value pair. This gives visibility to the AWS account administrator when there is lot number of instances.
• The instances should be tagged based on their department, environment like Dev/SIT/Prod. Etc. this gives a clear view of the costing on the instances under one common tag.

Configure Security Groups

Step 1: In this next step of configuring Security Groups, you can restrict traffic on your instance ports. This is an added firewall mechanism provided by AWS apart from your instance’s OS firewall.

You can define open ports and IPs.

• Since our server is a web-server, we will do the following things

1. Creating a new Security Group.
2. Naming our SG for easier reference.
3. Defining protocols which we want to be enabled on my instance.
4. Assigning IPs which are allowed to access our instance on the said protocols.
5. Once, the firewall rules are set- Review and launch.

Review Instances

Step 1: In this step, we will review all our choices and parameters and go ahead to launch our instance.

Step 2: In the next step you will be asked to create a key pair to login to you an instance. A key pair is a set of public-private keys.

AWS stores the private key in the instance, and you are asked to download the public key. Make sure you download the key and keep it safe and secured; if it is lost you cannot download it again.

1. Create a new key pair.
2. Give a name to your key.
3. Download and save it in your secured folder.

• When you download your key, you can open and have a look at your RSA private key.

Step 3: Once you are done downloading and saving your key, launch your instance.

• You can see the launch status meanwhile.

• You can also see the launch log.

• Click on the ‘Instances’ option on the left pane where you can see the status of the instance as ‘Pending’ for a brief while.
• Once your instance is up and running, you can see its status as ‘Running’ now.
• Note that the instance has received a Private IP from the pool of AWS.

• Now am going connect with EC2 instance with Git Bash.
• Select EC2 Instance and click on Connect and then copy the SSH client.

• Open the Key pair .pem file corresponding folder and right click to open the Git bash terminal.

• Paste the copied SSH client from the EC2 Connect in Git Bash.

Let’s install the MongoDB.

Once you are ready with the terminal, let’s start installing MongoDB in it. Since we are going to set up this installation in minutes, I would request you to go through below link in the references section for the below-mentioned command for its detailed usages if you need.

Step 1: Import the public key used by the package management system.

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2930ADAE8CAF5059EE73BB4B58712A2291FA4AD5

Step 2: Create a list file for MongoDB. (for Ubuntu 16.04)

echo “deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.6 multiverse” | sudo tee /etc/apt/sources.list.d/mongodb-org-3.6.list

Step 3: Reload local package database.

sudo apt-get update

Step 4: Install the MongoDB packages.

sudo apt-get install -y mongodb-org

• This will install the latest stable version of MongoDB.
• That’s it, We have installed MongoDB latest stable version in our Amazon EC2 instance.

In the following process, We will first create an admin, who generally has root access to your database and then admin will create a user who can be created with specific roles and access to specific databases. Finally, we will be using user credentials to connect to our database and start working on it.

Let’s start the MongoDB

Step 1: Start MongoDB. Issue the following command to start mongod.

sudo service mongod start

Step 2: Status MongoDB. Issue the following command to start mongod.

sudo service mongod status

Step 3: Verify that MongoDB has started successfully by typing the following command,

cat /var/log/mongodb/mongod.log

That’s it. You are successfully running MongoDB instance in your remote instance right now.

Note: Issue the following command to stop mongod (Don’t execute till needed)

sudo service mongod stop

Note: Issue the following command to restart mongod(Don’t execute till needed)

sudo service mongod restart

It’s time to secure your MongoDB.

Enabling access control on a MongoDB deployment enforces authentication, requiring users to identify themselves. When accessing a MongoDB deployment that has access control enabled, users can only perform actions as determined by their roles.

The following procedure first adds a user administrator to a MongoDB instance running without access control and then enables access control.

Access Control? — Yes, when you run the MongoDB with AccessControl, you are saying, Start secure MongoDB instance.

admin — This user can administrate user and roles such as: create users, grant or revoke roles from users and create or modify customs roles.

Procedure

Step 1: Create a directory to store data and set permission to it.

//Creating a path to store DB data.
sudo mkdir -p /data/db
//Giving yourself permission to write in that folder.
 
sudo chown $USER /data/db

If you have any difficulties in setting up this folder and permission, you may refer to this StackOverflow link.

Step 2: Start MongoDB without access control.

mongod --port 27017 --dbpath /data/db

Step 3: Let the MongoDB instance run on this terminal instance. Now let’s work on another remote terminal instance (Say Terminal 2) so that we will be switching to and fro after creating a user and to restart Mongo instance.

Open a new terminal and connect to your EC2 instance. Now we will be working on Mongo Shell which is used to run commands or queries in your created database.

Now we are going to create Mongo Shell by issuing the below command.

mongod --port 27017

Now your terminal will look like a below image, which is ready to run commands in it.

Step 4: Create the user administrator (admin)

In the admin database, add a user with the userAdminAnyDatabase role. This database acts as admin DB where we creating it only for authentication purpose.

Issue the following command to switch to admin DB even though you haven’t created it. It will be created automatically when you issue the command.

use admin

After switching to admin DB, let’s create an admin by issuing the following command.

db.createUser({user: "admin", pwd: "adminUser123", roles: [{role: "userAdminAnyDatabase", db: "admin"}]})

Which is prettified as, (Above prettified command, do not run twice)

>db.createUser({user: “admin”, pwd: “admin123”, roles: [{role: “userAdminAnyDatabase”, db: “admin”}]})
Successfully added user: {“user” : “admin”,
“roles” : [{
“role” : “userAdminAnyDatabase”,
“db” : “admin”
}]}

Now you have created a user called admin with userAdminAnyDatabase the role.

Disconnect the mongo shell by pressing Ctrl+C.

Step 5: Switch back to an old mongod instance (Terminal 1) where it is running. Let’s restart the MongoDB instance with access control to gain admin access to our databases. Remember, you are starting your Mongo instance with access control now.

Re-start the mongod instance with the –-auth command line option.

mongod --auth --port 27017 --dbpath /data/db

Step 6: Switch back to Terminal 2 (mongo shell instance).

Connect and authenticate as the user administrator by issuing the below command.

mongo — port 27017 -u “admin” -p “adminUser123” — authenticationDatabase “admin”

Step 7: Create additional users (user) as needed for your deployment.

Once authenticated as the user administrator, use db.createUser() to create additional users. You can assign any built-in roles or user-defined roles to the users. The following operation adds a user myTester to the test database who has a readWrite role in the test database

Issue the following command to create a database called test and use it.

use test

Now let’s create a user in this database, assigning specific roles to her/him.

db.createUser({user: "user", pwd: "user123", roles: [{role: "readWrite", db: "test"}]})

Which is prettified as, (Above prettified command, do not run twice).

db.createUser(
  {
    user: "user",
    pwd: "user123",
    roles: [ { role: "readWrite", db: "test" } ]
  }
)

Now you have successfully created a user with specific roles and access to the database.

Step 8: Connect and authenticate as a user now.

Now, connect to the instance with the user role by issuing the following command in the same Mongo Shell.

mongo — port 27017 -u “user” -p “user123” — authenticationDatabase “test”
ubuntu@ip-172-30-0-100:~$ mongo --port 27017 -u "crypto" -p "crypto3" --authenticationDatabase "test"
MongoDB shell version v3.6.9
connecting to: mongodb://127.0.0.1:27017/
Implicit session: session { "id" : UUID("db37bac3-32b8-4d2d-9e50-9aecd8e496a3") }
MongoDB server version: 3.6.9
Server has startup warnings:
2018-12-24T11:39:22.563+0000 I STORAGE  [initandlisten]
2018-12-24T11:39:22.563+0000 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2018-12-24T11:39:22.563+0000 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2018-12-24T11:39:23.295+0000 I CONTROL  [initandlisten]
2018-12-24T11:39:23.295+0000 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2018-12-24T11:39:23.295+0000 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2018-12-24T11:39:23.295+0000 I CONTROL  [initandlisten]
2018-12-24T11:39:23.295+0000 I CONTROL  [initandlisten]
2018-12-24T11:39:23.295+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2018-12-24T11:39:23.295+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2018-12-24T11:39:23.295+0000 I CONTROL  [initandlisten]
2018-12-24T11:39:23.295+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2018-12-24T11:39:23.295+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2018-12-24T11:39:23.295+0000 I CONTROL  [initandlisten]
>

You are now securely authenticated as a user to the database called test. If you want to perform some query operations, you can issue the below command and test now.

Now, let us learn how to perform CRUD (Create/Read/Update/Delete) operations in MongoDB.

Let us insert a document to a player_details collection. You must be connected to a database for doing any insert. It is done as follows:

db.player_details.insert ( { playerid: “ramesh”, email: “ramesh@g.com”}
WriteResult({ “nInserted” : 1 })

To retrieve (Select) the inserted document, run the below command. The find() command will retrieve all the documents of the given collection.

db.player_details.find()

NOTE: Please observe that the record retrieved contains an attribute called _id with some unique identifier value called ObjectId which acts as a document identifier.

If a record is to be retrieved based on some criteria, the find() method should be called passing parameters, then the record will be retrieved based on the attributes specified.

db.player_details.find({"Playerid":"ramesh"})
{ "_id" : ObjectId("5c20d365e297553432ce8003"), "playerid" : "ramesh", "email" : "ramesh@g.com" }

Updating a document in a collection (Update)

In order to update specific field values of a collection in MongoDB, run the below query.

db.player_details.update({"Playerid":"ramesh"},{$set:{"Playerid":"bala"}})
result:
{ "_id" : ObjectId("5c20d365e297553432ce8003"), "playerid" : "bala"}

Removing an entry from the collection (Delete)

Let us now look into the deleting an entry from a collection. In order to delete an entry from a collection, run the command as shown below :

db.player_details.remove({"Playerid":"ramesh"})
WriteResult({ "nRemoved" : 1 })

Note that after running the remove() method, the entry has been deleted from the player_details collection.

Run shell commands on an EC2 from a Lambda function

Execute one or more shell command on a Linux EC2 on AWS based on a particular event. you can use a Lambda function that connects to your EC2 trough SSH and execute the command.we should create a Lambda function

Entering Lambda functions

For a much simpler context, you can use a Lambda function that connects to your EC2 through SSH and executes the command.

Let’s start by creating the Lambda function:

• First of all, go to your AWS console and select Lambda from the services list.
• Click on “Create function”.

• Provide corresponding details while the function lambda creation.

• Following screen-shot shows Created lambda function

• Here I selected “code entry type” as upload a .zip file, and change Handler name as a lambda function.

Before executing the command we need to set up the SSH object, by passing as parameters all the credentials needed to connect to the EC2 instance. As you can see, you’ll need to pass the .pem keypair file data and that’s why you’ll need to put this file in the Lambda’s ZIP package. Also, please remember to set the correct file permissions to allow the Lambda container to access and read the file.

• Following lambda function code shows connected from lambda function to EC2 and run shell commands on EC2 from lambda function.
• Click on this link for source code https://github.com/balaramesh800436/lambdaToEc2

import boto3
import botocore
import paramiko
import pyasn1
import io
import ast


def lambda_handler(event, context):
    ssh = paramiko.SSHClient()
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    key = paramiko.RSAKey.from_private_key_file("./CryptoTradingApp.pem")
    client = paramiko.SSHClient()
    client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    print "the data"
    '''
    Connect/ssh to an instance
    Here 'ubuntu' is user name and 'instance_ip' is public IP of EC2. //instance ID: i-016aadcfb774beab4
    '''
    client.connect(hostname='18.237.161.248', username="ubuntu", pkey=key)
    '''
    Execute a command(cmd) after connecting/ssh to an instance
    '''
    jsontr = {"PLAYERID":"Ramesh","EMAIL":"rbala@g.com"}
    print "the end data"
    commands = ['mongo --eval \'db.PLAYER_DETAILS.insert({\"PLAYERID\":\"mouri\",\"EMAIL\":\"anusha@g.com\"})\'',
				'mongo --eval \'db.PLAYER_DETAILS.find()\'',
				'mongo --eval \'db.PLAYER_DETAILS.update({\"PLAYERID\":\"mouritech\"},{$set:{\"PHONENO":\"123456\"}})\'',
				'mongo --eval \'db.PLAYER_DETAILS.remove({\"PLAYERID\":\"mouri\"})\'']
    #commands = ["mkdir Trace"]
    print "finalCommand execute"
    print (commands)
    for finalCommand in commands:
        print (finalCommand)
        stdin, stdout, stderr = client.exec_command(finalCommand)
        response = str(stderr.read())
        print(response)
        print(str(stdout.read()))
    '''
    close the client connection once the job is done
    '''
    client.close()

As you can see, you can log in the Lambda console the output of all the steps performed, and you will be able to see those logs in Cloudwatch.

Ok, that’s it. I hope you found this case useful! See you next time.