User Command Logs — AWS Cloudwatch

This Blog has been moved from Medium to blogs.tensult.com. All the latest content will be available there. Subscribe to our newsletter to stay updated.

Why are the logs very important to us?

In a production environment, the logs are an important factor to know the reason for IT disaster. If a server goes down in an IT company, it causes loss to the business. If that happens, usually system administrators are blamed for it.

As a System Administrator of a company, he needs to know what are the major reasons for the failure and where to identify them.

This is where the concept of “LOG” comes. Logs will give you all the details of the failure. For every service that runs, whether it’s Linux or Windows Environment, there will be a section for generating the logs and it monitors all the processes of that services like stop, start, error generations, etc..

In a Linux Environment, for running any services we need administrative or sudo access. In a production company, there will be a number of employees who are working with the Linux environment. They need administrative access to configure or install the services. Here it is very difficult to find out who is accessing and the commands used by them. So in such situations, we need the help of user logs, to identify which user is responsible for the commands.

In AWS it is pretty easy to monitor such logs by using the AWS services called Amazon Cloudwatch. Amazon Cloudwatch is a monitoring service for the AWS cloud, which will collect and track metrics, collects and monitors the log files, sets an alarm, and automatically reacts to the changes in AWS. In this case, we are using the EC2 instance for collecting the logs.

Here I have mentioned, few configurations for capturing the user log:

Ubuntu version 16.04

1. Modify the /etc/bash.bashrc

open /etc/bash.bashrc file by using any editor, and append the below line at the end of the file

export PROMPT_COMMAND='RETRN_VAL=$?;logger -p local6.debug "$(whoami) [$$]: $(history 1 | sed "s/[ ][0–9]+[ ]//" ) [$RETRN_VAL]"'

PROMPT_COMMAND is an environment variable, The contents of this variable are executed as a regular Bash command just before Bash displays a prompt. by using logger it sends the message into the Syslog service by using Syslog facility levels ‘local6’ with security level debug.

After this run the following command

# source /etc/bash.bashrc

2. Create a new file /etc/rsyslog.d/bash.conf

Here redirecting the local6 messages from Syslog service to /var/log/commands.log file. For that, create the file and add the below line in the bash.conf

local6.* /var/log/commands.log

3. For logrotate, add /var/log/commands.log into /etc/logrotate.d/rsyslog

4. Restart the rsyslog

# service rsyslog restart

How to capture it in Cloud watch

1. Assign IAM role to the EC2 Instance — AmazonEC2RoleforSSM
2. To set up an AWS log agent, run the following commands in ubuntu machine

# curl https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O
# sudo python ./awslogs-agent-setup.py — region us-east-1
“or” 
# sudo python3 ./awslogs-agent-setup.py — region us-east-1

To download and run it in a standalone system, use the following commands and follow the prompts

# curl https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs- agent-setup.py -O
# curl https://s3.amazonaws.com/aws- cloudwatch/downloads/latest/AgentDependencies.tar.gz -O
# tar xvf AgentDependencies.tar.gz -C /tmp/
# sudo python ./awslogs-agent-setup.py — region us-east-1 –dependency- path /tmp/AgentDependencies

3. Add commands.log to AWS Logs Agent to sync with CloudWatch

For that, we can edit the file : /var/awslogs/etc/config/command_logs.conf

Add the following lines to this file

[ec2-commands-log]
datetime_format = %Y-%m-%d %H:%M:%S
file = /var/log/commands.log
log_stream_name = {instance_id}-commands-log
log_group_name = ec2-commands-log

4. Restart awslogs

#service awslogs restart

5. Check on Cloudwatch -> logs section

NOTE: The Default Region will be in “us-east-1” it depends upon your ec2 instance If you are in some other region Edit and change region on the file /var/awslogs/etc/aws.conf

In RedHat/AWS Linux

1. Modify the /etc/bashrc

open */etc/bashrc file by using any editor, and append the below line at the end of the file

export PROMPT_COMMAND='RETRN_VAL=$?;logger -p local6.debug "$(whoami) [$$]: $(history 1 | sed "s/[ ][0–9]+[ ]//" ) [$RETRN_VAL]"'

PROMPT_COMMAND is an environment variable, The contents of this variable are executed as a regular Bash command just before Bash displays a prompt. by using logger it sends the message into the Syslog service by using Syslog facility levels ‘local6’ with security level debug.

After this run the following command

# source /etc/bashrc

2. We need redirecting the local6 messages from Syslog service to file. /var/log/commands.log For that, Add the below line on /etc/rsyslog.conf file

local6.* /var/log/commands.log

3. For logrotate, add /var/log/commands.log into /etc/logrotate.d/rsyslog

4. Restart the rsyslog

# service rsyslog restart

TO CAPTURE IN cloud watch

1. Assign IAM role to the EC2 Instance — AmazonEC2RoleforSSM

Run the following commands in AWS Linux machine

# yum update -y
# yum install -y awslogs
# sudo service awslogs start
# sudo chkconfig awslogs on

2. Add commands.log to, for AWS Logs Agent to sync to CloudWatch. For that, Open /etc/awslogs/awslogs.conf file, and add the following lines to this:

[ec2-commands-log]
datetime_format = %Y-%m-%d %H:%M:%S
file = /var/log/commands.log
log_stream_name = {instance_id}-commands-log
log_group_name = ec2-commands-log

3. Restart awslogs

# service awslogs restart

4. Check on Cloudwatch -> logs section

NOTE: The Default Region will be in “us-east-1” it depends upon your ec2 instance. If you are in some other region edit and change the region on the file /var/awslogs/etc/aws.conf