How to install Cloudwatch agent and make configuration file
This is the third phase of this project. This phase is a combination of four tasks. I will show every task step by step.
Task 1: Installing CloudWatch agent on web server
For installing cloudwatch agent, run following command in cloud9 editor.
sudo yum install -y amazon-cloudwatch-agent
Task 2: Creating configuration file for the CloudWatch agent
Here I will use “sudo” before my command as it gives me permission error. Run the command without sudo first, if it is okay in your side then you do not need to run with sudo. But if it gives deny permission error, then run the command with sudo.
sudo wget https://aws-tc-largeobjects.s3.us-west-2.amazonaws.com/CUR-TF-200-ACCAP4-1-79925/capstone-4-clickstream/s3/config.json
After that run command to move config file to bin folder of amazon-cloudwatch-agent
sudo mv config.json /opt/aws/amazon-cloudwatch-agent/bin/
To see what is inside this file, run following command —
sudo cat /opt/aws/amazon-cloudwatch-agent/bin/config.json
You will see something like following image. You see the access file path and error file path. Remember, cloudwatch agent will find these files in these path. So, if you do not have this folder into this path, you have to make it.
Task 3: Configuring httpd.conf log format as JSON
In this section, you have to follow few steps.
Step 1: Find the configuration file, named httpd.conf . Then back up it. To do this I have run following commands -
First find the file path -
sudo find /etc -name httpd.conf
You will see something like this
Then back up it by running following command -
sudo cp /etc/httpd/conf/httpd.conf /home/ec2-user/environment/httpd.conf
Then you will see a file in left hand menu, named httpd.conf liek following image —
Step 2: As you can edit httpd.conf file into cloud9, run following command -
sudo ln -s /etc/httpd/conf /home/ec2-user/environment/httpdconf
Then you can see a folder in left side menu like following image —
Then run following command to give edit permission in this folder.
sudo chown -R ec2-user /etc/httpd/conf
Step 3: Then double click on httpd.conf file and see it in cloud9 editor and then start editing this file.
Then you have to comment out the line ErrorLog “logs/error_log" around 182 and then copy and paste following lines after that like image —
ErrorLog "/var/log/www/error/error_log"
ErrorLogFormat "{\"time\":\"%{%usec_frac}t\", \"function\" : \"[%-m:%l]\", \"process\" : \"[pid%P]\" ,\"message\" : \"%M\"}"
Then configure access log around the line 191, you have to comment out the line -
LogFormat “%h %l %u %t \”%r\” %>s %b \”%{Referer}i\” \”%{User-Agent}i\”” combined
and then add following lines after this -
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "{ \"time\":\"%{%Y-%m-%d}tT%{%T}t.%{msec_frac}tZ\", \"process\":\"%D\", \"filename\":\"%f\", \"remoteIP\":\"%a\", \"host\":\"%V\", \"request\":\"%U\", \"query\":\"%q\",\"method\":\"%m\", \"status\":\"%>s\", \"userAgent\":\"%{User-agent}i\",\"referer\":\"%{Referer}i\"}" cloudwatch
Then comment out entire <IfModule logio_module>
section like following image -
Then search this line CustomLog "logs/access_log" combined
. You will see this line around 219, then add a new line after this like following image -
CustomLog "/var/log/www/access/access_log" cloudwatch
Then save this file and close it.
Task 4: Using updated configuration file for the CloudWatch agent
In this task, you also have to follow some steps.
Step 1: First make access log directory and error log directory in cloud watch agent configuration file defined path.
Create new access and error log directories so that the directory locations that you specified in the httpd.conf file exist on the server. Run following command one after another -
sudo mkdir -p /var/log/www/error
sudo chown -R apache:apache /var/log/www
sudo mkdir -p /var/log/www/access
sudo chown -R apache:apache /var/log/www
Step 2: Then restart the httpd service —
sudo systemctl restart httpd
Step 3: Now you have to start your cloudwatch agent. Before running start command, you have to run following command first to find out where cloudwatch.json file live -
sudo find / -name '*cloudwatch*json*' 2>/dev/null
You will find a path like following image.
Then use your path in the replace of <> bracket in following command
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:<configuration-file-path>
After changing path, it will look like
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/doc/amazon-cloudwatch-agent-schema.json
Then run start command to start cloudwatch agent-
sudo systemctl start amazon-cloudwatch-agent.service
Afer this check the status of this -
service amazon-cloudwatch-agent status
You will see like following image —
If you get problem to run CloudWatch agent then follow these steps -
First, run the command to see which error occurred -
sudo journalctl -xe -u amazon-cloudwatch-agent.service
It will give the problem detail.
In case of some people following three commands work -
sudo amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -s
sudo systemctl start amazon-cloudwatch-agent.service
service amazon-cloudwatch-agent status
If you still get problem and problem like this -
Then follow these steps —
- Go to cloud9 console and run following command -
sudo cat /opt/aws/amazon-cloudwatch-agent/bin/config.json
2. Copy the result and save it into a separate editor as you can use it later. Copy your own result. I am giving my result here.
{
"agent": {
"metrics_collection_interval": 60,
"run_as_user": "cwagent"
},
"logs": {
"logs_collected": {
"files": {
"collect_list": [
{
"file_path": "/var/log/www/error/*",
"log_group_name": "apache/error",
"log_stream_name": "{instance_id}",
"retention_in_days": 180
},
{
"file_path": "/var/log/www/access/*",
"log_group_name": "apache/access",
"log_stream_name": "{instance_id}",
"retention_in_days": 180
}
]
}
}
},
"metrics": {
"aggregation_dimensions": [
[
"InstanceId"
]
],
"append_dimensions": {
"AutoScalingGroupName": "${aws:AutoScalingGroupName}",
"ImageId": "${aws:ImageId}",
"InstanceId": "${aws:InstanceId}",
"InstanceType": "${aws:InstanceType}"
},
"metrics_collected": {
"collectd": {
"metrics_aggregation_interval": 60
},
"disk": {
"measurement": [
"used_percent"
],
"metrics_collection_interval": 60,
"resources": [
"*"
],
"ignore_file_system_types": [
"sysfs", "devtmpfs"
]
},
"mem": {
"measurement": [
"mem_used_percent"
],
"metrics_collection_interval": 60
},
"statsd": {
"metrics_aggregation_interval": 60,
"metrics_collection_interval": 10,
"service_address": ":8125"
}
}
}
3. After that run following command into cloud9 -
sudo nano /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json
4. Then the result you get from step 2, paste it here. It will look like this. If you scroll, then you can see full copied text here.
5. Then save it and exit it by using “crtl+x”. After giving “crtl+x”, you will see option like this. Then press “shift+Y” and then press “enter”.
6. After that run following command -
sudo systemctl restart amazon-cloudwatch-agent.service
7. Check your CloudWatch agent status -
service amazon-cloudwatch-agent status
Hope it will fix problem.
Congratulations!!! You have done third phase of this project. Go to next phase.
If you find this post helpful, please give a clap in this post and follow me in medium and lets connected in linked in.