How to delete existing CloudWatch Log data and upload new simulated data and get insight from log data

This is the seventh phase of this project. This phase is a combination of five tasks. I will show every task step by step.

After completing this phase, you will see a dashboard like this —

Dashboard

Task 1: Understanding the requirements

In this task, there are no technical things to do. The requirements of next few tasks are discussed.

The cafe owner wants to see a pie chart and bar chart for different information like —

i) A pie chart that shows the 10 cities that had the most website visitors who accessed the menu page.

ii) A log table that shows the 10 cities that had the most website visitors who made a purchase.

iii) A pie chart that shows the 10 regions that had the most website visitors who accessed the main page of the website.

iv) A bar chart that shows the 10 regions that had the most website visitors who made a purchase.

Task 2: Using the example logs that include geolocation information

1. In first step of this task, you can see geo location data into “samplelogs” folder. Run following commands one after another —

 cd ~/environment

head -1 samplelogs/access_log_geo.log | python -m json.tool

cat samplelogs/access_log_geo.log | wc -l

2. Then stop the CloudWatch agent though this command -

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a stop

Check the status -

 sudo service amazon-cloudwatch-agent status

3. Then go to aws console and go to CloudWach log groups and double click on “apace/access” folder.

Log groups and access folder

Then delete the log stream by clicking “Delete” button.

Delete log stream

4. Then replace the previous access_log.log file into this path /var/log/www/access with the file access_log_geo.log. Run following command -

sudo cp /home/ec2-user/environment/samplelogs/access_log_geo.log /var/log/www/access/access_log

Check this file is moved successfully or not -

sudo wc -l /var/log/www/access/access_log

In new file there are 5428 data in total. You can see it like following image

Geo log data

Now restart the CloudWatch agent -

sudo systemctl restart amazon-cloudwatch-agent.service

And check the status -

sudo service amazon-cloudwatch-agent status

CloudWatch agent status

Task 3: Building a dashboard to observe the geolocation data in CloudWatch Logs Insights

1. First go to CloudWatch console and click on “Dashboards” from left menu and get this page with “Create dashboard” button. Click on this button.

Then make a dashboard named “cafe-dashboard”.

Dashboard named cafe-dashboard

Then select all red marked options like following image and click on “Next” button —

pie chart

2. Then you will see a query editor. Run following query -

fields remoteIP, city
| filter request = "/cafe/menu.php"
| stats count() as menupopular by city 
| sort menupopular desc
| limit 10

And process all actions like as red marked steps one after another and lastly click on “Create widget” button

create widget

Then click on edit button like red marked 1 and rename with “Cities visiting the menu the most”. Then click on “Save” button (red marked 2). After that click on “+” sign (red marked 3) to create new widget in this dashboard.

Cities visiting the menu the most

Make other widget like previous one.

Task 4: Saving the log file to an S3 bucket

To save log file into s3, first you have to go to s3 console and you will see a bucket is created for you. You have to copy the bucket name. In my case it is accap4-logsbucket —bf041fc0

Bucket

Change your bucket name in following command and run it from cloud9 console.

 sudo aws s3 cp /home/ec2-user/environment/samplelogs/access_log_geo.log s3://accap4-logsbucket--bf041fc0/

Then you will see you log file is in the bucket. Then choose the checkbox and select “Query with S3 Select”

Object in s3 bucket

Then you have to select options in “Input settings” and “Output settings” like following images -

input settings

output settings

Then select template and then copy that query and run it like following images -

s3 select query

s3 select query format

Congratulations!!! You have successfully done all phases.

If you find this post helpful, please give a clap in this post and follow me in medium and lets connected in linked in.