How to delete existing CloudWatch Log data and upload new simulated data and get insight from log data
This is the seventh phase of this project. This phase is a combination of five tasks. I will show every task step by step.
After completing this phase, you will see a dashboard like this —
Task 1: Understanding the requirements
In this task, there are no technical things to do. The requirements of next few tasks are discussed.
The cafe owner wants to see a pie chart and bar chart for different information like —
i) A pie chart that shows the 10 cities that had the most website visitors who accessed the menu page.
ii) A log table that shows the 10 cities that had the most website visitors who made a purchase.
iii) A pie chart that shows the 10 regions that had the most website visitors who accessed the main page of the website.
iv) A bar chart that shows the 10 regions that had the most website visitors who made a purchase.
Task 2: Using the example logs that include geolocation information
- In first step of this task, you can see geo location data into “samplelogs” folder. Run following commands one after another —
cd ~/environment
head -1 samplelogs/access_log_geo.log | python -m json.tool
cat samplelogs/access_log_geo.log | wc -l
2. Then stop the CloudWatch agent though this command -
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a stop
Check the status -
sudo service amazon-cloudwatch-agent status
3. Then go to aws console and go to CloudWach log groups and double click on “apace/access” folder.
Then delete the log stream by clicking “Delete” button.
4. Then replace the previous access_log.log file into this path /var/log/www/access with the file access_log_geo.log. Run following command -
sudo cp /home/ec2-user/environment/samplelogs/access_log_geo.log /var/log/www/access/access_log
Check this file is moved successfully or not -
sudo wc -l /var/log/www/access/access_log
In new file there are 5428 data in total. You can see it like following image
Now restart the CloudWatch agent -
sudo systemctl restart amazon-cloudwatch-agent.service
And check the status -
sudo service amazon-cloudwatch-agent status
Task 3: Building a dashboard to observe the geolocation data in CloudWatch Logs Insights
- First go to CloudWatch console and click on “Dashboards” from left menu and get this page with “Create dashboard” button. Click on this button.
Then make a dashboard named “cafe-dashboard
”.
Then select all red marked options like following image and click on “Next” button —
2. Then you will see a query editor. Run following query -
fields remoteIP, city
| filter request = "/cafe/menu.php"
| stats count() as menupopular by city
| sort menupopular desc
| limit 10
And process all actions like as red marked steps one after another and lastly click on “Create widget” button
Then click on edit button like red marked 1 and rename with “Cities visiting the menu the most
”. Then click on “Save” button (red marked 2). After that click on “+” sign (red marked 3) to create new widget in this dashboard.
Make other widget like previous one.
Task 4: Saving the log file to an S3 bucket
To save log file into s3, first you have to go to s3 console and you will see a bucket is created for you. You have to copy the bucket name. In my case it is accap4-logsbucket —bf041fc0
Change your bucket name in following command and run it from cloud9 console.
sudo aws s3 cp /home/ec2-user/environment/samplelogs/access_log_geo.log s3://accap4-logsbucket--bf041fc0/
Then you will see you log file is in the bucket. Then choose the checkbox and select “Query with S3 Select”
Then you have to select options in “Input settings” and “Output settings” like following images -
Then select template and then copy that query and run it like following images -
Congratulations!!! You have successfully done all phases.
If you find this post helpful, please give a clap in this post and follow me in medium and lets connected in linked in.