How to use CloudWatch log insight for analysis
This is the sixth phase of this project. This phase is a combination of three tasks. I will show every task step by step.
You have already seen in fifth phase that your simulated data is in CloudWatch now. So, you can get insight from these data by using CloudWatch log insight.
First go to CloudWatch console and then choose log insight (red marked 1) from left menu bar.
Then you have to choose an option from drop down (red marked 2). In your case it will be “apache/access”.
After that select a time (red marked 3) when you load your data into CloudWatch. If it is 2 hr ago, you have to select “last 2 hour”. So, choose the time depends on your data upload time.
Then you have to give query in query box (red marked 4) and click on “Run query” button. You will see results based on your query.
After that, you can save your query from “Action” drop down. You have to select “Save as” option from there.
You have to follow these process for next few queries.
Task 1: Determining the number of visitors who accessed the menu
In query section, you have to give this query and do all previously stated processes —
fields @timestamp, remoteIP
| filter request = "/cafe/menu.php"
| count(remoteIP) as Visitors by @timestamp
| sort @timestamp asc
You will see a result like following image. You can see 1000 records out of 49784 matched. You can also see that this gives results out of 187,053 records.
menu-visitors query result
Then click on “Actions” drop down and choose “Save as” option. Then you will get a form like following image, you have to select “Create new” for creating new folder and giving all information like image, click on “Save” button—
To store this result into a file named phase6-results.txt
, go to cloud9 console and open its editor. Then type following commands one after another —
log_group_name="apache/access"
query_string='fields @timestamp, remoteIP | filter request = "/cafe/menu.php" | count(remoteIP) as Visitors by @timestamp | sort @timestamp asc'
query_id=$(aws logs start-query \
--log-group-name $log_group_name \
--start-time $(date -u -d '4 day ago' '+%s') \
--end-time $(date -u '+%s') \
--query-string "$query_string" \
--output text --query 'queryId')
while true; do
status=$(aws logs get-query-results --query-id $query_id --output text --query 'status')
if [ "$status" == "Complete" ]; then
break
elif [ "$status" == "Failed" ] || [ "$status" == "Cancelled" ]; then
echo "Query failed or was cancelled. Exiting."
exit 1
else
sleep 5
fi
done
aws logs get-query-results --query-id $query_id --output text --query 'results' > phase6-results.txt
After running all these commands, you can see the file saved in left menu.
phase6-results.txt
Task 2: Determining the number of visitors who made a purchase
You have to follow all processes that you did in Task 1. Just change the query with this query —
fields @timestamp, remoteIP
| filter request = "/cafe/processOrder.php"
| count(remoteIP) as Visitors by @timestamp
| sort @timestamp asc
Then save it as a separate query, named purchasers
in “non-geo-results” folder.
Task 3: Determining the number of visitors who accessed the menu but didn’t make a purchase
In this step you have to got to cloud9 editor again and run following commands one after another —
log_group_name="apache/access"
query_id_1=$(aws logs start-query \
--log-group-name $log_group_name \
--start-time $(date -u -d '4 days ago' '+%s') \
--end-time $(date -u '+%s') \
--query-string 'fields Visitors | filter request = "/cafe/menu.php" | count(remoteIP) as Visitors' \
--output text --query 'queryId')
while true; do
status=$(aws logs get-query-results --query-id $query_id_1 --output text --query 'status')
if [ "$status" == "Complete" ]; then
break
elif [ "$status" == "Failed" ] || [ "$status" == "Cancelled" ]; then
echo "Query failed or was cancelled. Exiting."
exit 1
else
sleep 5
fi
done
aws logs get-query-results --query-id $query_id_1 --output text --query 'results' > query1_results.txt
query_id_2=$(aws logs start-query \
--log-group-name $log_group_name \
--start-time $(date -u -d '4 days ago' '+%s') \
--end-time $(date -u '+%s') \
--query-string 'fields Visitors | filter request = "/cafe/processOrder.php" | count(remoteIP) as Visitors' \
--output text --query 'queryId')
while true; do
status=$(aws logs get-query-results --query-id $query_id_2 --output text --query 'status')
if [ "$status" == "Complete" ]; then
break
elif [ "$status" == "Failed" ] || [ "$status" == "Cancelled" ]; then
echo "Query failed or was cancelled. Exiting."
exit 1
else
sleep 5
fi
done
aws logs get-query-results --query-id $query_id_2 --output text --query 'results' > query2_results.txt
paste <(awk '{print $2}' query1_results.txt) <(awk '{print $2}' query2_results.txt) | awk '{if ($1 != 0) printf "%.2f%%\n", ($2 / $1) * 100; else print "N/A"}' > percentage_difference.txt
So you can see this in console like these images —
Congratulations!!! You have done one more phase. Go to next phase.
If you find this post helpful, please give a clap in this post and follow me in medium and lets connected in linked in.