Threat Intelligence with Honeypots Part 2 — AWS CloudWatch Dashboards and Alerts
In the first part of this series, I posted how to create a honeypot using Docker and Sysdig. This part of the series I will go through how to create a dashboard in AWS CloudWatch so we can have a nice overview of the data. I’ll also through creating alerts so we can get notified when a new connection is successful.
If you want to follow along please check out my initial post https://medium.com/@davemound/threat-intelligence-with-honeypots-df06963384d3
With the honeypot in place and events being captured in the logs, the next step is to get those logs off the EC2 instance and into CloudWatch. The easiest way to accomplish this is to use the CloudWatch logs agent from AWS. https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/QuickStartEC2Instance.html
I used the python installer and followed the prompts to link the logs on the EC2 instance to my CloudWatch Logs.
Now we can confirm logs have been set up properly the first thing I wanted to do was set an Alarm, this is so I can be alerted when a new connection is made (ie we have someone connected to our honeypot)
To do this we select ‘Create Alarm’ from the Alarms page
For our metric we want to drill down into Logs -> Log Group Metrics and then select our ‘commands’ log group and the ‘IncomingLogEvents’ Metric.
For the next step we give the Alarm a name and a description, set the threshold (in our case IncomingLogEvents is ≥ 0) We need to change ‘Treat missing data as’ to Good and then set up our email account to send the notification to. With all that done our Alarm is sorted and we will be emailed whenever new commands are detected in the log.
Next up is creating a Dashboard for our data, so the first thing is to create one from the Dashboard menu.
When you initially create your dashboard it will ask you if you want to create an initial ‘Widget’, a dashboard is made up of one or more widgets and to start with let's look at creating a ‘Query results’ one.
Next, select the log group you want to look at, in our case, we’ll look at the ‘Failed Attempts’ log group, so select it from the drop-down in the wizard. The wizard should automagically pick up the field names from the log group and show them on the right-hand side of the designer in the ‘Discovered fields’ section. With the log group selected we can start testing our query, running it to see the output.
The CloudWatch Logs Insights Query Syntax is fairly intuitive, there is even a handy reference on the right above our ‘Discovered fields’ and you can also see sample queries from the designer too.
Let’s take a look at some output from our log though. Using the query
fields username, password, srcaddr, type, containerName, @timestamp
| sort @timestamp desc
| limit 100Hit ‘Run’ and you should see some logs records:
Now we can see our first issue, because of the way xinetd is working all the ‘srcaddr’ fields being captured by Sysdig are showing as a local IP. Not to worry though we have it appended to the ‘containerName’ so we need to parse that out. We can do that in the query! Change the query to
fields username, password, srcaddr, type, containerName, @timestamp
| sort @timestamp desc
| limit 100
| parse containerName "honeypot-*" as IPAnd run it again. You should now see that we’ve parsed out the IP from the containerName into its own field.
When you’re happy with the query, save it and it will get added to the dashboard.
I’ll leave it as an exercise for the reader to continue making widgets you think are handy to have for your dashboard. I’ve added a capture of mine so you can get an idea.
That’s it for this update, hopefully I’ve managed to give you a good insight into how you can use AWS CloudWatch to gain insight into your data.
There is still more valuable information that we can extract from our data though and in the next blog post I’ll take you through how you can easily enrich the data and we’ll create a new dashboard with the enrichments added.