Monitoring and Alerting On-Premise IIS Hosted Application Availability with AWS Cloudwatch
In this article, I will cover a monitoring and alerting infrastructure for on-premise IIS hosted applications with AWS Cloudwatch.
In a hybrid cloud architecture, we would have workloads running both in the cloud and on-premise. In our case, let's assume we have a legacy application (eg- SOAP API ) hosted in an IIS container on a windows host in an on-premise data center, and we have a requirement to set up monitoring and alerting when the application is unavailable or goes offline.
We will use the following tools to cater to the above requirement.
- AWS Cloudwatch agent — to ship server logs to Cloudwatch.
- Slack and the Opsidian slack bot — to set up an alerting channel.
- AWS SNS — To send Notifications via HTTPS to slack.
- AWS Cloudwatch — to aggregate logs and to create log filters and alarms.
- AWS SSM Parameter store— to store Cloudwatch agent configuration file.
#1 Setting up Cloudwatch Agent to Ship HTTERR IIS error logs
Incoming requests to the server first route through HTTP.SYS before being handed to IIS. These errors get logged in HTTPERR. Common errors are 503 Service Unavailable and similar types of issues. The built-in error messages and error codes from HTTP.SYS is very detailed, this is where we can find out when our application is unavailable.
HTTPERR Logs can be found at C:\Windows\System32\LogFiles\HTTPERR
I have a test application running on port 8091, lets stop the application pool relevant to the test app and see how HTTP.SYS captures it in HTTPERR error logs.
HTTP status code — 503 service unavailable, states that the IIS Application pool is not available to handle the http request.
Now that 503 — service unavailable error’s can be captured through HTTPERR, we need to ship those logs via Cloudwatch agent to AWS Cloudwatch.
- Setup the AWS CLI on the server with a new IAM user. Make sure the IAM user has the AWS Managed Policy — CloudwatchagentServerPolicy attached.
- Setup the Cloudwatch agent on the server, follow this tutorial to setup the agent.
Now that the agent is installed on the server lets configure the agent to ship the HTTPERR logs to Cloudwatch.
Agent configuration is stored in a JSON file. read more on the agent configuration file here. Check this configuration file i have created that has HTTPERR log shipping enabled.
Note — make sure you use the prefix AmazonCloudWatch-, when naming the file as the AWS managed policy we attached to the user only grants permission for files with the above prefix to be uploaded to AWS SSM Parameter store.
The “logs” section of the config file sets the config details to ship custom log files such as the HTTERR log files.
Next lets upload our config file to AWS SSM Parameter store, so that we can easily manage it.
Now that the configuration file is uploaded to parameter store, we can start the cloudwatch agent in the server.
Navigate to the installation path of the cloudwatch agent, usually it is C:\Program Files\Amazon\AmazonCloudWatchAgent. You will find a powershell script file, named — amazon-cloudwatch-agent-ctl.ps1.
Run the following command
#2 Setup Opsidian bot and a SNS Topic with HTTPS Subscription
Follow this link to install the opsidian bot to Slack. Once we have setup the bot with the required permissions to AWS, we also need to make sure that we add the bot to our preferred slack channel. This channel will be used to notify on application downtime.
Run the following command to obtain a HTTPS endpoint for SNS subscription.
Now create a new SNS topic and use the HTTPS endpoint from step 5 to create a new HTTPS Subscription (step 2–5).
Once you have completed the above steps (step 2- 5), we would have setup the communication channel from AWS to Slack, now we can move on to setting up log filtering and alarms in Cloudwatch.
#3 Setting up a Log Filter and Alarm in Cloudwatch
Lets navigate to the log group we mentioned in the Cloudwatch agent configuration file to find the logs that have been shipped from our server to Cloudwatch.
HTTPERR uses the following format for logging :
date time | c-ip | c-port | s-ip | s-port | cs-version | cs-method | cs-uri | streamid | sc-status | s-siteid | s-reason | s-queuename
We will concentrate on the sc-status (HTTP Status Code) and s-siteid (I.E Application Pool Id) to create the log filter, We need to look for HTTP status code 503-service unavailable and Application pool Id — 6 which is the App pool id in IIS for the TestApp i set up. Lets create the metric filter as shown below.
Now that the metric filter is setup, we can go on to setup a Cloudwatch alarm that will trigger the SNS topic we created in Step #2.
Navigate to the metric we created above and click on create alarm.
The Cloudwatch alarm details should be as shown below:
we need to select the appropriate SNS topic we created in Step #2 in the “Actions” section shown above.
Now we have completed setting up, lets take look at how Opsidian notifies on service unavailability in Slack.
Bingo!! we have completed setting up Alerting. Lets move on to setting up a dashboard to monitor service unavailability over time. We can create a Cloudwatch dashboard and include the following line graph as a widget.
Note : we have only data related to one data point, over time you will be able to see a more detailed graph.
Summing Up
I have shown you how to setup a Monitoring and Alerting infrastructure for On-Premise IIS hosted applications with AWS Cloudwatch. We setup Cloudwatch Agent for log shipping to AWS, then we setup the Opsidian bot in slack with a SNS https subscription to open up a notification channel. We created a metric filter in Cloudwatch to capture 503-service unavailable logs captured by HTTPERR, we went on to create an alarm for the metric filter. The alarm we created triggered the SNS topic which in-turn notified us in slack via Opsidian.
References
https://support.microsoft.com/en-us/help/820729/error-logging-in-http-apis#3
