Cross-Region application monitoring using Amazon CloudWatch Synthetics and AWS CloudFormation

Infrastructure of the project

What’s behind this project?

Before an application could be delivered to the market for end users, the company needs to test the potential issues facing the application. It’s essential to predict and project the performance of the application in supported geographies and further isolate the root cause of any hidden problems. By using synthetic monitoring, company is able to emulate business processes or user transactions from different geolocations and monitor performance and availability alike.

In this project, we will break down our projects into two parts. On the one hand, we dive deep to deploy a global synthetic monitoring solution across multiple geographic regions using Amazon CloudWatch Synthetics and AWS CloudFormation StackSets, on the other hand, we will also showcase how to monitor the website performance results using Amazon CloudWatch cross-region dashboards in a centralized location.

Notes: In this project, we’ll attempt to apply infrastructure as code as much as we can to accomplish the tasks intended.

Project prerequisites:

1 AWS non-root account

2 AWS CLI installed

Creating a non-root user

Based on AWS best practice, root user is not recommended to perform everyday tasks, even the administrative ones. The root user, rather is used to to create your first IAM user, groups and roles. Then you need to securely lock away the root user credentials and use them to perform only a few account and service management tasks.

Notes: If you would like to learn more about why we should not use root user for operations and more about AWS account, please find more here.

Login as a Root user

Create a user under IAM service

Choose programmatic access

Attach required policies

Create user without tags

Keep credentials (Access key ID and Secret access key)

Installing AWS CLI

Visit here and download Mac packer

Download MacOS pkg installer

Install it successfully

To verify your aws cli installation

$ aws --version
aws-cli/2.0.46 Python/3.7.4 Darwin/19.6.0 exe/x86_64

Part One:

Deploy a global synthetic monitoring solution across multiple geographic regions using Amazon CloudWatch Synthetics and AWS CloudFormation StackSets

Step 1: Complete prerequisites for AWS Cloudformation Stack Sets operations

2 prerequisites for creating our AWS Cloudformaton Stack Sets:

1. AWS Organizations

2 . Enable CloudFormation StackSets using Trusted Access

Creating AWS Organizations

By applying following code using AWS CLI, we are about to create an organization with following attributes

$ aws organizations create-organization
{
    "Organization": {
        "Id": "o-ir8giv37pm",
        "Arn": "arn:aws:organizations::XXXXXXX:organization/o-ir8giv37pm",
        "FeatureSet": "ALL",
        "MasterAccountArn": "arn:aws:organizations::XXXXXXXX:account/o-ir8giv37pm/XXXXXXXX,
        "MasterAccountId": "XXXXXXX",
        "MasterAccountEmail": "XXXXXXXXX@gmail.com",
        "AvailablePolicyTypes": [
            {
                "Type": "SERVICE_CONTROL_POLICY",
                "Status": "ENABLED"
            }
        ]
    }
}

In AWS console, we cross check organization that we create

Organization created

Email confirmation sent from AWS

Email confirmation required

Email verified

Email verified

Enable CloudFormation StackSets using Trusted Access

Under Cloudformation Stacksets, we need to enable trusted access

Enable trusted access

Successfully verified trusted access with AWS organizations

Trusted access verified

In your AWS account, check whether you already have an AWS Identity and Access Management (IAM) role named AWSCloudFormationStackSetAdministrationRole, if not create the IAM role. You must use this name. You can do this by creating a stack from the following AWS CloudFormation template, available online at https://s3.amazonaws.com/cloudformation-stackset-sample-templates-us-east-1/AWSCloudFormationStackSetAdministrationRole.yml.

In order to create a stackset, we create our builds using following codes

$ aws cloudformation create-stack-set --stack-set-name AWSCloudFormationStackSetAdministrationRole --template-body file://AWSCloudFormationStackSetAdministrationRole.yml --capabilities CAPABILITY_NAMED_IAM
{
    "StackSetId": "AWSCloudFormationStackSetAdministrationRole:c5038b08-5fb7-4162-956c-27f1d98018e5"
}

To verify our build-ups under AWS console StackSets

We also build up a AWSCloudFormationStackSetExecutionRole then create the buildup

$ aws cloudformation create-stack-set --stack-set-name AWSCloudFormationStackSetExecutionRole --template-body file://AWSCloudFormationStackSetExecutionRole.yml --parameters ParameterKey=AdministratorAccountId,ParameterValue=4643925XXXXX --capabilities CAPABILITY_NAMED_IAM
{
    "StackSetId": "AWSCloudFormationStackSetExecutionRole:96753913-897b-46b5-bb03-410fd64b4540"
}

Step 2: Create and deploy a CloudWatch Synthetics canary in different AWS Regions

Here is the cloudformation script and create a file locally and name it aws-synthetics-stackset.yml

This CloudFormation script will create the following resources:

• A Synthetics canary with inline code.
• Amazon Cloudwatch alarms for SuccessPercent and Durationmetrics.
• An Amazon Simple Storage Service (Amazon S3) bucket to store the canary results.
• An AWS Lambda function to execute the canary and associated IAM roles.

To use StackSets to deploy the canary in different AWS Regions:

In us-east-1 region, we first create our StackSet named

$ aws cloudformation create-stack-set --stack-set-name aws-synthetics-stackset --region us-east-1 --template-body file://aws-synthetics-stackset.yml --parameters ParameterKey=CanaryDurationAlarmName,ParameterValue=mycanary-durationalarm ParameterKey=CanaryName,ParameterValue=mycanary ParameterKey=CanarySuccessLowAlarmName,ParameterValue=mycanary-successlowalarm ParameterKey=URLToMonitor,ParameterValue=https://www.amazon.com/ --capabilities CAPABILITY_NAMED_IAM --execution-role-name AWSCloudFormationStackSetExecutionRole --administration-role-arn arn:aws:iam::4643925XXXXX:role/AWSCloudFormationStackSetAdministrationRole

Using code above, we about to create a StackSet in us-east-1 region like shown below

aws-synthetics-stackset

Then we can’t wait for our imagic to occur after applying code below. What it does is to apply StackSet into different regions within our account

$ aws cloudformation create-stack-instances --stack-set-name aws-synthetics-stackset --deployment-targets Accounts=4643925XXXXX --regions '["us-east-1","us-west-1","eu-west-2","ap-south-1","ap-southeast-2"]'
{
    "OperationId": "938d6fa4-5fd3-4289-a30b-f511870009be"
}

Here are the stacks being created in 5 different regions US East (N.Virginia) — us-east-1, US West (N.California) — us-west-1, EU (London) — eu-west-2, Asia Pacific (Mumbai) — ap-south-1, and Asia Pacific (Sydney) — ap-southeast-2

us-east-1

us-west-1

ap-south-1

ap-southeast-1

eu-west-2

Note: This operation can take up to five minutes. Wait until status of all stacks in selected AWS Regions change to CURRENT .

Current shown for all 5 stack instances

Tips: I was pretty struggled to figure out create-stack-instances will initialize stacks in different designated regions and apply what .yml file intended infrastructure. The best solution to tackle any AWS issues is definitely AWS documentations. For this one, please check out here

Step 3: Create a cross-region dashboard to visualize the results

To enable cross-region functionality to CloudWatch:

1. Navigate to Amazon CloudWatch console
2. Choose the Region where you want to create the dashboard.(N.Virginia (us-east-1).
3. From the left navigation pane, choose Settings.
4. In Cross-account cross-region, choose Configure.
5. In View cross-account cross-region, choose Enable.
6. In Enable account selector section, Choose account settings or keep them at default.
7. Choose Enable.

CloudWatch page to locate settings

Locate view cross-account cross-region and enable it

Choose account id input and enable it

Enabled view cross-account cross-region

To create a cross-region dashboard to view your canaries:

1. In the left navigation pane, choose Dashboards, and then choose Create dashboard.
2. Enter a name for the dashboard (for example, aws-synthetics-dashboard), and then choose Create dashboard.
3. Select a widget type to configure (for example, Line graph), click Next, choose Metrics, and then choose Configure.
4. In All metrics, scroll down and search for Synthetics. From Custom Namespaces, choose CloudWatchSynthetics.
5. Choose By Canary and then choose mycanary and duration to be visualized in the dashboard.
6. From the Choose region dropdown list, choose the AWS Regions where the canary is deployed and choose mycanary and duration.
7. Choose Create widget.
8. Choose Add widget and repeat steps 5 through 8 for the SuccessPercent metric.
9. Add more widgets of different types and metrics and edit the name of the widget, if needed.
10. Choose Save dashboard.

Display of the widgets

Step 4: Change the canary centrally and redeploy to all Regions

In this section, I am going to edit the CloudFormation template to change the schedule of the canary to run every 10 minutes instead of 5 minutes. This is to illustrate that you can change this canary centrally, in a single place, and then deploy the changes to multiple Regions at once.

Open the aws-synthetics-stackset.yml local file in a text editor and change the following line:

Schedule: {Expression: 'rate(5 minutes)', DurationInSeconds: '3600'}

to:

Schedule: {Expression: 'rate(10 minutes)', DurationInSeconds: '3600'}

Simply apply the following code for aws cloudformation update-stack-set

$ aws cloudformation update-stack-set --stack-set-name aws-synthetics-stackset --template-body file://aws-synthetics-stackset.yml --deployment-targets Accounts=4643925XXXXX --regions '["us-east-1","us-west-1","eu-west-2","ap-south-1","ap-southeast-2"]' --capabilities CAPABILITY_NAMED_IAM --execution-role-name AWSCloudFormationStackSetExecutionRole --administration-role-arn arn:aws:iam::4643925XXXXX:role/AWSCloudFormationStackSetAdministrationRole
{
    "OperationId": "d10c2be8-0799-44c5-8362-fc3ff90fb570"
}

We are able to make update for 10 minutes across regions

Our project is all done. But, we need to clear up resources created in AWS unless we’d like to see expenses going up to the roof!

Clear Up

Since we built up our project all with code, now it’s much easier to clear up our using infrastructure as code as well

Before we can delete a stack set, all of its member stack instances must be deleted

$ aws cloudformation delete-stack-instances --stack-set-name aws-synthetics-stackset --accounts "4643925XXXXX" --regions "us-east-1" "us-west-1" "eu-west-2" "ap-south-1" "ap-southeast-2" --no-retain-stacks
{
    "OperationId": "63c844ee-3628-47cf-87b7-4c4a5241045a"
}

While doing so, we encountered dependencies issues like shown below

Failed to delete stacksets

We found issue under stack instances that it was caused by dependencies of resources under S3 bucket

s3 object dependencies

Heading towards S3 bucket to find out the names of our resources

s3 buckets

To delete S3 resources

$ aws s3 rm s3://cw-syn-results-us-east-1-464392XXXXX --recursive
$ aws s3 rm s3://cw-syn-results-us-west-1-4643925XXXXX --recursive
$ aws s3 rm s3://cw-syn-results-eu-west-2-4643925XXXXX --recursive
$ aws s3 rm s3://cw-syn-results-ap-south-1-464392XXXXX --recursive
$ aws s3 rm s3://cw-syn-results-ap-southeast-2-4643925XXXXX --recursive

Now let’s try again to delete-stack-instances

$ aws cloudformation delete-stack-instances --stack-set-name aws-synthetics-stackset --accounts "4643925XXXXX" --regions "us-east-1" "us-west-1" "eu-west-2" "ap-south-1" "ap-southeast-2" --no-retain-stacks
{
    "OperationId": "80211360-bb99-4708-b186-cd72897eb9c2"
}

Awesome, we made it this time around

Final push is to delete our stacksets

$ aws cloudformation  delete-stack-set --stack-set-name aws-synthetics-stackset

Deletion of stackset

Notes: Along with it, the stacks created by this stackset in different regions were also deleted, which is the reason why we chose to use stackset to build our project. So we are able to delete them simultaneously when needed

stacks created by stackset in other regions deleted as well

Just to cross check, our resources for cloudwatch in other regions were also deleted since it was also created by stackset

cloudwatch resources deleted as well

Now let us delete the two remaing stacksets that we created for our IAM roles

$ aws cloudformation  delete-stack-set --stack-set-name AWSCloudFormationStackSetAdministrationRole
$ aws cloudformation  delete-stack-set --stack-set-name AWSCloudFormationStackSetExecutionRole

stacksets all cleared

Conclusions:

First of all, what a project it was, we dived in deep by applying the power of stackset as well as infrastructure as code.

As we built our project from scratch using infrastructure as code, we were able to better nagivate in our structures. Without jumping from our page in AWS console to another, we deployed our resources seemlessly. Besides, the code in place is much easier for us to make updates, deletions and more.

In terms of cloudformation stackset, we witnessed how it shined its power by applying stacks in different regions or even different accounts if needed in AWS. More importantly, it also gave us the edge to delete all resources seemlessly at the end of this project, which will reduce the cost of operation by large if applied in a production environment.

Throughpout the project, we also encountered a few errors such as unable to delete our stackset due to S3 bucket dependencies. This allowed us to be aware of dependencies issues when building up and cleaning up our infrastructures. Again, as long as we follow along the error alerts provided by AWS, we should be able to accomplish intended outcome with ease.