How I Built an AWS App Without Spending a Penny — Other Services
This is part 6 of a 6-part series. See the previous parts where we built the frontend, backend, and pipeline.
- Part 1 — Introduction
- Part 2 — The Frontend
- Part 3 — The CLI + SAM
- Part 4 — The Backend
- Part 5 — The Pipeline
I saved this section for other miscellaneous services I used while building this AWS project. AWS is kind of big.
Trusted Advisor
Trusted Advisor is a free tool you can use to check if what you’re building in AWS follows their well-architected best practices, including cost optimization, performance, security, fault tolerance, and service limits. Under a Basic Support plan, you only have access to the service limits and some security checks. But these other checks aren’t hidden at all! They’re faded, but still visible to all users.
In addition, you can find all the checks at https://docs.aws.amazon.com/awssupport/latest/user/trusted-advisor-check-reference.html. So, you can still browse all the checks for all the services you’re using and see if your account is compliant. This is where I discovered many of the best practices that I tried to apply for this project. The only thing upgrading to a Business or Enterprise plan would do is allow AWS to automatically check these for you.
If you want to get notified whenever a Trusted Advisor check fails, you can use EventBridge and SNS to subscribe to error or warning events from Trusted Advisor. The architecture looks like this:
And the CloudFormation template looks like the following:
AWSTemplateFormatVersion: "2010-09-09"
Description: Subscribe to errors and warnings from Trusted Advisor
Parameters:
Email:
Type: String
Description: The email address that will receive SNS notifications for the TA alerts
# Simple regex from https://stackoverflow.com/a/201378
AllowedPattern: "(?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|\"(?:[\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x21\\x23-\\x5b\\x5d-\\x7f]|\\\\[\\x01-\\x09\\x0b\\x0c\\x0e-\\x7f])*\")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\\[(?:(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9]))\\.){3}(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9])|[a-z0-9-]*[a-z0-9]:(?:[\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x21-\\x5a\\x53-\\x7f]|\\\\[\\x01-\\x09\\x0b\\x0c\\x0e-\\x7f])+)\\])"
Resources:
EventBridgeRule:
Type: AWS::Events::Rule
Properties:
Description: Push Trusted Advisor notifications to an SNS topic
EventPattern:
source:
- aws.trustedadvisor
detail-type:
- Trusted Advisor Check Item Refresh Notification
detail:
status:
- ERROR
- WARN
State: ENABLED
Targets:
- Id: EmailTarget
Arn: !Ref SNSTopic
# Failed events are retried up to 185 times within 24 hours by default
# This causes false drift detection :(
# DeadLetterConfig:
# Arn: !GetAtt DLQ.Arn
SNSTopic:
Type: AWS::SNS::Topic
Properties:
# Encrypt using the default SNS SSE key
# Key aliases: aws kms list-aliases
KmsMasterKeyId: alias/aws/sns
Subscription:
- Endpoint: !Ref Email
Protocol: email
TopicName: TrustedAdvisorNotifications
SNSTopicPolicy:
Type: AWS::SNS::TopicPolicy
Properties:
PolicyDocument:
Version: "2012-10-17"
Statement:
# Allow EventBridge to send email alerts using SNS
- Sid: AllowEventBridgeServicePrincipalWriteOnly
Effect: Allow
Principal:
Service: events.amazonaws.com
Action: sns:Publish
Resource: !Ref SNSTopic
Condition:
ArnEquals:
"aws:SourceArn": !GetAtt EventBridgeRule.Arn
- Sid: AllowSSLRequestsOnly
Effect: Deny
Principal: "*"
Action: sns:Publish
Resource: !Ref SNSTopic
Condition:
Bool:
"aws:SecureTransport": false
Topics:
- !Ref SNSTopic
DLQ:
Type: AWS::SQS::Queue
DeletionPolicy: Delete
UpdateReplacePolicy: Delete
Properties:
# Defaults:
# DelaySeconds (delay queue) = 0s (time before messages appear in the queue)
# MessageRetentionPeriod = 4 days (345,600s) (time messages stay in the queue)
# VisibilityTimeout = 30s (time before messages reappear in the queue)
MessageRetentionPeriod: 345600
ReceiveMessageWaitTimeSeconds: 5 # long polling if > 0
SqsManagedSseEnabled: true
DLQPolicy:
Type: AWS::SQS::QueuePolicy
Properties:
# Allow EventBridge to send messages to the DLQ
PolicyDocument:
Version: "2012-10-17"
Statement:
- Sid: AllowEventBridgeServicePrincipalWriteOnly
Effect: Allow
Principal:
Service: events.amazonaws.com
Action: sqs:SendMessage
Resource: !GetAtt DLQ.Arn
Condition:
ArnEquals:
"aws:SourceArn": !GetAtt EventBridgeRule.Arn
- Sid: AllowSSLRequestsOnly
Effect: Deny
Principal: "*"
Action: "sqs:*"
Resource: !GetAtt DLQ.Arn
Condition:
Bool:
"aws:SecureTransport": false
Queues:
- !Ref DLQ
Outputs:
DLQURL:
Description: DLQ URL for EventBridge
Value: !Ref DLQ
To help generate the EventPattern for Event Rules, you can go to the AWS console and start creating a rule for Trusted Advisor checks. Under Event pattern, set the desired properties and it will display the code in JSON, which can be translated to YAML in the CloudFormation template.
We don’t need a Lambda function for this, so we can set the target directly to SNS. Note that we can’t filter by specific checks without upgrading our Support plan, but this will still work for all the basic checks. Like many of the other services mentioned earlier, you can create a DLQ to catch errors, but I found that this can cause false drift detection, which causes the pipeline to fail. I believe this is a bug, but I can’t open a technical support case under the Basic Support plan. Oh well…
Compute Optimizer
Another free tool AWS provides is Compute Optimizer. It guides users on how to save money and improve performance for EC2 instances, Auto Scaling groups, EBS volumes, Lambda functions, and ECS services on Fargate. You must enable this feature in the AWS console. By default, it analyzes your CloudWatch metrics over the past 14 days to provide recommendations. These standard checks are free, but you can opt-in to enhanced checks to analyze 3 months of data for around 25₵ per resource per month, assuming they’re running for a full month. The only service we used in this project is Lambda and we haven’t spent a penny, so Compute Optimizer isn’t going to give us any recommendations because we’re already being cost effective!
Health
AWS Health (formerly known as AWS Personal Health Dashboard) is a service health page for all AWS services. You can use this to determine if any services are down or under maintenance in a particular region and how it will affect your apps. Like with Trusted Advisor, you can use EventBridge and SNS to subscribe to notifications whenever a health event is posted. The architecture diagram is the same, just replace Trusted Advisor with AWS Health. And the only difference in CloudFormation is the EventBridge rule resource:
EventBridgeRule:
Type: AWS::Events::Rule
Properties:
Description: Push AWS Health notifications to an SNS topic
EventPattern:
source:
- aws.health
detail-type:
- AWS Health Event
detail:
service:
# Services to get notified for health alerts
- CLOUDFORMATION
- S3
- CLOUDFRONT
- APIGATEWAY
- LAMBDA
- DYNAMODB
- IAM
- CODEBUILD
- CODEDEPLOY
- CODEPIPELINE
- KMS
- SQS
- ECR
- COGNITO
- ROUTE53
- CLOUDWATCH
- XRAY
State: ENABLED
Targets:
- Id: EmailTarget
Arn: !Ref SNSTopic
# Failed events are retried up to 185 times within 24 hours by default
# This causes false drift detection :(
# DeadLetterConfig:
# Arn: !GetAtt DLQ.Arn
I listed all the services that were relevant to my project, but you are free to add or remove as many services as you’d like. It’s all free! (Just make sure to double-check the console first to see how all the services are named under the event pattern. Usually, it’s all caps with no spaces or special characters.)
Well-Architected Tool
If you want another tool to check if you’re meeting AWS’s Well-Architected Framework (or are trying to study for the Solutions Architect exam), you can use, well, the AWS Well-Architected Tool. This is a free questionnaire for analyzing your app’s architecture. You create a workload and then answer questions that align with the 6 pillars of the Well-Architected Framework: operational excellence (are we monitoring our apps?), security (are our apps secure?), reliability (can our apps recover from disasters?), performance efficiency (are we optimizing performance on AWS?), cost optimization (we know this one 😉), and sustainability (we’re not burning fossil fuels, right?). You can even upload your architecture diagram for reference. Once you finish answering all the questions, you’ll get a list of high and medium risks for each pillar. Keep in mind that this tool is fully automated. No humans are reading your responses.
Admittedly, this tool can be much for a personal project. A lot of the questions revolve around things like teams, ticketing, monitoring, and security audits. It seems more tailored for someone higher up in a technical organization, like a director or a tech lead. If you’re on an Enterprise Support plan, you’ll have access to a Technical Account Manager (TAM) for architectural guidance from a human. (Tip: Discovery support allows the Well-Architected Tool to incorporate Trusted Advisor checks into its analysis. But I don’t recommend enabling this if your support plan doesn’t cover all the Trusted Advisor alerts. You will get a bunch of failed API logs in CloudTrail. For some reason, I couldn’t turn off this setting in the console or the CLI but was able to do so via boto3. Then I was able to delete the corresponding AWSServiceRoleForWellArchitected_Discovery role.)
Route 53 & CloudWatch
I’m combining both Route 53 and CloudWatch since they’re both used to monitor health checks for your apps and APIs. Although Route 53 is more commonly used as a DNS service, it can also serve as a health checker for any endpoint. And CloudWatch is used to monitor that health check to see if it’s unhealthy. CloudWatch can automatically monitor existing resources like CloudFront, API Gateway, and Lambda. Standard metrics from automatic dashboards are provided for you at no charge. You can browse these in the console by going to CloudWatch dashboards and clicking on the Automatic dashboards tab. We can create health checks for CloudFront on both the front end and back end, then tie it with a CloudWatch alarm that sends an SNS notification whenever the site or API goes down. On both templates, we can add the following to enable health checks:
Parameters:
Email:
Type: String
Description: The email address that will receive SNS notifications for the health check alarm
# Simple regex from https://stackoverflow.com/a/201378
AllowedPattern: "(?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|\"(?:[\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x21\\x23-\\x5b\\x5d-\\x7f]|\\\\[\\x01-\\x09\\x0b\\x0c\\x0e-\\x7f])*\")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\\[(?:(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9]))\\.){3}(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9])|[a-z0-9-]*[a-z0-9]:(?:[\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x21-\\x5a\\x53-\\x7f]|\\\\[\\x01-\\x09\\x0b\\x0c\\x0e-\\x7f])+)\\])"
Resources:
# Route 53
HealthCheck:
Type: AWS::Route53::HealthCheck
Properties:
HealthCheckConfig:
# 3 health checks every 30s by default
# Optional features: HTTPS, STR_MATCH, RequestInterval < 30, MeasureLatency
Type: HTTP
FullyQualifiedDomainName: !GetAtt CloudFrontDistribution.DomainName
Port: 80
ResourcePath: / # for the microservices, replace with /health
# CloudWatch
HealthCheckAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmDescription: Check the health of the AWS Shop site and send an email whenever it's unhealthy # update for the microservices
# In Route 53, monitor the health check status in the past 5 minutes (either 0 or 1).
# If this unitless value is < 1 at any point, transition to an ALARM state.
# https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/aws-services-cloudwatch-metrics.html
Namespace: AWS/Route53
MetricName: HealthCheckStatus
Dimensions:
- Name: HealthCheckId
Value: !Ref HealthCheck
Period: 300 # 5 minutes
EvaluationPeriods: 1
Statistic: Minimum
ComparisonOperator: LessThanThreshold
Threshold: 1
Unit: None
AlarmActions:
- !Ref SNSTopic
# SNS
SNSTopic:
Type: AWS::SNS::Topic
Properties:
# Encrypt using the default SNS SSE key
# Key aliases: aws kms list-aliases
KmsMasterKeyId: alias/aws/sns
Subscription:
- Endpoint: !Ref Email
Protocol: email
SNSTopicPolicy:
Type: AWS::SNS::TopicPolicy
Properties:
PolicyDocument:
Version: "2012-10-17"
Statement:
- Sid: AllowSSLRequestsOnly
Effect: Deny
Principal: "*"
Action: sns:Publish
Resource: !Ref SNSTopic
Condition:
Bool:
"aws:SecureTransport": false
Topics:
- !Ref SNSTopic
(Make sure to modify the Lambda function for the microservice to check for the “GET /health” route key, then return a 200 status.)
It’s very important to keep the free tier in mind. We can only have up to 50 health checks and 10 alarms within the free tier. Most importantly, the health check must not include any enhanced features, such as HTTPS, string matching, more frequent health checks (beyond every 30 seconds), and latency measurements. This is where I initially messed up. I used HTTPS and was greeted with a $1/month bill. Thankfully, the billing alarm that I created at the very beginning alerted me of this, so I only owed 8₵ over a few days (which got waived because the bill was so small). So, make sure to only do health checks on HTTP (port 80). The health check will still pass since it considers a 2xx or 3xx response to be successful. In this case, it will return a 301 response since CloudFront redirects all HTTP requests to HTTPS. The downside is that the health check can’t determine if the actual website is down since Route 53 doesn’t follow redirects.
To create the CloudWatch alarm, like with EventBridge, it’s easier if you start creating it in the console first. You get a nice graph to visualize what kind of metric you want to monitor. In this case, we want to check the HealthCheckStatus metric provided by Route 53. It reports 1 if the endpoint is healthy and 0 if it’s unhealthy. So, we need to trigger the alarm whenever this metric is 0. The way to set this up in CloudWatch can admittedly be confusing, but I tried to explain in the comments how to read out each property in the template. Since CloudWatch metrics are recorded over a period (in this case 5 minutes), we need to check if the minimum value at any point is < 1. This way, even if the route goes down briefly, we’ll still get notified since the minimum won’t be 1. The alarm action is tied to an SNS topic that will send us an email notification. The last piece of this metric is the HealthCheckId. This isn’t a standard property found on CloudFormation. It’s a dimension, which uniquely identifies a metric using a resource-specific value. This ID is important to reference the correct health check from Route 53. (We don’t want to use the AlarmIdentifier property in CloudFormation for Route 53. Basic health checks can’t have a region associated with them.)
Once everything is set up, you should receive an email confirming your SNS subscription and, hopefully, your Route 53 health checks report as healthy. But we’re not done with the free tier yet. Keep in mind what this health check is doing. Route 53 is pinging CloudFront every 30 seconds using the endpoint specified to see if it returns a 200 response. (If we assume CloudFront is caching the data, we don’t need to worry about any additional pings to S3 or API Gateway.) It’s also doing health checks across multiple regions to make sure the resource isn’t down due to a specific region. I kept my health checks simple by just returning an OK response, but you can add additional checks if the system is more complicated. Regardless, this is an API call made regularly to CloudFront. Remember when I commented out the logging property of the CloudFront distribution? If this was enabled, every health check would be logged to S3. Now CloudFront will aggregate all the logs every 15 minutes, but S3 objects are still being created, which requires making a PUT request. S3 will charge 1₵ for every 25K GET, SELECT, and other requests per month, but also for every 2K PUT, COPY, POST, & LIST requests per month. Let’s do the math. If this health check is running 24/7 every month, this will result in 4 calls/hour * 24 hours/day * 31 days/month (accounting for the longest months) = 2976 PUT requests/month. (Note that DELETE requests count in the 25K category, such as if we included a lifecycle rule that deletes objects after 1 day, so they’re not counted here.) We will have already exceeded our budget 2/3 into the month. We unfortunately can’t make the health checks less frequent to stay under the 2K limit. We can only do every 30s or every 10s.
This is also where I initially messed up. I used S3 Storage Lens to track the total storage and number of objects across all my S3 buckets. And soon enough, I found that the culprit was these CloudFront logs. But wait, I added a lifecycle rule like all the other buckets. Shouldn’t all the objects be deleted after 1 day? Not exactly. I originally had Object Lock set to 30 days, and this overrides the lifecycle rule to make sure all objects follow this retention policy. Even if I change the Object Lock settings to 1 day, the existing objects aren’t affected. So, I was stuck with thousands of log objects for a month until the Object Lock expired. Then the lifecycle rule took effect. Make sure to enable “Show versions” when browsing an S3 bucket to see if any delete markers are still present, since those still take up storage space. Now with those issues resolved, we can keep the health checks for the API and website, just without any CloudFront logs.
Conclusion
Phew! That was a lot. Any longer and I would’ve written a whole thesis! But I hope this inspires you to create something in AWS. I was intimidated at the beginning by the huge plethora of services. But as I showed, there’s plenty you can do in AWS under the free tier, especially with serverless tools. And with the right protection and automation, you can get alerted if you go over your budget and act immediately. Even with how long this article has been, it’s best to play around with AWS yourself to fully learn its services. There are plenty of gotchas in the world of AWS that catch newcomers off-guard, but as they say, you learn through failure. You can check out the GitHub repo at https://github.com/Abhiek187/aws-shop if you want to browse the full source code. Thanks for reading!
P.S. I added a new part to this series all about user authentication using Cognito. Please check it out here.