A Taste Test of AWS Services for the Software Quality Engineer
Six essential services for quality engineering in AWS ecosystems.
Written in collaboration with Taylor Wagner and Deborah Calderon
Have you ever wondered how to leverage AWS services to aid testing and improve software products? In this article, we’ll introduce six AWS services and describe how they can be used to help build quality software products in the cloud.
Along with each service’s introduction, we’ve provided examples of how they have been utilized by a quality engineer in a real world application, plus other AWS services beyond the six of focus. We encourage everyone to further research the services mentioned.
Unless otherwise noted, all images are provided by the authors.
Amazon CloudWatch and AWS CloudTrail
A major focus of quality engineering is foresight. When designing and implementing test automation, QEs must anticipate software outcomes. They validate expected outcomes as well as confirm that unexpected outcomes will yield failing tests.
Amazon CloudWatch streamlines monitoring resource utilization, application performance, and operational health by collecting metrics that can be configured to automatically respond to account actions by setting an alarm, maintaining logs, or creating events. Coupling Amazon CloudWatch Logs with AWS CloudTrail—a service that logs and saves a history of all API calls on the AWS account—allows AWS customers to maintain full visibility into their AWS account’s resources. For QE, any service that can provide insight and and information on an application is vital to testing efficiency.
To gain a better understanding, let’s walk through an example. Imagine you have an application that is ingesting a good amount of data and that data needs to run through multiple Lambda functions. The process begins with Lambda A function executing a query on Database A. Lambda A function will then take the results of the query and store a csv file into an S3 bucket. Finally, Lambda B function can then input the data from the file into Database B, where several calculations are set to occur in order to determine a number of metric values, one metric being a number of errors. Amazon CloudWatch Logs Insights metric values can evaluate AWS CloudTrail logs for specific values. A threshold can be set for the “number of errors” metric that can trigger an Amazon CloudWatch Alarm to inform team members of execution concerns.
Incorporating this automation greatly speeds the validation process because the logs from multiple Lambda functions can be reviewed at the same time and filtered for a specific use case without having to review thousands of logs that aren’t helpful. Through this process, you may even find bugs such as data not running through the whole process or data being duplicated, which allows the QE to create fail safes to decrease the number of errors. If you love a good data visualization, Amazon CloudWatch offers the capability to set up custom dashboards to view the metrics on a graph.
To get a better look at these AWS services, check out the documentation:
- AWS Lambda: https://aws.amazon.com/lambda/
- Amazon CloudWatch: https://aws.amazon.com/cloudwatch/
- AWS CloudTrail: https://aws.amazon.com/cloudtrail/
- Amazon S3: https://aws.amazon.com/s3/
AWS X-Ray
Quality engineers often work with complex programs with many slow API requests. When these requests build up, queuing can cause application lag, which in turn provides a poor experience for the user. This is where AWS X-Ray comes in! AWS X-Ray is a service that can be used to identify these slow requests in the inner workings of the program.
AWS X-Ray creates a visual mapping of the API requests as each one moves through the application, which helps QEs efficiently debug. AWS X-Ray collects data traces from the requests and responses. With the collection of those data traces, AWS X-Ray creates a detailed map that displays how many traces were received along with the time the traces took to process. From there, a QE can analyze the traces, find the ones that took the longest, and communicate the findings with the developers to improve the flow of the requests throughout the architecture.
In order to make analysis easier, AWS X-Ray also has a filtering system to view specific requests, or ones that were created within a certain timeframe. AWS X-Ray can go as deep as identifying the type of browser that traces come from as well. AWS X-Ray is also security conscious and offers some additional features that are helpful to maintain safe systems by encrypting the data for when sensitive information is being sent. This can be configured to fit any security needs requested from the team.
For QEs, these features can be beneficial to their work by making the debugging process significantly more efficient and helping with discovering issues such as network bottlenecks. It can be used instead of or in combination with Amazon CloudWatch if more detail is expected. Find more information about AWS X-Ray here: https://aws.amazon.com/xray/.
AWS Device Farm
Most companies require that their applications work on multiple browsers and devices, and this can mean a significant amount of time expended for quality engineers. This is where AWS Device Farm comes in as a solution to shorten the time it takes to test applications across a large number of required platforms.
AWS Device Farm has the ability to run custom, automated tests concurrently across multiple browsers and/or devices. During that process, it will log test failures that were caused by any of the browsers or devices. AWS Device Farm will also take screenshots and videos of the tests that were executed. The tests can also be automated to execute whenever needed, such as when commits are pushed. Engineers can also customize their testing environments settings, such as the Wi-Fi used, the locations found, and much more for precise testing. AWS Device Farm offers many options including supported frameworks, built-in tests, and manual testing.
How will these features be beneficial to a QE? The remote testing and customizable tests can lead to deeper insights on specific parts of the browsers or devices that you want to test. All the logs, including videos and screenshots, can be used to more accurately debug the issues caused in separate environments. And, finally, the automation can help ensure the devices consistently work with any changes made during development. Overall this service is instrumental for implementing specialized use-cases in applications that you, as a QE, may contribute to. More can be found about AWS Device Farm here: https://aws.amazon.com/device-farm/.
AWS Fault Injection Simulator
Cloud architectures and microservices are becoming increasingly complex. This complexity leads to hard-to-predict failures and interruptions. Chaos engineering is the disciplined approach to identifying failures before they become outages. AWS Fault Injection Simulator (FIS) is based on the principles of chaos engineering. This service allows for running controlled fault injection experiments based on actions, targets (AWS resources), and guardrails. These experiments assist users in identifying where the system may be deficient and improve those points proactively.
AWS FIS implementation is simple:
- Create an experiment via a pre-built template to generate the desired disruptions. The template also includes stop conditions so that an experiment will stop if it reaches a previously defined Amazon CloudWatch Alarm.
- Save the experiment. Once saved, run the experiment or include it in a pipeline.
- View the experiment results. Based on the initial AWS FIS setup, results may be monitored in the AWS FIS console or in other services, such as Amazon CloudWatch and AWS CloudTrail.
- If necessary, take actions to improve the performance and resiliency of the architecture.
Why should QEs care about AWS FIS or Chaos Engineering? Because engineers should be concerned not only for the code and application behavior, but for what their team is producing in its entirety. For example, an application on AWS infrastructure has been built for a client. QEs have verified the application works by manually testing and writing a robust test automation framework that now lives within the app. Providing additional checks related to fault tolerance, resiliency and application performance with FIS can also be built into the quality of the product resulting in a well-architected and tested application.
Find additional AWS FIS information here: https://aws.amazon.com/fis/.
Amazon Macie
Using machine learning (ML) and pattern matching, the data security service, Amazon Macie, can find and safeguard sensitive data. When initially enabled, Macie will kick off a scan of existing S3 buckets. This scan populates a dashboard that identifies buckets that are set to public, shared outside of a user’s organization, and are not encrypted. Users can then set up jobs to run scans for sensitive data by bucket(s). Jobs can be based on frequency (daily, weekly, monthly) and include/exclude managed identifiers, custom identifiers, and keywords.
There may be times when QEs must ensure Personal Identifiable Information (PII) or Protected Health Information (PHI) has been redacted from logs or files. Imagine an API that adds customer credit card payments to a digital wallet. One of the requirements is that all PII should be redacted from any Amazon CloudWatch Logs produced by the API. Amazon Macie would be an important security check to use in this situation — start by enabling Amazon Macie and scan existing data for PII. Next, add an Amazon CloudWatch Event to trigger a Lambda function that would copy any new logs to an Amazon S3 bucket. Finally, scan the new data in the S3 bucket for PII.
This is just a brief summary of how Amazon Macie can help QE’s explore ways of protecting sensitive data. Check out Amazon’s “Getting Started” tutorial and documentation for detailed steps on how to implement Amazon Macie with the following link: https://docs.aws.amazon.com/macie/latest/user/getting-started.html.
Next steps and implementation
While AWS offers a large range of services to meet the needs of the most diverse applications, we hope to have introduced and explained the benefits of a few services that are helpful in the interest of quality. We encourage all readers to do further research and try out these services on their own as well as looking into other services AWS offers. For easy practice, try checking out AWS sandbox options for free to minimal costs.
We are wishing our best in the pursuit of quality practices in the cloud!
