Serverless Data Driven Testing with DynamoDb and logging to TestRail.
- When you need to verify all possible states of a resource and the total permutations of those states is in the hundreds or thousands.
- Verifying reports can be challenging and time consuming. Often there are hundreds or even thousands of states (permutations) the data reported on can be in.
- By defining all permutations in some form of formatted data, such as .csv, QA Engineers can use that data as input to create: test data and test cases.
- These test cases can be stored in DynamoDb, executed by a lambda function and linked to a test management tool such as TestRail to record those results.
- This may sound like an ideal candidate for unit tests, however if the report is behind multiple microservices and some complex queries the unit tests needed for this can become unwieldy to manage and out of the traditional purview of QA.
But first! Reasons why serverless is high ROI for test automation:
- Easy to deploy. No need to build servers.
- Provides autonomy where QA can build the right tool for the right job.
- Low cost.
- Fast. Instead of smushing all your tests together you can have totally distinct processes for different test suites and scenarios.
- Tests run in the wild (not on your computer).
- Scalable — You can easily add new microservices for new features or extend an example like this to sync prod data into test suite.
- Plug into CI/CD pipeline fairly easily. For example link gitlab to API Gateway.
- Piece of cake to run on a schedule — just set a cron job on the lambda.
- Experimentation — You can experiment, learn and grow you and your colleagues’ skills for very cheap.
- Not monolithic. Due to the nature of serverless and microservices, QA Engineers will be compelled to build independent testing tools/services that are less tightly coupled than a traditional monolithic test automation architecture.
Things you’ll need:
- IAM role with proper policies: https://serverless.com/framework/docs/providers/aws/guide/iam/
- DynamoDb https://serverless.com/dynamodb/
- Lambda to execute tests https://serverless.com/framework/docs/providers/aws/guide/intro/ See events for reference on events which trigger lambda
- An AWS sdk (for SSM client if you API key for test rails, dynamo connection). https://aws.amazon.com/tools/. Personally used Boto3.
- Testrail client http://docs.gurock.com/testrail-api2/start (or whatever test reporting tool you use)
- A .csv parsing library
Walk through with a “service” account example.
- Capture all permutations in a .csv
- Say the report you need to verify accounts for is part of a POS system for any type of service which would have monthly charges (rentals, subscriptions etc.). This report essentially tells you yeses or nos (true or false) about these accounts.
- Lets imagine there are 8 true/false attributes about an account that the report cares about:
- Paid up to date?
- outstanding fine?
- High risk of losing account?
- High value account?
- VIP Customer?
- Customer email (visible only to certain user groups)
- Customer phone (visible only to certain user groups)
- Customer payment info (visible only to certain user groups)
This gives us 256 (2⁸) states an account can be in.
Paid up to date?,outstanding fine?,High risk of losing account?,High value account?,VIP Customer,Customer email,customer phone,customer payment,included,test_rail_idFALSE,TRUE,FALSE,FALSE,FALSE,TRUE,TRUE,FALSE
- This would go for 253 more rows..
… I skipped the test data creation portion because we’re using a make believe service as an example but the idea of using the .csv as an input for creating test would be similar to:
- A script would loop through each row and create an account based on the true/false values. For example, if Paid up to date? = true then process current month balance due or if outstanding fine? = true then invoice a rental or use an existing service to set a fine on the account.
- Each true/false permutation will need to become an attribute which identifies the resource you expect to see. So an account might have name of FALSE-TRUE-FALSE-FALSE-FALSE-TRUE-TRUE-FALSE.
2. Parse each row in the .csv to create each test case in TestRail.
- Parse the CSV to generate test case names built on the true/false permutations.
- TestRail API responds with the the test case ID. Append each ID to the CSV.
Paid up to date?,outstanding fine?,High risk of losing account?,High value account?,VIP Customer,Customer email,customer phone,customer payment,included,test_rail_idFALSE,TRUE,FALSE,FALSE,FALSE,TRUE,TRUE,FALSE,C16481
3. Create DynamoDb item
Your DynamoDb item should at the very least have the following keys:
- Test suite name (this is your partition key)
- TestRail test run ID
- Test cases (These contain the unique resource identifiers. See picture in next section)
4. Add the test cases to DynamoDb by parsing the CSV and creating a list of each test case with the case ID.
- The test_case_name (optional) and test_data are generated by combining the column values.
- The same item could provide an input to execute the test, time of execution and how the test was invoked (such as a cron job or CI/CD).
5. The execution.
There’s no code examples here because its mostly done using existing libraries and posting the code wouldn’t do much more than duplicate the docs.
- Deploy a lambda with an IAM role set up that can query and write to Dynamo.
- The lambda receives an event. (this can be from an SNS message, cron job, API Gateway, kinesis stream etc.).
- Create a test run in TestRail using the TestRail API.
- Add the IDs from the test case list in DynamoDb. This creates an empty test run of all those test cases.
- TestRail responds with the TestRun ID.
- Set the test_run_id in DynamoDb item to the ID from the TestRail response.
- Execute the request to get the data under test. (Might be a REST API or a lambda outputting to an SNS topic which your “QA lambda” would have as a trigger.)
- Upon receiving the response, grab the DynamoDb item.
- From the DynamoDb item, loop the list of the cases and search for each test_data value in the response. If the the included key = true for that test case then verify its there. If the included key = false then verify its not there.
- If there are failures, for each failure create a list of test case IDs which failed and then use the TestRail API to to fail the Test Run (which you have the ID for from the Dynamo item). lIf no failures then pass all the tests in the test run.
While this may seem excessive, as mentioned before, its important to consider the complexity of the backend logic which needs to account for all these possible permutations. For example, if using a relational DB, the implementation may be a highly complex query that then plugs into an equally complex backend.
Furthermore, if there are different groups and access levels for the reports, the permutations can go from 256 to 512 to 768 real quick. There may be important PII or security reasons that certain groups see certain things and this is highly valuable to test.
Extending to API automated testing:
- This approach can be extended to more traditional API tests, managing the test content (the url, request type, inputs, headers and expected response status code and/or content) in dynamo and driving test execution through a very minimal lambda function. I implemented a POC of this with the Swagger pet store API https://petstore.swagger.io/.
Extending to UI automated testing:
- One could do theoretically do the same with UI automation by creating pages as items in dynamo populated by the element locator and available actions. There would be a secondary set of items in dynamo which are the test steps themselves.
- This would also be executed by a minimal code base that only needs code to evaluate each dynamo item and execute actions on elements. The rest of the code would be stock code for whichever UI automation tool you’re using. POC coming maybe one day.
This testing approach was conceived with help, in no small part, from my colleagues at Logicworks.