An Introduction to Vega: Observe.AI’s Test Automation-as-a-Service

In a typical distributed systems architecture, with multiple microservices and components, code changes and deployments are very frequent. The same amount of time and resources are required to validate the feature and certify it.

And testing a fully functional product due to a new change in a service/component in a few hours, without impacting other dependent features, is just as important as the feature you’re delivering. But rapidly changing requirements and integrations with different services, apps, or products make it a likely candidate for defects.

With changing product development strategies, a distinct approach to test automation is also required.

That’s why we developed a distributed test execution framework called Vega. Vega is a microservices-based, on-demand test automation-as-a-service solution that addresses all the above-stated problems with the simple idea of on-demand test execution from anywhere.

Vega provides an easy-to-use interface to trigger the automation run. Vega processes the request right from building a runnable suite to creating a test runner and run the selected and dependent test to ensure everything works seamlessly.

Vega helps running automated tests on-demand to help you find potential bugs. Vega behaves as a test automation service as well as an automation framework. It then generates the response, updates the test management platform, and reports any regressions that may surface from running the tests.

How Vega Works

Vega is a restful, web service-based test automation framework built on top of spring boot that accepts requests drawn from any source that you provide. In the background, a post request handler puts the incoming requests to an SQS queue, an available worker node picks the request and passes it to the test runner for further processing. Test runner then converts the request to executable TestNg based suite and triggers the execution.

Vega also supports the execution of asynchronous tests (tests that validate multiple systems) by supporting record and replay mechanisms. It records all the completed steps and puts the test in a waiting queue (since the underlying system components take time to process the request) to wait for the completion of tasks and periodically picks it from the queue and resumes it from the last recorded point, therefore, saving resources and calculation time.

This system design was motivated primarily by problems that developers have while testing an omnichannel product.

Parallelism and Notification

Vega supports the running of tests in parallel and notifications to Slack by simply providing the details in configuration attributes. Users can specify the number of parallel threads (max up to 5) to fasten the test execution.

Reporting and Logging

Vega is fully integrated with Test Rail for updating the status of test execution for easy reporting and actionable insights. Its report API can also be used to integrate it with any other test management tool.

Vega’s GET report API includes the logs generated by tests, it is also integrated with Loggly for the logs analysis.

Vega Sequence Diagram

POST Handler: End-users or on-demand test runners will call this API with params like features, parallelism configuration, etc. This is an async API, the response of which is a unique suite ID and status URL.

DB: Once the POST request is received Vega makes an entry in MongoDB with the unique suite ID and other fields like request body, the status of the suite, creation time, etc.

Queue: Requests are kept in a queue. This is how Vega achieves async processing.

Test runner: TestRunner is responsible for reading the requests from the queue, forming the TestNg supported suite, and start the execution of test automation.

Invoking an automation test suite

Tests can be easily triggered by making a POST API call. Eg.

curl --location --request POST 'https://testdns.observe.ai/test' \
--header 'Content-Type: application/json' \
--data-raw '{
    "configuration": {
        "sendSlackNotification": true
    },
    "features": [
        "P0", "SANITY"
    ]
}'

The above request returns an immediate response with a unique suite id and status URL.

{
    "testSuiteID": "e05adfdb-23fd-4d95-b012-55e50543edab",
    "currentStatus": "CREATED",
    "creationDate": "2021-09-28T09:49:50.160+0000",
    "requestedBy": "test-user@oai",
    "statusURL": "http://testdns.observe.ai/test/e05adfdb-23fd-4d95-b012-55e50543adad"
}

Users can call the status URL and get the status of their automation suite in real-time. The response contains the overall suite status, the status of individual test cases along logs for easier debugging.

{
 "testSuiteID": "6152e53ea93882542b1ea489",
 "currentStatus": "COMPLETED",
 "creationDate": "2021-09-28T09:49:50.160+0000",
 "requestedBy": "test.user@oai",
 "totalPassed": 4,
 "totalFailed": 0,
 "totalSkipped": 0,
 "totalWaiting": 0,
 "testExecutionDetails": [{
  "id": "6152e54349fbfb3431b603f1",
  "rowCreated": "2021-09-28T09:49:55.643+0000",
  "rowUpdated": "2021-09-28T09:49:56.345+0000",
  "testClassName": "com.observeai.vega.testcases.auththinclient.AuthThinClientTest",
  "testMethodName": "restrictedAccessAuthTest",
  "testMethodParameters": "role,select,IAM_TICKET,user,1,thinclient/hello,hello",
  "testCaseId": null,
  "testStatus": "PASSED",
  "testIdentifier": "e05adfdb-23fd-4d95-b012-55e50543edab::com.observeai.vega.testcases.auththinclient.AuthThinClientTest::restrictedAccessAuthTest::role,select,IAM_TICKET,user,1,thinclient/hello,hello",
  "logs": [
   "Generated authContext header is:: eJxNjcsKwjAURP9l1t24zU6kSPGx0LiSIjW9aDEm4SYXkdJ/N4EqLucwc2aEROKmh8ICFTpjvLj0y2kwD0r6HSiDZrm76Ga1qXVpSrrPvBgyEbZQTqytEPyL+PQvZope2NDR+EBr9hK2Q0xQ5xG3kvbdk77ra5dPXV+7/jCvYi6CvSWlIlkyCW2FWFRZXzimdvoAW/NEXA==",
   "Validating decompression is successful!",
   "Calling application endpoint with auth-context header",
   "Request metadata : [",
   "URI  :https://testdns.observe.ai/thinclient/hello",
   "Method   :GET",
   "Authorization Header   :[]",
   "Request Body   : ]",
   "Response metadata : [",
   "Status code   :200 OK",
   "Response Body   :{\"message\":\"hello\"} ]",
   ":::::::::::::::::: PASSED - com.observeai.vega.testcases.auththinclient.AuthThinClientTest # restrictedAccessAuthTest ::::::::::::::::::"
  ],
  "testSuiteId": "6152e53ea93882542b1ea489"
 }]
}

Deployment of Vega

From the test developer’s side, only a single command is required to push code to Vega hosted service: git push.

What goes on behind the scenes is more sophisticated. Every time a developer pushes commits to our master git repository, a post-receive hook starts the Jenkins build, which creates a docker image in amazon ECR.

A backend cd deployment tool called Harness watches this build, and when build is successful, it triggers the deployment pipeline and deploys it to Amazon ECS. This ensures that Vega is with the latest code all the time.

For most revisions, the time between git push and release is about 10 minutes.