How Hydra Lab Empowers Microsoft Mobile Testing and Test Intelligence
Hydra Lab: Intelligent Cloud Testing Made Easy — microsoft/HydraLab (github.com)
What is Hydra Lab?
The following video could give you an overview of Hydra Lab on its functionality status and mission.
In other words, with Hydra Lab, you can build a private test cloud on your own which provides RESTful API service and support for DevOps system integration. In other words, it’s like you can have a private self-manageable “Firebase Test Lab”.
Why we need Hydra Lab
We open-sourced Hydra Lab under github.com/microsoft in 2022 Dec, and its open-source nature ensures better control and customization, removing the following concerns when it comes to the integration of cloud testing platform:
- [Credibility]: Is it okay to upload your debug app package to a 3rd party platform? Could we trust them? Is there a compliance requirement for using it?
- [Customization]: Does the 3rd party platform have the compatible real devices to run my test? Is cross-platform scenario supported?
- [Cost]: Is the pricing of those platforms affordable?
- [Community]: Could we plug in our devices to the cloud and share them across geos & teams?
Hydra Lab is a free, customizable, credible framework built on top of Appium, Espresso, Maestro, XCTest, etc. It has been geared with test integration glue code and scalable designs and test lifecycle definition for further support of new runners and test intelligence. The Hydra Lab team put consistent effort in contributing to the mission of “Intelligent Cloud Testing Made Easy” and designing “Your Intelligent Test Cloud”, we have built standardized CI and CD procedures in Azure DevOps with UT and API+E2E acceptance testing as quality gatekeepers and have shipped 27 stable releases so far.
How Hydra Lab powers testing in Microsoft
After integrating Hydra Lab into our procedures over the past two years, our in-house service now streamlines test automation for a wide variety of mobile products, including Phone Link, Link to Windows for Android and iOS, Office Union for Android, Teams Android, etc.
In Microsoft, we built a Hydra Lab center service leveraging the framework, and the collaboration model or the architect can be illustrated by the following diagram:
We deployed the Hydra Lab center docker image container in Azure as an app service, which expose AAD oauth based RESTful API to internal users, and our internal users can use Azure DevOps pipeline to define a task to queue Espresso or Appium test task to the service. And after receiving a new request with test specification, the center will choose the proper test agent and devices to run the test. The runner will take care of the test framework driver communication, and after the test is finished, the agent will send the result to center, and it will be visualized in center web. So our user will be able to view the test report.
And our user can refer to the Test agent setup · microsoft/HydraLab Wiki (github.com) wiki to onboard their machine and register it to the center.
Hydra Lab support seamless integration through our Hydra Lab Azure DevOps Plugin — Visual Studio Marketplace. We hope the this can automate the entire test process and act as a key component of entire DevOps workstream.
Get Started with Hydra Lab
If you want to try out the Hydra Lab service and play with it, we provide an uber docker image, as long as you have docker and ADB installed on your machine, simply type the following command:
docker run -p 9886:9886 ghcr.io/microsoft/hydra-lab-uber:latestThen you can open http://localhost:9886/portal, and start exploring the basic Hydra Lab features such as test device management, test task management, test runner, report viewer, test video navigator and so on.
For more details about onboarding, please refer to the GitHub wiki: Deploy Center Docker Container · microsoft/HydraLab Wiki.
Test + AI: integration with GPT/LLM
This year, we aim to incorporate Large Language Models (LLMs) into our platform, improving our capabilities in test result analysis, exploratory testing, and test case generation. Historically in software testing, the “monkey test” has been a well-received method. This involves the test driver initiating random operations on the screen or IO devices, mimicking the haphazard actions of a monkey, to assess the reliability of the app. The appeal of the monkey test lies in its simplicity — there’s no need for maintaining test cases — and its potential effectiveness as an initial smoke test. However, its main limitation is the coarse nature of random actions. Imagine if we could refine this method by introducing a more intelligent “monkey” that can truly comprehend the app and engage with it as a human would?
We introduce the SEE (Start-Extract-Evaluate) exploration model, which operates as follows: Initially, launch the application and configure the environment. Then, pinpoint and comprehend the interactive elements, transforming their descriptions into vectors or other formats that are manageable by computers. With this processed data, formulate decisions, implement actions, and assess the outcomes. Subsequently, revert to the extraction stage for the new state. This cyclical process draws parallels with the Markov Reward Process.
Post-exploration, we can utilize the outcome data as the basis for generating test cases. The challenge with black-box test case generation stems from the enormous context of a black box, which is difficult to analyze due to the volume of data. Thus, we aim to translate this into structured data that a machine learning model or a Large Language Model (LLM) can process. These innovations are underway at Hydra Lab.
Development Practice
The core contribution team is leveraging HydraLab/CONTRIBUTING.md as coding convention agreement. In general, we use PlantUML to describe our architecture and draw dev design diagram, define components and class relations. Therefore, we can easily review the design of a feature and comment on it within a PR.
We encourage UT in each new code change, and we have built BVT in CI and API acceptance testing in CD. If you are interested in contribution, we are ready to embrace you in the team.
