Storycheck for Web3 Apps: Converting Text into Test Code
A few weeks ago, we teased how it could be possible to convert user stories into test code automatically using natural language processing.
Now, we want to show that it is possible — as we release our beta of Storycheck in a public playground. You can access it here:
What is Storycheck for Web3 Apps?
StoryCheck for Web3 apps is for projects based on Ethereum. It is an experimental app testing playground as well as an API served via Gradio on port 7860.
It takes a markdown formatted user stories with steps written in natural language as input, parses the text, and executes the steps in a virtual web browser (via Playwright) closely emulating the actions of a real user. Storycheck uses RefExp GPT to predict UI element coordinates given a referring expression.
Note: Storycheck is currently most reliable for testing the UI of smartphones and tablets.
Example User Story Input
# Creating a new DAO LLC on Arbitrum One via SporosDAO.xyz
## Prerequisites
- Chain
- Id 42161
- Block 82121263
## User Steps
1. Browse to https://app.sporosdao.xyz/
2. Click on Connect Wallet in the top right corner
3. Click on circle icon in the top right corner
4. Click on Create a new company button
5. Click on Go! right arrow button left of Available
6. Select On-chain name text field
7. Type Test DAO
8. Select Token Symbol text field
9. Type TDO
10. Click on Continue button in the bottom right corner
11. Select Address text field above Enter the wallet address
12. Type 0x5389199D5168174FA177908685FbD52A7138Ed1a
13. Select text field below Initial Tokens
14. Type 1200
15. Select text field under Email
16. Type test@email.com
17. Scroll down
18. Click on Continue button
19. Click on Continue button right of Back button
20. Scroll up
21. Click on the checkbox left of Agree
22. Scroll down
23. Click on Continue button
24. Scroll up
25. Click on Deploy Now button
## Expected Results
- Wallet transactions match snapshotPrerequisites Section
The prerequisites section normally includes ChainId and Block parameters, which allow the test to execute from a deterministic blockchain state, which respectively allows for predictable results.
User Story Section
The format of user steps in this section resembles the HOWTO documentation of a web3 app. Teams may use the same markdown in their documentation (e.g. gitbook, notion, docusauros) and execute it with StoryCheck to make sure that the latest web app behavior is in sync with docs.
Expected Results Section
The Expected Results section currently implements a default transaction snapshot check similar to jest snapshot matching. The first time a test is run, all write transactions going through window.ethereum are recorded and saved. Subsequent runs must match the write transaction. If there is a mismatch, then one of two changes took place in the UI under test:
- Developers changed the frontend code in a significant way. This warrants a careful code review and update of the user stories.
- There is malicious injected code that changes the behavior of the app. A big red alert is in order! App infrastructure is compromised: hosting providers, third party libraries, or build tools.
- There is a bug in some of the third party dependencies that affects UI behavior. Developer attention required to track down and fix the root cause.
High Level Design
How to Contribute
If you’re interested in contributing, please join our discord and/or start a new discussion in github.
You can find everything you need to get started in our public repo, including access to the pre-configured Gitpod and instructions for how to run locally or in another dev environment.
About GuardianUI
GuardianUI is the testing and monitoring platform for web3 frontends. Our SaaS platform integrates and automates end-to-end testing, application performance monitoring for web3 critical paths, and real-time alerting and observability to ensure deployed applications create the expected smart contract interactions for users.
