Testing the Cryptorun smart contract: a tale of obsessive perfection
Better be careful when bugs could lead to theft
Some context to start with
You’ve probably read before about the whole Cryptorun challenge. This article is dedicated to the critical question of testing the smart contract powering the challenge. As a refresher, you should know that this smart contract is a fundraising where the release of funds is conditioned to an Oracle (my GPS). The full source code is available here, and this previous blog post might interest you as well. Be ready to face Blockchain’s unique software engineering challenges when it comes to security and testing!
Why should one test smart contracts?
One should always ensure good test coverage for any piece of software. You brush your teeth every morning. Elementary hygiene, right? Same for testing — testing is just part of the essential stuff you need to perform so that your code stays fit. Now, when it comes to Blockchain development, additional challenges arise that every crypto developer must be acutely aware of:
- You’re dealing with actual value flows (ETH for example), just like a bank dealing with fiat currency. This sets high exigences for your code if you want users to trust you and send any value to your contract
- Blockchain is immutable (#1), which means that it’ll be close to impossible to patch buggy contracts once they’re deployed. Even when a bug is caught, you can end up in situations where you just can’t prevent users from continuing to use your contract, leading to ever-increasing numbers of frustrated folks
- Blockchain is immutable (#2), which means that if a buggy transaction arises you won’t be able to revert it. Value might get lost or blocked irreversibly, or you might send ETH to a wrong address with no opportunity to retrieve your values
- Blockchain technology is still very young and lacks unified documentation (Ethereum went live only around 2015). As a consequence, no central knowledge base exists yet for developers to rely on as a checklist for safe development (there’s no such thing as the OWASP Top 10 for web apps, for example). So-called coding anti-patterns are not yet deeply rooted in developers brains when it comes to Blockchain development.
In the crypto world, no industry-wide security rules currently exist, and it is left to users to conduct their own due diligence when using a DApp
Reading this, you might be left wondering why on earth people from the crypto community are actually using any smart contracts / DApps at all. So many risks out there, with such a young technology! In the fiat currency world, software actors dealing with money are subject to stringent requirements and very close scrutiny from auditors so that the risk of bugs is limited. In the crypto world, no such security rules currently exist, and it is left to users to conduct their own due diligence when using a DApp. For large projects, security audits are most often performed using a mix of open-source community reviews, bounty programs, and private audits (e.g., with Zeppelin).
Smart contract analysis for Cryptorun
Let’s focus on the specifics of testing our Cryptorun smart contract. Good testing starts with an analysis of what we want our application to do and not to do. Once we’ve laid out a structure for the analysis, we can start performing individual tests for all the good and bad cases identified. Given the smart contract will be dealing with real value and engage the reputation of BeCode towards several stakeholders, we have the green light to be super paranoiac in the testing!
Setting the scene: stakeholders
Let’s first define the different stakeholders interacting with the smart contract:
- BeCode: the non-profit that will be the ultimate beneficiary of the donations
- Thomas: the coder and deployer of the contract, who will also run the challenge (that’s me!)
- Donors: the friendly folks (including you, hopefully) that will accept to contribute a few Ethers for BeCode
- Hacker: an hypothetical person with malicious intents
- Oracle: the bridge from the smart contract to Thomas’ GPS
Setting the scene: smart contract lifecycle
Let’s first understand the lifecycle of the smart contract along its states. This type of application is known as a state machine: it moves from one state to the next, with specific actions being possible only at certain states. The allowed states are the following:
- Ongoing: the contract is open to all donations, while BeCode and Thomas publicise the challenge.
- Accomplished: the smart contract learned from the Oracle that Thomas successfully ran the challenge. BeCode can thus withdraw all donations. No additional donations are possible. This is the end state of the smart contract if the challenge is not successful.
- Failed: the smart contract learned from the Oracle that Thomas did not manage to run the challenge. Donors can withdraw their donations. This is the end state of the smart contract if the challenge is not successful.
The state transitions are triggered via a call to the Oracle (leading to accomplished or failed state), and ultimately by BeCode to close the challenge after donations have been withdrawn.
Setting the scene: what could go wrong?
Now that we know what we want our smart contract to do, we can list the different issues that could arise and that we have to pressure-test against.
- Donations block: it is impossible for a donor to send Ether to the contracts when the challenge is ongoing
- Value lock: it is impossible for BeCode to withdraw all the Ether after a successful challenge, or for donors to withdraw their individual donations after a failed challenge
- Theft: it is possible for a hacker to withdraw all the Ether after a successful challenge, or to withdraw individual donations after a failed challenge
- Cheating: it is possible for BeCode to withdraw the donations if the challenge is not in the accomplished state
- Failing update: after a query to the Oracle, the contract status does not correctly update to the challenge status
- Unwanted Oracle query: it is possible for a hacker to query the Oracle to refresh the challenge state (this query consumes gas and hence depletes the donations)
Summarising the attack scenarios
With the scene now set, we can summarise in a nice table what we should test for and against:
You can see that the whole testing exercise will thus mostly consist of ensuring that unauthorised actions are blocked. The so-called happy path of authorised actions is relatively straightforward compared to the universe of unhappy possibilities.
Doing things elegantly: tests automation
As you can imagine from the above structuring work, it would be quite tedious to test all these scenarios manually. This would become painfully time-consuming and error-prone, especially so as the smart contract needs to be tested after each code modification to ensure no breaking changes were introduced. Luckily enough, Blockchain development like other coding environments does benefit from a suite of nice testing tool that allow us to automate the testing.
Setting up a Blockchain test environment with Ganache and Truffle
To be able to test our smart contract, we need to deploy it on a Blockchain. For obvious speed and cost reasons, a local test Blockchain is the preferred approach when developing Ethereum Smart Contract. For this we use Ganache (previously known as TestRPC), which is now part of Truffle, the popular development framework for Ethereum.
Unit testing the contract: Mocha & Chai to the rescue
When it comes to writing the actual test suite, we must conform with the standard way of asserting programs correctness in Solidity (the language used to code the smart contract). To this end, we can luckily use Mocha & Chai, standard javascript testing tools. Here’s a typical template for a unit test:
See? It really is that simple, and reuses good ol’ JS you’re likely familiar with.
The special case of the Oracle: using Ethereum bridge
Before we can run any tests, we need to account for the peculiar nature of our smart contract. Indeed, it is not only connected to the Blockchain, but is also in contact with an Oracle (via Oraclize, a wonderful tool). We must reflect this by allowing it to connect to this Oracle for testing purposes.
Luckily enough, the folks at Oracle have been so kind as to provide us with a nice helper tool for this: Ethereum Bridge. The setup is not so easy (find it all here in the readme of Cryptorun), but once you’re done you can use the Oraclize methods seamlessly.
Dealing with the Oracle — when tests modify their environment
And now for the really tricky part of testing the Cryptorun smart contract! Via Oraclize, our smart contract is connected to the GPS Oracle (an AWS Lambda function proxying Strava’s API). When we test the smart contract, we need to have it react to changes from the Oracle, by querying the Oracle’s API.
In standard software engineering, you would simply catch the call to the API before it flies on the web, and proxy it with a mock response depending on your testing needs (like we do for WebMock in ruby, for example).
Here this approach is not possible. Indeed, the Ethereum bridge connects to Oraclize, which in turn connects to the Oracle. You cannot simply mock the call between Oraclize and Oracle.
Therefore here I devised a cool little trick using a test AWS Lambda, that can be updated remotely from the CLI via a function wrapped into a test helper function. For this we use a Lambda ENV variable representing the challenge status, that can easily be updated remotely. See for yourself:
So we just need to use a test endpoint for the Oracle in our smart contract code, and thanks to this trick we can update the Oracle status remotely with no difficulty. This allows for example the kind of assertions like this:
How cool is that? Of course, given that it modifies its environment, this test suite is highly sensitive to race conditions (concurrency). Running tests in parallel results in multiple Oracle updates that can lead to thorny conflicts. But for the scale of our tests this is completely acceptable.
The test suite: 23 assertions to success
Now we’re finally ready: we know what to test and how to test it. You can find the full test suite here. It contains 23 scenarios / assertions, and counts ~500 lines of codes, which is approximately 4x more lines than the smart contract itself — a healthy ratio if you’re obsessed with perfection like I am for these bits of code. With that being done I can finally go running!
Next steps
Did you enjoy reading this? Then certainly it would be nice if you could provide feedback to pressure-test the current contract implementation. Also, please do visit Cryptorun.brussels to participate in the challenge: ETH donations for BeCode are welcome, as well as your muscles to run with me!
