Production environment out of your control or how to test blockchains

I would begin by saying that anyone who is into the tech industry and at least a little familiar with the blockchain will have a very good idea of ​​the basics of this technology. I think it is more correct to speak not about getting rid of delusions, but about some new knowledge, additional training in the development and testing of public blockchains (private blockchains are much closer to what programmers usually do).

I think it’s harder to get used to the fact that the production environment is completely out of the control of the developer. It is not easy to predict what will be the performance and simulate the necessary conditions. For me, this was the main difficulty when I wrote a first app for Ethereum two years ago. Everything went smooth for us: there was a well-established pipeline, the app was successfully launched in the virtual blockchain (which is included in the set of tools provided by Ethereum). We deployed the app to Testnet, and it worked fine there.

But when deployed to the main net, it becomes much more difficult to predict how the application will behave with the user. For example, there are numerous and unpredictable delays. Many serious problems are not found in the test environment, but only in production.

Suppose you have a contract in Ethereum, and this contract, among other things, dynamically creates wallets for users. But another contract of the same application may be launched before the creation of wallets begins, and this will happen due to the fact that the corresponding block will be obtained first. Thus, in front of you, there will be many exploitable traps that can often not be detected before production. Therefore, it is not enough for you to test the application, you need to predict in advance how it will behave in production.

For example, a very serious bug was recently discovered in a Parity multi-signature wallet. The wallet, in essence, was incapacitated due to the fact that an outsider got access to a function that was supposed to be private. Thus, they managed to disable the entire contract. This situation should not have arisen, but since it happened, nothing can be done, because this code has already appeared on thousands of machines, because of the node synchronization, and cannot be rolled back. Therefore, when thinking about architecture and testing, you should think about the worst possible scenarios and how to mitigate the possible damage in a situation where you can’t revoke the code.

Usually, when writing a website as a last measurement, you can always turn off your server. Of course, it will cost you money, but the opportunity is there. And in the case of a public blockchain, you cannot even do this: once the expanded code can no longer be revoked.

How about massive crypto exchanges exploits and thefts?

According to the Metcalfe effect, the network with the most users will be the most successful and popular in the future. Due to this effect, of all public blockchains, the most development is conducted on Ethereum, though EOS is definitely trying to catch up. Ethereum community created a lot of tools and libraries.

For example, there is a startup OpenZeppelin, which provides access to various libraries with the most basic functionality — for example, they have SafeMath library, which prevents overflows. Thus, they take care of some very simple things — for example, make sure that you include open source libraries for security in your project.

Being responsible for other people’s money is always hard, but the threat is not only theft. In the case of the most serious hacks, it was often not theft that took place, but the access to money was closed: this was the case, for example, in the case of the bug in the Parity wallet I already mentioned. Money does not go anywhere, but the owner can not get it. From a security point of view, this is a completely new threat that must also be borne in mind. You should not think about whether someone can steal the money, but about whether the system is safe, whether someone can cancel the contract.

When I talk about testing blockchains for security, I always try to convey the idea that vulnerabilities can be not only in smart contracts. They may also be caused by the fact that the developers are focusing entirely on the smart contract, and because of this, they lose sight of the application as a whole. Of course, this is less of a role if you simply provide a database or API. However, many people are developing web applications or mobile applications based on blockchains, and there may be very significant vulnerabilities due to the fact that developers do not think properly. For example, last year there was a hack of the Italian crypto exchange BitGrail, and the hackers stole more than $150M USD worth of Nano tokens.

On the backend, there was no problem interacting with the wallet. But I still can not believe that no one noticed the mistake made by the developer of the frontend. When a user withdraws money from his wallet, a check with the account balance was created on the client side, and because of this, any attacker with the most basic knowledge of Chrome tools could withdraw money, each time changing his balance to a higher amount. Many took advantage of this, and there was a major drain of funds.

Thus, trying to make the blockchain as safe as possible, developers often ignore elementary security, say, the frontend. And this is not the first vulnerability of the frontend, which the attackers took advantage of. Interestingly, the development teams in question are all fairly experienced; but in general, the process of software development life cycle is not going the way they plan. In addition, testers are experts on only a few projects with blockchains. Most of these projects are still experimental, and because of this, many have not yet had time to recruit the usual team for commercial development. Therefore, testing often suffers. This is unfortunate because many public blockchains have very good testing tools.

Does testing blockchain projects look similar to testing “normal” projects?

Rather yes, then no. The only difference is that you need to learn how to work with a production environment that is out of your control and poorly predictable. Obviously, this requires a very good knowledge of the technical side of things.

But if your work already requires you to automate tests, test APIs, test performance, then all these skills will be useful to you. The only difference is that you need to think about user expectations and user experience. It must be borne in mind that the general public is not yet familiar with the specifics of this technology, and it is necessary to inform users that nothing can be done with the stored information, and that a certain transaction can take an arbitrary amount of time. It helps shape user expectations.

Thus, the tester needs to think more about the quality of the product as a whole than about finding specific bugs. Here, I think, the skill needed is an ability to predict how the user will interact with the application — understanding of the UX concepts.

Therefore, I believe that developers of public blockchains should have experience in using these blockchains and an understanding of the basics of this system. Just reading about it is not enough. When I hold meetings and answer questions, I always demonstrate what I am saying in practice. Previously, I used the Bitcoin wallet for this purpose, but after I got deeper into the Ethereum and layer 2 scaling solutions, I switched to Ethereum related examples.

In order to learn, I ask you to install this wallet, show how the transaction is carried out and how to verify its implementation through the Block Explorer. Even such a five-minute experience helps to understand that you are dealing with a public and transparent network.

All other necessary skills are common to any kind of testing: the ability to understand code, use tools like Docker, Jenkins, and so on.

Who needs it? Crypto is dead

Well, I think we are now only at the beginning of the growth in demand for testers. Basically, this demand is not coming from startups, but from banks, exchanges, and various internal projects.

Of course, there is a possibility that this technology will be much less common than we are now expecting. But as an example, we can look at a cluster of interrelated technologies that have arisen around artificial intelligence and machine learning. There are constantly looking for new employees; Previously, it was mainly researchers and developers, now there is a demand, including for testers. I think the blockchain will evolve along a similar trajectory.

The idea of ​​a data structure that does not need an external security system is attractive, and people see how it can help reduce costs and how much new business models it can generate. Despite all the changes, I think that this basic principle will continue. Perhaps few are now ready to specialize in the blockchain, but at least you can add it to your toolkit. For example, working in big data, I only recently began to notice the demand for testers with experience with Hadoop. Projects with this technology existed before, but there were not looking for people with specialized experience. I believe that the blockchain will develop in the same way. Those experimental technologies that are widely known today will be the most promising in the labor market tomorrow.