Securing the Money God

Published in

Reflexer

10 min readJul 6, 2021

At Reflexer, saying we take security seriously is an understatement. Although, like most of the teams in the space, we are small and do not have anyone specifically in charge of security, ever since we started building the Money God we have directed a big chunk of our resources into securing the system.

In this article we will shed some light on everything we do in regards to the Money God's security, aiming to inform both our community and fellow builders on our approach.

The scope here is limited to smart contracts. We will not be touching on opsec nor on securing other parts of the infrastructure (maybe we will in a future article).

Security 101

This is an intro on security for non technical readers. If you know this stuff, you can skip to the next section.

Security (and risk management in general) is all about layering. The more sensitive/valuable the asset you are trying to preserve, the more safeguards you should implement around it.

If you take a look at the Dover Castle, pictured on the beginning of the article, you will immediately notice its layers. Castles were built in a way that protects the owner and his riches from external parties. In the picture you will notice that the castle in question is sitting on the top of a hill (vantage point against invaders) and has two layers of walls with several towers (used for both observation and defence in an attack). The king, or owner of the castle, would surely sit safely in the core of the middle tower, protected by several layers of walls, warriors and guards.

The medieval castle example has been beaten to death on infosec articles for a reason. It makes it very easy to visualize the security in depth approach. This approach still stands up to this day. Just pay attention next time you walk in a bank branch and you will be able to see several safeguards right away: cameras, armed guards, metal detectors, a somewhat fortified building among others. The bulk of the funds will be in another hidden layer, where only a few select branch employees will have access to.

The gist of it is that no security safeguard is a silver bullet. A motivated attacker will be able to breach at least some of them. Layering ensures that even if some safeguards fail, the other ones will still stand.

System security follows the exact same pattern. Corporations will deploy systems using many network layers, each being separated by a firewall. Sensitive databases will often be on inner layers and will be accessible only by relevant systems (and personnel). Each of the layers can also have an IPS or IDS deployed. Most often, a honeypot will also be in place (a network layer with useless data in it, that will be easily accessible by outsiders and closely monitored by the security team providing insights on prospective attackers). Sensitive data will first be encrypted and then stored on the database. Backups will be performed continuously and stored offsite somewhere safe, allowing corporation to roll back their systems in case of compromise. The list goes on, as a large corporation it will include dozens or even hundreds of preventive and detective/corrective safeguards.

Web3

Now, when we get to Web3 systems, we have to rethink the approach around security right from the beginning. While in the Web3 context some of the previous safeguards are no longer relevant (you won't find anyone worried about uptime), it also rendered some of them completely useless.

The core of web3 applications (smart contracts) run on an open, permissionless network (one can no longer just prevent access to parts of the system using firewalls). Backups are also a thing of the past, data integrity is ensured by the protocol itself (good thing!) and the data in a widely distributed network will be available in the thousands of nodes connected to it. Even though you could also backup the state of smart contracts, there is no way you'd be able to restore them if something goes wrong.

This is what makes smart contract development so unforgiving. If you leave just one bug behind, it will surely be found (and abused) by a motivated actor. Once the attacker drains the protocol there is very little you can do (it's also easier for attackers to flee off with the funds, but this is out of the scope for this article).

Monitoring for attacks is also ineffective, since flashloans allow for attacks to be performed within one transaction. Once one notices the problem, the damage will have already been made and will be irreversible.

This is why it feels natural to compare smart contract development to hardware development. You basically need to rely on preventive controls, as any incidents after launch will be disastrous and will not allow for remediation. On hardware development this can mean a multi million dollar recall. In the case of smart contracts a multi million dollar theft.

We can still draw examples from legacy software. Notably, life support systems and systems on which life depends on (such as systems used in aircrafts). These systems will go through strict processes ensuring they behave as intended in all situations, as well as through certification processes that ensure failures are non fatal. Even with all these checks in place, some systems still have failures down the line.

What do we do at Reflexer?

Now, I’d like to switch gears and present most of the safeguards we continuously perform when developing RAI and also while touching production. We are of course trying to improve our current workflow and we would be more than happy to hear your suggestions!

Keep in mind there are several different ways of approaching smart contract security. If you are a fellow builder you should not take this as the definitive recipe for safe smart contracts, but use it as inspiration and implement the layers you feel are important (there are many safeguards not listed here, that could make more sense to your application).

The resources you have at your disposal (including dev time) should be used with care, focusing on the most critical of the controls you aim to implement. While we all would like to follow the whole process outlined on the Nasa Systems Engineering Handbook (the de-facto standard for development of mission critical applications), we do not have the resources our colleagues from the aerospace industry have at their disposal. This is fine though, as long as you focus your resources to the controls that mitigate the most risk.

The Fork

As you might know by now, Rai is a fork of Multi-Collateral DAI. We were very fortunate of having this starting point, as MCD was developed by some of the brightest minds in the space. Not only that, but the system was formally verified, audited and went through a very generous bug bounty before hitting production. It was then battle tested for more than a year before RAI hit mainnet.

The changes we made include all features you do not see on MCD (the PID controller, liquidation protection, different auction houses, and others), and we also translated the cryptic terminology to readable English. We feel the variable name change improves security, as it allows for one to understand the system much quicker, without requiring familiarity with an unusual terminology.

Unit Testing

Unit tests are the base layer for securing smart contracts. Often neglected, they will ensure one irons out all surface bugs. They are the base layer, not only because they are effective, but also because of the benefits they offer compared to the amount of time needed to build them.

Coders coming from other industries might find the testing exhaustive and repetitive, but we're building unfixable applications, so we better err on the side of caution.

We aim for 100% coverage on our testing and test all possible paths, requires and state changes.

Fuzzing

We fuzz A LOT. Fuzzing, for non technical readers, is the process of trying a given function with a really large amount of parameters to ensure it performs as intended. It is useful for finding edge cases, and also highlighting the bounds at which the application works. As opposed to unit tests that emulate only a handful of scenarios, fuzzing will try thousands of different parameter combinations trying to find situations where things go wrong.

Lately we've been using the new fuzz feature on Dapp Tools within our unit tests, and for full on fuzzing our tool of choice is Echidna by Trail of Bits.

Guarded Launch

A guarded launch is a pre deployment of the system on mainnet, with the aim of letting it run in the wild with a TVL cap. It puts the system on a stress test impossible to replicate on any test (real funds + black hats), but with a limited impact in case anything goes wrong.

We ran Proto Rai on mainnet for almost 3 months. During the period 612K worth of ETH were deposited in the contract. The system was settled back in January 2021, with no incidents taking place while it ran.

Audits

Our industry is very audit-centric. Users tend to see an audit as a stamp of approval for a given project, and audit reports are often used as promotion tools.

Part of the community have been bashing audits recently, due to their high price and the number of incidents reported on audited contracts.

Both couldn't be further from the truth. Audits should be approached as expert reviews, and not as the single measure one takes to secure smart contracts. They cannot guarantee an application contains no bugs, but are an invaluable tool in highlighting things the developer did not foresee.

It's also unfair to blame audit companies for exploits. Ask any auditor and you will hear stories about code sent for audits without unit tests, in rare cases even code that does not even compile without errors. In these cases the effectiveness of an audit will be reduced, as auditors of course spend time with things that would be easily highlighted by unit tests. Likewise, if the audit report listed several critical bugs, it should be taken as a sign that the smart contracts need more work. Generally the more critical bugs are presented in the report, the more likely unfound bugs are still lurking within the codebase.

We are strong believers in the effectiveness of audits. Our core contracts were reviewed by OpenZeppelin (and also informally reviewed by the mighty samczsun, thanks Sam!) and we have had several audits with Solidified and Quantstamp for our periphery contracts (Quantstamp is auditing our staking contracts as I write this). Check out the reports here.

Bug bounty

Some in our industry doubt the effectiveness of bug bounties, but they can be a great tool for finding things the other layers failed to. As a general rule of thumb, the more sets of eyes that spend time scanning your code, the better, and well funded bug bounties excel at getting attention.

One of the best examples of this is MCD. The contracts were well tested from the start, most of them formally verified, and then audited by a reputable company.

The contracts were then submitted to hackerOne and the bug hunter lucash-dev found a critical bug in one of them. If the bug bounty was skipped, the bug could've reached production, and cause irreparable damage.

We set up our bug bounty back when Proto Rai was first deployed. It has been continuously running ever since. Details here.

Testnet/Mainnet fork testing

We deploy all our contracts to the Kovan network for testing purposes and for seeing them in action for longer timeframes.

Recently, we also started testing all the changes we push to production on a mainnet fork. Deploying contracts within complex systems means you have to perform several actions to ensure upgrades work as intended. This will include setting contracts up, authorizing them where it's needed, giving allowances and more. Failing to perform one of these might cause a perfect contract to fail when executed in production.

Risk based approach

This is the last item of the list, and it's not a layer like the others, but instead a guide on how and when to use each layer.

Even the largest corporations out there will have limited resources focused on security. Taking a risk based approach will ensure that more time is spent where the highest risks are, making for a more efficient use of resources.

A practical example are the audits. We do not audit every contract we produce. Audits are the most expensive control on the list, and therefore we save them for high complexity/high risk contracts.

As a practical example, we perform all the steps previously described apart from auditing when it comes to smart contracts that automate parameter setting inside the protocol. These contracts are low complexity, do not hold any funds and in the case everything goes wrong and they fail, they wont harm the system in a meaningful way and can easily be replaced.

Now, in the case of contracts that are highly complex and hold user funds (i.e: staking or an auction house), we will perform all the security measures and then on top of that, an audit.

Conclusion

These checks and processes are what we apply on a day to day basis while we build smart contract infrastructure. We haven't had any incidents up to now and these controls certainly have helped us in avoiding surprises.

We also push ourselves to stay diligent, and never get too comfortable when it comes to security. Continuously improving the process is always a priority.

If you have feedback, critique or any suggestion feel free to ping us on discord.