Ethereum Security Analysis Tools: An Introduction and Comparison
MythX vs Slither vs Securify
Security is hard. And it’s harder when you have user’s money at stake. Fortunately, the Ethereum ecosystem has matured over the last few years, and there are now various high-quality tools that will scan your Ethereum smart contracts for security vulnerabilities. (Spoiler: None of them can replace a real audit).
In this article, we’ll introduce three of the more popular security analysis tools—Slither, MythX, and Securify—and walk through the following steps for each so you can see how they compare:
- Setup process
- Running scans
- Analysis of results
As we’ll discuss in the last section, these tools are not exactly a 1:1 comparison and results should be taken with a grain of salt. Also note that all tools will be run on macOS Catalina for this article, so the steps and commands may differ depending on your operating system. This article will assume you have experience with Python 3, Git, and Solidity.
Tool Introductions
Let’s start with a high-level overview of each tool. As far I can tell, none of these tools support Vyper and they all require you to be using Solidity.
Slither
Slither was developed by the security organization Trail of Bits, and describes Slither as “a Solidity static analysis framework written in Python 3. It runs a suite of vulnerability detectors, prints visual information about contract details, and provides an API to easily write custom analyses. Slither enables developers to find vulnerabilities, enhance their code comprehension, and quickly prototype custom analyses.”
Slither also comes with a handful of other useful tools, such as a flattener, the ability to detect conformance to various ERCs such as ERC20 and ERC721, and checks for contracts using the delegatecall proxy pattern for upgradable contracts.
The Trail of Bits team also has written a few blog posts about the tool, which you can find here and here.
MythX
MythX—a ConsenSys project—can “automatically scan for security vulnerabilities in Ethereum and other EVM-based blockchain smart contracts. MythX’s comprehensive range of analysis techniques — including static analysis, dynamic analysis, and symbolic execution — can accurately detect security vulnerabilities to provide an in-depth analysis report. With a vibrant ecosystem of world-class integration partners that amplify developer productivity, MythX can be utilized in all phases of your project’s lifecycle. Our team of security experts is dedicated to the research and improvement of our tools and techniques used to fortify your code.”
The MythX team has their own blog, which can be found here, as well as a curated list of related resources and tools. It’s also worth noting that MythX is a full security service, and not just a scanning tool.
Securify
The last tool we’ll look at is Securify, developed by ChainSecurity. Securify supports 38 vulnerabilities, implements novel context-sensitive static analysis, and can only analyze contracts written in Solidity versions 0.5.8 or above.
Note that we’re looking at Securify v2.0 here, which is the successor to the original Securify.
Setup Process
We’ll assume you have a project ready to use, but if not feel free to clone the FakerDAO project to follow along. This is a good project for this article since it’s fairly simple, as the only contract used after deployment is Faker.sol in the contracts folder.
Slither
Slither is just one tool in Trail of Bits’ suite of Ethereum security tools, so if you’re interested in the rest you can check them out here. This means there are two ways to install Slither:
- Install the full suite of tools using Docker via
docker pull trailofbits/eth-security-toolbox - Install only Slither using Python
Either approach works great, but we’ll be using Python later anyway so we’ll use approach 2 here. Note: An earlier version of this article incorrectly stated the eth-security-toolbox is 15GB in size. I had an old version on my machine, hence the error. Thanks to Dan Guido and the Trail of Bits team for pointing this out.
In the project folder, create Python virtual environment using python3 -m venv venv and activate it using source ./venv/bin/activate. We can now install Slither by running pip install slither-analyzer. (Make sure to add venv to your .gitignore file).
And you’re done! Slither is now ready to use, but we’ll hold off on using it until we install the rest of the tools.
MythX
Recall that MythX is more than just a scanning tool, so the setup will be a bit more involved. First you’ll need to create an account by heading to the MythX website and clicking Sign Up.
Now let’s generate our API key and save it in a file called .env with the contents export MYTHX_API_KEY=yourApiKey. Make sure to use that exact name, since MythX will look for a variable with that name. Be sure this file does not get committed to a repository.
You’ll see theres a few options for running MythX—a VSCode plugin, a Truffle plugin, a Remix plugin, and a command line client. We’ll try to keep things tool-agnostic here so, with our Python virtual environment still active, let’s install the command line client with pip install mythx-cli.
That was a bit more involved than Slither since we needed to create an account and get an API key, but still pretty easy. You’ll see shortly why we needed to create an account.
Securify
This can be installed manually with Python, but the process was quite involved and I was unable to get Securify to run using that approach. Instead, we’ll use Docker to install this tool. If you’ve never used Docker, it lets you create or install what are called containers that can run on any machine, regardless of the operating system with all dependencies included. If you don’t know how to use Docker, that’s ok—just follow these installation instructions and later you’ll simply run the specified commands.
In the project root, clone the repository with git clone https://github.com/eth-sri/securify2.git, then cd into that directory and build the Docker container with sudo docker build -t securify . (don’t forget the period).
Once that finishes, we’re done installing all the tools and can start running some scans.
Running Scans
Ok, now it’s time for the fun stuff. Let’s learn how to run each of the tools, and afterwards we’ll compare the results. Make sure to cd back to the project root first.
Slither
If we want to run Slither on every contract in our project, it’s as simple as running slither . (note the dot). We only need to run it on one file, so instead we’ll run slither path/to/contract.sol , which in our example case becomes slither contracts/Faker.sol .
If you’re using the FakerDAO example, the first thing you might notice is that @ style imports are not supported. So you’ll need to change import statements from import “@openzeppelin/contracts-ethereum-package/contracts/math/SafeMath.sol”; to the more explicit import “node_modules/@openzeppelin/contracts-ethereum-package/contracts/math/SafeMath.sol”;.
Now try running the command again, and you’ll get the results printed to the terminal very quickly! The results of the various detectors are shown, where a detector is simply a category of bugs/vulnerabilities. The full list of supported detectors for Slither can be found here.
You should now see the results color-coded by severity, with high-severity detectors printed first, followed by the medium-severity and low-severity outputs.
They’re sorted by detector type, so after a section of results you’ll see a line starting with Reference: which contains a link to the documentation where you can learn more about that particular detector. And that’s just about it! You can see an example of the output below. After reading through each individual message and the corresponding documentation, you should be better informed to determine what changes you need to make to your code.
Slither also comes with an assortment of printers, which print different contract information. For example, a call graph can be printed with the call-graph printer, inheritance relations can be printed with the inheritance or inheritance-graph printers, and function IDs can be easily obtained with the function-id printer. Printers do not directly point you to security issues, but instead give you additional tools to help check things and make sure the code is structured as expected.
For example, we can get the contract’s call graph using slither contracts/Faker.sol — print call-graph. This will create a .dot file for each contract, and viewing it will show a call graph like this one:
It’s also worth noting that Trail of Bits offers a paid product called Crytic, which enables access to additional detectors, GitHub integration, and more. I haven’t used this tool so cannot comment on it, but it seems similar in Scope to the MythX platform, in the sense that it’s a full service as opposed to just a scanning tool. Without Crytic, you will need to manually version control any scan results that you want to keep and reference later.
MythX
MythX offers different plans, and each unlocks different depths of analysis. The paid plans offer 7-day free trials, so we’ll leverage that to take a look at three plans here.
Recall we configured an environment variable earlier, so first run source .env to set that. Now we can run mythx analyze to analyze every Solidity file, but here we’ll use mythx analyze contracts/Faker.sol to just analyze one file. (There’s a few other options you can specify as well which you can read about here). The first time this is run it can take quite a bit of time as dependencies are compiled, so please be patient! Subsequent runs will be faster.
Just like with Slither, @ style imports are not supported, so make sure to change the import paths to explicitly list the node_modules folder. Once complete, you’ll see an output like the one below. Be aware that by default this command will run MythX in Quick scan mode—soon we’ll look at the Standard and Deep scan modes.
Notice the warning that the free version states it does not detect all vulnerability types. However, I was unable to find a page containing a list of which vulnerabilities can and cannot be detected on each plan type. Though it wasn’t totally clear from the documentation, my understanding is that the MythX tiers provide access to functionality as follows:
- Free plan → Quick scan only
- Developer plan → Quick scan + Standard scan
- Professional plan → Quick scan + Standard scan + Deep scan
After upgrading to the Developer version and re-running, you’ll see that warning is removed, and because it was the same scan type there were no additional vulnerabilities found.
So we just ran a Quick scan, now let’s run the Standard and Deep scans. These modes take up to 5, 30, and 90 minutes respectively. To run a Standard scan, use the --mode flag, e.g. mythx analyze contracts/Faker.sol --mode standard.
Below are the results of the Standard and Deep scans.
For the FakerDAO contract we are using here, the Quick scan was nearly instant, the Standard scan took about 20 minutes, the Deep scan took about 45 minutes. And we can see the increasing tiers do in fact find more potential vulnerabilities. In the next section we’ll compare these results in more detail.
Recall that MythX is more than just a scanning tool. In the images above you’ll notice a URL to a dashboard containing a report. The API key you created is linked to your MythX account, so you can login anywhere and get a history of all analyses. This includes the date and time it was ran, the source files tested, and the vulnerabilities found. Below are three different pages you’ll find in your dashboard, shown for the Standard scan we ran above.
Securify
On to the last tool of the bunch. Securify currently only supports flat contracts, meaning our import statements are not supported. So to use Securify you’ll have to manually replace the imports with the full Solidity files instead. There are flattener tools that can do this for you, such as solpp, truffle flattener, solidity-flattener, another one called solidity-flattener, and even one that comes with Slither as mentioned above.
Once you’ve done that, you can run an analysis with sudo docker run -it -v <full/path/to/contracts>:/share securify /share/<contract.sol>. In our case, since we only want to run the Faker.sol contract we’ll use sudo docker run -it -v $(pwd)/contracts:/share securify /share/Faker.sol.
Here’s a quick overview of what that command does if you are unfamiliar with Docker:
sudo docker run: Run a docker container as an admin (I’m not sure why it needs admin access)-it: Run the container as an interactive process (this lets you enter commands into a terminal within the Docker container)-v $(pwd)/contracts:/share: Share our localcontractsdirectory, specified by the full filepath$(pwd)/contracts, with the Docker container’s/sharedirectory. The Securify tool lives within the container, so this lets the container read our local contract files.securify /share/Faker.sol: The command to run within the Docker container to execute the analysis
Now we wait for the analysis to run, and you should see something like the images below. The output here is a bit confusing because the colors don’t seem to correspond to the severity—notice how yellow messages are both Medium and High, while red messages are Info, Medium, High, and Critical. Furthermore, notice how the top two messages are both a Medium level Missing input validation, but they are shown in different colors. Be careful when reading the output here to avoid misinterpreting anything.
Analysis of Results
Before getting to the results, we should note that this article is far from comprehensive and there is plenty that can be expanded upon, such as the following:
- These tools are not exactly a fair 1:1 comparison. For example, MythX will run a static analysis, a symbolic analyzer, and a fuzzer. Slither, on the other hand, is only a static analysis tool, and the associated symbolic analyzer and fuzzer are broken out into separate tools called Manticore and Echidna, respectively.
- We did not dive too deep into Slither’s printers and did not cover its API.
- We did not touch on the other tools offered by Trail of Bits, such as its fuzz tester Echidna or symbolic analyzer Manticore.
- We did not look at the full suite of services offered by each tool, such at CI/CD integration, GitHub integration, or support.
- We did not account for the cost of the products.
Ok, now we’re ready to compare results!
We’ll use the SWC Registry — the Smart Contract Weakness Classification Registry — to compare against. This contains a list of potential contract weaknesses and vulnerabilities, along with explanations, references and test cases for each.
Below is a table listing the results of each scan. A few notes regarding this table:
- SWCs that were not found by any tool are hidden for clarity
- Not every tool mapped directly to this registry, so there’s parentheses explaining the specific issue as described by the tool.
- SWC IDs with an asterisk are made-up ones that I added because I could not find a good match for a tool’s output in the SWC Registry. The names of these “new” SWC IDs were pulled from the tool outputs.
- I did not manually verify the validity and accuracy of each report against the contract, so take this table with a grain of salt.
- Do not assume that the one with the highest count is the best.
Without further ado, here are the results!
So what can we take away here? A few thoughts that come to mind:
- No security tool is thorough enough to replace an audit! If there’s one thing you remember, let it be this.
- No single security tool caught everything: Only Slither mentioned the strict inequalities, only MythX mentioned timestamp dependence, and only Securify mentioned input sanitization and uninitialized state variables.
- Even when tools caught the same vulnerability category, they catch different instances and frequencies of it.
It’s also worth noting that the Securify outputs in particular are a bit odd and contain a few false positives. Aside from the confusing color-coding mentioned above, some of the outputs don’t quite make sense and there is no provided URL to find more information. For example, the Uninitialized State Variable message is failing to detect that those variables were initialized in the constructor. Similarly, it’s not clear what the concerns about Unrestricted write to storage are, and what inspection it’s looking for as the remedy.
Security scanning tools like these certainly have their place in the contract development process, and should always be used. Using multiple tools instead of relying just one seems to be the best approach for catching potential bugs, but no tool or combination of tools can replace a full audit.
If you must deploy to mainnet without an audit, please litter your website and code with warnings and consider capping the maximum amount of funds held by your contract.
Good luck!
Disclaimer: This article was written as part of a Gitcoin bounty funded by the MythX team in exchange for an objective comparison of the above tools.
