The Last Introduction to Web3 / Ethereum For Developers
Over the past couple of months, we’ve been onboarding at least a dozen web3 developers to our team. In order to get them started, we’ve written numerous demos, built prototypes, and even initiated an internal Confluent “ledger academy”, which gets lost in so many details you can spend an entire week parsing through it. Fact is: if you’re looking to get started, the interwebs already contain countless useful resources to help you. Still, for those just setting out on their web3 journey, it can be really overwhelming.
So, I decided to assemble this guide to point you in the right direction, as well as answer some questions that I have answered many times over during onboarding processes. If you want to get started for real, make sure to follow all the official sources I’ve listed. The goal of this document is to walk you from “basic” to “advanced”. If I’m missing something valuable, please let me know in the comments.
Much of the code you’ll read throughout this document is written in TypeScript. Why? Decentralized applications’ frontends usually run inside browser contexts. Plus: we cannot think of any chain, protocol or service that doesn’t come with a Javascript-compatible SDK. So, we’re considering Java-/Typescript the lingua franca to working with decentralized tech stacks. If you’re not familiar with it: NVM lets you install many different, isolated local node instances without leaving any traces on your harddisk, yarn and pnpm are splendid package managers and Visual Studio Code is an IDE that will never let you down.
A side note on the overwhelmingly hard to understand Javascript’s “bundler” ecosystem: we found that building web3 applications using the latest bundlers / compilers like esbuild, Vite or even Parcel is depressingly unpredictable. We definitely recommend staying on the established Create React App / webpack bundling track (Nextjs is also a fantastic framework to build decentralized apps).
web3: a primer
Web3 is a collective phrase for decentralized tech stacks. It’s mostly associated with Ethereum / IPFS but it’s also a heavily used phrase for Polkadot / Kusama / Substrate / Solana apps. In a nutshell, it stands for any application that doesn’t rely on a “Single Point of Failure”. More precisely, it describes a way of development, deployment and interaction of applications that are unstoppable and uncontrollable by anyone, thanks to distributed infrastructure such as blockchains.
Web3 apps are permissionless by definition. Everyone can access and interact with them. That doesn’t mean everybody can do what they want; there are well known methods to add access restrictions, role based access and even identity checks to protect them from random access. In contrast to traditional role management systems, those rules are enforced by cryptography and blockchain-based smart contracts. Once written correctly, they are fully uncompromiseable and unbreakable.
Since web3 apps are running in a decentralized environment, they’re also referred to as Dapps. While not restricted to them, Dapps mostly make use of either “content addressable storage networks” such as IPFS, and some kind of unstoppable, consensus based database, most likely a blockchain such as Ethereum.
Ethereum: a world computer.
Ethereum is an account-based blockchain running on interplanetary scale that executes custom binary code during transaction validation. It’s (still) using a Proof of Work consensus model that incentivizes miners to spend a lot of computing power to secure the network. Ethereum (mainnet) is migrating towards a dramatically more environment friendly Proof of Stake Consensus since 2015 and the current ETA of that “migration” is around 2022.
Both consensus models are proven to be secure in a massively adversary environment and they are locking hundreds of billions of USD value on many chains. There are lots of other consensus models out there and most of them deal with increasing transaction throughput, ease of setup, resource optimization etc. Though, to be honest, as of today not one of them has been proven to work as well as PoW.
The original “distributed consensus” term goes back to fail safe systems from the 70s, where people described the “Byzantine Generals Problem” leading to algorithms like PBFT (see Cosmos / Tendermint). Other very well understood (permissioned and not distributed) consensus algorithms are Paxos and Raft, which are used in distributed database / event streaming systems such as Apache Cassandra, Kafka or etcd.
Do you actually need a blockchain?
A common question people ask is: “Do you actually need a blockchain for this?” and mostly the answer will be no. For a lot of traditional business applications, using blockchain technology doesn’t add much value, but tenfold more complexity instead. There are two reasons when introducing blockchain technology makes absolute sense, though.
- First, when you need to interact with other economic parts on an existing, most likely public blockchain. On Ethereum, you find ten thousands of contracts that keep trillions of tokens, are worth billions of dollars and are all publicly available, waiting to accept your transactions.
- Second, if you’re building huge industrial consortia consisting of hundreds of players that neither know nor trust each other nor can agree on common infrastructure components.
Blockchains bear a promise of total transparency and interoperability. That’s why they are an ideal tech stack to build database-like systems that many partners can attach to without having to deal with ever changing API definitions or political onboarding processes. Identity ledgers like Sovrin / Indy or global supply chain systems with thousands of unacquainted logistics partners are very good candidates.
You most certainly will not need blockchains to enhance operations of only one company and you for sure won’t need one to run behind a single ecommerce application (which might be a different story if you’re building global marketplaces as we do at Turbine Kreuzberg).
Smart Contracts
Ethereum is the foundational tech stack for blockchain based applications. Even though it can seem hard to grasp in the beginning, it’s really the simplest to understand. As a developer, you don’t have to understand all of its inner workings. Still, there is one thing you really must get: you can write programs on your local machine in various programming languages, the most prominent one being Solidity, a Turing complete language, you compile them with a binary / intermediary compiler (solc), wrap the resulting byte code in a transaction and “deploy” that to the Ethereum blockchain.
The deployed code is what is called a “smart contract”. It can be accessed by all other acting entities on the Ethereum chain, either Externally Owned Accounts (EOAs), which are humans or wallets operated by bloodbags or other smart contracts. The runtime that executes smart contracts is called “Ethereum Virtual Machine” / EVM. Since contract code is triggered by transactions that are issued on a globally consented unique application state, the result of its execution will lead to exactly the same result every single time. Therefore, smart contracts are nothing but a fully deterministic, distributed state machine.
Wallets and Keys
Let’s quickly define what we mean by interacting here: every operation on the Ethereum ledger is triggered by something called a transaction. In its simplest form, a transaction is moving funds from one account to another. Most often, they’re used to invoke functions on smart contracts. Each transaction is signed with the private key of an EOA (a user) and can contain payload data.
Private keys are the fundamental security guarantee, your “login credentials” to the blockchain world, if you will. They’re an ultimate secret and are therefore kept inside wallet applications secured on the end users’ devices. Usually a “wallet” represents some 32 byte random seed number that can be recovered by / stored in a 12 / 24 words phrase, which you should put on a piece of paper and store in a secure place (read: not digitally!). You can use this seed phrase to recover your random wallet seed and derive an arbitrary amount of accounts from it using a keychain derivation path according to BIP-32 (official source). Note: inthe developer world a seed phrase is often referred to as Mnemonic. The most well known and widely used wallet in the Ethereum space for developers and users alike is called Metamask, a browser extension you should download and install right now.
WARNING — Even though this should be obvious, never, ever, on an astronomical scale of endless NEVER! should you use your seed phrase anywhere other than in a wallet application you trust. Don’t even put it on any “trusted” cloud setup, don’t copy and paste it into computer memory and for sure not on any website, for god’s sake. Don’t even think about sending it in an email! It belongs to you. When developing, make quadruple sure that the seeds you use for development are really different from the ones you use privately.
Transaction Priority / Gas
A final concept to understand for decentralized apps is how they deal with congestion. If everyone in the world tries to update the global state at the same time, a distributed state machine based on a single state tree (as Ethereum is) cannot write all their transactions simultaneously. It must wait for the global consensus layer to agree on a transaction ordering. This imposes a major bottleneck in the operational speeds of blockchain networks. That leads us to the question: how does the protocol know, under high load, which transactions it should accept first? Ethereum’s answer (and there are others) is the concept of Gas.
Gas is a virtual value unit that is required by the EVM to execute any state changing code. When a transaction triggers smart contract code and this code will change the contract’s state (e.g. their account balance) during execution, e.g. by assigning a new value to a member variable of the contract, this opcode execution costs a predefined amount of Gas.
Gas itself doesn’t translate to a monetary value. Instead, users of the network must estimate a competitive price compared to all other users (commonly calculated in a fractional unit of Eth, the so called Gwei or Shannon) for their transaction. They then send this amount along with their transaction, effectively making it a transaction fee. Gas is “consumed” during contract execution and surplus gas is refunded to the caller’s account. If your transaction fails because the executing smart contract reverts its execution, you won’t lose any assets involved in that transaction, but you’ll lose the gas you sent along.
An advanced word of notice: before the rollout of EIP-1559, all gas costs had been paid out as an incentive for block miners, which led to a somewhat overincentivized network situation, where people earned far more money than necessary for the security and stability of Ethereum. After EIP-1559, gas is consumed by miners totally differently than it has been before (the so called “base fee” is actually burnt by the protocol), but you as a developer shouldn’t notice the effects of this update at all.
You can find deep introductions on Ethereum at Ethereum’s funding company Consensys, and more developer centric introductions at EthHub. As I said, the most used wallet app by far is Metamask, which runs as browser extension or as a mobile app (React Native). You can also start with the very basics and sign transactions using Consensys’ official EthSigner, or choose another Dapp wallet.
There’s a deep official documentation for Solidity and you can code right away using the browser based Remix IDE or using Visual Studio Code extensions. To play around with seed phrases and keys, here’s a basic hd wallet repo, and the Truffle Suite’s HD Wallet provider or the “original” EthereumJS wallet repo.
Network Nodes, Private- and Testnets
The fundamental idea of blockchains is that you don’t have to (read: must never) trust anyone to add transactions to the ledger database — no bank, no service, no SaaS. Hence you can run a blockchain node on your own machine and talk to it via its JSON-RPC interface (which is a tiny spec to invoke exposed functions using JSON objects, eventually using HTTP as transport layer).
While it’s certainly fun to do so, you don’t need your own node to interact with any network (besides your local, single node devnet of course, see below). Services like Alchemy, Infura or Anyblock offer access with quite generous free tiers to Ethereum nodes on all networks. Dapps that run in browsers and are connected to Metamask actually access the blockchain nodes that Metamask relies on (they’re effectively run by Infura, a Consensys company).
But let’s say you’re feeling adventurous and want to run it for yourself. That’s why we’re developers, right? Then you have several options for Ethereum: the official node software is Geth (Go), a formerly abandoned and now slightly more active one is OpenEthereum (Rust) and then there’s Besu (Java, Hyperledger / IBM) and Nethermind (.NET).
In their default config, all those Ethereum clients are compatible to Ethereum mainnet: upon startup they’ll try to find neighbor nodes, exchange messages with them, download block headers, verify and rebuild the Ethereum chain state on your machine and finally join the network. You can even use those nodes to mine Ethereum blocks on your own, but that’s far out of scope of this article and totally unfeasible to achieve on average developer boxes.
Syncing Ethereum mainnet is a humongous task and needs a lot of computing power, RAM, bandwidth and storage alike (you’ll need Terabytes of NVMe SSD space!). Depending on the client you use, you have several options to sync. In Geth terms they’re called “light”, “fast” and “full” and you can activate a “tracing” feature that will add even more state data to it.
If you want to use a node to simply interact with smart contracts, a non-archive node does the job. Syncing requires ~3–7 days on a well equipped machine. Archive nodes keep state trees for all chain states in history and consume more than 7TB SSD hard disk space. They can be used if you want to analyse smart contract history, e.g. when writing applications that need to figure out what happened in the past to do predictions about the future, writing wallet apps with history support or chain explorers.
As a developer, you certainly won’t start your Ethereum development endeavours on mainnet. Therefore, the community is running 4 official testnets which differ in consensus protocols and speed, but from a developers perspective they’re perfectly compatible to mainnet: Ropsten, Kovan, Rinkeby and Goerli. Most of those networks are either set up with a low difficulty PoW or use a controlled consensus module, also known as “Proof of Authority” that allows predefined signers (or sealers or verifiers) to sign and issue blocks.
The Ethereum base currency has no real value on testnets so you can use faucet apps which are giving away testnet Ether for free (Ropsten, Kovan, Rinkeby, Goerli). Which testnet is best suited for your development needs depends on many factors, and they’re not all stable all the time. Here’s a Stackoverflow answer that might clarify things better than I ever could. To sync the Goerli testnet using geth, you simply can:
geth --goerli --syncmode "full" dumpconfig > goerli.toml
geth --goerli --config goerli.tomland depending on your bandwidth your node should be in sync after ~10 hours of downloading and validation. At the time of writing (2021) the full Goerli blockchain consumes around 43GB of harddisk space.
Running a local network / devnet
You can actually use Ethereum clients to setup your very own PoA network, too, which requires running several nodes and validators that operate on the Clique or AuRA PoA consensus modules.
Deploying and executing smart contracts on testnets still is cumbersome, error prone and laggy, so as a developer you can go for a much simpler solution: running your very own Ethereum devnet locally. For this you only need a node that runs on the most primitive, unvalidated “instant seal consensus”. The most well known node software to run a local Ethereum devnet is ganache (part of the truffle suite). You can run it as a standalone desktop app which comes with its own chain explorer or using Docker. Here’s a docker-compose snippet we often use to share a ganache instance with several apps:
version: "3.7"
services:
ganache:
image: trufflesuite/ganache-cli
entrypoint:
- node
- /app/ganache-core.docker.cli.js
- --deterministic
- --db=/ganache_data
- --mnemonic=${MNEMONIC}
- --chainId=5779
- --networkId=5779
- --gasPrice=5e9
volumes:
- .ganache:/ganache_data
ports:
- "7545:8545"For testing reasons you can run a fully fledged ganache node right inside your test suite. This is how we do it:
//@ts-ignore
import ganache from 'ganache-cli';
import Web3 from 'web3';const MNEMONIC = '...';const provider = ganache.provider({
mnemonic: MNEMONIC,
total_accounts: 10,
default_balance_ether: 100
});
const web3 = new Web3(provider);export default web3;
Now is the right time to wonder about the “provider” and “web3” objects. Let’s dig into them.
Interacting with Ethereum from your code: web3.js, ethers and providers
It’s sometimes hard to decide the terms “agent”, “wallet” and “provider”. Most commonly a provider is a service object that connects to a node and talks to it using its low level JSON RPC API. Additionally, a provider is usually “authenticated” upon construction, which means that it acts on behalf of the user. When writing non production code or tests, you’re usually using providers with direct access to private keys or Mnemonics but in a real environment you’re talking with a given provider’s web3 interface so you’ll never get in touch with a user’s private key or seed phrase.
Mental note: you will never ask for any user’s private key, even if it might simplify your life. If you’re tempted to do so, you certainly need to learn more about wallets, their APIs, environment variables or different ways to implement the problem you’re trying to solve.
For now, here’s a simple code sample that starts a local Ganache based Ethereum node and then uses web3.js to interact with it. When visiting the /transact endpoint, it’ll send 1_000_000 wei (the smallest amount you can break an Ether into) from account[0] to account[1] and displays the transaction receipt.
Note, that web3.js uses a (rather weird, but very practical) concept of PromiEvents that mix a promise with an event emitter. The local Ganache node creates a block with that transaction immediately so another call to / will show you the current balances. In a real world application (also on testnets), the transaction will take some time (15 seconds on testnets, longer on mainnet) to pass through the consensus protocol. You can use the event emitter part of a PromiEvent to listen for transaction confirmations and finality.
On a high level, you just witnessed how transactions are published to the Ethereum network, using web3’s sendTransaction method:
const receipt = await web3.eth.sendTransaction({
from: accounts[0],
to: accounts[1],
value: 10*1e5
})That looks fairly simple, but there’s quite a bit of stuff going on under the hood:
- it checks whether the bound provider to
web3.ethhas access to the wallet ofaccounts[0]. It has, because that’s how we created it. - it assembles an Ethereum transaction request object out of the given parameters.
- it guesses a gas limit (maximum gas for execution) for the operation involved (transferring an eth value from one account to another costs 21000 gas)
- it increments the latest nonce of
account[0]so the chain knows about the ordering of incoming transactions of that account. - it guesses the right network configuration for the transaction according to the underlying provider object.
- it uses
account[0]‘s private key to sign the transaction object. - it assembles a signed transaction request that looks like:
{
"nonce": 4,
"gasPrice": {
"type": "BigNumber",
"hex": "0x0f4240"
},
"gasLimit": {
"type": "BigNumber",
"hex": "0x5208"
},
"to": "0x6FaAB23372f372aB4B8F70C54CD15b4262847f32",
"value": {
"type": "BigNumber",
"hex": "0x0f4240"
},
"data": "0x",
"chainId": 0,
"v": 27,
"r": "0xb41e39cbee80b6a2d4d7ab7754553e3ac36901fc00656aad407d032650eb72e7",
"s": "0x1230e25a4968cb62bbdd0aa857f6886ff810f03ef7339d20413e45955ed0038a",
"from": "0xDC0a90a51E34D7b497df931679b47e0697B82d5E",
"hash": "0xee24c3cd5f2bc9ce0eb429c67e08ee959233526958bf1b497c440f3a3a7caeac",
"type": null
}- it promotes the signed transaction in binary encoding to the provider, using JSON-RPC
- it passes through the transaction receipt to the client
- it attaches a subscriber to the provider to get notified when the transaction is finally confirmed.
ethers.js
Besides web3.js, there’s the acclaimed ethers.js library, which is getting even more attention than web3.js lately and for many reasons. The most crucial difference is its consequent distinction of providers and signers. To demonstrate that, I rewrote the above web3 example for ethers.js with manual transaction signing (ethers.js of course also comes with a sendTransactions convenience method). Again, make sure to invoke the /transact endpoint to send transfer transactions:
Smart Contract Development
While there are others, the official language to write EVM compatible Smart Contracts is Solidity. If you know how to write Javascript and have a slight idea about hexadecimals, binary operations and object oriented coding, you should be able to learn and use Solidity within days.
The simplest way of getting your hands dirty using this language is the integrated, web based IDE Remix: https://remix.ethereum.org/. It lets you code, import, compile, interact and optimize smart contract code. There are so many tutorials written on getting started with it that I won’t repeat the steps here. My reading recommendation that goes beyond trivial aspects is this story by Burakcan Ekici from end of 2020.
To give you the simplest peek possible into Solidity, consider this code example:
It demonstrates the basic concepts: a smart contract feels like a class in other languages, it contains member variables and methods, it consequently uses typed parameters and it can extend other contracts (Solidity supports multiple inheritance). One special feature lies in the so called modifiers that are inherited by parent contracts and can annotate functions. Modifiers often act as “entry” guards, containing boundary conditions for a function call. In our simple example the onlyOwner modifier on store (inherited from Ownable) instructs the EVM to check that only the “owner”, who initially is set to the contract creator inside the derived parent constructor call, may invoke the store function.
Once you’ve written a contract like that, you can compile it using a Solidity compiler, inside your IDE, using your favorite toolchain or just right inside Remix. A solidity compiler creates two important things: first, executable bytecode that can be interpreted by the EVM and, second, a so called Application Binary Interface (ABI) that contains a user- and machine readable definition of your contract’s public methods. You need the ABI, or parts of it, to instruct clients (e.g. your Web3 / Javascript Dapp), how to interact with a deployed instance of your contract code.
To actually deploy the contract, you send a large transaction containing its bytecode to an Ethereum network, including an appropriate amount of Gas (each contract deployment modifies the state of the blockchain just as every other writing transaction).
Once a block has been mined that contains the code, the contract will get an unique address that others can use as a transaction target. A contract actually behaves like an externally owned account, only that it doesn’t have its own private key. It can call other contracts (when triggered by an external transaction) and store monetary value (e.g. Ether) and data and will react to transactions of any account (contracts or EOAs) according to the rules that are set inside the contract’s code.
I deployed the storage example contract on the Kovan testnet, and you can checkout its code, the bytecode and its ABI here: https://kovan.etherscan.io/address/0x7DA77f8a834369dDc5e9e47407C9746Ed55C3b72#code. Etherscan is by far the most prominent platform to view onchain data. If you’re wondering how the source code ended up visible on it: Etherscan allows smart contract developers to upload their sources for an already deployed contract, verifies that it’s compiling to exact deployed bytecode and then displays it to all users. This process is called “validation” and eventually involves flattening your source code to one large file or using a tool that uploads all sources and dependencies at once, the most powerful of them being Sourcify.
If you consider interacting with the contract from within a Dapp, you’ll need the aforementioned ABI that’s generated during compilation. For our Storage contract it looks like this:
Libraries like ethers.js or web3.js usually wrap ABIs with a dynamic class. You can use it to transparently invoke contract methods using a connected contract instance in your client code. Client code that interacts with our contract could look like this:
Smart Contract Development Environments & SDKs
I already introduced you to one IDE that many developers use build smart contracts on the fly: Remix. It’s a great tool to get you started, comes with lots of plugins for virtually any contract related workflow, it has a decent user interface and is very well documented.
When you start building Dapps, it falls short though, because you’re going to have to integrate your contract code with your Dapp, share and update the ABIs inside your developer workflow or add backend code or CLI tools. There are two commonly used toolchains that allow you to build Ethereum based apps inside non trivial project repos: Hardhat (formerly Builder) and Truffle.
In a nutshell, both are great. Emotionally, hardhat is closer related to people who like the ethers.js library and truffle is closer to web3.js. Another difference is that Truffle comes with a contract based migration concept to deploy your contracts, whereas Hardhat lets you as a developer write scripts to get your contracts deployed. Hardhat shines with (in my opinion!) slightly better intuitive support for automated tests, full Typescript integration and autogeneration of boilerplate code for typesafe usage of smart contracts, made possible by a library called Typechain.
Other handy tools to have a look at to get started are Austin Thomas Griffith’s scaffold.eth, Paul Razvan Berg’s Create Eth App, and OneClickDapp that generates a Dapp out of a deployed contract codebase.
OpenZeppelin: audited smart contract primitives
Building a baby storage contract like the one shown above is the tiniest tip of the iceberg above what you can achieve with smart contracts. If you’re in the Ethereum space, you likely will have heard about the tokenization of things, about Defi protocols, yield farming, decentralized exchanges, NFTs, DAOs and alike. All those are blockchain applications, built on smart contracts. Many of them make use of some very basic primitives in the Ethereum space that are known by their respective EIP numbers: ERC20, ERC721 and ERC1155, which all are token standards.
Before you try to wrap your head around them yourself, rest assured: it’s nearly impossible to write safe code to implement even their most simple parts on your own. Hence nearly everyone uses a collection of base libraries to implement the most general use cases of smart contracts, and it’s called OpenZeppelin Contracts (OZ).
Besides token standards OZ has implementations for ownership and role based access, timelocks, counters, governance, contract proxies, enumerable mappings, introspection, math and string operations. They’re actually everywhere and I couldn’t encourage you more than to checkout them right now. Their documentation comes with lots of examples that made me understand tokens far better than any article could explain.
Contract Upgrades
You already might’ve wondered how you could update code on the blockchain once it has been deployed, for example if you found a bug . The short answer is: you cannot. That’s why most projects run through very thorough and costly auditing processes before they launch their contracts on mainnet. A tool that can find the most obvious issues in your code automatically is Remix’ static analyzer but rest assured: smart contract security is far harder to achieve than running a static code analysis tool on your code. Here are some articles that might give you a perspective of how hard it is to make EVM code bullet proof in practice.
A highly acclaimed feature that OpenZeppelin therefore has in stock are upgradeable contracts. Their name is slightly misleading since they’re still far from being upgradeable. Instead, they’re a pattern that makes use of some advanced Solidity internals to enable so called contract proxies.
In a nutshell the idea is to first deploy a contract proxy as well as a first implementation contract and tell the proxy to use that implementation. The proxy delegates all calls to the implementation contract, but all state changes (i.e. the storage) are persisted on the proxy’s address space itself. If you decide to “upgrade” your contract with new code, you’re actually deploying a full new implementation and point the proxy to it. There are some tradeoffs here, e.g. the replacement of constructors with initializing functions and a non updateable storage layout that effectively doesn’t let you update variable names and force you to extend your variable list, instead of being able to replace old variables.
Upgradeable contracts will increase your development speed and your beta releases but using them for contracts that lock millions of dollars might not be the best idea, depending how you look at their implications.
It’s a wrap 🌯!
There’s no way to put a TL/DR; under this article, but if I had to put it shortly: Blockchain development is not the easiest task, but honestly it isn’t that hard either. What is really hard to get right, is dealing with the dependencies between your Dapp packages. I’ve become a huge fan of pnpm powered monorepos for that job. Security is another deal breaker that I just briefly touched upon during the last sections of this article.
Now, I wish you the very best of luck in your endeavours in the smart contracts & Dapp space. If you’re interested in getting in touch with me or my company Turbine Kreuzberg to consult on contract development, web3 tech and decentralization, just drop us an email or DM me on Twitter.
