How to craft your own oracle queries in HTTP GET using Witnet

The Witnet oracle is capable of querying any publicly available API. This tutorial will help us walk through how to create a Witnet Data Request from scratch and deploy it to a network like Ethereum. With this approach, you will have full control over the data sources and any data request parameter to be perfectly tailored to your and your contracts use case.

Why the Witnet oracle is a great solution

When writing a smart contract, developers are unable to query events outside the blockchain. This is due to a variety of factors that are beyond the scope of this article. The thing to remember is that blockchains are typically deterministic and censorship resistant. This means we need a way to get data without breaking the properties of blockchains; referred to as the oracle problem.

The key points:

• Decentralization: Witnet is a layer one blockchain, so instead of trusting node operators to provide truthful information, there are instead cryptoeconomic guarantees that force them to act within the best interest of the network.
• Developer experience: As we will explore, a Witnet data request is fully customizable from how where to get data from, how to aggregate it, what to do with the dishonest nodes, and how to reward honest nodes.
• Security: There are tens of thousands of nodes operating within the Witnet network and they are all required to post collateral in order to fulfill data requests. Not only do your data requests get fulfilled, but they also get put into a block in the Witnet blockchain for all eternity; meaning that your data is immutable.

Creating your first Witnet data request

How to GET started

We are going to keep this HTTP GET tutorial exclusive to price feeds, as its the most common query right now, but you can use any other kind of APIs.

Step 1

We must first install Witnet dependencies if we are already using Truffle or HardHat.

yarn add witnet-solidity-bridge
yarn add witnet-requests --dev

You can take a more in-depth look at the Witnet-Solidity-Bridge here on our Github.

Step 2

This one is a simple one… We’re going to put our oracle queries in a directory called Witnet for both safekeeping and easy organization.

Step 3

Let’s begin by crafting our own beautiful oracle query. Let’s use BTC/USD for our example.

When crafting oracle queries to be executed by the Witnet oracle, we generally give the advice that you should use sources that have high amounts of volume. We don’t utilize aggregators like CoinMarketCap or CoinGecko because they don’t actually have volume.

By high amounts of volume, we typically go with sources like Binance, Coinbase, Gemini, Kraken, Bitfinex, SushiSwap, Uniswap, etc.

Keep in mind that this doesn’t mean you can’t query the Witnet oracle. Even if there is only one exchange that a coin you need queried is listed, you can still query it- you just need to be aware of the risks.

For our Bitcoin/US Dollar price feed we are going to use 3 sources; Binance, Kraken, and Coinbase.

Go back to that directory you created back in step 2, the one called Witnet, or whatever you decided to call it. Create a file called BtcPrice.js.

We are now going to edit the file and import the witnet-requests library using this:

import * as Witnet from "witnet-requests"

Step 4

We now need to define our first data source that we want the Witnet node operators to query. Let’s use Binance. Below is how you define these requests.

const binance = new Witnet.Source("https://api.binance.US/api/v3/trades?symbol=BTCUSD")
  .parseJSONMap()
  .getFloat("price")
  .multiply(10 ** 6)
  .round()

Many developers might understand how this string works, but for those who don’t let’s walk through it really fast.

The link in the first line is the headless browser that nodes are going to to find the piece of data. .parseJSONMap() puts the data in the correct format.

The next one, .getFloat tells the node to search for the value “price” within that link.

The function .multiply(10**6) gives us the 6 decimal places we’re looking for. The final function .round() will round to a specific decimal place so our price feed is nice and neat.

Since we are using Bitcoin as our example, and there is a ton of volume on Bitcoin on most exchanges, we can define 2 more sources (as mentioned earlier, 3 should be the default). Kraken and Coinbase are stellar sources for this example.

Adding a second and third API data source only require new blocks below the first one we wrote above, otherwise known as a Witnet.Source block.

Coinbase:

const coinbase = new
Witnet.Source(“https://api.coinbase.com/v2/exchange-rates?currency=BTC")
 .parseJSONMap()
 .getMap(“data”)
 .getMap(“rates”)
 .getFloat(“USD”)
 .multiply(10 ** 6)
 .round()

Kraken:

const kraken = new
Witnet.Source(“https://api.kraken.com/0/public/Ticker?pair=BTCUSD")
 .parseJSONMap()
 .getMap(“result”)
 .getMap(“XBTCZUSD”)
 .getArray(“a”)
 .getFloat(0)
 .multiply(10 ** 6)
 .round()

Step 6

Here’s where the fun begins. We are now able to specify how to aggregate these pieces. Within Witnet, there are two steps as part of the aggregation of these differing pieces of information that nodes have gone out and gotten for us; filter and reducer.

When using the filter, you get to specify how to validate if the data received from nodes is good enough for your use case. For example, we can filter our or remove any outlier that claims wrong information or goes against the general consensus. This is useful for flash crashes.

When using the reducer you specify how to aggregate together results from multiple sources. After the outliers have been filtered out we now have to calculate the mean from all the honest nodes.

Aggregators offer a lot of flexibility when working with the Witnet oracle. Developers can choose how they want their filter and reducer to operate, but many developers default to the one that has worked in the past because of it’s simplicity.

The function that aggregates these remaining data points together that is considered the pinnacle is this:

const aggregator = Witnet.Aggregator.deviationAndMean(1.5)

Where 1.5 is the allowed difference from the average (this is our filter). Any data point that is 1.5x off the average will be filtered out. The average mean will then be recomputed for nodes who passed this filter.

Step 7

We are now able to specify the aggregation of data reporters, also know as Tally in the Witnet network.

The tally is a second layer that adds to security and truthfulness without trusting any one node. Tally functions look very much like aggregator functions. The most common form is below:

const tally = Witnet.Tally.deviationAndMean(2.5)

Step 8

We have now pretty much reached the end, and now we need to work out the kinks and get ready to ride. Step 8 consists of quite a few steps, so we will take this slow.

Let’s begin by looking at our oracle query. This example below is everything we’ve done up to this point, and it includes some comments for additional guidance.

const queryconst query = new Witnet.Query()
.addSource(binance)
.addSource(coinbase)
.addSource(kraken)
.setAggregator(aggregator) // Set the aggregator function
.setTally(tally) // Set the tally function
.setQuorum(10, 51) // Set witness count and minimum consensus percentage
.setFees(5 * 10 ** 9, 10 ** 9) // Witnessing nodes will be rewarded 5 $WIT each
.setCollateral(50 * 10 ** 9) // Require each witness node to stake 50 $WIT
// Do not forget to export the query object 
export { query as default }

Below are a list of things that are new based on this code block above.

.setQuorum: Allows us to choose how many nodes should go to our inputted sources. The more nodes we utilize, the more secure our feed will be. In the code block above, we can see the there is a (10, 51). 10 is the minimum nodes required to complete this request and they should have a 51% consensus among them; otherwise, abort.

.setFees: This is the parameter that will state how much to pay nodes that retrieve our vital information. In the code block above, we can see the (5 * 10 ** 9, 10 ** 9). 5 WIT for each node, and 1 WIT for miners including the transaction in their block.

.setCollateral: This asks nodes who want to fulfill our request to post the required collateral we want them to. In the code block above, we see (50 * 10 ** 9). Basically, stake 50 WIT to be eligible to fulfill a request (after the multiplication and decimal arithmetic is said and done).

Step 9

The actual request that will be fulfilled from the Witnet side is complete, and now we just need to compile the information correctly. In this case, it needs to be a Solidity contract, as Witnet is not available on non-EVM compatible chains at the time of this writing.

Let’s run a command line from the project directory:

npx rad2sol - write-Contracts

By commanding this, the JavaScript file will be automatically analyzed into Witnet bytecode and wrapped into a small Solidity contract. This will find it’s home in the ./contracts/requests directory for whenever you are ready to import it into your Solidity contract.

Let’s start testing

Now that we have completed the actual request, let’s see how they work and react to our real world events.

We are going to run the query locally to preview the result so that we can move on to the Solidity part. This is to ensure valid data sources and that our aggregation methods are working correctly.

The witnet-requests library provides a command to try oracle queries locally, let’s use it now.

npx witnet-toolkit try-query - from-solidity

The command here will output an execution report. All you need to do is double check that the result of each data sources makes sense and the tally stage is working.

Import the query into Solidity

Now we can import our Witnet request into our Solidity contract.

Step 1

// Import the UsingWitnet library that enables interacting with Witnet
import "witnet-ethereum-bridge/contracts/UsingWitnet.sol";
// Import the BitcoinPrice request that you compiled before
import "./requests/BtcPrice.sol";

Step 2

Now we need to make our contract inherit from UsingWitnet. That is simply one line:

contract PriceFeed is UsingWitnet {

This requires the constructor to pass the address of the WitnetRequestBoard that is specific to the network you are building on. It will most likely be a different address between different layers and chains, so bear that in mind.

constructor ()
UsingWitnet(WitnetRequestBoard(0x9E4fae1c7ac543a81E4E2a5486a0dDaad8194bdA)){

To deploy with convenience on different EVM chains, and in order to not change the code you can make the constructor receive the WitnetRequestBoard as an argument and provide the address in our migration scripts when deploying.

constructor (WitnetRequestBoard _wrb) UsingWitnet(_wrb) {

Aside from this, the only change needing to be made in our contract is to define the query as a property of the contract and to instantiate it from the constructor. See below:

contract PriceFeed is UsingWitnet {
Request public query;
uint256 requestId;
uint64 latestPrice;
constructor (WitnetRequestBoard _wrb) UsingWitnet(_wrb) {
query = new BtcPriceRequest();
 }
}

There are a few properties to focus on; requestId and latestPrice. The former will store the identifier of the last updated price, the latter will keep the latest result.

Step 3

We have finally made it to the point where we can submit the query to the actual Witnet network. The payable function must call the _witnetPostRequest. This is going to look like this:

function requestUpdate() public payable {
requestId = _witnetPostRequest(query);
}

Witnet has a special contract enabling the communication between EVM chains and the Witnet sidechain, this is referred to as the WitnetRequestBoard. Calling _witnetPostRequest will get the bytecode posted there.

Step 4

Because a Witnet query is an asynchronous process, it can take a few minutes to have the query fulfilled. While this might sound like a long time it has proven both effective in security and decentralization and been totally fine for all use cases.

The result will be reported back to the WitnetRequestBoard, which is a storage that your contract can read from. Other contracts are also able to read this feed after its fulfilled, and so is your contract! Let’s use this block below:

function completeUpdate() public witnetRequestResolved(requestId) {
Witnet.Result memory result = _witnetReadResult(requestId);
if (witnet.isOk(result)) {
lastPrice = witnet.asUint64(result);
} else {
// You can decide here what to do if the query failed
 }
}

Once we have read the result with Witnet.Result, we now need to decode the right Solidity type, like .asUint64.

Important to note: the witnetRequestResolved(requestId) modifier prevents calling the function before the resolution of our query. We can use witnet.isOk(result) to see the status of our query.

Step 5: deploy!

As you probably know, deployment instructions are very specific to the Solidity toolkit, so its lucrative to get it right when deploying with Truffle or Hardhat.

Our contract needs to get the address of the WitnetRequestBoard passed as an argument. Use this page here to find the contract address.

Specific to Truffle:

rad2sol can automatically generate migrations for the Witnet libraries and your own contracts if used like this:

npx rad2sol - write-contracts - write-witnet-migrations - write-user-migrations

If you look at the migrations folder, you will find two files; 1_witnet_core.js and 2_user_contracts.js. The former deploys all the Witnet related contracts for deploying on a local or private network. It can also dynamically link them for those on a public network. The latter contains autogenerated migrations scripts for consumer contracts.

The compiler will create default values for your additional constructor arguments in your contract, too.

Last but not least, make sure to double check the default argument that the compiler has inserted for you because they may not make sense for what you need.

Congratulations! You deployed a contract with a price feed using Witnet!

If you had any trouble, or you want to give us some feedback join our community. If you want to take your contract to the next level with a Witnet Foundation grant for marketing, resources, gas fees, etc, check the grant program.

Let us know what you think:

Website | GitHub | Twitter | Discord | Telegram | Reddit | YouTube