Technical Assessment of Augur

It’s been almost a month since the release of Augur, an Ethereum-based prediction market. In that time we’ve had a number of people ask us about how SportCrypt is different from Augur, and in particular if it makes sense to use Augur instead of, or perhaps as a back-end for our service.

In short, the answer is no. This article is the long-version explanation. I hesitate to be critical of an innovative product in such a nascent area of technology, but this field is too important to not discuss it openly.

I don’t claim to be an expert on Augur. I have read the whitepaper and most of the contract code, and I downloaded the client as soon as it was released and have made several trades since. Everything here is accurate to the best of my knowledge, but if you find any errors or material omissions then please post in this reddit thread and I will acknowledge and apply all due corrections.

Prediction Markets

Let’s start with some basics that are no doubt obvious to many but deserve stating all the same.

Prediction markets cannot make predictions on their own. They are systems where people can make bets between themselves. If people are willing to bet enough money on an event it may be possible to infer some information from their aggregate actions — perhaps enough to make an informed prediction. But no matter how sophisticated a prediction market is, it cannot be used to make predictions unless enough people find the events it hosts interesting or profitable enough to bet on.

The uphill battle with prediction markets has always been overcoming the disconnect between what academics would like to predict with prediction markets and what people actually enjoy betting on. It would no question be a benefit to society if we had accurate, market-driven predictions about politics, progress in the arts and sciences, the weather, and so on, but historically people have shown to mostly be interested in betting on less enlightened activities. Games of chance such as dice have been bet on throughout recorded history. The Romans loved to bet on chariot races. In 17th century England they bet on a sport (using the term loosely here) where two contestants kick each-other in the shins until one of them falls down.

On the other hand, if you’re inclined to look at it in a certain light, the entire global financial system can be regarded as a series of prediction markets where investors are betting on price changes of various assets. Certainly this is the case with instruments such as derivatives and CFDs where the underlying assets are not necessarily even owned by any of the participants.

Barring any rapid evolution in human nature, it seems there are really two avenues for prediction markets: the sports path or the finance path. This is already apparent on Augur where, following an initial surge of World Cup soccer betting, nearly all the activity has been on financial derivatives (in particular cryptocurrency futures).

On vs. Off-Chain Orderbooks

There are basically two models for orderbooks on Ethereum exchanges (along with some oddballs and hybrids) which I will attempt to quickly summarise.

In the on-chain model used by Augur, the orderbook lives in a smart-contract on the blockchain. In order to add orders you must submit transactions to the Ethereum network and they must in turn be mined.

In the off-chain model used by EtherDelta, IDEX, and SportCrypt, the orderbook lives elsewhere, usually on a centralised server but potentially on a neutral message board or peer-to-peer network. In order to add an order no transactions need to be submitted to the Ethereum network.

(There is also a distinction between on-chain vs. off-chain matching. I will assume on-chain matching so some of the following does not apply to IDEX.)

The Ethex exchange has written an article advocating on-chain orderbooks which is worth a read, however I would argue that one person’s “frivolous offer” is another person’s liquidity and that adding unnecessary costs is in nobody’s interest.

The article also doesn’t mention an important advantage of on-chain orderbooks: They have the ability to guarantee price-time priority during matching. With an off-chain orderbook it is up to the trader’s client software to pick which order(s) to match against. The trader may pick any order in the orderbook, regardless of it having the best price or being added the earliest. Think of on-chain being like the US equity market where the trade-through rule is in effect, and off-chain being like the global forex market where different banks may offer different quotes.

Augur’s orderbook maintains price priority but I’m not sure about time priority. It looks like you may be able to jump the queue by selecting appropriate bestOrderId and worstOrderId parameters during order creation (I haven’t confirmed this).

On the other hand, the major disadvantage of an on-chain orderbook is that it makes it expensive and difficult to create markets on numerous events. It’s expensive because of gas fees, and difficult because you have to monitor Ethereum network congestion and your account is effectively stuck until all its pending transactions are mined.

Since users demand many different instruments and variations thereof, and it is difficult to predict which ones will be popular beforehand, order creation should be inexpensive and frictionless — something not possible with on-chain orderbooks.

Order Expiration

During the World Cup one of our users pointed out an interesting phenomenon occurring on Augur. On several games most of the trading volume was actually happening after the soccer game had finished. I believe what happened is that many people created offers to trade these games without understanding that these offers would be valid until manually cancelled, even after the game had finished.

So naturally when the game did finish, opportunistic traders snapped up the orders on the losing side: Free money.

When creating an order on Augur there doesn’t appear to be any way to set an expiration time, and I can find no mention of order expiry in the contract code. The consequence is that you need to explicitly cancel your orders (using Ethereum transactions) when you wish for people to stop trading on them. This means that to make a market on a game you need to pay gas to add your orders to the orderbook, and then pay more gas to remove them prior to the kick-off (assuming they weren’t fully filled). There is also risk involved: what if your cancellations get delayed because of Ethereum network congestion?

In response some users tried creating markets like France + Croatia Total Goals Over 2.5 in the 2018 FIFA World Cup Finals? (Market closes before game starts to prevent sniping). Unfortunately this will not work since, by design, markets can be traded on for at least a week after their scheduled market end times.

Escrowed Funds

When you create an order in Augur (assuming you don’t yet own any shares), the amount of funds that will be required if somebody were to fully fill this order are transferred into the Augur smart contracts and held in escrow. When you cancel the order they are transferred back. This is so that there are always enough funds in the contract for all orders in the orderbook to be filled.

Based on my testing, this is the case even when a single account creates bid and ask orders on the same market. This is capital inefficient since if either of the orders were to be filled then the market maker would at that point have shares available which could serve as collateral for the opposite order.

The consequence is that if you would like to be a market maker on N different Augur markets, and would like to put up M ETH on both sides of each of those markets, you will need to escrow N*2*M ETH. On SportCrypt you only need M ETH, although usually you would want more than that so that if the entirety of one of your orders is filled all your other markets are not taken down (except for the opposite order on the given market, which would of course have a position as collateral). SportCrypt’s orders are in this sense similar to one-cancel-all orders (but not exactly, see the section on Collateral Flexibility in this article).

Oracles

Augur’s basically the printing press / internet of finance, *and* the Amazon on top to explore the long tail of everything that results / is created from it, *and* a solution to the oracle problem.
 — Joey Krug, founder of Augur

What Mr. Krug means by the oracle problem is that on blockchains such as Ethereum it is possible to construct self-contained smart contracts which don’t require any trusted parties in order to act fairly. However, as soon as you need to make the behaviour of the contract depend on some external information (ie, not already on the blockchain) someone or something needs to provide that information. This provider is called an oracle.

Augur’s innovation is what its creators call a decentralised oracle. Rather than trusting someone to provide the correct information so that a smart contract can, for instance, pay out the winner of a bet, in Augur there is a consensus-seeking game where anonymous participants (“reporters”) stake tokens on what they believe to be the correct version of the information. Eventually, potentially after multiple rounds of disputes and progressively increasing stakes, the system will reach a consensus and those participants who backed the “correct” version of the information will receive fees collected by the smart contract as compensation.

Since this all may seem a bit abstract, let’s take an example. At the time of this writing the top event by open interest (similar to trading volume — the distinction is not important here) is “Will price of Ethereum exceed $500 at the end of 2018?”. Every event has some additional details. In this case there is an extra description “As reported by Coinbase.pro (last trade in 2018)”, a resolution source of “https://pro.coinbase.com/trade/ETH-USD”, and some other specifics such as market end time.

Once the betting period has concluded and the consensus-forming process has begun, the participants in the game will go to the coinbase.pro website described in the details and use the value they see there as the correct information.

The plainly apparent issue with this particular example is that the trusted party has not been eliminated at all. It has merely been shifted to the operators of the coinbase.pro website. If this operator chooses to provide incorrect information then there is nothing in the Augur protocol that prevents it.

In this instance, the Augur consensus game does not add any additional assurance: it is always less reliable then had coinbase acted as a trusted oracle and directly reported the information to the contract as a signed message. In other words, in this case the consensus-seeking game played by the Augur participants can only add grading risk, not remove it because in the best-case scenario the consensus game will converge to whatever the coinbase.pro website says it should, as per the specified market details.

In fact, had Coinbase directly submitted the information to the smart contract then we could have had an assurance that they aren’t reporting conflicting versions of the information. In our example the coinbase.pro website could return different results to different people, or different results over time.

At the time of this writing the top 5 markets on Augur have either coinbase.pro or coinmarketcap.com listed as resolution sources. These 5 markets represent over 80% of Augur’s total open interest.

Grading Risk

However, Augur market descriptions don’t need to refer to a centralised reporting source. The description and the additional details are free-form text fields so market creators can describe the details and conditions of events with unlimited flexibility, and it is up to the reporters to interpret them. For binary markets they can either decide in favour of one of the outcomes, or mark the event as invalid.

Because descriptions are not structured or standardised, prior to betting it is very important to read the event and its additional details in full. There are cases where topic-specific rules may be assumed, the title conflicts with the details, and many other clearly (or not-so-clearly) ambiguous markets.

For every market you are considering betting on you need to consider carefully how the reporters will interpret it since this may be different from your interpretation. This is what is meant by grading risk. Even when the details are clearly written there is still a chance that the contract could be graded with a different interpretation than you were expecting, or be cancelled as invalid.

As an example, consider the market “Will England and Croatia Score More Than 2 Total Goals In Their 2018 World Cup Semifinal Match”. 2 goals were scored in the first 90 minutes, and then Croatia won by scoring an additional goal in extra time. So, many reporters made the reasonable assumption that this contract should be graded as YES. Unknown to them, it is common in soccer betting to only count the goals scored in the first 90 minutes, and this was clearly described in the additional details: “The scores used are taken after 90 minutes of regulation plus stoppage time. Does NOT include extra added time or penalty kicks.” (amusingly these rules were verbatim copied from the SportCrypt details).

This event is currently in dispute round 3, although partially due to a reddit discussion, it is likely that this event will eventually (the game took place almost 3 weeks ago) be graded correctly as NO.

You may have noticed that the above discussions took place on reddit. In the absence of authoritative guidelines and/or standardised event details, there will necessarily be a need to debate the wording and intent of the free-form details and coordinate the interpretation according to community norms, such as they are. Unsurprisingly, these discussions so far are occurring predominately on the official Augur reddit messaging board, and the official Augur discord chat, both of which are moderated by the Forecast Foundation (described below).

If an event you are participating in is interpreted in a way you didn’t expect, unless you are confident that reporters will eventually “see reason”, are prepared to risk even more money, and fully understand the consensus-seeking game as described in the whitepaper and implemented in the smart contracts, currently your best recourse may be to argue your case on reddit or discord, and hopefully one of the community “thought leaders” will be sympathetic to your cause.

Eventually there may be more formalised norms and understandings between reporters. The groups coordinating these could be referred to as standards bodies, or perhaps, if one feels less charitable, reporting cartels. Here is a proposal along those lines: A Constitution Proposal for Augur REP Voters, and here is a site attempting to “accelerate the adoption of reporting standards”.

Cost of the Oracle

Yes, everyone I’ve seen building derivatives projects with a couple rare exceptions are all easily replicable on Augur (which is kinda part of the point!)
 — Joey Krug

Even if we concede that the decentralised oracle system has advantages, they must be weighed against its costs.

The most obvious cost is the explicit market fee. In order to create an incentive for the reporters, half of the market fees are put up as a reward (the other half goes to the market creator). There is a more detailed discussion of fees below.

An additional cost is the opportunity cost for the time it takes the consensus-seeking game to conclude. In the best possible scenario this is 7 days after the scheduled end of the market which in common practice seems to be set several hours or more after the actual event is anticipated to end. However, any of the following may delay it further:

  • If the designated reporter doesn’t report right away, add up to 3 more days.
  • If the initial report isn’t made right at the end of a fee window, add up to 7 more days (so on average 3.5 more days).
  • If there are any disputes, add on more (potentially a lot more).

While waiting for the market to be graded, your funds are locked up in a smart contract. If you can find a willing trader you may be able to trade your position for ETH so as to cash out early, but they will of course want a portion of your winnings as compensation (discussed further in the fee section).

For many types of betting this waiting period is undesirable. For example, suppose you want to hedge the closing price of some asset every day. Assuming no disputes and instant designated reporting, in the steady state 10.5 times your daily hedging costs would be tied up waiting for reporting (supposing your hedging bets have fair market value of 50%).

Similarly, many sports bettors prefer to bet on several games every day and to roll their winnings into new bets hours or minutes after games finish. (In the past we’ve had complaints that 30 minute grading was too long.)

This suggests that Augur will be more useful for longer-term bets where the reporting duration can be amortised over the lifetime of the market.

Lastly, even if you stay away from disputable markets altogether there is still a systemic risk that Augur will enter a fork state, which effectively shuts the entire system down for up to 60 days!

Dogfooding

We believe we’ve solved the main host of problems there, and now augur is really just an engineering problem as opposed to a research *and* engineering one, which is good. Research is like wandering in the wilderness, we’re done wandering.
 — Joey Krug

In order to create a new event in Augur a special bond denominated in REP tokens must be posted by the creator. For reasons explained in the Augur whitepaper, the amount of REP that must be posted depends on the market price of REP.

But of course the market price of REP is information from the external world and cannot be computed by the smart contract on its own. What is needed is an oracle to provide this information to the smart contract. This is how the currently published version of the whitepaper says this will be handled:

Augur gets information about the price of REP in the same way it gets any other information about the real world: through an Augur market.

This is the exact use-case that Augur’s decentralised oracle has been designed for, and a perfect opportunity for the project to try some beneficial dogfooding. However, just prior to launch the current working version of the whitepaper has had the above sentence changed to:

Augur gets information about the price of REP via a collection of trusted third-party price oracles (Footnote: In the future, Augur may be able to learn the price of REP without having to trust third parties. This is an active area of research.)

This raises more questions than it answers. Who are these trusted third-party oracles and why do we trust them? Why was an Augur market not sufficient? Why hasn’t this been mentioned in any public documentation other than in an unpublished edit to the whitepaper?

And just in case the irony has escaped anyone, according to a literal interpretation of this edit, having a smart contract learn external information without needing trusted third-parties — the very “oracle problem” claimed to have been solved by Augur — is an active area of research.

Forecast Foundation

And it’s all being built with a small team of around 15 people. If a startup were building this it’d be 150–200 people, and honestly imo, probably not much faster.
 — Joey Krug

The Forecast Foundation is the legal entity which collected the funds from Augur’s ICO and employs programmers tasked with developing the Augur software, managers, marketing professionals, etc.

The Foundation insists that it is solely a software development organisation and it is up to their users to assume any and all legal risk. This is the main reason for the click-through agreement you have to agree to when you first connect to Augur.

I’m not qualified to know if this legal reasoning is sound, but to me Augur seems more like a regular startup than a pure-crypto project like, for example, Bitcoin. Here is what is publicly known about the Foundation:

It is a private corporation registered in Estonia. It has contractual agreements with pre-sale REP holders. It has registered trademarks on the Augur name and logo (although the name has been disputed). It owns the copyright on the smart contract code and the sole client implementation, and has a patent on the technology. It probably possesses the private keys that allow it to set the REP bond needed to create markets (see above). It serves the official releases of the Augur client from its github project, and moderates the official reddit and discord channels (where links to unofficial channels such as an Augur users group on Telegram have been censored).

Fees

As with most things, due to a combination of explicit and implicit fees the exact cost of using Augur is difficult to calculate. I have itemised the separate fees I can think of and expanded on them a bit in the following list:

  • The market fee shown in the Augur interface is the fee collected at the end of reporting and is split between the market’s creator and the reporters. The first market I traded on (Will France Defeat Belgium in the 2018 FIFA World Cup Semifinals?) had a 2.51% fee which was slightly on the high side but not extreme (as a comparison, matchbook.com’s commission is 1.5% for takers and 0.75% for makers).
    One of the downsides of Augur’s unstructured approach to markets is that you need to check the market fee prior to each market you participate in as it can vary considerably and there have already been scam markets posted with 50% or higher fees.
  • The bid-ask spread is a necessary attribute of any betting exchange. For regular bookmakers this is how they make all their money: Buying low and selling high. As a bettor, the wider the spread the more expensive this type of fee is.
    Although the spread is a function of many things, an important one is how cost effective and convenient it is for market makers to operate. As described above, due to Augur’s on-chain orderbook, lack of order expiry, and strict fund escrow it is more expensive and difficult to be a market maker than should be necessary.
  • Gas fees are a unique feature of exchanges built on the Ethereum blockchain. Essentially any interaction with the blockchain requires the sender pay a fee proportional to prevailing market conditions (usually reasonable unless in the middle of a gas price war), and the amount of work done by the destination contract.
    Augur’s gas usage is extremely high. A trade uses at least a decimal order of magnitude more gas than it does on EtherDelta/SportCrypt/etc. Partly this is because a price-priority orderbook in a smart contract does require a lot more work to be done on-chain, and partly because of some pretty dubious coding practices. As well as being very high, gas usage is also unpredictable since it can vary a lot depending on the state of the orderbook. Lastly, because orders do not have expiry times, you must also pay gas to cancel every order that wasn’t fully filled before kick-off time.
  • Grading risk must be factored in as a part of the fees as well. As described above, Augur markets have non-trivial grading risk due to the free-form nature of the market details and unknown incentives of the reporters. An interesting strategy might be to take advantage of dispute optionality: deliberately bet on slightly ambiguous markets and if and only if you lose, initiate a dispute (perhaps colluding with thought leaders or reporting cartels).
  • Opportunity cost must be accounted for since markets on Augur take such a long time to finalise. Instead of having your funds locked into a market waiting to be reported on, you could be deploying them elsewhere (perhaps another Augur market).
  • Similarly, consider foreign exchange risk. If your reporting or expenses are denominated in a currency other than ETH, you should be aware that large fluctuations in the value of ETH could wipe out your profits while waiting for the market to be reported on. The solution is forex hedging which involves its own set of fees.
  • Finally, for liquid markets there may be “early-exit” orders available. This is where somebody offers to buy your winning shares for a price close to the expected final price so that you can cash out immediately instead of waiting for the market to be reported on.
    In order for this to be worthwhile for the other party, exiting early will involve paying a fee equal to all the above fees, plus a profit. This is because the early-exit market makers will need to be compensated for their gas fees, the market fee which they will have to pay, and the grading risk, opportunity cost, etc. that they will have to bear. Needless to say, if a market is in the middle of a dispute the grading risk and opportunity cost will be higher.

Conclusion

Based on the above analysis, I don’t believe Augur is suitable for our purposes. I think a sports betting platform needs to offer the following attributes that Augur does not:

  • Structured market descriptions
  • Fast grading times
  • Low and predictable fees
  • Inexpensive and convenient order creation
  • Order expiry
  • Efficient escrow/collateral model for market makers