Ethereum Challenges While Launching IPLv2

Kitties, confirmation times, pending transactions, high fees — our experience from the battlefield

Our goal is to enable real mainstream users to earn and spend Kin on a daily basis and in as many different digital services as possible. To do so we need to place Kin in the hands of real users and enable them to seamlessly spend and earn. This is the purpose of IPLv2, the first Kin product which harnesses “on-chain” Ethereum transactions as part of production operations of Kik, the messaging mobile app.

Kin, however, is a decentralized ERC20 token on the Ethereum blockchain. The Ethereum blockchain is designed to mine new blocks approximately every 12 seconds. This means that even in ideal network conditions, confirming a spend transaction can create a delay which will affect user experience (e.g. a user that spends Kin on a sticker pack shouldn’t wait 12 seconds until he can use it). Furthermore, a user earning Kin may wish to spend it right away; they shouldn’t have to wait until its “on-chain” balance is confirmed and updated in order to spend his Kin.

Mainstream users expect an instantaneous confirmation time and high service availability, so they don’t necessarily understand how the Ethereum blockchain works; their main concern is getting valuable services and they wouldn’t/shouldn’t have to understand the underlying technology.

To make our challenge even more difficult, real Kin “on-chain” transactions on the Ethereum blockchain incurs a real cost. To reduce costs, we can adjust transaction fees by lowering a transaction’s “gas price”; however, a low “gas price” can cause an even longer confirmation time and hence negatively affect user experience. Moreover, setting a very low “gas price” can cause transactions to become effectively ignored by miners and eventually rejected completely. We shouldn’t allow such a scenario to happen as a user might never get his earned kin or won’t be able to spend kin and enjoy the product.

So how can we mitigate these challenges and ensure great user experience while lowering operational costs?

First, we introduce a “pending balance” — instead of just showing the user’s “on-chain” balance, we also calculate any incoming or outgoing pending balances. This Ethereum network capability allows us to update the balance in a “real time” fashion. In addition, we allow the client-side to send a spend transaction without checking user’s “on chain” balance. In this case, server-side logic can calculate the transaction likelihood to eventually be confirmed (based on user balance, pending transactions, gas price, etc.) and can decide to provide service immediately, even before final “on-chain” confirmation. Combining this process allows us to provide a seamless experience while lowering transaction costs.

The Plot Thickens

While working on the project, we mostly test on Ropsten testnet. Before the final launch, we want to run a full scale, end-to-end test on Ethereum Mainnet with as many testers as possible. This will require us to supply QA with real Kin and Ether. Initially we thought, “No problem, we can just ask accounting to send us the required funds and we are good to go.” However, reality had a different plan — the CryptoKitties craze started just a few days before and was said to be responsible for ~10% of the Ethereum traffic; by the time we were ready for final QA testing, it already clogged the entire network. We had to wait 48 hours until transactions were confirmed and QA’s Ethereum account balances updated (even though we used very high gas prices). Even then there was no point in starting to test— selected IPLv2 users need to opt in and accept the term of service in order to receive initial Kin award (and also small amount of ETH to fuel transactions). Without this ETH, all of a user’s transactions would be immediately discarded. User experience with a 48-hour delay is far from being acceptable :(

Not only was the network clogged, but all network transaction fees are paid in ETH, whose price more than doubled in the last 3 months. A high ETH price together with a congested network condition increased transaction fees dramatically in a very short time span. We basically had to wait for the network to settle down before starting QA testing and eventually start awarding real users.

As workarounds to these issues, we are monitoring the network state and the gas price and adjusting transactions fees accordingly. The first onboarding award transaction is the most critical and requires a faster confirmation time (and a higher fee). Our BI team monitors the onboarding funnel and, together with our server team, adjusts the gas price to both ensure a good onboarding experience and to minimize the “user acquisition” cost. Any earning and spending transaction gas price after the initial onboarding can be relatively lower as we only require that the transaction will eventually be confirmed, thus allowing a longer confirmation time and lower fees.

Overall we were extremely successful in providing a seamless “on-chain” end-to-end user experience. The key was to abstract blockchain behavior/design from the user’s eye as much as possible.

In order to scale IPLv2 even further, we still need to reduce transaction fees plus confirmation time and to address the volatility in cost and speed. Furthermore, a 48-hour or even a 1-hour down time might be expected by cryptocurrency users but is not acceptable for any user-facing service.

This project taught us so much and our teams will keep working to overcome these challenges, so stay tuned...