Storing, Calculating and Communicating Big Numbers
Building PlayGame from scratch brings the engineering team to something new every day. One of those things is dealing with very large numbers. The smallest unit of Ether is Wei. This post is written to address how to store, calculate and communicate big numbers.
Our first rule of thumb is that we will always communicate in Wei anywhere in the stack. A single Ether is equal to 10 to the power of 18 Wei. As you can see, this is a very big number. We communicate in Wei so that there will be no rounding and precision is lost during operations. More about the units here.
Disclosure: Before PlayGame, we’ve built a stateless trading bot. We got a little bit of experience there to prepare for building a cryptocurrency platform. We know a little bit more now.
Most programming languages and databases will come with these constraints with Integers:
- Positive integers max size is usually 2⁶⁴-1
- JavaScript’s Number.MAX_SAFE_INTEGER is 2⁵³-1
Our frontend is based entirely in React so I figure I put in a language specific constraint. The backend is Python3 based and we can do arbitrary sizes for integer without losing precision.
The big questions is how do we store the numbers?
I’ve looked into databases and the max integer limit holds at 2⁶⁴-1. You’d ask why is this a problem right? Here’s how Ether is represented to explain:
1 Ether = 10^18 Wei>>>((2 ** 64) - 1) - (1000000000 * (10 ** 18))
-999999981553255926290448385
PXG has 18 decimals with a supply of 1 billion tokens, hence the above equation.
The answer on how to store: Strings. This affected the whole cycle of not just storing but also calculating and communicating very big numbers.
I choose MongoDB since it store strings as UTF-8. Since Python3 also store strings as UTF-8, let’s see how big the storage would take up.
>>> a = '1000000000000000000000000000'
>>> sys.getsizeof(a)
77The largest string value to be stored is exactly 77 bytes. A MongoDB BSON document can have a maximum size of 16 Megabytes which is more than enough.
With the information above, let’s simulate how big the storage would be with these assumptions:
- 1 million users
- 1000 ledger entries per user
>>> users = 1000000
>>> ledger_entries = 1000
>>> size_of_entry = 77
>>> users * ledger_entries * size_of_entry / 1024 / 1024 / 1024
71.71183824539185There we have it, 71.7 GB of storage. Not bad since storage is cheap nowadays. Since we’re using MongoDB, the data will be around 35% smaller due to being saved as BSON. This would reduce the total size to 25 GB.
Storing numbers as strings has its own computing overhead. To avoid precision loss, we need to calculate using the CPU. There are ways to mitigate this which I would gladly write about in the next postings.
PlayGame is hiring new talents, join our Telegram community and mention me there. Cheers!
