Security and Scalability
How we keep Samsa secure and reliable
When we launched Samsa, our mission was to create an automated advising product for the cryptocurrency markets. Samsa is automated in that we deploy Samsa strategies (set by users or by our indexes) across users’ own cryptocurrency accounts. And Samsa is a product in that it’s intended for retail consumers, which means we have to deploy early to customers and iterate on their feedback. In the development of Samsa, this has led to a fundamental conflict between waterfall development of security and both the scalability features and agile iteration on the Samsa product.
Our philosophy at Samsa is to lean towards agile. We launched an MVP of the Samsa platform in January, onboarded $2.2M in customer capital over the first weekend, and then ceased marketing to focus on learning from an engaged group of customers (thanks to cliffy, JeremyC, cooperdooper, amirpc, AEye, and more). We believe that achieving product market fit starts with listening to customers. And in order to listen to customers, you have to give them something to talk about. For more on agile methodology, check out Agile Make No Sense by John Cutler.
Crypto is a special case of agile. Should you move fast and break things when you’re helping manage millions of dollars of customer capital? Ask Mt Gox, Bitstamp, DAO, or Bitfinex. Or more recently, Coinrail. Obviously any cryptocurrency product needs to have first-rate security, and security is not a trial-and-error process. And Samsa, as an automated platform, has tens of thousands of user services that need to be both secure but also highly reliable to ensure users’ funds are managed according to their settings. As an institutionally-backed business, we have a responsibility to our employees as well as our investors to not let the business “implode” via a disaster event, regardless of our scale, and we identified three areas we would focus on as a higher priority than agile:
- Security — User API keys should be secured with cryptography that can only be decrypted using resources off the private network.
- Scalability — The Samsa Platform must scale to 100,000+ users without major service disruptions (scheduled downtime is OK).
- Compliance — Samsa and its products should comply with applicable regulations, primarily United States Securities and Commodities laws.
In this article, we’re talking about 1 and 2.
Samsa’s launch product is Samsa Track, an analytics and automation tool for users’ own exchange accounts. To use Samsa Track, users upload their API keys to Samsa on their account page as seen below. Samsa uses these keys to view user accounts (analytics) and affect trades (automation). To ensure a good product experience, and make sure Samsa can trade when a user is not logged in, Samsa stores these keys on our servers.
Security of these API keys should be taken very seriously, as a recent Binance “hack” illustrates. While not necessarily gaining access to the underlying client funds, a hacker can use trade permissions of a hack to pump an altcoin and then sell during the pop. For a good explanation of API key security issues, check out this article by DDEX.
The first and most important aspect of securing client API keys is not having control of any account permissions a user doesn’t explicitly grant to Samsa. We show the user which permissions are shared with Samsa and provide the capability to change them at any time:
- View — the capability to view the users’ account and trade histories, used for analytics
- Trade — the capability to trade on the users’s account, used for automation
- Transfer — the capability to affect deposits or withdraws, never used by Samsa
We encourage users to grant only the permissions required for using the features of Samsa they need: View for analytics and Trade for automation.
Samsa protects uploaded API keys using public key cryptography. Public key cryptography can be confusing, so here’s a great guide by Panayotis Vryonis if you want to learn more. But for understanding how Samsa uses it, here’s what you need to know:
- A public key is used to encrypt the API keys.
- Only the private key can be used to decrypt the API keys.
Our public key is stored on our public servers, where it can be used to immediately encrypt and hide the user API key without further transmission of the data. The private key is stored on our dispatchers and relayers, which are the parts of the system that create tasks for viewing, monitoring, and trading customer accounts. Critically, the dispatchers and relayers are not exposed to public networks. Therefore they play a similar role in our system to a “secure enclave” you may have in your smartphone.
Without being a coder, you can see that the public keys we store on our public servers don’t even have the capability to decrypt user credentials:
The private keys, which are stored in our enclave, do have the cabaility to decrypt API keys:
For transparency, we have made our cryptography-specific code available publicly on Github. But you may have heard the phrase “Don’t Roll Your Own Crypto.” Don’t worry, this library simply wraps the popular and battle-tested pyNacl library in a way that makes it easier for our servers to digest.
In addition to public key cryptography, Samsa uses a number of industry best practices for securing its systems including but not limited to:
- Storing secrets in environment files
- Limiting permissions in AWS roles
- Restricting ports of production servers as needed
- Data-at-rest encryption of databases
- Production hardening of front end and back end
For more information on our security, contact firstname.lastname@example.org.
We believe that the world financial system (commodities and securities) will be converted into crypto; exchanges like Coinbase will become the next NASDAQ, leveling the playing field for financial; and platforms like Samsa will sit on top of these exchanges as the next incarnation of Betterment, Vanguard, or Robinhood. To put it simply, we are working on a huge problem. And this means Samsa needs to be scalable!
The Problem We Have That Everyone Has
We can start by saying scaling is an industry-wide problem. Millions of people are moving their investments to crypto, driving trillions of events (orders, page loads, transfers) to exchange servers each day. And while the markets may be down for the moment, two major trends will drive orders of magnitude more traffic to crypto infrastructure:
- Regulated crypto exchanges will open the floodgates to trillions of dollars of institutional investment capital, which is often traded using sophisticated algorithms that generate higher exchange activities.
- Asset-based tokens will migrate potentially hundreds of trillions of dollars of investable assets, and portions of these assets’ investor base, to crypto exchanges for the first time.
These features either already exist or will be added based on user milestones, potentially in combination with a waitlist or scheduled downtime.
As a platform that processes user events, Samsa is implementing many at-scale features that are typically rolled out as a platform scales in user adoption. These features either already exist or will be added based on user milestones, potentially in combination with a waitlist or scheduled downtime:
- Compilation of front end software and distribution on an edge network
- Edge distribution of media (images, audio, etc.)
- Load balancing and scaling of public servers
- Database sharding, replication, and scaling
- Media caching of public data such as index price histories
DevOps and web infrastructure scaling are something of a trial and error process: you hypothesize about bottlenecks, watch user experience data, and run a team responsive to the situation on the ground. If we succeed at exciting users, we have a team excited to scale the platform. But we have another problem to deal with..
The New Problem Samsa Introduces
Samsa is different from crypto exchanges, because we are introducing a critical new technology to the crypto space: automated trading.
Manual trading drives events to crypto exchanges, which are processed as they come.
Let’s say an exchange has 1,000 orders that place an average of 3 order per day. This exchange has to process 3,000 order events a week.
Automated trading happens automatically (!) and periodically. Unlike events, which occur only when a user takes an action, Samsa relies on services which create events for users automatically. Here are some services that Samsa runs for each user, and on each exchange:
- Account monitoring — Samsa checks the balance of each account every 15 minutes = 672 events / user / week
- Account rebalancing — Samsa rebalances user accounts based on a user-specified frequency (let’s say 4 hours) and generates trades (let’s say 5 trades per rebalance) = 280 events / user / week
- Order monitoring — Samsa monitors placed orders every 5 minutes, so assuming the trades above are monitored for an hour after placement =3,360 events / user / week
- Analytics — Samsa compiles trade histories every hour to provide analytics on trade histories = 168 events / user / week
So to service the same set of 1,000 users, Samsa has to process as many as 4,480,000 events per week.
To adapt to this massive number of events, Samsa uses a distributed task queue as a decentralized group of workers that auto-scales to accommodate task queue load. Tasks are generated by dispatchers and relayers, and then consumed by workers. This is a decentralized computing architecture. The workers themselves can concurrently process multiple tasks simultaneously, using a local record keeping system to make sure they don’t talk to exchanges more often then permitting (and keeping us from getting blacklisted from the exchange!) This is a parallel computing architecture.
One of the coolest aspects of this architecture is that tasks can split up subtasks and submit these subtasks to the task queue for processing. For example, a large account with many trades on a given rebalance interval is split into subtasks so its trades are given the same priority as a small account.
Another cool aspect is the way we randomize trade execution for rebalancing. Exchange accounts are initialized with random group id’s and offsets to smooth out trading throughout the 24 hours a day these markets are open. Trade execution randomization ensures steady load across the system throughout the day and may even help stabilize the crypto markets long-term.
Samsa also runs a lot of analytics jobs across our entire user base such as calculating funds managed on the platform, ranking trader performance, and computing user account metrics. These analytics tasks are performed in large batches and don’t congest our real time processing queue, regardless of how complicated the analytics get. This combined system of real-time and batch processing is called a lambda architecture.
Ultimately, we aren’t building this platform to be scalable as a goal in and of itself, but rather we are trying to achieve reliability as we scale. To this end, Samsa has taken the immediate step of adding status monitors to the Samsa website, which can be viewed by selecting “Status” in the website footer. The scoreboard is up, and we will continue to do our best!
We took a gamble that, based on our interactions with customers after our initial launch, we could build a solid foundation for a Samsa which will excite Samsa customers for years into the future. We’re excited to make our months-long investment in security and scalability public for the first time. But now the real work begins. I hope you join us as Samsa customers as we iterate the Samsa product together!