Designing an Advanced Algo-Trading Infrastructure with AWS: From Data Acquisition to Order Execution
Introduction
Welcome to Alphagrow’s Trading Platform: Empowering Automated Cryptocurrency Trading through Data-Driven Algorithms
At Alphagrow, our skilled team is committed to developing innovative, data-driven algorithms that enable automated cryptocurrency trading. Although the algorithms are the center pieces of our platform, they are part of a larger and more complex system that demands various components to realize their full potential.
In this blog post, we will examine the typical logical flow within a trading platform and explore the AWS services that we’ve employed in our trading platform design. We will focus on the following successive stages:
- Acquiring Market Data
- Transforming Data
- Generating Trading Signals
- Executing Orders
- Post-Processing Orders
In a nutshell, our architecture capitalizes on widely-used AWS services such as:
- AWS Lambda: A FaaS, i.e Function As A Service, ideal for short-lived jobs
- ECS Fargate: Facilitates deploying docker containers for long-lived jobs
- DynamoDB: A scalable, distributed, low-latency database for document storage
- AWS RDS: An AWS service for hosting and maintaining SQL databases like MySQL and Postgres
- S3: AWS solution for large-file storage
If you are short on time, you can jump straight to the last section “Putting it all together” where we provide a summary diagram of our infrastructure.
1. Acquiring Market Data
A crucial function of our trading platform is collecting market data from exchanges, which serves as the driving force behind our algorithms. Exchanges generally provide this data via Websockets and REST APIs. We categorize the data into two types: time-series and ad-hoc. Throughout the rest of this article, we will use Binance as the integrated partner exchange; however, the logic is applicable to any other exchanges.
a) Time-Series Data
This type of data encompasses elements like prices and bid-ask spreads. We gather it using Websockets to ensure up-to-date information while avoiding overloading the exchange API or surpassing rate limits. Time-series data is stored in an SQL database, managed by AWS RDS, which allows for quick read operations and easy querying.
b) Ad-Hoc Data
Ad-hoc data, such as asset lot sizes and trading pairs, doesn’t necessitate frequent updates. We obtain this information using REST APIs and store it in DynamoDB for its flexibility, speed, scalability, and minimal maintenance requirements. AWS Lambda, a cost-effective and short-lived FaaS solution, is employed for executing REST API calls.
2. Transforming Data
Once acquired, market data must be transformed to be compatible with our algorithms. This may involve cleaning data or truncating time-series data to create fixed-length data frames. Since our time-series data frames can contain thousands of timepoints, we use S3 for storage, which supports large file objects and flexible data structures and formats.
3. Generating Trading Signals
To calculate trading signals, our algorithms primarily leverage market data and portfolio information, with the latter referring to open positions and related details. Both types of data are stored across various storage solutions (SQL database/RDS, file storage/S3 bucket, document storage/DynamoDB).
An AWS Lambda service retrieves the data, combines it, and applies our proprietary algorithms to generate trading signals. These signals are then sent to the Orchestrator, which stores them in memory and initiates the order execution process if conditions are met.
4. Executing Orders
The Orchestrator sends an order request to the “Booker” service, also running on AWS Lambda. The Booker first checks the order’s validity, ensuring there is enough portfolio space and the asset isn’t “blacklisted.” If the order is valid, the Booker executes it on the crypto exchange via an API call. The order information is then stored in our AWS RDS SQL table and DynamoDB for further analysis.
5. Post-Processing Orders
After executing an order, additional tasks are necessary, such as updating the portfolio balance and broadcasting the order to notification systems and third-party platforms. We once again use AWS Lambda to run these post-processing tasks. The Booker triggers these tasks, and their status (Pending, Failed, Success) is stored in DynamoDB. Tracking task status allows for retries if the task fails.
Putting it all together
Conclusion
In summary, Alphagrow’s advanced trading platform showcases the power of data-driven algorithms in optimizing cryptocurrency trading. By leveraging a suite of AWS services, we have built a robust, scalable, and maintainable infrastructure that efficiently handles various aspects of the trading process, from acquiring and transforming market data to generating trading signals, executing orders, and post-processing orders.
Our cloud-native platform, designed with modularity in mind, allows us to tackle latency challenges while emphasizing scalability and maintainability. This approach enables us to streamline our operations, adapt to rapidly evolving market conditions, and deliver consistent performance for our clients.
By sharing our insights and experiences, we hope to provide valuable information for those seeking to develop or enhance their trading platforms. The combination of cutting-edge algorithms and a carefully engineered infrastructure lays the foundation for success in the competitive world of automated cryptocurrency trading.
