Quick Links: Github We have been using Airflow as a key workflow management tool at Robinhood for a while. In addition to the traditional use cases like analytics and metrics aggregation, we use Airflow for certain brokerage operations as well, including our clearing system. However, as Airflow usage within the…

Amazon’s compute-intensive C5 instance family has multiple instance types which can make it difficult to choose the right instance type for your use-case. In this post, we will look at the differences between the C5, C5d and C5n instance types. C5 instance family Amazon added the C5d instance type at the beginning of…

AWS recently launched Fargate which lets users build and deploy containerized applications without having to manage the underlying servers themselves. Previously, if you wanted to deploy a container on AWS, you would have to first provision a cluster and then deploy your container onto that cluster. …

One of the benefits of using a cloud provider like AWS is the ease of spinning up resources such as a new EC2 instance or storing more data in S3. As developers, we are generally more focused on getting our product or feature to the users as quickly as possible…

Amazon recently introduced the next generation of their compute-optimized instances: EC2 C5 instances. Compute instances are well suited to compute-heavy workloads such as batch processing, distributed analytics, high-performance computing etc. If you are considering whether you should migrate your workload from the previous generation instance families to C5, then the…