Trading Data Analytics — Part 0: Parquet Files and MinIO S3
Building upon our prior articles detailing the development of a comprehensive Trading Data Pipeline system for ingesting, transforming, and storing financial data, we now embark on the Trading Data Analytics series.
In our earlier series, we generated numerous partitioned Parquet files containing comprehensive financial data, including technical indicators, for our instruments. However, accessing this data needs conducting our data analysis from the same host or establishing mapped network folders. While this approach is functional, it could be enhanced by utilizing an object storage solution that offers remote accessibility, like MinIO.
MinIO is an open-source object storage server built for cloud-native and containerized environments. It is compatible with Amazon S3 cloud storage service, providing an alternative option for storing and accessing data in object storage format.
Creating a Minio container
Create a file named docker-compose-minio.yamlcontaining the following contents:
