Stock Market Real-Time Data Analysis
Ever since I came to the realization that I am about to graduate from the university I started thinking about my future and career. I looked at the world… see where things going in my home country… judge my skills and knowledge… and last but not least my financial situation. After one year of considering all my options and doing some fact-checks on my life then I came to a conclusion. So I want to have full control of my life. Both financially and professionally.
At that very moment of the conclusion I drew after a lengthy-thinking process, I decided to take part in the finance world. The idea of being the actual driver of your finance together with my profession sounded perfect to me and my living style. Eventually, that path led me into the world of quantitative finance and I attended an MA in Quantitative Finance program in Warsaw but I had to drop out due to personal reasons shortly after I finished the first semester successfully. However, I kept improving myself both in Software Development and Finance/Trading. Python was super helpful to combine these two disciplines as it has been used widely at investment banks and hedge funds where its capabilities in data analysis and machine learning are proven good use cases for the industry.
You can find my other articles about Python and Quantitative Finance on my website here: https://www.quantitativepy.com
In Wall Street We Trust!
In this article, I will share my journey to build a stock market real-time data analysis project that will help me to identify the recent trends and strengths in the stock markets. Probably it will be a long journey however I will try not to bore my readers with all unnecessary points and just stick to the core idea of the project and explain how I overcame the challenges and why and how I do certain things in order to solve those challenges.
I hope it will be useful for everyone who has a similar interest as mine.
Read even more exclusive content by subscribing QuantitativePy Insights newsletter below:
The Architecture Design
You can see the basic structure of the system design I have been thinking of.
Technology Stack
- Programming Language — Python
- Amazon Web Service (AWS)
S3 (Simple Storage Service)
Athena
Glue Crawler
Glue Catalog
EC2
In basic principles, the system will work as follows:
1- Python script will feed the raw data from the sources to the Kafka EC2 cluster as a producer.
2- Kafka cluster will do data gathering-storing-analyzing jobs
3- After the initial analysis is done, Kafka will distribute the data to the Amazon S3 bucket service
4- As a last step, the system will utilize the AWS Glue and Crawler for data querying and monitoring features.
Some glossary you might face while building such a system on your own:
- Stream: Unbounded sequence of ordered immutable data.
- Stream Processing: Continuel calculations and processing of streamed data
- Immutable Data: The data type that can’t be changed once created.
- Event: Trigger effect regarding something that has occurred in the system.
- Broker: A single member of the Kafka server.
- Cluster: A group of Kafka Brokers serves together.
- Node: A single computing instance.
- Zookeeper: To orchestrate the Kafka Brokers on the go.
That’s it for this post! I will be giving more details in the next posts while keep building the system.
Read even more exclusive content by subscribing QuantitativePy Insights newsletter below:
