Does President Trump’s tweet has any correlation with stock market prices?
A Deeper Look For Insights
Topic : President Trump’s Tweets vs Stock Market Prices — An Analogy
Abstract:
In this article, I will primarily explain my analysis of President Trump’s tweets and the stock market prices and figure out if there is any correlation between these two.
To accomplish this, we perform 4 tasks. First, we mine tweets using Twitter API and subsequently process them for analysis. Second, we retrieve the market prices from Yahoo Finance and process them for analysis. Third, we will manually mark the tweets that contains texts with stock market information. Finally, we will combine the data and visualize our findings.
Note: The third task could be accomplished using machine learning techniques that searches the keywords correlating to the changes in stock prices but I want to keep it simple for this article and we will manually mark the tweets pertaining stock market information.
Introduction:
Everyday there are large amounts of tweets posted through twitter. According to ‘internetlivestats.com’, on average, around 6,000 tweets are tweeted on Twitter every second which corresponds to 500 million tweets per day. If each has a maximum of 150 characters and over 500 million tweets sent per day then that's over 75 billion characters generated per day on Twitter.
This large volume of data has been arguably providing valuable insights about public sentiments on various topics and has led to a recent growth in research of public’s sentiment and its relationship to financial markets. Determining the public’s sentiment by retrieving online information on the market can be valuable in creating trading strategies.
Especially when it comes to tweets by the President Trump, it is believed to certain extent that there is a correlation between his tweets and the market prices. Of course there are many other factors that are involved in the movement of stock prices, but President’s and public’s tweet are included as one of the factors.
Although there has been debates around if President Trump’s tweet is the background noise for markets or not, in this article we will manually mark the tweets posted by President Trump on the stock market and have a look at the market prices changes. There might be plenty of reason why the stock market plunged but we will focus on our goal to see if President Trump’s tweet has any effect on the market.
Analysis:
- Tweet Collection
For collecting the tweets, Twitter provides a powerful API : the Search API. The Search API is a REST(Representational State Transfer) API that allows us to request specific tweets. REST API simply uses HTTP methods to perform different operations. Depending on the request the API sends us a list of JSON/Twitter lists response that contains tweets data. This contains information such as username, location, time stamp, full texts tweeted etc. For our purpose of visualization, we focus mainly on the user name, tweet id, full tweet texts, location, time stamp and device used to tweet. Note: To access twitter API we need API Key for authentication. Once the API key is created we can easily access the API.
Before you run the code to get the tweet data, you need the following information. Twitter API documentation is self explanatory, but if you couldn't find the link, Click here.
- Step 1: Generate access tokens(authentication) : Login to your Twitter account on developer. Twitter.com. Navigate to the Twitter app dashboard (https://developer.twitter.com/en/apps) and create an app if you have not already created one. If you have created a twitter app already then open the Twitter app for which you would like to generate access tokens. Navigate to the “Keys and Tokens” page. Select ‘Create’ under the “Access token & access token secret” section.
- Step 2: Python installation and other necessary libraries: I have another article on the basics of installation and web scrapping. You can find the necessary links in it. https://medium.com/@chitu_rk/web-scrapping-with-python-part-1-a4962d455ea5
Once you have the above information with you, you can go ahead and run the following code.
from twitter import Twitter,OAuth
import pandas as pd
import numpy as np
from bs4 import BeautifulSoup
class Twitt:
def __init__(self):
consumer_key =
consumer_secret =
access_token =
access_token_secret =
self.t = Twitter(auth= OAuth(access_token,access_token_secret,consumer_key,consumer_secret))
def user_details(self):
pythonStatuses = self.t.statuses.user_timeline(screen_name="****", count= 10,tweet_mode='extended')
return pythonStatusesNext you can write the rest of the code to take the required information from the returned tweet status and write the data to a text file or csv. The output of your text file should look like this.
Now create a visualization to get deeper insights of the tweets collected. You can use any visualization tool of your preference, I used Tableau.
Open Tableau and Import the text file and then click on the sheet to create the following visualization.
2. Stock Market Prices Collection
After analyzing the tweet data separately, we collect the stock data for the same duration. To do this, you can use yahoo finance API to gather data for the duration of trading day or you can download the historical data CSV from yahoo finance. I used both methods for practice purpose but if you are a starter in web crawling field I would recommend you to download the historical data CSV from yahoo finance. The link to download the data: https://finance.yahoo.com/quote/YM=F?p=YM=F
Note: You can download the historical data for stock indices of your interest, I downloaded for 5 major indices : Dow 30, S&P 500, NASDAQ,CBOE volatility index and Russell 2000.
The output csv file should look like this.
Now create a visualization to get deeper insights on the stock market data collected.
Open Tableau and Import the csv file and then click on the sheet to create the following visualization.
3. Streamline Tweet Collection With A Tag
After looking at the data collected from the twitter, we need to associate each tweet with a label “Yes/No” if the tweet text has “Stock market” information.
- Open the text file in which you saved the crawled tweet data.
- For each twitter text, manually verify if the tweet has stock market information. Easiest way is to find and search(CTRL + F) for texts such as ‘Stock', 'Markets’ etc.
- Read the tweet and label the tweet ‘Yes’ like the one shown in the screenshot below.
Note: The reason we are manually marking the label is because we need a way to look for correlation between the tweets and the stock prices. As mentioned earlier we are manually marking the tweet text, but if interested you can create a machine learning model to use specific emoticons for sentiment classification or examine specific words from tweets that are associated with changes in stock prices., a technique that greatly reduces manual tweet tagging.
4. Visualize The Market Correlation
The final step after marking the labels for correlation would be to combine both the data sets and come up with a visualization that shows us the correlation between the tweets and the stock market variation.
Again, we can use tableau to help us with the visualization. We can blend the data from two datasets based on the ‘date’ and come up with a creative visualization such as the one below.
Open Tableau and Import the text file and then add the csv files as well. Make a full outer join on both data sets and join them based on the ‘Date’. Add filters to extract nulls if any. Now click on the sheet to create the following visualization.
The results of this visualization is certainly interesting. We can see information about meaningful events that indicates the stock market variation based on tweets. We also see there is a correlation between President Trump’s tweet and stock market prices.
For example, On December 4,2018, President Trump tweeted the following.
According to Fortune Magazine, as the markets opened on Tuesday(December 4, 2018),
President Trump began tweeting about his administration’s trade negotiations with China.U.S. stocks plunged as rising concerns that an economic downturn lies ahead and mixed signals from the White House on the state of U.S.-China trade talks rattled investors who had been in a buying mood earlier this week.
How much did the market price go down?
The Dow Jones Industrial Average lost 799.36 points. The S&P 500 Index lost 90.31 points, while the Nasdaq Composite dropped 283.09 points. Small-cap stocks were hit hard, with the Russell 2000 Index falling 68.21 points.
Do we have this data in our findings? Of course we have. If you want to look at the variations of stock prices of these tweets in our viz, here you go.
Conclusion:
Our scope of this article is narrowed and we cannot say that the President Trump’s tweet was the only factor for the stock price downfall. According to Greg Valliere, chief global strategist at Horizon Investments, there were plenty of reasons why the stock market plunged that particular week — a disastrous press conference by Fed Chair Jerome Powell, fears of a decelerating economy, continued anxiety over a trade war, etc.
However, the scope of our analysis was to analyze and visualize the correlation between the tweets and stock market prices by manually marking the correlation label, it would be an interesting area of future studies to naturally train a neural network model with words and sentiments that seem to correlate to changes in stock prices. In which case we might be able to see stronger correlation of words with stock prices variation.
Tableau Visualization:
Find the interactive visualization in my Tableau Public link. You can play around with different dates and different indices in the viz. But I recommend you to follow all the process specified above and create a visualization of your own. It would be a good practice.
Thanks for reading and awaken the visual greatness within!
When you visualize for the joy of visualizing rather than with the intention of correcting some deficiency, your thoughts are more pure and therefore more powerful.” — Abraham Hicks
