Twitter Credibility Assessment With A Jupyter Notebook Bug
A couple of months ago I was working on a project when I got this error in my Jupyter Notebook while I tried running my code. It took me some time to understand the issue and fix the bug. The reason why I felt the need to write about it is because I couldn’t find any proper information about this issue when I searched the web.
So before I talk about the issue I would like to share what was the project I was working on. I was creating a twitter credibility assessment application that was supposed to fetch the tweets of a user in real time and save it in a json file. The id generated with each tweet was extracted and stored in a separate csv file. Then we extracted the features of another set of real time tweets fetched using Twython library where the id was compared with the one fetched with tweets.
Twython is basically a python wrapper for the Twitter API which supports both normal and streaming twitter APIs.
Feature extraction was done using the given code:
This generated a file with all the defined attributes fetched from the tweets. Following image shows the structure of the csv file generated after applying the above process:
After creating the above file normalization was done to fetch more important features from it and put it in a dataframe and then a Random Forest Classifier was applied to assign labels of highly credible(HC), non credible(NC),controversial(C) and neutral(N) and accuracy was predicted by comparing these labels with the manually defined tags.
You can have a look into my project here:
Now the error was generated when I installed Twython package using pip3 and tried importing it in Jupyter Notebook.
So I firstly ran the command pip3 install twythonwhich successfully installed the package on my laptop. But when I tried importing it in Jupyter Notebook it gave an error “No module named twython”. I thought there might be some problem with this specific package so I switched to a similar package Tweepy. But I got a similar error with it too. So I firsty checked if I have python installed in my system or not. I was working in a virtual environment and I installed all my packages into it.
After this I checked whether my notebook is recognizing my installed twython and tweepy packages or not. To my surprise when I ran the command !pip3 list| grep 'tw' I got the following output:
So what was happening is the packages were installed in my virtual environment but I was not able to use them. I looked for similar kind of errors on internet but couldn’t find anything that could help me here. I thought might be the packages are broken so I even reinstalled pip3, created the environment and reinstalled all the packages again but nothing worked.
Then after a lot of efforts all going in vain, I noticed that I have been running my Jupyter Notebook using the Anaconda Navigator so I probably should give a try installing the packages using conda. So I ran the command conda install -c conda-forge twythonand tried importing the package in my Jupyter Notebook.
And it worked!!
So after analyzing this a bit I realized that when you are working on Jupyter Notebook or any other application using Anaconda Navigator always prefer installing packages using conda rather than pip or pip3 because those packages will get installed on your virtual environment but are not supported by conda so ultimately the application won’t recognize it.
