Concluding Popbot

‘Popbot’ is the outcome of the initial idea to create a bot which represents a region and composes tweets based on trending topics in the region. The further stages which would make the bot interact more directly with individual users was not implemented nor was deeper knowledge through news articles implemented.

Throughout the project, I tried and tested many different possible directions for the bot. Some of my tests focused on taking a machine learning approach however learning about how machine learning works, the many different methods and ultimately the complex implementations, lead to a change in direction. The possibilities for machine learning were amazing however I faced many challenges, the most difficult was finding and understanding the breadth of knowledge required for simple tasks. It was difficult to find usable methods for learning on a data set with some of the more popular libraries, I made some limited progress using Tensorflow however progressing the learning from predicting a single trait to predicting sentences was beyond the scope of what could be achieved within the time.

Another limitation of machine learning was getting the right data in large quantities, the amount of data required is enormous and although there are some great sets available for use, the information I needed was difficult to get easily. I wanted to construct sentences based on tweets of a trend, the first step would be to train a neural network on hundreds of thousands of tweets so it could start to gain an understanding of language in the context of language used when tweeting not well crafted written language. It was also necessary to have the algorithm learn from trending tweets which would provide data in smaller amounts. This challenge was too much to handle within the time considering the processing time to train the network.

I decided to look elsewhere in favour of a more manageable approach to the problem. I settled on some NLP libraries, mainly Spacy in the beginning as it offered some powerful part-of-speech tagging which had been trained on a large corpus, it was accurate and fast for the most part — it lacked finesse with tweets however. I made a lot of progress analysing the data and pulling interesting statistics from tweets however it was difficult to find a practical application for the data within Spacy itself. I attempted to process the tweets in Spacy first followed by some other stages to take the data and construct sentences. The result was very random and couldn’t pass as readable.

I slowly began to integrate other frameworks and even re-wrote sections using Textblob so I could work more closely with the primitive types and have more control over how the tweets were interpreted. The bot was able to form more coherent sentences but they were basically ‘Mad Libs’, key words would be replaced in a randomly selected tweet with top trending words which ‘looked’ similar. This approach had a high chance of creating tweets which made no sense but could be interesting when it worked.

The final evolution of the bot is what was described in my post, Making a Compromise, which takes elements of the previous iterations however it combines this with a much more structured approach and includes an emoji representing overall positive or negative sentiment toward the trend. The bot also uses the trend’s query to attempt to find relevant gifs to use in the tweet. The combination is not always perfect but the the viewer can often interpret it in interesting ways.

The bot is available here:

The code for this iteration of the bot is available on GitHub, it includes commented out parts which were cut due to the ‘Mad Libs’ effects described however these portions offer more fine-grain control.

The next step for the bot if I were to continue working on it would be to integrate Tracery.js; an option I almost took which is part of the reason why I changed from Python to JS and Node. I think that by collecting the top words used in tweets and creating categorised word bags like I did originally is a good approach. The implementation of the sentence building however is the most difficult to achieve believably, I think that by analysing some good sentence structures and building a robust grammar in tracery could result in much better tweets that are human readable and contain much richer information.

I think my original vision for the bot was a bit over ambitious however I think I achieved the first stage. I think there is great potential for bots and through my research, I would like to create more bots that utilise platforms like twitter.

Title Image: Source