Technology evolves fast. It has become an important part of our lives, it has given us many potentials and heavily influenced our habits and thoughts. As it progresses and advances much faster than the human kind, what could happen when it eventually surpasses us, and the possibilities it provides us with form negative consequences on the society? Should we have too much power?
In short, this topic is what the British science fiction series, Black Mirror explores. Most episodes are set in a dystopian near-future societies, imagining outcomes of different impacts that new technologies could have on people. Almost every episode leaves you with a bitter taste in your mouth. One of the episodes that was not too easy to watch was Be Right Back, the first episode of the second season.
I will try to explain the plot without spoiling it, so I will not go into details. After her boyfriend’s death, a woman signs up for an online service that uses data from the deceased’s social networks to create a chat platform that communicates just like them, in order to console the aggrieved party. It wouldn’t come as a surprise if i said that it didn’t end well, would it? Nevertheless, however creepy it seems, I can not help but wonder if such an algorithm is possible. Could they have done that?
Let’s split the problem. First, they needed person’s data to create a personality for their product. Secondly, they needed a software that is used for communication. And finally, in order to enable the phone calls with the robot, they needed both audio data and voice-recognition and pronunciation software. In the first part, we’re going to discuss collecting data.
In the series, it is implied that they used the person’s Facebook and Twitter posts to imitate their manner of speech: the expressions, vocabulary, even their sense of humor. To extract data from websites, one might need a software called Web scraper. Web scraping represents the action of collecting data from a website. They are relatively easy to build, and there are many advanced scraper software out there. The best way to understand how something works is to build it, so we might write a simple twitter scraper in Python. The purpose of the following scraper is to extract tweets written by the official account of Black Mirror series, and print them out on the console.
There are tools that make scraping easier. For example, Selenium is a very useful framework when it comes to web applications. Although it is not used in our scraper, I have found Beautiful Soup to be very handy when it comes to parsing HTML code. For starters, we need to provide a way to scroll down the page while looking for tweets, and a timer that pauses our script occasionally. This should be done because we don’t want to burden the server: it must be able to respond to other users’ requests.
import timefrom selenium import webdriverfrom selenium.webdriver.common.keys import Keys
Next, we must specify our browser and the link to the Twitter page from which we want to extract data. Our URL will be the URL to the targeted account, as shown in code below. Alternatively, if we wanted to scrape from search results, we would search for “blackmirror” and use the resulting URL for our script.
browser = webdriver.Chrome()addr = u’https://twitter.com/'query = u’%40blackmirror&src=typd’url = addr + querybrowser.get(url)
Now, let’s pause the execution to make sure that our page is properly loaded and that we’re not sending to many requests to the server.
time.sleep(1)When we explore the Twitter feed ourselves, it is easy to scroll down once we have read the displayed content. Same needs to be done here, and this is why we need Selenium framework: it provides us with page navigating. By sending keys to our browser, as we have done in code below, we can simulate pressing a key on our keyboard.
body = browser.find_element_by_tag_name(‘body’)for _ in range(5):body.send_keys(Keys.PAGE_DOWN)time.sleep(0.5)
To get the content, we will need to inspect our page to see which class contains the text that we want to print. Once we know that, we can extract the text that we need from the browser and print it.
tweets = browser.find_element_by_class_name(“js-tweet-text-container”).textfor tweet in tweets:
print(tweet)
Simple, right? These were the basics to scraping Twitter posts. Facebook scraper could be written in a similar way. The tools that I used provide many more features, and can be used to build powerful applications, and I encourage you to experiment with them: you will be amazed at what can be done.
The next topic to discuss is the second fragment of our problem, building a conversational software using data as an input. For now, I will just mention a few similar software: Siri, Cortana and Alexa.
