Week#2 Tracking of Students’ Facial Expressions

Figure 1: New Class for Dataset

What Have We Done Last Week?

Last week, we researched studies similar to our project and benefited from shared papers about it. In these projects, we have observed that algorithms that obtain results from the instantaneous facial expressions of people are designed. Our aim is to design a program that detects the student’s participation in the lesson from the facial expression during the flowing video. For this reason, we stated that we could not find a project that achieved results close to the results we wanted.
At the same time, we started a dataset search suitable for our project. We have seen that the most comprehensive dataset we can feed our algorithm with is AffectNet — Sample. AffectNet — Sample has compiled a dataset of 8 different human facial expressions with approximately 5500 photos for each facial expression. But as we mentioned last week, we observed that many classes were miscategorized. In addition to these, we mentioned that there are some train sets that may be suitable for students and that we can create them ourselves.

What Have We Done This Week?

This week, we cleaned some of the data from the dataset and created a new class to improve our dataset.

Data Cleaning Process

Last week, we started to create solutions to the problems we encountered with the dataset. One of them was the misclassification of some images in the dataset. You can see this in the images below.

Figure 2: Images from the Neutral Class

You see a photo of Trump classified as Neutral.

Figure 3: Trump

We thought that this and many similar photos were not suitable for this class. In addition, we detected many examples of misclassifications for other classes as well.

Therefore, we decided to delete the photos one by one that we thought were misclassified among 44000 photos, even though we knew it would be difficult.
So far, we’ve deleted about 400 of what we think were misclassified photos. We will continue this process next week, as we think that this number is not enough yet.

Data Generation Process

Since the project we developed will work on students, we thought that there are some special facial expressions that students can show during the lesson. One of them is the ‘Sleepy’ facial expression. Since this class is not included in the dataset we will be using, we had to specifically collect this dataset. For this, we decided to take photos on Google with the web scraping method.

We decided to use selenium which is a web testing library for this and implemented two different methods to browse and download images.

The function you see below saves the photos in the folder.

# crating directory to save images
def download_image(url, folder_name, number):
    # writing image to file
    response = requests.get(url)
    if response.status_code == 200:
        with open(os.path.join(folder_name, "image" + str(0) + str(number) + ".jpg"), 'wb') as file:
            file.write(response.content)

The following function scans the photos one by one at the entered URL address. Precautions have been taken against some problems that may occur in this function. The first one of these is that Google images create a container named ‘Related Photos’ for every 25 photos. The program will get an error because no photos are stored in this container. To avoid this, containers with multiples of 25 are skipped.
Apart from this, another important problem is the problem of resolution. You may have noticed that some high-resolution photos need to be loaded when clicked. During this time, we see the low-resolution version of the photo. To solve this problem, we defined a timer. Thanks to this timer, the program waits up to 10 seconds for the photo to be uploaded, if the photo is not uploaded, it downloads the low-resolution version and continues.
In addition, “driver.execute_script(“window.scrollTo(0,0);”)” automatically scrolls down and accesses more photos.

def webscrapper(driver, url):
    driver.get(url)

    # we need to scroll the page to see every image on the page
    driver.execute_script("window.scrollTo(0,0);")

    # we need to reach to the containers
    page_html = driver.page_source
    pageSoup = bs4.BeautifulSoup(page_html, 'html.parser')
    containers = pageSoup.findAll('div', {'class': "isv-r PNCib MSM1fd BUooTd"})
    number_of_containers = len(containers)
    print("Found %s image containers" % number_of_containers)

    for i in range(1, number_of_containers + 1):

        # to avoid the 'Related Searches' on the page
        if i % 25 == 0:
            continue

        # getting the url of the small preview image
        preview_image_xPath = """//*[@id="islrg"]/div[1]/div[%s]/a[1]/div[1]/img""" % (i)
        preview_image_element = driver.find_element("xpath", preview_image_xPath)
        preview_image_URL = preview_image_element.get_attribute("src")

        # clicking on the container
        xPath = """//*[@id="islrg"]/div[1]/div[%s]""" % (i)
        driver.find_element("xpath", xPath).click()

        # starting a while True loop to wait until we get the the URL inside the large image view
        time_started = time.time()

        while True:
            # xPath stays same
            image_element = driver.find_element("xpath",
                                                """//*[@id="Sva75c"]/div/div/div/div[3]/div[2]/c-wiz/div[2]/div[1]/div[1]/div[2]/div/a/img""")
            image_url = image_element.get_attribute('src')

            if image_url != preview_image_URL:
                # this breaks loop, because we got the high resolution of the image
                print(
                    "The preview of the image and larger size of the image have different resolutions. Got the high resolution image.")
                break

            else:
                # making a timeout if the full resolution image can not be loaded
                current_time = time.time()

                if current_time - time_started > 10:
                    print("Timeout! Program will download a lower resolution image and move on")
                    break


        # downloading
        try:
            download_image(image_url, folder_name, i)
            print('....................Downloaded %s/%s' % (i, number_of_containers))
        except:
            print('....................Could not download the image index with %s' % (i))

We will add the full version of this code to the project’s repo in the coming days.

Below you will see some outputs obtained when the program runs.

Figure 4: Downloading Images

Figure 5: The page that opens when the program runs

With this function, we entered different URL addresses and created a data set. You can see it in Figure 1. However, as you can see in Figure 6, some images contain animated faces instead of human faces.

Figure 6: Animated human face

We decided to clean up some images like this and finalize the dataset after we had the entire ‘Sleepy’ class.

Conclusion

We made some changes in our dataset in order to increase the accuracy of the algorithm we will set up and to improve our program. We cleaned up some images and added a new class to try to customize our dataset.

Thanks for your time.

by Arda Altunkaya and Gökçenaz Akyol.

References

[1] “Web Scraping (Downloading) Google Images using Python + Selenium Webdriver”.

https://www.youtube.com/watch?v=Yt6Gay8nuy0&ab_channel=IvanGoncharov. [Accessed; Nov 19, 2022].

[2] “Python Google Image Web Scraping”.

https://www.youtube.com/watch?v=QZn_ZxpsIw4&t=189s&ab_channel=OhYicong. [Accessed; Nov 19, 2022].

[3] “Selenium with Python”.

https://selenium-python.readthedocs.io/. [Accessed; Nov 19, 2022].