How I collected data for my PhD research during the COVID-19 lochdown
The University of Glasgow closed down due to the outbreak of Coronavirus (COVID-19), therefore I decided to collect my research data at home. A students’ accommodation half a mile from the University.
My research focuses on studying the physical properties of garments (such as shape, weight, stiffness, etc.) to enable a robot to fold garments. To avhieve this, my aim is to first learn the shape change of several pieces of garments when they are grasped from the ground to a point in the air and then released from this point to the ground.
For the purpose of collecting images of clothes being grasped and released, I planned to use a Baxter robot (https://cobotsguide.com/2016/06/rethink-robotics-baxter/) to do the grasping and dropping task, however, when COVID-19 hit Scotland, I had to come up with a way to collect the data by myself. Instead of using Baxter to grasp the clothes and release them down, I chose to take advantage of my own arm to do the same job. At the end of the day, it does not matter who does the job, my focus at the moment is to understand how a neural network encodes the dynamic properties of clothing.
After deciding to use my arm to complete my data collection, I had to choose an effective way to segment the grasped clothes from the background, which will be a hindrance if they were not removed properly before training a neural network. The reason is that it will be difficult for a neural network to generalize all the details of a full image. A green curtain background is a traditional way for film producers to add some effects into shots, especially in science fiction films, because such background can be easily eliminated. Inspired by this idea, I decided to use a green curtain as my background and I would wear a similarly green garment for the purpose of eliminating the influence of my arms as shown in the image below.
Another prerequisite is selecting garments. The selected garments should be distinct from each other, and their physical properties should be different enough so that they will help a neural network to learn the dynamic characteristics of the garments. Therefore I choose pants, shirts, sweaters, towels and t-shirts, and each of them has a distinctive colour and a unique shape.
Doing research at home means that you will be less supported compared with laboratory equipment you can use at the University. The only equipment I took from the laboratory was an ASUS Xtion Pro camera, which is able to capture depth images. Depth images play a vital role in analysing data because they provide dynamic distance information about grasped garments.
Figure 1 shows an example of how I grasped garments using my arm. From the image, it can be seen that I was grasping a shirt when I was wearing a green sweater so that the influence of my arm can easily be eliminated. Although there were some shadows, they can also be eliminated easily by a simple colour threshold algorithm. After capturing the data, the next step is to segment the clothes from the images, which means eliminating the green background. Instead of using RGB values as a threshold, I chose to use the HSV values[1] [https://en.wikipedia.org/wiki/HSL_and_HSV] of the images to set the threshold. From the HSV colour space plot in Figure 2, we can see that the HSV values of the green background and the green sweater are within a concentrated area, which means it is easy to remove by thresholding HSV values (i.e. Values axis in Figure 2).
From Figure 2, we know that it should be easy to segment the clothes from the green background and green clothes by using their HSV values as a threshold. Therefore, from this idea, I got segmented images, of which an example is shown in Figure 3, its corresponding segmentation mask is shown in Figure 4, and an example of segmented depth images is shown in Figure 5. Depth images capture the distance between this pixel and the camera.
Eventually, I finished the data collection task and in two weeks I captured 80,000 colour and depth images and 40,000 masks.
In conclusion, in order to collect data at home, I have changed my flat to make an environment that fulfils the requirements of a laboratory setting, which included setting up a green background, wearing a piece of working clothing of similar colour with the background and choosing the right threshold to segment the clothes from the background.
My next step is to measure the weights of the selected garments and train a neural network to classify the weight of these garments (light, medium, or heavy). In this way, I can devise the neural network to learn about the physical properties of garments.
All in all, I hope Scotland obtains the victory from the COVID-19 so I can go back and continue my research at the University. Special thanks to the NHS for fighting against and beating this deadly virus. We are all in this together!
About Myself
My name is Li Duan, and I am a 2nd year PhD student at the Computer Vision and Autonomous System (CVAS) group of the University of Glasgow. My reserch focus is on investigating continuous perception and manipulation of robots.
