Our plastic QA engineer, or how to get data without people.
What essentially machine learning is? The technology for making the world a better place or just a matrix multiplication? Machine learning algorithms already changed the world. But if you the one who create such an algorithm, most of the time you deal with matrixes. The weights of your model are a matrix. The dataset you learning is just a bigger matrix. Maybe the only connection to the real world is a dataset origin. The more representative your data is, the more useful will your model be. But what you should do if you have no data? How to build a model without a representative dataset? The obvious answer is to find such data. In this story, I’ll tell the funniest experience of collecting such dataset in my career.
In Neatsy, we are creating a mobile app that use an iPhone camera as a foot scanner to capture 3D models for predicting a comfortable sneaker fit. This story is about a scanner module.
Calculating the scanner’s accuracy is quite simple. You need to gather people, manually measure them (this is a truly challenging part, and you can read more here), give them the scanner and compare the ground-truth measurements with the scanner’s. The task becomes more complicated when you want to use this data to train your model. In this case, you will have to log every user interaction, for example, all camera frames, the user collected. It will allow you to reproduce scanner algorithm, change it, and score on the same dataset. Even more complex and even impossible task becomes if you want to change the scanner UI. Let me clear out what I mean. There are a lot of ways to collect foot photos. You can design your scanner so that user must stand during the process, or sit, or keep it on the knee and all these variants change the geometry of a human foot! Also, you can collect video or separate frames. You can use, or not to use reference objects, such as standard paper sheet or a standard plastic card. When we started our project we were not sure which variant is the best, so we decided to build something that will work with any reasonable accuracy, and only then collect the dataset to adjust the algorithm.
So we built a foot segmentation module, a morphable model for a foot, combined it in a pipeline and finally got the first results. The challenge we got at that point was how to understand if the results are reasonable enough to start collecting dataset for adjusting the algorithm. The results were not even close to that we wanted — they were very unstable, and the trick was to find where the problem is. Two reasons may spoil the stability of the algorithm — the human foot is a changing object, so there always will be some lack of stability in the scanner, but maybe our way to scan was more unstable than other. The second reason may be in the algorithm. It may be just not good enough. We have tested all separate components of the algorithm on their own datasets, and they had quite good accuracy. So the only thing we need was to test the whole pipeline from camera frames to measurements. Testing the algorithm on our foot didn’t solve the challenge. We have ordinary feet, that are always changing, while we need the one that will never change. The solution is obvious. We need an artificial foot.
The ideal model for our case we quickly found on the Internet. The silicone foot model with detailed shape and skin. The only disadvantage was the delivery time. We would have to wait for about two months, which was unacceptable for us. So we decided to find something in our city.
The next morning we were already seeking for a mannequin store in the Moscow industrial zone. The store focused on clothing stores. Such dummies do not have to repeat human anatomy accurately. Just being similar is enough. However, we decided to try our luck. The choice was wide, but nearly all the figures did not fit us. Either they had no developed fingers — all but the thumb merged into one, or they were standing on a special platform attached to the heel so that we could not rest our foot on the floor, as a person does. Fortunately, we found a mannequin with matching legs, but they refused to sell us just the feet, so we bought the whole thing.
Having got a plastic leg, it remains only to add colour diversity. The dummy was completely black, so we decided to paint one of the legs beige. To buy the paint, we went into the nearest hardware store. A consultant was a bit surprised at our request. He told that not every plastic can be painted, and we should choose the paint depending on mannequin type. Unfortunately, we didn’t find any stamps on the figure. So we took the most universal one, and in addition, the piece of sandpaper to make the foot rugged, so the paint would stick better.
With all this stuff we returned to the office. As our lessor said, painting right in the office was a bad idea. Unable to find any technical room, we decided to paint the leg just on the street. Rubbed the foot with sandpaper, applied paint, and went for a walk until the paint dries.
The resulting foot gave us the answer where the algorithm problem was. For the constant foot, our algorithm outputted different results from time to time, so the stability problem was not in the input, but in the algorithm. When we understood where the problem was, we fixed it. And after some work we released our application.
