Neucraft — Exploring traditional crafts using AI — Part II — Prototyping with GANs
In the last blog entry, we reported on the data collection at the traditional Asahiyaki kiln, where a collection of artworks dating back 400 years were taken out of the warehouse. In this blog post, we would like to report some of what we have learned so far.
The collection of pieces taken from Asahiyaki’s warehouse spans a great variety in terms of their colors, patterns and shapes. Although there is a tendency towards bowl-shaped pieces, there are also figurines and other objects with specific shapes such as incense holders. We were also surprised to see some works with vivid colors, almost fluorescent. Below we can see a number of randomly chosen pictures from the data.
All of the photos for this project were taken under very close conditions, with controlled shadows etc., thus bring ready to use for training an AI model without any pre-processing such as background removal. We first started training the generative AI model using 1000 photos taken from the front.
Training and generation using StyleGAN
We started off with StyleGAN3, the latest version of StyleGAN developed by NVIDIA and known especially for its high quality in generating human faces. The video below, which shows the results from the model during training, shows that we started with a hazy output, but as we went through the training steps, we gradually succeeded in generating objects that looked like pottery with well-defined contours.
However, upon closer inspection, we discovered a mysterious pattern on the surface of every object as shown in the left side photo below and no amount of training could remove this artifact (i.e., the AI creates features that do not exist in the real thing). Artifacts are a common problem in generative AI; for example, in the case of human face generation, it is well known that the shape of the ears inevitably looks unrealistic. As the StyleGAN algorithm is improved over time, some of these problems go away, but it is still not universally effective.
After some investigation, it seems that the cause of this problem is inherent to StyleGAN3, which was solved this time by going back and using the previous StyleGAN2 (see the photo on the right).
Looking at the generation results of the model trained on the GPU for about 5 days using StyleGAN2, at first glance, the results were of high quality, approaching that of the original data used for training. In particular, stable generation results were obtained for the bowl-shaped objects.
On the other hand, with regard to vases and object-type pieces, there are scattered examples that, upon closer inspection, appear to be partially cut or have eerie, odd shapes. It is also interesting to note that, as we looked further into the various generated results, some unusual “in-between” designs surfaced, in which a handle is attached to a type of pottery that should not normally have a hand attached (hohin, a type of teapot on the left and a tea bowl on the right). While these designs could be considered impractical, some of them may serve as inspiration for new ideas.
Controlling the generative process
We showed these results to Mr. Matsubayashi, the head of the Asahiyaki kiln, and discussed the future use of the generative model. One important thought that came out of the discussion was that, while it is practical in a sense that the more the models are trained and the higher the internal evaluation metrics are, the more they produce outputs that become indistinguishable from the real thing, on the other hand, producing something that looks exactly like existing Asahiyaki pieces will not lead to the emergence of new ideas. In fact, we have experienced this kind of situation in our past projects, where it got us thinking “the more realistic the AI is, the less creative it is” and it can be said that this is a challenge inherent to current machine learning approaches.
We have begun several attempts to resolve this situation. When actually generating an image from a generative model, different results can be obtained by giving it a random value that serves as the seed. The results of changing the ψ (psi, the standard deviation of randomness) from a minimal value of 0.1, to a default value of 1, to a larger value of 2 are shown below.
When ψ is extremely small, all the products have the same color and shape. At a first glance it might look like a boring result, but it showcases what a typical example of a Asahiyaki bowl might look like. When ψ is at its default value, the variability of the products has a spread that feels similar to the variability of the training data, but at larger values, the colors and shapes can be more vivid or bizarre than in reality. This includes things that are not necessarily practical or even makeable, but may serve as a stimulus for a bit of “wild” thinking.
How to Generate More Novelty from Statistical Models
Some believe that learning techniques based on statistical theory, such as machine learning, are not suited for generating new ideas in the first place, since they are inevitably based on learning and generation learned from past precedents. Such challenges are not limited to creativity, but are also faced, for example, in research and development to find new substances using AI. In practice, as in the above example, it is effective to solve this problem by having AI produce results that successfully deviate from past examples to obtain ideas, but what is most effective is to expand the training data.
So far, we have only looked at and used data taken from the Asahiyaki kiln, but there are many other types of ceramic art in the world. Even looking only within Japan, there is a wide range of styles. Thus, by combining the information outside of the training data we have seen so far with this training data, it will be possible to obtain output that could not be “imagined” in the AI model that has only seen Asahiyaki. Merging the style of Asahiyaki and another kiln, or bringing just a little bit of influence from multiple different styles into Asahiyaki pieces, it is hard to imagine what kind of results are possible until we try it out.
In the next blog entry, we hope to show you how generative AI models can become even more creative.
