Week 6 — Image Transformation According to Art Style
Hi everyone! Last week, we have talked about our clustering results after applying the clustering on extracted features from the VGG19. Today, we will talk about more feature extraction methods that we have applied.
After applying the clustering on activation values extracted from the last layer activation functions, we have only got the object-based clustering method. When we research to find the best methodology to find style-based features of the artworks, we found a method which is proposed by A Neural Algorithm of Artistic Style paper. According to the paper, we can use the top layers of the VGG19 to extract texture information which composes style-based information. These layers are Conv1_1, Conv2_1, Conv3_1, Conv4_1, and Conv4_1. You can see these layer below:
After extract the activation values of these layers, these features need to extra pre-processing step to get rid of object structure and positioning information which is called Gram-Matrix. The Gram-Matrix method is applied according to the given formula:
In the proposed model, these features are used directly in style transformation as an image processing operation. In the selective clustering paper, these features are used in the Robust Continuous Clustering algorithm to find the artist’s artworks that mainly contain his/her style. We have inspired this solution to separate the artist’s works into different sub-styles, and After this feature extraction step, we have composed all 5 feature vector and create one vector to use in the K-means clustering. When we applied this methodology to Van Gogh artworks, we got good results on 3 number clusters. You can see 2 different clustering sets below:
We will apply style-based clustering. Our final step is to run the cycle-gan method on different clusters which is generated by object-based and style-based clustering methods and evaluating the final results. See you next week!
Contributors:
References:
- https://arxiv.org/pdf/1508.06576.pdf, A Neural Algorithm of Artistic Style
- https://kops.uni-konstanz.de/bitstream/handle/123456789/45588/Deng_2-o7wcklpfg6zr4.pdf?sequence=1, Selective clustering for representative paintings selection
