We taught a computer how to say “Go Bears!” then oil painted it

“What do you do for fun?”

I guess I enjoy creating cool stuff, and art is pretty cool.

Image for post
Image for post
Training montage of neural style transfer of ‘The Great Wave off Kanagawa’ onto a Lion

What if computers were the target audience of our art?

While this question is admittedly quite absurd, who knows, perhaps one day with the advent of artificial general intelligence, instead of our technology serving to entertain us, we might serve as entertainment providers for them…

Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
2D representations of the gram matrices of the 5 style layers of the neural net, as well as the learned “style” they correspond to (bottom right)

Neural Style transfer is not a new technique

It has been around since 2015 (Gatys et al.), thus we claim no novelty on this front. The sequence below is an example of how Jacky and I personally applied the transfer of style from one image (painting by Zlatko Music Art) to the content of another (bear photo), to generate a entirely new, stylized image. Technical details of our implementation of neural style transfer (which differs in various ways from Gatys et al’s) to generate images such as the one shown below can be found here, and further examples of our style transfer work (including onto videos and 360 videos!) can be found here.

Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Left: Adobe stock image of a bear, Center: Artwork by Zlatko Music Art, Right: Our neural style transfer generated art

But can we really consider these generated images as novel pieces of art? If so, who is the artist?

Is it the machine (i.e. the trained neural net)? The human who trained the machine with specific parameters and who chose the input and training images? Or the photographers and artists who create the input images (used for style, content and training)?

The novel step: Machine Learning as a muse

We coded a neural style transfer algorithm to generate a computationally stylized bear, we found that to be pretty cool, and are currently working on new generative art projects using neural networks. However, for this particular project, it is this next step that we consider to be the novel one: we used our own computationally generated image as inspiration to paint our own mixed media art!

Image for post
Image for post
“bear” — mixed media spray and oil painting by Ivan Jayapurna & Jacky Lu

Written by

Student, Researcher, Writer?

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store