Deep Photo Style Transfer

In 2016 researchers published the paper Image Style Transfer Using Convolutional Neural Networks in which they presented a technique that enables a Neural Network to transfer a style from an artwork onto a photograph. This technique became popular and it is now available in apps like Prisma or Facebook filters.

In 2017 researchers from Adobe and Cornell University published a new paper called Deep Photo Style Transfer in which they take the methods from the original style transfer paper and improve it in order to achieve photo realistic style transfer.

As seen the the previous image, (a) represents the original image and also the image from where the style will be transferred. On the following images, (b) to (f), style transfer is done and we can see the results where lambda represents the intensity of the transfer. Researchers found that a good value is lambda=10⁸.

In order to achieve this results, they proposed two core ideas to the original model of style transfer:

• A photorealism regularization term in the optimization function.
• An optional guidance to the style transfer process based on semantic segmentation of the inputs to avoid the content-mismatch.

From left to right, the first image is the original image. The second image is the image from where the style will be transfered. Image third and fourth are other techiques. Finally the last image belongs the the proposed method

The results are surprising, the model is able to keep the original structure of the image while applying the colorspace of the target image. It will not be long until we see photographs apps implementing this kind of technique for their users.