Using deep learning with style transfer

The Socratic Kiss, II

Background

This is Part 3 of 3 in a tutorial series for getting started with GPU-driven deep learning. These tutorials are intended for anyone who is interested in figuring out what it takes to get started with deep learning on a personal machine that costs less than $2000. This assumes you have some familiarity with machine learning and deep learning. This series will cover:

1. Part 1: How to build a GPU deep learning machine
2. Part 2: Setting up Ubuntu and Docker for deep learning
3. Part 3: Using deep learning with style transfer

Recommended Reading

• Instructions for using Neural Style (Justin Johnson)

Papers

Leon A. Gatys, Alexander S. Ecker, Matthias Bethge:

• A Neural Algorithm of Artistic Style
• Image Style Transfer Using Convolutional Neural Networks

K. Simonyan and A. Zisserman:

• Very Deep Convolutional Networks for Large-Scale Image Recognition

Pedro Domingos

• A Few Useful Things to Know about Machine Learning

Examples of Style Transfer

• Style Transfer (Gene Kogan)
• Style Studies (Kyle McDonald)

Other Resources

• Neural Doodle (Alex Champandard)
• Machine Learning for Artists (Gene Kogan, Francis Tseng)

Docker Commands

• docker images displays a list of Docker images on your machine
• docker ps displays a list of active Docker containers
• docker rm -f <container_ID> shuts down an active container
• More commands

Installing the Style Transfer Docker Image

When you use Docker, any changes you make inside of a Docker container will be lost unless you take some action to preserve the changes. I recommend having a dedicated directory for Docker projects and for style transfer, specifically. When we launch our container, we can then create a link between certain directories in the Docker container to the project-specific directories we’ve created on our local machine. If we store any data in these directories when we’re running the container, then that data will be available after the container is shut down. We can also make data from our local machine accessible to the container by putting it in the linked folder.

Setting Up Project Directory Tree

1. Open up a new terminal session.
2. Because this is a freshly installed OS, I’m going to create a directory called “projects” in my user directory. I’ll type mkdir ~/projects.
3. Create a directory for your docker projects. I’m going to create a directory called “docker_projects” inside of my “projects” folder. To do this, I’ll type mkdir ~/projects/docker_projects.
4. Create a directory for our neural style project. I’ll type mkdir ~/projects/docker_projects/style_transfer.
5. Create three new directories called “images”, “outputs”, and “models” inside of your style transfer project folder. I’ll type mkdir ~/projects/docker_projects/style_transfer/{outputs,images,models}.
6. Navigate to the inside of your style_transfer directory. I’ll type cd ~/projects/docker_projects/style_transfer/. This step is important for when we use $PWD to copy the models and to attach the local folders to the container folders. Alternatively, you can define the file path.
7. Open a new tab in terminal by typing ctrl+shift+t and switch to it.

Setting Up The Neural Style Docker Image

1. Install the the Docker image: docker pull danhigham/cuda-neural-style. This image is several GBs and could take a few minutes to download.
2. Run the container: nvidia-docker run -it danhigham/cuda-neural-style.
3. Navigate to the models folder: cd models.
4. Run the script to download the models: sh download_models.sh.
5. Switch to the other tab in terminal. Determine the container ID of the neural style container by typing docker ps.
6. Copy the contents of the “models” directory in the Docker container to the “models” directory in our local style transfer project folder by typing: docker cp <container_ID>:/root/torch/neural-style/models/. $PWD. In my case, cp b04d2f650ff4:/root/torch/neural-style/models/. $PWD. Now our models will persist after we close the container.
7. Switch back to the terminal tab with the Docker session. Close the container by typing exit and selecting enter.

From now on, when we want to open a style transfer container, we can use the following code when we’re inside of our style transfer project directory:

nvidia-docker run -it -v $PWD/images:/root/torch/neural-style/images -v $PWD/outputs:/root/torch/neural-style/outputs -v $PWD/models:/root/torch/neural-style/models danhigham/cuda-neural-style:latest

Neural Style Transfer

Now that we have our models, we’re ready to do some neural style transfer! Note, there are images available in examples/inputs and examples/outputs.

Download this image and save it as “klimt_the_kiss.jpg” in “images”.

The Kiss by Gustav Klimt

Download this image and save it as “data.jpg” in “images”.

Download this image and save it as “socrates.jpg” in “ images”.

Socrates — Encyclopedia Britannica © Araldo de Luca/Corbis

Launch the neural style container. Run the following code

th neural_style.lua -style_image images/klimt_the_kiss.jpg -content_image images/socrates.jpg -output_image outputs/socrates_kiss_01.png -backend cudnn

It may be a minute or so before you see any response after executing the code. Eventually, you will see something that looks like the following:

Note, if your style losses are NaN values, then you’re going to end up with completely black images. If this happens, you can cancel the execution by selecting ctrl+c. Consider trying to adjust the parameters to try and fix this. Using the adam optimizer, adjusting the image size, or adjusting the weights is often helpful. You can find more information here.

You can find your new stylized images in the “output” folder.

The Socratic Kiss, I

Now try using “data.jpg” as a style image, making sure to rename the output file so that you don’t accidentally erase your last set of outputs. Hint: In terminal, you can press the up arrow on your keyboard to scroll through previously executed commands.

After you run style transfer again, you should see something like the following in the “output” folder.

Socratic Datum

Congratulations! You’ve successfully used GPU-driven deep learning for neural style transfer. I encourage you play with different parameters, to use your images of your own art, and to have fun!

This is a living document, subject to change. If you find any inaccuracies or have any comments, reply below!