Video Style Transfer with art.app

We’re excited to be releasing a first implementation of video processing in art.app!

This is a tutorial walk-through to get you started. Since rendering video takes a huge amount of time and computing resources, to make it useful and let users (you!) be in control we’ve implemented the following features:

  1. You can process single frames out of your video, so that you can try different styles at different snapshots of your video. This is useful to see how the different frames will look like before embarking on filtering the whole video.
  2. Once you decide to render a whole video, we let you incrementally process frames in sequence, and show you the video as it is being rendered. You can decide to stop or continue rendering at any time, and we only charge for the frames that were rendered.

So, let’s get started!

Choosing an input

Choose a nice video to filter, like the following one:

You can upload it to art.app just like any regular photo, selecting it in the Choose file… button. Please use only MP4 or MOV files, and not larger than 30MB. If your video is long, you may need to trim it in several parts and upload separately, then join together the filtered parts. In this tutorial we’ve trimmed to just the first 11 seconds of the above video.

Testing single frames

We’re ready to test a few single frames! Choose “Create” from the Renders page and select the video as input (or just click on “New render with this video” from the video page). Then you can choose the filters you want to try on the video. We chose two effects, then click “Next”, and arrive at the options page:

The options page, one video input selected and two styles.

Here you can see two new options just for video: “Video processing” and “Output video size”. Click on “Single frames” so that you can filter individual frames of the video as opposed to whole:

Choosing single frames

We choose two different frames (highlighted in orange in the above screenshot). Also, set “Output video size” to “MD”. Click on “Go”, and the 4 combinations of style/single frames will be scheduled:

Single frames being rendered

Here are the results:

Well done! At this point, you can iterate this process until you find the style and video you like the most. You can choose different filtering settings (like brush size and so on), and choose different output size. You can also test on different single frames to make sure the style will apply well to the whole video. Once you’re happy, read on to see how to render the whole video.

Rendering a whole video

Select a video again and choose one style, just like above. However, leave “Whole video” selected, then click “Go!”. This time, you’ll see a new entry on your renders page like so:

A video render job

Click on the image, and you’ll be taken to the video render page, which looks like this initially:

Initial video render page

The progress is 1 / 697: meaning frame 1 is being rendered out of a total of 697. (It’s important to know that we’re working with 60fps, so the whole video is about 11 seconds long.) There are two tabs here: “Video” and “Frames”, both are empty at the beginning! After a while, the first frame is rendered, and the page should refresh into this:

First frame rendered!

At this point, the video render is paused so that you can evaluate results. You can either render 4 more frames, or turn on “auto” to render more frames in batch. Let’s do 4 more frames, click on “Render next 4 frames”.

This step will take a few minutes to process, since besides filtering each frame we need to compute something called the “flow”, that will allow us to relate each frame with the next one so that there’s no flicker in the final video.

Once done, you’ll see that progress is now “9”:

When you’re happy with the results and want to advance faster, you can click on the dropdown menu and click on the batch options, as shown above. Credits will be deducted as each frame is being processed, don’t worry! Also, you can stop the auto render at any time.

When you reach frame 20, a video of the frames filtered so far is going to be rendered. Each new 20 frames the video is going to be updated. Otherwise, you can select the “Frames” tab and browse the frames separately.

The final video is here!, behold:

Final result

At this point you can either save the video (right-click on it, then click “Save”), or download all frames: click on the pulldown menu, then “Download frames”. This will prepare a ZIP file with all the individual PNG files, you can process them in your favorite program like After Effects for example.

Rendering a long video in parallel

Rendering can take a lot of time per frame, and since we need to compute the correlation between frames each video has to be processed sequentially. However, you can split your input video in several chunks and process each video separately: then you can process in parallel, and the process will go much faster.

We can’t wait to see what you will come up with! Style transfer is a magical technology, and videos look quite stunning with it. Of course, this is all beta software, so you may run into difficulties as you’re one of the first brave beta testers! If you do run into difficulties, please contact us at contact@art.app , and we’ll be happy to help. We’ll reimburse credits if things go wrong on our side of course!