RunwayML vs. Pika Labs Throwdown
And have my AI Video Skills improved over six months?
I made my first AI video project just under 6 months ago, on June 8, 2023. It was a 45 second commercial for a fictional beer company called ‘South Pole Brewing’. I had made a lot of progress with Midjourney and I was excited to try using RunwayML to generate video. I spent 2 minutes with ChatGPT writing a script, 15 minutes editing that script, and then 2 minutes in Eleven Labs generating the audio. I didn’t know much about RunwayML at the time, so I started with straight ‘text-to-video’ prompts. I just wanted a record, a milestone, of what the technology was capable of at that moment. You can view the results below (or just skip, it’s awful) before we get into the main course of today’s article.
The general consensus on technique has come pretty far since then, and almost every tutorial I see recommends an ‘image-to-video’ workflow where you generate an image in Midjourney (or the tool of your choice), maintain a consistent aspect ratio (I like 16:9), and use that as the basis of your generative video. Theoretically, you can add a prompt to this in both RunwayML and Pika Labs and art direct your video, but to be honest, I’ve never had any success with this. In Pika, I see no result and in RunwayML, things almost always get worse. So at this point, I just upload the image and take what I get. However, I usually generate at least 50% more images than I need so I can cherry pick from the results.
Both tools now offer good camera controls like Zoom and Pan which adds to a cinematic effect, and RunwayML has their new Motion Brush, but more on that later. Pika Labs works via Discord and it becomes a lot less confusing if you open up a private thread by messaging the Pika Bot. Simply right-click the Pika icon and select ‘Message’ and you’ll be working in your own space. Pika has a few more controls and a possible frame rate of 24, but you need to add -fps 24 to your prompt. It also has a movement scale of 1–5, so I was adding -move 5 to all my prompts. After a while, I just kept attaching the following command to each image I uploaded. ‘-fps 24 -camera zoom in -move 5’, varying the camera command as needed. Pika’s clips are 3 seconds each and I generated about 25 of them and used the best 15 for my 45 second commercial. Pro Tip: You don’t need to wait for one video to finish generating before starting the next, just go for it, i rapid succession. I was done in under 15 minutes.
Next, I headed over to Eleven Labs to try out their new ‘Speech to Voice feature. I’ve used Eleven Labs for a number of projects, but the pacing and intonation can be stiff at times. With this new feature, you can speak, vary your inflection and pacing, and have this reflected in the output voice of your choice. It was really impressive and after my second attempt, I downloaded a new audio file. I quickly threw things together in CapCut, added some music and effects, and here you have it, in about two hours total time, the Pika Labs version.
On to RunwayML, who should have a bit of an advantage since their web based interface is easier to use, and has a few more features, including their new ‘Motion Brush’. I’ve only used this a little, but it let’s you highlight portions of the image to be animated. It’s great for clouds and smoke and fire, but terible for hands and faces. On a side note, I think that RunwayML must have an amazing PR firm, because the all the media articles I read just gush about them and say the same thing about their ‘revolutionary technology’. To be honest, I think they’re over-rated, and I’ve burned through a lot of credits on their site with mediocre results. However, I’m trying to set aside my bias for this article and produce the best work possible. This will take a little longer because I want to vary my technique with the motion brush and take notes aftyer each image, so it’s going to be ‘one at a time’.
The first attempt worked out beautifully and I had several successful generations before I had any epic failures. RunwayML was off to a good start, until this happened.
And the more I tried to use the ‘Motion Brush’, the more warped and distorted my subjects became. However, as I backed off, the quality of movement did improve. Fortunately, the video clips from RunwayML are 4 seconds long, so I would only need 11 to fit into my 45 second commercial. For the sake of efficiency and comparison, I used the same audio and music tracks for both versions. While the individual rendering of the files took a long time, dropping them into the same Capcut file and rendering a second version went really quickly. Altogether, the RunwayML version also took about 2 hours, but that was reaping the benefits of all the prep work (Midjourney, audio, etc) from the original effort. If I’d started from scratch, it would have been over three hours.
Overall, I’m coming away with an improved opinion of RunwayML. The ‘Motion Brush’ produced amazing results for clouds and water and hair, but weird distortions for faces and hands. When used with restraint for subtle motion it created some really elegant results. However, RunwayML is expensive and I burned through a lot of credits and had a little more than a 50% success rate. Pika Labs is free, not as powerful, but produces great results most of the time. Which one do you prefer? Leave a comment and let me know.
If you like this kind of content, consider subscribing to my newsletter and getting it delivered to your inbox 3x a week. https://erikfadiman.substack.com/
