The Best AI+Automation Tools for Video Professionals
Artificial Intelligence is becoming an essential part of the professional toolkit. And there are a growing number of solutions for content creation.
I’ve compiled a list of what I believe are the most compelling current tools out there for video professionals, which both improve the efficiency of editing workflows and allow for new creative exploration. Innovation in this field will undoubtedly speed up in the coming years, so it’s important for creators to embrace automation to stay competitive — not just with other humans, but with competing AI as well.
AI is a broad field, but two terms important to know when it comes to AI+Video are: Computer Vision (CV allows computers to see and also understand what is being seen in images and video) and Deep Learning (creates digital brain-like networks so AI can learn by experiences and acquire skills without human involvement). If you’ve ever used a Snapchat filter, deposited a check with mobile banking, or switched to a tropical virtual background on Zoom, you’ve used them. Onto the list.
AI in Pro Editing Software
Adobe — The biggest player in the game, Adobe has been working on a suite of AI tools under its Adobe Sensei initiative for a few years. Their primary focus is efficiency. And while the tools aren’t yet one-click perfect, they can collectively save hours of time per project. Here are a few notable examples:
- Auto Reframe — automatically reframes the subject of a video shot in widescreen, when you want to export it to social platforms (in square or vertical).
- Color Match — changes the color and light values to all clips, based on your reference. This isn’t just copying and pasting your LUT onto multiple clips (which many editors are probably guilty of!) — but it instead analyzes each clip and makes adjustments to be consistent with your reference.
- Audio Ducking — automatically adjusts music levels during dialogue, and creates keyframes for fine adjustment.
Kamua — Kamua has a suite of tools, with its standout feature similar to Adobe’s Auto Reframe (above), though their own version, called AutoCrop, is much more robust, faster, and more accurate — thanks to its advanced Computer Vision algorithms.
On top of that, it’s browser-based, so that means that you don’t have to have a fast computer (heck you can even use it on an iPad). Where Kamua really shines is if you have a library of videos you need to crop/convert — this feature will save hours, if not days.
One of its other signature tools is AutoCut, which uses AI to create individual clips from your final footage. From there you can easily take a clip and automatically create a GIF from it, or assemble a selection of clips on their timeline to easily create a cutdown version. The editor doesn’t have the robust features of say Adobe Premiere, but for many applications when re-working existing videos, this should be powerful enough. You can even tell it what your ideal video length is and how many different versions you want, and it will auto-generate a ton of cut-downs, which you can then test out to see which ones perform best. Here’s a list of a bunch more features in detail.
Naive — This app uses AI to analyze all the footage in a project, then isolates the bad clips (which may be shaky or overexposed) and automatically creates a sequence in Premiere Pro. It also synchronizes the video & audio (similar to Pluraleyes) and creates proxy files. Depending on the kind of projects you work on (i.e. event, travel, and wedding videos), this could be a great efficiency tool.
Descript — This tool uses AI to auto-transcribe your podcasts or videos, then spits it into a text editor, similar to Google Docs. From there, you just edit the document, deleting words (i.e. your ‘ums’ and ‘likes’), cutting out unnecessary bits, and moving sentences or paragraphs around to finesse your narrative.
On the back-end, Descript will re arrange and edit all those audio-files or video clips together. You can then use their timeline editor and built-in effects to create a final file, or export an Adobe Audition or Premiere file. They even have a tool that creates an audiogram for you, though it isn’t as robust as other services like getaudiogram.
If there’s one automation tool that will speed up any conversation based video project, it’s auto-transcription, and Descript is one of the leaders in this space.
Descript also recently launched Overdub, which is honestly one of the more jaw dropping tools out there. It first has you do a 10 minute training where their AI learns your speech patterns, and from there, you can create audio using your own voice …SIMPLY. BY. TYPING. Here it is in action:
This has some awesome time saving applications. For instance, if you have your voiceover talent or podcast host train on this software, and then during the editing process, you’re missing something, instead of going back and having to re-record, you can instead use Overdub.
If this feels like an unwieldy tool (which is open-source, btw) with potentially sinister applications, rest assured, it’s already gone there. Deepfake voice scams are already a thing. For Descript’s part, they implement a verification process. But it’s inevitable that this technology will lead to a lot more than just an AI making Jay-Z speak out Billy Joel’s We Didn’t Start The Fire.
Individual AI Tools
Topaz Labs — If you’ve ever increased the size of a clip in your editing software, you know that all it does is make the video blurry. But this tool uses AI to allow you to upscale (or UpRez) the original clip. The results are impressive, and it has a host of interesting applications:
- You can upscale your HD video to 8K (!) to create super crisp and clear images on even the best displays.
- It opens up new creative possibilities during the editing process, as you can now digitally zoom, crop, and reframe videos while maintaining resolution.
- You can take historic footage and upscale them so that something from 100 years ago can now be 4K.
Speaking of historic footage, you might have seen a video where they modernized an old black and white film like this clip going down Market St in 1906 (days before the earthquake, btw):
This was created by Denis Shiryaev at Neural Love. To accomplish this, he used an upscale AI, plus a suite of other tools (not yet commercially available):
Colorization AI — This employs a neural network to colorize a black and white film. There’s obviously a novelty to seeing films like the above in color, but one could argue it’s also more engaging. You may remember Peter Jackson’s 2018 World War I documentary They Will Not Grow Old, which achieved this as well, but actually had humans (gasp!) painstakingly colorize each frame.
It’s important to note that these most likely aren’t the actual colors (because, well, the camera never captured them) so the AI is guessing based on what it thinks is correct. This opens up many questions, like issues surrounding historical accuracy as some say it’s fictionalizing the past. And one thing I personally wonder is if in the future, black and white films just won’t exist, as AI has colorized all of our historic footage libraries.
Frames Per Second Boosting AI — This tool, officially called Depth-Aware Video Frame Interpolation (DAIN), was created by computer scientist Wenbo Bao. And what it’s doing is analyzing a film clip and then literally creating new frames and splicing them in. It’s wild and has some cool creative applications:
- It can create better motion for any video, as the more frames you have, the smoother the video looks. That’s why it’s an important tool in modernizing clips like the above Market St film. At that time, it was mostly likely shot at 8 or 16 frames per second, but the AI has converted it into a smooth, 60 FPS film.
- You can also convert modern footage into high quality slow motion video. For those that don’t know, you can’t just slow down the speed of any clip in an editor, as you get really choppy results (see left image on the video below). You instead need dedicated cameras that can capture slow motion (60, 120, 240 FPS, etc), which have historically been expensive. This AI tool lowers the bar for entry and allows you to make creative decisions about slo-mo during the editing process, even if you didn’t use a slow motion capable camera.
As of now, there is one DAIN app currently in alpha, but it is otherwise not commercially available.
Unscreen — This is a simple tool, which is doing some heavy lifting to remove the background from your video. For specific use-cases, this is a game changer, as previously you had to shoot on a green screen to get the same effect (or spend hours and hours cutting out your subject). And though it’s not perfect, it works pretty darn well, and opens up a whole new set of creative opportunities for editors.
**One thing to note – there are models for the Colorization, Upscale, and Background Removal within RunwayML – a super powerful Machine Learning App designed for creators who don’t code. But the process may be more involved than some may care for, so these a la carte apps may be the better choice. But if you’re curious, you should definitely dive into Runway!
All-In-One AI Video Creation
As social media marketing has skyrocketed, automated video creation tools are popping up everywhere. The options range from completely hands-off, to hybrid solutions where the AI gets you more than half way and your job is to finesse. Many are pretty lackluster, but there are a few standouts.
THE BEST: Lumen5 — This end-to-end solution ticks off most of the boxes for short social content. As you can see below, you can literally start making a video by adding a link to a blog post, though I’d recommend feeding it a script.
There is a ton of AI at play with Lumen5, notably:
- AI will automatically cut down and summarize the content (if its a blog post)
- Your text is then built into scenes, and the AI will automatically lay in relevant background videos and graphics
- AI determines the length of the scenes based on average reading times
- AI positions your text to avoid covering background faces
What it spits out is an entirely AI-built first pass of your video. And this version should give you some perspective on how good your AI competition actually is at this point.
From here, you can tweak your video using their well designed editor to replace footage (btw, they’ve licensed millions of clips and images through Shutterstock), add or remove scenes, modify the design and color scheme, add music, and more.
One of the best features (though this isn’t unique to Lumen5) is that it will automatically resize the video in the editor, based on the platform you want to send it to:
Runner Ups
vimeo Create —vimeo recently acquired AI video tool Magisto and has launched the beta of vimeo Create. While it is not currently anywhere as powerful as Lumen5, these guys obviously know video, have their own stock libraries, a vibrant community of creators, and as always a strong design aesthetic (they have some pretty rockin templates). Definitely one to keep an eye on.
Offeo —If you’re looking for a tool to use for quick social ads using imagery, then Offeo is right for you. They have an insane (and somewhat dizzying) number of tools, but get past the learning curve and it’s quite powerful. It’s basically Canva for ads.
I found some of their advanced effects and usage of alpha channels, on par with After Effects functionality, albeit done 10x faster – it’s no surprise that it was built by a motion graphics studio. Offeo could be an extremely useful tool for social media marketers, motion designers, and ad agencies (who definitely wouldn’t be able to tell their clients they used it!)
Trash – Some of the team from Vine recently released an iOS app more focused on the TikTok crowd. It uses AI to find the most compelling clips and auto-stitches everything together. Without a robust feature-set, it may be hard to get creators to migrate to this on top of the apps they’re already using — but their Deep Learning AI is constantly getting smarter, and could lead to some interesting results as it evolves.
As a video editor myself, I look at apps like Lumen5 and honestly feel a bit threatened. Tools like this are built so that companies no longer have to rely on editors. Though at the same time, there’s no reason why editors can’t start using Lumen5 and Offeo, leveraging their expertise to improve quality, while also being able to produce content at the high volume these tools were built for. In fact it’s vital that editors begin to embrace more automation tools like this.
Admittedly the videos that services like Lumen5 create are a very specific type of content. And one might say, well, AI wouldn’t be able to easily edit something more complex like a recipe video, right?
If you’ve ever watched more than 1 Tasty video, then you’re aware that many internet videos follow a consistent pattern. Whether it’s a recipe video, a How To, or a wedding film – there are clear templates. And keep in mind, AI (and more specifically deep learning) is quite good at finding patterns. The more of these videos we feed it, the better it will become at editing them.
Then you might say, that’s just internet videos, but more creative film endeavors will still need humans to edit, right? Well in 2017, this Stanford project built a tool that not only automatically edits dialogue scenes in films, but does so based on user-inputted performance notes (i.e. start wide, emphasize character, intensify emotion, etc). The tool ended up cutting a scene that took a professional 3 hours to edit, in a matter of minutes. At this rate, I wouldn’t be surprised if we see the first feature film, assistant-edited by an AI, in a matter of years.
It’s important to note that whether it’s a simple social ad, a recipe video, or a film – anything the AI creates will still need human oversight to yield a good final product.
But in the coming years, AI will undoubtedly evolve from helping us out with particular tasks to becoming our creative collaborators. They’ll help us make better content on quicker timelines and will allow small teams to produce studio-level projects.
For people who understand narrative, have ideas, and are fluent with automation tools, there will definitely still be work. But for many editors, if they don’t evolve their skillset, In the Blink Of An Eye, they might just become obsolete.
How could I write this article without making an AI-assisted video of it:
