Adding ultra-efficient AI filters into your video-conferencing app with Vectorly and AWS Chime
If you're building a video-conferencing application in 2021, you've likely heard of Amazon Chime, an SDK which helps companies build real-time video and audio communications into their apps.
There are already several providers in this space such as Twilio, Agora and Daily.co, but Amazon Chime has specifically made it easy to integrate custom Video Filters into your app with their Video Processor AI.
Vectorly offers an SDK with pre-built, ultra-efficient, video filters such as Virtual Backgrounds and AI Upscaling, that you can add into any web-based video conferencing application.
It seems like a perfect fit, so I decided to write a quick guide to show you how to leverage the Video Processor API and Vectorly's AI Filters SDK to add features like Virtual Backgrounds and AI Upscaling into an AWS Chime app with just a few lines of code.
Getting Started
Beyond having an AWS account and credentials, you’ll need to sign up for Vectorly, and get your account token from the dashboard
Then, clone the Vectorly AI Filters Examples Repository, which is a fork of the AWS Chime starer repository.
git clone https://github.com/Vectorly/ai-filters-examples.gitAnd go to the AWS Chime SDK demo folder
cd ai-filters-examples/upscaler/amazon-chime-sdk-demoIf you pay attention, there will be two files we have created in app/meetingV2/videofilter , VectorlyUpscalerProcessor.ts and VectorlyBackgroundFilter.ts
In both of these files, there will be a few lines of code which instantiate the video filter
You'll have to put in your vectorly token in the tokenfield of the config object. Once you do that, you should be ready to build!
Building
Once you have made the changes in the Vectorly files, go back to the aws-chime-sdk-demo folder, and run
npm installThat should install the main dependencies, including Vectorly's SDK. Once everything is loaded, you should be able to start the app with
npm run start:fast --app=meetingV2It might take a while to build, but if it worked, the demo application should be available at http://localhost:8080/
Testing Vectorly's Filters
Go ahead and join a meeting. After selecting a camera, you should enter the main room, where you can enable your camera stream.
✅ Great, you've got the video stream running.
You can enable video filters by clicking on the dropdown menu next to the semi-circle icon, where you will see several filter options, including the Vectorly filters.
Let’s go try them out! For demo purposes, let's go ahead and click Vectorly Background. It might take a second or two to load, but if it does, you should be able to see your video feed with a virtual background enabled.
If everything worked, congrats! You've built your first AWS Chime app with Vectorly's AI filters enabled 🎉.
Performance
Before we go, let's quickly take a look at the performance. If you go to the filters list, you'll see two options for background segmentation: VectorlyBackgroundFilter and Segmentation (BodyPix).
Try running the demo in a chromium browser (Chrome/Edge/Brave/Opera), and use the performance profiler to monitor CPU usage.
First, let's load the Segmentation (Bodypix) model. If you do, you'll see a lot of dropped frames, and fairly high CPU usage
Next, let's try Vectorly's Background Filter. If you do, you should see little to no CPU usage
We've worked hard to make sure we have the most perform-ant AI models available for web. You can find more benchmarks and info about our SDK's performance on our docs page.
Wrapping up
With these files, you should be able to easily add AI Upscaling and Virtual Backgrounds into your AWS Chime app.
If you're curious about AI Upscaling and Virtual Backgrounds beyond just the 'how-to' of integration, here are a few resources!
