How Adobe’s Enhanced Smart Tags Capability Empowers Marketers to Find the Most Relevant UGC Video

Co-Author: Bryan Russell

Marketers are spending a whopping $10 billion a year on content in the US alone, with half of that going into content creation. It may come as a surprise, though, that about 20 cents of every dollar spent creating content actually goes to waste due to inefficient processes and output, amounting to a staggering $1 billion yearly loss.Pair that figure with the fact that, in 2018, only 28 percent of marketers reported significant returns to their content marketing campaigns, and we can safely say that content marketing has a serious scaling problem.

As content and screens multiply, leveraging user-generated content (UGC) is the key to alleviating the scaling challenges brands face in an increasingly content-hungry and personalized world. This is because UGC is not only cost effective — in most cases free by obtaining the rights — but also more authentic with better performance. 64 percent of social media users seek UGC before making a purchase and UGC videos receive 10x more views than branded videos (source). That’s why some of the world’s most respected brands, like Apple or Starbucks, have made UGC a pillar of their content strategy for years now. However, this traditionally requires marketers to sift through pages of social media posts to find the few golden nuggets they can repurpose.

Turning to AI to find the best UGC for the job

To address these challenges facing marketers, Adobe is tapping into computer vision to help automate the UGC curation efforts that were previously done by hand. Smart Tags, powered by Adobe Sensei, automatically scans images and identifies the key objects, object categories, and aesthetic properties to use as descriptive tags. This allows marketers to filter out image content with tags that do not match their search criteria. Yet while Smart Tags has been an effective tool for images, video is by far the most consumed media type on the web today.

Adobe’s Smart Tags.

Video is growing at a massive pace — according to Cisco, video will account for 82 percent of all web traffic by 2021 and the number of videos posted on Instagram grew 4X last year. This poses a serious challenge for marketers and technologists alike as to date, curating video content has been laborious since a user needs to manually watch lots of videos to find relevant footage. Videos are much heavier and have a temporal dimension, making them more challenging than images to classify, filter and curate. This is precisely why we partnered with Adobe Research and Adobe’s Search team to enhance our current Smart Tags capability in Adobe Experience Manager to handle UGC video classification.

How we built Smart Tags for video in Adobe Experience Manager

We sought out to automatically output a set of relevant tags for a given video. This ultimately resulted in the Video Auto Tag Adobe Sensei service which produces two sets of tags for a video of up to 60 seconds in length. The first is a set corresponding to depicted objects, scenes and attributes in the video, and the second corresponds to depicted actions in the video. These tags are used to improve search and retrieval of videos.

Our system builds on in-house Adobe image auto-tagging technology that was trained on a large collection of images from an internal Adobe image dataset and can predict tags over a large vocabulary. As a video consists of a sequence of frames, we first apply the image auto-tagger to the frames and aggregate the outputs across time to produce a final tag set for the video. This process results in a set of tags that typically correspond to the objects, scenes, and attributes depicted in the video, since the image auto-tagger has been trained to make such predictions.

Video Smart Tags for UGC

In addition to objects, scenes, and attributes, it is important to recognize temporally varying events — actions and activities — in a video. Example actions and activities include “drinking” and “jumping”. This is addressed by adapting the image auto-tagger to predict actions by training on a curated set of ‘action-rich’ videos with accompanying action labels derived from user metadata from an internal Adobe video dataset. The action auto-tagger is applied across multiple frames in the video, aggregating the results over time to produce the final action tag set for the video.

Takeaways for marketers, data scientists, and their development teams

UGC is an essential tool to help reduce content marketing costs, improve the effectiveness of campaigns, and tackle the scale issues marketers face today. Teams supporting these marketers are now empowered through computer vision — particularly video understanding — to better leverage UGC, the most valuable and popular content format on the web, by accelerating workflows.

We’re offering use of the Video Auto Tag service to a select group of beta participants who are interested in enterprise use cases. Click here to find out more.