Attributing Instagram Screenshots via Visual Search and Adobe Analytics

Shazam-like visual search technologies are proliferating, making “every moment shoppable.” They assist consumers in product discovery and assist brands in capitalising upon that intent. Less discussed is that image intelligence technologies can also be applied against the screenshot or supplied image. This has value in supporting marketers in the identification of influential channels and in attributing the value against them.

Ben Hutchinson
Sep 9 · 7 min read

This article explores this topic with a simple example illustrating how Instagram screenshots can be attributed using Microsoft Cognitive Services, Adobe I/O, and Adobe Analytics.

When a series such as Game of Thrones trends, it often spawns the development of a whole series of ancillary markets. While the Monopoly boards and, er, Adidas White Walker Ultraboost shoes might arguably be expected, influence can subsequently unfold in less anticipated areas, from baby names to college degrees to the emergence of entire cottage industries — read, the 4500 Game of Thrones tours in Croatia “to learn about the evil exploits of King Joffrey” as a case in point.

Yet this small example and the associated increase in tourism witnessed serve as a reminder of just how consumer influence and inspiration can strike— i.e., at any time—marking a myriad of routes to product discovery. Being able to screenshot a Games of Thrones episode, an influencer post, or Copacabana paving patterns, and map that image against any form of product or content catalogue in seconds is something we have all come to expect. A proxy for consumer adoption might simply come from referencing Pinterest, its burgeoning share price reporting 600m monthly visual search queries last year.

Pinterest notwithstanding, the application of visual search is not the sole domain of the major platforms. Mainstream retailers such as IKEA, ASOS Stylematch, Argos, or Walmart can all now count image-based search tools across their digital estate. Collectively, they are all awoken by a desire to capitalise upon our collective picture-taking “muscle memory;” the promise of improvements in conversion and benefits in shortening consideration cycles for a mobile-native, socially-influenced customer-base lure them in. And while Walmart even reports corollary benefits in returns and customer satisfaction, referring back to the Game of Thrones tourism, I would argue that one the most interesting applications lies with easyJet and their “Look & Book” feature released last year.

The feature, which lets people book flights based on a photo, relies on image recognition technology to match a photo with one of their served holiday destinations. It not only recognises that destination, but also marks the nearest airports and pre-populates booking fields in the app as appropriate. Its existence recognises and reflects both the rise in usage of Instagram as a search engine by consumers but also a common visual fabric to Instagram screenshots which makes them technically easier to process. One simple justification of this is that “Instagram photos are more likely to be geo-tagged.” Coupled with image or character-recognition technologies, avenues have opened up to easyJet to leverage consumer “screenshotting” and their emerging Instagram influence; the raw image is harnessed to offer consumers assistance and reduce friction in their path to purchase, automating and supporting them in logistics, planning, and the core goal of destination discovery.

Building on this, it is this notion of a common visual structure against a supplied Instagram screenshot — for example, a relatively persistent logo, common navigation placement and UI, etc. — that could also lend itself to channel identification use cases, extending the value of visual search beyond improving the end consumer’s experience and into the realm of the marketer.

Adjacent to identifying the next holiday destination or artisan Danish lighting manufacturer, the simple goal here would lie in extending the application to identify whether the channel of influence could be predicted from a supplied image, drawing upon a AI model trained to identify the commonalities of the Instagram fabric.

The technology could help support marketers in attributing value to channels and campaigns where influence is tougher to quantify and also generally in augmenting customer intelligence. Magazine adverts, Instagram screenshots, linear TV advertising will all carry business costs that at times lack direct referring clicks that those advertising investments can be tied against and as influencers prove, sources of consumer influence can clearly veer beyond a brand’s owned media. See an example of Adidas’ Creator Club 6% commission investments as one example in this regard, but the underlying concept remains simple: image intelligence can be used to gain a deeper understanding of user journeys and the commercial influence of external channels. That insight can add value in informing future content production, channel investments, resources and advertising direction.

The remainder of this article will therefore depart from consumer examples and will offer a basic technical demonstration and discussion showing how image intelligence can be used to support marketers that leverage visual search in their attribution of Instagram and the new kinds of questions it can begin to answer. Screenshots, links, and an explanation are covered below.

Attributing Instagram screenshots using Microsoft Cognitive Services, Adobe I/O, and Adobe Analytics

From a storytelling perspective, the Visual Search Demonstration Page can be considered as representing a company that sells stock assets — er, such as Adobe Stock. The products that this demonstration company merchandises lend themselves to visual discovery, particularly since the Adobe Stock catalogue numbers in excess of 90 million assets.

Conveniently, Adobe Stock offers a Visual Search API and hence this can be leveraged as a visual search tool enabling the stock asset catalogue to be searched by simply suppling an image URLs or uploading a graphic.

This Visual Search Demonstration Page therefore takes an image URL — there are three hosted screenshot images for use on the page — and upon image submission, is designed (and limited) to throw back three related images as determined by Adobe Stock’s Visual Search tool.

At the same time, the supplied URL is also appraised by an AI-model that has been trained using Microsoft’s Custom Vision Service. This service has enabled a custom model to be built, trained to simply identify Instagram or Facebook as one of the associated channels within the boundaries of a visual screenshot. In tandem with Adobe I/O Runtime, Adobe’s serverless platform that lets you run custom code in the cloud, a prediction is generated and then sent back to the browser relating to the channel it has identified from the supplied image URL. Please note that from a privacy perspective (and to be candid regarding how robust this demonstration really is), only 30-40 screenshots have been used in the training of this model along with the associated social network logos and all screenshots utilised contain content from Adobe’s own social channels. For the sake of clarity, Microsoft’s pricing relating to this service is also currently found here.

Finally, when a visitor clicks one of the three images shown in the pop-up, Adobe Analytics is notified that an add-to-cart event has occurred. Data relating to the product (i.e., one of the three Adobe Stock images displayed) is sent to Adobe Analytics along with the source of visual influence, namely the predicted social channel. As you can see in the screenshots below, this enables the social screenshot and channel to be tied against individual products and revenue within the Adobe Analytics Analysis Workspace application, along with any other engagement metrics that are being collected by Adobe Analytics. Value crediting could be easily reappraised through capabilities such as Attribution IQ which lets you apply different attribution models on the fly.

In closing, this demonstration clearly represents a simplified architecture, sparsely-trained model, and relatively simple use case. This article aims to provide food for thought in terms of the accessibility of these services, maturity of content intelligence, and how the technologies can be woven together. It is worthy of consideration in a period in which visual search adoption and investments anecdotally seem to be rising and hence are naturally going to come up against constant value, ROI, and measurement discussions. There are natural and clear extensions to Instagram channel identification use case such as in the identification of other channels such as media, magazine, television, tube adverts, QR codes, etc, many of which have had long-standing attribution challenges, along with other use cases that could leverage further content intelligence technologies such as OCR. Most importantly though, the demonstration also illustrates just how democratised some of these technologies are. The derivative implication of this is that while there is a clear value exchange for both consumers and businesses from these technologies (and equally, the practice of analysing and crafting experiences based upon text-based search queries is long-established), adoption clearly needs to be considered and measured, with privacy principles top of mind.



P.S. If you would like to explore the theme of visual search more fully, an encyclopedia of many of interesting examples and trends can be found by Clark Boyd.

Adobe Tech Blog

News, updates, and thoughts related to Adobe, developers, and technology.

Thanks to Erin Pierce Haymond

Ben Hutchinson

Written by

Adobe Inc, London

Adobe Tech Blog

News, updates, and thoughts related to Adobe, developers, and technology.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade