The rapid growth in global digital advertising spending has led to an explosion in formats, tools and techniques: display, search, mobile, video, in-app, in-game, native, programmatic and so on. Consumers find themselves targeted by a growing array of promotional messaging that seems designed to interfere with anything they may be trying to do. In response, they flee from advertising-heavy sites and go find other sites, leaving publishers scrambling for new ways to attract and retain readers, while forcing advertisers to chase consumers across more and more sites.

How can publishers satisfy both their readers and their advertisers? How can advertisers determine the best format and placement of their messaging? I propose that a key to success is to focus on the behavior of consumers – not from a statistical perspective, but from the perspective of how individuals behave.

The success of any piece of advertisement hinges on how effectively it can reach consumers, and how effectively it can engage the consumers it has reached. There are many ways–some sophisticated, others downright annoying–of increasing the likelihood that an ad will reach a consumer. I will discuss the problem of reaching consumers in a different article. In this article, I focus instead on the problem of understanding how well an ad that is visible to a consumer matches his/her interest in order to drive engagement.

We can think of different forms of digital advertising as falling along a continuum of how effectively they engage consumers, ranging from indifference to intent. I call this the advertising Engagement Scale, as depicted in Figure 1. Display advertising is at the left end of the Engagement Scale: consumers are likely to ignore a banner ad along the rail of an article they are reading. At the right end of the Engagement Scale, doing a search clearly shows the consumer’s intent to find information about a specific topic: consumers are much more likely to click an ad for a product for which they just searched.

In practice, as depicted by the shaded bars in Fig.1, both display ads and search ads cover a wide portion of the Engagement Scale: sometimes we see banner ads that are very interesting to us; and sometimes we get ads with search results that just make no sense. But in general, the Engagement Profile of display advertising is on the left side of the Engagement Scale, while the Engagement Profile of search advertising is on the right.

More importantly, Fig.1 suggests that there is a substantial gap between display (indifference) and search (intent). This gap is evident in the relative performance of display and search ads: typical banner ads generate click-through rates (CTR) of about 0.1%, while keyword-based ads generate on average, about 2% CTRs. The same gap also manifests itself in the economics of advertising: Google, which dominates the digital advertising market, makes the bulk of its revenues from search advertising, even though people only spend about 20% of their time online doing searches.

The gap in the Engagement Scale is where consumers show interest: they are not indifferent, but they are also not showing intent. How can advertisers elicit more interest in consumers?

One possibility is to improve the level of engagement of ads by increasing the inherent usefulness or value of ads, as is done for example with native advertising. Another possibility is to be more selective about which consumers see which ads; an example is retargeting, which selects display ads on a site based on the visitor’s recent behavior on other sites.

A different approach is to look for contexts in which consumers are already more highly engaged. The most conspicuous example is social media: consumers spend a lot of time on Facebook and Twitter. Not surprisingly, both companies are gaining significant market share in advertising, and both the price and performance of ads on these platform clearly indicate that they fall in the interest gap of the Engagement Scale.

It is tempting to argue that the success of Twitter and Facebook ads is due to the implicit trust in content that appears alongside social media. However, I propose that Twitter and Facebook are so successful because they create a context in which consumers willingly explore content that interests them. As I have discussed elsewhere, exploration is what you do when want to discover something that, by definition, you did not know ahead of time; in contrast, search is what you do when you want to find something specific (such as the address of a restaurant).

Although the distinction between search and exploration has not yet been recognized by the mainstream, the value of content discovery is evident from the success of companies such as Pinterest, Instagram, Vine and StumbleUpon. These and other platforms populate the “interest gap” in our Engagement Scale, and they all describe themselves as exploration or discovery platforms.

Thinking about the effectiveness of advertising in these terms can help advertisers and publishers alike make better decisions about how they invest their resources. However, this analysis also reveals a fundamental asymmetry: on the one hand, advertisers can place their ads anywhere along the Engagement Scale, based on where they get the best returns; on the other hand, publishers see a disconnect between their content–which tends to be in the “interest” portion of the Engagement Scale–and the ads they serve, which dominate the left side of the Engagement Scale.

The misalignment between content and advertising limits severely a publisher’s potential for advertising revenues. It must be immensely frustrating for traditional publishers such as the New York Times or USA TODAY to spend vast resources generating high-quality content, while seeing the majority of ad revenues going to Google, Twitter and Facebook.

It is no surprise, then, that publishers are flocking to new technologies that promise to increase engagement and advertising revenues. Some of these techniques, like retargeting and native advertising, certainly help. However, they typically only generate modest shifts along the Engagement Scale and in some cases have a negative impact.

These observations suggest that a sound strategy for publishers is to focus on technologies that target the interest gap in the Engagement Scale. In particular, given that they are in the business of generating engaging content, publishers should focus on methodologies that align advertising more effectively with their content.

In fact, promising results are coming from companies that improve the exploration and discovery of content directly on the publisher’s sites. The best examples are companies such as Outbrain and Taboola, which provide links to related content at the bottom of articles – a portion of the site reached by readers who were clearly interested in the content. These companies place themselves squarely in the interest gap of the Engagement Scale, where exploration / discovery reside. In fact, Outbrain describes itself as “the world’s largest and most trusted content discovery platform.” My own company, Infomous, has seen amazing results using Visual Exploration of publisher content: in one example we developed a module for USA TODAY that led to roughly 50% engagement and 10% CTRs, and average page dwell times of nearly 10 minutes.

A complementary approach, which is also gaining traction, is to leverage the user communities that develop around many online publications, in the form of comments, forums or social media. Disqus and Livefyre have emerged as leaders for reader comments, while companies like Citia and RebelMouse have developed elegant ways to leverage social media and other content on Publisher sites. Our own experience with Infomous shows that social media exploration on a brand or web site can also lead to high engagement: for more than two years The Economist included and Infomous “cloud” of reader comment, which consistently kept readers engaged for nearly 15 minutes (average page visit). Figure 2 shows how all these companies, as well as publishers and advertisers, can be mapped to the Engament Scale.

Sadly, instead of taking a strategic approach that leverages their core strength, many publishers are chasing advertising technologies that may lead to a transient spike in revenues, but that will ultimately alienate readers because of their irrelevance, or, in some cases, their intrusiveness.

I am convinced that soon everyone will understand the difference between exploration and search, and how exploration can be leveraged to monetize content without alienating site visitors. In the more immediate future, publishers and advertisers alike should take the necessary steps to align their advertising and their content more effectively. And I hope they will embrace the Engagement Scale as a guide in evaluating the impact of different platforms and technologies.