Media Measurement Still Requires a Human Touch

Published by Graeme Harris, was head of corporate communications for UBS Canada

It was a logical assumption. With AI capabilities growing exponentially, PR execs expected media-measurement software to work without human intervention, and reduce costs. So why do some measurement firms insist on having humans intervene when preparing reports? Skeptics might see it as a ploy to charge more. Just as data capture is more than aggregation, measuring earned and social media entails more than merely evaluating mentions.

It’s not just happening in PR. Facebook hired humans to remove fake news from its platform. Apple chose humans rather than algorithms to highlight top news stories, Tim Cook said June 26.

The Dirty Data Problem

Rage over dirty data remains a problem with media measurement, and it’s understandable. Even with technology, too many irrelevant stories show up in reports, tone is completely mis-applied in a too-large portion of the analysis, and the analytics delivered are not consistently selected for insight.

Ask Tom Lange, Union Pacific’s AVP for corporate communications.

“Every three hours, somebody is hit by a train and the most recent…study shows 94% of the time, it’s due to reckless driver behavior,” Lange says. “Nonetheless, we conduct…UPCARES events where we work with local law enforcement to educate drivers about rail crossing safety. The education comes in the form of traffic citations for violating crossing signals, or…as movie tickets or gasoline cards as positive reinforcement for good behavior. Obviously these are good local media stories but the headlines and/or leads almost always use the words ‘accident,’ ‘violations,’ ‘deaths,’ and ‘injuries,’ so automated media platforms tone them as negative. However, for us, these are positive stories.”

Toning is significantly better when humans participate in it. No doubt the cost usually is significantly higher. To some, the ROI is just not there. No wonder many practitioners have taken measurement in-house, use off-the-shelf software and let interns sort it. Or worse, they choose not to measure.

There’s a Solution

All is not lost. Some measurement firms are developing technology to reduce the need for human intervention, which should trim costs. How is this being done?

Good firms use multiple services to ensure they capture everything, which creates duplication, so something or someone must sort it out. Luckily, technology has evolved to score one hit and then simply count the duplications.

But how do we sort out relevance and tone problems? Is it another case of AI to the rescue?

Machine-learning technology has evolved to perform basic filtering and sorting by learning from sets of analysis humans performed previously. Humans are teaching the computers. Human intervention, though, still is needed to keep AI current with changing idioms, sarcasm, Trumpisms, etc, according to studies by MIT, Berkeley, and Northwestern University.

In other words, machines are only as good as the input and they don’t know what they don’t know. A big challenge with blindly applying machine-learning techniques is it’s possible to train machines incorrectly. You need to make sure the trainer is hitting the target consistently. This is why you should use trained analysts, not the summer intern.

But technology continues to evolve; machines will get smarter. They will identify topics, not just words and facial recognition for photos and television appearances. The goal will be to have machines develop predictive capabilities to tell PR where an issue is headed.

Big Questions

This leads to big questions: If AI is not quite ready, when will it be? How much human intervention, if any, will be needed in the future?

If the most advanced academic research is correct, it appears even after the singularity the Cyborgs will need human intervention for media analytics. While AI and Natural Language technology will improve, 10% to 20% of the job will require human intervention…depending on what you want to know from the results. It’s like a driverless car. Do we need a steering wheel? Perhaps not, but 10%-20% of the time, we just might.

With the current amount of human intervention, savings could be substantial. In other words, if you use human-centric analytics today, over time your media measurement costs should decline or you should be getting more insights for the same money.

As the data gets better, we will want to use it more in decision-making and strategy development. It will also get more convenient to access, so you’ll be asking Alexa to give you the morning media brief while having breakfast. But for the foreseeable future, you’ll likely have humans guiding the AI behind your analytics.