AI Algorithms May Not Be As Smart As You Think

We are in the midst of a wave of AI-driven advertising. Every advertising platform from Google to Facebook has launched its version of AI advertising products in recent years and is heavily recommending those products to marketers.

These marketing products, such as Performance Max campaign from Google and Advantage Plus Shopping campaign from Facebook, promise a utopia of hands-free advertising that automatically provides marketers with the best return on investment.

If you talk to any of the Facebook and Google representatives, their recommendations will almost always be: just set a budget, leave the algorithm alone, and it will achieve the best outcome for you without intervention.

The claim they have made on the power of the algorithm is incredibly bold, especially when they provide little transparency on the inner workings of the algorithms when delivering your ads, and provides even less detailed reporting for marketers to make accurate assessments of the delivery effectiveness.

As independent thinkers, we must examine those marketing claims carefully, and the purpose of this article is to show you two ways of “Turing Test” your AI algorithms to understand if they are really behaving as smart as a human advertiser in providing ROI for your business.

I will not provide a definitive conclusion on whether those algorithms are good or bad for your business. Otherwise, I am just making another claim just like the ones that are telling you that the algorithm works.

Instead, we are offering you a way through our free product to plug in the data yourself and make your conclusions, learn more at www.humanlytics.co.

The Turing Assumptions

The theories behind analyses today are heavily based on the popular “Turing Test (or Imitation Game)”, devised by Alan Turing, one of the forefathers of the modern computer.

To put it simply, our assumption is that if the AI algorithm is truly smart, it must behave and perform somewhat indistinguishable compared with that of an intelligent human marketer when the human is facing a similar situation.

The Promotional Period Analysis

The first situation is during a sales or promotional period.

The graph we are going to use is a simple dual y-axis time series chart, with spend on one side and conversion/conversion rate on the other axis.

You can also add a black line on the graph to indicate the budget level you have set for the campaign/platform in question, though we will not include it in the free version due to UI complications.

The time period here is the key here. We want to set the graph around a period of significant expected conversion rate fluctuations — this typically happens during a sale or during shopping holidays like BFCM.

For the conversion/conversion rate axis, you have a lot of options here, and each will give you a different perspective on your data.

The first one is between the choices of conversion vs conversion rate. Conversion rate is a good measurement of the conversion potential of a single traffic landed on the site, and conversion is simply the overall revenue or orders on your website.

I found the conversion rate to be less exciting if you cannot filter it by the specific advertising platform/campaign as organic traffic tends to have a significantly different conversion rate and obscure the data you see — so your choice here should be conversions or revenue if you want to do a simple analysis (which is what we have for the free version).

On the other hand, if you can zoom in on traffic from a specific campaign or platform, the conversion rate can serve as a power indicator to help you identify the scaling potential of your platform/campaign, as it has some predictive power on the expected value of each session from each traffic sources — more on that in future articles.

Then you can choose which platform you want the conversion/conversion rate data to come from. For a typical business situation, you can choose between GA4 and the advertising platform in question — whether that’s data from Google or Facebook.

Overall, I trust GA data (and will use it for the free analysis) more than that from the advertising platforms due to the differences between data collection protocols. However, GA is mostly last-touch and will not give you insights into how your ads are helping with conversions in the future (though that is slowly changing with GA4’s attribution features).

Let’s assume we are running a 7-day sale of the store’s product and revisit the above graph to discuss expectations.

We expect the revenue and conversion rate for the store to go up during the sales period, with it being the highest approaching the last days of the sale.

After the sale, we expect the revenue to drop either back to or below the regular level before the sale, then it will gradually return to the regular level if there is a dip.

In response to this expected change, an intelligent marketer should increase spend during the sales, especially towards the end of it, to benefit from the increased conversion rate of the sale, and decrease spend after the conclusion of the sale when the conversion rate is at its lowest.

Similar to humans, a “smart” AI algorithm should detect the change in conversion rate across those different periods and make adjustments accordingly to best leverage the time of higher conversion rate during the sales, and also drop the spend back to a regular level or even below allotted budget after the conversion rate has gone down. So maybe not as sharp of a change, but it should be present.

Over and over again, I have seen situations in which the spend level of the AI algorithms just remained flat (or even lower) during the period of promotions, like the graph indicated below.

When this happens, we are missing two opportunities for profit for the business — potential lost revenue from not scaling sufficiently during the promotional period and overspending after the promotional period when the conversion rate is expected to be lower than usual.

While this might not seem significant for your business, based on our experience, companies typically lose an estimated 10% of their profit from a sale due to spending inefficiently from their sales period (more from overspending than missing opportunities).

Day of the Week Analysis

Whereas the above analysis examines the AI algorithm’s strength when responding to short-termed extreme events, the day-of-the-week analysis examines another aspect of fluctuations in our day-to-day business activities — seasonalities.

The best seasonality we can analyze in the e-commerce industry is the weekly seasonality, which presents itself in the form of different conversion rates and revenue from Monday through Sunday.

Almost all of our clients have weekly seasonality, with weekdays (Monday — Thursday) having significantly different conversion rates than the weekend.

We can plot a dual-axis bar chart like the one presented above to understand how seasonality affects our revenue and how our spend reacts to the seasonality.

The y-axes are the same as the previous chart, with spend on one side and conversion/conversion rate on another (with similar caveats applied in choosing the data for conversions/conversion rate)

The difference here is that on the x-axis we have a categorical variable called day of the week: Monday Thru Sunday.

Instead of plotting out every single point of spend and conversion within a specific period, we are instead taking the average of all days within the relevant analysis period (3 months is recommended), so we have the average spend and conversion rate for each of those weekdays.

For this analysis, we should expect our spend trends with our weekly seasonality of revenue. For example, if your weekends are generally weaker than your weekdays, you should spend less over the weekend so you can recover most profitability and spend more money on weekdays to get more revenue.

This is in fact an area where human media buyers fall short in terms of management, as day-to-day management and responding to change in conversion rates can be extremely labor intensive especially when we want to create an analysis for this for every platform and every campaign.

On the other hand, a smart AI algorithm can detect changes in conversion rate across different weekdays easily and somewhat automatically, making adjustments without human intervention to optimize the company’s profitability.

Unfortunately, when it comes to the reality of existing AI algorithms, I typically see a situation like the following: Whereas the changes in your daily revenue are noticeable, the AI algorithms usually do not spend the most on those days you get the most conversions.

On a weekly basis, I see Facebook and Google tend to spend more over the weekend regardless of your revenue level and conversion rate, likely due to the higher deliverability of your campaigns rather than it will be more profitable for your business.

Furthermore, Facebook and Google tend to have extreme days of spend that seemingly just happen randomly. For example, on the weekend of April 23rd of 2023, many companies saw their Facebook spending significantly exceeding their campaign budgets for no reason and returning almost no results for the business.

Just like the previous analysis, you will be losing money on overspending if you spend too much on days on which you are not the highest in conversion rate, and vice versa, losing potential revenue if you are not spending enough on typical strong days of your business.

Bringing it all together

Though refraining from criticizing them too much, my experiences with AI-driven marketing products such as Performance Max and Advantage Plus have been mostly disappointing.

However, this is not to say that they are not smart — they are smart in a different way than most advertisers imagine.

AI algorithms are good at finding ways to deliver ads at a low cost for marketers. It might search across different audience groups, time periods, and geographic locations to ensure your CPC is the lowest possible for the most results.

But we must understand the interests of the advertising platforms — they will not stop delivering your ads when they see the delivery today is not bringing you sufficient revenue to cover the cost — it will always prioritize delivering as long as it satisfies the parameters and restrictions that you have set in place (CPC requirements and budget).

To further support this point, we have found hints of Google Performance Max shifting budgets up when the monthly ROI of the client is above the target ROAS you have set for the campaign, usually resulting in higher spend and ROI lowering to the target ROAS. Some may argue that this is simply an attempt at scaling the ads, though those scaling attempts rarely come at a level of success that justifies the increase.

With a similar strain of thought, it becomes very important for us as marketers to place those strict restrictions on our AI marketing ads so they don’t go wild and deliver to a degree that is not profitable for us (even though sometimes they still do, but must less likely).

Restrictions we can place include:

• ROAS or CPA target
• Delivery limitations on time of the day and day of the week
• Budget

Marketers often hesitate to do any of the steps above because those platform representatives will tell you that “it will reset learning in the campaigns,” so let me address this point briefly.

Overall, I believe (happy to be proven wrong) those campaigns are not being sent to ground zero of learning when you make those changes but instead trigger the learning process in which it has to search for the new optimal given the circumstances you have given them.

This learning process is not only not scary in most cases (as you are still around your current optima) but also encouraged in a pure machine learning sense as it prevents you from being trapped at a “local maxima” — an area that is somewhat optimal for your business but not most optimal.

The bottom line of this topic is — our business environment changes drastically on a month-by-month basis, it is probably better for us to have an algorithm that is continuously learning and adapting to new business situations than being trapped on a fixed “optimal” point in which it uses the same strategy to respond to different situations in your business.

In conclusion, look at your data and conclude whether AI marketing works for you. I have gradually advised my clients to shift back to a more manual marketing style and have found slightly more efficient success than automated algorithms, though the verdict is still out.

As usual, you can do both analyses presented in this article for free at www.humanlytics.co by plugging in your data, and you will be able to do this with all of our analyses presented in this series.

Next, we will explore the method of click matching to estimate the contribution of advertising platforms to your revenue.

In the meantime, follow us on Medium, Youtube (for a video discussion of this analysis), and TikTok (@humanlytics) for more marketing analytics tips.