Evaluate the impact of your App Store Optimization efforts — Part 1: Search

One of the primary goals of App Store Optimization (ASO) managers is to grow the user base of their mobile product, thanks to measures impacting the quantity and quality of the leads finding their app through the stores’ search engines.

So, how do you evaluate the true impact of your keyword optimization efforts? Sure, holding down the top positions for some keywords is important, but does it really tell you if and how you gained new users? Using only ranks to evaluate the impact of your keyword optimization strategy isn’t reliable.

In this article, you will learn how to evaluate the impact of your ASO measures towards search looking at steps to conduct for 3 different categories of metrics:

1. Operational indicators: Extract insights from the keywords you rank for
2. Supporting KPIs: Measure search traffic and downloads
3. Top-level KPIs: Evaluate what impact your efforts have on the business

Step 1: Extract insights from the keywords you rank for

As you start to implement keyword optimization measures, you should see the total number of keywords your app rank for improving, as well as your ranking for these keywords. Monitoring keyword ranking indicators should help you define whether you’re targeting the right keywords.

What you should monitor:

1. The number of keywords indexed. If you’re just starting with ASO, whether you are ranking in the top positions doesn’t matter too much. However, seeing an increase in how many keywords your app is ranking for will let you know straight away if your strategy is working. The number of keywords indexed should stabilize over time as there are only a limited number of terms you can target with the character constraint on the store pages (with all other things besides ASO measures being equal).

2. Keyword distribution. If you’re targeting the right keywords, you should see ranking positions improve. If not, it’s about time to start playing around with your keywords and analyze their performance more into detail. Keep in mind that the lower your rank, the higher the volatility will be — leading you to often experience “keyword dances”. Once you get closer to the number one spot, you should see smaller movements.

Here is how you can set up your dashboard for monitoring your keyword distribution and number of keywords indexed (inspired by Sistrix Toolbox)

3. Rankings for target keywords. Your target keywords should be highly relevant to your app, drive a high search volume, and allow you to group all your search terms into clusters based on the similarity between two strings, or by similarity between search intent. Learn how to pick these keywords in my article about doing keyword research.

4. Visibility for clusters based on your target keywords. Choosing highly competitive search terms as target keywords means that your app could be nowhere to be found in the search results — even though you’ve included these words in your title. Does it mean that you’re not going anywhere? Not necessarily! Your app is likely going up for long-tail search terms. Because a big chunk of keywords is long-tails, especially on Google Play, these need to be monitored carefully.

• To monitor your clusters on the App Store, you can use a daily visibility score aggregating the information about all the keywords in the cluster, taking into account both their Search Popularity and the ranking of your app. Some tools provide this functionality.
• Things get a bit more complicated on Google Play — mostly because it exists at the moment no first-party search volume data. I advise monitoring the Google Play Console on a monthly basis to see if you see any interesting keyword appearing in your Acquisition Reports.

Beyond the keywords you’ve identified with your research, I highly recommend using Apptweak’s keyword movement detector if you’re using their tool (Note: I haven’t had the chance to check if similar features are available in competitive tools). It shows you the keywords that have had the highest-ranking movements from one date to another to easily identify if (and when) the Apple or Google algorithms have picked up your keyword updates.

How to track your ranks?

There are lots of different tools available for keeping track your app’s position in search results. Determine which one is the best for you depending on your needs, and whether you want to use their dashboards or create your own using their APIs.

Bonus — What you could monitor:

• Track the latest OS version available to ensure that the ranks you track are the ones that most users will see in the stores.
• Track app and non-app results to get a better understanding of the visibility you get with each position. Besides apps (organic and ads), you can also find stories, in-app purchases, and bundles on the App Store.
• Track search results layout in Google Play. For certain terms (typically brands), Google displays a snippet of the first result’s store listing, which is followed by ads and numerous recommendations — as you can see on the screenshots below. Getting this information could help you understand why holding the top position for a term isn’t driving much traffic.

If you’re not #1 on the Play Store, be ready to be pushed below-the-fold behind ads, personalized recommendations and similar apps (Screenshots: XDA-Developers)

Step 2: Measure search traffic and downloads

So, you’re now monitoring where your app shows up in search results for a given query on the App Store and Play Store. Should you stop there? Seeing your app reaching the top positions feels great, but you don’t want to get stuck in the quagmire of vanity metrics. It boils down to several unknowns: we only have assumptions about the download distribution in search results for both stores, and it exists nowadays no search volume data for Google Play.

This is why you need to also measure the search traffic landing on your page, as well as downloads generated.

What you should monitor for the App Store:

Measuring the App Store traffic is not as straightforward as it seems. First, the App Store discloses no information on the keyword level in App Store Connect. Second, App Store Connect search metrics include Apple Search Ads. I recommend monitoring the following metrics:

• Search Impressions, the number of times your app was viewed in the search section of the App Store, and Search App Units, the number of first-time app downloads per Apple ID. See definitions in App Store Connect Help. Please note that these include numbers from Apple Search Ads. So how do you get organics? You could deduct it the following metrics:
• Apple Search Ads Impressions, as well as Apple Search Ads New Downloads. See Apple Search Ads definitions. The latter represents app downloads from users who have never before downloaded your app before, hence not including app updates or redownloads—similar to App Units. It gets a bit trickier for impressions: I find it unclear whether they’re counted the same way in ASA (the number of times your ad was shown) and App Store Connect (the number of times your app was viewed, with the option to display only per “Unique Device”).

Here is how you can set up your dashboard for monitoring your search performance on the App Store

Bonus — What you could monitor:

• Add more granular information to your dashboard and monitor conversion rates per cohort from ASA to inform your organic strategy
• Discuss with your data team and see how you can assign a score to each keyword based on their search popularity to estimate the contribution of each keyword movement to the variance in impressions. I would recommend leveraging this cheat sheet published by Merlin Penny following a study he conducted with SHQ. This allows you to get an idea of the number of impressions your app would get if ranked number 1 on the App Store in the US.

What you should monitor for Google Play:

On Google Play, you have access to acquisition data per search term. Yayy! However, it is only global — and organics are grouped altogether on the country level, making it impossible to differentiate between search and browse traffic 🤯.

Edit November 2020: Google Play updated its console earlier this month- making available keyword data per country… but removing the split between organic and paid. I recommend reading this great article written by Nadir Garouche explaining how to isolate ads.

Here is how I suggest proceeding:

• Track your acquisition data per search term — such as store listing visitors, first-installers…. I differentiate search terms into 3 categories: brand terms, competitive terms, and generic terms. Only an increase in generic terms is the result of effective keyword optimization. I’ll come back to this.
• If you’re focusing on a few markets, monitor your organic acquisition data per country. Before claiming victory after seeing an increase in any organic metrics, keep in mind that if there has been a change in search in one country, it should also be reflected on the global level (acquisition data per search term). If not, the change is likely attributed to Explore — check if you’ve been featured, if there has been a change in user acquisition, or in your visibility in Similar App Cards…

Why you should monitor these metrics:

Tracking your data per keyword (type) on the Play Store can be tedious. Let me share with you three anecdotes proving the utility of it to evaluate the impact of your efforts.

• Competitive terms. A few years back, the organic installs of one of the app I was responsible for rose suddenly: I thought that my keyword optimization efforts were finally paying off. Looking more granularly, I realized that the rise was all due to a competitor with no app investing in PR — and for which my app was ranking #1... Keyword optimization had no impact.
• Brand terms. I see on my dashboard that one of my apps has seen its installs quadruple overnight. Looking at the data per keyword, I notice that it’s entirely attributed to our brand. The app was quoted in a TV show and users massively searched for it. Once again, it’s a fantastic thing for the app, but ASO hasn’t proved useful.
• Generic terms. A couple of weeks later, another app download’s soared overnight in one market— mostly on the App Store. Why only on the App Store? The Play Store data tells that there has been an increase in visitors for one particular keyword. Checking the rank of this keyword helped me explain part of the enigma — the app was ranking way higher on the App Store. I knew I would need to double down on my efforts for this keyword as it had proven effective.
  You might wonder: what led to an increase? Checking on Adjust the hourly downloads showed me that all downloads happened at the same time. Checking the TV program led me to realize that a show had broadcast somebody using an app similar to the one of my company — however without quoting it, leading users to search for a generic term I had been targeting for months.

How to track these metrics?

Discuss with your data engineering team to automate the downloading of App Store Connect data, and from Apple Search Ads Custom Reports. Google Play organic country data is available through their API. Data per search term on Google Play should be scraped — it can be done using libraries and frameworks such as Selenium or Scrapy.

Step 3: Evaluate what impact your efforts have on the business

Seeing month-over-month growth for downloads or impressions will very likely provide you with a sense of accomplishment, but your ASO efforts could be driving thousands of downloads of users who will never actually use your app. To evaluate what impact your efforts have on your business, you need to understand how the rest of your marketing team measures the success of acquisition, and what are the top-level KPIs for your business/app.

As an ASO Manager, you may rely heavily on the data provided by the stores. However, it is siloed, not standardized, and doesn’t correspond to the data you see in your attribution tools. It also doesn’t allow you to track meaningful events post-install. Therefore, when making an attempt at comparing organic installs to any other acquisition channel, you should stick to attribution providers' definitions. There’s no need to go too much into detail to explain how ASO is returning to the company’s bottom line. There are several ways to do it.

First, you need to understand how favoring users with a specific intent — by prioritizing some keywords- impacts key metrics. Let’s take the example of Spotify: should they prioritize “free music” over “music offline”? The first term might have higher search popularity but could convert fewer users to subscribers in comparison to the latter. Compare over time KPIs such as the share of paying users and the average revenue per paying user organic vs non-organic, as well as the Customer Lifetime Value of various cohorts. Feel free to read another article I wrote about increasing the quality of your organic traffic leveraging ASA data. Thinking beyond the install is key to the long-term success of your ASO strategy and business.

Second, I recommend having a look at this presentation from Dor Alaluf (Playtika) who tried to give a dollar value to ASO. He compares the organic baseline to the uplift brought by ASO activities such a review prompting or replying to reviews. Then, he multiplies the number of extra installs by the average CPI of his company to understand how much the company has saved thanks to ASO.

Final thoughts

Defining metrics is critical to understand the impact of your ASO measures. It helps you assess your work objectively, identify the impact of a change, and prioritize the right initiatives. Reporting on the right KPIs will also contribute to increasing your credibility by setting up targets and defining success.

In the next article in this series, I will detail how to evaluate the impact of your conversion rate optimization measures. Feel free to reach out if you’re interested in reading about any other topic :).