Sensor Tower — First Thoughts

So I came across a really interesting service (sensor tower) to promote my @diveadvisorapp on both iTunes and Play Market.

It really was what I was looking for, it gave me a simple metric of Search Volume, Difficulty, Competitors and ways to monitor rankings of keywords aswell as ‘Spy’ on competitors. It helped me quite a bit, but a little bit into the process I began to realize that something was a little strange and started to question things a little bit.

Here’s the thing

1. Sensor Tower is quite expensive
2. The cost of using a bad/inaccurate service is actually a LOT more than the cost of the service itself.

Anyhow, so this is what the core sensor tower looks like:

Left to right: Traffic, Difficulty, # of Apps and your apps rank for keyword.

Question 1: How is Traffic Derived?

This is probably the most important question since if sensor tower gets this wrong, you are optimizing for thin air. Now, a response to my question:

@nolabelinc Good question. There aren’t traffic numbers available by keyword, so we had to create our own algorithm to estimate traffic… Read the comments in this post to get a better idea of what Traffic Score is based on: http://ow.ly/Ag7FV

Following that link, the only piece of information I could find that was relevant was this response:

The Traffic Score is an estimate of potential traffic, and it ranges from 0 — 10, with only a few very popular keywords like “Facebook” having a score of over 8. A healthy traffic score is anywhere above 4 for single words. We pull data from a number of sources, like autosuggestions when typing in the store, frequency of word usage in common crawl data, length of terms, difficulty of typing on the iPhone keyboard, traffic estimates from the web, etc.

Lets break this down:

1. autosuggestions when typing in the store

Well, perhaps I am wrong, but this would only provide the confirmation of a keyword being indexed, not sure how it would provide clues into the volume of that word. So let me mess around.

Random search 1

Ok, so I see what they mean now, scuba is probably more popular than scuba diving and scuba diving is probably more popular than scuba diving games. Its NOT based on alphabetical order. I’m guessing this would give sensor tower a hierarchial system atleast.

2. frequency of word usage in common crawl data

My guess would be that they have a crawler that crawls through the app store and aggregates all descriptions, titles, keywords etc. Then it probably analyzes the text, extracts the keywords and then correlates that with the ranking of that app for that particular search. Ok, makes sense.

3. length of terms

Not too sure what they mean by this, but my guess would = longer search terms are generally less likely to have higher volume than the composite of words within them. For example above, ‘scuba diving’ ranks above ‘scuba diving games’. Unless they mean something different, its kind of a risky assumption. For example, “marine weather app” is probably going to get more volume than “marine”. Would be interesting to get a comment from them.

4. difficulty of typing on the iPhone keyboard, traffic estimates from the web, etc

The first part (difficulty), definitely not clear on that. The traffic estimates from the web, again, can be a useful assumption when compared to basing optimization on no information at all, but also very risky. An example of this is simply that the same search term can have very different volume changes between android and play market. Therefore there can be big differences between search volume on web and app stores. Users are simply looking for a different kind engagement when running a mobile web search vs. app store search.

Answer to Question 1

Given the limited data, sensor tower are probably doing the best they can BUT here is the key problem that I see, ALL and absolutely ALL their information is relative — with nothing absolute.

I.e: based on my understanding and their rather limited answer — they are ONLY able to tell you:

1. The keywords that relate to a certain niche or ‘market’
2. Their rankings relative to each other

The implication of this is that you have no idea of actual volumes and answers to questions like if I DOMINATE keywords “x”, “y” and “z” — how many actual downloads can I expect? My point of view is further reinforced in question 2.

Question 2: How is Difficulty Derived?

This is another mega question. You could base your whole strategy in optimizing for a keyword with an assumption of low difficulty and waste time in not knowing why you can’t rank.

Now here is what I asked:

@SensorTower also. How do you get ‘Difficulty’? If I have less “apps” how can difficulty be higher? Inputs = ? #data pic.twitter.com/SiIVGCpW2S

@nolabelinc Other apps with more downloads will make a keyword more competitive. Read this: http://ow.ly/Ag8ht

So I carefully studied the post…

And actually found no clues as to how this DIFFICULTY metric is actually derived, except in a comment posted by sensor tower:

Unfortunately, there is no way to get certain information about particular apps, such as downloads, unless you are the developer of that app. In addition, nobody except Apple knows what the ranking formula is and it will change over time, so it all comes back to looking at your scores, testing and refining.

This really made me sad ☹

Firstly, this indirectly confirmed my conclusion from Question 1. If sensor tower had info on number of downloads, they could correlate that to rankings and get SOME estimate of absolute numbers.

But secondly, what is this difficulty based on?

This next part of that same comment put the nail in the coffin:

You can estimate the strength of your competition by looking at how they rank for their keywords, the number of their reviews, looking at the data on their Sensor Tower App Profile Page, etc., but those are just qualitative indicators.

Ranking for keywords only gives you info about a specific competitor and a specific keyword. Ok, so “xyz” app is ranking #1 for “yyz” keyword — what does that tell me about “yyz” difficulty versus “dyz” keyword in the context of the whole market? Nothing really…

The number of reviews is another DANGEROUS metric since there is absolutely NO possibility of deriving some kind of universal correlation between downloads and reviews. Some apps have 1% users reviewing them and some have 20%. It should not be used as an insight for number of downloads.

So lets get specific here. I want to pick between 2 words: “marine life” and “marine animals”

Now the weird thing is, for marine life, there are 66 apps that rank but difficulty is 2.9. For marine animals, its 77 apps but difficulty is 2.1…

Here is the thing, if sensor tower dont actually have the data on the number of combined downloads or even number of downloads of the leaderboard, how can this be so? The ONLY thing that could skew this would be the number of reviews based on which they would have to make a systemic guess into the number of downloads. They simply dont have any other information on this.

UNLESS of course, they have historical data and make some other assumptions like — “if the top 10 apps ranking for this keyword haven’t changed in x days then its more Difficult than a vice-versa situation”.

Anyhow, my conclusion and desire, is for sensor tower to write a blog post = “Insight on how we get our metrics” and maybe even “The limitations/assumtions of our algorithm that generate our metrics”.

Granted, they are working with limited info, but us users need to be aware of the risks we are taking by trusting sensor tower with our time (money) and money.

And once again, they have built and amazing UX around their data, BUT if the data is ultimately unreliable, that all becomes irrelevant.

Thanks for reading.

Timur from No Label Inc.