Jan 16, 2015
Apple, Apps and Algorithmic Glitches
A data analysis of iTunes’ top chart algorithm
On October 29th and December 18th, 2014, something very strange happened to the iTunes top apps chart. Like an earthquake shaking up the region, all app positions in the chart were massively rearranged, some booted off completely. These two extremely volatile days displayed rank changes that are orders of magnitude higher than the norm — lots of apps moving around, lots of uncertainly.
If you build apps for iOS devices, you know that the success of your app is contingent on chart placement. If you use apps on iPhones and iPads, you should realize just how difficult it is for app developers to get you to download their app. Apple deploys an algorithm that identifies the Top Apps across various categories within its iTunes app store. This is effectively a black box. We don’t know exactly how it works, yet many have come to the conclusion that the dominant factor affecting chart placement is the number of downloads within a short period of time.
If a bunch of people all of a sudden download your app, you climb up the charts, and as a results, gain significant visibility, which results in many more downloads. Some estimate that topping the charts may lead to tens of thousands of downloads per day.
Encoded within the iTunes app store algorithm is the power to make or break an app. If you get on its good side, you do really well, and if not, you lose.
If these volatile days are deliberate, shouldn’t we be informed? There are over 9 million registered developers who have shipped 1.2 million apps into iTunes. Algorithmic glitches on wall street can set off hundreds of millions of dollars in losses. What’s the dollar cost to entrepreneurs affected by these iTunes glitches? These are people who pour countless hours and resources into adding value to Apple’s ecosystem. Whether running experiments or A/B tests, shouldn’t Apple show due respect by taking issues like this seriously?
While the app store’s ranking algorithm is opaque, there’s much to be learned by looking at its output over time. In his work on Algorithmic Accountability, Nick Diakopoulos highlights ways to investigate the inner-workings of algorithmic systems by tracking inputs and outputs.
Analyzing this type of data gives us a way to hold accountable systems of power, in this case, Apple and its algorithm.
Perhaps Apple is not aware of these glitches? Or maybe my data is flawed? I’ll let you be the judge of that. I did manage to find another person complaining about abnormal chart rank fluctuation around the same time. If you’ve witnessed something similar, please add a note or get in touch.
Others are also manipulating the system
There is clear value in gaining top iTunes chart placement. And where there’s value you’ll always find people gaming the system. Since the top app chart algorithm is heavily reliant on downloads within a short period of time, the practice of boosting has become quite common. By carefully planning an advertising campaign along with incentivized downloads one can gain momentum and enough downloads within a short period of time to drastically increase their chart rank. This isn’t that much different than Google SEO — by choosing the right keywords at the right time, you can make your website much more visible.
The plot below shows chart placement for Beats Music and Soundcloud between March and June, 2014. The higher the curve on the y-axis, the lower the placement in the top free apps chart. The vertical red lines mark Saturdays and Sundays.
Note how SoundCloud’s placement stays fairly steady while slightly fluctuating throughout the week. Now Beats Music is completely different curve. Throughout March and most of April we see a steady decline in chart placement until the app is completely booted off the charts. Within a few days, it spikes sharply back up to spot #5, staying there for 10 days before falling sharply again. This does not look like a typical organic spike, especially since there were seemingly no significant product launches or updates at that time.
These are clear signs of boosting.
Curiously, only a few weeks later, on May 28th, Apple announced that it will acquire Beats Music. The timing of this campaign just before the acquisition seems somewhat dubious.
Perhaps a sign of deliberate manipulation to impress M&A assessment, or simply a mere coincidence. Having access to this type of data gives us the ability to call Beats Music out.
Apple turns a blind eye towards these practices.
Now, let’s dive into the data
Over the past year I’ve been collecting iTunes chart data to get a sense for how the algorithm works. At betaworks we build and ship many iOS apps. If we knew a bit more about the way apps are ranked we could make better decisions along the way, especially as we launch new services.
Every day, my script hits a number of RSS feeds that Apple publishes with rank listings across multiple charts, and saves the result into a mysql database. For the sake of this analysis, I’ll use data from two time periods; 3/14–5/14 and 8/14–1/15. For this reason, you’ll see a gap between mid June and August in the charts below.
The dataset includes 2358 different apps that reached a position within the ‘top free apps’ iTunes chart over the period of 273 days. Some of the apps consistently remained in a top chart position, while others may have made a minor appearance only for a few days.
For example, the top 5 apps that accrued the highest number of days on the charts are: LINE, Emoji Keyboard 2, Spotify Music, Vine and Pandora Radio. The apps with the longest persistent chart appearances (consecutive days in the top list) is quite different: InstaCollage Pro, Clash of Clans, Shazam, SoundCloud and RetailMeNot Coupons. And the most volatile apps, displaying the steepest climbs and falls, are: Game of War, NFL Mobile, Dunkin’ Donuts, LinkedIn and the Bible.
Here’s a typical graph showing chart position for Tinder and Uber.
A few notes:
- Tinder and Uber consistently display opposing weekly patterns: while one is at its peak, the other is at its trough (remember: the higher the line in the graph, the LOWER its’ iTunes rank).
- Thanksgiving and Christmas — not a great time for Tinder dates nor Uber cab rides.
- Over the past year Uber’s chart placement has only been getting stronger. With all the negative media and threats of bans from November onwards, you’d think less people would use the service. But it has consistently stayed within the top 50 spots in the chart. Unless Uber is paying more for chart placement, it doesn’t seem like there’s a drop in new downloads.
- There’s some odd behavior around the end of October, where Tinder jumps up the charts, and mid December where both apps disappear from the charts for a single day. These are the two glitches we’ll dive into later.
Let’s look at another example.
Social network and messaging apps display clear weekly fluctuations in chart rank, while effectively maintaining their average rank over time. LINE, a popular messaging app, is an exception, displaying two significant spikes throughout Spring 2014. In mid-March, LINE rises steeply to a top 50 position, slowly losing rank over the next month and a half, only to regain that position for another month in May. This may be due to a number of factors.
In mid March ‘14, LINE launches its Premium Call service, allowing users to make phone calls to non-LINE users with flat rates no matter where you’re calling from. One explanation for the spike, is that this new feature prompted tens of thousands of users to download the app within a short period of time. Alternatively, LINE’s marketing team may have implemented some boosting strategy, prompting downloads, resulting in higher chart placement.
If we normalize each time series according to its standard deviation, we can clearly see the aligned weekly fluctuations amongst the apps.
Viber, WhatsApp, Twitter and Facebook Messenger are closely correlated in terms of their weekly cycles, displaying their highest ranking mid-week, and lowest on Mondays. Snapchat and Vine display very different, much shorter cycles. Snapchat is almost opposite of Facebook Messenger and Twitter, where there seems to be significantly higher usage throughout the weekend, driving the app up the ranks towards Monday, its best day on the charts. This makes sense, since iTunes users information from the previous coupld of hours to calculate the current rank for an app. So when an app peaks on Mondays this implies that there were heightened number of downloads on the Sunday.
Which brings us to another app that generally sees many downloads on Sundays: the Bible.
One of the most popular ‘utility’ applications in the app store, the Bible, consistently stays in the top list throughout the year. With that, some periods are much more volatile than others. As you can see above, the Bible clearly fluctuates on a weekly basis, where Mondays are consistently the highest ranked days (= many downloads on Sundays). The Bible’s all-time highest ranked day is, unsurprisingly, Easter, where it reached number 24 in the top charts. Additionally, we can clearly observe a rise in volatility — as we head towards the summer, there is a larger difference between Sundays and Monday rankings; less users are downloading the Bible throughout the week.
Using Correlation
Calculating correlation between apps helps us a way to compare usage patterns between mobile apps. Correlation is a measure of the mutual relationship between two objects. The higher the correlation between two apps, the more similar their app ranking fluctuates over time, the darker the cell in the matrix below. (larger matrix with top 100 apps linked here)
We can clearly see a number of groups that emerge above — these are the darker regions displaying a number of apps that have high correlation amongst each other. The plots below help illustrate why the apps were grouped together.
Games are a completely different ballpark. (and need their own blog post).
Algorithmic Glitches
Now that we’re familiar with the data, let’s move on to to the real puzzle.
When we plot out all app ranking over the year, we can clearly identify a number of days where something looks odd. On both October 29th and December 18th we see significant volatility, where half of the apps are booted off the charts, the rest move up in ranks, and a large number of single-day new appearances happen. The next day everything shifts back to normal; order is restored.
I’ve spent many hours with this data, pivoting on all parameters, trying to find some justification for these dips. Perhaps holidays? Day of the week? Massive Synchronized marketing promotions? But none leads to an answer that makes sense.
The only possible explanation, the only way in which there could be so much volatility all of a sudden, is algorithmic.
This is especially poignant when comparing to previous months of data.
We can display volatility by taking the first derivative of our time series data, effectively displaying the difference between two adjacent days. This means that if on Tuesday an app was ranked at #20, and on Wednesday its rank moved to #35, we’d plot a line that climbs up 15 points on the y-axis. In the plot below you’ll notice how steady the volatility is in aggregate, except the two specific days.
Now let’s take a look at a different measure — persistence.
We define persistence as the number of consecutive days an app appears in the top rank list. Plotting persistence across all apps, we get this seemingly Zipfian distribution, sometimes called the long tail or power law, commonly identified in data from social systems.
When we zoom in, we see a form of step function. A large number of apps appear in the top chart for exactly 129 consecutive days. This is certainly not organic, and supports our thesis that an automated mechanism was involved. Like turning off the switch all of a sudden, we see a large number of apps booted off the charts at exactly the same time. I’ve seen this happen in the past with Twitter bot nets (see comment for more context).
Regardless of how we look at these two days, the underlying data feels strange. We see almost no apps dropping in ranks between the 17th and the 18th of December. Half of the apps rise in rank, while the rest are booted from the chart. The booted apps are ones that would typically not drop off the charts, including: Dropbox, Amazon, Google Maps, LinkedIn, LINE, Twitter and WhatsApp.
In their place, 88 apps in the chart for a single day. This includes: Fandango Movies, Unroll Me, Bubble Witch 2 Saga, GIF Keyboard, Yahoo and Amex Mobile. There are quite a few games in this list, but that’s not surprising. Generally speaking, the iTunes top list includes anywhere between 20–40% games. The breakdown by category over time doesn’t point to any peculiar outlier on December 18th, although it does show a minor dip in social networking apps (light blue).
This is important
Analyzing this data gives us a fascinating peek into the iTunes walled garden. The better we understand the inner workings of the charts, the more predictable they become, the better decisions we can make as we build, launch and promote apps. Apple owns this ecosystem and we’re completely at their beck and call.
Yet the more glitches and erratic behavior we see in their governing algorithm, the less trust and faith we have in the system as a whole. No doubt the ecosystem is already being gamed, but there’s a sense of unfairness when it is the algorithm that is manipulated right underneath our feet. Especially without any warning.
Running these types of data analyses help us hold Apple accountable for potential biases or flaws in their algorithm — an algorithm that is so powerful, it decides who makes money, and who loses.
Google does a decent job engaging around changes to its search algorithm . But search is Google’s core business. This is far from core to Apple’s cash cow.
Almost negligible.
Thoughts, questions, ideas? Find me on twitter — @gilgul
