Taking care of the merchants, the apps, and the reviews

In his book The Hard Thing About Hard Things, Ben Horowitz recounts a quote from his old boss, Jim Barksdale: “We take care of the people, the products, and the profits… In that order.”

Working on the App Store at Shopify, we operate under a similar philosophy: we take care of the merchants, the developers, and the App Store… In that order. Merchant’s experiences with the apps they install is paramount. If we don’t earn — and maintain — the trust of merchants, it doesn’t matter how developer’s feel and it certainly doesn’t matter if the App Store exists. Without merchant trust, developers don’t get installations of their app. Without app installs, the App Store has no purpose.

Merchants put a huge amount of faith in other merchant’s app reviews. Reviews also influence our ranking algorithm, which determines the long term success of an app, making reviews all the more important. Developers know this, and this knowledge has resulted in the pursuit of positive reviews at any cost. So much so, that occasionally, you’ll see reviews like these:

because it’s making me do this before I even try the app (3/5)

Jury still out … (1/5)

What we’re looking at in this example is an app that requires merchants to leave a review part way through a tutorial in order to advance. This is insanity — I truly have no idea how a developer thinks this is a good idea. Nevertheless, this is rampant, and it’s something that we’re taking very seriously. How can merchants trust anything — the reviews, the apps, the App Store — if this is what it’s turned into?

This week we took a fairly large step in maintaining merchant trust. We implemented a minimum character count for all reviews going forward (30 characters), and removed all reviews that don’t meet that standard. What does this look like? It’s removing about 18% of the almost 100,000 reviews across the App Store. It’s removing 0% of some app’s reviews, and 70% of others. There is a significant percentage of the removed reviews that are simply “great app” — that’s it. This isn’t helpful to other merchants, and it’s obvious that in almost every case, this review was obtained in a way that isn’t good for anyone.

It’s easy, when given a role like “Developer Advocate” or “Developer Relations”, to think that serving developers is what’s most important. It is my job to make sure developers are happy with our platform. I believe, fundamentally, that the best way to do this is to make merchants happy first.

Some developers aren’t happy about review removal — it’s interesting, because the ones I’m hearing complain are the ones who are having the most reviews removed. Maybe if they had focused on the people (the merchants), the product (the app), and the profits (the reviews), in that order… Things may have been different for them. I’m excited to continue rewarding the developers that build with a merchant-first attitude.

Thank you to Josh Gosse and Liz Couto for their editing help.

This post was also published on my personal blog.