Stargazing
1 year ago, I started a spreadsheet. For 9 months we’d been rewriting Firefox for Android from the ground up, and on May 16, 2012 we released the first beta version to the Android market. I started a spreadsheet to track our star rating. I was nervous.
9 months of rewrite. In web time— in mobile web time— that’s years. Rewrites are almost always the wrong call. You get to throw away bad code, but you throw away good code, too. Hard won, battle-hardened fixes are expensive to throw away, and the code is never cleanly labelled “baby” and “bathwater.” The decision to rewrite Firefox was one of the loudest weeks I’ve had. Even once we’d decided, my hand hovered over the send key for a long time.
3.5 stars. We didn’t pull the old version from the market during the rewrite. We wanted to keep those users safe with regular security updates; that meant keeping the product online. It meant taking black eyes every day from users who tried us, who loved us for our desktop product, and who were disappointed. And because the Android rating system rounds, our already painful 3.7-star rating rounded down to 3.5. We were dead last.
7:1. If you want to maintain a 4.5 star rating, it takes seven 5-star reviews to counteract each 1-star review. Star ratings are deeply flawed for all kinds of reasons: selection bias, survivorship bias, false dilemmas, unidimensionality, reporting bias, et cetera. I know this. And still I watched them. When you try to reach people through an app store, your star rating is the first assessment of your product they’ll see. And when you try to make something great, reviews are real pieces of feedback from real human beings and they are painful and they are frustrating and they are golden.
1181 reviews in the first week. 588 of them were 5-stars. 105 of them were 1-star. I argued with my screen over 1-star comments about bugs we had already fixed. I swooned at 5-star reviews that said they were reversing earlier 1-star reviews.
6 weeks after beta, we pushed to release. By this point beta had climbed to 3.8, which rounds to 4 stars. Fists were pumped. Release sat at 3.6.
Time passed.
200,000 reviews and 6 releases later, this week, today, Firefox and Beta both show 4.5 (rounded) stars in the market and the team is still going strong. I’m immensely proud of the work they’ve done. It’s made me reflective and maybe a bit wordy. I want to have some profound and pithy lesson to separate then and now. Something that I can package up and hand to you. We certainly learned lessons, profound lessons, but in repeating them they sound trite:
Listen. Care about what people think. Be hungry for feedback. Don’t work forward from the tech you have to the product you can build; work backward from the product your users deserve to the tech you’ll need to get there. Ask for help, and accept it even when it hurts to admit that you need it. Don’t throw things away lightly, but be able to throw them away in those rare cases where it’s necessary. Surround yourself with the most excellent people you can find. That one helps a lot.
The haters will say I over-focus on star ratings. There are certainly lots of other good things to measure, and lots of bad things to say about stars. But the stars abide. They are inescapable. They sit front and center in your primary distribution channel. Those five little stars. And I believe they do say something about what we’ve built.
Building good things is hard and my hat is off to anyone who earnestly tries. If the good thing you build reaches people through an app store, I want you to know that I know how you feel. And I’m rooting for you.
