My biggest failure
On Launching Penultimate 6 for iPad
I’ve made a lot of mistakes I guess—both personally and professionally.
This is the story of the biggest design mistake I’ve made. It doesn’t have a big happy ending, but sometimes mistakes are just mistakes—not lessons.
Let’s start with celebration
With champagne glasses raised high, we celebrated. It was 10 in the morning so we called them mimosas. I had bought a couple bottles of champagne, orange juice and snacks for the whole team so we could celebrate our success. While we all sat there and quietly celebrated, there was a sense of great accomplishment, but also a tangible sense that we could have done better. Timid optimism I guess you could say.
Whenever you launch something, you’re always keenly aware of all the shortcomings — of all the things you wish you had done better. That’s natural, I guess.
I had no idea, though, that I was not launching into the world my greatest success, but my biggest failure.
Penultimate had been at 4.5 stars in the App Store for years but would drop to 1 star for a painfully long time, and even today, exactly one year later, it’s only gotten back to 3.5 stars.
The Launch (and the why)
On 10.29.2014, early in the morning, we launched Penultimate 6. It was a major, ground-up redesign — and not one part of the product was left untouched. These types of redesigns are tricky at best. You know some people will be unhappy, but the hope is that the new changes will improve the product for a solid core of users and open it up to many more.
We spent a lot of time weighing the idea of even starting the redesign and finally decided that for a few reasons, the app had reached as far as it would go in its current form. Evernote had bought Penultimate a few years prior when it was top on the charts of paid iPad apps. It was rooted in original iOS skeumorphic metaphors that severely constrained the new features we could add, as well as how the data in Penultimate could be used in the Evernote ecosystem. There’s only so much you can do with an actual physical notebook.
After months of research, surveys, exploration, user testing, beta testing, iteration and playing with all these versions on our iPads, we decided to launch publicly.
The realization — a cacophony of criticism
Within a few hours after the launch, we had promotion from Apple, millions of people updating — and a few grumbling users. By day two, we had a growing list of serious complaints. By day three I wanted to crawl in a hole and hide from the world.
Making big changes to an app always comes with a bit of resistance. That’s to be expected. Generally speaking, people don’t like change — especially to something they love. I was ready for the complaints, because they are necessary sometimes in order to move forward. What I wasn’t ready for was how accurate and insightful the criticisms would be.
What went wrong
There’s a lot but I’m going to keep this to my own personal responsibility. There were probably four areas that, done better, would have made for a much more successful redesign.
Testing & Segmentation
Showing designs to users early and often is an extremely important part of the design process for me. Penultimate was no exception. However, I made one really key mistake. Segmentation. Evernote has over 150 million users, and Penultimate users are a subset of those users. Because of this, we included a lot of Evernote users in our user studies— even those that did not, and would not, use a handwriting app on their iPad—assuming that they would serve as a good proxy for Penultimate. This was a huge oversight.
False assumptions
What we got back from user testing was largely positive because the voices of the heaviest Penultimate users actually ended up being mutually exclusive from heavy Evernote users. We thought they had a lot in common, but it turns out that most people that are using an iPad for handwriting are in a very different mindset than someone that is on the go and typing into their phone. This seems obvious in hindsight.
I also assumed that the skeumorphic metaphor of a notebook was dated and no longer desirable — that users would prefer the shiny new UI paradigms afforded by iOS7 (i.e. “flat design”). This was hugely false. The metaphor of a physical page was not antiquated at all. Paper is not a thing of the past. In fact, those metaphors are powerful in their ability to help users with a sense of space and navigation inside of the app.
Fixed deadlines
We were working with a physical product — the Evernote edition Jot Script Stylus. Physical products have much longer timelines and don’t have the flexibility of release dates like software does. If Target says they are going to put your product in stores on November 1st to be there for Black Friday, that’s what they are going to do, whether you are ready or not.
I made promises that we would be ready to launch on a certain day. For many reasons, we simply weren’t ready when we should have been. Even so, I felt the pressure to meet the dates we had set, and I allowed us to ship the product before it was at an acceptable quality level, and before it had undergone further user testing. This was silly. External timelines are the enemy. Other companies have their own objectives and should never dictate the quality of your app. #lessonlearned
Communication and auto-updates
For many users, iOS automatically updates their apps. I knew this. Somehow though, I didn’t take into account the gravity of the fact that a user would open up their app one day and it would be completely different. The worst story I heard was someone that was taking notes in a university class, switched apps to look something up, and when they came back to the app, it had updated and they could no longer find their notes. I don’t think they use Penultimate anymore.
We should have had a much more significant experience inside the app to explain the new change and what users could expect. Just the general shock of a huge change and having to figure it out by themselves was one of the hardest things to overcome. We probably lost a lot of people just because they didn’t feel like having to learn a new app.
I’m sorry
This was a project I put a lot of my heart into. I led the product direction, the product management, and the majority of the design. It was one of the most painful things I’ve had to endure to sit and listen to thousands of pieces of feedback — most of which were incredibly true. It hurts when you realize that your mistakes were harmful to someone. And when that someone is backed by millions of similar voices, it is borderline unbearable.
I’m sorry that I didn’t take the chance to make this app what it could have been, and I’m even more sorry that my decisions had the impact that they did.
Triage
We immediately took the feedback to heart—along with their pain—and started making improvements. I reached out to people on the forums to ask them what we could do better and then sent them prototypes of what we planned to do. We wanted to get it right next time. We reintroduced pages, we fixed bugs, we improved zooming, we scrambled to get in fix after fix. We apologized.
But in some ways it was too late. Users felt betrayed. And now, because of that betrayal, our bar had been raised and it was even harder to meet those new and higher expectations.
Where do we go from here
I’ve probably learned a lifetime of lessons from my mistakes on Penultimate, but I’ll have to save those for another time — maybe next year. For now, I hope there’s a lot of value is simply admitting that I made mistakes. It’s too easy to turn our failures in to big wins with a big lesson. Sometimes we need to just live with the pain for a while.
And while I made some (big) mistakes, I’m also remarkably proud of the Penultimate redesign. In fact, it’s one of my favorite pieces in my portfolio. Even with all these mistakes, I think we made a pretty awesome product.
So yeah, I’ve learned a lot. But let’s call this what it was. It was my biggest failure.
One final thought
This is in no way meant to be a negative reflection on Evernote or anyone else on the team. I have a ton of respect for everyone I worked with on this. I decided to leave Evernote 6 months ago to help startups launch new products (and hopefully avoid a lot of these mistakes). Evernote has continued to make improvements. The app is now back up to 3.5 stars and heading in the right direction due to their hard work.
You can download Penultimate for your iPad if you’d like to see it for yourself.
