When a Site Refresh Becomes a Regression — A Story of Startup Anxiety

(Originally published on April 15, 2014)

We launched MoonClerk about 14 months ago and from our start to this month (April, 2014), we’d kept the same design for our marketing website. I figured it was time for a refresh and I wanted to make some improvements so I worked with the same designer who helped with our original site to come up with a new design. Now, our original site wasn’t broken. In fact, it had been converting well. We had been converting 8.96% of our unique visitors into trialing customers over the previous 6 months. That’s pretty awesome for a SaaS web app, but who doesn’t want to improve? We flipped the switch on April 1, and well… things didn’t go as planned. I experienced some of what I’ve experienced before — the stereotypical panic, anxiety, and hard lessons that come with running a bootstrapped startup.

For some context, here’s what the above-the-fold portion of our original website looked like. You can see see the original site on Wayback Machine.

Some of the updates my designer, Justin Hall (who’s smart, awesome, and great to work with by the way) wanted to make were aesthetic in nature — updates like lighter backgrounds, different textures, more subdued illustrations, etc. Because we’ve been adding happy, paying customers at a pretty fast clip, I wanted to add a Testimonials page to our main navigation, so we needed to design a new page for that. I had also developed a short compilation testimonial video that I wanted to appear on the homepage. Overall, I wanted testimonials to be a bigger part of our site. Previously, they just appeared as blog posts.

At the same time, I had also had been doing some research on improving the headline copy for our homepage. Even though we weren’t moving to a long form sales page, I tried to pull some nuggets from this blog post by GrooveHQ. I interviewed a good many of our customers and discovered a common thread in the reasons why they used and liked MoonClerk. The first is that it’s easy. The second is that it’s customizable and flexible. I’ve always liked philosophy of “Don’t tell me you’re funny, tell me a joke that makes me laugh.” I like to apply that same philosophy to “easy.” I don’t want to say on our homepage that we’re easy-to-use. Everyone says that. Heck, even web apps that require months of training and implementation will claim to be easy. No, I’d prefer to have a website that’s easy to understand and an app that people enjoy using. I want to show easy, not say it. So, I decided to focus on the flexible part. After a lot of iterations, I came up with the phrase, “MoonClerk gives you the flexibility to accept recurring and one-time payments according to how you do business.” I also hate when I go to a website and I don’t immediately know what it’s for so I wanted to make it abundantly clear that we’re all about recurring payments and that the signup process is quick. So, I came up with the simple headline “Instantly Accept Recurring and One-Time Payments”.

We launched the redesigned site on April 1. Here’s what the above-the-fold portion of the homepage looked like:

The first week of April after we launched the site refresh was a complete bloodbath, at least in the eyes of a startup founder. Our conversion rate from unique visitors to trialing customers plummeted by 50% — half of what it had previously been. That’s the bad news. The good news — our unique visitors were slightly up (but not by much) and our conversation rate from trialing customers to paying customers stayed the same (we hadn’t changed anything in our app.) So, in my eyes, we were obviously losing people because the site refresh wasn’t working. At the time, I couldn’t prove that quantitatively. We don’t have a ton of traffic and, after all, it could have just been that we were having an uncharacteristically bad week. Since then, after reading this blog post by Kyle Rush, I’ve learned that I did have enough data to make a decent determination as to the effect of the site changes. At the time I didn’t know that. I was basing my judgement and emotions on a combination of data and gut. Something in me said that our bad week wasn’t just a fluke. For some reason, the site wasn’t converting well.

I came up with three different options that I could take:

1. I could leave the site as-is and gather more data to see if the new site was really the problem or if the first week of April was just a fluke.
2. I could punt and revert back to the old site and then regroup. After all, the old site wasn’t broken. With this option, I could A/B test with slight variations that slowly brought me back to the refreshed site.
3. I could try to make some educated guesses as to why the new site wasn’t converting and just revert those elements back to what I had before. With this option I could A/B test one-by-one some of the main changes I had made that I though were negatively affecting conversions.

I chose option 3. Overall I felt that the majority of the aesthetic changes weren’t what was causing the plummet in conversions. For example, we had removed the texture in the header where our main navigation lives. I doubted something like that would severely affect conversions. I also doubted that putting a testimonials page in the main navigation or slightly changing the background color of the site would negatively affect conversions. In addition, I doubted that changing the icons and a little bit of the wording on the content below the fold on the homepage was drastic. We didn’t really make any changes to our blog, FAQs/Help, or Pricing pages. So, my educated guess was that the huge dip in our conversion rate had to be the changes we made to the above-the-fold area on our homepage.

That’s the part of our site that I reverted back to the original. Here’s what we changed the site to on April 8 (you can visit http://moonclerk.com but the site may have changed since the publication of this post):

As you can see:

• I removed the Watch Testimonials button (previously this pulled up a modal window that played the video.)
• I reverted the headline copy back to what it was before.
• I added the “Explore for free. No credit card required” text back.
• I went back to the original background image and app/checkout screenshots on a computer and iPhone.
• I removed the subhead copy that talked about our flexibility.

It’s been a week since we reverted the above-the-fold area back to close to its original state and it appears our conversion rate is not only back to what it was before April, but is actually slightly higher. So, I’m making an educated guess that my gut was right: The conversion rate drop wasn’t a statistical anomaly; it was actually dropping. And, the reason for the drop was the group of changes we made above the fold on the homepage. I’m not sure yet which changes caused the drop. Maybe it was a combination of changes. Maybe it was just one of them. I’m now going to start A/B testing to see if I can figure that out.

It’s a cliche to say that the life and emotions of a startup founder are like a roller coaster, full of ups and downs. But, there’s a reason cliches exist. It’s because they most often ring true. That’s the case here. March had been our best month ever — by far. On March 31, I was feeling so encouraged. We had momentum. Then, the first week of April, as I was watching our conversion rate drop, I felt like a failure. Surely I had made a poor decision. Why didn’t I A/B test? How could I have screwed up so bad? Could we come back from this and still have a good month? Was I even making a good decision to revert back to our original above-the-fold design? Why had I been trying to fix what wasn’t broken?

I’m not a Vulcan — I don’t know if I’ll ever be able to separate my emotions from rational analysis. If we have a bad day at MoonClerk, I don’t automatically think, “Oh, today is just a regression to the mean. I know things will be better over the next few days. This is just statistics at play.” No, I think, “Crap. If we don’t have an awesome day tomorrow, we’re never going to recover. The pressure is overwhelming. I’m in charge of making this business a success. What am I even doing trying to start a business?” It’s easy to know in your head that if you flip a penny 100,000 times, roughly 50,000 of those times will be heads and 50,000 will be tails. But, if “tails” means you’re losing and you’re only thinking about the next 100 flips, when you get tails 10 times in a row, you start to panic.

I may not do a great job of controlling my emotions, but there are always nuggets I can choose to take from any hard situation and see if I can apply them down the road.

1. The main nugget I took from all of this is if I’m running a SaaS business, I need to A/B test major changes. I guess if I ran a service business and my income wasn’t really tied to my conversion rates — if my site was more of a brochure — then it wouldn’t be as big of a deal. But, given that conversions are my lifeblood, I’m going to start A/B testing in the future.
2. The second nugget I took is that in order to keep my emotions in check, I need to evaluate worst case scenarios. Few startup decisions are irreversible. In this instance, worst case was that I would revert back to the old site and have lost 50% of my customers for 7 days. In the scheme of things, that’s not so bad. I can recover from that.

Business is hard. I’ll never have it all figured out. In the future, I want to focus on what is going right (increased traffic, awesome trialing-to-paid conversion rates, happy customers,) work efficiently, and grow in wisdom from the situations that do cause me to panic.

Originally published at blog.doddcaldwell.com.