You’ve got it. It’s the perfect evolution of your brand, encompassed in a 152 x 152 pixel square. Leadership is happy, your team feels great about the design process, and your colleagues are begging for new branded swag.
The design of your new app icon may be done, but your work isn’t! Where the creative work ends, the analytical work starts: it’s time to test and roll out your icon.
In our recent app icon rollout process, we identified six key steps every marketer should go through when updating an icon. These steps will help you to:
- Understand the impact your new design has on key audience segments (at different stages of interaction with your brand)
- Set up a framework to understand the impact the icon is having on key business metrics.
We’re sharing the six steps we took to test and roll out the icon changes. Our goal was to minimize risk and optimize for understanding, not speed and the whole test and rollout process took less than two months. Trust us, it was well worth the time! These steps can be applied to nearly any type of app:
1. Pick the Right Time of Year
The first decision you have to make is when to kick off the rollout process. Ideally, you should do this during a stable time of year with as few external variance drivers as possible. You’ll also want to select a time that mitigates the risk of any unintended effects of the change. Our icon design was completed in November, but we chose April to launch our test. There aren’t any large holidays in April, and our typical New Year Resolution seasonality has usually worn off by then. This stable time gave us a clearer view into key behavior metrics and allowed us the freedom to test the way we wanted to.
2. Test Icon with New Users
- Audience: New Users
- Test Channel: Google Play Store
- Question: Does the new icon make people more or less likely to download your app?
To understand the icon impact, set up a simple A/B test in your Google Play store listing. We assume there’s a reasonable correlation between Android and iOS users when looking at these sorts of tests, and the Google Play store platform has an easy test interface for these experiments. Aim to keep all other marketing efforts constant during the testing period.
Our hypothesis was that the new logo would have no impact on the number of downloads. As long as we saw no statistically significant decrease in the amount of downloads with the new icon, we would move forward with our testing.
Here are the results from our test:
In less than two weeks of running this test, we learned that our new app icon was not expected to have a significant impact on the download behavior of new users. Since there were no issues identified, we moved forward with our plan.
3. Survey Existing Users
- Audience: Existing Users
- Test Channel: Survey
- Key Questions: Is our new icon recognizable as the Lose It! brand to our existing users, or will it cause confusion? How do we expect existing users to react to this change?
Set up your survey so you can answer the following questions, both for your existing icon and your new icon:
- Is the icon recognizable as your brand’s icon?
- Does the icon align with your brand values?
- How do people feel about your icon?
Ideally, you’ll leave this step with an understanding of the comparative difference between the two icons.
Split your recently active members based on operating system type and icon version:
Be sure to ask the same set of questions to all the groups, just switching out the icon that you’re asking about. Here’s a link to view the survey we sent to our users.
The results of our survey were surprising:
- Findability: When asked to select the app icons that they had seen before out of a list of orange-colored app icons, 95% of respondents who were shown our new app icon selected it, and 96% of respondents who were shown our old app icon selected it. This told us that if placed on a user’s home screen, they would likely be able to find the new icon just as easily as they were able to find the old one.
- Recognition: When asked to select out of the same group of icons which was most recognizable as the Lose It! app icon, 98% of respondents who were shown the new app icon recognized it as Lose It!, and only 94% of respondents who were shown the old app icon recognized it as Lose It!. This confirmed that the changes we made to the app icon didn’t create confusion among our members.
- Keyword Association: The keywords selected for each of the two app icons were very similar. The old app icon skewed more motivating and the new skewed more modern. This confirmed that our new app icon was still evoking the feelings we want to be associated with our brand.
- Preference: At the end of the survey we revealed both versions to respondents and asked which they preferred. Respondents who had been shown the old icon throughout the survey preferred the old icon 65% of the time. If shown the new icon throughout the survey, however, the preference for the old icon decreased to 41%. This told us that pre-seeding the new icon throughout the user experience could help increase preference for the new icon.
- Qualitative Feedback: We included an opportunity for respondents to share their open-ended feedback so we could capture overall thoughts about the icon change. The comments we received were centered around a dislike for the flat design trend and were not focused on our specific app icon changes.
If we had seen a negative response to our new icon during this survey, it might have indicated that the change we were making to our icon was too drastic. The information we received through the survey helped us shape our rollout strategy and craft the communication material that we used during the switch.
Now you’re ready to release the new icon. In your next build to the Android and iOS app stores, include the new icon without changing any other brand touchpoint.
4. Monitor New User Behavior
- Audience: New Users
- Test Channels: Apple Search Ad Performance & iOS Installs
- Question: What impact is our new icon having on branded search & install performance?
Closely monitor the Apple Search Ad performance and App Installs over at least a week to understand the impact the new icon has on acquisition efforts. Since you’ve already tested the impact of the app icon on the Android download behavior, you can focus on the iOS behavior.
To fully understand the impact of your change, you might want to involve your data science or business intelligence team in this stage. Using the historical data you have on Search Ad performance and App Installs, predict the expected range of activity over the testing week. This will give you a really clear indication if users are behaving as expected when the new icon is being used.
Search Ad Performance
The goal is to understand if your audience recognizes your brand when they see an advertisement with the new icon, so click-through rate (CTR) is the metric to focus on. We looked at branded search performance to understand the impact of the icon on an audience that was already familiar with our brand.
Even though our app store presence still had our old logo in the app card images, we wanted to understand if the main logo change led to a statistically significant change in download behavior.
These two metrics gave us the confidence that the new icon wasn’t having a significant negative impact on the acquisition efforts of our business, so we moved forward with our rollout plan.
5. Monitor Existing User Behavior
- Audience: Existing Users
- Test Channels: App usage data
- Question: What impact is the new icon having on existing user engagement?
Arguably more important than new user testing is how you listen to your existing users during a brand change. Making it harder for existing users to find your app on their phone could hurt key engagement metrics. Our data science team predicted app engagement during the testing week, and we compared our actual engagement to their predicted values. The app icon changed over slowly for existing users, following the standard iOS rollout phasing.
Once we determined that the new icon wasn’t having a negative impact on our key business metrics, it was time to roll it out across the other touchpoints. Since we had the whole team bought into the key metrics ahead of time, it was really easy to make the ‘go’ decision and have all hands on deck to complete the rollout.
We had been closely working with the other teams around the company to create the icon files needed to complete the rollout. With the assets ready to go, it took less than two weeks after testing to update the icon across every brand touchpoint.
6. Listen to Your Members
- Audience: Existing Users
- Test Channels: Social media, customer support, app reviews
- Question: What are your customers saying about the new icon?
As tempting as it is to want to justify your design changes to your audience, the most important thing you can do is listen to your members. Pay attention to the constructive criticism you receive and learn how your customers react to brand changes.
Because the changes we were making to our icon were incremental, we were expecting (and hoping for) very little in the way of feedback. We tracked the qualitative feedback through app store reviews, social media comments, and inbounds to our customer care team.
Even after you’ve successfully made it through this rollout process, there is still a risk that less active members will have a hard time finding the new app icon. Continue to monitor your user reactivation data periodically to make sure it remains in line with your expectations.
We’ve been monitoring the feedback from our members for the last couple of months, and have learned that members would have appreciated a heads up prior to the icon switching over. The process described above puts testing above customer communications, so you’ll have to decide if the change in your icon merits a different strategy.
We are tempted as marketers to make big brand changes all at once but this process has taught us that slowing down can add a lot of value back to the business. We loved our icon before we rolled it out, and now we’re confident that our users love it as well. Through this process, we got smarter about the impact of our brand at key decision points, we learned the preferences of our customers and we built trust internally for our marketing process.
We’re proud of the results from taking a data-backed approach to an app icon rollout and we hope you’re able to use some of these practices in yours! Have you taken a similar approach to brand changes at your company? I’d love to learn about them!