What I Learned: Sometimes great products are diamonds in the rough.

Decision to Update Local Search

Since 2011, Local Search has been the second-highest bookings digital product for Vistaprint (after Websites).

In May 2014, it was decided in Product Quarterly planning that the (then) Planet Express team at Webs would work on rebuilding the product; first launching it to Webs users, and then migrating it to Vistaprint users with the assistance of the O-Line team.

VP API Work in Advance of Project

In June 2014, Seth Moore, a new hire on the O-Line team at Vistaprint started work on the isListed and Profile Submission pieces of the application.

According to Seth, this application:

“Provides methods for dealing with local search business services (Google Maps, Facebook, etc.) It provides querying, which returns all of the services that a business is or isn’t listed [on]. Whether or not a business is “listed” is subjective since there are different levels of being listed. This service deems a business as being listed if a business is found that matches its name [and zip code].

The local search web service also provides an interface to submit businesses for being listed with all of the various local business services through a third-party service, Localeze.”

Seth spent roughly 1–2 months building the Local Search Web Service with architectural assistance from Arthur Correa.

Throughout his work, Seth worked closely with the Planet Express team at Webs to discuss service requirements, testing, architecture, access, and design.

It’s worth pausing here and acknowledging the incredibly significant contributions that Seth and Arthur made to the Local Search foundation.

However, it’s also worth noting that both Seth and Arthur work at Vistaprint in Lexington, MA. This is both a physically different location and a different code-base (C#) than the Webs SpaceX team which is in Silver Spring, MD.

The impact of this choice echoed both positively and negatively through the Local Search Epic.

What I Learned: There’s no such thing as a free lunch.

SpaceX Takes Ownership

Originally, the Planet Express team planned to accept Seth’s Local Search Web Service following the completion of their Internationalization epic. However, due to product and resource timing, they chose to complete the Package and Price Restructuring Epic first.

In July 2014, near the end of the Package and Price Restructuring Epic, Planet Express spiked extensively on the Local Search service that Seth and Arthur constructed.

Additionally, they developed the technical architecture plan for the Webs product. The new SpaceX team began in earnest on the Local Search MVP in August 2014.

What I Learned: I needed/need to learn more about technical architecture.

MVP Build

In planning for the Local Search MVP Epic, the new SpaceX team estimated that it would take roughly 4 sprints (8 weeks) to take the Local Search Web Service written by Seth Moore and launch an MVP product to Webs users.

There were several key features required for the Local Search MVP:

1. Search My Business: By submitting a business name and zip code, I should be able to see how many directories I am listed on by querying the “isListed” API service.

2. Purchase Local Search: After searching how my business ranks, I should be able to purchase a product from Webs that gives me a Local Search product entitlement.

3. Complete a Business Information Form: If I have purchased the Local Search product, I should be able to complete a form that contains all of the important business information I want to submit to Localeze (our data partner).

4. Submit My Information to Localeze: Once the form is completed by the Local Search buyer, we needed to connect to the Local Search Web Service to submit this information to Localeze.

5. Profile Page: After successful submission, users should have a page where they can see the information they previously submitted to Localeze.

Additionally, we should repurpose the lookup services in #1 above to show if a customer is listed on the directories after using the Webs product.

6. Dashboard Tab for Local Search: The MVP version of the Webs Local Search product will live on the Webs dashboard (only accessible to Free/Paid Webs users).

The SpaceX team completed these product features in roughly the 4 sprint (8 week) estimate. The MVP was silently launched to a 10% of Webs users on the dashboard on Thursday, 10/2/14.

What I Learned: There’s a fine line between MVP and shipping crap.

What I Learned 2: It’s difficult to determine when a product is ready to launch, but it’s the Product Owner’s job to explain the value of learning from real users to their team.

Pre-Launch UserTesting and Alpha Test

Once SpaceX took ownership of the Local Search product, Alex Mitchell sent several emails to current Webs Premium subscribers asking if they would be interested in “An Exciting New Offering from Webs” that helps their business “Get listed in over 100 directories online”.

If users were interested, they were prompted to complete a survey that included most of the information about their business that they would be ultimately be able to submit to Localeze through the Webs Local Search product.

Out of the 1,000 users contacted, approximately 50 responded to the survey with detailed information about their business. Out of this population of 50 responses, 30 contained all of the required data to submit a listing to Localeze.

This population helped the team understand some of the key challenges with the Localeze product (time to listing, what directories were more/less reliable, etc.) and allowed Alex to interact with users at an early stage in the process and truly understand the visibility challenges that small business owners experience.

Once the MVP was launched, these users were given promo codes to access the product at a discounted rate.

Ideally, Alex would have liked to have given these users a free subscription; however, there were technical limitations that would have required extra work.

What I Learned: Get a product to users as fast as possible, even if you’re still pulling all the strings behind the scenes.

Pre-Launch “Early Bird” Email List

Throughout the MVP build process, Alex Mitchell, the product manager for the Local Search product, ran Qualaroo surveys on the Webs dashboard to assess demand and understand pricing.

In a new tactic for Webs product launches, Alex collected emails from the population that demonstrated interest in the concept ahead of launch.

This will be referred to as the “Early Bird” email list. The list reached approximately 3k users in size by 10/2/14.

What I Learned: Every launch should include at least 1 thing you’ve never done before.

Pre-Launch Blog Post

While the Local Search product was still being built, Alex wrote a blog post on the importance of Local Search for SMBs.

The article was intended to build Webs user interest in Local Search and highlight the challenge of manually listing yourself in all of the important directories online.

See: Get Found Online With Local Search

What I Learned: Product Owners need to market and promote too.

Pre-Launch Internal Testing

One of the most successful components of the Local Search epic was the pre-launch internal testing. On 9/24, approximately 2 weeks prior to launch, the SpaceX team brought their computers to the lunch area and invited Webs employees to test out the newest product in the Webs family.

The results of the testing were incredible. Over 30 employees created sample accounts with the tool, searched for businesses they knew, and offered creative (and often painful!) feedback. After the 2 hour internal testing period, we had an additional 50 tickets of work. Some of these tickets were of course substantially more important than others.

While some of the feedback was challenging or scary to hear, it was incredibly helpful in showing the team what still had to be done before launch.

What I Learned: Feedback can be Painful, but Suck it Up and Listen for The Most Painful and Most Repetitive Experiences

MVP Launch

Following the product launch on 10/2/14, the SpaceX team closely monitored the user experience. After no errors were discovered, the product was ramped from 10% to 25% early on 10/3/14 and 100% by the end of the day on 10/3/14.

In the first outbound launch notification, the “Early Bird” email was sent on 10/9/14 to approximately 3k early opt-ins with a 20% promo code for the product.

The next week, notifications were sent to Paid Subscribers (10/14) and Free Users (10/16), with each email including a 20% promo code for Local Search.

The product had its PR launch on 10/20/14. The public PR launch included:

· Blog Post: http://www.webs.com/blog/2014/10/21/introducing-webs-local-search-get-business-listed-today/

· Social Promotion

· Facebook Cover Photo Takeover

· Press Release(s)

o http://www.prweb.com/releases/2014/10/prweb12263286.htm

o http://thesocialmediamonthly.com/give-local-some-love-foolproof-ways-to-conquer-local-search/

· Affiliate Network Outreach

Post-MVP Analytics

Very quickly after launch, Alex realized that we weren’t tracking enough data on our Local Search buyers. For a product like this, purchases and bookings alone weren’t sufficient.

Matt Halbe, a developer on SpaceX helped set up a Mixpanel funnel that covered the main steps of the user experience: Landing Page Visit, Search for Business, Shopping Cart, Purchase, and Local Search Profile Submission.

It’s common with this type of additional analytics work that you can’t see the full value initially. You might even choose not to do the work because the on-paper benefits don’t add up:

“Sure, it’s interesting to know the conversion rates of each step, but what do they really tell you for a brand-new product (at least on Webs)?”

Well, with each of these stages, Matt also passed user-inputted data. For example with the business search event in Mixpanel, he passed the business name and zip code entered by the user. Although this was largely an afterthought for Matt and Alex at the time, this data turned out to be incredibly valuable.

In the early days of Webs Local Search, a significant minority of the buyers were non-US.

Since the product doesn’t work outside of the US, these purchases almost always resulted in refunds. With a quick look at Mixpanel, Alex was able to see how many users were searching for international addresses.

Based on this analysis, the team quickly worked to make it clearer that the product was US only.

What I Learned: More data and measurement is always better, especially with a new product.

Post-Launch MVP UserTesting

After the Local Search product launch on Webs, Alex Mitchell quickly started UserTesting with www.usertesting.com He ran 5 tests on 10/23/14 and 5 tests on 10/29/14.

The NPS (Net Promoter Score) for this initial round of UserTesting on the MVP product was -11.

NPS is calculated as (% of Promoters — % of Detractors). Promoters are defined as scores of 9 and 10, while Detractors are defined as scores of 1–6.

While this initial round of UserTesting was somewhat discouraging, it’s worth putting the scores into context. The Local Search MVP product had been built in only 8 weeks.

Additionally, the NPS score can be compared to the NPS score of -32 for the Vistaprint Local Search product.

Finally, this round of UserTesting was extremely valuable because it exposed the most glaring issues with the Local Search MVP:

1. The search experience wasn’t good: At the launch of the MVP, our isListed API service could only return if a user was or wasn’t listed on Google. Additionally, the return from the search sounded like we did search many more directories. When a customer was relatively saavy, or knew where their business was listed online, we broke their trust at the first search.

2. What is 100+ Directories/What Other Directories are Included? Users were confused exactly where they were being submitted. There wasn’t a link to the 100+ Directories so users were confused if this was the name of the directory or what other directories were included in the 100+.

3. Submission Errors: 2 of our 10 test users ran into issues when they submitted their correctly formatted, correctly entered business forms. One of the surprising benefits of user testing was seeing exactly how users on different browsers, different internet speeds, in different countries experienced errors in the product.

4. Challenges with the Category Picker: Sometimes it’s hard to see that a user function is confusing until you actually see a user test it and speak through their thoughts. One of these surprising challenges for our users was selecting categories for their business.

We had a free text form (with no placeholder text) that didn’t offer any clear indication that it had search functionality. Most users clicked in the box and then proceeded to try to find their business type in a non-alphabetically sorted list that was almost 2,000 categories long!

We quickly realized we needed to improve this experience:

See full details on the MVP UserTesting here: Local Search User Testing

What I Learned: UserTesting/User Simulation is the most important thing you can do as a Product Manager when you launch a new product.

MVP Name Test

Soon after launching the MVP Local Search product on Webs, Alex wanted to test the name. The product was getting a substantial amount of clicks per day (2–3k), so volume for the test wouldn’t be a concern.

The test was simple, test the champion “Local Search” against two competitors: “Local Listings” and “Get Found Online”. As is the case for most simple tests at Webs like this, the test was run through the Optimizely testing platform.

The results 2 weeks later were conclusive:

“Local Listings” gets clicked on 11% more

Quickly, the name was changed to Local Listings. Unfortunately, this backtracking was annoying for the team, but it’s the type of test that is difficult to execute prior to launch.

Once the migration to Vistaprint is complete, we’ll execute a similar name test to see if we can improve on the champion “Local Search” name there.

What I Learned: Sometimes you need to backtrack to get on the right path

Phase II Milestones

Following the MVP launch on 10/2, the team took a 2–3 sprint break from the now-named “Local Listings” to focus on building the Webs Referral program.

However, Alex and the team set their sights on planning for Local Listings Phase II, where they would take what they had learned from UserTesting, Internal Testing, and actual users and decide what features to build next.

After taking into account all of these data input sources, the team began on Phase II Local Listings in the beginning of November 2014. The key priorities for Phase II were:

1. Improve Stability of the Local Listings Application/Track Errors: Stability issues in both the development and live environments were causing development challenges and paying user downtime.

2. Improve Submission Form, Edit Listing: The submission form, while functional, was in need of several important UX enhancements. Additionally, SpaceX needed to add the functionality to edit a listing, a feature that didn’t make it into the initial MVP work.

3. Launch Improved Business Search: As the team saw in UserTesting, our business search wasn’t great and several things needed to be done to enhance the user experience there.

4. Add Additional Directories to isListed Service: At the launch of the MVP, there was only one matching service in isListed for Google. The O-Line team added Yelp in late October, but 2 directories were still far too few for an effective Local Listings product.

5. Create an External Presence for Local Listings: The MVP Local Listings product was only launched to the Webs dashboard. Partly as a merchandising technique, but more importantly as a way to test iFrame-ing elements of the Local Listings application, the team sought to launch the Local Listings application at an external site: Webs Local Listings External Site.

6. Show Success Sooner: As we reviewed earlier, one of the biggest complaints with the Vistaprint Local Listings product is that customers really didn’t know if it was working.

In Phase II, SpaceX truly wanted to show success sooner. This included adding a lookup service for Localeze/Neustar, where every customer would be listed instantly and default a green check for 100+ Directories.

Following this work, the minimum green checks for any user would be 2. Success shown instantly.

7. Polish + Look Towards VP Migration: A key element of Phase II work was looking forward towards migration. At each sprint planning meeting, the team considered whether the feature in question was needed by Vistaprint.

Although we wanted to make Webs customers happy, we needed to rationalize that the Vistaprint population was considerably (10–20X/month) larger. This goal prioritized things like Adding Photos to listings ahead of adding emails for Webs users.

What I Learned: Phase II’s are exciting. It’s where you get to take what you learned from real users and put work against it.

Stability Issues (Thanksgiving 2014)

Thanksgiving 2014 was not kind to the nascent Local Listings application. Over the period of the 4 day holiday, Local Listings was down for a collective 48 hours. There were several key stability issues that caused downtime in the Local Listings application:

· Networking Issues with Vistaprint Servers

· oAuth Authentication

· Server Load Balancer

· API Code Errors (ex. API could be broken by user)

· MongoDB Connection Errors

A full overview of the errors experienced and the flow of the Local Listings application can be found here: N:\Product\Alex — Product — ROI\! Local Listings Product\Instability Analysis

What I Learned: Stability isn’t sexy. But downtime is even uglier.

Directories Added

One of the main goals of Local Listings Phase II was to add more directories to the search and profile pages. The SpaceX team delivered strongly on this mission, launching services for Neustar/Localeze, YP.com, and Yahoo over the 2014 holiday season.

What I Learned: Shower your users with success and they’ll stick around

Paid Search

After the initial “Local Search” external page (www.webs.com/features/local-search) was launched in late November 2014, Alex wanted to experiment with pushing more traffic there. At the time, the external page was receiving a minimal amount of traffic from organic keywords like “Business Listing” and “Business Directory”.

In the first week of December, Webs began a small scale paid search campaign targeting these types of search queries on Google.

Initially, the results weren’t too great. The cost of the keywords was approximately 50% more than the LTV (Lifetime Value) of the Local Search buyers. However, there were improvements to be made.

For the first 3 weeks of the paid search campaign, the Local Search/Local Listings external page still featured the “old/generation 1" search experience. In late December 2014, this was replaced by the iFrame experience above.

Since the new experience was launched in late December, approximately 396 visitors have come from CPC sources at a cost of ~$2/click.

From these 396 visitors, there have been 7 Local Listings buyers with a total LTV of $1,182.

This puts the ROMI (Return on Marketing Investment) at

1.49 or 49% above cost

What I Learned: Learn why something isn’t working. Fix it. Measure the results.

Continued UserTesting

UserTesting was an integral part of the Local Listings product development from MVP through the completion of Phase II. Although UserTesting.com testers were non-Webs users, they still were small businesses that could comment effectively on the Webs Local Listings product.

From October 2014 to early-January 2015, Alex ran UserTesting.com tests with 23 different users. Each of these ~30 minute sessions provided a wealth of information that helped him prioritize backlog items, identify errors, and improve poor user experiences.

As we talked about earlier, the Webs Local Search product began with an NPS of -11 in October 2014. However, after a significant portion of the Phase II improvements, 4/5 users tested on 1/9/14 were “Promoters”, while the 5th tester responded to the NPS question with an 8/10 score.

What I Learned: UserTesting is great. When you see improvements based on your work, its incredible.

Live User Feedback — Qualaroo

In addition to the UserTesting through www.usertesting.com, the SpaceX team implemented Qualaroo within the Local Listings application to get real-time NPS data from actual Local Listings buyers.

If a user is a “Detractor”, a follow-up question provides a free-response input for why they are not likely to recommend. If a user is a “Promoter”, a follow-up question nudges a user towards providing a testimonial, which we would like to use on our product in the future.

Additionally, as an “unexpected benefit” of Qualaroo in our application, we’ve been able to offer targeted promo codes to users. For example, if a user is about to abandon the business search page after being there for 30 seconds, we can popup the following promo code offer and try to retain them:

What I Learned: Qualaroo can serve many purposes for a new product and can target customers incredibly well.

2015 Wrap-Up

As the team entered 2015, they were excited to wrap up the goals of Phase II still outstanding, as well as some new ones that had surfaced as they thought about VP migration:

1. Build a Webs isListed Service: Bundle the individual services written by SpaceX into a maintainable, extendable, and understandable API that can be called by both Webs and Vistaprint.

2. Add Statistics to Local Listings: In a further effort to show Local Listings users that the product is working, we’ll expose pageviews from Local sources.

3. Upload Photo/Logo With Business Listing: A significant portion of Vistaprint Website/Local Search users have a logo for their business. In mid-January, we’ll add this feature to our business information form.

4. Add Facebook to isListed: As we near the end of Local Listings, SpaceX will aim to add Facebook as another directory on the isListed API.

What I Learned: Once you’ve built momentum, it’s tough to stop building.

Data Partner Evaluation

In late 2014 and early 2015, we also explored additional data partners that could improve on the submission and listings aspects of the Local Listings product.

While Localeze is cheap, it is a black box and offers no guarantee a user will be listed. Additionally, our team had to research on our own to determine what directories were the most likely for our users to get listed on.

After evaluating these data partners, the team decided to continue to use Localeze for our base offering and conduct demand testing to explore a higher end, DIFY solution in early 2015.

If there is appetite at a higher price point for a more robust product, we plan to explore a partnership with Yext.

What I Learned: You generally get what you pay for with data partners.

Migration to Vistaprint

The O-Line team will migrate the Webs Local Listings product to Vistaprint in several key stages in March-April 2015.

1. Launch New External Page

2. Replace Local Search Product with Webs Local Listings Product via iFrame

3. Price/Name/Merchandising Testing

Local Listings Marketing Videos (Unfinished)

With the Local Listings videos, Alex and George Goodman, Webs’ videographer, wanted to depart from the typical feature-based videos that Webs and Vistaprint were known for.

Instead, they wanted to highlight the problem of missing/incorrect listings to business owners in an engaging, realistic, and humorous way.

1. Piano/Music Teacher
2. Locksmith
3. Florist

Password(s): local

Great Decisions in Local Listings Epic

1. Internal and UserTesting: If you’re not doing these as a product owner, you’re not doing your job

2. Launch strategy: Have a diverse launch strategy because you don’t know what channel will resonate strongest with your customers

3. “Show Success Sooner”: Make sure customers feel like they are getting what they pay for and make that feeling happen as quickly as possible

4. Spending a lot of time with users: Beyond internal and external testing, you should be simulating, emailing, and surveying your users consistently.

Poor Decisions in Local Listings Epic

1. Having someone at a different location, on a different team, with a different schedule, create a vital service that our application relies on

2. Alex should have brought a key member or two from the team to Lexington to meet with Arthur Correa and other individuals responsible for the application architecture

3. Architecture and Stability are IMPORTANT!

4. Error monitoring should be set up before errors occur, not after