Product Festival 2017: Feedback, Emotions, and Data

I attended Product Festival NY 2017 today, a day long conference in NY showcasing case studies and presentations from some great thought leaders in the product space. A lot of great lessons were shared — here are some of my favorite takeaway notes.

Jason Schwartz, Spotify

Jason discussed the process of identifying the “light bulb moment” once you’ve found your product market fit. Specifically, your product has taken off, user acquisition is climbing, and you don’t know why — how do you identify the Aha! moment that converted these users so that you can direct as many leads to this happy path as possible?

  1. Identify Behavior that might have led to the happy path. For example, you might notice that the majority of the users who converted to your product uploaded at least three pictures to your platform. We can then hypothesize that uploading three pictures is instrumental in getting users to take the happy path to conversion. So how do we test it?
  2. Instead of building an expensive feature, Jason recommends two low cost ways for a product manager to test her behavior hypothesis. The first is external messaging — bring users back to your app by contacting them off your platform to encourage them to complete the identified behavior. In this example, we might send an email encouraging a user to come back to the platform and upload 3 pictures for an incentive — like a greater chance of finding a match or for a store credit.
  3. The next low cost way is to guide your user in-app to complete the identified behavior. Now that you’ve brought them back to the platform, provide whatever guidance or education necessary to complete the task — whether it’s colorful, attention-grabbing tool tips or clear, concise copy outlining the steps for uploading pictures.
  4. Finally, measure for causality. Having implemented these low cost message and education methods, do users who see these messages actually have an improved conversion rate? Jason notes: the answer is probably no. It typically takes several, several iterations of this process before you actually find the behavior hook.

Blake Barnes, Instagram

Blake went through some problems they identified at Instagram and the design solutions that they implemented to fix them, but there were two main takeaways that I found to be universally applicable to all product managers:

  1. Consider the user when you change or A/B test a new feature. Specifically, the user experiences a jarring loss of control and disorientation when big changes occur in an app that they use daily. Be thoughtful about when and how often you introduce these changes.
  2. The metrics that you pick for measuring the success of your change should fulfill three criteria: quick, relevant, and simple.
  • Quick metrics have a short feedback loop (i.e. it would take several months for you to get a good Monthly Active Users number)
  • Relevant metrics should measure an action that would be directly impacted by your change (i.e. if users are already using your app daily, then Daily Active Users wouldn’t be a good indicator of your feature’s success)
  • Simple metrics should be easy to capture, like a tap or a swipe.

Mindy Zhang, Dropbox

If you follow my social media at all — my Instagram stories, in particular — you’ll have noticed that I snapped nearly every single slide of Mindy’s presentation, because I freaking love her message: what if we sought feedback for ourselves the way we seek feedback for our products?

  1. Maybe then instead of being nervous and offended, we would invite feedback with appreciation and curiosity.
  2. Maybe then we’d be more proactive in seeking feedback, and we’d have a better idea about what is a one-time, situational, biased comment, and what is a thoughtful, critical area of improvement that would reduce friction between you and your stakeholders.
  3. Maybe then instead of once or twice yearly reviews, we can respond iteratively to frequent feedback by speccing a small change, implementing it and getting feedback to speed up our personal upskilling.

Brittany Menutti, Google Daydream

Brittany had a few great messages, but her case studies really drove home her message of “Your community is your greatest asset.”

  1. While at Etsy, she implemented a personalized feature release email to Etsy sellers who she identified should benefit the most from the new features. The emails invited the users to respond directly with feedback, but the users gave much more than that. In addition to feedback about the feature, they also began suggesting enhancements, feedback on other features, and became active community members on the Etsy forums, despite having been nominally quiet members previously.
  2. While at Google Daydream, Brittany received a 3 page long email itemizing every single bug and flaw in their current system. Instead of ignoring it — as most product managers might — she responded, line by line, to every concern. That user became a huge asset to the product team and eventually collaborated on new features with Daydream.

Devang Thakkar, Artsy

Devang touched on a concept that was pretty new to me: using emotions as a guiding principle for building your product.

  1. Look at the current process — what emotion is evoked during the process? In the case for Artsy, the traditional art auctioning world featured an auctioneer narrating bidder actions and eventually announcing the winner in a celebratory, albeit awkward and restrained manner.
  2. Identify other parallel/unrelated domains with similar processes. For Artsy, the unrelated domain that resonated was a football (soccer) game on TV: the sports announcer narrates the players actions and eventually announces the winner. The celebration here, however, is one filled with excitement that includes the announcer and the audience too, not just the players.
  3. Mimic the traits of the unrelated domain to evoke similar emotions that the users take home with them. Artsy created a lot of excitement for its auctioneers, bidders, and watchers by using techniques from football announcers. They created a platform where the offline and online bidding action were captured on screen in real-time with narrative notifications, countdown buzzers, and announcements for the winners.

Inga Chen, Squarespace

Inga’s background in data products shed some light on the way we present stats to our users.

  1. Appropriately, she also discussed the role of emotions, but in data products. Understand the opportunities to use simple numbers to evoke emotions with your users — what is a great number for them to have achieved?
  2. Google Analytics, while free, really doesn’t filter or interpret the data for you — it’s a rich, cluttered environment that can be overwhelming. On the other end, if you present too little data with no background, the user doesn’t know what to do with the stats you present them. Find the essential complexity between the numbers and help your users interpret the opportunity presented by the data.
  3. Ideally, your product should be collecting sufficient data to allow personalization via machine learning, but what if you don’t? See what data you do have, and label it where possible, and then start collecting data, yesterday.

Alix Fitzgerald & Caleb Rotach, Betterment

Betterment introduced a messaging feature for customers to discuss their specific investment questions with licensed financial advisors — this is an interesting play in particular, as it helps them stand out from other roboadvisor companies by showing off the humans behind the platform. Alix & Caleb used this messaging feature to walk us through the product process at Betterment.

  1. Build a strong core — common advice, but what they did that I found especially interesting was the tight documentation and communication process that allowed every single team member to convey their feature goals so concisely that any one of them was able to reach out to stakeholders with their team’s strong, unified message.
  2. Align on success — again, sounds common, but a particular tactic they used was to create a key metrics scoreboard (with proxies, where necessary) that was shared broadly with everyone they could get their hands on in order to optimize their metrics and align them with their broader team, division, and company.
  3. Protect the magic — this was a new one for me. They talked a lot about their feelings about the product and their customer needs, and kept those central to the product decisions and fought for these feelings whenever challenged by developers or designers. They even included the “magic” as a category for QA.

Lauren Berk, Tesla

First and foremost, I need to impress upon you how this is far and beyond my most favorite slide of the entire day. It is Lauren’s — very accurate — representation of the PM role: one that should be focused on the user and customer, but typically gets side tracked by the volume of emails and loud shouting from their business and engineering counterparts. It’s easy to confuse your stakeholders with your customers, if for no reason but that you hear from them more frequently and in more direct channels, and this was a great reminder.

We’re all familiar with the story point system for estimation and prioritization of enhancements and bugs, but Lauren suggested an excellent addition of weight based on who the request comes from and/or largely benefits:

Once you add weight as a criteria based on the enhancement origination, with all things being equal, it makes it much easier to prioritize fixes for customers.

Nickey Skarstad, Airbnb

I found Nickey’s marketplace talk the most actionable and info-packed. (On the other hand, I spend a big chunk of my days working with Bloomerent on their flower sharing market place, so I may be a bit biased.) Nickey suggests that it’s a misnomer that supply and demand are equally important in a two-sided market place, and in actuality, if you focus your energy on building high quality supply, the demand will follow. “Your supply will ultimately set you apart.” She paired three lessons with really elucidating case studies:

  1. Build with intention for quality and give your sellers the support necessary to create your “castle in the sky” as it will create a benchmark for future sellers to mimic. For the Airbnb Experiences product, they sent a team of professional videographers, photographers and graphic designers to capture trip experiences all around the world and turned each of their product listings into high quality, standalone content. The stunning imagery sends a strong message to future sellers about what Airbnb expects from them.
  2. Friction around supply acquisition is fine. This was a surprising finding she shared: if you make your product too easy to use, you may wind up having poor quality sellers. When Etsy improved the seller onboarding process to bring more sellers to the platform, they found that they were able to onboard 20% more sellers, but these sellers were less successful and hurt sales overall by bringing down the average quality of all Etsy listings. When they reintroduced friction in the onboarding process, it helped filter out the sellers that didn’t want it badly enough.
  3. Let your users be growth hackers. When she gave incentives to her sellers on Etsy, they became brand evangelists. Etsy introduced a “treasury” feature that allowed sellers to recommend products from other sellers — those who recommended and brought traffic to other sellers made it to the front page! An entire side community arose of sellers finding each other to help each other promote their products. “Small businesses are scrappy and will hack their own growth.”

Justin Gallagher, Trello

We wrapped up with Justin’s presentation on lessons learned building Trello from the ground up to where they are today. It was a great finisher recapping the mistakes made and lessons learned.

  1. They overthought things and overengineered their features. Build for now.
  2. They let users vote on feature requests and users quickly got disappointed when their features weren’t built because they didn’t align with the product roadmap. Connect with passionate users, but set expectations carefully.
  3. They ignored their own feedback from their developers because they didn’t want to build a tool just for developers, and when other users echoed that feedback, they doubly ignored it by assuming those users must also be developers and thus not the target market. Collect feedback you can trust by A/B testing to generate traceable data points.
  4. They added a flat-rate paid plan and forgot to update it when organizations with hundreds of users started using Trello. Be diligent about paying down product debt.
  5. They added too many nouns by giving every new feature a name; this sharply increased the learning overhead for users and turned an elegant product into something clunky. Document product and design principles and build accordingly.

Great conference with lots of practical takeaways that I can apply immediately to the products I’m working on. Looking forward to next year’s!