Swarm Testing is rapid user testing for startups
For startup PM’s: How we got 300+ users to test our B2B prototype in 1 month
Context
At Teem, we build consumer experience software that we sell to enterprises. We build apps for everyday employees to more efficiently go about their day-to-day. In this dynamic, our customers are enterprise companies, but our users are “consumers” within those enterprises. Our customer contacts are usually directors of IT who make our products available to the end users, within their company.
Ideas, without execution, are just ideas. A major part of executing on an idea, and turning it into a product, is testing your assumptions and iterating based on your findings. It’s a critical part of ensuring you’re building for impact and ultimately, finding product-market fit. An important step in how I build a product is user testing.
We were looking for consumer-type users, without an agenda specific to our offering, to test our product.
Don’t take the path of least resistance
It’s easy to get someone to test your designs and provide feedback. It’s not as easy in B2B Saas to identify and engage target users who can provide unbiased input. The more familiar your testers are of your product, the more biased the input will be. Feedback from people within your company is the path of least resistance, but also the most tainted. Power-users or contacts at your favorite customers are easily accessible, but often have feature-driven agendas when “testing” your designs. We were looking for consumer-type users, without an agenda specific to our offering, to test our product.
Identify and engage target users who can provide unbiased input.
Getting 300+ users to test your designs
The secret? Sugar and Swag. Here’s what we did:
Rather than make it a burden on the customer, we turned user testing into a benefit. I didn’t have much of a budget, so I had to figure out how to incentivize people to test the prototype. We couldn’t afford to hand out gift cards, especially because people don’t value cards under $10, so we chose something most people enjoy — donuts.
We called up our local customers and asked if we could come by and set up a table in their lunchroom. The table would have 75 free donuts, a few t-shirts, and a cool prototype anyone could test if they wanted to. That was it, we promised to be set up and out of the office in less than 2 hours. Once we offered free donuts for their coworkers, our contact at the company got excited (because they would be the one who facilitated free donuts) and made it happen.
It was simple. As people walked by we asked if they wanted a donut — most people wanted a donut. We then said they could have one if they spent 30 seconds and tested our prototype — most people considered 30 seconds a fine compromise for a donut.
In that first swarm testing session, we had around 80 people test the prototype and provide input. We then went back to the office, spent a few days iterating the prototype from what we learned, and did it again at another office. We did this about 6 times with design iterations after each session.
Swarm testing was incredibly efficient and effective — but isn’t right for every product.
Do this before using this method
- Make sure this method makes sense for your product first. We were building an interactive kiosk for the office, and the swarm testing method was a perfect fit.
- Prepare a strategy for processing a lot of users in a short amount of time with the space you’ll have. You want to avoid congestion and long queues.
- Prepare primers, questions, and tasks for testing your prototype. Primers are your assumptions and unknowns. Then put questions together that specifically shed light on your primers. Tasks are what you want the user to try and do with your prototype to see if the product is intuitive enough.
Swarm testing, as we now call it, came out of the need to connect with hard to reach users. I was surprised at how effective it was, but that’s the joy of a startup —we came up with the idea and knew within a week if we were on to something. Morgan Williams, who led product design for this did a write-up on this same experience; it’s a better read IMO.
-dal
we did this back in Q3 2018
Thanks for reading! If you found my ramblings helpful go ahead and give it a few 👏 so others can find it.
This story is published in The Startup, Medium’s largest entrepreneurship publication followed by +426,678 people.
Subscribe to receive our top stories here.
