How to run a lean product testing session in your community

User experience is key. To achieve great UX, you have to understand how your users experience your product.

Being a member of the local community of startups and young companies we want to help others to be close and connected to their users and empower them to create great products. Thus, we share our mindset of user-centricity and proper user research. For this, we encourage them to get in touch with users and provide proper tools and best practices to enable them to continue their journey of getting to know their users on their own.

Gain valuable insights from testing your product in your community.

We — as the team behind consider.ly — ourselves took part in many sessions of testing communities (such as usability-testessen.org). It always was a blast and we got away with tons of new valuable input for our products.

Nevertheless, for our taste, the frequency of these events was too low, often booked out early, and generally, startups struggled to participate for the first time (“We’re not ready yet.”, “We don’t have something to show.”, etc.).

“We’re not ready to show our product yet.” — Mostly always wrong.

Being persuaded of the concept, we started our own testing sessions for the community. As organizers, we aimed at a moderate effort to organize, having non-professionals as testers, and developing a setup that can easily be held monthly. And our community loves it so far.

So how do we do it? In the following, we give you a detailed description of our setup, so that you can implement it in your community yourself.

Lean testing session vs. lab approach

Our goal with the lean testing sessions is to give a small but valuable glimpse of the benefits of a regular lab testing session. For sure, an event like this will not replace or try to reproduce the depth of a typical lab usability test. Still, we highly value the quick wins and accept certain tradeoffs:

• general input over specific target group testers
• multiple participants over long single sessions
• simple setup over a dedicated environment

With this in mind, we organize sessions where multiple products from different companies are tested during a single event.

The lean concept for a product testing session

We brainstormed what would be the bare minimum of requirements of a testing session which provides benefits for someone who wants to get a product tested. What are the common requirements which meet the testing needs of all interested companies and products? Here’s what we came up with:

• 12 (+3) minutes of testing rounds
  One testing round takes 12 minutes followed by 3 minutes of re-setup and documentation.
• 5 (different) testers at one station
  We stick to Nielsen’s rule of thumb that says that five testers find 85% of the key usability problems (which is the primary goal here instead of learning about your exact target group).
• 1.5 hours in total
  The total duration of a session sums up to 1.5h (5 slots of 15min plus 15min of setup and administration).
• Minimum of 6 stations
  Upon invite, we ask every station to bring a tester to the event. Those testers often already know their product and thus are left out at their station. To still ensure 5 testers at one station we need a minimum of 6 stations. Then no one will have an empty slot as a tester, as a station, or visit one station twice. Due to the fact that each station brings a tester, it’s easy to handle cancellations (if the count of remaining stations is 6 or greater).
• One location
  This event can be held at every location where you can set up small stations. So a bigger conference room or multiple connected rooms can be easily used.

Preparation and conduct cheatsheet

Organizers

Preparation

• organize or book the venue
• find 6+ stations to attend

On the event

• organize the setup
• welcome everyone and give a short introduction
• ring the bell for every testing round
• order pizza

Stations

Preparation

• decide on which product (prototype) to test
• prepare a short verbal intro and 2–3 tasks to solve
• find someone who will attend as a tester

On the event

• run the product testing interview
• finish in time
• gather and document valuable insights

Testers

On the event

• on the end of a timeslot continue to the next station
• try to solve the given tasks and articulate all observations, expectations, and experiences

The usual setup for our lean testing sessions.

The setup at the venue

Here is the breakdown of the setup for our lean product testing session:

A) Location and equipment

• 3 chairs and 1 table per station
• Internet via wireless LAN
• Each station brings its own devices for its products (usually a laptop or smartphone)
• Printed-out station numbers on DIN A4 sheets. 1,2,3,4, …
• A bell to announce the end of the slots

Note: In our experience, there is no need for a beamer or screen.

B) Stations and testers

• Stations set up their desk on their own
• The station number should be visible when entering or walking around the room
• The stations should be placed in order so that the next station (next number) is easy to find for the participants
• After a 12 min testing slot is finished the testers move on to the next station (beginning at 1 if the last station is reached) with a break of 3 minutes

C) Food and beverages

This point is optional — but a good thing to foster intense networking after the session. Our solution is to provide crates of water, juice, and beer as well as ordering pizza at a delivery service. In our experience, this results in 6€ to 8€ per participant and overall costs of 200€ to 300€ per session, which is usually paid for by one of the participating companies or the location provider.

Pizza and cold beverages are always very welcome on testing sessions.

Best practices for organizers

• Prepare a few sentences for opening the session.
• Order pizza at the start of the session with a dedicated arrival time (end of the session).
• Create a mailing list with all interested and participating teams.
• Use the same location each time — so you know the rooms, available furniture, and general infrastructure. Kudos to our local startup network and incubator CyberLab which lets us run our testing sessions in their premises!

The location should provide enough space so that everyone can arrange their testing station.

Feedback about our lean testing sessions

Since our first sessions, we are overwhelmed by the positive feedback from stations and testers about the learnings and live experienced interaction of users with their products.

Some younger companies that we had to convince to participate for the first time immediately saw the benefits and are now promoters and regular attendees. For us, this is a huge success and motivates us to continue organizing lean product testing sessions.

Closing thoughts

As a young startup, it is a great opportunity to provide something valuable to the community and enable first contact and experiences with usability and UX.

Especially the described lean approach allows a high testing frequency and limits the effort to organize, host, and moderate the sessions. We spread the word in our local community with this lean testing session — but we’d love to see more similar events and spread of the user-centric mindset.

Feel free to adapt those ideas and concepts within your community! And if you already have your first experiences: What are your thoughts and adaptions?

Keep Testing!

Yours,
Jonas

PS: As a startup that provides the UX insights repository consider.ly, we for sure recommend every station to properly document their findings. Because all new insights, observations, and new hypothesis could be crucial for their business.

This post first appeared on the consider.ly blog on August 12, 2019.