Your UX research is wrong (and I just found out mine is too…)

In a recent project, I followed the general steps

1. Competitive Analysis
2. Design new and better product
3. Complete UX Research to Validate Designs

But when presenting my findings to the team, I was told my UX Research was wrong.

Let me explain it a bit more

I’m currently a Product Designer at iContribute. We are designing and developing a WebApp to allow Organizations to Post Events — unfortunately, we have a major time constraints.

The main portion of the WebApp is a form, the form organizations need to fill out with all the event information. In a perfect world, we would have more time and be able to do more research, but as all designers know — this is not a perfect world. Oh yeah, one more constraint — this is a volunteer position a.k.a. no research budget.

The Nitty Gritty

Let me explain what studies I completed, then I’ll get into where it went wrong. The two main studies I conducted were

1. Open Card Sorting —to make sure the way we were grouping information on the “create event” form made sense
2. Tree Testing — to make sure the Site Map was logical to the user

I used UXtweak to conduct these studies. Its a free platform that makes it super simple to set-up studies (not-spons — there are a ton of free platforms like this all with benefits and downfalls)

Open Card Sorting

For those who don’t know, this is a study where participants are given several cards (think post its with words written on them) and are asked to group them and title those groups. For this, I used all the information we are asking from the organization to create an event.

All the cards in my card sorting study

After all the participants complete the study, UXtweak then spits out data analysis on how these answers were similar — aka a similarity matrix.

I took the results and analyzed them. I was looking for how groups of questions related — the darker the box between the questions, the more frequently users matched those questions in the same group.

Before I go on to explain why I was told this is all wrong, let me go through the next study (I’ll be quick- I promise.)

Tree Testing

So first you find an old oak tree.. just kidding — bad dad joke.

Tree Testing is essentially testing your Site Map on participants to evaluate the hierarchy according to how it performs in a real-world scenario — will links lead to where users think they will, do pages have the information that users expect, etc.

I was testing my Site Map design that was derived from the competitor analysis. It was simple like the MVP — The landing page was a Dashboard that was home base for three pages — New Events, Event Details, and Setting/ Account Information.

When I tested this design with participants, the results were very positive — 95.2% of participants successfully ended on the correct answer and 81.0% of participants found the correct answer without having to backtrack.

With all this data validating my designs, I thought fore-sure my team would feel confident to move forward. The designs were based on research, and had data to back them up.

Why I was told the Research was Insignificant

So like, my background is engineering. My formal education is in engineering. I took a statistics class in University — and I did well.

So why is this data unreliable, well

When the study’s aim is to measure the frequency of something or to describe people’s behavior, we recommend following the calculations made for probability sampling. This often translates to a sample of about 1,000 to 2,000 people. When a study’s aim is to investigate a correlational relationship, however, we recommend sampling between 500 and 1,000 people.

a.k.a the site I was using only allowed for 10 participants which makes my data not statistically significant. But a lot of people use these free sites for guerrilla testing, there must be some way I can prove that my findings are valuable — and valid enough to base designs on.

Why I wont throw out my Research

While mathematically, my data was not statistically significant because I was limited to 10 participants on a free testing site, there must be some validity to my over 90% success rate. Here is why I am moving forward with my results and trusting my data without statistical significance.

Statistical significance testing is a great tool to validate tests and analyses, but it doesn’t mean that your data is accurate or unbiased. Participants can lie and give you incorrect information, and your surveys can be biased by a non-uniform representation of certain demographics.

Remember, statistical significance is a great tool to make decisions with increased confidence, but it is not a mathematical silver bullet.