Quantifying the qualitative data.
This is the fifth post in our series of learnings from the wearable. I thought it would be good to talk a bit about our testing methodology and take a look at the validity of our data set.
Reigning in the Qualitative
Our data gathering on this project included audience observation, post-test surveying, and post-test interviews. One of the most fascinating things we did on the project was how we worked to quantify the qualitative data that was coming in through tester interviews.
Three quarters of the way through this process, we had 40 pages of interviews in google docs — transcriptions of the onsite interviews that followed a person’s testing. This was an incredible amount of information and I read every word, but I noticed my personal bias would get in the way and I’d naturally gravitate to those comments that represented by own view of product — something that has to be avoided.
Parsing interviews into meaningful and accurate data was also difficult because testers would often give us mixed messages in what they’d say about something. “I like X, but if it were more like Y then I’d really like it.” If not mixed messages, some statements would contain information about different features in the same statement, “I like this, and this, and this, but not this thing” — counting statements (or parts of statements) would be difficult and likely misrepresent the data.
Interview data was, also, polarizing with large portions of users on one side of the divide or the other — we had a lot of opinion without the ability to see a critical mass. We needed a way to parse interviews into quantitatively meaningful information, so we went into the interviews and pulled some very honest and very opposing quotes from users — we specifically looked for ones that had no gray area. We then published those quotes in a new survey and started asking people which statement they agreed with the most; essentially we took our most polarizing opinions on our most difficult challenges and made our audience pick a side.
To say this was incredibly useful was an understatement. This gave us a quantitative view based on the qualitative data and while one can’t replace the other, the two together are very powerful and provided greater clarity.
Validity
The wearable is being tested over the course of three months, January through March. During this time, we will have had roughly 49,189 unique/individual visitors to the Barnes, with March figures being projected at this stage. Of these, a random sample of 400+ visitors will be testing the wearable, taking the wearable with them during the course of their visit. Upon return, they answered a survey first, and then we interviewed them about their experience.
As a result our sample size was 0.81% of the general audience. Using a sample size calculator, we are 95% confident that the true findings will fall within +/- 5% of the reported results.
It should be noted that while most of our sample received the same survey questions, there were some instances where questions were tweaked or added based on the particular feature we were testing at that time. This means, in select cases, the sample size was much smaller and the margin of error larger. Nevertheless, in all of these cases, there was a clear trend in the data before we moved to a new test.
This represents the balance of testing in an iterative project — sample sizes and confidence ratings are important, but they need to be balanced with the practicality of project testing needs. Still, this is the largest testing sample I’ve ever worked with and having a front lines team getting this into the hands of as many visitors as possible has been critical for the project findings.
The Barnes Foundation wearable digital prototype is funded by the Barra Foundation as part of their Catalyst Fund.
Want more info? Read more about the Barnes Wearable on Medium and follow the Barnes Foundation publication, where we’ve got multiple authors writing about our projects.
