Unmoderated Testing with Validately Part 2

This is a follow-up to my previous post on Validately (https://medium.com/@TheRealTang/unmoderated-testing-with-validately-part-1-563be74486f5#.jvkxgtyx0 ). I have had a chance this week to go through analyzing data from a sample test I created for Grubhub. I had created 3 tasks and questions focusing on finding and ordering food for delivery. Immediately, the tool wins in helping the researcher compile the results for the survey questions. This helps immensely by reducing the need to review the answers from each individual user (with the exception of free text answers). All of this can be easily exported into a CSV file that can be analyzed further and integrated into your final report. Another interesting feature is being able to see the paths users took or the page exits. This may be able to help manage the exceptions if you have a high volume of testers. Lastly, there is a “Findings” feature that allowed the reviewer to capture clips which helps in bookmarking examples or making a highlight reel.

Despite the usefulness of the tool, I found a couple of things challenging in general for unmoderated tests. The users helping with the testing must inherently be comfortable with thinking aloud as I could not remind them in person (as its unmoderated). One way you can overcome this is to add that reminder in each task before the user begins. Another issue is that I couldn’t tell if the users encountered a technical difficulty until the person has already submitted the test. Obviously this some to question whether to keep an uncompleted test for the task that were complete or to scrap this tester session all together. This is also leads to ensuring that there should be a pilot done. In my first try, I did not have a pilot done. I found that some questions could be better worded and that I missed a critical task. I decided to re-create my user testing again (so I guess my first try was my pilot). Overall, I would definitely use this tool again. I will try to see if this tool will work on applications that live inside the corporate firewalls.

There is a great talk done by Kyle Soucy on unmoderated remote usability testing (http://www.slideshare.net/usableinterface/unmoderated-remote-usability-testing-good-or-evil-4446847 ) that points out the benefits and issues with this style testing. Please make sure to check it out if you have not.