Evaluation Methods
& why we need them
I have been doing research to help identify possible thesis topics, and focusing my search around the methodology of evaluating the effectiveness of a proposed implementation. I am very eager to begin my Doctoral work and help to bring change to the field of User Experience and Human Computer Interaction. I have many years of practical experience and a passion to research and mentor. I see Doctoral research and education as a way to push myself, my peers, and the industry to better the processes we have, define new strategies, and help to educate the next generation of UX practitioners. It is often said that “the legacy we leave behind is all we have in the end”, to this I say no… that is simply not true. We can leave behind so much more then a legacy or a memory. We are able to impart ourselves in society and work for positive change, not seeking a legacy, or a name in a book. By helping to shape the world around us we cause a wave in time, nameless and silent it’s cause and effect will shape the future of mankind.
I previously covered the need to understand how to quantify if a solution was successful, in the proposal for a solution.
There are many various methods, they differ greatly in academic setting vs. corporate implementation. One thing is clear there are no standards or concise methods to justify the user experience process. As a designer or even architect it is important to be able to measure the effectiveness of your product. As the UX field continues to grow and progress, it is of growing importance to validate the process used. The need has been defined and is evident by the growth, the only way to stand out or shape the field from here is to be able to provide more concise and accurate products.
Looking into the problem
Looking into current research or availability for methods and their practices I quickly found that much of the research is limited or outdated. With the increasing speed and growth for not only digital interactions, but technology, a more expansive user experiences is emerging. Limited methods that use very simple interactions or usability testing as a basis, limits the capabilities for design phases. I found a great academic paper User Experience Evaluation Methods in Academic and Industrial Context, that reviews 30 evaluation methods and their relevance. It does stop short of covering the most effective or the most accurate, it doesn’t review the effectiveness of the possible methods.
What is the difference between the purpose of usability and user experience evaluation methods?, has a straight forward overview of the methods and their implementations. The simple table format brings to light the possible evaluation methods and the best use/outlook for the applicable product.
Moving forward
There are countless publications relating to UX design and process concepts and methods, however, many of them cover the concepts of design and the principles for user specific needs that can be addressed through design elements.
Visual communication through interactions, psychological understand of human intuition and natural interactions, Human Computer Interaction, Psychology of play, user flow, behavior analysis. All of these are the building blocks for UX, it’s the reason so many people with various backgrounds are able to be so successful as a UX designer. UX design at the core is being able to understand people, and empathize with their perspective. This also creates a new challenge of being able to justify the methods used, solidify an approach by validating the process that was implemented. Much like software development, each process will be a little different and each approach newer and expanded from the one before it. We can ensure that solid software development takes place by the effectiveness of the program, debugging and ensuring it functions as intended with the optimum performance.
The biggest shortcoming with the UX approach is the process is being proposed above the implementation understanding of effects on the user. As more and more people see the booming field and bring their diverse backgrounds it is difficult to merge the various experiences to a unified process. If any user experience designer was to look into the field of user experience as a process, it would be not only a challenge to rectify the process and understand the uses. It would be a difficult challenge to implement change. There are no unified avenues of education or solidified processes.
This is the key point I propose look further into, how do we bring unification to the UX/UI evaluation processes that are used for academic and industrial applications. Understanding how we define the effectiveness and being able to evaluate a specific implementation will help to identify the origin of the process and shape how we approach not only a problem but validate the solution.
Current processes
As I previously stated there are many processes for validation. Looking deeper into the current subsets of processes and evaluation methods there is a common thread to them, a unifying principle. “What do users think of the product?”. By looking into user reactions and surveying people mass amounts of data comes in then the work to digest the info to make sense and verify a proposition begins.
By taking the approach of design, ask, refine, it turns into a search for validation. The process needs to be a proposal, design to meet a set objective, validate, refine.
Here are the most common current processes for evaluation.
- Lab studies with individuals
- Lab studies with groups
- Field studies (short, e.g. observation)
- Field studies (longitudinal)
- Surveys (e,g. online)
- Expert evaluations
- Mixed methods
Though these are very broad and their use is not logically defined to a specific formula or use. They leave a foundational framework that is ready to be built into a process that can be analyzed and conformed to a proven method that can fit academic and professional settings alike.
In conclusion
Looking to the future I plan to push myself and continue to research and analyzing my process, as well as those with the teams I work with, to better understand the current applications of evaluation methods and needs for improved methods.
Continuing in my academic pursuit, I will look deeper into the world of analytical analysis, as well as continue to push my understanding of the psychology that UX is built on.
