The Most Ambitious UX Project I Ever Worked On — Before I Became a Designer
(This case study is a part of my upcoming UX portfolio. Any comments and constructive criticism are more than welcome!)
This story happens half way around the world, in the city of Belgrade, Serbia. At the time I was still pursuing psychology full time, and design was just something I did on the side.
I still remember thinking: “I never shy away from a challenge”, as I proudly presented the idea to my mentor. I was already two years into my Master’s Degree program at University of Belgrade’s Department of Psychology, and it was time for me to start working on the research project, the final, and the most crucial step in obtaining the degree.
Always being an overachiever, I was determined to do something meaningful with it. I was going to make a difference!
Through my undergrad studies and my work as a Teacher’s Associate in our Department, I was intimately familiar with the tests used to assess cognitive abilities of school children in Serbia. Most of the tests were old, some dating back as far as the 1950s. We were desperately in need of modern assessment tools.
When I learned about Kaufman’s Assessment Battery for Children, Second Edition (KABC™-II), a highly acclaimed new clinical instrument developed and used in the U.S., I knew I found just what I was looking for.
I decided I was going to be the one to bring this instrument to Serbia.
This challenge, however, proved to be slightly more difficult than my overly enthusiastic self had first imagined. The research that followed became the most challenging and the most complex project I have ever worked on.
To accomplish, what at many times seemed impossible, I had to sharpen my skills and develop new ways of thinking.
I’d like the share these lessons and show how they continue to be a source of invaluable help for my UX design career.
What I learned:
1. Setting the Right Goals
How to convert impossible goals into a minimum viable product.
2. User-Centered Design
How to apply iterative user-centered methodology.
3. Problem Solving
How to identify and efficiently solve problems.
4. User Testing
How to lead a team to successfully test over 100 children.
5. Data Analysis
How to analyze multiple data sets and draw correct conclusions.
6. Don’t Give Up
How to push-on through frustration and finish what I started.
1. SETTING THE RIGHT GOALS
Bringing a clinical instrument from one country to another is a long and complex process. It’s basically done in two steps:
- Adaptation of the instrument to be linguistically and culturally appropriate for the new country.
- Standardization of the adapted instrument, by administering it to a representative sample of the population (usually involving thousands of people), and using the acquired data to define a standardized scoring system against which all subsequent testing in the new population would be compared.
All of this is normally done by a team of psychologists. Yet, due to the requirements of my program, I had to conduct most of the work by myself.
Obviously, I had to revise my initial goals to something a little bit more attainable.
I started by making a list of my research goals and the necessary steps to achieving them. I dissected KABC-II to come up with a list of all the features and elements that needed to be adapted.
Through careful analysis (and a lot of head bashing) I placed these items in an “importance vs. feasibility” matrix (although I didn’t label it as such back then), and came up with a ranked list of goals and features I absolutely had to focus on in order to produce a valuable research project.
Through this meticulous process I ended up doing the following:
- I limited the goals of the research — Instead of coming up with a fully fledged, ready-to-use adapted version of KABC-II (which would have been impossible for me to do), I decided to create a prototype which I could test on a large enough sample of subjects (dozens instead of thousands). By analyzing the results, I would then come up with recommendations for further development, thus creating a solid foundation for other psychologists to build on.
- I narrowed down the breadth of KABC-II —Kaufman’s battery is made up of 18 individual tests. Through careful analysis and planning, I was able to narrow down that number to 10 tests to use in my prototype.
By shifting my goals to creating a prototype others could build upon, I was able to minimize the amount of work I had to do, while still producing a meaningful research.
Due to sheer necessity, I ended up creating a minimum viable product, years before I would learn about it in Dan Olsen’s amazing book.
2. USER-CENTERED DESIGN
Research in psychology is governed by a similar methodology to that of the User-Centered Design.
The specifics may be different, but the iterative process of discovering problems, formulating possible solutions and testing their impact is common to both. Not surprising, since both methodologies draw on what is know as the “scientific method”, a body of techniques and processes that’s been around since the time of Galileo (perhaps much longer than that).
We learned a lot about research methodology in my undergrad studies, and had a chance to put it to practical use in several research projects. However, it wasn’t until my Master’s program that I really got to sharpen my skills in using it.
On a global level, I used the iterative process to organize my research into four broad phases:
- Adapting the KABC-II prototype.
- Testing the adapted prototype on a large sample of school children.
- Statistical analysis of the collected data and determining the success of the adaptation process.
- Writing the research paper to detail the process and give recommendations for further improvements and development.
I also relied on the iterative research process to guide the translation and modification of 10 of the original KABC-II tests I used in my prototype. The process looked like this:
- Adaptation of a test to fit Serbian population.
- Guerrilla-testing the adapted version on a small number of children (1–5).
- Analyzing the collected data.
- Improving the adaptation based on these findings.
- Rinsing and repeating until arriving at a satisfactory version for testing on a large sample of children.
This experience taught me to always test my solutions with real users, as fast as possible .
I learned that I was definitely *not* my users (something that is arguably much easier to grasp in psychology than in design), and the sooner I can include them in my process, the better my product is going to be.
3. PROBLEM SOLVING
I think the ability to efficiently solve problems is one the core attributes of a great designer.
Even though my Master’s research wasn’t in the field of design, it forced me to solve some of the most challenging problems I ever encountered.
I already mentioned some of them, like finding a way to narrow down the scope of my research. Most problems, however, cropped up during the adaptation process of KABC-II.
To help you understand why, let me take a moment to explain what it means to “adapt” a clinical instrument, and why it is necessary.
In psychology, every assessment tool is loaded with inherent linguistic and cultural biases (even the so-called “culture-free” ones like KABC-II). This makes an instrument suitable for the culture in which it was created, but not for any other — not without adapting it first. This process usually includes translation, but goes much further than that, to encompass any and all changes necessary to maximize the cultural fit of the original instrument for the new culture.
The problem lies is in knowing what to change, and what not to. With 10 distinct tests to adapt, it was like solving one giant puzzle after another.
One test in particular became the bane of my existence for weeks (months?), pushing the upper limits of my deductive reasoning (and my frustration tolerance).
It relied heavily on language structure and grammar to test a child’s cognitive functioning. The problem is Serbian grammar is much more complex than English, and some core concepts of English language, like definite / indefinite articles, don’t exist in Serbian.
The effects of language barrier were so severe, that after many iterations of modifying and guerilla-testing with a small group of kids, I ended up basically recreating the whole test.
This experience sharpened my skills at being able to identify a problem, understand its impact on the target users and most importantly — to ask questions.
The nature of the challenges, the consequences of failing to solve them, and the tiring trial-and-errors, drilled into my head the need to truly understand the problem space before leaping into solutions.
(Raising my frustration tolerance level was an added bonus.)
4. USER TESTING
I conducted the standardized testing of the adapted KABC-II prototype in three elementary schools in Belgrade, over a period of three months.
I lead a team of five psychology students, who helped test the children and collect additional information from the teachers. Together, we administered the prototype to over 100 students, ages 6 through 9.
I ended up personally conducting around 60 test sessions, including those I performed during the adaptation cycle.
It had been a valuable learning experience. Although testing in psychology and in UX research have a fundamentally different focus (testing a person vs. testing a product), the same basic principles apply to both.
My research really helped me to master these principles, but it also did wonders for my confidence level. Before this research, I always dreaded having to test somebody, but after working with 60 or so kids, I developed a very thick skin.
5. DATA ANALYSIS
The process of analyzing the information I collected during the standardized testing was the most complex data analysis I ever performed.
It consisted of multiple sets of data based on various sources, including the achievement of children on the adapted prototype and observations made by the test administrators during testing.
Based on the painstaking statistical analysis, I determined the following:
- My adaptation of KABC-II prototype was successful in terms of design and format of the adapted testing material, and in terms of fostering a consistent, well established testing procedure.
- My efforts of adapting the 10 individual tests were mostly successful, with only two tests out of 10 needing serious revisions.
I’ve always been very detail-oriented, but this research forced me to “up my game” when it comes to being meticulous about collecting and analyzing data. I learned the hard way the value of being patient and double checking everything.
6. DON’T GIVE UP— THE MOST IMPORTANT LESSON I LEARNED
I started this journey with desire to challenge myself. I wanted to make a difference, do something meaningful.
I came up with an awesome idea for my research and I was so pumped up.
I was going to be the one to bring a shiny new clinical instrument to my home country. I was going to propel our cognitive assessment into the 21st century! I already saw my name in the psychology textbooks! I was ready to take on the world…
…Five months later, I was still working on adapting the individual KABC-II tests. I just finished another disappointing testing session and was facing yet another round of changes. All of the initial euphoria had long faded away; the dreamy vision of my academic future was gone, replaced by the bleakness of frustration and anxiety.
I was never one to quit before finishing what I started, but this time I was ready to give up. I was sick of endless work and wanted to walk away from it.
Yet, I was too invested. My wife, my family and friends, all gave me their unwavering support despite my lack of progress, as did my mentors. I knew giving up would mean letting them all down. It would mean letting myself down. Plus, I’d lose any hope of having an academic career in psychology.
So I forced myself to push on through the frustration. I decided to take on one challenge at a time, and see it through to resolution.
It was difficult, but eventually I started making progress, little by little. I started getting closer to finishing the research. Then after all the testing and data analysis was done, I still had to write my research paper — a monumental task in itself. It took me a while, but I finished that as well.
“Aim high, push-on through the lows” — this lesson was the most difficult one to learn, but proved to be the most valuable one. I reaped its benefits time and again, both in my design career and in my personal life.
Because I didn’t give up, I attained my Master’s Degree, with a mark of distinction. I don’t know if my name ever went into any textbook, but I’m happy I made a contribution. It also didn’t hurt my ego to see my research mentioned on national TV. :)
Even though, in the years that followed, I moved on from psychology and dedicated myself fully to design, the lessons I learned and the experiences I gained from my psychology studies and my Master’s research have continuously been my ever-present allies. And since I started my career in UX design, they’ve become more relevant then ever.