In-Class Testing & Feedback, 04/23/2019
Part One & Two — Describe the Testing Sessions and Outcomes:
Today, we did our in-class test to get feedback from peers and test if our design of learning experience would be effective to support patients. To makes sure the participants better understand our focus area and patients type we’re targeting at, we first set up the scene and background information for the participants to get a general understanding of the disease:
Scene and background:
“Inflammatory Bowel Disease is a chronic illness that causes severe abdominal pain and currently has no cure. One of the ways to manage the symptoms is to adapt to a recommended diet. However, an ideal diet differs for everyone and for each stage, the recommendations can also be different. Therefore, we aim to teach the patient the basic principles of what to eat/avoid during flare/remission and also how to maintain a balanced meal.”
We then handed the participants a paper acting as a doctor’s note and asked them to imagine they are newly diagnosed and are supposed to follow these directions. We explained this experience would be on a website shown to patients on an iPad during in-patient and they will have a physical diary printed out by TC or nurse to bring with them at discharge.
The doctor’s instruction is the starting point of the learning experience. This is also the information the patients will input in the websites while onboarding. After the brief introduction, we showed the participants the on-boarding screen for the learning website.
Onboarding:
Phase I — Learning principles:
After the onboarding and patients input doctor’s instruction phase, the learning principles phase starts. Elva and I prepared several different activities for testing purpose, hoping to get more feedback and suggestions on what activities are more effective for learning. We presented four exercises/interactions and got feedback from four pairs of participants. The feedbacks were very helpful!
Exercise 1. — See a list of food, when hovering over, the food card (detail) will show up and patients see the tags, as well as where it lies on the safe/trigger scale.
Feedback:
- Like how this is personalized to me (“safe or not safe for ME to eat”) and directly relatable to themselves.
- This is more like a tool than a learning experience because you are just told the answer directly, but if you use it over a period of time, that could encode into your memory.
- The scale is useful, it can also be gamified to be more interactive and help with learning.
- Currently, the safe/trigger evaluation is on individual types of food, in real life, meals are probably more complicated and composed of several different foods listed in our cards (e.g. a hamburger has bread, ham, cheese, onion, tomato, etc in it), is there a way to see if what a combination of food would lie on the scale?
- This could be a handy tool, like an app, to check for food choice safe/not on the go.
Exercise 2. — Sort food by a given nutrition content (e.g. sort food by fiber, sort food by protein; sort food by fat)
Patients will be presented a list of food with the same type, e.g. see below — a list of fruits, and will be asked them to sort the fruits by the content of fiber. The interaction is when patients click the fruit card, it’ll flip and the back side of the card shows all detailed numbers of the fruit content. (e.g. 30% fiber, 20% protein, 20% fat, 30% carb)
In order to test the learning, we actually made a paper version of these cards to test the interaction, with the purpose of observing the participants’ behaviors, and see how they react to learning in this way.
Observation:
- None of the pairs get the sorting almost right without looking at the number on the back.
- Most of the people like to guess first, then flip over the cards to check the number to adjust the ranking; only one person does not want to guess at all and looked at the number from the start, then ranking based on numbers.
- People said “wow”, “I didn’t know that”, “who would have thought that” a few times throughout, which indicated surprise. When we asked if the surprising part would help with learning, participants responded positively and indicating this would make them more engaged, as well as help them memorize better. (I’ve actually learned another learning theory in another class, which is called ‘Invent-and-tell’- the idea is to have learners try on their own first and then show explanation and tell them the correct answers and why. This method works best for teaching conceptual level, which I think works well in our case)
Feedback:
- This game is interesting and engaging, attracts attention (cool activity)
- When there is an unexpected nutritional fact, that’s beneficial for memorizing. (e.g. coconut actually have the highest fiber, banana is low fiber)
- It would be better if there is a line between low and high fiber, currently, it is laid out low to high but you don’t know how low/high causes problem
- The surprise of finding out something is counter-intuitive is beneficial for memory encoding
- Once arranged, can it have all the numbers shown above, so that it is more clear which are actually high vs low (there is a lot of space between 2.8 and 6.5 which are the middling choices)
Phase II-Learning by doing (planning meals):
Exercise 3 — Plan meals with a calorie goal and balanced nutrition, while avoiding trigger food
Feedback & Questions:
- Question: is this like planning a real meal or just simulation?
- This may be useful at the moment in the hospital, but how would people practice it afterward? Can this be an app to use in a restaurant when ordering or cooking so that is more accessible?
- Question: is this one meal or one day? Doing it by meal will make more sense.
- This is gamified, participants liked it.
- How does it show the nutrition balance besides calories? Can I just add all meat onto the plate and hit the calorie goal? Will there be a chart of nutrition after planning three meals for a day?
- There could be more emotional feedback besides calorie go up like abdominal pain level rises if choose the wrong food
- It can be made more challenging by telling feedback at the end and has to go back and find out which food is wrong
- Would want this to be an app, so that the interactions are more fluent.
- In regard to the feedback of picking trigger food — instead of saying “Wrong”, it’s better to go to a section that explains why this may be a trigger food, which could help them revisit the principle, so that can also apply what they learned from this experience to other scenarios.
Exercise 4. (More advanced)-Use comprehensive principles to judge if a food is okay to eat/try to avoid during a flare.
In this activity, we provided the participants with more food (real-life food, e.g. sandwiches, pizza, etc), which may consist of different ingredient and have participants judge/analysis if the food is safe or not.
This activity does not currently have a digital version, the participants are supposed to judge if some common foods are okay to eat during a flare based on the principles given in the doctor’s note.
Observation:
- majority of the choices was obvious, but people are not sure about waffle, sushi, coffee, etc.
- Some food has different properties that confuse people (both high protein but also high fiber).
- The uncertainty triggered thinking and discussion.
Feedback:
- This is an interesting activity, but maybe another format so that it does not feel like another sorting game.
- This is more comprehensive and less scaffolding, the time of introducing this activity can be towards the end of learning during in-patient.
Besides these activities, we also showed the participant the draft for the food diary. We have two versions of the diary.
Phase III — Keeping diary (post discharge)
The nurse will have the print ready for the patients, based on their learning and their personalized input during the learning. The prints include safe food list, trigger food list, a diary book.
Diary Version 1
Feedback:
- maybe do not need so much
- the good/bad is confusing, where am I supposed to write about what
- why “where I went”
Version 2
Feedback:
- Most participants commented the design is pretty, would want to use it.
- Participants also mentioned that the layout is pretty straight forward but would be nice if there’s a bit more instruction on how to keep the diary at discharge. (either at the end of the learning experience or through nurse’s instruction)
- There was one participant did say that probably would not use these journals, if she wants a reference, she’ll just go back to the website.
- One good point was one of the participants mentioned that having a physical journal may be embarrassing to take out and write, especially in public, or in a restaurant. It’s probably more appropriate to take notes on the phone given it’s more.
- Young people would like an app better, but older people may prefer paper.
Part Three-How the Outcomes inform Our Next Step
Today’s testing session was really fruitful for us as we gathered so many good feedbacks and suggestion, while meantime participants responded very positively to our solution. To move forward, the outcomes informed our next steps in the following aspects:
- Consolidate the design of the activities. For our designs of several different activities, each of them has their strength while also has drawbacks, we might want to think about how to consolidate these several activities. It’s very likely for us to have multiple activities and patients do different activities on different days so they would feel less bored.
- Our current learning experience/flow seems pretty fluent and makes sense! Moving forward, we will keep optimizing the learning experience to make the whole process more cohesive.
- We will discuss and make a matrix to compare the pros and cons of several different forms to implement our solution. We could also show that to UPMC and see the value of each one.
Part Four-Outstanding Questions
- Should we pivot and focus more on physical products rather than digital products?
- In our final pitch, can we pitch to UPMC that we can implement this in different forms, and see which way they think could be more cost-effective? Our solution could easily be switched between website/app/physical games.
